Showing posts with label WSL. Show all posts
Showing posts with label WSL. Show all posts

Friday, 12 July 2019

Digging into the WSL P9 File System

Windows 10 version 1903 is upon us, which gives me a good reason to go looking at what new features have been added I can find bugs in. As it's clear people seem to appreciate fluff rather than in-depth technical analysis I thought I'd provide a overview of my process I undertook to look at one new feature, the P9 file system added for the Windows Subsystem for Linux (WSL). The aim is to show my approach to analyzing a feature with the minimum amount of reverse engineering, ideally with no disassembly.

Background

When WSL was first introduced it had a pretty poor story for interoperability between the Linux instance and the host Windows environment. In the early versions the only, officially supported, way to interop was through DrvFS which allows you to mount local Windows drives into the Linux environment. This story has changed over time such as adding support to start Windows executables from Linux and better NTFS case-sensitivity support (which I blogged about already).

But one fairly large pain point remained, accessing Linux files from Windows applications. You could do it, the files are stored inside the distro's package directory (%LOCALAPPDATA%\Packages\DISTRO\LocalState\rootfs), so you could open them directly. However WSL relies on various tricks to deal with the mismatch between Windows and Unix-style filesystem semantics, such as storing the UID/GID and file permission bits in extended attributes. Modifying these files using an unenlightened Windows application could result in corruption of the file state which in the worse case could break the distro.

With the release of 1903 the WSL team (if such a thing exists) looks to be trying to solve this problem once and for all. This blog introduced the new feature, accessing Linux files via a UNC path. I felt this warranted at least a small amount of investigation to see how it works and whether there's any quick wins or low-hanging fruit.

Understanding the Feature

The first thing I needed was to setup a x64 version of 1903 in a Hyper-V VM. I then made the following changes, which I would always do regardless of what I end up using the VM for:
  • Disabled SecureBoot for the VM.
  • Enabled kernel debugging through BCDEDIT. Note that I tend to be paranoid enough to disable NICs in the VM (and my success of setting up alternative debug transports is mixed) so I resort to serial debugging over a named pipe. Note that for Gen 2 Hyper-V VMs you can't add a serial port from the UI, instead you need use the Set-VMComPort PowerShell cmdlet.
  • Install my tooling, such as NtObjectManager and SysInternals suite, especially Process Monitor.
  • Enabled the Windows Subsystem for Linux feature.
  • Install a distro of choice from the Windows Store. Debian is the most lightweight, but any will do for our purposes. Note that you don't need to login to the Store to get the distro, though the app will do its best to convince you otherwise. Don't listen to its lies.
With a VM in hand we can now start the investigation. The first thing I do is take any official information at face value and use that to narrow the scope. For example reading the official blog post I could determine the following:
  • The feature uses the Plan 9 Filesystem Protocol to access files.
  • The files are accessed via the UNC path \\WSL$\DISTRO but only when the distro is running.
  • The P9 server is hosted in the init process when the distro starts.
  • The P9 server uses UNIX sockets for communication.
Based on those observations the first thing I want to do is try and find how the UNC path is implemented. The rationale for starting at the UNC path is simple, that's the only externally observable feature described in the blog post. Everything else, such as the use of P9 or UNIX sockets could be incorrect. I'm not expecting the blog post to outright lie about the implementation, but there's sometimes more important details to get right than others. It's worth noting here that you should increase your skepticism of a feature's technical description the older the blog post is as things can and will change.

If we can find how the UNC path is implemented that should also lead us to whether P9 is used as well as what transport the feature is using. An important question is whether these files are really accessed via the UNC path, which would imply kernel support, or is it only in Explorer? This is important to allow us to track down where the implementation lies. For example it's possible that if the feature only works in Explorer it could be implemented as a shell extension, similar to how MTP/PTP is supported.

To determine whether its a kernel driver or a shell extension it's as simple as opening the UNC path using the lowest possible function, which in this case means calling a system call. Invoking a system call will also eliminate the chance the WSL UNC path is implemented using some new feature added to the Win32 APIs. As my NtObjectManager module directly calls the NtOpenFile system call we can use that to do the test. I ran the following PowerShell command to check on the result:

$f = Get-NtFile \??\UNC\wsl$\Debian\bin\bash

This command successfully opens the BASH executable file. This is a clear indication that we now need to look at the kernel to find the driver responsible for implementing the UNC path. This is commonly implemented by writing a Network Mini-Redirector which handles a lot of the setup with the Multiple UNC Provider (MUP) and the IO Manager.

At this point the assumption would be the mini-redirector would be implemented in the LXCORE system driver which implements the rest of WSL. However a quick check of the imports with the DUMPBIN tool, shows the driver doesn't import anything from RDBSS which would be crucial for the implementation of a mini-redirector. 

To find the actual driver name I'll go for the simplest, brute force approach, just list all drivers which import RDBSS and see if any are obvious candidates based on name. You could achieve this in one of many ways, for example you could implement a PE file parser and check the imports, you could script DUMPBIN, or you could just GREP (well FINDSTR) for RSBSS, which is what I'll do. I ran the following:

c:\> findstr /I /M rdbss c:\windows\system32\drivers\*.sys
c:\windows\system32\drivers\csc.sys
c:\windows\system32\drivers\mrxdav.sys
c:\windows\system32\drivers\mrxsmb.sys
c:\windows\system32\drivers\mrxsmb20.sys
c:\windows\system32\drivers\p9rdr.sys
c:\windows\system32\drivers\rdbss.sys
c:\windows\system32\drivers\rdpdr.sys

In the FINDSTR command I just list all drivers which contain the case insensitive string RDBSS and print out the filename only (unless you enjoy terminal beeps). The result of this process is a clear candidate P9RDR. This also likely confirms the use of the P9 protocol, though of course we should never jump the gun on this. 

We could throw the driver into a disassembler at this point and start RE, but I don't want to go there just yet. Instead, in the spirit of laziness I'll throw the driver into STRINGS and get out all printable debug string information, of which there's likely to be some. I typically use the SysInternals STRINGS rather than the BINUTILS one, just as I usually always have it installed on any test system and it handles Unicode and ANSI strings with no additional argument. Below is some of the output from the tool:

c:\> strings c:\Windows\system32\drivers\p9rdr.sys
...
\Device\P9Rdr
P9: Invalid buffer for P9RDR_ADD_CONNECTION_TARGET_INPUT.
P9: Invalid share name in P9RDR_ADD_CONNECTION_TARGET_INPUT.
P9: Invalid AF_UNIX path in P9RDR_ADD_CONNECTION_TARGET_INPUT.
P9: Invalid share name in P9RDR_REMOVE_CONNECTION_TARGET_INPUT.
...
\wsl$

We can see a few things here, firstly we can see the WSL$ prefix, this is a good indication that we're in the right place. Second we can see a device name which gives us a good indication that there's expected to be communication from user-mode to kernel mode to configure the device. And finally we can see the string "AF_UNIX" which ties in nicely with our expectation that Unix Sockets are being used.

One this which is missing from the STRINGS output is any indication of the Unix socket file name being used. Unix sockets can be used in an "abstract" fashion, however typically you access the socket through a file path on disk. It's most likely that a file is how the driver and communicates with the socket (I don't even know if Windows supports the "abstract" socket names). Therefore if it is indeed using a file it's not a fixed filename. The kernel has support for a socket library so again maybe this would be the place we could go disassembling, but instead we'll just do some dynamic analysis using PROCMON.

In order to open a socket from a file there must be some attempt to call the IO Manager to open it, this in turn would likely be detectable using PROCMON's filter driver. We can therefore make the following assumptions:

  • The file open can be detected in PROCMON.
  • The socket file will be opened in the context of the first process to open the UNC path.
  • The open request will have the P9RDR driver on the call stack.
The first assumption is a general problem with PROCMON. There are ways of opening files, such as inside another filter driver which cannot be detected by PROCMON as it never receives the request. However we'll assume that is can be detected, of course if we don't find it we might have to resort to disassembly or kernel debugging after all. 

The second assumption is based on the fact the WSL distribution isn't always running, therefore any Unix socket file would only be opened on demand, and for reasons of laziness is likely to be in the same process that first makes the request. It could push the request to a background thread, but it seems unlikely. By making this assumption we can filter PROCMON to only show open file requests from a known process.

The final assumption is there to filter down all possible open file requests to the ones we care about. As the driver is a mini-redirector the call chain is likely to be IO Manager to MUP to RDBSS to P9RDR to UNIX SOCKET. Therefore we only care about anything which goes through the driver of interest. This assumption is more important if assumption 2 is false as it might mean that we couldn't filter to a specific process, but we'll go with it anyway on the basis that it's useful technique to learn.

Based on the assumptions we can set PROCMON's filters for a specific process (we'll use PowerShell again) and filter for all CreateFile operations. The Windows kernel doesn't specifically differentiate between open and create calls (open is a specific case of create) so PROCMON doesn't either.

PROCMON Filter View showing filtering on powershell process name and CreateFile operation.

What about the call stack? As far as I can tell you can't filter on the call stack directly, instead we'll do something else. But first gather a trace of a PowerShell session where you execute the Get-NtFile command show earlier in this blog post. Now we want to save the trace as an XML file. Why an XML file? First, the XML format is easy to access, unlike the native PML format. However, the real answer is shown in the following screenshot.

PROCMON Save Dialog showing options for XML output including stack traces.

The screenshot shows the options for exporting to XML. It allows us to save the call stacks for all trace events. It will even resolve symbols, however as we're only interested in the module on the stack not the name we can select to include the stack trace, but not symbol resolving. With an exported trace we can now filter the calls based using a simple XPath expression. The following is a simple PowerShell script to run the XPath query.

$xml = [xml]$(Get-Content "LogFile.XML")
$xml.SelectNodes("//event[stack/frame[contains(path, 'p9rdr')]]/Path[text()]")

The script is pretty simple, if you "cast" a text file to an XML object (using [xml]) PowerShell will create an XML DOM Document from the text. With the Document object we can now call SelectNodes with an appropriate XPath. In this case we just want to select all Path of all events which have a stack trace frame containing the P9RDR module. Running this script against the capture results in one hit:

%LOCALAPPDATA%\Packages\DISTRO\LocalState\fsserver

DISTRO is the name of the Store package you installed the distro from, for example Debian is installed into TheDebianProject.DebianGNULinux_76v4gfsz19hv4. With a file name of fsserver it seems pretty clear what the file is for, but just to check lets open the event back in PROCMON and look at the call stack.

PROCMON call stack opening fsserver showing AFUNIX driver and P9RDR.

I've highlighted areas of interest, at the top there's the calls through the AFUNIX driver, which demonstrates that the file is being opened due a UNIX socket connection being made. At the bottom we can see a list of calls in the P9RDR driver. As symbol resolving is enabled we can use the symbol information to target specific areas of the driver for reverse engineering. Also now we know the path we can put this back into PROCMON as a filter and from that we can confirm that it's the init process which is responsible for setting up the file server.

In conclusion we can at least confirm a few things which we didn't know before.
  • The handling of the UNC paths is handled entirely in kernel mode via a mini-redirector. This makes the file system more interesting from a security perspective as it's parsing arbitrary user data in the kernel.
  • The file system uses UNIX sockets for communication, this is handled by the kernel driver and the main init process.
  • The socket protocol is presumably P9 based on the driver name, however we've not actually confirmed that to be true.
There's of course still things we'd want to know:
  • How is the UNC mappings configured? Via the device driver?
  • Is the protocol actually P9, if so what information is being passed across?
  • How well "fuzzed" are the protocol parsers.
  • Does this file system have any other interesting behaviors.
Some of those things will have to wait for another blog post.









Sunday, 17 February 2019

NTFS Case Sensitivity on Windows

Back in February 2018 Microsoft released on interesting blog post (link) which introduced per-directory case-sensitive NTFS support. MS have been working on making support for WSL more robust and interop between the Linux and Windows side of things started off a bit rocky. Of special concern was the different semantics between traditional Unix-like file systems and Windows NTFS.

I always keep an eye out for new Windows features which might have security implications and per-directory case sensitivity certainly caught my attention. With 1903 not too far off I thought it was time I actual did a short blog post about per-directory case-sensitivity and mull over some of the security implications. While I'm at it why not go on a whistle-stop tour of case sensitivity in Windows NT over the years.

Disclaimer. I don't currently and have never previously worked for Microsoft so much of what I'm going to discuss is informed speculation.


The Early Years

The Windows NT operating system has had the ability to have case-sensitive files since the very first version. This is because of the OS's well known, but little used, POSIX subsystem. If you look at the documentation for CreateFile you'll notice a flag, FILE_FLAG_POSIX_SEMANTICS which is used for the following purposes:

"Access will occur according to POSIX rules. This includes allowing multiple files with names, differing only in case, for file systems that support that naming."

It's make sense therefore that all you'd need to do to get a case-sensitive file system is use this flag exclusively. Of course being an optional flag it's unlikely that the majority of Windows software will use it correctly. You might wonder what the flag is actually doing, as CreateFile is not a system call. If we dig into the code inside KERNEL32 we'll find the following:

BOOL CreateFileInternal(LPCWSTR lpFileName, ..., DWORD dwFlagsAndAttributes) { // ... OBJECT_ATTRIBUTES ObjectAttributes; if (dwFlagsAndAttributes & FILE_FLAG_POSIX_SEMANTICS){ ObjectAttributes.Attributes = 0; } else { ObjectAttributes.Attributes = OBJ_CASE_INSENSITIVE; } NtCreateFile(..., &ObjectAttributes, ...); }

This code shows that if the FILE_FLAG_POSIX_SEMANTICS flag is set, the the Attributes member of the OBJECT_ATTRIBUTES structure passed to NtCreateFile is initialized to 0. Otherwise it's initialized with the flag OBJ_CASE_INSENSITIVE. The OBJ_CASE_INSENSITIVE instructs the Object Manager to do a case-insensitive lookup for a named kernel object. However files do not directly get parsed by the Object Manager, so the IO manager converts this flag to the IO_STACK_LOCATION flag SL_CASE_SENSITIVE before handing it off to the file system driver in an IRP_MJ_CREATE IRP. The file system driver can then honour that flag or not, in the case of NTFS it honours it and performs a case-sensitive file search instead of the default case-insensitive search.

Aside. Specifying FILE_FLAG_POSIX_SEMANTICS supports one other additional feature of CreateFile that I can see. By specifying FILE_FLAG_BACKUP_SEMANTICS, FILE_FLAG_POSIX_SEMANTICS  and FILE_ATTRIBUTE_DIRECTORY in the dwFlagsAndAttributes parameter and CREATE_NEW as the dwCreationDisposition parameter the API will create a new directory and return a handle to it. This would normally require calling CreateDirectory, then a second call to open or using the native NtCreateFile system call.

NTFS always supported case-preserving operations, so creating the file AbC.txt will leave the case intact. However when it does an initial check to make sure the file doesn't already exist if you request abc.TXT then NTFS would find it during a case-insensitive search. If the create is done case-sensitive then NTFS won't find the file and you can now create the second file. This allows NTFS to support full case-sensitivity. 

It seems too simple to create files in a case-sensitive manner, just use the FILE_FLAG_POSIX_SEMANTICS flag or don't pass OBJ_CASE_INSENSITIVE to NtCreateFile. Let's try that using PowerShell on a default installation on Windows 10 1809 to see if that's really the case.

Opening the file AbC.txt with OBJ_CASE_INSENSITIVE and without.

First we create a file with the name AbC.txt, as NTFS is case preserving this will be the name assigned to it in the file system. We then open the file first with the OBJ_CASE_INSENSITIVE attribute flag set and specifying the name all in lowercase. As expected we open the file and displaying the name shows the case-preserved form. Next we do the same operation without the OBJ_CASE_INSENSITIVE flag, however unexpectedly it still works. It seems the kernel is just ignoring the missing flag and doing the open case-insensitive. 

It turns out this is by design, as case-insensitive operation is defined as opt-in no one would ever correctly set the flag and the whole edifice of the Windows subsystem would probably quickly fall apart. Therefore honouring enabling support for case-sensitive operation is behind a Session Manager Kernel Registry valueObCaseInsensitive. This registry value is reflected in the global kernel variable, ObpCaseInsensitive which is set to TRUE by default. There's only one place this variable is used, ObpLookupObjectName, which looks like the following:

NTSTATUS ObpLookupObjectName(POBJECT_ATTRIBUTES ObjectAttributes, ...) { // ... DWORD Attributes = ObjectAttributes->Attributes; if (ObpCaseInsensitive) { Attributes |= OBJ_CASE_INSENSITIVE; } // Continue lookup. }

From this code we can see if ObpCaseInsensitive set to TRUE then regardless of the Attribute flags passed to the lookup operation OBJ_CASE_INSENSITIVE is always set. What this means is no matter what you do you can't perform a case-sensitive lookup operation on a default install of Windows. Of course if you installed the POSIX subsystem you'll typically find the kernel variable set to FALSE which would enable case-sensitive operation for everyone, at least if they forget to set the flags. 

Let's try the same test again with PowerShell but make sure ObpCaseInsensitive is FALSE to see if we now get the expected operation.

Running the same tests but with ObpCaseInsensitive set to FALSE. With OBJ_CASE_INSENSITIVE the file open succeeds, without the flag it fails with an error.

With the OBJ_CASE_INSENSITIVE flag set we can still open the file AbC.txt with the lower case name. However without specifying the flag we we get STATUS_OBJECT_NAME_NOT_FOUND which indicates the lookup operation failed.

Windows Subsystem for Linux

Let's fast forward to the introduction of WSL in Windows 10 1607. WSL needed some way of representing a typical case-sensitive Linux file system. In theory the developers could have implemented it on top of a case-insensitive file system but that'd likely introduce too many compatibility issues. However just disabling ObCaseInsensitive globally would likely introduce their own set of compatibility issues on the Windows side. A compromise was needed to support case-sensitive files on an existing volume.

AsideIt could be argued that Unix-like operating systems (including Linux) don't have a case-sensitive file system at all, but a case-blind file system. Most Unix-like file systems just treat file names on disk as strings of opaque bytes, either the file name matches a sequence of bytes or it doesn't. The file system doesn't really care whether any particular byte is a lower or upper case character. This of course leads to interesting problems such as where two file names which look identical to a user can have different byte representations resulting in unexpected failures to open files. Some file systems such macOS's HFS+ use Unicode Normalization Forms to make file names have a canonical byte representation to make this easier but leads to massive additional complexity, and was infamously removed in the successor APFS. UPDATE: It's been pointed out that Apple actually reversed the APFS change in iOS 11/macOS 10.13.

This compromise can be found back in ObpLookupObjectName as shown below:

NTSTATUS ObpLookupObjectName(POBJECT_ATTRIBUTES ObjectAttributes, ...) { // ... DWORD Attributes = ObjectAttributes->Attributes; if (ObpCaseInsensitive && KeGetCurrentThread()->CrossThreadFlags.ExplicitCaseSensitivity == FALSE) { Attributes |= OBJ_CASE_INSENSITIVE; } // Continue lookup. }

In the code we now find that the existing check for ObpCaseInsensitive is augmented with an additional check on the current thread's CrossThreadFlags for the ExplicitCaseSensitivity bit flag. Only if the flag is not set will case-insensitive lookup be forced. This looks like a quick hack to get case-sensitive files without having to change the global behavior. We can find the code which sets this flag in NtSetInformationThread.

NTSTATUS NtSetInformationThread(HANDLE ThreadHandle, THREADINFOCLASS ThreadInformationClass, PVOID ThreadInformation, ULONG ThreadInformationLength) { switch(ThreadInformationClass) {
case ThreadExplicitCaseSensitivity: if (ThreadInformationLength != sizeof(DWORD)) return STATUS_INFO_LENGTH_MISMATCH; DWORD value = *((DWORD*)ThreadInformation); if (value) { if (!SeSinglePrivilegeCheck(SeDebugPrivilege, PreviousMode)) return STATUS_PRIVILEGE_NOT_HELD; if (!RtlTestProtectedAccess(Process, 0x51) ) return STATUS_ACCESS_DENIED; } if (value) Thread->CrossThreadFlags.ExplicitCaseSensitivity = TRUE; else Thread->CrossThreadFlags.ExplicitCaseSensitivity = FALSE; break; } // ... }

Notice in the code to set the the ExplicitCaseSensitivity flag we need to have both SeDebugPrivilege and be a protected process at level 0x51 which is PPL at Windows signing level. This code is from Windows 10 1809, I'm not sure it was this restrictive previously. However for the purposes of WSL it doesn't matter as all processes are gated by a system service and kernel driver so these checks can be easily bypassed. As any new thread for a WSL process must go via the Pico process driver this flag could be automatically set and everything would just work.

Per-Directory Case-Sensitivity

A per-thread opt-out from case-insensitivity solved the immediate problem, allowing WSL to create case-sensitive files on an existing volume, but it didn't help Windows applications inter-operating with files created by WSL. I'm guessing NTFS makes no guarantees on what file will get opened if performing a case-insensitive lookup when there's multiple files with the same name but with different case. A Windows application could easily get into difficultly trying to open a file and always getting the wrong one. Further work was clearly needed, so introduced in 1803 was the topic at the start of this blog, Per-Directory Case Sensitivity. 

The NTFS driver already handled the case-sensitive lookup operation, therefore why not move the responsibility to enable case sensitive operation to NTFS? There's plenty of spare capacity for a simple bit flag. The blog post I reference at the start suggests using the fsutil command to set case-sensitivity, however of course I want to know how it's done under the hood so I put fsutil from a Windows Insider build into IDA to find out what it was doing. Fortunately changing case-sensitivity is now documented. You pass the FILE_CASE_SENSITIVE_INFORMATION structure with the FILE_CS_FLAG_CASE_SENSITIVE_DIR set via NtSetInformationFile to a directory. with the FileCaseSensitiveInformation information class. We can see the implementation for this in the NTFS driver.

NTSTATUS NtfsSetCaseSensitiveInfo(PIRP Irp, PNTFS_FILE_OBJECT FileObject) { if (FileObject->Type != FILE_DIRECTORY) { return STATUS_INVALID_PARAMETER; } NSTATUS status = NtfsCaseSensitiveInfoAccessCheck(Irp, FileObject); if (NT_ERROR(status)) return status; PFILE_CASE_SENSITIVE_INFORMATION info =
(PFILE_CASE_SENSITIVE_INFORMATION)Irp->AssociatedIrp.SystemBuffer; if (info->Flags & FILE_CS_FLAG_CASE_SENSITIVE_DIR) { if ((g_NtfsEnableDirCaseSensitivity & 1) == 0) return STATUS_NOT_SUPPORTED; if ((g_NtfsEnableDirCaseSensitivity & 2) && !NtfsIsFileDeleteable(FileObject)) { return STATUS_DIRECTORY_NOT_EMPTY; } FileObject->Flags |= 0x400; } else { if (NtfsDoesDirHaveCaseDifferingNames(FileObject)) { return STATUS_CASE_DIFFERING_NAMES_IN_DIR; } FileObject->Flags &= ~0x400; } return STATUS_SUCCESS; }

There's a bit to unpack here. Firstly you can only apply this to a directory, which makes some sense based on the description of the feature. You also need to pass an access check with the call NtfsCaseSensitiveInfoAccessCheck. We'll skip over that for a second. 

Next we go into the actual setting or unsetting of the flag. Support for Per-Directory Case-Sensitivity is not enabled unless bit 0 is set in the global g_NtfsEnableDirCaseSensitivity variable. This value is loaded from the value NtfsEnableDirCaseSensitivity in HKLM\SYSTEM\CurrentControlSet\Control\FileSystem, the value is set to 0 by default. This means that this feature is not available on a fresh install of Windows 10, almost certainly this value is set when WSL is installed, but I've also found it on the Microsoft app-development VM which I don't believe has WSL installed, so you might find it enabled in unexpected places. The g_NtfsEnableDirCaseSensitivity variable can also have bit 1 set, which indicates that the directory must be empty before changing the case-sensitivity flag (checked with NtfsIsFileDeleteable) however I've not seen that enabled. If those checks pass then the flag 0x400 is set in the NTFS file object.

If the flag is being unset the only check made is whether the directory contains any existing colliding file names. This seems to have been added recently as when I originally tested this feature in an Insider Preview you could disable the flag with conflicting filenames which isn't necessarily sensible behavior.

Going back to the access check, the code for NtfsCaseSensitiveInfoAccessCheck looks like the following:

NTSTATUS NtfsCaseSensitiveInfoAccessCheck(PIRP Irp, PNTFS_FILE_OBJECT FileObject) { if (NtfsEffectiveMode(Irp) || FileObject->Access & FILE_WRITE_ATTRIBUTES) { PSECURITY_DESCRIPTOR SecurityDescriptor; SECURITY_SUBJECT_CONTEXT SubjectContext; SeCaptureSubjectContext(&SubjectContext); NtfsLoadSecurityDescriptor(FileObject, &SecurityDescriptor); if (SeAccessCheck(SecurityDescriptor, &SubjectContext FILE_ADD_FILE | FILE_ADD_SUBDIRECTORY | FILE_DELETE_CHILD)) { return STATUS_SUCCESS; } } return STATUS_ACCESS_DENIED; }

The first check ensures the file handle is opened with FILE_WRITE_ATTRIBUTES access, however that isn't sufficient to enable the flag. The check also ensures that if an access check is performed on the directory's security descriptor that the caller would be granted FILE_ADD_FILE, FILE_ADD_SUBDIRECTORY and FILE_DELETE_CHILD access rights. Presumably this secondary check is to prevent situations where a file handle was shared to another process with less privileges but with FILE_WRITE_ATTRIBUTES rights. 

If the security check is passed and the feature is enabled you can now change the case-sensitivity behavior, and it's even honored by arbitrary Windows applications such as PowerShell or notepad without any changes. Also note that the case-sensitivity flag is inherited by any new directory created under the original.

Showing setting case sensitive on a directory then using Set-Content and Get-Content to interact with the files.

Security Implications of Per-Directory Case-Sensitivity

Let's get on to the thing which interests me most, what's the security implications on this feature? You might not immediately see a problem with this behavior. What it does do is subvert the expectations of normal Windows applications when it comes to the behavior of file name lookup with no way of of detecting its use or mitigating against it. At least with the FILE_FLAG_POSIX_SEMANTICS flag you were only introducing unexpected case-sensitivity if you opted in, but this feature means the NTFS driver doesn't pay any attention to the state of OBJ_CASE_INSENSITIVE when making its lookup decisions. That's great from an interop perspective, but less great from a correctness perspective.

Some of the use cases I could see this being are problem are as follows:
  • TOCTOU where the file name used to open a file has its case modified between a security check and the final operation resulting in the check opening a different file to the final one. 
  • Overriding file lookup in a shared location if the create request's case doesn't match the actual case of the file on disk. This would be mitigated if the flag to disable setting case-sensitivity on empty directories was enabled by default.
  • Directory tee'ing, where you replace lookup of an earlier directory in a path based on the state of the case-sensitive flag. This at least is partially mitigated by the check for conflicting file names in a directory, however I've no idea how robust that is.
I found it interesting that this feature also doesn't use RtlIsSandboxToken to check the caller's not in a sandbox. As long as you meet the access check requirements it looks like you can do this from an AppContainer, but its possible I missed something.  On the plus side this feature isn't enabled by default, but I could imagine it getting set accidentally through enterprise imaging or some future application decides it must be on, such as Visual Studio. It's a lot better from a security perspective to not turn on case-sensitivity globally. Also despite my initial interest I've yet to actual find a good use for this behavior, but IMO it's only a matter of time :-)