path: Windows paths may contain non-utf8-representable sequences

[lilyball]

It turns out that Windows paths may contain unpaired UTF-16 surrogates, which is not legally representable in UTF-8. This means that WindowsPath is not capable of representing such paths, as it uses ~str internally.

[SimonSapin]

It turns out that Windows paths may contain unpaired UTF-16 surrogates

@kballard What is this based on? Trying to use CreateFileW directly to create such a file failed for me. Test case:

extern crate libc;

use std::ptr;
use std::io::IoError;

fn main() {
    for name in [&['a' as u16, 0xD83D, 0xDCA9, 'b' as u16, 0],
                 &['a' as u16, 0xD83D,         'b' as u16, 0]].iter() {
        let handle = unsafe {
            libc::CreateFileW(name.as_ptr(),
                              libc::FILE_GENERIC_WRITE,
                              0,
                              ptr::mut_null(),
                              libc::CREATE_ALWAYS,
                              libc::FILE_ATTRIBUTE_NORMAL,
                              ptr::mut_null())
        };
        let is_invalid = handle == libc::INVALID_HANDLE_VALUE as libc::HANDLE;
        println!("{} {} {}", name, is_invalid, IoError::last_error())
    }
}

Output:

[97, 55357, 56489, 98, 0] false unknown error (OS Error 0: The operation completed successfully.
)
[97, 55357, 98, 0] true unknown error (OS Error 87: The parameter is incorrect.
)

[lilyball]

@SimonSapin I don't know the precise details, but there exist portions of Windows in which paths are UCS2 rather than UTF-16. I ignored it because I thought it wasn't going to be an issue but at some point someone (and I wish I could remember who) showed me some output that showed that they were actually getting a UCS2 path from some Windows call and Path was unable to parse it.

[SimonSapin]

Maybe CreateFileW is doing an explicit check that not all parts of the API are doing. sigh

[lilyball]

My recollection is the bad path came from iterating a temporary directory on this person's filesystem. But I don't remember for certain.

[SimonSapin]

I’d be interested to see a test case, because this seems like it would affect every other language or library that tries to use filenames as Unicode.

[lilyball]

I'd love a test case too. Not being a Windows user is making it hard for me to actually test this stuff.

[SimonSapin]

FWIW I used a virtual machine image from http://www.modern.ie/ and a nightly build to run the test above.

[klutzy]

@kballard @SimonSapin #13338?
I think it depends on OS version and possibly even on locale. I ran your code on win7/kr and got:

[97, 55357, 56489, 98, 0] false unknown error (OS Error 0 (FormatMessageW() returned error 15100))
[97, 55357, 98, 0] false unknown error (OS Error 0 (FormatMessageW() returned error 15105))

[SimonSapin]

@klutzy false for is_invalid and OS Error 0 looks like CreateFileW was successful. Do you see the two files created in the current directory?

[SimonSapin]

Oh, wait, "success" here is not good news as it means you managed to create a file with a broken name. Does io::fs::readdir() trigger this failure, then?

[klutzy]

@SimonSapin

Do you see the two files created in the current directory?
Does io::fs::readdir() trigger this failure, then?

Yes and yes. :'( I'm curious if it only occurs on recent OSes. (maybe >= 7?) Could you confirm OS name of VM you use?

[SimonSapin]

Windows 7 Enterprise SP1, 32-bit, en-US locale

[klutzy]

Ugh. I have no idea now.

Anyway, I'm ok to just ignore non-unicode filenames: I don't think there will be many use cases regarding it.

[lilyball]

Perhaps. If it really is that rare, then maybe it's ok. It certainly simplifies things. And the programmer always has the option of dropping down and doing the system calls directly if they need to handle this.

But it makes me a bit uncomfortable regardless.

However, as I'm not a Windows user, I'll defer to Windows users on this subject.