path/filepath: use Windows GetFinalPathNameByHandle instead of os.Readlink to implement EvalSymlinks

[darstahl]

What version of Go are you using (go version)?

go version go1.8.1 windows/amd64

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows

What did you do?

When a directory junction points to a path in the NT Namespace (See this MSDN page for details) that starts with \\?\ os.Readlink strips the leading \\?\ which results in returning an invalid path.

This requires cmd to create the Junction, so doesn't seem to work on play.golang.org. I've uploaded a repro to https://github.com/darrenstahlmsft/ReadJunction

To run the repro, run mountvol.exe which will print out:

Possible values for VolumeName along with current mount points are:

    \\?\Volume{7d396648-0527-4fed-9cae-5d961ec7311a}\
        C:\

Replace the folderPath in main.go with the retrieved volume GUID path then run main.go.

What did you expect to see?

When run in go1.7.5 this works as expected and returns:

$ go run main.go
Junction created for test_junction <<===>> \\?\Volume{7d396648-0527-4fed-9cae-5d961ec7311a}\

Readlink returned: \\?\Volume{7d396648-0527-4fed-9cae-5d961ec7311a}\

What did you see instead?

When run in go1.8.1 the returned path is invalid:

$ go run main.go
Junction created for test_junction <<===>> \\?\Volume{7d396648-0527-4fed-9cae-5d961ec7311a}\

Readlink returned: Volume{7d396648-0527-4fed-9cae-5d961ec7311a}\

/cc @jstarks

[alexbrainman]

@darrenstahlmsft this is a reasonable request, but I do not know how to fetch the information you require.

At this moment we use Windows DeviceIoControl call with FSCTL_GET_REPARSE_POINT parameter to fetch symlink target (see syscall.Readlink function). I tried calling DeviceIoControl with your example, and I get \??\Volume{790f333b-6087-11e4-82c0-806e6f6e6963}\. How do I convert that into \\?\Volume{790f333b-6087-11e4-82c0-806e6f6e6963}\, as you expect? What are the rules? Maybe you could ask your colleagues at Microsoft.

How did you encountered this problem? What is it that you are trying to do?
Maybe you should use path/filepath.EvalSymlinks instead?
The path/filepath.EvalSymlinks will not work for your scenario at this moment, but I beleive I know how to make it work.

Thank you.

Alex

[darstahl]

@alexbrainman \??\Volume{790f333b-6087-11e4-82c0-806e6f6e6963}\ is the NT path for the volume. For the case of a path beginning with \??\Volume it can be manually converted to \\?\Volume. I'll look into seeing if there is a better way to generalize this, or a different way to get symlink targets that returns a DOS path rather than an NT path. The old behaviour (pre 1.8.x) of returning the \??\ path is preferable to the current implementation at the very least.

I am using a slightly forked version of filepath.EvalSymlinks that allows for paths starting with \\?\. See (https://github.com/moby/moby/blob/master/pkg/symlink/fs_windows.go).

[alexbrainman]

For the case of a path beginning with ??\Volume it can be manually converted to \?\Volume

So we should just replace \??\Volume at the start with \\?\Volume? Do you have some reference for that?

I'll look into seeing if there is a better way to generalize this, or a different way to get symlink targets that returns a DOS path rather than an NT path.

That would be nice. Thank you.

The old behaviour (pre 1.8.x) of returning the ??\ path is preferable to the current implementation at the very least.

Did os.Readlink change during go1.8? What change are you referring to? Regardless, I don't think we could change os.Readlink or syscall.Readlink now. But maybe we could just return whatever DeviceIoControl(...FSCTL_GET_REPARSE_POINT...) returns as is? Add new syscall.ReadlinkNTName or something?

I am using a slightly forked version of filepath.EvalSymlinks that allows for paths starting with \?.

Well, if your problem is filepath.EvalSymlinks, then tell me more about what is broken with filepath.EvalSymlinks. Just provide some examples.
I was planing to change filepath.EvalSymlinks to use GetFinalPathNameByHandle instead (see https://blogs.msdn.microsoft.com/oldnewthing/20100212-00/?p=14963/ for details). That will not need to use broken os.Readlink.

Alex

[jstarks]

@alexbrainman Using GetFinalPathNameByHandle is definitely the right approach. I think that should resolve @darrenstahlmsft's issues.

Unfortunately it's Vista+ only, so you'll have to fall back to the old behavior for XP (it's still supported, right?).

[alexbrainman]

@alexbrainman Using GetFinalPathNameByHandle is definitely the right approach. I think that should resolve @darrenstahlmsft's issues.

Thank you for confirming. I am happy to retrofit this issue to fixing filepath.EvalSymlinks. But can you provide some examples of where filepath.EvalSymlinks is broken, please? I have some ideas, but nothing I can put into a test that will run on our builders.

Also I noticed GetFinalPathNameByHandle can return "paths with drive letter" or "nt paths". We should use the former for compatibility reasons. But sometimes (see #18555) paths inside container lead to "nt path" only. If we ask for "paths with drive letter", the GetFinalPathNameByHandle fails. But it succeeds if we ask for "nt path". Should

1. filepath.EvalSymlinks fail for these?
2. filepath.EvalSymlinks return "nt path" for these?
3. filepath.EvalSymlinks fail for these, but we provide new syscall.EvalSymlinksNTPath (or whatever name) that always return "nt path"?

Unfortunately it's Vista+ only, so you'll have to fall back to the old behavior for XP (it's still supported, right?).

I am not worried about XP. XP does not support symlinks and Docker. And there aren't many XP users left. We will leave XP as is.

Alex

[jstarks]

Sorry for the delays in following up; I missed Alex's response.

The easiest case where this is broken is when you have an NTFS volume mounted to a mount point (e.g. C:\foo) without having a separate drive letter for it. In this case, as I understand it EvalSymlinks on this path will fail. I don't know how to set this up easily in your test environment, unfortunately. You could create, format, and mount a VHD, but this will require administrative privileges and some fiddling to get right. I'm not aware of any call that would fail with a desktop or server OS in its default clean-install configuration.

Regarding #18555, I suspect you should just fail if the drive letter option doesn't work. We want to fix this at the Windows level so that GetFInalPathNameByHandle just works. This will take a little while for us to fix, but I don't think Go should encode a workaround for what's really a Windows container bug.

[alexbrainman]

I don't know how to set this up easily in your test environment, unfortunately.

Fair enough. We already have plenty of filepath.EvalSymlinks tests. We would have to rely on them.

I suspect you should just fail if the drive letter option doesn't work.

SGTM

I don't think Go should encode a workaround for what's really a Windows container bug.

SGTM

I also changed this issue description. Hopefully new description covers what you want.

Alex

[gopherbot]

Change https://golang.org/cl/55250 mentions this issue: path/filepath: re-implement windows EvalSymlinks

[gopherbot]

Change https://golang.org/cl/55612 mentions this issue: path/filepath: re-implement windows EvalSymlinks

[gopherbot]

Change https://golang.org/cl/135295 mentions this issue: path/filepath: correct symlink eval for symlink at root