Repository navigation
windows: preserve hard links when committing a container - #53624
Conversation
8a57541 to
442798d
Compare
thaJeztah
left a comment
There was a problem hiding this comment.
Thanks! Overall looks good, but left some comments
| nlinks, id, err := r.LinkInfo() | ||
| if err == nil && nlinks > 1 { |
There was a problem hiding this comment.
Nit; can scope these vars to the if;
if nlinks, id, err := r.LinkInfo(); err == nil && nlinks > 1 {| } else { | ||
| nlinks, id, err := r.LinkInfo() | ||
| if err == nil && nlinks > 1 { | ||
| slash := filepath.ToSlash(name) |
There was a problem hiding this comment.
This one looks to be only used if it's not yet present, so could inline this;
linkRecords[id.FileID] = filepath.ToSlash(name)| slash := filepath.ToSlash(name) | ||
| if prev, ok := linkRecords[id.FileID]; ok { | ||
| hdr := backuptar.BasicInfoHeader(name, 0, fileInfo) | ||
| hdr.Mode = 0644 |
There was a problem hiding this comment.
If you're updating; perhaps also use the canonical (0o644 ) format here, to prevent gofumpt from updating it later 😅
hdr.Mode = 0o644There was a problem hiding this comment.
In case useful; here's what I had locally; feel free to copy (and amend your commit); I also added a short comment to explain what the intent is;
// For files with multiple hardlinks, write the first occurrence normally
// and record its path. Subsequent occurrences are emitted as hardlinks to it.
if nlinks, id, err := r.LinkInfo(); err == nil && nlinks > 1 {
if prev, ok := linkRecords[id.FileID]; ok {
hdr := backuptar.BasicInfoHeader(name, 0, fileInfo)
hdr.Mode = 0o644
hdr.Typeflag = tar.TypeLink
hdr.Linkname = prev
if err := t.WriteHeader(hdr); err != nil {
return err
}
continue
}
linkRecords[id.FileID] = filepath.ToSlash(name)
}| func writeTarFromLayer(r hcsshim.LayerReader, w io.Writer) error { | ||
| linkRecords := make(map[[16]byte]string) | ||
| t := tar.NewWriter(w) |
There was a problem hiding this comment.
Unrelated to this PR; I can open a follow-up PR for that, or if you want, you can add a second commit for it; looks like there's a potential resource leak here; the writer isn't closed on early returns, so we should probably add a defer (and a named output variable);
func writeTarFromLayer(r hcsshim.LayerReader, w io.Writer) (retErr error) {
linkRecords := make(map[[16]byte]string)
t := tar.NewWriter(w)
defer func() {
if err := t.Close(); err != nil && retErr == nil {
retErr = err
}
}()Then remove the return t.Close() at the end of the function (just return nil)
vvoland
left a comment
There was a problem hiding this comment.
Thanks!
2 suggestions from my side:
- The verification you mentioned could be a nice integration test (probably also for non-Windows too?)
- Can be squashed to one commit, no need to separate fix from the tests
writeTarFromLayer wrote a full copy for every name of a file, so files that shared a file ID in a layer became independent copies when a container was committed. The link relationship was lost and the payload was stored once per name. The layer reader already exposes LinkInfo, which reports the link count and file ID of the current file, so no new API is needed. Record the file ID of the first name seen for a file and emit any later name as a tar hard link entry pointing at that first name. The import side already handles tar.TypeLink through legacyLayerWriter.AddLink, so no change is needed there. Unit tests drive writeTarFromLayer through a scripted LayerReader, so the behaviour can be verified without a layer, a container, or a particular Windows build. Note that this depends on the Windows container layer export reporting a link count greater than one for such files. That requires a corresponding change in the Windows container storage layer, which is being made separately. On builds without it the layer export writes independent copies, the link count is one, and this change is inert with no behaviour difference. Measured on a build that carries the matching OS change, a container writing a 200 MB file with two additional hard linked names produced a committed layer consuming 608 MB before this change and 208 MB after. Signed-off-by: Varun Gokulnath <vagokuln@microsoft.com>
The tar writer was only closed on the success path, so any early return from the loop left it unclosed. Use a named return value and close it in a defer, preserving the first error seen. This is a pre-existing issue, unrelated to hard link handling. Suggested by thaJeztah during review of moby#53624. Signed-off-by: Varun Gokulnath <vagokuln@microsoft.com>
442798d to
b783799
Compare
|
Thanks both, updated. @thaJeztah I took your suggested block as is, so the vars are now scoped to the @vvoland squashed the fix and the tests into one commit. On the integration test, I looked at this and I do not think one would be meaningful today, for two reasons. For Windows, the link count comes from the layer export performed by the Windows container storage layer. On every currently shipping Windows build that export writes an independent copy per name, so For non-Windows, I do not think there is anything to test. This is specific to the Windows graphdriver writing its own tar. The Linux path goes through Once the OS side lands I am happy to follow up with an integration test if you still want one. |
|
Code changes look good to me, but can you remove this snipped from the second commit message? We try to avoid referencing GitHub PR / ticket numbers from commit messages, as they tend to produce much noise on Github;
|
The tar writer was only closed on the success path, so any early return from the loop left it unclosed. Use a named return value and close it in a defer, preserving the first error seen. This is a pre-existing issue, unrelated to the hard link handling in the preceding commit. Signed-off-by: Varun Gokulnath <vagokuln@microsoft.com>
b783799 to
c958419
Compare
|
Done, removed the reference. The second commit message now just notes that it is a pre-existing issue unrelated to the hard link handling in the preceding commit. Only the message changed, the tree is identical. |
- What I did
Preserve hard links when committing a Windows container.
writeTarFromLayerwrote a full copy for every name of a file, so files sharing a file ID in a layer became independent copies in the committed image. The link relationship was lost and the payload was stored once per name.- How I did it
The layer reader already exposes
LinkInfo, which returns the link count and file ID of the current file, so no new API was needed.For a file reporting more than one link, the file ID of the first name seen is recorded. Any later name with the same file ID is written as a
tar.TypeLinkheader pointing at that first name instead of streaming the contents again.The import side already handles
tar.TypeLinkthroughlegacyLayerWriter.AddLink, so no change was required there.- How to verify it
Unit tests are included in
daemon/graphdriver/windows/windows_test.go. They drivewriteTarFromLayerthrough a scriptedLayerReader, so the behaviour can be checked without a container or a particular Windows build:They assert that a later name for a file ID becomes a
tar.TypeLinkentry carrying no payload, that later names point at the first name rather than chaining to the previous one, and that distinct file IDs and files with a single link are never linked. Reverting the change inwindows.gomakes them fail.Please also read this part, because there is a dependency for the end to end behaviour.
NewLayerReaderobtains its link count from the layer export produced by the Windows container storage layer. On current Windows builds that export writes an independent copy for every name, soLinkInfoalways reports a link count of one, the new branch never runs, and this change is inert with no behaviour difference. A corresponding fix to the Windows container storage export is being made separately, and only with both changes present is the link preserved end to end.Verified on a Windows Server build carrying that matching OS change:
docker commit src img:latestInspect the resulting image layer on the host with
fsutil hardlink listagainstFiles\hltest\original.dat.Before this change the layer reports one link, meaning three independent copies. With this change it reports three names sharing one file ID.
A
docker save,docker image rm,docker loadround trip was also run and the reloaded layer still reports three links with intact file contents.Note that hard links in a read only layer are not observable from inside a running container, because the container filter presents layer files as separate per name objects. This is existing behaviour and is also true of hard links that ship in base images. Verification should be done against the image layer on the host.
- Size impact
Measured on the same source container, with a 200 MB payload and two additional hard linked names, so 600 MB of logical content:
The saving scales with the number of names sharing the file ID.
- Description for the changelog