Skip to content

feat(storage): implement Session Bundle filesystem codec - #2013

Merged
MicroGery merged 7 commits into
apache:mainfrom
MicroGery:feat/session-bundle-filesystem-codec
Aug 5, 2026
Merged

feat(storage): implement Session Bundle filesystem codec#2013
MicroGery merged 7 commits into
apache:mainfrom
MicroGery:feat/session-bundle-filesystem-codec

Conversation

@MicroGery

Copy link
Copy Markdown
Contributor

Summary

  • implement deterministic POSIX USTAR V1 writing and strict canonical decoding
  • add fixed-parameter Node.js Zstandard framing, archive SHA-256 verification, and a shared streaming validator
  • implement pack, inspect, and hydrate with explicit quotas, safe extraction, source-change detection, sibling staging, cleanup, and atomic publication
  • add golden, corruption, security, round-trip, and independent-process hydration coverage

Testing

  • npm run format:check
  • npm run lint
  • npm run build
  • npm run typecheck
  • 15 targeted Session Bundle storage tests

Closes #1528

@MicroGery
MicroGery force-pushed the feat/session-bundle-filesystem-codec branch from 20aa3bc to f52b1b1 Compare August 3, 2026 15:03
@MicroGery MicroGery self-assigned this Aug 3, 2026

@likun666661 likun666661 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found three issues in the filesystem codec, including a reproducible publication-integrity race. Details are inline.

Comment thread packages/storage/src/session-bundle-file-service.ts
Comment thread packages/storage/src/session-bundle-file-service.ts Outdated
Comment thread packages/storage/src/session-bundle-file-service.ts

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review comments. Two items need a decision (directory fsync on publish; process-level crash coverage); the rest are small. Full items are inline. Notes that did not fit a line: ZERO_BLOCK (:73) is unused, the if (index < 0) break; in splitUstarPath is unreachable, and Number.isSafeInteger(parsed) in parseOctal cannot fire; the golden tar.zst hash depends on Node's bundled zstd encoder (it held across Node 24/26 and macOS/Linux, and a future encoder change breaks it loudly, which is the intended behavior; a version note next to the constant would help attribution); and a crash before publication leaves an orphan staging directory with no public cleanup API, which the issue contract mentions but the codec does not expose yet.

Comment thread packages/storage/src/session-bundle-file-service.ts Outdated
Comment thread packages/storage/src/session-bundle-file-service.ts Outdated
Comment thread packages/storage/src/session-bundle-file-service.ts
Comment thread packages/storage/src/__tests__/session-bundle-file-service.test.ts Outdated
Comment thread packages/storage/src/__tests__/session-bundle-file-service.test.ts
Comment thread packages/storage/src/__tests__/session-bundle-file-service.test.ts Outdated
Comment thread packages/storage/src/__tests__/session-bundle-ustar.test.ts

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one acceptance-criteria gap and a few non-blocking follow-ups:

  1. Orphan hydration staging cleanup: the SIGKILL test correctly proves publication atomicity, but it also leaves .maka-session-bundle-staging-* permanently behind. Issue #1528 explicitly requires orphan cleanup, while the current service only cleans failures observed by the same process. Please either implement a destination-scoped cleanup mechanism with persistent ownership evidence, or explicitly defer this criterion and link a follow-up before closing #1528. Prefix matching alone would not prove ownership after a crash.

  2. Duplicated layout validation: validateArchiveEntryBeforeContent and SessionBundleCanonicalTreeDigestBuilder independently maintain path ordering, duplicate/conflict, explicit-parent, and payload-root state. Preflight must remain before hydration writes, so the existing builder cannot simply replace it, but a shared path/layout validator—or a builder API with preflight/finalize phases—would reduce drift. I see this as a maintainability follow-up rather than a current correctness failure.

  3. Cheap regression coverage: please consider adding operation-level coverage for read-side source_changed and io_failure. Hydration cleanup before publication is already covered by the transport-digest mismatch test, and schema/codec decoding is already covered at the manifest owner, so I would not duplicate those tests here.

  4. Portable-path policy: the USTAR layer rejects Windows-reserved segments even on POSIX. That may be the right portability decision, but it is stricter than the path rules currently recorded in #1528; documenting it as part of the V1 format would make the compatibility boundary explicit.

I do not think the root engine needs to be raised to Node 24 for API availability: Node documents native Zstd support from 22.15.0, and the current golden Session Bundle tests pass under Node 22.19.0. The persistent publication lock file also appears intentional; unlinking flock files can create split-inode locking races, and the cooperative-writer limitation is already recorded in the code.

@likun666661 likun666661 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One remaining publication-cleanup issue found during re-review; details inline.

Comment thread packages/storage/src/session-bundle-file-service.ts Outdated

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. The current head closes the previously identified acceptance gap: orphan hydration staging now has an explicit cleanup API and persistent ownership records; layout validation has one shared owner; the missing operation-level regressions and portable-path documentation are also present. The storage suite and current CI are green.

Two narrow cleanup races remain as non-blocking follow-ups:

  1. After link(temporaryPath, destination) succeeds, cleanup ownership is recorded only after lstat(temporaryPath). If that temporary pathname disappears in this window, pack() returns io_failure while leaving the published destination behind. I reproduced this with a controlled link shim.

  2. If the initial ownership-record write fails, the empty or partial .owner.json cannot be parsed by destination cleanup and remains permanently. Keeping the opened record’s inode binding and removing that exact inode when staging setup fails would close this path.

Neither issue changes the core codec correctness or normal cooperative publication path, so I consider this mergeable. They are worth retaining as focused cleanup follow-ups.

@likun666661 likun666661 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found three reproducible cleanup-boundary issues on the current head. The first can recursively remove a substituted unrelated directory, the second can leave a published pack destination after returning an error, and the third leaves malformed ownership residue after an initialization write failure. Details and repro conditions are inline.

Comment thread packages/storage/src/session-bundle-file-service.ts Outdated
Comment thread packages/storage/src/session-bundle-file-service.ts Outdated
Comment thread packages/storage/src/session-bundle-file-service.ts

@likun666661 likun666661 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved the current head. The two remaining pre-anchor crash and post-link stat-failure windows are narrow, non-blocking follow-ups.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(storage): define and implement the Session Bundle filesystem codec

3 participants