Skip to content

refactor: adopt monorepo layout#770

Merged
kvinwang merged 505 commits into
masterfrom
chore/monorepo-layout
Jul 13, 2026
Merged

refactor: adopt monorepo layout#770
kvinwang merged 505 commits into
masterfrom
chore/monorepo-layout

Conversation

@kvinwang

Copy link
Copy Markdown
Collaborator

Summary

  • group the existing core repository under dstack/ and merge the meta-dstack history under os/yocto/
  • introduce a backend-neutral guest-OS interface, artifact manifest, shared rootfs payload, image assembler, and release entrypoints
  • update SDKs, workflows, documentation, submodules, license scopes, and consumers for the new paths
  • separate scripts by ownership: common image tools under os/image/, Yocto-only helpers under os/yocto/, isolated legacy host/QEMU tools under os/yocto/tools/, and repository utilities under tools/
  • harden image selection and release validation, including unified CPU/GPU image fallback and excluding UKI-only archives from dstackup

Build and migration notes

  • imported histories are retained, so this PR intentionally contains a large commit range; the integration work is summarized by the 12 first-parent commits
  • make os is the native default guest-OS build, while make os-image remains the reproducible container entrypoint
  • Docker Compose's first Yocto fetch contains roughly 670 independently checksummed sources; BitBake resets the displayed 0–100% progress for each source under one do_fetch task, and completed downloads resume from the cache

Validation

  • make os — full native Yocto build, bare and UKI archives generated
  • release metadata Git revision, UKI contents, and artifact-manifest schema/path validation
  • make core-check
  • make core-test
  • make sdk-test / direct root invocation of ./sdk/run-tests.sh
  • cargo test --manifest-path dstack/Cargo.toml -p dstackup (17 passed)
  • Rust SDK crate packaging checks
  • prek run on all changed files
  • reuse lint
  • Bash syntax, ShellCheck, submodule, stale-path, and relative Markdown-link checks

Non-blocking note

The existing JavaScript dependency tree reports 1 low and 3 moderate npm audit findings; all 99 JavaScript tests pass and this PR does not change those dependencies.

Update dstack/ and corresponding refactor
- Consolidate four separate image recipes (dstack-cloud-rootfs,
  dstack-dev-rootfs, dstack-cloud-nvidia-rootfs, dstack-nvidia-dev-rootfs)
  into a single dstack-rootfs.bb with conditional includes
- Add multiconfig support with separate TMPDIR per flavor to avoid
  build conflicts
- Build flavors serially in Makefile to avoid BitBake runqueue deadlock
- Update mkimage.sh to support --flavor parameter for multiconfig builds
- Copy multiconfig to new build directories in dev-setup
- Move local.conf and multiconfig/ to meta-dstack/conf/
- Update dev-setup to always sync config to build directory
- bb-build is now fully excluded from git (build artifact only)
- dstack-0.6.0/ -> dstack-cloud-0.6.0/
- dstack-nvidia-0.6.0.tar.gz -> dstack-cloud-nvidia-0.6.0.tar.gz
- Update download URL to meta-dstack-cloud repo
Move write_authenticode_hash_if_missing call to main flow instead of
only in create_gcp_artifacts. This ensures the hash file is always
generated when UKI exists, regardless of GCP image creation.
kvinwang added 15 commits July 10, 2026 05:23
Add an nvattest recipe building NVIDIA's attestation-sdk CLI (pinned
SRCREV, Apache-2.0). GPU TEE attestation runs inside `dstack-util setup`
(dstack-prepare.service), driven by the app-compose requirement
`requirements.verify_gpu` (default true): the conf-compute ready state
is set only after nvattest succeeds (previously nvidia-persistenced set
it unconditionally), and attestation failure -- including a non-CC GPU
or CC mode left off by the host -- aborts boot before key provisioning.
Deployments can opt out with `requirements.verify_gpu: false`, which
sets the ready state without verification.

The recipe ships only the CLI plus a dstack-prepare.service ordering
drop-in (After=nvidia-persistenced/nvidia-fabricmanager); there is no
dedicated attestation service or unit gating. GPU-less instances are
unaffected (dstack-util skips when no NVIDIA GPU is on the PCI bus).

Build notes:
- regorus (Rego engine, Rust) is embedded via Corrosion; use the
  prebuilt meta-rust-bin toolchain (cargo-bin-native), with host/target
  linker split for cargo build scripts (host triple == target triple)
- USE_SYSTEM_DEPS=ON: link sysroot openssl/curl/libxml2/xmlsec1 instead
  of ExternalProject-downloaded copies
- RUSTFLAGS remap-path-prefix keeps build paths out of libnvat
  ([buildpaths] QA / reproducible builds)
- only nvattest + libnvat are shipped; cargo proc-macro dylibs are
  excluded from the image
meta-nvidia: gate GPU ready state on local TEE attestation (nvattest)
Repair guest OS and SDK entrypoints, make release image selection safe, and relocate imported helpers to clear ownership boundaries. Document the multi-source docker-compose fetch behavior and validate release artifacts before publishing.
Copilot AI review requested due to automatic review settings July 13, 2026 01:17
kvinwang added 2 commits July 12, 2026 18:19
Keep the repository-wide Rust pin at the monorepo root so both the core workspace and SDK simulator inherit Rust 1.92. Update the CI action refs to the same toolchain and clean simulator sockets after core tests.
Copilot stopped reviewing on behalf of kvinwang due to an error July 13, 2026 01:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@kvinwang
kvinwang merged commit 8e9c13c into master Jul 13, 2026
16 checks passed
@kvinwang
kvinwang deleted the chore/monorepo-layout branch July 20, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants