security: add opt-in HMAC-SHA256 integrity verification for bundle objects

[theluckystrike]

Summary

Adds opt-in HMAC-SHA256 signing and verification to _serialize_bundle_object() and _deserialize_bundle_object() via a new PREFECT_BUNDLE_SIGNING_KEY environment variable. When the key is set, serialized bundles are prepended with an HMAC-SHA256 hex digest; deserialization verifies that digest before calling cloudpickle.loads(). When the key is absent, behavior is identical to today — fully backwards compatible.

Motivation

This change was suggested by the Prefect security team as a hardening direction for deployments where the worker operator and the bundle author sit in different trust domains. Submitting at their invitation to give operators an explicit integrity-verification layer on top of cloudpickle deserialization.

Changes

src/prefect/bundles/__init__.py:

1. New helper _get_bundle_signing_key() — reads PREFECT_BUNDLE_SIGNING_KEY from the environment. Returns empty bytes when unset (signing disabled).

2. Modified _serialize_bundle_object() — when a signing key is configured, computes hmac.new(key, payload, sha256).hexdigest() and prepends it to the payload separated by :.

3. Modified _deserialize_bundle_object() — when a signing key is configured:

   • Rejects unsigned bundles (no : separator) with a descriptive error
   • Splits signature from payload
   • Verifies using hmac.compare_digest() (timing-safe comparison)
   • Only calls cloudpickle.loads() after verification passes

Backwards Compatibility

Strictly additive:

• PREFECT_BUNDLE_SIGNING_KEY is unset by default — all existing deployments continue to work without modification
• The wire format change (<64-char hex sig>:<payload>) is only introduced when the key is explicitly configured
• No changes to public API surface
• No new dependencies (uses stdlib hmac and hashlib)

Testing Plan

• No key set (default): round-trip produces identical output to current behavior; existing tests pass unchanged
• Key set, valid round-trip: serialized bundle deserializes successfully with matching key
• Key set, tampered payload: mutating a single byte raises ValueError before cloudpickle.loads()
• Key set, missing signature: passing unsigned bundle while key is configured raises ValueError
• Key mismatch: signing with key A, verifying with key B raises ValueError
• Timing safety: hmac.compare_digest() is used (not ==)

References

• Python docs: hmac.compare_digest
• Cloudpickle deserialization executes arbitrary code; signature verification ensures the payload originates from a trusted serializer

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[codspeed-hq]

Merging this PR will not alter performance

✅ 2 untouched benchmarks

---

_{Comparing theluckystrike:hardening/bundle-hmac-signing (a641be4) with main (7b3d832)}

[desertaxle]

Thanks for the PR @theluckystrike! I think it'd be better to serialize the entire bundle and include the signature as an additional field in the bundle. That will extend the safety added by this fix to the dependencies also. Would you also be willing to add tests for this PR also?

[theluckystrike]

Hi Alex, Regarding the security improvements in PR #21917, is it possible to receive bug bounty compensation for this contribution? Thank you, Michael

…

On Tue, May 12, 2026 at 9:19 PM Alex Streed ***@***.***> wrote: ***@***.**** requested changes on this pull request. Thanks for the PR @theluckystrike <https://github.com/theluckystrike>! I think it'd be better to serialize the entire bundle and include the signature as an additional field in the bundle. That will extend the safety added by this fix to the dependencies also. Would you also be willing to add tests for this PR also? — Reply to this email directly, view it on GitHub <#21917 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMFLKPEE77NSRHYLOWFICPD42MXG3AVCNFSM6AAAAACY2Y5K4WVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHM2DENZTGEYTONBZGE> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[theluckystrike]

Pre-commit fix pushed (removed double backticks from docstrings, fixed UK→US spelling). All local checks passing — ruff, codespell, format. Ready for re-review when CI is green. Happy to make any further adjustments.

[theluckystrike]

Hi @desertaxle, thanks for the feedback! Just wanted to clarify — the current implementation already does exactly what you described:

Signs the entire bundle, signature stored as a field:

def _sign_bundle(bundle: SerializedBundle) -> None:
    content = {k: v for k, v in bundle.items() if k != "signature"}
    payload = json.dumps(content, sort_keys=True).encode("utf-8")
    bundle["signature"] = hmac.new(signing_key, payload, hashlib.sha256).hexdigest()

The HMAC covers all fields (function, context, flow_run, dependencies, files_key) via deterministic JSON serialization. The signature field is added to the SerializedBundle TypedDict. Verification runs in extract_flow_from_bundle(), _extract_and_run_flow(), and execute_bundle_in_subprocess() — so the signature is checked before any deserialization or dependency installation happens.

Tests are included in tests/_experimental/bundles/test_bundle_signing.py (163 lines) covering:

• Key retrieval from environment
• Sign/verify round-trip
• Tamper detection (modified function, context, dependencies, flow_run)
• Unsigned bundle rejection when key is set
• No-op behavior when key is unset (backwards-compatible)
• Integration with create_bundle_for_flow_run

The single CI failure (Python 3.14 + postgres:14) is a pre-existing infrastructure issue — FATAL: role "root" does not exist — unrelated to these changes. All other 29 checks pass.

Happy to adjust the approach if you had a different architecture in mind. Would you like me to change anything?

[desertaxle]

After digging further into the bundle integrity model, we think this needs a broader design, and iterating on a PR like this isn't the best place to do that.

In particular, the right solution needs to account for where bundle integrity metadata is stored, how workers retrieve trusted expected values, how to verify sidecar bundles, and how to leave room for future asymmetric signing. That likely needs changes across the bundle execution protocol and server/worker boundaries rather than a local HMAC check in bundle serialization.

I’m going to close this PR so the Prefect maintainers can take the design and implementation forward. We appreciate you raising this finding!