fix: constrain ComplexParam Java deserialization - #2678

Rana Singh (ranadeepsingh) · 2026-08-26T10:04:58Z

Related issues and context

Follow-up to #2513 and the remaining MSRC recommendation in IcM 31000000568481.

What does this pull request change?

This PR closes the remaining unrestricted Java deserialization path used by
ComplexParam model persistence. Previously, a crafted model artifact could
run a deserialization callback before the loaded object reached its eventual
type cast.

The change:

makes unfiltered Serializer.read and model-loading paths fail closed;
passes an optional, type-specific DeserializationClassFilter from
ComplexParam into the existing serializer;
adds constrained policies for Array[Byte], Spark SQL/ML data types, and
LightGBMBooster;
validates object-array component classes and dynamic-proxy interfaces;
rejects SerializedLambda, including when a broader java.lang. prefix is
otherwise required;
preserves explicit readUnsafe and readFromHDFSUnsafe APIs for artifacts
the caller has independently established as trusted; and
provides the opt-in
spark.synapseml.legacy.allowUnsafeJavaDeserialization=true compatibility
switch for trusted legacy model artifacts.

The legacy switch restores unrestricted Java deserialization and is therefore
disabled by default. It must not be enabled for untrusted or externally
supplied artifacts.

Behavior and compatibility

Writers and serialized bytes are unchanged.
Existing standard data types remain Java object streams.
Pipeline stage, pipeline array, and DataFrame serializers are unchanged.
Existing SafeObjectInputStream(InputStream, Set[String]) and Serializer
overloads remain available.
Policy rejections use a dedicated InvalidClassException subtype, so normal
Java compatibility errors such as serialVersionUID mismatches retain their
original exception behavior.
Open-ended legacy values, including persisted UDF closures and custom UDTs,
now require explicit trusted-legacy opt-in.

This PR does not change model metadata or paths, native Spark pipeline
persistence, Python/R loaders, dependencies, workflows, release tooling, or
the on-disk format.

What to review

Reviewers should check:

whether each built-in object graph has the narrowest practical class policy;
whether every safe-to-unsafe fallback requires explicit trusted-artifact
intent;
whether array, dynamic-proxy, Scala module-proxy, and SerializedLambda
handling prevents callbacks before policy rejection; and
whether existing model artifacts retain the documented compatibility path
without weakening the secure default.

How was this patch validated?

Reproduced the vulnerable baseline: direct and ComplexParam loads ran
crafted readObject callbacks before failing.
Ran focused security, compatibility, persistence, DataType, LightGBM,
UDF persistence, model equality, and fuzzing tests on the master
baseline.
Ran the exact 14-file change set against the Spark 4.0 and Spark 4.1
compatibility branches, including Scala module-proxy and ordinary Java
compatibility-error regressions.
Ran Core and LightGBM main/test Scalastyle, code generation,
generated-wrapper configuration checks, pinned Black, and
git diff --check.
Completed exact-head GitHub and Azure validation.

Azure build
234437857
validated merge commit 9e21e0b1f53b70f4d300ec8aa90414f47be04f80
with exact PR head c2b3d78d23fbf42180e3a9b0dc849b58c8004f74.
Every published job check is green. The Azure aggregate remains
partiallySucceeded because the first PythonTests deep-learning-hf attempt
failed before product execution when msdata.visualstudio.com presented an
unrelated *.azureedge.net certificate; the selective retry passed.

The regressions prove fail-closed rejection before callbacks, legitimate
round-trips with unsafe fallback disabled, unchanged Java stream bytes,
trusted compatibility for persisted UDFs and custom UDTs, stream cleanup,
proxy and array filtering, preserved ordinary Java serialization errors, and
Scala 2.12/2.13 LightGBM object graphs.

Does this PR change any dependencies?

No.
Yes.

Does this PR add a new user-facing feature?

No. This is a security hardening and compatibility change.
Yes.

azure-pipelines · 2026-08-26T10:05:07Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

github-actions · 2026-08-26T10:05:13Z

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

fix: Fix LightGBM crashes with empty partitions
feat: Make HTTP on Spark back-offs configurable
docs: Update Spark Serving usage
build: Add codecov support
perf: improve LightGBM memory usage
refactor: make python code generation rely on classes
style: Remove nulls from CNTKModel
test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

Rana Singh (ranadeepsingh) · 2026-08-26T10:05:47Z

/azp run

azure-pipelines · 2026-08-26T10:06:03Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Pull request overview

This PR hardens SynapseML/SparkML model persistence and loading paths to fail closed against unsafe Java deserialization, unbounded metadata/object graphs, and path-traversal/link/glob issues—while preserving an explicit trusted-legacy compatibility switch for artifacts that cannot be safely constrained.

Changes:

Introduces a session-scoped “trusted legacy artifact” gate (spark.synapseml.legacy.allowUnsafeJavaDeserialization) and routes unsafe persistence (UDF closures, custom readers, BallTrees, DataFrames, etc.) behind explicit opt-in.
Adds hardened, budgeted model metadata/path handling (canonical containment, link resolution, bounded metadata enumeration and decoding budgets, recursive model context) for Pipeline/ComplexParams loads.
Migrates DataTypeParam persistence toward bounded JSON handling and adds/updates extensive regression coverage for security and compatibility.

Show a summary per file

File	Description
reviews/serializer-deserialization/task-2513-attempt-1-review-1-gpt-5.6-sol.md	Adds a long-form review artifact documenting the hardening work and verification steps.
lightgbm/src/test/scala/com/microsoft/azure/synapse/ml/lightgbm/params/VerifyLightGBMBoosterParam.scala	Adds a regression ensuring LightGBM booster param loads under constrained legacy handling.
lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/params/LightGBMBoosterParam.scala	Adds a narrow deserialization class policy for `LightGBMBoosterParam`.
core/src/test/scala/org/apache/spark/ml/VerifyArtifactPathResolver.scala	Adds tests for metadata listing guarantees and Java 8 linkage constraints.
core/src/test/scala/com/microsoft/azure/synapse/ml/stages/UDFTransformerSuite.scala	Adds trust-gating tests for persisted UDFs and session propagation through PipelineSerializer.
core/src/test/scala/com/microsoft/azure/synapse/ml/stages/LambdaSuite.scala	Enables trusted legacy load path for serialization fuzzing where needed.
core/src/test/scala/com/microsoft/azure/synapse/ml/recommendation/RankingTrainValidationSpec.scala	Enables trusted legacy load path for serialization fuzzing in ranking suites.
core/src/test/scala/com/microsoft/azure/synapse/ml/param/VerifyEvaluatorParam.scala	Adds explicit-trust tests for evaluator param persistence.
core/src/test/scala/com/microsoft/azure/synapse/ml/param/VerifyEstimatorArrayParam.scala	Adds coverage for stage-array persistence (safe writable vs trusted legacy cases).
core/src/test/scala/com/microsoft/azure/synapse/ml/param/VerifyDataTypeParam.scala	Adds coverage for JSON DataType persistence and legacy-stream gating/tripwires.
core/src/test/scala/com/microsoft/azure/synapse/ml/param/VerifyDataFrameParam.scala	Adds regression guarding against linked Parquet parts escaping artifact containment.
core/src/test/scala/com/microsoft/azure/synapse/ml/param/VerifyArrayParamMapParam.scala	Adds explicit-trust coverage for `ArrayParamMapParam` persistence.
core/src/test/scala/com/microsoft/azure/synapse/ml/nn/VerifySchemas.scala	Extends SafeObjectInputStream/BallTree coverage and validates fail-closed NN loading defaults.
core/src/test/scala/com/microsoft/azure/synapse/ml/io/split1/ParserSuite.scala	Enables trusted legacy load path for serialization fuzzing in parser suites where applicable.
core/src/test/scala/com/microsoft/azure/synapse/ml/core/utils/VerifySafeObjectInputStream.scala	Adds focused unit tests for SafeObjectInputStream resource and policy enforcement.
core/src/test/scala/com/microsoft/azure/synapse/ml/core/test/fuzzing/Fuzzing.scala	Adds a controlled trusted-legacy mode for serialization fuzzing and Pipeline/PipelineModel round trips.
core/src/test/scala/com/microsoft/azure/synapse/ml/core/serialize/VerifyModelLoadEnvironment.scala	Adds environment regressions around session scoping, trusted loading, and provider behaviors.
core/src/test/scala/com/microsoft/azure/synapse/ml/core/serialize/VerifyMetadataBudgets.scala	Adds regression ensuring aggregate decoded-metadata budgets are enforced.
core/src/test/scala/com/microsoft/azure/synapse/ml/core/serialize/ValidateComplexParamSerializer.scala	Significantly expands hardening regressions across metadata, paths, trust gates, and reader safety.
core/src/test/scala/com/microsoft/azure/synapse/ml/automl/VerifyFindBestModel.scala	Enables trusted legacy load path for serialization fuzzing in AutoML persistence tests.
core/src/main/scala/org/apache/spark/ml/StageReaderInspector.scala	Adds bytecode-based reader classification without class initialization.
core/src/main/scala/org/apache/spark/ml/Serializer.scala	Centralizes hardened serializer routing, trust scoping, Pipeline serializers, and safe read/write behavior.
core/src/main/scala/org/apache/spark/ml/ModelLoadContext.scala	Introduces shared load/write budgets (nodes, depth, metadata physical/decoded) across nested model loads.
core/src/main/scala/org/apache/spark/ml/DataTypeSerializer.scala	Implements bounded DataType JSON persistence with legacy-stream detection and UDT gating.
core/src/main/scala/org/apache/spark/ml/ComplexParamsSerializer.scala	Hardens ComplexParams read/write (session assignment, budgets, metadata accounting, and native pipeline boundaries).
core/src/main/scala/org/apache/spark/ml/ArtifactPathResolver.scala	Adds canonical containment, link resolution, bounded metadata enumeration/decoding, and safe directory validation helpers.
core/src/main/scala/com/microsoft/azure/synapse/ml/param/TransformerArrayParam.scala	Persists transformer arrays via hardened PipelineArraySerializer and validates load-time types.
core/src/main/scala/com/microsoft/azure/synapse/ml/param/EstimatorArrayParam.scala	Persists estimator arrays via hardened PipelineArraySerializer and validates load-time types.
core/src/main/scala/com/microsoft/azure/synapse/ml/param/DataTypeParam.scala	Adjusts DataTypeParam imports to align with new DataTypeSerializer path.
core/src/main/scala/com/microsoft/azure/synapse/ml/param/DataFrameParam.scala	Marks DataFrameParam as not supporting untrusted deserialization by default.
core/src/main/scala/com/microsoft/azure/synapse/ml/param/ByteArrayParam.scala	Adds an explicit deserialization class policy (primitive byte arrays).
core/src/main/scala/com/microsoft/azure/synapse/ml/param/BallTreeParam.scala	Marks BallTree params as requiring trusted loading (fail closed by default).
core/src/main/scala/com/microsoft/azure/synapse/ml/nn/BallTree.scala	Changes ConditionalBallTree default load to fail closed and adds explicit `loadUnsafe`.
core/src/main/scala/com/microsoft/azure/synapse/ml/core/utils/SafeObjectInputStream.scala	Expands SafeObjectInputStream with resource limits, filter composition, and class allowlisting model.
core/src/main/scala/com/microsoft/azure/synapse/ml/core/serialize/ComplexParam.scala	Adds per-param deserialization policy hooks and trust gating for unsafe complex params.

Review details

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Files reviewed: 34/35 changed files
Comments generated: 2
Review effort level: Lite

## Summary Clarify class-policy rejection messaging, remove machine-local paths from the review evidence, and isolate new UDF test imports so the security patch replays cleanly onto the spark4.1 compatibility branch. ## Prompting Intent Resolve every current-head pull-request comment and the release-branch compatibility failure without weakening the deserialization policy or changing branch-specific Spark 4.1 UDF behavior. ## Linked Sources - Pull request: microsoft#2678 - Class-policy feedback: microsoft#2678 (comment) - Review-artifact feedback: microsoft#2678 (comment) - Failed compatibility build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=232896880 ## Rationale The class policy supports both exact names and package prefixes, so the error must describe the combined policy. Repo-relative evidence paths avoid leaking workstation details. Moving imports into the added tests keeps the functional master change intact while avoiding overlap with spark4.1's branch-only UDF helper object; a local three-way replay confirmed the complete patch applies cleanly to the current spark4.1 tip. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) · 2026-08-26T10:23:48Z

/azp run

azure-pipelines · 2026-08-26T10:24:00Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Review details

Files reviewed: 35/35 changed files
Comments generated: 3
Review effort level: Lite

## Summary Express Spark text framing from its explicit newline bytes and rename two persistence tests so their names identify the parameter type under test. ## Prompting Intent Resolve the current-head review comments precisely while preserving Spark 3.5's actual text-output framing semantics and keeping the security boundary tests understandable across master and spark4.1. ## Linked Sources - Pull request: microsoft#2678 - Framing review: microsoft#2678 (comment) - Evaluator test review: microsoft#2678 (comment) - ParamMap test review: microsoft#2678 (comment) - Spark 3.5 TextOptions: https://github.com/apache/spark/blob/v3.5.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextOptions.scala ## Rationale Spark's text writer does not use the platform line separator by default; its TextOptions contract explicitly uses UTF-8 newline bytes. Computing the byte length from that literal documents the dependency without introducing incorrect Windows-specific accounting. Accurate test names improve failure diagnostics without changing coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) · 2026-08-26T10:40:52Z

/azp run

azure-pipelines · 2026-08-26T10:41:06Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

core/src/main/scala/org/apache/spark/ml/Serializer.scala:88

typeToSerializer routes any Array[_] whose element type is a subtype of PipelineStage to PipelineArraySerializer and then casts it to Serializer[T]. Since arrays are reified on the JVM, PipelineArraySerializer will return an Array[PipelineStage], which cannot be safely treated as (for example) Array[Estimator[_]]/Array[Transformer] and can produce ClassCastException for any caller that goes through Serializer.typeToSerializer (outside the specialized Param overrides).

Consider returning a small adapter serializer for pipeline-stage arrays that converts to/from Array[PipelineStage] while preserving the requested runtime component type.

  def typeToSerializer[T](
      tpe: Type,
      sparkSession: SparkSession,
      classFilter: Option[DeserializationClassFilter]): Serializer[T] = {
    (if (tpe <:< typeOf[PipelineStage])              new PipelineSerializer(sparkSession)
     else if (isPipelineStageArray(tpe))             new PipelineArraySerializer(sparkSession)

core/src/main/scala/com/microsoft/azure/synapse/ml/core/utils/SafeObjectInputStream.scala:218

SafeObjectInputStream.ResourceFilter is a shared singleton instance of DeserializationResourceFilter, but DeserializationResourceFilter is stateful (declaredArrayBytes accumulates across checkInput calls). This makes the exposed filter instance easy to misuse and can make future tests order-dependent if they call ResourceFilter.checkInput on multiple “allowed” inputs.

Prefer exposing a factory (e.g., def newResourceFilter(...)) that returns a fresh per-stream filter, and update VerifySafeObjectInputStream to construct a new instance for assertions.

  private[utils] val ResourceFilter: ObjectInputFilter =
    new DeserializationResourceFilter(defaultResourceLimits)

Files reviewed: 35/35 changed files
Comments generated: 0 new
Review effort level: Lite

## Summary Preserve the requested JVM component type when generic serializers load PipelineStage arrays, provide fresh stateful resource filters per stream, and disambiguate Spark Transformer from Scala reflection for Spark 4.1 builds. ## Prompting Intent Resolve all suppressed current-head review findings and the spark4.1 compatibility compile failure without weakening the secure Pipeline serializer or its model-wide resource controls. ## Linked Sources - Pull request: microsoft#2678 - Suppressed current-head review body: microsoft#2678 - Failed spark4.1 build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=232900732 ## Rationale JVM arrays are reified, so returning `Array[PipelineStage]` through a generic `Serializer[Array[Estimator[_]]]` cast is not type safe. A small adapter now validates elements and allocates the exact requested component array. Resource filters track aggregate array bytes and therefore must never be shared across streams. Fully qualifying Spark's Transformer avoids a Scala 2.13 reflection name collision while remaining source-compatible with the master baseline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) · 2026-08-26T11:19:57Z

Resolved both suppressed findings from the 9aa866acbd automated review in 0e360a5827:

Generic PipelineStage-array serializers now allocate and validate the requested reified component type, with direct Array[Estimator[_]] and Array[Transformer] round-trip coverage.
DeserializationResourceFilter is now created fresh per stream/test instead of exposing a shared stateful singleton.
The same commit fully qualifies Spark Transformer, and the complete patch now passes spark4.1 core/Test/compile locally under Java 17 / Scala 2.13.

Master core compilation, main/test Scalastyle, VerifyEstimatorArrayParam, VerifySafeObjectInputStream, and ValidateComplexParamSerializer all pass.

Rana Singh (ranadeepsingh) · 2026-08-26T11:19:57Z

/azp run

azure-pipelines · 2026-08-26T11:20:10Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Review details

Files reviewed: 36/36 changed files
Comments generated: 1
Review effort level: Lite

## Summary Use overflow-safe multiplication and addition for declared array byte accounting, rejecting arithmetic overflow before it can weaken the deserialization resource budget. Add regressions for both per-array multiplication and aggregate addition overflow. ## Prompting Intent Resolve the current-head automated review finding on the model deserialization hardening PR while preserving fail-closed JEP 290 resource enforcement and cross-version compatibility. ## Linked Sources - Pull request: microsoft#2678 - Review comment: microsoft#2678 (comment) - Prior remediation: microsoft#2513 ## Rationale Exact arithmetic makes overflow an explicit rejection instead of allowing signed Long wraparound. This keeps the stateful aggregate budget monotonic and avoids saturation logic that could obscure malformed stream accounting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) · 2026-08-26T11:35:23Z

/azp run

Rana Singh (ranadeepsingh) · 2026-09-02T13:49:50Z

/azp run

azure-pipelines · 2026-09-02T13:50:06Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Copilot review overview

🟡 Changes recommended

A couple of new “constrained graph” tests can be masked by the test-default legacy deserialization opt-in, so they don’t currently prove the allowlists are sufficient without unsafe fallback.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 2

New issues introduced by this change (2)

Severity	Finding
	`core/src/test/scala/com/microsoft/azure/synapse/ml/param/VerifyDataTypeParam.scala` — This test currently runs with spark.synapseml.legacy.allowUnsafeJavaDeserialization enabled by…
	`lightgbm/src/test/scala/com/microsoft/azure/synapse/ml/lightgbm/split6/VerifyLightGBMBoosterParam.scala` — This test runs with spark.synapseml.legacy.allowUnsafeJavaDeserialization enabled by default via…

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

core/src/main/scala/com/microsoft/azure/synapse/ml/core/utils/SafeObjectInputStream.scala:23

The Scaladoc says classes must be "explicitly allowed", but this stream also permits classes via allowedPrefixes; the wording is misleading for the new DeserializationClassFilter behavior.

## Summary Force standard DataType and LightGBM booster round trips through their class policies with the trusted-legacy fallback disabled. Clarify that SafeObjectInputStream accepts both exact names and configured prefixes. ## Prompting Intent Resolve every finding from the automated review of a517ca6 while preserving the smallest IcM fix, the fail-closed production default, and cross-version test coverage. ## Linked Sources - DataType policy finding: microsoft#2678 (comment) - LightGBM policy finding: microsoft#2678 (comment) - Current pull request: microsoft#2678 - IcM incident: https://portal.microsofticm.com/imp/v5/incidents/details/31000000568481/msrc ## Rationale The global test-fixture opt-in is necessary for open-ended repository fixtures, but policy-specific tests must not inherit it or an incomplete allowlist could pass through unsafe fallback. Scoped config restoration proves the constrained paths directly without changing production behavior or duplicating allowlist logic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) · 2026-09-02T14:32:53Z

Addressed the exact-head suppressed finding in 0957337: SafeObjectInputStream documentation now accurately describes both exact-name and prefix policies. No production behavior changed.

Rana Singh (ranadeepsingh) · 2026-09-02T14:33:06Z

/azp run

azure-pipelines · 2026-09-02T14:33:20Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Copilot review overview

🔵 Needs a closer look

Serializer.readFromHDFS(..., classFilter) wraps all InvalidClassExceptions as security-policy violations, which can misreport ordinary compatibility failures (e.g., serialVersionUID mismatch) and should be differentiated.

Review tier: Lite
Findings: None

Issues resolved since last review (2)

Severity	Finding
	`lightgbm/src/test/scala/com/microsoft/azure/synapse/ml/lightgbm/split6/VerifyLightGBMBoosterParam.scala` — This test runs with spark.synapseml.legacy.allowUnsafeJavaDeserialization enabled by default via… View resolved comment
	`core/src/test/scala/com/microsoft/azure/synapse/ml/param/VerifyDataTypeParam.scala` — This test currently runs with spark.synapseml.legacy.allowUnsafeJavaDeserialization enabled by… View resolved comment

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

core/src/main/scala/org/apache/spark/ml/Serializer.scala:200

readFromHDFS(spark, path, classFilter) treats every InvalidClassException as a class-policy rejection and wraps it as a SecurityException. InvalidClassException is also thrown for non-security reasons (e.g., serialVersionUID mismatch / local class incompatible), so this can misreport ordinary compatibility failures as a security-policy violation.

Rana Singh (ranadeepsingh) · 2026-09-02T15:29:08Z

Addressed the exact-head suppressed compatibility finding in 77dbc60. SafeObjectInputStream now throws a dedicated InvalidClassException subtype for policy rejections, and Serializer translates only that subtype. Ordinary incompatibilities such as serialVersionUID mismatches remain InvalidClassException; the new HDFS-path regression passes on master, Spark 4.0, and Spark 4.1.

Rana Singh (ranadeepsingh) · 2026-09-02T15:29:12Z

/azp run

azure-pipelines · 2026-09-02T15:29:25Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Copilot review overview

🔵 Needs a closer look

It changes core serialization/deserialization behavior and security-critical defaults, warranting final human review despite strong targeted test coverage.

Review tier: Lite
Findings: None

Suppressed comments (1)

core/src/main/scala/com/microsoft/azure/synapse/ml/core/utils/SafeObjectInputStream.scala:80

SafeObjectInputStream now supports both exact class allowlisting (allowedClasses) and prefix allowlisting (allowedPrefixes), but the rejection message still claims "Only classes with approved package prefixes may be deserialized." This is misleading when a filter uses exact class names without prefixes (e.g., LightGBMBoosterParam).

    if (!allowed) {
      throw new DeserializationClassRejectedException(
        className,
        "Deserialization of this class is not allowed. " +
          "Only classes with approved package prefixes may be deserialized."
      )

Rana Singh (ranadeepsingh) · 2026-09-02T15:46:36Z

Resolved the final exact-head wording finding in 05cc9e3: the rejection now describes the configured exact-name or prefix policy, matching both supported allowlist forms. This is diagnostic-only; no deserialization behavior changed.

Rana Singh (ranadeepsingh) · 2026-09-02T15:46:40Z

/azp run

azure-pipelines · 2026-09-02T15:46:53Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Copilot review overview

🔵 Needs a closer look

It changes core model persistence deserialization behavior in security-sensitive code paths, so it warrants final human verification for compatibility and regression risk.

Review tier: Lite
Findings: None

Rana Singh (ranadeepsingh) · 2026-09-02T17:58:59Z

Full CI triage for final head 05cc9e3d3f:

Azure build 234173105 ran from refs/pull/2678/merge and reached every affected path. The deserialization suite passed 7/7, FuzzingTest passed 10/10, VerifyDataTypeParam passed 11/11, VerifyLightGBMBoosterParam passed within a 35/35 shard, and the public UDF persistence suite passed within a 240/240 shard.
The 14 failing leaf jobs all also fail on exact-target build 234106471 for master SHA 1db3c40b1a (that target build additionally failed Databricks CPU shard 3).
Representative logs show 403 Forbidden from the Face API, null/empty responses across unrelated Cognitive suites, generated Python/R fixture failure downstream of that Face 403, and the same Databricks “Understand and Search Forms” notebook failure. The trusted-fixture Spark setting is present, and no failed job reports a deserialization-policy rejection.

These are reproduced target-baseline/service failures, so changing the serializer patch would not address them. The Azure aggregate remains the sole engineering blocker; once the shared service/credential issue is repaired, this head needs a fresh /azp run. Exact-head Copilot review reports no findings or suppressed comments.

## Summary Fail closed at the remaining unrestricted Java object deserialization boundary used by ComplexParam persistence. Add per-type class policies for byte arrays, Spark data types, and LightGBM boosters while preserving the existing on-disk format and providing an explicit trusted-legacy opt-in for unconstrained model graphs. ## Prompting Intent Investigate the remaining MSRC recommendation after the BallTree fix in microsoft#2513, apply the smallest complete mitigation without the earlier broad persistence redesign, preserve compatibility wherever security permits, and prove the change through focused regressions, supported Spark branches, and the SynapseML PR-readiness loop. ## Linked Sources - Prior BallTree mitigation: microsoft#2513 - Follow-up pull request: microsoft#2678 ## Rationale Use the MSRC-recommended per-type allowlist approach at Serializer.read rather than changing model formats, signing artifacts, or replacing native Pipeline persistence. This keeps existing serialized bytes readable across versions and limits the patch to the vulnerable boundary. Types whose graphs are inherently open-ended, such as UDF closures, remain loadable only through an explicit trusted-legacy switch because silently deserializing them would preserve the reported code-execution risk. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Document why Scala ModuleSerializationProxy targets remain subject to the active deserialization class policy and add a regression proving a disallowed module is rejected before its initializer runs. ## Prompting Intent Resolve the exact-head automated review findings that questioned whether ModuleSerializationProxy could bypass the new ComplexParam class filters, verify the claim against the Scala 2.13 runtime used by the supported Spark 4 branches, and fix or disprove it with executable evidence. ## Linked Sources - LightGBM review thread: microsoft#2678 (comment) - SafeObjectInputStream review thread: microsoft#2678 (comment) - Pull request: microsoft#2678 ## Rationale Scala 2.13.17 stores the target in ModuleSerializationProxy as Class<?> rather than as an unchecked module-name string. ObjectInputStream resolves that class descriptor through SafeObjectInputStream before readResolve can access MODULE$. The regression constructs the proxy reflectively, grants only the proxy class, and verifies the referenced module class is rejected without initialization. Keeping this validated path preserves legitimate Scala singleton values used by Spark data types and LightGBM while still enforcing the per-type policy. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Make the deserialization error message accurate for both missing policies and rejected classes, and move the persisted-UDF public-path regression into a standalone suite that replays cleanly onto supported Spark 4 branches. ## Prompting Intent Address the suppressed exact-head review finding, triage the failed release compatibility check from its logs, preserve the same public UDF migration proof, and keep the MSRC fix small rather than changing branch-specific production behavior. ## Linked Sources - Pull request: microsoft#2678 - Failed compatibility run: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=233926848 ## Rationale The prior exception text claimed every failure lacked a constraint even when a class had been rejected by an active policy. A neutral base message lets each call site supply the accurate reason. The Spark 4.1 job failed before compile because its branch-specific UDF suite intentionally hoists lambdas for Scala 2.13; placing the new persistence regression in its own suite removes that test-only replay conflict without changing coverage or production scope. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Opt trusted repository-generated Scala, Python, and R fixtures into the explicit legacy deserialization path while keeping production fail-closed behavior unchanged. Keep exploit regressions in safe mode and replace the discoverable test-only ML stage with a plain Params holder. ## Prompting Intent Address the failed full CI run for the smallest possible IcM remediation, without expanding the production change, altering persisted formats, or weakening the secure default. Run the SynapseML PR-readiness loop and preserve focused exploit coverage. ## Linked Sources - IcM incident: https://portal.microsofticm.com/imp/v5/incidents/details/31000000568481/msrc - Initial remediation: microsoft#2513 - Current pull request: microsoft#2678 - Failure evidence: Azure Pipelines build 234131717 linked from PR microsoft#2678 ## Rationale Central test-harness opt-in covers fixtures created and controlled by the repository across Scala and generated language tests, avoiding suite-by-suite exceptions. Dedicated security tests explicitly disable the switch before loading crafted payloads, so the exploit contract remains enforced. This is smaller and safer than broadening production allowlists for arbitrary closure graphs or enabling unsafe deserialization by default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Force standard DataType and LightGBM booster round trips through their class policies with the trusted-legacy fallback disabled. Clarify that SafeObjectInputStream accepts both exact names and configured prefixes. ## Prompting Intent Resolve every finding from the automated review of a517ca6 while preserving the smallest IcM fix, the fail-closed production default, and cross-version test coverage. ## Linked Sources - DataType policy finding: microsoft#2678 (comment) - LightGBM policy finding: microsoft#2678 (comment) - Current pull request: microsoft#2678 - IcM incident: https://portal.microsofticm.com/imp/v5/incidents/details/31000000568481/msrc ## Rationale The global test-fixture opt-in is necessary for open-ended repository fixtures, but policy-specific tests must not inherit it or an incomplete allowlist could pass through unsafe fallback. Scoped config restoration proves the constrained paths directly without changing production behavior or duplicating allowlist logic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Introduce a dedicated InvalidClassException subtype for deserialization policy rejections and translate only that subtype into security guidance. Preserve ordinary Java compatibility failures such as serialVersionUID mismatches, with a regression through the HDFS serializer path. ## Prompting Intent Address the suppressed finding from the automated review of 0957337, exhaust review feedback, and keep the IcM remediation accurate without expanding its scope or weakening the safe default. ## Linked Sources - Current pull request: microsoft#2678 - Reviewed head: microsoft@0957337 - IcM incident: https://portal.microsofticm.com/imp/v5/incidents/details/31000000568481/msrc ## Rationale InvalidClassException is raised both by the allowlist and by Java serialization compatibility checks. A typed policy exception preserves existing InvalidClassException behavior for direct stream callers while letting Serializer distinguish policy failures without brittle message matching. It also avoids an unsafe retry for incompatibilities that fallback cannot repair. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Update the SafeObjectInputStream rejection message to describe both exact-name and prefix policies. ## Prompting Intent Resolve the final suppressed wording finding from the automated review of 77dbc60 without changing behavior or expanding the IcM remediation. ## Linked Sources - Current pull request: microsoft#2678 - Reviewed head: microsoft@77dbc60 - IcM incident: https://portal.microsofticm.com/imp/v5/incidents/details/31000000568481/msrc ## Rationale LightGBM uses exact class names while other policies use prefixes, so the error should describe the configured policy generically. This keeps diagnostics accurate with no runtime or compatibility change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot review overview

🔵 Needs a closer look

It changes a security-critical, core persistence/deserialization path with broad compatibility surface area that warrants final human review despite strong targeted tests.

Review tier: Lite
Findings: None

Rana Singh (ranadeepsingh) · 2026-09-04T03:41:16Z

/azp run

azure-pipelines · 2026-09-04T03:41:29Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Rana Singh (ranadeepsingh) requested a review from Brendan Walsh (BrendanWalsh) as a code owner August 26, 2026 10:04

Copilot AI lite review requested due to automatic review settings August 26, 2026 10:04

Copilot started reviewing on behalf of Rana Singh (ranadeepsingh) August 26, 2026 10:05 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

Comment thread core/src/main/scala/com/microsoft/azure/synapse/ml/core/utils/SafeObjectInputStream.scala Outdated

Comment thread reviews/serializer-deserialization/task-2513-attempt-1-review-1-gpt-5.6-sol.md Outdated

Copilot AI review requested due to automatic review settings August 26, 2026 10:23

Copilot started reviewing on behalf of Rana Singh (ranadeepsingh) August 26, 2026 10:23 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 26, 2026 10:40

Copilot started reviewing on behalf of Rana Singh (ranadeepsingh) August 26, 2026 10:40 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 26, 2026 11:19

Copilot started reviewing on behalf of Rana Singh (ranadeepsingh) August 26, 2026 11:20 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

Comment thread core/src/main/scala/com/microsoft/azure/synapse/ml/core/utils/SafeObjectInputStream.scala Outdated

Copilot AI review requested due to automatic review settings August 26, 2026 11:35

Copilot started reviewing on behalf of Rana Singh (ranadeepsingh) September 2, 2026 13:50 View session

Copilot AI reviewed Sep 2, 2026

View reviewed changes

Comment thread core/src/test/scala/com/microsoft/azure/synapse/ml/param/VerifyDataTypeParam.scala

Comment thread ...c/test/scala/com/microsoft/azure/synapse/ml/lightgbm/split6/VerifyLightGBMBoosterParam.scala

Copilot AI reviewed Sep 2, 2026

View reviewed changes

Rana Singh (ranadeepsingh) mentioned this pull request Sep 3, 2026

fix(ci): repair publishing and Cognitive service test credentials #2691

Merged

Rana Singh (ranadeepsingh) and others added 7 commits September 3, 2026 19:43

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Brendan Walsh (BrendanWalsh) approved these changes Sep 4, 2026

View reviewed changes

Uh oh!

Conversation

Rana Singh (ranadeepsingh) commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related issues and context

What does this pull request change?

Behavior and compatibility

What to review

How was this patch validated?

Does this PR change any dependencies?

Does this PR add a new user-facing feature?

Uh oh!

azure-pipelines Bot commented Aug 26, 2026

Uh oh!

github-actions Bot commented Aug 26, 2026

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 26, 2026

Uh oh!

azure-pipelines Bot commented Aug 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 26, 2026

Uh oh!

azure-pipelines Bot commented Aug 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 26, 2026

Uh oh!

azure-pipelines Bot commented Aug 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 26, 2026

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 26, 2026

Uh oh!

azure-pipelines Bot commented Aug 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 26, 2026

Uh oh!

Rana Singh (ranadeepsingh) commented Sep 2, 2026

Uh oh!

azure-pipelines Bot commented Sep 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Rana Singh (ranadeepsingh) commented Sep 2, 2026

Uh oh!

Rana Singh (ranadeepsingh) commented Sep 2, 2026

Uh oh!

azure-pipelines Bot commented Sep 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Rana Singh (ranadeepsingh) commented Aug 26, 2026 •

edited

Loading