Raise the datamodel-code-generator floor for the codegen toolchain - #70670

potiuk · 2026-07-29T09:21:18Z

The codegen dependency groups in task-sdk and airflow-ctl pinned
datamodel-code-generator==0.41.0. A number of published advisories affect
releases in that range; the highest fix floor among them is 0.64.0, so this
requires >=0.71.0 to resolve clear of all of them with headroom rather than
landing on the boundary release.

Regenerating the models with the newer generator changes their shape: it no
longer emits a spurious = None default for fields the OpenAPI spec lists as
required but nullable. The regenerated models now match the spec instead of
silently accepting payloads that omit those fields.

No Cadwyn migration is involved. No datamodel, route, or spec file is
touched — the server-side contract is byte-identical. This is purely a change
in how the generator renders the same spec. The client pins
airflow-api-version, so Cadwyn shapes every response to the version the
client was generated against.

Test fixtures that relied on the old leniency now pass those fields
explicitly. Two of them were silently dropping a value by passing schema_=
instead of the schema alias — the old implicit default masked the bug.

The supervisor schema snapshot (schema/schema.json) is regenerated as a
consequence, since the comms models inherit from the generated datamodels.

Supersedes the datamodel-code-generator half of #70660, which bumps the same
pin but does not regenerate the models and so does not carry the fixture
updates that change requires.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

The codegen dependency groups pinned datamodel-code-generator==0.41.0. A number of published advisories affect releases in that range; the highest fix floor among them is 0.64.0, so require >=0.71.0 to resolve clear of all of them with headroom rather than landing on the boundary release. The newer generator no longer emits a spurious "= None" default for fields the OpenAPI spec lists as required but nullable, so the regenerated models now match the spec instead of silently accepting payloads that omit those fields. This is a client-side rendering change only -- no schema, route, or API version is touched, so no Cadwyn migration is involved. Test fixtures that relied on the old leniency now pass those fields explicitly; two of them were silently dropping a value by passing "schema_=" rather than the "schema" alias.

The regenerated models drop the implicit `= None` for spec-required nullable fields, so every construction has to name them. Three test call sites and the TypeScript SDK's generated types were still relying on the old leniency. The fernet-key rotation test is a third instance of the `schema_=` mistake the generator's leniency used to hide: the value was being dropped, not passed. `ts-sdk/src/generated/supervisor.ts` derives from the same supervisor schema snapshot, so it moves with it.

potiuk · 2026-07-29T16:14:24Z

Pushed 4fa3fcf,
which clears the CI failures from the model regeneration. The generated models no longer carry a
spurious = None for fields the spec marks required but nullable, so the remaining call sites
have to name them:

test_rotate_fernet_key_command.py — schema_= → schema=. A third instance of the alias
bug already fixed twice in this PR: the value was being dropped, and the old implicit default
hid it.
providers/amazon/.../test_base_aws.py — names the six now-required ConnectionResult fields.
providers/openlineage/.../test_listener.py — see below.
ts-sdk/src/generated/supervisor.ts — regenerated; it derives from the same supervisor schema
snapshot this PR already regenerates.

cc @mobuchowski @kacpermuda for the OpenLineage change — 26 tests in
TestOpenLineageListenerAirflow3 were failing on SdkDagRun validation. Two points worth a look:

data_interval_start / data_interval_end are passed as None, not datetimes. The
runtime-TI tests assert nominal_start_time=None / nominal_end_time=None, so a real
datetime would trade one failure for another. None reproduces exactly what the old implicit
default produced.
The three new keys (data_interval_start, data_interval_end, partition_key) go through
the same if field in SdkDagRun.model_fields guard the existing state key already used, so
the file keeps working against the older SDKs the provider supports (apache-airflow>=2.11.0).

No production code changed here — test fixtures plus one generated file.

The Helm tests / Release Helm failure is unrelated to this PR: chart/Chart.yaml pulls
postgresql from the retired charts.bitnami.com/bitnami index, so it fails on any PR that
runs helm tests.

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

github-actions · 2026-07-30T06:31:32Z

Backport failed to create: airflow-ctl/v0-1-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	airflow-ctl/v0-1-test

You can attempt to backport this manually by running:

cherry_picker b301429 airflow-ctl/v0-1-test

This should apply the commit to the airflow-ctl/v0-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

Ruff 0.16 widened its default rule set from ~250 to ~510 enabled rules. Airflow's config uses extend-select, which layers on top of whatever Ruff currently enables by default, so the bump would have silently pulled in 261 unreviewed rules and produced sweeping autofix churn across the repo. Pin `select` to the rule groups Airflow has always effectively had enabled, so future Ruff releases cannot change our lint surface out from under us; everything intentionally on is still listed in extend-select as before. Two rules remain newly in scope under the pin, because they sit inside sections Airflow selects wholesale. Both are deferred behind `ignore` rather than fixed here: a version bump should not also carry semantic changes, and both need per-site judgement that deserves its own reviewed change. ISC004's autofix is unsafe, and LOG004's call sites can only be classified by tracing whether an exception is genuinely in flight at each one. Ruff 0.16 also started formatting Python blocks inside Markdown, which is why a task-sdk doc changes alongside the bump rather than in its own commit: its example was a bare statement that the formatter rewrites into a one-element tuple, which would have changed what the example teaches. Showing the entry in its surrounding context keeps it both correct and stable to format. The lockfile additionally picks up datamodel-code-generator 0.71.0. That is not part of this change: apache#70670 raised the declared floor to >=0.71.0 without regenerating uv.lock, leaving the lock pinned at a version its own constraint excludes, so any `uv lock` has to correct it.

…l test The regenerated model exposes the field as `schema_` with `Field(alias="schema")` and no `populate_by_name`, so pydantic requires the alias `schema=` at construction. Matches main's fix (#70670): construct with `schema=` plus `# type: ignore[call-arg]`.

ConnectionResponse.schema_ has Field(alias=schema) and no populate_by_name, so the source validates responses by the schema alias. The mock handlers dumped the model by field name, breaking the round-trip. Dump connection_response / connections_response mocks with by_alias=True (matches main #70670).

…toolchain (#70670) (#70976) * Raise the datamodel-code-generator floor for the codegen toolchain (#70670) (cherry picked from commit b301429) Manual backport: datamodel-code-generator floor bumped 0.33.0 -> >=0.71.0 and models REGENERATED against v3-3-test's own spec (not copied from main). Test constructions in test_operations.py updated for the now-required nullable fields. test_task_command.py is main-only; main's divergent test_operations.py tail (ctl task-instance ops not on this branch) intentionally not backported. * Fix ConnectionResponse construction to use schema alias in airflow-ctl test The regenerated model exposes the field as `schema_` with `Field(alias="schema")` and no `populate_by_name`, so pydantic requires the alias `schema=` at construction. Matches main's fix (#70670): construct with `schema=` plus `# type: ignore[call-arg]`. * Serialize connection mock responses by alias in airflow-ctl tests ConnectionResponse.schema_ has Field(alias=schema) and no populate_by_name, so the source validates responses by the schema alias. The mock handlers dumped the model by field name, breaking the round-trip. Dump connection_response / connections_response mocks with by_alias=True (matches main #70670). * Regenerate airflow-ctl datamodels after merging v3-3-test The #70127/#69719 reverts (now on v3-3-test) removed BackfillDagRunResponse, BackfillDagRunCollectionResponse and bundle_version from the API spec. Regenerated generated.py from the current spec with datamodel-code-generator>=0.71.0 so it reflects the reverts in the new-tool format. mypy airflow-ctl clean; no tests reference the removed models.

Ruff 0.16 widened its default rule set from ~250 to ~510 enabled rules. Airflow's config uses extend-select, which layers on top of whatever Ruff currently enables by default, so the bump would have silently pulled in 261 unreviewed rules and produced sweeping autofix churn across the repo. Pin `select` to the rule groups Airflow has always effectively had enabled, so future Ruff releases cannot change our lint surface out from under us; everything intentionally on is still listed in extend-select as before. Two rules remain newly in scope under the pin, because they sit inside sections Airflow selects wholesale. Both are deferred behind `ignore` rather than fixed here: a version bump should not also carry semantic changes, and both need per-site judgement that deserves its own reviewed change. ISC004's autofix is unsafe, and LOG004's call sites can only be classified by tracing whether an exception is genuinely in flight at each one. Ruff 0.16 also started formatting Python blocks inside Markdown, which is why a task-sdk doc changes alongside the bump rather than in its own commit: its example was a bare statement that the formatter rewrites into a one-element tuple, which would have changed what the example teaches. Showing the entry in its surrounding context keeps it both correct and stable to format. The lockfile additionally picks up datamodel-code-generator 0.71.0. That is not part of this change: apache#70670 raised the declared floor to >=0.71.0 without regenerating uv.lock, leaving the lock pinned at a version its own constraint excludes, so any `uv lock` has to correct it.

…toolchain (#70670) (#70976) * Raise the datamodel-code-generator floor for the codegen toolchain (#70670) (cherry picked from commit b301429) Manual backport: datamodel-code-generator floor bumped 0.33.0 -> >=0.71.0 and models REGENERATED against v3-3-test's own spec (not copied from main). Test constructions in test_operations.py updated for the now-required nullable fields. test_task_command.py is main-only; main's divergent test_operations.py tail (ctl task-instance ops not on this branch) intentionally not backported. * Fix ConnectionResponse construction to use schema alias in airflow-ctl test The regenerated model exposes the field as `schema_` with `Field(alias="schema")` and no `populate_by_name`, so pydantic requires the alias `schema=` at construction. Matches main's fix (#70670): construct with `schema=` plus `# type: ignore[call-arg]`. * Serialize connection mock responses by alias in airflow-ctl tests ConnectionResponse.schema_ has Field(alias=schema) and no populate_by_name, so the source validates responses by the schema alias. The mock handlers dumped the model by field name, breaking the round-trip. Dump connection_response / connections_response mocks with by_alias=True (matches main #70670). * Regenerate airflow-ctl datamodels after merging v3-3-test The #70127/#69719 reverts (now on v3-3-test) removed BackfillDagRunResponse, BackfillDagRunCollectionResponse and bundle_version from the API spec. Regenerated generated.py from the current spec with datamodel-code-generator>=0.71.0 so it reflects the reverts in the new-tool format. mypy airflow-ctl clean; no tests reference the removed models.

Ruff 0.16 widened its default rule set from ~250 to ~510 enabled rules. Airflow's config uses extend-select, which layers on top of whatever Ruff currently enables by default, so the bump would have silently pulled in 261 unreviewed rules and produced sweeping autofix churn across the repo. Pin `select` to the rule groups Airflow has always effectively had enabled, so future Ruff releases cannot change our lint surface out from under us; everything intentionally on is still listed in extend-select as before. Two rules remain newly in scope under the pin, because they sit inside sections Airflow selects wholesale. Both are deferred behind `ignore` rather than fixed here: a version bump should not also carry semantic changes, and both need per-site judgement that deserves its own reviewed change. ISC004's autofix is unsafe, and LOG004's call sites can only be classified by tracing whether an exception is genuinely in flight at each one. Ruff 0.16 also started formatting Python blocks inside Markdown, which is why a task-sdk doc changes alongside the bump rather than in its own commit: its example was a bare statement that the formatter rewrites into a one-element tuple, which would have changed what the example teaches. Showing the entry in its surrounding context keeps it both correct and stable to format. The lockfile additionally picks up datamodel-code-generator 0.71.0. That is not part of this change: apache#70670 raised the declared floor to >=0.71.0 without regenerating uv.lock, leaving the lock pinned at a version its own constraint excludes, so any `uv lock` has to correct it.

potiuk requested review from amoghrajesh, ashb, bugraoz93, choo121600, dheerajturaga, henry3260, jason810496, jscheffl, kaxil and shahar1 as code owners July 29, 2026 09:21

boring-cyborg Bot added area:airflow-ctl area:DAG-processing area:task-sdk backport-to-airflow-ctl/v0-1-test labels Jul 29, 2026

potiuk added this to the Airflow 3.3.1 milestone Jul 29, 2026

potiuk requested review from mobuchowski and o-nikolas as code owners July 29, 2026 15:19

mobuchowski approved these changes Jul 29, 2026

View reviewed changes

shahar1 merged commit b301429 into apache:main Jul 30, 2026
279 of 280 checks passed

vatsrahul1001 mentioned this pull request Aug 3, 2026

[v3-3-test] Raise the datamodel-code-generator floor for the codegen toolchain (#70670) #70976

Merged

1 task

vatsrahul1001 mentioned this pull request Aug 7, 2026

Status of testing of Apache Airflow 3.3.1rc1 #71274

Open

Navigation Menu

Uh oh!

Raise the datamodel-code-generator floor for the codegen toolchain - #70670