fix(migrations): move UPDATEs inside disable_sqlite_fkeys in migration 0097

[Lee-W]

Why

Migration 0097 failed on SQLite with FOREIGN KEY constraint failed when running DROP TABLE dag. PRAGMA foreign_keys=off (issued inside disable_sqlite_fkeys) is silently ignored when a transaction is already active — SQLite limitation. The two UPDATE statements before disable_sqlite_fkeys triggered SQLAlchemy's autobegin, starting a transaction before the PRAGMA ran.

How to reproduce

1. Bring up a 3.1.8 instance

breeze start-airflow  --backend sqlite --load-example-dags --use-airflow-version 3.1.8

2. Run a few Dags randomly. We need rows in the Dag run and task instance (or whatever rows that references Dag using FK)

3. Go to the Shell tab in the breeze terminal

4. Run

uv pip install apache-airflow==3.2.0
uv run airflow db migrate

What

Move the UPDATE statements inside disable_sqlite_fkeys so PRAGMA foreign_keys=off executes before autobegin opens a transaction.

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Generated-by: [Claude] following the guidelines

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[Lee-W]

@vatsrahul1001 It looks like we used the wrong approach for testing, and I applied the wrong constraint during RC1 testing. I didn't have time to test it on RC2 until it was released yesterday 🤦‍♂️

After running the following command we discussed, I found out there's no dag_run in the DB.

breeze start-airflow  --backend sqlite --load-example-dags --use-airflow-version 3.1.8
breeze start-airflow  --backend sqlite --load-example-dags

So here's what I do for verifying this PR is correct (how to reproduce was detailed in the description)

Variables

OLD_VERSION=3.1.8
NEW_VERSION=3.2.0  # or a specific SHA-1

Step 1: Start Airflow $OLD_VERSION

breeze start-airflow \
    --backend sqlite \
    --load-example-dags \
    --use-airflow-version $OLD_VERSION 

Step 2: Populate the database

Run a few Dags manually to create rows in Dag run, task instance, and other tables that reference Dags via foreign keys.

Step 3: Save the database and exit Breeze

cp /root/airflow/sqlite/airflow.db /files/airflow_$OLD_VERSION.db

Exit Breeze with stop_airflow.

Step 4: Checkout the target version and start Breeze

git checkout $NEW_VERSION
breeze shell \
    --backend sqlite \
    --mount-sources selected

Step 5: Restore the saved database

cp /files/airflow_$OLD_VERSION.db /root/airflow/sqlite/airflow.db

Step 6: Run the migration and test

airflow db migrate
airflow db downgrade --to-revision 937cbd173ca1 -y
airflow db migrate

[Dev-iL]

Can we modify the migration test (or add a new regression test) to capture this sort of issues?

[jason810496]

Can we modify the migration test (or add a new regression test) to capture this sort of issues?

Perhpas we need to raise Dev List about adding unit test or any better testing approach to harden the whole DB migration. I had discussed with Wei and other folks before, but we haven't settled down the concore approach.

[Lee-W]

Can we modify the migration test (or add a new regression test) to capture this sort of issues?

I can try to find time to do it in another PR, but I think this is needed for users who use SQLite as a backend to upgrade to 3.2.x.

Perhpas we need to raise Dev List about adding unit test or any better testing approach to harden the whole DB migration. I had discussed with Wei and other folks before, but we haven't settled down the concore approach.

I'm not sure whether we should bring it up on the dev list without at least some idea of what it is. But if anyone already has an idea to do it correctly, they're more than welcome to take action. 🙂

[Lee-W]

I'm going to merge it and backport it. will create another issue for the testing thing

[github-actions]

Hi maintainer, this PR was merged without a milestone set.
We've automatically set the milestone to Airflow 3.2.1 based on: backport label targeting v3-2-test
If this milestone is not correct, please update it to the appropriate milestone.

This comment was generated by Milestone Tag Assistant.

[github-actions]

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test