Support onSchemaChange for incremental tables#2101

SuchodolskiEdvin · 2026-03-03T14:21:33Z

This change introduces support for the onSchemaChange option in incremental tables for the BigQuery adapter. It adds the incrementalSchemaChangeBody() strategy to handle schema changes.

End-to-end tests have been created to verify the new functionality.

apilaskowski · 2026-03-03T21:28:57Z

/gcbrun

kolina

I haven't taken a deep look, but in the current logic we add granular SQL statements as separate tasks.

I'd be great to preserve this consistency (as incremental schema change generates quite a lot of statements).

spatel11 · 2026-03-09T15:29:53Z

+
+    // Create temp table for incremental data.
+    finalDmlSql += `
+CREATE OR REPLACE TEMP TABLE ${dataTempTableName} AS (


Not a maintainer of this repo but:

I've noticed this in the GCP Dataform compiled version of an EXTEND schema change incremental table, but for the life of me I cannot discern why this intermediate table is needed. It doesn't seem to be used for anything other than the MERGE statement, and the INSERT on line 420. Could the query be used directly in the MERGE, and in the INSERT? Materializing into an intermediate table increases the cost of the query, sometimes by quite a lot.

The dataTempTableName is used to ensure the columns selected for the MERGE or INSERT operation exactly match the schema determined after potential ALTER TABLE operations, as reflected in the temp_table_columns variable. The final DML statement is constructed dynamically within EXECUTE IMMEDIATE using column lists derived from temp_table_columns (e.g., dataform_columns_list). To use these dynamic column lists in the SELECT part of the MERGE's USING clause or the INSERT's SELECT statement, the source data must be in a table format. Using the raw table.incrementalQuery directly would not allow dynamic column selection based on dataform_columns_list within the EXECUTE IMMEDIATE context.

Hey, we actually don't have a specific reason for doing it. It is clean up leftover from the when we were using store procedure inside the execute immediate. @SuchodolskiEdvin it might be worth running some tests on the engine directly with the generated queries with table.incrementalQuery instead of temp table.

@spatel11 Thanks for pointing this out. Please feel free to open a feature request in our public tracker and we will take a look and plan this change.
https://issuetracker.google.com/issues?q=status:open%20componentid:1193995&s=created_time:desc

Done, thank you: https://issuetracker.google.com/issues/493440483

ikholopov-omni · 2026-03-11T16:27:44Z

+    });
+  }
+
+  private safeCallProcedure(


The name should also describe the fact that procedure is being removed in all cases.

ikholopov-omni · 2026-03-11T16:30:17Z

-                )
-          )
-        );
+        const onSchemaChange = table.onSchemaChange || dataform.OnSchemaChange.IGNORE;


?? operator is more appropriate here.

SuchodolskiEdvin · 2026-03-12T15:50:08Z

I haven't taken a deep look, but in the current logic we add granular SQL statements as separate tasks.

I'd be great to preserve this consistency (as incremental schema change generates quite a lot of statements).

SuchodolskiEdvin · 2026-03-12T16:07:48Z

@kolina

WDYT, about such devision:

Task 1: Create the Procedure.
Task 2: Execute the block that Calls the Procedure and handles exceptions.
Task 3: Drop the Procedure.
This structure aligns with the CLI's granular task pattern for operations sent to BigQuery.

Task 1 (the CREATE PROCEDURE statement) still contains the main, complex SQL logic from incrementalSchemaChangeBody(). Trying to divide the contents of the procedure body into multiple separate Task.statement calls from the CLI side is indeed hard to achieve and not feasible. This is because BigQuery SQL Scripting features (like DECLARE, SET, variable scope, control flow) require the entire logic to run within a single script execution context, which the BEGIN...END block of the procedure provides.

kolina · 2026-03-17T16:02:46Z

@kolina

WDYT, about such devision:

Task 1: Create the Procedure. Task 2: Execute the block that Calls the Procedure and handles exceptions. Task 3: Drop the Procedure. This structure aligns with the CLI's granular task pattern for operations sent to BigQuery.

Task 1 (the CREATE PROCEDURE statement) still contains the main, complex SQL logic from incrementalSchemaChangeBody(). Trying to divide the contents of the procedure body into multiple separate Task.statement calls from the CLI side is indeed hard to achieve and not feasible. This is because BigQuery SQL Scripting features (like DECLARE, SET, variable scope, control flow) require the entire logic to run within a single script execution context, which the BEGIN...END block of the procedure provides.

This sounds reasonable to me. I'd recommend refactoring it a bit so it'll be a bit more structured and ordered

separate SQL generation pieces into separate small sub-functions
have high-level functions that will be calling these sub-functions so the overall algorithm (i.e. steps to assemble a query) and SQL generation pieces are not intertwined

kolina

Ideally we should add an e2e test checking a generated query like here

kolina · 2026-03-20T09:31:16Z

+      emptyTempTableName,
+      dataTempTableName,
+      shortEmptyTableName


Do we need to pass both shortEmptyTableName and emptyTempTableName here?

We could remove emptyTempTableName from the parameters and just compute it inside incrementalSchemaChangeBody() using shortEmptyTableName. However, the parent function buildIncrementalSchemaChangeTasks() still needs emptyTempTableName anyway to pass it to safeCallAndDropProcedure(). Since the parent has to compute it, passing it down to the child prevents us from duplicating the resolveTarget logic in two different places. So I would stay with this implementation.

Can you pass ITarget instances as function arguments and call resolveTarget or name where you actually need a specific string?

It'll be more readable than passing these strings that have quite similar names

kolina · 2026-03-20T10:31:19Z

+);`;
+  }
+
+  private executeMergeOrInsertSql(


So it seems that you added a new logic of generating MERGE or INSERT statements from what we had before in mergeInto / insertInto functions.

I think we should avoid duplicating this logic and different behaviour when incremental schema change is enabled / disabled. So let's consolidate them

Consolidating them entirely is tricky because they operate differently: mergeInto/insertInto generate static SQL, while executeMergeOrInsertSql generates dynamic SQL via EXECUTE IMMEDIATE. Forcing both into one function hurts readability.

To reduce duplication I suggest we extract the shared ON clause logic (handling uniqueKey and updatePartitionFilter) into a single helper method, and rename the original methods to clarify they are for static SQL.

WDYT?

Consolidating them entirely is tricky because they operate differently: mergeInto/insertInto generate static SQL, while executeMergeOrInsertSql generates dynamic SQL via EXECUTE IMMEDIATE. Forcing both into one function hurts readability.

I assume this logic with EXECUTE IMMEDIATE was copied from GCP code? If static SQL in CLI didn't use it before, I don't think you should introduce it as part of your changes: merge / insert logic should stay consistent for all modes of schema updates including IGNORE.

We can bring EXECUTE IMMEDIATE to synchronize GCP and CLI, but it can be done later.

SuchodolskiEdvin · 2026-04-13T11:07:27Z

Ideally we should add an e2e test checking a generated query like here

I was wondering if we can actually create an e2e test for the onSchemaChange parameter? During a dataform run --dry-run, Dataform connects to BigQuery to fetch the current warehouseState. Because our example_incremental table doesn't actually exist in the test BigQuery project yet, tableMetadata comes back as undefined. When that happens, shouldWriteIncrementally() correctly evaluates to false, and Dataform natively falls back to generating a standard CREATE OR REPLACE TABLE.

Because the CLI does not expose a flag to inject a fake warehouseState JSON, I am not sure how we can force the E2E test to simulate a pre-existing table. I still can create the test to verify the parser and the dry-run fallback.

Technically we could run a real dataform run first to create the table in BigQuery, followed by a --dry-run to check the generated MERGE query. However, this would mutate the test BigQuery project, potentially introducing flakiness if tests run concurrently, and it would slow down the suite. The SQL generation is already mostly covered in the execution_sql_test.ts unit tests by mocking tableMetadata.

WDYT?

ikholopov-omni · 2026-04-14T17:29:57Z

Ideally we should add an e2e test checking a generated query like here

I was wondering if we can actually create an e2e test for the onSchemaChange parameter? During a dataform run --dry-run, Dataform connects to BigQuery to fetch the current warehouseState. Because our example_incremental table doesn't actually exist in the test BigQuery project yet, tableMetadata comes back as undefined. When that happens, shouldWriteIncrementally() correctly evaluates to false, and Dataform natively falls back to generating a standard CREATE OR REPLACE TABLE.

Because the CLI does not expose a flag to inject a fake warehouseState JSON, I am not sure how we can force the E2E test to simulate a pre-existing table. I still can create the test to verify the parser and the dry-run fallback.

Technically we could run a real dataform run first to create the table in BigQuery, followed by a --dry-run to check the generated MERGE query. However, this would mutate the test BigQuery project, potentially introducing flakiness if tests run concurrently, and it would slow down the suite. The SQL generation is already mostly covered in the execution_sql_test.ts unit tests by mocking tableMetadata.

WDYT?

you can create a separate dataset per-run and prepare the table with the desired state in operation action before running the one you actually want to test. Just make sure to have a tear-down mechanism.

kolina · 2026-04-14T19:42:54Z

you can create a separate dataset per-run and prepare the table with the desired state in operation action before running the one you actually want to test. Just make sure to have a tear-down mechanism.

Igor's proposal is good

kolina · 2026-04-14T18:44:25Z

+      emptyTempTableName,
+      dataTempTableName,
+      shortEmptyTableName


Can you pass ITarget instances as function arguments and call resolveTarget or name where you actually need a specific string?

It'll be more readable than passing these strings that have quite similar names

kolina · 2026-04-14T18:51:15Z

+    database: string,
+    schema: string,
+    targetName: string,


You can also just pass ITarget here

kolina · 2026-04-14T19:15:38Z

+    table: dataform.ITable,
+    qualifiedTargetTableName: string
+  ): string {
+    const query = this.getIncrementalQuery(table);


Do I understand correctly that this query will be inserted twice into generated SQL? Could we avoid it?

Yes, the query is inserted twice into the final SQL script. It goes into the CREATE OR REPLACE TABLE ... statement to extract the schema metadata, and then again into the final MERGE or INSERT statement. AFAIK, we cannot avoid injecting it twice, because BigQuery requires it to materialize the INFORMATION_SCHEMA metadata before running the validation rules.

Right now I can avoid recalculating it twice. Locally I've updated the code so getIncrementalQuery() is only called once in the incrementalSchemaChangeBody() and after the result of it passed down as a constant. The GCP code works the same way.

kolina · 2026-04-14T19:24:31Z

+);`;
+  }
+
+  private executeMergeOrInsertSql(


Consolidating them entirely is tricky because they operate differently: mergeInto/insertInto generate static SQL, while executeMergeOrInsertSql generates dynamic SQL via EXECUTE IMMEDIATE. Forcing both into one function hurts readability.

I assume this logic with EXECUTE IMMEDIATE was copied from GCP code? If static SQL in CLI didn't use it before, I don't think you should introduce it as part of your changes: merge / insert logic should stay consistent for all modes of schema updates including IGNORE.

We can bring EXECUTE IMMEDIATE to synchronize GCP and CLI, but it can be done later.

kolina · 2026-04-14T19:34:19Z

+      /create or replace procedure `project-id.dataset-id.df_osc_.*`\(\)\s+options\(strict_mode=false\)/i
+    );
+    expect(procedureSql).to.include(
+      `"Schema mismatch defined by on_schema_change = 'FAIL'. Added columns: %T, removed columns: %T"`
+    );
+    expect(procedureSql).to.match(/call `project-id.dataset-id.df_osc_.*`\(\)/i);
+    expect(procedureSql).to.include("EXCEPTION WHEN ERROR THEN");
+    expect(procedureSql).to.match(/drop procedure if exists `project-id.dataset-id.df_osc_.*`/i);


Can we just match the whole query?

If there are non-deterministic places in logic, you can inject an interface for UUID generation to have a determinitic output in tests.

For readability you can put expected "golden" SQL into separate file and read it in tests for comparison

- Updated e2e tests - Added golden files - Consolidated merge and insert functions

SuchodolskiEdvin · 2026-04-22T09:02:05Z

Ideally we should add an e2e test checking a generated query like here

I was wondering if we can actually create an e2e test for the onSchemaChange parameter? During a dataform run --dry-run, Dataform connects to BigQuery to fetch the current warehouseState. Because our example_incremental table doesn't actually exist in the test BigQuery project yet, tableMetadata comes back as undefined. When that happens, shouldWriteIncrementally() correctly evaluates to false, and Dataform natively falls back to generating a standard CREATE OR REPLACE TABLE.
Because the CLI does not expose a flag to inject a fake warehouseState JSON, I am not sure how we can force the E2E test to simulate a pre-existing table. I still can create the test to verify the parser and the dry-run fallback.
Technically we could run a real dataform run first to create the table in BigQuery, followed by a --dry-run to check the generated MERGE query. However, this would mutate the test BigQuery project, potentially introducing flakiness if tests run concurrently, and it would slow down the suite. The SQL generation is already mostly covered in the execution_sql_test.ts unit tests by mocking tableMetadata.
WDYT?

you can create a separate dataset per-run and prepare the table with the desired state in operation action before running the one you actually want to test. Just make sure to have a tear-down mechanism.

Done

SuchodolskiEdvin · 2026-04-22T09:07:13Z

@kolina

Coming back to #2101 (comment)

Yes, dynamic EXECUTE IMMEDIATE logic was adapted from the GCP. Ok, I will keep the merge / insert logic consistent with the existing static SQL in the CLI for now and avoid expanding the scope of this PR too much.

I will revert the dynamic DML generation and reuse the existing mergeInto and insertInto functions for the final data loading across all schema change strategies. I'll push the updated implementation shortly!

- Updated e2e tests - Added golden files - Consolidated merge and insert functions

This change introduces support for the onSchemaChange option in incremental tables for the BigQuery adapter. It adds the incrementalSchemaChangeBody() strategy to handle schema changes. End-to-end tests have been acreated to verify the new functionality.

- Split sql queries to sun-functions - Fixed function names

- Updated e2e tests - Added golden files - Consolidated merge and insert functions

kolina · 2026-04-30T21:27:40Z

+        };
+
+        expect(executionGraph).deep.equals(expectedRunResult);
+        expect(statement).to.include("CREATE OR REPLACE PROCEDURE");


ideally you can use NodeJS BigQuery client (see how we use it in our code here) and just check the schema after running this action in non-dry mode.

- extend onSchemaChange e2e test timeout (#2175)

* Support onSchemaChange for incremental tables This change introduces support for the onSchemaChange option in incremental tables for the BigQuery adapter. It adds the incrementalSchemaChangeBody() strategy to handle schema changes. End-to-end tests have been created to verify the new functionality. * fix: Private functions moved after public functions * - Split onSchemaChange queries to sub-tasks - Split sql queries to sun-functions - Fixed function names * fix: Address feedback for PR dataform-co#2101 and fix presubmit failures - Updated e2e tests - Added golden files - Consolidated merge and insert functions

- extend onSchemaChange e2e test timeout (dataform-co#2175)

* ISSUE-2103 Handle trailing and trailing statement comments * Bump vm2 from 3.10.2 to 3.11.3 (#2168) * Bump vm2 from 3.10.2 to 3.11.3 Bumps [vm2](https://github.com/patriksimek/vm2) from 3.10.2 to 3.11.3. - [Release notes](https://github.com/patriksimek/vm2/releases) - [Changelog](https://github.com/patriksimek/vm2/blob/main/CHANGELOG.md) - [Commits](patriksimek/vm2@v3.10.2...v3.11.3) --- updated-dependencies: - dependency-name: vm2 dependency-version: 3.11.3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Fix vm2 sandbox execution errors and CallSite path stripping The upgrade of vm2 to 3.11.3 tightened sandbox security, stripping file path information from V8 CallSite objects during Error.prepareStackTrace and breaking the resolution of symlinked modules. This caused critical test failures ("Unable to find valid caller file") during Dataform compilation. This commit resolves these issues by: * Resolving `projectDir` to its absolute realpath before sandbox compilation to circumvent symlink boundaries (especially important for bazel runfiles). * Injecting `global.__dataform_current_file` into the NodeVM compiler configuration with a try-finally block to track cross-boundary macro scopes. * Adding a fallback in `core/utils.ts:getCallerFile` to read the injected global file path when the V8 stack trace path is stripped or undefined. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Rafal Hawrylak <hawrylak@google.com> * Bump DF_VERSION from 3.0.56 to 3.0.57 (#2173) * Support onSchemaChange for incremental tables (#2101) * Support onSchemaChange for incremental tables This change introduces support for the onSchemaChange option in incremental tables for the BigQuery adapter. It adds the incrementalSchemaChangeBody() strategy to handle schema changes. End-to-end tests have been created to verify the new functionality. * fix: Private functions moved after public functions * - Split onSchemaChange queries to sub-tasks - Split sql queries to sun-functions - Fixed function names * fix: Address feedback for PR #2101 and fix presubmit failures - Updated e2e tests - Added golden files - Consolidated merge and insert functions * feat: implement .jitCode() support for Assertion actions (#2170) - Add `jit_code` field to the `Assertion` message in `protos/core.proto` - Fix missing protobuf imports and configuration in `protos/BUILD` - Introduce `JitAssertionResult` type alias and `jitCode` method to `Assertion` builder (`core/actions/assertion.ts`) - Add strictly typed `jitContextable` parameter to `Session.sqlxAction` in `core/session.ts` - Update assertion `compile()` to handle `jitCode` execution bypassing standard string evaluation - Parse and apply `jitContextable` safely for assertions in `core/session.ts` - Add unit test coverage in `core/main_test.ts` to verify compilation output * fix: Resolve presubmit failure introduced by PR #2101 - extend onSchemaChange e2e test timeout (#2175) * Bump DF_VERSION from 3.0.57 to 3.0.58 (#2176) * Guard workflow_settings.yaml and dataform.json require with existence check (#2178) * Refactor CompileChildProcess to use BaseWorker (#2179) * Rename worker timeout error to "Compilation timed out" (#2186) * Extract task creation into executionSql helpers (#2180) * Add JiT compilation support for assertion actions (#2193) Adds JIT_COMPILATION_TARGET_TYPE_ASSERTION alongside the existing TABLE / OPERATION / INCREMENTAL_TABLE target types, plus a matching JitAssertionResult { string query } proto and a jitCompileAssertion() dispatcher in core/jit_compiler.ts. Assertions reuse SqlActionJitContext since AssertionContext implements IActionContext, and the result is a single SELECT query string (matching the existing JitAssertionResult = string alias in core/actions/assertion.ts). * Fix caller-file error (#2177) and enforce Core/CLI version match (#2191) vm2 3.11.3 strips file paths from V8 CallSite objects inside the sandbox, so `getCallerFile()` in @dataform/core can no longer resolve the current file from the stack. Track it via a host-side stack exposed through `__df_enter`/`__df_exit`/`__df_current` sandbox helpers and read it through a getter on `global.__dataform_current_file`. For @dataform/core <= 3.0.56 (no global fallback in getCallerFile), patch the bundle text at load time to consult the global. Gated on the installed Core version so newer bundles are never touched. Add a CLI/Core compatibility check: reject installed Core whose major differs from the CLI's, or whose minor is below the CLI's minor. Resolve the installed Core's version by running `require("@dataform/core").version` inside the index-generator vm so any install layout (package.json, workflow_settings.yaml stateless install, JiT) reports the version actually loaded. The error tells the user the exact `dataformCoreVersion:` line to set in `workflow_settings.yaml`. Skipped when the CLI version can't be determined (unbundled local dev). Mirror the same wrapper pattern in `testing/run_core.ts`. Add `compile rejects @dataform/core with incompatible version` test in `cli/index_compile_test.ts` that drives the workflow_settings.yaml stateless install path with dataformCoreVersion: "2.9.0" (latest 2.x on the registry) to exercise the real `npm i` install flow. Add `compile succeeds with @dataform/core <= 3.0.56 via caller-file shim` test that installs @dataform/core@3.0.50 and asserts the compiled action's `fileName` matches the source path, end-to-end proving the bundle-text patch and host-side file stack restore getCallerFile under vm2 3.11.3. * Raise default compile timeout to 60s and improve timeout error message (#2192) Default compilation timeout was 30s. Bump to 60s so larger projects don't hit it on a single compile pass. Extends the "Compilation timed out" error to point at --timeout with concrete examples (--timeout=2m, --timeout=1h) so users can self-mitigate without reading docs. * Bump DF_VERSION from 3.0.58 to 3.0.59 (#2194) * Bump vm2 from 3.11.3 to 3.11.4 (#2188) Bumps [vm2](https://github.com/patriksimek/vm2) from 3.11.3 to 3.11.4. - [Release notes](https://github.com/patriksimek/vm2/releases) - [Changelog](https://github.com/patriksimek/vm2/blob/main/CHANGELOG.md) - [Commits](patriksimek/vm2@v3.11.3...v3.11.4) --- updated-dependencies: - dependency-name: vm2 dependency-version: 3.11.4 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Extract BigQueryClientProvider for injectable BQ clients (#2181) * Extract BigQueryClientProvider for injectable BQ clients * Address review: collapse BigQueryClientProvider to a function type * Bump protobufjs from 7.5.5 to 7.5.8 (#2172) Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.5.5 to 7.5.8. - [Release notes](https://github.com/protobufjs/protobuf.js/releases) - [Changelog](https://github.com/protobufjs/protobuf.js/blob/protobufjs-v7.5.8/CHANGELOG.md) - [Commits](protobufjs/protobuf.js@protobufjs-v7.5.5...protobufjs-v7.5.8) --- updated-dependencies: - dependency-name: protobufjs dependency-version: 7.5.8 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add go_package option to db_adapter.proto and jit.proto (#2197) * Migrate to bazel 7.3.2 (#2187) * Migrate to bazel 7.3.2 * Use fixed ref for github action * Update gcloud to 572.0.0 (#2207) KMS has an org-policy enabled, requiring the usage of client certificate aware endpoints when performing the decryption of credentials. Updating gcloud, so that it uses an updated compatible endpoint. * Add metadata field to AssertionConfig (#2208) Mirrors TableConfig.metadata / ViewConfig.metadata so callers can attach key/value data that lands on Assertion.action_descriptor.metadata.extra_properties. * Bump DF_VERSION from 3.0.59 to 3.0.60 (#2209) * Ignore trailing positional arguments in the global flag parser (#2199) The flag parser in common/flags/index.ts runs in a static initialiser at module load and parses process.argv. Once it had seen a --flag, any later token that was neither another flag nor a flag value caused it to throw "Arg neither flag name nor flag value", crashing the entire CLI before yargs ran. yargs accepts positional arguments after flags (for example `dataform run --some-flag value my_project`), so this lightweight parser was rejecting valid argument orderings. Ignore any token that is neither a flag nor a flag value instead of throwing, exactly as already happened for positional arguments before the first flag; yargs remains responsible for parsing positionals. Extract the parsing loop into an exported parseArgv function so it can be unit-tested in isolation, and add common/flags/index_test.ts covering the regression and related orderings. Fixes #2198. * Bump protobufjs-cli from 1.0.0 to 1.3.3 (#2202) * Bump protobufjs-cli from 1.0.0 to 1.3.3 Bumps [protobufjs-cli](https://github.com/protobufjs/protobuf.js) from 1.0.0 to 1.3.3. - [Release notes](https://github.com/protobufjs/protobuf.js/releases) - [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md) - [Commits](protobufjs/protobuf.js@protobufjs-cli-v1.0.0...protobufjs-cli-v1.3.3) --- updated-dependencies: - dependency-name: protobufjs-cli dependency-version: 1.3.3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * fix(tools): patch generated proto typings to use ES6 import for 'long' This avoids syntax errors in older rollup-plugin-dts which does not support 'import Long = require("long")' syntax. We patch the generated ts.d.ts file to use 'import Long from "long"' instead. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Nick Nalivaika <ikolina@google.com> * Bump js-yaml from 4.1.1 to 4.2.0 (#2205) Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 4.1.1 to 4.2.0. - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](https://github.com/nodeca/js-yaml/commits) --- updated-dependencies: - dependency-name: js-yaml dependency-version: 4.2.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: add getContents() for reading external markdown files into table descriptions pr fix (#2164) * initial attempt at md reader * adding relevant tests and better path resolving for md * cleaner get contents path handling * escaping newlines for md * adding documentation * add check if file in rootdir for get contents and update documentation on getContents to include limits * cleaner and filesystem agnostic check if getContents target is in the root dir * making sure root dir has seperator on check if getcontents file is in dir * removing node module usage * removing redundant normalize and root variable * adding seperator when checking if file is in rootDir * Bump protobufjs from 7.5.8 to 7.6.3 (#2204) Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.5.8 to 7.6.3. - [Release notes](https://github.com/protobufjs/protobuf.js/releases) - [Changelog](https://github.com/protobufjs/protobuf.js/blob/protobufjs-v7.6.3/CHANGELOG.md) - [Commits](protobufjs/protobuf.js@protobufjs-v7.5.8...protobufjs-v7.6.3) --- updated-dependencies: - dependency-name: protobufjs dependency-version: 7.6.3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(gcloud): support client certificate environment in gcloud actions and tests (#2213) - Enable `use_default_shell_env` in `gcloud_secret` rule to pass host environment variables (like `CLOUDSDK_CONTEXT_AWARE_USE_CLIENT_CERTIFICATE`) to the gcloud tool. - Pass `CLOUDSDK_CONTEXT_AWARE_USE_CLIENT_CERTIFICATE` to bazel test in `scripts/publish`. * Prevent .sqlx files in actions.yaml configs with a clear error (#2189) Putting a .sqlx file under the ``filename`` field of an entry in ``definitions/actions.yaml`` used to fall through to ``nativeRequire``, which then failed with a cryptic error far from the source of the mistake. ``.sqlx`` files are loaded directly from the ``definitions/`` directory by the sqlx compiler, not referenced through ``actions.yaml``. Add an explicit validation step in ``loadActionConfigs`` that runs before each action is constructed. If the action's ``filename`` ends in ``.sqlx`` (case-insensitive), emit a ``compileError`` naming the action type, the offending filename, and pointing at the fix. The action is then skipped so we don't double-up with a downstream ``nativeRequire`` error. The check covers every action type that accepts a ``filename``: ``table``, ``view``, ``incrementalTable``, ``assertion``, ``operation``, ``declaration``, ``notebook`` and ``dataPreparation``. Data preparations are included because a ``.dp.sqlx`` file is also auto-compiled from ``definitions/`` - referencing one from ``actions.yaml`` doesn't resolve its path, ignores its ``config {}`` block and double-registers the action, so it's the same mistake. ``.dp.yaml`` data preparations are unaffected. Closes #1785 * Bump tmp from 0.2.4 to 0.2.7 (#2183) Bumps [tmp](https://github.com/raszi/node-tmp) from 0.2.4 to 0.2.7. - [Changelog](https://github.com/raszi/node-tmp/blob/master/CHANGELOG.md) - [Commits](raszi/node-tmp@v0.2.4...v0.2.7) --- updated-dependencies: - dependency-name: tmp dependency-version: 0.2.6 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump @tootallnate/once from 2.0.0 to 2.0.1 (#2171) Bumps [@tootallnate/once](https://github.com/TooTallNate/once) from 2.0.0 to 2.0.1. - [Release notes](https://github.com/TooTallNate/once/releases) - [Changelog](https://github.com/TooTallNate/once/blob/v2.0.1/CHANGELOG.md) - [Commits](TooTallNate/once@2.0.0...v2.0.1) --- updated-dependencies: - dependency-name: "@tootallnate/once" dependency-version: 2.0.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump fast-uri from 3.0.6 to 3.1.2 (#2161) Bumps [fast-uri](https://github.com/fastify/fast-uri) from 3.0.6 to 3.1.2. - [Release notes](https://github.com/fastify/fast-uri/releases) - [Commits](fastify/fast-uri@v3.0.6...v3.1.2) --- updated-dependencies: - dependency-name: fast-uri dependency-version: 3.1.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * remove extra utils_test.ts from api build * allow pbts command to run on mac * refactor concatenateQueries to be a bit more readable * update on_schema_change tests to reflect the new format * update format for api spec * replace drop table in statement comment test because it was causing a test failure * add new tests for comments within a query * update test count in bigquery spec * multi line comment tests * update new golden tests to updated format * fix bigquery spec test to add ending semi colons --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Stephen <spatel11@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Rafal Hawrylak <hawrylak@google.com> Co-authored-by: rafal-hawrylak <33635872+rafal-hawrylak@users.noreply.github.com> Co-authored-by: Edvin <56413069+SuchodolskiEdvin@users.noreply.github.com> Co-authored-by: Nick Nalivayka <ikolina@google.com> Co-authored-by: ikholopov-omni <73560138+ikholopov-omni@users.noreply.github.com> Co-authored-by: August Cayzer <august@cayzer.me> Co-authored-by: Amman <87317538+AmmanuelT@users.noreply.github.com> Co-authored-by: August Cayzer <a@cyzr.me>

SuchodolskiEdvin requested a review from a team as a code owner March 3, 2026 14:21

SuchodolskiEdvin requested review from krushangSk17 and removed request for a team March 3, 2026 14:21

apilaskowski assigned SuchodolskiEdvin Mar 3, 2026

ikholopov-omni self-requested a review March 4, 2026 12:55

Tuseeq1 self-requested a review March 4, 2026 19:29

kolina requested changes Mar 7, 2026

View reviewed changes

spatel11 reviewed Mar 9, 2026

View reviewed changes

ikholopov-omni requested changes Mar 11, 2026

View reviewed changes

SuchodolskiEdvin closed this Mar 12, 2026

SuchodolskiEdvin reopened this Mar 12, 2026

SuchodolskiEdvin force-pushed the on_schema_change branch from c1c02cc to 5e3f9f9 Compare March 17, 2026 20:32

kolina requested changes Mar 20, 2026

View reviewed changes

SuchodolskiEdvin added a commit that referenced this pull request Apr 13, 2026

fix: Address feedback for PR #2101

c0ea194

SuchodolskiEdvin force-pushed the on_schema_change branch from 5e3f9f9 to c0ea194 Compare April 13, 2026 11:04

kolina reviewed Apr 14, 2026

View reviewed changes

SuchodolskiEdvin added a commit that referenced this pull request Apr 22, 2026

fix: Address feedback for PR #2101

a9ac2bf

SuchodolskiEdvin added a commit that referenced this pull request Apr 22, 2026

fix: Address feedback for PR #2101

5408bd1

- Updated e2e tests - Added golden files - Consolidated merge and insert functions

SuchodolskiEdvin force-pushed the on_schema_change branch from c0ea194 to 5408bd1 Compare April 22, 2026 09:00

SuchodolskiEdvin added a commit that referenced this pull request Apr 22, 2026

fix: Address feedback for PR #2101 and fix presubmit failures

fdade1b

- Updated e2e tests - Added golden files - Consolidated merge and insert functions

SuchodolskiEdvin force-pushed the on_schema_change branch from 5408bd1 to fdade1b Compare April 22, 2026 09:55

SuchodolskiEdvin added a commit that referenced this pull request Apr 22, 2026

fix: Address feedback for PR #2101

23aae8f

SuchodolskiEdvin added a commit that referenced this pull request Apr 22, 2026

fix: Address feedback for PR #2101 and fix presubmit failures

c5a5cec

- Updated e2e tests - Added golden files - Consolidated merge and insert functions

SuchodolskiEdvin force-pushed the on_schema_change branch from fdade1b to c5a5cec Compare April 22, 2026 10:58

SuchodolskiEdvin added a commit that referenced this pull request Apr 22, 2026

fix: Address feedback for PR #2101 and fix presubmit failures

c77e723

- Updated e2e tests - Added golden files - Consolidated merge and insert functions

SuchodolskiEdvin force-pushed the on_schema_change branch from c5a5cec to c77e723 Compare April 22, 2026 13:03

SuchodolskiEdvin added a commit that referenced this pull request Apr 22, 2026

fix: Address feedback for PR #2101 and fix presubmit failures

da913c2

- Updated e2e tests - Added golden files - Consolidated merge and insert functions

SuchodolskiEdvin force-pushed the on_schema_change branch from c77e723 to da913c2 Compare April 22, 2026 13:17

SuchodolskiEdvin added 5 commits April 29, 2026 10:38

fix: Private functions moved after public functions

3578b05

- Split onSchemaChange queries to sub-tasks

25279fb

- Split sql queries to sun-functions - Fixed function names

fix: Address feedback for PR #2101

65a0b3c

fix: Address feedback for PR #2101 and fix presubmit failures

0e7c329

- Updated e2e tests - Added golden files - Consolidated merge and insert functions

SuchodolskiEdvin force-pushed the on_schema_change branch from da913c2 to 0e7c329 Compare April 29, 2026 11:11

SuchodolskiEdvin requested review from ikholopov-omni and kolina April 30, 2026 13:04

kolina approved these changes Apr 30, 2026

View reviewed changes

Merge branch 'main' into on_schema_change

05678e1

ikholopov-omni approved these changes May 25, 2026

View reviewed changes

SuchodolskiEdvin merged commit 636d085 into main May 25, 2026
10 checks passed

SuchodolskiEdvin deleted the on_schema_change branch May 25, 2026 14:37

SuchodolskiEdvin added a commit that referenced this pull request May 26, 2026

fix: Resolve presubmit failure introduced by PR #2101

dc19645

- extend onSchemaChange e2e test timeout (#2175)

spatel11 pushed a commit to spatel11/dataform that referenced this pull request Jul 8, 2026

fix: Resolve presubmit failure introduced by PR dataform-co#2101

bbbc192

- extend onSchemaChange e2e test timeout (dataform-co#2175)

Uh oh!

Conversation

SuchodolskiEdvin commented Mar 3, 2026

Uh oh!

apilaskowski commented Mar 3, 2026

Uh oh!

kolina left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SuchodolskiEdvin commented Mar 12, 2026

Uh oh!

SuchodolskiEdvin commented Mar 12, 2026

Uh oh!

kolina commented Mar 17, 2026

Uh oh!

kolina left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SuchodolskiEdvin commented Apr 13, 2026

Uh oh!

ikholopov-omni commented Apr 14, 2026

Uh oh!

kolina commented Apr 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SuchodolskiEdvin commented Apr 22, 2026

Uh oh!

SuchodolskiEdvin commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SuchodolskiEdvin commented Apr 22, 2026 •

edited

Loading