If you run Argo CD past a handful of applications, you’ve probably built an app-of-apps. You create a root Application whose source is a directory of more Application manifests, and those point at the real workloads. It’s a reasonable way to bootstrap a cluster, or a set of clusters, from one place. When the directory got large, you may have switched to ApplicationSets and let a generator produce the child Application objects instead of maintaining them by hand.

I’m not writing this to argue against the pattern. It solves a real problem. But it’s worth looking at why you end up reaching for it, because the reasons line up almost exactly with the capabilities we’ve been building into ConfigHub: querying configuration across a fleet, propagating values between related pieces, and checking and gating changes before they’re applied.

What you’re actually asking app-of-apps to do

Once you have more than a few applications, you stop caring about them one at a time and start needing to operate on the group. Concretely, that means:

• Seeing all the apps in a set together, not app by app.
• Sharing values across them — the destination cluster and namespace, the registry, a region, a common set of labels or policy resources — without restating those values in every child.
• Expressing ordering and dependencies between them, and deciding what happens to the children when the root changes or is deleted.
• Having one place to bootstrap and review the whole thing.

App-of-apps gives you a way to do each of these on top of Git. Grouping becomes a directory. Shared values get duplicated across the children, or threaded through an ApplicationSet generator. Ordering becomes sync-wave annotations. The single place to review is the root Application.

People reach for app-of-apps when the apps are heterogeneous and sometimes there are thousands of them. With the Git version, changing one of those apps means the usual commit, push, and review, even when you only wanted to find it and edit a value. In a real setup the manifests for each environment often live in separate repos, so there’s no single place to see or manage them.

In ConfigHub the apps are stored as data you can query across Spaces, which makes them easier to find and change.

The same app-of-apps, backed by data

To make this concrete, here’s the setup I ran. Four Spaces in ConfigHub:

• guestbook-aoa holds two Units: root (the root Application CR) and app-of-apps (the three child Applications).
• guestbook-dev, guestbook-staging, and guestbook-prod each hold one guestbook Unit — the actual workload, a Deployment and a Service. All of them are Variants of the guestbook-base Space.

The shape of the pattern is unchanged. What’s different is where the manifests live: instead of a Git directory, each piece is a Unit that Argo pulls over OCI. The root Application points at the OCI address of the app-of-apps unit:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: oci://oci.hub.confighub.com/unit/guestbook-aoa/app-of-apps
    targetRevision: latest
    path: "."
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated: { prune: true, selfHeal: true }

And the app-of-apps unit contains the three children, each pointing at the guestbook unit in its environment’s Space and at its own cluster:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook-dev
  namespace: argocd
spec:
  project: default
  source:
    repoURL: oci://oci.hub.confighub.com/unit/guestbook-dev/guestbook
    targetRevision: latest
    path: "."
  destination:
    name: dev-cluster
    namespace: guestbook
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions:
      - CreateNamespace=true
---
# Two other Applications here
# ... guestbook-staging -> staging-cluster, guestbook-prod -> prod-cluster

Creating the units is two commands:

cub unit create --space guestbook-aoa app-of-apps app-of-apps.yaml
cub unit create --space guestbook-aoa root root.yaml

Argo CD doesn’t know or care that the source is ConfigHub rather than Git — it sees an OCI repo. Which means the app-of-apps mechanics all still work: root syncs, children appear, children sync, workloads deploy. The difference is what you can do with the configuration before and around that sync loop.

Keeping the two systems readable together

ConfigHub and Argo CD name different things, and a single change flows through both, so it’s worth being deliberate about how the names line up. The convention I settled on: a Space’s OCI worker and target are named after the Argo CD cluster that Space’s config is deployed to. Term by term:

• A Space (a bucket of Units, one per environment) maps to an environment, or a set of Applications — here guestbook-dev, guestbook-staging, guestbook-prod, and guestbook-aoa.
• A Unit (versioned KRM config data) is the manifests an Application syncs (its spec.source) — the guestbook Unit becomes the guestbook-dev Application’s source.
• The app-of-apps Unit is the directory of child Applications, fanning out to three; the root Unit is the root Application CR, the one you bootstrap.
• A Worker plus Target (OCI) has no Argo counterpart — it’s ConfigHub’s publishing mechanism — and I name them after the destination cluster: dev-cluster, staging-cluster, prod-cluster, in-cluster.
• A Unit’s OCI address oci://…/unit/<space>/<unit> is the Application’s spec.source.repoURL, the pull source.
• The live Cluster, registered in Argo and addressed by name, must equal the child’s destination.name.

So guestbook-dev’s target is dev-cluster, which is also the child Application’s destination.name and the cluster registered in Argo CD — followed end to end. The guestbook-aoa Space deploys to the host cluster Argo CD itself runs in, so its target is in-cluster, Argo CD’s own name for that cluster. Targets have no Argo CD equivalent — which is why naming them after the destination cluster is what ties the two systems together at a glance.

The set of apps, as a query

With the units published, looking at the set of apps is a query over data that already exists:

$ cub unit list --space "guestbook-*" --where "Labels.Component = 'guestbook'"
SPACE              UNIT          TYPE                 GATED  UNAPPLIED
guestbook-dev      guestbook-ui  apps/v1/Deployment   no     no
guestbook-staging  guestbook-ui  apps/v1/Deployment   no     no
guestbook-prod     guestbook-ui  apps/v1/Deployment   yes    yes

Nothing generates that list. The Units are the data, and the set is a where clause over them. Each Unit is materialized — plain YAML with real values, no templates — and it’s exactly what Argo pulls and syncs:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: guestbook-ui
spec:
  replicas: 2
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: guestbook-ui
  template:
    metadata:
      labels:
        app: guestbook-ui
    spec:
      containers:
      - image: gcr.io/google-samples/gb-frontend:v5
        name: guestbook-ui
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: guestbook-ui
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: guestbook-ui

Most of what follows comes down to that difference: the thing you query and change is the same thing that gets deployed.

Checking changes before they’re applied

This is the capability I’d miss most going back to a plain app-of-apps. Normally a bad child config syncs and the violation turns up afterward, in the cluster, if an admission controller or a scan catches it at all. Catching it earlier means rendering every child in CI and running your checks there, which brings back the per-repo commit, push, and render steps.

In ConfigHub, we have a concept called Triggers, and validation and policy tools like Kyverno, OPA Gatekeeper, Kubescape, and others can all be built as Triggers. They run on every change to the configuration data, before it’s released. When a check fails, the trigger attaches an Apply Gate, or a warning if you’d rather not block. Across the same set of apps:

$ cub unit list --space "guestbook-*" --where "GatedCount > 0"
SPACE           UNIT          GATE          REASON
guestbook-prod  guestbook-ui  no-critical   container 'guestbook-ui' runs as root (PodSecurity restricted)

The check runs when the data changes, not when Argo decides to sync it. In the setup above, a gated guestbook-prod unit simply never gets a new latest published to oci://…/unit/guestbook-prod/guestbook, so Argo keeps syncing the last good version — the gate blocks release regardless of whether the change came from a person, from CI, or from an agent.

Seeing what a change affects

A template-based app-of-apps can only show you the source you edited, not the set of live configs it expands to. Because ConfigHub stores the materialized data along with the upstream and downstream relationships between Units, you can look at what a change reaches before releasing it:

$ cub unit tree --node guestbook-base --downstream
guestbook-base
|- guestbook-dev/guestbook-ui
|- guestbook-staging/guestbook-ui
`- guestbook-prod/guestbook-ui   [gated]

That’s a review an app-of-apps doesn’t give you: not the template you changed, but the live targets it expands to and which of them is currently blocked.

Bootstrapping

The one place a Git-based app-of-apps and this setup look identical is the very first step: something has to put the root Application into the cluster. Here it’s one line, straight from ConfigHub:

cub unit data --space guestbook-aoa root | kubectl apply -f -
kubectl get applications -n argocd

Argo CD takes over from there: it pulls app-of-apps over OCI, creates the three child Applications, each pulls its guestbook unit and deploys into its vcluster, creating the guestbook namespace on the way. End state:

NAME                SYNC STATUS   HEALTH STATUS
root                Synced        Healthy
guestbook-dev       Synced        Healthy
guestbook-staging   Synced        Healthy
guestbook-prod      Synced        Healthy

Wrapping up

App-of-apps is a sensible way to get grouping, shared values, dependencies, and a single point of control out of Git and templates, which weren’t built to provide them. What the setup above shows is that you don’t have to choose between the pattern and Configuration as Data: the root, the fan-out, and the children all still work when Argo CD pulls them from ConfigHub over OCI. The things app-of-apps makes awkward — sharing a value without duplicating it, checking the whole set before it syncs, turning on a policy everywhere, seeing what a change affects — are the things you get directly when the configuration behind those Applications is stored as data with relationships between the pieces.

If you already run an app-of-apps, the shared values and the fleet-wide policy rollout are the two things I’d try moving into ConfigHub first. You can do it for a single set of apps without touching the rest — the setup above is one root and three children on a laptop-sized kind cluster — which makes it easy to compare against what you have now.

ConfigHub is in preview — you can try it here. If you do, I’d like to hear how it lines up with your current Argo CD setup; you can find me on LinkedIn.

Argo CD App-of-apps and Configuration as Data was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

I got creative. And what emerged is an architecture I now believe every serious Apps Script developer should have in their toolbox: a two-half pipeline that compresses, chunks, and reassembles massive datasets by moving every heavy computation out of Apps Script entirely — into a local Node.js preprocessing step and the user’s own browser. Apps Script itself becomes the thinnest possible proxy: a stateless, sub-second string-returning bridge between Google Drive and the client.

This is the deep-dive story of that pipeline. Here’s what we’ll cover:

• The four killer constraints that make the naive approach impossible
• The complete compress-chunk-upload-reassemble flow
• the Vite + clasp + gws toolchain that makes it all work
• Real measured numbers from production
• The trade-offs (plus improvements) you’ll need to weigh if you adopt this pattern yourself.

The Four Walls You Will Hit

Before we talk solutions, let’s put our finger on the problems. If you’ve ever tried to serve a large dataset with Apps Script, you already know the proble. Maybe you’ve tried to orchestrate multiple executions through google.script.run; and maybe that as enough of a work around, and maybe not (spoiler alert: not in this project). Here's what I've had to deal with.

1. The 6-minute execution budget. Every Apps Script execution has a hard wall clock limit — six minutes for consumer accounts, thirty for Workspace plans. Parsing millions of cells from hundreds of spreadsheets within a single server function? Not happening. I explored the classic workaround — splitting work across multiple executions with triggers and PropertiesService — way back in 2021. The pipeline we’re building here takes a more radical approach: move the heavy lifting out of Apps Script entirely.

2. google.script.run only speaks ECMAScript primitives. The bridge between your browser client and the Apps Script V8 runtime — that magical google.script.run.withSuccessHandler() we all rely on — can only pass strings, arrays, objects, numbers, and booleans. Native Apps Script objects like Blobs, Drive Files, or Calendars? They die on the bridge. This means your return values must survive JSON serialization through the IFRAME sandbox's postMessage channel. Binary data? Forget it.

3. There’s an undocumented but very real payload ceiling. Google doesn’t publish a precise byte limit for google.script.run return values, but anyone who's pushed the envelope knows the score: payloads above tens of megabytes trigger "Argument too large" exceptions, or worse — they silently return undefined. Community testing and production scars put the practical ceiling somewhere in the 20–30 MB range. A separate 50 MB blob upload limit through HtmlService is documented, but that's a different mechanism entirely.

4. CacheService caps at 100 KB per key-value pair. You might think, "Fine, I'll cache the parsed data server-side." Nope. CacheService is designed for configuration snippets and short-lived tokens, not megabyte-scale datasets. It cannot serve as your datastore.

These four constraints are non-negotiable. They are the laws of physics in the Apps Script universe. The pipeline we’re about to walk through treats them not as obstacles to overcome but as boundary conditions to route around.

The Two-Halves Architecture

The core insight is surprisingly simple once you state it: move all the work that doesn’t fit inside Apps Script’s quotas to the two places that have none — the developer’s machine and the user’s browser.

Half One: The Local Preprocessing Pipeline (Node.js, Unbounded)

This runs on your machine or a CI runner, well outside Google’s execution sandbox. It:

• Reads the master aggregated JSON snapshot — hundreds of megabytes — straight from disk.
• Compresses it with raw DEFLATE at maximum compression level using fflate, a pure-JavaScript library weighing roughly 8 kB that runs identically in Node.js and the browser.
• Encodes the binary stream to Base64 so it can survive the google.script.run string channel.
• Slices that Base64 string into fixed-size chunks (13 MB each, chosen through empirical testing — more on that below).
• Uploads each chunk as a plain text file to a designated Google Drive folder via the gws CLI.
• Writes a manifest file mapping chunk indices to their Drive file IDs.

Half Two: The Google Apps Script Runtime (V8 Engine, Heavily Bounded)

Deployed as a Sheets-bound or standalone project serving an HtmlService web application. The server code is almost absurdly thin:

• The manifest is imported at build time as a static JSON asset bundled into the IIFE.
• One function returns the manifest. One function per chunk reads a Drive file by ID and returns its text content as a plain string.
• It does not parse, decompress, aggregate, or transform the data in any way. It is a pure Drive-to-string proxy.

The Client Browser (Alpine.js SPA)

• Fetches the manifest, then downloads all chunks in parallel through runGas calls.
• Concatenates chunks in memory, decodes Base64, decompresses with fflate (the same library used to compress), parses the JSON, and indexes the resulting dataset with non-reactive closure caching — ensuring searches stay instantaneous even across millions of cells.

That’s the architecture. Now let’s look at the toolchain that builds it.

How the Code Gets Built and Deployed

None of this toolchain was wired together by hand. It all comes from the Apps Script Engine Template — a reusable project starter I built and open-sourced. One command — npx apps-script-engine my-project — and you get a fully scaffolded project with Vite, clasp, environment management, and a build pipeline that produces exactly what Apps Script needs: a single-file client bundle for HtmlService, an IIFE bundle for the server (with npm dependencies like fflate baked in), and thin wrapper functions so the GAS runtime can call your exported functions.

I first open-sourced this template back in August 2024, and it has grown — through performance optimizations that yielded 10× speed improvements and, as of v2.0, full TypeScript support — into the complete toolkit you’re looking at now.

The key result for this pipeline: you get to npm install fflate and import { decompressSync } from 'fflate' in your Apps Script HTML code, and it works in production. The template handles the bundling, the IIFE wrapping, and the export hoisting automatically. Deployment is a single command:

npm run build:addon:push
  → npm run build:ui     (Vite single-file: Alpine.js + Tailwind → index.html)
  → npm run build:gas    (Vite IIFE: server.js + fflate + manifest → server.iife.js + exports.js)
  → npm run push         (clasp push -f)

That’s it. The template is what makes this architecture reproducible — you scaffold, you write your server functions, and you build. The internals of AST traversal and IIFE wrapping are handled for you.

The Data Movement Pipeline: Step by Step

This is where the magic really happens. Let’s walk through each stage with actual code.

Step 1 — Extraction from Google Sheets (External, Upstream)

The master dataset originates from an external extraction process that walks a folder hierarchy of Google Sheets, reads their data ranges programmatically via the Google Sheets API, and aggregates everything into a single JSON snapshot.

This extraction is deliberately kept outside the web application and outside the build pipeline. It’s an I/O- and CPU-intensive task — opening hundreds of spreadsheets, reading millions of cells, normalizing structure — that would never fit inside the 6-minute budget. By hoisting it into an unbounded execution environment (a Node.js script on a developer machine, or a scheduled Cloud Function), the web application never touches a single source spreadsheet cell at runtime. It only ever reads the pre-compiled, pre-compressed binary artifact.

The output is a JSON file committed to version control as the authoritative snapshot.

Step 2 — Compression

const rawData = fs.readFileSync("outputs/master_data_combined.json", "utf8");
const u8 = fflate.strToU8(rawData);
const compressed = fflate.deflateSync(u8, { level: 9 });

fflate is the unsung hero here. It’s a pure-JavaScript compression library — roughly 8 kB, zero native dependencies, identical behavior in Node.js and in the browser. This symmetry is critical: the same library compresses during the offline build and decompresses in the user’s browser at runtime. No format compatibility issues, no environment-specific APIs like Node’s zlib or the browser's CompressionStream.

We use raw DEFLATE at compression level 9 (maximum). The “raw” part means we omit the gzip header and CRC trailer, saving a handful of bytes. fflate.decompressSync on the client autodetects the format, so this choice is transparent.

Real measured compression from a 341 MB JSON snapshot: the compressed binary stream comes out to roughly 85 MB — a compression ratio of about 4×. Clinical and structured data with high entropy compresses less aggressively than repetitive log data, but 4× is enough to make chunking feasible.

Step 3 — Base64 Encoding

const b64 = Buffer.from(compressed).toString("base64");

This is the tax we pay for staying purely within Apps Script. The compressed binary stream must cross the google.script.run bridge, which serializes everything as JSON through the IFRAME sandbox. Binary types like Uint8Array or ArrayBuffer are either rejected or silently corrupted. So we encode to Base64 — converting arbitrary binary into a plain ASCII string safe for JSON transport.

The cost is a fixed 33% expansion: 85 MB of compressed binary becomes roughly 113 MB of Base64 text. We accept this because introducing a Cloud Run sidecar or a CDN-hosted file would defeat the purpose of keeping the entire runtime surface within a Sheets sidebar. The architecture pays the Base64 tax to avoid paying the infrastructure tax.

Step 4 — Chunking

const CHUNK_SIZE = 13 * 1024 * 1024; // 13 megabytes per chunk
const chunksCount = Math.ceil(b64.length / CHUNK_SIZE);

The 113 MB Base64 string is sliced into consecutive 13 MB substrings. For our dataset, that’s 9 chunks: 8 full-size at ~13.6 MB each and one tail chunk at ~9.7 MB.

The 13 MB threshold wasn’t pulled from thin air. It satisfies three constraints simultaneously:

• Stays well under the practical google.script.run payload ceiling. Production experience shows payloads above 20–30 MB frequently trigger serialization failures. Thirteen megabytes leaves comfortable headroom.
• Keeps each server function sub-second. DriveApp.getFileById().getBlob().getDataAsString() on a 13 MB text file completes in well under one second. All nine parallel calls easily stay within the 6-minute limit.
• Enables full concurrency. Nine independent chunks fetched via Promise.all means wall-clock download time is bounded by the slowest single chunk, not the sum of all nine.

Each chunk is written to disk in two formats: .js files (ES modules for Vite dev server mock mode) and .txt files (the actual upload payloads for Google Drive).

A manifest file ties it all together:

{
  "root_folder": "<drive_folder_id>",
  "extracted_at": "2026-07-09T09:02:22.435Z",
  "chunks": [
    {
      "index": 0,
      "drive_id": "1Qlb...",
      "local_path": "outputs/chunks/chunk_0.js"
    },
    {
      "index": 1,
      "drive_id": "1FGA...",
      "local_path": "outputs/chunks/chunk_1.js"
    }
  ]
}

Step 5 — Upload to Google Drive via gws CLI

gws drive +upload outputs/chunks/chunk_0.txt \
  --parent 1wUx-2z4AilM-uEL3i-ZNvStJzhf9Cc7O \
  --name chunk_0.txt

gws (Google Workspace CLI, v0.22.5) is an open-source command-line tool from github.com/googleworkspace/cli. It wraps the Google Workspace REST APIs behind a clean, consistent interface: gws <service> <resource> <method> [flags]. The +upload convenience helper auto-detects MIME type, constructs a multipart upload to files.create, sets the parent folder and filename, and returns the created file's metadata as JSON — including the Drive file ID.

Since our pipeline script runs in Node.js, we invoke gws via child_process.spawnSync, parse the JSON response, and write the id field back into the manifest:

function runGws(args) {
  const result = spawnSync("gws", args, { encoding: "utf8" });
  if (result.status !== 0) {
    console.error(`Error running gws ${args.join(" ")}:`, result.stderr);
    return null;
  }
  return JSON.parse(result.stdout);
}

For local development, npm run db:deploy:local runs the entire compress → encode → slice → write flow but skips the upload. Chunks are marked with drive_id: "local_only" in the manifest, and the Vite dev server's custom middleware plugin serves the local .txt files over HTTP. The client-side code path — fetch chunk text → concat → atob → decompressSync → JSON.parse — is identical in local development and production. Only the data source changes.

Step 6 — Recovery from Drive: The Thin Server

Here’s the entire server-side data path — two functions:

import manifest from "../../../outputs/dlv_chunks_manifest.json";

export function getDlvChunksManifest() {
  return manifest;
}

export function getCompressedDlvData(chunkIndex) {
  const chunk = manifest.chunks.find((c) => c.index === chunkIndex);
  if (!chunk || !chunk.drive_id) {
    throw new Error(`Chunk ${chunkIndex} not found.`);
  }
  if (chunk.drive_id === "local_only") {
    throw new Error(
      `Chunk ${chunkIndex} has local_only placeholder. Deploy data first.`,
    );
  }
  const file = DriveApp.getFileById(chunk.drive_id);
  return file.getBlob().getDataAsString();
}

Let me emphasize what’s happening here — and what’s not happening. The manifest is imported at build time and bundled into the IIFE. No runtime API call needed to discover what chunks exist. DriveApp.getFileById().getBlob().getDataAsString() reads the entire file and returns a plain string — the one type google.script.run can legally return. For a 13 MB text file, this call completes in well under one second. The function is idempotent and stateless: read file, return string. No parsing, no transformation, no caching, no mutation. It is trivially auditable and virtually immune to exceeding the 6-minute budget.

Step 7 — Stitching and Decompression in the Browser

The client orchestrates reconstruction in six stages:

// 1. Fetch the manifest (one cheap call)
const manifest = await runGas("getDlvChunksManifest");

// 2. Fetch all chunks in parallel
const chunkPromises = manifest.chunks.map((c) =>
  runGas("getCompressedDlvData", [c.index]),
);
const chunkResults = await Promise.all(chunkPromises);

// 3. Stitch - string concatenation restores the unified Base64 stream
const compressed = chunkResults.join("");

// 4. Decompress
const jsonStr = decompressBase64(compressed);

// 5. Parse into JavaScript objects
const rawData = JSON.parse(jsonStr).data;

// 6. Flatten the hierarchical structure into a flat array

The decompressBase64() function is a four-stage synchronous pipeline:

function decompressBase64(b64) {
  // Stage 1: Base64 decode via the browser's built-in atob()
  const binString = atob(b64);

  // Stage 2: Manually copy char codes into a Uint8Array
  const bytes = new Uint8Array(binString.length);
  for (let i = 0; i < binString.length; i++) {
    bytes[i] = binString.charCodeAt(i);
  }

  // Stage 3: DEFLATE decompress via fflate (the same library used to compress)
  const decompressed = fflate.decompressSync(bytes);

  // Stage 4: UTF-8 decode back to a JavaScript string
  return fflate.strFromU8(decompressed);
}

A few behavioral notes worth highlighting:

• Parallelism is real. Promise.all dispatches all chunk fetches concurrently. Each google.script.run call is a separate IFRAME → server → Drive round-trip, and they overlap rather than serialize. Wall-clock download time is bounded by the slowest chunk, not the sum.
• Stitching is a simple .join(''). Because Base64 was sliced at arbitrary character boundaries, per-chunk decoding isn't possible — the entire stream must be reassembled first. But .join('') is O(N) and near-instant even for ~113 MB strings in modern engines.
• The charCodeAt loop (Stage 2) is the CPU bottleneck. Converting an ~85 MB binary string to a Uint8Array requires iterating every byte. It takes several hundred milliseconds.
• decompressSync blocks the main thread. For an 85 MB compressed payload decompressing to ~341 MB, expect a 1–3 second UI freeze. We'll discuss fixes below.

Step 8 — Non-Reactive Closure Caching

Once decompressed, the dataset lives as a read-only JavaScript array. Alpine.js’s reactive proxy is powerful, but re-rendering DOM on every state change across millions of cells would be catastrophic. The solution: a non-reactive closure cache with a cheap selection signature:

const cache = {
  maps: null,
  headerIndex: new Map(),
  mergedKey: null,
  mergedRows: null,
  displayKey: null,
  displayRows: null,
};

If the signature (composite string of active IDs, count, and search query) matches the cache key, the pre-computed array reference is returned instantly. Only on a cache miss does the application run the indexing and mapping algorithms, which use pre-compiled header index maps for O(1) lookups instead of O(N) scans over millions of cells.

This is the final link: the pipeline got the data into the browser; the cache keeps it fast once there.

The gws CLI: A Closer Look

I want to spend a moment on gws because it's one of those tools that deserves more attention than it gets. The Google Workspace CLI is open-source, community-maintained, and explicitly labeled "not an officially supported Google product" — but for our purposes, it's exactly what we need.

Its command structure is beautifully uniform:

gws <service> <resource> [sub-resource] <method> [flags]

So gws drive files list, gws sheets spreadsheets get, gws gmail users messages list — all follow the same pattern. Authentication supports multiple strategies: a pre-obtained OAuth token via environment variable, a credentials JSON file, client ID/secret, or interactive browser-based login via gws auth login.

The +upload helper is the killer feature for our pipeline. One command replaces multiple lines of API construction that would otherwise be needed. The trade-off — invoking gws as a child process via spawnSync — adds negligible overhead for a script that runs infrequently.

Real Measured Numbers

Theory is nice. Here are the actual numbers from a production 341 MB master JSON snapshot:

What This Architecture Achieves

Let’s count the wins explicitly, because there are several.

It bypasses the 6-minute execution wall. No single server function performs heavy work. Each is a sub-second Drive read. All CPU-intensive work runs offline in Node.js or in the user’s browser — neither of which has a 6-minute budget.

It bypasses google.script.run type and size limits. Everything crossing the bridge is a plain ECMAScript string. The 13 MB chunk size stays comfortably below the undocumented serialization ceiling. Base64 encoding ensures binary data survives JSON serialization intact.

It bypasses the 100 KB CacheService ceiling. The dataset lives in Google Drive files, not in CacheService. Drive imposes no meaningful per-file size limit for text files, and it acts as a free, durable, permission-scoped content delivery mechanism. No separate hosting. No database server. No CDN.

It bypasses Google Sheets throughput limits at runtime. The source data is never read from spreadsheets at runtime. The only Sheets interaction is reading a small configuration sheet (typically under 10 KB).

It remains a genuinely low-code application. There is no Cloud Run instance, no database server, no API gateway, no custom hosting. The entire runtime surface is a Google Sheets sidebar served by HtmlService. A non-technical user launches it from a spreadsheet menu. The "infrastructure" is a single Drive folder.

Deployment is two commands:

• npm run db:deploy — compress, chunk, upload data to Drive
• npm run build:addon:push — compile UI + server, push to Apps Script via clasp

The Trade-offs (Every Architecture Has Them)

I wouldn’t be doing my job if I didn’t tell you where this hurts.

Base64 Expansion Tax (+33%)

Every byte crossing the bridge is 33% larger than it needs to be. The alternative — transmitting raw binary — simply isn’t available through the Apps Script IFRAME bridge. You could eliminate this tax by introducing a Cloud Run sidecar or a CDN-hosted file, but that defeats the “zero-infrastructure” design goal.

All-or-Nothing Reassembly

Because Base64 was sliced at arbitrary character boundaries, the entire stream must be present before atob() and decompressSync() can run. Promise.all fails fast — if any single chunk download fails, the whole load fails. There's no partial recovery and no progressive rendering.

Synchronous Decompression Freezes the Browser

fflate.decompressSync() blocks the main thread for 1–3 seconds. The application yields before CSV export (30 ms setTimeout to paint a spinner) but does not currently yield during decompression. It's janky. It's fixable. But it's there.

Deploy is Atomic and Manual

npm run db:deploy clears all previous chunks, recompresses, and re-uploads from scratch. A network interruption mid-upload leaves orphaned files and an incomplete manifest. The fix is to re-run — there's no transactional upload, no resume, no rollback.

Manifest Baked at Build Time

The manifest is statically imported into the server bundle. When new data is deployed, the Apps Script project must be rebuilt and pushed too. Data updates and code deployments are coupled. This is a deliberate simplicity trade-off, but it’s a trade-off.

No Incremental Updates

Adding one row changes all chunks because the entire dataset is recompressed from scratch and the Base64 stream is sliced anew. Simple and safe, but wasteful if only a fraction of the data changed.

Memory Footprint

The full decompressed dataset (~341 MB as JavaScript objects) lives entirely in browser memory. Modern desktop browsers handle this fine. Mobile or older hardware? Not so much. The non-reactive cache prevents re-allocation storms, but the baseline footprint can’t be reduced without switching to a lazy-loading or indexed query-backend model.

What I’d Improve Next

Every project has a v2 wishlist. Here’s mine.

1. Decouple the Manifest from the Source Code

Store the manifest as a standalone file on Drive and have the server read it at runtime:

export function getDlvChunksManifest() {
  const manifestFile = DriveApp.getFileById(MANIFEST_FILE_ID);
  const jsonText = manifestFile.getBlob().getDataAsString();
  return JSON.parse(jsonText);
}

This decouples data deployment from code deployment entirely. New chunks can be uploaded without touching the Apps Script project. The cost is one extra DriveApp call on startup — sub-100 ms for a ~1 KB file.

2. Offload Deployment to a Cloud Function + Cloud Scheduler

Package the compression and upload logic into a Google Cloud Function triggered by Cloud Scheduler. A service account with Drive write access handles everything on Google’s infrastructure — no developer machine involved. Combined with a decoupled manifest, data updates become fully automated: new data appears → Cloud Function compresses and uploads → runtime manifest on Drive is updated → browser clients pick up new chunks on next page load.

Cloud Functions have a 9-minute timeout (15 minutes for gen2), comfortably fitting the full cycle. The free tier includes 2 million invocations per month.

3. Asynchronous Decompression via Web Worker

Move fflate decompression into a Web Worker to keep the main thread responsive:

const worker = new Worker("/decompress.worker.js");
worker.postMessage(compressedBytes);
worker.onmessage = (e) => {
  const jsonStr = e.data;
  // continue with parsing and indexing
};

This eliminates the 1–3 second UI freeze during decompression. The trade-off is additional complexity in worker bundling — fflate must be available in the worker scope.

4. CI/CD Pipeline Integration

Integrate into GitHub Actions: on push to main with changes to the data snapshot → trigger the Cloud Function. On push with changes to gas/src/ → run build:addon:push. GitHub Environments manage dev/UAT/production clasp configurations, with credentials in CI secrets.

5. Incremental Updates via Chunk-Aware Architecture

Instead of slicing the Base64 stream at byte boundaries, chunk at the logical level — groups of spreadsheets, date ranges, or alphabetical shards. Each logical chunk is independently compressed and uploaded with a version identifier. The client compares local version tags and only downloads changed chunks. This enables partial updates at the cost of less uniform chunk sizes and more complex version management.t

The Bigger Lesson

This architecture exists because of constraints. But here’s the thing I want you to take away: those constraints didn’t limit what we built — they defined it. The four walls of Apps Script — 6-minute execution, string-only return types, a practical payload ceiling, and a tiny cache — forced an architecture that is, in retrospect, better than the “just spin up a backend” alternative. It’s simpler to deploy. It has fewer moving parts. It costs nothing to run. This isn’t the first time I’ve pushed Google Apps Script far past its intended boundaries — I once built an entire LLM fine-tuning pipeline using Apps Script as the orchestration layer for Vertex AI. The constraints didn’t stop that project either. They shaped it. And the pattern generalizes beyond GAS: if you ever need to serve large datasets through an API gateway with tight quotas, the compress-chunk-reassemble pipeline is now in your toolkit.

The toolchain — Vite’s IIFE bundling, the AST extraction plugin, clasp, and gws — bridges the gap between modern JavaScript development and the constrained Apps Script runtime. You get npm dependencies, environment switching, and automated deployment without sacrificing the low-code, Sheets-sidebar deployment model that makes Apps Script so compelling in the first place.

That’s the whole story. I’d love to hear how you’re handling large datasets in your own Apps Script projects — or if you’ve hit the same walls and found different ways around them. Drop a comment below.

Have you pushed the google.script.run payload ceiling yourself? What broke first?

Related Articles

If this deep-dive resonated with you, here are a few more from the toolbox:

• Kickstart Your Apps Script Projects with the Pinnacle of My Development — The Apps Script Engine Template — The original announcement of the template that makes this entire architecture reproducible. Covers the philosophy, the scaffold, and the “why” behind the build pipeline.
• Make Apps Script Fast Again! With Apps Script Engine — How the template enables 10× performance gains in Apps Script web apps by eliminating callback hell, bundling npm dependencies, and providing a proper local dev environment.
• Apps Script Engine v2.0 is Here! Your Workflow Just Got a TypeScript Power-Up — The migration from Jest to Vitest, the addition of full TypeScript support, and what v2.0 means for production-grade Apps Script development.
• How Apps Script Became the Ultimate LLM Fine-Tuning Tool — Another case study in pushing Apps Script far beyond its intended limits: using Google Sheets as a human-in-the-loop interface for Vertex AI fine-tuning pipelines.
• Beat the Clock! Dodging the Maximum Script Runtime in Google Apps Script — The classic approach to the 6-minute problem: chunking work across multiple script executions with triggers, timers, and PropertiesService. A useful contrast to the offline-compression strategy in this article.

Happy building. 🚀

About Me

Get my book Kickstart Google Apps Script. I am an AI-first Google Cloud and Workspace developer and a Workspace Google Developer Expert (GDE), CTO and co-founder of Mentormatic, and founder of Wurkpaces.dev

How I Shipped 341 MB of JSON Through Google Apps Script — and Lived to Tell the Tale was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Most engineering teams have already answered the first question about AI: does it help write and understand code? Yes. Copilot-style assistants are now a normal part of how developers work.

The harder question comes next, and fewer teams have answered it yet: what happens when you want AI involved not just in writing code, but in running the process around it — checking release scope, verifying gates, drafting a change request, summarizing an incident, deciding whether a sprint is actually ready to start?

That’s a different kind of problem. It’s not about model quality. It’s about operations: who’s allowed to trigger this, which systems can it touch, what needs a human sign-off, and can you reconstruct exactly what happened afterward. This is the layer NopsAI is built for, so it’s worth being specific about what that actually means in practice.

AI touches every stage of the SDLC — the tooling around it doesn’t catch up evenly

Across planning, development, testing, release, and operations, the pattern is similar: an LLM can summarize, draft, or flag something useful. Backlog grooming, PR review, test-gap analysis, release notes, incident write-ups — all reasonable jobs for a model.

The complication is that each of these stages has different systems of record, different people with access, and different consequences if something goes wrong. A model that can reason about release readiness isn’t the same as a system that can safely execute the checks, pause for approval, and log what it did — and most teams find that gap only after they’ve already wired an AI step into something that touches production.

The existing tool categories, and where each one actually stops

It’s easy to lump “AI + automation” tools together, but they solve different layers, and it’s worth being precise about that instead of pretending they overlap more than they do:

AI coding assistants (Copilot and similar) operate at the repository level — code generation, explanation, PR-adjacent work. Their job is developer productivity, not cross-system orchestration, and they’re not trying to be anything else.

CI/CD platforms (GitHub Actions, GitLab CI, Jenkins) are the right home for deterministic build/test/deploy work. They’re mature, they’re native at container and pod execution, and there’s no reason to replace them. What they’re generally not built for is per-step LLM profiles, MCP tool access, or caller-scoped authorization before a step runs — that’s just outside their design center.

Workflow automation platforms (n8n, Zapier, Make) are good at connecting services quickly through triggers and nodes. They handle scripts-and-AI-steps-in-one-flow reasonably natively, but GitOps-owned configuration, Kubernetes runtime pools, and run-level evidence tend to be custom work bolted on, not native to the platform.

Agent frameworks (LangGraph, LangChain, CrewAI) give engineers the building blocks — tools, memory, state, routing, human-in-the-loop patterns. They’re strong on agent roles and reusable steps, but identity/SSO, GitOps configuration, and caller-scoped AAA before dispatch generally aren’t what these frameworks are trying to solve; that’s left to whoever builds around them.

None of this is a knock on any of these tools — it’s just that “can an LLM be part of this workflow” and “can this workflow be governed, repeatable, and audited” are separate questions, and most of the current landscape answers the first one, not the second.

The actual bottleneck

Here’s where most teams get stuck, and it’s rarely the model. Turning “an LLM can do this” into “this is safe to run against our systems” surfaces a set of operational problems that coding assistants, CI/CD, workflow tools, and agent frameworks each solve a piece of — but usually not all of, and usually not consistently across a team.

Problem: everyone has the same AI access. Once an agent is wired into Jira, GitHub, and ServiceNow, it’s tempting to give it broad access to all three and let it figure out what it needs per task. In practice, that means a summarization step and a “file a change request” step end up with identical tool access, so a bad prompt or a bad plan can reach further than the task actually required.

The fix isn’t “trust the model more” — it’s scoping access at the step level. A pipeline, step, or task gets an approved LLM profile and an approved MCP tool profile assigned to it specifically. This is one of the things NopsAI’s pipeline model does directly: the summarization step and the change-request step don’t inherit each other’s access just because they’re in the same run.

Problem: no one owns “who’s allowed to trigger this.” Once a workflow can touch production — approve a release, close an incident, modify a runbook — “does this pipeline exist” isn’t the right question anymore. The right question is “is this specific caller allowed to run this specific step against this specific system, right now.” Most tools check the first and skip the second.

NopsAI checks authorization against the calling identity before a step dispatches, not just at the pipeline level — so access control follows the caller, not just the workflow definition.

Problem: pipeline config lives in a UI, not in review. A workflow-automation canvas is fast to build in and easy to quietly change — often by whoever last had edit access, with no review trail. That’s fine for an internal automation nobody depends on; it’s a problem the moment the workflow touches a release or an incident.

NopsAI keeps pipeline definitions GitOps-owned, so a change to what the pipeline does goes through the same review path as any other code change.

Problem: sensitive work runs wherever there’s spare capacity. Source code, credentials, infrastructure state, and production data increasingly pass through AI steps. Treating every one of those steps as “just another API call to a chat model” ignores that the execution environment itself matters — where it runs, what it can reach, what’s logged.

NopsAI dispatches steps to Docker or Kubernetes runners chosen deliberately per step, rather than treating every AI task as an undifferentiated external call.

Problem: AI can quietly become the thing that ships the change. A model drafting a release note is low-risk. A model whose draft gets auto-applied to a production system with no one checking is a different thing entirely — and it’s an easy line to cross by accident once a workflow is “basically working.”

NopsAI treats approval as a step in the pipeline’s own DAG — a human gate the AI-generated output has to pass through before a production-affecting action happens, not a side process someone has to remember to run.

Problem: the evidence ends up in five different places. The CI/CD result is in GitHub. The approval is in ServiceNow. The reasoning is in a chat log someone may or may not have saved. Reconstructing “what happened and why” after the fact means stitching together systems that were never designed to be stitched together.

NopsAI attaches trigger, resolved context, tool calls, model usage, approvals, and outputs to the run itself — one record, not a reconstruction project.

Taken together, this is what “governed AI execution” actually means in practice — not a slogan, but six specific gaps that show up the moment AI moves from answering questions to running operational work.

This is what NopsAI is, concretely — a self-hosted, Git-aware platform where those six things (scoped AI access, caller-checked authorization, GitOps-owned config, controlled runtime, in-pipeline approval, and run-owned evidence) aren’t separate add-ons, but the operating model the pipeline is built around from the start, across planning, releases, incidents, security, reporting, and operations. What that looks like in an actual workflow:

Sprint planning. Today, a scrum master manually checks Jira backlog and velocity, reads Confluence for open design questions, and cross-checks Monday.com for dependencies — before the meeting even starts. With a scheduled NopsAI pipeline, that context gets pulled automatically, a fast LLM profile summarizes story readiness, and a planning brief is ready before anyone opens a tab. The failure mode this addresses isn’t “AI can’t summarize a backlog” — it’s that planning currently starts with manual research instead of decisions, and half-ready stories quietly enter the sprint because nobody caught it in time.

Release preparation. Today, a release manager filters Jira by fix version, checks GitHub for merged PRs, checks CI/CD gate results, updates the Confluence runbook, and files a ServiceNow change for CAB approval — by hand, differently each time depending on who’s doing it. A governed pipeline runs the same checks every time: verify scope against merged PRs, validate CI/CD gates, pause before production, generate release notes and the change request together, and keep it all as one record. The value isn’t that an LLM can write release notes — it’s that scope mismatches, unmerged PRs, and failed gates get caught by the same checks every single release, instead of depending on whoever happens to be running it that week.

Where this genuinely doesn’t matter much yet

Worth saying plainly: if your team’s main use of AI is code generation inside a normal CI/CD pipeline, none of this is necessary. A coding assistant plus GitHub Actions is a complete, sensible setup for a lot of teams, and adding a governance layer on top of that is solving a problem you don’t have yet.

This starts to matter once a few things are true at the same time: a workflow crosses multiple systems (Jira, GitHub, ServiceNow, Confluence, whatever your stack is), different people should have different access to trigger or approve it, AI tool access needs to be scoped rather than blanket, some action in the workflow needs a human sign-off before it’s real, and someone — security, compliance, an auditor, your future self during an incident postmortem — needs to be able to reconstruct exactly what happened and why.

How it relates to what you already have

None of this requires ripping anything out. A Git event can trigger a NopsAI pipeline; that pipeline can check CI/CD status and report back to the repo. Copilot keeps doing code-level work; NopsAI governs the release-readiness process around the feature it produced. An n8n flow or a LangGraph agent can still be the thing that gets invoked — NopsAI is the layer that decides whether it’s allowed to run, what it can access, and what gets recorded when it does.

Starting point

The realistic way to evaluate this isn’t a generic demo — it’s picking one workflow that’s already repetitive, already crosses a few tools, and already causes friction around evidence or approval: sprint-readiness prep, release prep, incident follow-up, QA-blocker triage, or security-remediation coordination are common starting points. Define the trigger, the systems it touches, which steps are deterministic versus AI-assisted, what needs approval, and what evidence needs to be kept — and you have something concrete to test against, rather than a hypothetical.

Where AI Fits in the SDLC — and Why “Assisting” and “Executing” Are Different Problems was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Did Vercel Just Reinvent React Native?

If we all released Software at the pace of Vercel we would be acquired, sunset, and open-sourced by Friday.

Anyway, they’ve shipped again.

It’s called native from Vercel Labs.

Not react “native”.

Not “native” base.

Just native.

Mononymous, like Cher.

Native SDK is a toolkit for building native desktop apps.

You write views in declarative markup and logic in TypeScript or Zig, and the SDK’s own engine draws every pixel into real OS windows.

No browser, no WebView, and no JavaScript runtime anywhere in the binary.

Read that again, because you write TypeScript and it ships without a JavaScript engine.

Hold that thought.

First, here’s an entire working UI:

<column gap="12" padding="16">
  <row gap="8" cross="center">
    <text grow="1">Counter</text>
    <button size="sm" variant="ghost" on-press="reset">
      Reset
    </button>
  </row>
  <row gap="8" main="center" cross="center" grow="1">
    <button variant="secondary" on-press="decrement">-</button>
    <text>{count}</text>
    <button variant="primary" on-press="increment">+</button>
  </row>
  <status-bar>total: {count}</status-bar>
</column>

Here’s where it stops looking like React Native.

No useState. No useEffect. Nobody in a GitHub thread is asking why it fired twice.

The markup is a dumb template: it can read values {count} and shout messages on-press="increment", and that's all it's allowed to do.

Every message from every corner of the UI funnels into one function, update, which takes the current state and the message and returns the next state. The whole window re-renders from that.

If you’ve used useReducer or Redux, it's that, as the entire application.

// src/core.ts, compiled to NATIVE CODE at build time
export type Msg =
  | { readonly kind: "increment" }
  | { readonly kind: "decrement" }
  | { readonly kind: "reset" };

export function update(model: Model, msg: Msg): Model {
  switch (msg.kind) {
    case "increment":
      return { ...model, count: model.count + 1 };
    case "decrement":
      return { ...model, count: model.count - 1 };
    case "reset":
      return { ...model, count: 0 };
  }
}

So where did the JavaScript engine go?

At build time, a TypeScript-to-native transpiler compiles that core straight into the executable.

Node is required to build, then vanishes from the binary like Keyser Söze.

It was never really there.

What’s left is a small example calculator that is 3.6 MB and opens in roughly 100 milliseconds, according to their own macOS measurements.

You could email it. The calculator. Through Gmail.

About Zig: it’s the low-level systems language the entire toolkit is built in, think C with better error messages. You never have to touch it, but if you want to, native init my_app --template zig-core scaffolds the identical counter with its core written in Zig.

It’s not a blank-slate toolkit either.

A catalogue of 45-plus built-in components such as buttons, tabs, dialogues, charts, tables, virtual lists, and a markdown renderer ships with considered typography and spacing, so the scaffolded app looks intentional the first time the window opens.

Every app embeds an automation server.

The agent that wrote your app can launch it, read the live widget tree the way it would read a DOM, click the button it just invented, assert on the real state, and screenshot the window to confirm the app isn’t lying about any of it.

Then, buried in the Web Content docs, the funniest architecture decision of 2026.

Native SDK includes a Bridge. Yes… a Bridge.

JavaScript in the WebView calls native handlers by sending asynchronous JSON messages, capped at 16 KiB. JavaScript on one side. Native on the other. Serialised JSON in the middle.

This community spent half a decade escaping that exact architecture, and Vercel has rebuilt it from memory like a traumatised architect.

Let’s take full inventory, actually.

Declarative views? Check.

Logic driving a native renderer? Check.

An asynchronous serialised JSON bridge to JavaScript? Check.

iOS and Android targets? Experimental, but check.

Congratulations to Vercel on inventing React Native 0.59, an architecture so historic that this community holds conferences about having survived it.

👉 Native SDK

Let Your Agent Use the App It Just Built

Your coding agent writes a login screen, tells you it’s done, and moves on.

It never actually opened the app.

The Maestro MCP fixes that.

It hands your agent an iOS simulator, Android emulator, or physical Android device to drive.

Your agent reads the screen hierarchy, taps through the flow, grabs screenshots, and confirms the feature it just wrote actually works. Then it saves a repeatable end-to-end test, so the same flow doesn’t quietly break next week.

The Maestro Viewer embeds the live device right inside your coding agent, showing every command as it runs.

It all ships inside the Maestro CLI.

Point Claude Code, Cursor, Codex, Copilot, or Gemini at it. For Claude Code, it’s one line:

mcp add maestro -- maestro mcp

Your agent stops saying “this should work” and starts checking.

👉 Maestro MCP

Your OTA Update Is Out for Delivery

Zepto (@ZeptoNow) is the Indian quick-commerce company famous for delivering groceries in ten minutes.

At some point, an engineering team in Bengaluru looked at OTA platforms re-shipping the entire JavaScript bundle for a one-line fix, and treated it like a delivery arriving in eleven minutes.

A production Hermes bundle runs about 20 MB, conservatively.

Push a one-character fix to a million installs, and your CDN moves 20 terabytes: roughly $1,700 of data at CloudFront list price, spent shipping 19.999 MB of JavaScript to phones that already had it, over metered mobile data.

OTA at this scale earned its own talk at React Universe Conf 2025.

So Zepto built Delta.

Delta ships the diff instead.

Binary delta patching (computing the byte-level difference between two files) means the device downloads only the bytes that changed between the bundle it has and the bundle it needs.

The delta-cli computes the patch between releases and uploads it to your own S3 bucket, while delta-server (a self-hosted stack of Lambdas behind CloudFront) decides which patch each device receives, keyed on the appId, jsVersion, and bundleVersion from a manifest.json baked into your app at build time.

On the device, the SDK checks for an update, downloads the patch, verifies it, applies it with bspatch (the applying half of bsdiff, a binary diffing tool) to reconstruct the full bundle, and swaps it in on the next launch.

The swap itself is one override:

// MainApplication.kt
class MainApplication : Application(),
  ReactApplication, IDeltaDelegate {
    override val reactNativeHost: ReactNativeHost =
    object : DefaultReactNativeHost(this) {
      // Delta decides which bundle boots
      override fun getJSBundleFile(): String {
        return Delta.getJSBundlePath()
      }
    }
  override fun onCreate() {
    super.onCreate()
    Delta.initialize(this)
  }
}

According to their own production numbers, Zepto saw 80% of users on the latest version within 24 hours, across 15 million update requests per day, with a P90 (P90 is the value that 90% of all measurements fall at or below) patch download time of 305 milliseconds.

305 milliseconds.

Quicker than your ex moving on to someone who actually has their shit together.

Somewhere in Bengaluru, a rider is doing 60 through traffic with a bag of onions, and he is now only the second-fastest thing at the company.

👉 react-native-delta

React Native Finally Learns to Let Go

The user starts uploading a 40 MB video on mobile data, changes their mind, and taps the X.

That X is your job.

And fetch has no cancel method.

No .stop(), no .nevermind().

That upload finishes in the background, burns through 40 MB of the user’s data plan, and your cancel button is just a placeholder icon.

An AbortController is the only way to kill an in-flight fetch.

You pass its signal into the request, and the X calls abort(). The promise is rejected, the connection dies, and the upload stops transferring bytes.

Your app moves on with its life.

It’s the web platform’s institutionalised form of ghosting.

AbortController has worked in React Native for years, but it was never React Native's code.

Hermes doesn’t ship browser APIs, so at startup, React Native bolts the web globals on in a file called setUpXHR.js, and the AbortController it uses is an npm package abort-controller by Toru Nagashima (@mysticatea).

A package last published 7 years ago.

Which means it predates half the modern API.

AbortSignal.timeout(), AbortSignal.any(), the static AbortSignal.abort(reason), and signal.throwIfAborted() simply did not exist in React Native.

Spotted doing the rounds on X this week, because a core web primitive quietly frozen in 2019 is exactly the kind of thing this ecosystem finds under the sofa.

Now, a commit from @retyui has landed on main.

The unmaintained package (and its event-target-shim sub-dependency) are gone.

In its place, React Native forked the source in-tree, wired it up to React Native’s own EventTarget, and added all four missing methods. The Hermes team did the same thing to the promise package years ago, so there's precedent for adopting abandoned dependencies and raising them as your own.

The payoff is that the patterns you already use on the web now JustWork™:

const controller = new AbortController();

fetch(url, {
  // Cancel manually OR after 5s, whichever first
  signal: AbortSignal.any([
    controller.signal,
    AbortSignal.timeout(5000),
  ]),
});
controller.abort(); // the X, doing its one job

AbortSignal.timeout() gives every request a deadline without a hand-rolled setTimeout, and AbortSignal.any() merges signals so one fetch can be cancelled by whichever fires first.

It landed on main against 0.87, so it will be available in the next React Native release.

And somewhere out there, an X is about to do something for the first time in its life.

👉 Abort Controller

Authored by Luke Farrell and edited by Fran Ríos.

If you’re enjoying this newsletter, why not help us grow? Share it with your friends, family, and even that one coworker who always “forgets” to update dependencies. Use this link: https://thereactnativerewind.com/.

Vercel Reinvents React Native, Sub-Second OTA Updates, and an X Button Finally Doing Its Job was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

⭐ Github Source: https://github.com/dimastatz/audiotrace

When your AI agent types, you can see everything it does. LangChain traces every step, LangSmith replays every run, OpenTelemetry hangs spans off each call. You know what the model saw, what it said, how long it took, and what it cost. The moment that same agent picks up a phone, the lights go out. A voice agent’s entire interaction lives inside audio recordings. The transcript, the customer’s mood, the awkward four-second silence, the moment it talked over the caller, the point where the conversation went sideways — all of it is in there.

But to your existing observability stack, that file is opaque. LangSmith sees the tokens you fed the LLM; it does not see the audio that reached a human ear. So most teams do the only thing they can: they listen to a handful of calls by hand and hope the sample is representative. That doesn’t scale, and it misses the thing that makes voice agents hard — their behavior drifts. You tweak a prompt, swap a model, change a TTS voice, and the agent gets subtly slower, colder, or starts missing intents. No unit test catches it, because the regression lives in the audio.

This series is about closing that gap. In this first post I’ll lay out the mental model; the next two get hands-on with a tricky signal-extraction problem and with wiring voice signals into CI.

The artifact is richer than you think

Here’s what’s actually recoverable from a single call recording:

- **Transcript** — what was said, by whom, with timestamps.

- **Quality** — silence gaps, interruptions, speaking pace, pitch variance.

- **Sentiment** — the caller’s mood, and where it shifted.

- **Latency** — how long each stage (STT, LLM, TTS) took to respond.

- **Cost** — what the call cost, attributed per stage.

- **Events** — the detected intent, whether the caller dropped off, and compliance flags.

That’s a lot of signal locked inside one file. The reason teams rebuild this from scratch at every company is that prying it loose means bolting together speech recognition, speaker separation, audio analysis, a sentiment model, and a pricing sheet — and then maintaining all of it.

Two ways to pull meaning out of audio

The key insight that makes this tracable: there are really two different kinds of questions you can ask of audio, and they want two different tools.

1. Measure it — classical signal processing. Deterministic math run straight on the waveform: energy, pitch, the length of a silence. Cheap, exact, no training data. It shines for physical questions:

- How long was the pause?

- How fast did someone speak?

- Is this voice high-pitched or low?

You *measure* the answer instead of guessing at it.

2. Estimate it — learned models. Statistical systems like Whisper or a sentiment classifier that have ingested enormous amounts of data and *estimate* an answer. They own everything that turns on meaning rather than physics:

- What words were said?

- Who is speaking?

- Is the caller upset?

No hand-written rule survives real speech here — you need a model. Most of the craft is knowing which question belongs to which bucket: reach for a model to **estimate meaning**, for signal processing to **measure physics**. (In the next post you’ll see that when a model isn’t available, a measurement can sometimes stand in for it — that turns out to be a surprisingly useful trick.)

One report, split along that line

I packaged this into a small open-source library called [AudioTrace](https://github.com/dimastatz/audiotrace). You hand it a recording; it hands back one structured, typed report — split along exactly that measure-vs-estimate line. The acoustic layer (silence, pace, pitch) is signal processing; the semantic layer (transcript, sentiment, intent) is models.

pip install audiotrace

import audiotrace

report = audiotrace.analyze(

audio=”call_recording.wav”,

metadata={“agent_version”: “v2.1”, “provider”: “vapi”},

)

print(report.quality.overall_score) # 0.87

print(report.quality.speaking_pace_wpm) # 168.0

print(report.sentiment.caller_frustration) # False

print(report.latency.total_ms) # 4200

print(report.events.drop_off) # False

print(report.cost.total_usd) # 0.063

The return value is a Pydantic `CallReport`, so it’s typed, validated, and trivial to serialize. You can emit it as OpenTelemetry spans, hang it off your LangChain and LangSmith traces, or assert on it in a CI check — which is exactly where this series is headed.

One decision shaped everything: it runs locally

Call recordings are about as sensitive as data gets. So AudioTrace runs entirely on your machine — no audio leaves the box, and the open models download once. Privacy here shouldn’t be an upgrade you pay for; it should be the default.

What’s next

The two-layer model sounds tidy, but the interesting part is what happens when the “right” tool isn’t available. In the next post I’ll walk through a concrete example: labeling **who is speaking** without the gated model everyone reaches for — and why a few dozen lines of pitch measurement beat it for the common case.

⭐ The repo is at https://github.com/dimastatz/audiotrace.

Issues and PRs welcome — it’s early, and provider integrations are exactly the kind of contribution that helps most.

Keep building!

Your AI Voice Agent Is a Black Box. Here’s How to Open It. was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

A local-first QA prototype that scores field mappings, rejects risky candidates, validates accepted patches locally, and opens a reviewable pull request only after PASS.

V0.3 of API Drift Healer successfully proved a concept: it could handle a narrow contract-drift case (like mapping userEmail to email_address), patch the outdated field, rerun the test, and open a pull request, but only if the healed test passed locally.

However, a critical question remained unresolved: How should the tool actually decide if a field mapping is safe enough to attempt in the first place? Mapping userEmail to email_address based on an OpenAPI update seems completely reasonable. But what happens when the tool encounters pairs like displayName to email_address, firstName to last_name, or userId to customer_id? Just because two fields share the same string type or contain loosely related words doesn't mean they represent the same underlying data.

This is exactly what V0.4 addresses. The tool now introduces a deterministic field matcher designed to rigorously evaluate a candidate rename long before it touches your test files. The original rule has not changed: No PASS, No PR. But V0.4 adds another requirement: No strong match, no patch.

What Changed in V0.4

V0.3 proved the basic workflow:

• detect a failing test
• patch one outdated field
• rerun the healed test
• generate a report
• open a pull request only after PASS

V0.4 adds a dedicated matching engine with:

• field-name normalization
• rule-based field concepts
• qualifier conflict detection
• runtime value-type checks
• OpenAPI type checks
• runtime value-format detection
• OpenAPI format checks
• deterministic scoring
• structured SAFE PATCH or REJECT decisions
• 48 automated tests

The real improvement is not that the tool can accept more mappings.

It is that the tool can explain why it accepts one mapping and rejects another.

Recreating the Contract Drift

The demo still starts with a small and familiar mismatch.

The local API expects:

{
  "name": "Test User",
  "email_address": "qa_user@example.com"
}

The OpenAPI contract defines email_address as required:

required:
  - email_address

It also defines its type and format:

email_address:
  type: string
  format: email

But the YAML test case still sends:

body:
  name: Test User
  userEmail: qa_user@example.com

The test runner sends the outdated payload and receives:

[FAIL] Expected 201, got 400
Error: missing required field: email_address

Despite the [FAIL] output, the API is actually running fine, the endpoint works, and the expected status code logic is still correct. The only real problem is that our request payload is simply outdated. Instead of a manual debugging session, this exact failure now becomes the perfect input for the V0.4 deterministic matching flow.

The V0.4 Safe Healing Flow

The local flow now looks like this:

Original test FAIL
        ↓
OpenAPI drift detected
        ↓
Candidate field rename found
        ↓
Field names normalized
        ↓
Field concepts checked
        ↓
Qualifier conflicts checked
        ↓
Type and format checked
        ↓
Deterministic score calculated
        ↓
SAFE PATCH or REJECT
        ↓
Healed test generated only when accepted
        ↓
Healed test PASS
        ↓
Report generated
        ↓
Optional GitHub PR

Changing one YAML key is easy.

The difficult part is deciding whether that change should be attempted at all.

V0.4 moves that decision into a separate module:

field_matcher.py

The matcher always returns a structured result.

Its final decision is one of two values:

SAFE PATCH

or:

REJECT

The same input produces the same result.

There is no LLM deciding which field should be renamed.

Normalizing Different Field Styles

APIs and test files do not always use the same naming style.

The same field may appear as:

• userEmail
• user_email
• user-email
• UserEmail
• user email

The matcher first converts field names into normalized tokens.

For example:

normalize_field_name("userEmail")

returns:

["user", "email"]

And:

normalize_field_name("email_address")

returns:

["email", "address"]

This allows the matcher to compare structured parts of a name instead of comparing only the raw strings.

The current normalizer handles:

• camelCase
• snake_case
• kebab-case
• PascalCase
• spaces
• common acronym patterns

Normalization is only the first signal.

Two fields can still share a token without representing the same value.

Rule-Based Field Concepts

The matcher uses a small and explicit concept dictionary.

Current concepts include:

• email
• phone
• name
• id
• date

For example:

userEmail

and:

email_address

both contain the concept:

email

That supports the mapping.

But:

displayName

and:

email_address

do not share a concept.

Both values may be strings, but they represent different data.

This is rule-based concept matching.

It is not an embedding model and it does not use an LLM.

The concept list is intentionally small in V0.4. It can be extended later as more field patterns are tested.

Qualifier Conflicts

Fields may share the same main concept and still describe different values.

Consider:

firstName -> last_name

Both fields contain the name concept.

A simple matcher may treat that as a good sign.

But their qualifiers conflict:

first -> last

V0.4 treats this as a hard conflict and rejects the candidate.

The same guard can reject mappings such as:

• primaryEmail -> secondary_email
• createdAt -> updated_at
• userId -> customer_id

A high similarity score cannot override a hard qualifier conflict.

This rule matters because string similarity alone can make bad mappings look more convincing than they really are.

OpenAPI Type and Format Checks

Field names are not the only source of evidence.

The matcher also checks:

• the runtime value
• the detected runtime type
• the OpenAPI target type
• the detected runtime format
• the OpenAPI target format

For the safe email example, the source value is:

qa_user@example.com

The runtime type is:

string

The target OpenAPI field also expects:

type: string
format: email

The runtime value looks like an email, and the target field expects an email.

Both signals support the candidate mapping.

Now consider:

displayName = "Test User"

The value is still a string.

But it does not look like an email.

If the target schema expects:

type: string
format: email

the matcher detects a format conflict.

This is why checking only the Python type is not enough.

V0.4 currently detects formats such as:

• email
• UUID
• date-time
• phone
• plain or unknown string

The format checks use deterministic rules and heuristics.

They are safety signals, not semantic proof.

Runtime format detection also depends on the quality of the test data. Placeholder values or intentionally invalid fixtures may cause a reasonable mapping to be rejected.

For this prototype, rejecting an uncertain candidate is preferable to applying the wrong patch.

Deterministic Scoring

After collecting the matching signals, the matcher calculates a deterministic score.

The current acceptance threshold is:

0.700

The score considers signals such as:

• shared field concepts
• type compatibility
• format compatibility
• normalized string similarity
• qualifier conflicts
• strong matching anchors

However, the numerical score is not the only rule.

A candidate must also:

• have a strong matching anchor
• avoid hard qualifier conflicts
• avoid hard format conflicts

A candidate can therefore be rejected even when some of its individual signals look positive.

The current threshold is a conservative prototype value.

It will need further calibration against a larger set of real field mappings and rejected examples.

The score should be treated as an explainable decision signal, not a universal probability of correctness.

Example 1: Accepting an Email Rename

Candidate:

userEmail -> email_address

Decision:

Score: 0.772
Threshold: 0.700
Confidence: High
Decision: SAFE PATCH

Supporting evidence includes:

• shared concept: email
• source value type matches the OpenAPI target type
• detected email format matches the OpenAPI target format
• no hard qualifier conflict
• no hard format conflict

The healer generates:

body:
  name: Test User
  email_address: qa_user@example.com

It then runs the healed test:

[PASS] Expected 201, got 201

The mapping passes the current matcher rules and the resulting test passes locally.

That makes it eligible for the report and optional PR flow.

Example 2: Rejecting Name-to-Email

Candidate:

displayName -> email_address

Decision:

Score: 0.158
Threshold: 0.700
Confidence: Low
Decision: REJECT

Reasons include:

• no shared field concept
• source value does not match the target email format
• no strong matching anchor
• score is below the threshold
• hard format conflict detected

The tool does not create a healed file.

It does not patch the test.

It does not create a branch or pull request.

This is an important part of V0.4.

The matcher is designed to reject uncertain candidates, not to force every failure into a generated fix.

Example 3: Rejecting First Name to Last Name

Candidate:

firstName -> last_name

The fields share the name concept.

Their normalized strings are also somewhat related.

But the matcher detects:

Qualifier conflict: first -> last

Final result:

Score: 0.587
Decision: REJECT

The shared concept is not enough to override the qualifier conflict.

Testing the Safety Rules

A matching engine should not only prove that accepted cases work.

It should also prove that risky cases are rejected.

V0.4 currently includes:

45 matcher unit tests
+
3 healer integration tests
=
48 automated tests

The matcher tests cover:

• field-name normalization
• field concepts
• qualifier conflicts
• runtime type detection
• runtime format detection
• OpenAPI type compatibility
• OpenAPI format compatibility
• deterministic scoring
• accepted mappings
• rejected mappings

The integration tests verify the full healer behavior.

Safe case:

userEmail -> email_address
Result: healed file created

Unsafe format case:

displayName -> email_address
Result: rejected, no healed file

Unsafe qualifier case:

firstName -> last_name
Result: rejected, no healed file

Current test result:

Ran 48 tests

OK

These rejection tests are as important as the successful case.

A matcher that only tests its happy path is mostly testing its optimism.

Applying the Patch

Once a candidate passes the matching checks, the actual patch is intentionally simple:

request_body[new_field] = request_body.pop(old_field)

For this demo, it changes:

userEmail: qa_user@example.com

into:

email_address: qa_user@example.com

The output is written to:

api_test_case.healed.yaml

This file is generated output.

It can be recreated from the original YAML test during each healer run.

The complexity belongs in the decision, not in the assignment.

The Main Guardrail

The tool continues only when all current safety conditions are satisfied:

Original test fails
+
Exactly one required OpenAPI field is missing
+
Exactly one request field is invalid
+
Matcher returns SAFE PATCH
+
Healed test passes locally

If the candidate is rejected, the process stops.

If the healed test still fails, the process stops.

If multiple required fields or invalid fields are found, the process stops.

No branch.

No PR.

No pretend-success.

Rejecting an uncertain mapping is cheaper than confidently patching the wrong field.

And the original rule still applies:

No PASS, No PR.

Making the Decision Explainable

A patch alone is not enough.

The reviewer should be able to understand:

• what failed
• what changed
• why the mapping was accepted
• how strong the matching evidence was
• whether the healed test passed

V0.4 generates:

heal_report.md

The report includes:

• root cause
• applied fix
• original status
• healed status
• match score
• safety threshold
• confidence
• final decision
• matching evidence

Example:

Match score: 0.772
Safety threshold: 0.700
Confidence: High
Decision: SAFE PATCH

The evidence section may include:

Shared concept: email
Value type matches the OpenAPI target type
Email format matches the OpenAPI target format
Normalized string similarity: 0.435
No hard conflict found

The report does not simply say that a patch was generated.

It shows why the candidate passed the current checks.

Local Report Mode and PR Mode

By default, the tool runs locally:

python auto_healer.py

Local mode:

• runs the failing test
• evaluates the candidate mapping
• creates a healed test only when accepted
• runs the healed test
• generates the report
• skips GitHub PR creation

PR mode must be enabled explicitly.

On macOS or Linux:

CREATE_PR=true python auto_healer.py

On Windows PowerShell:

$env:CREATE_PR="true"
python auto_healer.py

Before creating a branch, the tool checks that the Git working tree is clean.

If unrelated local changes are present, the PR flow stops.

Generated fixes should not be mixed with unrelated work from the developer’s local environment.

The Human-Reviewable Pull Request

When PR mode is enabled and the healed test passes, the tool can open a GitHub pull request.

The PR includes:

• root cause
• applied field rename
• validation result
• confidence
• match score
• safety threshold
• matching evidence

The tool does not push directly to main.

It does not merge automatically.

A human reviewer keeps control over the final decision.

The V0.4 implementation was also developed through a protected feature-branch workflow and released as:

v0.4.0

Current Limits

API Drift Healer is still a focused prototype.

The current flow handles:

1 missing required field
+
1 invalid existing field

It does not yet solve multiple field mappings in the same request.

For example, a request containing:

userEmail
phoneNumber
firstName

while OpenAPI requires:

email_address
phone_number
first_name

needs a more advanced matching process.

A future version should:

1. compare every invalid source field with every missing target field
2. calculate a candidate score matrix
3. remove candidates with hard conflicts
4. find the safest one-to-one global assignment
5. reject the full patch when the result remains ambiguous

The current prototype also has limited support for:

• nested objects
• arrays
• optional fields
• newly added required fields
• removed fields
• type changes
• multiple endpoints
• placeholder values with unrealistic formats

The goal is not to automatically rewrite everything.

The goal is to inspect broadly and automatically change only candidates that pass strict, explainable checks.

Roadmap

The next major steps are:

• multiple-field candidate matching
• one-to-one global field assignment
• nested object and array support
• CLI arguments and configurable safety modes
• Postman collection adapter
• VS Code .http adapter
• pytest and requests adapter
• GitHub Actions integration

The core matcher should remain independent from the input format.

A YAML test, Postman collection, .http file, or Python test should eventually be converted into the same internal model.

The same drift engine could then analyze each format without duplicating the matching logic.

Where AI Fits Later

The matcher remains deterministic.

That is intentional.

For this workflow, I do not want an LLM deciding whether a test field should be silently rewritten.

A safer architecture is:

Deterministic code decides.
Tests validate.
AI explains.
Humans review.

AI may later summarize matching evidence, explain rejected candidates, or help a reviewer understand a complex drift report.

But it should not bypass deterministic safety checks.

Tests still need to prove that the accepted patch works.

Letting an LLM freely edit integration tests is how YAML becomes modern art.

Final Thoughts

The field replacement itself is not the interesting part.

Changing userEmail to email_address takes one line of Python.

The real work happens before and after that line:

• detecting the contract drift
• evaluating the candidate mapping
• rejecting conflicting or weak matches
• applying the accepted patch
• running the healed test
• explaining the decision
• opening a pull request only after PASS
• keeping a human in the loop

That is what V0.4 was built to prove.

V0.3 showed that a generated patch could be validated locally. V0.4 focuses on the decision that comes before the patch: whether the candidate mapping should be attempted at all.

Trust does not come from automation alone.

It comes from automation that can explain why it made a change and show that the updated test passes.

For API Drift Healer, that trust still starts with one rule:

No PASS, No PR.

Building a Safer API Drift Healer: Deterministic Field Matching in V0.4 was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

I previously wrote about some Kubernetes namespace policy management tools I built on ConfigHub: namespace-manager, rbac-manager, network-policy-manager. Next, I built some workload-oriented management tools.

Workloads have some interfaces that are specific to them, such as command-line flags, environment variables, application configuration, probe endpoints, and so on. The configuration of those properties is best performed with the schemas and constraints for those interfaces in hand. I’ll write another post about that.

Workloads also have security and operational aspects that are cross-cutting and at least partly determined by the deployment environment, such as operating-system capabilities, computational resources, scaling automation, network access, and observability integrations. The workload may have some requirements of these aspects, but their configuration is not unique to the workload and is often at least partly determined by factors outside the workload implementation, such as the choice of autoscaling tool or observability tool, as well as the application or infrastructure platform, which is Kubernetes for this discussion. Typically the security posture is determined at the organization level, as well, or at least at a larger scope than just one workload.

With configuration as data, not all changes to configuration need to be implemented in a single monolithic generator. They can be decoupled and performed by interoperable tools, just as when building tools against APIs.

To demonstrate this, I built a suite of tools, in one day, each with its own agent skills:

• workload-manager: ensures workloads follow best practices, such as resource specifications, probes, pod disruption budgets, and security context
• scheduling-manager: checks and sets node selectors, node affinity, and tolerations according to some pre-defined profiles
• observability-manager: manages Prometheus ServiceMonitor resources and Open Telemetry collector sidecar containers
• autoscale-manager: checks and edits HorizontalPodAutoscaler and/or KEDA resources, and can convert HPA to KEDA

All of these tools can enforce desired postures, as well, by installing ConfigHub Triggers via guardrails commands.

As with the policy tools, these tools can provide lists of the relevant resources across clusters, snapshot summaries, and detailed lists of findings.

For example, from the workload manager:

CLUSTER                                 WORKLOADS  PDBS  UNITS  GATED  UNAPPLIED
dev-cluster                             19         0     33     0      21
prod-cluster                            19         0     38     0      24

2 clusters, 38 workloads, 0 pdbs, 71 units (0 gated, 45 unapplied)

SEVERITY  ANALYZER   CLUSTER          NAMESPACE  KIND         NAME     MESSAGE
...
HIGH      security   prod-cluster     appvote    Deployment   vote     container "vote" does not set runAsNonRoot: true
HIGH      security   prod-cluster     appvote    Deployment   worker   container "worker" allowPrivilegeEscalation is not false
HIGH      security   prod-cluster     appvote    Deployment   worker   container "worker" does not set runAsNonRoot: true
...
MEDIUM    probes     dev-cluster      appchat    Deployment   backend  container "backend" no readiness probe
MEDIUM    resources  dev-cluster      appchat    Deployment   backend  container "backend" no cpu limit
MEDIUM    resources  dev-cluster      appchat    Deployment   backend  container "backend" no cpu request
MEDIUM    security   dev-cluster      appchat    Deployment   backend  container "backend" readOnlyRootFilesystem is not true
...
LOW       hygiene    dev-cluster      appvote    Deployment   redis    container "redis" terminationMessagePolicy is not FallbackToLogsOnError
...
39 workloads (0 pass, 21 warn, 18 fail)
...
  dev-cluster/appchat Deployment/backend:
    [hygiene] container "backend" terminationMessagePolicy is not FallbackToLogsOnError
    [probes] container "backend" no liveness probe
    [probes] container "backend" no readiness probe
    [resources] container "backend" no cpu limit
    [resources] container "backend" no cpu request
    [security] automountServiceAccountToken is not false
    [security] container "backend" allowPrivilegeEscalation is not false
    [security] container "backend" does not drop ALL capabilities
    [security] container "backend" does not set runAsNonRoot: true
    [security] container "backend" readOnlyRootFilesystem is not true
    [security] container "backend" seccompProfile is not RuntimeDefault
...

(terminationMessagePolicy is a pet feature of mine. I find it useful.)

If this were a typical policy tool and you were using helm, cdk8s, tanka, or other configuration as code tool, then you’d have to track down the right repo and configuration files, render the configuration and feed it to the policy tool, ask your AI agent to implement fixes for the correct environment(s), review the changes, and render the configuration again, for all potentially affected environments, and feed it to a validation tool.

But this is configuration as data, so those checks ran against fully materialized configuration and you can just change the configuration directly with the tools.

Want a PodDisruptionBudget?

% ./bin/cub-workload ensure-pdb appchat-prod/frontend --max-unavailable "10%" -o table
Dry run — would create PDB Unit appchat-prod/frontend-pdb:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: frontend-pdb
  namespace: appchat
spec:
  maxUnavailable: 10%
  selector:
    matchLabels:
      app: frontend

Re-run with --commit --change-desc "…" to create the Unit. It is not applied until you apply it.

That automatically matched the namespace and labels of the specified workload, similar to kubectl expose, which makes it even simpler than kubectl create pdb. For a single workload, an AI agent could also author the resource from whole cloth, especially if it had a sample to start from, as in our agent skills or in our package of resource samples.

Want spreading via pod anti-affinity? ./bin/cub-workload ensure-spread appchat-prod/frontend. CPU and memory resources? set-resources. Probes? set-probes . Security context and disabled service account token? harden . There are lots more features I don’t have space to show in this post. There’s no need to write YAML by hand when tools are perfectly capable of doing it. Remember, these tools are operating on automatically versioned storage, not operating on the clusters directly.

To be agent-friendly, the commands execute in dry-run mode and output JSON by default. Agents can execute the commands and generate summary tables regarding the current state and changes they made. With approval, they can “commit” changes by performing the same operations without dry run. The changed configuration will be validated automatically, and humans can still review and approve the changes before releasing them to the clusters, and can revert or adjust them as needed.

I’m also experimenting with writing higher-level intent to the resources using annotations. I’ll write about that in a future post once I have more experience with it.

Under the hood, the tools work through a combination of ConfigHub filters, functions, such as set-container-resources and set-pod-container-security-context-defaults, client-side analysis, client-side mutations using the function SDK, saved function invocations, triggers, and other ConfigHub features.

Because the configuration data is authoritative, these tools can interoperate with other tools that read and write the configuration data, including other ConfigHub functions, such as set-container-image-reference and set-replicas, and AI agents editing the configuration. Tools can be single-purpose or multi-purpose, as desired, but they don’t need to be monolithic, covering every field of Kubernetes Deployment, StatefulSet, and DaemonSet resources. The tools can also be versatile. They can analyze, summarize, generate, backfill, promote, validate, and enforce.

And because we store configuration in a database instead it being sprawled across many git repositories owned by different teams, the tools don’t need to clone lots of repositories, make ad hoc edits, commit and push changes, and create lots of PRs. Instead, they can call APIs. I’ll write another post about the “initiative” system that we’re working on for changes that span multiple teams.

I’m sure the tools still have rough edges and need more work to make them usable — I’ve barely had time to try them yet, but the point is that we CAN build and use tools if we’re not using configuration as code. The tools can operate on variants of the same application across clusters, and across multiple different applications, not just a single application, using standard APIs. Composable, interoperable tools are more reusable than all-encompassing templates.

That last point is pretty important. Yes, for one application, you could use a heavily parameterized template library to create, say, a HorizontalPodAutoscaler, and add some values to drive it. But every Helm chart exposes different input values, so there’s no standard API that one could build tooling around.

The common alternative automated mutation methods are less transparent and more complicated operationally:

• kustomize overlays
• mutating dynamic admission control
• Operators (e.g., to inject sidecar containers or create PDBs)

ConfigHub makes it possible to directly modify configuration while maintaining multiple similar-but-different variants of the same configuration through mechanisms that help keep them in sync. That’s one of the key challenges we’re solving with ConfigHub.

I think the approach of building tools on top of configuration as data has a lot of promise. Try the tools and let me know what you think. Or build your own and let me know how they turn out.

What cross-cutting customizations do you frequently need to make to Kubernetes workload configurations? What are common changes that aren’t supported by helm charts you use? How do you override them? What common operational changes do you need to make beyond just replicas and resources? How often do people in your organization “break glass” and change clusters directly? What are the reasons? Have you migrated from HPA to KEDA, or Ingress to Gateway, or Nginx Ingress to Traefik? Do you have operations, platform, or security teams that need to modify configurations owned by application teams? What kubectl commands or other tools should we adapt to ConfigHub?

Reply here, or send me a message on LinkedIn, X/Twitter, or Bluesky, where I plan to crosspost this.

You could also try out ConfigHub, which is now in preview.

If you found this interesting, you may be interested in other posts in my Kubernetes series or my IaC / declarative configuration series.

Building tools to manage cross-cutting concerns in Kubernetes was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.