Grab Tech

Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption

Fri, 10 Jul 2026 00:00:00 +0000

Introduction: The evolution of Grab’s Data Lake

At Grab’s scale, managing petabytes of data across billions of S3 objects demands more than a storage layer. It demands a robust architectural primitive that supports the high-concurrency needs of a modern “Lakehouse.” Our goal is full storage-compute separation, leveraging S3 as an elastic foundation for both near-real-time metrics and large-scale batch transformations.

For years, the vast majority of our tables were Hive Parquet, managed through the Hive Metastore with a directory-based layout. This model served us well, but as data volume grew, the directory-and-metastore approach became the limiting factor. We are now transitioning to a table-centric architecture built on modern table formats, treating data as a first-class primitive to ensure consistency and performance across our internal data transformation platforms: Slide, which powers batch transformations, and Hugo, which handles online-to-data-lake ingestion. Along the way, we also built the UnifiedSparkCatalog, a unified Spark catalog that hides table-format differences from users entirely, which we are open-sourcing alongside this post.

The catalyst for change: Challenges with Hive Parquet

For years, Hive Parquet was the backbone of our Data Lake, representing the vast majority of our tables. However, as data volume scaled, the architectural limitations of directory-based storage became apparent. We identified four primary bottlenecks:

Catalog latency: The Hive Metastore (HMS) became a centralized failure point. High concurrency during metadata access led to O(n) listing overhead, where query planning time scaled linearly with partition count, crippling throughput.
The small file problem: The directory layout left us with severe file fragmentation. Certain Machine Learning (ML) datasets had an average file size under 1 MB, with thousands of files in each partition. At this scale, the overhead of S3 object listing and metadata request latency drove up Application Programming Interface (API) costs and slowed scan operations.
Operational toil: Data engineers faced constant manual overhead for partition registration. Without native ACID support (no native UPSERT or DELETE), teams relied on complex workarounds to manage data changes carefully.
The broken information loop: A fundamental disconnect existed between the catalog and storage. Because the HMS, not the storage layer, was treated as the source of truth, direct S3 modifications frequently left the catalog stale and out of sync with the actual state on disk.

Why Iceberg? Strategic alignment and future-proofing

We evaluated several open table formats before selecting Apache Iceberg as our default. The deciding factors came down to community governance, engine compatibility, and long-term flexibility.

Recent industry momentum, including growing cloud-native support for Iceberg, further validates this direction. We are positioning Grab to be format-agnostic in the long term, but Iceberg provides the most mature foundation today.

Adopting Iceberg at scale

Migrating an established lake is not a flag flip. Our challenge was rolling out Iceberg across a lake that was overwhelmingly Hive Parquet, queried by many engines and teams, without breaking the downstream consumers that depended on those tables. Rather than converting everything at once, we moved the highest-value tables first. The efficiency gains across our production workloads have been substantial. Here are representative examples:

Query performance via Z-ordering: On a high-traffic navigation dataset, we achieved roughly a 10x improvement in query runtime. Z-ordering co-locates rows with similar values across specified dimensions, enabling Trino to leverage data skipping and min/max statistics to prune irrelevant files during query planning. This reduced query runtime from 70 seconds to 6 seconds.
S3 API cost reduction: For a heavily queried operations table, daily S3 API costs were reduced by up to 95% with no changes to the queries themselves. Larger file sizes and the elimination of expensive object listing during query planning drove most of the savings.
Compute savings: For a dataset used in funnel analysis, we reduced cluster resource usage by approximately half. A separate ML feature pipeline also improved feature freshness for downstream models.

The UnifiedSparkCatalog: Making mixed formats transparent

Migrating to Iceberg solved our storage and metadata problems, but it surfaced a new one at the developer-experience layer. Modern table formats like Delta, Iceberg, and Hudi each implement their own custom catalog that extends Spark’s SessionCatalog. In a standard Spark runtime, only one catalog implementation can be set as the default spark_catalog. Supporting additional formats requires explicit catalog declarations, meaning users must reference tables with format-specific prefixes like iceberg_catalog.schema.table or delta_catalog.schema.table.

With Iceberg, Delta, Hudi, and Hive tables now coexisting and tables actively migrating between formats, this created two problems: engineers had to know the underlying format of every table they queried, and any format migration silently broke every downstream query that hardcoded a prefix.

The UnifiedSparkCatalog is our answer. It is a unified Spark catalog that abstracts the complexity of working with mixed table formats so users never need to think about which format a table uses. We took inspiration from Trino’s Table Redirection, a feature that transparently points a query at the right connector when a table’s format differs from the catalog it was queried through. Our Spark equivalent works as follows:

How it works

Table detection: The catalog loads metadata from the Hive Metastore.
Format identification: A TableTypeDetector utility identifies the format based on metadata properties (e.g., the provider field) or path-based inference.
Operation routing: The catalog delegates the operation to the correct format-specific catalog (Iceberg’s SparkCatalog, Delta’s DeltaCatalog, etc.) without requiring any prefix from the user.

Key design decisions

Lazy initialization: Catalogs for each format are initialized only when first needed, reducing startup overhead. If a format’s JAR is missing from the classpath, initialization continues gracefully. The catalog simply skips that format rather than failing the entire session.
Naming as spark_catalog: The catalog reports its name as spark_catalog because Spark treats this name specially for legacy Hive Data Manipulation Language (DML) operations. Many internal Spark code paths check for this exact name to determine whether to use Hive-compatible logic for inserts, updates, and deletes. Using any other name would break legacy Hive table operations.
Catalog reuse: Before creating a new catalog instance, the system checks whether one already exists in Spark’s catalog manager. This preserves compatibility with plugins like OpenLineage, which inspect catalog class types for lineage extraction.
Fallback behavior: If a table is not found in the expected format-specific catalog, the system falls back to the base session catalog, ensuring robust behavior for standard Hive tables.

We are open-sourcing UnifiedSparkCatalog alongside this blog post. The code and documentation are available here.

Lessons learned and overcoming hurdles

Scaling Iceberg across a large ecosystem revealed several technical nuances:

Hive lock contention: We encountered “zombie locks” in the HMS that blocked commits. We traced this to a low read timeout on the metastore side under high load. Adjusting retry intervals and increasing the timeout resolved the issue.
Timestamp handling: Spark 3.4 introduced TIMESTAMP_NTZ (no time zone), while Iceberg defaults to TIMESTAMP_LTZ (local time zone). This caused compatibility issues with legacy Hive views. We resolved it through a custom migration workflow and targeted patches to our Trino deployment to ensure consistent casting.
Storage tier costs: Generating Iceberg metadata involves reading historical data, which can trigger a one-time cost spike as files move between S3 storage tiers. To manage this, we prioritize migrations based on a table’s scan frequency and API operation costs rather than migrating the entire lake at once.

Conclusion: The road ahead

Apache Iceberg is now foundational to Grab’s data strategy. It is the default format for Slide and Hugo, and adoption is expanding across our compute platforms.

Looking forward, we are experimenting with Storage Partitioned Joins to eliminate shuffle stages in Spark and monitoring the Apache XTable project to maintain interoperability between formats. Our journey does not end with adoption. We will continue contributing back to the ecosystem, starting with the upcoming release of the UnifiedSparkCatalog.

Acknowledgments: This journey was made possible by the dedicated efforts of the Data Engineering, Infrastructure, and Search & Personalization teams at Grab.

Join us

Grab is Southeast Asia’s leading superapp, serving over 900 cities across eight countries (Cambodia, Indonesia, Malaysia, Myanmar, the Philippines, Singapore, Thailand, and Vietnam). Through a single platform, millions of users access mobility, delivery, and digital financial services, including ride-hailing, food delivery, payments, lending, and digital banking via GXS Bank and GXBank. Founded in 2012, Grab’s mission is to drive Southeast Asia forward by creating economic empowerment for everyone while delivering sustainable financial performance and positive social impact.

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Migrating Counter Service storage: Design choices and learnings

Fri, 03 Jul 2026 00:00:00 +0000

Introduction

Counter Service is used across Grab’s anti-fraud platform to answer time-windowed count questions, such as recent ride requests by a user or failed payment attempts on a card. The service handles tens of thousands of queries per second (QPS) with about a billion requests per day, while maintaining strict requirements around latency and reliability to support real-time fraud rule evaluation.

For most of its life, Counter Service was backed by a wide-column database that served the workload reliably as the service scaled. As part of a broader infrastructure review mandated at an organizational level, our database team evaluated alternatives to this storage that many services relied on, including Counter Service. Based on their assessment, Aerospike emerged as a good fit for our use-case. We also used the migration as an opportunity to decouple storage concerns from business logic, a necessary first step for this migration, and one that would reduce the effort required for future storage changes. As part of the same effort, we revisited the data model and access patterns in detail, which helped us identify and apply several straightforward optimizations.

This post walks through how we did it. What we built on the reader-side to make the migration safe, how we redesigned the writer-side data model around the new backend, and what we ran into during the gradual rollout.

Setting the stage

Counter data is stored in three time granularities: 15-minute, hourly, and daily buckets. A typical read would be along the lines of, “give me the count for key X over the last 90 minutes”, which the service decomposes into the smallest possible set of buckets, one hourly in the middle, a few 15-minute buckets at the edges, fetches them, and sums.

In the original setup, each granularity was stored in a separate table with a composite primary key:

TABLE daily_count (
    key      TEXT,         -- partition key
    day_ts   TIMESTAMP,    -- clustering key
    count    BIGINT,
    PRIMARY KEY (key, day_ts)
);

The clustering column gave us convenient range queries, that is needed for the Counter Service. On the write path, each incoming counter event triggered a read-modify-write, three parallel SELECT across the three tables, an in-memory increment, then a batch write. This produced four network round-trips per event.

As this service is a core part of Grab’s fraud detection ecosystem and handles high query volume, migrating its underlying storage required a careful rollout plan. We had three requirements:

Ramp traffic to the new backend gradually and roll back at any point with a config change.
Monitor both the original and new storage paths to verify data integrity before switching over.
Complete the migration without downtime.

We also wanted the migration machinery to be reusable for future storage changes. The migration is divided into three workstreams, which we’ll walk through below:

Preparing the reader service.
Identifying the best integration mechanism for the new storage.
Updating the writer pipeline.

Reader: Separating the data access layer

The reader is a Rust service. Before any migration work began, the reader’s business logic had tight coupling with the storage layer. Session creation, query building, fan-out orchestration, and the data types those queries returned were all intertwined in a single flat file. The main application state struct (AppState) held a raw database session handle and prepared query references. Every handler, gRPC Remote Procedure Calls (gRPC) or HyperText Transfer Protocol (HTTP), received the bare session as a parameter. Variable names baked the storage technology into the business layer.

This made the storage migration difficult to attempt directly. We couldn’t add a second storage backend without forking the orchestration logic, and we had no way to test the read path in isolation from a real database session. So we did the migration prep in three stages.

Stage 1: Extracting the storage code

The first stage shipped no behavioural change. We deleted the monolithic storage file and split its contents in two:

storage/legacy.rs: wrapped session creation, prepared statements, and query execution behind a LegacyStorage struct.
batch_read_ops.rs: kept only the orchestration logic: time-range splitting, channel-based fan-out, and aggregation.

AppState started holding an Arc<LegacyStorage> instead of a raw session handle. The PreparedQueries struct lost its statements (those moved inside LegacyStorage). We renamed every storage-specific identifier in business code to generic storage_* names.

The result was a hard fence. After Stage 1, the database driver crate was reachable only from inside the storage module. Nothing in the business logic or handlers imported it any more.

Stage 2: The storage facade

With the seam in place, we introduced the actual abstraction. A new storage/ module with mod.rs, legacy.rs, aerospike.rs, and mock_storage.rs as siblings became the only place driver crates were reachable from.

The idiomatic Rust approach would have been a trait with associated types, but our backend selection is runtime (a config string parsed at startup), and associated types propagate upwards through every consumer. The alternative, trait objects with boxed futures adds a heap allocation per query, which we wanted to avoid at our QPS.

We chose a concrete facade with enum dispatch:

struct Storage {
    legacy:    LegacyStorage,
    aerospike: AerospikeStorage,
    mock:      MockStorage,
    settings:  StorageSettings,
}

execute_queries(backend: BackendType, ...) {
    match backend {
        Legacy    => self.legacy.execute(...),
        Aerospike => self.aerospike.execute(...),
        Mock      => self.mock.execute(...),
    }
}

A match statement at the request boundary, which made it easier to reason about and debug. The facade then routes everything to the original backend without the rest of the code knowing or caring.

Each backend’s execute_queries honours the same contract: take a Vec<QueryCandidate> and a HashMap<BatchIndex, Sender<...>>, and emit (index, value, timestamp, granularity) tuples into those channels. The orchestration layer above doesn’t need to know whether a candidate became a paginated row stream or a single batch read with client-side map filtering, both write into the same channels in the same shape.

On top of the facade we layered three config-driven operating modes that map to the migration phases:

Single: one backend serves the request.
WithShadow: the primary serves the response; the secondary runs asynchronously in the background for parity comparison.
WithSplit: a deterministic percentage of traffic is served by each backend. Used for the live cutover.

The mode and traffic percentages are read from a service config, allowing the reader to move from legacy-only to Aerospike-only without code changes. The transition starts in Single(legacy), then shadow reads are enabled with WithShadow(primary=legacy, secondary=aerospike, pct=X). The shadow percentage is gradually ramped from 5% to 20%, 50%, and finally 100%, while parity is verified through metrics. Optionally, the system can then move into WithSplit(primary=legacy, secondary=aerospike, split=X), where live traffic is gradually shifted from the original backend to Aerospike, for example from 5% to 30%, 70%, and then 100%. Once Aerospike is fully validated and serving all traffic, the reader moves to Single(aerospike).

Stage 3: Shadow comparison and metrics

Each storage call carries metadata like backend, role (primary/secondary/shadow), and mode, attached as tags to every metric. When Aerospike was added, existing dashboards showed per-backend breakdowns without changes.

We placed the mode dispatch at the handler level rather than inside the storage layer to validate the full request path, not only the rows returned by storage. This also lets the response return as soon as the primary completes, while the shadow runs as a fire-and-forget background task.

Writer: redesigning the data model

Since the two systems use different storage engines, it wasn’t clear that a one-to-one port of our original schema would work. We tried three approaches.

Approaches 1 and 2: Row-per-bucket

We first tried mirroring our original row-per-bucket model. Approach 1 used Aerospike’s Secondary Index (SI) to recover range queries; approach 2 skipped SI and computed the exact set of primary keys client-side via BatchGet.

Both hit the same wall: Aerospike’s primary index is 64 bytes per record, kept in memory. At billions of records, index memory becomes the constraint. SI added overhead and operational complexity we didn’t need.

Approach 3: Map-based schema

The third approach was structurally different from the first two and was the most compact of the options. Rather than storing one record per bucket, which kept us in the same cardinality regime, we collapsed all bucket counts for a single counter into one record. The values were stored as a sorted map keyed by bucket timestamp:

Set:           helium_hourly
Primary key:   "{counterKey}"
Bins:
   counts: KEY_ORDERED_MAP({
        1773369000000: 1,
        1773372600000: 3,
        1773376200000: 7,
        ...
   })

The map keys are bucket timestamps in milliseconds. The map values are running counts. One record holds the entire time series for one counter at one granularity.

Reads become straightforward: fetch the record, iterate the map, sum the entries within the requested window. Each Get returns a bounded number of map entries (determined by Time To Live (TTL) and bucket size), and client-side filtering of that many entries is negligible.

Writes use MapIncrementOp, an atomic server-side increment of a value at a given map key, creating the entry on first access. Combined with MapRemoveByKeyRangeOp for pruning stale entries, every write is one atomic operation:

ops = [
    MapIncrementOp(counts, bucketTsMs, delta),
    MapRemoveByKeyRangeOp(counts, 0, cutoffMs),
    PutOp(key_bin, counterKey),
]
client.Operate(policy, key, ops)

For TTL management, we couldn’t use Aerospike’s record-level expiry directly. A single record holds many timestamps, so record-level TTL would either keep everything or drop everything. Instead, we prune stale map entries explicitly on every write using MapRemoveByKeyRangeOp. The record-level TTL stays as a safety net for counters that stop receiving writes.

The two backends produce very different network shapes for the same logical query. The original backend returns many small paginated row streams, one per (key, granularity). The server filters by time range using the clustering column. Aerospike returns one batch response with the entire counts map per key, and the client filters the map to the requested range. The reader’s storage layer hides this difference: both paths emit (index, value, timestamp, granularity) tuples into the same per-index channels, and the orchestrator above sums them the same way.

The third approach performed best in testing. By collapsing many bucket records into a single record per counter, we reduced the total record count by more than an order of magnitude, which also reduced primary index memory. It also produced a smaller on-disk footprint, since the long counter key is stored once per record instead of being repeated across every bucket. The schema was chosen to fit the access pattern, with the index and disk savings following naturally.

The pipeline continues writing to the original backend as the primary, while Aerospike is added as a separate asynchronous shadow write behind a deterministic rollout logic. This lets us ramp Aerospike gradually and eventually cut over to it fully.

Reader: How each backend actually serves a query

The two storage backends sit behind the same execute_queries contract on the reader service, but what they do internally for a single batch read looks very different.

Figure 1. How a single read request flows through each backend.

The reader takes a batch of counter queries and decomposes each into one or more sub-queries per granularity (a 90-minute window for instance, becomes one hourly sub-query and two 15-minute sub-queries). In the original backend, each sub-query becomes its own prepared statement bound with (start_ms, end_ms, key), and the storage layer fires all of them concurrently as a stream of futures with buffer_unordered capping in-flight queries to a tuned bound. Each query returns a paginated row iterator, the server uses the clustering column to filter by time range and rows stream through to per-index channels as they arrive. So a single user request can produce many small queries, each a separate network round-trip to the partition master holding key, with results dribbled back over a paginated stream.

On Aerospike, the storage layer first groups all sub-queries by granularity, then issues one BatchOperate per granularity. Each sub-query becomes a single primary-key read against the appropriate set; the server returns the entire counts map for that key in one record. The client iterates the map and emits only the entries whose timestamps fall inside the requested range. This keeps the code simple, and at our map sizes the overhead is negligible. There’s no streaming, a batch read either succeeds or fails as a unit and there are at most three network round-trips per user request, one per granularity, regardless of how many sub-queries there are.

This reflects the different design philosophies of the two systems. Wide-column stores typically expect client-side fan-out for reads, while Aerospike’s batch API is designed for exactly this multi-key pattern.

A few issues with the Aerospike Rust client also surfaced during rollout, as it was less mature than its Go counterpart. For example, when we started, the officially available Rust client was synchronous, so every batch read had to be bridged through tokio::task::spawn_blocking with some amount of custom plumbing. Once the official async client was released, we removed that layer and saw measurable improvements in both p50 and p99 latency. The other issue was Domain Name System (DNS). The client resolved seed hostnames only during initialization and did not re-resolve them when the cluster topology refreshed. As a result, a full staging cluster replacement, with new IPs behind the same hostnames, left the client stuck on the old IPs until restart. We filed the bug upstream, and a fix shipped in a subsequent release. We also reproduced the scenario locally with a Docker-based end-to-end test and ran additional staging drills to confirm recovery before continuing the rollout.

Experiment with indexing

We run Aerospike in its default storage configuration, Hybrid Memory Architecture (HMA), where the primary index sits in Random-Access Memory (RAM) and the data sits on Solid-State Drive (SSD). The other relevant mode keeps both index and data in Dynamic Random-Access Memory (DRAM), which is more expensive and not something that fits our use-case. Even in HMA, the primary index grows linearly with record count. At our scale, that growth was a foreseeable issue.

To raise the memory ceiling, we tried moving the primary index itself from RAM to local Non-Volatile Memory Express (NVMe) while keeping data on SSD. We expected the extra index latency to be invisible within our overall request budget. In practice, we started seeing p99 spikes that did not track overall QPS. Instead, they followed I/O activity on hot keys. We observed that when many concurrent lookups land on the same record, the in-memory index handles them more prudently compared to a disk backed index. Adding more and better nodes improved things slightly but did not mitigate the issue. Consequently, we reverted back to in-memory index with a memory-optimized instance type.

Overall impact

The migration delivered gains across infrastructure, performance, and data footprint. Most of these improvements trace back to the schema redesign like collapsing rows into maps, rather than the database change itself.

The primary index currently uses about 50 GB of the roughly 100 GB usable memory per node. The same dataset is around 1 TB on disk, compared with around 3 TB on the original setup. This is primarily attributed to our adoption of the map-based schema discussed earlier.

In production, p99 read latency was consistently better than the original setup, with roughly 50% improvement across our read paths. The write path now uses a single atomic increment operation, replacing the read-modify-write pattern we had built previously.

The new setup costs roughly 45–50% less per node compared to our original setup. We also reduced the replication factor from 3 to 2, saving roughly a third of both storage and primary index memory. RF=2 can be awkward in databases that depend on write quorum, but Aerospike’s master-replica model still keeps an authoritative copy available after a single-node loss. That gives us meaningful fault tolerance even at RF=2. The remaining risk, a simultaneous multi-AZ failure, was acceptable for this workload because the writer continues producing increments from the source event stream. Any lost counter data can self-heal as new events arrive.

Conclusion

This migration ultimately came down to aligning the storage design with the workload. These results would not have been achieved by simply swapping one storage system for another. As the service evolved over time, our initial design choices became less optimal, and the migration surfaced opportunities to rethink them. The gains came from focusing on optimization opportunities, redesigning the data model, and cleanly separating storage concerns. Through shadow reads and writes, followed by a gradual rollout, we completed the migration with zero downtime and no data-integrity issues. The result is a system that fits its workload well and a foundation that makes future storage changes safer and easier to attempt.

Join us

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Scaling out Distroless adoption With AI

Mon, 22 Jun 2026 00:00:00 +0000

Distroless adoption at Grab

Grab is migrating from heavy base images to Distroless images to reduce security risks. By limiting each container to the application and its runtime dependencies, we shed non-essential binaries and associated Common Vulnerabilities and Exposures (CVEs).

This migration is more than a compliance mandate; it is a strategic security decision to build a more resilient environment.

Why Distroless requires rigorous testing

Distroless adoption risk: Runtime failure

Shifting to Distroless images introduces a critical technical risk: Runtime Failure. A service might build perfectly in Continuous Integration (CI), but fail at the deployment stage due to:

Missing shared objects: Binaries might require specific libraries (.so files) present in Ubuntu but absent in Distroless.
Implicit links: Third-party tools might expect specific system utilities or directory structures.

Testing is required to ensure two things:

The service spins up with the correct configuration.
All runtime dependencies remain intact.

Scaling this verification across thousands of services manually? That would take years, unless we found a way to automate the trust.

The testing methodology

As we perform changes to the Dockerfile definition of our services, it is important for us to include the corresponding test strategy to ensure that the changes that we make do not introduce a regression to our running services. Assessing the change introduced to our services, the lowest possible testing boundary would be that of what we define as medium tests in Grab.

Medium tests in Grab

At Grab, we categorize our test suites into 3 main sizes: small, medium and large. Small tests refer to functional tests whereby mocks are introduced via dependency injection. Large tests refer to end-to-end tests that run on actual services in our staging environment where nothing is mocked.

Figure 1: Architecture diagram of a medium test environment

Medium tests sit between the small and large tests. External dependencies (such as service-to-service dependencies) are mocked with a network proxy layer similar to WireMock, while internal dependencies like MySQL are left unmocked and are started instead with Testcontainers. The system under test is built into a Docker image, run as a container, and exercised through its endpoints; the tests then assert on the responses. That setup lets us catch Dockerfile changes that would break the service in production. A further benefit is that the whole flow can run inside Continuous Integration (CI) so that problems surface before anything reaches Continuous Deployment (CD).

Figure 2: Happy path for Distroless changes

This makes medium tests effective and efficient for testing changes to the services associated with Distroless adoption. We could now largely scale up our adoption process by:

Raising batch Merge Requests to dockerfile definitions for Distroless adoption.
Running medium tests in CI.
Upon passing the medium tests, automatically merge the changes and trigger CD.

Introduction of toil

The above methodology works well for services that already have medium tests. However, we soon ran into a blocker when rolling it out to services that do not yet have a medium test setup. Inherently, scaffolding medium tests for a service is a tedious task. Most of the toil comes from first identifying internal dependencies, then bringing up the corresponding test containers during tests, and then connecting those dependencies to the service under test by updating the test environment configuration.

Figure 3: Current gap

These tasks are not challenging but are generally tedious to set up. They also cannot be fully automated because every service combines internal dependencies differently, and because configuration is defined and used in different ways across codebases. Roughly 400 services in scope still lack a medium test setup, which became a major obstacle to our Distroless migration campaign.

Since each step needs flexible execution and is only moderately complex, artificial intelligence (AI) is a natural way to accelerate Distroless adoption.

AI: The toil buster

Figure 4: Solution leveraging AI

AI was a good fit because the work we needed to automate produced a clear deliverable and we could tell in a deterministic way whether it had succeeded. Success was straightforward: the CI pipeline would turn green after running the basic health-check medium tests. With a measurable end goal and a reliable success signal, we pursued an agentic workflow rather than a one-off generation attempt.

The starting point

We started by adopting skills to guide the agent on how to carry out the medium-test work and how to get past repo-specific blockers. These skills gave context for scaffolding basic medium tests, setting up internal dependencies, and debugging issues in the code. Once those foundations were in place, we rolled the approach out to a batch of 20 services, completed by the AI in about two working days. That run confirmed our core hypothesis: the agent could scaffold medium tests first, then rely on those tests to show that our Dockerfile change (using distroless image as a base image) had not introduced regressions.

Teaching an agent to test

By then, the real shift was turning “can do the task” into “can repeat the behavior.” We captured the medium-test knowledge as a list of skills, which were grounded in Grab’s internal medium test SDK.

Next, DevSecOps wrapped those skills into an Entrypoint Skill, an orchestrator that runs a multi-phase workflow across services. The result is a single agent loop that moves from candidate detection to scaffolding, fixing failures, and CI verification, without treating each service as a brand-new, one-off problem.

Figure 5: Workflow overview for medium test generation

Leveraging the skills we have acquired, we utilized Claude Code, Anthropic’s agentic coding tool. This tool takes a list of services and processes them in a batch.

Detect: Is this a deployable service or a library? Is it still maintained? The agent skips anything that does not qualify, so human time is only spent on real candidates.
Scaffold: Using Grab’s scaffolding tool, the agent generates the medium test boilerplate.
Fix: The scaffold rarely works on the first try due to the unique setup of each repository. This includes missing environment variables, database dependencies at startup, and port mismatches. The agent reviews its knowledge base to pattern-match errors against known fixes.
Raise MR: Once the medium test passes locally, the agent creates a draft merge request on GitLab with a description explaining the service-specific changes and the rationale behind them.
Monitor CI: The agent polls the pipeline, reads job logs on failure, and attempts CI-specific fixes. If the same error persists after two attempts, it flags the issue for human review.
Repeat: It pushes the fix and moves to the next service while the pipeline runs. The agent does not sit idle waiting for CI. It starts scaffolding the next service in parallel and checks earlier pipelines as results arrive.

What made it work

Getting the workflow to function was the easy part. Getting it to function reliably across hundreds of services required deliberate design choices.

MCP: The agent never leaves Claude Code. GitLab interactions such as creating branches, raising MRs and reading pipeline error logs happen through a Model Context Protocol (MCP) server. When it needs Grab-specific context (what a service does, who owns it), it queries Glean, Grab’s enterprise search tool, via MCP instead of guessing. For code-level context (how a service is structured and how dependencies are wired across repos), it queries Sourcegraph through its own MCP integration.
Guardrails over autonomy: The agent can only touch test files and CI configurations; application code is off-limits, enforced before every commit. It might not hollow out tests to force a pass. If it cannot fix an issue, it escalates.
Knowledge that compounds: We maintain a feedback loop for scaffolding, mocking, and known failure patterns. After each batch, we review what blocked the agent and fold recurring fixes back into the skill. The agent’s improvement is driven primarily by the iterative refinement of its instructions, rather than changes to the underlying model.
Integrating Scripts with skills: For deterministic tasks like boilerplate generation, scripts are far more reliable than raw AI logic. By integrating these scripts as “skills,” we also optimize the agent’s performance in context window management. During test execution, standard output often produces hundreds of lines of repetitive logs that could exhaust token limits or distract the model. Using a script as an intermediary allows us to programmatically filter logs, extracting only the specific error messages or stack traces required for debugging. This ensures the AI receives a clean, actionable summary rather than being overwhelmed by noisy data.
Token efficiency: Batches across many services burn tokens quickly. We use a compressed communication style that cuts model output by about 75% while keeping technical detail and dropping filler. Proper communication is reserved for MR descriptions and messages to service owners.
Isolated execution: Each service gets its own context window (sub-agents). Long sessions processing dozens of services do not bloat the main conversation, keeping the agent focused and responsive.
Human-in-the-loop: Every MR is a draft: a human reviews before anything merges. Humans also decide which learnings become permanent knowledge. The agent proposes; people approve.

From tests to migration at scale

With medium tests in place across our service fleet, we had the safety net we needed. The next step was automating the Distroless migration itself.

The Patch-Test-Compare loop

Figure 6: The patch-test-compare loop

Before any Dockerfile changes, the system runs the service’s existing medium tests to establish a baseline. Pre-existing test failures are baselined, allowing for a clear distinction between legacy issues and new regressions introduced by the Distroless patch.

Next comes Distroless patching. The system inspects each service’s Dockerfile for OS-level package dependencies; scanning apt-get install lines and filtering out packages already included in the distroless base image. Two scenarios to consider here:

If no extra packages are needed, it is a straightforward base image swap.
If packages are detected, the system generates a multi-stage build: a builder stage installs the required packages, then copies only the necessary shared libraries into the Distroless runtime stage. The result is a minimal image that still contains everything the service needs to run.

After patching, the same medium tests run again. Results fall into clear categories: pass (tests still green - safe to migrate), regression (tests fail - the patch introduced a problem), or already failing (the suite was failing before we changed anything). Regressions trigger an automated remediation step: a separate AI agent inspects the container for missing shared libraries and attempts to fix the Dockerfile. If it cannot resolve the issue, the service is flagged for human review.

Scaling with batch changes

The previous section explains the patch-test-compare loop; this section addresses how we apply patching changes to more than one service simultaneously. To migrate at scale, we use batch change tooling that applies the Dockerfile transformation across dozens of repositories simultaneously, creating merge requests automatically. The system handles both standalone GitLab repositories and Grab’s shared Go monorepo, adapting the patching and MR strategy to each.

Impact on our services

Medium test generation at zcale

With medium tests in place, regressions are more likely to be caught before code reaches staging, which provides the safety guarantee we needed. Each generated test also becomes a lasting safety net for the service, not just for the Distroless migration but for future changes as well. Over roughly 1.5 months, the agent raised more than 100 medium test MRs across repositories, bringing more services into compliance with Grab’s “shift-left” testing initiative.

Distroless adoption

The campaign moved the needle across our service fleet: overall distroless adoption within our scope has grown substantially since we began using AI to drive the work.

Autonomous with oversight

For typical cases, the agent autonomously handles the majority of medium test generation and Dockerfile migration work with little human intervention. Engineers remain in the loop, reviewing every draft MR and making the final call on what merges.

Engineering bandwidth reclaimed

Manually generating a basic medium test requires familiarity with Grab’s internal SDK, typically taking one to three days per repository for developers new to the framework. For approximately 400 services lacking these tests, this equates to 400-1,200 engineer-days. By leveraging AI, we reduced the effort to roughly 0.1 days per service, compressing over a year’s worth of work into a fraction of the time. This allowed the team to focus on higher-leverage tasks, such as improving migration tooling, handling edge cases, and advancing the roadmap beyond Distroless.

Conclusion

The integration of Distroless images and enhanced medium test coverage significantly strengthens the security and verifiability of Grab’s services. This initiative demonstrates AI’s capacity to handle the heavy lifting required for large-scale migrations.

Join us

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Palana (Part 2): Architecting isolation, identity, and auditability for AI agents

Sun, 21 Jun 2026 00:00:00 +0000

Introduction

In Part 1, we introduced Palana, Grab’s Kubernetes-native secure execution platform for autonomous AI agents. We discussed the underlying need for isolated environments and covered its core design principles: treating isolation as the unit of trust, keeping credentials out of agent hands, and mediating all network access. In this second part, we’ll dive under the hood into Palana’s architecture, look at the agent lifecycle, and share the key lessons we learned from putting this system into production.

Architecture overview

The core request path looks like this:

Figure 1. Palana architecture overview.

The agent pod runs in a namespace owned by one user and one agent. It gets default-deny style network policy, domain name system (DNS), access to required platform services, and a persistent /data volume. Browser traffic enters through Traefik. LLM traffic goes to the LiteLLM wrapper in the gateway namespace. General Hypertext Transfer Protocol (HTTP) and Hypertext Transfer Protocol Secure (HTTPS) egress goes through the proxy namespace. Secrets are read from Vault only by the component authorized to use them.

The operator is responsible for turning a user request into the concrete Kubernetes shape:

The user creates an agent through pcli (Palana command-line interface) or the portal.
Palana writes a UserAgent or Agent custom resource with the raw user identity.
The operator creates the user and agent namespaces, service accounts, role bindings, storage, network policies, and ingress.
The user runs a template or container image.
Admission webhooks inject proxy environment variables and enforce pod-level restrictions.
Logs, policy decisions, and activity signals are emitted to observability systems.

Agent lifecycle

From a user’s perspective, the basic workflow is intentionally small:

./pcli login
./pcli create demo
./pcli secrets add demo GRABGPT_API_KEY token=<token>
./pcli run demo --template claudecodeui

Behind those commands, Palana provisions an isolated execution environment:

Namespace: agent-{sanitized-user}-{agent}
Service account: bound only to that namespace
Storage: an Amazon Elastic File System (EFS)-backed persistent volume claim (PVC) mounted at /data
Ingress: an agent-specific hostname protected by Concedo-backed browser auth
Egress: forced through platform proxies, except for approved internal platform services
Secrets: split between agent-readable and proxy-only Vault paths
Policies: proxy egress, network egress, and optional inter-agent peering rules

The same lifecycle is exposed in the portal for users who prefer a browser user interface (UI).

How Palana handles identity

Human authentication uses Concedo OpenID Connect (OIDC). pcli login performs a browser-based authorization code flow with Proof Key for Code Exchange (PKCE) and stores the resulting identity in an isolated kubeconfig. Browser access to agent UIs is protected by OAuth2-Proxy through Traefik forward auth.

The important detail is that Palana keeps the raw user identity, such as an email address, as the authoritative owner on the custom resource. That raw identity is used for Kubernetes role-based access control (RBAC) subject matching. Sanitized forms are used only where Kubernetes object names, labels, namespaces, or Vault paths require safer strings.

This split prevents a common class of identity bugs: the display-safe or path-safe version of a user ID should not accidentally become the authorization subject.

In the future, we will integrate Palana via SPIFFE (Secure Production Identity Framework for Everyone) and SPIRE (SPIFFE Runtime Environment) with the rest of our service mesh, to provide an agentic identity — a combination of user and agent instance id — that can then be controlled as a subset of a user’s capabilities. This gives us a first step into “agents on behalf of users” with cut-down permissions while the wider industry firms up the approaches via Open Authorization (OAuth) and other controls.

How Palana handles secrets

Palana’s Vault layout is designed around least privilege:

kv/agents/{user}/{agent}/{secret}
kv/proxy-secrets/{user}/{agent}/{secret}

The first path is for secrets the agent is allowed to read through its per-agent Vault role. The second path is for credentials the agent can use only through the proxy. For each proxy-only secret, Palana can create an agent-visible placeholder value. The placeholder is inert unless the request goes through the approved proxy path.

This gives teams a practical migration path. Existing clients can often be configured with a token-looking value, while Palana keeps the real token out of the runtime.

How Palana handles LLM access

LLM calls go through litellm-proxy-wrapper, which sits in front of LiteLLM and GrabGPT. The wrapper derives agent identity from Kubernetes context rather than trusting client-provided headers. It then looks up the per-agent GrabGPT credential in Vault and forwards the request to the correct upstream route.

The agent config uses internal base URLs such as:

http://litellm-proxy.gateway:4000/aws/v1
http://litellm-proxy.gateway:4000/unified/v1

That design gives us three useful properties:

Agents do not need raw upstream LLM credentials.
LLM traffic is attributable to a specific agent.
Provider routing and credential handling can evolve centrally.

How Palana handles network access

Network control is split into two layers.

At Layer 3 and Layer 4, Kubernetes NetworkPolicy and Cilium enforce which pods can talk to which namespaces, services, and classless inter-domain routing (CIDR) blocks. Agent namespaces are locked down to the platform paths they need: DNS, Vault, the egress proxy, the LLM gateway, and the Kubernetes application programming interface (API) patterns the platform explicitly supports.

At Layer 7, the proxy policy controls HTTP and HTTPS destinations by host, method, and agent identity. Open Policy Agent (OPA) evaluates per-agent policy. The proxy logs allow and deny decisions in structured form.

This split is deliberate. NetworkPolicy is good at containment. The proxy is good at application-aware decisions and audit. This allows us to be very expressive in the restrictions we place on our agents — by default, they get nothing; if they should have access to an internal service they get only that service, and cannot be used as an entry point to the wider internal environment.

Observability and operations

Palana treats observability as part of the safety model, not a nice-to-have. The platform emits structured logs for proxy decisions, Git activity, LLM requests, agent lifecycle, and idle-shutdown decisions. Operators can query activity by namespace, user, host, decision, or component.

One example is idle shutdown. Long-running agents are useful, but idle workloads consume cluster resources and expand the surface area that platform teams must monitor. Palana’s reaper records the most recent observable activity for each UserAgent. It combines signals from gateway/proxy logs, Git activity, Slack-routed agent messages, and Prometheus network activity. After a configurable idle threshold, it can warn the user and stop the workload while preserving /data, RBAC, namespace, and Vault state.

This is a good example of the platform philosophy: stop the compute, keep the state, and make resumption easy.

In addition, as we move into agentic operations, we use the many signals generated by Palana itself to aid our agents. For example, we have an agent that can monitor user workloads and provide advice and assistance if it spots issues — say, an agent is consistently out of memory (OOM), the ops agent can see that and message the user with instructions on how to increase the allocated memory. We don’t need to special-case every possible issue; instead we have agents that understand Palana logs and are able to communicate with the users themselves.

What we learned

Agent platforms need security controls at the platform layer

Prompt-level guardrails and model policies are useful, but they are not enough. Agents call tools, tools call services, and services use credentials. Palana puts controls where the action crosses a trust boundary: identity, egress, secrets, ingress, Git, and Kubernetes API access.

The user experience matters as much as the control

If the secure path requires every team to learn Terraform, Vault policy syntax, Kubernetes RBAC, and proxy configuration before they can try an agent, teams will work around it. Palana uses pcli, templates, and the portal to make the safe path the easy path.

Separating “can read a credential” from “can cause a credentialed request” is powerful

Proxy-only secrets are one of the highest-leverage design choices. They let agents perform authenticated work without turning the agent filesystem, logs, process environment, or prompt context into a credential store.

A namespace boundary is simple, but it compounds

Per-agent namespaces give us a consistent place to apply RBAC, storage, network policy, logging labels, resource quotas, and lifecycle controls. The pattern is easy to reason about during incidents: identify the namespace, identify the owner, inspect the policy, and isolate if needed.

Long-running agents need lifecycle management

Once agents persist for days or weeks, “run a container” becomes an incomplete product. Users need resume semantics. Operators need idle cleanup. Security teams need audit history. Platform teams need a way to rotate credentials, update images, and stop workloads externally.

What’s next

Palana is increasingly becoming a substrate for larger autonomous systems rather than only a place to run individual agents. Emerging patterns include:

Supervisor systems that route work to a pool of scoped agents.
Slack-native agents that wake up, handle a task, and scale back down.
Remote development environments backed by persistent cloud state.
Agent swarms where each worker has a separate namespace and credential scope.
Operational agents that investigate platform health and propose or apply small fixes under policy.
Security experiments around supply chain monitoring, token rotation, transport layer security (TLS) inspection, and automated isolation.

The north star is not “let every agent do anything”. It is to make useful autonomy boring to operate: attributable, inspectable, revocable, and recoverable.

Conclusion

AI agents are most valuable when they can act in real environments. That is also when they become risky. Palana gives Grab a way to keep both sides of that tradeoff in view: teams can move quickly with self-service agent environments, while the platform keeps isolation, identity, secrets, network access, and auditability as defaults.

We expect the underlying tools and models to keep changing. The platform primitives are more durable. Agents will vary, but they will still need a place to run, a way to authenticate, a boundary around their actions, and a record of what happened.

That is the role Palana is designed to play.

Join us

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Palana (Part 1): Why Grab built a secure platform for autonomous AI Agents

Fri, 19 Jun 2026 00:00:00 +0000

Abstract

Artificial intelligence (AI) agents are moving from experiments into everyday engineering workflows. They can read code, call application programming interfaces (APIs), run tests, create merge requests, answer Slack messages, and keep long-running state. That makes them useful, but it also changes the risk model - especially as agents get more autonomous in their use of tools. An agent with network access, credentials, tools, and memory is no longer just a chat interface. It is a workload that can act.

The more capability we give to the agents, the more valuable they get - but they also get riskier, and maintaining controls and oversight gets more challenging. We need isolated environments, with clear intentional capabilities added rather than just inheriting “everything on your laptop”.

Palana is Grab’s Kubernetes-native platform for running those workloads safely. It gives each agent an isolated namespace, persistent storage, controlled ingress, proxy-mediated egress, Vault-backed credential injection, large language model (LLM) routing, Git access controls, structured audit logs, and emergency kill switches. It is currently used to run hundreds of agents, including remote development environments, Slack automation, OpenClaw workers, Hermes agents, and other long-running internal systems.

In this post, we share why we built Palana, what it does, and how its architecture lets teams experiment with autonomous agents without giving up control over identity, secrets, network access, and operational visibility.

Introduction

The first wave of AI coding tools lived close to the user: an integrated development environment (IDE) plugin, a chat window, or a command-line assistant running on a developer’s laptop. That model is familiar and easy to adopt, but it has limits. Long-running agents need persistent state. Team workflows need shared access through Slack or web user interfaces (UIs). Security teams need to inspect what an agent is doing, and apply highly granular controls over what an agent can do. Platform teams need a way to stop, resume, update, and audit the workload.

As usage grew, we started seeing the same question in different forms:

How do we let agents do useful work inside the company without treating every new agent as a bespoke infrastructure project?

The answer was not simply to “run agents in containers”. Containers help package the runtime, but they do not answer the harder platform questions:

Which user does this agent act on behalf of?
What credentials can it use?
Can it see another user’s state?
Can it connect directly to the internet?
How do we inspect LLM, Git, and Hypertext Transfer Protocol (HTTP) activity after something goes wrong?
How do we stop an agent quickly without trusting the agent to cooperate?
How do we give teams a self-service experience without handing them cluster-admin access?

Palana is our answer to those questions.

What Palana is

Palana, an in-house proprietary system built by the CyberSecurity team at Grab, is a secure execution substrate for autonomous and semi-autonomous agents. The name comes from a Sanskrit root associated with protection, maintenance, and care. That maps well to the platform’s purpose: Palana is not trying to be the agent’s brain. It is the environment that contains, observes, and sustains the agent while it works.

At a high level, Palana provides:

A Kubernetes namespace per agent, with role-based access control (RBAC), resource quotas, network policy, and storage scoped to that agent.
A command-line and portal experience for creating, running, stopping, configuring, and inspecting agents.
Persistent /data storage so long-running agents can preserve memory, caches, repositories, and session state across restarts.
Browser and shell access for interactive workloads such as Claude Code UI, OpenCode, IDEs, ttyd, or Secure Shell (SSH)-backed development flows.
LLM access through a LiteLLM wrapper that injects per-agent GrabGPT credentials from Vault.
HTTP and HTTPS egress through an Envoy and ext-authz proxy path, with Open Policy Agent (OPA) policy checks and structured request logs.
Proxy-only secrets, where agents can reference placeholder tokens but cannot read the underlying credentials directly.
Git access through a bastion path so repository operations are attributable and policy-controlled.
Kill switches and idle shutdown so the control plane can isolate or stop workloads from outside the agent process.

This combination lets Palana support several categories of work:

Secure OpenClaw and agent-framework testing.
Cloud development environments accessible from a browser or SSH client.
Fast prototyping and testing for agentic workloads in a secure environment.
Slack-connected agents such as cts-aergia and Claude-to-Slack workflows.
Long-running task agents such as Hermes, Matlock, Butler, and custom team automations.
Higher-order systems where agentic supervisors launch or route work to scoped agents.

Why we built it

The immediate need came from security research. We wanted a place to run and investigate OpenClaw and related agent frameworks without exposing the broader internal network or placing raw credentials inside the agent runtime. That use case forced us to design for containment from the beginning.

The broader need quickly became developer productivity. Once the basic primitives existed, Palana became useful for remote coding, Slack automation, internal assistants, long-lived experiments, and agentic operational workflows. Grabbers wanted agents that could keep context over days or weeks, run from corporate infrastructure, access approved internal services, and survive laptop sleep, local dependency drift, or network changes.

The security and productivity goals reinforce each other. If the safe path is self-service and ergonomic, teams are more likely to use it. If the productive path is observable and policy-controlled by default, and the appropriate security is baked into the system automatically, platform teams do not have to retrofit controls after adoption.

Design principles

Palana’s architecture follows a few principles that shaped most of the implementation.

Isolation is the unit of trust

Each agent gets its own namespace, service account, storage, network policy, and Vault scope. Agents should not see each other’s pods, secrets, or filesystem state by default. Inter-agent communication is possible, but it goes through explicit peering rules rather than ambient pod-to-pod reachability.

This means the platform does not have to assume every agent framework has perfect multi-tenant isolation internally. A framework designed as a single-user assistant can still be hosted safely by giving each user or worker its own Palana boundary.

Credentials are never given to the agent

Traditional application hosting often gives credentials to the workload as environment variables or mounted files. That is risky for agent workloads because the agent may execute tools, run untrusted code, summarize files, install packages, or expose a web UI.

Palana separates two kinds of secrets:

Agent-readable secrets live under the agent’s own Vault path and are available only to that agent’s service account.
Proxy-only secrets are stored under a separate Vault path and are read by the proxy layer, not by the agent.

For proxy-only secrets, the agent sees a placeholder such as TOKEN_GITHUB_PAT or TOKEN_GRABGPT_API_KEY. When an outbound request travels through the proxy path, the proxy replaces the placeholder header with the real credential from Vault. The remote service receives a valid token, but the agent process never stores the token in its own environment or config.

This pattern is especially important for LLMs, source control, API integrations, and browser-like tools where prompt injection or dependency compromise could otherwise expose long-lived credentials.

Egress is a control point

Agents can be useful only if they can call tools and services. Instead of forbidding network access, Palana makes network access observable and policy-mediated.

Agent pods receive proxy configuration automatically. External HTTP and HTTPS traffic flows through Envoy. Envoy asks ext-authz-proxy to identify the calling pod, evaluate policy with OPA, log the request, and optionally inject credentials. HTTPS traffic can be terminated by the proxy’s man-in-the-middle (MITM) listener for header inspection and replacement, with the generated certificate authority (CA) distributed to agent pods.

This gives the platform a place to answer questions that normal Kubernetes networking cannot answer alone:

Which agent made this request?
Which user owns that agent?
Which host and method were requested?
Was the request allowed or denied?
Which placeholder credentials were replaced?
Did the request go to an internal service, an LLM gateway, GitLab, or the public internet?

The control plane must stay outside the agent

Palana assumes an agent might become confused, compromised, or uncooperative. Operational controls therefore live outside the agent process. The operator reconciles namespaces and policies. The proxy controls egress. The portal and pcli (Palana command-line interface) manage lifecycle. The kill switch is enforced with network policy. Idle shutdown is handled by a separate reaper CronJob.

That separation matters. A kill switch that asks the agent to stop is a feature. A kill switch that removes the agent’s network path is a safety control.

Use Kubernetes primitives where they fit

Palana is intentionally Kubernetes-native. Agents are represented by custom resources. The operator reconciles namespaces, RBAC, storage, services, ingress, and network policies. Users can interact through pcli or the portal, while platform engineers can still inspect the underlying Kubernetes objects when debugging.

This gives us a layered experience: simple workflows for users, direct primitives for advanced operators, and infrastructure-as-code for the deployed platform.

Conclusion

By centering the design around isolation, controlled egress, and proxy-mediated secrets, Palana provides a secure foundation for AI agents to operate within Grab. In Part 2, we will dive deeper into the under-the-hood architecture of Palana, exploring how it orchestrates agent lifecycles, handles LLM routing, and maintains operational visibility.

Join us

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

From decentralized Docs-as-Code to a centralized repository: Evolving Grab's documentation strategy

Fri, 29 May 2026 00:00:00 +0000

Introduction: The journey of documentation at Grab

In early 2021, Grab adopted a Docs-as-Code approach to address gaps in our technical documentation processes, as illustrated in our blog post Embracing a Docs-as-Code. Inspired by the practices of other market leaders, we integrated documentation into our engineers’ workflows, making it part of the codebase.

This approach addressed our initial documentation challenges by creating a single source of truth for engineers to search and build knowledge, making documentation upkeep necessary and less of an afterthought. After four years of use, we transitioned to a centralized documentation repository. This change was not about abandoning Docs-as-Code but adapting it to meet new and growing organizational needs.

This post walks through the motivations, benefits, and lessons from each phase of this journey, showing how our documentation strategy evolved.

What is Docs-as-Code?

Docs-as-Code is an approach that manages documentation with the same tools and workflows engineers use for source code. Content is written in plain-text Markdown, which is easy to edit in any code editor. Markdown is a lightweight markup language that uses simple, readable symbols like # for headings and * for lists to format text, and can be rendered to HTML and other outputs. It lives in version-controlled repositories (e.g., GitLab), so documentation evolves alongside code. Updates go through the same merge request reviews and automated CI/CD checks.

This integrated model lets teams at Grab build, test, and publish documentation as part of a pipeline. We then surface it through a centralized internal developer portal for easier discovery and implement governance for quality assurance.

Ideal use case at Grab

Imagine an engineer responsible for maintaining documentation for a product or platform managed by their team. This engineer creates comprehensive documentation in Markdown, containing pages such as an overview, a getting-started guide, how-tos, troubleshooting, FAQs, and related references for a specific platform. The documentation is included in the merge request and published on the documentation portal immediately after the code is merged. This seamless integration fosters a sense of ownership over the documentation. However, while this scenario is ideal, implementing it in practice presents significant challenges.

Problems we solved with Docs-as-Code

Before adopting a Docs-as-Code model, documentation was often scattered across Google Docs, slide decks, wikis, and ad hoc text files, which led to version confusion, poor discoverability, and gaps in quality assurance. Centralizing documentation in version-controlled repositories next to the code creates a single source of truth, ties updates to the same pull/merge request reviews, and enables automated checks such as link validation, style linting, and preview builds.

Industry practice reflects this shift: Kubernetes maintains its documentation as Markdown on the Kubernetes website, uses GitHub as a repository, and builds the site with Hugo, encouraging doc updates alongside feature work.

When documentation is embedded with the code and flows through the same CI/CD pipeline, engineers are more likely to update it in tandem with code changes. This method keeps the content up to date and in sync with releases by default. The TechDocs team can also set standardized metrics to uphold quality across all documentation and implement quality gates and blockers to ensure each document meets quality standards.

The limits of decentralized repositories

As Grab’s engineering footprint expanded, our decentralized Docs-as-Code approach began to strain at scale, surfacing friction that made documentation harder to discover, maintain, and ship with confidence.

Fragmented user experience and uneven standards

When documentation is scattered across many repositories and managed independently by teams, information architecture, voice, terminology, and granularity diverge. Similar concepts end up with different names, pages follow inconsistent navigation and templates, and redundant or misaligned guidance proliferates.

Ultimately, the search experience becomes noisy and unreliable as multiple versions of “the truth” surface. The impact shows up as longer onboarding, more tech support escalations, slower incident response when runbooks differ by team, and eroding trust that eventually pushes people toward tribal knowledge.

For the TechDocs team, decentralization made it hard to enforce standard templates, formatting, and quality gates. With documentation spread across many repositories, each with different or no linters, CI setups, and conventions, running organization-wide automation (linters, link checkers, readability checks) or applying uniform review steps was unreliable. This resulted in limited oversight and persistent inconsistencies, which degraded the user experience and trust in the documentation.

Difficulty keeping pace and staying discoverable

A fast-moving platform means decentralized documentation ages quickly and becomes hard to find. Frequent infrastructure and framework releases introduce breaking changes and deprecations. Teams struggled to stay informed, leading to missed opportunities for optimization and potential security risks due to outdated practices.

Meanwhile, with content sprawled across many repositories, managing and tracking the content became increasingly challenging for the team overseeing TechDocs. When teams changed the location of their source repositories, they often failed to notify the managing team, making it difficult to keep track of newer and updated locations. This lack of coordination created significant hurdles in discovering relevant documentation and maintaining a centralized record, ultimately impacting productivity and delaying decision-making.

Why we transitioned to a centralized repository

A centralized repository allowed us to address these scaling challenges while keeping the benefits of Docs-as-Code:

AI-driven enhancements

We are no longer writing only for human engineers. As we integrate more AI tools into our developer experience, our documentation also serves as the knowledge base for internal agents. A centralized, Markdown-based format gives agents clean, readable content in one location, which supports better integration, faster comprehension, and more accurate responses.

Improved quality assurance

Centralizing our documentation enabled the managing team to run automated linters for quality checks across all content. This helped ensure consistent standards, reducing manual oversight and minimizing the risk of errors. Contributors were also required to use the appropriate template for each document type, ensuring a consistent structure by default.

Unified search experience

The unified search experience changes how engineers access information. They can search for any topic and find relevant documentation without navigating multiple repositories. A global search overlay combines two methods: fuzzy page-title search for quick navigation and Glean-powered search across all TechDocs content. Glean is an enterprise search and AI assistant platform that integrates with internal tools to help users find and use information more efficiently. This search capability saves time and helps engineers stay informed.

Streamlined contribution process

While the decentralized model allowed engineers to use the GitLab web IDE, local editors, and GitLab CLI commands for faster updates, the transition to a centralized system helped streamline this process by offering a consistent editing environment. Even with these advanced tools, the centralized repository provided a unified location for all documentation, reducing the need to navigate across multiple repositories.

Centralization also gives the TechDocs team clearer visibility into documentation behavior and health. After implementing a centralized repository, the team extracted statistics on user activity: a new update is merged roughly every 50 minutes, with roughly 27 commits per day, and approximately 63% of changes being small to medium improvements. These signals point to ongoing documentation maintenance, with frequent touch-ups that fix typos, clarify steps, and keep guidance current rather than sporadic bulk updates. The image below illustrates how Grabbers use the centralized repository in practice.

Reflecting on the evolution

The transition was not without its hurdles. To bridge the gap left by decentralized Docs-as-Code workflows, we implemented:

Automated syncs: We synchronized critical content from service and platform repositories into a central hub to prevent gaps, while keeping the overlap period short to avoid two sources of truth and missed updates as legacy repos were retired.
Training sessions: We ran hands-on workshops to help engineers navigate the new platform and understand its benefits.
Continuous feedback: We set up surveys and regular check-ins to refine tooling and processes based on real-world usage.

Conclusion: choosing what works for your context

Docs-as-Code with decentralized and centralized repositories are not mutually exclusive; they excel in different contexts and can be combined. Decentralized authoring works well when engineers are the primary contributors and documentation naturally ships with code. Centralization becomes valuable when you optimize for organization-wide discoverability, consistency, governance, and analytics. We conclude with these findings from our shift to a centralized Docs-as-Code repository:

Use decentralized Docs-as-Code when teams need autonomy and documentation is tightly coupled to services.
Use a centralized repository when you need a single source of truth for discovery, standardized templates and style, consistent CI checks, ownership metadata, and clearer compliance and review gates.
Consider a hybrid approach: authors create documentation in service repos and publish to a central portal with shared templates, ownership metadata, automated quality checks, and centralized discovery and governance.

At Grab, decentralized Docs-as-Code fostered strong ownership early on. As we scaled and our audience broadened, a centralized repository and unified discovery surface became essential to maintain consistency, improve findability, and support diverse user needs. Documentation strategies evolve with the organization. The goal is not picking one model forever, but recognizing the signals to pivot and adapting so engineers can reliably find the right information at the right time.

Join us

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

The Hugo evolution: Engineering Grab's unified, one-click data ingestion platform with Apache Flink

Fri, 22 May 2026 00:23:00 +0000

Introduction

Data drives every decision we make at Grab. As our operations scale, so does our need for robust, real-time data ingestion and processing frameworks. Enter Hugo: our self-service data platform that has long empowered teams to seamlessly route data into our Data Lake. Today, Hugo is evolving. We have taken previously siloed onboarding workflows and transformed them into one seamless, unified journey to truly democratize data ingestion and maximize efficiency.

In this blog, we’ll share how Hugo turns complex engineering hurdles into a frictionless, self-service reality. By moving away from siloed workflows, we’ve achieved a unified pipeline experience where one-click RDS CDC and self-service Kafka ingestion are the new standard.

Background

Figure 1. Hugo - Ingests data from every source into Grab's data lake.

Hugo was originally designed as a self-service platform for batch-oriented data ingestion into the Data Lake, built on a single computation engine, Spark. It provided a centralized and streamlined onboarding experience for data sources such as MySQL, Aurora, PostgreSQL, and DynamoDB.

As the organization’s data platform evolved toward near real-time ingestion, Hugo expanded to support streaming pipelines from Kafka and MySQL binlog. This evolution introduced a more distributed architecture, where ingestion workflows spanned multiple systems, including Kafka Connect, Sprinkler (an in-house Go-based S3 writer), and Hugo.

The siloed past: A multi-platform hurdle

While powerful, the expanded architecture introduced significant onboarding friction. Creating a single data pipeline now requires users to coordinate across multiple platforms, each with its own configuration model and operational semantics. As a result, the onboarding journey became fragmented and difficult to navigate, especially for new users.

The common challenge during onboarding was helping users understand how configurations mapped across systems.

For MySQL CDC pipelines, users often asked, “I’ve already configured Kafka Connect, what values do I need to provide in Hugo?” after setting up a Kafka Connect job. This revealed a gap in abstraction between systems, requiring users to manually translate concepts and configurations across different platforms.

For Kafka pipelines, users frequently struggled with schema evolution in the data lake. Common questions included: “How should I update the data lake schema?” and “I’ve already updated the Protobuf schema for this Kafka topic, why isn’t the latest schema reflected in the data lake?” These issues highlighted unclear expectations around schema propagation and synchronization across the pipeline.

This multi-step, cross-system dependency increased cognitive load, slowed down onboarding, and created coordination overhead between platform teams and users.

The Hugo evolution: A unified ingestion platform

Hugo’s new, deeply automated ingestion framework, built with a custom automation layer and Apache Flink, has unified workflows and retired Sprinkler and Kafka Connect. This evolution converted manual, artisanal work into a streamlined, self-service experience, with custom automation serving as the “intelligent chassis” for the entire user journey.

The Hugo ingestion architecture: Engineering a unified flow

One-click MySQL CDC pipelines

The transition to a unified modernized pipeline powered by Flink CDC shifts the data ingestion architecture from a fragmented, high-maintenance toolchain into a single, end-to-end orchestrated platform. By reading the database binlog directly and embedding the lifecycle within a centralized control plane, the modernized approach drastically reduces operational overhead, eliminates data mismatch risks, and cuts onboarding times from days to minutes. Below are the core advantages of adopting Flink:

Minimal operational overhead: It reduces the footprint from 4 disparate components (Kafka Connect, topics, Sprinkler app, and Spark) to just 2 core components managed via a single control plane.
Eliminated schema risk: It replaces brittle, manually coded Go DTOs, which caused frequent schema deviations, with automated schema detection and dynamic validation.
Streamlined architecture: It eliminates the intermediary Kafka hop. Flink reads the MySQL binlog directly and pushes straight to a queryable Hive table via an integrated Spark compaction process.
Instant onboarding: It shifts deployment from a multi-team, ticket-heavy process taking days to a single-engineer, self-service setup completed in minutes.

Figure 2. Data ingestion with MySQL CDC to data lake

Self-service Kafka ingestion

The most significant architectural shift in the self-service Kafka ingestion pipeline is the move from manual, fragile schema handling to an automated, resilient system. This comparison highlights the operational pain points eliminated by adopting Flink’s approach.

Legacy Sprinkler approach (manual and static)

Static registration and hardcoding: It required manual registration of streams within the Go monorepo and relied on hardcoded mappings in entities.go to convert Protobuf to Avro.
Custom dependencies: Avro schema was generated indirectly from custom DTO structs, not directly from the Protobuf definition.
Manual schema evolution: Any field change required a multi-step manual process: updating .pb.go and entity files, followed by a manual pipeline rebuild.

New Flink approach (automated and dynamic)

Dynamic runtime fetching: Flink pipelines dynamically retrieve the Protobuf schema from Confluent Schema Registry on startup, removing the need for hardcoding and manual stream registration.
Reduced operational overhead for schema changes: Schema updates are propagated through the CI pipeline to the Schema Registry, removing the need for hardcoded mapping changes. The Flink pipeline can detect updated schemas and resume from the latest checkpoint after restart, though manual restart intervention is still required.
Click-to-query: Engineers can now ingest streaming data from Kafka topics into queryable Hive tables through a few clicks in the Hugo UI. Hugo automatically orchestrates the multi-stage background work, from Flink consumption and S3 writing to Spark compaction, ensuring data is query-optimized and ready for immediate use.

Figure 3. Data ingestion with Kafka to Datalake.

Impact

The platform’s new onboarding workflow has significantly reduced a previously multi-day process to mere minutes, enabling faster iteration and improving overall onboarding efficiency. This dramatic change has fundamentally altered how our teams interact with data.

Figure 4. Kafka Flink.

Figure 5. CDC Flink.

The onboarding workflow is intentionally designed with early validation guardrails to proactively surface prerequisite and governance-related issues before pipeline creation proceeds.

For Kafka sources, user drop-offs between the “Create Kafka Source” and “Kafka Sink” stages are primarily driven by validation checks such as topic ownership verification and topic activity requirements, for example topics with zero message volume. Additional drop-offs between the “Kafka Sink” and “Create Source Pipeline” stages typically occur when the proposed output table name already exists in the data lake, preventing duplicate table creation.
For MySQL sources, drop-offs are mainly associated with unmet database onboarding prerequisites, including credential setup, binlog user configuration, binlog format requirements, and binlog expiration settings.

In addition, the streamlined self-service experience encourages exploratory usage, allowing teams to familiarize themselves with the onboarding workflow and platform capabilities before fully committing to pipeline creation.

Summary

The new architecture engineered a custom automation layer that successfully retired the reliance on Kafka Connect and Sprinkler for the data lake, turning artisanal work into a streamlined, one-click experience. This transformation provides a direct boost to developer productivity.

The key impact metrics are:

Onboarding time reduction: The time required to set up data pipelines has been dramatically reduced and is now measured in minutes.
- Kafka pipelines: approximately 6 minutes.
- MySQL CDC pipelines: approximately 3 minutes.
Adoption: Since the release, the number of new Kafka and CDC pipelines onboarded in the last year is more than the total number of pipelines onboarded in the previous five years.

What’s next

These enhancements are just one step in our broader vision for optimized and self-service data ingestion. Currently, Flink is the default only for Kafka source pipelines. Flink onboarding for MySQL CDC pipelines is impact- and cost-driven. Our strategic roadmap includes:

Next-generation formats: We are investigating the adoption of Apache Iceberg as the data lake table format to further improve pipeline SLA and costs, and improve performance.
Seamless schema evolution: Schema changes still require some manual effort from pipeline owners, including manually restarting Flink pipelines. In Hugo, we aim to make schema evolution a zero-touch experience by automatically detecting changes, validating compatibility, and updating tables without disruption.

Join us

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Scaling developer experience: How we improved Android Studio in a large monorepo

Fri, 15 May 2026 00:23:00 +0000

Introduction

Long integrated development environment (IDE) sync/indexing times can quietly erode developer productivity, making code navigation sluggish, spiking memory usage, and slowing down Jetpack Compose preview updates, turning the IDE into a bottleneck rather than a helpful tool. For Android engineers working in a large monorepo, this was a daily reality. In this post, we will share how we built a custom Focus plugin that dramatically reduced Android Studio sync times by leveraging our existing investments, such as the Gradle-to-Bazel migration workflow.

Our Android monorepo at scale

The Grab passenger Android (PAX) repository contains roughly 2,000 Android modules and 11,000,000 lines of code. As the repository grows year over year, a natural consequence of scaling our superapp, which combines ride-hailing, food delivery, payments, and more into a single application, is the increase in time required to build and sync the project.

What makes this growth especially pronounced today is the shift in how code gets written. Development assisted by artificial intelligence (AI) has enabled engineers to produce more code faster than before. At the same time, non-engineering personnel such as designers, product managers, and other non-technical contributors have started making changes to low-risk features under engineering provision. Together, these two forces are pushing the codebase to grow at its fastest rate ever, which in turn compounds the pressure on every developer’s IDE and build tooling to keep up.

We previously adopted Bazel to speed up incremental and cached builds, but build time was only part of the picture. We intentionally kept Android Studio syncing with Gradle, so developers get fast Bazel builds while the IDE uses the standard Gradle toolchain, thereby preserving compatibility and avoiding the friction and tooling gaps of full Bazel IDE integration. This trade-off gives us the best of both worlds, but it also means Gradle sync remains a first-class concern. Even though Bazel handles the builds, Android Studio still depends on Gradle sync to import the project model that powers IDE features such as code navigation, autocompletion, and error highlighting. That sync process, which evaluates every module declared in settings.gradle, had quietly become a major pain point.

The problem

Over time, we noticed a growing number of reports stating that IDE syncs were too slow and memory-intensive. A single full sync could take more than 35 minutes on a cold start. The pain was especially acute after a rebase or branch checkout. Since these operations often modify build configuration files, Android Studio would detect the changes and trigger a full re-sync just to restore basic IDE functionality.

We conducted a developer experience survey to quantify the issue. From 55 responses, the results painted a clearer picture:

76% said long sync times significantly or very significantly impacted their productivity.
60% were unsatisfied or very unsatisfied with IDE sync time.
47% were unsatisfied or very unsatisfied with Compose preview update speed.
82% said they would benefit from the option to exclude modules from syncing.

Figure 1. Results of developer experience survey.

The survey validated our anecdotal feedback: developers were frustrated. Slow, sluggish IDE performance was eroding productivity and disrupting flow. We set out to determine whether developers really needed to load every module to work on just one.

Investigation

Root cause

The root cause was straightforward: module count. With roughly 2,000 modules in the codebase, a full sync required Gradle to configure every single module, including parsing build files, resolving dependencies, and generating IDE project models, regardless of whether the developer actually needed them. A developer working on the Payments feature still had to wait for Gradle to process Food, Transport, Mart, and every other module. The configuration time and resulting memory consumption grew roughly in proportion to module count, and the count kept rising.

Exploring community solutions

We looked at existing solutions in the Android community. One promising candidate was the Focus plugin from Dropbox. Here’s how the Focus plugin works:

The developer runs a Gradle command to focus on a specific module (e.g. ./gradlew :module:focus).
The Gradle task calculates the dependency graph, generates a separate focused settings file, and writes a .focus marker file that tells Gradle to use it instead of the full project settings.
The developer syncs the IDE, which now only configures the focused modules.

This approach works because instead of syncing the entire repository, the developer only configures the module they are working on, along with its required dependencies. Everything else is excluded.

For example, if you are working on the Payments module, only Payments and its dependency chain get loaded. Food, Transport, and Mart modules are excluded entirely.

Figure 2. Focus mode sync vs. full sync.

Depending on the size of the target module, this approach can cut the number of loaded modules by 50% or more, especially with a well-structured modularization architecture. We wanted to adopt this approach and saw an opportunity to improve it further by leveraging our existing Gradle-to-Bazel migration workflow.

Our solution: Building a custom Focus plugin

The Dropbox Focus plugin was a great starting point, but it introduced several friction points in our setup:

We would need to move all non-essential declarations from settings.gradle into a separate settings-all.gradle file.
We would need guardrails to ensure new modules are declared in the correct file.
Most critically, focusing on a module requires running a Gradle task (e.g., ./gradlew :module:focus), which itself goes through Gradle’s configuration phase and adds a noticeable delay before a developer can even start an IDE sync.

We set out to address each of these issues.

Challenge 1: Eliminating the configuration phase

The Dropbox Focus plugin recalculates the dependency graph every time a developer runs the focus command. This means every Focus operation pays the cost of Gradle’s configuration phase, parsing every build.gradle file in the project to resolve the full dependency tree.

We realized we already had this information. Our build infrastructure includes Grazel, which migrates Gradle build files to their Bazel equivalents via a migrateToBazel task, and our Continuous Integration (CI) validations ensure that both are aligned. This task already traverses the full dependency graph for migration purposes.

Our insight: generate a dependency graph as a static file during migrateToBazel and reuse it for focus operations.

Figure 3. Focus flow vs. Grab’s customized focus flow.

By pre-computing and persisting the dependency graph, we skip the Gradle configuration phase entirely. The focus operation becomes a fast, local file lookup instead of a lengthy Gradle computation. The developer simply selects their module and syncs.

The dependency graph is stored as a JSON file, which is lightweight and fast to read. The trade-off is that it requires a migrateToBazel run to stay up-to-date. When creating a new module or changing module dependencies, developers need to rerun ./gradlew migrateToBazel to regenerate the graph. We accepted this because developers already had to run migrateToBazel before merging to master (to ensure Bazel files are current). The graph stays fresh as part of their existing workflow, and no extra step is required.

Challenge 2: Minimizing developer friction with a Gradle plugin

We did not want to introduce a process that adds cognitive load. Migrating all module declarations to a new settings.gradle file would require every team to change their workflow. Instead, we adopted a more elegant approach.

The `include` shadow trick

In a standard Android project, modules are declared in settings.gradle using the include function:

include 'app'
include 'payment'
include 'food'
// ... hundreds more

The include function is part of the Gradle Settings API. In Groovy, you can define a local closure variable with the same name as an existing method. Since Groovy resolves local variables before delegate methods, the closure effectively shadows the original include method and all subsequent include calls in the script invoke the closure instead.

We created a custom Gradle plugin with a focusInclude function that decides whether to include or exclude a module based on the current focus configuration. By adding just three lines to the top of settings.gradle, we redirect all include calls through our plugin:

// After applying the focus plugin in the buildscript block
def include = { module ->
 com.grab.focus.GradleFocusPluginKt.focusInclude(settings, module)
}
include 'app'
include 'payment'
include 'food'

The rest of the file remains untouched. Every existing include call now passes through focusInclude, which checks whether the module should be loaded based on the developer’s focus selection. If no focus is active, all modules are included as usual with zero behavior change.

This approach meant zero migration effort for feature teams. The settings.gradle file stays as-is, and the plugin integrates seamlessly.

Early implementation: Property-based focus

In the early days of this plugin’s development, the way to specify focus modules was via a Gradle property in the command line:

./gradlew build -Pmodules-to-sync=":app,:payment"

The focusInclude function reads this Gradle property. If present, it activates focus mode and only includes the specified modules (and their transitive dependencies resolved from the graph file). If absent, all modules are included normally.

Challenge 3: Making it seamless with an Android Studio plugin

Figure 4. User flow.

Manually passing Gradle properties on the command line was functional but not ideal. We needed a better developer experience. The Gradle property approach opened the door to IDE integration; this led to an Android Studio plugin (an IntelliJ plugin) being built that automates the entire flow through a user interface (UI):

Module selection: The plugin presents a list of all available modules, parsed from the pre-computed dependency graph file. Developers select which modules they want to work on.
Dependency count indicator: Since we have the full dependency graph, the plugin displays how many transitive dependencies each module requires. This gives developers immediate visibility into module “weight” and encourages teams to keep their modules lean.
Automatic argument injection: The plugin uses two IntelliJ Gradle extension points to inject the -Pmodules-to-sync property: a GradleResolverExtension that adds the argument during project sync, and a GradleTaskManagerExtension that injects it before any Gradle task execution (including Compose preview builds). The developer just clicks sync; the plugin handles the rest.

Figure 5. Example of Automatic Argument Injection.

Beyond the core functionality, we added several usability enhancements to the plugin:

Indirect focus indicator: Modules that will be synced as a transitive dependency of a focused module are marked as “indirectly focused,” giving developers visibility into exactly what will be loaded.
Search and filtering: With hundreds of modules, finding the right one matters. The plugin supports fuzzy matching and regular expression (regex) search to quickly narrow down the module list.
Sort by dependency count: Modules can be sorted by name or by dependency count, making it easy to spot the heaviest modules at a glance.
Status bar widget: A persistent “Focus: X/Y” indicator in the IDE status bar shows how many modules are currently focused out of the total, with a click-through to the Focus tool window.
State persistence: The developer’s focus selection is saved and restored between IDE sessions, so they do not need to reselect modules after restarting Android Studio.

Encouraging lean module architecture

An unplanned but welcome side effect of the focus plugin was that it nudged teams toward a cleaner module architecture. With dependency counts now visible in the IDE, developers became more aware of their module’s size, which in turn encouraged a clearer separation between interface and implementation.

Interface module (e.g., :payment-api): Contains only the public API definitions (interfaces, data classes, contracts). This is the module that other teams depend on. Because it has no implementation details, it carries very few transitive dependencies.
Implementation module (e.g., :payment-impl): Contains the actual implementation of those interfaces. This module typically has a larger dependency footprint, but only the owning team needs to load it.

By depending on the interface module rather than the implementation module, teams avoid pulling in a large tree of transitive dependencies. This keeps the dependency count low for consumers, which directly translates to faster focus sync times and leaner Compose preview builds.

How we measure

Instrumentation: The PAX IDE plugin

The PAX IDE plugin is a mandatory install for every PAX Android engineer in Grab. This gives us a consistent, organization-wide data collection baseline without requiring any opt-in. The plugin registers four IntelliJ Platform listeners that automatically capture metrics on every relevant IDE event:

IntelliJ API	What it tracks
`GradleSyncListenerWithRoot`	Sync time
`ProjectIndexingActivityHistoryListener`	Indexing time
`ProjectIndexingActivityHistoryListener`	Scanning time
`PerformanceListener`	IDE freezes

Each metric event is enriched with shared context captured at event time: IDE version and build number, heap memory usage, focus state (enabled/disabled, number of focused modules), Operating System (OS) info, and project name. This means every data point is automatically segmented by whether focus mode was active, which is exactly what we need for before/after comparisons.

What each metric captures

Sync time: We implement GradleSyncListenerWithRoot and calculate wall-clock duration from syncStarted() to syncSucceeded() or syncFailed(). This covers the full Gradle configuration, dependency resolution, and IDE model generation phase.
Indexing time: ProjectIndexingActivityHistoryListener.onFinishedDumbIndexing() provides a ProjectDumbIndexingHistory object. We read history.times.totalUpdatingTime, the time IntelliJ spent updating its symbol index after the sync.
Scanning time: ProjectIndexingActivityHistoryListener.onFinishedScanning() provides a ProjectScanningHistory object. We read history.times.totalUpdatingTime and history.times.scanningType (full vs. partial) for additional segmentation.
IDE freezes: PerformanceListener.uiFreezeFinished(durationMs) is called by the platform whenever the Event Dispatch Thread (EDT) is blocked long enough to be classified as a freeze. The duration arrives directly as a parameter.
IDE memory usage: Captured at the moment of each metric event via Runtime.getRuntime(). Captures used memory (totalMemory - freeMemory) and max heap. Attached to every event as part of the shared context.
IDE version: From ApplicationInfo.getInstance(), captures version name, full version string, and build number. Also attached to every event, enabling per-version breakdowns.

Survey

After each successful sync, the plugin triggers an in-IDE notification prompting developers to fill out a short survey. The notification respects developer attention; it uses a weekly reset cycle with a “Don’t remind me again” option that appears after the second prompt. These periodic qualitative check-ins complement the telemetry data and help surface pain points that raw numbers alone may not capture.

Establishing the baseline

The plugin collects focus_enabled on every event. Therefore, baseline numbers come directly from the same pipeline; they are simply the subset of metric events where focus_enabled = false. This means the before/after comparison is an apples-to-apples measurement from the same instrumentation, same engineers, same codebase, with no separate manual benchmarking required.

Results

Compose preview build

The focus approach also improved Jetpack Compose preview builds. Compose previews require a module build to render, and with fewer modules loaded, the IDE has significantly less indexing overhead. A typical UI module has just 5–10 local dependencies. With the focus plugin, a developer configures only those modules instead of all 2,000. Developers consistently report that Compose previews feel significantly more responsive in focus mode.

As a best practice, we recommend that teams separate their UI into dedicated modules containing only composable functions and minimal dependencies. This maximizes the benefit of focus mode for preview builds.

Memory usage

In focus mode, excluded modules are not configured by Gradle and not indexed by the IDE, significantly reducing both build-process and editor memory consumption from approximately 10 GB down to 2 GB. This frees up memory for Bazel builds and other tooling. Developers reported fewer freezes, faster code navigation, and more responsive autocompletion.

Sync time

We observed a dramatic reduction in per-sync IDE sync time. A full sync previously took around 26 minutes at the 95th percentile (p95). With the Focus plugin, sync times dropped to under 2 minutes for typical feature work. The p95 remains higher for modules with deep dependency trees, but in practice, sync times vary significantly depending on module size. A typical UI module with 5 to 10 dependencies syncs in roughly 2 minutes, while heavier modules with deep dependency graphs take longer. For most developers working on focused feature work, the improvement is dramatic.

Tradeoffs

Focus mode does come with limitations. IDE features like “Find Usages” and cross-module refactoring only cover the focused modules; developers occasionally need to expand their focus set or temporarily switch to a full sync for repo-wide operations. In practice, this has been a minor inconvenience compared to the productivity gained.

Conclusion

IDE sync time is one of those problems that slowly degrades the developer experience without a single dramatic breaking point.

Our solution combined three key ideas:

Reuse existing infrastructure: By generating the dependency graph during migrateToBazel, we eliminated the expensive Gradle configuration phase without adding a new build step.
Minimize adoption friction: The Groovy include shadow trick let us integrate the focus mechanism with just three lines of code, requiring zero changes from feature teams.
Invest in user experience (UX): The Android Studio plugin turned a manual, error-prone process into a one-click operation with useful module health indicators.

The results spoke for themselves. IDE sync time dropped from 35 minutes to under 1 minute (depending on module size). IDE memory consumption fell from 10 GB down to 2 GB, freeing up headroom for Bazel builds to run alongside the IDE. Compose preview update times improved significantly due to reduced indexing overhead. And adoption was frictionless. Engineers went from a manual, multi-step process to a simple Select → Focus → Sync flow with native IntelliJ integration.

As the codebase continues to grow, accelerated by AI-assisted development and a broader contributor base, we are also investing in guardrails to keep quality in check. An area we are actively exploring is using skills.md to guide AI coding agents when they generate new modules, encoding architectural conventions and dependency rules directly into the context that AI tools consume. This helps ensure that AI-generated code lands in the right shape from the start, rather than accumulating structural debt that compounds the sync and build problems described above.

Join us

Grab is a leading superapp in Southeast Asia, operating across the deliveries, mobility, and digital financial services sectors, serving over 900 cities in eight Southeast Asian countries: Cambodia, Indonesia, Malaysia, Myanmar, the Philippines, Singapore, Thailand, and Vietnam. Grab enables millions of people every day to order food or groceries, send packages, hail a ride or taxi, pay for online purchases or access services such as lending and insurance, all through a single app. We operate supermarkets in Malaysia under Jaya Grocer and Everrise, which enables us to bring the convenience of on-demand grocery delivery to more consumers in the country. As part of our financial services offerings, we also provide digital banking services through GXS Bank in Singapore and GXBank in Malaysia. Grab was founded in 2012 with the mission to drive Southeast Asia forward by creating economic empowerment for everyone. Grab strives to serve a triple bottom line. We aim to simultaneously deliver financial performance for our shareholders and have a positive social impact, which includes economic empowerment for millions of people in the region, while mitigating our environmental footprint.

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Enhancing Flink deployment with shadow testing

Thu, 07 May 2026 00:23:00 +0000

Introduction

Ensuring the reliability of Apache Flink deployments in Grab is crucial for the availability of our business-critical, real-time applications. While all applications are tested in a staging environment before getting promoted to the production environment, there is still a class of issues that can only surface when deploying in the production environment, e.g.:

The new version of the application is unable to cope with the volume or the nature of production traffic.
The new version of the application is unable to resume from a production checkpoint or savepoint taken by the previous version of the application.
Certain environment-specific dependencies or configurations are malfunctioning or misconfigured.

When an application faces such issues upon deployment in production, our in-house deployment system automatically rolls it back after 10 minutes of observation, leading to a downtime of the application for about the same duration.

In this article, we will describe how Grab’s data streaming team (Coban) has enriched the traditional deployment pipeline for Flink applications with a Shadow Testing stage that eliminates this downtime during deployment failures, enhancing the availability of our Flink applications during this critical moment of their lifecycle.

Shadow Testing is a testing technique whereby a new version of an application (Shadow) is deployed in parallel with the current version of the application (Main), but without impacting it. It involves replicating production data to the new version of the application and comparing its behavior with the current version of the application to identify potential issues and regressions.

Architecture overview

Figure 1. Overall architecture of Shadow Testing.

We integrated Shadow Testing directly into the production environment, alongside the Main application (1). The Shadow application is deployed next to it via the same deployment process (2). An environment variable isShadow=true as well as a distinct jobID are injected for runtime differentiation, enabling the Shadow application to produce its results to distinct, isolated sinks that do not interfere with those of the Main application (3).

Deployment flow

Shadow Testing is embedded within our normal Flink deployment pipeline to make it a seamless experience for the users of our platform.

Figure 2. Deployment flow diagram.

The deployment flow is as follows.

A user triggers a deployment of their Flink application in Grab’s in-house deployment tool. At this step, they decide whether they want to enable Shadow Testing for this particular deployment.
The deployment pipeline validates the input parameters provided by the user.
If the user has not opted for Shadow Testing, the deployment flow directly jumps to step 8 and deploys the latest version to the Main application. However, if the user has enabled Shadow Testing, the deployment flow first goes through the Shadow Testing stages described in steps 4 to 7.
The Shadow Kubernetes manifest is baked with its set of distinctive parameters:
- The application name is prefixed with shadow- which propagates to all the Kubernetes objects that are part of the Shadow application
- An environment variable isShadow is injected and set to true. It instructs the Shadow application to produce its results to the shadow sinks.
- A distinct Job ID is attributed
- The target Kubernetes namespace is overridden with a shadow namespace
The Shadow application is deployed into the shadow Kubernetes namespace.
The Shadow application runs for a configured period of 1 hour by default to reach a steady state. The status of the job manager is monitored to determine the success of the Shadow Testing. If the Shadow application is stable, the Shadow Testing is considered successful.
The user is prompted to continue with the deployment of the Main application.
The Kubernetes manifest of the Main application is baked with its standard parameters and the environment variable isShadow is set to false.
The Main application is deployed in its standard Kubernetes namespace.
After 10 minutes of observation, the deployment pipeline determines if the Main application is healthy by querying the status of its job manager. If it is healthy, the Main application is considered successfully deployed. Otherwise, the deployment pipeline automatically triggers a rollback to the previous version.

During the deployment, the user can leverage our standard observability stack to monitor the behavior of the Shadow application. For example, in the case of an Apache Kafka sink, they can compare the number of messages produced by the Main and Shadow applications.

Figure 3. Tracking of the Kafka messages.in_rate metric for the respective Kafka sink topics of the Main application (purple) and Shadow application (blue) at the beginning of the Shadow deployment stage.

Besides, our standard Datadog dashboard that comes with each application can conveniently be toggled to view the metrics of the respective Shadow application.

Connector implementation

Our standard sink and source connectors, provided by our platform, ensure the absence of interference with the Main application during Shadow Testing. For example, Kafka source connectors use distinct consumer group IDs, while the various sink connectors direct the data to dedicated shadow sinks.

The Flink application evaluates the isShadow environment variable to set up the connectors at runtime.

if (isShadow){
    // Shadow Testing operation
}
else {
    // Normal operation
}

The following table shows how some typical connectors are dynamically configured if isShadow=true:

Type	Connector	Dynamic configuration
Source	Kafka	The consumer group ID for the Shadow application is suffixed with -shadow. This is crucial so as to consume a full copy of the data stream without interfering with the Main application. Main application: `consumerGroup = <application_name>` Shadow application: `consumerGroup = <application_name>-shadow`
Source	Change Data Capture	The Server ID range of Debezium is shifted to the next non-overlapping range of the same size. This enables the Shadow application to get a full copy of the database binlog stream without interfering with the Main application. Note that the misleading Server ID naming is because Debezium acts as a pseudo-replica of the database server. Main application: `serverId = 1001-2000` Shadow application: `serverId = 2001 - 3000`
Sink	Kafka	The cluster endpoint is replaced with that of a Kafka cluster dedicated to Shadow Testing, set up with auto.create.topics.enable=true and 8h retention. Main application: `brokers = <flink-kafka>:9092` Shadow application: `brokers = <flink-kafka-shadow>:9092`
Sink	S3	The S3 bucket name is replaced with that of a bucket dedicated to Shadow Testing, set up with a 7-day retention lifecycle policy. Main application: `s3://<flink-s3>/<application_name>` Shadow application: `s3://<flink-s3-shadow>/<application_name>`
Sink	Metrics	The StatsD prefix configuration is overridden. A shadow. prefix is added. Main application: `flink.<application_name>.<metric_name>` Shadow application: `shadow.flink.<application_name>.<metric_name>`
Sink	Logs	The Shadow Kubernetes manifest prefixes the Shadow application name with shadow-. The resulting name becomes available as a field in Kibana, enabling discriminated filtering. This tweak is done at the Kubernetes manifest level, not at the Flink application level. Main application: `app_name = <application_name>` Shadow application: `app_name = shadow-<application_name>`

Conclusion

Our Shadow Testing framework represents a meaningful step forward in enhancing the reliability of our Flink applications during deployment. By leveraging and enriching the existing components of our platform, we have created a robust system that enables our users to confidently increase their Deployment Frequency and reduce their Change Failure Rate.

What’s next

To drive wider adoption, we intend to support more source and sink connectors. By expanding the range of supported connectors, we could empower teams to leverage Shadow Testing across a broader spectrum of applications.

For connectors that are less frequently used, we consider implementing a no-op approach combined with metrics collection to expose a minimal set of actionable data points.

We will remain focused on making Shadow Testing accessible, scalable, and adaptable to various applications. Stay tuned as we continue to push the boundaries of innovation and deliver solutions that enhance reliability and efficiency across our systems.

Join us

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Data Mesh at Grab (Part II): The foundational tools behind certification

Thu, 30 Apr 2026 00:23:00 +0000

Introduction

In Part I, we discussed why Grab is investing in a data mesh, referred to as the Signals Marketplace within Grab, as part of our evolving data culture. We also explained how data certification aids teams in reliably reusing data across different domains. However, cultural change doesn’t occur through principles alone; it happens when tools reshape people’s daily behaviors. Therefore, it is crucial for us to develop effective platforms that integrate these practices.

This follow-up focuses on these platforms that make certification work at Grab:

Hubble – the central metadata management platform with a built-in certification engine.
Genchi – the data quality observability platform.
Data Contract Registry – the central service for managing data contracts.

Together, these platforms turn data mesh principles into an operational system that scales across hundreds of thousands of datasets, streams, attributes, and metrics.

Hubble: The data discovery and governance layer

Hubble is Grab’s central metadata management platform and data catalog for all data assets, including datasets, dashboards, metrics, Machine learning (ML) models, and more. Built on top of the open‑source DataHub and heavily extended for Grab’s needs, it is the discovery and governance layer for the Signals Marketplace.

Figure 1. Hubble cataloging data assets across various data platforms in Grab.

What Hubble does for data mesh

For certification and data mesh, Hubble provides a few key capabilities:

Search and discovery: Data analysts/scientists, engineers, and product managers use Hubble to search the rich swath of data assets, then inspect schemas, documentation, lineage, and usage statistics in one place. This replaces back-channel questions and tribal knowledge with a single, self-serve catalog.
Ownership and domains: Every asset is tied to a domain and explicit technical and business owners. This enforces the domain ownership model that Signals Marketplace depends on. Producers are clearly accountable for the quality and lifecycle of the data products they publish.
Data contracts and documentation: Data contracts, classifications, and rich documentation live as structured metadata attached to each asset, not scattered across wikis and slide decks. Producers and consumers share the same source of truth when they ask questions like, “what does this table guarantee?”.
Lineage and impact analysis: Hubble provides table, column, and metric-level lineage so owners can see which teams and pipelines depend on their data before making breaking changes or deprecations. Instead of guessing, they can answer “what will I break?” with a single click.
Certification status: The familiar “Hubble green tick” turns trust into a first-class signal in the catalog. Assets that meet Grab’s certification criteria are clearly marked with a drill-down view of which criteria are satisfied (ownership, documentation, contracts, quality tests, upstream certification) and which are still missing.

From a consumer’s point of view, this collapses a lot of uncertainty into a simple workflow: search → filter to “certified” → pick the most suitable asset. They don’t need to reverse-engineer reliability from table names and hearsay.

Hubble’s system architecture

The open-source DataHub architecture is fundamentally event-driven and designed for high extensibility, moving away from the “passive” catalogs of the past, towards a “living” metadata graph. DataHub models everything as an Entity (e.g., a Dataset), composed of multiple Aspects (atomic versioned metadata like SchemaMetadata or Ownership), connected by Relationships (e.g., DownstreamOf).

Since introducing DataHub as Grab’s central data catalog in 2022, we’ve tailored it to Grab’s specific needs while continuously rebasing onto the latest open-source DataHub releases. This lets us adopt new capabilities from the community quickly, contribute improvements back, and evolve Hubble without forking away from the main project.

On top of the DataHub foundation, Hubble ingests metadata from Grab’s source platforms in two ways. Source systems either push changes as they happen, or Hubble periodically pulls metadata via Airflow jobs into the central metadata service that exposes GraphQL and REST APIs. Every change is then published to Kafka as metadata events (low-level change logs for indexers and audits, and higher-level semantic events for workflows), keeping Hubble’s search and lineage indices fresh and allowing downstream integrations like certification, deprecation notices, and governance automation to react to metadata changes in near real time.

This architecture is crucial for a data mesh because metadata evolves more rapidly than organizational changes. As domains change, tables are relocated, or pipelines are refactored, Hubble continuously updates, allowing certification to keep pace without the need for manual re-audits.

Figure 2. Hubble’s system architecture.

Computational certification via the certification engine on Hubble

A data asset’s certification state is not a manual label, it is computed by an event-driven certification engine built on the DataHub Actions framework. As source platforms for tables, streams, metrics, and user attributes push metadata into Hubble, every change becomes a metadata event. The engine subscribes to these events and re-evaluates the certification state of the data asset based on the predetermined certification criteria.

Conceptually, assets move between four states:

Uncertified: Never met all criteria.
Certified: Asset itself meets all required criteria.
CertifiedPlus: Stricter conditions than Certified, requiring both the asset and all of its upstreams to meet the criteria.
Revoked: Previously certified but now out of compliance.

The state diagram below captures how assets move between these states as metadata and upstream health change.

Figure 3. Certification state diagram.

While each asset type can add its own nuances (for example, metrics and attributes may also require explicit endorsement from business data owners), the core certification criteria are consistent:

Ownership and domain: Clear domain assignment plus accountable technical and business data owners.
Documentation and semantics: Table/metric documentation and, where relevant, column-level descriptions so consumers understand what the data means.
Lineage and upstream trust: For stronger levels like CertifiedPlus, upstream lineage must be present, and upstream assets must themselves be certified.
Contracts and runtime quality: Linked data contracts that spell out expectations, backed by required Genchi tests for freshness, volume/completeness, schema stability, and critical business checks.
Governance signals: No conflicting deprecation flags or policy violations (for example, missing required classifications on sensitive data).

If an asset satisfies the base criteria, the engine writes a certification aspect back into Hubble and surfaces it as a green tick; if the asset later falls out of compliance, certification is revoked, and downstream assets are re-evaluated as needed. This is because all certification changes are stored as time-series metadata and driven by events rather than a one-off checklist; certification becomes a continuous, metadata-driven process that keeps pace as the entity evolves.

Genchi: The data quality observability layer

Genchi is Grab’s in-house, self-service data quality observability platform. It allows teams to define and run data quality tests on their datasets, receive alerts when something goes wrong, and integrate those tests directly into data contracts so that issues are caught and contained before they impact downstream consumers.

Figure 4. Genchi job configuration page, where dataset owners enroll a table and set up freshness, volume, schema, and other data quality tests for it.

What Genchi does for data mesh

In a data mesh, every domain is responsible for the quality of the data products it publishes. Genchi is the guardrail that makes that responsibility practical at scale. At its core, Genchi turns “good data” into clear, testable pillars that a data asset must satisfy:

Freshness and timeliness: Is the data recent enough to trust? Genchi runs data freshness and pipeline-freshness checks so owners know when today’s numbers are really “today’s” and can spot delayed loads before they hit dashboards or models.
Completeness and volume: Are we seeing all the records we expect? Volume and completeness tests compare current loads against historical baselines or source systems to flag partial backfills, silent drops, or suspicious spikes.
Structural stability (schema): Did the shape of the data change? Schema checks detect added/removed columns and type changes so that teams don’t discover breaking changes only after pipelines or reports start failing.
Semantic validity (values and business rules): Do the values themselves make sense? Column-level and X-Validation tests enforce constraints like uniqueness, ranges, patterns, cross-table reconciliations, and more advanced anomaly detection on row counts and null percentages.

Wrapped around these pillars is the operational layer:

Genchi runs these checks continuously (on schedule or on pipeline completion), emits real-time alerts, and helps teams drill into failing records and trends instead of debugging blind.
Its health signals flow into the catalog and incident tooling, so consumers see quality status alongside metadata and contract breaches are handled consistently.

The result is a mesh where domains publish data products with explicit, machine-enforced quality guarantees, and consumers can safely reuse them without a central team hand-holding every request.

Genchi’s system architecture

The Genchi system is designed to handle validation workflows triggered by user actions or automated schedules. It utilizes Temporal for reliable workflow orchestration and Kafka for event-driven data distribution to downstream consumers.

Figure 5. Genchi’s system architecture.

Triggering tests on pipeline completion with Sync with Pipeline (SWP)

Before SWP, Genchi tests lived on their own cron schedules, completely decoupled from the Airflow pipelines that actually produced the data. Teams had to manually copy pipeline crons into Genchi, juggle offset/lookback math to point at the “right” pipeline batch, and hope that nothing drifted over time. The result: misconfigured pipeline duration Service Level Agreement (SLA) checks, and tests sometimes running too early, too late, or against the wrong batch. This leads to noisy alerts and false-positive Data Production Issue (DPI).

To solve this, Genchi leans on Lighthouse, Grab’s pipeline execution and monitoring service. Lighthouse tracks when Airflow jobs start, finish, and which data interval they cover, and exposes that as structured execution events that the rest of the observability stack can consume.

SWP then flips Genchi from “best-effort cron alignment” to event-driven orchestration. Instead of guessing when a pipeline should have finished, Genchi listens to Lighthouse execution events. When a pipeline run is completed, Lighthouse emits an event with the run’s schedule and data interval; Genchi consumes that event, spins up an ad-hoc validation run aligned to that execution, and runs data-quality tests on the corresponding slice of data.

Pipeline-freshness is modeled as its own run type, separate from the data-quality tests that run on pipeline completion. Instead of inspecting rows, it is triggered asynchronously on the same schedule as the pipeline and tracks when each run actually completes in Lighthouse. This gives data producers an intuitive way to get alerted when a pipeline exceeds its expected runtime, and to review historical runtime behavior for any drift over time.

In practice, this makes test orchestration both simpler and more trustworthy. Users no longer need to think about crons or offsets for their data validation jobs. “Run after my pipeline finishes” becomes the default, with advanced overrides for custom schedules when needed. Misconfigured freshness tests and noisy DPIs drop, because Genchi is now anchored to real pipeline execution signals rather than approximations.

Figure 6. Genchi pipeline-freshness run page for a table, showing the historical runtime patterns and the SLA status.

Data Contract Registry: The producer–consumer agreement layer

At Grab, a data contract is the explicit, versioned agreement between a producer and its consumers that defines the data’s shape and semantics, the quality and availability guarantees around it, and the rules for how and for how long it may be used. The Data Contract Registry is the source of truth for these agreements, while Genchi and platform-specific observability stacks continuously verify that reality still matches what the contract promises.

What Data Contract Registry does for data mesh

Within Grab’s data mesh, the Data Contract Registry is the producer–consumer agreement layer: it centralizes contracts for key assets (data lake tables, Kafka streams, metrics) so expectations on shape, quality, SLAs, and lifecycle live in one canonical place instead of being scattered across individual platforms. That single source of truth underpins Hubble certification (only assets with valid contracts can be certified), gives Kinabalu (Grab’s central incident lifecycle orchestrator) the context it needs to open and route DPIs when checks fail, and lets Ouroboros (a table lifecycle management tool) interpret lifecycle clauses consistently.

This is important for a data mesh because it transforms the concept of “data as a product” from a mere slogan into an operational reality. Contracts provide domain teams with a clear and enforceable method to specify their guarantees, offering consumers a solid foundation for trust and data reuse across domains. Importantly, a contract is only valuable if its promises are verifiable and enforceable. Schema expectations, quality checks, and SLAs are all connected to concrete tests and health endpoints, enabling downstream platforms to automatically detect breaches and manage or mitigate breaking changes, rather than treating the contract as static documentation.

On top of storing contracts, the registry also manages contract changes. When a contract evolves, say a schema tweak, a new freshness SLA, or a planned deprecation, it identifies the right stakeholders (direct downstream owners, heavy query users, and, for critical assets, deeper dependencies) and pushes targeted Slack notifications. Producers get a structured way to roll out changes safely while consumers get timely, actionable signals instead of surprise breakages so that both enforcement and change management are baked into the mesh.

In addition to storing contracts, the registry also manages contract changes. When a contract evolves such as a schema adjustment, a new freshness SLA, or a planned deprecation, it identifies the appropriate stakeholder (direct downstream owners, heavy query users, and critical assets with deeper dependencies) and then sends targeted Slack notifications to these stakeholders. This process provides producers with a structured method to implement changes safely, while consumers receive timely and actionable alerts, preventing unexpected disruptions. As a result, both enforcement and change management are seamlessly integrated into the data mesh.

The data contract specification

Under the hood, a data contract in the registry is a JSON construct that follows the contract specification. Grab’s data contract specification is inspired by the public Data Contract Specification, but adapted to our environment so that contracts plug directly into our observability stack and automated incident management workflows.

Notably, we embed data health and test health URLs in the contract itself. Each data-quality rule points to a concrete health endpoint, so Kinabalu can determine contract breaches and create DPIs by calling those test health URLs, without hard-coding what “healthy” means. For example, a completeness test on the latest partition can be marked healthy only if the last N days are complete, not just because the most recent test run happened to pass. The data health URL at the contract root then lets Kinabalu fetch the overall diagnosis and decide who the DPI should be assigned to. More on this will be covered in Part III.

Here’s a simplified example of a contract for a data-lake table:

{
  "specification_version": 1,
  "asset_urn": "urn:li:dataset:(urn:li:dataPlatform:hive,genchi.validation_jobs,PROD)",
  "entity_type": "datalake_table",
  "health_url": "https://example-hugo.grab.com/assets/genchi.validation_jobs/health",
  "contract_details": {
    "contact": {
      "oncall_group": "oncall-genchi",
      "slack_channel": "ask-genchi"
    },
    "terms": {
      "usage": "Source of truth for all genchi validation jobs.",
      "limitations": "Not suitable for real-time use cases.",
      "notice_period_in_days": 14
    },
    "schema": [
      {
        "type": "genchi",
        "health_url": "https://example-genchi.grab.com/assets/genchi.validation_jobs/tests/fundamental_schema_test/health",
        "uid": "fundamental_schema_test",
        "version": 1
      }

    ],
    "sla": {
      "freshness": [
        {
          "type": "genchi",
          "health_url": "https://example-genchi.grab.com/assets/genchi.validation_jobs/tests/fundamental_freshness_test/health",
          "uid": "fundamental_freshness_test",
          "version": 1
        }
      ],
      "lifecycle": null
    },
    "data_quality": [
      {
        "type": "genchi",
        "health_url": "https://example-genchi.grab.com/assets/genchi.validation_jobs/tests/fundamental_completeness_test/health",
        "uid": "fundamental_completeness_test",
        "version": 1
      }
    ]
  }
}

The contract in the registry only stores references to enforceable rules, which are an identifier and version, like the example above, rather than the full configuration body. This keeps contracts lightweight and tool-agnostic, while giving rule-enforcement tools (Genchi for data-quality tests, Ouroboros for table lifecycle) and Kinabalu a stable handle to resolve the actual rule definition in their own systems, without duplicating configuration or letting it drift across platforms.

Conclusion

By bringing Hubble, Genchi, and the Data Contract Registry together, we provide the foundational tools to build trust in certified data assets. Hubble enables discovery and establishes domains and ownership; the Data Contract Registry captures explicit expectations between producers and consumers; and Genchi continuously validates those promises with tests on freshness, volume, schema, and business rules. Hubble’s certification engine then evaluates these ownership, contract, and quality signals to decide whether an asset meets Grab’s standards and surfaces that as a visible certification state. As a result, consumers can confidently default to certified assets, usage converges on a smaller, better-governed pool of datasets, and certification becomes a mechanism that changes producer behavior and guides consumer choice. We saw this convergence in practice. In just one year since the Signals Marketplace campaign began in 2024, the number of P80 datasets (the most used tables that account for 80% of all queries) has dropped by over 58%.

This data foundation is especially important in an AI-first future for Grab. Certified streams, tables, metrics, and attributes give AI agents and automated analytics a default substrate they can rely on. With Hubble and Genchi, data producers have clear ownership, contracts, and observability. Data consumers can discover and trust certified assets without guesswork, and platform teams can measure and improve Signals Marketplace health over time (for example, queries on certified assets, lineage depth, and cost). Together, these capabilities turn “data mesh” from a slogan into an operational, AI-ready marketplace of reliable, reusable signals that power decisions across Grab.

Figure 7. Building trust in certified data assets through discovery, contracts, and continuous validation.

What’s next

In the next blog, we’ll zoom into the DPI process itself with Kinabalu as the incident lifecycle orchestrator:

How Genchi test failures and data contract breaches turn into DPIs.
How DPI is assigned based on root causes and what sets the priority level.
The patterns we’ve seen in “noisy” vs actionable DPIs, and what we’ve changed in our platforms.
How we’re using automation and agents to reduce DPI toil and close the loop back into certification.

We’ll walk through concrete case studies showing how a single broken table moves from first failure, to diagnosis and fix, to updated contracts and a more resilient certified data asset.

Join us

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!