Grab Tech

GrabMart data for faster disease outbreak detection

Tue, 13 Oct 2026 00:23:00 +0000

Abstract

Retail purchases can supplement clinical surveillance because changes in demand for some over-the-counter (OTC) products may appear before confirmed case reports. This article describes an early-stage concept for analysing aggregated GrabMart transaction data to identify unusual changes in product demand. The concept is not a deployed system, and it would require validation by public health experts, privacy review, and regulatory approval before any implementation.

Introduction

Public health agencies use sources such as clinical case reports, laboratory results, and hospital records to monitor disease activity. Retail data may offer an additional signal when demand changes across groups of symptom-related products. It cannot establish a diagnosis or replace clinical evidence.

When using indirect signals, we should assess whether they remain relevant to the outcome we want to measure. This includes considering how changes in user behaviour could affect their interpretation. A retail-based signal can experience similar drift, selection bias, and false alarms. Any useful design must therefore combine statistical validation with careful privacy and governance controls.

Background

Syndromic surveillance tracks patterns that may indicate emerging illness before diagnoses are confirmed. Retail surveillance applies this idea to changes in demand for products associated with symptoms, such as fever remedies or cough medicines.

The proposed method groups eligible products into broad symptom classes and measures changes in aggregate demand over time. An anomaly score can indicate that observed demand differs from an expected seasonal pattern. The score is a statistical signal, not an estimate of how many people have a disease.

The design uses the following terms:

Proposed solution

The proposed pipeline would process only transactions that meet documented eligibility rules. It would map eligible SKUs to symptom classes, aggregate counts across approved time periods and geographic areas, and compare each series with an appropriate baseline. Public health experts would review the resulting Early Outbreak Detection Index alongside independent evidence.

The method deliberately excludes identity linkage and individual-level health inference. It also avoids using a single purchase to classify a customer, target an advertisement, adjust an insurance decision, or change access to a product.

Conceptual data flow

Eligible transactions
        |
        v
Reviewed SKU-to-symptom mapping
        |
        v
Aggregate by approved time period and geographic area
        |
        v
Compare with seasonal and historical baselines
        |
        v
Privacy and quality checks
        |
        v
Aggregate signal for authorised public health evaluation

Analysis and validation

Several statistical methods could model expected demand, including seasonal time-series models and spatial scan statistics. Model selection should follow retrospective evaluation against approved public health data. Evaluation should measure sensitivity, false-alert rate, calibration, geographic stability, and performance during seasonal changes.

The system would also need ongoing monitoring. Product catalogues change, purchasing behaviour shifts, promotions affect demand, and access to GrabMart varies across populations and locations. These factors can produce anomalies unrelated to disease activity.

Privacy and governance

Privacy protection starts with data minimisation. The analysis should use only eligible product categories, coarse geographic areas, and aggregate time windows approved for the stated public health purpose. It should not expose raw transactions or personally identifiable information (PII).

The release process could apply minimum-count thresholds and differential privacy (DP), subject to a documented privacy analysis. Privacy specialists would need to set and review any privacy budget. Access controls, retention limits, audit logs, and contractual restrictions would govern any approved data exchange.

An independent review should cover privacy law, health-related inference, fairness, scientific validity, and the risk of harm from false alerts. An authorised public health body, rather than an automated model, should decide whether and how to communicate a possible outbreak.

Potential applications

If research validates the signal and the required approvals are in place, aggregate trends could support:

public health research alongside established surveillance sources;
operational planning for broad categories of health supplies; and
evaluation of whether retail demand contributes useful evidence during an emerging event.

These applications should use aggregate trends only. The signal should not drive personalised medical advice, individual targeting, insurance decisions, or automatic changes to medicine search rankings.

Limitations

Retail demand is an indirect and incomplete measure of illness. Purchases may reflect promotions, weather, media coverage, household stocking, or changes in product availability. GrabMart customers are not representative of the full population, and the available product mix differs by market.

Mapping a product to a symptom class is also uncertain because many products have several uses. Aggregation and privacy controls reduce disclosure risk but can weaken statistical utility. These limitations require local validation and documented thresholds before anyone interprets the index.

Conclusion

Aggregated retail transactions may add a timely signal to public health research, but they cannot replace clinical surveillance. A responsible design must minimise data, prohibit individual health inference, test the model against independent evidence, and place public health experts in control of interpretation and communication.

This concept remains in the early planning phase. No production deployment, measured performance result, or public health integration is claimed.

Join us

Grab is a leading superapp in Southeast Asia, operating across the deliveries, mobility, and digital financial services sectors. Serving over 900 cities in eight Southeast Asian countries: Cambodia, Indonesia, Malaysia, Myanmar, the Philippines, Singapore, Thailand, and Vietnam. Grab enables millions of people every day to order food or groceries, send packages, hail a ride or taxi, pay for online purchases or access services such as lending and insurance, all through a single app. We operate supermarkets in Malaysia under Jaya Grocer and Everrise, which enables us to bring the convenience of on-demand grocery delivery to more consumers in the country. As part of our financial services offerings, we also provide digital banking services through GXS Bank in Singapore and GXBank in Malaysia. Grab was founded in 2012 with the mission to drive Southeast Asia forward by creating economic empowerment for everyone. Grab strives to serve a triple bottom line. We aim to simultaneously deliver financial performance for our shareholders and have a positive social impact, which includes economic empowerment for millions of people in the region, while mitigating our environmental footprint.

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Smarter personalization: How property data helps us understand user price sensitivity

Tue, 06 Oct 2026 00:23:00 +0000

Introduction

Existing systems estimate user price sensitivity primarily from spending behavior or demographic proxies. They do not systematically account for residential property values, which can indicate a user’s financial circumstances. This omission creates three limitations:

Limited property insights: Existing profiles do not account for property values.
One-dimensional profiles: Users with similar spending patterns but different living standards receive the same classification.
Regional variation: Manual classification does not adapt well to differences between property markets.

This invention adds property data to user profiles to produce affluence segments that account for regional property markets. The invention remains unimplemented and is not deployed in any live market.

Background

Property values vary across buildings, neighborhoods, and regions. A useful classification method must account for these differences while processing property records in several formats. The same classification method could support other industries that use affluence segments.

Solution

A four-step method that combines public property data with clustering techniques to classify user affluence was created:

Data collection and enrichment.
Building-level property price estimation.
Property tier classification.
User mapping to affluence tiers.

The method combines external property data, large language model (LLM) enrichment, and cluster-based segmentation. The resulting profiles can inform personalized offers, targeted marketing, and operational planning.

Data gathering and processing

The system collects public property data from external sources. These records include transaction prices, building types, and related attributes in raw text formats. LLMs standardize the records and extract key attributes such as postcode, building address, transaction price, price per square meter, property size, and floor level. This stage produces a structured property-level dataset for analysis and modeling.

Building-level price estimation

Under the proposed method, transaction records for properties within the same building would be aggregated to calculate a weighted average sale price. The weighting would account for transaction recency and data volume, with the aim of reflecting recent market conditions.

When transaction data is limited, the method could apply rules based on comparable properties in the area. It could also evaluate LLM-assisted extraction of contextual signals from appropriately licensed external sources, such as real estate listings. Any LLM-derived signals would require validation against independent reference data before contributing to a property-value estimate.

The invention has not been implemented, deployed, or tested in a live market. Its model and version, validation methodology, error bounds, and operational safeguards remain subjects for future technical evaluation.

Property tier classification

A hierarchical clustering model groups buildings into affluence tiers based on their estimated resale values. The model accounts for regional differences and local market-value distributions across property types and locations. This stage assigns each building to an affluence tier.

User mapping to property tiers

The system maps would use appropriately aggregated geographic signals to associate users with broad property-market segments and assign a corresponding affluence classification.

Potential future applications

This invention presents a conceptual approach for exploring new ways to support personalization, advertising, and operational planning. It has not been implemented, deployed, or tested in a live market. Subject to technical validation and approval from Privacy, Legal, PR, and the relevant product owners, potential applications could include:

Personalization: Tailoring promotions, discounts, subscription plans, and recommendations for optional products and services.
Advertising: Supporting more relevant connections between advertising content, merchants, and broad audience segments, alongside tailored marketing campaigns.
Operational planning: Informing aggregated analysis used to plan and prioritize delivery and transport capacity.

Learnings and conclusion

This invention presents a conceptual approach that combines publicly available property data, LLM-assisted data structuring, building-level value estimation, hierarchical clustering, and contextual tier assignment. It demonstrates how regional property-market context could complement existing signals, while highlighting dependencies on data quality, model accuracy, geographic coverage, privacy, and fairness.

The approach shows potential applications in personalization, advertising, and operational planning, but these are illustrative, not demonstrated outcomes.

Join us

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Powering AI-led research through simulation

Mon, 05 Oct 2026 00:00:00 +0000

The short story

Consider a Friday evening. A food order arrives from a mall in the city center. One driver is nearby; another is finishing a drop-off and will be available shortly; a second order from the same mall may or may not appear in the next two minutes. Dispatch the nearby driver now, or hold briefly for a batching opportunity? The decision window is short.

A fulfillment marketplace makes these decisions continuously. Each one is small. Across a city, those decisions determine whether your dinner arrives hot and whether a driver’s hour is well spent.

And that is one decision. There are dozens more: how far to look for a driver, when two orders are worth combining, which of three waiting trips gets the one free bike, how long to keep trying before giving up. These decisions interact, and the setting that is right for Friday at seven is wrong for Tuesday at two. Together they define a space of possible strategies far larger than anyone could exhaustively explore.

We sample only about a dozen points in that space each year. Not for want of ideas: implementing each strategy costs weeks of engineering, and a production experiment takes weeks more to judge. Some questions have no production answer at all. Nobody can run last Saturday again with twenty percent fewer drivers. We were never short of compute resources, and never short of data. We were short of attempts.

So we built sim-rs, a simulator that makes each attempt take minutes, and connected agents that run experiments on it.

sim-rs is self-contained: it requires no production services or databases. The dispatch lifecycle runs in one compiled program alongside a separate dispatch service that is spawned locally and operates offline. Historical booking and driver data go in as plain files, together with a configuration describing the strategy to test. Simulated bookings, trips, drivers, and a single metrics report come out. Processing one city-day of marketplace activity takes tens of minutes, from raw input to finished report. One command in, one report out: a workflow as practical for a software agent as for an engineer.

That compact contract is what makes the simulator AI-friendly. Agents can change bounded components, run reproducible experiments, receive verdicts they cannot alter, and help keep both production logic and behavioral models current.

Building blocks

The essential components

To model a marketplace, the simulator needs four core concepts:

Bookings: requests to move something (a passenger, a meal, a parcel), represented in one format regardless of the business vertical.
Drivers: simulated workers with a location, a shift, a vehicle, and a queue of work.
Trips: units assigned to drivers, containing one booking or several batched into a multi-stop route.
Ticks: simulated time, advancing in fixed steps. Nobody waits in real time.

At each tick, the simulator runs the following sequence:

Demand arrives. Historical (or synthesized) bookings whose time has come enter the pool.
Supply moves. Drivers come on shift, advance along routes, go idle, and reposition.
Pre-dispatch cancellations are applied. Cancellation models, including survival-analysis models trained on real behavior, determine which bookings leave the pool.
Dispatch happens. Bookings are batched into trips, trips are matched to drivers by an optimization solver, and a recycling step decides, for each trip, whether to send it now, hold it, or split it back into bookings and try again later.
Post-dispatch cancellations are applied. These include passenger and driver cancellations.
Outcomes are recorded. Every booking, trip, and driver outcome is written to the run outputs.

Crucially, this is a closed loop: today’s dispatch decisions change where drivers end up, which changes what’s possible next tick. That feedback produces second-order effects that a static replay, one that scores historical decisions without updating future supply, cannot show. Supply may dry up in a hot zone, or one bad dispatch rule may cascade into a wave of cancellations.

Figure 1. Simulator sequence.

That loop is only the mechanism. Making it a lab takes three further properties: configurability; reproducibility and auditability; and reliable comparison.

1. Interchangeable stages. The major dispatch stages are pluggable. Batching, allocation, cancellation, routing, driver movement, and recycling are each exposed through an interface with interchangeable implementations selected by configuration. This turns a fixed pipeline into an experimental platform: the component under study can be swapped while everything around it stays unchanged. The same mechanism determines which marketplace is being simulated. Ride-hailing, food delivery, parcel delivery, or all three sharing one driver pool are configuration choices within the same codebase, not separate simulators.

2. Self-contained runs. Each experiment is a small, portable package containing its configuration files, input data, source revision, metrics report, and detailed outputs. Anyone holding that package can recreate the setup, whether a reviewer, teammate, agent, or the original author six months later. The report identifies the configured components and headline metrics, while the supplementary per-booking, per-trip, and per-driver outputs let reviewers recompute those metrics and investigate unexpected outcomes without relying on separate notes or systems.

3. Experiments inside the simulation. Production marketplaces have experimentation platforms, so our simulator ships with one too. Experimentable fields can define multiple treatment arms assigned through the same time-based switchbacks, spatial splits, or cell-and-hour schemes used in production. Each booking records its resolved arm for traceability. Running these A/B/n comparisons together exposes every arm to the same demand, drivers, marketplace dynamics, noise, and biases. By reproducing both the conditions and assignment design of a marketplace experiment, simulated insights are more likely to translate into effects observed in production.

Automating the research cycle

Agents do two jobs here, and both need the same environment: searching for strategies that beat the incumbent, and keeping the simulator sufficiently faithful for that search to mean something.

Searching for better strategies

We connect agents to that interface through autoresearch, an automated loop that runs the research cycle end to end. An agent proposes a change, implements it in source code, tests it, and acts on the verdict. A candidate survives only if it improves the target metric and passes every required check; otherwise it is rolled back and the loop continues. The objective may be a marketplace outcome such as orders-throughput or a software measure such as execution time. What matters is a repeatable command-and-metric contract that both humans and agents can use.

The first failure mode we encountered was specification gaming. Given a target and a loophole, an agent may find the shortest path to the number rather than the improvement we intended. Ours discovered that changing fields used by pre-dispatch cancellation could reduce cancellations and raise completion without improving a single dispatch decision. The metric moved; nothing real had improved. Rather than relying on instructions alone, we built four safeguards into the environment:

Core data is immutable. Modules under test may change only the state exposed by their interfaces. An attempt to manipulate protected fields fails to compile instead of producing a misleading result.
The verdict is computed, not judged. The simulator computes its own metrics, which the agent can read but not redefine. Executable checks apply calibrated tolerance bands, verify equivalent outputs where required, and run the tests. The agent never judges its own work.
Changes are scoped, checked, and reversible. Each experiment runs on its own branch within a declared writable scope, which is checked against the resulting diff. The grading machinery remains outside that scope, regressions cost only a discarded branch, and every surviving change is bounded enough to review in full.
Changes are checked at two levels. It first rejects changes that violate type contracts, ownership rules, interface boundaries, or concurrency requirements. The evaluation harness then applies checks suited to the objective: performance work must preserve expected outputs, while strategy experiments must satisfy domain constraints and outcome thresholds.

The speed that matters is the speed of an adaptive research loop, not simulation runtime alone: each iteration uses previous results to propose the next change, then implements, compiles, runs, scores, and decides whether to keep it. The simulator returns a verdict in milliseconds for benchmarks or tens of minutes for a full-day replay, while autoresearch carries each verdict into the next attempt without human handoffs. This turns an overnight run into a connected sequence of evidence-driven experiments, with the safeguards above ensuring that faster iteration compounds reliable evidence rather than mistakes.

One outcome is the familiar purpose of simulation: discovering better marketplace strategies. When asked to explore trip-recycling policy, the loop turned an emerging human intuition into a concrete rule: hold batched orders only as long as service-level deadlines permit, maximizing the chance that another nearby order joins the trip. Because the result is human-readable code, engineers can review, audit, and deploy it like any other pull request.

The same machinery also improves the simulator itself. Over several nights of unattended operation, we aimed the agents at two hot paths in its dispatch logic: the checks that decide which drivers are eligible for a trip, and the pass that adjusts the cost of every driver-and-trip pairing before the solver chooses an assignment. Across roughly 150 logged experiments, three out of four attempts failed to build or were rejected by later gates. The eligibility checks ended up about 10 times faster, and the cost pass about 24 times faster. Because these paths run repeatedly in every replay, improvements compound across subsequent research. Faster and more efficient runs enable testing of more ideas across more markets and dates, repeat runs to separate signal from noise, and validate results more rigorously. Shorter runs also tighten the feedback loop from verdict to next proposal, so improving the instrument accelerates and strengthens every search performed with it.

Execution time is only one possible objective: pointing the same machinery at orders-throughput changes the research question, not the loop itself. Regardless, whatever the objective, the result is only as trustworthy as the simulator behind it. An agent can optimize only the world it is given; if that world has drifted from production, a faster loop will merely produce misleading answers sooner.

Keeping the simulator faithful

A simulator is a claim about the world: that this is how the system behaves and that this is how the people within it respond. Both halves of that claim decay. Production logic changes continuously, while models of passenger and driver behavior grow stale as new product features reshape how people act. A simulator that has drifted from the world produces misleading insights and innovations that fail in production. So the second job we give agents is to keep the simulator faithful.

The system half is a translation problem, and coding agents are particularly good at it. An agent can read a component’s production implementation and implement equivalent logic behind the simulator’s corresponding interface, translating directly from source code rather than from a written description that may already be stale. The result remains a candidate until parity checks show that it matches the production behavior being modeled.

The behavioral half cannot be translated, because no source file tells us how a passenger will behave. It must be inferred from what people actually do. Here, we redirect the same autoresearch loop from dispatch-code optimization to behavioral modeling. It proposes, fits, and scores candidate models, exploring which features predict cancellation and how their effects should be represented. The test is not only how closely a model explains its training data, but also how well it generalizes to held-out data. A model that fits last Tuesday perfectly and next Tuesday poorly does not make the simulator more faithful; it makes the lab more confidently wrong.

The pieces are simple: a realistic marketplace model, swappable parts, repeatable experiments, honest scoring, and an easy way to undo failures. Together, they give agents a safe place to test and improve ideas. Everyone is racing to give AI a bigger brain. We got further by giving it a better lab: a place where it can try a thousand ideas, be wrong cheaply, and receive an honest verdict.

Join us

Grab is Southeast Asia’s leading superapp, serving over 900 cities across eight countries (Cambodia, Indonesia, Malaysia, Myanmar, the Philippines, Singapore, Thailand, and Vietnam). Through a single platform, millions of users access mobility, delivery, and digital financial services, including ride-hailing, food delivery, payments, lending, and digital banking via GXS Bank and GXBank. Founded in 2012, Grab’s mission is to drive Southeast Asia forward by creating economic empowerment for everyone while delivering sustainable financial performance and positive social impact.

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Automated merchant review summary system with integrated feedback

Fri, 25 Sep 2026 00:00:00 +0000

Introduction

Merchant reviews contain useful details about food quality, portion size, packaging, and value. Finding these details often requires reading many comments. Aggregate ratings simplify comparisons, but they do not explain what shaped each score.

We built the Automated Merchant Review Summary System to turn written reviews into concise summaries. Consumers can scan an overall summary or focus on a specific topic. Merchants can identify what customers value and which parts of the experience need attention.

This article focuses on how we generate and maintain consumer-facing review summaries. A misleading or incomplete summary can affect how consumers view a merchant. Merchant-facing recourse and dispute processes sit outside this pipeline and are not covered in this process flow.

Problem statement

A long, unstructured list of reviews creates two related problems. Consumers spend time searching for comments that match their priorities, while merchants must combine individual comments to identify recurring themes.

Consumers may see hundreds or thousands of reviews for a merchant. Reading only the first few can produce an incomplete picture, while the aggregate star rating provides little context. Two restaurants with the same rating may differ in food quality, delivery experience, portion size, or value.

Merchants face a related problem. Reading and classifying every review takes time, and the task becomes harder as review volume grows. Recurring complaints and positive patterns can remain hidden among individual comments.

Existing approaches

Manual review analysis

Consumers and merchants can read reviews one by one and form their own conclusions. Their conclusions depend on which reviews they read and how they interpret them. This method becomes impractical as review volume grows.

Aggregate star ratings

An average score quickly indicates customer opinion but does not explain the reasons behind it. A merchant can see overall performance but not the themes that may require attention.

Keyword search and filtering

Keyword search helps readers who already know what to look for. It does not reveal unexpected themes, and literal matching may separate phrases with the same meaning. For example, “too cold”, “not warm enough”, and “lukewarm” all describe a food-temperature issue.

Our solution

The system uses representative sampling, large language model (LLM) summarisation, automated quality checks, and consumer feedback. It generates an overall summary and topic-based summaries for areas such as taste, packaging, and portion size.

The pipeline has three flows:

Summary generation and validation: The pipeline samples reviews, generates overall and topic-based summaries, and checks whether the source reviews support each claim.
Summary maintenance: The system monitors new reviews and consumer feedback, then regenerates a summary when either reaches a configured threshold.
Summary display and feedback: The interface retrieves a validated summary for the merchant’s review page and records whether consumers found it helpful.

Architecture

The generation pipeline prepares the source reviews, produces summaries, and validates the output before publication.

Figure 1. Representative review sampling flow.

Data preparation and representative sampling: The pipeline calculates the merchant’s rating distribution. A sentiment model assigns each written review a score from one to five. Stratified sampling then selects a smaller set of reviews that follows this distribution, reducing the chance that one rating group dominates the summary. However, matching the rating distribution does not guarantee that the sample represents every period, language, customer segment, or aspect of the merchant experience.

Summary generation with an LLM: The system sends the sampled reviews to an LLM with instructions to produce:

A concise summary of the sampled reviews.
Topic-based summaries for categories such as taste, packaging, and portion size.

The system can update the topic set when new themes appear in consumer feedback.

Figure 2. Summary generation, validation, and feedback flow.

Quality control (QC) with an LLM: A separate LLM receives the generated summary and the sampled reviews. Summaries that do not meet the validation criteria are withheld from publication. Passing this automated assessment reduces, but does not eliminate, the risk of unsupported claims. The maintenance flow updates stored summaries in response to incoming reviews and consumer feedback:

New-review trigger: A monitoring service counts reviews submitted after the latest summary. When the count reaches a configured threshold, the service starts the generation and validation flow.
Consumer-feedback trigger: The interface records whether consumers found a summary helpful. When negative feedback reaches a configured threshold, the system generates and validates a new summary.

Figure 3. Consumer feedback analysis and prompt refinement flow.

Impact

Review summaries are designed to reduce the amount of text consumers need to scan. Topic-based summaries can surface themes that are not visible in an aggregate rating alone. The quality-control step rejects unsupported claims. The regeneration triggers incorporate new reviews and consumer feedback into later summaries.

Learnings and conclusion

Useful review summaries depend on more than generation quality. The source sample must reflect the intended population, each claim must remain grounded in source reviews, and the output must change with the underlying feedback. Automated LLM validation is one layer of defence, not sole proof of accuracy, and feedback-driven regeneration needs anti-abuse controls to remain trustworthy.

Representative sampling, separate generation and validation stages, and feedback-based regeneration form a practical system for maintaining consumer-facing merchant review summaries. This article describes that technical pipeline; it does not cover the full set of merchant recourse options when a summary is disputed.

Join us

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Data Mesh at Grab (Part III): Operationalizing data reliability with automated DPIs

Fri, 28 Aug 2026 00:00:00 +0000

Introduction

In the first two parts of this series, we described how Grab approaches data mesh through the Signals Marketplace: a way for teams to publish, discover, and reuse trusted data products across domains. Part II introduced the foundational tools behind certification: Hubble for metadata and ownership, Genchi for data quality observability, and the Data Contract Registry for explicit producer-consumer guarantees.

Certification is the starting point for a trusted data marketplace. It gives downstream consumers confidence in an asset’s ownership, documentation, lineage, and quality controls. Certification does not eliminate runtime failure. A certified table can still arrive late. A certified metric can still be affected by a broken dependency. A certified Kafka stream can still violate a freshness expectation.

Keeping certified data products reliable in production requires more than defining standards upfront. Teams need a consistent way to detect failures, diagnose the root cause, fix the issue, and verify recovery. That is where Data Production Issues (DPIs) come in. At Grab, DPIs turn data quality signals into an operational workflow.

The DPI lifecycle

A good DPI should be clear enough to act on, and it should close automatically when the underlying condition recovers. From the beginning, we designed the DPI lifecycle to be automated, with minimal human-in-the-loop.

The lifecycle starts when Kinabalu, Grab’s incident orchestrator, observes that a data asset may no longer satisfy its contract. The contract captures the reliability expectations that matter for the asset, along with the health checks, exposed through Test Health application programming interfaces (APIs), that evaluate those expectations.

The orchestrator stays decoupled from platform internals. It does not need to know how each platform computes freshness, completeness, or other quality dimensions. It only needs to ask whether the relevant contract tests are healthy. If one or more contract tests are unhealthy, the contract is considered breached, and the DPI lifecycle begins.

Figure 1. Automated DPI workflow.

Triaging DPIs: From alerts to confirmed contract breaches

Data platforms emit many alerts. An Airflow schedule may be delayed, a data quality test may fail, or a pipeline job may exit unexpectedly. These alerts are useful, but they are not automatically DPIs. Triage decides whether an alert represents a real contract breach for a data asset.

As introduced in Part II, a data contract is an explicit, versioned agreement between a data producer and its consumers. It outlines the data’s schema, freshness, completeness, and other semantic guarantees. These guarantees are codified and enforced through data quality tests in Genchi.

When the incident orchestrator evaluates contract tests, it distinguishes an individual test run result from the overall health of a test. A test run can pass or fail at a point in time, but the test itself may only be considered healthy after the underlying issue has been fully resolved. For example, consider a completeness test that checks whether the T-1 daily partition is complete. If the test failed two days ago but passed yesterday and today, the test may still be considered unhealthy until the partition from two days ago has been backfilled and verified as complete.

The orchestrator also deduplicates around the active unhealthy condition. If an asset already has an open DPI for the same breach, new signals update the existing DPI with additional context rather than creating parallel issues. DPIs that share the same underlying root cause can also be grouped. This keeps responders focused on solving the underlying issue rather than chasing a stream of repetitive alerts.

During triage, the workflow also gathers context for the DPI: affected asset, breached contract, unhealthy tests, data interval, and upstream and downstream dependencies. Not every alert becomes a DPI. Triage protects the operational workflow from noise by promoting only meaningful contract breaches into production issues.

Diagnosing DPIs: Assigning owners with root cause analysis (RCA)

Once a DPI is created, the system must answer why the data is unhealthy, who should fix it, and how.

Not every data issue should be assigned to the data asset owner. A data product may be unhealthy because of a platform incident, a failed producing job, or a delayed upstream dependency. Assigning every issue to the asset owner creates unnecessary handoffs and slows down resolution.

This is where the Data Health API matters. It answers the question: “What kind of failure made this asset unhealthy?” The Data Health API keeps the error taxonomy small:

UPSTREAM_ERROR: the asset is unhealthy because an upstream dependency is late, failed, or unavailable.
PLATFORM_ERROR: the asset is unhealthy because the underlying platform or infrastructure is impaired.
JOB_ERROR: the asset is unhealthy because the producing job or pipeline failed.
DATA_ERROR: the asset is unhealthy because the produced data violates quality expectations.

The taxonomy is not meant to replace platform-specific diagnostics. The high-level Data Health API gives the orchestrator just enough structure to assign DPIs and manage their lifecycle consistently. An ingestion platform, streaming platform, metrics platform, or machine learning (ML) platform can still maintain detailed internal error catalogs, logs, retry states, and debugging tools. Platforms remain free to evolve their internals, while the incident orchestrator consumes a stable API contract, so the DPI workflow can interoperate across heterogeneous systems.

A simplified Data Health API response might look like this:

Disclaimer: The fields in this API response are mock data generated for demonstration purposes and do not represent real operational metrics.

{
  "assetId": "urn:li:dataset:(urn:li:dataPlatform:hive,schema.table_A,PROD)",
  "healthStatus": "UNHEALTHY",
  "errorCategory": "UPSTREAM_ERROR",
  "context": {
    "upstreamAsset": "urn:li:dataset:(urn:li:dataPlatform:hive,schema.table_B,PROD)",
    "reason": "upstream data has not arrived for the expected data interval."
  },
  "lastCheckedAt": "2026-06-15T08:30:00Z"
}

From this response, the orchestrator can see that table_A is unhealthy because of an upstream dependency rather than a problem in the asset itself. It then traces the active DPI for the upstream asset and links the table_A DPI to that upstream issue. The downstream DPI can inherit the same owner as the upstream DPI, keeping related failures grouped under the team best positioned to resolve the root cause.

The DPI process works only when the issues it raises can be assigned and fixed. If DPIs are frequently noisy, duplicated, or difficult to act on, users will eventually learn to ignore them. Diagnostic accuracy matters because it keeps DPIs useful for the people who receive them. It also creates a forcing function for each data-producing platform to improve its diagnostics. To produce accurate RCA, platforms need to incorporate signals from their dependencies and surrounding systems, not just their own local failure state.

Grab operationalizes DPI diagnosis across its internal data platforms. Our ingestion platform, Hugo, is a primary example of this approach, as outlined in a previous tech blog. Hugo’s intelligent diagnosis architecture uses a three-layered system to automatically detect, analyze, and troubleshoot data pipeline failures within its domain, as shown in Figure 2.

Figure 2. Hugo diagnosis architecture.

Modern data platforms generate alerts from many independent systems. Individually, these signals show only a partial view of a dataset. Hugo consolidates platform-specific signals into a unified diagnostic workflow to pinpoint root causes and recommend pipeline remediations. The diagnosis architecture consists of three stages:

Signal collection collects events from multiple signal sources to build a full view of the dataset and pipeline health.
Alert diagnosis creates a structured alert context, classifies the alert, routes it to the appropriate diagnoser, and identifies the root cause using specialized diagnosis logic.
Diagnosis result persists the structured diagnosis output, including the identified root cause, affected dataset, and recommended fix or action.

For example, when a dataset fails, the workflow orchestrator notifies Hugo with a job failure event. Hugo then routes the alert to its internal diagnostic layer to check for conditions such as upstream database replica lag, storing both the diagnosis and recommended fix alongside the affected dataset.

Decoupling signal ingestion, diagnosis, and result management makes it straightforward to add new signal sources and specialized diagnosers. Immediate RCA removes the need for manual log inspection, which shortens remediation and feeds directly into automated resolution workflows.

Resolving DPIs: Auto-healing first, human judgment when needed

After triage and RCA, the final stage of the DPI lifecycle is resolution. The lifetime of a DPI is a proxy for data downtime: it begins when a contract breach is detected and ends when the affected dataset becomes healthy again. Reducing that window requires more than identifying the correct issue. It also depends on recovering safely and consistently from recurring failure modes.

Many incidents are routine and recoverable, such as transient compute interruptions, database connection timeouts, S3 throttling, or upstream pipelines that are delayed rather than permanently broken. Instead of relying on manual intervention for every incident, Hugo automates recovery for these well-understood failure patterns. Once the diagnosis workflow identifies the root cause, it produces a structured diagnosis result containing the affected dataset, the root cause, and the recommended resolution strategy. The auto-resolution workflow then consumes this result to execute the appropriate remediation automatically. Figure 3 shows Hugo’s auto-resolution architecture in two stages.

Figure 3. Hugo auto-resolution architecture.

Resolution execution applies the recommended resolution strategy, such as retrying a failed job, waiting for an upstream dependency, or executing a custom resolver. After the action completes, the system verifies both pipeline health and data correctness to confirm the issue has been fully resolved. If a failure cannot be resolved safely through automation, such as in cases of data corruption, invalid records, or application code defects, the workflow escalates the incident for human intervention.
Notification and audit records every resolution attempt and its outcome, while notifying the appropriate engineering teams. That record supports operational analysis, auditing, and later improvements to resolution policies.

For example, a dataset may miss its freshness Service Level Agreement (SLA) because the workflow orchestrator becomes temporarily unresponsive and fails to submit the scheduled ingestion job. The diagnosis workflow identifies the incident as a pipeline execution failure and recommends a retry strategy. Hugo automatically retries the job, verifies that the pipeline completes and data health is restored, then logs the recovery and notifies the responsible team. This end-to-end process, from incident detection to resolution, runs automatically without manual intervention.

Hugo closes the loop between detection, diagnosis, and recovery. Rather than stopping at identification, the platform turns diagnosis results into targeted remediation, so routine operational issues can be resolved automatically while preserving human oversight for complex or high-risk incidents. Separating diagnosis from execution also lets new diagnosis capabilities and resolution strategies evolve independently without changing the overall architecture.

The impact is already evident in production. 86.9% of DPI incidents were automatically resolved, significantly reducing manual operational effort. By automating routine recoveries, engineers spend less time performing repetitive operational tasks and more time building new platform capabilities, while overall data downtime is significantly reduced.

Conclusion

Certified data products still need to prove their reliability in production. Freshness delays, upstream failures, platform incidents, and data quality violations can all break consumer trust, even when an asset has already met certification standards.

Automated DPIs are the operating model for managing these failures. By turning contract breaches into structured production issues, the DPI lifecycle makes data reliability operational: triage separates real breaches from alert noise, diagnosis identifies the likely failure domain, ownership routing reduces handoffs, and resolution closes the loop through auto-healing or human intervention when needed.

The most important outcome is not simply that issues are detected faster. It is that data downtime becomes visible, measurable, and reducible. With every DPI tracked from detection to recovery, teams can understand where time is spent, which failure modes repeat, and where automation can safely reduce operational toil. To date, more than 95% of DPIs are raised automatically rather than by humans, with a mean time to resolve (MTTR) that is 6 times faster for automated DPIs than for manually raised ones.

For Grab, this shifts data reliability from reactive firefighting to a managed production workflow. Automated DPIs help keep trusted data products trustworthy after certification, so downstream teams can depend on them with greater confidence.

What’s next

Across the three-blog series, the story is how Grab turns data mesh from an operating principle into an artificial intelligence (AI)-ready foundation for the company.

Part I: Building trust through certification. Grab needed the Signals Marketplace because the business had scaled across mobility, deliveries, financial services, and many data-producing domains. The old model of relying on a central Data Engineering team could no longer keep up. Certification became the mechanism for making high-quality data products visible, reusable, and accountable. With clear ownership, data contracts, and measurable adoption, Grab moved more consumption toward trusted assets, reduced duplication, and created stronger incentives for teams to curate the data they publish.
Part II: The foundational tools behind certification. Trust becomes operational through platforms. Hubble covers discovery, lineage, ownership, and the certification engine. Genchi runs continuous data quality observability across freshness, completeness, schema, and business-rule checks. The Data Contract Registry formalizes producer-consumer expectations as versioned, enforceable contracts. Combined, these systems keep data certification an actively maintained standard rather than a static label.
Part III: Operationalizing data reliability with automated DPIs. Certification tells consumers which data products should be trusted; DPIs keep that trust true in production. Kinabalu evaluates contract breaches, deduplicates noisy alerts, assigns ownership, and tracks recovery. Data Health APIs make RCA portable across platforms, while Hugo’s diagnosis and auto-resolution patterns show how common failures can be remediated faster and with less operational toil. The result is a measurable reduction in time to resolve and a stronger feedback loop back into certification.

The bigger takeaway is that Grab’s data moat is not just the volume of data we have. It is the system that makes our data trustworthy, discoverable, reusable, and continuously reliable. This foundation is what lets us embrace the agentic world: AI agents can search certified assets, reason over contracts and lineage, trust quality signals, detect production issues, draft RCA, and eventually suggest or execute safe remediation. In that world, data reliability becomes a compounding advantage. The better our foundations are, the more confidently Grab can build agentic experiences on top of them.

We would like to thank all the data practitioners across Grab, including engineers and analysts to data scientists and product teams, who have invested in certification, contracts, and data quality to build a solid foundation for AI agents and AI-powered experiences. We are equally grateful for the unwavering sponsorship, strategic guidance, and hands-on support from our leadership (Mohan Krishnan and Nikhil Dwarakanath), without which this long-term data foundation initiative would not have been possible.

Join us

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Building Jarvis Pro: Route first, answer later

Fri, 21 Aug 2026 00:00:00 +0000

Introduction

The first Jarvis Pro prototype could produce answers that sounded right.

That was the problem.

One early answer looked polished: it named the merchant, summarized the week, and recommended pushing promotions before the next review. It was also wrong. The merchant’s order volume was down, but the sharper issue was operational: more outlets were paused and fulfilment had slipped. Sending more demand into that setup would have made the merchant look worse.

That failure changed how we judged the system. Fluent was not enough.

Jarvis Pro is the AI assistant we built for Grab account managers. Its job is to help them turn account data into better merchant conversations: what changed, why it changed, and what to do next. They rarely ask clean dashboard questions. They ask: “I am meeting this merchant tomorrow. What should I tell them?” or “Which accounts in my portfolio need attention this week?”

Those questions hide decisions: scope, access, business diagnosis, and metric definition. If the system gets those wrong, confidence becomes a liability.

So the core design became: route first, answer later.

In an internal offline evaluation (not a measure of production performance or business impact), routing matched the expected safe route for 99.4% of 351 realistic prompts drawn from labelled eval sets from the first half of 2026. In a focused portfolio and brand answer-quality suite, the average score moved from 78.5 to 91.0. These figures come from offline launch-readiness evaluation only; they are not business-impact proof.

Why dashboards were easier

A dashboard answers a bounded question:

“Show net sales for merchant X last week.”

An account review question has to diagnose the work to be done:

“This merchant softened this week. Should I push promos, ads, or operations fixes before the review?”

If outlets were paused, more traffic can backfire. If average order value fell, the next action may be menu or bundle design. If a dashboard, warehouse table, and local spreadsheet disagree, the first task is metric reconciliation, not recommendation.

We wanted Jarvis Pro to classify the task first, then load only the context needed for that task.

The route as a contract

The launch router was a prompted LLM classification step constrained by a routing guide: supported task types, allowed context, answer shapes, and conditions that should force a handoff or refusal.

A broad merchant review routes differently from an ads campaign recap or an out-of-scope request. A route is not just a label; it decides the structure, metrics, tool path, and guardrails.

Figure 1. Jarvis Pro routes before it answers. The route determines context, tools, and guardrails; evaluation checks each layer, not only the final response.

Here is a simplified example. The merchant and numbers are illustrative.

User prompt

I am seeing Merchant A soften this week. I need talking points for tomorrow’s review. Should I push promos, ads, or operations fixes?

Router decision

Ilustrative example only. No actual, live, or non-public data was used.

{
  "route": "brand_performance_action_plan",
  "scope": {"merchant": "Merchant A", "country": "Singapore"},
  "memory": "use current question context",
  "context": ["brand action plan", "approved metric catalog", "ads context"],
  "guardrails": ["no private competitor metrics", "no unsupported projection"]
}

Answer extract

Merchant A’s issue looks like order volume, not basket size. Net sales fell 7.8% week-on-week and orders fell 9.1%, while average order value increased 1.4%.

Start with operations before promotions. Temporary pause rate rose from 2.1% to 4.8%, and fulfilment fell from 94.6% to 91.8%. Extra demand may leak unless the merchant fixes availability first.

Talking point: “Before we add promo depth, let’s recover lost availability. If we bring pause rate back below 2.5%, we should have a cleaner read on whether the remaining gap is demand, menu, or ads.”

The route prevented three common mistakes: jumping straight to discounts, inventing competitor spend, and answering from a generic growth template.

Memory had to stay narrow

Here, “memory” means saved user context such as country, role, portfolio scope, and preferences. That context is useful when a user says “my portfolio”. It is risky when saved defaults override explicit context, missing identity becomes a guess, or a memory request crosses user boundaries.

We put memory before answer generation and kept the decision small: proceed, ask for onboarding information, prefer explicit context, use saved defaults, look up portfolio scope, or refuse. Backend permissions and row-level controls remain the authorization layer.

That extra checking costs time. Jarvis Pro does route classification, memory checking, context selection, warehouse or specialist tool calls, then generation. To keep the wait usable, we loaded route-specific context, ran memory before expensive retrieval, consolidated warehouse queries, capped tool calls, and returned unavailable cells as N/A instead of looping until the conversation stalled.

That tradeoff was deliberate: a slower first token was better than a fast unsafe recommendation.

Reconcile the metric before blaming the model

Even with good routing and memory, an assistant is only as good as the numbers it pulls.

When a user says “the number is wrong”, several failures can look identical: wrong source, different metric definitions, different entity mapping, or stale data. One reconciliation pass showed that what looked like model error was sometimes just a freshness mismatch between reporting surfaces.

We built regression checks that normalised source values and compared daily rows across approved metric paths. The point was not the row count. It was knowing whether to fix source selection, metric guidance, or the caveat shown to the account manager.

How we evaluated it

One aggregate score would have hidden the failures we cared about.

The routing set had 351 prompts labelled against the routing guide. Each prompt had an expected route family, meaning the broad business category, plus an expected route and any handoff or refusal. “Accepted route accuracy” meant the selected route was exact or semantically equivalent and safe. A wrong business family, missed handoff, or unsafe scope failed.

The answer-quality suite had 501 total cases scored on a 0-100 rubric covering template fit, metric use, diagnosis, next action quality, caveats, and guardrail compliance. Within that suite, the 150-case portfolio and brand subset improved from 78.5 to 91.0. A wrong merchant, wrong country, fabricated metric, unsupported projection, or private competitor detail could fail a case. User isolation was treated as a hard evaluation requirement. All scores were measured offline against fixed rubrics for launch readiness; they do not reflect production commercial outcomes.

That caught the answer we most wanted to avoid: plausible, polished, and operationally unsafe.

The lesson we would reuse

The final paragraph is too late to resolve ambiguity. Jarvis Pro has to earn the right to answer: route the task, check memory and access, load the right evidence, cap the tools, then judge failures at each layer.

Offline evals gave us confidence in system behaviour, not commercial uplift. Measuring that needs production telemetry: recommendations shown, actions taken, accounts affected, and outcomes.

The assistant should not merely sound like a great account manager. It should first prove it understands the account.

Join us

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Grab Bench: Evaluating AI on Grab-shaped production work

Wed, 12 Aug 2026 00:00:00 +0000

Introduction

What worried us wasn’t the hallucination, it was the subtle plausibility. Answers an engineer could easily read past and accept: a right-looking Structured Query Language (SQL) query, a plausible tool call, an innocent profile update, or a patch that satisfied the surface tests.

When we analyzed the row-level failures, a clear pattern emerged:

SQL generation: kept the query shape but changed the underlying metric.
Tool calling: selected the right tool family but drifted on parameters.
Profile updates: cited every event instead of only the evidence that supported the claim.
Coding agents: passed visible tests while missing a hidden stateful invariant.

Grab Bench bridges this exact gap. Grab Bench is a configurable eval (evaluation) harness for artificial intelligence (AI) systems on Grab-shaped work. It runs model providers through task plugins, records one row per case/model pair, and uses deterministic scorers or large language model (LLM) judges depending on the task. We treat the eval like software: version it, run baselines, keep score records, and make the failure modes visible enough for a team to debug.

This write-up focuses on the design choices behind that work.

The problem: plausible is not correct

Public leaderboards are still useful; we read them too. They just answer a different question. A product team needs to know whether a model can preserve a metric definition, obey an internal tool contract, stay cautious with weak evidence, or make a code change without breaking behaviour hidden from the prompt.

The hard part is that real examples are rarely reusable as-is. Production traces, schemas, user records, and internal workflows need protection. So the benchmark has to preserve the shape of the work without depending on the work itself.

That constraint shaped Grab Bench from the beginning. Some surfaces stay internal. Others use synthetic or redacted cases. Either way, the case has to keep the thing that makes the work hard: metric faithfulness, tool-parameter discipline, evidence grounding, safety boundaries, or repository-level behaviour.

What Grab Bench runs

The harness is deliberately ordinary. A YAML configuration defines providers, models, task settings, sampling, concurrency, judge settings, and output paths. The runner loads rows, checks whether each model supports the required modality and application programming interface (API) family, calls the task plugin, and writes row-level records plus model summaries for dashboards.

The unusual part is that each task owns its contract:

Query generation cares about preserving metric and schema intent.
Tool use compares canonical tool names and parameters.
Multimodal pair matching scores constrained yes/no decisions.
Passenger-profile reasoning checks grounded claims, evidence, uncertainty, action quality, and safety.
Agentic coding runs visible and hidden workspace tests, plus hard-failure and anti-gaming checks.

This is why the row record matters. A leaderboard can tell us that one model is ahead. It cannot tell us whether the loss came from a fabricated evidence identifier (ID), a weak action, a hidden invariant, latency, cost, or a genuine capability gap.

Each run also keeps the unglamorous fields that make reruns possible: token use, latency, judge latency where applicable, skip reasons, resolved configurations, and dashboard-ready summaries. Without those fields, the next comparison starts from memory instead of evidence.

Figure 1 is deliberately boring: add a plugin; providers, records, and dashboards stay shared.

Figure 1. Grab Bench keeps execution shared while task plugins own request shaping, parsing, and scoring.

Design choice 1: make the cases safe, not generic

A useful eval case should feel familiar to the people who own the system. It should include distractors, stale context, ambiguous evidence, and the kind of boundary conditions that make production work tricky.

In passenger-profile reasoning, each case is a synthetic evidence ledger: rides, food, support, app events, saved places, promotions, and noise. All cases use synthetic data with no live user records. The model must return strict JavaScript Object Notation (JSON). Claims must come from an ontology; values must be valid for that claim; evidence IDs must exist; weak or sensitive inferences should be suppressed, not laundered into confident prose.

The scorer is deliberately mechanical where it can be: schema validity, claim correctness, evidence faithfulness, confidence calibration, action quality, and safety. It distinguishes required claims from acceptable auxiliary claims and forbidden claims, so a model can get credit for useful extra evidence without getting a pass on unsafe or unsupported inferences.

A simplified case might ask whether a passenger has a stable weekday commute:

The evidence ledger contains repeated morning rides from a home-like saved place to an office-like area, plus unrelated food orders and stale support contacts.
A good answer returns a claim such as weekday_commute = likely_home_to_office_commute, cites only the commute evidence IDs, and keeps confidence within the allowed range.
The scorer checks that the claim and value exist in the ontology, that every cited evidence ID exists, and that the cited rows actually support the claim.
If the model cites every event, fabricates an ID, adds a dietary-preference claim from one old order, or recommends an unsafe action, the row gets explicit failure tags or a score cap.
The result is still a number, but the row also says what failed, which is what an engineer needs to fix the prompt, scorer, data, or model choice.

For agentic coding, the repository is synthetic too, but it asks for a real-shaped change: default ride insurance across backend services, API compatibility, mobile helpers, analytics events, rollout controls, migration compatibility, idempotency, concurrency, and cancellation lifecycle. A patch that only satisfies visible tests is not enough.

The safety comes from using synthetic data. The pressure comes from keeping the real contract intact.

Design choice 2: score contracts, not confidence

LLM judges are useful for open-ended tasks such as SQL, where correctness can depend on business intent and query shape. But for many surfaces, the benchmark should not ask another model whether an answer seems good.

Grab Bench uses deterministic scoring when the task contract allows it. Passenger-profile reasoning scores ontology values and evidence IDs. Tool use compares canonical tool names and parameters. Multimodal pair matching scores exact labels. Agentic coding scores visible and hidden tests, maintainability, efficiency, and hard-failure gates.

The audit trail is the point. A fluent answer should not get credit for missing the contract. The row needs to say whether the model misunderstood the task, ignored a constraint, exceeded a budget, or produced something plausible but unsupported.

Design choice 3: make shortcuts visible

Benchmarks get weaker when shortcuts work. The scorer has to make those shortcuts visible.

In the reasoning benchmark, fabricated evidence IDs, unsupported claims, broad cite-everything behaviour, unsafe actions, and forbidden sensitive claims trigger penalties or caps. In the coding benchmark, hidden-test tampering, network-access patterns, oversized patches, case-id leakage, visible-only overfit, and implausible difficulty curves are blocked or investigated.

Baselines make that visible. Empty output, schema-only output, cite-all-evidence output, unsafe-sensitive output, no-op coding agents, and reference agents are not busywork; they are checks on the scorer. If a shortcut baseline can pass, the benchmark is not ready.

This is not about assuming bad faith. It is about refusing to reward behaviour that would fail the moment it left the harness. A profile update that cites every event has not shown evidence discipline. A SQL answer that changes the metric has not preserved intent. A coding agent that passes only visible tests has not earned trust.

Internal reproducibility and hidden pressure

The package has to be inspectable and hard to overfit at the same time. Engineers need to rerun the harness, read score records, and understand failures. Certification still needs unseen cases, or we end up optimising prompts against the examples everyone can see.

Grab Bench handles this with a split between teaching artifacts and certification artifacts. Teaching artifacts explain the task contract, scorer, examples, baselines, and canaries. Certification artifacts keep hidden splits, seeds, raw outputs, and full comparison evidence behind the right access boundaries.

One dataset cannot do all of that honestly. Shared examples are for learning the method. Hidden cases are for checking generalisation. Row-level outputs are for debugging. Aggregates are for comparison.

Before a comparison run is trusted, the package also has to pass gates: oracle or reference solutions behave as expected, weak baselines fail, redaction passes where applicable, score spread remains useful, and canaries catch harness regressions. Here, a canary is a deliberately simple or malformed case with a known expected result, such as a no-evidence profile update that must be rejected.

Figure 2. Teaching artifacts and certification artifacts share the same harness but need different access boundaries.

What we learned

The most useful Grab Bench output is often not the leaderboard. It is the failure taxonomy.

We saw that more reasoning is not a universal good. It can help planning-heavy tool use and hurt tasks that need literal schema discipline. Evidence selection is also part of reasoning: citing everything is not safer when only a few rows are direct support. For agentic coding, category-level results matter because a model can handle API contracts while missing stateful invariants.

We also learned not to treat prompt or model settings as universal. A setting that helps one task can make another worse. That pushed us toward task-level reports, not one global recommendation, and toward comparisons that show failure tags alongside scores.

Most of all, evals need hygiene: versions, baselines, gates, dashboards, and scope limits.

One limit is worth stating plainly: synthetic evals do not prove production uplift. They tell us whether a model respects the contract under controlled pressure. Live retrieval quality, user impact, and rollout decisions still need separate evidence.

What comes next

Next, we want the benchmark surfaces to look more like pipelines. Instead of scoring only the final answer, we want to separate retrieval, reasoning, action selection, latency, cost, and safety where the task supports it.

We also want packages to be easier for other teams to reuse. A good eval should not depend on one team remembering how it works; it should be documented, versioned, and safe enough for others to run.

Grab Bench is our attempt to make AI evaluation boring in the useful way: configuration in, rows out, failures explained, shortcuts caught. The question is not which model wins in the abstract. It is which model is ready for this work, under these constraints, with these failure modes.

The test I would apply to any eval is simple. If a cite-everything baseline can pass, the eval is not measuring evidence discipline. If a visible-test-only agent can pass, it is not measuring production behaviour. The useful conversation starts when the benchmark can show the shortcut and make it fail.

Join us

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

How AI is transforming analytics at Grab

Sat, 01 Aug 2026 00:23:00 +0000

Introduction

At Grab, analytics sits close to almost every decision that matters. Our north star is the democratisation of intelligence, ensuring that anyone making a business call has immediate access to trustworthy answers.

Over the last two years, model capability has crossed a threshold enabling this shift. Agents now do in minutes what used to take a week: preparing the data, writing queries, running deep analysis and developing insights for business opportunities, designing experiments and interpreting the results, drafting the commentary that follows, and more. Our throughput is no longer rate-limited by how fast an individual can write code, build a deck, or run a deep-dive. It is rate-limited by how fast we can frame the right problem, judge the right answer, and influence the right decision.

As autonomy climbs, an analyst’s impact moves from producing the artefact to owning the question and the call behind it, and the role evolves to become part builder, part advisor, part strategist, owning the loop rather than running it. That unlocks two things at once: work we already do, faster and at lower marginal cost, and work we could never staff before, sitting beside every product manager, business owner, and operator at the moment they decide.

The ladder

We were heavily inspired by Dan Shapiro’s framing of five levels for AI coding. We use a similar ladder that defines how much of the loop an agent should own and where human judgement stays for every analytics loop.

One distinction runs across every level: who owns the loop, and where human judgement is required.

Level	What	Human role	Agent role
L2	AI-Assisted	Owns and executes every step; uses AI to draft, suggest, summarise	Drafts SQL, suggests a visualisation
L3	Human plans, agent owns steps, human reviews	Frames the question, picks the metric, the segment, and the comparison frame, reviews evidence, owns the recommendation	Discovers data, writes and runs the query, sanity checks, drafts the write-up, flags caveats
L4	Agent plans, agent owns workflows, human reviews	Sets intent and guardrails; reviews at gates (anomaly, novel scope, sensitive cut); owns the stakeholder relationship and sign-off	Orchestrates discovery through query, analysis, validation, narrative and publish; runs validation, escalates exceptions
L5	End-to-end autonomous	Sets objectives, quality bars, risk thresholds, escalation rules; reviews exceptions only	Detects anomalies and opportunities, runs the loop, surfaces insight, evolves the metric layer, context and skills

Human judgement remains at every level, and autonomy never removes accountability. Humans own problem framing, canonical metric definitions, the causal story behind a move, business-case assumptions, the go/no-go, and the stakeholder relationship. A higher level means more of the mechanical loop sits with the agent and more human attention concentrates on the ambiguous, high-stakes work.

Making the climb

Five core capabilities move a workflow up the ladder. They also gate the climb in order: L3 needs execution and certified context, L4 needs gates and agentic review good enough that reviewing only at gates is honest, L5 needs a learning loop that closes.

Execution: A stack that runs the loop end to end rather than a notebook/workflow a human drives.
Knowledge: Metrics certified at the right grain, discoverable in our catalogue, grounded in context an agent can read. Ambiguous definitions cause most analytics slop.
Control: Repeatable expectations become mechanical checks, while human review handles what a rule cannot.
Review and governance: Agents check their own output against the gates and escalate on defined triggers. We govern definitions, targets, risk and exceptions.
Learning: When an agent fails the same way twice, we encode the fix into context documents, golden datasets, evals and gates.

What this looks like in practice

What follows is a set of explorations from the last two years. Some run in production today, while others are still teaching us where the limits are.

Loops that run end to end

Spartan is our end-to-end agentic analytics workflow, embedded across surface areas (like Slack), and most of its usage comes from people who are not analysts. On any given day, the Slack channel enables a range of analytics actions: from ads salespeople pulling spend breakdowns for a named merchant, to campaign managers sizing audiences for a target segment, and country teams asking why a number moved week on week. All of it in plain business language.

Figure 1. Index architecture across our knowledge base.

Two requests from July best demonstrate how it works. A commercial manager asked why revenue fell in the Philippines mid-market segment in the last two weeks of June. Separately, a product manager asked for a summary of a frequency-cap experiment on the ads surface. Both arrived as natural language questions in Slack and took entirely different routes through the system.

The router reads the first as a root-cause question and sends it down the diagnostic path. It identifies the best analysis framework for ads revenue, which is codified knowledge of how the metrics in that domain relate to each other, which dimensions are worth decomposing, and what counts as a meaningful move. Then it works through segment, market and campaign type against certified metrics to isolate what changed. The second question never touches the data lake. The router reads it as an experiment question, selects the experiment skill, pulls the pre-computed scorecard and the test’s own metadata from our experiment platform, and summarises the read rather than recomputing it. This is powered through 50+ skills and 120+ analysis frameworks that sit behind that routing decision. Underlying that is an index that tells the agent what to search, context that tells it how to query, and a framework that tells it how to think. Because the frameworks are shared rather than living in an analyst’s head, the interpretation compounds instead of being re-derived every time someone asks.

The second example of such a loop is Scarlet, which powers near-self-healing pipelines (L4). When a pipeline fails, an agent runs the root-cause analysis, triages, and then either fixes it or hands it to the team that owns the upstream problem. It escalates when the failure sits outside its documented runbooks or the pre-defined gates fire.

Figure 2. Scarlet in action on Slack.

Context that maintains itself

Context sets an agent’s ceiling. An agent that does not know a metric’s grain, its exclusions, and its caveats will guess and confidently produce wrong outputs at speed and at scale.

Realising the criticality of this, we have dedicated platform investment, as well as dedicated functional bandwidth to generate context docs.

Figure 3. ContextIQ.

Context goes out of date faster than anyone maintains it by hand, so we build the maintenance into our workflows. We built ContextIQ, and its Context Lifecycle Manager, to treat context as something with a lifecycle rather than a document somebody wrote once. A newer skill of ours reads an instrumentation spec alongside the existing context, proposes the SQL changes that follow from it, and updates the context document in the same pass. We work the problem from the other direction too. When we categorise an agent failure in production, we patch the context document behind it.

Two analysts recently used our internal agents to understand how the packaging fee is stored as a configuration. Having found the answer, the agent opened a merge request that committed both a certified-context table reference and a golden-dataset test case, so the next agent to ask the same question would find the answer already documented and the check already in place. One of the analysts spotted a false positive in it. The agent corrected itself and reopened the merge request. That is the learning capability working as designed, and it happened without anyone setting out to demonstrate it.

Loops that run unattended

The step from L3 to L4 is mostly the step from interactive to scheduled, and it is where we go down the path of autonomous execution, because no human is watching at the moment the work runs.

We have built root-cause analysis (RCA) as a platform capability, and it powers our automated metric and OKR commentaries, which are published through automated agents (configurable cadence). It judges whether a move is meaningful against standard deviation over six months and year on year, walks the metric tree to find which country, segment or funnel stage carried it, and correlates the operational metrics that moved alongside. Importantly, it also scans internal context for what teams changed on the ground, such as delivery fee and incentive moves, merchant visibility shifts, and experiments shipped in the same period. It also compares the movement against the same period in the previous year, which separates a seasonal effect from a real one and enables it to report a Songkran (Thai New Year) dip as amplified rather than merely expected. All of it is grounded in our own context documents, which keeps the narrative about the business rather than generic model output. The analytics owner is tagged on every report, and edits sync back so corrections land in the system.

Figure 4. OKR commentary shared through RCA agent.

Analysts as builders

The clearest evidence that our centre of gravity has moved is BriX, an internal portal we built and run ourselves.

Figure 5. Home page of BriX.

The premise is to configure once, host everywhere. We configure a system prompt, a set of context files, a model, the MCP connections and an interface once, and what comes out is a purpose-built analytics surface for a particular team or job. Each one inherits certified data, permissions and reusable agent skills rather than being wired up from scratch, and it runs wherever the work already happens: in Slack, invoked from inside an IDE, or on a schedule with nobody watching. We have grown usage more than 10x since September 2025, with strong retention, and every function at Grab now has users on it. Our aim is to put L3 workflows in the hands of people who are not advanced users.

We run it without a product manager, a technical programme manager or a designer. Our data engineers own the product, the platform, the support queue and the eval loop, with Claude Design doing the interface work and the builders triaging their own bugs. In the first half of this year, they shipped 31 production deployments, 283 merge requests and 60 features.

Three of our apps show the range:

Insights Lab is the general-purpose surface: a stakeholder asks for a metric, a breakdown or a root-cause in natural language, and the agent loads a specialist skill and answers off certified metrics rather than from memory.
We built Funnelytics to enable easy understanding of our consumer funnels. A funnel question used to mean an analyst writing the query and then assembling the view in Tableau or Power BI, and doing it again the next time someone wanted a slightly different path through the app. Now a stakeholder picks the events they care about and Funnelytics queries the raw event stream, builds the Sankey and funnel views, and writes the summary. If they cannot find the right instrumentation, which happens often on products still being redesigned, a live debugger lets them tap through the app on their own phone and watch the events fire.
Monte (like Monte Carlo) runs simulations to put a probability on a business outcome. You give each uncertain input a range rather than a single value, and it runs ten thousand scenarios to return the likelihood of hitting a target.

Figure 6. Interface of Insights Lab and Funnelytics.

Outside the portal, the same instinct shows up in smaller ways. Our analysts have been building more bespoke tools that enable better workflows for themselves and stakeholders.

The path forward

In February, 44% of the tickets our analysts closed were mechanical (data preparation, alerting, reporting); by June, that share had fallen to 30%. That capacity was redirected to other higher-leverage work, such as building new workflows to enable stakeholder self-serve, as well as more time spent on generating deeper insights for business opportunities.

Figure 7. Comparison of percentage of tickets closed in Q1 vs Q2.

Importantly, our cycle times reduced by ~33%.

Figure 8. Comparison of time taken to resolve a ticket in Q1 vs Q2.

The sharpest version of this sits in a Slack channel where self-serve agents are enabled. In March, an analyst had to step into half of them; by May, it was under a quarter. The share answered with no human involvement rose from 53% to 67% for metric questions, 63% to 90% for data pulls, and 50% to 81% for SQL requests. Just under three in four of the threads were started by someone outside the analytics team, and 85% of them got a first response inside a minute. Nearly every thread is logged as a ticket on the team’s board, and roughly two-thirds of the data exploration tickets on that board now arrive through the channel rather than through an analyst, and are solved by our data agents. For the ~230 tickets that arrived via the channel, if we apply a conservative assumption of 1–2 days per ticket, that is 230 to 470 business days of stakeholder asks that would have been in the backlog.

None of these arrived on a roadmap. They came from analysts who saw a loop worth automating and built it, which is why the climb is uneven. These have been strong proof points for us to believe our investments are working, and many of these workflows are starting to operate at scale. We will keep experimenting and iterating, and we expect to get a fair amount of it wrong. An analyst who owns a loop, sets its quality bar and reviews its exceptions is doing a different job from one who answers questions. Most of our team is somewhere in that transition today, and we truly believe it is changing what analytics is at Grab.

Join us

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Special thanks to the contributors:
A3T Braintrust - Maanas Prabhakar, Sneh Agarwal, Nikhil Dwarakanath, Chun Kai Phang, Raghav Garg, Avishek Ghosh, Ankush Khandelwal, Nishant Gupta, Harshita Shah, Lara Pureum Yim, Alex Dobrev
Spar/tan 3.0 - Lara Pureum Yim, Kenrick Tan, Tushar Bose, Yong Kai Lim, Dandy Kennedy, Glen Low, Zian Li
BriX - Sneh Agarwal, Rishi Raj, Kartik Srinivasan, Aseem Kumar, Eddy Lim, Wen Zhong Tan
Funnelytics - Chengxuan Feng, Wingyiu Ng, Zoey Zhou
ContextIQ - Defi Mediana, Karen Koh, Anitha Sreedhar, Wanyi Zhu
Butler Bot - Adhika Putra
Monte - Nikhil Dwarakanath

Crowdsourced taxonomy verification: A feedback-driven framework for refining knowledge graph relationships via online search interactions

Thu, 30 Jul 2026 00:00:00 +0000

Introduction

The efficacy of semantic search relies on the accuracy of the underlying Knowledge Graph (KG). In high-velocity domains like on-demand food delivery or e-commerce, the catalog of entities like dishes, products, and merchants changes rapidly.

Current methods for KG construction and maintenance face three critical challenges:

Inaccuracy and hallucination from Large Language Models (LLMs): Automated models often infer relationships based on statistical text co-occurrence rather than semantic reality. For instance, an LLM might incorrectly classify “Pho” as a child of “Italian Noodle Soup” due to linguistic similarity, leading to irrelevant search results.
Scalability limits of manual verification: Traditional verification relies on human annotators or domain experts. This approach is slow, expensive, and unable to keep pace with dynamic catalogs containing millions of entities. For example, daily changes in restaurant menus or grocery stock keeping units (SKUs).
Error propagation in ranking: Inaccurate graph edges propagate errors downstream. If a parent-child relationship is wrong, query expansion algorithms will retrieve irrelevant items, directly degrading Click-Through Rate (CTR) and user trust.

We introduce a feedback-driven verification engine that operationalizes the search interface as a validation environment. Key contributions include:

User feedback-driven verification: The system treats unverified graph edges as hypotheses. Instead of accepting them as truth, it tests them against live traffic by injecting them into search suggestions and measuring user engagement.
Hierarchical relationship refinement: Unlike systems that only validate entities (nodes), this framework validates structural links (edges). It confirms whether entity A is truly a parent, child, or sibling of entity B, ensuring structural integrity.
Adaptive exploration: The system employs a greedy exploration policy. It intelligently balances exploitation by showing known good results with exploration through injecting unverified candidates to gather data without degrading the user experience.

Background

Automated KG construction using LLMs and unstructured content extraction can scale quickly across large, dynamic catalogs. However, relationships inferred from text co-occurrence or vector similarity do not always reflect semantic reality. Manual verification by domain experts remains accurate but does not scale to millions of entities that change daily.

When inaccurate edges enter the graph, ranking and query expansion systems propagate those errors to users. Incorrect parent-child or sibling links lead to irrelevant search results, reduced CTR, and lower user trust. An additional solution is required that can validate graph structure continuously, at scale, without relying solely on manual curation.

Solution

The overall workflow of this invention is shown in the following figure. The details of each step are explained in this section.

Figure 1. The system architecture.

The proposed framework functions as a closed-loop validation ecosystem. It is composed of four integrated modules designed to continuously cycle data from the KG to the user interface and back, using real-world interactions to separate semantic truth from artificial intelligence (AI) hallucinations.

The verification process follows a continuous, iterative loop that cycles data from the backend graph to the frontend user interface and back. This four-step procedure operationalizes the human-in-the-loop validation mechanism:

Hypothesis generation
Candidate injection
Signal aggregation and scoring
Graph update logic

Architecture details

KG core

The central repository acts as the source of truth, storing entities such as dishes, products, or merchants, and the connections between them. To manage the verification process, the system introduces a specialized metadata layer that classifies every connection (or edge) into one of two distinct states:

Verified edges: These are established relationships that have been validated either by high historical traffic or human confirmation. They represent the safe structure of the graph. For example, “Sushi” is definitely a child of “Japanese Cuisine”, and is used to power standard search results.
Candidate edges: These are probabilistic, unverified relationships generated by automated LLMs or content scrapers. They are treated as hypotheses waiting to be proven. For example, if an LLM ingests a blog post and predicts that “Pho” is related to “Italian Noodle Soup,” this link is stored as a candidate edge, invisible to the main search algorithm until validated.

Search and injection module

This module sits between the KG and the user, intercepting the query execution pipeline. Unlike standard ranking algorithms, which strictly optimize for relevance by showing only the best results, the injection engine employs a balanced strategy known as exploration vs. exploitation.

The injection mechanism: When a user performs a search, the system retrieves a list of high-confidence results (exploitation). Simultaneously, it deliberately retrieves a small subset of candidate edges related to the query. It injects these unverified candidates into specific, lower-risk slots within the user interface, such as the third or fourth position in a related searches chip carousel.
Risk management: To prevent user frustration, the system limits the number of candidates shown per session. This ensures that the user is primarily served helpful, verified content, while still providing enough data points to test new hypotheses.

Behavior tracking module

To accurately measure whether a candidate relationship is valid, the system tracks user micro-interactions with high granularity. It captures not just the final click, but the precise context in which the interaction occurred to determine semantic intent.

Contextual anchoring: The system logs the specific search term, also known as the anchor, used by the user. A click on “Pho” is only counted as a vote for the relationship if the user was searching for “Noodle Soup” at the time.
Signal classification: Signals are assessed in aggregate to estimate the relevance of a candidate relationship. Higher-intent engagement contributes stronger positive evidence, lighter exploratory behavior contributes weaker positive evidence, and lack of engagement or explicit negative actions contributes negative evidence.

This is an offline processing unit that acts as the final judge. It aggregates thousands of individual user signals to update the topology of the KG.

Relevance scoring: Instead of complex formulas, the engine calculates a simple confidence ratio. It looks at the total number of times a candidate was shown versus the number of positive interactions it received.

Graph topology updates:

Promotion (verify): If the confidence ratio exceeds a verification threshold. For example, if the candidate performs as well as known good items, the edge is upgraded from candidate to verified. It becomes a permanent part of the graph and is shown to all users.
Demotion (prune): If the candidate consistently fails to garner engagement or receives negative signals, it falls below a pruning threshold. The system automatically deletes this edge, effectively correcting the AI’s hallucination and cleaning the dataset.

Implementation

Hypothesis generation

The process begins by identifying a target subject, referred to as the anchor entity. For example, the specific dish “Pho”. The system queries the KG to retrieve a set of potential relationships. This retrieval includes both verified neighbors, where relationships are already confirmed by experts, and candidate neighbors, where the relationships are predicted by AI models but not yet proven.

Candidate injection

Once a hypothesis is selected, the system exposes it to real users to gather evidence. When a user actively searches for the anchor entity, the system dynamically injects the candidate neighbor into the search results.

User interface (UI) implementation: The candidate is presented alongside verified items, typically in a related categories carousel or a refine search chip list. This reflects standard relevance experimentation in search, with safeguards to ensure the experience remains controlled and measurable.
Exposure logging: The system logs an impression event specifically linking the anchor to the candidate. This record serves as the baseline, documenting that the user saw the relationship, which is essential for calculating future engagement rates.

Signal aggregation and scoring

Instead of using a raw count of clicks, the system calculates a sophisticated relationship confidence score by aggregating user interactions over time. This scoring model uses a weighted tier system to distinguish between casual interest and strong intent.

Weighted interaction logic: The system assigns a higher value to actions that require more effort or commitment. For example, a “Purchase” or “Add-to-Cart” action is weighted significantly heavier than a simple click, as it indicates a strong validation of the relationship. Conversely, scrolling past the item quickly or skipping is treated as a negative signal.
Normalization: To ensure fairness, the total weighted score is normalized against the total number of times the candidate was shown. This prevents niche items with low total traffic but high accuracy from being unfairly penalized.

Graph update logic

Periodically, the verification engine evaluates the confidence score against predefined benchmarks to update the KG’s topology. This is a binary decision process:

Validation (cementing the edge): If the accumulated confidence score exceeds a strict validation threshold, the system concludes that the relationship is genuine. The status of the edge is updated from candidate to verified. This permanently adds the relationship to the graph, ensuring it appears in future standard searches without the need for further testing.
Rejection (pruning the edge): Conversely, if the score falls below a rejection threshold, indicating that users consistently ignore or reject the suggestion, the system concludes the relationship is an AI hallucination. The edge is severed or removed from the graph. This pruning action cleans the dataset, preventing the system from making the same bad recommendation again.

To demonstrate the framework, consider a validation scenario in food delivery taxonomy. An LLM-based ingestion pipeline flags a candidate parent-child link
Noodle Soup → Dry Mee Pok and stores it as an unverified candidate edge in the KG, ready for live validation.

User-triggered validation:

When a user searches for “Noodle Soup,” the search module injects the candidate alongside verified results. For example, in a “Refine by Dish” filter carousel, and logs an impression linking the anchor query to the candidate.

Outcome collection:

User interactions like clicks, dwell time, scroll behavior, and conversions are captured and weighted over the validation window. The verification engine aggregates these signals and updates the graph: relationships that meet the validation threshold are promoted to verified status; those that fail are pruned or re-mapped to a more appropriate parent node.

Impact

By injecting unverified candidate edges into live search results and recommendation interfaces via a multi-armed bandit (MAB) exploration strategy, the system leverages implicit user feedback to validate semantic truth. This dynamic, human-in-the-loop mechanism effectively prunes erroneous connections and reinforces accurate taxonomies without the need for manual curation, significantly enhancing search relevance in dynamic domains such as food delivery and retail.

The case study demonstrates how the framework validates candidate relationships through live user traffic, collecting interaction signals and updating the graph without manual curation.

Learnings and conclusion

The feedback-driven verification engine operationalizes the search interface as a validation environment for KG relationships. By classifying edges as verified or candidate, injecting candidates through an exploration vs. exploitation strategy, and aggregating weighted user signals, the system promotes accurate relationships and prunes AI hallucinations at scale.

Unlike approaches that validate only entities, this framework validates structural links, confirming whether entity A is truly a parent, child, or sibling of entity B. The food delivery case study shows how a user-triggered search can initiate validation and outcome collection at scale, without manual intervention.

What’s next

Hierarchical confidence tiers

To safely graduate new connections into the production graph, we are introducing a dual-measurement trust system that requires both volume and variety before a new connection goes live: support mass (product hits, graph depth, recency) and corroboration (unique sessions, anonymous cohorts, and temporal spread). Connections must climb a strict state machine: proposed → shadow eligible → canary eligible → production, advancing only when both metrics meet progressively higher thresholds; if a snapshot causes metrics to fall below a tier’s floor, the connection is automatically demoted.

Adversarial and spam resistance

To prevent bad actors, bots, or highly repetitive users from manipulating the search graph, we are building a multi-layered defense system. We enforce per-merchant rate limits and anti‑abuse controls: hourly caps per session/device, exponential backoff for rapidly repeated actions, and a short (few‑hour) freeze of promotions from any user cohort after declines or “irrelevant” signals. For bot and Sybil attack defense, traffic flagged by abuse systems is excluded from trust calculations (but logged for analysis); votes must come from diverse network subnets or cohort buckets, and each bucket is subject to a daily contribution cap.

Join us

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

Agent platform (Part 1): How we help Grab build and run AI agents at scale

Fri, 24 Jul 2026 00:00:10 +0000

Part 1: From one support bot to a framework

At Grab, AI agents have evolved from interesting team prototypes into production services used every day by millions of merchants, drivers, and consumers. Today, more than 500 services run on our internal agent framework, over 50 Model Context Protocol (MCP) servers are registered on our remote MCP framework, and a single Large Language Model (LLM) gateway fronts every model call across the company, handling billions of tokens each month.

None of this was designed up front. It began as the plumbing behind one internal support bot, which then expanded because the same problems kept resurfacing for every team trying to ship an agent. This series tells the story of what the platform eventually became. This Part 1 of the blog focuses on the beginning: the architecture of our AI support bot, the specific pain points we hit while scaling and iterating on it, and how each of those failures became a core building block in the framework we now call LLM-Kit.

The bot that started it

Imagine you have a question for the Technical Infrastructure (Tech Infra) team - the engineers who run the cloud platforms, databases, developer tooling, and AI infrastructure behind Grab’s ecosystem. Instead of immediately paging an on-call engineer, a bot first triages the request, checks the team’s documentation, runbooks, and past Slack threads, and tries to answer directly in the thread. If it still cannot resolve the issue, it routes the ticket to the right human, with the relevant context already attached.

That is what we built with the Tech Infra Support Bot.

In the first half of 2023, Tech Infra handled thousands of support tickets, many of them repeated questions that had already been answered somewhere internally. Before LLMs, the bot’s role was mainly operational; performing tasks like helping track acknowledgments and response times for on-call engineers. With the arrival of GPT-4-32k, we evolved it into a GPT-powered Level-0 support layer that could answer documented questions before a human needed to be paged.

The first production version was a Go service organized around two planes:

A reasoning plane. At Level-0, it was a single-agent loop. It takes the user’s question, decides which tools to call, executes those calls, feeds the results back into the prompt, and returns an answer. The default model at the time was gpt-4.1; today, we have evolved to the latest reasoning models.
A tool plane. The tools provided the bot’s core working context. Retrieval flowed through Glean, which covered Confluence, TechDocs, internal drives, and Jira. Other tools handled log search through Kibana, GitLab runbook and file access, Slack conversation search, and a small set of Hypertext Transfer Protocol (HTTP) plugins. In the first version, tools and prompts were defined in per-channel JavaScript Object Notation (JSON) configs and resolved at request time. As models became more capable, we later standardized the tool set across channels.

A trimmed version of that tool config looked like this:

"agent_plugins": [
  {"name": "glean_search",        "type": "common", "metadata": {"wiki_space_collection": ["..."]}},
  {"name": "runbook_search",      "type": "common"},
  {"name": "gitlab_runbook_reader","type": "common"},
  {"name": "gitlab_read_file","type": "common"},
  {"name": "kibana_log_search",   "type": "common", "metadata": {"index": "k8s*"}},
  {"name": "slack_conversation_tool", "type": "common"}
]

It worked, but it taught us, the hard way, why a demo agent is not a production agent.

What it takes to scale and improve quickly

As we worked on improving the agent, we kept running into the same kinds of friction. Over time, those pain points formed clear patterns, and they were the same ones we saw other teams run into as well.

Vibe check is not an evaluation strategy. The bot had a base prompt, and each Slack channel could configure its own prompt, tools, and documentation filters. But the workflow was essentially: configure it, ship it, and hope it reduced toil. There were no real evaluations, just optimism that it would work.
Fast model and provider switching is essential. The AI landscape moves incredibly fast: a new state-of-the-art (SOTA) model appears on Tuesday, and a highly efficient open-source alternative shows up on Thursday. Switching providers should not feel like open-heart surgery. A unified Software Development Kit (SDK) and an LLM API gateway remove the need to refactor payload schemas, rewrite error handling, or integrate each provider from scratch. If moving from OpenAI to Anthropic, or routing to an open-source model endpoint, takes more than a few config changes, technical debt is already slowing you down.
Observability cannot be an afterthought. When an answer was wrong, figuring out “why” meant grepping logs across three separate systems: the agent workflow, the tool calls, and the model call. There was no shared trace tying them together. That level of friction is survivable for an internal tool; it is unacceptable for a customer-facing agent.
Everything around the agent took longer than the agent itself. Auth (OIDC), secrets management (Vault), per-environment config, vector database integration, LLM tracing, health probes, and metrics were not agent-specific problems. However, they all had to be solved before anything could be shipped. The reasoning loop took a whole afternoon. The production wrapper took two weeks.

The pattern was clear: the hard part of building an agent was not the agent itself, but everything around it that had to be in place before it could safely run in front of users. So we began pulling those shared components out of the bot and consolidating them into a unified framework.

Extracting the framework: LLM-Kit

LLM-Kit emerged when we stopped solving these problems service by service and started solving them once, centrally. It is intentionally not a new agent abstraction or a Domain-Specific Language (DSL). Instead, it is a curated set of integrations and scaffolding built around Grab’s existing infrastructure, pipelines, secret management, and observability. Just as importantly, we chose to build a framework rather than a heavy centralized platform. In a space evolving this quickly, a platform would have locked teams into rigid assumptions that would soon become outdated. A framework let us meet developers where they already were: standardizing the plumbing while preserving the freedom to iterate quickly. Looking back, that was the right first choice. Each part of LLM-Kit is a direct response to one of the failures described above.

We first wrote about LLM-Kit’s structure and code architecture in a 2024 blog post. Two years and a few hundred agents later, the overall shape is still recognizable, but almost every underlying layer has changed. Poetry was replaced by uv; we standardized on the OpenTelemetry stack; LangChain evolved into LangGraph and Deep Agents; and some tools moved onto our MCP framework.

It starts with a template. The entry point is a user interface (UI) form. An engineer fills in an application name and a few details, and gets back a GitLab repository with the production wrapper already assembled. Under the hood the template stamps out a full FastAPI service:

/
├── app/
│   ├── server.py              # FastAPI app factory: mounts routes + middleware, boots OTel + statsd
│   ├── agents/
│   │   ├── simple_react_agent.py   # a single-agent LangGraph ReAct loop (agent <-> tools)
│   │   ├── mcp_react_agent.py      # the same loop, but tools are pulled from remote MCP servers
│   │   └── simple_react_agent.png  # auto-exported graph diagram (generated in dev)
│   ├── routes/
│   │   ├── api.py             # router aggregator
│   │   ├── health_check.py    # liveness/readiness probe
│   │   ├── oidc.py            # OIDC login/callback (skipped in proxy-auth mode)
│   │   └── evalshub_eval.py   # runs ROUGE / BLEU / LLM-as-judge evals on the agent
│   ├── core/config.py         # AppConfig (pydantic-settings) + INI/secret parsing
│   ├── tools/word_length_tool.py   # an example tool to copy from
│   ├── utils/prompts.py       # prompt/message assembly helpers
│   └── storage/connection.py  # Postgres + pgvector engine and connection pooling
├── sdk/         # a generated, typed client SDK (protobuf) other services import
├── configs/
│   ├── dev.ini / stg.ini / prd.ini   # one config per environment
│   └── secret.ini.example     # secret template; real values resolve from Vault at deploy
├── databases/postgresql/      # SQL migrations (pgvector extension bootstrapped for you)
├── scripts/
│   ├── db.py / db.sh          # migration runner
│   └── gunicorn_conf.py       # production server/worker config
├── tests/
│   ├── unit_tests/            # starter unit tests (e.g. the health check)
│   └── evalshub_evaluation/   # golden test cases the eval route runs against
├── Dockerfile                 # multi-stage, distroless
├── Makefile                   # setup / run / test / lint targets
├── pyproject.toml             # uv build backend + pinned deps
└── .pre-commit-config.yaml

Three things are worth pulling out of that tree:

app/agents/ is the part you actually own. You get two working agents to fork from rather than a blank file: simple_react_agent.py is a single-agent LangGraph ReAct loop, and mcp_react_agent.py is the same loop wired to pull its tools from remote MCP servers. Both compile to a LangGraph StateGraph with a retry policy and a 30-second per-step timeout, and in dev the graph is auto-exported as a diagram. This is a real step up from the bare LangChain agent initialization we scaffolded in 2024.
app/routes/evalshub_eval.py ships evals on day one. The template comes with an endpoint that runs Recall-Oriented Understudy for Gisting Evaluation (ROUGE), Bilingual Evaluation Understudy (BLEU), and LLM-as-judge evaluators over a set of golden test cases in tests/evalshub_evaluation/. The thing we most wished the support bot had, is now in the box before a builder writes a line of their own logic.
Everything else is the production wrapper. core/config.py, storage/, configs/, databases/, scripts/, the distroless Dockerfile, and the pyproject.toml (now uv, not the Poetry we used in 2024) are the auth, secrets, persistence, packaging, and deploy plumbing that every service needs and that no team should have to write from scratch.

The day-one wiring that used to take two weeks or more now takes about an hour. The rest of this section is what “pre-wired” means, layer by layer.

Config and secrets are solved once. Apps declare environment configs as initialization (INI) files with secret interpolation, so secrets resolve from Vault at boot, and a single secret.ini.example is enough to run any LLM-Kit app locally:

[CONFIG]
GRABGPT_API_KEY=${SECRET:GRABGPT_API_KEY}
OTEL_EXPORTER_OTLP_ENDPOINT=<otel-collector-endpoint>
POSTGRES_POOL_RECYCLE=1800

Model access behind one resolver. Every model call goes through the GrabGPT Gateway, which is OpenAI-compatible. LLM-Kit’s job is just to resolve the right endpoint (per environment, and per data tier) and inject the key so application code never hard-codes a provider again:

from openai import OpenAI
from llm_kit.grabgpt import resolve_grabgpt_base_url, resolve_grabgpt_api_key

client = OpenAI(
    base_url=resolve_grabgpt_base_url("prd", "public"),  # provider chosen centrally
    api_key=resolve_grabgpt_api_key(),
)

That one indirection is what later lets a platform team change which provider serves a model, configure fallback routing, set budgets, and manage cost attribution, without a single application touching its code.

Tracing wired in, not bolted on. A single instrumentor auto-instruments FastAPI, outbound HTTP, LangChain, and MCP, and stamps every span with Kubernetes resource attributes (pod, namespace, image, service version). Structured logs auto-inject the trace and span IDs, so logs and traces correlate in Grafana/Kibana for free:

exporter = OTLPSpanExporter(endpoint=app_config.otel_exporter_otlp_endpoint)
OTELInstrumentor(exporter=exporter, excluded_urls=["health_check"]).instrument_app(app)

The three systems, no shared trace problem turns into one end-to-end trace across every LLM call, tool call, and retrieval step.

Tools can be exposed through MCP servers built on our MCP framework. Instead of hardwiring a large set of tool functions inside the agent process, the agent connects to MCP servers and discovers their tools at runtime. That means adding a new capability can be as simple as registering an MCP server, rather than redeploying the agent.

client = MultiServerMCPClient({
    "mcp-gitlab-remote": {
        "transport": "streamable_http",
        "url": "<remote-mcp-gitlab-endpoint>/mcp/",
        "headers": {"Authorization": "Bearer <token>"},
    }
})
tools = await client.get_tools()   # schema negotiated, no redeploy

An agent is just another service in the ecosystem, with gRPC on both sides. Most of Grab’s backend communicates over gRPC, and agents are rarely standalone; other services call them, and they in turn call other internal services. The template is designed to support both directions.

On the serving side, the scaffold includes a Protocol Buffers (protobuf) contract (sdk/.../.proto, with a sample Hello remote procedure call (RPC)) and a generated, typed client SDK package that other teams import to call your agent without hand-writing HTTP. make gen-proto regenerates the Python stubs from the .proto, and a gen-proto-check Continuous Integration (CI) step fails the build if the committed stubs drift from the contract. A gRPC server runs alongside FastAPI (default port 8087, multi-worker-safe via SO_REUSEPORT) and ships a standard gRPC health service out of the box:

$ grpcurl -plaintext localhost:8087 grpc.health.v1.Health/Check

On the calling side, LLM-Kit ships a channel provider so an agent never hardcodes an address. The auto provider tries Istio, then Consul, then a static fallback, health-checks the channel it selects, and runs a background monitor that re-selects after a few consecutive failures:

from llm_kit.grpc.channel_providers.auto import (
    AutoGrpcChannelProvider, AutoGrpcChannelProviderConfig,
)

provider = AutoGrpcChannelProvider(logger, AutoGrpcChannelProviderConfig(
    client_name="my-agent",
    service_key="some-internal-service",   # resolved via Istio / Consul
    enable_istio=True, enable_consul=True,
))
channel = provider.get_channel()           # first healthy channel, auto-reselected on failure
stub = SomeServiceStub(channel)

This is the less glamorous side of being production-ready. Before an agent can deliver value, it needs to both accept calls from and make calls to the rest of the company’s services using the same transport the broader system already relies on.

What’s next

LLM-Kit solved building and shipping one agent. At 500 agents, the problems were no longer framework problems. They were platform problems: who can change which model everyone calls, how one team safely reuses another team’s tools, and how you know an agent got better and not just different after a prompt change. We built three answers for that layer: the GrabGPT Gateway, a remote MCP framework, and an evals platform. Part 2 starts with the gateway — one endpoint, five providers, and what it takes to make “swap the model” a configuration change instead of an incident.

Join us

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!