https://yaooqinn.github.io/ Recent content on Kent Yao Hugo -- 0.157.0 en-us Sun, 31 May 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/query-engines/llm-cardinality-estimator/ Sun, 31 May 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/query-engines/llm-cardinality-estimator/ Cardinality estimation is the heart of the optimizer. A team from Peking University and ByteDance fine-tunes Llama-3 8B to do CardEst, and on workloads like IMDB and STATS the 99th-percentile Q-error drops by up to 74.1% versus the strongest baseline (PRICE) — the accuracy win is real. But end-to-end, it backfires: on JOB-light and ErgastF1 the LLM’s more accurate plans are dragged down by its own inference latency, with total time exceeding even the strongest baseline PRICE. The real engineering contribution isn’t the model — it’s the gate that uses the optimizer’s own cost model as a bouncer: call the LLM only for high-cost sub-queries, leave the rest to the old methods. https://yaooqinn.github.io/posts/query-engines/llm-index-tuning-vs-dta/ Sat, 30 May 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/query-engines/llm-index-tuning-vs-dta/ A Microsoft team evaluates LLM-driven index tuning on real enterprise customer workloads. On query 22 of Real-R, the SOTA commercial tuner DTA recommends indexes that cause a near-10x regression; on the same query, GPT-5 cuts execution time from 10 seconds to 4. The LLM wins precisely where the what-if cost model is wrong. But that intuition is high-variance, can’t be bolted into the existing architecture, and can’t be validated cheaply — it’s not a replacement for DTA today, it’s a source of the candidate indexes DTA can’t see. https://yaooqinn.github.io/posts/query-engines/rule-rewrite-blindspot-dsb/ Wed, 27 May 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/query-engines/rule-rewrite-blindspot-dsb/ On TPC-H 10GB, a state-of-the-art learned rewriter cuts mean execution time from 69.84s to 37.57s — a 46% win. On DSB 10GB, the same rewriter takes 32.62s to 31.93s — a 2.1% non-event. The gap isn’t query difficulty; it’s whether the benchmark is in the rewriter’s training distribution. “Rule-based systems are stable and reliable” is often a benchmark artifact, not an engineering fact. https://yaooqinn.github.io/posts/query-engines/prompt-anatomy-for-plan-generation/ Wed, 27 May 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/query-engines/prompt-anatomy-for-plan-generation/ DBPlanBench gets GPT-5 to deliver a 4.78× geometric-mean speedup on DataFusion TPC-H SF10 by letting the model rewrite physical plans directly. I read its sql_optimization_prompts.py end to end — 120 lines, 30 of methodology, 90 of contract. That ratio is the most transferable thing in the paper. https://yaooqinn.github.io/posts/query-engines/llm-as-plan-tuner-not-optimizer/ Wed, 27 May 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/query-engines/llm-as-plan-tuner-not-optimizer/ Putting the LLM after the optimizer, emitting JSON patches for local plan tuning, is easier to reason about as engineering than asking it to replace the cost-based optimizer. https://yaooqinn.github.io/posts/spark/branch-flip-analysis-from-postgres-to-spark/ Tue, 26 May 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/branch-flip-analysis-from-postgres-to-spark/ An ETH paper finds 21 previously unknown performance bugs in PostgreSQL, MySQL, CockroachDB and MariaDB by flipping optimization branches on and off. The technique is conceptually simple, the surface in Spark is unusually inviting, and the open-source engine community already ships one of the building blocks. https://yaooqinn.github.io/posts/query-engines/llm-only-rewrite-doesnt-work/ Tue, 26 May 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/query-engines/llm-only-rewrite-doesnt-work/ On TPC-H 10GB, asking GPT-4o to rewrite SQL takes mean execution time from 78.81s down to 74.92s — almost nothing. Swap in an open 14B model, feed it plans, add a reward, fine-tune once, and the same workload drops to 29.67s. Whether LLMs can help SQL rewriting is not a question about model strength; it’s a question about whether you’re willing to give the model the signals it actually needs. https://yaooqinn.github.io/posts/spark/llm-for-join-order-an-apache-spark-perspective/ Mon, 25 May 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/llm-for-join-order-an-apache-spark-perspective/ Databricks and UPenn put an LLM agent to work as an offline join-order tuner and got P90 latency down 41% / geomean 1.288× speedup on JOB’s 113 queries — beating even perfect cardinality estimates. From the trenches of an open-source query engine, here is what that result does and does not prove. https://yaooqinn.github.io/posts/spark/sql-metrics-part5-gluten-internals/ Thu, 02 Apr 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/sql-metrics-part5-gluten-internals/ Part 5 of the SQL Metrics deep dive. How Gluten maps Substrait plan nodes to Velox operators, aggregates metrics across pipelines, walks the MetricsUpdaterTree, and handles aggregation sub-phases and shuffle metrics. https://yaooqinn.github.io/posts/spark/sql-metrics-part6-in-action/ Thu, 02 Apr 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/sql-metrics-part6-in-action/ Part 6 of the SQL Metrics series. A real-world walkthrough of TPC-DS q99 at SF10000 with Gluten/Velox, reading every metric to understand what happened during execution. https://yaooqinn.github.io/posts/spark/understanding-sql-metrics/ Wed, 01 Apr 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/understanding-sql-metrics/ Part 1 of a 3-part deep dive into Apache Spark’s SQL metrics system. Covers the 5 metric types, a complete reference of 100+ metrics across all operators, and how to read the numbers in the Spark UI. https://yaooqinn.github.io/posts/spark/sql-metrics-part2-internals/ Wed, 01 Apr 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/sql-metrics-part2-internals/ Part 2 of the SQL Metrics deep dive. How metrics flow from tasks to driver, and how Adaptive Query Execution uses shuffle statistics to rewrite plans at runtime. https://yaooqinn.github.io/posts/spark/sql-metrics-part3-extension-api/ Wed, 01 Apr 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/sql-metrics-part3-extension-api/ Part 3 of the SQL Metrics deep dive. How to extend Spark with custom metrics via the DataSource V2 API, how the UI renders them, and how to query metrics programmatically. https://yaooqinn.github.io/posts/spark/sql-metrics-part4-gluten/ Wed, 01 Apr 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/sql-metrics-part4-gluten/ Part 4 of the SQL Metrics deep dive. How Apache Gluten bridges native Velox/ClickHouse metrics back to Spark’s SQL Metrics framework, adding 60+ metrics that vanilla Spark doesn’t have. https://yaooqinn.github.io/posts/spark/spark-declarative-pipelines/ Sat, 28 Mar 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/spark-declarative-pipelines/ Apache Spark 4.1 introduces Spark Declarative Pipelines (SDP) — a declarative framework that lets you define what your data should look like, not how to compute it. As a Spark PMC Member, here’s my take on what this means for data engineering. https://yaooqinn.github.io/posts/spark/spark-advisor/ Fri, 20 Mar 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/spark-advisor/ spark-advisor is an agent skill that turns your AI coding assistant into a Spark performance engineer — diagnosing slow jobs, detecting skew, comparing benchmark runs, and producing actionable tuning recommendations. https://yaooqinn.github.io/posts/spark/spark-history-cli/ Wed, 18 Mar 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/spark-history-cli/ spark-history-cli brings the Spark History Server to your terminal — an interactive REPL and one-shot CLI that covers all 20 REST API endpoints. List apps, inspect jobs, drill into stages, check SQL executions, and download event logs without ever opening a browser. It also ships as a GitHub Copilot CLI skill. https://yaooqinn.github.io/posts/spark/sql-execution-page-modernization/ Sat, 14 Mar 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/sql-execution-page-modernization/ The SQL execution detail page in Spark’s Web UI used to show jobs as comma-separated IDs. Now it has a full Associated Jobs table with status, duration, stage progress, and task progress bars — so you can debug SQL queries without clicking through each job individually. https://yaooqinn.github.io/posts/spark/dark-mode-spark-ui/ Fri, 06 Mar 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/dark-mode-spark-ui/ Apache Spark’s Web UI now supports dark mode — a long-awaited quality-of-life improvement for developers who spend hours debugging jobs. Here’s why we built it and what it means for the Spark community. https://yaooqinn.github.io/posts/spark/sql-plan-visualization/ Thu, 05 Mar 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/sql-plan-visualization/ The Spark SQL plan visualization just got a major upgrade — compact node labels, clickable metric panels, and edge row counts that make join explosions immediately visible. https://yaooqinn.github.io/about/ Mon, 01 Jan 0001 00:00:00 +0000 https://yaooqinn.github.io/about/ About Kent Yao https://yaooqinn.github.io/projects/ Mon, 01 Jan 0001 00:00:00 +0000 https://yaooqinn.github.io/projects/ Open source projects