container name: spark-kubernetes-executor
container state: terminated
exit code: 137
termination reason: OOMKilled
The log every team running a native Spark accelerator at scale eventually sees.
Vinoth Chandar
@byte_array
Founder @Onehousehq, Creator of @apachehudi, Built the World's first #DataLakehouse, Distributed/Data Systems, Linkedin, Uber, Confluent alum. (views are mine)
Joined April 2009
- "BYOC or serverless, pick one" is how most compute vendors frame it. Both options are real, and both are worse than they should be.
- Open table formats won the open architecture argument years ago. No one serious is relitigating it. The part we skipped is operational, and I keep watching good teams get burned.
- GROUP BY ROLLUP over N keys computes N+1 levels of subtotals. Spark computes that by copying every input row N+1 times before it aggregates anything. That copying is the Expand operator, and it's the real cost of your rollup query.
- Latency claims are cheap, so here are the receipts. A real customer's 12 hour Athena trace replayed through Lakegres, across 570 production tables: p0.001 goes from 1000ms to 119.86ms.
