Hyperparam

Hyperparam is building tools for the future of AI data - An estimated four quadrillion tokens are being produced every year, and that number is accelerating. Every new AI data center, every GPU brought online, and every model deployed adds a rapidly compounding ocean of unstructured text. This includes chat logs, agent traces, reasoning tokens, tool calls, and system outputs. Unlike structured data, which has benefitted from decades of mature tooling, this new class of data is largely unreadable at scale. Most of it is logged, stored, or quietly discarded, despite containing immense value. The problem is no longer building intelligence. It is understanding what is already happening. That is why we are excited about Hyperparam. They are building a new category of data engine designed specifically for massive, unstructured text. By bringing high performance data infrastructure directly into the browser and pairing it with AI native analysis, Hyperparam makes it possible to explore, simulate, and reason over datasets no human could ever read manually. Teams can replay entire histories under new prompts or models, ask fuzzy real-world questions that SQL could never express, and understand the downstream impact of changes before they ship. AI is already live and operating at scale, and Hyperparam is one of the few teams truly paying attention to what that means. | Kenny Daniel |

Squirreling is a JavaScript SQL engine I built to make interactive, ad hoc data exploration possible in the browser. I wrote about the thought process behind it and how optimizing for the browser leads to different design choices than anything else out there. https://lnkd.in/g4BH3AYN

Rendering 30M+ points on MapLibre on WebGPU in 10s! Some work in progress over Christmas: a WebGPU-based layer for loading millions of features in seconds from a standard geoparquet file. After crashing MapLibre's built-in layers with around 2.5M points (https://lnkd.in/e9xrqcTc) and bumping into lags with deck.gl with 3M points (https://lnkd.in/ekYNrtM6 thought I'd take a stab myself relying on a modern WebGPU implementation after parsing the GeoParquet file with Hyperparam's Hyparquet (which is insanely fast and reliable!). Results on my Mac: - Up to 3M points it's basically lag-free - Up to 10M it's absolutely usable with small lags only on low zoom levels (panning and zooming up to some higher zoom levels where less points fall in view). The lag is proportional to the point radius however. So short radius -> almost no lag! - With around 30M (1.1Gb file) low zoom levels become the bottleneck where all points need to be rendered. There are significant lags (comparable to my tests with deck.gl layers at around 3M points, maybe slightly worse) but it still works. It's fully fluid though on high zoom levels! - At around 40-50M points I encounter weird bugs where something runs OOM. Still investigating whether I can eventually push it to 100M points! From my experience now I'd say that this is probably possible but it can't be called interactive anymore when you get only 1 frame per minute :D Strangely, I found Safari to work more reliably for up to 40M points than Chrome, crashing already at 25M points. Still performance seems better on Chrome with 20M points. I guess Chrome has some stricter OOM prevention policies? Test data from Foursquare, free background maps from OpenFreeMap. I'll release everything open source after the holidays and will probably record a short tutorial. Happy holidays and merry Christmas everyone! WebGL / WebGPU

If code were effectively free, would you still design software the same way? In this post, I reflect on how my process of software architecture and design has changed thanks to AI. I spend a lot of time thinking about context locality, testability, and dependency structure. And this is reflected in some of my recent projects like Hyparquet and Squirreling. We need to update our existing “best practices” around software development for the world where code is “too cheap to meter”.

The slow painful part of debugging LLM chat logs is inspection. If it takes too long to actually look at the data, you're not going to do it, and your iteration speed is capped.