The other day, I was thinking about how I pick my stack.

“Built” means nothing to me anymore. Agents can produce sophisticated code — well-structured, true rocket science. They can build almost anything without breaking a sweat. Tests green, always.

I choose battle-tested implementations. I let time decide which implementations win and which perish. Those that survive must have gone through the brutal cycles of reality: functional issues, security issues, performance issues. Even the most tasteless software, when exposed to enough user feedback, eventually ends up in a slick state (or dies).

I choose mature technologies. Agents can implement anything. The value lies in technologies that actually solve real problems — that have found product-market fit. Only time can tell this.

The next time you want to add that 5-star dependency, how about you let your agent vibe-copy it instead?

I am back at FOSDEM, in Brussels. Unlike last year, I flew in for the weekend. The bus was terribly late. And here’s the first situation I found myself in:

Having confirmed with one of the volunteers that the queue situation was a real thing indeed, I hurried back to the end of the line. What if I don’t get in? Ten to twenty minutes later, a flock of people got out of the stadium-like auditorium, and every one of us found ourselves a seat inside. Patience pays off!

The Future of Git: What’s Next?

Now we are back! A Git maintainer / GitLab staff engineer revealed that developer experience is their key priority. Sourcing inspiration from Jujutsu — a Git-compatible frontend.

Some features to watch for: we’ll be able to switch branches without stashing (essentially auto-commit). We’ll be able to edit commits. Child branches will automatically rebase on parent updates.

Maintainers will on enabling stronger SHA-256 (instead of SHA-1) hash functions for file checksums, reducing the chance of someone sneaking in malicious code with the same hash but different contents.

Overall, an okay keynote, though it spent too much time on the details.

Lightning talks - love the format!

Afterwards, I rushed to the databases hall, where the lightning talks had already started. Now, this was the FOSDEM I remembered — punchy five-minute talks, perfect for our TikTok fried brains.

RonDB: Unified ML Feature Stores

The first talk on the line was the RonDB session. RonDB promises to unify both offline and online stores (think classical BigQuery / ClickHouse + Redis) into a single database for ML feature building and serving. That’s a nice operational benefit. In my current place, though, we already benefit a lot from Feast, which achieves this unification as a software-level abstraction. Still, it sounds cool if you’re starting from scratch.

DuckDB: Is Spark dead already??

Next-up on the line was DuckDB. I am always excited about DuckDB.

DuckDB has somewhat recently unveiled its Iceberg support. It’s realistically read-only, since they don’t support upserts (merge into), which is highly critical for idempotent writes. I gave it a shot once , and it seems it has limitations for some catalog types as well. Despite these drawbacks, the progress is clear, and I bet in a few years or so Iceberg ecosystem will reach similar maturity as Parquet, with different mature implementations in various languages. Just a few more years, and you can drop Spark - for medium data use-cases though.

Grafana Labs: SQL + Analytics for operational metrics

Lastly, I was surprised to learn that even Grafana Labs is spearheading a SQL + analytics stack like dbt as a valuable technology direction on top of operational use cases. That’s such an understatement, in my opinion — DWHs love appends, and operational metrics are all time-series appends; alerts and metrics are really just windows + group-bys on columns. Perfect efficiency! And when was the last time you actually enjoyed writing PromQL?

Lunch Break

Time for lunch. The typical FOSDEM menu is slightly overpriced, fuel-like food: burgers, fries, pizzas. But hey, can’t complain; the conference itself is free.

Containers, ML models as containers?

Having fueled myself—and knocked back an espresso shot — I rushed back, this time to the containers track room. I wanted to see something new, outside of databases and Python.

Basically, docker model run. Did Docker eventually manage to create a perfect ML model packaging format? The model, its dependencies, I/O schemas—able to run in any container-like environment? Only for LLMs, though…

In essence, you can package your favorite LLMs supported by frameworks like VLLM into an OCI-compatible artifact, storing it on GCP’s Artifact Registry or your own hosted registry (argument: you don’t want private stuff on HuggingFace). Neat—but I was really hoping they had solved this for any arbitrary model, like scikit-learn with custom preprocessing (and maybe even beat MLflow).

Apache Arrow: The Future of Data Exchange

Next on the line was the Apache Arrow talk:

I had been waiting for the Apache Arrow talk. Matt is an engaging presenter. Arrow remains a common in-memory format, allowing different systems to communicate without serializing or deserializing data.

An argument he focused on this year is that row-based communication, like JDBC, is wasteful. Since most sources have mostly converged to columnar memory layouts, transferring data in columnar Arrow format eliminates the need to transpose it. CPU is the bottleneck, not network speed anymore. Hence comes Columnar, building a faster JDBC with Arrow at its core.

Read up on why exchanging data in columnar format is so beneficial in this rather historical paper (from 2017, haha) by one of the DuckDB co-authors.

Lakehouse Migration for Streaming: Lessons Learned

A cherry on top of the day was this talk on a lakehouse migration for a high-write streaming stack:

The talk served as an assurance for us, since we are exploring the lakehouse architecture at my current workplace.

They moved from a classic shared-nothing DB to shared-everything on object storage. In shared nothing approach, each node holding the full state to avoid read/write bottlenecks. This causes storage to balloon (multiply with each replica), also crashed replicas can’t serve since they need to be catching-up from WAL. By switching to stateless nodes with the Rust-based DataFusion query engine, they cut costs by 50% (no replica storage duplication) and improved speed (thanks to DataFusion itself mostly). An architecture usually reserved for DWHs, I was genuinely surprised they pulled this off for streaming.

Eventually, unwinded after the day with some 9% Belgian beers.

Day 2: Rust and Python

The previous day, we had discussed with some colleagues that the next day looked rather empty. Overall, it did turn out to be less packed than the first one, but nonetheless — punchy.

I started the day meeting an old colleague. The first few talks were a miss, thanks to the classic mistake of focusing too much on the details.

We eventually sneaked into the Rust track, packed full, for the talk by ClickHouse’s CTO himself!

Clickhouse: C++ to Rust

Engaging delivery on ClickHouse’s journey from C++ to Rust. The CTO argued it only makes sense to swap C/C++ for Rust if:

• There’s a Rust extension with a significant community around it, whereas C++ doesn’t have one. A good example is delta-rs, a Rust-based delta-log implementation ClickHouse adopted to support reading and writing from Delta Lakehouses.
• Rust is cool. It serves as a marketing tool to attract maintainers.

On the other hand, a full rewrite is a bad idea because:

• Rust is too hyped—its underlying value might turned out to be often lower than it seems.
• Separate dependency management can be tricky. For example, ClickHouse had issues with both baked-in OpenSSL and the system’s dynamically linked OpenSSL, because the Rust dependency relied on a transitive version that differed from the C++ setup.

ty: The Fast Python Type Checker

Energized, we all jumped from Rust for databases into Rust for Python developer tooling. Meet Ty - the new fast Python type checker.

The main point — ty is fast not just because it’s written in Rust, but also because it leverages incremental computation. Whenever part of your source changes, it only recomputes the dependent parts for type checking. Ty’s been out of beta for a while—definitely give it a shot!

Wrap-up

We ended the day in one of the many FOSDEM indoor cafés, with some beers and fries. Met a few old colleagues, so we eventually skipped the rest of the presentations just to catch up…

Intense, knowledge-packed, social weekend. Hard to pin down a main theme — there was a bit of everything for everyone. That’s FOSDEM. See you in 2027!

I’ve switched to uv for all of my Python projects—both legacy and new—over the past year. It offers many interesting features, but I’ve narrowed it down to three that make it an absolute joy to use.

It can manage Python versions

Not only UV can install dependencies, but can also install Python (-s). Great dev-ex, one command for both. An example - I constantly switch between 3 Python apps, all with different Python versions. With UV, I can go to project’s dir, run:

uv run pytest tests

and it installs / switches to project's Python. Infers straight from pyproject.toml:

[project]
requires-python = ">=3.12,<3.14"

uv run will pull-in Python 3.13, at most. No need for a separate tool like pyenv, on top pyenv does not understand pyproject.toml’s Python.

It is fast

Faster than pip, or frankly any package manager. It feels real-time. Because of Rust™. Speed is rarely a tradeoff (from a user point of view).

It can spin-up virtual envs on the fly (the script mode)

A year ago I thought this one was a killer feature. Putting it last though, I barely used it a few times. Nonetheless, it’s cool. Here’s a situation I had - I was debugging some parallel http calls, it was convenient to just run the script with:

uvx --python 3.13 --with pandas test_concurrent_calls.py

Pulled-in Py13 + Pandas. Slight difference from a year ago - change uv -> uvx, to leave no venv trace. Neat, right?

Cons?

I’ve yet to find any. Once I couldn’t install some Torch dependency because UV kept insisting incompatibility with my system. That’s useful though, but I knew what I was doing and got blocked. Conda packages? Unsupported, but didn’t need those, at least for production apps.

To another year - now, please improve (speed of) type checking 🐍.

Some time ago, I was toying around with an idea for a tech product. I was procrastinating on building a landing page. Pixel pushing, as I used to call it, is an excruciating process. So, I asked an LLM¹ to do it for me. “Surprise me,” I thought. In a few minutes of prompting, I was amazed. Here it was — an interactive website with callouts, a contact form, and more. But that joy vanished quickly. Looking deeper, it felt too generic. I tried multiple times, but nothing felt right. I didn’t really know what made a good landing page, so LLMs couldn’t help me much either. I parked the idea of an autonomous coding assistant then.

Later, I found myself procrastinating again. I’d been delaying an idea to give a breath of fresh air to my personal blog. I had borrowed a simple idea from Andrej Karpathy’s blog: move posts into the landing page and the bio section into a separate about page. It meant some yawn-inducing HTML and CSS. Once more, I called on an LLM¹. It nailed it. With some interim feedback and a quick code review, I had a good result in a few minutes instead of an hour.

Sonnet adjusted the UI just as instructed. Sonnet also coded-up this HTML to illustrate the argument.

Makes me think. In the first situation, I had no vision of the end result — I didn’t know what makes a good landing page. I could barely give the agent useful feedback. It saved some typing time but produced no good results. In the second, I had a clear end state in mind, so a bit of guiding brought us to a satisfactory end result, in no time. Both tasks’ scope was slightly different though, but I believe the key difference was having a clear end state.

I can finally un-procrastinate my side projects! They offer no monetary incentive (unlike jobs) so the process of shipping is painful - considerate amount well-scoped work with obvious - so boring - implementation effort, that I would always put aside. No more. Exciting times ahead!

¹ In both cases, I used Claude Sonnet 3.5 and 4, via GitHub’s Copilot VSCode extension. The latter experiment obviously benefited from a more powerful model, but I doubt it was a significant factor.

Free threading (no-GIL) is the main feature of Python 3.14, setting it apart from its predecessors. We can process multiple tasks at the same time (so, parallelise) in Python with pure threads. Just see the benchmarks and here.

Not everyone might share my enthusiasm - but here’s why it matters:

“I can just use multiprocessing.” Yes, but it comes at a cost — slow inter-process communication. Every argument must be pickled for message passing; objects can be slow to pickle, and some aren’t picklable at all. You need to minimize sync points and be selective about the data exchanged between processes.

“We already have C and Rust extensions.” Like C NumPy, obviously use one if you can. But would you ever write one yourself just to parallelise?

“We have async I/O.” Useful only if you’re I/O bound i.e. tons of external calls - I’ve written about that before.

“I can scale horizontally”. Sure — but if you care about minimizing single-request latency, horizontal scaling only increases throughput. The time to process an individual request remains the same as long as the machine itself is unchanged, we merely add additional machines to serve more requests.

So, in theory, we have absolutely None ergonomic means for parallelisation in Python. 1Starting with 3.14, no-GIL aims to change that.

Here are a few practical use cases where I’ll immediately start exploring no-GIL:

Algorithm or ML model inference. You want fast, customer-facing predictions. Batch APIs are full of Python glue for feature fetching and pre/post-processing, but inputs are easy to parallelise. It’s all CPU-cycles. At my previous job, we used multiprocessing to cut recommendation endpoint latency by several times (cost = super selective about message passing) — wish we had no-GIL back then.

Existing async I/O apps. I suspect many async I/O apps also do decent amount CPU work that can’t currently be parallelised due to the GIL. I’m already excited to benchmark existing implementations — with parallel event loops on the way, we might see latencies drop!

I don’t think data pipelines will benefit from no-GIL though. Most don’t have latency requirements (rather throughput) and we already offload bulk of CPU work to specialised SQL / DF engines as Polars, DuckDB, PySpark (you are not going to be rewriting them soon).

Hopefully, this convinces you to try free threading / no-GIL. I am excited myself. Especially with uv, it’s as a easy as:

uv run --python=3.14t myscript.py

Happy 🧵-ing!

One of the ideas from A Philosophy of Software Design that struck me was that good method interfaces should be shallow. Fewer parameters make it more obvious how to use a method - less cognitive load incentivizes folks to actually use your API. Fewer parameters mean less levers, so lower chance of misuse. Fewer parameters reduce coupling, making method changes less risky.

Resist the urge to add yet another parameter to your API. Your users will thank you.

I love Kubernetes — for production, it’s rock-solid. But throw me a notebook that needs 0.5 TB of RAM and a GPU, and suddenly K8s feels like a chore.

Here’s what that usually means:

1. Write a Dockerfile.
2. Build the image.
3. Push it to a registry.
4. Write a deployment spec.
5. Deploy it to the cluster.
6. Set up secrets and environment variables.
7. Wait for the pod to start.
8. Port forward to Jupyter within the pod.
9. Finally, run the notebook.

This gets even more interesting if you forget to shutdown the pod. 24 * 10 USD / h = 240 USD / day, down the drain 🤦.

Meet kuberun.
Just decorate your function with @kuberun and it will run on a Kubernetes cluster as if it were local. It quietly handles the Docker build, push, deployment, and wiring — you just write Python:

@kuberun(requirements=["torch", "transformers", "pillow"], cpu=32, mem="64Gi")
def image_contents(image_bytes):
    from transformers import pipeline
    from PIL import Image
    import io
    import torch

    # Load and convert the image bytes to PIL
    image = Image.open(io.BytesIO(image_bytes))
    if image.mode != "RGB":
        image = image.convert("RGB")

    # Choose a captioning model—e.g. BLIP or ViT‑GPT2
    captioner = pipeline("image-to-text", model="Salesforce/blip-image-captioning-base")
    # Alternatively: model="nlpconnect/vit-gpt2-image-captioning"

    # Run the pipeline
    outputs = captioner(image)

    # outputs is a list of dicts like [{'generated_text': "..."}]
    caption = outputs[0]["generated_text"]

    return {"description": caption}


...

caption = image_contents(image_bytes)["description"]

As soon as the function returns, the pod and any related resources get immediately cleaned-up.

Zooming in:

@kuberun(requirements=["torch", "transformers", "pillow"], cpu=32, mem="64Gi")
def you_func(arg):
    ...

Under the hood, it builds a rather standard templated Docker image and generates a just enough Kubernetes Pod spec. The image is pushed to the configured registry, and the spec is applied to the cluster (within a default namespace, configurable though). Arguments and results are pickled — like in Ray or Python’s multiprocessing— and exchanged through a shared volume with a sidecar container.

Builds are very slow, and I haven’t figured out annotations, secrets, environment variables yet (without manual spec writing, which defeats the purpose). Still, the concept gives me energy, so give it a try—and share your feedback via an issue!

Happy experimentation!

¹ I sort of accidentally found out that the name kuberun had been already used for some fancy ssh to container tool. It seems to be deprecated. Fair if I borrow the name. It’s a pet project after all…

I was attending a DuckDB meetup in Leuven, where a presenter was detailing how they replaced Spark with DuckDB to cut costs, when rather abruptly, someone in the audience asked, “How do I know if I should really use DuckDB now, me being not a data engineer? What if my query is heavier than expected? What if my data fluctuates and just sometimes goes beyond a single machine?”. The presenter admitted that you set up the architecture once, do your best benchmarking, and hope for the data dynamics to stay the same.

I was intrigued, but not really convinced, yet.

Catalog is the bridge

Next day after the event, I dove into the idea that the query engine should be automatically selected based on the query itself. Having some scars from hefty BigQuery bills, I built a SQL executor that switched between DuckDB and BigQuery. The challenge though was that I couldn’t use the same query with both engines due to the lack of shared catalog support.

Fast forward a year, and we now have Unity Catalog support for DuckDB and Polars — yes, the same catalog well supported by distributed engines like Spark and platforms like Databricks. Since we only care about read-only access, I think we can go pretty far with this.

I quickly threw together a prototype — let’s see it in action:

1) Launch the Unity catalog:

$ git clone https://github.com/unitycatalog/unitycatalog.git unity-catalog 
$ cd unity-catalog
$ bin/start-uc-server

should start it at :8080.

2) Load a decent chunk Clickbench data into UC:

spark.read.parquet("data/hits.parquet") \
    .limit(5_000_000) \
    .write \
    .format("delta") \
    .option("overwriteSchema", "true") \
    .mode("overwrite") \
    .option('path', '<abs_path>/hits_delta') \
    .saveAsTable("default.hits")

that’s all it takes to expose a Unity Catalog table backed by a Delta table (files) to other query engines!

3) Pull-in frugal-engine:

$ pip install git+https://github.com/astronautas/frugal-engine

4) Set-up the executor:

import logging
from frugal_engine.main import setup

logging.basicConfig(level=logging.INFO)
setup()

This sets up the hybrid engine with a single-node executor and a cluster executor as a fallback.

Now, let’s run a few experiments 🧪. We’ll try a simple select first:

# %%sequel deliberately, not to interfere with %%sql
%%sequel 
SELECT * 
FROM unity.default.hits LIMIT 10

You should see in the logs that it picked a single node executor since the data size is relatively small:

INFO:frugal_engine.main:single_node_executor has been chosen, because the top source count is less than or equal to the threshold: 6000000
INFO:frugal_engine.main:Executing query in single_node_executor: SELECT * 
FROM unity.default.hits LIMIT 10

Let’s assume our machine is tiny and lower the threshold for single-node workflows:

from frugal_engine.main import setup
setup(threshold=100_000)

%%sequel 
SELECT * 
FROM unity.default.hits LIMIT 10

INFO:frugal_engine.main:Executing query in cluster_executor: SELECT * 
FROM unity.default.hits LIMIT 10

From the logs, we see it falls back to a cluster engine since it “thinks” a single machine won’t process that many rows in a reasonable time. It should also fall back if the single-node executor reaches its memory limits—or even proactively, using ML to forecast that it won’t finish in a reasonable time.

Identical SQL, totally different engines! Convenient-ish to use - check. What about the promise of improved performance / reduced costs?

Quantifying the cost savings

We can’t draw firm conclusions on costs yet, as proper testing with a Spark cluster, Delta access from S3, load full Clickbench dataset - it’s a complex evaluation. I’ll tackle this in the next chapter.

With usability and (some) cost aspects explored, let’s examine what makes the package tick.

DuckDB UC - not quite there yet

I initially tried to use DuckDB for the single-node executor, but… it didn’t go well.

I was so hyped by many talks about how DuckDB should at least have read-only support for Unity Catalog. It works fine for pre-loaded OSS UC tables:

import duckdb
import logging

duckdb.sql("""
	install uc_catalog from core_nightly;
	load uc_catalog;
	install delta;
	load delta;
""")

duckdb.sql(f"""
CREATE SECRET (
	TYPE UC,
	TOKEN 'not-used',
	ENDPOINT 'http://127.0.0.1:8080',
	AWS_REGION 'us-east-2'
);
""")

duckdb.sql("ATTACH 'unity' AS unity (TYPE UC_CATALOG);")


logging.basicConfig(level=logging.DEBUG)
duckdb.sql(f"select * from unity.default.marksheet limit 10;")

But as soon as you commit something to UC via another engine, like Spark, DuckDB halts:

IOException Traceback (most recent call last) Cell In[4], line 4 1 import logging 3 logging.basicConfig(level=logging.DEBUG) ----> 4 duckdb.sql(f"select * from unity.default.hits_2 limit 10;") 

IOException: IO Error: Invalid field found while parsing field: type_precision

CatalogException Traceback (most recent call last) Cell In[5], line 4 1 import logging 3 logging.basicConfig(level=logging.DEBUG) ----> 4 duckdb.sql(f"select * from unity.default.hits_2 limit 10;") 

CatalogException: Catalog Error: Table with name hits_2 does not exist! Did you mean "information_schema.views"?

I spent hours trying to figure it out. Fair enough, the extension is tagged as experimental, and there’s a related GitHub issue, but I expected clearer error messages. 👎

Polars to the rescue

Polars, on the other hand, works flawlessly! Unlike DuckDB, you can’t just attach a catalog—there’s no dedicated Polars client or anything like polars.sql yet. To achieve this, we’ll create a wrapper around Polars UC:

class PolarsClient:
    """
    Just an interface similar to Spark and DuckDB i.e. exposing sql, but going through UC catalog
    """
    def __init__(self, unity_catalog_uri: str, memory_limit: int = None):
        self.unity_catalog_uri = unity_catalog_uri
        self.catalog = pl.Catalog(unity_catalog_uri, require_https=False) # last flag - for local testing
        self.memory_limit = None

    def sql(self, query: str) -> pl.DataFrame:
        tables = [table for table in sqlglot.parse_one(query).find_all(sqlglot.exp.Table)]

        assert len(tables) == 1, "Only one table is supported as a source for now"

        df_scanned = [self.catalog.scan_table(str(table.catalog), str(table.db), str(table.this)) for table in tables]
        df_scanned = df_scanned[0]

        var_name = next(name for name, val in locals().items() if val is df_scanned)

        query_for_pl = sqlglot.parse_one(query).from_(var_name).sql()
        return pl.sql(query_for_pl).collect()

See what we did here? We used sqlglot to retrieve the source tables, pre-load them into LazyFrame, and replace references in the original SQL query. Nasty, but it gets the job done!

def setup(unity_catalog_uri: str = "http://localhost:8080", threshold: int = 6_000_000):
    spark = _build_local_spark_client(unity_catalog_uri)
    polars_client = PolarsClient(unity_catalog_uri)
    client = HybridClient(spark_client=spark, polars_client=polars_client, threshold=threshold)

    # non-invasive way to add a magic function
    get_ipython().register_magic_function(lambda line, cell=None: client.sql(line or cell),
        magic_kind="line_cell", magic_name="sequel")

setup isn’t that interesting—it just introduces a new magic function, sequel, that delegates queries to HybridClient. The real interesting part is engine the decision function.

Using query information to select the best engine

    def _count_heuristic(self, query: str) -> Choice:
        tables = [table for table in parse_one(query).find_all(exp.Table)]
        counts = [self.polars_client.sql(f"select count(*) from {table.catalog}.{table.db}.{table.name}").to_series()[0] for table in tables]
        top_count = max(counts)

        logger.debug(f"Top source count: {top_count}")
        
        if top_count <= self.threshold:
            logger.info(f"{Choice.POLARS.value} has been chosen, because the top source count is less than or equal to the threshold: {self.threshold}")
            return Choice.POLARS
        else:
            logger.info(f"{Choice.SPARK.value} has been chosen, because the top source count is greater than the threshold: {self.threshold}")
            return Choice.SPARK

Essentially, we select the most appropriate engine for the query based on source table statistic - the row count - and then delegate. It’s a simple heuristic, but it illustrates the point.

    choice = self._count_heuristic(query)
    
    if choice == Choice.POLARS:
        try:
            logger.info(f"Executing query in {Choice.POLARS.value}: {query}")
            # polars - unsupported dialect, duckdb might be closest?
            query = sqlglot.transpile(sql=query, write="duckdb")[0]
            return self.polars_client.sql(query).to_pandas()
        # fixme - too generic exception
        except Exception as e:
            logger.error(f"Error: {e}, fallbacking to {Choice.SPARK.value}")                
            query = sqlglot.transpile(sql=query, write="spark")[0]

        return self.spark_client.sql(query).toPandas()
    else:
        logger.info(f"Executing query in {Choice.SPARK.value}: {query}")
        return self.spark_client.sql(query).toPandas()

We use sqlglot to transpile between dialects, but unfortunately Polars SQL is not a supported dialect yet. Finally, if you look closely, we fallback to the default distributed Spark executor if we encouter an unforseen issue - e.g. unsupported SQL operation, memory issues, etc.

Next Step – Databricks

Hope you at least got intrigued by the idea that it’s possible to balance the benefits of single and cluster type query engines without changing analyst workflows. Next, when I have time, I’ll extend this to Databricks and test cost savings in a more realistic production setting - think Spark cluster, Delta tables in S3, annoying authentication issues, etc.

Stay tuned!

This weekend, I participated in FOSDEM 2025, which impressed me more than I had imagined. Boy, was it big—over 1,000 talks! Here are a few notable takeaways:

Apache Arrow is a top dog, with a turbulent future

The first panel I attended on day one was Apache Arrow’s. It felt surreal to meet the contributors face-to-face—they all seemed like genuinely nice people who simply know their niche very well.

I’ve been aware of Arrow for a few years, and this panel only reinforced my confidence in it as the standard in-memory analytics data format for interoperability between languages, processes, and systems. With identically laid-out bytes in memory (based on their spec) for Python, Go, Rust—heck, even Java (crazy, right?) — you can operate on the same shared memory block across different processes and even different stacks without any serialization or deserialization. This is why DuckDB can pass data to Pandas without any overhead, why Spark (with the Arrow option enabled) can pass data to Pandas without any overhead, and why Polars can do the same.

An additional benefit is that it’s columnar, meaning bytes are laid out in a way that allows highly efficient operations on single or multiple columns (e.g., sum, count, etc.). As an example, Polars leverages it as a in-memory storage format with great success.

→ Arrow enables different languages, processes, and systems do work on and communicate at maximum speed — think of it as the English language for software.

Another cool application they mentioned was using Arrow at Hugging Face: efficiently handling data larger than memory for LLMs with their Datasets library. We all know how to stream data from disk when RAM is limited - implementing reading in batches - but with Arrow, you can dump memory bytes to disk 1:1 (pure bytes without any changes) and memory-map the disk to “pretend” it’s RAM — kind of similar to swapping - but with efficient read-in since no deserialization occurs. So, basically, you dump bytes into a .arrow file (with PyArrow), straight into /tmp/, and then mmap, which is much faster and less code than doing the same with Parquet files. Extremely useful for loading datasets larger than memory in a convenient fashion.

A bit of drama — Voltron Data, the main contributor to Arrow, was hit by layoffs last year. The majority of its staff had to find other jobs, meaning a significant portion of Arrow maintainers left. People expressed their commitment to finding ways to support the Arrow project (e.g., through foundations) to ensure its continued maintenance, but they seemed quite concerned nonetheless.

With Arrow, nothing prevents us from implementing a dataframe library in Go, right?

Graph databases will not conquer the world, but they have their niches

Great historical talk by one of DuckDB’s developer advocates, Gábor Szárnyas, who is apparently a reputable graph database researcher as well! Key takeaway: graph databases excel at nested/recursive multi-join problems because (a) writing SQL for something like the shortest path quickly turns into a mess of LISTAGG, multiple joins, etc., and (b) they leverage factorization under the hood—flattening certain columns into lists to reduce joins (e.g., storing a person’s friend list as a list instead of separate rows). This makes them dominant in this niche.

However, graph DBs won’t replace RDMS - they are neither faster (usually since they are schemaless), nor better supported (very fragmented landscape due to overfitting to specific use-cases).

On a positive note, his group has made massive leaps on a graph DB benchmarking suite, which should move the field forward either way.

Opentelemetry + | insert your favorite OLAP engine | is the way to go

There was a talk from a Clickhouse developer advocate, who once more preached Clickhouse for telemetry. He had a point though.

I don’t have years of SRE experience to make any strong recommendations here, but having dabbled a bit with the classic Prometheus + Grafana setup, I’ve noticed two issues:

1. PromQL is a terrible language—hard to read and write, especially compared to SQL.
2. Prometheus time-series DB is relatively slow, given the sheer amount of data it has to process.
3. Prometheus instrumentation client is okay, but why lock yourself into Prometheus when you can use OpenTelemetry and switch between different backends?

OK, you may have to set-up some things e.g. a UI, but it’s not the hard part i.e. you can set-up Granana with some SQL sources e.g. the Clickhouse, and you’re good to go.

From my perspective, it’s just that we have ingrained habits of setting up these telemetry stacks. I have to agree with the overly excited ClickHouse presenters: the future telemetry stack is OpenTelemetry (to reduce vendor lock-in) combined with any real-time OLAP database, like ClickHouse. Sums, counts, percentiles, etc., are OLAP’s bread and butter—typically all one- or multi-column operations.

Will benchmark sometime soon. This entire area can be IMHO disrupted™.

Wasm will not replace containers, but it is the definition of cool

Dan Phillips showcased a pretty cool project he’s been developing, which essentially takes a Dockerfile, runs box build -f Dockerfile, and produces a WASM binary that can even run in a browser. In general, this results in much lower startup times and better isolation, which I definitely agree with.

Here’s the catch: no, containers are not disappearing. WASM has slower runtime performance because the WASM runtime acts like a VM executing WASM assembly-like bytecode. They are reimplementing every syscall to match the same interface, which requires an astonishing amount of effort. On top of that, they’re reimplementing every Dockerfile command.

Nonetheless, it’s a cool project, the author seems like a great enthusiastic engineer, and I can see it being used for short-running FAAS (short runtime, but the syscalls are the real challenge).

CNCF is a big deal

To be fair, I’d never heard of CNCF, but they’re such a professional non-profit and the parent company of Kubernetes, Prometheus, and many other projects. They shared some thoughts on how they’re not too concerned if some individual contributors disappear for reasons like family commitments, but the bigger issue arises when projects are maintained by a single entity that can just reorganize overnight. Apache Arrow, is that you?

Keep up the good work, CNCF!

Distributed Systems are (again) notoriously hard to build

“Was Leslie Lamport Right?” — that’s the title of the main talk in the distributed systems track. Apparently, the legendary analyst and educator in distributed systems even before he had a distributed system at home. I learned a bit about Lamport clocks and how distributed systems sync in discrete and relative time, not wall-clock time. It made me think that offloading as much state as possible to S3 might not be such a bad idea, and that building one should be (and remain) a highly rewarding side project.

FSFE is also a big deal

Massive respect for the folks who have been advocating for nearly 20 years for extendable and replaceable software. Although the presenter stirred the pot a bit with the audience — some folks didn’t buy the idea that mobile device software is not modifyable — let’s take Apple, FSFE’s current target - sure, you can change iOS into something open-source, but that’s only feasible if you’re a 10x hobbyist engineer, not regular folk IMHO. Best of luck to them!

Conclusion

The conference beat to dust every commercial conference I have ever been to. It was an eye-opening experience tech-wise and great for meeting old and existing colleagues for some networking. Brussels is the perfect setting for it as well - well connected and just the right European vibes. Great food and drinks too…

Will return next year.

I’ll give it a few years until MLflow dominates the model package format space, with alternatives like SageMaker models fading away, and sharing pure weights becoming an arcane art. But until that dominance is absolute, I’ve been thinking that there’s another quite obvious way to package models: just store them as wheels. Packaging == persisting the trained model.

Let’s create a project to train a model:

$ uv init --lib model
$ mkdir registry # the model registry

Within the model package (src/model/main.py), we’ll a create dummy decision tree model, with typical train/predict interface:

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
import joblib
from pathlib import Path
from sklearn.preprocessing import StandardScaler
import numpy as np
from sklearn.pipeline import Pipeline

def train():    
    # Load dataset
    iris = load_iris()
    X, y = iris.data, iris.target

    # Split the dataset into training and testing sets
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

    # Create a pipeline with a scaler and a decision tree classifier
    pipeline = Pipeline([
        ('scaler', StandardScaler()),
        ('classifier', DecisionTreeClassifier())
    ])

    # Train the pipeline
    pipeline.fit(X_train, y_train)

    # Dump the trained pipeline to a file
    Path(Path(__file__).parent / 'data').mkdir(parents=True, exist_ok=True)
    joblib.dump(pipeline, Path(__file__).parent / 'data' / 'model.pkl')
    
    print("Pipeline trained and saved to model.pkl")

def predict():
    # Load the trained model
    model = joblib.load(Path(__file__).parent / 'data' / 'model.pkl')
    print("Model loaded from model.pkl")

    # Make a prediction
    X_new = [[6.1, 2.8, 4.7, 1.2]]
    y_new = model.predict(X_new)

We pin the model dependencies and the Python version (pickling should be backward compatible):

[project]
name = "model"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
authors = [
    { name = "Lukas Valatka", email = "<email>" }
]
requires-python = ">=3.9"
dependencies = [
    "pandas == 2.2.3",
    "scikit-learn == 1.6.1",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

Train the model:

$ uv run python -c "from model.main import train; train()"

A /data folder should appear within src/model, containing the trained model. This is where the magic happens—we bundle the trained model together with the source code. (Yes, it’s a bit of an abomination, but bear with me.)

Register the model:

$ uv build # this step should be part of a training pipeline
$ mkdir ../registry/model
$ mv dist/* ../registry/model

Make our fake model registry go live:

$ cd ../registry
$ python -m http.server 8000 # package index is just /package folders served

that’s it - we should be able to pull-in our trained model and predict:

$ uv run --with model==0.1.0 --index-url http://127.0.0.1:8000 --extra-index-url 'https://pypi.python.org/simple' --index-strategy unsafe-best-match python -c "from model.main import predict; predict()"

we can also add our model to a serving application, as follows:

pyproject.toml

[project]
name = "serving-app"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
authors = [
    { name = "Lukas Valatka", email = "<email>" }
]
requires-python = ">=3.9"
dependencies = [
    "model == 0.1.0",
    "fastapi[standard]",
    "pandas",
    "pydantic",
]

[project.scripts]
serving-app = "serving_app:main"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[[tool.uv.index]]
name = "local"
url = "http://localhost:8000"

all dependencies such as scikit-learn and pandas are at exact versions they were used for training, for full reproducibility. Also reflected within the lockfile - for reproducible builds!

If you try to create the venv within an incorrect Python environment e.g. Python 3.8, it will complain:

$ uv run --python 3.8 --with model==0.1.0 --index-url http://127.0.0.1:8000 --extra-index-url 'https://pypi.python.org/simple' --index-strategy unsafe-best-match python

  × No solution found when resolving `--with` dependencies:
  ╰─▶ Because the current Python version (3.8.20) does not satisfy Python>=3.9 and pandas==2.2.3 depends on Python>=3.9, we can conclude that pandas==2.2.3 cannot be used.
      And because model==0.1.0 depends on pandas==2.2.3 and you require model==0.1.0, we can conclude that your requirements are unsatisfiable.

So, how does this all compare to the all mighty mlflow model format:

• With MLflow, you need to explicitly list model dependencies in your serving-app OR have some script automating this (pull in model, uv add deps). It’s not a native functionality.
• How do you make sure Python version is compatible between training and serving? Need to manually check this.

Just a thought experiment. I still believe MLflow is the go-to format due to its flavors, and storing large files within your Nexus will make SREs frown big time.

But who knows? Maybe in the future, MLflow might just consider disguising models as standard Python packages—achieving both framework flexibility via its flavors and interoperability with the Python ecosystem.

EDIT 2025-02-19 - it appears Simon Willison himself put an LLM in package. Same way, pip install the package+model and run with uvx. So maybe the wheel format is not that of a bad idea after all? 🤔 Kudos to my colleague Lode for sharing this.