There are many approaches you could take to implement a query language. I chose Python because I enjoy working with it, and because it’s a very popular language in the data processing space. I also knew upfront that I wanted my custom language to somehow run on top of Apache Spark because I work with it every day and the data I was interested in querying was already stored as Parquet and registered in a central catalog. That context alone greatly reduces the space of possible implementation approaches, and that’s a great thing! I didn’t need (or want) to design a custom storage format or query engine, both of which are major, major undertakings. I just needed the ability to compile my custom language down into queries that Spark could then execute for me.

If you’re implementing your own language, there are broadly three steps you’ll need to take:

1. Define a grammar, usually in some form of EBNF.
2. Use that grammar to construct a parse tree from the source text.
3. Transform that parse tree into whatever final form your program should take.

EHQL has a number of interesting features, but it would be too much to walk through implementing them all in this post. Instead, I’m going to focus on the following example query and use it to implement a skeleton of the language that will nonetheless demonstrate the key technologies and techniques involved.

-- This is an example EHQL query.
history contains:
  "oil change"
  "transmission fluid change"

This query finds all vehicles in our database that have had both an oil change and transmission fluid change at any time in their maintenance history.

Here is a self-contained solution you can run locally to follow along as I build out the implementation.

Defining the Grammar

If you’re working with Python, the right library to use to define your grammar is Lark.¹ I believe in the JVM ecosystem most people use ANTLR. ANTLR does have a Python implementation, but a quick survey of both parsing libraries tells me that Lark is faster, more popular, and more feature rich. And while I personally haven’t worked with ANTLR to compare, I can say that I enjoyed working with Lark and didn’t feel its design was obtuse or constraining in any way.

Here is the Lark grammar for the minimal skeleton of EHQL that we are walking through in this post:

# ehql.lark
%import common.ESCAPED_STRING -> QUOTED_STRING
%import common.WS_INLINE
%import common.SQL_COMMENT

%ignore WS_INLINE
%ignore SQL_COMMENT

%declare _INDENT _DEDENT

?start: _NEWLINE* history_clause

history_clause: "history" "contains" ":" _history_body
_history_body: _NEWLINE _INDENT [history_pattern _NEWLINE]+ _DEDENT
history_pattern: event_name
event_name: QUOTED_STRING

_NEWLINE: (/\r?\n[\t ]*/ | SQL_COMMENT)+

Lark grammars are expressed in EBNF, with some added features specific to Lark. The %import statements at the top, for example, are importing predefined rules that Lark offers so we can use them in our own grammar. When you import a rule, you can also choose to rename it, which is what ESCAPED_STRING -> QUOTED_STRING is doing. start is the default rule Lark will look for as the root of the grammar, though you can rename it. The leading ? in ?start tells Lark to exclude it from the parse tree if it has a single child, which it does.

By convention, rules in block capitals are typically terminals, which basically means rules that don’t in turn break down into more rules. In other words, terminals match some text and stop there.

• QUOTED_STRING matches strings with double quotes like "hello".
• WS_INLINE matches inline whitespace like spaces and tabs, but not newlines.
• SQL_COMMENT matches SQL-style comments like -- this is a comment.

The two %ignore statements tell Lark to ignore text that matches those terminals. This is basically how we implement support for comments and make our language robust to trivial whitespace differences. For example, history contains: and history    contains: will both parse the same thanks to %ignore WS_INLINE.

Handling Significant Indentation

EHQL is like Python in that it uses indentation to logically group statements together. The way this is implemented in the grammar is via these statements:

%declare _INDENT _DEDENT

history_clause: "history" "contains" ":" _history_body
_history_body: _NEWLINE _INDENT [history_pattern _NEWLINE]+ _DEDENT

_NEWLINE: (/\r?\n[\t ]*/ | SQL_COMMENT)+

The key structure is captured by _history_body, which contains one or more history patterns, each separated by a newline. The patterns are all indented one level relative to the parent history_clause. The _NEWLINE terminal is a bit interesting because it matches whitespace after the newline as well as comments; this is crucial for Lark to handle indentation and inline comments correctly. Also interesting are the _INDENT and _DEDENT terminals. They don’t have definitions in our grammar but they are declared and then used as part of _history_body. This is another consequence of how Lark handles languages with significant indentation, specifically.

Why all the strangeness with how _NEWLINE, _INDENT, and _DEDENT are defined? The gist of it is this: Parsing text into a structured syntax tree generally happens in two stages, lexing and parsing. Lexing a text means scanning the raw characters and converting them into tokens, which would be matched by the terminals of our grammar (i.e. the rules in block capitals). Parsing takes the tokens we generated and composes them into a structured tree based on the rules of our grammar.

When the lexing stage encounters a newline, the number of _INDENT or _DEDENT tokens it needs to generate will depend on what happened before and what happens after the newline. We won’t be implementing support for any and all in this post, but consider this example:

-- After each newline, does the lexer need to indent or dedent?
history contains: -- indent
  any of: -- indent
    "oil change" -- no indent or dedent
    "transmission fluid change" -- dedent 2x

In other words, the lexer needs to maintain some state about the indentation level as it goes through the text if it’s going to generate indent and dedent tokens correctly. But lexers are typically stateless, so they cannot track this kind of information. I believe they are designed this way mainly so they can be simple and fast. So the way Lark addresses this problem is with a postlex processor that runs after lexing but before parsing. The postlexer does track some state and generates the indent and dedent tokens we need.

Lark offers a dedicated postlexer for significant indentation. All we need to do is subclass it and tell it which of our grammar’s terminals correspond to the key newline, indent, and dedent tokens.

from lark.indenter import Indenter


class EHQLIndenter(Indenter):
    NL_type = '_NEWLINE'
    OPEN_PAREN_types = []
    CLOSE_PAREN_types = []
    INDENT_type = '_INDENT'
    DEDENT_type = '_DEDENT'
    tab_len = 2

If EHQL supported breaking up long expressions across multiple lines using parentheses, we’d also have to specify the names of those tokens. That’s because the indentation level would have to be frozen while inside the parentheses, regardless of what leading whitespace there was on each line. Since EHQL doesn’t support this, we can leave the *_PAREN_types parts of EHQLIndenter empty.

Finally, the tab_len parameter is there to translate the tab character \t into the appropriate number of spaces when determining indentation level. Note it is not there to tell Lark how many spaces translate into a level of indentation, which is something I was initially confused by. The indentation level of a given line is determined automatically by the postlexer based on the number of leading spaces that line has relative to the previous line. There is no global rule that translates a fixed number of spaces into a specific indentation level. This means that separate blocks of code can be written with different numbers of leading spaces but parse to the same indentation level, just like in Python. In practice, EHQL’s convention is to use two spaces per indentation level, as compared to Python’s four.

Constructing the Parse Tree

Alright! We have our grammar, and we have the appropriate instructions for the postlexer that’s going to handle EHQL’s indentation. Now we can put them together and parse our example EHQL query into a parse tree.

from lark import Lark
from pathlib import Path
from textwrap import dedent

grammar = Path("ehql.lark").read_text()
parser = Lark(grammar, parser='lalr', postlex=EHQLIndenter())
query = dedent(
    """
    -- This is an example EHQL query.
    history contains:
      "oil change"
      "transmission fluid change"
    """
)
parse_tree = parser.parse(query)
print(parse_tree.pretty())

First, a brief note about parser='lalr'. Lark supports a few parsing algorithms that you can choose from when parsing your grammar. The main two are Earley and LALR(1). Earley is slower but much more flexible in what grammars it can handle. However, LALR(1) is powerful enough to parse “real” languages like Python and Java, so unless you’re doing something exotic I would start with LALR(1) and only switch to something else if necessary. Notice also that, since our language has significant indentation, we had to pass our indenter specification to the postlex parameter.

Running the above will parse our sample query and print the following parse tree:

history_clause
  history_pattern
    event_name  "oil change"
  history_pattern
    event_name  "transmission fluid change"

Now that’s pretty neat! We can see nodes in this tree that correspond to our grammar rules. Comments and inline whitespace have been correctly ignored thanks to the %ignore directive. Newlines and indentation tokens have been correctly converted into the appropriate structure and then dropped from the parse tree. This is important because we don’t want or need to see tokens like _NEWLINE, _INDENT, or _DEDENT in the tree itself; we just need event_name to be a child of history_pattern, and history_pattern a child of history_clause, etc. This is a key detail of our Lark grammar: Rules that start with an underscore are parsed into the appropriate structure but then discarded so they don’t litter the final parse tree.

Converting the Parse Tree into a Query

At this point, we’re ready to convert this parse tree into a query that Apache Spark can execute for us. Spark has a broad set of capabilities, so “execute a query” can take many different forms. In the case of EHQL, what it means is to execute a DataFrame query over Spark Connect.

Drafting a Target Solution

In my post on designing EHQL, I worked through a reference query in SQL. Spark can execute SQL just fine, but I prefer to work with its DataFrame API. The main reason is that the DataFrame API is accessible in the host language – in our case, Python – so you get proper IDE support like you would with any library, rather than having to embed or awkwardly build SQL strings. The DataFrame API also offers a fluent interface that makes it natural to chain operations, similar to how you would in Unix shell programming.² Spark compiles both SQL and DataFrame queries down into the same intermediate representation and executes them with exactly the same optimizer, so there is no runtime difference between using these different interfaces. It’s all about what interface works better for your use case.

Another key decision I made was to use Spark Connect to interface with Spark, rather than the classic DataFrame API. The DataFrame API is the same, but the difference is that whereas before your client would need to run a fairly heavy JVM to host the driver for your Spark application, with Spark Connect your client becomes a thin and lightweight layer that doesn’t know anything about the JVM or Spark’s internals. Instead, it just builds logical query plans from your DataFrame queries, sends them over the wire to the server, and gets the results back. This makes working with the DataFrame API much more like working with SQL in that your client is fully decoupled from the server and can be embedded basically anywhere, even on the most resource-constrained hosts.³ And the best part is that using Spark Connect doesn’t require anything other than a small change in how we initially connect to Spark, as you’ll see just below.

What does our query look like in the DataFrame API? Porting the original SQL from my previous post is pretty straightforward.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, bool_or


# `.remote()` is what creates the session using Spark Connect.
spark = SparkSession.builder.remote("local[*]").getOrCreate()

(
    spark.table("maintenance_history")
    .withColumns({
        "has_oil_change": col("work_done") == "oil change",
        "has_tx_fluid_change": col("work_done") == "transmission fluid change",
    })
    .groupBy("vehicle_id")
    .agg(
        bool_or("has_oil_change").alias("has_oil_change"),
        bool_or("has_tx_fluid_change").alias("has_tx_fluid_change"),
    )
    .where(
        col("has_oil_change")
        & col("has_tx_fluid_change")
    )
)

This, roughly, is what we want to convert the above parse tree into.

Choosing the Right Lark Visitor

Lark offers a few different ways of transforming your parse tree into its final form. These ways differ on two different axes:

• They process the tree top down vs. bottom up.
• They reconstruct the tree vs. edit it in place.

Our language skeleton is simple enough that it makes most sense to use Lark’s Transformer, which processes the tree bottom up, building a new tree as it does so. If we were implementing more of EHQL’s design we might want to process the tree differently. Consider event aliases. Since aliases used in one part of a query are defined in another part, we may need to process the tree top down or even in multiple passes to build our final query.

Building the Query Bit by Bit

Step 1: event_name

Let’s start with a very simple implementation to show you how a Transformer works in practice.

from lark import Transformer, v_args


class EvaluateEHQLSkeleton(Transformer):
    @v_args(inline=True)
    def event_name(self, quoted_string: str):
        return quoted_string.strip('"')


transformed_tree = EvaluateEHQLSkeleton().transform(parse_tree)

print(parse_tree.pretty())
print(transformed_tree.pretty())

This will print the initial parse tree followed by the result of transforming that tree using the EvaluateEHQLSkeleton transformer.

history_clause
  history_pattern
    event_name  "oil change"
  history_pattern
    event_name  "transmission fluid change"

history_clause
  history_pattern       oil change
  history_pattern       transmission fluid change

There are a few things going on here – and this is a good snippet to tinker with if you are following along – but the most important to call out are the following:

• The event_name method in our transformer is automatically matched by Lark to the corresponding grammar rule, which is very neat. Since we didn’t define methods for the other rules, the corresponding tree nodes remain unchanged.
• Note how our event_name method strips the double quotes from each event name. We get a new tree with those nodes replaced by the output of that method.
• The v_args(inline=True) decorator simplifies the logic of our method a bit since normally it would take as input a list of the node’s children. Since we know the event_name node has exactly one child – the quoted string capturing the event name – the decorator simplifies the method’s interface so we can accept the string directly.

Step 2: history_pattern

Let’s go to the next level up our parse tree by filling in a definition for history_pattern:

import pyspark.sql.functions as sqlf
from lark import Transformer, v_args


class EvaluateEHQLSkeleton(Transformer):
    @v_args(inline=True)
    def history_pattern(self, event_name: str):
        event_name_slug = event_name.lower().replace(" ", "_")
        condition_name = f"__has_{event_name_slug}"
        return (
            condition_name,
            sqlf.col("work_done") == event_name,
        )

    # Include methods from previous steps.
    ...

Running the transformation with this updated definition now gives us this before and after of our parse tree:

history_clause
  history_pattern
    event_name  "oil change"
  history_pattern
    event_name  "transmission fluid change"

history_clause
  ('__has_oil_change', Column<'==(work_done, oil change)'>)
  ('__has_transmission_fluid_change', Column<'==(work_done, transmission fluid change)'>)

Do you see how the parse tree is slowly turning into a set of Spark data structures? All we did in our history_pattern rule is create Spark column expressions for each of the maintenance events in our query, and we paired those with aliases in a plain Python tuple. One more step and we’ll have the DataFrame that we want.

Step 3: history_clause

Now we can piece everything together at the root of the tree and build our final query:

import functools
import operator
import pyspark.sql.functions as sqlf
from lark import Transformer, v_args


class EvaluateEHQLSkeleton(Transformer):
    def __init__(self, spark):
        self.spark = spark

    # We don't use `v_args` here because there are a variable number of
    # patterns and it's best to accept them as a list.
    def history_clause(self, history_patterns):
        return (
            self.spark
            .table("maintenance_history")
            # This adds the columns like `__has_oil_change` which we
            # defined in `history_pattern()`.
            .withColumns({
                condition_name: column
                for (condition_name, column) in history_patterns
            })
            .groupBy("vehicle_id")
            # This aggregation with `bool_or` helps us identify if a vehicle
            # had a particular event anywhere across its maintenance history.
            .agg(*[
                sqlf.bool_or(sqlf.col(condition_name)).alias(condition_name)
                for (condition_name, _) in history_patterns
            ])
            .where(
                # PySpark uses the `&` bitwise operator to `AND` conditions
                # together. So this breaks down to something like:
                #     col("condition1") & col("condition2") & ...
                functools.reduce(
                    operator.and_,
                    [
                        sqlf.col(condition_name)
                        for (condition_name, _) in history_patterns
                    ]
                )
            )
        )

    # Include methods from previous steps.
    ...


result = EvaluateEHQLSkeleton(spark).transform(parse_tree)
# The result is now a DataFrame, so we call `.show()` instead of `.pretty()`.
result.show(truncate=False)

This final step is a bit long because this is where we piece everything together into the final result, but it should be easy to follow as it maps directly to the original SQL I wrote for this use case.

We’ve fully transformed the original parse tree into a DataFrame query that returns the vehicles we’re looking for:

+----------+----------------+-------------------------------+
|vehicle_id|__has_oil_change|__has_transmission_fluid_change|
+----------+----------------+-------------------------------+
|3344      |true            |true                           |
+----------+----------------+-------------------------------+

And that’s it! We have a working skeleton implementation of EHQL that runs simple queries by converting them into DataFrames and running them on Spark.

Building out the Rest of EHQL

EHQL has a lot of features that we didn’t cover here. To implement them, we’d need to add the appropriate rules to our grammar; for each rule, we’d need a matching implementation in our Lark visitor that describes how to convert the parsed tokens returned by that rule into the right Spark code. It may not be possible or desirable to keep the implementation in a single Transformer class; we may need multiple passes to build the right query, or we may want to first build an abstract syntax tree (AST) before converting it into the final query. An AST would more directly capture the semantic structure of a query versus the grammatical structure returned by the parse tree. For our skeleton of EHQL, those two things are very similar, but as we build out more of the language it may make more sense to construct a proper AST before building the final DataFrame.

I hope that this walk through of designing and implementing EHQL inspires you to try your hand at making your own custom query language. I may try to build out more of EHQL and release it as a proper demo project. But for now, here is a standalone script that captures everything we walked through in this post. Try it out!

I can’t share the exact project since it was in service of a group at my then-employer. But in this post I will walk through a broadly equivalent hypothetical scenario and the design process I used: mapping out the problem, trying existing solutions, and finally designing my own query language.

Understanding the Need

First, why would you want to design a custom query language? There are many possible reasons; in my case, the decision to take this on emerged from an understanding of three things:

1. The user base I’m trying to help.
2. The data these users want to query.
3. These users’ use cases.

I imagine this combination of factors is what drives the creation of most query languages.

User Base: Analysts

Our target user base is non- and semi-technical analysts specializing in vehicle maintenance data.

The non-technical analysts may have experience with something like Excel and may know how to write basic formulas for that system, but they are not familiar with SQL or programming otherwise.

The semi-technical analysts know a fair amount of SQL but are not professional software developers. That is, they know how to write basic queries that involve joins or aggregates, but they’re not as comfortable using more advanced techniques like window functions or correlated subqueries.

Data: Vehicle Maintenance Logs

The data these users are interested in is vehicle maintenance data being streamed into a central database. The data is updated daily and spans 10+ years, covering 100 million vehicles across the United States.

While the implementation details of how exactly the data is stored² are interesting – the format, the storage system, partitioning scheme, etc. – what matters most for our purposes is the logical or abstract schema of the data.

This data can be described abstractly as two tables with the following schemata:

vehicle
    vehicle_id
    make
    model
    year

maintenance_history
    vehicle_id
    time
    work_done
    detail

Example “rows” from each table:

vehicle:

maintenance_history:

Use Case: Defining Vehicle Populations

Our target user base is interested in finding vehicles that fit a specific profile and maintenance history, mainly to market products or services to the owners of those vehicles. A common industry term for this is vehicle population.

Here are some example vehicle populations:

1. All vehicles that have had an intake upgrade, exhaust upgrade, and their fuel/air ratio reconfigured (i.e. rice rockets).
2. Toyotas that threw a P1155 diagnostic code three or more times but have not had an oxygen sensor replaced yet.
3. Vehicles that had a transmission fluid change and then, within six months, had a full transmission rebuild.

There are many more use cases I brainstormed for my target user base. However, wanting to constrain and focus my design, I distilled them down to the above small set of example cases that I felt best captured my users’ needs. These are the reference queries that I used to guide my design process.

I don’t know if this is how query languages are typically designed. But I can say that this process of profiling my users and distilling their needs into a focused set of queries was the most important exercise I did before designing the language. It grounded my work and gave me something to test design ideas against.

Is there an Existing Solution?

This can’t be the first time someone has encountered a problem like this. Maybe there’s something out there we can just pick up off the shelf and use.

First Stop: SQL

My natural – and correct, in my humble opinion – impulse was first to wonder if I could just use SQL. The data is already structured as tables. And SQL is a robust, flexible query language that is widely used and has stood the test of time. Perhaps all I needed to do was create a few views to make certain query types easier to express and that would be good enough.

So here I worked through another exercise which I found to be just as critical to my design process as the previous exercise: I expressed my reference queries in SQL to see what they would look like. Was there a reusable pattern to these queries? Were the queries themselves simple enough to encapsulate somehow so that our semi- and non-technical users could write them? Ultimately, do I really need to bother developing a custom query language or not?

Consider the first reference query we documented earlier:

All vehicles that have had an intake upgrade, exhaust upgrade, and their fuel/air ratio reconfigured.

This is a query for vehicles of any type that have three separate events in their maintenance history. The events can be in any order; we just want to see that all three are present for a given vehicle in order to return it in the result set for this query. A large chunk of my users’ queries – about half – are roughly of this nature, so any solution I came up with needed to work really well for this simple use case.

My first attempt at writing this in SQL looked something like this:

SELECT vehicle_id
FROM maintenance_history
WHERE
        work_done = 'intake upgrade'
    AND work_done = 'exhaust upgrade'
    AND work_done = 'fuel/air ratio reconfig'
;

This query is wrong, but it’s the kind of query I want to be able to write to satisfy this use case. It’s wrong because we’re looking for three separate rows with the same vehicle_id that have these three different values for work_done, whereas the above query tries to find a single row that matches all values, which obviously won’t work. Changing the ANDs to ORs won’t help either because that will find vehicles that have any of these events in their maintenance history, whereas we want vehicles that have all of them.

This process of fumbling through what I feel like I want to be able to write turned out to be another key in helping me design a better solution. We’ll come back to this point later.

A correct SQL query for this use case would be:

SELECT vehicle_id
FROM maintenance_history
WHERE work_done IN (
    'intake upgrade',
    'exhaust upgrade',
    'fuel/air ratio reconfig'
)
GROUP BY vehicle_id
HAVING COUNT(DISTINCT work_done) = 3

This works but it’s inflexible. If we slightly change our query to search for cars that upgraded either their intake or exhaust (or both), in addition to reconfiguring their fuel/air ratio, we can’t use this query pattern anymore. That’s because the aggregation, as it is structured here, destroys the information about what specific work was done on the vehicle.

We want to aggregate each vehicle’s maintenance history so we can query across the entire history at once, without losing the detail that enables us to express finer conditions like “had this or that” or “had this and then that”.

After some experimentation, this is the query pattern I came up with:

WITH history AS (
    SELECT *,
        work_done = 'intake upgrade' AS has_intake_upgrade,
        work_done = 'exhaust upgrade' AS has_exhaust_upgrade,
        work_done = 'fuel/air ratio reconfig' AS has_ratio_reconfig
    FROM maintenance_history
    -- This filter here is important for performance but is not required
    -- for the correct result.
    WHERE work_done IN (
        'intake upgrade', 'exhaust upgrade', 'fuel/air ratio reconfig'
    )
)
SELECT
    vehicle_id,
    BOOL_OR(has_intake_upgrade) AS has_intake_upgrade,
    BOOL_OR(has_exhaust_upgrade) AS has_exhaust_upgrade,
    BOOL_OR(has_ratio_reconfig) AS has_ratio_reconfig
FROM history
GROUP BY vehicle_id
HAVING
        has_intake_upgrade
    AND has_exhaust_upgrade
    AND has_ratio_reconfig

Obviously, this solution is lengthier than the previous one. But as a query pattern it breaks our query down into two logical stages: first, we extract the key attributes of each vehicle’s history (the various BOOL_OR(...) parts), and then we filter on those attributes (in the HAVING clause). This separation makes the pattern quite flexible.

For example, let’s revisit the variation on the first query that we considered:

cars that upgraded either their intake or exhaust (or both), in addition to reconfiguring their fuel/air ratio

This can be expressed with largely the same query as above. All we have to change is the HAVING clause:

HAVING
        (has_intake_upgrade OR has_exhaust_upgrade)
    AND has_ratio_reconfig

We’ve worked through only the first reference query, but it’s already striking how verbose and cumbersome our SQL solution is compared to the plain English description of our target vehicle population. Other queries may have aspects that don’t cleanly fit the pattern we came up with, and more complex queries will of course require even lengthier SQL to express.

While I worked through translating about a half dozen queries into SQL as part of my exploration of the problem, I won’t go through that all here. But my conclusion was that I couldn’t see how I would simplify this for users who would find this amount of SQL intimidating or downright inaccessible. Each query needs to extract different attributes from the maintenance history, so somehow restructuring the data in a set of views to make queries simpler doesn’t seem like a viable approach.

Why not use an LLM?

So the SQL is lengthy. So what? LLMs are very good at writing code for constrained problems like this. We could just give an LLM the plain English descriptions of the vehicle populations we want and let it build the SQL automatically. It’s an attractive idea. For some, an LLM and a thin interface are a good enough replacement for a custom query language. For this situation, however, LLMs don’t work as a complete solution.

Natural English doesn’t have the precision of a programming language, so an LLM first has to interpret potentially ambiguous descriptions. Did it interpret the description correctly? Someone has to validate the generated SQL. As we saw in our walkthrough of the first reference query, the SQL is much more verbose than the English description it captures. And many queries, like our other two reference queries, will be more complex than that.

Over the course of a year, these analysts write hundreds – perhaps even thousands – of queries describing vehicle populations. These populations are designed based on the analysts’ business knowledge and conversations with clients. The analysts need to understand and defend the details of each population they build. How can they do that if they cannot understand the SQL being generated? Even the semi-technical analysts who do know some SQL would quickly fatigue from having to review so much generated SQL day in and day out.

In other words, the problem is mainly one of accountability and accessibility. With an accessible query language tailored to the business problem at hand, these analysts can own their work. A custom language would also make LLMs much more useful to them than if they were working with SQL. They’d be able to generate and debug queries that they can actually understand. And with a query language that’s much more constrained than SQL, LLMs are much more likely to get translations to and from English correct on the first try.

LLMs don’t change this fundamental fact: Programming is thinking, and thinking clearly is much easier when you’re working with the right abstractions. Our analysts, whether they think of themselves as such or not, are doing a form of programming.

Other Prior Art

I did two rounds of research into prior art. The first took place before I started work on my design in late 2024. At the time I wasn’t sure what to look for or how to explain the problem in a domain-agnostic way that wasn’t tied to the topic of vehicle maintenance. I was already familiar with Cypher and GraphFrames. Though my problem wasn’t a graph problem, I did take inspiration from their syntaxes even if I didn’t use either directly.

I remember searching for “time series analysis” and “time series query language”, as I thought that was the closest description to what I was doing. But the results seemed to have a focus on statistical analysis rather than pattern matching, which I would later understand was a better description of what I wanted. One interesting find from this round of searching was Kusto, whose first-class pipelining reminds me of PRQL.

In the course of writing this post I did another round of prior art research, this time guided by my experience designing and implementing my new language, and also using the assistance of an LLM. I found more relevant results this time, first among them being the term Complex Event Processing (CEP), which seems to be an industry term that best describes our problem domain. There are a lot of CEP engines out there. Some of them are proprietary; most extend SQL or offer a DSL that is very SQL-like.

A surprising find in this area was SQL’s MATCH_RECOGNIZE, which had somehow escaped my notice before. Oracle first released this feature in 2013 and it was later standardized as part of SQL:2016. This seems to be SQL’s solution to the problem of event pattern matching. It’s not widely supported as of early 2026 (including neither by PostgreSQL nor by Apache Spark) and, most importantly, it doesn’t simplify the expression of our reference queries enough to make them accessible to my users. But it’s an interesting development nonetheless!

The most interesting find I made while researching CEP engines for this post is Elasticsearch’s Event Query Language (EQL). It’s designed primarily for threat detection, but for some of our reference queries it seems to fit the problem quite well. If my company was already using Elasticsearch it would have been worth digging a bit into EQL as a solution, or at least as a starting point for one.

The last bit of prior art I want to make note of is Logica. It’s not a CEP engine, but rather a logic programming language in the Datalog family. What always impresses me about Datalog is how elegantly it expresses ideas that take much more prose to replicate in SQL. And what’s special about Logica in particular is that it compiles programs into SQL that you can run at scale (though Apache Spark is currently not one of its supported dialects). If I were starting over today, I would look into the possibility of building a thin wrapper around Logica as the DSL for my users.

Distilling the Essence of the Problem

In looking for an existing solution to the problem I was trying to solve for my intended users, I started in the right place: SQL, the de facto universal query language. And while at the time I took on this work I did not have the insight that I was looking for a CEP engine (or something like that), the process of talking to users and implementing their examples in SQL gave me insight into the “essence” of the queries. It wasn’t something that hit me suddenly like an “a-ha!” moment; it was more like seeing a shape emerge from the mist and progressively get clearer until you could tell what it was.

The analysts are interested in querying vehicle maintenance data on two different axes: one is the vehicle itself—that is, its properties like make, model, and so forth; the other is the vehicle’s maintenance history, which is a timeline of maintenance events like “oil change on this date”, “tire rotation on that date”, and so on.

The underlying representation of the data is, in fact, a separate matter from this abstract model of how the analysts think about their domain and phrase their queries. The closer the new query language tracks this abstract model, independent of the underlying data storage picture, the more natural analysts will find it.

Of course, this type of problem is not unique to vehicle maintenance data. Anything that has a history could fit this model: sensors that take measurements (e.g. temperature, humidity, etc.), people making purchases, even volcanos erupting. The key is that we have an entity which has some kind of history.

The analysts’ queries against a vehicle’s history look for a combination of events or event patterns across the entire history. This is also key, as other query languages may facilitate querying for individual events or querying across multiple entities, which isn’t the focus here.

As I thought about how best to express the event patterns these users wrote, I found it useful to sketch out maintenance histories on a timeline. When I sketched this out on paper, the image that came to mind was that of overlaying pattern strips on the timeline to see if the timeline as a whole matched what the query was looking for.

The physicality of this image synthesized so many thoughts and ideas – ranging from fuzzy to precise – into a concrete picture for me. It also gave me inspiration for a suitable syntax, which I’ll get to later in this post.

Before that, we must first carry out that most enjoyable of tasks when creating something new, and give our new query language a name. The obvious choice is Vehicle Query Language (VQL), but a more general name that captures the core abstractions and query patterns would be Entity History Query Language (EHQL).

Designing the Syntax

At this point I felt I had a good command of the problem, having worked through a clear problem description, reference queries, example implementations of some of those queries, and an abstract data structure. It was a lot of prep work, but at this point I was finally ready to design the syntax itself.

The central problem of designing a query syntax was coming up with something approachable for my semi- and non-technical users, while still offering something expressive and precise enough to cover their typical query needs. The plain English of the reference queries was already quite good—expressive, easy to understand, and well-suited to the domain of describing vehicle populations. It’s also much more concise than the equivalent SQL we had to write. What it lacked was the structure and precision of a programming language.

So the problem becomes: Starting with the plain English queries, can we add just enough structure to make them precise, while preserving the clarity and ease-of-use? Well, my first attempt at writing an SQL query got close to capturing exactly that. So I stripped it down a bit:

-- Reference Query 1: All vehicles that have had an intake upgrade,
--   exhaust upgrade, and their fuel/air ratio reconfigured.
maintenance_history WHERE
        work_done = 'intake upgrade'
    AND work_done = 'exhaust upgrade'
    AND work_done = 'fuel/air ratio reconfig'

Since we’re designing for a narrow use case, we don’t need SQL’s generality; every degree of freedom is a degree of complexity. Instead of general tables and relations, our query language only needs to interact with two data abstractions, which we can elevate into keywords: vehicle and maintenance_history. In this case each abstraction does happen to map to an underlying table of data, but this is not directly relevant to the design of the query language. The language is expressing filters on data abstractions which are decoupled from whatever underlying storage implementation we have. When we come to implement our language we’ll have to bridge this gap between data abstraction and concrete storage, but that’s for later.

Finally, we can just drop the entire SELECT clause since we are just defining vehicle populations and each query would return a set of distinct vehicle IDs. If our language needed the ability to select specific vehicle attributes, or perhaps even select the events that matched the specified history patterns (which is an interesting problem), then we would need some syntax to express that. Saying “no” to this capability for now makes our design job easier, and we may end up never needing this capability anyway.

Significant Indentation

If my users could write a query like this, that would already be a big win. But I wanted to simplify this further based on an important piece of feedback I had gotten from users about their experience working with SQL and other custom query languages at the company: They hated interacting with long or deeply nested lists of conditions or values. These lists demanded careful placement or balancing of binary operators, commas, and parentheses. One user, a semi-technical analyst, commenting on a fairly lengthy query that required a lot of perfectly balanced parentheses, called it “disgusting”.

This is where I thought to borrow the most distinctive feature of Python: significant indentation. The indentation would group conditions or values in a clear visual block that was easier to read and required less fidgety syntax to get right.

-- Reference Query 1: All vehicles that have had an intake upgrade,
--   exhaust upgrade, and their fuel/air ratio reconfigured.
history contains:
  work_done = "intake upgrade"
  work_done = "exhaust upgrade"
  work_done = "fuel/air ratio reconfig"

I made some other changes that seemed natural to me:

• Just history, which is more concise but still clear from the context that it refers to the maintenance history.
• contains instead of where since we are searching across the entire history and the former captures the semantics of the filter more accurately.
• Double quotes for string literals instead of single quotes, since that’s more common. I considered allowing both, and if users found this choice to be a sticking point I could revisit it. But it’s better to have a single way to do this.

Instead of significant indentation, I considered going with bulleted lists:

history contains:
  - work_done = "intake upgrade"
  - work_done = "exhaust upgrade"
  - work_done = "fuel/air ratio reconfig"

This would probably also be very accessible to my users as it leverages a common document formatting idiom, but the bullets add visual noise and my users and I liked the pure whitespace version better.

One reasonable concern about significant indentation for a custom query language is that queries are likely to be embedded in another language that does not have significant indentation. The argument is that this leads to bugs over time as the indentation is changed in the surrounding language, which subtly changes the semantics of the embedded, indentation-sensitive query. This concern doesn’t apply in our case since the query language was designed with its own execution environment in mind and designed specifically for non-programmers.

Against Boolean Operators

Alright, so our new language is starting to take shape. Let’s consider the variation on our first reference query, where two conditions are or-ed together. How do we do that with significant indentation? Do we need to reintroduce the parentheses and boolean operators that our users complained about? Not quite.

Continuing a theme, I adopted Python’s all and any built-in functions.

history contains:
  all of:
    work_done = "fuel/air ratio reconfig"
    any of:
      work_done = "intake upgrade"
      work_done = "exhaust upgrade"

This is surprisingly readable, and we can make it a bit more concise without losing any readability by making the of optional. More significantly, we can also make all the default, top-level group condition, since that naturally matches the most common use cases, allowing users to just drop that part entirely.

-- same semantics as the query just above
history contains:
  work_done = "fuel/air ratio reconfig"
  any:
    work_done = "intake upgrade"
    work_done = "exhaust upgrade"

Simple Cases Should be Simple

Recall that by far our most common use case is to just list some number of events that must be present somewhere in the history. We really want to make this use case as pleasant as possible without marring the language in more complex cases. I thought it would be best to just drop the work_done, knowing that this is typically what users are searching on.

-- still the same semantics
history contains:
  "fuel/air ratio reconfig"
  any:
    "intake upgrade"
    "exhaust upgrade"

And if users need to specify a filter on detail in addition to work_done, we can extend this idea by borrowing the hierarchical notation of filesystem paths.

history contains:
  "diagnostic" / "P1155"
  "oil change" / "0W-20"

Each condition in the above query expresses an equality filter on work_done and detail, respectively, which are the two attributes users are filtering on the vast majority of the time. Users really liked the conciseness of this syntax. Simple queries have simple expressions in our language, and that’s a good sign.

But what if users want to use something more complex to describe events, like additional attributes, or comparison operators like in or like, which some of them would know from their exposure to SQL? It’s tempting to reintroduce the parentheses and boolean operators we have so far avoided.

history contains:
  work_done = "oil change" and detail like "0W-%"
  work_done = "tire replacement" and (detail = "Bridgestone" or detail = "Michelin")

This is not so bad. The use is limited enough that users are unlikely to get to “disgusting” levels of nesting or balancing syntax in this way. But we could perhaps still avoid having to allow this type of syntax back in by further leveraging blocks.

history contains:
  event:
    work_done = "oil change"
    detail like "OW-%"
  event:
    work_done = "tire replacement"
    any:
      detail = "Bridgestone"
      detail = "Michelin"

I think this is better and more consistent with the “spirit” of the new language that is taking shape.

More Complex Filters

Moving on to the second reference query, there were several new filter types that the language needed to support: vehicle attributes, event counts, and the absence of events.

The vehicle attributes seemed like the easiest to address using all of the design choices I’d already made so far. They are separate from the maintenance history and so should get their own top-level block. Otherwise, I could reuse the rest of the syntax I’d already come up with.

vehicle has:
  make = "Toyota"
  -- extra filters just for demonstration purposes
  model = "Camry"
  year >= 2015
history contains:
  ...

If we wanted to decouple the language from vehicle maintenance records, entity has would be more appropriate. The choice of has seems natural since we are not searching across a history of events but rather looking for individual vehicles that have specific properties.

Next came the event count filter. I tinkered with options like count(>= N of ...) or a dedicated block like at least N of:, but none of them really appealed to me. I settled on a new key phrase {at most|exactly|at least} N occurrences of. It’s a bit lengthy but allows for very naturally worded expressions. The wording also disambiguates it from a phrase like at least N of, which I might use in other contexts (perhaps as a block) to filter on at least N conditions being true.

-- Reference query 2
vehicle has:
  make = "Toyota"
history contains:
  at least 3 occurrences of "diagnostic" / "P1155"
  no "o2 sensor replaced"

For the absence of an event, a plain no seemed the obvious choice.

History Patterns

We come now to the third reference query, which presents a couple of interesting challenges. The first is how to express these pattern overlays in an accessible manner. The visual abstraction is pretty intuitive; I needed to translate it into clear, structured text.

I confess that here I was very attracted to the idea of using something inspired by Cypher or GraphFrames’s motif finding syntax. We are not working with graphs, but a timeline of events is commonly represented as nodes on a line just as we visualized it earlier in this post. So my first cut at this problem took direct inspiration from that syntax.

history contains:
  "transmission fluid change" -[<= 6 months]-> "transmission rebuild"

What I like about this is that it closely mirrors the pattern overlay visualization I sketched out earlier on in the design process. It’s also quite concise and extends well to longer patterns like A -[<= 6 months]-> B -[<= 1 year]-> C. The main problem with it is the -[<duration>]-> syntax, of course, which is quite unfriendly to non-programmers, especially compared to the rest of the language.

history contains:
  "transmission fluid change" then after at most 6 months "transmission rebuild"

I went with this more verbose English syntax.³ It supports the common variations on history patterns my users care most about. The then makes the order of events explicit. A common variation users also asked for is one where the order doesn’t matter, just that the events are within a certain window of time of one another. A Cypher-inspired syntax would probably be something like -[<duration>]- (i.e. without the arrow heads), but in this case a simple within <duration> of is direct and clear.

-- history patterns
A then B
A then after {at most|exactly|at least} N {days|months|years} B
A then within N {days|months|years} B -- shorthand for `after at most`, which is very common
A within N {days|months|years} of B

An important note about the semantics of reusing an event: Each history pattern is independent. I think as a default this matches what most users would expect, but it has some important consequences.

-- matches BCAB
history contains:
  A then B
  B then C

-- does not match BCAB
history contains:
  A then B then C

One is that we have no way to express patterns that refer to the same occurrence of a specific event. It’s a more advanced use case that we can leave to the future, should the need arise.

Unfinished Ideas

In the course of designing this new language I sketched out a lot of ideas that seemed relevant but didn’t make the cut for the initial release. They are good candidates for subsequent releases of the language, with more refinement and validation from the user base.

with events:
  tx_change:
    work_done = "transmission fluid change"
    detail = "AW-1"
    ... -- more conditions as necessary
  tx_rebuild:
    work_done = "transmission rebuild"
history contains:
    tx_change then after at most 1 year tx_rebuild

with events:
  recent_tune_up:
    work_done = "tune up"
    -- as convenient comparison operator for attributes
    -- avoids need to use SQL functions like `NOW()` and `DATEDIFF()`
    time within past 1 year
history contains:
  recent_tune_up then after at most 6 months "diagnostic" / "P0524"
  -- as lookback filter for all patterns with a block
  within past 2 years:
    at least 2 occurrences of "oil change"

population "rice-rockets":
  history contains:
    "intake upgrade"
    "exhaust upgrade"
    "fuel/air ratio reconfig"

population "rice-rockets-jp":
  vehicle has:
    membership in population "rice-rockets"
    any:
      make = "Toyota"
      make = "Honda"
      make = "Mazda"

history contains:
  within past 1 year:
    anything

history contains:
  A then within 6 months B
  A then within 6 months C

history contains:
  for the same occurrence of A:
    A then within 6 months B
    A then within 6 months C

history contains:
  no A then A then within N days B

history contains:
  first occurrence of A then within N days B

Design is a Non-Linear Process

I’ve walked you through my design process step by step, but the real design process was not this linear. Some ideas went through many iterations before I settled on something. Sometimes you run with an idea you think is solid until you get to the implementation and then realize there is something incoherent or incorrect about it; the realization takes you back to the beginning to rework the design. Sometimes you step away from the problem for days or weeks, during which ideas incubate subconsciously, and emerge evolved when you return to the problem.

As for the design of the language itself, you may find it good or bad. That’s not so much the point of this post.⁴ Rather, what I wanted to share was the process. It was fun. It was intellectually satisfying. And it wasn’t too difficult. If the big picture problem I described here reminds you of something you’re facing, consider designing your own query language. Even if it doesn’t see the light of day⁵, you will learn a lot in the process.

In the next post I will walk through how to implement a portion of this new query language so we have a working prototype!

Thanks to Ciprian and Eddie for reading a draft of this post.

A constraint – or, similarly, a validation check¹ – is a description of what your data should or shouldn’t look like. A query serves the same purpose. It describes data that has a specific “shape” or satisfies certain properties. Anything that can be thought of as a constraint can also be expressed as a query.

Consider one of SQL’s “built-in” constraints, PRIMARY KEY. It’s such a commonly used constraint that, presumably, SQL’s authors felt it merited a dedicated keyword. But a primary key constraint can also be expressed as a plain query.

-- `PRIMARY KEY` expressed as a plain SQL query.
-- `id` is our primary key column.
SELECT id, COUNT(*) AS id_count
FROM some_table
GROUP BY id
HAVING (
    id_count > 1
    OR id IS NULL
);

As long as this query returns nothing, our constraint holds. If some data violates our constraint, then this query will return precisely the violating data.

This ability to express constraints as queries isn’t specific to SQL. Any query language can do this. Let’s express this same constraint using Apache Spark’s DataFrame API, which is roughly equivalent to SQL in expressiveness:

# `PRIMARY KEY` expressed using PySpark's DataFrame API.
(
    some_table
    .groupBy("id")
    .count()
    .where(
        (col("count") > 1)
        | col("id").isNull()
    )
)

As another quick example, here is FOREIGN KEY expressed as an SQL query:

-- `FOREIGN KEY` expressed as a plain SQL query.
-- book.author_key is a foreign key to author.author_key.
SELECT b.*
FROM 
    book b
    LEFT OUTER JOIN author a
        ON b.author_key = a.author_key
WHERE
        b.author_key IS NOT NULL
    AND a.author_key IS NULL
;

And again using Spark’s DataFrame API:

# `FOREIGN KEY` expressed using PySpark's DataFrame API.
(
    book
    .join(author, on="author_key", how="left_outer")
    .where(book["author_key"].isNotNull())
    .where(author["author_key"].isNull())
    .select(book["*"])
)

If every row in book points to a valid row in author (or points to no author at all), then these queries return nothing. If a book points to an author not in our author table, then our foreign key constraint has been violated and these queries will return the rows from book that point to a bad author key.

Not Just the Built-In SQL Constraints

My feeling is that any possible constraint or check you can think of can be expressed in this way, using the query languages we are already familiar with. And since we are using mature and well-understood query languages, we can reuse all the constructs and patterns they provide: functions, subqueries, common table expressions, and so on.

To that end, let’s express a more complex constraint as a query, using this example scenario described in Designing Data-Intensive Applications²:

[Y]ou are writing an application for doctors to manage their on-call shifts at a hospital. The hospital usually tries to have several doctors on call at any one time, but it absolutely must have at least one doctor on call.

You can’t express this constraint using any of SQL’s built-in keywords, but it’s pretty straightforward as a query.

-- Doctor on-call constraint expressed as SQL.
SELECT COUNT(*) >= 1 AS sufficient_on_call_coverage
FROM doctors
WHERE on_call;

If we have at least one doctor on-call, then sufficient_on_call_coverage is TRUE, otherwise it’s FALSE. And as before, this constraint can be expressed in another query language like a DataFrame API, not just SQL:

# Doctor on-call constraint expressed using PySpark's DataFrame API.
from pyspark.sql.functions import count

(
    doctors
    .where(col("on_call"))
    # I don't call .count() here but do it instead inside the
    # .select() so this query is semantically equivalent to the SQL one.
    .select(
        (count("*") >= 1).alias("sufficient_on_call_coverage")
    )
)

At this point, you can probably imagine any number of arbitrary data constraints expressed as queries. When the constraint holds the query returns TRUE (or, alternately, it returns nothing), and when the constraint is violated the query returns FALSE (or, alternately, it returns the violating data). You can make the query as complicated as you’d like, as long as it fits this pattern.³

Straight from the SQL specification: CREATE ASSERTION

This idea that arbitrary queries can be used to express data constraints is not new. As part of my research for this post I stumbled on a feature of SQL that’s been part of the specification since 1992: CREATE ASSERTION

The idea of SQL assertions is that you can specify constraints on your data via queries that return TRUE when the constraint holds and FALSE when it is violated.

So here’s our primary key constraint re-expressed as an SQL assertion.

CREATE ASSERTION some_table_primary_key
CHECK (
    NOT EXISTS (
        SELECT id, COUNT(*) AS id_count
        FROM some_table
        GROUP BY id
        HAVING (
            id_count > 1
            OR id IS NULL
        )
    )
);

If some_table should ever come to have more than one row with the same id, or a row with a NULL id, then the inner SELECT will return some results, causing the NOT EXISTS check to return FALSE, thus violating our some_table_primary_key constraint.

Here’s the doctor on-call constraint expressed as an SQL assertion.

CREATE ASSERTION sufficient_on_call_coverage
CHECK (
    (
        SELECT COUNT(*)
        FROM doctors
        WHERE on_call
    ) >= 1
);

Any constraint you can express as an SQL query can be tweaked to fit this form of an ASSERTION. Popular DataFrame APIs like Spark’s don’t offer assertions, but we could easily imagine a couple of potential DataFrame equivalents to SQL assertions. Here, again, is the on-call doctors assertion:

from pyspark.sql.functions import count

# Hypothetical DataFrame Assertion API, inspired by Spark's
# `DataFrame.createGlobalTempView()`.
(
    doctors
    .where(col("on_call"))
    .select(count("*") >= 1)
    .createAssertion("sufficient_on_call_coverage")
)

# Another hypothentical DataFrame Assertion API, inspired by the Delta
# Live Tables API.
@assertion
def sufficient_on_call_coverage():
    return (
        doctors
        .where(col("on_call"))
        .select(count("*") >= 1)
    )

Efficiently Checking Assertions

CREATE ASSERTION is, unfortunately, vaporware—no popular database supports it.⁴ Why is that? There probably isn’t a definitive answer to this question, but we can take some educated guesses at the reasons.

First, generic assertions are not a critical feature. The other constraints SQL supports – PRIMARY KEY, FOREIGN KEY, CHECK – cover most applications’ practical needs. Anything more complex that would need to be expressed as an assertion can instead be implemented in your application layer or as part of a stored procedure.

A deeper reason is that assertions are very difficult to check efficiently. An assertion can touch an arbitrary number of tables and involve potentially expensive operations. Every time a table referenced in an assertion is modified, the assertion needs to be rechecked. That means running a potentially expensive query that may involve joins or aggregations to confirm the assertion still holds. While the assertion is being checked, potentially large ranges of data may need to be locked to ensure consistency in the face of concurrent readers and writers. And if the assertion fails, the original modification to the referenced tables needs to also fail and be rolled back. Compare this to the traditional constraints supported by SQL, which are typically limited to the rows being updated and do not require looking outside of that narrow range.

To express this problem differently, a key aspect of what makes assertions expensive to check is that they are expensive to maintain. That is, given a database in a consistent state with some active assertions, how do you check that an assertion still holds when an arbitrary change is made to the database? Naively, this means running the query that defines the assertion. If the query involves a join and aggregation across two large tables, that means computing the join and aggregation against those large tables from scratch. In the case of our doctors example, that means scanning the table to count how many doctors are on-call.

But why should we run anything from scratch? Given an incremental update to our database, we’d ideally want to be able to incrementally update the queries that define our assertions. That is, instead of recomputing the output of an assertion query from scratch, we want to incrementally update the output given the incremental changes to the input tables referenced in the query. So if one doctor updates their on-call status, we should be able to recompute the on-call constraint by looking at that one changed row.

This problem is starting to look like incrementally maintaining a materialized view. To recap how we got here: A general data assertion or constraint can be modeled as a query with a name. A named query is a view. We need to persist the output of this view in order to check it quickly, meaning we need a materialized view. And when the view output changes, we want a way to efficiently update it without recomputing the whole view. So we conclude that – in its essence – maintaining a data constraint is the same problem as maintaining a materialized view.

So what?

So data constraints can be modeled as queries, and efficiently checking arbitrary constraints is the same problem as efficiently updating a materialized view. What does that get us? If nothing else, understanding how these seemingly separate ideas are deeply connected provides some mental clarity.

I find that valuable for its own sake, but there are perhaps some practical benefits we can derive from this understanding, too. The main one being: If your data must conform to some constraints or invariants – that is, things that must always be true (like, there must always be at least one on-call doctor) – then express them declaratively. That could be in SQL, some DataFrame API, Datalog, or something else.

Your data platform may not support realtime enforcement of complex constraints, but you can still periodically check for constraint violations using these expressive, high-level queries written in a declarative language. Declarative checks like this would be much easier to write, understand, and maintain compared to relatively low-level, imperative data tests. And you can run them against your real production data to find problems, not just against test data.

There already are modern systems for applying constraints or validation checks to data lakes, like Deequ and Great Expectations. These systems offer their own custom APIs for expressing checks, instead of building on widespread query languages like SQL or DataFrames. I feel this is a missed opportunity. On the other hand, I suspect at least one benefit they have derived from building custom APIs is to make it easier to compute groups of validation checks efficiently, e.g. by avoiding repeated scans of the same data that may be referenced by multiple constraints.

Other projects that are already based on building graphs of declarative queries – like dbt, Materialize, ksqlDB, and Delta Live Tables – could likewise take advantage of the idea laid out in this post. Some of these projects already allow users to define row-level constraints on a dataset, analogous to the typical SQL CHECK constraint. But what if you could declare the whole dataset itself to be a constraint? Whenever an update to the pipeline triggers a refresh of the dataset, the constraint is rechecked.

Wrapping Up

Model your data platform as a database. Use a declarative query language to interact with it. If you do both of these things, you can then use that same language to define the constraints on your data platform. That’s the message of this post.

This idea is not new. Many teams run their data platform on a traditional relational database where this idea fits most naturally. In fact, Oracle admins have literally been able to implement complex constraints as materialized views – perhaps the most direct application of this post’s idea – since the early 2000s.

But you don’t have to be running on an actual relational database to take inspiration from this idea. Declarative query languages have stood the test of time and been reimplemented for the modern data lake. You can use them today to describe your data in ways that a simple CHECK constraint cannot capture. And hopefully, in the near future we will see more direct support for efficiently maintained, general data constraints.

Thanks to Matthew for reading a draft of this post.

The output of this data pipeline is a function of the input. In other words, the output is derived from the input by running the input through the pipeline.

This is an important characteristic of the output. As long as the input data and pipeline transformations (i.e. the pipeline code) are preserved, the output can always be recreated. The input data is primary; if lost, it cannot be replaced. The output data, along with any intermediate stages in the pipeline, are derivative; they can always be recreated from the primary data using the pipeline.

The Logical View

Let’s represent our hypothetical “Top-Selling Authors” pipeline as a directed graph, where the nodes represent datasets and the edges represent transformations of those datasets. Furthermore, let’s color each dataset in the graph based on whether it’s primary or derivative.

Most data pipelines, if you zoom out far enough, look something like this. You have some source data; it gets sliced, diced, and combined in various ways to produce some outputs. If someone were to wipe out all the derived data in this pipeline, you’d be able to regenerate it without any data loss. The pipeline could include any number of arbitrary steps, like copying files from an FTP share, or scraping data from a web page. It doesn’t matter as long as the pipeline produces the same output when given the same input.

Any time someone queries the output of the pipeline, it’s logically equivalent to them running the entire pipeline on the source data to get the output they’re looking for. In this way, a pipeline is a view into the source data.

Materializing the View

Of course, data pipelines don’t work this way in practice. It would be a waste of resources and a long wait for users if every query triggered a series of computations stretching all the way back to the primary data. When you ask for this month’s top-selling authors, you expect a quick response.

Hence, the typical real-world pipeline materializes its output, and often also several of the intermediate datasets required to produce that final output. Materializing a dataset simply means saving it to persistent storage, as opposed to repeatedly computing it on the fly. So when you ask for that list of authors, whatever system answering your query can start from the closest materialized dataset, as opposed to starting at the source or primary data.

So we’ve turned our view into a materialized view. “View” represents the logical transformations expressed in the pipeline. “Materialized” represents the fact that we cache the output of the pipeline, and perhaps also some of the intermediate steps. A complex set of interdependent data pipelines can be conceptualized in this way, as a graph of materialized views.

Note that this concept can be applied very broadly, and not just to what we think of as “normal” data pipelines:

• A traditional web cache alleviates read traffic from the primary database, which is the source of truth. The cache is derivative and can be regenerated from the database at any time. The data in the cache is materialized so that incoming queries do not need to go all the way back to the database to get an answer.
• A build system compiles or assembles source code into artifacts like executables or test reports. The artifacts are derivative, whereas the source code is primary. When you run a program over and over, you reuse the artifacts output by your build system, as opposed to recompiling them from source every time.

Updating a Materialized View

Materializing the output, though a practical necessity for most pipelines, adds an administrative cost. When the source data changes, the materialized views need to be updated. Otherwise, the data you get from the view will be stale.

To update a materialized view, there are two high-level properties you typically care about: the update trigger, and the update granularity. The former affects the freshness of your output, which impacts end-users of the data, and the latter affects the performance of your update process, which impacts the engineers or operators responsible for that process.

Update Trigger

The update trigger is the event that prompts a refresh of the materialized view—e.g. by running your pipeline against the latest source data.

That event may be a file landing in a shared drive, or some data arriving on an event stream, or another pipeline completing. For some pipelines, the update trigger may just be a certain time of day, in which case it might be more useful to talk about the update frequency rather than trigger.

A typical batch pipeline, for example, might run on a daily or hourly cadence, whereas a streaming pipeline may run every few seconds or minutes, or whenever a new event is delivered via some sort of event stream. Whenever the pipeline runs, it updates its output, and the whole process can be viewed as a refresh of the materialized view.

Update Granularity

The update granularity refers to how much of the materialized view needs to be modified to account for the latest changes to the source data.

A common update granularity is the full refresh. No matter how small or large the change to the source data, when the pipeline runs it throws away the entire output table and rebuilds it from scratch.

A more sophisticated pipeline might rebuild only a subset of the table, like a date partition. And an extremely precise pipeline may know how to update exactly the output rows that are impacted by the latest changes to the source data.

The update trigger and granularity are independent. You can have a pipeline that runs every second and does a full refresh of its output, and you can have a pipeline that runs once a day but carefully updates only the rows that it needs to.

Typical Examples

Let’s explore these two properties a bit using our example pipeline that computes the top-selling authors of the month.

The Daily Batch Update

Every night at 1 a.m., an automated process looks for a dump of the latest purchases from the previous day. The dump is a compressed CSV file.

The update process uses this dump to recompute the month’s sales numbers for all authors. It replaces the entire output table with all-new calculations for all authors. Many of the authors’ numbers may not have changed since the last update (because they had no new sales in that time period), but they all get recomputed nonetheless.

This is a very typical example of a batch pipeline. It has a scheduled update trigger at 1 a.m. every night, and an update granularity of the entire output.

The Live-Updating Table

In this version of our top-selling authors pipeline, individual purchases are streamed in as they happen, via a stream processor like Apache Kafka. Every purchase on this stream triggers an update to the calculation of top-selling authors.

The update process uses each individual purchase to incrementally recompute the sales total for the relevant author. If an author has no new sales over a given span of updates, their sales total is not recomputed (though their rank in the top-selling authors may need to be updated).

This is an example of a precise streaming pipeline. The update trigger is the purchase event that is streamed in, and the update granularity is the sales total for a single author.

The Declarative Data Lake

We previously discussed the idea of conceptualizing your data lake as a database. And here we’ve shown how you can conceptualize your data pipelines as materialized views.

But what if we could take this idea further than just as a conceptual tool? What if you could actually implement your data pipelines as a graph of materialized views?

Taken far enough, the promise of such an idea would be to build a declarative data lake, where the code that manages the lake focuses more on defining what the datasets are and less on how to mechanically build or update them.

Two relatively new projects express aspects of this vision in clear but different ways, and they merit some discussion here: dbt and Materialize.

dbt: Pipelines as Batch-Updated SQL Queries

The core of dbt is an engine for building a graph of SQL queries. Parts of any given query can be generated dynamically using a templating language (Jinja), and queries can reference other queries.

Every query has a configured materialization strategy, which defines whether the results of the query are generated ahead of time, and if so, how they are stored and updated.

If the results are materialized, they can be updated with a full refresh or incrementally, though there are some restrictions on what kinds of updates can be done incrementally. Updates are typically triggered on a schedule.

Materialize: Pipelines as Live-Updated Materialized Views

Materialize is an engine for building live, incrementally updated materialized views from streaming sources like Apache Kafka. A view can reference other live-updated views, as well as fixed tables.

The primary interface for creating these views is plain and elegant: A CREATE MATERIALIZED VIEW SQL statement.

Conceptually, this is roughly the same statement that is available in traditional relational databases. Materialize’s implementation however, allows for very efficient incremental updates against very flexible and expressive queries. Materialize’s capabilities are based on relatively new research done by its creators.

Conclusion

The ideas presented in this post are not new. But materialized views never saw widespread adoption as a primary tool for building data pipelines, likely due to their limitations and ties to relational database technologies. Perhaps with this new wave of tools like dbt and Materialize we’ll see materialized views used more heavily as a primary building block in the typical data pipeline.

Regardless of whether we see that kind of broad change, materialized views are still a useful design tool for conceptualizing what we are doing when we build data pipelines.

Get clear on what data is primary and what is derivative. Map your pipeline to the concept of a graph of transformations with materialized, intermediate datasets, each with a specific update trigger and update granularity.

The exercise should help bring some conceptual order to even the messiest pipelines.

Read the discussion about this post on Hacker News.

Working on that problem gave me a hands-on introduction to several recurring themes in software engineering. One of those themes is the divide between two ways of thinking about data: thinking about the state of something, and thinking about the transition of something from one state to another.

The state of something describes what it is at a specific point in time, and a transition describes a change from one state to another—i.e. how the state changed at a particular point in time. There are many other ways to describe the same concepts, but I’ll stick with state and transition for this post. In this player wallet scenario, a player’s balance is a piece of state, whereas the individual transactions that the player makes are transitions on that state.

In this post I’d like to share all the common threads I’ve found in software design and data management once I started to think about this divide.

Perspective: State-First or Transition-First

When you have a problem that requires you to manage both states and transitions, like in the player wallet example, one of those ways of thinking about the problem will tend to dominate. What I’ve noticed is that, whichever way is the primary way you think about the problem, you end up needing to build automatic and efficient ways of deriving the other representation of your problem.

• If states are the primary way you think of a problem, then when a piece of state changes you need to automatically derive the transitions that implement that change.
• If transitions are the primary way you think of a problem, then as the number of transitions grows large you need to build a way to efficiently query the state at a specific point in time (which is typically “right now”).

Our wallet example is a typical example of a “transitions-first” problem. The primary data are the individual transactions against a balance, since that corresponds most naturally to the activity we’re capturing, so we need to build an efficient way to derive the current balance from the history of transactions.

There are many problems that fit this “transitions-first” pattern, as well as problems that fit the “state-first” pattern. Let’s look at some examples of each, and see how in each case the secondary way of thinking about the problem needs to be automatically derived.

State-First Thinking

Infrastructure Management

Terraform is a tool for managing infrastructure that uses declarative configuration files to describe the infrastructure under its control. For example, here is a simple configuration that describes a single EC2 instance running on AWS:

provider "aws" {
  profile = "default"
  region  = "us-east-1"
}

resource "aws_instance" "example" {
  ami           = "ami-00b882ac5193044e4"
  instance_type = "t2.micro"

  tags = {
    Name  = "TerraformExample"
    Owner = "Nick"
  }
}

The configuration doesn’t explain how to create this infrastructure. It simply describes what the infrastructure is. In other words, it describes the state of the infrastructure.

When you deploy this configuration, Terraform compares the desired configuration against what is already out there and automatically figures out what operations are required to change the deployed infrastructure to match the configuration. To use the terminology we’re using in this post: The user specifies the desired infrastructure state, and Terraform automatically derives the required transitions to bring about that state.

So when you first deploy the above configuration against an empty environment (at least to Terraform’s knowledge), Terraform reports what actions it will take to bring about the specified infrastructure:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_instance.example will be created
  + resource "aws_instance" "example" {
      + ami                          = "ami-00b882ac5193044e4"
      ...
      + tags                         = {
          + "Name"  = "TerraformExample"
          + "Owner" = "Nick"
        }
      ...

Plan: 1 to add, 0 to change, 0 to destroy.

Suppose that after you deploy this infrastructure, you update your configuration to change the tags on the instance:

resource "aws_instance" "example" {
  ...

  tags = {
    Name        = "TerraformExample"
    Owner       = "Bob"
    Environment = "Dev"
  }
}

When you instruct Terraform to deploy this change, it figures out how to modify the existing infrastructure to match your desired configuration, even though you haven’t specified how it would do that, only what infrastructure you want at the end of the day:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_instance.example will be updated in-place
  ~ resource "aws_instance" "example" {
        ami                          = "ami-00b882ac5193044e4"
        ...
      ~ tags                         = {
          + "Environment" = "Dev"
            "Name"        = "TerraformExample"
          ~ "Owner"       = "Nick" -> "Bob"
        }
        ...

Plan: 0 to add, 1 to change, 0 to destroy.

As you can see in the plan, Terraform detects that only parts of the infrastructure need to be changed, and it figured out exactly how to change them to match the desired state.

As a Terraform user, you think about infrastructure primarily as what you want its current state to be, and Terraform figures out for you how to transition your infrastructure to match your desired configuration.

This seems like a natural way to approach infrastructure management, but you can probably imagine how a transitions-first approach to this problem would play out. Instead of specifying “I want one instance” in your configuration, you’d say “Add one instance”, and so on. You’d then have to carefully run just the appropriate steps as they are needed, or somehow build idempotency into each operation so it’s safe to rerun them without careful pre-planning. Otherwise, you’d likely end up creating duplicate infrastructure or make unwanted changes.

Database Schema Migrations

Relational databases are typically tightly coupled to the applications they back. As an application changes, the database schema backing it often also needs to change. But where an application can be updated with a simple code push, the database needs more care because it’s stateful. In other words, it’s carrying all this data that you want to maintain; you typically don’t want to update your database schema by dropping the whole database and redeploying it from scratch, which is in effect what you do when you deploy a new version of your application—replace the old application code entirely with the new. Instead, you want to migrate the database schema in-place, preserving all the data.

When I worked as a database developer, one of my tasks was to plan and execute migrations like this. I’d compare the current database schema against the new one that needed to be deployed, and hand craft a migration script that would ALTER tables and make any other necessary changes to mutate the schema as needed.

-- Version 1
CREATE TABLE person(
  id INT PRIMARY KEY,
  first_name VARCHAR(200),
  last_name VARCHAR(200)
);

-- Version 2
CREATE TABLE person(
  id INT PRIMARY KEY,
  first_name VARCHAR(200),
  last_name VARCHAR(200),
  birth_date DATE
);

-- Derived v1->v2 migration script
ALTER TABLE person
ADD COLUMN birth_date DATE;

Every release of an application had an associated database schema as well as a migration script to upgrade a database from the previous schema version. The full database schema at a given version was the primary description of the database, and the migration script was derived from the comparison of the full schema at two different versions.

To fit this into the common thread I’m tracing in this post, database schemas fit the state-first mode of thinking. The state of your database schema at a given version is primary (i.e. what the schema is), and the transitions from one schema version to another are secondary (i.e. how to get the schema to that state).

There are tools for approaching database schemas in this fashion, like Redgate SQL Compare for SQL Server and OnlineSchemaChange for MySQL. You give these tools two full schemas, and they compute the appropriate migration script. There are some risks to performing automatic migrations in this fashion. There may be semantic changes to your schema or strict availability requirements that an automated schema migration tool cannot satisfy without human input. But I think these tools address the problem of schema migrations in a conceptually natural way.

Source Control: Commits

Consider git. Depending on what you’re doing, git seamlessly moves between a state-first and transition-first view of the world. One of the primary things you do with git is create new commits to capture changes to your codebase. When you create a commit, git derives the diff for the commit by comparing the current state of your codebase against its state at the most recent commit.

diff --git a/.travis.yml b/.travis.yml
index c7938b6..c1a3c91 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,6 +1,5 @@
 language: python
 python:
-  - "3.4"
   - "3.5"
   - "3.6"
 # Work-around for Python 3.7 on Travis CI pulled from here:

In other words, you as the developer focus simply on what you want your code to look like now, and git figures out for you how to capture that as an incremental change from the most recently committed state of the code. You specify the desired state of the code, and git computes the transition from one state of the code to the other.

The derived transitions have a number of uses which you are probably familiar with. We’ll take a look at some of them in the next section.

Transition-First Thinking

In contrast to these examples of state-first thinking, we have transition-first thinking. This form of thinking naturally fits many real-world problems that are oriented around capturing or responding to events: a customer bought something; a user added a comment; a player made a move.

We saw how bank transactions fit this way of thinking; the individual transactions are primary, and the account balance is derived from those transactions. Let’s take a look at a few more examples of transition-first thinking and see how the state of the world ends up being derived from those transitions.

Social Media Activity

“Like, comment, and subscribe!” – a common refrain on social media platforms – also captures a straightforward example of a transition-first problem. Users each add individual likes to a post, for example. Each such action is recorded in the backing database. That’s the primary record of what happened.

When people view a given post, however, they don’t see all the individual likes (at least not by default). What they see is a summary which is derived from all the individual likes and represents the current state of the total. From the perspective of the backend database, the individual likes are primary; the derived total is secondary.

Turn-Based Games

Games like chess are well represented as a sequence of transitions. Each player takes a turn to make a move, and at any point in time the state of the game can be derived from the history of moves that have been made leading up to that point.

These are the first 10 moves of each player in Game 1 of the 1996 match between Garry Kasparov and Deep Blue.

1. e4 c5
2. c3 d5
3. exd5 Qxd5
4. d4 Nf6
5. Nf3 Bg4
6. Be2 e6
7. h3 Bh5
8. 0-0 Nc6
9. Be3 cxd4
10. cxd4 Bb4

The state of the board shown in the image is derived from this sequence of moves.

Image Editing

Image editing tools like Photoshop track your edits to the image you are working on, allowing you to easily undo changes or quickly flip between past image states.

You can think of the current state of the image as being derived from the history of edits to the original image. Each edit is a transition from one image state to another. The edits are the primary activity the user engages in, and the resulting image follows from them.

Source Control: Code Review

This is the other side of source control systems like git. Git lets you focus on the state of your code when you make a commit, and derives from that the transition from one state of the code to another. But after that, almost everything else in git starts with the transitions first and derives from those the state of the code.

A great example of a transitions-first interface is what happens when you submit a code change for review, like a GitHub Pull Request.

Instead of presenting the entire code base at once, the pull request focuses on a set of code changes – a “diff” – that a person can review in isolation. Focusing on this limited transformation of the code is what makes code review practical. And when a pull request gets merged in, git uses the encapsulated set of changes to efficiently update all downstream clones of the repository. The diff is primary, and the state of the codebase is derived from a series of transitions (i.e. diffs).

Thanks to Ivan, Michael, and Cip for reading drafts of this post.

In a previous post we discussed why many teams use both Postgres and Redshift, or some equivalent, in their data stack. In this post let’s look at the broader collection of data systems that constitute the modern data lake and give you, the newcomer, a mental map of them organized around a longstanding and very useful abstraction—the database.

What is a Database?

First, what is a database? In the abstract, it’s a system for storing and retrieving data. For the purposes of this post, however, I want to take a more practical view inspired by the classic graphical database client interface that is widespread in the software industry.

So in practical terms, this is a database:

Specifically, it’s a collection of three components that work together as one system:

1. Catalog: The Catalog tracks what data you have – i.e. schemas, tables, and columns – and where in the Storage layer to find it. The Catalog also tracks statistics about the data, like how many rows are in a table, or what the most common values in a specific column are. The Query Engine uses these statistics to figure out how to execute a query efficiently.
2. Query Engine: The Query Engine is what takes your query, in this case a SQL query, and translates it into specific machine instructions that will fetch and assemble the data you asked for. In other words, it takes a declarative query describing what you want and translates it into instructions detailing how to get it. The Query Engine uses the Catalog to lookup the datasets referenced in the query and find them in Storage.
3. Storage: The Storage layer holds all of the database’s data. Its job is to store all the rows of data for all the tables in the database and retrieve or update them as requested.

Every traditional relational database system, like Postgres or MySQL, comes with all three of these components packaged into one coherent system. They work together seamlessly, but they’re also inseparable. You cannot, for example, query or update the data in the database using regular Unix tools like grep or sed; you have to go through the database’s query engine. And while some databases let you use the database’s query engine to query data from outside of its own storage layer, it’s very much a secondary capability that you wouldn’t want to rely on heavily.

Breaking up the Database

In the years since this formula was first developed and perfected, there’s been an explosion of new database and data processing technology: Graph databases, document databases, column-oriented databases, stream processing systems, and more. Among these new technologies is the group of distributed data processing systems – also known as “Big Data” tools – dominated by the Hadoop ecosystem. This ecosystem includes systems like Apache Spark.

Broadly speaking, what distinguishes these systems from traditional databases is that they enable you to process a) large amounts of data b) in varied formats c) quickly d) and affordably. They do that by distributing the work to process data over a large number of cheap machines that are clustered together, and by allowing you to process data as it is on your storage system. In other words, instead of sending your data to the query engine, you send your query engine to the data. This contrasts with a traditional database system, where you would need to load the data into a specialized format in an area managed exclusively by the database. So to give a simple example of something you could do with these systems, which wasn’t as easy or practical to do before, you could process 20 TB of plain old CSV data distributed across 100 cheap machines in a few minutes.

When these systems were first being developed, the focus was on making them scalable and fault-tolerant, and the programming APIs weren’t very friendly. Over time, these distributed data processing systems evolved to recreate the convenience and productivity of the traditional database system. Instead of MapReduce, you could now query data using plain old SQL. And instead of referring to data by fixed paths on a filesystem, you could now refer to them by abstract schema and table names, just like in a traditional database.

In effect, the people building these systems took the three components of the traditional database – Catalog, Query Engine, and Storage – and reinvented each as a stand-alone component for the distributed, massively scalable world. These components interoperate through shared catalog and storage formats.

This means you can store your data in one place, like S3, and query it using multiple tools, like Spark and Presto. These query engines will have the same view of the available datasets by pointing to a shared instance of the Hive Metastore.

Another key point is that storage is split up into formats and systems. Instead of having your data in a closed format on a single server operated on by a single database system, you can have data in multiple, open formats (like CSV or Parquet), across several storage systems. And because the data formats are not specific to any query engine, data created by one query engine can easily be read by another.

Example: The Spark “Database”

Apache Spark is extremely popular with teams building data lakes. If you’re reading this post, chances are that you’ve used it at some point. But if your experience with Spark was limited to its RDD or DataFrame APIs, you may not have realized that it can be integrated with these other systems to create a logical database with SQL as the primary query language. So let’s take a quick look at how to do that, keeping in mind that you can do something similar for many other “Big Data” query engines.

Spark comes with a command-line utility called spark-sql. It’s similar, for example, to Postgres’s psql. It gives you a SQL-only prompt where you can create, destroy, and query tables in a virtual database. By default, the catalog for this database is stored in a folder called metastore_db, and the data for the tables in the database is stored in a folder called spark-warehouse, typically in Parquet format. That’s already pretty neat, but you can take this further by calling ./sbin/start-thriftserver.sh from the Spark home directory. This will start up a JDBC server that you can connect to with any old database client, like DBeaver. That will give you the full “Spark is a database” experience. I won’t go over how to do this in detail, since that’s not the focus of this post, but the documentation for Spark’s JDBC server and SQL CLI is here.

We can extend this experience to the cloud. If you work with Spark on Amazon EMR, you can connect Spark to the AWS Glue Data Catalog. This gives Spark the same view into your datasets that several other AWS services have, including Amazon Athena and Amazon Redshift Spectrum. In other words, you can have one catalog, managed by AWS Glue, one location for your data, on S3, and any number of different services or query engines updating or querying that data using SQL. And just as you can with Spark running locally, on EMR you can start a JDBC server and connect to it with a regular database client.

Final Thoughts

I hope this post connected some dots for you about the various distributed data systems out there. There are many ways to conceptualize a data lake. Thinking of it as a database – i.e. a combination of catalog, query engine, and storage layer – provides a familiar abstraction that will help you mentally map out many of the technologies in this space.

This idea is more powerful than just as a conceptual tool, though! After all, a team may use these same technologies to build a data lake without integrating them to create that cohesive “database” package. What are they missing out on? As we touched on earlier, by actually building your data lake around the database abstraction, you can can shift the focus of your work away from where the data is or how to manipulate it, and instead focus on what data you want. Let’s explore this idea in a future post.

Thanks to Michelle, Yuna, Cip, and Sophie for reading drafts of this post.

It’s a great question. In fact, it’s very common for teams to use a combination of databases in their stack, especially a combination like Postgres and Redshift. Let’s explore why.

Understanding Database Types via their Access Patterns

There are a lot of database types in the world. A very powerful way to understand any given database is via the access patterns it is designed for.

At a high level, we can divide databases into two broad categories:

• transactional
• analytical

(Note: We’re not talking about transactions in the sense of ACID; we’re focusing just on the transactional access pattern.)

Transactional Databases

A transactional database is what most people think of when they hear “database”. It’s typically a database that backs online, interactive operations, like for a store or a game, where users expect instant responses to their queries.

Each query typically touches a very small amount of data, since the user executing the query is usually only reading or writing data about themselves, like updating their profile information or noting a new purchase. Tables tend to be narrow and highly normalized. At any one time there may be thousands or tens of thousands of such queries executing against a transactional database, as a multitude of people interact with the service that the database backs.

Query times in well-operated transactional databases are typically measured in milliseconds or less. Engineers spend a lot of time designing table indexes to enable the database to sift through the minimum number of rows required to answer a query, and tuning database parameters to keep as much data in memory as possible and minimize disk I/O.

In summary, transactional databases are designed for:

• point reads/writes, i.e. small amounts of data per query
• high concurrency, i.e. many queries running at the same time
• very low latency, i.e. quick query response times

Examples of transactional databases include all the popular database systems you’ve heard of:

• MySQL
• Oracle
• Postgres
• SQL Server

Analytical Databases

An analytical database is designed for a very different access pattern. Instead of backing an online store or game, an analytics database typically backs pipelines or tools that help users analyze large swathes of data.

A typical analytics query will touch a large range of data, like a reporting query that summarizes sales numbers by day for an entire quarter. Compared to a transactional database, there will only be a small number of queries running at one time against an analytics database, and each query will usually only touch a handful of columns in any given table. Tables tend to be wide (i.e. they have many columns) and highly denormalized.

Query times in an analytics database will typically be on the order of seconds or minutes. Indexes, which help queries quickly find their target row, aren’t relevant to analytics databases because queries rarely target a single row. Instead, engineers optimize analytical databases by reorganizing how the data is stored on disk to minimize the number of columns that need to be read, and by compressing the data so that large chunks of data can be read quickly. Trying to hold most of your data in memory is typically not possible or even necessary for analytics workloads.

In summary, analytical databases are designed for:

• bulk reads/writes, i.e. large amounts of data per query
• lower concurrency, i.e. fewer queries running at the same time
• higher latency, i.e. longer query response times

Popular analytical database systems include:

• Vertica
• Redshift
• Teradata

Typical Usage Examples

Now we can answer the question that opened this post: Why use both Postgres and Redshift?

A typical pattern is for teams to use both to build an analytics product. For example, consider a team building a product that tracks visits to your website and then shows you a handy chart summarizing your web traffic over the past few weeks.

The team uses Redshift to bulk load detailed event data tracking every visit to your site and then aggregate it down to a set of summary statistics and key metrics. They then load that summarized data – for example, total visits to your site per day for the past 12 weeks – into Postgres and serve it up from there to a website or API endpoint for users to access. Redshift answers a relatively small number of queries that crunch a lot of data and take a lot of time each, as part of a batch update pipeline, while Postgres answers a larger number of lighter queries that touch smaller amounts of data each, as users browse the summary statistics for their website.

It’s also common for the flow between the systems to go the other way around. Consider a team building an online game–an MMORPG, perhaps.

They use Postgres to back online game operations and track what actions a player is taking in the game. Those actions affect the online world and develop the player’s character as they are playing the game. The game only needs to know what a player has done in the current session, so to keep the transactional database light, the team regularly moves data for old sessions from Postgres to Redshift. In Redshift, analysts study player behavior across long stretches of time and try to answer questions like “What is the most popular path players take through our world?” or “Where are players quitting our game, and why?” Postgres handles the flurry of detail-level activity to serve thousands of online players, while Redshift answers big picture queries for a handful of in-house analysts.

Final Thoughts

There are many more ways to understand and categorize database systems:

• by the consistency guarantees they provide;
• by the levels of transaction isolation they provide;
• by how they scale to handle additional load;
• by the query languages and data structures they support;
• or by how they lay out data on disk, to name a few.

In this post we’ve focused just on access patterns, though databases designed for different access patterns typically do so by differing on these other axes, too.

What about systems like Amazon Athena and Spark SQL, by the way? Many teams with data-intensive workflows tend to use these tools as well. And they certainly look like databases, though there’s something weird about them. Roughly speaking, systems like Athena and Spark SQL can be categorized as analytical databases, but there’s more to them than that. We’ll explore these systems in more detail in a follow-up post.

Thanks to Michelle, Yuna, Sam, Cip, Fabian, and Roland for reading drafts of this post.

Apparently, you can solve this problem using a formal specification language like TLA+. I don’t know much about this topic, but it appears that a formal specification language is much like a programming language in that it lets you describe the behavior of a system. However, it’s much more rigorous and builds on mathematical techniques that enable you to reason more effectively about the behavior of the system you’re describing than you can with a typical programming language.

In a recent discussion on Hacker News about TLA+, I came across this comment which linked to a fun and simple example showing how to solve the Die Hard 3 problem with TLA+. I had to watch the first two lectures from Leslie Lamport’s video course on TLA+ to understand the example well, but once I did I was reminded of the idea of property-based testing and, specifically, Hypothesis.

So what’s property-based testing? It’s a powerful way of testing your logic by giving your machine a high-level description of how your code should behave and letting it generate test cases automatically to see if that description holds. Compare that to traditional unit testing, for example, where you manually code up specific inputs and outputs and make sure they match.

How not to Die Hard with Hypothesis

Hypothesis has an excellent implementation of property-based testing for Python. I thought to myself: I wonder if you can write that Die Hard specification using Hypothesis? As it turns out, Hypothesis supports stateful testing, and I was able to port the TLA+ example to Python pretty easily:

from hypothesis import note, settings
from hypothesis.stateful import RuleBasedStateMachine, rule, invariant


# The default for `max_examples` is sometimes not enough for Hypothesis
# to find a falsifying example.
@settings(max_examples=2000)
class DieHardProblem(RuleBasedStateMachine):
    small = 0
    big = 0

    @rule()
    def fill_small(self):
        self.small = 3

    @rule()
    def fill_big(self):
        self.big = 5

    @rule()
    def empty_small(self):
        self.small = 0

    @rule()
    def empty_big(self):
        self.big = 0

    @rule()
    def pour_small_into_big(self):
        old_big = self.big
        self.big = min(5, self.big + self.small)
        self.small = self.small - (self.big - old_big)

    @rule()
    def pour_big_into_small(self):
        old_small = self.small
        self.small = min(3, self.small + self.big)
        self.big = self.big - (self.small - old_small)

    @invariant()
    def physics_of_jugs(self):
        assert 0 <= self.small <= 3
        assert 0 <= self.big <= 5

    @invariant()
    def die_hard_problem_not_solved(self):
        note("> small: {s} big: {b}".format(s=self.small, b=self.big))
        assert self.big != 4


DieHardTest = DieHardProblem.TestCase

Calling pytest on this file quickly digs up a solution:

self = DieHardProblem({})

    @invariant()
    def die_hard_problem_not_solved(self):
        note("> small: {s} big: {b}".format(s=self.small, b=self.big))
>       assert self.big != 4
E       AssertionError: assert 4 != 4
E        +  where 4 = DieHardProblem({}).big

how-not-to-die-hard-with-hypothesis.py:17: AssertionError
----------------------------- Hypothesis -----------------------------
> small: 0 big: 0
Step #1: fill_big()
> small: 0 big: 5
Step #2: pour_big_into_small()
> small: 3 big: 2
Step #3: empty_small()
> small: 0 big: 2
Step #4: pour_big_into_small()
> small: 2 big: 0
Step #5: fill_big()
> small: 2 big: 5
Step #6: pour_big_into_small()
> small: 3 big: 4
====================== 1 failed in 0.22 seconds ======================

Note: I last tested this against Python 3.13, pytest 8.4, and Hypothesis 6.138.

What’s Going on Here

The code and test output are pretty self-explanatory, but here’s a recap of what’s going on:

We’re defining a state machine. That state machine has an initial state (two empty jugs) along with some possible transitions. Those transitions are captured with the rule() decorator. The initial state and possible transitions together define how our system works.

Next we define invariants, which are properties that must always hold true in our system. Our first invariant, physics_of_jugs, says that the jugs must hold an amount of water that makes sense. For example, the big jug can never hold more than 5 gallons of water.

Our next invariant, die_hard_problem_not_solved, is where it gets interesting. Here we’re declaring that the problem of getting exactly 4 gallons in the big jug cannot be solved. Since Hypothesis’s job is to test our logic for bugs, it will give our state machine a thorough shake down and see if we ever violate our invariants. In other words, we’re basically goading Hypothesis into solving the Die Hard problem for us.

I’m not entirely clear on how Hypothesis does its work, but I know the basic summary is this: It takes the program properties we’ve specified – including things like rules, invariants, data types, and function signatures – and generates data or actions to probe the behavior of our program. If Hypothesis finds a piece of data or sequence of actions that get our program to violate its stated properties, it tries to whittle that down to a minimum falsifying example—i.e. something that exposes the same problem but with a minimum number of steps. This makes it much easier for you to understand how Hypothesis broke your code.

Hypothesis’s output above tells us that it was able to violate the die_hard_problem_not_solved invariant and provides us with a minimal reproduction showing exactly how it did so. That reproduction is our solution to the problem. It’s also how McClane and Carver did it in the movie!

Final Thoughts

All in all, I was pretty impressed with how straightforward it was to translate the TLA+ example into Python using Hypothesis. And when Hypothesis spit out the solution, I couldn’t help but smile. It’s pretty cool to see your computer essentially generate a program that solves a problem for you. And the Python version of the Die Hard “spec” is not much more verbose than the original in TLA+, though TLA+’s notation for current vs. next value (e.g. small vs. small') is elegant and cuts out the need to have variables like old_small and old_big.

I don’t know how Hypothesis compares to TLA+ in a general sense. I’ve only just started to learn about property-based testing and TLA+, and I wonder if they have a place in the work that I do these days, which is mostly Data Engineering-type stuff. Still, I found this little exercise fun, and I hope you learned something interesting from it.

Thanks to Julia, Dan, Laura, Anjana, and Cip for reading drafts of this post.

Read the discussion about this post on Lobsters.