jdlm.info

Bootstrap Confidence Intervals in SQL for PostgreSQL and BigQuery

Sun, 11 Jun 2023 12:00:00 +0000

A confidence interval is a good way to express the uncertainty in an estimate. This post is about how to calculate approximate confidence intervals in portable (mostly) standard SQL using bootstrapping. We’ll also see that BigQuery is surprisingly fast at running the required bootstrap calculations, which makes it easy to add a confidence interval to nearly any point estimate you calculate in BigQuery.

The code for this article is open source.

Background: Confidence Intervals and the Bootstrap

Let’s start with some background on confidence intervals and the bootstrap, illustrated with a small example. If you already know all about these, feel free to skip to the queries.

Suppose we want to find the average mass of an (adult, domestic) cat ¹, and we’ve started by selecting 10 cats at random and measuring their masses in kilograms:

Name	Mass (kg)
Apollo	3.2
Bean	2.4
Casper	6.9
Daisy	3.2
Ella	5.1
Finn	3.5
Ginger	5.9
Harley	3.3
Iago	5.5
Jasper	5.4
Mean	4.4
Std Dev	1.5

The sample mean for these ten cats is 4.4kg, with a standard deviation of 1.5kg. What can we now say about the population mean for all cats? The sample mean is our best estimate for the population mean, but it may be inaccurate, because it is based on a (very) small subset of the population; this inaccuracy is called sampling error. A confidence interval quantifies sampling error by calculating an interval that we can be confident, up to a defined confidence level, contains the population mean. For example, one of the methods we’ll use below calculates a confidence interval at the 95% level for the average mass of a cat as [3.4kg, 5.5kg] based on these 10 measurements.

Before looking at how to calculate that interval, it’s worth spending a few words on what it means. Firstly, it is not a claim that 95% of all cats weigh between 3.4kg and 5.5kg; instead, it is a claim that the mean over all cats is likely in this range. Secondly, it is not strictly speaking a claim that the population mean lies in that range “with probability 0.95” ². Instead, the preferred interpretation is that, if one were to repeat the whole experiment many times, each time re-running all the data collection and analysis on a new sample to calculate a 95% confidence interval, the (fixed) population mean would lie outside the calculated interval in only 5% of the repeats. This is somewhat sobering, firstly because we have no way of knowing if the one experiment we actually did was one of the unlucky 5%, and secondly because, even if we do everything right, if we do a lot of different experiments we should expect to be wrong in 5% of them. That is still, however, better than one is likely to do with point estimates alone!

On to the confidence interval calculations without bootstrapping. Under the (here reasonable) assumption that the population distribution is normal, which is to say that the distribution of the mass of all cats is normal, a confidence interval can be obtained from the quantiles of the $t$-distribution. In particular, for a sample of size $n$ with sample mean $\bar{x}$ and (sample) standard deviation $s$, the endpoints of a $C \times 100 \%$ confidence interval are given by \begin{equation} \bar{x} \pm Q_t\left(\frac{1-C}{2}, n - 1\right) \frac{s}{\sqrt{n}} \label{t-ci} \end{equation} where $Q_t(\alpha, k)$ denotes the quantile function of the $t$-distribution with $k$ degrees of freedom, evaluated at quantile $\alpha$, where in this case $\alpha = \frac{1-C}{2}$ and $k = n - 1$. The second term in the above equation is the standard error of the sample mean, scaled by the $Q_t$ quantile. For a 95% confidence level and sample size 10, $\alpha = 0.025$, $k = 9$, and the $Q_t$ factor is -2.26, which leads to the interval [3.4kg, 5.5kg], as noted above ³.

The process for finding the bootstrap confidence interval looks very different. Rather than a formula, it is an algorithm. To calculate a bootstrap confidence interval, we repeatedly resample this original sample, with replacement, and compute the mean for each resample. The first such resampling might be:

Name	Mass (kg)
Apollo	3.2
Bean	2.4
Casper	6.9
Casper	6.9
Casper	6.9
Casper	6.9
Ella	5.1
Ella	5.1
Finn	3.5
Finn	3.5
Mean	5.1
Std Dev	1.8

Here we’ve chosen Casper four times and left out several of the other cats altogether, as can happen when resampling with replacement. Casper is a rather heavy cat, which has pulled up the mean for this resample to 5.1kg. The second resample might then be:

Name	Mass (kg)
Casper	6.9
Casper	6.9
Daisy	3.2
Finn	3.5
Finn	3.5
Finn	3.5
Ginger	5.9
Harley	3.3
Harley	3.3
Jasper	5.4
Mean	4.6
Std Dev	1.6

This time, some of the cats that were missing from the first sample reappear, and there are fewer Caspers pulling up the mean, so it returns to 4.6kg, closer to our original sample. If we then repeat this process 998 more times to collect a total of 1000 resampled means, we are likely to find something like the following bootstrap distribution:

To form the desired 95% confidence interval, the simplest approach is to simply read off the the 2.5% and 97.5% quantiles from this empirical bootstrap distribution as the 95% confidence interval, which in this case gives the interval [3.6kg, 5.3kg].

One way to think about bootstrapping is as a ‘what if’ sensitivity analysis with the observations in a sample. It essentially asks for each observation, what would our estimate look like if we had not collected this observation? Or if we’d seen this observation several times instead of just once? If the sample is large, or if the observations are fairly similar, then losing or double counting a few of them shouldn’t make a big difference, and bootstrap distribution will be sharp and the confidence interval tight. If, on the other hand, the sample is smaller, or there is more variability in the observations, then the bootstrap distribution will be more dispersed and the confidence interval wider.

This example hints at several important caveats:

Bootstrap CIs require a lot of computation. There is no hard and fast rule for the number of resamples that one should use, but 1000 is generally regarded as the minimum for calculation of confidence intervals. This means that instead of computing a statistic once, bootstrapping requires that it be computed thousands of times. Fortunately, the required computation can be efficiently parallelized, as we shall see below.
Bootstrap CIs are approximate. In many common cases, there are more accurate and efficient methods of calculating confidence intervals. They should be used where possible. Bootstrapping is still a useful technique for more complicated cases or as an additional check on other methods.
The ‘percentile bootstrap’ approach of calculating the confidence interval directly from the quantiles of the bootstrap distribution is simple to implement and intuitively appealing, but it is known to produce intervals that are too narrow, particularly for small sample sizes. This may explain why the bootstrap interval obtained above was narrower than the $t$ interval. There are some ways to correct for this ⁴, but this approach will do for now.

The Bootstrap in SQL

Now that we’ve seen the idea behind the bootstrap, let’s see an example of how to implement it in SQL. Suppose we have scaled up our cat weighing experiment and now have a table, cats, with 10000 rows. Each row records a unique identifier for the cat and its mass in kilograms.

> SELECT count(*) FROM cats;
count
-------
10000
(1 row)

> SELECT * FROM cats ORDER BY random() LIMIT 10;
  id  |       mass       
------+------------------
 3091 | 4.25136586892616
 5680 | 4.34285504738124
 5895 | 5.63979916384868
 1952 |  5.1710561140116
 3847 | 4.82984705465995
 2861 | 4.33598448266297
  592 | 6.52217568717482
 2915 | 3.87406259543517
 6338 |  4.0396866933194
 6726 | 4.71685250612103
(10 rows)

The mean mass for this sample is given by:

SELECT avg(mass) FROM cats;
        avg        
-------------------
 4.492052081409403
(1 row)

The following query calculates a 95% CI around this estimate using 1000 resamples (for PostgreSQL; the query for BigQuery is here):

WITH bootstrap_indexes AS (
  SELECT generate_series(1, 1000) AS bootstrap_index
),
bootstrap_data AS (
  SELECT mass, ROW_NUMBER() OVER (ORDER BY id) - 1 AS data_index
  FROM cats
),
bootstrap_map AS (
  SELECT floor(random() * (
    SELECT count(data_index) FROM bootstrap_data)) AS data_index,
    bootstrap_index
  FROM bootstrap_data
  JOIN bootstrap_indexes ON TRUE
),
bootstrap AS (
  SELECT bootstrap_index,
    avg(mass) AS mass_avg
  FROM bootstrap_map
  JOIN bootstrap_data USING (data_index)
  GROUP BY bootstrap_index
),
bootstrap_ci AS (
  SELECT
    percentile_cont(0.025) WITHIN GROUP (ORDER BY mass_avg) AS mass_lo,
    percentile_cont(0.975) WITHIN GROUP (ORDER BY mass_avg) AS mass_hi
  FROM bootstrap
),
sample AS (
  SELECT avg(mass) AS mass_avg
  FROM cats
)
SELECT *
FROM sample
JOIN bootstrap_ci ON TRUE;

Let’s take it one CTE at a time:

bootstrap_indexes enumerates the bootstrap resamples, 1 to 1000, as bootstrap_index.
bootstrap_data generates a contiguous sequence of row numbers, one for each row in the input data, data_index. (Here I’ve used id as the natural way to order the rows, but you could use anything.) The - 1 is important, because it makes the sequence start from 0 rather than 1, and the next CTE will generate random indexes starting at 0.
bootstrap_map performs the resampling with replacement by generating 10000 random integers in the range of the data_index for each of the 1000 resamples. The JOIN bootstrap_indexes ON TRUE produces the full Cartesian product of the bootstrap and data indexes, so for 1000 resamples of 10000 observations, there are 10 million rows in this CTE.
bootstrap computes the mean mass for each resample by joining the bootstrap data with the bootstrap map by data_index, grouping by bootstrap_index, and calculating the mean within each group (i.e. within each resample).
bootstrap_ci uses the percentile_cont ordered-set aggregate function to find the 2.5% and 97.5% percentiles of the empirical bootstrap distribution.
sample computes the most likely estimate as we did above.

Finally, we put them all together to get a single row with the most likely estimate, mass_avg, as above, and the confidence interval bounds, mass_lo and mass_hi:

     mass_avg      |      mass_lo      |     mass_hi      
-------------------+-------------------+------------------
 4.492052081409403 | 4.470893493719024 | 4.51113963545513
(1 row)

That is, our estimate here is 4.49kg with 95% CI [4.47kg, 4.51kg]. In this case, the data were generated with a true mass of 4.5kg, so the mean is not far out, and the true rate is within the 95% confidence interval, as we’d expect to happen 95% of the time. (If you rerun the same query on the example data, you may get somewhat different numbers due to randomness in the bootstrap sampling, but with 1000 resamples they should not be very different very often.)

This query takes ~20s to run on my instance, and EXPLAIN ANALYZE shows most of that time is spent joining the bootstrap_map and bootstrap_data back together in the bootstrap CTE. Let’s see if we can speed it up.

The Poisson Bootstrap in SQL

The overall effect of resampling with replacement is that each of the observations in the original sample is used a random number of times in any given resample. One way to think of this random number of times is as a bootstrap weight for each observation in each resample. Returning to the cats example above, the first two resamples could instead have been expressed in terms of bootstrap weights, as follows:

Original Sample		Resample 1	Resample 2	…	Resample 1000
Name	Mass (kg)	Bootstrap Weight	Bootstrap Weight	…	Bootstrap Weight
Apollo	3.2	1	0	…
Bean	2.4	1	0	…
Casper	6.9	4	2	…
Daisy	3.2	0	1	…
Ella	5.1	2	0	…
Finn	3.5	2	3	…
Ginger	5.9	0	1	…
Harley	3.3	0	2	…
Iago	5.5	0	0	…
Jasper	5.4	0	1	…

For example, Casper appears in the first resample 4 times and so has weight 4. For each resample, the weights sum up to the number of observations in the original sample, namely 10. To calculate the mean mass for each resample, we take the weighted average of the cat masses using the bootstrap weights.

In general, for a sample of $n$ original observations, the bootstrap weights for each resample jointly follow a $\textrm{Multinomial}(n,\frac{1}{n},\ldots,\frac{1}{n})$ distribution. The approach in the Poisson bootstrap is to make two simplifying approximations to how we generate these bootstrap weights:

Approximate the $n$–dimensional multinomial distribution with $n$ independent $\textrm{Binomial}(n, \frac{1}{n})$ distributions. The advantage is that the independent binomial distributions for each observation can be sampled in parallel ⁵. The disadvantage is that the total number of observations in a resample, which was constrained to be exactly $n$ in the multinomial case, may not add up to $n$ in the binomial case. When computing a statistic like a mean, where we divide through by the number of observations, this turns out not to make much difference, provided $n$ is large enough to avoid very sparse resamples ($n \gtrapprox 100$).
Approximate the $\textrm{Binomial}(n, \frac{1}{n})$ distribution by a $\textrm{Poisson}(1)$ distribution. This is a good approximation for any reasonably large $n$, and it avoids having the weights depend on $n$, which is again helpful for parallel running.

These simplifications allow us to avoid the join that was the most expensive part of the ‘pure’ bootstrap; instead, the query attaches the Poisson weights directly to the observations and computes the required weighted mean. The Poisson bootstrap query for the cats example with 1000 resamples looks like this (for PostgreSQL; the query for BigQuery is here):

WITH bootstrap_indexes AS (
  SELECT generate_series(1, 1000) AS bootstrap_index
),
bootstrap_data AS (
  SELECT mass, bootstrap_index, random() AS bootstrap_u
  FROM cats
  JOIN bootstrap_indexes ON TRUE
),
bootstrap_weights AS (
  SELECT bootstrap_data.*, (CASE
    WHEN bootstrap_u < 0.367879441171442 THEN 0
    WHEN bootstrap_u < 0.735758882342885 THEN 1
    WHEN bootstrap_u < 0.919698602928606 THEN 2
    WHEN bootstrap_u < 0.981011843123846 THEN 3
    WHEN bootstrap_u < 0.996340153172656 THEN 4
    WHEN bootstrap_u < 0.999405815182418 THEN 5
    WHEN bootstrap_u < 0.999916758850712 THEN 6
    WHEN bootstrap_u < 0.999989750803325 THEN 7
    WHEN bootstrap_u < 0.999998874797402 THEN 8
    WHEN bootstrap_u < 0.999999888574522 THEN 9
    WHEN bootstrap_u < 0.999999989952234 THEN 10
    WHEN bootstrap_u < 0.999999999168389 THEN 11
    WHEN bootstrap_u < 0.999999999936402 THEN 12
    WHEN bootstrap_u < 0.99999999999548 THEN 13
    WHEN bootstrap_u < 0.9999999999997 THEN 14
    ELSE 15 END) AS bootstrap_weight
  FROM bootstrap_data
),
bootstrap AS (
  SELECT bootstrap_index,
    sum(bootstrap_weight * mass) / sum(bootstrap_weight) AS mass_avg
  FROM bootstrap_weights
  GROUP BY bootstrap_index
),
bootstrap_ci AS (
  SELECT
    percentile_cont(0.025) WITHIN GROUP (ORDER BY mass_avg) AS mass_lo,
    percentile_cont(0.975) WITHIN GROUP (ORDER BY mass_avg) AS mass_hi
  FROM bootstrap
),
sample AS (
  SELECT avg(mass) AS mass_avg
  FROM cats
)
SELECT *
FROM sample
JOIN bootstrap_ci ON TRUE;

Let’s again take it one CTE at a time:

bootstrap_indexes is as it was in the pure case.
bootstrap_data generates 1000 $\textrm{Uniform}(0,1)$ random numbers for each of the 10000 observations; like bootstrap_map in the ‘pure’ bootstrap query above, it has 10 million rows.
bootstrap_weights converts these variates from the uniform distribution to the Poisson distribution using inverse transform sampling, in which we invert the Poisson cumulative distribution function ⁶. The CASE statement here is basically an unrolled loop generated from this R code; it encodes that, when drawing from the $\textrm{Poisson}(1)$ distribution, one obtains 0 with probability 0.368, 0 or 1 with probability 0.736, 0, 1, or 2 with probability 0.920, and so on, up to 15 where the probability is so close to 1 that we start to hit the limits of 64-bit floating point numbers.
bootstrap again computes the mean mass for each resample, but this time it does so by finding a weighted average using the bootstrap weights.
bootstrap_ci and sample are exactly as before.

This query produces essentially the same results, but in ~12s rather than ~20s for the ‘pure’ bootstrap query above, which may not seem like much given all the extra mathematics, but it can be a larger savings for larger datasets.

Benchmark Results

So, let’s see some timings for the pure and Poisson approaches and PostgreSQL and BigQuery, as we vary the size of the sample:

These are wall clock times for 1000 bootstrap resamples. The Postgres instance used here was a Google Cloud SQL instance with 4 vCPUs and 16GiB of RAM, running Postgres 15.2. Each point is based on 10 trials. The error bars are (of course!) bootstrap 95% CIs.

The main conclusions are that the Poisson queries run faster than the pure queries, and that BigQuery is a lot faster than Postgres on both kinds of queries. Execution times with BigQuery remained essentially constant over the whole range. This is basically because BigQuery parallelized the bootstrap across many nodes, whereas Postgres ran it serially. The CPU times that BigQuery reported for queries that I ran manually were comparable to the wall clock times for Postgres.

I had hoped Postgres would also parallelize the queries, but it did not. It only ever used one out of its four available cores. The docs indicate that random is currently labelled as PARALLEL RESTRICTED, which might be a contributing factor.

It would be unwise to draw any conclusions about the absolute or relative costs of Postgres and BigQuery from these results, but I did learn a few things related to costs, so here they are. I spent £35 on running the Cloud SQL instance for a few days and £12 on 100 “flat rate flex slots” for BigQuery for a few hours. That said, the Cloud SQL instance was not always busy, and I had to rerun some tests after all my results perished in a make accident ⁷. Had I avoided that, it probably would have finished in about half the time (and cost). The queries with less than $10^6$ cats all ran fine in BigQuery’s on-demand pricing model and apparently fit within the free tier. For the larger datasets, I hit a limit (understandably) on the amount of CPU time they were using for the bootstrap resamples, which was very large compared to the size of the input data that the query was billed on. To get around that, I had to reserve some capacity, which required putting in a quota increase request but was otherwise a fairly painless process.

Finally, we should check that the generated intervals are correct. Here is a comparison with the boot package from R:

Each row shows the distribution of the 95% confidence interval endpoints over 100 bootstrapping trials for a fixed sample of 100 cats, as computed with R as the baseline, and Postgres and BigQuery using the ‘pure’ and Poisson bootstrap queries above. The plots also include the $t$ intervals from equation \eqref{t-ci} and, because this is synthetic data, the normal CI calculated from the true population variance used to generate the dataset.

There is good agreement between the distributions for R and the two SQL queries, which indicates that the queries are computing the right things. All of the bootstrap percentile intervals undercover somewhat with respect to the $t$ interval, which as noted above is a common problem for percentile intervals. There are some methods that attempt to correct for this ⁴. For this particular sample, the $t$ interval is also too narrow compared to the interval obtained using the true population variance, but on average it would cover correctly here.

Conclusions

We have seen how to implement bootstrap confidence intervals in (mostly) standard SQL. The full set of example queries is here. The queries run remarkably quickly in BigQuery, and they are usable for relatively small samples, at least, in Postgres. Using the Poisson approximation can significantly speed up the queries.

Where a standard formula exists for confidence intervals, it is usually best to use it. However, if you do need to bootstrap, and all you have is SQL, it turns out you can do it.

Footnotes

This seems like the sort of thing that should be known, but estimates vary. Googling cat produces an info box with a range of 3.6kg–4.5kg, apparently without a definition (what quantiles?) or source. Wikipedia says 4kg-5kg. National Geographic gives a rather broader range of 5lb-20lb, which is 2.3kg–9.1kg. Anyway, it is just an example. ↩
For that, one instead wants a Bayesian interval estimate, often called a credible interval. Fortunately, the two kinds of intervals do agree in many important cases. ↩
In this formula, the exact $Q_t$ quantile for a 95% confidence interval is often replaced with the constant 1.96, which is the corresponding quantile for the normal distribution. This approximation is good for large samples, since the $t$ distribution approaches the Normal distribution as the sample size increases. On a sample of size 10, it yields something more like a 92% confidence interval. Put differently, if we lazily use 1.96 instead of the correct $t$ quantile, we should expect to be wrong 8% of the time instead of 5% of the time, which is about 60% more often! Most spreadsheets now have a T.DIST function, so using the right number is not much more work. Maybe databases will catch up eventually. ↩
There are several ways to calculate a confidence interval from the bootstrap distribution, of which the percentile bootstrap is the simplest. Queries for the “Studentized” bootstrap are available in the companion repo here for Postgres and here for BigQuery. The Studentized intervals seem to have better coverage, but in my experience they are quite sensitive to outliers. I am not sure there is any broad consensus on which one of these approaches is best overall. ↩ ↩²
It may be helpful to think of this in physical terms. The physical analogy for the multinomial distribution would be rolling an $n$-sided die $n$ times; the weight of an observation would be the number of times the die lands on the corresponding face. For the binomial approximation, it would be flipping a (very) unfair coin with probability $\frac{1}{n}$ of coming up heads; each observation gets its own coin, which is flipped $n$ times, and the observation’s weight is the number of heads. Instead of rolling one giant die, which might look more like a disco ball, for the whole sample, the binomial approximation lets us flip the $n$ independent coins in parallel. ↩
It might be tempting to combine the bootstrap_data and bootstrap_weights CTEs, but CASE WHEN random() < x THEN y WHEN random() < z THEN w ... will generate a new random number for each WHEN, rather than repeatedly testing the same random number against the breaks of the target CDF. That might be an interesting way to sample from a Geometric distribution, but it is not what we want here. ↩
If you hit Ctrl-C to interrupt make, it deletes the target by default, because it doesn’t want to leave half-built files hanging around. That is not what you want when the target is a CSV with all your results in it. The solution is to mark the target as .PRECIOUS. ↩

Testing with Node and Docker Compose, Part 3: End-to-End

Sun, 24 May 2020 12:00:00 +0000

This post is the third in a short series about automated testing in node.js web applications with Docker. So far, we have looked at backend testing and frontend testing. This post will be about end-to-end testing, which covers both frontend and backend together.

To illustrate, we’ll continue building and testing our example application: a simple TODO list manager, which comprises a RESTful backend and a React frontend:

In particular, we’ll:

Use Docker Compose projects to create separate development and test environments, while sharing some services between the two for efficiency.
Upgrade our bin scripts to manage these separate ‘projects’.
Extract the storage layer from the backend so that the end-to-end tests can use it.
Set up end-to-end tests with puppeteer in Docker.
Promote some frontend integration tests using jsdom from the previous post to end-to-end tests, reducing the number of mocks we need to maintain.

As usual, the code is available on GitHub, and each post in the series has a git tag that marks the corresponding code.

Separate Test and Development Environments

The main idea of end-to-end testing is to stand up the whole system, put it into a given state, and then run automatic tests against it. In most cases, this means wiping out the state in the database before each test run, so we’d rather not run them against a development environment used for more manual or exploratory testing. Instead, we’d like to have separate development and test environments.

The simplest approach is to have two completely separate environments, but this can be expensive. For a large application, just running two copies of every service can tax your development machine. And we have to keep these two environments in sync in terms of installed packages and code. Sharing resources between the two environments can help with these problems.

In this case, the shared resources will be the application’s postgres server (with separate logical databases for development and test), to save having to run two full postgres instances, and its node_modules folders, to keep dependencies in sync between development and test environments. Of course, in a larger application, there might be more that could be shared. In diagram form, what we’re aiming for is:

Notably:

The development and test environments each have their own network and on it their own instances of the application’s services. This means that development and test services can use the same names in both environments without collisions, which reduces the amount of configuration that has to differ between the environments.
The postgres database server lives in a third shared environment, with a presence on both development and test networks, so both development and test applications can talk to the same database server (but use different logical databases).
For faster edit-test cycles, and to keep code in sync between development and test environments, both the development and test services need to have the source code on the host bind mounted in, which means two instances of the ‘node modules trick’ with a nested node_modules volume per service. To keep the dependencies in sync, both the development and test instance of each service mount the same shared node_modules volume. That way changes to packages in the development environment are automatically reflected in the test environment.

To set this up, we use several Docker Compose features: project names, variable substitution, Compose file merging, and external links and volumes.

Project Names

Compose prepends a project name to all of the resources that it creates in Docker so that they don’t conflict with those of other projects. This is why, in the last post, the backend and frontend services in the Compose file produced containers called todo_backend_1 and todo_frontend_1¹:

$ docker-compose ps
Name                    Command               State           Ports
---------------------------------------------------------------------------------
todo_backend_1    docker-entrypoint.sh npx n ...   Up
todo_frontend_1   docker-entrypoint.sh npx w ...   Up      0.0.0.0:8080->8080/tcp
todo_postgres_1   docker-entrypoint.sh postgres    Up      5432/tcp

By default, Compose uses the basename of its working directory, but this can be overridden with the --project-name flag. Here we’ll use --project-name todo_development and --project-name todo_test to create two projects for the development and test environments, respectively. We’ll also create a third project for the shared environment just called todo.

Variable Substitutions and Merging Compose Files

Running the same Compose file with different project names lets us create isolated and identical environments, but usually what we actually want is nearly-identical environments with small differences. For example, the development and test environments should be as similar as possible, but we need them to use different logical databases. Compose provides two useful features that enable this.

Firstly, for single-value changes, we can use variable substitution to inject environment variables from the host into the Compose file. Here we’ll use an environment variable called ENV to control the database name in the database connection string, for example.

Secondly, for larger structural changes to Compose files, the --file flag lets us load multiple Compose files, which Compose intelligently merges together. Here we’ll have a root Compose file with the shared configuration for the development and test environments, and then an environment-specific Compose file for each environment.

External Networks, Links and Volumes

To allow Compose to find resources defined in other environments, we’ll use the external and external_links properties in the Compose file. The development and test projects will refer to networks and volumes in the shared project using external, and the services’ link to the postgres database container is an external link.

Putting this all together, we need three new Compose files and some modifications to our existing Compose file from part 2, which we’ll take in turn. First, there’s the compose file for the shared environment, which sets up the networks, the postgres database, and the node_modules volumes.

`docker-compose.shared.yml`

version: '3.7'

networks:
  development_default:
  test_default:

services:
  postgres:
    image: postgres
    environment:
      POSTGRES_HOST_AUTH_METHOD: trust
    networks:
      - development_default
      - test_default

volumes:
  backend_node_modules:
  frontend_node_modules:

We’ll run this docker-compose.shared.yml Compose file with the project name set to todo, so the development_default network here will have the todo prefix added, yielding todo_development_default as its full name in Docker. The test network will similarly end up being called todo_test_default. The shared postgres sits on both networks.

The volumes similarly pick up this todo prefix and so end up being called todo_backend_node_modules and todo_frontend_node_modules in Docker.

The main docker-compose.yml file then has to change to reference these shared resources, as follows:

`docker-compose.yml`

diff --git a/todo/docker-compose.yml b/todo/docker-compose.yml
index 37983ad..69fbc29 100644
--- a/todo/docker-compose.yml
+++ b/todo/docker-compose.yml
@@ -1,15 +1,22 @@
 version: '3.7'

+# Use the network set up in the shared compose file.
+networks:
+  default:
+    external: true
+    name: todo_${ENV}_default
+
 services:
   backend:
     build:
       context: .
       target: development-backend
     command: npx nodemon server.js
-    depends_on:
-      - postgres
     environment:
+      DATABASE_URL: postgres://postgres:postgres@postgres/${ENV}
       PORT: 8080
+    external_links:
+      - todo_postgres_1:postgres
     volumes:
       - ./backend:/srv/todo/backend
       - backend_node_modules:/srv/todo/backend/node_modules
@@ -24,17 +31,15 @@ services:
     environment:
       HOST: frontend
       PORT: 8080
-    ports:
-      - '8080:8080'
     volumes:
       - ./frontend:/srv/todo/frontend
       - frontend_node_modules:/srv/todo/frontend/node_modules

-  postgres:
-    image: postgres:12
-    environment:
-      POSTGRES_HOST_AUTH_METHOD: trust
-
+# Use the node_modules volumes set up in the shared compose file.
 volumes:
   backend_node_modules:
+    name: todo_backend_node_modules
+    external: true
   frontend_node_modules:
+    name: todo_frontend_node_modules
+    external: true

The main changes here are to:

Redefine the default network as external and point it to the appropriate network from the shared environment. Here we use our ENV environment variable, which is set to either development or test, and Compose variable substitution to switch between the two networks. Our bin scripts will handle setting this ENV variable, as we’ll see shortly.
Remove the postgres service and instead point the backend service at the shared postgres service using an external link. The external link has to refer to the full container name, which due to the shared environment’s todo prefix turns out to be todo_postgres_1; however, we alias it to just postgres within the current environment, so nothing else needs to know about this. Removing the postgres service also means that we have to remove the backend service’s dependency on it; instead we have to take responsibility for making sure that postgres starts up. The bin scripts will again help with this.
Mark the named node_modules volumes as external and point them at the shared volumes.
Stop exposing port 8080 on the host for the frontend by default. We’re going to run the development and test environments at the same time, so they can’t both expose the same port.

This brings us to our third and fourth Compose files, which are specific to the development and test environments, and are fortunately very short. The development Compose file lets us expose 8080 for the frontend service only in the development environment:

`docker-compose.development.yml`

version: '3.7'

services:
  frontend:
    ports:
      - '8080:8080'

The test Compose file doesn’t (yet) have anything to add on top of the common configuration from the root docker-compose.yml, so it is empty but for its Compose version number:

`docker-compose.test.yml`

version: '3.7'

The `bin` Scripts

At this point you may be thinking that it would be a pain to list out all these Compose files and environment variables on the command line, and you would be right. It’s no longer practical to run docker-compose directly — instead we will run it only through helper scripts.

The first of these scripts is bin/dc, which wraps the docker-compose command so it takes as its first argument the environment we want it to run in:

`bin/dc`

#!/usr/bin/env bash

#
# Run docker-compose in the given environment.
#
# Usage: bin/dc <d[evelopment]|t[est]|s[shared]> <arguments for docker-compose>
#

source "${BASH_SOURCE%/*}/.helpers.sh"

docker_compose_in_env "$@"

It supports abbreviations, so bin/dc d ps or bin/dc development ps will both run docker-compose ps in the development environment.

The lynchpin of the whole approach is the docker_compose_in_env function, which is defined in helper file so other bin scripts can use it, so let’s look at that:

`bin/.helpers.sh`

set -e

# Run docker-compose in the given environment.
function docker_compose_in_env {
  local ENV=$(get_full_env_name $1)
  case $ENV in
  development | test )
    ENV=$ENV docker-compose \
      --project-name todo_$ENV \
      --file docker-compose.yml \
      --file docker-compose.$ENV.yml \
      "${@:2}"
    ;;
  shared )
    docker-compose \
      --project-name todo \
      --file docker-compose.shared.yml "${@:2}" ;;
  * ) echo "Unexpected environment name"; exit 1 ;;
  esac
}

# Get the full environment name, allowing shorthand.
function get_full_env_name {
  case $1 in
    d | development ) echo development ;;
    t | test ) echo test ;;
    s | shared ) echo shared ;;
    * ) echo "Expected environment d[evelopment]|t[est]|s[hared]"; exit 1 ;;
  esac
}

In the development and test environments, the docker_compose_in_env function

sets the Compose project name to todo_development or todo_test,
sets the ENV variable to either development or test for Compose to use for variable substitutions, and
loads both the root docker-compose.yml Compose file and the appropriate environment-specific Compose file.

It then passes the rest of its arguments on to docker-compose with a somewhat cryptic "${@:2}", which makes sense when you know that $@ expands to all arguments, and the :2 says we should drop the first one, $1, which is the environment name.

The shared environment is simpler: it runs docker-compose with the project name set to just todo ² and loads the shared Compose file, docker-compose.shared.yml.

Another important script that uses the docker_compose_in_env function is the bin/up script, which is responsible for starting up the environments. It requires the following changes, compared to last time with a single environment:

`bin/up`

diff --git a/todo/bin/up b/todo/bin/up
index 3fa0a0e..d4d6d75 100755
--- a/todo/bin/up
+++ b/todo/bin/up
@@ -1,20 +1,20 @@
 #!/usr/bin/env bash

-set -e
+source "${BASH_SOURCE%/*}/.helpers.sh"

-docker-compose up -d postgres
+docker_compose_in_env shared up -d

 WAIT_FOR_PG_ISREADY="while ! pg_isready --quiet; do sleep 1; done;"
-docker-compose exec postgres bash -c "$WAIT_FOR_PG_ISREADY"
+docker_compose_in_env shared exec postgres bash -c "$WAIT_FOR_PG_ISREADY"

 for ENV in development test
 do
   # Create database for this environment if it doesn't already exist.
-  docker-compose exec postgres \
+  docker_compose_in_env shared exec postgres \
     su - postgres -c "psql $ENV -c '' || createdb $ENV"

   # Run migrations in this environment.
-  docker-compose run --rm -e NODE_ENV=$ENV backend npx knex migrate:latest
+  docker_compose_in_env $ENV run --rm backend npx knex migrate:latest
 done

-docker-compose up -d
+docker_compose_in_env development up -d

The script essentially does the same things, but it runs the database-related commands in the shared environment, and the migrations in the development or test environment, as appropriate.

Other useful bin scripts include bin/test to run all the tests (now in the test environment), bin/stop to stop all the containers, and bin/down to tear things down (now optionally preserving the data in the shared environment).

The End-To-End Test

Now that we have a separate test environment, we’re nearly ready to add an end-to-end test into that environment. The final bits of preparation are to separate out the model layer from the backend into its own package, here called storage, and then to write an end-to-end-test package to contain the end-to-end test and its dependencies.

The reason to extract a storage package is that both the backend service and the end-to-end test will need to access the database, and that will be easier if we can share the database-related code between them. Our TODO application is actually simple enough that we could test everything using only public interfaces, using a black box approach. However, in more complicated applications, end-to-end tests often benefit from a more grey box approach — at minimum they need to be able to efficiently reset the database state in between tests, and it is helpful to be able to set up more complicated test scenarios by using the model layer directly.

The full diff for splitting out the storage package is long, but essentially it moves the database dependencies, configuration, migrations and test helpers, and our model class, Task, into the new package:

$ tree storage
storage
├── index.js
├── knexfile.js
├── migrations
│   └── 20190720190344_create_tasks.js
├── node_modules
├── package-lock.json
├── package.json
├── src
│   ├── knex.js
│   └── task.js
└── test
    └── support
        ├── cleanup.js
        └── knex-hook.js

The backend service’s package.json then gains a local path dependency on the storage package, instead of depending on the database packages directly:

`backend/package.json`

diff --git a/todo/backend/package.json b/todo/backend/package.json
index 36855a1..40c2363 100644
--- a/todo/backend/package.json
+++ b/todo/backend/package.json
@@ -13,9 +13,7 @@
   "dependencies": {
     "body-parser": "^1.19.0",
     "express": "^4.17.1",
-    "knex": "^0.19.5",
-    "objection": "^1.6.11",
-    "pg": "^7.12.1"
+    "storage": "file:../storage"
   },
   "devDependencies": {
     "mocha": "^6.2.0",

The local path dependency lets us update the storage package locally without having to publish it to npm every time, which would be extremely tedious. This approach also requires the usual changes to the Dockerfile to npm install the package and changes to the docker-compose.yml file to bind mount the source for the storage package into the containers that need it.

Now we can create the end-to-end-test package that also depends on the storage package and declares our other test dependencies, which are here mocha to drive the test and puppeteer to drive the headless browser that will exercise the frontend. Its Dockerfile requires some specific setup to allow run puppeteer in a container, but it is well documented.

So, finally, here is the end-to-end test:

`end-to-end-test/test/todo.test.js`

const assert = require('assert')
const puppeteer = require('puppeteer')

const { Task } = require('storage')
const cleanup = require('storage/test/support/cleanup')

const BASE_URL = process.env.BASE_URL

before(async function() {
  global.browser = await puppeteer.launch({
    executablePath: 'google-chrome-unstable',
    headless: process.env.PUPPETEER_HEADLESS !== 'false'
  })
})

after(async function() {
  await global.browser.close()
})

beforeEach(cleanup.database)

describe('TO DO', function() {
  this.timeout(10000)

  beforeEach(async function() {
    await Task.query().insert([{ description: 'foo' }, { description: 'bar' }])
  })

  it('lists, creates and completes tasks', async function() {
    const page = await global.browser.newPage()
    await page.goto(BASE_URL)

    await waitForNumberOfTasksToBe(page, 2)
    let tasks = await page.$$('.todo-task')

    // Complete task foo.
    assert.strictEqual(await getTaskText(tasks[0]), 'foo')
    let complete = await tasks[0].$('button')
    complete.click()
    await waitForNumberOfTasksToBe(page, 1)

    // Create a new task, baz.
    await page.type('.todo-new-task input[type=text]', 'baz')
    await page.click('.todo-new-task button')
    await page.waitForSelector('.todo-task button')
    await waitForNumberOfTasksToBe(page, 2)

    // Complete task baz.
    tasks = await page.$$('.todo-task')
    assert.strictEqual(await getTaskText(tasks[1]), 'baz')
    complete = await tasks[1].$('button')
    complete.click()
    await waitForNumberOfTasksToBe(page, 1)

    // Only bar should remain.
    tasks = await page.$$('.todo-task')
    assert.strictEqual(await getTaskText(tasks[0]), 'bar')
  })
})

async function waitForNumberOfTasksToBe(page, n) {
  await page.waitForFunction(
    `document.querySelectorAll(".todo-task").length == ${n}`
  )
}

async function getTaskText(elementHandle) {
  return elementHandle.$eval('span', node => node.innerText)
}

Key points:

A global puppeteer instance, global.browser is shared by all the tests (though here there’s only one test).
The BASE_URL environment variable points it at the frontend container (in the test environment), which proxies requests from the browser through to the backend container, as discussed in the previous post.
Before each test, it uses functionality from the shared storage package to first reset the database state, with cleanup.database, and then seed the database with two starter Tasks, foo and bar.
The test itself is based mainly on query selectors. For example, page.$$('.todo-task') finds all of the elements on the page with the todo-task CSS class, which are the list items for the tasks, as per the frontend. More on this later.
The waitForNumberOfTasksToBe helper function uses puppeteer’s waitForFunction primitive to poll until the given number of tasks are on the page. This is the main way in which the test handles the asynchronous loading and rendering of data.

Now, to run the test, we need to tell Compose how to start a container for it. To do this we create a separate Compose file that we’ll merge together with the other Compose files using the --file flag discussed above:

`docker-compose.end-to-end-test.yml`

version: '3.7'

services:
  end-to-end-test:
    build:
      context: .
      dockerfile: end-to-end-test/Dockerfile
    cap_add:
      - SYS_ADMIN # for puppeteer
    command: npm test
    depends_on:
      - frontend
    environment:
      DATABASE_URL: postgres://postgres:postgres@postgres/test
      BASE_URL: http://frontend:8080
    volumes:
      - ./end-to-end-test:/srv/todo/end-to-end-test
      - end_to_end_test_node_modules:/srv/todo/end-to-end-test/node_modules
      - ./storage:/srv/todo/storage
      - storage_node_modules:/srv/todo/storage/node_modules

volumes:
  end_to_end_test_node_modules:
  storage_node_modules:
    name: todo_storage_node_modules
    external: true

Notably:

This Compose file defines an end-to-end-test ‘service’ that we’ll only ever run as a one-off command, with docker-compose run instead of docker-compose up.
The end-to-end-test service defined here depends on the frontend service, which is defined in the main Compose file. It therefore depends on the backend service too, through the frontend. Compose will automatically bring up all the application services required for the end-to-end test. It won’t automatically bring up the postgres service in the shared environment, however, because it’s external — we have to run the bin/up script to make it work.
It sets the test database URL and the base URL where it can find the frontend as environment variables.
Like other node services, the source is bind mounted into the container, and we again apply the node modules trick.

So, we’ve again reached a point in this series where we can run npm test in a container:

bin/up
bin/dc t --file docker-compose.end-to-end-test.yml run --rm end-to-end-test

This runs docker-compose in the test environment (t) with our extra Compose file and runs the tests in the container. It’s a lot to type, so we’ll usually run it through the bin/test script, which runs all the tests:

`bin/test`

#!/usr/bin/env bash

source "${BASH_SOURCE%/*}/.helpers.sh"

docker_compose_in_env test run --rm backend npm test "$@"
docker_compose_in_env test run --rm frontend npm test "$@"
docker_compose_in_env test --file docker-compose.end-to-end-test.yml \
  run --rm end-to-end-test

The last line runs the end-to-end tests:

$ bin/up
...
$ bin/test
...
Starting todo_test_backend_1 ... done
Starting todo_test_frontend_1 ... done

> end-to-end-test@1.0.0 test /srv/todo/end-to-end-test
> mocha --timeout 10000



  TO DO
    ✓ lists, creates and completes tasks (1724ms)


  1 passing (2s)

Revisiting a Frontend Integration Test

In the previous post about frontend testing, we had a frontend integration test running with jsdom that made fairly heavy use of request mocking to simulate responses from the backend. Now that we have the infrastructure for end-to-end tests, we have the option of just letting the jsdom tests talk to the backend, so let’s see how that looks.

The main change required is that the frontend tests (but not the frontend itself) also needs to load the storage package and to be able to talk to the database and the backend. We can add the required config to the test environment Compose file that we stubbed out earlier:

`docker-compose.test.yml`

version: '3.7'

services:
  frontend:
    environment:
      DATABASE_URL: postgres://postgres:postgres@postgres/test
      BASE_URL: http://backend:8080

Then we can remove the mocking from the frontend integration test, which makes it much more succinct:

`frontend/test/integration/todo.test.js`

diff --git a/todo/frontend/test/integration/todo.test.js b/todo/frontend/test/integration/todo.test.js
index bb0d6dd..bd0a56e 100644
--- a/todo/frontend/test/integration/todo.test.js
+++ b/todo/frontend/test/integration/todo.test.js
@@ -1,23 +1,23 @@
+import assert from 'assert'
 import React from 'react'
 import {
-  cleanup,
+  cleanup as cleanupReactTest,
   fireEvent,
   render,
   waitForElement,
   waitForElementToBeRemoved
 } from '@testing-library/react'

-import fetchMock from '../support/fetch-mock'
+import { Task } from 'storage'
+import { database as cleanupDatabase } from 'storage/test/support/cleanup'
+
 import App from '../../src/component/app'

 describe('TO DO App', function() {
-  afterEach(cleanup)
-  afterEach(fetchMock.reset)
+  beforeEach(cleanupDatabase)
+  afterEach(cleanupReactTest)

   it('lists, creates and completes tasks', async function() {
-    // Load empty list.
-    fetchMock.getOnce('path:/api/tasks', { tasks: [] })
-
     const { getByText, getByLabelText } = render(<App />)

     const description = getByLabelText('new task description')
@@ -25,49 +25,24 @@ describe('TO DO App', function() {

     await waitForElementToBeRemoved(() => getByText(/loading/i))

-    // Create 'find keys' task.
-    fetchMock.postOnce('path:/api/tasks', {
-      task: { id: 1, description: 'find keys' }
-    })
-    fetchMock.getOnce('path:/api/tasks', {
-      tasks: [{ id: 1, description: 'find keys' }]
-    })
     fireEvent.change(description, { target: { value: 'find keys' } })
     fireEvent.click(addTask)

     await waitForElement(() => getByText('find keys'))

-    // Create 'buy milk' task.
-    fetchMock.postOnce('path:/api/tasks', {
-      task: { id: 2, description: 'buy milk' }
-    })
-    fetchMock.getOnce('path:/api/tasks', {
-      tasks: [
-        { id: 1, description: 'find keys' },
-        { id: 2, description: 'buy milk' }
-      ]
-    })
     fireEvent.change(description, { target: { value: 'buy milk' } })
     fireEvent.click(addTask)

     await waitForElement(() => getByText('buy milk'))

-    // Complete 'buy milk' task.
-    fetchMock.deleteOnce('path:/api/tasks/2', 204)
-    fetchMock.getOnce('path:/api/tasks', {
-      tasks: [{ id: 1, description: 'find keys' }]
-    })
-
     fireEvent.click(getByLabelText('mark buy milk complete'))

     await waitForElementToBeRemoved(() => getByText('buy milk'))

-    // Complete 'find keys' task.
-    fetchMock.deleteOnce('path:/api/tasks/1', 204)
-    fetchMock.getOnce('path:/api/tasks', { tasks: [] })
-
     fireEvent.click(getByLabelText('mark find keys complete'))

     await waitForElementToBeRemoved(() => getByText('find keys'))
+
+    assert.strictEqual(await Task.query().resultSize(), 0)
   })
 })

Compared to the fully end-to-end test we wrote with puppeteer above, which depended on a lot of CSS classes to make the query selectors work, this mostly end-to-end test with jsdom uses React Test Library matchers that are hopefully much less fragile and more closely resemble how the user uses the application, improving the fidelity of the test ³. One disadvantage is that we lose the ability to run the frontend integration test in a normal browser, where loading the storage package is impossible, but we could still run the rest of the frontend tests in a normal browser. End-to-end testing with jsdom instead of a full headless browser is worth considering for many applications.

Conclusions

In this post we’ve added end-to-end tests to our example TODO list application. We’ve seen how to:

Create separate test and development environments with Docker Compose, while still sharing some resources between environments for efficiency.
Write scripts to help manage the more complicated docker-compose commands required for this approach.
Extract the model layer into its own package to allow tests to access the database.
Use puppeteer via Docker in the test environment for an end-to-end test using a headless browser.
Use jsdom in the test environment for a lighter weight approach to (mostly) end-to-end testing, blurring the boundary somewhat between frontend integration testing and end-to-end testing.

Next time, we’ll expand into the world of multiple backend services and explore some strategies for managing and testing them. Until then, happy testing!

If you’ve read this far, you should follow me on twitter, or maybe even apply to work at Overleaf. :)

Footnotes

The numeric suffix is so you can run multiple instances of the same container with the --scale flag to docker-compose up. We won’t use this feature here. ↩
One disadvantage of hard coding the project name in this way is that you can’t run two instances of the project in different folders, which the default Compose behavior of using the directory name as the project name allows. Setting it based on $(basename $(pwd)) would bring this back. ↩
There is a package, pptr-testing-library, that brings some DOM Testing Library-style queries to Puppeteer. At the time of writing, some of the key features around waitForElement are not yet implemented, but it’s a good start and a promising direction! ↩

Testing with Node and Docker Compose, Part 2: On the Frontend

Sun, 12 Jan 2020 20:00:00 +0000

This post is the second in a short series about automated testing in node.js web applications with Docker. Last time, we looked at backend testing; this time, we’ll look at frontend testing.

To illustrate, we’ll continue building and testing our example application: a simple TODO list manager. So far, we’ve built and tested a RESTful API to manage the task list:

# Create a task 'foo'.
$ curl --header 'Content-Type: application/json' \
  --data '{"description": "foo"}' \
  http://localhost:8080/api/tasks
{"task":{"description":"foo","id":1}}

# List the tasks, which now include 'foo'.
$ curl http://localhost:8080/api/tasks
{"tasks":[{"id":1,"description":"foo"}]}

# Complete task 'foo' by its ID.
$ curl -X DELETE http://localhost:8080/api/tasks/1

In this post, we’ll build and test a frontend that is hopefully more user friendly than curl! Here’s what it will look like:

In particular, this post will cover:

building a simple frontend with React, Bootstrap and webpack,
building the frontend with a multi-stage Dockerfile, for both development and production,
running backend and frontend containers in Docker Compose, and
frontend testing with jsdom, React Testing Library, fetch-mock and testdouble.js.

Subsequent posts will extend the approach developed here to include end-to-end testing and then multiple backend services.

The code is available on GitHub, and each post in the series has a git tag that marks the corresponding code.

The TODO List Manager Frontend

Let’s start with a tour of the frontend we’ll be developing and testing. The frontend comprises three React components, App, NewTask and Task, and a taskStore singleton. Here’s an overview of how they interact:

The taskStore is a singleton that holds the frontend’s copy of the task list and makes requests to the backend API in response to user actions. (The term ‘store’ comes from redux, but this application doesn’t actually use redux.)
App is a coordinating component. It renders the top level UI, a NewTask component and one Task component for each task. It initiates the initial request for tasks when it’s rendered, and it listens for the updated list of tasks from the taskStore. It puts the task list into its state, so when the task list changes, React will re-render the relevant parts of the UI.
The NewTask component is responsible for creating new tasks, and the Task component is responsible for displaying a task allowing the user to mark it as completed. Both components call through to the taskStore when the user takes an action, which then updates the task list, which triggers a re-render as required.

Here’s what some of the code looks like:

`frontend/src/task-store.js`

const TASKS_API_ROOT = '/api/tasks'

class TaskStore {
  constructor() {
    this.tasks = []
    this.listener = () => {}
  }

  listen(listener) {
    this.listener = listener
  }

  unlisten() {
    this.listener = () => {}
  }

  async list() {
    const response = await fetchJson(rootUrl())
    if (!response.ok) throw new Error('Failed to list tasks')
    const body = await response.json()
    this.tasks = body.tasks
    this.listener()
  }

  async create(description) {
    const response = await fetchJson(rootUrl(), {
      method: 'POST',
      body: JSON.stringify({ description })
    })
    if (!response.ok) {
      if (response.status === 400) {
        const body = await response.json()
        throw new Error(body.error.message)
      }
      throw new Error('Failed to create task')
    }
    await this.list()
  }

  async complete(id) {
    const response = await fetchJson(itemUrl(id), { method: 'DELETE' })
    if (!response.ok) throw new Error(`Failed to complete ${id}`)
    await response.text() // ignore
    await this.list()
  }
}

async function fetchJson(url, options) {
  const response = await fetch(url, {
    headers: {
      Accept: 'application/json',
      'Content-Type': 'application/json'
    },
    ...options
  })
  return response
}

function rootUrl() {
  return new URL(TASKS_API_ROOT, window.location.origin)
}

function itemUrl(id) {
  if (!id) throw new Error(`bad id: ${id}`)
  return new URL(id.toString(), rootUrl() + '/')
}

const taskStore = new TaskStore()

export default taskStore

A few notable points:

The taskStore singleton implements a very simple event system: one listener can call taskStore.listen with a callback to be notified when the task list changes. In this case, there’s only one listener, the App component, so this is sufficient, but of course a more general event system could be used if needed. Using events here helps reduce coupling between the store and the rest of the frontend; in particular, the logic in the store doesn’t depend on React.
The taskStore could try to be smart and update its local tasks array in response to create and complete calls, but for now it just re-requests the whole task list from the backend whenever it knows the list has changed. This keeps the store simpler.
It uses fetch to talk to the backend through a small wrapper function, fetchJson, which sets up the required headers. And it uses URL to build the API URLs, to slightly reduce the amount of manual URL string munging required.

Next let’s look at the App component, which uses the task list from the store:

`frontend/src/component/app.js`

import React from 'react'

import NewTask from './new-task'
import Task from './task'

import taskStore from '../task-store'

export default class App extends React.Component {
  constructor(props) {
    super(props)
    this.state = {
      tasks: null,
      listError: null
    }
  }

  componentDidMount() {
    taskStore.listen(() => this.setState({ tasks: taskStore.tasks }))
    this._listTasks()
  }

  componentWillUnmount() {
    taskStore.unlisten()
  }

  async _listTasks() {
    try {
      await taskStore.list()
    } catch (listError) {
      this.setState({ listError })
    }
  }

  render() {
    let listItems
    if (this.state.listError) {
      listItems = (
        <li className="list-group-item">
          Failed to load tasks. Please refresh the page.
        </li>
      )
    } else {
      if (this.state.tasks) {
        listItems = this.state.tasks.map(({ id, description }) => (
          <Task id={id} description={description} key={id} />
        ))
      } else {
        listItems = <li className="list-group-item">Loading&hellip;</li>
      }
    }

    return (
      <div className="container">
        <div className="row">
          <div className="col">
            <h1 className="mt-5 mb-3 text-center">TO DO</h1>
          </div>
        </div>
        <div className="row">
          <div className="col">
            <ul className="list-group">
              <NewTask />
              {listItems}
            </ul>
          </div>
        </div>
      </div>
    )
  }
}

It looks fairly long, but most of it is just error handling, latency compensation and some rather verbose HTML for bootstrap styles.

The component starts off in a ‘Loading…’ state with no tasks.
It asks the taskStore to start the first data fetch in the background and notify it of the results, using React’s componentDidMount and componentWillUnmount lifecycle methods.
Once the data comes back, it will render the list, or, if an error occurred display a (not particularly great) error message.

The child components handle most of the UI. Let’s look at the Task component:

`frontend/src/component/task.js`

import PropTypes from 'prop-types'
import React, { useState, useEffect } from 'react'

import taskStore from '../task-store'

const Task = ({ id, description }) => {
  const [completing, setCompleting] = useState(false)

  useEffect(() => {
    let unmounted = false

    async function complete() {
      try {
        await taskStore.complete(id)
      } catch (err) {
        alert(err.message)
      } finally {
        if (!unmounted) setCompleting(false)
      }
    }
    if (completing) complete()

    return () => {
      unmounted = true
    }
  }, [completing])

  return (
    <li className="list-group-item todo-task">
      <form
        onSubmit={e => {
          setCompleting(true)
          e.preventDefault()
        }}
      >
        <p>
          <span>{description}</span>
          <button
            className="btn btn-success float-right"
            type="submit"
            style={{ minWidth: '3em' }}
            disabled={completing}
            aria-label={`mark ${description} complete`}
          >
            ✓
          </button>
        </p>
      </form>
    </li>
  )
}

Task.propTypes = {
  id: PropTypes.number,
  description: PropTypes.string
}

export default Task

This is a smaller component, but again there are a few things to comment on:

Since the component is mostly driven by its props and has relatively little state, I’ve written it as a functional component that uses React hooks. It has one state variable, completing, that tracks whether there is currently a request in progress and disables the button.
Because the taskStore updates the list asynchronously, the taskStore.complete call can finish after the component has been unmounted, in which case trying to update the state is an error. The unmounted variable tracks when the component is unmounted, to avoid this.
The button sets an aria-label to be friendlier to screen readers, for improved accessibility. We’ll also see that it’s helpful in testing, later.

The NewTask component is similar.

Finally, to bring it together, we also need an index.html file and an entrypoint to load up all our various polyfills and render the App component.

Packaging Backend and Frontend

Now that we have the frontend code, let’s see how to build it and get it working with the backend express application that we developed in part 1. The main points are:

Put the backend and the frontend into separate packages. Each can then have its own package.json file and dependencies (and dev dependencies). Frontend applications these days have a lot of dependencies that are not needed on the backend. npm audit reports that our todo demo app frontend pulls in 10,997 packages, most of which are related to webpack.
In development, run the backend and the frontend in separate containers. The backend container runs the backend service as usual, and the frontend container, which is what our browser will talk to, runs an instance of webpack-dev-server configured to serve the frontend and proxy the API endpoints through to the backend container.
For production, build the frontend using a multistage Dockerfile shared between the backend and the frontend. This lets us build the frontend in one stage with webpack and then copy the resulting build artifacts from dist into the production backend image in a later stage, so the backend can simply serve them as static files.

This may be clearer in diagram form:

Let’s start with the multistage Dockerfile responsible for producing the images for both the backend and the frontend. It uses the techniques that I covered in the first Docker post to avoid running as root, and to the use of slim images for smaller image sizes, so I’ll skip the details related to those things and talk about the overall structure below.

`Dockerfile`

#
# Backend for Development
#
FROM node:12 AS development-backend

RUN mkdir -p /srv/todo/backend && chown -R node:node /srv/todo

USER node

WORKDIR /srv/todo/backend

COPY --chown=node:node backend/package.json backend/package-lock.json ./

RUN npm install --quiet

#
# Frontend for Development
#
FROM node:12 AS development-frontend

RUN mkdir -p /srv/todo/frontend/dist && chown -R node:node /srv/todo

USER node

WORKDIR /srv/todo/frontend

COPY frontend/package.json frontend/package-lock.json ./

RUN npm install --quiet

#
# Frontend Build for Production
#
FROM development-frontend AS build-frontend

COPY frontend .

RUN npm run build

#
# Backend for Production
#
FROM node:12-slim AS production

USER node

WORKDIR /srv/todo/backend

COPY --from=development-backend --chown=root:root /srv/todo/backend/node_modules ./node_modules

COPY --from=build-frontend --chown=root:root /srv/todo/frontend/dist ./dist

COPY . .

CMD ["node", "server.js"]

The stages are laid out linearly in the file, but it may be easier to follow in this graph, which shows what each stage derives or copies files from:

The development-backend stage installs the dependencies for the backend ¹. It doesn’t actually copy the application source into the image, because we’ll instead bind mount those files into the container with Docker Compose, below.
The development-frontend stage does the same for the frontend.
The build-frontend stage runs the webpack build. It starts from development-frontend stage rather than a node base image, so it can use the packages we just installed, and it copies in the frontend source files, which are the inputs for the webpack build.
The production stage copies in the dependencies from the development-backend so it can run the backend express application, and it copies the dist folder, which is the output of the webpack build, so it can serve the frontend ².

In development, we won’t actually build the production stage, so it’s just the development-backend and development-frontend that we’ll need in this post. Those will be referenced in the docker-compose file, which will change from part 1 to the following, with an additional frontend service:

`docker-compose.yml`

version: '3.7'

services:
  backend:
    build:
      context: .
      target: development-backend
    command: npx nodemon server.js
    depends_on:
      - postgres
    environment:
      PORT: 8080
    volumes:
      - ./backend:/srv/todo/backend
      - backend_node_modules:/srv/todo/backend/node_modules

  frontend:
    build:
      context: .
      target: development-frontend
    command: npx webpack-dev-server
    depends_on:
      - backend
    environment:
      HOST: frontend
      PORT: 8080
    ports:
      - '8080:8080'
    volumes:
      - ./frontend:/srv/todo/frontend
      - frontend_node_modules:/srv/todo/frontend/node_modules

  postgres:
    image: postgres:12
    environment:
      POSTGRES_HOST_AUTH_METHOD: trust

volumes:
  backend_node_modules:
  frontend_node_modules:

Key points are:

The name of the Dockerfile stage targeted by the backend service changed from just development to development-backend, as discussed above.
The new frontend service runs webpack-dev-server. It depends on the backend service so it can proxy requests for the API back to the backend.
The exposed port 8080 moved from the backend service to the frontend service, so we can view the frontend in the browser.
The node_modules volume trick described in the first Docker post is repeated for the frontend, so there are now two volumes that contain the node_modules. I have also changed the paths to reflect the separation into backend and frontend packages.

Finally, we need the to set up webpack.config.js to do the proxying. Here’s the relevant part of the webpack config that proxies requests from the browser to the frontend under /api through to the backend service:

`frontend/webpack.config.js`

// ... setup ...

module.exports = {
  mode: process.env.NODE_ENV === 'production' ? 'production' : 'development',
  devServer: {
    // Allow connections from outside the container (not much use otherwise).
    host: process.env.HOST || '0.0.0.0',
    // Proxy api routes through to the todo backend.
    proxy: [
      {
        context: ['/api'],
        target: `http://backend:${process.env.PORT}`
      }
    ]
  },
  entry: './src/index.js',
  // ... more config ...

With all that in place, we can now bring up the frontend and backend with the bin/up script from part 1, and see the UI on http://localhost:8080. Here’s that gif again:

The Tests

Now that we have a frontend, let’s look at how to test it. I’ve written three types of tests, model tests, component tests, and integration tests. Like on the backend, these types of tests can be understood in terms of the different layers of the Model-View-Controller (MVC) architecture.

The model tests test the model layer, which contains the application’s core business logic. In this case, the entire model layer for the frontend is essentially the TaskStore class.

The component tests test the controller and view layers. In React applications, the controller and view layers tend to be combined together with in the same component. For example, the Task component has logic for dealing with in progress operations and error handling, as well as the JSX code that actually generates the DOM for the UI. So, at least for a React application, it makes sense to test these two layers together, component-by-component.

Finally, the integration tests test that the models and components work together as expected. In diagram form:

Notably, the frontend tests don’t include the backend. I have instead faked the HTTP requests to the backend API for frontend testing; more on this in the conclusion.

Integration Tests

Let’s start with the integration tests. In this case, there’s only one, which is essentially a ‘happy path’ test for the whole frontend — listing, creating and completing tasks.

`frontend/test/integration/todo.test.js`

import React from 'react'
import {
  cleanup,
  fireEvent,
  render,
  waitForElement,
  waitForElementToBeRemoved
} from '@testing-library/react'

import fetchMock from '../support/fetch-mock'
import App from '../../src/component/app'

describe('TO DO App', function() {
  afterEach(cleanup)
  afterEach(fetchMock.reset)

  it('lists, creates and completes tasks', async function() {
    // Load empty list.
    fetchMock.getOnce('path:/api/tasks', { tasks: [] })

    const { getByText, getByLabelText } = render(<App />)

    const description = getByLabelText('new task description')
    const addTask = getByLabelText('add task')

    await waitForElementToBeRemoved(() => getByText(/loading/i))

    // Create 'find keys' task.
    fetchMock.postOnce('path:/api/tasks', {
      task: { id: 1, description: 'find keys' }
    })
    fetchMock.getOnce('path:/api/tasks', {
      tasks: [{ id: 1, description: 'find keys' }]
    })
    fireEvent.change(description, { target: { value: 'find keys' } })
    fireEvent.click(addTask)

    await waitForElement(() => getByText('find keys'))

    // Create 'buy milk' task.
    fetchMock.postOnce('path:/api/tasks', {
      task: { id: 2, description: 'buy milk' }
    })
    fetchMock.getOnce('path:/api/tasks', {
      tasks: [
        { id: 1, description: 'find keys' },
        { id: 2, description: 'buy milk' }
      ]
    })
    fireEvent.change(description, { target: { value: 'buy milk' } })
    fireEvent.click(addTask)

    await waitForElement(() => getByText('buy milk'))

    // Complete 'buy milk' task.
    fetchMock.deleteOnce('path:/api/tasks/2', 204)
    fetchMock.getOnce('path:/api/tasks', {
      tasks: [{ id: 1, description: 'find keys' }]
    })

    fireEvent.click(getByLabelText('mark buy milk complete'))

    await waitForElementToBeRemoved(() => getByText('buy milk'))

    // Complete 'find keys' task.
    fetchMock.deleteOnce('path:/api/tasks/1', 204)
    fetchMock.getOnce('path:/api/tasks', { tasks: [] })

    fireEvent.click(getByLabelText('mark find keys complete'))

    await waitForElementToBeRemoved(() => getByText('find keys'))
  })
})

The test uses React Testing Library, which provides test helpers that simulate a user interacting with the React application. The library’s motto is “The more your tests resemble the way your software is used, the more confidence they can give you.”, which I generally agree with.

The getByText and getByLabelText functions returned by React Testing Library’s render function let us query for UI elements using text visible to the user or, in some cases, screen readers via aria properties. The fireEvent function lets us then click on or type in those elements. Finally, the asynchronous waitForElement and waitForElementToBeRemoved functions let the test wait for an expected change to the UI.
The test starts by rendering the top level App component, which causes the TaskStore to request the task list from the API. The test uses fetchMock before rendering the component to set up a fake response that returns an empty task list when the TaskStore requests it. The test then largely repeats in this pattern — mock the requests we expect, manipulate the UI, and then wait for it to reach the expected state.

Model Tests

As mentioned, this application’s model layer is essentially all in one class, TaskStore. Since the integration tests exercise the happy path well, including getting the TaskStore to make requests and verifying them with fetchMock, the model tests can focus mainly on error handling.

`frontend/test/model/task-store.test.js`

import assert from 'assert'
import td from 'testdouble'

import fetchMock from '../support/fetch-mock'
import taskStore from '../../src/task-store'

describe('TaskStore', function() {
  afterEach(fetchMock.reset)

  afterEach(function() {
    taskStore.unlisten()
  })

  it('lists tasks', async function() {
    const listener = td.function()
    taskStore.listen(listener)

    fetchMock.getOnce('path:/api/tasks', {
      tasks: [{ id: 1, description: 'foo' }, { id: 2, description: 'bar' }]
    })
    await taskStore.list()
    assert(taskStore.tasks.length === 2)

    td.verify(listener())
  })

  it('handles 500 error on listing', async function() {
    taskStore.listen(shouldNotCallListener)
    fetchMock.getOnce('path:/api/tasks', 500)
    try {
      await taskStore.list()
    } catch (err) {
      assert.strictEqual(err.message, 'Failed to list tasks')
    }
  })

  it('handles 400 error on task creation', async function() {
    taskStore.listen(shouldNotCallListener)
    fetchMock.postOnce('path:/api/tasks', {
      status: 400,
      body: { error: { message: 'test: create failed' } }
    })
    try {
      await taskStore.create('')
    } catch (err) {
      assert.strictEqual(err.message, 'test: create failed')
    }
  })

  it('handles 500 error on task creation', async function() {
    taskStore.listen(shouldNotCallListener)
    fetchMock.postOnce('path:/api/tasks', 500)
    try {
      await taskStore.create('find keys')
    } catch (err) {
      assert.strictEqual(err.message, 'Failed to create task')
    }
  })

  it('handles 500 error on task completion', async function() {
    taskStore.listen(shouldNotCallListener)
    fetchMock.deleteOnce('path:/api/tasks/1', 500)
    try {
      await taskStore.complete(1)
    } catch (err) {
      assert.strictEqual(err.message, 'Failed to complete 1')
    }
  })

  function shouldNotCallListener() {
    throw new Error('should not call listener')
  }
})

Like the integration tests, these tests use fetchMock to mock the HTTP requests that the store makes. They also use (one small part of) testdouble to verify that the TaskStore’s listener is called at the expected times, namely when the UI should be updated with the new task list.
The first model test is a ‘sanity test’ that overlaps a bit with the integration tests, in that it’s a ‘happy path’ test. We could write similar happy path model tests for the other public methods of the task store, but it would not be add much value in this case — the happy path for the single consumer of this TaskStore is relatively simple and well covered ³ by an integration test, so testing it exhaustively at model level would yield diminishing returns. If the ‘happy path’ were more complex, for example with more branching into many ‘happy paths’, it might make sense to only test some of the happy paths at integration test level and test the rest at model level. And if there were many consumers for this TaskStore, having more complete model testing might help to clarify the contract that it has with those consumers.
The remaining tests are for error conditions. fetchMock makes it easy to simulate errors from the backend, such as status 400 or 500 responses — a good use case for mocking.

Component Tests

Finally, we have the component tests. The ‘happy path’ integration test did exercise the components, but it didn’t make any assertions about latency compensation or error handling, which is where most of the complexity in the components comes from. So, these areas are a good focus for the component tests. Let’s look at the tests for the Task component:

`frontend/test/component/task.test.js`

import React from 'react'
import td from 'testdouble'
import { cleanup, fireEvent, render, wait } from '@testing-library/react'

import Task from '../../src/component/task'
import taskStore from '../../src/task-store'

describe('Task', function() {
  afterEach(cleanup)
  afterEach(td.reset)

  const testId = 123
  const testDescription = 'find keys'

  let complete, completeResolve, completeReject
  beforeEach(function() {
    const { getByText } = render(
      <Task id={testId} description={testDescription} />
    )
    complete = getByText('✓')

    const taskStoreComplete = td.replace(taskStore, 'complete')
    const completePromise = new Promise((resolve, reject) => {
      completeResolve = resolve
      completeReject = reject
    })
    td.when(taskStoreComplete(testId)).thenReturn(completePromise)

    fireEvent.click(complete)
  })

  it('completes a task', async function() {
    // The button should be disabled while we're submitting.
    await wait(() => complete.disabled)
    completeResolve()
    await wait(() => !complete.disabled)
  })

  it('handles failure to complete', async function() {
    const alert = td.replace(global, 'alert') // yes, I'm using alert

    // The button should be disabled while we're submitting.
    await wait(() => complete.disabled)
    completeReject(new Error('test message'))
    await wait(() => !complete.disabled)

    // Show the user the error message (ideally would be friendlier).
    td.verify(alert('test message'))
  })
})

The component tests use React Testing Library to render the component under test and make assertions about it, and testdouble to mock the interactions with the TaskStore. We could instead include the TaskStore in the system under test and mock the requests it makes with fetch-mock, but since the taskStore singleton presents a relatively simple interface for mocking, and we’d have to mock somewhere, I’ve gone with mocking at the model level here. (See also part 1 for a discussion of when and what to mock.)
Both tests here relate to what happens when the user clicks the complete button for the task, so the code to set up the state for the test is in a shared beforeEach. If we had a wider range of states and behaviors to set up, some of it would probably be better moved to a separate describe with its own beforeEach, but here I’ve just put it all together to keep it simple (YAGNI).
The tests check that the button is disabled (to prevent double clicks) and re-enabled in both the success and error cases, and that an error message is shown in the error case.

The tests for the NewTask component and the App component are similar.

One thing that these component tests don’t cover is how things look, or indeed anything about the structure of the DOM they generate, other than the presence of appropriate buttons or text. Assertion-based testing like that above tends to be quite time-consuming when used at that level of detail, both initially to create and over time to update when anything changes. And, in my experience, subtle bugs still tend to slip through, for example due to CSS making things invisible or unclickable. So, here I’ve left the visual testing to manual testing. Another approach that seems interesting is snapshot testing, in which we just record what was rendered so that any changes can be noted, but I have not so far used it in anger. Some of the visual testing can also be achieved with end-to-end tests, which will be the subject of the next post in this series.

Running the Tests

So, we have finally reached the part where we can run npm test in a container. In particular, the tests will run in the frontend container. They are set up to run in node with jsdom and babel, which requires some setup when running mocha. This allows us to run the tests on the command line in a simulated browser environment, which is quicker to set up than a full browser.

$ docker-compose run --rm frontend npm test
Starting todo_postgres_1 ... done
Starting todo_backend_1     ... done

> frontend@1.0.0 test /srv/todo/frontend
> mocha --require test/setup



  App
    ✓ handles a failure to list tasks (587ms)

  NewTask
    ✓ creates a new task
    ✓ handles failure to create

  Task
    ✓ completes a task
    ✓ handles failure to complete

  TO DO App
    ✓ lists, creates and completes tasks (198ms)

  TaskStore
    ✓ lists tasks
    ✓ handles 500 error on listing (49ms)
    ✓ handles 400 error on task creation
    ✓ handles 500 error on task creation
    ✓ handles 500 error on task completion


  11 passing (1s)

That said, it’s nice to also be able to run the tests in a real browser, and there is a handy webpack loader, mocha-loader that can handle this. I added a short script to get it to expose the test runner on another port using Docker Compose’s --publish flag:

`bin/browser-test`

#!/usr/bin/env bash

#
# Run the frontend tests in a browser on http://localhost:8181 .
#

set -e

docker-compose run --rm --publish 8181:8181 --use-aliases frontend \
  npx webpack-dev-server 'mocha-loader!./test/index.js' \
  --port 8181 --hot --inline --output-filename test.js

One caveat is that we need to provide an entrypoint, test/index.js, that imports all the other tests, rather than letting mocha find them on disk itself. This is not so bad, and it’s still possible to run individual tests with --grep. Here’s what it looks like in the browser:

Conclusion

We’ve built, packaged and tested a frontend application using webpack and React, and we’ve wired it up to our backend using Docker and Docker Compose. In particular, we’ve seen that putting the backend and the frontend into the same multistage Dockerfile allows Docker to copy the frontend build artifacts to the backend so the backend can serve them, and that webpack-dev-server’s proxy feature is great for exposing the backend API to the frontend.

We’ve seen three kinds of frontend tests that, like the tests on the backend, focus on different layers of the model-view-controller architecture. In particular, model tests test the frontend model layer, mocking out the HTTP requests to the backend. Component tests test the view and controller layers, mocking out the model layer. And a ‘happy path’ integration test checks that the models and components work together, again mocking the backend requests and responses.

This highlights a potential problem with our tests: we could change the way the backend works in such a way that the frontend would break, and the tests would all still pass, because the mocked requests in the frontend tests would be out of date. Next time, we’ll set up some end-to-end tests that will test the backend and frontend together. See you then!

Onward to part 3!

If you’ve read this far, you should follow me on twitter, or maybe even apply to work at Overleaf. :)

Footnotes

This Dockerfile copies the backend dev dependencies (in addition to the runtime dependencies) into the production image. If you would rather not do this, it can be avoided by adding an additional build-backend step that installs only the production dependencies. However, because the Dockerfile is already quite long, I’ll leave this as an exercise for the reader. ↩
This approach of bundling the single latest version of the frontend in the production is very clean and container-y and works well at small scale. However, assuming webpack is set up to use asset hashing, you can easily get into a situation where changes to the frontend will result in 404s on assets during or soon after a deploy, because clients are asking for old versions of assets that aren’t in the new image. There are a few ways to fix this:
- If using blue-green deployments, putting a pull-mode CDN or caching reverse proxy, such as CloudFlare, in front of the application should in most cases keep old versions of the frontend assets around for long enough. Another option is to add a build cache to the image build process to copy some older versions of the frontend assets in, as well as the latest version. This is similar to how Heroku’s build packs and build cache work.
- If using rolling deployments, which are popular in Kubernetes environments, a push-mode CDN is required, because otherwise requests for new assets might land on pods that are still running the old code, resulting in 404s on the new assets instead of the old ones. The approach here would be to extract the assets from the image before it’s deployed, e.g. with docker cp from a temporary container, and push them up to the CDN. Or you could do a two-phase deploy, with the old backend plus new assets and then the new backend plus new assets.
It’s never easy. ↩
This article is long enough without also talking about measuring code coverage, but I think it is a good thing to do. Here it’s simple enough to see which tests cover which code, but in a larger application it is often less clear. Code coverage is particularly helpful when the time comes to optimize the test suite. The process of identifying redundant tests or those that add little coverage relative to their running times can then be quantified and to some extent automated. It can also help identify dead code to be deleted. ↩

Testing with Node and Docker Compose, Part 1: On the Backend

Sat, 19 Oct 2019 22:00:00 +0000

My last post covered the basics of how to get a node.js application running in Docker. This post is the first in a short series about automated testing in node.js web applications with Docker.

It boils down to running npm test in a Docker container, which may not seem like it should require multiple blog posts! However, as an application gets more complicated and requires more kinds of testing, such as frontend testing and end-to-end testing across multiple services, getting to npm test can be nontrivial. Fortunately, Docker and Docker Compose provide tools that can help.

To illustrate, we’ll build and test an example application: a small TODO list manager. Here’s what it will look like when finished, at the end of the series:

In this post, we’ll start with the backend, which is a small node.js service that provides a RESTful API for managing the task list. In particular, we’ll:

cover some (opinionated) background on web application testing,
set up a node.js service and a database for development with Docker Compose,
write shell scripts to automate some repetitive docker-compose commands,
see how to set up and connect to separate databases for development and test, and
finally, run npm test in the container!

Subsequent posts will extend the approach developed here to include frontend and end-to-end testing, and then to multiple services. The Compose setup in this post is pretty standard and provides the foundation from which we’ll build up to using some more advanced features as the application grows.

The code is available on GitHub, and each post in the series has a git tag that marks the corresponding code.

The TODO List Manager Backend

Let’s start with a quick tour of the service we’ll be developing and testing. It uses PostgreSQL for the datastore and Express for the web server. For convenient database access, it uses knex.js with Objection.js as the object-relational mapping layer.

It has just one model: a Task on the TODO list. Each task has an identifier and a description, which can’t be null and must have a sensible length. Here’s the code:

`src/task.js`

const { Model } = require('objection')

require('./knex') // ensure database connections are set up

class Task extends Model {
  static get tableName() {
    return 'tasks'
  }

  static get jsonSchema() {
    return {
      type: 'object',
      required: ['description'],

      properties: {
        id: { type: 'integer' },
        description: { type: 'string', minLength: 1, maxLength: 255 }
      }
    }
  }
}

module.exports = Task

The service exposes a RESTful API for managing the tasks, which is implemented in the usual way:

`src/app.js`

const bodyParser = require('body-parser')
const express = require('express')

const Task = require('./task')

const app = express()

app.use(bodyParser.json())

// Is the service up?
app.get('/status', (req, res) => res.sendStatus(204))

// List tasks.
app.get('/api/tasks', async (req, res, next) => {
  try {
    const tasks = await Task.query().orderBy('id')
    res.json({ tasks })
  } catch (error) {
    next(error)
  }
})

// Create a new task.
app.post('/api/tasks', async (req, res, next) => {
  try {
    const task = await Task.query().insert({
      description: req.body.description
    })
    res.json({ task })
  } catch (error) {
    if (error instanceof Task.ValidationError) {
      res.status(400).json({ error: { message: error.message } })
      return
    }
    next(error)
  }
})

// Check the id route param looks like a valid id.
app.param('id', (req, res, next, id) => {
  if (/^\d+$/.test(req.params.id)) return next()
  res.sendStatus(404)
})

// Complete a task (by deleting it from the task list).
app.delete('/api/tasks/:id', async (req, res, next) => {
  try {
    await Task.query().deleteById(req.params.id)
    res.sendStatus(204)
  } catch (error) {
    next(error)
  }
})

module.exports = app

The API endpoints are relatively thin wrappers around the ORM, because there’s not much logic in a TODO list app, but we’ll still find some worthwhile things to test.

Behind the scenes, there is also some boilerplate for database connection strings, database access, database migrations, and running the server.

Dockerfile and Compose for Development

Next let’s see how to get the service running in development. I’ve followed the approach in my previous post to set up Dockerfile that handles getting node and its dependencies installed, so here let’s focus on the Compose file:

`docker-compose.yml`

version: '3.7'

services:
  todo:
    build:
      context: .
      target: development
    command: npx nodemon server.js
    depends_on:
      - postgres
    environment:
      PORT: 8080
    ports:
      - '8080:8080'
    volumes:
      - .:/srv/todo
      - todo_node_modules:/srv/todo/node_modules

  postgres:
    image: postgres:12
    environment:
      POSTGRES_HOST_AUTH_METHOD: trust

volumes:
  todo_node_modules:

It’s short but fairly dense. Let’s break it down:

The Compose file defines two services, our todo API service and its database, postgres.
The todo service is built from the current directory, using the development stage of the multi-stage Dockerfile, like in my last post.
The command runs the service in the container under nodemon, so it will restart automatically when the code changes in development.
The todo service depends_on the postgres database; this just ensures that the database is started whenever the todo service starts.
The todo service uses the PORT environment variable to decide what port to listen on in the container, here 8080. The ports key then tells compose to expose port 8080 in the container on port 8080 on the host, so we can access the service on http://localhost:8080.
The volumes are set up to allow us to bind the service’s source files on the host into the container for fast edit-reload-test cycles, like in my last post.
There’s not much to the postgres service, because we’re using it as it comes. It’s worth noting that the image is fixed to version 12, which is the latest at the time of writing. It’s a good idea to fix a version (at least a major version) to avoid accidental upgrades.

Development Helper Scripts

Now that Compose is set up, let’s see how to use it. While not strictly speaking required, it is often helpful to write some shell scripts to automate common tasks. This saves on typing and helps with consistency. The most interesting script is the bin/up script, which handles the initial setup and can also be safely re-run to make sure you are up to date after pulling in remote changes:

`bin/up`

#!/usr/bin/env bash

set -e

docker-compose up -d postgres

WAIT_FOR_PG_ISREADY="while ! pg_isready --quiet; do sleep 1; done;"
docker-compose exec postgres bash -c "$WAIT_FOR_PG_ISREADY"

for ENV in development test
do
  # Create database for this environment if it doesn't already exist.
  docker-compose exec postgres \
    su - postgres -c "psql $ENV -c '' || createdb $ENV"

  # Run migrations in this environment.
  docker-compose run --rm -e NODE_ENV=$ENV todo npx knex migrate:latest
done

docker-compose up -d

Taking it from the top:

The set -e tells the script to abort on the first error, instead of continuing and getting into even deeper trouble. All shell scripts should start with this.
It starts up the database and then runs the built-in postgres pg_isready utility in a loop, waiting until postgres finishes starting up (which is usually pretty quick, but not instant). If we didn’t do this, subsequent commands that need the database might fail sporadically.
Then it creates two databases, one called development and one called test and runs the database migrations in each one. The application uses the connection string for the appropriate database depending on what NODE_ENV it is running in. More on this later.
Finally it brings up the rest of the application with up -d, which runs it detached, in the background.

The application is just an API at this point, so here’s what it looks like when exercised with curl on localhost:8080 — an interface that only a developer could love:

# Create a task 'foo'.
$ curl --header 'Content-Type: application/json' \
  --data '{"description": "foo"}' \
  http://localhost:8080/api/tasks
{"task":{"description":"foo","id":1}}

# List the tasks, which now include 'foo'.
$ curl http://localhost:8080/api/tasks
{"tasks":[{"id":1,"description":"foo"}]}

# Complete task 'foo' by its ID.
$ curl -X DELETE http://localhost:8080/api/tasks/1

The Tests

Now that we’ve seen the service, let’s look at the tests. I’ve chosen two kinds of tests for the example service, model tests and integration tests. These terms are borrowed from Ruby on Rails, which I think encourages an approach to testing that is sound for many kinds of web applications.

Model tests test the ‘model’ layer of the Model-View-Controller (MVC) architecture, which most web applications follow to at least some degree. The model layer contains the application’s core business logic. The controllers are responsible for translating between the model layer and the view layer, which comprises, for most web applications, HTML or JSON responses to HTTP requests. Integration tests test that the models, controllers and views work together. In diagram form:

Note that both model and integration tests have access to the database; the database is not mocked, because it is an essential part of the application. There are certainly cases where mocks are the right tool for the job, but I think the primary database is rarely one of them. I’ve included an appendix with some further discussion on this point.

Model Tests

So, let’s see some examples of model tests. The example app doesn’t have much in the way of core business logic, so these particular model tests border on trivial. However, they illustrate at least one thing that model tests often do: they exercise the Task model to check that some valid data can be inserted and some invalid data can’t be:

`test/model/task.test.js`

const assert = require('assert')

const cleanup = require('../support/cleanup')

const Task = require('../../src/task')

describe('Task', function() {
  beforeEach(cleanup.database)

  it('can be created with a valid description', async function() {
    const description = 'a'.repeat(255)
    const task = await Task.query().insert({ description })
    assert.strictEqual(task.description, description)
  })

  it('must have a description', async function() {
    try {
      await Task.query().insert({
        description: ''
      })
      assert.fail()
    } catch (error) {
      assert(error instanceof Task.ValidationError)
      assert(/should NOT be shorter than 1 characters/.test(error.message))
    }
  })

  it('must not have an overly long description', async function() {
    try {
      await Task.query().insert({
        description: 'a'.repeat(1000)
      })
      assert.fail()
    } catch (error) {
      assert(error instanceof Task.ValidationError)
      assert(/should NOT be longer than 255 characters/.test(error.message))
    }
  })
})

A few remarks:

The tests are written with mocha and the built-in node assertions. (It is usually worthwhile to use a library such as chai for more kinds of assertions, but I didn’t want to overload the example app with tons of libraries, and node’s assert does the job.)
The cleanup.database hook at the top is a helper function I wrote to clear out the database before each test. Compared with cleaning up more selectively, this is a brute force approach, but it does help avoid flakey tests by making sure each test starts with a clean slate.
These model tests are for the Task model, which is a model in the ORM sense of the word. However, model tests can test other kinds of code that aren’t coupled to the ORM and database, too. If your application affords some pure functions (woo hoo!) or plain objects (also good!), you can still test those in model tests. Just omit the cleanup.database hook.

Technically, the main thing that distinguishes a model test from an integration test is that it doesn’t require spinning up or interacting with the express app. The model layer in MVC should be independent from the controller and view layers where possible — it should not care if it’s running in a background job or a service using websockets instead of plain HTTP. If it does, parts of it probably belong in the controller or view layer, where they will be easier to test with integration tests.

Integration tests

That brings us to integration tests, in which we do start up the express app and test it primarily by making HTTP requests. I wrote another test helper to start the express application in a global mocha before hook, so it starts once for the whole test suite. The helper also puts a testClient object on mocha’s this with convenience methods for making requests against the app, such as this.testClient.post. Here are some integration tests:

`test/integration/todo.test.js`

const assert = require('assert')

const cleanup = require('../support/cleanup')

const Task = require('../../src/task')

// Ensure the global test server is started, for this.testClient.
require('../support/test-server')

describe('todo', function() {
  beforeEach(cleanup.database)

  describe('with existing tasks', function() {
    let exampleTasks
    beforeEach(async function() {
      exampleTasks = await Task.query().insert(
        ['foo', 'bar'].map(description => ({ description }))
      )
    })

    it('lists the tasks', async function() {
      const response = await this.testClient.get('/api/tasks')
      assert(response.ok)
      const body = await response.json()
      assert.strictEqual(body.tasks.length, 2)
      assert.strictEqual(body.tasks[0].description, 'foo')
    })

    it('completes a task', async function() {
      const response = await this.testClient.delete(
        `/api/tasks/${exampleTasks[0].id}`
      )
      assert.strictEqual(response.status, 204)

      const remainingTasks = await Task.query()
      assert.strictEqual(remainingTasks.length, 1)
      assert.strictEqual(remainingTasks[0].id, exampleTasks[1].id)
    })
  })

  it('creates a task', async function() {
    const response = await this.testClient.post('/api/tasks', {
      description: 'foo'
    })
    const body = await response.json()
    assert.strictEqual(body.task.description, 'foo')
  })

  it('handles a validation error on create', async function() {
    const response = await this.testClient.post('/api/tasks', {})
    assert(!response.ok)
    assert.strictEqual(response.status, 400)
    const body = await response.json()
    assert.strictEqual(
      body.error.message,
      'description: is a required property'
    )
  })

  it('handles an invalid task ID', async function() {
    const response = await this.testClient.delete(`/api/tasks/foo`)
    assert.strictEqual(response.status, 404)
  })
})

A few talking points:

The tests generally follow a state verification pattern, in which we put the system into an initial state, provide some input to the system, and then verify the output or the final state, or both. For example, the setup for the describe('with existing tasks', ...) block creates two tasks in the database, and then the it('completes a task', ...) test makes a DELETE request and verifies that the service (1) produces the correct response code, 204, and (2) puts the database into the expected state, in which there is only one uncompleted task left.
The tests are gray box tests, in that they are allowed to reach into the database (ideally through the model layer) to affect and inspect the state of the system. Here the API is complete enough that we could write these tests as black box tests using only the public API, but that is not always the case. Having access to the model layer in integration tests gives a lot of useful flexibility.
The integration tests aim to cover all the success and error handling cases in the app’s controllers, but they don’t exhaustively test all of the possible causes of errors in the model layer, because those are covered in the model tests. For example, the it('handles a validation error on create', ...) test checks what happens if the description is missing, but there isn’t an integration test for the case where a description that is too long, because there was a model test for that.

This effect is a major contributor to the often talked about test pyramid, in which we have more model tests than integration tests. In this example, the model layer is too simple for that pattern to emerge, but the test pyramid is a good ideal to strive for in a large application. Integration tests generally take longer to run and require more effort to write than model tests, because there is much more going on — making requests, receiving and deserializing responses, etc.. Other things being equal, it’s usually best to test at the lowest level you can.

Running the Tests

With model and integration tests in hand, let’s see how to run them. We’ve seen above that the bin/up script creates separate development and test databases, so we have to set up the service to use them. This happens mainly in the knexfile:

const common = {
  client: 'postgresql'
}

module.exports = {
  development: {
    ...common,
    connection: 'postgres://postgres:postgres@postgres/development'
  },
  test: {
    ...common,
    connection: 'postgres://postgres:postgres@postgres/test'
  },
  production: {
    ...common,
    connection: process.env.DATABASE_URL
  }
}

The development and test connection strings tell the application how to connect to postgres as the default postgres user, which has default password postgres, running on the host postgres, as declared in our Compose file. (It is a bit like that buffalo buffalo sentence.) In production, we assume that the service will be provided with a DATABASE_URL environment variable, because hard coding production credentials here would be a bad idea.

Then in the service’s package.json we can set up the test scripts to run with NODE_ENV=test, which is how the service knows to connect to the test database:

"scripts": {
  "test": "NODE_ENV=test mocha 'test/**/*.test.js'",
  "test:model": "NODE_ENV=test mocha 'test/model/**/*.test.js'"
},

The test:model script runs only the model tests, which can be useful if you just want to run some model tests without the overhead of starting the express app in the global before hook, as mentioned above.

So, we are now ready to run npm test in a Docker container:

$ docker-compose run --rm todo npm test
Starting todo_postgres_1 ... done

> todo@1.0.0 test /srv/todo
> NODE_ENV=test mocha 'test/**/*.test.js'



  todo
    ✓ creates a task (123ms)
    ✓ handles a validation error on create
    ✓ handles an invalid task ID
    with existing tasks
      ✓ lists the tasks
      ✓ completes a task

  Task
    ✓ can be created with a valid description
    ✓ must have a description
    ✓ must not have an overly long description


  8 passing (349ms)

which is the name of the game for this blog post. (Or we can use the bin/test helper script, which does the same thing.)

Finally, it’s worth noting that to run a subset of the tests, mocha’s grep flag works, provided that it is after a -- delimiter to tell npm to pass it through to mocha. For example,

$ bin/test -- --grep Task

runs only the tests with names containing Task.

Conclusion

We’ve seen how to write and run model and integration tests for a simple node.js web service with Docker and Docker Compose. Model tests exercise the core business logic in the model layer, and integration tests check that the core business logic is correctly wired up to the controller and view layers. Both types of tests benefit from being able to access the database, which is not mocked.

The Docker Compose setup for this project was pretty simple — just one container for the application. We’ll see some more advanced Docker Compose usage in subsequent posts, together with scripts like bin/up to help drive them.

Next time, we’ll add a frontend so we can manage our TODO list without curl, and of course we will add some frontend tests. See you then!

Onward to part 2!

If you’ve read this far, you should follow me on twitter, or maybe even apply to work at Overleaf. :)

Appendix: Views on Testing

This series of posts makes some assumptions about the kinds of automated tests that we’re trying to write, so I should say what those assumptions are. I’ll start with the big picture and work back to the practical.

So, why do we test? We test to estimate correctness. Good testing lets us iterate quickly to improve correctness by providing accurate and cheap estimates.

I say ‘estimate’ here because we can in principle measure ‘ground truth’ correctness by letting the system loose in the real world and seeing what happens. If we can release to production quickly and get feedback quickly through great monitoring, and if the cost of system failure is low, we might not need to estimate. For example, if we deliver pictures of cats at scale, we might just ship to production and measure; if we make antilock braking systems, not so much. In most domains, it is worth investing in testing so we can accurately predict and improve correctness before we go to production.

The main way to achieve high prediction accuracy is through high test fidelity. As they say at NASA, fly as you test, test as you fly. For high fidelity, the system under test should closely resemble the one in production, and it should be tested in a way that closely resembles how it is used in production. However, fidelity usually comes at a cost.

There are two main costs to testing: the effort to create and maintain the tests, and the time to run the tests. Both are important. Software systems and their requirements change frequently, which requires developers to spend time adding and updating tests. And those tests run many, many times, which leaves developers twiddling their thumbs while they wait for test results.

Testing effectively requires finding the right tradeoff between fidelity and cost for your domain. This dynamic drives many decisions in testing. One example in this post is the decision to write both model tests and integration tests. We could just test everything with integration tests, which would be high fidelity but also high cost. Model tests are lower fidelity, in that they only test a part of the system, but they are generally easier to write and faster to run than integration tests, and hence lower cost.

The use of fakes (test doubles / mocks / stubs / etc.) is another important example; it can reduce test runtimes at the expense of lower fidelity and more effort to create and maintain the fakes. If replacing a component with a fake makes tests for other components much easier to write or faster to run without too much loss of fidelity, and it is not too much work to create and maintain the fake, it can be a good tradeoff.

Generally it makes sense to fake a component when two conditions hold: it is slow or unwieldy, and its coupling with the rest of the system is low. For example, if you want to test that your system sends an email when a user registers, you don’t need to stand up a full SMTP server and email client in your test environment; it is better to just fake the code that sends the email. Third party APIs called over HTTP often fall into this category, too, especially because there are good tools, such as vcr and polly.js, that make it easy to fake them.

One component I think it seldom makes sense to fake is the database. In most applications, coupling with the database is high, simply because some business logic is best handled by the database. The humble uniqueness constraint, for example, is very difficult to achieve without race conditions in application code but trivial with help from the database. And if you are fortunate enough to have a database that can do joins and enforce some basic data integrity constraints, you should definitely let it do that instead of endlessly rewriting that logic in application code. Given this coupling, including the database in the system under test increases fidelity and reduces costs to write and maintain tests, at the expense of longer test runtimes.

Fortunately, there are other ways to decrease test runtimes besides using fakes. A key property of testing is that it is embarrassingly parallel. Tests cases are independent by design, so it is straightforward (though not necessarily trivial) to run them in parallel. Some frameworks, such as rails, can do this out of the box, and scalable Continuous Integration services make it easy to bring a lot of compute power to bear on your tests at relatively low cost. Of course, massive parallelism doesn’t help so much if you are running the tests on your laptop, but in most cases you don’t have to rerun the whole test suite to make progress on a single feature — usually only a subset of the tests will be relevant. Then you can push the code to the beefy CI boxes to get a final check that the change hasn’t broken something in an unexpected part of the application.

So, to sum up, in this series of blog posts I’m advocating for high fidelity (don’t fake too much) and low costs to write and maintain tests (don’t spend too much time tending fakes), with high investment in testing infrastructure to offset longer test runtimes. This does certainly make test environments more complex, but Docker and Docker Compose provide many useful tools for managing this complexity, which is one of the motivations for this series of blog posts.

Lessons from Building Node Apps in Docker (2019)

Fri, 06 Sep 2019 16:00:00 +0000

Update (2019-10-10): This post was discussed on Hacker News.

Way back in 2016, I wrote Lessons from Building a Node App in Docker, which has now helped over a hundred thousand people Dockerize their node.js apps. Since then there have been many changes, both in the ecosystem and how I work with node in Docker, so it was due for an overhaul.

In this updated tutorial, we’ll set up the socket.io chat example with Docker, from scratch to production-ready. In particular, we’ll see how to:

Get started bootstrapping a node application with Docker.
Not run everything as root (bad!).
Use binds to keep your test-edit-reload cycle short in development.
Manage node_modules in a container (there’s a trick to this).
Ensure repeatable builds with package-lock.json.
Share a Dockerfile between development and production using multi-stage builds.

This tutorial assumes you already have some familiarity with Docker and node. If you’d like a gentle intro to Docker first, I’d recommend running through Docker’s official introduction.

Getting Started

We’re going to set things up from scratch. The final code is available on github here, and there are tags for each step along the way. Here’s the code for the first step, in case you’d like to follow along.

Without Docker, we’d start by installing node and any other dependencies on the host and running npm init to create a new package. There’s nothing stopping us from doing that here, but we’ll learn more if we use Docker from the start. (And of course the whole point of using Docker is that you don’t have to install things on the host.) We’ll start by creating a “bootstrapping container” that has node installed, and we’ll use it to set up the npm package for the application.

The Bootstrapping Container and Service

We’ll need to write two files, a Dockerfile and a docker-compose.yml, to which we’ll add more later on. Let’s start with the bootstrapping Dockerfile:

FROM node:10.16.3

USER node

WORKDIR /srv/chat

It’s a short file, but there already some important points:

It starts from the official Docker image for the latest long term support (LTS) node release, at time of writing. I prefer to name a specific version, rather than one of the ‘floating’ tags like node:lts or node:latest, so that if you or someone else builds this image on a different machine, they will get the same version, rather than risking an accidental upgrade and attendant head-scratching.
The USER step tells Docker to run any subsequent build steps, and later the process in the container, as the node user, which is an unprivileged user that comes built into all of the official node images from Docker. Without this line, they would run as root, which is against security best practices and in particular the principle of least privilege. Many Docker tutorials skip this step for simplicity, and we will have to do some extra work to avoid running as root, but I think it’s very important.
The WORKDIR step sets the working directory for any subsequent build steps, and later for containers created from the image, to /srv/chat, which is where we’ll put our application files. The /srv folder should be available on any system that follows the Filesystem Hierarchy Standard, which says that it is for “site-specific data which is served by this system”, which sounds like a good fit for a node app ¹.

Now let’s move on to the bootstrapping compose file, docker-compose.yml:

version: '3.7'

services:
  chat:
    build: .
    command: echo 'ready'
    volumes:
      - .:/srv/chat

Again there is quite a bit to unpack:

The version line tells Docker Compose which version of its file format we are using. Version 3.7 is the latest at the time of writing, so I’ve gone with that, but older 3.x and 2.x versions would also work fine here; in fact, the 2.x series might even be a better fit, depending on your use case ².
The file defines a single service called chat, built from the Dockerfile in the current directory, denoted .. All the service does for now is to echo ready and exit.
The volume line, .:/srv/chat, tells Docker to bind mount the current directory . on the host at /srv/chat in the container, which is the WORKDIR we set up in the Dockerfile above. This means that changes we’ll make to source files on the host will be automatically reflected inside the container, and vice versa. This is very important for keeping your test-edit-reload cycles as short as possible in development. It will, however, create some issues with how npm installs dependencies, which we’ll come back to shortly.

Now we’re ready to build and test our bootstrapping container. When we run docker-compose build, Docker will create an image with node set up as specified in the Dockerfile. Then docker-compose up will start a container with that image and run the echo command, which shows that everything is set up OK.

$ docker-compose build
Building chat
Step 1/3 : FROM node:10.16.3
# ... more build output ...
Successfully built d22d841c07da
Successfully tagged docker-chat-demo_chat:latest

$ docker-compose up
Creating docker-chat-demo_chat_1 ... done
Attaching to docker-chat-demo_chat_1
chat_1  | ready
docker-chat-demo_chat_1 exited with code 0

This output indicates that the container ran, echoed ready and exited successfully. 🎉

Initializing an npm package

⚠️ Aside for Linux users: For this next step to work smoothly, the node user in the container should have the same uid (user identifier) as your user on the host. This is because the user in the container needs to have permissions to read and write files on the host via the bind mount, and vice versa. I’ve included an appendix with advice on how to deal with this issue. Docker for Mac users don’t have to worry about it because of some uid remapping magic behind the scenes, but Docker for Linux users get much better performance, so I’d call it a draw.

Now we have a node environment set up in Docker, we’re ready to set up the initial npm package files. To do this, we’ll run an interactive shell in the container for the chat service and use it to set up the initial package files:

$ docker-compose run --rm chat bash
node@467aa1c96e71:/srv/chat$ npm init --yes
# ... writes package.json ...
node@467aa1c96e71:/srv/chat$ npm install
# ... writes package-lock.json ...
node@467aa1c96e71:/srv/chat$ exit

And then the files appear on the host, ready for us to commit to version control:

$ tree
.
├── Dockerfile
├── docker-compose.yml
├── package-lock.json
└── package.json

Here’s the resulting code on github.

Installing Dependencies

Next up on our list is to install the app’s dependencies. We want these dependencies to be installed inside the container via the Dockerfile, so the container will contain everything needed to run the application. This means we need to get the package.json and package-lock.json files into the image and run npm install in the Dockerfile. Here’s what that change looks like:

diff --git a/Dockerfile b/Dockerfile
index b18769e..d48e026 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,5 +1,14 @@
 FROM node:10.16.3

+RUN mkdir /srv/chat && chown node:node /srv/chat
+
 USER node

 WORKDIR /srv/chat
+
+COPY --chown=node:node package.json package-lock.json ./
+
+RUN npm install --quiet
+
+# TODO: Can remove once we have some dependencies in package.json.
+RUN mkdir -p node_modules

And here’s the explanation:

The RUN step with mkdir and chown commands, which are the only commands we need to run as root, creates the working directory and makes sure that it’s owned by the node user.
It’s worth noting that there are two shell commands chained together in that single RUN step. Compared to splitting out the commands over multiple RUN steps, chaining them reduces the number of layers in the resulting image. In this example, it really doesn’t matter very much, but it is a good habit not to use more layers than you need. It can save a lot of disk space and download time if you e.g. download a package, unzip it, build it, install it, and then clean up in one step, rather than saving layers with all of the intermediate files for each step.
The COPY to ./ copies the npm packaging files to the WORKDIR that we set up above. The trailing / tells Docker that the destination is a folder. The reason for copying in only the packaging files, rather than the whole application folder, is that Docker will cache the results of the npm install step below and rerun it only if the packaging files change. If we copied in all our source files, changing any one would bust the cache even though the required packages had not changed, leading to unnecessary npm installs in subsequent builds.
The --chown=node:node flag for COPY ensures that the files are owned by the unprivileged node user rather than root, which is the default ³.
The npm install step will run as the node user in the working directory to install the dependencies in /srv/chat/node_modules inside the container.

This last step is what we want, but it causes a problem in development when we bind mount the application folder on the host over /srv/chat. Unfortunately, the node_modules folder doesn’t exist on the host, so the bind effectively hides the node modules that we installed in the image. The final mkdir -p node_modules step and the next section are related to how we deal with this.

The `node_modules` Volume Trick

There are several ways around the node modules hiding problem, but I think the most elegant is to use a volume within the bind to contain node_modules. To do this, we have to add a few lines to our docker compose file:

diff --git a/docker-compose.yml b/docker-compose.yml
index c9a2543..799e1f6 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -6,3 +6,7 @@ services:
     command: echo 'ready'
     volumes:
       - .:/srv/chat
+      - chat_node_modules:/srv/chat/node_modules
+
+volumes:
+  chat_node_modules:

The chat_node_modules:/srv/chat/node_modules volume line sets up a named volume ⁴ called chat_node_modules that contains the directory /srv/chat/node_modules in the container. The top level volumes: section at the end must declare all named volumes, so we add chat_node_modules there, too.

So, it’s simple to do, but there is quite a bit going on behind the scenes to make it work:

During the build, npm install installs the dependencies (which we’ll add in the next section) into /srv/chat/node_modules within the image. We’ll color the files from the image blue:
```
/srv/chat$ tree # in image
.
├── node_modules
│   ├── accepts
...
│   └── yeast
├── package-lock.json
└── package.json
```
When we later start a container from that image using our compose file, Docker first binds the application folder from the host inside the container under /srv/chat. We’ll color the files from the host red:
```
/srv/chat$ tree # in container without node_modules volume
.
├── Dockerfile
├── docker-compose.yml
├── node_modules
├── package-lock.json
└── package.json
```
The bad news is that the node_modules in the image are hidden by the bind; inside the container, we instead see only an empty node_modules folder on the host.

However, we’re not done yet. Docker next creates a volume that contains a copy of /srv/chat/node_modules in the image, and it mounts it in the container. This, in turn, hides the node_modules from the bind on the host:

/srv/chat$ tree # in container with node_modules volume
.
├── Dockerfile
├── docker-compose.yml├── node_modules
│   ├── accepts
...
│   └── yeast├── package-lock.json
└── package.json

This gives us what we want: our source files on the host are bound inside the container, which allows for fast changes, and the dependencies are also available inside of the container, so we can use them to run the app.

We can also now explain the final mkdir -p node_modules step in the bootstrapping Dockerfile above: we have not actually installed any packages yet, so npm install doesn’t create the node_modules folder during the build. When Docker creates the /srv/chat/node_modules volume, it will automatically create the folder for us, but it will be owned by root, which means the node user won’t be able to write to it. We can preempt that by creating node_modules as the node user during the build. Once we have some packages installed, we no longer need this line.

Package Installation

So, let’s rebuild the image, and we’ll be ready to install packages.

$ docker-compose build
... builds and runs npm install (with no packages yet)...

The chat app requires express, so let’s get a shell in the container and npm install it with --save to save the dependency to our package.json and update package-lock.json accordingly:

$ docker-compose run --rm chat bash
Creating volume "docker-chat-demo_chat_node_modules" with default driver
node@241554e6b96c:/srv/chat$ npm install --save express
# ...
node@241554e6b96c:/srv/chat$ exit

The package-lock.json file, which has for most purposes replaced the older npm-shrinkwrap.json file, is important for ensuring that Docker image builds are repeatable. It records the versions of all direct and indirect dependencies and ensures that npm installs in Docker builds on different machines will all get the same dependency tree.

Finally, it’s worth noting that the node_modules we installed are not present on the host. There may be an empty node_modules folder on the host, which is a side effect of the binds and volumes we created, but the actual files live in the chat_node_modules volume. If we run another shell in the chat container, we’ll find them there:

$ ls node_modules
# nothing on the host
$ docker-compose run --rm chat bash
node@54d981e169de:/srv/chat$ ls -l node_modules/
total 196
drwxr-xr-x 2 node node 4096 Aug 25 20:07 accepts
# ... many node modules in the container
drwxr-xr-x 2 node node 4096 Aug 25 20:07 vary

The next time we run a docker-compose build, the modules will be installed into the image.

Here’s the resulting code on github.

Running the App

We are finally ready to install the app, so we’ll copy in the remaining source files, namely index.js and index.html.

Then we’ll install the socket.io package. At the time of writing, the chat example is only compatible with socket.io version 1, so we need to request version 1:

$ docker-compose run --rm chat npm install --save socket.io@1
# ...

In our docker compose file, we then remove our dummy echo ready command and instead run the chat example server. Finally, we tell Docker Compose to export 3000 in the container on the host, so we can access it in a browser:

diff --git a/docker-compose.yml b/docker-compose.yml
index 799e1f6..ff92767 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -3,7 +3,9 @@ version: '3.7'
 services:
   chat:
     build: .
-    command: echo 'ready'
+    command: node index.js
+    ports:
+      - '3000:3000'
     volumes:
       - .:/srv/chat
       - chat_node_modules:/srv/chat/node_modules

Then we are ready to run with docker-compose up ⁵:

$ docker-compose up
Recreating dockerchatdemo_chat_1
Attaching to dockerchatdemo_chat_1
chat_1 | listening on *:3000

Then you can see it running on http://localhost:3000.

Here’s the resulting code on github.

Docker for Dev and Prod

We now have our app running in development under docker compose, which is pretty cool! Before we can use this container in production, we have a few problems to solve and possible improvements to make:

Most importantly, the container as we’re building it at the moment does not actually contain the source code for the application — it just contains the npm packaging files and dependencies. The main idea of a container is that it should contain everything needed to run the application, so clearly we will want to change this.
The /srv/chat application folder in the image is currently owned and writeable by the node user. Most applications don’t need to rewrite their source files at runtime, so again applying the principle of least privilege, we shouldn’t let them.
The image is fairly large, weighing in at 909MB according to the handy dive image inspection tool. It’s not worth obsessing over image size, but we don’t want to be needlessly wasteful either. Most of the image’s heft comes from the default node base image, which includes a full compiler tool chain that lets us build node modules that use native code (as opposed to pure JavaScript). We won’t need that compiler tool chain at runtime, so from both a security and performance point of view, it would be better not to ship it to production.

Fortunately, Docker provide a powerful tool that helps with all of the above: multi-stage builds. The main idea is that we can have multiple FROM commands in the Dockerfile, one per stage, and each stage can copy files from previous stages. Let’s see how to set that up:

diff --git a/Dockerfile b/Dockerfile
index d48e026..6c8965d 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,4 +1,4 @@
-FROM node:10.16.3
+FROM node:10.16.3 AS development

 RUN mkdir /srv/chat && chown node:node /srv/chat

@@ -10,5 +10,14 @@ COPY --chown=node:node package.json package-lock.json ./

 RUN npm install --quiet

-# TODO: Can remove once we have some dependencies in package.json.
-RUN mkdir -p node_modules
+FROM node:10.16.3-slim AS production
+
+USER node
+
+WORKDIR /srv/chat
+
+COPY --from=development --chown=root:root /srv/chat/node_modules ./node_modules
+
+COPY . .
+
+CMD ["node", "index.js"]

Our existing Dockerfile steps will form the first stage, which we’ll now give the name development by adding AS development to the FROM line at the start. I’ve now removed the temporary mkdir -p node_modules step needed during bootstrapping, since we now have some packages installed.
The new second stage starts with the second FROM step, which pulls in the slim node base image for the same node version and calls the stage production for clarity. This slim image is also an official node image from Docker. As its name suggests, it is smaller than the default node image, mainly because it doesn’t include the compiler toolchain; it includes only the system dependencies needed to run a node application, which are far fewer than what may be required to build one.

This multi-stage Dockerfile runs npm install in the first stage, which has the full node image at its disposal for the build. Then it copies the resulting node_modules folder to the second stage image, which uses the slim base image. This technique reduces the size of the production image from 909MB to 152MB, which is about a factor of 6 saving for relatively little effort ⁶.
Again the USER node command tells Docker to run the build and the application as the unprivileged node user rather than as root. We also have to repeat the WORKDIR, because it doesn’t persist into the second stage automatically.
The COPY --from=development --chown=root:root ... line copies the dependencies installed in the preceding development stage into the production stage and makes them owned by root, so the node user can read but not write them.
The COPY . . line then copies the rest of the application files from the host to the working directory in the container, namely /srv/chat.
Finally, the CMD step specifies the command to run. In the development stage, the application files came from bind mounts set up with docker-compose, so it made sense to specify the command in the docker-compose.yml file instead of the Dockerfile. Here it makes more sense to specify the command in the Dockerfile, which builds it into the container.

Now that we have our multi-stage Dockerfile set up, we need to tell Docker Compose to use only the development stage rather than going through the full Dockerfile, which we can do with the target option:

diff --git a/docker-compose.yml b/docker-compose.yml
index ff92767..2ee0d9b 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -2,7 +2,9 @@ version: '3.7'

 services:
   chat:
-    build: .
+    build:
+      context: .
+      target: development
     command: node index.js
     ports:
       - '3000:3000'

This will preserve the old behavior we had before we added multistage builds, in development.

Finally, to make the COPY . . step in our new Dockerfile safe, we should add a .dockerignore file. Without it, the COPY . . may pick up other things we don’t need or want in our production image, such as our .git folder, any node_modules that are installed on the host outside of Docker, and indeed all the Docker-related files that go into building the image. Ignoring these leads to smaller images and also faster builds, because the Docker daemon does not have to work as hard to create its copy of the files for its build context. Here’s the .dockerignore file:

.dockerignore
.git
docker-compose*.yml
Dockerfile
node_modules

With all of that set up, we can run a production build to simulate how a CI system might build the final image, and then run it like an orchestrator might:

$ docker build . -t chat:latest
# ... build output ...
$ docker run --rm --detach --publish 3000:3000 chat:latest
dd1cf2bf9496edee58e1f5122756796999942fa4437e289de4bd67b595e95745

and again access it in the browser on http://localhost:3000. When finished, we can stop it using the container ID from the command above ⁷.

$ docker stop dd1cf2bf9496edee58e1f5122756796999942fa4437e289de4bd67b595e95745

Setting up `nodemon` in Development

Now that we have distinct development and production images, let’s see how to make the development image a bit more developer-friendly by running the application under nodemon for automatic reloads within the container when we change a source file. After running

$ docker-compose run --rm chat npm install --save-dev nodemon

to install nodemon, we can update the compose file to run it:

diff --git a/docker-compose.yml b/docker-compose.yml
index 2ee0d9b..173a297 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -5,7 +5,7 @@ services:
     build:
       context: .
       target: development
-    command: node index.js
+    command: npx nodemon index.js
     ports:
       - '3000:3000'
     volumes:

Here we use npx to run nodemon through npm ⁸. When we bring up the service, we should see the familiar nodemon output ⁹:

docker-compose up
Recreating docker-chat-demo_chat_1 ... done
Attaching to docker-chat-demo_chat_1
chat_1  | [nodemon] 1.19.2
chat_1  | [nodemon] to restart at any time, enter `rs`
chat_1  | [nodemon] watching dir(s): *.*
chat_1  | [nodemon] starting `node index.js`
chat_1  | listening on *:3000

Finally, it’s worth noting that with the Dockerfile above the dev dependencies will be included in the production image. It is possible to break out another stage to avoid this, but I would argue it is not necessarily a bad thing to include them. Nodemon is unlikely to be wanted in production, it is true, but dev dependencies often include testing utilities, and including those means we can run the tests in our production container as part of CI. It also generally improves dev-prod parity, and as some wise people once said, ‘test as you fly, fly as you test.’ Speaking of which, we don’t have any tests, but it’s easy enough to run them when we do:

$ docker-compose run --rm chat npm test

> chat@1.0.0 test /srv/chat
> echo "Error: no test specified" && exit 1

Error: no test specified
npm ERR! Test failed.  See above for more details.

Here’s the final code on github.

Conclusion

We’ve taken an app and got it running in development and production entirely within Docker. Great job!

We jumped through some hopefully edifying hoops to bootstrap a node environment without installing anything on the host. We also jumped through some hoops to avoid running builds and processes as root, instead running them as an unprivileged user for better security.

Node / npm’s habit of putting dependencies in the node_modules subfolder makes our lives a little bit more complicated than other solutions, such as ruby’s bundler, that install your dependencies outside the application folder, but we were able to work around that fairly easily with the nested node modules volume trick.

Finally, we used Docker’s multi-stage build feature to produce a Dockerfile suitable for both development and production. This simple but powerful feature is useful in a wide variety of situations, and we’ll see it again in some future articles.

My next article in this series will pick up where we left off about testing node.js services in Docker. See you then!

Onward to the series about testing!

Thanks to Hope Thomas, Ali Smith and Miguel Serrano for reviewing drafts of this article.

If you’ve read this far, you should follow me on twitter, or maybe even apply to work at Overleaf. :)

Appendix: Dealing with UID mismatches on Linux

When using bind mounts to share files between a Linux host and a container, you are likely to hit permissions problems if the numeric uid of the user in the container doesn’t match that of the user on the host. For example, files created on the host may not be readable or writable in the container, or vice versa.

We can work around this, but first it’s worth noting that if your uid on the host is 1000, everything is fine for Dockerized development with node. This is because Docker’s official node images use uid 1000 for the node user. You can check your uid on the host by running the id command, which prints it out. For example, mine currently says uid=1000(john) gid=1000(john) ....

A uid of 1000 is fairly common, because it is the uid assigned by the ubuntu install process. If you can convince everyone on your team to set their uid to 1000, everything will work fine. If not, here are a couple of workarounds:

Run the service as root in development by simply omitting the USER node step from the development stage of the Dockerfile (introduced in the Docker for Dev and Prod section). This ensures that the user in the container (root) will be able to read and write files on the host. If the user in the container creates any files, they’ll be owned as root on the host, but you can always fix that by running sudo chown -R your-user:your-group . on the host.

You can (and should) still run the process as an unprivileged user in production.
Use Dockerfile build arguments to configure the UID and GID of the node user at build time. We can do this by adding a few lines to the development stage of the Dockerfile:
```
FROM node:10.16.3 AS development

ARG UID=1000
ARG GID=1000
RUN \
  usermod --uid ${UID} node && groupmod --gid ${GID} node &&\
  mkdir /srv/chat && chown node:node /srv/chat

# ...
```
This introduces two build args, UID and GID, which default to the existing value of 1000 if no arguments are given, and changes the node user and group to use those IDs before creating any files as the user.

Each developer with a non-1000 uid/gid has to set these args for Docker Compose. One way to do this is to use a docker-compose.override.yml file that is not checked into version control (i.e. is in .gitignore), to set the args, like this:
```
version: '3.7'

services:
  chat:
    build:
      args:
        UID: '500'
        GID: '500'
```
In this example, the uid and gid in the container will be set to 500. There may be some easier ways of doing this one day. Again, these changes only need to be done in the development stage, not production.

Footnotes

Fundamentally, it doesn’t matter where the files go in the container. /opt would also be a very reasonable choice. Another option would be to keep them under /home/node, which simplifies some file permissions management in development but requires more typing and makes less sense in production, where I’ll advocate letting root own the application files as a way of keeping them read only. In any case, /srv will do. ↩
Both the 2.x and 3.x versions of the Docker Compose file format are still being actively developed. The main benefit of the 3.x series is that it is cross-compatible between single-node applications running on Docker Compose and multi-node applications running on Docker Swarm. In order to be compatible, version 3 drops some useful features from version 2. If you are only interested in Docker Compose, you might prefer to stick with the latest 2.x format. ↩
Some of this trickery in the Dockerfile can be removed if we allow the npm install build step to run as root. If we do, we can and should still use the unprivileged node user at runtime, which is where most of the security benefits reside. A Dockerfile to run the build as root and the container as the node user would look more like this:
```
FROM node:10.16.3

WORKDIR /srv/chat

COPY package.json package-lock.json ./

RUN npm install --quiet

USER node
```
This is cleaner, without the need for some mkdir and chown tricks, at the expense of running npm install as root at build time. Overall, I think the modest increase in complexity is worth it to avoid running the build as root, but you might decide that you prefer the cleaner Dockerfile.

One caveat if you build as root is that when you want to later install new dependencies you need to run a shell as root instead of the node user, as in docker-compose run --rm --user root chat bash and then npm install --save express. This is a bit like “sudoing” to install packages, which is a fairly familiar experience. ↩
We could instead use an anonymous volume to contain the modules, just by omitting the name:
```
diff --git a/docker-compose.yml b/docker-compose.yml
index c9a2543..5a56364 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -6,3 +6,4 @@ services:
     command: echo 'ready'
     volumes:
       - .:/srv/chat
+      - /srv/chat/node_modules
```
That would be shorter, but it is very easy to forget to clean up anonymous volumes, which leads to a profusion of anonymous modules with no indication which container they came from. You can still clean them up with docker system prune, but that is a bit of a ‘sledge hammer to crack a nut’. The named volumes approach is a bit more verbose but also more transparent.

(Extra credit: you might wonder where those dependency files in the volume actually get stored. In short, whether using named or anonymous volumes, they live in a separate directory managed by Docker on the host; see the Docker docs about volumes for more info.) ↩
The eagle eyed reader may have noticed that we don’t have to docker-compose build to get the dependencies installed before docker-compose up. This is because it is running with the node modules in the chat_node_modules named volume. The next time we do a build, npm will install the dependencies from scratch into the image, but for installing packages day-to-day, we can just run npm install in the container without having to rebuild.

If you ever find yourself in a situation where you want to get rid of the named volume and start from scratch, you can run docker volume list to get a list of all volumes. The full name of your node modules volume will depend on your docker compose project. In my case, the volume of interest is docker-chat-demo_chat_node_modules, which can be removed if we first remove the container with docker-compose rm -v chat and then the volume itself with docker volume rm docker-chat-demo_chat_node_modules. ↩
Docker also provides an official alpine image variant that is even smaller. However, these size savings come in part from using a completely different libc and package manager than the Debian-based images. Unless you are deploying to embedded systems where space is at a premium, the complexities that may arise due to these differences may not be worth it, especially when the Debian-based slim images already offer substantial savings. ↩
You might notice that it takes about 10s to stop. This is because the socket.io chat example does not correctly handle the SIGTERM signal, which Docker sends it when it’s time to stop, to perform a graceful shutdown. Extra credit: add this code to the end of index.js:
```
process.on('SIGTERM', function() {
  io.close();
  Object.values(io.of('/').connected).forEach(s => s.disconnect(true));
});
```
then rebuild the production image and try to run and docker stop the container again. It should disconnect any clients and stop promptly after that change. ↩
An alternative to using npx would be to put the command in an npm script, such as for npm start. However, running npm scripts in a container can easily cause the SIGTERM from Docker to get lost, leading to non-graceful (10s) shutdowns even when the child properly handles the signal; see ⁷. This is because older versions of npm did not pass the signal to the child, and, even though that’s now fixed, the child typically runs in a shell, and most shells (bash, at least) do not pass signals to their children. One workaround is to use exec in the script, as in exec nodemon server.js, which replaces the shell process with the node process, but it’s easy to forget. Moreover, running a command through an npm script creates some overhead, namely an additional node process. So, other things being equal, it’s best to define long running commands in the Dockerfile (with the exec form of CMD) or Docker Compose files instead. For short-running scripts, such as npm test, where signal handling is not a concern, npm scripts are usually fine. ↩
You may notice that nodemon says that typing rs will restart it. That won’t work if we use docker-compose up to bring up the service, because our terminal is not connected to nodemon’s standard input when we do that. If we run docker-compose run --rm chat instead, rs should work as usual; this can be useful when working on one service in particular. ↩

From Zero to Driverless Race Car with Deep Learning

Tue, 01 Jan 2019 16:30:00 +0000

This post is based on the following talk.

Abstract:

Learn how to train a driverless car to drive around a simulated race track using end-to-end deep learning — from camera images to steering commands. Key techniques used include deep neural networks, data augmentation, and transfer learning. This was a course project, so I'll introduce the key ideas and talk about the practical steps needed to get it working. You'll also see a lot of very dangerous driving.

Introduction

Today I’m going to talk about how to build a driverless race car using deep learning.

I should start by saying that this was a personal project, but it does have a connection to my work. By day, I’m CTO of Overleaf, the online LaTeX editor. Overleaf now has over three million users, and the first two, namely my co-founder and I, were driverless car researchers! We started Overleaf while we were working on the Heathrow Pod, which was the world’s first driverless taxi system. It launched in 2011, and it’s still running, so if you ever have some time at London’s Heathrow Airport, you should take a ride on the pods out to business parking and back to Terminal 5.

The key thing allowed us to put driverless taxis into active service way back in 2011 is that the taxis run on their own roads. It’s a closed system, and we used a fairly traditional systems engineering approach to design and build the system and prove that it was safe.

Nowadays most driverless car research is about how we get them to work safely on public roads, where we need to worry about human drivers and pedestrians and cyclists and traffic lights and stop signs, and lots of other messy things.

One of the pioneers in this area is Sebastian Thrun, who was a professor at Stanford and leader of the winning team in the 2005 DARPA Grand Challenge, which in many ways inaugurated the modern driverless car era. He later left to start Udacity, one of the leading MOOC providers, and they launched a driverless car course in 2016 ¹, the Self-Driving Car Engineer nano-degree. It sounded great and with my copious free time (!) I enrolled.

This talk is based on one of the labs in that course, which is in turn based on a 2016 paper from NVIDIA, called End-to-End Learning for Self-Driving Cars. What the NVIDIA team showed is that it’s possible to take images from a front-mounted camera on a car, feed them into a convolutional neural network, which we’ll talk about, and have it produce steering commands to drive the car. So, just like you look at the road ahead and decide whether you need to steer left or right, they did the same thing with a neural network.

What’s remarkable about this is that it’s very different from the systems engineering approach that we usually take in driverless car engineering. In that approach we break the overall problem of driving the car down into lots of subproblems, such as object detection, object classification, mapping, planning, etc., and have different subsystems responsible for each of those subproblems. The subsystems are then connected up to make the whole system. Here, however, we’re just going to train one big monolithic neural network, which will somehow drive the car. It feels a bit like magic.

Our task for this lab was to reproduce this magic, and this talk follows steps I went through to do so, which were broadly:

Collect training data in a simulator by driving around the track manually.
Train a deep neural network using the camera images and steering angles.
Use the network to drive the car around the track (also in simulation).

In the middle, I’m going to handwave quite a lot about some of the theory. You may wish to skip that part if you are already familiar with convolutional neural networks.

Collecting Training Data

The first thing we need to do is collect training data by driving the car around in the simulator. We’re going to train the neural network to imitate me, so here I am trying to drive mostly in the center of the road, like we want the neural network to do.

You can see in the top right a ‘recording’ sign, which means that we’re recording the camera images and my steering angle for each frame, 10 times each second.

You might notice that I’m not very good at this. One of the key reasons is that the only way to steer the car is with the left and right arrow keys. So, the first thing we need to do is take my raw steering inputs and clean them up by smoothing them.

The spiky blue line is the raw steering input from me pressing the arrow keys. The green and red lines are the results of applying exponential and Gaussian smoothing, respectively. I tried both kinds of smoothing, and the Gaussian smoothing turned out to work better when actually driving the car.

The next hurdle to overcome in this training process is that if I only show the car how to drive in the center of the road, it will never gain any experience of what to do if it ever finds itself off-center. We can solve this problem by recording recoveries in training, in which I stop the recording, drive off to the side of the road, start recording, and then drive back into the middle:

By doing this, we teach the car how to get back to the center of the road, hopefully without teaching it too much about driving off the road.

After driving around the track several times in both directions and doing recoveries, I ended up with about 11 thousand frames of training data: ²

Dataset	Rows (Frames)
Normal	5,273
Recoveries	6,037
Total	11,310

That is in some sense quite a lot, but in the world of deep learning, it is not very much at all, and I think we would struggle to train a network from scratch using just these data. To get around that, we’ll use a technique called transfer learning.

Transfer Learning

Transfer learning means that we take someone else’s network that they have already trained for some other task, extract a little bit of it, and repurpose it for our task.

Here we’re going to use the Inception v3 network, which was trained by Google for an image classification competition. It takes as input an image, runs lots of computations, and outputs the class of the image — that is, what kind of thing the image shows. If you feed in the image on the left, it will tell you that that is a Siberian Husky, and if you feed in the image on the right, it will tell you that that is an Eskimo Dog. (It is interesting to note that some of the classifications are quite fine; I’m not sure I could tell the difference!)

The Inception network is a very large — it has over 25 million parameters, which Google has trained with considerable effort and expense. So, how would we go about building on the work that they’ve done? The answer is surprisingly simple: we ‘lobotomize’ the network and just take the first few layers (indicated by the red dashed box below).

The reason this works is that these first few layers of the network, 44 layers to be precise, turn out to be relatively generic image processing stuff — different kinds of edge detectors, for example. It is the later stages of the full Inception network that encode the difference between Siberian Huskies and Eskimo Dogs, and all of the other image classes, which we don’t need. So, for our application, we’re just going to tack a relatively small neural network on to the end of our Inception prefix network:

That means that we only have to train these last few layers ourselves, leaving the pre-trained Inception prefix network alone, so we can hopefully get away with much less training data than if we started from scratch.

I should add the architecture of our last three layers was chosen after some trial and error with even simpler architectures, and this was the simplest one that seemed to work.

Convolutional Neural Networks

So, how does this network actually work? This is where I will talk about some of the theory. To build a convolutional neural network, which is the kind of deep neural network that this is, we need three basic building blocks:

Convolution
Resizing (in particular making the images smaller)
Activation Functions

Let’s take each of these things in turn. I’m going to use this camera frame as an example:

One way to look at it is that our overall aim for this network is to go from a camera frame at 320 by 160 pixels down to a single number, which is the steering angle that the car should apply when it sees this image.

Let’s start with our first building block, convolution. Convolution is a simple idea but also a very general one. To use it, we define what is called a kernel (it has many names, but I will call it a kernel), which is a small matrix of numbers. Here we’ll use a 3 pixel by 3 pixel kernel.

We start by lining up our 3 pixel by 3 pixel kernel with the 3 pixel by 3 pixel patch in the top left corn of the image. We then multiply each pixel in the image with the corresponding value from the kernel and add them up to give us the first pixel in the output image ³. Then we slide our kernel over the rest of the pixels in the input image, pixel by pixel, each time yielding one more pixel in the output image: ⁴

This operation is very simple and can be done efficiently, but it is also very powerful. If you use a program like Photoshop, most of the tools in the image filters menu will be using a convolution under the hood, each using a different kernel. The identity kernel is not very interesting, because it just copies the input to the output, but we can also choose kernels for edge detection, blurs, sharpens, and much more.

To use convolution in a neural network, the key insight is that instead of very carefully engineering these kernels ourselves, we let them be learned from the training data. And we let the network learn a lot of different kernels. The prefix of the Inception network that we’re using has about 700 thousand parameters, many which are are kernel parameters, which Google has trained using 1.2 million images.

When we run convolutions with so many kernels, we basically take one input image and produce lots of output images — one for each kernel. If we were to keep adding more and more images, we would run out of memory, so we also need our next building block: resizing.

This does what it says on the tin: every few convolutions, we resize the image to make it smaller. There are lots of ways of doing this, such as max pooling.

When we resize, we lose some spatial resolution, but we gain depth. It lets us take our flat input image, and by repeated convolutions and resizing, get to a sausage shaped ‘image’ that is lower resolution but also deeper. The network gains, in some handwavy sense, understanding with this depth — it started out with a patch of image that was just pixels, and now it has mapped those pixels into a set of features that can carry more meaning to help it solve the task at hand.

Our final building block is the Activation Function, which is where the ‘neural’ in ‘neural network’ comes from.

A real neuron is an incredibly complicated thing with amazing dynamics and lots of interesting behaviors, but we are going to model it in a very simple way. Essentially, it has dendrites, which are ‘inputs’ connected to upstream neurons, and if those inputs add up to at least some threshold, this neuron is going to activate and send a signal through its axon to downstream neurons.

Mathematically, we can represent this as a very simple squashing function. If the sum of the inputs is negative, it outputs a value near zero, which means that it is not activated. If the sum of the inputs is positive, it returns a value near one, which means it is activated.

These three building blocks are all we need. We just repeat them over and over again. It’s worth noting, however, that convolution (of the kind used here) is a linear operation. If it were not for the little bits of nonlinearity that we get from (most kinds of) resizing and the activation function, the composition of convolutions would collapse down to one big linear function. However, once we add these relatively tame non-linearities into the mix, we go from being able to represent only linear functions to being able to approximate any function, which is pretty amazing.

Completing the Network

So, that’s the end of my theory bit. Let’s see what it does in practice using our example camera image. When we feed it through the first 44 layers of the Inception network, we get back 256 greyscale ‘feature images’. The width and height of each feature image are about a factor of 10 smaller than the input image, but there are many more of them — as noted above, the output is much smaller and deeper than the input.

Here are nine feature images, starting mostly at random from feature image number 42, out of our stack of 256. Each image shows one of the responses that the network gives to our example input image. A light pixel means that the neuron at that point in the feature image is activated — it’s responding to some characteristic of the corresponding part of the input image. A dark pixel means it isn’t activated.

If we overlay the response with the input image, it becomes easier for us to interpret what they’re responding to:

In feature image number 42 (top left), for example, we can see some neurons responding fairly strongly to the edges of the road. When driving, it is pretty important to know where the edges of the road are, so this feature image may be useful. Number 43 (top center) seems to be responding to the road surface, which may be similarly useful. It’s also picking up some of the background, but we can also see that number 48 (middle right) is responding mainly to the background. So, it seems like some combination of these feature images would give us useful information.

It’s important to note that we didn’t tell this part of the network how to find features that might be helpful for our task. In fact, this part of the network was trained by Google for a completely different task, namely image classification, but it seems to have some features that look like they may be useful for our task, which is encouraging.

This brings us back to the bit of the network that we actually train, which are the three layers that we’ve added at the end.

Our first layer is another convolution. This is a ‘one by one’ convolution, which means a kernel size of 1 pixel by 1 pixel. A 1x1 convolution picks a set number of linear combinations of its input feature images; it is often used for dimensionality reduction. This architectural choice is motivated by the discussion above about how some linear combination of features images 42, 43 and 48 seems like it might be pretty good at finding the road — we want to let the network pick the most useful combinations of the 256 feature images from above.

In this example, we’ll pick 64 linear combinations of the 256 features. Here are some new feature images after that 1x1 convolution. They look similar, but they are generally smoother and brighter than the feature images before the 1x1 convolution.

Again we can see some features responding strongly to the edges of the road, for example feature image number 45 (out of the 64 new feature images this time).

Finally, we have two fully connected layers. From now on, we won’t be able to visualise the results so easily, because we flatten all of the outputs from the convolution into one large list of numbers. A fully connected layer then computes a set number of linear combinations of all of those numbers.

The diagram on the right shows a schematic for one of those linear combinations — that is, one neuron – in one fully connected layer. A fully connected layer consists of many such neurons, each connected to all of the outputs of the previous layer. Here the weights we train are the w_i, and the inputs are the x_i. And again we apply an activation function, here denoted f, to each neuron to introduce some nonlinearity, so our two fully connected layers don’t just collapse down into one big linear function.

The outputs of the first fully connected layer feed into the second fully connected layer, and the output of the second fully connected layer is, at last, our steering angle! With our network architecture set out, we are ready to start training.

Time to Train

Well, almost ready. Even just for three layers that we need to train, there are quite a few hyperparameters that we need to set before we can fully define the network and the training scenario. How many kernels should we use in our 1x1 convolutions? How large should each fully connected layer be? What kind of smoothing should we do on the steering angle? And many more.

There are smart ways to search hyperparameters, but in this case I just tried all possible combinations in a large grid search. It takes a while to run the whole grid, but it’s just computation — set it off before bed, and by the time you get home from work, new data are waiting.

Each point in the grid gives one network to train and evaluate. Then we can evaluate the performance of each of the networks to choose the best hyperparameter settings.

Fortunately, the actual training is made very easy by great libraries, such as Keras. Here’s an example of the Keras training output for one of the networks:

Layer (type)                     Output Shape          Param #     Connected to
====================================================================================================
convolution2d_1 (Convolution2D)  (None, 17, 37, 64)    16448       convolution2d_input_1[0][0]
____________________________________________________________________________________________________
flatten_1 (Flatten)              (None, 40256)         0           convolution2d_1[0][0]
____________________________________________________________________________________________________
dense_1 (Dense)                  (None, 32)            1288224     flatten_1[0][0]
____________________________________________________________________________________________________
dense_2 (Dense)                  (None, 1)             33          dense_1[0][0]
====================================================================================================
Total params: 1304705
____________________________________________________________________________________________________
Epoch 1/30
27144/27144 [==============================] - 160s - loss: 79.0047 - val_loss: 0.0780
Epoch 2/30
27144/27144 [==============================] - 147s - loss: 25.4130 - val_loss: 0.0692
Epoch 3/30
27144/27144 [==============================] - 148s - loss: 8.4912 - val_loss: 0.0670
Epoch 4/30
27144/27144 [==============================] - 148s - loss: 2.9383 - val_loss: 0.0638
…
Epoch 12/30
27144/27144 [==============================] - 148s - loss: 0.0851 - val_loss: 0.0572
Epoch 13/30
27144/27144 [==============================] - 148s - loss: 0.0785 - val_loss: 0.0568
Epoch 14/30
27144/27144 [==============================] - 148s - loss: 0.0802 - val_loss: 0.0546
Epoch 15/30
27144/27144 [==============================] - 147s - loss: 0.0769 - val_loss: 0.0569
Epoch 16/30
27144/27144 [==============================] - 147s - loss: 0.0793 - val_loss: 0.0560
Epoch 17/30
27144/27144 [==============================] - 148s - loss: 0.0832 - val_loss: 0.0574

There’s a lot going on in this output, and I’d like to remark on a few things. At the start of the output, we have Keras’s summary of the model we’re training, which includes numbers of parameters to fit. Remember that we’re keeping the layers from the Inception network that Google trained completely fixed, so we’re only worried about training our three layers at the end of the network. ⁵

In the second section, we have the training progress, which Keras prints as it goes. I’ve split the training data I collected into a training set (80%) and a validation set (20%). ⁶ At each Epoch, Keras reports the mean absolute error in the network’s predictions on the training set (the loss) and the validation set (the val_loss).

For each of our three layers, training starts with randomly initialized weights. As you might expect, the initial loss with random weights is pretty terrible, starting at around 79 ⁷. However, Keras uses that loss to refine the weights for the next epoch, feeds the training set through again, and sure enough the loss drops with each successive epoch. After 17 epochs, the loss is orders of magnitude lower, at 0.08. Training stops when the validation set loss, val_loss starts to increase — to run more epochs would likely lead to overfitting.

Run the Model with Lowest Validation Loss!

So, after repeating that training process on hundreds of networks for all our various hyperparameters, let’s hook up the network with the best validation set loss to the car and see how it drives!

We put the simulator into autonomous mode, then the car starts moving. I haven’t talked about throttle control, but basically it sets the throttle to keep the car at about 10mph in this case.

We can see the car weaving, and then correcting, but it overcorrects, and eventually it runs off the road and crashes. Still, it is not a bad start — it was clearly trying to stay on the road, which is encouraging.

Try Lots of Things…

And so the debugging begins. There were lots of arbitrary decisions not in my initial grid of hyperparameters, so I started by adding many more to the grid. For example:

Different loss functions — how should we measure the error? I tried mean squared error and mean absolute error; the latter seemed a bit better.
Different activation functions — I tried sigmoid, tanh and ReLU; tanh seemed a bit better.
Different regularization — penalizing the weights so that they do not become too large is a common technique to avoid overfitting, and I tried several different L2-regularization weights.
Different distributions for initial weights (reduce variance of Normal) — this did solve some convergence problems during training.
Different layer sizes — how many neurons in each hidden layer?
Different network architectures — try adding more layers? Or fewer layers?

However, none of the things I tried really moved the needle. After thrashing around for a few days without getting anywhere fast, I stopped fiddling with the neural network and added a print statement to the control loop. That quickly revealed the actual problem:

The controller was spending too much time processing each frame, so it was only actually able to steer about three times per second. If you imagine trying to steer, but you can only touch the wheel three times a second, it does seem pretty tough. Further investigation revealed that it was spending most of its time in the Inception prefix layers. One solution would have been to buy a faster laptop. However, it turned out that it was possible to use fewer inception layers, in particular the first 12 layers, instead of the first 44 (it looks like 7 in the diagram, but some are not visible):

With Lower Latency

Let’s see how it does with lower latency (about 0.1s instead of 0.35s):

It now does much better. I’ve sped up the video, because it makes it much further. The weaving is much reduced, and it’s staying mostly in the middle. It’s not following a great ‘racing line’, but that’s probably because I didn’t follow a great racing line when I was training it.

Eventually it gets to a turn where there are some trees ahead of it, and it doesn’t make the turn. Still, progress!

The resolution for this problem was basically to add more training data — I drove around that corner a few more times, and eventually it learned to make it. At around the same time, Udacity released a bunch of training data from someone who had a steering wheel, instead of having to do the arrow keys and smoothing, so I added that in too.

I also did some augmentation with the training data, which in retrospect was fairly obvious: you can mirror every image in your original training set and negate the steering angle, and you have another training example.

The Final Result

With all this additional training data, the network managed the following:

I changed the throttle control to let it drive at 30mph, which was the cap in this version of the Udacity simulator, so it’s going quite a bit faster. You can still see some ‘shimmy’ but it is no longer crashing. Success!

Does it Generalize?

So, we’ve seen that for this race track, we can train the network and make it go around. What if we put it on a totally different race track? Fortunately Udacity provided just a second track in the simulator, so let’s try it out. Note that this is exactly the same network as in the previous video, and it has not seen any part of this new track in its training data.

Behold, it drives! Again it shows sensitivity to trees, with a near miss at 0:24 and eventually a crash, both with trees in its field of view. Still, considering that this is a very simple network, as neural networks go, and it hasn’t had a huge amount of training, the fact that it does this well on an unseen track, albeit one in the same simulator, is I think pretty good.

Conclusions

In this post we:

Took a neural network designed to classify pictures of dogs.
Repurposed it to drive a car.
Taught it to drive with ~35 minutes’ training.
Watched it drive around an unseen race track (for a little while).

That’s amazing!

Moreover,

In 2015, very few people would have even thought this was possible.
In 2016, thousands of people like me were doing it in their spare time in an online course.
In 2018, I mentored (very slightly) Josh, a high school student who did it all himself on an RC car with a Raspberry Pi.

That’s also amazing!

I prepared this talk for the 2018 Holtzbrinck Publishing Group AI Day. I would like to thank the organizers for giving me the impetus to finally write this talk and for providing the video.

The code for this project is open source.

If you’ve read this far, perhaps you should follow me on twitter, or maybe even join our team. :)

Footnotes

Udacity actually ran two driverless car courses. The first was CS373: Programming a Robotic Car, which I completed in 2012, and is still offered for free. The second was the Self Driving Car Engineer nano-degree. Both were great! ↩
The simulator actually takes three camera images in each frame, one on the left wing, one in the center, and one on the right wing. I also added the wing cameras to the training data, together with an empirically determined adjustment to the target steering angle to account for the different wing camera position. In that sense the number of examples is 33,930 (3 times 11,310), but it’s not clear how much these images add, since they are very similar to the central camera image. I didn’t have time to go into this in my talk. ↩
Here I have glossed over an important detail: the input image is a color image, so it actually has three channels, one each for red, green and blue (RGB). The convolutions in convolutional neural neworks operate on all of the channels in their input, so the kernels here should actually be thought of as three-dimensional kernels in this case — 3 pixels by 3 pixels by 3 channels. I did not have time to get into this detail in my talk (and drawing that would have been hard!). ↩
This image is from An Introduction to different Types of Convolutions in Deep Learning, by Paul-Louis Pröve, which is an excellent place to find out more about convolution. ↩
Because the Inception prefix layers remain fixed through training, a useful optimisation is to run the training data through the prefix network once and save the outputs. The saved outputs are then fed into the training process for the later layers many times. Quite a lot of the code is concerned with doing this, but it does make a big difference to training times. The output features from the prefix are usually called ‘bottleneck’ features, and the slides with example features from the Inception prefix network are taken from these bottleneck features. ↩
The eagle-eyed reader may notice that Keras reports 27,144 images in the training set, which is larger than the 11,310 points I reported in my table. See ². The size of the training set is 80 percent of 11,310 samples times 3 frames per sample, which is 27,144. ↩
That is not 79 degrees; the angle has been scaled so that the range [-1, 1] corresponds to [-25˚, 25˚], so it is effectively doing donuts during the first training epoch. ↩

The Mathematics of 2048: Optimal Play with Markov Decision Processes

Sun, 18 Mar 2018 23:00:00 +0000

Updates

2018-04-10 This post was discussed on Hacker News.

So far in this series on the mathematics of 2048, we’ve used Markov chains to learn that it takes at least 938.8 moves on average to win, and we’ve explored the number of possible board configurations in the game using combinatorics and then exhaustive enumeration.

In this post, we’ll use a mathematical framework called a Markov Decision Process to find provably optimal strategies for 2048 when played on the 2x2 and 3x3 boards, and also on the 4x4 board up to the 64 tile. For example, here is an optimal player for the 2x2 game to the 32 tile:

Loading…

The random seed determines the random sequence of tiles that the game adds to the board. The ‘strategy’ the player follows is defined by a table, called a policy, that tells it which direction it should swipe in every possible board configuration. In this post, we’ll see how to construct a policy that is optimal, in the sense that it maximizes the player’s chances of reaching the target 32 tile.

It turns out that the 2x2 game to the 32 tile is very hard to win — even when playing optimally, the player only wins about 8% of the time, which probably does not make for a very fun game. The 2x2 games are qualitatively quite different to the 4x4 games, but they’ll still be useful to introduce the key ideas.

Ideally we’d be able to find an optimal policy for the full game on the 4x4 board to the 2048 tile, but as we saw in the previous post, the number of possible board configurations is very large. This makes it infeasible to construct a complete optimal policy for the full game, at least with the methods used here.

We will however be able to find an optimal policy for the shortened 4x4 game to the 64 tile, and fortunately we’ll see that optimal play on the 3x3 boards looks qualitatively similar, in an admittedly hand-wavy way, to some successful strategies for the full game.

The (research quality) code behind this article is open source.

Markov Decision Processes for 2048

Markov Decision Processes (MDPs) are a mathematical framework for modeling and solving problems in which we need to make a sequence of related decisions in the presence of uncertainty. Such problems are all around us, and MDPs find many applications in economics, finance, and artificial intelligence. For 2048, the sequence of decisions is the direction to swipe in each turn, and the uncertainty arises because the game adds new tiles to the board at random.

To set up the game of 2048 as an MDP, we will need to write it down in a specific way. This will involve six main concepts: states, actions and transition probabilities will encode the game’s dynamics; rewards, values and policies will be used to capture what the player is trying to accomplish and how they should do so. To develop these six concepts, we will take as an example the smallest non-trivial 2048-like game, which is played on the 2x2 board only up to the 8 tile. Let’s start with the first three.

States, Actions and Transition Probabilities

A state captures the configuration of the board at a given point in the game by specifying the value of the tile, if any, in each of the board’s cells. For example, is a possible state in a game on a 2x2 board. An action is swiping left, right, up or down. Each time the player takes an action, the process transitions to a new state.

The transition probabilities encode the game’s dynamics by determining which states are likely to come next, in view of the current state and the player’s action. Fortunately, we can find out exactly how 2048 works by reading its source code. Most important ¹ is the process the game uses to place a random tile on the board, which is always the same: pick an available cell uniformly at random, then add a new tile either with value 2, with probability 0.9, or value 4, with probability 0.1.

At the start of each game, two random tiles are added using this process. For example, one of these possible start states is . For each of the possible actions in this state, namely Left, Rright, Up and Down, the possible next states and the corresponding transition probabilities are ²:

In this diagram, there is an arrow for each possible transition to a successor state, on the right hand side. The weight of the arrow and the label indicate the corresponding transition probability. For example, if the player swipes right (R), both 2 tiles go to the right edge, leaving two available cells on the left. The new tile will be a 4 with probability 0.1, and it can either go into the top left or the bottom left cell, so the probability of the state is $0.1 \times 0.5 = 0.05$.

From each of those successor states, we can continue this process of enumerating their allowed actions and successor states, recursively. For , the possible successors are:

Here swiping right is not allowed, because none of the tiles can move right. Moreover, if the player reaches the successor state , highlighted in red, they have lost, because there are no allowed actions from that state. This would happen if the player were to swipe left, and the game were to place a 4 tile, which it would do with probability 0.1; this suggests that swiping left may not be the best action in this state.

For one final example, if the player moves up instead of left, one of the possible successor states is , and if we enumerate the allowed actions and and successor states from that state, we can see that swiping left or right will then result in an 8 tile, which means the game is won (highlighted in green):

If we repeat this process for all of the possible start states, and all of their possible successor states, and so on recursively until win or lose states are reached, we can build up a full model with all of the possible states, actions and their transition probabilities. For the 2x2 game to the 8 tile, that model looks like this (click to enlarge; you may then need to scroll down):

To make the diagram smaller, all of the losing states have been collapsed into a single lose state, shown as a red oval, and all of the winning states have been similarly collapsed into a single win state, shown as a green star. This is because we don’t particularly care how the player won or lost, only that they did.

Play proceeds roughly from left to right in the diagram, because the states have been organized into ‘layers’ by the sum of their tiles. A useful property of the game is that after each action the sum of the tiles on the board increases by either 2 or 4. This is because merging tiles does not change the sum of the tiles on the board, and the game always adds either a 2 tile or a 4 tile. The possible start states, which are in the layers with sum 4, 6 and 8, are drawn in blue.

Even for this smallest example, there are 70 states and 530 transitions in the model. It is possible significantly reduce those numbers, however, by observing that many of the states we’ve enumerated above are trivially related by rotations and reflections, as described in Appendix A. This observation is important in practice for reducing the size of the models so that they can be solved efficiently, and it makes for more legible diagrams, but it is not essential for us to move on to our second set of MDP concepts.

Rewards, Values and Policies

To complete our specification of the model, we need to somehow encode the fact that the player’s objective is to reach the win state ³. We do this by defining rewards. In general, each time an MDP enters a state, the player receives a reward that depends on the state. Here we’ll set the reward for entering the win state to 1, and the reward for entering all other states to 0. That is, the one and only way to earn a reward is to reach the win state. ⁴

Now that we have an MDP model of the game in terms of states, actions, transition probabilities and rewards, we are ready to solve it. A solution for an MDP is called a policy. It is basically a table that lists for every possible state which action to take in that state. To solve an MDP is to find an optimal policy, which is one that allows the player to collect as much reward as possible over time.

To make this precise, we will need our final MDP concept: the value of a state according to a given policy is the expected, discounted reward the player will collect if they start from that state and follow the policy thereafter. To explain what that means will require some notation.

Let $S$ be the set of states, and for each state $s \in S$, let $A_s$ be the set of actions that are allowed in state $s$. Let $\Pr(s’ | s, a)$ denote the probability of transitioning to each successor state $s’ \in S$, given that the process is in state $s \in S$ and the player takes action $a \in A_s$. Let $R(s)$ denote the reward for entering state $s$. Finally, let $\pi$ denote a policy and $\pi(s) \in A_s$ denote the action to take in state $s$ when following policy $\pi$.

For a given policy $\pi$ and state $s$, the value of state $s$ according to $\pi$ is \[ V^\pi(s) = R(s) + \gamma \sum_{s’} \Pr(s’ | s, \pi(s)) V^\pi(s’) \] where the first term is the immediate reward, and the summation gives the expected value of the successor states, assuming the player continues to follow the policy.

The factor $\gamma$ is a discount factor that trades off the value of the immediate reward against the value of the expected future rewards. In other words, it accounts for the time value of money: a reward now is typically worth more than the same reward later. If $\gamma$ is close to 1, it means that the player is very patient: they don’t mind waiting for future rewards; likewise, smaller values of $\gamma$ mean that the player is less patient. For now, we’ll set the discount factor $\gamma$ to 1, which matches our assumption that the player cares only about winning, not about how long it takes to win ⁵.

So, how do we find the policy? For each state, we want to choose the action that maximizes the expected future value:

\[ \pi(s) = \mathop{\mathrm{argmax}}\limits_{a \in A_s} \left\{ \sum_{s’} \Pr(s’ | s, a) V^\pi(s’) \right\} \]

So, this gives us two linked equations, and we can solve them iteratively. That is, pick an initial policy, which might be very simple, compute the value of every state under that simple policy, and then find a new policy based on that value function, and so on. Perhaps remarkably, under very modest technical conditions, such an iterative process is guaranteed to converge to an optimal policy, $\pi^*$, and an optimal value function $V^{\pi^*}$ with respect to that optimal policy.

This standard iterative approach works well for the MDP models for games on the 2x2 board, but it breaks down for the 3x3 and 4x4 game models, which have many more states and therefore take much more memory and compute power. Fortunately, it turns out that we can exploit the particular structure of our 2048 models to solve these equations much more efficiently, as described in Appendix B.

Optimal Play on the 2x2 Board

We’re now ready to see some optimal policies in action! If you leave the random seed at 42 and press the Start button below, you’ll see it reach the 8 tile in 5 moves. The random seed determines the sequence of new tiles that the game will place; if you choose a different random seed by clicking the ⟲ button, you will (usually) see a different game unfold.

Loading…

For the 2x2 game to the 8 tile, there is not actually much to see. If the player follows the optimal policy, they will always win. (As we saw above, when building the transition probabilities, if the player does not play optimally, it is possible to lose.) This is reflected in the fact that the value of the state remains at 1.00 for the whole game — when playing optimally, there is at least one action in every reachable state that leads to a win.

If we instead ask the player to play to the 16 tile, a win is no longer assured even when playing optimally. In this case, picking a new random seed should lead to a win 96% of the time for the game to the 16 tile, so I’ve set the initial seed to one of the rare seeds that leads to a loss.

Loading…

As a result of setting the discount factor, $\gamma$, to 1, the value of each state also conveniently tells us the probability of winning from that state. Here the value starts at 0.96 and then eventually drops to 0.90, because the outcome hinges on the next tile being a 2 tile. Unfortunately, the game delivers a 4 tile, so the player loses, despite playing optimally.

Finally, we’ve previously established that the largest reachable tile on the 2x2 board is the 32 tile, so let’s see the corresponding optimal policy. Here the probability of winning drops to only 8%. (This is the same game and the same policy used in the introduction, but now it’s interactive.)

Loading…

It’s worth remarking that each of the policies above is an optimal policy for the corresponding game, but there is no guarantee of uniqueness. There may be many optimal policies that are equivalent, but we can say with certainty that none of them are strictly better.

If you’d like to explore these models for the 2x2 game in more depth, Appendix A provides some diagrams that show all the possible paths through the game.

Optimal Play on the 3x3 Board

On the 3x3 board, it is possible to play up to the 1024 tile, and that game has some 25 million states. Drawing an MDP diagram like we did for the 2x2 games is therefore clearly out of the question, but we can still watch an optimal policy in action ⁶:

Loading…

Much like the 2x2 game to 32, the 3x3 game to 1024 is very hard to win — if playing optimally, the probability of winning is only about 1%. For some less frustrating entertainment, here also is the game to 512, for which the probability of winning if playing optimally is much higher, at about 74%:

Loading…

At the risk of anthropomorphizing a large table of states and actions, which is what a policy is, I see here elements of strategies that I use when I play 2048 on the 4x4 board ⁷. We can see the policy pinning the high value tiles to the edges and usually corners (though in the game to 1024, it often puts the 512 tile in the middle of an edge). We can also see it being ‘lazy’ — even when it has two high value tiles lined up to merge, it will continue merging lower value tiles. Particularly within the tight constraints of the 3x3 board, it makes sense that it will take the opportunity to increase the sum of its tiles at no risk of (immediately) losing — if it gets stuck merging smaller tiles, it can always merge the larger ones, which opens up the board.

It’s important to note that we did not teach the policy about these strategies or give it any other hints about how to play. All of its behaviors and any apparent intelligence emerges solely from solving an optimization problem with respect to the transition probabilities and reward function we supplied.

Optimal Play on the 4x4 Board

As established in the last post, the game to the 2048 tile on the 4x4 board has at least trillions of states, and so far it has not been possible to even enumerate all the states, let alone solve the resulting MDP for an optimal policy.

We can, however, complete the enumeration and the solve for the 4x4 game up to the 64 tile — that model has “only” about 40 billion states. Like the 2x2 game to 8 above, it is impossible to lose when playing optimally. This does not make for very interesting viewing, because in many cases there are several good actions, and the choice between them is arbitrary.

However, if we reduce the discount factor, $\gamma$, that makes the player slightly impatient, so that they prefer to win sooner rather than later. It then looks a bit more directed. Here is an optimal player for $\gamma = 0.99$:

Loading…

The value starts around 0.72; the exact initial value reflects the expected number of moves it will take to win from the randomly selected start state. It gradually increases with each move, as the reward for reaching the win state gets closer. Again the policy shows good use of the edges and corners to build sequences of tiles in an order that’s convenient to merge.

Conclusion

We’ve seen how to represent the game of 2048 as a Markov Decision Process and obtained provably optimal policies for the smaller games on the 2x2 and 3x3 boards and a partial game on the 4x4 board.

The methods used here require us to enumerate all of the states in the model in order to solve it. Using efficient strategies for enumerating the states and efficient strategies for solving the model makes this feasible for models with up to 40 billion states, which was the number for the 4x4 game to 64. The calculations for that model took roughly one week on an OVH HG-120 instance with 32 cores at 3.1GHz and 120GB RAM. The next-largest 4x4 game, played up to the 128 tile, is likely to contain many times that number of states and would require many times the computing power. Calculating a provably optimal policy for the full game to the 2048 tile will likely require different methods.

It is common to find that MDPs are too large to solve in practice, so there are a range of proven techniques for finding approximate solutions. These typically involve storing the value function and/or policy approximately, for example by training a (possibly deep) neural network. They can also be trained on simulation data, rather than requiring enumeration of the full state space, using reinforcement learning methods. The availability of provably optimal policies for smaller games may make 2048 a useful test bed for such methods — that would be an interesting future research topic.

Appendix A: Canonicalization

As we’ve seen with the full model for the 2x2 game to the 8 tile, the number of states and transitions grows quickly, and even games on the 2x2 board become hard to draw in this form.

To help keep the size of the model under control, we can reuse an observation from the previous post about enumerating states: many of the successor states are just rotations or reflections of each other. For example, the states

and

are just mirror images — they are reflections through the vertical axis. If the best action in the first state was to swipe left, the best action in the second state would necessarily be to swipe right. So, from the perspective of deciding which action to take, it suffices to pick one of the states as the canonical state and determine the best action to take from the canonical state. A state’s canonical state is obtained by finding all of its possible rotations and reflections, writing each one as a number in base 12, and picking the state with the smallest number — see the previous post for the details. The important point here is that each canonical state stands in for a class of equivalent states that are related to it by rotation and reflection, so we don’t have to deal with them all individually. By replacing the successor states above with their canonical states, we obtain a much more compact diagram:

It’s somewhat unfortunate that the diagram appears to imply that swiping up (U) from somehow leads to . However, the paradox is resolved if you read the arrows to also include a rotation or reflection as required to find the actual successor’s canonical state.

Equivalent Actions

We can also observe that in the diagram above it does not actually matter which direction the player swipes from the state — the canonical successor states and their transition probabilities are the same for all of the actions. To see why, it helps to look at the ‘intermediate state’ after the player has swiped to move the tiles but before the game has added a new random tile:

The intermediate states are drawn with dashed lines. The key observation is that they are all related by 90° rotations. These rotations don’t matter when we eventually canonicalize the successor states.

More generally, if two or more actions have the same canonical ‘intermediate state’, then those actions must have identical canonical successor states and transition probabilities and therefore are equivalent. In the example above, the canonical intermediate state happens to be the last one, .

We can therefore simplify the diagram for further if we just collapse all of the equivalent actions together:

Of course, states for which all actions are equivalent in this way are relatively rare. Considering another potential start state, , we see that swiping left and right are equivalent, but swiping up is distinct; swiping down is not allowed, because the tiles are already on the bottom.

MDP Model Diagrams for 2x2 Games

Using canonicalization, we can shrink the models enough to just about draw out the full MDPs for some small games. Let’s again start with the smallest non-trivial model: the 2x2 game played just up to the 8 tile (click to enlarge):

Compared to the figure without canonicalization, this is much more compact. We can see that the shortest possible game comprises only a single move: if we are lucky enough to start in the state with two adjacent 4 tiles, which happens in only one game in 150, we just need to merge them together to reach an 8 tile and the win state. On the other hand, we can see that it is still possible to lose: if from state the player swipes up, they reach with probability 0.9, and then if they swipe up again, they reach with probability 0.9, at which point the game is lost.

We can further simplify the diagram if we know the optimal policy. Specifying the policy induces a Markov chain from the MDP model, because every state has a single action, or group of equivalent actions, identified by the policy. The induced chain for the 2x2 game to the 8 tile is:

We can see that the lose state no longer has any edges leading into it, because it is impossible to lose when playing optimally in the 2x2 game to the 8 tile. Each state is also now labelled with its value, which in this case is always 1.000. Because we’ve set the discount factor $\gamma$ to 1, the value of a state is in fact the probability of winning from that state when playing optimally.

They get a bit messier, but can build similar models for the 2x2 game to the 16 tile:

If we look at the optimal policy for the game to the 16 tile, we see that the start states (in blue) all have values less than one, and that there are paths to the lose state, in particular from two states that have tile sum 14:

That is, even if we play this game to the 16 tile optimally, we can still lose, depending on the particular sequence of 2 and 4 tiles that the game deals us. In most cases, we will win, however — the values of the start states are all around 0.96, so we’d expect to win roughly 96 games out of a hundred.

Our prospects in the game to the 32 tile on the 2x2 board, however, are much worse. Here is the full model:

We can see a lot of edges leading to the lose state, which is a bad sign. This is confirmed when we look at the diagram restricted to optimal play:

The average start state value is around 0.08, so we’d expect to win only about 8 games out of a hundred. The main reason becomes clear if we look at the right hand edge of the chain: once we reach a state with tile sum 28, the only way to win is to get a 4 tile in order to reach the state . If we get a 2 tile, which happens 90% of the time, we lose. It’s probably not a very fun game.

Appendix B: Solution Methods

To efficiently solve an MDP model like the ones we’ve constructed here for 2048, we can exploit several important properties of its structure:

The transition model is a directed acyclic graph (DAG). The sum of the tiles must increase, namely by either 2 or 4, with each move, so it is never possible to go back to a state you have already visited.
Moreover, the states can be organized into ‘layers’ by the sums of their tiles, as we did in the first MDP model figure, and all transitions will be from the current layer with sum $s$ to either the next layer, with sum $s+2$, or the one after, with sum $s+4$.
All states in the layer with the largest sum will transition to either a lose or win state, which have a known value, namely 0 or 1, respectively.

Property (3) means that we can loop through all of the states in the last layer, in which all successor values are known, to generate the value function for that last layer. Then, using property (2), we know that the states in the second last layer must transition to either states in the last layer, for which we have just calculated values, or to a win or lose state, which have known values. In this way we can work backward, layer by layer, always knowing the values of states in the next two layers; this allows us to build both the value function and the optimal policy for the current layer. In general, this approach to solving MDPs is called backward induction, which is a particular type of dynamic programming.

In the previous post we worked forward from the start states to enumerate all of the states, layer by layer, using a map-reduce approach to parallelize the work within each layer. For the solve, we can use the output of that enumeration, which is a large list of states, to work backward, again using a map-reduce approach to parallelize the work within each layer. And like last time we can further break up the layers into ‘parts’ by their maximum tile value, with some additional book keeping.

The main solver implementation is still fairly memory-intensive, because it has to keep the value functions for up to four parts in memory at once in order to process a given part in one pass. This can be reduced to one part in memory at a time if we calculate what is usually called $Q^\pi(s,a)$, the value for each possible state-action pair according to policy $\pi$, rather than $V^\pi(s)$, but that solver implementation proved to be much slower, so all of the results presented here use the main $V^\pi$ solver on a machine with lots of RAM.

With both solvers, the layered structure of the model allows us to build and solve a model in just one forward pass and one backward pass, which is a substantial improvement on the usual iterative solution method, and one of the reasons that we’re able to solve these fairly large MDP models with billions of states. The canonicalization methods in Appendix A, which reduce the number of states we need to consider, and the low level efficiency gains from the previous post are also important reasons.

Thanks to Hope Thomas for reviewing drafts of this article.

If you’ve read this far, perhaps you should follow me on twitter, or even apply to work at Overleaf. :)

Footnotes

There is also some nuance in how tiles are merged: if you have four 2 tiles in a row, for example, and you swipe to merge them, the result is two 4 tiles, not a single 8 tile. That is, you can’t merge newly merged tiles in a single swipe. The original code for merging tiles is here, and the simplified but equivalent code I used to merge a line (row or column) of tiles is here with tests here. ↩
The graph diagrams here come from the excellent dot tool in graphviz. ↩
There are several other possible objectives. For example, in the first post in this series, I tried to reach the target 2048 tile in the smallest possible number of moves; and many people I’ve talked to play to reach the largest possible tile, which is also what the game’s points system encourages. These different objectives could also be captured by setting up the model and its rewards appropriately. For example, a simple reward of 1 per move until the player loses would represent the objective of playing as long as possible, which would I think be equivalent to trying to reach the largest possible tile. ↩
Technically, we need one more special state, in addition to the win and lose states, to make this reward system work as described. The equations that we develop for $\pi$ and $V^\pi$ assume that all states have at least one allowed action and successor state, so we can’t just stop the process at the lose and win states. Instead, we can add an absorbing state, end, with a trivial action that just brings the process back into the end state with probability one. Then we can add a trivial action to both the lose and win states to transition to the absorbing end state with probability 1. So long as the end state attracts zero reward, it will not change the outcome. It’s also worth mentioning that there are more general ways of defining an MDP that would provide other ways of working around this technicality, for example by making the rewards depend on the whole transition rather than just the state and by making policies stochastic which, as a side effect, means that we can handle states with no allowed actions, but they require more notation. ↩
In addition to being well founded in economic theory, the discount factor is often required technically in order to ensure that the value function converges. If the process runs forever and continues to accumulate additive rewards, it could accumulate infinite value. The geometric discounting ensures that the infinite sum still converges even if this happens. For the processes with the reward structure we’re considering here, we can safely set the discount factor to 1, because the process is constructed so that there are no loops with nonzero reward, and therefore all rewards are bounded. ↩
There is a caveat for the 3x3 and 4x4 policies: the full optimal policies for every state are too large to ship to the browser (without unduly imposing on GitHub’s generosity in hosting this website). The player therefore only has access to the policy for the states that have a probability of at least $10^{-7}$ of actually occurring when playing according to the optimal policy. This means that, unfortunately, roughly one in hundred readers will choose a random seed that takes the process to a state that is not included in the data available on the client, in which case it will stop with an error. These states are selected by calculating the transient probabilities for the absorbing Markov chain induced by the optimal policy. The mathematics are essentially the same as those in the first post about Markov chains for 2048. ↩
Of course, I’m not claiming here to be great at 2048. The data in my first post suggest otherwise! ↩

The Mathematics of 2048: Counting States by Exhaustive Enumeration

Sun, 10 Dec 2017 00:00:00 +0000

Updates

2017-12-11 This post was discussed on Hacker News.

This post is the third in a series. Next: Optimal Play with Markov Decision Processes.

So far in this series on the mathematics of 2048, we’ve seen that it takes at least 938.8 moves on average to win, and we’ve obtained some rough estimates on the number of possible states using combinatorics.

In this post, we will try to refine those estimates by simply counting every reachable state by brute force enumeration. There are many states, so this will require some computer science as well as mathematics. With efficient processing and storage of states, we’ll see that it is possible to enumerate all reachable states for 2x2 and 3x3 boards.

Enumeration of all states for the full game on a 4x4 board remains an open problem. I ran the code on an OVH HG-120 instance with 32 cores at 3.1GHz and 120GB RAM for one month, during which it enumerated 1.3 trillion states, which is a respectable half a million states per second, but that turns out to be nowhere near enough. At present, the best we we’ll be able to do on the 4x4 board is to enumerate all states for the game played up to the 64 tile.

Overall, the results show that the combinatorial estimates from the last post were substantial overestimates, as suspected. However, the state space for the full game is still very large. If a large nation state or tech company decided that 2048 was their top priority, which may not be the craziest thing that’s happened this year, I think they could probably finish the job.

The (research quality) code behind this article is open source, mainly in ruby and C++.

Counting States

Here a state captures a complete configuration of the board by specifying the value of the tile, if any, in each of the board’s cells. Our overall goal is to count all of the states that can actually occur in the game and no more. In the previous post, the estimates from (very basic) combinatorics counted many states that can’t actually occur in the game. By enumerating states systematically from each of the possible start states, we ensure that the states we count are actually reachable in play.

We also have some freedom in choosing which states are interesting enough to count. Because the game ends when we obtain a 2048 tile, we won’t care about where that tile is or what else is on the board, so we can condense all of the states with a 2048 tile into a special “win” state. Similarly, if we lose, we won’t care exactly how we lost; we can condense all of losing states into a special “lose” state.

Canonicalization and Symmetry

Many other non-interesting states that we’d like to avoid counting arise from the fact that some states are trivially related to each other by rotation or reflection. For example, in the game on a 2x2 board, the states

and

are just mirror images — they are reflections through the vertical axis. If we swiped left in the first state, it would be essentially equivalent to swiping right in the second. We can therefore reduce the number of states we have to worry about by treating these states as equivalent. In general, the number of such equivalent states is the number of elements in the dihedral group for the square, $D_4$, which is 8. For the first state above, these eight states are:

In this diagram, which is called a cycle graph, ⤴ denotes a rotation counterclockwise by 90° and ↔ denotes a reflection about the vertical axis. The three states at the top are obtained by one or more rotations; for example, ⤴² means two rotations of 90°, which add up to a rotation of 180°. The four states at the bottom are obtained by zero or more rotations followed by a reflection.

When the tiles in a state are arranged in a symmetrical pattern, the number of equivalent states may be less than 8, because the symmetry means that some of the states in the above diagram will be identical. For example, the state has only four, because it is symmetric along the diagonal. However, particularly on 3x3 and 4x4 boards, states with symmetric arrangements of tiles are not so common, so overall we can expect to reduce the number of states we need to count by roughly a factor of 8 by choosing only one of these equivalent states as the canonical state.

States as Numbers

Which of the equivalent states should we choose as the canonical state? To answer this question, it will be helpful to think about states as numbers. We’ll also see that this has significant computational benefits in the appendices.

To write a state as a number, we can start in the top left and read the cells by rows; for each cell, we write a $0$ digit if the cell is empty, and the digit $i$ if the cell contains the $2^i$ tile. For example, on a 2x2 board, the state would be written as the number $2310$, because $4 = 2^2$, $8 = 2^3$, $2 = 2^1$, and the last cell is empty.

Since we’re only interested in numbers up to 2048, which is $2^{11}$, we could in principle write any state as a number in base 12 (rather than base 10, as we are accustomed to). However, because computers run on binary numbers, it will be more convenient to think of each state as a number in base 16 — that is, hexadecimal. The state above would therefore be more properly written as 0x2310 for computer scientists or $2310_{16}$ for mathematicians.

To find the canonical state, we simply try all eight possible rotations and reflections, convert the resulting states to numbers, and then pick the smallest one. For our example state, , the candidates and their corresponding numbers are:

State	Number
	`0x2310`
	`0x3201`
	`0x1023`
	`0x2130`
	`0x0312`
	`0x1203`
	`0x0132`
	`0x3021`

The state has the smallest number, 0x0132, so that is the canonical state for . In general, this system for choosing canonical states tends to put tiles with larger values toward the bottom right corner.

Enumeration

Now we’re ready to start enumerating states — it’s all computation from now on. The idea is to generate all possible (canonical) start states, then for each one try each possible move, then for each move generate all possible (canonical) successor states. In code (ruby) form, the basic algorithm for enumerating states looks like this:

def enumerate(board_size, max_exponent)
  # Open all of the possible canonicalized start states.
  opened = find_canonicalized_start_states(board_size)
  closed = Set[]

  while opened.any?
    # Treat opened as a stack, so this is a depth-first search.
    state = opened.pop

    # If we've already processed the state, or if this is
    # a win or lose state, there's nothing more to do for it.
    next if closed.member?(state)
    next if state.win?(max_exponent) || state.lose?

    # Process the state: open all of its possible canonicalized successors.
    [:left, :right, :up, :down].each do |direction|
      state.move(direction).random_successors.each do |successor|
        opened.push(successor.canonicalize)
      end
    end

    closed.add(state)
  end

  closed
end

The code for generating start states is here, and the code for the rest of the State methods, such as random_successors and canonicalize, is here.

If we run this code for the game on the 2x2 board played up to the 32 tile, which we’ve previously established is the highest tile reachable on the 2x2 board, we get 57 states (click to enlarge):

In this diagram, each edge is a possible state-successor pair. For example, from the state , we could swipe left or right, which would result in a 4 tile, and then the game adds a 2 or 4 tile at random, which leads to , , or after canonicalization; or we could swipe up or down, which would leave the two 2 tiles unmerged and lead to or after canonicalization.

Not shown are the special ‘lose’ and ‘win’ states, so in total we have 59 states for the 2x2 game to the 32 tile. This compares favorably to the estimate of 529 states from the (simple) combinatorics arguments in the previous blog post. We’ve saved about one order of magnitude!

When we try to run this ruby code on the 3x3 or 4x4 boards, however, we quickly hit two problems: it’s very slow, and it runs out of memory for the closed set. I’ve included three appendices with the details of how to speed up the calculations and manage the large amounts of data involved, but first let’s see some results.

Results

The numbers of states for the various games we’ve looked at in this series of blog posts are:

Board Size	Maximum Tile	Combinatorics Bound	Actual
2x2	32	529	59
3x3	1024	786,513,819	25,179,014
4x4	64	2,816,814,934,817	40,652,843,435
4x4	2048	44,096,167,159,459,777	$\gg$ 1.3 trillion

Just like the 32 tile is the highest reachable tile on the 2x2 board, the 1024 tile is the highest on the 3x3 board. Exhaustive enumeration of states shows that the 3x3 game contains about 25 million states, which is a factor of 31 lower than the rough ‘Combinatorics Bound’ from the previous post. For the game on the 4x4 board to the 64 tile, which is the largest game on the 4x4 board that I was able to completely enumerate, the factor is even larger, at 69. As expected, the (very basic) combinatorics bounds were quite loose, because they count many states that can’t occur in the game or are trivially related to each other.

For the 3x3 game to the 1024 tile, there are too many states to draw a diagram like the one for the 2x2 game above. However, we can gain some insight into that 25 million figure by counting states in groups by (1) the sum of the tiles on the board and (2) the value of the maximum tile on the board. The sum of the tiles on the board increases by either 2 or 4 with each move ¹, so the game generally progresses from left to right on this graph:

Early in the game, when the sum of the tiles is small, the number of states grows fairly smoothly and linearly with the sum of tiles. However, later in the game when the board fills up, there are sharp drops around where the sum of tiles reaches a larger power of two, for example at around sums 128 and 256. These drops indicate that the 3x3 game is tightly constrained by the small size of the board — there are not many ways to survive past these drops without merging most of the tiles together into a larger one.

It’s also notable that the same structure seems to repeat each time a larger maximum tile is reached (that is, each time the shade of blue in the plot gets darker). The 64, 128, 256 and 512 max tile curves each have a similar slope at the start and a ‘step’ at about 26,000 states per tile sum. In terms of gameplay, this repetition reflects the fact that once you merge most of the tiles together to get the next largest one, the board is mostly empty again, except for the newly merged tile, so the game sort of ‘resets’ at that point.

We might hope that the game on the 4x4 board would also show some of these characteristics, but at least up to tile sum 380, this is apparently not the case. After running the enumeration for one month and counting over 1.3 trillion states, the results to date for the full game of 2048 look like:

We see smooth and uninterrupted growth in the total number of states. Whereas the game on the 3x3 board topped out at about 80 thousand states per tile sum, the game on the 4x4 board shows no sign of slowing down at 27 billion states per tile sum ².

We can still see the rise and fall in the number of states with each maximum tile value, and it becomes clearer if we unstack these counts and plot them on a logarithmic scale:

The top line in black shows the total number of states, summing over all the maximum tile values, which are again shown in shades of blue. Each blue arc shows the growth and later decay in the number of states with a given maximum tile value, as the game progresses. The bending down of the total (black line) shows that the growth in the number of states per tile sum tapers off as the game progresses, but the numbers are already quite large.

Conclusion

We’ve improved our estimates for the number of states in the game of 2048 on the 2x2 and 3x3 boards by one to two orders of magnitude, compared to the previous (basic) combinatorial estimates. The number of states for the game on the 4x4 board remains too large to enumerate in full, but we have at least managed to completely enumerate the states for the 4x4 game to the 64 tile, and we’ve made an attempt at enumerating the states for the full game.

The explicit enumeration of states counts only states that can be reached in actual game play, but even with that restriction there are still some surprising states included in the count. For example, in the last figure above, the arc that shows the number of states with at most a 32 tile does not stop until tile sum 348. There are four states with tile sum 348 and no 64 tile, one of which I chose for the cover image of this post: ³

It contains seven 32 tiles and seven 16 tiles arranged in a nice striped pattern that makes it very difficult to merge any of them. I find it quite surprising that it’s possible to play to such a state without losing, and indeed it seems quite improbable, especially if one is ‘playing well’.

In the next post, we’ll explore what it means to ‘play well’ in a rigorous way by modeling the game of 2048 as a Markov Decision Process and finding an optimal policy for the 2x2 and 3x3 games — that is, we will find a strategy for playing those game that we can show mathematically to be at least as good as any other possible strategy.

Appendix A: Bit Bashing for Efficiency

Profiling the ruby code above revealed that most of the time was being spent on state manipulation — for example, moving tiles, counting available tiles, and reflecting or rotating states to find canonical states. Let’s see how we can speed it up.

The hexadecimal numerical representation for states has a convenient property: for the 4x4 board, we need to store 16 numbers, where each number takes 4 bits, for a total of 64 bits. Now that we all use 64-bit computers, this is an auspicious number: the whole board state can fit into a single 64-bit (8-byte) machine word. For example, the 4x4 state (which is a winning state, because it has a 2048 tile)

would be written 0x022b013020100000 in hexadecimal, or more clearly with the addition of line breaks to make a 4x4 grid:

With the state represented as a 64-bit integer, we can also implement many common manipulations on states very efficiently using bit mask and shift operations, often without loops ⁴. For example, the C++ function to reflect a 4x4 board state horizontally looks like:

uint64_t reflect_horizontally(uint64_t state) {
  uint64_t c1, c2, c3, c4;
  c1 = state & 0xF000F000F000F000ULL;
  c2 = state & 0x0F000F000F000F00ULL;
  c3 = state & 0x00F000F000F000F0ULL;
  c4 = state & 0x000F000F000F000FULL;
  return (c1 >> 12) | (c2 >> 4) | (c3 << 4) | (c4 << 12);
}

While initially quite opaque, all this is doing is shuffling bits around. The role of first bit mask, 0xF000F000F000F000ULL, becomes clearer if we omit the 0x and ULL, which just tell the compiler that this is non-negative 64-bit integer in hexadecimal, and again add line breaks to make a 4x4 grid:

F000
F000
F000
F000

The binary representation of the hexadecimal digit F is 1111, so the effect of this bit mask is to make c1 contain only the values in the first column of the board and zero bits everywhere else. The bit shift c1 >> 12 at the end of the function moves the first column 12 bits, which is to say three 4-bit cells, to the right, which makes it the last column. Similarly, c2 selects the second column, and c2 >> 4 moves it one cell to the right, and so on with the third and fourth columns to reverse the order of the columns.

Some functions take a bit more work to decipher. If you’re in the mood for a puzzle, here’s a function that counts the number of cells available (value zero) in a 4x4 state (you can find my explanation in comments here):

int cells_available(uint64_t state) {
  state |= (state >> 2);
  state |= (state >> 1);
  state &= 0x1111111111111111ULL;
  uint64_t count = ((state * 0x1111111111111111ULL) >> 60);
  if (count == 0 && state != 0) return 0;
  return 16 - count;
}

Profiling (with perf) showed that such tricks made a big difference — compared to the obvious implementations with arrays and loops, these functions require very few CPU instructions and contain few or no branches, which allows the CPU to keep its instruction pipelines full.

Appendix B: Layers and MapReduce for Parallelism

The next challenge is that we can’t keep the whole closed set in memory. To break up the state space into manageably sized pieces, we can use the following property of the game: The sum of the tiles on the board increases by either 2 or 4 with each move. This property holds because merging two tiles does not change the sum of the tiles on the board, and the game then adds either a 2 or a 4 tile. ¹

This property is useful here because it means that we can organize the states into layers according to the sum of their tiles. We can therefore generate the whole state space by working through a single layer at a time, rather than having to deal with the whole state space at once.

To parallelize the work within each layer, we can use the MapReduce concept made famous by Google. The ‘map’ step here is to take one complete layer of states with sum $s$ and break it up into pieces; then, for each piece in parallel, generate all of the successor states, which will have either sum $s + 2$ or $s + 4$. The ‘reduce’ step is to merge all of the pieces for the layer with sum $s + 2$ together into a complete layer, removing any duplicates. The pieces with sum $s + 4$ are retained until the next layer, in which states have sum $s + 2$, is processed, at which point they will be included in the merge. This may be easier to see in an animation (made with d3):

In the map step for each piece, it is feasible to maintain the set of successor states in memory. To make the merge in the reduce step efficient, we want the map step to output a list of states for each piece in order by their state numbers so we can merge the pieces together in linear time. Here Google again comes to our aid: they have released a handy in-memory B-tree implementation that plays well with the C++ standard template library. B-trees are most commonly found in relational database systems, where they are often used to maintain indexes on columns. They keep data in order and provide logarithmic time lookup and also logarithmic time insertion — much better than logarithmic plus linear time insertion into a sorted list — with relatively little memory overhead.

To break the state space up into even smaller pieces, which reduces the amount of work we need to do in each merge step, we can exploit an another property of the game: the maximum tile value on the board must either stay the same or double with each move. This property holds because the maximum tile value never decreases, and when it does increase, it can only increase as the result of merging two tiles — that is, even if you have for example four 16 tiles in a row and you merge them, after one move the result is two 32 tiles, not one 64 tile.

Together with the property above, this means that from a list of states in which all states have tile sum $s$ and maximum tile value $k$, the generated successors will all fall into one of four pieces:

Piece	1	2	3	4
Tile Sum	$s+2$	$s+2$	$s+4$	$s+4$
Max Tile Value	$k$	$2k$	$k$	$2k$

There is a bit more bookkeeping to keep track of which pieces need to be merged together at each step, but it is basically the same idea. Here are the 57 states (excluding the ‘win’ and ‘lose’ states) for the 2x2 game to the 32 tile again, this time with the states grouped by tile sum and maximum tile value, here written $s / k$:

For example, in the leftmost piece, both states have tile sum 4 and maximum tile value 2. The transitions from that piece are to pieces with tile sum either 6 or 8 and maximum tile value 2 or 4. Compared to the original diagram of these states without grouping, it’s also easier to see that you always transition to a state with a tile sum that is either 2 or 4 larger, like in the first post in this series.

Appendix C: Encoding and Compression

When working with billions or trillions of states, and each state takes 8 bytes, even fitting them all on disk is not trivial (or at least not cheap). To reduce the storage space required, and also the amount of input/output required, we can exploit the fact that the states are stored as sorted lists of integers — rather than storing each integer in full, we can store the differences between successive integers. These differences will generally be smaller than the integers themselves, so it is usually possible to store the differences in a smaller number of bytes. Using a variable-width encoding scheme to store the differences ensures that we use only the number of bytes we need. ⁵

For example, the list of states for the 4x4 game to the 2048 tile with tile sum 380 and maximum tile value 128 contains 21,705,361,721 states ². At 8 bytes per state, that would be roughly 161GiB. However, with variable-width encoding, it takes only 35GiB — a compression factor of 4.6 — or about 1.7 bytes per state.

For longer term storage and transport over Internet, I also tried several compression programs on the variable-width encoded data. To evaluate the different programs, I used a smaller list of states with tile sum 260 and maximum tile value 32; it contained 30,954,422 states, which is about 1/700th the number in the 380/128 layer mentioned above, and it weighed in at 64MiB after variable-width encoding. For each of the programs, and for each of their supported compression levels, I measured the elapsed time for compression and the resulting compressed size using this smaller list of states.

Three programs emerged on the resulting Pareto frontier: Facebook’s Zstandard, Google’s Brotli and 7-zip. After scaling size and time up by a factor of 700 to estimate performance on the larger 380/128 layer, the frontier looks like:

Here closer to the origin is better; we see Zstandard performing best for relatively fast and light compression, and 7zip performing best for relatively slow and heavy compression. From the graph, we can see that Zstandard at compression level 11 is fairly close to the origin, and it turned out to minimize the particular linear cost function that I used. The results on the smaller layer suggested that Zstandard level 11 would reduce the 380/128 layer from 35GiB to about 10GiB. In fact, it did much better: it reduced it from 35GiB to 3.8GiB, for another factor of 9.2, or about 1.5 bits per state. I think that’s quite remarkable.

Thanks to Hope Thomas for reviewing drafts of this article.

If you’ve read this far, perhaps you should follow me on twitter, or even apply to work at Overleaf. :)

Footnotes

This property was called called Property 3 in the previous post, and we also used it anonymously in the first post. It is a very useful property. ↩ ↩²
These figures come from the one month build on the full 4x4 game, which used an additional technique for reducing the size of the state space: for each state, it looked one move ahead to see whether the next move was a definite win or a definite loss; if it was, the state was collapsed to a special ‘one move to win’ or ‘one move to lose’ state. This technique was not very effective, however, because states that are close to a loss tend to have few very successors anyway, and because win states only occur much later in the game, and the enumeration process never reached a win state. I therefore haven’t included it in the main body of the article, but the code is here. The state counts mentioned in this article for the partial enumeration of states with the 4x4 board to the 2048 tile are slightly reduced compared to what they would be without this technique, but only slightly. ↩ ↩²
In case you are wondering, I faked that screenshot; I didn’t actually play the game to that state! ↩
The bit bashing techniques used here are based mainly on 2048-ai by Robert Xiao and 2048-c by Kuang-che Wu, with some help from Bit Twiddling Hacks by Sean Eron Anderson. The original motivation for 2048-ai and 2048-c was to allow real time search for 2048 AI bots. ↩
Variable width encodings are most commonly encountered in the UTF-8 character encoding for unicode, which you are using as you read this webpage. They are also sometimes used for storing integer primary key indexes in some databases, which also face the problem of efficiently storing large lists of sorted integers. The implementation used here is libvbyte from Christoph Rupp, based on work by Daniel Lemire. ↩

The Mathematics of 2048: Counting States with Combinatorics

Sun, 17 Sep 2017 01:00:00 +0000

Updates

2017-09-25 There was some lively discussion about this article on Hacker News.

This post is the second in a series. Next: Counting States by Exhaustive Enumeration.

In my last 2048 post, I found that it takes at least 938.8 moves on average to win a game of 2048. The main simplification that enabled that calculation was to ignore the structure of the board — essentially to throw the tiles into a bag instead of placing them on a board. With the ‘bag’ simplification, we were able to model the game as a Markov chain with only 3486 distinct states.

In this post, we’ll make a first cut at counting the number of states without the bag simplification. That is, in this post a state captures the complete configuration of the board by specifying which tile, if any, is in each of the board’s cells. We would therefore expect there to be a lot more states of this kind, now that the positions of the tiles (and cells without tiles) are included, and we will see that this is indeed the case.

To do so, we will use some (simple) techniques from enumerative combinatorics to exclude some states that we can write down but which can’t actually occur in the game, such as the one above. The results will also apply to 2048-like games played on different boards (not just 4x4) and up to different tiles (not just the 2048 tile). We’ll see that such games on smaller boards and/or to smaller tiles have far fewer states than the full 4x4 game to 2048, and that the techniques used here are relatively much more effective at reducing the estimated number of states when the board size is small. As a bonus, we’ll also see that the 4x4 board is the smallest square board on which it is possible to reach the 2048 tile.

The (research quality) code behind this article is open source, in case you would like to see the implementation or code for the plots.

Baseline

The most straightforward way to estimate the number of states in 2048 is to observe that there are 16 cells, and each cell can either be blank or contain a tile with a value that is one of the 11 powers of 2 from 2 to 2048. That gives 12 possibilities for each of the 16 cells, for a total of $12^{16}$, or 184 quadrillion (~$10^{17}$), possible states that we can write in this way. For comparison, some estimates put the number of possible board configurations for the game of chess at around $10^{45}$ states, and the latest estimates for the game of Go are around $10^{170}$ states, so while $10^{17}$ is large, it’s certainly not the largest as games go.

For 2048-like games more generally, let $B$ be the board size, and let $K$ be the exponent of the winning tile with value $2^K$. For convenience, let $C$ denote the number of cells on the board, so $C=B^2$. For the usual 4x4 game to 2048, $B=4$, $C=16$, and $K = 11$, since $2^{11} = 2048$, and our estimate for the number of states is \[(K + 1)^C.\] Now let’s see how we can refine this estimate.

First, since the game ends when we obtain a $2^K$ tile, we don’t particularly care about where that tile is or what else is on the board. We can therefore condense all of the states with a $2^K$ tile into a special “win” state. In the remaining states, each cell can either be blank or hold one of $K - 1$ tiles. This reduces the number of states we have to worry about to \[K^C + 1\] where the $1$ is for the win state.

Second, we can observe that some of those $K^C$ states can never occur in the game. In particular, the rules of the game imply two useful properties:

Property 1: There are always at least two tiles on the board.

Property 2: There is always at least one 2 or 4 tile on the board.

The first property holds because even if you start with two tiles and merge them, there is still one left, and then the game adds a random tile, leaving two tiles. The second property holds because the game always adds a 2 or 4 tile after each move.

We therefore know that in any valid state there must be at least two tiles on the board, and that one of them must be a 2 or 4 tile. To account for this, we can subtract all states with no 2 or 4 tile, of which are $(K-2)^C$, and also the states with just one 2 tile and all other cells empty, of which there are $C$, and the states with only one 4 tile and all other cells empty, of which there are again $C$. This gives an estimate of \[K^C - (K-2)^C - 2C + 1\] states in total. Of course, when $K$ or $C$ is large, this looks pretty much just like $K^C$, which is the dominant term, but this correction is more significant for smaller values.

Let’s use this formula to tabulate the estimated number of states various board sizes and maximum tiles:

Maximum Tile	Board Size
	2x2	3x3	4x4
8	73	19,665	43,046,689
16	233	261,615	4,294,901,729
32	537	1,933,425	152,544,843,873
64	1,033	9,815,535	2,816,814,940,129
128	1,769	38,400,465	33,080,342,678,945
256	2,793	124,140,015	278,653,866,803,169
512	4,153	347,066,865	1,819,787,258,282,209
1024	5,897	865,782,255	9,718,525,023,289,313
2048	8,073	1,970,527,185	44,096,709,674,720,289

We can see immediately that the 2x2 and 3x3 games have many orders of magnitude fewer states than the 4x4 game. We’ve also also managed to reduce our estimate for the number of tiles in the 4x4 game to 2048 to “only” 44 quadrillion, or ~$10^{16}$.

Counting in Layers

To gain some additional insight into these state counts, we can take advantage of another important property, which was also useful in the last post:

Property 3: The sum of the tiles on the board increases by either 2 or 4 with each move.

This holds because merging two tiles does not change the sum of the tiles on the board, and the game then adds either a 2 or a 4 tile.

Property 3 implies that states never repeat in the course of a game. This means that we can organize the states into layers according to the sum of their tiles. If the game is in a state in the layer with sum 10, we know that the next state must be in the layer with either sum 12 or sum 14. It turns out we can also count the number of states in each layer, as follows.

Let $S$ denote the sum of the tiles on the board. We want to count the number of ways that up to $C$ numbers, each of which is a power of 2 between 2 and $2^{K-1}$, can be added together to produce $S$.

Fortunately, this turns out to be a variation on a well-studied problem in combinatorics: counting the compositions of an integer. In general, a composition of an integer $S$ is an ordered collection of integers that sum to $S$; each integer in the collection is called a part. For example, there are four compositions of the integer $3$, namely $1 + 1 + 1$, $1 + 2$, $2 + 1$ and $3$. When there are restrictions on the parts, such as being a power of two and only having a certain number of parts, the term is a restricted composition.

Even more fortunately, Chinn and Niederhausen (2004) ¹ have already studied exactly this kind of restricted composition and derived a recurrence that allows us count the number of compositions in which there are a specific number of parts, and each part is a power of 2. Let $N(s, c)$ denote the number of compositions of a (positive) integer $s$ into exactly $c$ parts where each part is a power of 2. It then holds that \[ N(s, c) = \begin{cases} \sum_{i = 0}^{\lfloor \log_2 s \rfloor} N(s - 2^i, c - 1), & 2 \le c \le s \\\
1, & c = 1 \textrm{ and } s \textrm{ is a power of 2} \\\
0, & \textrm{otherwise} \end{cases} \] because for every composition of $s - 2^i$ into $c - 1$ parts, we can obtain a composition of $s$ with $c$ parts by adding one part with value $2^i$.

We now just need to make a few minor adjustments to the summation bounds: we would like to use powers of 2 starting at 2 and at most $2^{K-1}$, since if we have a $2^K$ tile the game is won. To this end, let $N_m(s, c)$ denote the number of compositions of $s$ into exactly $c$ parts where each part is a power of 2 between $2^m$ and $2^{K-1}$. This is given by \[ N_m(s, c) = \begin{cases} \sum_{i = m}^{K - 1} N(s - 2^i, c - 1), & 2 \le c \le s \\\
1, & c = 1 \textrm{ and } s = 2^i \textrm{ for some } i \in \{ m, \ldots, K-1 \} \\\
0, & \textrm{otherwise} \end{cases} \] following the same logic as above.

Now we have a formula for exactly $c$ parts, but we want a formula for up to $c$ parts. We can follow the same rationale as in the previous section: subtract off the states with no 2 or 4 tile, of which there are $N_3(s, c)$. According to property 1, we need at least 2 parts, so we start summing at $c=2$. This gives \[ \sum_{c = 2}^{C} {C \choose c} \left( N_1(s, c) - N_3(s, c) \right) \] as our estimate for the number of states with sum $s$. Here $C \choose c$ is a binomial coefficient that gives the number ways of choosing $c$ of the possible $C$ cells into which to place the tiles. Let’s plot it out.

In terms of magnitude, we can see that the 2x2 game never has more than 60 states in any layer, the 3x3 game peaks at about 3 million states per layer, and the 4x4 game peaks at about 32 trillion ($10^{13}$) states per layer. The number of states grows rapidly early in the game but then tapers off and eventually decreases as the board fills up. On the decreasing portion of the curve, we see discontinuities: particularly for higher sums, it may happen that there are no tiles that will fit on the board and sum to that value.

The upper limit on the horizontal axis arises because we can have $C$ values, each up to $2^{K-1}$, so the maximum achievable sum is $C 2^{K-1}$, or 16,384 for the 4x4 game to 2048.

Finally, it’s worth noting that if we sum the number of states in each layer over all of the possible layer sums from 4 to $C 2^{K-1}$, and add one for the special win state, we get the same number of states as we estimated in the previous section, which is a helpful sanity check.

Layer Reachability

Another useful consequence of Property 3 is that if two consecutive layers have no states, it’s not possible to reach later layers. This is because the sum can increase by at most 4 per turn; if there are two adjacent layers with no states, then the sum would have to increase by 6 in a single move in order to ‘jump’ to the subsequent layer, which is not possible. Finding the layer sums that contain no states according to the calculation above therefore allows us to tighten up our estimate by excluding states in unreachable layers after the last reachable layer. The largest reachable layer sums (without ever attaining a 2048 tile) are:

Board Size	Largest Reachable Layer Sum
2x2	60
3x3	2,044
4x4	9,212

This table also tells us that the highest tile we can reach on the 2x2 board is the 32 tile, because the 64 tile can’t occur in a layer with sum 60 or less, and similarly highest reachable tile on the 3x3 board is the 1024 tile. This means that the 4x4 board is the smallest square board on which it’s possible to reach the 2048 tile ². For the 4x4 board, the largest layer sum we can reach without reaching a 2048 tile (and therefore winning) is 9,212, but larger sums would be reachable if we did allow a 2048 tile.

Taking into account layer reachability, the new estimates for the number of states are:

Maximum Tile	Method	Board Size
		2x2	3x3	4x4
8	Baseline	73	19,665	43,046,689
8	Layer Reachability	73	19,665	43,046,689
16	Baseline	233	261,615	4,294,901,729
16	Layer Reachability	233	261,615	4,294,901,729
32	Baseline	537	1,933,425	152,544,843,873
32	Layer Reachability	529	1,933,407	152,544,843,841
64	Baseline	1,033	9,815,535	2,816,814,940,129
64	Layer Reachability	905	9,814,437	2,816,814,934,817
128	Baseline	1,769	38,400,465	33,080,342,678,945
128	Layer Reachability	905	38,369,571	33,080,342,314,753
256	Baseline	2,793	124,140,015	278,653,866,803,169
256	Layer Reachability	905	123,560,373	278,653,849,430,401
512	Baseline	4,153	347,066,865	1,819,787,258,282,209
512	Layer Reachability	905	339,166,485	1,819,786,604,950,209
1024	Baseline	5,897	865,782,255	9,718,525,023,289,313
1024	Layer Reachability	905	786,513,819	9,718,504,608,259,073
2048	Baseline	8,073	1,970,527,185	44,096,709,674,720,289
2048	Layer Reachability	905	1,400,665,575	44,096,167,159,459,777

This has a large effect on the 2x2 board, reducing the number of states from 8,073 to 905 for the game up to 2048, and it’s notable that the figure for the number of reachable states does not increase from 905 for maximum tiles over 32, because it’s not possible to reach tiles larger than 32 on a 2x2 board. It also has some effect on the 3x3 board, but on the 4x4 board, there is relatively little effect — we remove “only” about 500 billion states from the total for the game to 2048.

In graphical form, these data look like:

Conclusion

We’ve obtained some rough estimates for the number of states in the game of 2048 and similar games on smaller boards and to lesser tiles. Our best estimate so far for the number of states in the 4x4 game to 2048 is roughly 44 quadrillion (~$10^{16}$).

It is likely that this and the other estimates are substantial overestimates, because there are many reasons that states might be counted here but still not be reachable in the game. For example, a state like the one in the cover image for this blog post:

satisfies all of the restrictions we’ve considered here, but it is still not possible to reach it, because we must have swiped in some direction before getting to this state, and that would have moved two of the 2 tiles to the edge of the board. It may be possible to adapt the counting arguments above to take this (and likely other restrictions) into account, but I have not figured out how!

In the next post, we’ll see that the number of actually reachable states is much lower by actually enumerating them. There will still be a lot of them for the 3x3 and 4x4 boards, so we will need some computer science as well as mathematics.

If you’ve read this far, perhaps you should follow me on twitter, or even apply to work at Overleaf. :)

Footnotes

Chinn, P. and Niederhausen, H., 2004. Compositions into powers of 2. Congressus Numerantium, 168, p.215. (preprint) ↩
We could also have shown this using the Markov chain analysis from my previous post by removing all of the states with more than nine tiles, and seeing whether it was still possible to reach the 2048 tile. If we do this, we find that it is not. Interestingly, if we allow the Markov chain to continue to larger maximum tile values, the same analysis shows that it is possible to reach the 131072 (that is, $2^{17}$) tile on a 4x4 board, if the structure of the board is ignored. Whether this is true when there are structural constraints is still open, but I suspect it is not. It is, however, possible to play to at least the 8192 tile, as shown by this AI bot, and there is a ‘proof of concept’ video for the 131072 tile. ↩

The Mathematics of 2048: Minimum Moves to Win with Markov Chains

Sat, 05 Aug 2017 09:00:00 +0000

Updates

2017-09-25 There was some lively discussion about this series on Hacker News.

This post is the first in a series. Next: Counting States with Combinatorics.

As part of a recent revamp, 2048’s “You win!” screen started reporting the number of moves it took to win, which made me wonder: how many moves should it take to win?

In this post, we’ll answer that question by modeling the game of 2048 as a Markov chain and analyzing it to show that, no matter how well the player plays, the number of moves required to win the game is at least 938.8 on average. This gives us a benchmark — if you can win in around this number of moves, you’re doing pretty well.

The number of moves needed to win depends on random chance, because the game adds 2 and 4 tiles at random. The analysis will also show that the distribution for the minimum number of moves to win has a standard deviation of 8.3 moves, and that its overall shape is well-approximated by a mixture of binomial distributions.

To obtain these results, we’ll use a simplified version of 2048: instead of placing the tiles on a board, we’ll… throw them into a bag. That is, we’ll ignore the geometric constraints imposed by the board on which tiles can be merged together. This simplification makes our job much easier, because the player no longer has to make any decisions ¹, and because we don’t have to keep track of where the tiles are on the board.

The price we pay for relaxing the geometric constraints is that we can only compute a lower bound on the expected number of moves to win; it might be that the geometric constraints make it impossible to attain that bound. However, by playing many games of 2048 (for science!), I’ll also show that we can often get close to this lower bound in practice.

If you’re not familiar with 2048 or with Markov chains, that’s OK — I’ll introduce the necessary ideas as we go. The (research quality) code behind this article is open source, in case you’d like to see the code to generate the chain and the plots.

2048 as a Markov Chain

To represent our simplified ‘2048 in a bag’ game as a Markov chain, we need to define the states and the transition probabilities of the chain. Each state is like a snapshot of the game at a moment in time, and the transition probabilities specify, for each state, which state is likely to come next.

Here we will define each state to encode which tiles are currently in the bag. We don’t care about the order of the tiles, so we can think of it as a multiset of tiles. Initially, there are no tiles, so our initial state is simply the empty set. In diagram form, which we’ll add to below, this initial state looks like:

Setting up the Board

A sample of a dozen new game boards.

When we start a new game of 2048, the game places two random tiles on the board (see examples above). To represent this in the Markov chain, we need to work out the transition probabilities from the initial state to each of the possible successor states.

Fortunately, we can look at the game’s source code to find out how the game does this. Whenever the game places a random tile on the board, it always follows the same process: pick an available cell uniformly at random, then add a new tile either with value 2, with probability 0.9, or value 4, with probability 0.1.

For 2048 in a bag, we don’t care about finding an available cell, because we haven’t put any capacity constraint on the bag; we just care about adding either a 2 or 4 tile with the given probabilities. This leads to three possible successor states for the initial state:

$\{2, 2\}$ when both of the new tiles are 2s. This happens with probability $0.9 \times 0.9 = 0.81$.
$\{4, 4\}$ when both of the new tiles are 4s. This happens with probability $0.1 \times 0.1 = 0.01$ — that is, you are pretty lucky if you start with two 4s.
$\{2, 4\}$ when the new tiles are 2 and then 4, which happens with probability $0.9 \times 0.1 = 0.09$, or 4 and then 2, which happens with probability $0.1 \times 0.9 = 0.09$. We don’t care about order, so both cases lead to the same state, with total probability $0.09 + 0.09 = 0.18$.

We can add these successor states and their transition probabilities to the Markov chain diagram as follows, where the transition probabilities are written on the edge labels, and the new nodes and edges are shown in blue:

Playing the Game

With the first pair of tiles now placed, we’re ready to start playing the game. In the real game, this means that the player gets to swipe left, right, up or down to try to bring pairs of like tiles together. In the bag game, however, there is nothing to stop us from merging all pairs of like tiles, so that’s what we’ll do.

In particular, the rule for merging tiles in the bag game is: find all of the pairs of tiles in the bag that have the same value and remove them; then replace each pair of tiles with a single tile with twice the value ².

Once pairs of like tiles have been merged, the game adds a single new tile at random using the same process as above — that is, a 2 tile with probability 0.9, or a 4 tile with probability 0.1 — to arrive at the successor state.

For example, to find the possible successors of the state $\{2, 2\}$, we first merge the two 2 tiles into a single 4 tile, and then the game will add either a 2 tile or a 4 tile. The possible successors are therefore $\{2, 4\}$ and $\{4, 4\}$, which, as it happens, we have already encountered. The diagram including these two transitions from $\{2, 2\}$, which have probability 0.9 and 0.1 respectively, is then:

If we follow the same process for the successors of $\{2, 4\}$, we see that no merging is possible yet, because there is no pair of like tiles, and the successor state will either be $\{2, 2, 4\}$ or $\{2, 4, 4\}$, depending on whether the new tile is a 2 or 4. The updated diagram is then:

Layers and ‘Skipping’

We can continue adding transitions in this way. However, as we add more states and transitions, the diagrams can become quite complicated ³. We can make the diagrams a bit more orderly by using the following observation: the sum of the tiles in the bag increases by either 2 or 4 with each transition. This is because merging pairs of like tiles does not change the sum of the tiles in the bag (or on the board — this property also holds in the real game), and the game always adds either a 2 tile or a 4 tile.

If we group states together into ‘layers’ by their sum, the first few layers look like this:

For later layers, I’ve also omitted the labels for transitions with probability 0.9 (solid lines, unless otherwise labelled) and 0.1 (dashed lines), to reduce clutter.

Grouping the states into layers by sum makes another pattern clear: each transition (other than those from the initial state) is either to the next layer, with probability 0.9, or the layer after, with probability 0.1. (This is particularly clear if you look at the layers with sums 8, 10 and 12 in the diagram above.) That is, most of the time the game gives us a 2, and we’ll transition to the next layer, but sometimes we get lucky, and the game gives us a 4, which means we get to skip a layer, getting us slightly closer to our goal of reaching the 2048 tile.

The End Game

We could continue this process forever, but since we are only interested in reaching the 2048 tile, we’ll stop it at that point by making any state with a 2048 tile an absorbing state. An absorbing state has a single transition, which is to itself with probability 1 — that is, once you reach an absorbing state, you can never leave.

In this case all of the absorbing states are ‘win states’ — you have a 2048 tile and have therefore won the game. There is no way to ‘lose’ the bag game, because unlike in the real game, we cannot get into a situation where the board (or bag) is full.

The first state that contains a 2048 tile is in the layer with sum 2066. It’s notable that you can’t get a 2048 tile on its own on the board — it takes a few moves to merge the 1024 tiles, the 512 tiles, and so on, during which you accumulate more 2 and 4 tiles. This is why the sum of the tiles for the first state with a 2048 tile is higher than 2048.

Here’s what the graph looks like around this first winning state (see more), with the absorbing states colored red:

If we continue to add transitions until there are no non-absorbing states left, we eventually end up with 3487 states, of which 26 are absorbing; this completes the definition of the Markov chain. The diagram for the full chain is quite large, but if your device can handle a 5MB SVG file, here is a diagram of the full chain (you may need to scroll down a bit to see the start). When very zoomed out, it looks like this:

Sampling from the Chain

Now that we have put in the effort to model 2048 (in a bag) as a Markov chain, the simplest way to find out how many moves it takes before we are absorbed is to run simulations. In each simulation, we generate a single trajectory through the chain by starting at the initial state, then choosing a successor state at random in proportion to the transition probabilities, then repeating from that state. After one million simulation runs, the following distribution emerges for the number of moves to win:

The mean, which is marked by the vertical blue line, comes out at 938.8 moves, excluding the first transition from the initial state, with a standard deviation of 8.3 moves. So, that’s our answer for the minimum expected number of moves to win the game!

The theory of Markov chains also lets us calculate some of these properties directly using clever mathematics. In Appendix A, I’ll show how to calculate the mean and standard deviation for the number of moves without relying on simulation. Then, in Appendix B, I’ll use some of properties of the chain to offer at least a partial explanation of the shape of the distribution.

Putting Theory to the Test

Finally, to test these results in real life, I played a lot of 2048 (for science!) and, for the 28 games I won, recorded the number of moves it took and also the sum of the tiles on the board when I reached the 2048 tile ⁴:

Transcribing these numbers into a spreadsheet and plotting them leads to the following scatter plot:

I’ve marked the minimum expected number of moves, plus or minus one standard deviation, in blue, and the tile sum 2066, which we found to be the lowest sum of tiles for which it was possible to have a 2048 tile, in red.

The sum of the tiles on the board is important, because when it’s large, that typically means that I made a mistake that left a large tile stranded somewhere I could not merge it with any other tile. It then took many more moves to build up that tile again in a place where it could be merged (or to set up the board to try to get into the right place to merge) with another large tile.

If I were very good at playing 2048, we’d predict my results would cluster in the bottom left corner of the graph, and that most of them would lie between the dashed blue lines. In fact, we see that, while I sometimes get close to this ideal, I am not very consistent — there are plenty of points in the top right, with lots of extra moves and extra tiles.

This plot also highlights the fact that this analysis gives us only the minimum expected number of moves. There were a few games where I got lucky and won in less than 938.8 moves, including one win with 927 moves and a tile sum of 2076. (It is the second from the left in the bottom row of the montage above.) This is essentially because I got a lot of 4 tiles in that game, just by chance, and also because I didn’t make any major blunders that required extra moves.

In principle, there is non-zero probability that we could win the game in only 519 moves. We can find this by walking through the chain, always taking the transition for the 4, and counting the number of transitions required to reach a 2048 tile. However, the probability of this occurring is $0.1^{521}$, or $10^{-521}$; there are only about $10^{80}$ atoms in the observable universe, so you shouldn’t hold your breath waiting for such a game to happen to you. Similarly, if we are very unlucky and always get 2 tiles, we should still be able to win in only 1032 moves. Such a game is much more likely, with a probability of $0.9^{1034}$, which is about $10^{-48}$, but you probably shouldn’t hold your breath waiting for that game either. The average of 938.8 moves is much closer to 1032 than 519, because 2 tiles are much more likely than 4 tiles.

Conclusion

In this post we have seen how to construct a Markov chain that models how a game of 2048 evolves if it is always possible to merge like tiles. By doing so, we’ve been able to apply techniques from the theory of absorbing Markov chains to calculate interesting properties of the game, and in particular that it takes at least 938.8 moves to win, on average.

The main simplification that enabled this approach was to ignore the structure of the board, effectively assuming that we threw tiles into a bag, rather than placing them onto the board. In my next post, I plan to look at what happens when we do consider the structure of the board. We’ll see that the number of states we need to consider becomes many orders of magnitude larger (though perhaps not as large as one might think), and also that we will need to leave the world of Markov chains and enter the world of Markov Decision Processes, which allow us to bring the player back into the equation, and in principle may allow us to ‘solve’ the game completely — to find a provably optimal way of playing.

Appendix A: Analysis of the Markov Chain

Once that we’ve defined our Markov chain, we can bring some powerful mathematical machinery to bear to calculate its properties without simulation. Many of these calculations are possible only because our Markov chain is a special type of Markov chain called an absorbing Markov chain.

The criteria for being an absorbing Markov chain are that:

There must be at least one absorbing state. As we’ve seen above, there are 26 absorbing states, one for each winning state with a 2048 tile.
For any state, it is possible to reach an absorbing state in a finite number of transitions. One way to see that this holds for our chain is to observe that there are no loops other than for the absorbing states — except for the absorbing states, the chain is a directed acyclic graph.

The Transition Matrix

Now that we have established that we have an absorbing Markov chain, the next step is to write out its transition matrix in canonical form. A transition matrix is a matrix that organizes the transition probabilities, which we defined for our chain above, such that the $(i, j)$ entry is the probability of transitioning from state $i$ to state $j$.

For the transition matrix, $\mathbf{P}$, of an absorbing chain with $r$ absorbing states and $t$ transient (which means non-absorbing) states, to be in canonical form, it must be possible to break it up into four smaller matrices, $\mathbf{Q}$, $\mathbf{R}$, $\mathbf{0}$ and $\mathbf{I}_r$, such that: \[ \mathbf{P} = \left( \begin{array}{cc} \mathbf{Q} & \mathbf{R} \\\
\mathbf{0} & \mathbf{I}_r \end{array} \right) \] where $\mathbf{Q}$ is a $t \times t$ matrix that describes the probability of transitioning from one transient state to another transient state, $\mathbf{R}$ is a $t \times r$ matrix that describes the probability of transitioning from a transient state to an absorbing state, $\mathbf{0}$ denotes an $r \times t$ matrix of zeros, and $\mathbf{I}_r$ is the transition matrix for the absorbing states, which is an $r \times r$ identity matrix.

To get a transition matrix in canonical form for our chain, we need to decide on an ordering of the states. It suffices to order states (1) by whether they are absorbing, with absorbing states last, then (2) by the sum of their tiles, in ascending order, and finally (3) in lexical order, to break ties. If we do this, we obtain the following matrix:

It’s quite large, namely $3487 \times 3487$, so when zoomed out it just looks pretty much diagonal, but if we zoom in on the lower right hand corner, we can see that it does have some structure, and in particular it has the canonical form that we’re after:

The Fundamental Matrix

With the transition matrix in canonical form, the next step is to use it to find what is called the fundamental matrix for the chain, which will let us calculate the expected number of transitions before absorption, which is (finally!) the answer to our original question.

The fundamental matrix, $\mathbf{N}$, is defined in terms of $\mathbf{Q}$ by the identity \[ \mathbf{N} = \sum_{k=0}^{\infty} \mathbf{Q}^k \] where $\mathbf{Q}^k$ denotes the $k$th matrix power of $\mathbf{Q}$.

The $(i, j)$ entry of $\mathbf{N}$ has a particular interpretation: it is the expected number of times that we would enter state $j$ if we followed the chain starting from state $i$. To see this, we can we observe that, just as the $(i, j)$ entry of $\mathbf{Q}$ is the probability of transitioning from state $i$ to state $j$ in a single transition, the $(i, j)$ entry of $\mathbf{Q}^k$ is the probability of entering state $j$ exactly $k$ transitions after entering state $i$. If, for a given pair of states $i$ and $j$, we add up said probabilities for all $k \geq 0$, the summation includes every time at which we could possibly enter state $j$ after state $i$, weighted by the corresponding probability, which is what gives us the desired expectation.

Fortunately, the fundamental matrix can also be calculated directly, without the awkward infinite summation, as the inverse of the matrix $\mathbf{I}_t - \mathbf{Q}$, where $\mathbf{I}_t$ is the $t \times t$ identity matrix; that is, $\mathbf{N} = (\mathbf{I}_t - \mathbf{Q})^{-1}$. (The proof of this identity is left as an exercise for the reader!)

Expected Moves to Win

Once we have the fundamental matrix, we can find the expected number of transitions from any state $i$ to an absorbing state by summing up all of the entries in row $i$ — in other words, the number of transitions before we reach an absorbing state is the total number of transitions that we spend in all of the transient states along the way.

We can obtain these row sums for all states at once by calculating the matrix-vector product $\mathbf{N} \mathbf{1}$, where $\mathbf{1}$ denotes a column vector of $t$ ones. Since $\mathbf{N} = (\mathbf{I}_t - \mathbf{Q})^{-1}$, we can do this efficiently by solving the linear system of equations \[ (\mathbf{I}_t - \mathbf{Q})\mathbf{t} = \mathbf{1} \] for $\mathbf{t}$. The entry in $\mathbf{t}$ that corresponds to the initial state (the empty set, $\{\}$) is the number of transitions. In this case, the number that comes out is 939.8. To finish up, we just need to subtract $1$, because the transition from the initial state doesn’t count as a move. This gives our final answer as 938.8 moves.

We can also obtain the variance for the minimum number of moves as $ 2(\mathbf{N} - \mathbf{I}_t) \mathbf{t} - \mathbf{t} \circ \mathbf{t} $, where $\circ$ denotes the Hadamard (elementwise) product. For the initial state, the variance comes out as 69.5, which gives a standard deviation of 8.3 moves.

Appendix B: The Shape of the Distribution

Perhaps remarkably, we were able to calculate both the mean and the variance of the moves-to-win distribution using the fundamental matrix from the Markov chain. It would however be nice to have some insight into why the distribution is the shape that it is. The approach I’ll suggest here is only approximate, but it does match the empirical results from the simulation of the chain quite closely, and it provides some useful insights.

We’ll begin by revisiting an observation that we made above: the sum of the tiles on the board increases by either 2 or 4 with each transition (other than the first transition from the initial state). If we were interested in hitting a specific sum for the tiles on the board, rather than hitting a 2048 tile, then it’s relatively straightforward to calculate the required number of transitions using the binomial distribution, as we’ll see below.

So, the next question is, which sum should we aim to hit? From the Markov chain analysis above, we determined that there are 26 absorbing (winning) states, and we’ve also seen that they are in different ‘sum layers’, so there isn’t a single target sum — there are several target sums. What we need to know is the probability of being absorbed in each state, which is called an absorbing probability. We can then add up the absorbing probabilities for each of the absorbing states in a particular sum layer to find a probability of winning with a given target sum.

Absorbing Probabilities

Fortunately, the absorbing probabilities can also be found from the fundamental matrix. In particular, we can obtain them by solving the linear equations \[ (\mathbf{I}_t - \mathbf{Q}) \mathbf{B} = \mathbf{R} \] for the $t \times r$ matrix $\mathbf{B}$, whose $(i, j)$ entry is the probability of being absorbed in state $j$ when starting from state $i$. As before, we are interested in the absorbing probabilities when we start from the initial state. Plotting out the absorbing probabilities, there are 15 absorbing states for which the probabilities are large enough to plot (at least $10^{-3}$):

In particular, most games end in either the $\{2,2,8,8,2048\}$ state, which has sum 2068, or the $\{2,4,16,2048\}$ state, which has sum 2070. Summing up all of the absorbing states by layer sum gives the complete layer sum probabilities:

Binomial Probabilities

Now that we have some sums to aim for, and we know how often we are aiming for each one, the next question is, how many moves does it take to hit a particular sum? As noted above, we can think of this in terms of the Binomial distribution, which lets us calculate the probability of a given number of “successes” out of a given number of “trials”.

In this case, we’ll consider a “trial” to be a move, and a “success” to be a move in which the game gives us a 4 tile; as we’ve seen above, this happens with probability 0.1. A “failure” here is a move in which the game gives us a 2 tile, which happens with probability 0.9.

With this interpretation of successes, in order to hit a given sum $S$ in $M$ moves, we need $\frac{S}{2} - M$ successes out of $M$ moves. This is because each move counts at least $2$ toward the sum, which contributes a total of $2M$, and each success counts an additional $2$ toward the sum, for a total contribution of $2 \left(\frac{S}{2} - M\right) = S - 2M$; adding these contributions together leaves the desired sum, $S$.

The joint probability of obtaining a sum $S$ in a number of moves $M$ is then Binomial, and in particular \[ \mathrm{Pr}(M=m, S=s) = B\left(\frac{s}{2} - m; m, 0.1\right) \] where $B(k; n, p)$ is the probability mass function for the binomial distribution, which gives the probability of exactly $k$ successes in $n$ trials, where the probability of success is $p$, namely \[ B(k; n, p) = {n\choose k}p^k(1-p)^{n-k} \] where $n \choose k$ denotes a binomial coefficient.

Now if we know the target sum $S$ that we are aiming for, we can compute the conditional distribution of the number of moves given that sum, which is what we are interested in, from the joint distribution. That is, we can find $\mathrm{Pr}(M | S)$ as \[ \mathrm{Pr}(M | S) = \frac{\mathrm{Pr}(M, S)}{\mathrm{Pr}(S)}. \] where $\mathrm{Pr}(S)$ is obtained from the joint distribution by summing out $M$ for each possible sum $S$. It’s worth noting that $\mathrm{P}(S)$ is less than one for each possible sum, because there’s always a chance that the game ‘skips’ the sum by placing a 4 tile.

With these conditional distributions for each target sum in hand, we can then add them up, weighted by the total absorbing probability for the target sum, to obtain the overall distribution. This gives a fairly good match to the distribution from the simulation:

Here the simulated distribution is shown with the grey bars, and the colored areas show each of the conditional distributions, which are stacked. Each conditional distribution is scaled according to the total absorbing probability for its sum, and also shifted a few moves, with larger sums requiring more moves on average.

One interpretation of this result is that, if playing optimally, the number of moves to win is essentially determined by how quickly the player can get to a sum large enough to have a 2048 tile, which is in turn governed by the number of 4 tiles, which follows a binomial distribution.

Thanks to Hope Thomas and Nate Stemen for reviewing drafts of this article.

If you’ve read this far, perhaps you should follow me on twitter, or even apply to work at Overleaf. :)

Footnotes

If we do allow the player to make decisions, we have a Markov Decision Process, rather than a Markov chain. That will be the subject of a later blog post. ↩
Merging pairs of like tiles in this way captures an important nuance of the merging logic in the real game: if you have, for example, four 2 tiles in a row, and you swipe to merge them, the result is two 4 tiles, not a single 8 tile. That is, you can’t merge newly merged tiles on a single swipe. ↩
The diagrams here come from the excellent dot tool in graphviz. If we don’t give dot a hint by grouping the states together into layers by sum, laying out the full graph can take quite a while. ↩
The appearance of the game’s “You win!” screen changed several times over the months in which I collected this data. For the record, playing 2048 was not the only thing I did during these months. ↩

Piece	1	2	3	4
Tile Sum	\(s+2\)	\(s+2\)	\(s+4\)	\(s+4\)
Max Tile Value	\(k\)	\(2k\)	\(k\)	\(2k\)

jdlm.info

Bootstrap Confidence Intervals in SQL for PostgreSQL and BigQuery

Background: Confidence Intervals and the Bootstrap

The Bootstrap in SQL

The Poisson Bootstrap in SQL

Benchmark Results

Conclusions

Footnotes

Testing with Node and Docker Compose, Part 3: End-to-End

Separate Test and Development Environments

Project Names

Variable Substitutions and Merging Compose Files

External Networks, Links and Volumes

The bin Scripts

The End-To-End Test

Revisiting a Frontend Integration Test

Conclusions

Footnotes

Testing with Node and Docker Compose, Part 2: On the Frontend

The TODO List Manager Frontend

Packaging Backend and Frontend

The Tests

Integration Tests

Model Tests

Component Tests

Running the Tests

Conclusion

Footnotes

Testing with Node and Docker Compose, Part 1: On the Backend

The TODO List Manager Backend

Dockerfile and Compose for Development

Development Helper Scripts

The Tests

Model Tests

Integration tests

Running the Tests

Conclusion

Appendix: Views on Testing

Lessons from Building Node Apps in Docker (2019)

Getting Started

The Bootstrapping Container and Service

Initializing an npm package

Installing Dependencies

The node_modules Volume Trick

Package Installation

Running the App

Docker for Dev and Prod

Setting up nodemon in Development

Conclusion

Appendix: Dealing with UID mismatches on Linux

Footnotes

From Zero to Driverless Race Car with Deep Learning

Introduction

Collecting Training Data

Transfer Learning

Convolutional Neural Networks

Completing the Network

Time to Train

Run the Model with Lowest Validation Loss!

Try Lots of Things…

With Lower Latency

The Final Result

Does it Generalize?

Conclusions

Footnotes

The Mathematics of 2048: Optimal Play with Markov Decision Processes

Markov Decision Processes for 2048

States, Actions and Transition Probabilities

Rewards, Values and Policies

Optimal Play on the 2x2 Board

Optimal Play on the 3x3 Board

Optimal Play on the 4x4 Board

Conclusion

Appendix A: Canonicalization

Equivalent Actions

MDP Model Diagrams for 2x2 Games

Appendix B: Solution Methods

Footnotes

The Mathematics of 2048: Counting States by Exhaustive Enumeration

Counting States

The `bin` Scripts

The `node_modules` Volume Trick

Setting up `nodemon` in Development