Martin Hicks - Journal

I watched DynamoDB change under my conformance suite

Martin Hicks — Tue, 09 Jun 2026 00:00:00 GMT

My DynamoDB conformance suite went red this week, and the test that failed was real AWS.

That needs explaining. The suite runs a few hundred tests against real DynamoDB first, records exactly what the real thing does, then runs the same tests against the emulators people reach for locally - DynamoDB Local, LocalStack, Dynalite, my own engine - and scores how closely each one matches. Real DynamoDB is the reference. It sits at 100% by definition, because it's the thing everything else is measured against. So when the reference itself fails a test, something's up: either my test is wrong, or the real thing has moved underneath it.

It was the second one. Real DynamoDB had changed how it words a chunk of its validation errors. The empty-table-name case used to come back as Value '' at 'tableName' failed to satisfy constraint...; now it's Value at 'TableName'... - the echoed value gone, the field name in PascalCase. A pile of other bespoke messages picked up a 1 validation error detected: envelope they didn't have before. And PutItem with a { NULL: false } attribute, which used to be rejected outright, is now accepted and quietly stored as { NULL: true }.

None of that is written down anywhere. DynamoDB's exact error strings aren't part of any published contract. The only way to know what they are is to ask the service and record what it says, which is exactly what the suite does - so it noticed within a week.

Here's the part that caught me out. I'd assumed a change like this lands everywhere at once, so I fired the same inputs at four regions to be sure. It doesn't. eu-west-2 and eu-central-1 return the new wording. us-east-1 and ap-southeast-2 still return the old. Two regions on one side of the line, two on the other.

Most of that split is cosmetic - the wording moved, not the behaviour. The exception is { NULL: false }: two regions now accept a write the other two still reject. That one isn't the error text reading differently, it's the service taking different input depending on where you call it, and it's what makes "which region" change the answer rather than just the message.

So which is it - has AWS changed DynamoDB, or do these regions just behave differently? I think it's the first, and that the regional split is the change still in motion rather than a permanent fact of geography. Two things point that way. The two regions that moved moved in exactly the same way - same dropped value, same envelope, same { NULL: false } behaviour - which reads like one upstream change propagating outward, not two regions independently drifting apart. And AWS has no reason to deliberately fork European validation from everyone else's. The simplest story that fits the evidence is a staged rollout that's reached Europe and not yet reached Virginia or Sydney.

I can't prove that from the outside. I won't know the laggards have caught up until they do, and a regional fault that gets quietly reverted would look much the same from here. But "a change rolling out region by region" fits better than anything else, so that's how I'm reading it. Which makes the durable point not "DynamoDB differs by region" - if I'm right, that sorts itself out - but "DynamoDB's validation moves, it's undocumented, and the only way to know where it's got to is to ask each region directly."

That points at something I'd got wrong in the suite. I'd pinned the exact prose - the full error string, word for word - and the exact prose was never part of any contract AWS owes me. The upstream Smithy model says which constraints exist: this field is required, that one has a minimum length. It says nothing about how the resulting error reads. I was asserting on the one part AWS is free to change without telling anyone, and then acting surprised when it did.

So the direction I'm taking the suite is to assert the contract and stop pinning the wording. Check that the right exception type comes back, against the right field, for the right violated constraint - the things the model actually promises - and let the surrounding prose vary. Pin less, and the next reword stops being a CI failure and goes back to being the non-event it always should have been.

There's a temptation, having seen this, to build a standing "regional drift" monitor: run the strict checks against a row of regions on a schedule, light up whenever they disagree. I've decided not to, at least not yet. If this really is a rollout, the divergence trends to zero the moment it finishes everywhere, and I'd be left maintaining a view that watches nothing most of the time. So I took a one-off snapshot instead - that's how I know the spread is two and two - and I'll build the live view only if drift turns out to be a recurring pattern rather than a single event I've already characterised. No sense turning one honest number into a dozen confusing ones, or standing up infrastructure for something I've already had a good look at.

The smaller, sharper upshot is about the number the suite reports. It's always really been "conformance to DynamoDB in eu-west-2", not "conformance to DynamoDB". I'd quietly treated those as the same thing. They aren't, and the fix is to make the suite say which region it's measuring rather than imply a single global DynamoDB that doesn't exist.

If you've ever written a test that asserts on a DynamoDB error message, or leaned on one in your own code, this is worth sitting with. The string you pinned might be true in your region and not the next one over, and it might not stay true in yours for much longer. Mine changed under me.

The results are at paritysuite.org, and the suite's open source if you want to point it at your own region and see what comes back.

Build the squad. Own the platform. The org chart will catch up.

Martin Hicks — Wed, 03 Jun 2026 00:00:00 GMT

My wife Helen, who works in HR, sent me an episode of HR Disrupted last week - Lucy Adams interviewing Andy Doyle from Kantar on putting AI agents into production HR workflows.

Andy describes some genuinely impressive work. Kantar had around 4,000 HR policies spread across 63 markets. Twenty-three versions of a single policy were scattered across the intranet, many written by different people at different times, some long out of date. Reviewing and standardising all that by hand would have been the work of years. So they built a set of small, single-purpose agents instead, one to rewrite each policy into a standard format, another to flag where it broke local employment law. The whole lot went into one common framework in a week.

It's a real piece of engineering, which is what makes some of the rhetoric around it a shame: you don't train out hallucination, you constrain it with evals and guardrails. But the operational story underneath holds, and the lesson is in what they did, not how they describe it.

Listening to the podcast got me thinking less about Kantar and more about the next move for everyone who hears a story like that and wants to do something similar.

Why now?

The conversation about AI inside large organisations has shifted in the last six months, from a model problem to a deployment one. AWS's Ishit Vachhrajani describes it as moving from "generate and hope" to systems that plan, decide, and act with human guidance. You can still argue about whether the models are ready, and plenty of people do. But the teams already shipping agents into production - Andy's at Kantar among them - have stopped waiting for that argument to settle. The hard parts that remain, reliability and hallucination, get handled operationally now - not trained away. Those are deployment problems, not model problems. The bottleneck is the org chart.

Where in your organisation does this capability live? Who owns it, who governs it, who carries the risk when something goes wrong? These are organisational design and architectural decisions, and the next ten years of AI in your business will be shaped more by how you answer them than by which models you pick.

And the honest version of where this is heading is bigger than a governance question. The functional org chart - HR over here, IT over there, finance in the corner - has been the default shape of large organisations for the better part of a century. AI may prove to be one of the forces that starts to pull it apart. Get this right and you're not just deploying agents well, you're building the first working piece of an organisation that composes around the work itself rather than the functions that have always owned it. The first step towards that is much smaller.

Faced with the immediate question - where does this capability live - most leaders reach for one of two answers. Both look reasonable. Both have serious problems.

Two easy answers that don't quite work

The first is to outsource the capability wholesale. Buy an agentic platform from one of the workflow-automation or RPA vendors who've rebadged in the last eighteen months, let them run the platform layer, let their workflows define your agents.

This is a mistake, and a specific one. Not that you shouldn't get external help - most organisations will sensibly work with partners. The mistake is handing over the architectural decisions. Rent a vendor's abstraction and your data, governance, audit, and identity story is whatever they ship. When an agent does something it shouldn't, the loop back to the people who can change it goes through a support queue. You've outsourced the part of your business that's about to become strategic.

Building on a hyperscaler is genuinely different. Model behaviour, pricing, and roadmap are externally controlled either way, but on AWS, Azure, or Google you own the IAM, the audit, the cost controls, the integration patterns. You stand on their shoulders without handing over the decisions that shape how your capability evolves.

The second easy answer is to bolt the capability onto an existing function. Hand it to HR, hand it to IT, hand it to whoever has the closest claim, and declare the strategy done.

There are really two layers here, and they want different answers. The platform layer - IAM, audit, cost controls, the cloud foundation the agents run on - IT will rightly own as platform engineering. It's foundational and benefits from standardised delivery, same as any cloud platform.

The harder question is who owns the capability that uses the platform: designing the agents, deploying them into real business problems, shaping how people interact with them, deciding what good looks like. That's where the "give it to a function" answers break down.

Of the single-function answers, HR has the strongest case. Adoption really is the hard part, and behavioural standards and change management live closer to HR than to engineering. None of that is wrong. The problem is that bolting the capability onto any single function inherits that function's operating pattern, and strategically important work inside a shared service ends up under-invested and slow. The interesting question isn't "which function owns it?" It's whether you're building cross-disciplinary capability that lives in the business, or single-discipline capability that gets delivered to the business.

We've done this before

The thing that should give every senior leader pause is that we've already lived through a version of this. It was called Excel.

Spreadsheets put powerful data manipulation in the hands of capable people with no engineering scaffolding around them. Net positive on the whole, but it also gave us the London Whale loss at JPMorgan and a generation of businesses running critical processes on undocumented workbooks built by an analyst who left two roles ago. We're still cleaning that up decades later.

"Buy some licences and see what people can do" is the same energy. Powerful tools, capable hands, no scaffolding. Except an agent doesn't sit in a workbook waiting to be opened - it acts. Put one on top of a broken process and it won't just record the mess, it'll run it. We learned the lesson once. This time the spreadsheet could fire people.

What to do instead: build a squad

The answer isn't a new department. It's a deployment squad - and you can see the shape of one in the Kantar story, even if Andy doesn't call it that. The top agent builders are spread across HR, payroll, and analytics. The payroll manager who happened to be in the room when the Copilot licences went out became the AI manager. The head of analytics built the finance/HR reconciliation agent herself, eighty hours by her own count. The job title may be useful political cover, but it isn't the operating model. What Andy describes is a squad, not a function.

I'm extrapolating here. I haven't run an internal AI deployment squad inside a large enterprise. What I have watched, up close, is the pattern AWS Professional Services and the Partner Network use to ship production work inside customer organisations - and the forward-deployed engineer roles that Anthropic, OpenAI, and Google are now pitching look like the same shape. The underlying logic is one AWS worked out long ago: access alone isn't the bottleneck. Deployment is.

So build your own version. A small, cross-functional squad: platform engineers who can operate like internal forward-deployed engineers, IT people who already know your data and systems, HR expertise on behavioural standards and adoption, and line-of-business owners who know what good looks like. It embeds into specific problems and ships production agents. Not an innovation pod or a tiger team that spins up a demo and disbands. It owns what it ships, it integrates properly with the platform, and it stays.

The squad needs two things to work.

The first is platform discipline. The agents it builds are applications. They live on a platform layer that decides who an agent is allowed to be, what it can touch, what records it leaves, what it costs, and what it's allowed to do with sensitive data. Most of that comes from the cloud platform you already use. What it doesn't give you, you build and maintain yourself. Be clear from day one about which decisions belong to the platform and which belong to the agent, and resist the urge to rebuild platform-level safeguards inside every individual agent. Andy's team raced past the audit trail and had to come back to it later; he notes it would have been far more costly in a regulated industry. The discipline matters from the start, not as a retrofit.

The second is a sponsor senior enough to clear blockers but light enough not to slow it down, with the authority to ship without permission from every adjacent function. Andy had exactly this: permission to think bigger and permission to fail visibly. The sponsor's risk tolerance is what lets the squad ship and learn. The platform discipline is what decides whether the inevitable failures are recoverable embarrassments or unrecoverable disasters.

Where the squad sits on the org chart matters less than these two things.

One thing the podcast skips is how the rest of the organisation reacts. A cross-cutting squad shipping into work that IT, finance, or operations think of as theirs can be read as a threat, even when it's there to help. Handling that is real work, and it gets easier each time: squads that do it well don't stay one-offs. They become the prototype for how AI capability spreads, and the platform team scales with them.

What this means for HR, and for Helen

The more forward-looking parts of the profession have been moving from a roles-based view of the workforce to a skills-based one, where the unit of work is the skill, not the job title, and teams get composed from skills as the business needs them. Mercer's Jess Von Bank, in a recent People Management feature, pushes HR to design itself from first principles - for what the business needs now, not what the org chart has always done. A deployment squad, assembled from the skills a problem needs, is exactly what that model produces, which is why HR shouldn't try to own the AI capability but should help compose the team that delivers it. That's a craft the profession has spent a decade building, and it deserves better than to be flattened into change management.

For someone like Helen, that means becoming genuinely literate in the platform layer - what agents do, how they fail - without becoming an engineer. The more HR understands how these systems work in practice, the more useful it is to the teams that build and deploy them.

The squad is the starting move. Build enough of them and the question stops being "which function owns AI" and starts being "why are we organised around functions at all". That's a bigger argument, and not one for today - but you don't have to settle it to start.

Don't wait for a reorg to make it official. The future organisation doesn't arrive by redrawing the chart - it arrives one shipped agent at a time. The org chart will catch up when the work has earned it.

Dynoxide 0.10.0: it runs in the browser now

Martin Hicks — Fri, 29 May 2026 00:00:00 GMT

A DynamoDB-compatible database, running in a browser tab. No server, no network. The data lives in the origin private file system, and the query engine is the same Rust that backs the native build, compiled to WebAssembly.

That's the thing I'm most pleased about in dynoxide 0.10.0. It's a big release, and the first one that isn't a clean upgrade. So here's what shipped, why, and what bites you on the way up.

A database in the browser

Dynoxide started as an engine I wanted to embed in a native app. "Embed it anywhere" was always the pitch. 0.10.0 is the release where the browser counts as anywhere.

The native build backs onto SQLite through rusqlite. To run in a browser you need a SQLite that runs in WebAssembly, and that's wa-sqlite. Dynoxide talks to it over a wasm-bindgen bridge and persists to OPFS. Build it with --features wasm-sqlite and you get WasmDatabase, the same handlers exposed as async fn with no block_on in sight.

What works today: create table, put, get, delete, query, and scan, over base tables and both secondary index types (GSI and LSI). Index maintenance is atomic with the base write, same as native. It's enough to back a real client-side app.

What doesn't, yet: TTL returns a typed Unsupported error (I haven't implemented it yet). Streams are planned but not wired - the delivery design is the open question. And TransactWriteItems, tags, table-setting updates, stats, and bulk import all return a preview "not yet implemented" error. It's a preview, and I'd rather say so plainly than have you find out mid-build.

The detail I like most: no cross-origin isolation. SQLite in the browser usually needs COOP/COEP headers, because the common trick makes an async storage API look synchronous through a SharedArrayBuffer. Dynoxide sidesteps that by running wa-sqlite's synchronous OPFS VFS inside a Web Worker, where synchronous file handles are available directly.

So it drops onto ordinary static hosting. No special headers, no cross-origin isolation, no faff.

npm run build:wasm gives you a self-contained dist/ of three files, about 1.2 MB total. The harness under harness/ loads the exact same bundled worker a consumer would, so a green harness means the shipping artefact works, not a parallel build that happens to pass.

The trait that made it possible

You can't point one engine at both rusqlite and wa-sqlite if the data layer is welded to rusqlite. So the real work in 0.10.0 is underneath. There's a StorageBackend trait now, and Database is generic over it - Database<S>. The native rusqlite backend implements the trait, the browser one is WasmBridgeBackend, and both go through the same set of SQL builders.

A query fixed on one is fixed on both. That shared SQL is what lets me trust the browser build without a separate conformance run.

The handlers went async, because the browser backend is async by nature. Native keeps its old synchronous API. NativeDatabase drives each handler future to completion with block_on (via pollster), and because the native futures never actually suspend, that block_on never parks a thread.

It stays safe inside the tokio-based HTTP and MCP servers, and existing native code that names Database keeps compiling untouched.

The Docker image

Last time I said a Docker image was coming next. Here it is.

docker run -p 8000:8000 ghcr.io/nubo-db/dynoxide

FROM scratch, around 5 MB, multi-arch (amd64 and arm64). No shell, no OS, just the static binary. Against DynamoDB Local's ~225 MB Java image, it's a clean drop-in for the containerised test suites people actually run, and the one someone asked for.

It ships a HEALTHCHECK backed by a new dynoxide healthcheck subcommand, so docker ps and Compose health gates report status without you bolting on curl or wget (there's no shell in the image to run them with anyway). GHCR is the canonical home; Docker Hub and ECR Public get best-effort mirrors on each release.

What breaks

0.10.0 is the first breaking release, so this part matters.

If you run the binary, there's one change to know about: MCP over HTTP now requires a bearer token on every request. A loopback bind generates and persists a token on first run and prints a client-config snippet; a non-loopback bind won't start without one. Existing HTTP-transport MCP clients break until they send an Authorization: Bearer <token> header. The stdio transport is unchanged, and plain dynoxide serve (DynamoDB only, no MCP) is unchanged.

The reason is the obvious one. An MCP server reachable off-loopback with no auth is an open door to whatever it can touch. There's a SECURITY.md now that lays out the threat model, the token, and the Host and Origin allowlists behind it.

If you depend on the Rust crate, there are a few more. DynoxideError is now #[non_exhaustive], so an exhaustive match needs a _ => arm. Database being generic is source-compatible if you just name Database (it defaults to the native backend, and there's a NativeDatabase alias if you want to be explicit). And mcp::serve_http takes an HttpOptions struct now instead of a bare port.

The less glamorous half

Under the two headline features sits a stack of correctness fixes, most of them surfaced by the conformance suite I run alongside dynoxide. A few worth naming:

PartiQL DELETE and UPDATE now evaluate the whole WHERE clause, not just the key. The old behaviour could delete a row a filter should have excluded - a genuine data-correctness bug.
DescribeTable returns a stable TableId instead of minting a fresh UUID on every call.
Query and Scan over a GSI stop dropping items when several entries tie on the index key and the base table has only a partition key.
ConsumedCapacity now matches AWS on the transactional and PartiQL paths.

Where dynoxide actually stands against real DynamoDB and the other emulators lives at paritysuite.org. It moves as the suite grows and each engine changes, so I'm not going to pin a number here that's wrong by next week. Go and look at the live one.

What's next

Streams in the browser, once I've settled how delivery should work without a server to push from. And an npm package for the wasm build, so you can pull it in without wiring up the worker yourself - that path already works, it just isn't packaged.

The browser build is the proof I wanted. The engine isn't tied to one host any more. Native binary, Docker image, or a browser tab: same engine, same SQL, same behaviour.

Running iai-callgrind on Apple Silicon

Martin Hicks — Thu, 28 May 2026 00:00:00 GMT

My instruction-count benchmarks for Dynoxide, my embeddable DynamoDB engine in Rust, only ever ran in CI. The wall-clock benchmarks ran happily on my Mac, but the iai-callgrind track (benchmarks/benches/iai_core.rs) just sat there - because it needs Valgrind, and Valgrind doesn't run on Apple Silicon. Not "runs badly", not "needs a flag" - there's no aarch64 Darwin port at all. So on the machine where I write the code, I was guessing at performance until I pushed.

Which was a shame, because iai-callgrind is a lovely way to benchmark Rust code. Instead of wall-clock timing, which jitters with whatever else your machine happens to be doing, it counts CPU instructions under Valgrind's Callgrind. Same input, same count, every time. That makes it brilliant for catching the kind of regression that quietly adds 2% more work - exactly the sort of thing wall-clock noise buries. I wanted that locally, not just as a number CI reported back to me after the fact.

The fix is Docker. But there are a few catches that'll cost you an afternoon if nobody tells you about them first, so here they all are.

Run it in a native arm64 container

Valgrind on aarch64 Linux works fine. So the move is a native linux/arm64 container, not an emulated x86 one. Emulated x86 Valgrind is punishingly slow - you'd be running an instrumentation engine inside a CPU emulator. A native arm64 container runs at close to CI speed.

Start a long-lived container with the repo bind-mounted and an isolated target dir on a named volume. Match the rust: image tag to your project's toolchain.

REPO=/absolute/path/to/your/project

docker run -d --name dx-iai --platform linux/arm64 \
  --security-opt seccomp=unconfined \
  -v "$REPO":/repo \
  -v dx-iai-target:/target \
  -e CARGO_TARGET_DIR=/target -e CARGO_TERM_COLOR=never \
  rust:1.95-bookworm sleep infinity

The isolated CARGO_TARGET_DIR on a named volume earns its place twice over: it keeps your host target/ clean (it's a different target triple anyway), and it persists iai-callgrind's stored results between runs, which is what makes the before/after comparison work later.

That --security-opt seccomp=unconfined flag, though, is the one that isn't optional. Here's why.

The seccomp / personality() trap

iai-callgrind runs your benchmark binary under setarch -R to switch off ASLR - it needs stable memory addresses to get deterministic counts. setarch does that through the personality() syscall, and Docker's default seccomp profile blocks personality(). Leave the flag off and the run dies with:

setarch: failed to set personality to aarch64: Operation not permitted

That's a maddening error to land on cold, because nothing about it points at Docker's security profile. --security-opt seccomp=unconfined lifts the block, and that single flag is all it needs - Callgrind runs the target on its own synthetic CPU, so there's no ptrace and no extra capability to grant.

Installing Valgrind and the runner (mind the PATH)

Install Valgrind and the iai-callgrind runner inside the container. The runner version has to match the iai-callgrind crate version pinned in your Cargo.toml.

docker exec dx-iai bash -c '
  apt-get update && apt-get install -y valgrind &&
  cargo install iai-callgrind-runner --version 0.14.2
'

Note the bash -c, not bash -lc. A login shell re-runs /etc/profile, which resets PATH and drops the rust image's cargo bin off the end of it. Use a login shell here and cargo comes back "command not found" - which is a genuinely baffling thing to debug when you can see cargo sitting right there in the image.

Running the benchmarks

docker exec dx-iai bash -c 'cd /repo/benchmarks && cargo bench --bench iai_core --features iai-callgrind'

Two more traps:

Don't pass --locked. Resolving the iai-callgrind feature wants to touch Cargo.lock, and --locked aborts the whole run before it builds a single thing. The run will modify Cargo.lock as a side effect; if you'd rather not carry that change, reset it afterwards with git checkout -- benchmarks/Cargo.lock.

Don't pipe the run through | tail if you care whether it passed. The pipe swallows cargo's exit code, so a failed run cheerfully reports success. Ask me how I know.

Before and after

iai-callgrind stores results per benchmark in the target dir and auto-compares each run against the last. So the workflow is simple: run on your baseline, change the code, run again. The second run prints the delta per metric - -1.54%, [-1.01560x] and so on. The only requirement is that both runs share the same target dir, which is exactly why the setup above parks it on a named volume.

To compare two branches or commits, check the ref out in the host repo between runs. The container sees the change through the bind mount, no restart needed.

git -C "$REPO" checkout <baseline-ref>
docker exec dx-iai bash -c 'cd /repo/benchmarks && cargo bench --bench iai_core --features iai-callgrind'   # baseline
git -C "$REPO" checkout <changed-ref>
docker exec dx-iai bash -c 'cd /repo/benchmarks && cargo bench --bench iai_core --features iai-callgrind'   # prints the delta

The build cache lives in the target volume, so only the crates you actually changed recompile between runs.

The caveat that actually matters

Here's the one to keep in your head, because it's the difference between useful numbers and a wild goose chase: arm64 Callgrind instruction counts are not the same as x86 counts. Different architecture, different instructions, different totals. The numbers from your container won't line up with the x86 figures your CI publishes, and they're not meant to.

That's fine, as long as you use them for what they're good at. iai-callgrind on your Mac is for relative comparison - before versus after, on the same machine and the same arch. It is not for checking against an absolute figure from a different architecture. Treat the container's counts as "did my change make this cheaper or dearer", never as "does this match the published number". Get that backwards and you'll spend time chasing a regression that only exists because you compared arm64 against x86.

Cleanup

docker rm -f dx-iai
docker volume rm dx-iai-target

That's it: a native arm64 container, seccomp unconfined so setarch can disable ASLR, the right shell so cargo stays on PATH, and a clear head about relative versus absolute counts. None of it is hard once you know the traps. It's finding the traps that costs you the afternoon.

If you want to see what's actually being measured, the iai-callgrind suite lives in the Dynoxide repo. And if DynamoDB tooling is your sort of thing, Dynoxide itself might be worth a look.

How close is your DynamoDB emulator to AWS?

Martin Hicks — Tue, 26 May 2026 00:00:00 GMT

My DynamoDB conformance suite has a site now: paritysuite.org. Eight emulators, 684 tests, scored against live AWS DynamoDB.

The README only ever showed a single snapshot, frozen at whenever I last updated the table. The site shows everything around that snapshot. The homepage is the standings - every target ranked by how closely it matches real DynamoDB, with DynamoDB itself pinned at the top on a flat 100%. Each emulator gets its own page with run-over-run history - not just where a score landed, but where it moved and which tests pushed it. There's a support matrix that breaks every operation down target by target, and a runs archive going back through each scoring pass. Want to know whether LocalStack handles transactions, or where DynamoDB Local diverges on error wording? It's a couple of clicks now, not a repo checkout.

How it works

Every test runs against live AWS DynamoDB first. Whatever the real thing does is recorded as the expected answer, and an emulator only passes if it gives that same answer. The ground truth is never my reading of the docs - it's what DynamoDB actually returned when I asked it.

The results split into three tiers, because one number hides too much. Core is the everyday stuff: CRUD, queries, scans, batch operations. Complete adds the documented-but-less-common features like transactions, PartiQL, TTL and streams. Strict is the fiddly end - validation ordering, exact error wording, API limits, legacy shapes. A gap in Core breaks your app. A gap in Strict breaks the CI test that asserts on an error message.

Everything is checked through the standard AWS SDK against the target's HTTP endpoint. Nothing reaches inside the implementation. If your application would see it through the SDK, the suite checks it. If it wouldn't, it doesn't care.

And the suite grows. New tests land most weeks, so a score can move because the target changed or because I added coverage that exposed an old gap. A high score only means the target passes the tests that exist; behaviours the suite doesn't cover yet are blind spots, not passes. The methodology page has the full version.

What this run shows

The figures below come from the run dated 26 May 2026 - the site always shows the current ones.

Take LocalStack. It sits at 88% overall, which reads as middling until you split it: 98.6% on Core, 68.7% on Strict. That's not a mediocre emulator, it's a genuinely good one for everyday work that comes apart on error fidelity. The headline number flattens two completely different shapes of gap into one figure, which is precisely why the tiers exist. DynamoDB Local lands in almost exactly the same place - 97.7% Core, 69.2% Strict - which is no coincidence: LocalStack runs it under the hood. Same engine, same gaps, right down to the tier split.

This run also shows the suite breathing. It grew by 29 tests this pass, and the biggest movers all went down - Floci off 1.3 points, Dynalite 1.2, ExtendDB 1.1. Nothing regressed. The new tests just went looking where the old ones hadn't. That's the suite doing its job: coverage sharpens, scores dip, targets fix the gaps and climb back.

One result is worth singling out. Ask real DynamoDB for item collection metrics on a write with ReturnItemCollectionMetrics: SIZE and it hands them straight back. Ask DynamoDB Local, LocalStack, Dynalite or Ministack and you get silence - a documented response field they simply don't implement, AWS's own emulator included. Dynoxide, ExtendDB and Floci return it correctly. You'd never know unless something was checking.

Run it yourself

The whole thing is Apache-2.0 and every test is in the repo. Anything that speaks the DynamoDB HTTP API can be scored, so if you maintain an emulator, clone it, point it at your endpoint and see what comes back. You'll almost certainly find something. If you'd rather just see your emulator on the board, there's a suggest a target form.

If you spot a behaviour the suite doesn't cover yet, send a PR. The single rule is that the test has to pass against real DynamoDB first - if AWS rejects it, the test is wrong, not the emulator. That rule is the whole point. Targets are measured against real DynamoDB, never against each other, so two emulators agreeing on the same wrong answer can't quietly make it the standard. The suite is meant to be an independent reference the whole field can trust, not something any one project owns. No emulator author, me included, gets to mark their own homework.

And the field is genuinely getting good. Floci has come a long way since late April, from around 61% to the low 90s, with its Tier 3 score hauled up out of the twenties. ExtendDB only joined in late May and landed near the top straight away. It's an AWS-managed open-source project, Postgres-backed, that speaks the DynamoDB wire protocol. Worth a look.

Dynoxide, my own engine, sits in there on the same terms as the rest: same tests, no favours. Floci and ExtendDB are a couple of points behind and still climbing, and the whole field is closing on real DynamoDB.

Filing my first security advisory

Martin Hicks — Tue, 12 May 2026 00:00:00 GMT

Yesterday I logged into GitHub and noticed a Dependabot alert sitting on dynoxide. DNS rebinding CVE in rmcp, a transitive dependency. The alert had been raised a few days earlier - I just hadn't seen it, because I'd never set up email notifications for Dependabot on that repo. First lesson of the day, before the actual lesson of the day.

I worked through the fix and published a GitHub Security Advisory once the patch release was out. The fix itself was the bit I knew how to do. The GHSA was new ground, and that's the part I want to write about, because if you maintain something with users and you've never filed one, the process is less daunting than I'd built it up to be.

The notification gap

Worth dealing with this one first.

Dependabot raises alerts on your repo's Security tab automatically. By default, GitHub doesn't email you about them - they appear on the dashboard and that's it. If you don't log into the affected repo regularly, you won't see them.

The fix is at the repo level. Top right of the repo page, hit Watch → Custom, then tick Security alerts. You'll start getting emails for security events on that repo.

I can see why this isn't the default. If you've forked a load of old projects you don't actually maintain, the last thing you want is an inbox full of alerts for repos that aren't yours. The cost of that default is that on the repos you do maintain, you have to opt in deliberately. I hadn't.

There's a second timing thing worth knowing about. Even with email notifications set up, there's a gap between the upstream CVE being published and your Dependabot alert firing. For mine, the upstream rmcp advisory was published on 29 April. My Dependabot alert came through about nine days later. GitHub's Advisory Database reviews advisories before they propagate, and Dependabot itself runs on a scan schedule. The delay is mostly outside your control. Knowing it exists at least means you don't panic when you discover the upstream advisory predates your own alert.

What a GHSA actually is

A GHSA is a structured record on your repo's Security tab. Affected package, affected version range, fixed version, severity, CVSS vector, weakness category, description. The structure matters because of what happens when you publish it: GitHub's advisory database picks it up, and every downstream project that depends on your package gets a Dependabot alert. Often an automated upgrade PR.

In my case this was the whole point. The named CVE lives in rmcp. Anyone with a direct rmcp dependency already had an alert from the upstream advisory. But dynoxide users have rmcp transitively - it's in the lockfile, not the manifest - and the upstream alert doesn't fire for them. Without a dynoxide-specific GHSA naming the affected version range of dynoxide-rs and dynoxide, nothing reaches those users automatically.

The GHSA isn't ceremony. It's the actual distribution mechanism.

Order of operations

Release first, advisory second.

The advisory references the patched version. Dependabot's upgrade PRs for downstream projects will fail if that version doesn't exist on the registry yet. If you publish the GHSA before the release is live, you've broadcast "upgrade to a version that doesn't exist" to every project that depends on you.

So: cut the release, wait for CI to push to the registries, verify the patched version is actually available, then publish the advisory. The save-as-draft option on the GHSA form is there for this reason.

Last step is the social post, after opening the GHSA URL in a private window to check a logged-out viewer can actually see it. Until you hit publish, the advisory is only visible to repo admins and the URL returns nothing useful to anyone else.

Filing the form

The form lives at Security → Advisories → New draft security advisory. A few things that tripped me up.

Ecosystem. The dropdown doesn't list "cargo" - it lists "Rust". I scrolled past it twice. The mapping is to crates.io under the hood. npm is named npm.

Package names. Must match the registry exactly or Dependabot doesn't fire. GitHub validates this and shows a small "Package name found on Rust" confirmation when it's right. For dynoxide that's dynoxide-rs (the Cargo crate, because the name dynoxide was already taken) and dynoxide (npm).

Version ranges. Cargo-style semver: >= 0.9.3, < 0.9.13. I started at 0.9.3 because that's when the MCP HTTP transport landed. Earlier versions don't expose the vulnerable surface, so flagging them would just generate noise alerts for users on older versions who aren't actually affected.

CVE. The upstream CVE-2026-42559 already exists. There's a field for "I have an existing CVE ID" - use it. Don't request a new one for the same root cause.

CVSS. This was the field I felt least sure about. There's an in-form calculator and I had a go at deriving my own vector for dynoxide's exposure context. I marked Attack Complexity as High because the rebinding chain felt fiddly to me. That was wrong - upstream marked it Low, and Low is right, because public DNS rebinding tooling makes the attack cheap to execute. My score came out at 7.5 instead of upstream's 8.8.

The mistake was easy to make and easy to fix (GHSAs are editable after publishing), but it would have saved me both the agonising and the error to just match the upstream score in the first place. For a transitive-dep advisory where the exposure mechanism is essentially the same as upstream, divergent scores just confuse downstream readers. Match upstream unless you've got a strong reason not to.

CWE. Weakness category. The upstream advisory used CWE-346 (Origin Validation Error) and CWE-350 (Reliance on Reverse DNS Resolution). Same advice as CVSS: match upstream rather than reasoning from first principles.

Description. Markdown. I used Summary, Impact, Patches, Workarounds, References. The Workarounds section is the one I'd most strongly suggest including - if a user can't upgrade immediately, knowing how to mitigate is more useful than knowing the CVSS score. For dynoxide that was "use the stdio transport, don't pass --http."

The bug

The named CVE is DNS rebinding. You're running dynoxide mcp --http so your coding agent can talk to a local DynamoDB. You visit a malicious page in another tab. The page's JavaScript spoofs the Host header in requests to 127.0.0.1:PORT/mcp and the server processes them. The attacker can call any tool the running dynoxide instance exposes, including writes.

The rmcp 1.4+ fix is a Host-header allowlist: localhost, 127.0.0.1, ::1, anything else gets a 403. Upgrade rmcp, done.

While I was reading the patched rmcp source to make sure I understood what the upgrade actually changed, I clocked something. The Host check closes rebinding. It doesn't close cross-origin CSRF.

fetch('http://127.0.0.1:PORT/mcp', {
  mode: 'no-cors',
  method: 'POST',
  body: JSON.stringify({ /* call put_item */ })
})

The browser sends Host: 127.0.0.1 - legitimately, because that's the URL it's connecting to - and Origin: https://evil.com. The Host check passes. There's no Origin check by default. The request goes through.

A pedant's note: a no-cors POST with a JSON body actually can't set Content-Type: application/json (browsers strip it down to text/plain), so rmcp's protocol-level Accept and Content-Type checks would bounce this exact request before tool execution. The point still stands. Content-type validation isn't a security boundary - alternative request shapes that sidestep no-cors's restrictions (form POSTs, future MCP transport variants) reach the handler with valid framing. Origin is the right layer to enforce same-origin, not content-type.

This isn't a flaw in the upstream fix. The CVE is narrowly scoped to DNS rebinding, and the fix closes it. Cross-origin CSRF is a different shape of attack on the same transport surface, and the upstream rmcp team have it tracked as a defence-in-depth follow-up. rmcp 1.6.0 already ships an allowed_origins field on the same config - it just defaults to empty, which means "skip validation". Same pattern Host had before 1.4.

So dynoxide 0.9.13 sets both lists explicitly:

c.allowed_hosts = vec!["localhost".into(), "127.0.0.1".into(), "::1".into()];
c.allowed_origins = vec!["http://localhost".into(), "http://127.0.0.1".into()];

Plus a regression test covering all three paths: loopback Host with no Origin (passes, because native MCP clients don't send Origin), foreign Host (403), foreign Origin (403). The test asserts on the rejection message, not just the status code, so a future rmcp change that returns 403 for some other reason can't keep the test green while reopening the vulnerability.

What I'd tell past-me

The advisory is a positive signal. I was anxious about publishing it. Felt like it made dynoxide look amateur, like I was admitting fault. The opposite is true. Projects with zero advisories either have no users or aren't handling issues responsibly. Filing one is what well-run projects do.

Match the upstream CVSS and CWEs. Already covered above. The recalculation is defensible in theory but the consistent practice is simpler, faster, and less error-prone.

Turn on the email notifications. Worth the small ongoing noise on the repos you maintain.

One thing the whole exercise sharpened for me: the HTTP transport doesn't authenticate callers at all. The Host and Origin allowlists defend against browsers, but anything on the same machine that can reach the loopback port can call any tool. Worth saying why.

Why no auth (yet)

The MCP HTTP transport doesn't authenticate callers. That was a deliberate choice, not an oversight. dynoxide's main job is to be a small fast binary running in a CI job or on your own machine - both isolated environments where the only thing that can reach the loopback port is the process that launched it. The threat model is browser-borne attacks against that port, which is exactly what the Host and Origin allowlists handle. Adding token-based auth on top buys you very little: you'd be issuing yourself credentials to talk to yourself, and self-issued credentials tend to end up in dotfiles or pasted somewhere they shouldn't be.

What's changed the calculation is the Docker image I've been working on. A container is a different deployment shape: it's something you can hand to another developer, drop into a shared environment, or expose on a network where the caller isn't the process that launched it. The trust boundary widens and the assumptions behind shipping without auth stop holding.

So auth is the next piece of work. There's more design space here than I expected - the MCP spec is OAuth 2.1 or nothing, rmcp doesn't ship an auth hook of its own, and the path that actually fits a single-user local tool (a pre-shared bearer token wired in as a tower middleware) isn't really what the spec contemplates. More on that when it lands.

dynoxide 0.9.13 is out. Upgrade if you're running dynoxide mcp --http or dynoxide serve --mcp. The advisory is at GHSA-fvh2-gm75-j4j7 if you want a look at the finished form. If you've never filed a GHSA and you've got the option to file a draft on a private repo just to see the form, that's the cheapest way to take the unknown out of it before you have to do it for real.

Update (evening, same day)

One useful incidental find since publishing this article earlier this afternoon...

The RustSec → GHSA import is one-way: the GitHub Advisory Database imports from RustSec, not the other way round. So if you filed your advisory via the GitHub repository security advisory flow (as I did), it lands in GHSA but not RustSec. Anyone running cargo audit or cargo deny doesn't get an alert.

Fix: file a RustSec advisory at rustsec/advisory-db. It's a markdown PR. Takes about 20 minutes. Closes a real gap for Rust users and gives you a second distribution channel for any future advisory.

My PR is rustsec/advisory-db#2852 if you want a worked example.

Dynoxide patch notes: 0.9.10 to 0.9.12

Martin Hicks — Tue, 05 May 2026 00:00:00 GMT

Dynoxide now lives entirely in its own public repo, out of the private workspace it shared with Nubo. One repo, one CI pipeline, no shuttling private commits to test against the public version. That's the work behind 0.9.10 that doesn't show up in the changelog.

Three patch releases since.

0.9.10: making error messages actually match

v0.9.10 closed 16 places where dynoxide's error strings drifted from real DynamoDB.

Dynoxide's whole pitch is "behaves like AWS DynamoDB". Easy to claim and hard to keep. AWS doesn't publish a spec for the error strings their SDK clients see; they just emit them, and downstream code parses them.

The dynamodb-conformance project I run alongside dynoxide has been growing a tier-3 suite that asserts on exact error strings. Three rungs of strictness:

Rung 1: literal exact match
Rung 2: interpolated exact match, with test fixture values substituted in
Rung 3: anchored regex around the bits AWS controls (e.g. the Java toString dump of a request body)

601 tests now. Things you only notice when you assert:

tableName length validation is per-operation - 1 char on read/write, 3 on CreateTable
Select enum order matches AWS rather than alphabetical
Query and Scan Limit=0 messages are deliberately different
batch and transact empty/oversize requests use the standard 1 validation error detected: Value '...' at 'X' failed to satisfy constraint: ... envelope
UpdateExpression syntax errors include the AWS near: "..." window

Plus one real bug. TransactGetItems with a missing key was returning HTTP 500 instead of TransactionCanceledException with a ValidationError cancellation reason. The dedup loop was calling the server-fault helper before key validation. That's the kind of bug you only catch when your test suite holds error messages to byte-strictness.

0.9.11: MCP server hanging on Ctrl+C

v0.9.11 fixed dynoxide hanging on Ctrl+C when an MCP client (Claude Code, Cursor) was connected.

In a side project, dynoxide runs alongside four other processes via concurrently: two seed scripts and two vite servers. Plus Claude Code with .mcp.json pointing at dynoxide's MCP endpoint.

Ctrl+C to stop the dev session. npm run dev again.

error: failed to bind 127.0.0.1:8000: Address already in use

A second Ctrl+C would unstick it. Sometimes a pkill -9 dynoxide. Annoying.

The MCP server's shutdown path was using axum's with_graceful_shutdown(...), which drains in-flight connections before returning. Fine for the short-lived AWS-SDK requests on the HTTP side. Not fine for Streamable-HTTP MCP sessions that Claude Code holds open indefinitely. The drain phase waited forever, so the dynoxide process hung, so the port stayed bound, so the next npm run dev failed.

The fix: race the cancellation token against the serve future and drop the future on cancel. The listener closes, the function returns, the surrounding tokio::join! proceeds, the process exits. No drain phase.

tokio::select! {
    res = serve_fut => res?,
    _ = ct.cancelled_owned() => {}
}

The HTTP server keeps with_graceful_shutdown because AWS-SDK requests drain in milliseconds.

0.9.12: TIME_WAIT and the case of the still-bound port

v0.9.12 fixed port 8000 staying bound for around 60 seconds after a clean shutdown, if anything had connected during the session.

Shipped 0.9.11. Tested in the same project. Got the same error.

error: failed to bind 127.0.0.1:8000: Address already in use

But this time the dynoxide process had exited cleanly. "Shutting down..." printed, exit code 0. So why was the port still held?

netstat -an | grep 8000 showed it. TIME_WAIT entries. Both directions. The seed scripts had opened TCP connections to dynoxide during the run, those went through TIME_WAIT on close, and the kernel sits on TIME_WAIT for around 60 seconds so stray packets can't land on a fresh listener.

dynoxide's listener was using socket2 with a comment that read:

// Deliberately do NOT set SO_REUSEADDR - this is the whole point.

The comment was wrong. SO_REUSEADDR doesn't allow two live listeners to share a port. That's SO_REUSEPORT. SO_REUSEADDR only bypasses TIME_WAIT entries. The port-conflict detection (a TCP connect probe before the bind) still works either way.

One-line fix:

#[cfg(unix)]
socket.set_reuse_address(true)?;

Gated to Unix because Windows is different. SO_REUSEADDR on Windows lets another process hijack an active bind, which is the opposite of what you want. The Windows-correct answer is SO_EXCLUSIVEADDRUSE, and that's a different change for a different release.

After 0.9.12: npm run dev, Ctrl+C, npm run dev. Instant.

Coming next: a Docker image

Someone's asked for an official Docker image. Hadn't crossed my mind. If you're already running a Docker-based dev or CI workflow around DynamoDB Local, a FROM scratch dynoxide image is an easy drop-in: ~5 MB against the ~225 MB Java image. The release pipeline already builds static linux-musl binaries for npm and Homebrew, so the Docker work is wrapping one of those in a FROM scratch layer. No shell. No OS. Just the binary.

What I take from this

If I hadn't pushed dynoxide to byte-exact error matching, I wouldn't trust it as a drop-in for DynamoDB Local. I wouldn't have used it for real. I wouldn't have hit either of these bugs. The strictness of the conformance work is what gave me the licence to use it in anger.

And the other thing: comments lie. "This is the whole point" had been sitting in the code since at least v0.9.7. It looked authoritative. It was wrong. The bug stayed latent until I started running dynoxide next to scripts that opened a few thousand TCP connections in succession.

Worth questioning the "deliberately" comments in your own code. Especially the ones written by past-you.

A year with my Intel N100 home server: what changed

Martin Hicks — Sat, 02 May 2026 00:00:00 GMT

It's been a year since I built my Intel N100 home server, and a few things have happened since I wrote that post. Some of what I set up is still humming along untouched. Some of it I quietly tore out. And some of what I added wasn't on the original plan at all.

This is the honest "what I'd actually do differently" follow-up.

The boring infrastructure won

The most useful thing I can report is how little I've thought about this server. It's been up for 50+ days at a stretch between reboots - most recently for kernel updates, not because anything broke. Total downtime in a year has been under an hour, all of it self-inflicted.

The services I set up first are still the ones earning their keep:

AdGuard Home is invisible until I open the dashboard. The kids' devices, our laptops, the smart TV - they all quietly stop talking to ad and tracker domains, and nobody in the household notices except when I show off the daily query log.
WireGuard for the kids' devices does what I built it to do. They get filtered DNS at school, at friends' houses, on mobile data. I have not had to debug it once.
Time Machine over SMB has been completely reliable. Three Macs, no failed backups, no sparsebundle corruption.

That alone makes the £250 build worth it for me.

What I removed: Unbound

The biggest change to the original design was disabling Unbound.

The premise was great: AdGuard handles filtering, Unbound handles recursive resolution from the root servers down, no single upstream provider sees my full browsing history. Privacy through distribution.

In practice, the latency was a problem. Popular domains were fine - they cache locally and return instantly. But for the long tail of less-popular domains, recursive resolution means walking the DNS tree on demand, and each hop is a real network round-trip from a UK home connection to wherever the next nameserver lives. By the time you've gone root → TLD → authoritative → CDN → CNAME chain, you've spent enough milliseconds for a person to notice. Pages felt sluggish to load, and family members were the canary.

Switching AdGuard to forward directly to a single fast public resolver fixed it instantly. I lost the "no third party sees my queries" property, but I kept the filtering and gained the snap-back responsiveness people actually feel.

The lesson I'd pass on: measure DNS perceived latency from someone who doesn't know how DNS works, not from yourself looking at dig output. Your tolerance is much higher than theirs.

What I added: Tailscale

The unplanned addition that changed the most was Tailscale.

Originally I thought WireGuard could do remote access for me too - just spin up a personal config and tunnel home when I need to manage the server. Technically true. In practice, Tailscale's mesh VPN with magic DNS is so much more pleasant for "just let me reach my server from anywhere" that I never bothered.

Now the n100 lives at a fixed *.ts.net name regardless of where I am. I can SSH from a coffee shop without configuring anything. There's zero port forwarding on the router for my own access - Tailscale handles NAT traversal. SSH itself isn't exposed to the public internet at all.

I still keep WireGuard for the kids' devices because that's a single-purpose dependency I want to fully own. But for me, Tailscale won.

What else found its way onto the box

Three things I didn't plan for ended up earning their place:

AirConnect turns Chromecast and UPnP speakers into AirPlay targets. We had a couple of speakers that aren't Apple-aware, and the alternative was buying new hardware or waving phones at the wrong device for ten minutes. AirConnect runs as two systemd services. Worth it. The Google Home speakers we have are sometimes slow and drop out intermittently though, so I might not have perfectly cracked this yet.

Netdata gave me a real-time monitoring dashboard at http://n100-home:19999/. I won't pretend I check it daily - but every time I have checked it, it's earned its place. When I wondered whether Time Machine backups were CPU-bound, the answer was right there in a chart. When I worried about thermal throttling on the passively-cooled N100, the temperature traces said no. The investment in installing it is small; the times you need it, you really need it.

Fail2ban is the kind of thing you put on and forget. SSH isn't exposed to the public internet anyway (Tailscale-only), but Fail2ban means even on my own LAN there's a hard limit on bad-credential noise. Belt and braces.

What I'd change about the build

Going back to the hardware, my original post said I'd go SSD-only if I were rebuilding. I'd take that back.

The 2.5" HDD has absorbed something like 1.3 TB of Time Machine writes over the year. Putting that on an SSD would have eaten into its endurance for no real benefit - Time Machine's bottleneck is the network, not the disk. The split between fast NVMe for the OS and a slow spinning disk for backups has been quietly correct.

The other thing I overthought before the build was HDD noise. I went down a rabbit hole comparing "quiet" enterprise drives, almost talked myself into going SSD-only just to dodge the question, and even called the HDD the noisiest part of the system in the original post. In practice, a one-line hdparm -S 120 /dev/sda (run as a tiny systemd unit at boot) puts the drive into standby after ten minutes of idle, and since Time Machine is the only thing using it, I genuinely can't hear it from the same room. The compromise that wasn't.

The one mild regret on the build is disk size. The boot NVMe is 500 GB and is barely 5% used. I could have got away with 256 GB and saved a few quid. The Time Machine HDD, on the other hand, is now 73% full at 1.3 TB / 1.8 TB. I've capped Time Machine at 1.5 TB to leave headroom. Anyone planning the same setup should size the backup disk for at least 3× their largest Mac's data.

The numbers, a year in

Metric	Value
Uptime since last reboot	50 days (kernel update)
Total downtime in a year	< 1 hour
RAM used	1.6 GB / 16 GB
CPU load average	~1.0 of 4 cores
Idle power	6-8 W
Total annual electricity cost	~£20
Things that have broken on their own	None

There's still room for everything I could plausibly throw at it.

What's next

The infrastructure has gone quiet, which means there's headroom. CPU mostly idle, 14 GB of RAM untouched, and a backlog of self-hosted bits I've been meaning to try. Nextcloud for files I'd rather not park on someone else's cloud. Jellyfin for the family media library. Home Assistant to do home automation properly. A few others on the maybe pile.

Whatever earns its place, you'll hear about it.

Earlier in this series: Building a tiny Intel N100 home server · Running AdGuard Home and Unbound · WireGuard for safe browsing on kids' devices

Building a DynamoDB conformance suite

Martin Hicks — Mon, 13 Apr 2026 00:00:00 GMT

When I started building Dynoxide - a DynamoDB-compatible engine in Rust - I kept running into the same problem. I'd implement an operation, write tests against my understanding of DynamoDB's behaviour, and ship it. Then someone (usually me) would discover that real DynamoDB does something slightly different. A validation error with a different message. A condition expression that fails where I expected it to succeed. An edge case in number precision that I'd never considered.

Every emulator ships with "DynamoDB compatible" and you're expected to take their word for it. I needed a way to answer a simple question: does Dynoxide actually behave like DynamoDB?

No public conformance suite exists for DynamoDB - not from AWS, not from the community.

So I built one.

The idea

The principle is straightforward. Write a test. Run it against real DynamoDB. Record the result. That recorded result becomes the ground truth. Then run the same test against an emulator and compare.

If the emulator's response matches DynamoDB's, the test passes. If it doesn't, the emulator has a conformance gap. No opinions, no guesswork about what the documentation means - just "does this behave the same way the real thing does?"

I built it as a standalone project - dynamodb-conformance - so anyone can clone it, point it at their own DynamoDB-compatible endpoint, and get results. It uses the AWS SDK v3 for TypeScript and vitest as the test runner. Nothing exotic.

Tiers

Not all DynamoDB operations are equally important. If your emulator can't do a basic PutItem, nothing else matters. If it doesn't perfectly replicate the error message format for an obscure validation edge case, that's less critical.

So the suite is split into three tiers.

Tier 1 covers core CRUD operations - CreateTable, PutItem, GetItem, UpdateItem, DeleteItem, Query, Scan, BatchGetItem, BatchWriteItem. These are the operations every application uses. If an emulator fails here, it's not usable for real development.

Tier 2 covers advanced features - transactions, PartiQL, streams, tags, TTL, and table updates like billing mode changes. These are features that production applications rely on but that you might not hit in a simple prototype.

Tier 3 covers edge cases and error fidelity - validation ordering, exact error message formatting, API limits, and legacy API behaviour. The things that only matter when you need your emulator to be indistinguishable from the real service.

This tiered structure means a result like "100% Tier 1, 95% Tier 2, 80% Tier 3" actually tells you something useful. It means core operations work perfectly, advanced features are mostly there, and edge cases have some gaps. A flat "92% overall" hides whether the failures are in PutItem or in obscure validation corner cases.

Ground truth

The key design decision was making DynamoDB itself the source of truth rather than my interpretation of the documentation.

Every test in the suite has been run against a real DynamoDB table in eu-west-2. The expected values in the tests aren't what I think DynamoDB should return based on reading the docs - they're what DynamoDB actually returns. This matters more than you'd expect. DynamoDB's documentation is good, but it doesn't cover every edge case. Sometimes the only way to know what DynamoDB does is to try it and see.

A scheduled CI job runs the full suite against real DynamoDB weekly, so if AWS changes something - a new validation, a different error message, a behavioural tweak - the ground truth stays current. The suite can't go stale the way a set of hand-written expected values would.

Results

Here's where it gets interesting.

Target	Tier 1 (267)	Tier 2 (93)	Tier 3 (166)	Overall
DynamoDB (eu-west-2)	100%	100%	100%	526 / 526
Dynoxide	100%	100%	100%	526 / 526
LocalStack	98.9%	95.7%	81.9%	489 / 526
DynamoDB Local	98.9%	90.3%	81.9%	484 / 526
Dynalite	98.1%	10.8%	92.8%	426 / 526

Dynoxide passes every test. 526 out of 526, across all three tiers.

DynamoDB Local - AWS's own emulator - fails 42. LocalStack fails 37. Dynalite, which hasn't been actively maintained for a while, fails 57 (and that Tier 2 number of 10.8% tells you it's missing entire feature categories like transactions and PartiQL).

I want to be careful about what this means and what it doesn't.

526 tests is a lot, but DynamoDB is a massive service. There are aspects of it that the suite doesn't cover - DAX compatibility, on-demand capacity semantics, cross-region replication, fine-grained IAM condition keys. "100% conformance on 526 tests" is accurate. "100% DynamoDB compatible" is not, and I wouldn't claim it.

What the results do tell you is that for the operations and behaviours the suite covers - which include everything most applications use day to day - Dynoxide matches DynamoDB exactly. Not approximately. Exactly.

What DynamoDB Local gets wrong

Some of DynamoDB Local's 42 failures are genuinely surprising. This is AWS's own emulator. You'd expect it to be the gold standard for local DynamoDB development. In practice, it has real gaps across all three tiers.

Three of its Tier 1 failures are in ItemCollectionMetrics - it doesn't return them for PutItem, DeleteItem, or UpdateItem even when you ask for them with ReturnItemCollectionMetrics: SIZE. That's not an edge case. That's a documented feature of the API that silently does nothing.

In Tier 2, the entire tagging API is broken. All eight tag-related tests fail - TagResource, UntagResource, ListTagsOfResource. If you're writing infrastructure-as-code that tags tables and you're testing against DynamoDB Local, none of that is being validated. It also gets PartiQL INSERT semantics wrong - it treats INSERT as an upsert when real DynamoDB correctly rejects an INSERT on an existing item.

The bulk of the failures (30) are in Tier 3 - validation ordering and error message formatting. DynamoDB Local returns different errors than production for the same bad request. This might sound academic, but if you're writing error handling code against DynamoDB Local, it might not work the same way against the real thing. You'll get the right error type but the wrong message content, or the validations will fire in a different order.

These are the kind of differences you only discover when you have a conformance suite to catch them. Without one, they hide as production bugs that are impossible to reproduce locally.

Why bother?

I built this because I needed it for Dynoxide. I wanted confidence that the thing I was building actually worked the way DynamoDB works, not the way I assumed it works. But the suite is useful beyond that.

If you're choosing between DynamoDB emulators for your development workflow, you can run the suite yourself and compare. The numbers above aren't marketing claims - they're reproducible results from a public repository that anyone can verify.

If you maintain a DynamoDB emulator or compatibility layer, you can use it to find and fix gaps. Point it at your endpoint, see what fails, fix the failures. The tiered structure tells you which failures matter most.

And if you're using DynamoDB Local in your CI pipeline and wondering why a test passes locally but fails against real DynamoDB - the conformance suite might tell you why. The 42 things DynamoDB Local gets wrong are 42 potential sources of exactly that kind of bug.

526 tests is solid coverage, but there are DynamoDB behaviours the suite doesn't exercise yet. If you've hit a case where an emulator diverges from real DynamoDB and you think it should be covered, open a PR. The only requirement is that the test passes against real DynamoDB - include the results artifact from a run against your own AWS account and I'll verify it against the ground truth pipeline. More tests make the suite more useful for everyone, not just Dynoxide.

The suite is at github.com/nubo-db/dynamodb-conformance. Clone it, run it, check the results for yourself.

Dynoxide 0.9.8: fixing the orphan problem

Martin Hicks — Mon, 06 Apr 2026 00:00:00 GMT

Dynoxide 0.9.8 fixes a bug where backgrounding dynoxide in an npm script left it running after the parent process exited. The port stayed bound, and the next npm run dev failed with a port conflict. There were three separate causes.

The Rust server ignored SIGTERM

kill <pid> sends SIGTERM by default, but Dynoxide only handled SIGINT (Ctrl+C). The fix is a tokio::select! between ctrl_c() and a SIGTERM listener:

async fn shutdown_signal() {
    #[cfg(unix)]
    {
        use tokio::signal::unix::{SignalKind, signal};
        let mut sigterm =
            signal(SignalKind::terminate()).expect("failed to install SIGTERM handler");
        tokio::select! {
            _ = tokio::signal::ctrl_c() => {},
            _ = sigterm.recv() => {},
        }
    }
    #[cfg(not(unix))]
    {
        tokio::signal::ctrl_c()
            .await
            .expect("failed to install CTRL+C handler");
    }
    eprintln!("\nShutting down...");
}

This applies to all installs (Homebrew, cargo, GitHub Action), not just npm.

spawnSync blocks the event loop

The npm wrapper used spawnSync to launch the Rust binary. That blocks the Node.js event loop entirely - no signal handlers fire, no process.on('exit'), nothing. The shim can't forward signals to the child because it's stuck in a synchronous call.

Switched to async spawn with explicit signal forwarding (SIGINT, SIGTERM, SIGHUP), same pattern esbuild and Biome use. The child spawns with detached: true to avoid double SIGINT delivery - without it, Ctrl+C sends SIGINT to both the shim and the child via the foreground process group, and the shim forwards it again. Detaching makes explicit forwarding the only signal path.

There's also double-signal escalation: first Ctrl+C forwards SIGINT for graceful shutdown, second sends SIGKILL.

Backgrounded processes never get signals

The & in a typical npm script pattern like dynoxide & sleep 1 && npm run seed && react-router dev puts dynoxide outside the foreground process group. Ctrl+C goes to the dev server. SIGTERM from kill targets the parent shell. No signal reaches dynoxide at all.

The fix is polling process.ppid on a 1-second interval. When the parent dies, the OS reparents the process to PID 1 (or launchd on macOS) and the PPID changes. When detected, the shim sends SIGTERM to the child with a SIGKILL fallback after a grace period. The interval uses .unref() so short-lived commands like dynoxide --help exit immediately.

detached and PPID polling are coupled

detached: true means the OS won't automatically kill the child if the wrapper crashes or gets SIGKILL'd. Without PPID polling, a detached child orphans more easily than the old spawnSync behaviour. Removing the polling while keeping detached: true would make things worse, not better.

Upgrade

npm install --save-dev dynoxide@latest

Or if you're using Homebrew or cargo, the SIGTERM fix applies regardless of how you installed it.

Changelog and issue on GitHub.

Introducing Dynoxide: a fast, embeddable DynamoDB engine

Martin Hicks — Thu, 02 Apr 2026 00:00:00 GMT

I've been working with DynamoDB for the best part of a decade. It's my default database for most things I build at Si Novi, and I genuinely like it. The data modelling is satisfying once it clicks, the operational overhead is basically zero, and it scales without you having to think about it.

Last year, Si Novi started building Nubo, a native DynamoDB client for macOS and iPadOS, with Windows and Linux on the way. One of the features we wanted from the start was a built-in sandbox. A local DynamoDB instance running on-device that you could experiment with, no AWS credentials needed. Create tables, insert data, test access patterns. All offline. All on your machine. Maybe even on an iPad.

The problem was: how do you ship that?

DynamoDB Local's licence explicitly prohibits redistribution. You can't bundle it inside another application or sublicence it. Even setting the licence aside, the developer experience would have been awful. Asking users to separately download and install a Java-based emulator, with Docker or a JVM as a prerequisite, just to use one feature in your app? That's not a sandbox. That's homework.

And on iPadOS there's no path at all. No JVM. No Docker. Nothing.

So I started building something new. A DynamoDB-compatible engine in Rust, backed by SQLite, that compiles to a single native binary. Something I could embed directly into Nubo as a library.

That's how Dynoxide started.

From library to tool

The first version did exactly what I needed: a local DynamoDB that Nubo could bundle as its sandbox. Job done, in theory. But the thing that surprised me was just how small and fast it turned out to be.

A ~3 MB download (~6 MB on disk). Under 5 MB of RAM at idle. Cold startup in about 15 milliseconds, compared to over two seconds for DynamoDB Local. Numbers like that change what's practical. Instead of running one shared DynamoDB instance in your CI pipeline, you could spin up a fresh one per test. Isolated state. No cleanup step. No flaky tests from leftover data. Tear it down when you're done and the overhead is essentially nothing.

That realisation shifted what Dynoxide was becoming. It wasn't just infrastructure for Nubo any more. It was a tool in its own right.

The same properties that made it good for CI made it a natural fit for agentic development. A coding agent that needs to create tables, write data, or test queries can do all of that through Dynoxide's built-in MCP server. 34 tools, no setup required. Run dynoxide mcp and your agent has a full DynamoDB environment to work with.

And then I looked at how many Node.js projects still depend on dynalite for local DynamoDB. I owe a lot to dynalite. Using it through arc.codes and Architect is what turned DynamoDB from a database I'd reach for occasionally into the one I'd reach for first. Having a fast local emulator that just worked changed how I thought about building with DynamoDB. It made the feedback loop tight enough that I actually wanted to experiment. But dynalite hasn't seen updates in a while and newer DynamoDB features like transactions and streams aren't covered. Dynoxide supports both, along with the rest of the API surface. With an npm package shipping platform-specific binaries (the same approach esbuild and Biome use), it works as a drop-in replacement. Same workflow, one line change in your package.json.

Under the hood

Dynoxide implements the DynamoDB API as a translation layer in Rust, storing data in SQLite and compiling to a native binary with no runtime dependencies. DynamoDB Local takes the same general approach (SQLite as the storage engine is well documented) but ships as a Java application with a JVM dependency and a restrictive licence.

Because SQLite is an embedded database, Dynoxide doesn't have to run as a standalone server. It can be linked directly into another application, which is exactly how Nubo uses it. Encrypt the database file with SQLCipher and you've got a portable, encrypted, DynamoDB-compatible data store. Nubo uses this for its local cache: data you've previously accessed from AWS is kept in an encrypted database on-device, so it loads instantly and works offline.

To make sure Dynoxide actually behaves like the real thing, I built a conformance test suite. 526 tests, all validated against real DynamoDB on AWS. Every emulator gets the same suite. Dynoxide passes all 526. DynamoDB Local manages 92%. LocalStack gets 93%. Dynalite hits 81%.

The suite is public. Anyone can run it and verify.

Get started

Homebrew:

brew install nubo-db/tap/dynoxide

npm (drop-in dynalite replacement):

npm install --save-dev dynoxide

Cargo:

cargo install dynoxide-rs

Then just run it:

dynoxide

Full documentation is at dynoxide.dev, the source is on GitHub, and there's more detail on the Dynoxide project page.

Dynoxide is the foundation. But the thing I'm building on top of it is what I really want to talk about next. Nubo is a native DynamoDB client for macOS and iPadOS, and I think people who work with DynamoDB day to day will love it. If that sounds interesting, there's a mailing list at nubo.sinovi.uk.

Dynoxide stands on its own though. If you're running DynamoDB Local in CI and wondering why your pipeline is slow, or you're still using dynalite and need coverage for newer API features like transactions and streams, give it a try. I'd love to know what you think.

Si Novi will be at AWS Summit London on April 22nd. If you're there and want to talk DynamoDB, come and find me.

Automating my son's YouTube with Python and FFmpeg

Martin Hicks — Sat, 21 Mar 2026 00:00:00 GMT

My son Ben is 12. Back in August he decided he wanted to start a YouTube channel. We talked about what it could be and he landed on Premier League score predictions. Every gameweek he'd predict the scores for all 10 matches, record himself talking through his picks, then after the weekend we'd see how he did and record a review.

Simple concept. He was into it. I was into it. Let's go.

The manual era

The first video was recorded in the car on the drive home from France. I'd knocked together a Keynote presentation on my iPad with the fixtures for Gameweek 1. Team names, badges, blank spaces for Ben to write his predictions with Apple Pencil while screen recording. It was rough, but it worked. He recorded his picks, we uploaded it, and Ben's PL Predictions was born.

The workflow went like this: I'd create the Keynote slides manually, Ben would screen record on the iPad, I'd trim the video in iMovie, adjust the audio levels and brightness, then upload to YouTube Studio and fill in the title, description, and tags by hand.

For the review video I'd do the same thing again, but this time with actual results filled in and a little thumbnail showing what he'd predicted, so viewers could see both side by side as he talked through each result.

It was taking me about an hour on a Friday evening or Saturday morning. Not terrible, but I could feel the fragility of it. Ben was being consistent, recording every week without fail, and I didn't want me to be the reason he couldn't get a video out. If I was busy, or forgot, or just didn't fancy spending an hour creating slides and editing video, his streak would break. That didn't sit right.

JSON and Python: the first automation

After a couple of weeks I rebuilt the system. Instead of manually creating Keynote slides, I set up a JSON data model for each gameweek (fixtures, predictions, results, points) and wrote a Python CLI using Click that generates PowerPoint presentations from the data. The slides get copied to iCloud so they appear on the iPad automatically.

{
  "league": "EPL",
  "season": "2025-26",
  "gameweek": 30,
  "matches": [
    {
      "home": "ars", "away": "eve",
      "prediction": {"home": 2, "away": 0},
      "result": {"home": 2, "away": 0},
      "points": 3
    }
  ]
}

The scoring is simple: three points for an exact score, one point for the correct result (home win, away win, or draw), zero for wrong. The CLI generates prediction slides with blank scores for Ben to write on, and review slides with actual results, points, and a thumbnail of his predictions.

This got my weekly time down from an hour to about thirty minutes. Create the JSON with fixtures, run the command, copy to iCloud. After matches, add results, run the review command. Better, but I was still manually typing fixtures, manually watching his video to work out what he'd predicted, manually looking up results, manually editing in iMovie, and manually uploading to YouTube. Five of those steps are things a computer should be doing.

Automating everything else

I've just finished building the next phase, removing every remaining manual step. It's ready to go from the Gameweek 31 review and Gameweek 32 onwards. Here's what the pipeline looks like.

Fixtures from the FPL API

No more manual data entry. The Fantasy Premier League website exposes an undocumented API that the community has been using for years. It provides fixture data per gameweek, free, no authentication required.

$ python cli.py fetch-fixtures -w 31
Fetched 8 fixtures for GW31

The CLI maps FPL's numeric team IDs to our registry, handles postponed matches and writes the gameweek JSON automatically. It's worth noting this is an unofficial API with no formal terms of use for developers. For a personal project making a handful of calls per week, the risk is negligible, but it's not something I'd build a commercial product on.

Video transcription with Whisper

This is the one that surprised me most. After Ben records his prediction video, OpenAI's Whisper runs locally to transcribe the audio, then a regex parser extracts his predicted scores by matching them to fixtures by team name.

$ python cli.py transcribe -w 31
 1 | Bournemouth  | Man United  | 1 - 1  | matched
 2 | Brighton     | Liverpool   | 2 - 1  | matched
 ...
Matched 8/8 fixtures
Save these predictions to the gameweek JSON? [y/N]: y

It handles colloquial names like "Spurs," "Palace," and "Forest", and works out which score belongs to home and away based on context. I run the Whisper medium model with an initial prompt containing all 20 Premier League team names, which dramatically improves recognition accuracy. I tested it against Ben's existing recordings and it matched all predictions correctly: 10/10 for GW30 and 8/8 for GW31.

It asks for confirmation before writing to the JSON, so there's always a human check.

Results from the API

Same FPL endpoint, different data. After the weekend's matches:

$ python cli.py fetch-results -w 31
Updated 8 matches with results
Total points: 11

Merges results into the existing JSON, preserving Ben's predictions, and calculates points automatically.

Encoding publish-ready Shorts

This replaces iMovie entirely. A single command takes the raw iPad screen recording and produces a YouTube-optimised Short:

$ python cli.py encode-short -w 31 -t predict -i RPReplay_Final.MP4

Under the hood it runs a single Whisper pass (reused for both trim detection and SRT caption generation), detects the speech start via FFmpeg's silencedetect filter, finds the closing phrase ("see you next time") in the transcript, then encodes with normalised audio, proportional scaling and YouTube-recommended H.264 settings.

The trim is tight, half a second after the closing phrase, so it cuts before Ben swipes up to stop the screen recording and you don't see the iOS Control Centre.

Upload to YouTube

The final piece. One command uploads the encoded Short as a private video with auto-generated metadata:

$ python cli.py upload-short -w 31 -t predict
Title: GW31 Premier League Predictions #PremierLeague #matchweek31
Tags: Premier League, EPL, GW31, Football...
Thumbnail set
Captions uploaded
Privacy: PRIVATE

It generates the title, description (including season accuracy stats and an engagement question), tags with relevant team names, extracts and sets a thumbnail from the title frame, and uploads the SRT captions for search indexing.

Everything uploads as private. Ben and I review it in YouTube Studio and he publishes when he's happy. This was a deliberate decision. It's a child's channel and I wanted a human gate before anything goes public.

The YouTube API OAuth credentials are pulled from 1Password at runtime via the op CLI, so nothing sensitive sits on disk.

The weekly routine from Gameweek 32

What used to be nine manual steps should now be a handful of CLI commands plus Ben's two recordings:

fetch-fixtures then pptx-predict-table --icloud - pull fixtures, generate slides, send to iPad
Ben records prediction video
transcribe then encode-short then upload-short - process and publish
fetch-results then pptx-review-table --icloud - pull results, generate review slides
Ben records review video
encode-short then upload-short - process and publish
Review and publish manually in YouTube Studio

I'm looking forward to running it for the review once GW31's fixtures are all played, and both predictions and review for real in Gameweek 32 after the international break. If the numbers hold, my active time should go from an hour to under two minutes. And critically, none of those two minutes are things that block Ben. If I'm not around, someone else could run four commands, or I could do it from my phone over SSH. The next step is getting this set up on Ben's Chromebook so he can run the pipeline himself — once I work out why the option to enable Linux is disabled on his machine, he won't need me involved at all.

What I'm proud of

It's not the code. It's that Ben has stuck with it. He's at over a hundred subscribers, he's recorded every single gameweek this season, and he genuinely looks forward to it. He's learning that consistency matters more than virality, that showing up every week builds something, and that over a hundred people choosing to follow a 12-year-old's football predictions is pretty cool.

The automation exists to protect that consistency. Not to make the videos for him. He still records every one himself, writes his predictions by hand with Apple Pencil, and talks through his reasoning. The personality is all him. I just make sure the boring infrastructure never gets in the way.

If you're into Premier League predictions, or you want to support a kid who's been showing up every week, you can find Ben at youtube.com/@BensPLPredictions.

Recently Played: bringing back my Last.fm component

Martin Hicks — Sun, 08 Mar 2026 00:00:00 GMT

Around 2010, my personal website had a Last.fm widget showing what I'd been listening to. It was a small thing - just a few album covers and track names in the sidebar - but it was very me. Back then your personal site was an extension of yourself, and having your music taste ticking away in the corner felt right.

Fast forward fifteen years and I've rebuilt this site from scratch several times over. The Last.fm widget never made the cut again. Not for any good reason - it just fell off the list each time. When I was recently working on the sidebar I remembered it, and thought why not bring it back?

There's something I like about the full circle. The web has gone through its various phases since 2010 - the single page app years, the JavaScript-for-everything years - and come out the other side valuing progressive enhancement and server-rendered HTML again. My site is built with Eleventy and deployed from GitHub Actions to S3 with CloudFront. It felt fitting to bring back a feature from an earlier era, built with the principles I care about now.

So here's how it works.

The architecture

The component is a two-layer system: build-time SSR for the initial HTML, and client-side polling for live updates. It's dropped into the sidebar as a <now-listening> WebC element alongside <my-details>.

Build time:  listening.js → Last.fm API → tracks data → SSR into HTML
Runtime:     Client JS → polls Last.fm API every 60s → diffs → updates DOM
             ├── localStorage cache (2min TTL) for instant loads
             └── pauses when tab hidden, resumes on focus

Data layer: `src/_data/listening.js`

This is an Eleventy global data file that runs at build time. It calls the Last.fm API - specifically user.getrecenttracks - requesting the five most recent tracks for my account.

It has a three-second timeout so that if Last.fm is having a bad day, builds aren't left hanging. On any failure it returns { tracks: [] }, which means builds never break regardless of what the API does.

The raw API response gets mapped into clean objects - name, artist, album, url, art, and a nowPlaying boolean. All text fields are HTML-escaped and URLs are validated (only http: and https: schemes allowed) - this is the server-side XSS protection layer.

The data is then available to templates as listening.tracks via Eleventy's data cascade.

The component: `now-listening.webc`

The WebC component has three distinct parts.

Server-rendered HTML

A webc:type="render" script runs at build time, reading this.$data.listening.tracks. It generates the initial HTML so the page ships with real track data baked in. This means content is visible immediately on load - and for search engines or anyone browsing without JavaScript, this is the component. It's done. No spinner, no empty state.

A `<template>` element

An inert HTML template used by the client-side code for DOM cloning. It defines the track row structure with data-* attribute hooks - data-track, data-name, data-detail, data-art-placeholder, and data-now-playing. Nothing renders from this until JavaScript picks it up.

Client-side polling script

A self-executing IIFE (kept alive with webc:keep to prevent Eleventy from stripping it) that handles the live behaviour:

Polling - hits the Last.fm API every 60 seconds to keep the track list current.

localStorage cache - on page load, if a fresh cache exists (two-minute TTL), it renders from cache immediately and defers the first API call. This avoids the flash-of-stale-content problem on repeat visits or navigating between pages.

Diffing - compares JSON.stringify(tracks) against the last known state. If nothing has changed, the DOM stays untouched. No unnecessary reflows.

Visibility awareness - listens to visibilitychange to stop polling when the tab is hidden and restart when it becomes visible. If you leave the tab in the background for an hour, it's not hammering the API the whole time. Be a good citizen.

AbortController - cancels in-flight fetch requests when polling stops, so there's no risk of stale responses landing after the component has moved on.

Client-side XSS protection - uses the same safeUrl and esc helpers as the server layer. Content is inserted via textContent (inherently safe) and URLs are validated before being set as href or src attributes.

Design decisions

The main tradeoff worth calling out is the duplicated logic. The same API call and data mapping exists in both listening.js (server) and the client-side script. I could abstract it into a shared module, but the server code runs in Node during the Eleventy build and the client code runs in the browser. Keeping them self-contained means each layer is independently understandable and testable. The duplication is small and the mapping is straightforward - it's not the kind of logic that's likely to drift in dangerous ways.

The other thing I'm quietly pleased with is how little the component asks of the outside world. Between the visibility-aware polling, the localStorage cache, and the diff check before touching the DOM, it makes the minimum number of requests necessary. If you're pulling from someone else's API on every page load of your site, I think you owe it to them (and your users) to be thoughtful about it.

Was it worth it?

It took a couple of hours to build and it makes me smile every time I see it on the page. Sometimes the best features aren't the most technically ambitious - they're the ones that make a site feel like yours.

I might extract this into a standalone package at some point - the component is fairly self-contained and would slot into any Eleventy site with minimal config.

If you want to see it in action, it's in the sidebar at martinhicks.dev.

WireGuard for safe browsing on kids' devices

Martin Hicks — Mon, 28 Jul 2025 00:00:00 GMT

After setting up AdGuard Home and Unbound on my home server, DNS filtering was working perfectly for every device on the home network. But children's devices don't stay on the home network. They go to school, to friends' houses, connect to public Wi-Fi, and use mobile data. In all of those situations, the home DNS filtering was being bypassed entirely.

The solution was WireGuard VPN, configured so that children's devices always route their DNS queries back through the home server, regardless of which network they're on.

Why WireGuard

I've used OpenVPN in the past and it works, but WireGuard is a different league:

Fast. Built into the Linux kernel, minimal overhead. You don't notice it's running.
Simple. The entire codebase is around 4,000 lines of code. OpenVPN is over 100,000.
Battery friendly. It only sends packets when there's traffic. No keepalive polling draining the battery.
Reliable. Handles network changes gracefully. Switch from Wi-Fi to mobile data and the tunnel reconnects silently.

The N100's hardware AES-NI support means encryption overhead is essentially zero. WireGuard throughput on this box easily saturates the home broadband connection.

DNS-only vs full tunnel

This is the key design decision. There are two approaches:

Full tunnel routes all traffic through the VPN. Every web request, every app connection, everything goes home first and then out to the internet. This gives you full control but means all traffic takes a round trip through your home broadband, which adds latency and uses upload bandwidth.

DNS-only routing sends only DNS queries through the VPN. The actual browsing traffic goes directly to the internet via whatever network the device is on. But because DNS is resolved at home, AdGuard Home's filtering still applies everywhere.

I went with DNS-only routing for the children's devices. The filtering is the point, not inspecting traffic. This keeps browsing fast while still blocking ads, trackers and inappropriate content on every network.

Server setup

Install WireGuard

sudo apt install wireguard

Generate server keys

wg genkey | tee /etc/wireguard/server_private.key | wg pubkey > /etc/wireguard/server_public.key
chmod 600 /etc/wireguard/server_private.key

Server configuration

/etc/wireguard/wg0.conf:

Note: replace enp1s0 below with your server's actual NIC name. Run ip -br link to find it - modern Debian uses predictable names like enp1s0 or eno1, not eth0.

[Interface]
Address = 10.10.0.1/24
ListenPort = 51820
PrivateKey = <server_private_key>

# NAT for VPN clients
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o enp1s0 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o enp1s0 -j MASQUERADE

# Child's iPad
[Peer]
PublicKey = <client_public_key>
AllowedIPs = 10.10.0.2/32

Enable IP forwarding:

echo "net.ipv4.ip_forward = 1" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

Start and enable the service:

sudo systemctl enable wg-quick@wg0
sudo systemctl start wg-quick@wg0

Port forwarding

You'll need to forward UDP port 51820 on your router to the server's local IP. This is the only port WireGuard needs.

Client configuration

For each child's device, generate a key pair:

wg genkey | tee client_private.key | wg pubkey > client_public.key

The client configuration is where the DNS-only routing magic happens:

[Interface]
PrivateKey = <client_private_key>
Address = 10.10.0.2/24
DNS = 10.10.0.1

[Peer]
PublicKey = <server_public_key>
Endpoint = your-home-ip:51820
AllowedIPs = 10.10.0.0/24
PersistentKeepalive = 25

The PersistentKeepalive = 25 line tells the client to send a packet every 25 seconds. Without it, mobile carrier NATs quietly drop the connection's state after a minute or two of silence and the server can no longer reach the device until the device sends something first.

The critical line is AllowedIPs = 10.10.0.0/24. This tells WireGuard to only route traffic destined for the VPN subnet through the tunnel. Since DNS = 10.10.0.1 points to the server's VPN address, DNS queries go through the tunnel and hit AdGuard Home. Everything else goes direct.

If you wanted a full tunnel instead, you'd set AllowedIPs = 0.0.0.0/0 which routes everything through the VPN.

Getting it onto the devices

The easiest way is to generate a QR code:

sudo apt install qrencode
qrencode -t ansiutf8 < client.conf

Open the WireGuard app on iOS or Android, tap "Add tunnel", scan the QR code, done. The whole process takes about 30 seconds per device.

AdGuard Home integration

For this to work properly, AdGuard Home needs to listen on the WireGuard interface too. In AdGuard Home's settings, make sure it's bound to 0.0.0.0 (all interfaces) or specifically includes 10.10.0.1.

You can also configure AdGuard Home with per-client settings. I have the children's VPN IPs set up with stricter filtering rules (safe search enforced, adult content blocked) while my own devices have a lighter touch.

On-demand VPN on iOS

WireGuard's iOS app supports On-Demand activation per-network. In the app, edit the tunnel, enable On-Demand, and add your home Wi-Fi SSIDs to the Disconnect on demand list. The result: the tunnel comes up automatically anywhere except home, where the LAN already has filtered DNS via the router. iOS sometimes takes a few seconds to bring the tunnel up after a network change - but it's reliable enough that I haven't had to think about it for months.

The result

Whether a device is at home, on school Wi-Fi, at a friend's house, or using mobile data, DNS queries go through the home filtering system. Ads are blocked, trackers are blocked, and inappropriate content is filtered. All without installing any software beyond the WireGuard app, and with no noticeable impact on browsing speed.

The whole stack (N100 server, Debian, AdGuard Home, Unbound, WireGuard) has been running for months now without intervention. It's the kind of infrastructure that just fades into the background and works, which is exactly the point.

This is the final article in my home server series. Previously: building the N100 server and AdGuard Home with Unbound for private DNS.

Running AdGuard Home and Unbound on a home server

Martin Hicks — Sun, 20 Jul 2025 00:00:00 GMT

After building my Intel N100 home server, the next step was improving the network itself. The server was sitting there running 24/7, barely using any resources. Perfect for handling DNS.

My goals were straightforward:

Block ads and tracking across every device on the network
Block malicious and adult content domains
Stop relying on third-party DNS resolvers for privacy
Get better visibility into what's happening on the network

The solution I settled on was AdGuard Home for filtering, combined with Unbound for recursive DNS resolution. Together they give you a private, fast, ad-free DNS setup that you fully control.

Update - May 2026

A few months in, I disabled Unbound. The privacy story is genuinely great, but cold-cache lookups for less popular domains were noticeably slower than I wanted on a household network - enough that family members were noticing. AdGuard Home now forwards directly to a fast public resolver. The walkthrough below still works exactly as written if you'd like to try Unbound yourself; this is just my honest experience after living with the setup for a while.

The architecture

The DNS query flow looks like this:

Device → Router → AdGuard Home → Unbound → Root DNS servers
                   (filter)      (resolve)

Every device on the network sends DNS queries to AdGuard Home (via the router's DHCP settings). AdGuard checks the query against its blocklists. If the domain is allowed, it forwards the query to Unbound, which performs recursive resolution starting from the root DNS servers.

No Google DNS. No Cloudflare. No third party sees your queries.

AdGuard Home

AdGuard Home is a network-wide DNS filtering gateway. Think Pi-hole, but with a more polished interface and built-in support for DNS-over-HTTPS, DNS-over-TLS and DNS-over-QUIC.

Every device that uses the network DNS automatically gets:

Ad blocking across all apps, not just browsers
Tracker blocking for analytics, telemetry and fingerprinting domains
Malware domain blocking via regularly updated threat lists
Adult content filtering (configurable per-client, handy with children in the house)
Query logging with a clean dashboard showing top clients, blocked queries, and trends

Installation

I installed AdGuard Home directly on the host using their official script:

curl -s -S -L https://raw.githubusercontent.com/AdguardTeam/AdGuardHome/master/scripts/install.sh | sh -s -- -v

After installation, the web UI is available on port 3000 for initial setup, then moves to port 80 (or wherever you configure it).

Configuration

The key settings I changed from defaults:

Upstream DNS: Set to 127.0.0.1:5335 (Unbound, see below)
Bootstrap DNS: 9.9.9.9 (Quad9, only used to resolve filter list domains during startup)
Blocking mode: Default (0.0.0.0) which returns a null route for blocked domains
Rate limit: Disabled (it's a home network, not a public resolver)
Query log retention: 7 days

For blocklists, I use:

AdGuard DNS filter (default)
OISD blocklist (comprehensive, well-maintained)
Steven Black's unified hosts
A custom list for a handful of domains I want blocked specifically

Between these lists, around 15-20% of all DNS queries on the network get blocked. That's a lot of ads, trackers and telemetry that never reaches any device.

Unbound

Unbound is a validating, recursive, caching DNS resolver. Rather than forwarding queries to an upstream provider like 8.8.8.8 or 1.1.1.1, Unbound resolves domains itself by walking the DNS hierarchy from the root servers down.

Why bother with recursive resolution?

When you use a third-party resolver, they see every domain you look up. Even with DNS-over-HTTPS, you're trusting that provider with your full browsing history. With Unbound resolving recursively, no single upstream server sees the full picture. The root servers see you asking about .dev, the .dev TLD servers see you asking about martinhicks.dev, and the authoritative nameserver sees the full query. Privacy through distribution.

Installation and configuration

sudo apt install unbound

The main configuration file at /etc/unbound/unbound.conf.d/pi-hole.conf (the name is a leftover from many guides, works fine):

server:
    verbosity: 0

    interface: 127.0.0.1
    port: 5335

    do-ip4: yes
    do-udp: yes
    do-tcp: yes
    do-ip6: no

    prefer-ip6: no

    # Root hints for recursive resolution
    root-hints: "/var/lib/unbound/root.hints"

    # Trust glue only if it is within the server's authority
    harden-glue: yes
    harden-dnssec-stripped: yes

    use-caps-for-id: no

    # Reduce EDNS reassembly buffer size
    edns-buffer-size: 1232

    prefetch: yes

    num-threads: 1

    so-rcvbuf: 1m

    private-address: 192.168.0.0/16
    private-address: 169.254.0.0/16
    private-address: 172.16.0.0/12
    private-address: 10.0.0.0/8
    private-address: fd00::/8
    private-address: fe80::/10

The root hints file needs updating periodically (I have a cron job for this):

sudo wget -O /var/lib/unbound/root.hints https://www.internic.net/domain/named.root

Unbound listens on 127.0.0.1:5335, and AdGuard Home forwards allowed queries to it.

Performance

Running DNS locally is fast. After the initial recursive lookup, Unbound caches the result, so subsequent queries for the same domain return in under a millisecond. Even uncached lookups typically resolve in 20-50ms. In hindsight that's a touch optimistic - the popular stuff is fast, but the long tail of cold lookups is where I felt the latency tax most.

The N100 handles this effortlessly. CPU usage from AdGuard and Unbound combined is negligible, hovering around 1-2%. Memory usage was light when I first measured it, well under 100MB total. AdGuard's footprint grows over time as filter lists get larger, so don't be surprised to see it considerably higher months down the line.

The dashboard in AdGuard Home is genuinely useful too. Being able to see which devices are making the most queries, what's being blocked, and spotting unusual patterns has been eye-opening. You quickly learn just how chatty some devices and apps are.

Router configuration

The final step is telling your router to hand out the server's IP as the DNS server via DHCP. On most routers this is somewhere in the LAN/DHCP settings. Set the primary DNS to your server's local IP and remove any secondary DNS (otherwise devices will fall back to it and bypass filtering).

If your router doesn't support custom DNS settings via DHCP, you can configure devices individually, though that's less convenient.

This is the second article in my home server series. Previously: building the N100 server. Next: using WireGuard so kids' devices stay protected on any network.

Building a tiny Intel N100 home server

Martin Hicks — Thu, 03 Jul 2025 00:00:00 GMT

Over the past few years my home setup has gradually grown more complicated. Between family devices, work machines, backups, development tools and general tinkering, I'd slowly accumulated a mixture of cloud services, routers and older hardware doing different jobs. None of it was joined up, and none of it was particularly efficient.

I wanted a single small home server that could handle a few key tasks:

Network services (DNS filtering, VPN)
Time Machine backups for every Mac in the house
Some lightweight file storage
A place to run a few containers
Something quiet enough to live in the office and cheap enough to run 24/7

After a bit of research I settled on building a tiny Intel N100 based server.

Why the N100

The Intel N100 is a 4-core, 4-thread processor from the Alder Lake-N family. It was designed for low-power, embedded use cases, but it's surprisingly capable. The key specs that sold me:

6W TDP (yes, six watts)
Hardware AES-NI (important for VPN throughput)
Support for DDR4 or DDR5 depending on the board
Passive cooling on most boards

For a home server that runs DNS, VPN, backups and the occasional container, it's more than enough. This isn't a Plex transcoding box or a development build server. It's a quiet, reliable workhorse.

Hardware

The core of the build is an ASRock N100DC-ITX motherboard. This board is powered by a standard laptop-style DC barrel jack, which means no ATX PSU, no fan noise from a power supply, and a much smaller footprint.

The full parts list:

Component	Choice
Motherboard	ASRock N100DC-ITX
Case	SKTC MX01 mini-ITX
RAM	16GB Crucial DDR4 SODIMM
Boot drive	500GB Samsung 980 NVMe SSD
Storage	2TB WD Blue 2.5" HDD
Power	19V external laptop PSU

Total cost was around £250. The whole system is incredibly compact and almost entirely silent. The only moving part is the 2.5" HDD, which only spins up for backups.

Assembly

The build was straightforward, if a little tight due to the mini-ITX case:

Slot in the SODIMM RAM
Install the NVMe SSD (M.2 slot on the underside of the board)
Mount the motherboard on the standoffs
Install the 2.5" HDD in the drive bay
Connect SATA data and power
Route the DC barrel jack through the rear panel
Close up the case

The result is a very tidy little machine that sits quietly on a shelf next to the router. No fans spinning. You'd barely know it was on.

Operating system

For the OS I chose Debian 12 (Bookworm).

I considered a few alternatives:

Option	Why I didn't choose it
TrueNAS	Overkill for my storage needs, heavy
Unraid	Licence cost, more NAS-focused than I need
Proxmox	Great, but adds a virtualisation layer I don't need yet
Ubuntu Server	Would have been fine, but I prefer Debian's stability

Debian felt like the right balance of stability, simplicity and long-term maintainability. The N100 is well supported in the 6.1 kernel. Everything worked out of the box, including the Realtek NIC which can sometimes be problematic.

I'm running everything directly on the host for now, with Docker available for anything that benefits from containerisation. No VMs, no hypervisor. Keep it simple.

What it runs

The server now handles:

AdGuard Home for network-wide DNS filtering
WireGuard VPN so family devices stay protected outside the house
Time Machine backups over SMB for three Macs
Samba file shares for general household storage
Tailscale for low-effort remote access to the server from anywhere
AirConnect for bridging AirPlay to Chromecast and UPnP speakers
Netdata for live system monitoring on a local web dashboard
Fail2ban guarding SSH
Docker is installed for the occasional container, though I'm not running anything in it long-term right now

The DNS and VPN setup in particular has been a real quality-of-life improvement for the whole family. Some of these services get their own dedicated articles in this series; others are smaller pieces I'll possibly cover in shorter follow-ups.

Power usage

One of the key goals was efficiency. This thing runs 24/7, so every watt matters.

Typical measurements with a plug-in power meter:

6-8W idle (SSD only, HDD spun down)
10-12W when the HDD is active (backups running)
8-9W under light CPU load (DNS, VPN traffic)

At roughly 8W average, that's around 70 kWh per year, which costs about £20 in electricity. Compare that to an old desktop repurposed as a server pulling 60-80W idle, and the difference is stark.

What I'd change

If I were doing this build again:

I considered SSD-only, but in hindsight the spinning HDD has been the right call for Time Machine. Backups are network-bound, not disk-bound, and putting that volume of writes onto an SSD would chew through its endurance for no real gain. The original concern about HDD noise turned out to be overblown too - a one-line hdparm -S 120 /dev/sda (run as a tiny systemd unit at boot) puts the drive into standby after ten minutes of idle, and since Time Machine is the only thing using it, I genuinely can't hear it from the same room. I'd keep this same split.
A case with better airflow. The SKTC MX01 is compact and looks decent, but cable management is tight. Something like the Jonsbo N1 would give more room, though it's larger.

Overall though, I'm really happy with this build. It's been running for months now without a single issue, and it's become one of those invisible pieces of infrastructure that just works.

This is the first in a series of articles about my home server setup. Next up: running AdGuard Home and Unbound for private DNS filtering.

Remove trailing slashes with CloudFront and 11ty

Martin Hicks — Tue, 16 May 2023 00:00:00 GMT

Intro

This article will walk through the steps required to create a Cloudfront function to handle redirecting trailing slash URIs to non-trailing slash equivalents on your S3 hosted 11ty website.

Background

Several years ago we published our website at Si Novi using a hand-balled static site generator we built for ourselves, and deployed it to S3 with Cloudfront used for caching and routing our A record from Route 53.

For this old website we wanted to strip trailing slashes on URLs, so https://example.com/articles/some-article instead of https://example.com/articles/some-article/, personal preference I guess.

Anyway, to achieve this we used a Lamda@Edge function to handle the redirects for us - 301 redirecting from the trailing slash URI to the non-trailing slash URI - something we'd long achieved using .htaccess on an Apache server.

We published this function to the AWS Serverless Application Repository.

Fast forward a few years, and we now publish our website using Eleventy - still hosted on S3, still fronted with the Cloudfront CDN, but our long-standing redirect function no longer worked.

With 11ty we were hitting a redirect loop which we believe was due to it's use of subfolder index.html pages - our old hand-balled system created S3 objects like articles/some-article.html whereas Eleventy creates S3 objects like articles/some-article/index.html. The the old system resolved to the object correctly, whereas when using sub-directory within an index.html as 11ty and others do, this caused an infinite redirect loop.

Solution

1. Create a new CloudFront function

function handler(event) {
    var request = event.request;
    var uri = request.uri;
    
    var params = '';
    if(('querystring' in request) && (request.querystring.length > 0)) {
        params = '?'+request.querystring;
    }
    
    if(uri.endsWith('/')) {
        if(uri !== '/') {
            var response = {
                statusCode: 301,
                statusDescription: 'Permanently moved',
                headers:
                { "location": { "value": `${uri.slice(0, -1) + params}` } } // remove trailing slash
            }
    
            return response;    
        }
        
        
    }
    //Check whether the URI is missing a file extension.
    else if (!uri.includes('.')) {
        request.uri += '/index.html';
    }
    
    

    return request;
}

The above code achieves the same trailing slash removal as we had in our old Lambda@Edge function, but also includes an additional check to ensure that index.html is appended to any requests on their way to S3 (only if the request doesn't already include '.', so image/some-image.png will pass-through just fine ).

2. Publish the function

Save and publish your newly created function, in this example I've named it subfolder-index

3. Configure your Cloudfront distribution to route requests through the function

Modify your Cloudfront distribution's behaviour and set the published function to run on Viewer request.

Wrap up

That's it, you should now be seeing URLs redirected to your non-trailing slash URI preference, while still successfully serving the subfolder index.html file (which wont appear in the URL)

One benefit of using a Cloudfront function is it's cheaper to invoke than a Lambda@Edge function.

You can see our old Lambda@Edge function here, which still might be useful.

ps: If you're not hosting subfolder index.html files you can remove the else if from the Cloudfront function.

Enhance CSRF package - now supports multipart form data

Martin Hicks — Mon, 15 May 2023 00:00:00 GMT

I've just published a minor update to my CSRF plugin for Enhance projects.

v0.9.0 now supports multipart/form data, meaning you can easily use the <csrf-form></csrf-form> component for forms that contain file uploads.

usage:

<csrf-form method="post" action="/upload" enctype="multipart/form-data">
  <input type="file" name="file" />
</csrf-form>

outputs:

<form action="/si-novi/christopher-dee/media/upload" method="post" enctype="multipart/form-data">
  <input type="hidden" name="csrf" value="540c460e-946e-4c78-8c8d-63c4cd091ee9"> <!-- auto-generated hidden input with the unique csrf token for this request (use with verifyCsrfToken on your post handler) -->
  <input type="file" name="file">
</form>

Additionally, I've also improved the html generation for the component so you can now include all the following optional attributes and they'll pass-through into your HTML <form> element.

enctype
target
acceptCharset
autocomplete
id
novalidate
rel

Of course you can still use the standaone <csrf-input></csrf-input> if you'd prefer to use a standard form tag.

Get in touch

Any feedback on your usage of this plugin, bugs, or suggestions for improvements please get in touch on GitHub. I'd love to hear from you.

DynamoDB Streams locally with arc.codes & Enhance

Martin Hicks — Mon, 24 Apr 2023 00:00:00 GMT

Introduction

When working on a recent web app project using Enhance, I encountered a requirement for using both transactions and DynamoDB streams, two DynamoDB features that aren't supported by Dynalite - the fast in-memory DynamoDB engine that Architect uses to support local development.

In this article, I'll introduce my new plugin (arc-plugin-sandbox-stream) for working with DynamoDB streams locally within Arc and Enhance projects. This plugin enables developers to use DynamoDB streams locally in their arc or enhance sandbox environment.

Background

Architect got me hooked back in 2018 with its amazing local developer experience, it's a huge DX boon to be able to try ideas out on your own machine without being concerned about provisioning real infrastructure, or being tied to a location where network connectivity is guaranteed.

One of the services Architect spins up locally is Dynalite. It provides a fast in-memory DynamoDB engine. For lots of projects this engine is absolutely ideal, fast to launch, lightweight and not a process hog. But for this particular project I really needed streams and I wasn't prepared to sacrifice the local developer experience. So I set about researching how this could be achieved, if at all. Helpfully it's a question that has cropped up on Arc's discord and the thread of replies helped point me in the right direction.

I'd need to do a few things:

Switch to using DynamoDB Local
Create a middleware to poll the stream for new data
Invoke the corresponding function handler when new data is returned from the stream.

Using the plugin

The plugin kicks in when a @tables-streams pragma is discovered within your arc config, and queries the table meta data / stream meta data to retrieve iterators for each shard of the table's stream.

If data is found the plugin invokes the corresponding lambda function using arc's inbuilt invoke function.

Your function code will receive one or more results off the stream like so (as you'd expect if you've used streams on AWS):

[
  {
    "eventID": "7c1ac231-2c9d-4ba0-b3a2-1de2eb6602c0",
    "eventName": "MODIFY",
    "eventVersion": "1.1",
    "eventSource": "aws:dynamodb",
    "awsRegion": "ddblocal",
    "dynamodb": {
      "ApproximateCreationDateTime": "2023-04-24T12:19:00.000Z",
      "Keys": {
        "sk": {
          "S": "account:si-novi"
        },
        "pk": {
          "S": "account:si-novi"
        }
      },
      "NewImage": {
        "_type": {
          "S": "Organisation"
        },
        "name": {
          "S": "Si Novi"
        },
        "sk": {
          "S": "account:si-novi"
        },
        "created_at": {
          "S": "2023-04-24T12:19:07.409Z"
        },
        "id": {
          "S": "01GYSK850HWHGE36A2VTDG1CPK"
        },
        "pk": {
          "S": "account:si-novi"
        },
        "modified_at": {
          "S": "2023-04-24T12:19:07.409Z"
        },
        "slug": {
          "S": "si-novi"
        }
      },
      "SequenceNumber": "000000000000000000935",
      "SizeBytes": 310,
      "StreamViewType": "NEW_IMAGE"
    }
  }
]

Setup

1. Add the dependency to your project

npm install @hicksy/arc-plugin-sandbox-stream

2. Configure your project to use @tables-streams in .arc file

@tables
example
  pk *String
  sk **String

@tables-streams
example

3. Add the plugin to arc config

@plugins
hicksy/arc-plugin-sandbox-stream

4. Setup DynamoDB Local

There are various ways to use DyanmoDB Local. I opted for the bundled version that comes with AWS's NoSQL Workbench tool. But you can use which ever version you're comfortable with.

Just note that if you use the NoSQL Workbench version there's the following caveats I came accross:

Data storage is persistent in the NoSQL Workbench version - I can't find a way to set the inMemory flag available to the standalone / docker version (you'll need to modify any seed scripts to conditionally insert or drop and re-create the table)
You need to be running NoSQL Workbench while you develop and remember to toggle on DynamoDB Local each time (not a biggie really, but it'd be great to have the means to set it to autostart or run as a start-up background task)
The NoSQL Workbench version runs on port 5500, I can't see a way to change this, but it's not a problem for me.

5. Configure Arc to use DynamoDB Local

Arc environment params

Arc makes it really easy to switch out Dynalite for DynamoDB Local.

There's a couple of env vars you need to set to get you going.

Tell arc that you're using an external db
Change the port used for tables within arc
Set your region to ddblocal

You can do this one of several ways.

Either setting an environment variable ARC_DB_EXTERNAL=true / ARC_TABLES_PORT=5500 in an .env file, or using the prefs.arc file and setting the properties within the @sandbox section see arc's docs for more info

e.g.

.env file

APP_URL=http://localhost:3333
ARC_TABLES_PORT=5500

prefs.arc file

@sandbox
external-db true

Arc data seeds

As DynamoDB Local data is persistent when using the bundled NoSQL Workbench version, you'll need to modify any startup data seeds you have and update them to either conditionally insert data, or what I prefer, to delete the table and re-create it each time - this makes it more in keeping with all my other arc projects.

Arc table create or update

Sadly, as Dynalite doesn't support streams, arc's sandbox doesn't create the table with the required StreamSpecification param, so you'll need to supply that either when re-creating the table using createTable or supplying it via an updateTable call.

An example StreamSpecifcation looks like:

StreamSpecification: {
	StreamEnabled: true,
	StreamViewType: 'NEW_IMAGE' 
},

The project I'm working on currently uses Sensedeep's OneTable. So I can achieve the above like:

const client = new Dynamo({client: new DynamoDBClient(params)})
const table = new Table({
    name: DDB_NAME,
    client: client,
    logger: true,
    schema: AppSchema,
    partial: false
})

if(await table.exists()) {
	await table.deleteTable('DeleteTableForever');
}

await table.createTable({
    StreamSpecification: {
        StreamEnabled: true,
        StreamViewType: 'NEW_IMAGE' 
    },
});

You can also do this using the standard JS dynamodb client for any projects that aren't using an additional tool like onetable.

tip: use @architect/functions to infer the full name for your DynamoDB table's, which will be a combination of the name you've given and it's deployement environment etc

6. Modify the polling interval [optional]

By default the plugin will poll for new stream data every 10 seconds. You can override this by providing an alternate millisecond interval by updating your .arc file

@sandbox-table-streams
polling_interval 1000

Get in touch

Any feedback on your usage of this plugin, bugs, or suggestions for improvements please get in touch on GitHub. I'd love to hear from you.

A mistake the internet won't forget easily

Martin Hicks — Mon, 19 Dec 2022 00:00:00 GMT

tldr: If you visited my site between 2022-12-01 and 2022-12-18 you might not be seeing updated content on previously accessed pages. I've added a Clear-Site-Data header to this page to possibly rectify. If you don't see multiple blogs posts on the homepage after viewing this article, please manually refresh any page you previously visited - thank you 🙏

edit: 2022-12-20 - Following a brief discussion with Jake Archibold he offered the suggestion to use a fetch request with the cache: 'reload' property. The nice thing about this is that it's cross-browser, so even Safari on both Desktop and iOS should re-validate their local cache using this technique. It's still not something you'd want to put on every request - I've added an update below that describes this.

A couple of weeks or so ago, enthused by my move to Mastodon, the growing desire of owning your own content post-Musk, and an upcoming New Year's resolution I'd made with myself to be "less mute" on the internet in 2023 - I decided to re-publish my website and start sharing my thoughts in longer form writing.

Over the weekend (hungover, after some pre-Christmas drinks the night before) I realised I'd made a BIG mistake with the website deployment mechanism - accidentally setting a long-term cache-control header for all resources, not just the immutable ones 🤦🏻‍♂️.

Read on for what what caused the problem, how I've resolved it, the steps I've taken to mitigate it, and what I wish was possible to limit damage following a situation like this in the future.

Automatic deployments

I update this site using a GitHub action; triggering a build of my static site and transferring the web assets over to AWS S3 where I statically host my website.

It's within this S3 sync command that I made the galling mistake...

aws s3 sync ./_site s3://martinhicks.net --cache-control max-age=31536000 --delete --acl=public-read --follow-symlinks

If you're not familiar with the AWS S3 cli - this command transfers the directory /_site to the S3 bucket hosting this website.

As you can see I use a number of CLI params:

--delete: This tells the s3 sync command to delete any items that exist in the destination bucket, but aren't in the local source directory (perfect for clearing up old, no-longer needed objects).
--acl=public-read: Every object uploaded to the bucket has public read enabled - this being a public website, you need every asset to be available. Omitting this would prevent you from accessing the page, despite the bucket itself being public
--follow-symlink: I think this is a default actually, but added it in just to make sure any content symlinked into that source directory was actually transferred
--cache-control: The biggie, the doozy.

Here I inform S3 to set the Cache-Control header - a header S3 will include when returning a request for each individual web asset - a html page, a css file, an image, a sitemap.xml... i.e. everything I've just synced.

In haste I inadvertently set the max-age value to 31536000 - that's ONE YEAR - and I told S3 to do so for all my assets. Therefore for every request, S3 will return the header cache-control: max-age=31536000, which tells your web browser that it's allowed to cache the contents of the page you requested for up to one year.

Why is this a problem - caching is good right?

So firstly, caching is great.

It's definitely the right thing to do. You'll improve the speed in which return visitors access your page(s), and you'll receive a better Page Speed Rank from Google, which is a ranking factor on Google's search results.

Using a CDN is even better - I do that here too with AWS CloudFront storing a cache on the edge, closer to where you're accessing the internet from.

eg: you live in London, someone else from London has already accessed this page, you then visit the same page, and the CDN will serve you from the edge cache - win for you as the page will load much faster, bonus for a webmaster as the origin server won't have to handle the request

Anyway that's an aside.

Here I'm taking about the cache-control: max-age=31536000 header I was inadvertently serving.

The issue with that header is the liberal use of it for all my assets. It's absolutely the correct header to return for anything immutable - images, css etc.

If you visit my home page, you'll see that it has some changeable content - a list of latest blog posts that are updated when I publish a new article. So by setting this long-term cache on all web assets, I'd created a situation where the next time you visit my homepage you'd think I hadn't written any new content - you'd likely move on elsewhere - you most certainly wouldn't think;

I know - I'll hit refresh, just in case Martin has messed up his cache headers.

I had an inkling something wasn't right during development, but given I'm an impatient soul, while my site was being synced with S3, in another tab I would be furiously hitting refresh to see the changes

hitting refresh clears the cache, and hid the problem. It was only when I revisited a page on mobile (iOS Safari), that I realised older content was being served.

The correct way to cache content for a website

If you've read this far, you might be interested in seeing a more sensible mechanism for caching content using the S3 sync command.

I updated my GitHub action to make two separate calls to s3 sync.

//long-lived cache for images and css
aws s3 sync ./_site s3://martinhicks.net --cache-control 'max-age=31536000' --delete --acl=public-read --follow-symlinks --exclude '*' --include 'images/*' --include 'css/*'

//no-cache on everything else
aws s3 sync ./_site s3://martinhicks.net --cache-control 'no-cache' --delete --acl=public-read --follow-symlinks --exclude 'images/*' --exclude 'css/*'

The first command ensures that my immutable content (images and css) are served with a year long max-age. Telling the browser that once you've received an image or a css file for the first time to cache it, and keep it in cache for up-to a year. (i.e. your browser won't keep requesting it on subsequent pages, or reloads, over the internet. It'll serve that image from your local browser cache instead.)

The key props to that are as follows:

--exclude '*': tell s3 sync to exclude every item in the source _/site directory (nothing that isn't explicitly included will be transferred to S3)
--include 'images/*' & --include 'css/*: tell s3 sync to include anything within images and the css directory

With the second pass of S3 sync, we invert that - excluding the images and css directories. This run will upload every web asset that isn't in the two excluded directories while setting a cache-control header of no-cache (i.e. don't locally cache the HTML page in browser, re-request it from the CDN or origin as appropriate for every request)

Fixing the problem for returning visitors

Sadly, this isn't a problem that's easily fixed.

First off, you need to hope anyone who accessed page content with a long cache header, will revisit your site to read something new you've written - this is the only way to set a new header after all, as the old content, say the index page, will have already been stored long-term in the returning-visitor's browsers cache.

After studying MDN I found the Clear-Site-Data header, which is exactly what I need.

As MDN states:

The Clear-Site-Data header clears browsing data (cookies, storage, cache) associated with the requesting website. It allows web developers to have more control over the data stored by a client browser for their origins.

Great. So this article is being served with the following header.

Clear-Site-Data: "cache". Which I've acheived by creating a response headers policy on CloudFront that includes this header in the response for this specific path.

Which I sincerely hope means your browser has now been zapped by a Neuralyzer and forgotten any cached data it stored. It's good for all browsers except for Safari (desktop and iOS), and Firefox (although that's behind a feature flag, so hopefully wider roll-out will be in an iminent version).

~(If you're on iOS, maybe you could humour me by giving the homepage and journal page a refresh?)~

Is there a better solution?

I'm currently only applying the Clear-Site-Data header to this page and really hoping / relying on some of the people that read my earlier articles might stumble upon this one? Which is probably unrealistic...

It feels like I shouldn't apply this header on every new page I publish from this point forward - wouldn't that mean the browser will never cache any data? I guess I could include the header for a limited period on all html pages, but it still doesn't feel quite right.

What I really wish is that there was a more deterministic header I could apply, say something like:

Clear-Site-Data: 'cache' '2022-12-01:2022-12-18'

Wouldn't it be handy to be able to say "clear this site cache if the item was stored between the following dates"?

That way I'd be able to drop this header onto all future assets, and eventually anyone returning to my site will have the rogue cached pages cleared - without affecting the behaviour of anything cached before or after the incident date.

Disclaimer: I have no idea how the internals of a browser work. Maybe this is nonsense as the item in cache might just store its expires-at date, rather than the date it was accessed and originally stored?

To be honest, I'm probably over analysing the problem. There's been very little inter-page navigation here - most people have followed a link to a specific article, read it and then moved on. But it's still bugging me...

I'd be keen to hear from anyone who has any better ideas. Is there a more appropriate solution I could apply?

You can contact me at https://indieweb.social/@martinhicks

Update: 20th Dec 2022:

I outlined my issue on the Fediverse - and tagged in Jake Archibald and Alex Russell to see if they had any ideas - I wasn't expecting a response tbh, so was blown away that they took time out to offer their thoughts.

Alex rightly pointed out that Apple lagging behind on this spec (and many more, let's be clear), and the general issues holding up improving the web, are largely down to the lack of browser competition on iOS - something the Open Web Advocacy (OWA) group are looking to change. I completely agree and support this cause, and I'm going to reach out on OWA's Discord to see how I go about helping. If you care about the web, then why don't you too?

Jake offered the following suggestion - fetch(brokenUrl, { cache: 'reload' })

I tested this out on a random test URL and set the long max-time on its cache-control response header. I then hit a second page (after changing the initial test page's cache-control to no-cache), which included the suggested fetch call to the now stuck test page.

After hitting the page with this fetch command, the rogue stuck resource was re-requested by Safari, and when I returned to the original long-cached page it had it's new cache-control: no-cache applied.

It's perfect for what I need - a cross-browser solution, albeit with an additional couple of network requests as you need to specify individual stuck resources - there's no site-wide mechanism like with clear-site-data. But these quick tests proved that you can clear the cache of a previously visited page, from a second independent page (on the same domain).

This technique uses the Request.cache api to tell the browser to re-request the URL specified without using any existing local cache. And to re-populate its cache based on the rules of the new cache-contol header.

Like with clear-site-data you wouldn't want to apply this to all your pages, but if you can get your returning-vistors to view some new content* it's a great solution.

* perhaps a special article such as this, or if you've built a web-app you could email users who you know had accessed your service between a specific date period, and send them to a brand new landing page to peform the cleanse.

There's also an experimental spec proposal only-if-cached on this API too. Meaning in the future, we could get even more specific and only perform the cache reload if the problematic stuck resource is a) in cache for this specific visitor, and b) has a date earlier than the incident.

Anyway, for now.. I've added the following script to this page, and this page only:

<script type="module">
  
  async function clearRogueCache(url) {
      return await fetch(url, {
          "cache": "reload"
      });
  }

  await clearRogueCache("https://martinhicks.net");
  await clearRogueCache("https://martinhicks.net/articles");

</script>

There we go. Still hoping this article can catch any of the early readers of my site. But also hopeful that if you're reading this way in the future, and you've made a similar mistake, you might have more options at your disposal - lucky you.

Thanks again Jake and Alex - really appreciated.

Eleventy 2.0 & WebC

Martin Hicks — Sat, 17 Dec 2022 00:00:00 GMT

I recently rebuilt this website and my company website - sinovi.uk - using Eleventy 2.0 and their new WebC language for templating.

It's really good. And well worth checking out.

In this post I look at my experiences trying out Eleventy 2.0 and its new Web Component language, WebC.

Note

At the time of writing 11ty 2.0 is pre-release.

It’s just hit version canary-20 and judging by 11ty's recent toot, it’s official release is very close.

Background

I’d heard of the Eleventy project in passing via newsletters and Twitter posts for several years, but up until a few months ago I hadn’t tried to build anything with it.

I think I’d mentally stored it away as similar in vain to Gatsby or some similar react tool - ...there's a lot of those, right?

How wrong was I?

Static site generation

Back in 2018, at Si Novi, we rolled our own internal tool for static site generation, that I’ve now learnt felt pretty similar to 11ty in some respects, but lacked its finesse and feature set.

Our templates were HTML and we used a companion JSON file (ie index.html & index.json) to hook data into each page view using HTML attributes.

We had collections in an array within a standalone json file (eg articles.json). We even hooked it up to PHP blade templates for one test project.

There was a key thing wrong with it though - it was a bloody pain to use.

At the time we didn’t know about the front matter syntax, which in hindsight might’ve helped.

But the key issue was our node CLI processing engine was written using a jquery compatible dom lib - Cheerio, and a load of Grunt hooks.

It was fiddly; hard to maintain and lacked features, and we never quite found the time to make improvements alongside our client work.

Furthermore, we were having to stuff lots of properties into the JSON files to handle conditional templating logic particularly for reusing partial views / re-usable template snippets.

Maybe if we’d known about frontmatter, had chosen a templating engine like nunjucks, and had known about parse5, I’d still be maintaining it now - who knows?

What I do know (now) is that 11ty absolutely nails static site generation.

They have multiple templating languages, nested layouts, a sensible config and plugin system, and a really cool data cascade which provides lots of options for populating a page’s templating data or mutating a particular value prior to generating the page.

The docs site is veeeeerrrry comprehensive. To be honest… so big I found it overwhelming to begin with (it really melted my head for a bit), but it's a fantastic resource and a credit to the community of contributors.

My first look

I took my first spin of Eleventy when I heard about Enhance in September and noticed one of their deployment targets was 11ty.

Building out a few demo pages, I really liked what I saw in both projects but struggled with a few bits as I tried to understand how these worked together.

Attempting to learn two new things at once meant I was doing a disservice to both tools; hampering my understanding of each project, and limiting my discovery of any overlap / boundaries of concern when using them together.

I gave up.

But promised myself I’d go deeper into each tool individually when time allowed.

Learning Eleventy

A month or so later I started a fresh starter Eleventy project.

Making some really simple pages and layouts, I learnt about 11ty’s special dirs (like _includes & _data), and generally started to feel more comfortable with how Eleventy worked and it’s limitations.

Those limitations, for me, were the fact that sharing blocks of re-usable components felt difficult.

You could use 11ty’s shortcodes to create snippets, or nunjuck macros, but these felt to me like reusing strings of templates was just about ok, but making them configurable using properties or attributes wasn’t a great experience for me.

Having first used 11ty with Enhance, I’d been drawn to Enhance’s use of Web Components as reusable elements - it fit my more recent mental model of using components in htm or react.

Maybe I gave up too soon in my earlier experiments?

But then I heard about Webc - I decided to persevere with my original plan, and set about re-building the Si Novi and martinhicks.dev sites .

Webc - now we’re rocking

Webc is a brand new 11ty templating language.

It uses Web Components and requires Eleventy 2.0 (at time of writing this is pre-release and requires installing their canary package).

Using Webc with 11ty is seamless; You can use WebC to build an individual component, for re-use and you can use it for layouts. Basically anywhere you’d expect an 11ty template to work, .webc files work.

Meaning it works on its own, or can be used within their other templating languages too. And being 11ty you can mix and match templating languages throughout a project.

I like this 11ty feature a lot - there’s loads of useful 11ty code snippets on gist, on their website, and, well all over the internet written in Nunjucks or Liquid and choosing to use WebC doesn’t prohibit you from tapping into this rich seam.

For example I’m using an njk page to create my sitemap.xml and another to create my RSS feed.xml.


//sitemap.njk

---
permalink: /sitemap.xml
eleventyExcludeFromCollections: true
---
<?xml version="1.0" encoding="utf-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    {% for page in collections.all %}
        <url>
            <loc>{{ site.url }}{{ page.url | url | replace(r/\/$/, "") }}</loc>
            <lastmod>{{ page.date.toISOString() }}</lastmod>
        </url>
    {% endfor %}
</urlset>


//rss.njk
---json
{
  "permalink": "feed.xml",
  "eleventyExcludeFromCollections": true,
  "metadata": {
    "title": "Martin Hicks - Journal",
    "subtitle": "Martin Hicks is a software developer from Manchester, UK",
    "language": "en",
    "url": "https://martinhicks.dev/",
    "author": {
      "name": "Martin Hikcs",
      "email": "hello@martinhicks.net"
    }
  }
}
---
<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xml:base="{{ metadata.url }}" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>{{ metadata.title }}</title>
    <link>{{ metadata.url }}</link>
    <atom:link href="{{ permalink | absoluteUrl(metadata.url) }}" rel="self" type="application/rss+xml" />
    <description>{{ metadata.subtitle }}</description>
    <language>{{ metadata.language }}</language>
    {%- for post in collections.articles | reverse %}
    {%- set absolutePostUrl = post.url | absoluteUrl(metadata.url) %}
    <item>
      <title>{{ post.data.title }}</title>
      <link>{{ absolutePostUrl }}</link>
      <description>{{ post.data.description | htmlToAbsoluteUrls(absolutePostUrl) }}</description>
      <pubDate>{{ post.date | dateToRfc822 }}</pubDate>
      <dc:creator>{{ metadata.author.name }}</dc:creator>
      <guid>{{ absolutePostUrl }}</guid>
    </item>
    {%- endfor %}
  </channel>
</rss>

I also really like the fact I can just use WebC as a templating language for HTML generation.

This website currently has zero need for any client-side progressive enhancement, so I don’t need any of the web component features in the HTML served to the browser.

11ty allows you to build a component that just returns HTML, and if so, the generator treats that component output as just the inner HTML omitting the enclosing custom element tag.

If you did want to keep the component wrapper for some reason you pass an attribute of webc:keep.

//my-avatar.webc
//example html only web component
<picture>
    <source srcset="/images/5F8AB69C-FA08-4C25-B932-74D76EBB7721.webp" type="image/webp">
    <img src="/images/5F8AB69C-FA08-4C25-B932-74D76EBB7721.jpg" alt="Me and my wife, Helen, lying on the grass in summer. " :width="this.width" :height="this.height" class="max-w-[100px] md:max-w-[200px] mx-auto aspect-square ring-2 ring-zinc-500/40 rotate-45 rounded-full bg-zinc-100 object-cover"/>
</picture>

Webc with 2.0 means I could now build proper re-usable components, configurable with attribute props if required. No more snippets or Nunjucks macros.

<my-avatar width="200" height="200"></my-avatar>
<my-avatar width="100" height="100"></my-avatar>

Perfect.

Other wins are;

Using web components within the head element - this isn’t allowed by the Web Component spec I don't think, but given you may have a Webc component that just provides HTML you can use the webc:is attribute to upgrade a standard element to a WebC component just for templating purposes (but only if it just returns html)

Eg:

<script webc:is="json-ld" ></script>

The above script tag in my head, will be ran using my component json-ld, which basically adds some dynamic json-ld for articles on this site, and excludes if it's not an article page.

WebC components can include render only Js functions to iterate collections


---
meta:
title: "Articles"
description: "Occasional thoughts"
pagination:
  data: collections.articles
  size: 10
  alias: articles
  reverse: true
layout: layouts/main.webc
---


<container>
  <div class="flex flex-col mx-auto justify-center ">
    <h1 class="text-4xl font-bold tracking-tight text-zinc-800  sm:text-5xl mt-4">
      Journal
    </h1>

    <div class="grid grid-cols-1 gap-8 md:grid-cols-2 auto-cols-auto md:auto-rows-[1fr] mb-8">
      <script webc:type="render" webc:is="template">
        function () {
          //console.log(this.pagination)
          let articles = this.pagination.items;

          return articles.map((article, idx) => /*html*/`
                    
            <div class="prose relative pb-8">
                    <a href="${article.data.url}">
                        <picture class="flex w-full " >
                            <source srcset="${article.data.image.webp}" type="image/webp">
                            <img class=" full-width mb-2" src="${article.data.image.path}" width="345" height="236" alt="${article.data.image.alt}">
                        </picture> 
                    </a>
                    <span class="text-sm!">
                    ${article.data.date}
                    </span>
                    <h1 class="text-xl! mb-2 font-semibold"><a class=" no-underline" href="${article.data.url}">${article.data.title}</a></h1>
                    <p>${article.data.description}</p>
                    <a class="absolute bottom-2 " href="${article.data.url}">
                        Read the article
                    </a>
                </div>
                
                    `)
          .join("");
        }
      </script>
    </div>

    <hr>

  </div>

  <my-details mode="full"></my-details>
</container>

Slots

If you’ve used any Web Component tool or manually created your own, you’ll know that web components use ‘slots’ to control where nested elements or strings are displayed within the component template.

They’re super useful and help direct content to the correct placeholder without using attributes or similar.

eg:

//social-link.webc

<a class="group -m-1 p-1" :href="href" target="_blank" :aria-label="this.arialabel" :role="this.role" :rel="this.rel">
    <div class="flex items-center space-x-2">
        <slot name="icon"></slot>
        <slot name="content"></slot>
    </div>
</a>

Which is usable like:

<social-link rel="me" role="listitem" href="https://indieweb.social/@martinhicks">
    <icon-mastodon slot="icon" class="w-8 h-8"></icon-mastodon>
    <span slot="content">Follow on Mastodon</span>
</social-link>

nb icon-mastodon is another webc component - completely nest-able as you'd expect

Things to look out for in 2.0 / webc

Having not been a long time user of 11ty, I’ll leave any deep comparison between the two versions to seasoned experts.

There’s loads of new features in 11ty 2.0, some of which are breaking changes.

Their docs site does a good job of signposting these changes, and I’m sure when it’s released there will be loads written to guide users in migration.

What I’ve found:

Webc: Script and link tags

Since, I think, "@11ty/eleventy": "2.0.0-canary.18" or "@11ty/eleventy-plugin-webc": "0.8.0", you’ve been required to add webc:keep to any script or link tag that has an external src.

Thankfully the CLI warns you of these during build, throwing an error.

However, I found that several non external script tags in my head (two json-ld and the local google analytics gtag script) weren’t included in my published site for a week.

Adding webc:keep brought them back.

Maybe I misread the CLI warning, but I don’t think I did. And certainly the lack of them on an external src caused a build fail, whereas omitting them for locally src’d elements didn’t.

Luckily I don’t care much about either of those on my personal site, so no biggie.

**Given that I’m using a canary build and I haven’t been studiously keeping up with the change notes between pre-release build versions I’ll take the blame on this one.**

11ty 2.0 - The copy command doesn’t actually copy files locally during dev

This is new and intentional, for performance reasons, it sort of magically symlinks them internally during the local serve process.

So as far as the browser is concerned the images folder you’ve set to copy to the output dir, for example, and therefore is served from /images/myimage.jpg, hasn’t physically been copied to that location on your machine. That only happens during a production build.

Fine when you know but it’s a little confusing at first.

I’ve had to build a few production builds locally at times just to make sure my copy configs are set correctly.

Plug-ins have a whole bunch of new hooks

This is a big upgrade and I think will make integrating other tools way easier.

I think it's backwards compatible. I’m looking forward to playing around with this more.

WebC components aren’t automatically discoverable within a project by default.

I found this confusing.

Especially as most of my .webc components were simply returning HTML, I didn’t want to have to add a load of webc:import attributes per component.

Thankfully 11ty’s config system has you covered.

Adding the following to your .eleventy.js file, and placing all your components within /_includes/components makes them usable throughout the entire project without individually importing them.

eleventyConfig.addPlugin(pluginWebc, {
    // Glob to find no-import global components
		components: "src/_includes/components/**/*.webc",
});

Everything in project root by default

I think this is right, and not just a mistake I made. But the default 11ty starter had everything configured to run from the project root.

Eg, index.webc was in the same root folder as package.json and other non web assets.

I didn’t like this.

Again, 11ty config to the rescue, it’s super simple to tell 11ty where your input and output dirs should be. So it was quick get things how I wanted them.

return {
  dir: {
      input: "src",
      output: "_site"
    }
  }

It’d be great if the starter cli command could ask you where you’d like your source and output directories to be, and auto configure this for newbies like me.

Tailwind

Tailwind works really well with a component based system, so 11ty and webc is no different, outputting only the css for classes that you’ve actually used in your pages.

To make that work better, I pointed tailwind config to the output folder (_site in my case) to look for content, rather than the src directory, and made it run after 11ty prod build.

//tailwind.config.js
module.exports = {
  content: ['./_site/**/*.html'],
  plugins: [require('@tailwindcss/typography')]
}

This way any draft components I’ve built but not yet used, won’t have any of its unique classes included in the final production css output. Until they’re actually included in one of the html pages.

I also needed a way to allow component modification throughout the site (more so on Si Novi). To do that I used the Tailwind Merge package and created a helper function within .eleventy.js that can be used on each component.

eleventyConfig.addFilter("tailwindMerge", function(defaultClasses, overrideClasses) { 
  return twMerge(defaultClasses, overrideClasses)
});

This means I can pass in overriding css attributes at the point of using the component and the twMerge function (as it’s tailwind aware), replaces the defaults with the overrides.

Within a WebC component:

//link-primary.webc
<a :href="href" :class="tailwindMerge('underline hover:text-blue-500', this.class)"><slot></slot></a>

If I used this component, like so...

<link-primary class="hover:text-red-500">example link hovers red</link-primary>

... the hover on that instance of the component would be red.

Wrap up

I’m so glad I’ve found 11ty, it’s a great tool. Version 2.0 and WebC has made it sticky for me.

I haven’t used any of the more advanced webc features such as scoped css or bundling, but I’m sure I’ll test them out in due course.

Absolutely great job.

Keep an eye out for the next post in this series, which explains how to set up 11ty, AWS and GitHub actions to create a CI/CD build pipeline to deploy to S3.

You can view the source code of my site here.

Sharing Enhance elements between projects

Martin Hicks — Thu, 15 Dec 2022 00:00:00 GMT

This post introduces a new arc plugin for Enhance projects that I've recently published, with the catchy name shared-enhance-components-plugin.

Read on for details about how to use the plugin and why I came to write it.

Background

While I continued my exploration of the Enhance framework, I started to wonder how you'd go about importing elements from a shared library or UI toolkit.

In a React project, you'd import the library into the given component and then begin using its JSX tag within the render.

However, Enhance elements are dependency free by design, for good reasons.

So instead, any element you create within /app/elements or define within /app/elements.mjs (* more on this later), becomes automatically available in any Enhance page.

That doesn't mean you can't use dependencies where appropriate. Let's say you have a couple of elements, one of which can be used standalone, the other you always want a particular element included, you could do the following:

//app/elements/csrf-input

export default function CsrfInput({ html, state }) {
    const { attrs={}, store={} } = state
    const { name = 'csrf' } = attrs
    return html`
        <input type="hidden" name="${name}" value="${store.csrf_token}" />
    `
}

import CsrfInput from "./csrf-input.mjs";

export default function CsrfForm({ html, state }) {
    const { attrs={} } = state
    const { action = '', method = '' } = attrs

    return html`
    <form action="${action}" method="${method}">
        ${CsrfInput({html, state})}
        <slot></slot>
    </form>
`
}

You can simply call the pure function to return its "render", e.g. ${CsrfInput({html, state})}.

And I think looking at my browser's source, it means the dependent element is just the HTML entity, rather than being expanded into a web component. Which in this case is what I wanted, all be it by accident.

Pretty cool.

But what about if you want to bring in a shared UI library maintained by a different team or 3rd party?

After a search on the Enhance Discord (which is great by the way - definitely head there for help / guidance / cool things people are building), I couldn't find anything baked in.

I did see a comment from Macdonst (one of the begin/enhance team), who was explaining how you could do all of your imports in an /app/elements.mjs file, and then they would be available throughout your app just like a first-party element you've created in /app/elements - nice.

He went on...

The above is begging to be a plug-in.

So I thought why not.

What is this elements.mjs file anyway?

It's not mentioned much in their docs at all. In fact, at the time of writing, it's only referenced within the Deploy to Fastify section.

However, if you look at the code for the arc-plugin-enhance (which is the plugin that orchestrates arc to have a catch-all lambda for each /app/api/ route, among many other things), you'll see that 4 locations are checked for elements to enhance:

let pathToModule = join(basePath, 'elements.mjs')
let pathToPages = join(basePath, 'pages')
let pathToElements = join(basePath, 'elements')
let pathToHead = join(basePath, 'head.mjs')

i.e.

/app/elements.mjs
/app/pages (any mjs file in here)
/app/elements (any mjs file in here)
/app/head.mjs

The elements.mjs file is basically returning a keyed object mapping the tag name (e.g my-component) to a corresponding element function (e.g. MyComponent) (which can be located anywhere, like node_modules/some-package/elements/MyComponent.mjs)

import MyComponent from 'some-package/elements/MyComponent.mjs'

let elements = {

  'my-component': MyComponent
}

export default elements

So that's a pretty powerful in-built mechanism for linking things together right there.

The need for a potential plugin is that you may npm install a package with 10s, 100s of components you'd like to use, so having a plugin do most of the leg work is helpful.

Also, defining a map between the tag name and pure function doesn't seem to have any draw back - I'm fairly certain elements are only processed if they're referenced within a page and / or element - so mapping a lot of potentially unused components seems to have no issues

Building the plugin

Enhance (the way I'm using it), is wrapped with Architect. Meaning you can easily deploy your enhance app to AWS, or to Begin.

Because it uses Architect's sandbox to run locally, and its hydration mechanism to hydrate your Lambda function code before deployment, you can also tap into Arc's powerful plug-in system.

I'd previously had a go at an Architect plugin and got a little lost. Since then (18 months or so ago), they've massively expanded their plug-in docs and improved the available lifecycle hooks you can tap into.

I also had a good look at some of the plugins listed in their repository, which helped me figure things out.

After an afternoon or so's work I had an acceptable (just about) version of a plugin which works as follows:

You add a component package or UI library to the .arc file under the plugin's unique @ pragma - @shared-enhance-components-plugin - this informs the plugin which external packages to import component elements from.
On arc hydration (which happens during sandbox start, and pre arc deploy), the plugin analyses the available functions that can be imported from the given package:
- either from a specific folder, in which case each .mjs / js file will be mapped into elements.mjs
- or from an index.js file, in which case each named export will be mapped into elements.mjs
Tag names are inferred by de-camelcasing the function name
Any existing elements you've manually added to elements.mjs will be preserved (auto generated lines include an eol comment)

Example usage

I think it's pretty easy to use.

First install the plugin and tie it into your project's arc file.

npm install @hicksy/shared-enhance-components-plugin

//.arc

@app
myproj

@plugins
enhance/arc-plugin-enhance
hicksy/shared-enhance-components-plugin

nb: you have to drop the preceding @ on scoped package names in the .arc file so they don't collide with the pragma names

Install some shared elements

For example, we could install these example Enhance form elements - https://github.com/enhance-dev/form-elements.git

npm install git+https://github.com/enhance-dev/form-elements.git

Tell the plugin to use this package

//.arc

@app
myproj

@plugins
enhance/arc-plugin-enhance
hicksy/shared-enhance-components-plugin

@shared-enhance-components-plugin
hicksy/enhance-csrf 'elements'
enhance/form-elements

Note in the above example the package we've just installed is referenced just by it's name (dropping the @ again to avoid collision with arc pragmas) - we know by looking at this package all of it's components are named exports from a route index.js file.

For @hicksy/enhance-csrf package, we pass a second arg to the plugin, this tells the plugin that the components are individual files within the specific folder.

Start

npm start

The plugin-hook will fire, and the /app/elements.mjs file will be auto-populated with the tag-name's and import declarations as required.

// /app/elements.mjs

import CsrfInput from '@hicksy/enhance-csrf/elements/csrf-input.mjs' //automatically inserted by shared-enhance-components-plugin
import CsrfForm from '@hicksy/enhance-csrf/elements/csrf-form.mjs' //automatically inserted by shared-enhance-components-plugin
import { CheckBox } from '@enhance/form-elements/index.js' //automatically inserted by shared-enhance-components-plugin
import { FieldSet } from '@enhance/form-elements/index.js' //automatically inserted by shared-enhance-components-plugin
import { FormElement } from '@enhance/form-elements/index.js' //automatically inserted by shared-enhance-components-plugin
import { LinkElement } from '@enhance/form-elements/index.js' //automatically inserted by shared-enhance-components-plugin
import { PageContainer } from '@enhance/form-elements/index.js' //automatically inserted by shared-enhance-components-plugin
import { SubmitButton } from '@enhance/form-elements/index.js' //automatically inserted by shared-enhance-components-plugin
import { TextInput } from '@enhance/form-elements/index.js' //automatically inserted by shared-enhance-components-plugin

let elements = {

  'csrf-input': CsrfInput, //automatically inserted by shared-enhance-components-plugin
  'csrf-form': CsrfForm, //automatically inserted by shared-enhance-components-plugin
  'check-box': CheckBox, //automatically inserted by shared-enhance-components-plugin
  'field-set': FieldSet, //automatically inserted by shared-enhance-components-plugin
  'form-element': FormElement, //automatically inserted by shared-enhance-components-plugin
  'link-element': LinkElement, //automatically inserted by shared-enhance-components-plugin
  'page-container': PageContainer, //automatically inserted by shared-enhance-components-plugin
  'submit-button': SubmitButton, //automatically inserted by shared-enhance-components-plugin
  'text-input': TextInput //automatically inserted by shared-enhance-components-plugin
}

export default elements

Publishing to NPM

I've never published a package to NPM before so this was all new to me.

Previous node packages I've created have all been private and we'd opted to npm installing from the git repo address rather than purchasing a private packages account from npm.

Publishing was really straight forward (aside from accidentally deploying it unscoped, and then deciding I'd prefer to release it scoped to my username).

I followed this guide to npm publishing - thanks Zell!

And there you go...

A small plugin that hopefully eases the use of shared Enhance elements and hopefully making it more compelling for authors to share libraries of their elements for others to use?

There's no doubt a few wrinkles will be discovered, and the plugin code could do with a tidy, but I'm pretty happy with it.

Get in touch on GitHub if there's any issues, improvements or feature ideas.

Back to the Future

Martin Hicks — Mon, 12 Dec 2022 00:00:00 GMT

As 2022 draws to a close I wanted to reflect on a welcome change in the web framework world that shifts the needle back towards web-first principles. Which I'm all here for.

This year I’ve worked on a large web application using Remix, and the past week I’ve finally had free time to properly explore Enhance - something I've been super excited about since it was announced back in late Summer 2022.

Both frameworks share core principles;

web-platform aligned mechanism for storing state (i.e forms, sessions and http verbs - those foundational web principles that somehow had been forgotten)
progressive enhancement; a web app can (and should) be usable without browser JS
HTML served over-the-wire
file based routing

The fact as an industry we allowed ourselves to sleep-walk into a period where these principles weren't the norm is unbelievable.

Thankfully a long overdue course correction is taking place.

Remix

I really enjoyed building with Remix - it's well thought out, and makes working with React, well... more enjoyable.

We were able to scaffold out a broad approach to the app pretty seamlessly, and the build out was relatively pain free. We deployed the app to AWS using what Remix call their Grunge stack - which basically pairs Remix with Architect (a tool I'm already very familiar with) and provides a catch-all Lambda to power the server-side code.

I believe Remix can be deployed anywhere that supports the Node.js runtime, and I think it even runs at the edge on web-workers - which is pretty cool.

It's amazing how much cruft Remix helps take away from a typical React project- no more nonsense client side Redux stores, optimistic UI is extremely easy (no loading spinners), and with the file based routing it's far simpler to move things around as you're building out an app.

It's not without complexity, it's still React, right? You have to be pretty disciplined to not over engineer things, particularly the functional component, as you can get drawn to using (or overusing in my case) hooks etc to maintain state client-side rather than using the platform to best effect. (Aware this is my issue not necessarily Remix's).

A lot to like and definitely looking forward to using it again in 2023.

Enhance

For those new to it, Enhance is a standards based web framework underpinned by Web Components from the folks at Begin.

So far I've only had a chance to test out some very basic examples, but really like what I've seen, both in terms of playing around with it and the team's core mission.

Our mission is to enable anyone to build multi-page dynamic web apps while staying as close to the platform as possible.

YES!! 🙌

I'll write up another post soon I'm sure, but here's a quick summary of wins:

1. No need for client-side code

Components (or Elements as enhance refers to them), can be just HTML if you like, there's no need for any client-side JS at all.

export default function ProseHeader({ html, state }) {
  const { attrs } = state
  const { title = '' } = attrs

  return html`
    <h1><slot></slot></h1>
  `
}

This Element would be usable in any Enhance page like so:

<prose-header>Article title</prose-header>

And to the browser, would be served up expanded ready for enhancement if required:

<prose-header>
<h1>Article title</h1>
</prose-header>

One thing I wish were possible (and it may well be, I just haven't looked hard enough), is whether you could opt-out of the web component parent element all together for elements that don't require client-side enhancement i.e. they only contain HTML.

This is a feature I quite like in 11ty's webc - meaning if you're using webc as only a templating language, the outputted HTML is just pure HTML.

Like I say, I'm not sure if this is possible or not. Nor is it much of a pain point to be honest, just a nicety.

2. Progressive enhancement

Easy to progressively enhance client-side capabilities by adding a script into the string returned from the Element.

...rest of Element
<script type="module">
  class ProseHeader extends HTMLElement {
    constructor() {
      super()
      this.heading = this.querySelector('h1')
    }
    
    ...whatever powers you want to give the new Web Component
  }

  customElements.define('prose-header', ProseHeader)
</script>
...etc

3. Routing

File based routing, with an API mechanism backed in.

For example, let's say you have the following page /app/pages/index.html, and the following api route /app/api/index.mjs the returned JSON from either the get handler, or post handler of the API will be made available to any custom elements within index.html automatically.

The server-side mechanism automatically routes to the get handler if the request method is GET and likewise to the post handler if the request is a POST.

This is similar in nature to Remix's actions (post) and loaders (gets), but split across different files.

4. Automatic state

Server side state is populated into every custom element for a given route automatically

I think this is really powerful. No need to drill props, no need to maintain some sort of client-side state, if it's returned in the json prop from either a get or post handler it will be available in state.store in every element on the current page.

What a joy.

Summary

Overall two very interesting frameworks that I'm delighted exist.

They've both, for different reasons, helped me get excited about web development again and travelling back to the future.