It's a lot of work to manually syntax highlight everything, and it makes my HTML harder to read and maintain:

<code>q + s + r</code>
→
<code><span class="q">q</span> + <span class="s">s</span>
 + <span class="r">r</span> = 0</code>

Why not use a syntax highlighting library? These libraries take text as input and produce HTML as output. But on these interactive pages, the input is HTML. I have manually marked up the code with embedded interactive elements that respond to the reader's choices:

The syntax highlighting libraries I looked at don't work on HTML out of the box. It's possible to merge two highlighters together but it can be tricky. Let's see what the merging looks like. To keep the examples concise, I'll use <b> for tags I added manually and <i> for tags the highlighter added:

1. highlighting tags <i> enclosing the manual markup <b>:

   function <b>abc</b>_to_hex(p):
   →
   <i>function</i> <i><b>abc</b>_to_hex</i>(p):
   <i>        </i> <i><b>   </b>       </i>
                      [--------]
                   [----------------------]
   

2. manual markup <b> enclosing the highlighting tags <i>:

   <b>hex.q + 1</b>
   →
   <b>hex.<i>q</i> + 1</b>
   <b>    <i> </i>    </b>
          [------]
   [---------------------]
   

3. manual markup and highlighting tags crossing:

   var y_<b>prev = 3</b>
   →
   <i>var</i> <i>y_<b>prev</i> = 0</b>
   <i>   </i> <i>  <b>    </i>    </b>
              [--------------]
                   [-----------------]
   

   Oops, that's not right. Tags must be nested correctly. The </i> can't be until after the </b>. We have to split the <b> tag here:

   <i>var</i> <i>y_<b>prev</b></i><b> = 0</b>
   <i>   </i> <i>  <b>    </b></i><b>    </b>
                   [---------]    [---------]
              [------------------]
   

   Note that splitting it will break hover and some other interactions.

On the A* page I wrote a highlighter that takes HTML input and produces HTML output. It works for the samples on the A* page, but doesn't work on code in general. It's simple but fragile.

The code samples on the A* page aren't interactive. Interactivity complicates things. Actions on the page can change the text of the code being displayed. The simplest thing is to highlight it again when the code changes:

In an interactive system, the DOM has more than just the HTML: it can have event handlers, focus status, hover state, text selection, pointer capture, and other things. All of that is lost when we generate HTML from scratch. This type of problem is partly solved with a Virtual DOM library (React/Vue/etc.) that attempts to reuse existing DOM nodes where possible.

I wasn't sure how best to solve this problem so I decided to study the modules of an existing syntax highlighting library:

1. Tokenize: find the ranges in the text that need to be highlighted.
2. Markup: surround those ranges with markup such as <span class=keyword>.
3. Style: use CSS to apply colors to span.keyword.

I wanted to replace step 2 of an existing library with the new CSS Custom Highlight API, available in mid-2025. This is a non-destructive approach to highlighting, similar to how selecting text applies styling to the selected text without modifying the HTML. However, the Custom Highlight API also requires new CSS rules, so I have to replace step 3. And it turns out the only part I could reuse, step 1, is tiny (under 30 lines). So I decided to write everything myself:

1. Convert the DOM element to plain text using el.textContent. Don't make the same mistake I did of using el.innerText. That is almost right but has some differences. For example, <br> has text content "" and inner text "\n". That leads to off-by-one errors in a later step.
2. Tokenize the plain text, the same way a syntax highlighter library would. I looked at libraries such as rse/tokenizr and no-context/moo but they are overkill for my needs, around 10✕ as much code as I ended up with.
3. Construct Range objects for each token. The tokenizer gives me an index into the plain text but I need to convert that into an index into the specific DOM node. I can walk the tree with document.createTreeWalker() until I find the correct node.
4. Create a Highlight object to hold all the ranges for each token type (comment, keyword, etc.). Add the ranges to that highlight object.
5. Write CSS rules ::highlight(keyword) for each token type. There are limitations on the styling. These rules can't cause a reflow, so I can change foreground/background but not bold or any other change in fonts.
6. Use MutationObserver to watch for changes to the DOM element. When the text changes, create new Range objects to replace the existing ones. One limitation of this is that I'm watching the existing <pre> elements, not watching for new or removed <pre> elements. I made sure the page doesn't create any after initial load.

This worked really well! I can apply highlighting without messing up any of the interactivity.

On the Hexagons guide I decided to primarily use highlighting for non-syntax (variable names) rather than syntax (keywords, comments, etc.). The diagrams use three colors for the three axes of a hexagonal grid, and I use the same colors for the variables representing those axes:

The code highlighting on the Hexagons guide is now live, with 829 tokens highlighted. Try it out! If you want to see my code, it's code-highlighting.js.

Part 1: de-optimizing

For a long time I've been removing as many spaces as I can from the final HTML, to make the page smaller and faster to load. But it makes the pages hard to read and debug. When I write this:

  <ol>
    <li><i>Wilma</i> <b>Fred</b></li>
    <li>and          Dino</li>
  </ol>

my transform rules turn it into:

<ol><li><i>Wilma</i> <b>Fred</b></li><li>and Dino</li></ol>

In this example it's ok to remove the spaces between </li><li> but it's not ok to remove the spaces between </i> <b>. And it's often ok to collapse multiple spaces into one, but not inside <pre> or <script>. It's tricky. I looked through my XSLT and found that I had kept adding more rules over the years:

1. Remove a bunch of spaces.
2. Add a bunch of spaces to undo some of step 1.
3. Remove more spaces to undo some of step 2.
4. Add a bunch of spaces to undo some of step 1 and 3.
5. Remove more spaces to undo some of step 2 and 4.

What a mess! But each time I had added a rule, I hadn't been looking at the whole process. I was fixing one situation. If I did that again, I could add rule 6 to add more spaces to partially undo a previous step.

Why am I doing this at all?

When I started my web site in 1994, compression wasn't common, so removing 1000 spaces from the input would make the page load 3 seconds faster on a 2400 baud modem. The spaces mattered a lot to me. But in 2025? It's much less important, both because of compression and because internet speeds are much higher now.

These rules are tricky and I was still getting it wrong sometimes. I've written before about de-optimizing to make something easier to work with. I decided to de-optimize here by removing these whitespace rules. Not only will make it easier for me to debug my pages, it also makes it easier for anyone using View Source to see how I wrote the page.

Since browsers collapse multiple spaces in HTML into one, I expected only minor visual differences. An extra space here or there is hard to spot by quickly skimming a page. White space changes are subtle and hard to catch with manual inspection. I decided the best approach would be to break up the change into small steps, and take screenshots before/after each step.

I'm not the only one who wants to test the effect of a CSS change across an entire site. I used Playwright, as recommended in Florian Marending's blog. It can take screenshots using the three major rendering engines (Chrome's Blink, Firefox's Gecko, and Safari's WebKit). Instead of using their output format, I wrote my own, using pixelmatch to create a diff image:

for f in /tmp/yes/*firefox*.png
do 
    node_modules/pixelmatch/bin/pixelmatch $f /tmp/no/$(basename $f) /tmp/diff-$(basename $f) 0.1 || \
        echo "<div style='display:grid;grid-template-columns:repeat(3,1fr)'><img src=diff-$(basename $f)><img src=yes/$(basename $f)><img src=no/$(basename $f)></div>" \
             >>/tmp/diff.html
done

Usually a one space difference will cause the rest of the line to shift, showing up as a red bar. In this change both outputs could be considered correct:

In this change the whitespace rule had mangled the output in a way that I hadn't noticed:

I found no examples where the whitespace rules made things better. I found plenty of examples where they made things slightly worse, and a few that made things much worse, like this:

After testing my site locally, I made the change and pushed to production.

Part 2: re-optimizing

I had mentioned the whitespace experiments on social media and got a suggestion to switch from Gzip to Brotli. I knew Brotli existed but I hadn't looked into it. And wow, it is great! I ran Paul Calvano's compression tester to get a sense of how it would perform on my site. Here's the compression ratios for the hexagons guide:

The dynamic version makes the web server compress each time the browser requests the page. It has to balance compression time and compression ratio. The static version compresses ahead of time, and can use a slower algorithm with a better compression ratio. I also looked at Zstd. Caddy supports Zstd dynamic and static, and Brotli static only. Nginx has third party modules for Zstd dynamic and static, and Brotli dynamic and static.

Switching from Gzip dynamic to Brotli dynamic is a far bigger improvement than the small amount I saved on whitespace. I may try switching to Brotli static at some point, but I need to restructure my build step a bit.

So I now have a smaller faster web site with better formatting, and also learned a tool that will help me test future global changes. It was a good week!

I first looked through everything I had written. I already had started an "overview" page but hadn't gotten very far on it. I also have 22 separate pages that were "diary style", about what I did rather than what you should know.

The overview page covered how to use various SDF font libraries (msdfgen, stb_truetype, tiny-sdf, etc.). I wrote code for multiple libraries, had sketched out diagrams for various concepts, and had screenshots of outputs from each of those libraries.

At some point I realized the scope was too large. I had spent the most time with msdfgen and hadn't yet learned enough about the other libraries to write a proper guide. They all worked differently. I kept getting stuck. So I reduced the scope. In redesign 2 I decided to only use msdfgen, but show the various tradeoffs involved (atlas size, antialias width, shader derivatives, smoothing function).

I made several diagrams for concepts, such as:

And I started running tests. I wanted to compare the effect of atlas size, so I made lots of screenshots and started looking closely. I wanted to come up with a way to recommend a specific size. I wanted to make recommendations for all the other parameters. I showed all the commands I ran.

At some point I realized I could run tests forever. And I had already done that last year, and wrote it up in blog posts (one and two). Doing it again here didn't seem especially valuable. So I pivoted to a "how to" page. In redesign 3 I decided to show the concepts, then a JavaScript implementation using CPU rendering, and then another implementation using GPU rendering. I made new versions of the diagrams:

I was making progress on that page but it didn't feel like a Red Blob Games page. The page started out with tons of shell commands, and then showed lots of code. It felt like a page that only I would find useful. So I started over and designed a "concepts" page. In redesign 4 I focused on what effects I wanted, how SDF works, and how to use it to create those effects. I again reduced the scope by removing the implementation details. What I had already written, I moved to a separate (unpolished) page. And I never wrote a standalone downloadable project like I originally wanted.

Sometimes it takes a long time before I figure out what I actually want to write, and then everything falls into place:

I'm finally happy with the page. Take a look! I hope search engines point to it eventually.

I use both nginx and Caddy for my web sites.

In nginx I can put a permanent rewrite inside the server block:

server {
    …
    rewrite ^(.*)[\;:,.)]$ $1 permanent;
}

In Caddy the syntax is different:

https://www.redblobgames.com {
    …
    @trailing_punctuation path_regexp url ^(.*?)[;:.,\)]$
    redir @trailing_punctuation {re.url.1} 301
}

Now it will 301-redirect "https://www.redblobgames/)" to "https://www.redblobgames/".

For both of these web servers, I never remember what characters need to be escaped, and I kind of wish I could use existing programming language syntax with delimiters like /^(.*)[;:,.)]$/ or r'^(.*)[;:,.)]$'.

Interpreters/Compilers

The main theme for this year was text. I wanted to be very broad in applying this theme: interpreters/compilers, procedural generation, rendering, language models, text editors, search engines, and anything else related to text. Last year I bought and started reading munificent's excellent book Crafting Interpreters, but it wasn't until this year that I spent a lot of time on it. It creates a new language, Lox, and then teaches you how to write an interpreter and compiler for it.

• I implemented the first part, an interpreter. The code samples in the book are in Java, but I used Python instead. When I use the same language as the book/tutorial, I am more likely to copy code without fully understanding it. By using a different language, it forces me to understand the code so that I can reimplement it.
• I read but didn't implement the second part, a compiler. I instead used what I learned with Web Assembly (wasm). I hand-compiled some simple code from Lox to wasm, but didn't tackle the harder topics (closures and function calls). I also compiled some Javascript to wasm, and tried AssemblyScript. I used tools that compile wasm to C and wasm to x86 to understand better what the opcodes do. I wrote partial notes for myself.
• I've made LL(1) and LALR parsers before, and also an LL(1) parser generator which is now a Debian/Ubuntu package. I've long been interested in Pratt Parsers, and Crafting Interpreters finally got me to try it. Pratt Parsers turn out to be really cool for the "expression" part of a grammar. Also see Oil Shell's links to Pratt Parsing tutorials and matklad's blog post. After implementing the lexer and parser, I compiled to wasm, loaded the wasm module at run time, and ran the code.

I also read lots more about interpreters and compilers. I made notes and bookmarks. It's an endlessly fascinating area to study.

Text

This year's theme was text, and although intepreters/compilers were a large part of that, I also did other text-related projects.

• I had explored Signed Distance Field (SDF) font rendering in 2024 but this year I wanted to write a tutorial with a complete working example. Part of the motivation for this is that my blog posts are already showing up in search engines, and those blog posts are "what I tried" and not "what I recommend". Knowing that my pages are showing up in the Google search results makes me a bit embarassed by the page it's returning, and I want to make a better one. I had hoped to finish this in 2025 but it's still incomplete.
• I wrote a very simple search engine for my site, and wrote about it: Part 1 and Part 2. You can try it here.
• It seems like everyone's talking about "AI", especially Large Language Models (LLMs). Some people have very strong positive or negative feelings about them. I've had fairly negative feelings about them for the past ten years and am exhausted. As part of the text theme for this year, I decided I'd look into them a bit more.
  • I wrote a blog post about creative things we might be able to do with them if we could disassemble the chat interface into ingredients and then reassemble the ingredients into new uses.
  • I wrote a blog post about the wrong information ChatGPT writes about Red Blob Games, and how I might be able to harness that.
  • I tried using them occasionally for "side quests" like cropping images for my business cards, coming up with example code I can study, or reformatting/rewriting my code in a different style, but I just haven't put enough effort to learn how to use them effectively for anything other than simple things.
• I made a text editor tool that lets me quickly tweak the SVG of a diagram. I used CodeMirror with a plugin and "attached" it to SVG diagrams on my page. I can then press Alt while dragging the mouse to tweak a number, and see the results. Once I have something I like, I can paste the updated SVG back into the source page. The goal here was that I want to preserve the formatting and comments from the SVG I had hand written, and only make minimal edits with the interactive tool.
• I started watching for and fixing my text editing annoyances while using Emacs. As a side effect, I've been also watching for and fixing annoyances with Linux, Mac, shell, and other tools I use regularly. My computing environment is getting better!

I wanted to work on procedural name generation for maps, which would've connected my text projects to my map projects, but didn't get around to it. That's ok. I always have more ideas than time.

Maps

Maps are such a rabbit hole for me that I tell myself not to work on map projects unless someone other than me wants me to work on maps. It turns out this year I had two companies approach me about map work, so I gave myself permission to work on my other map projects too.

• I had originally written mapgen2 to have the generation algorithms only, and I had the UI "demo" as a separate repository that I hadn't shared. The idea was that you could use mapgen2 to generate a map and then attach it to your own game, which already has its own UI and rendering. But this year I decided to merge the two. It's easier to start with a complete repository that works and remove some files you don't need than to start with an incomplete repository and try to figure out how to make it work.
• I rewrote mapgen4's renderer from using regl.js to using WebGL directly. Regl is nice but there were WebGL 2 features I wanted to use. WebGL 2 has been widely available since 2022, and there are no plans to make Regl support it.
• I reimplemented mapgen4's river renderer. I had been drawing river segments on the CPU, putting those segments into a texture, and then using that render on the GPU. The new renderer is a shader that runs on the GPU.
• Since I had just gone through the mapgen4 rendering code, I had lots of ideas for improvements, and I implemented many of them, and fixed some bugs along the way.
• I started writing a guide to procedural river generation. Some of my tutorials on terrain generation don't include the rivers because rivers are tricky. I know how to make them work in mapgen2 and mapgen4 but I don't know how to generate them in general. But I am only halfway through this.
• I updated my voronoi map tutorial to include biomes and rudimentary rivers.
• I de-optimized parts of Mapgen4 to make it easier to experiment with.
• I modified mapgen2's renderer to work with mapgen4.
• I experimented with towns, roads, and trade routes.
• I played with noise functions.
• I tried increasing the paintability of mapgen4, allowing more elevations as well as a choice between plateaus and mountain peaks.
• Realm of the Mad God used my map generator for their 13 maps, but didn't keep track of the seeds. I have long been curious if I could find the seeds from the map. I searched map seeds to match existing maps and found half of them.
• I tried a stylized hill/mountain image.
• I made a biome painter on top of mapgen1.
• I added more to my jittered grid page but I think it's time to move it onto its own URL outside of /x/. I usually use /x/ for short-lived projects, and this page is slowly turning into a reference page.

It was good to play with mapgen4 again, and I started getting ideas of what might go into a mapgen5.

Site Management

This is the second year that I'm using my own home grown blogging software. I like it a lot better than what I was using before. But I didn't blog as much this year as last year. Part of that was that I was spending a lot of time on my own projects (particularly learning interpreters/compilers) but also because I spent some time working with Jetbolt Games on Galactic Assault Squad and another game studio on an unannounced project.

For the blog but also the rest of the site I've been wanting to add tags like "#hexagons" along with browsing by topic. I have hundreds of pages so it's going to take a while. I've made progress on this but want to label more of my site before I add browsing by topic.

My site is older than most any site out there, and most sites don't last that long. I have around 5000 links from my site to others. A lot of the links I have on my site have broken over the years. I made some good progress on fixing broken links but there's still more to do. It's tricky because some links report as good (status 200) but are actually broken in the sense that the content I linked to is gone, replaced by some generic marketing or spam. And some links report as bad (404) but are actually good, with the server reporting 404 to my crawler but 200 to a regular browser. So there's a lot of manual checking involved.

I continue to update my existing pages. For example this year I improved the conversions and spiral coordinates sections of the Hexagons Guide. I've also updated my pages using Flash applets to use Ruffle, and it's been great to make those experiments available again.

I significantly sped up my web site build. It had been fast initially but as I've added more and more to the site (now 26,079 individual files) it's gotten slower to process. I've been rewriting parts of it from using Bash to Python. If I need further optimizations I may use Rust or C++ in places. I've also switched my Python projects to use uv, which I'm enjoying very much.

I simplified my web site build by removing the Sass compiler. CSS has added the features I had been using from Sass. The build step is now faster, and the pages are slightly smaller.

Next year

I've been thinking about how to choose projects, and I ask myself:

1. Is it that I want it done?
2. Is it that I want to be doing it?
3. Is it that I want to have done it?

There are often times I want to have done something, but when I sit down to do it, I don't actually want to do it. That's ok. And sometimes the problem is that I don't know how to do it. That's ok too. But I need to get better at understanding what's stopping me, and not beating myself up about it. Here are some projects ideas for this year:

• A specialized text transformation language, using what I learned about interpreters/compilers this year. I'd like to design something specific to make example code for my Hexagon guide. I currently use Haxe macros to generate hexagon code in C++, Python, C#, Haxe, Java, JavaScript, TypeScript, Lua, and Rust.
• My guide to SDF font rendering.
• My guide to procedural river generation.
• Version 2 of my A* page. I have some ideas of how to improve this page. Last time I worked on it, I focusd on the diagrams, but the next round should be focusing on better explanations.
• Attempt 3 of explaining coordinate transforms. I've tried this twice before but didn't like the results. I have some ideas of how to approach it differently this time.
• Using SQLite as an ECS (Entity Component System). I've long thought of ECS as a rediscovery of the principles of relational databases. It might be interesting to use a relational database directly instead of an ECS as an intermediary.
• A "first pass" at all the topics ChatGPT falsely thinks I have written. I could spend my time complaining about its hallucinations, or I could instead consider it a list of ideas.
• A procedural name generator. I want to try something more interesting than Markov chains, and have lots of ideas.

Whichever ones I end up picking, I think they will be a lot of fun. Happy 2026 everyone!

A few years ago I had switched variables and calculations from Sass to CSS, but I still needed Sass for nesting. Since CSS nesting has now been supported for two years, I decided it should be safe for me to use it. And that means I no longer need Sass.

I divided the CSS into eight files that I can simply concatenate together (using cat), then esbuild to minify the result:

cat input1.css input2.css … \
  | esbuild --loader=css --minify --bundle --external:* \
            --outfile=output1.css

Since the web server uses gzip content encoding, does it really matter if I minify? Yes, it does, and the help from minify seems to be independent of the help from gzip:

The main goal of this change was to reduce dependencies. But it looks like I still have a dependency on esbuild instead of on sassc. How is that better?

I have been maintaining my site for over 30 years. It’s older than Wikipedia or Google. Most software doesn’t last that long, and I don’t expect sassc or esbuild to last as long as my web site. When I'm choosing tools I ask myself "what happens when this software disappears?"

There’s a difference between the sassc risk and the esbuild risk. I’m using sassc to translate Sass to CSS. If it disappears (it's already deprecated), I have to manually translate it, or find an alternative. I can't update the site without that software. I’m using esbuild only to minify. If it disappears, I lose an optimization but I don't have to do more work. I can continue updating the site without it. So in terms of dependencies I'm happier with esbuild than sassc.

I generally want to prioritize changes that make the site better for readers. This change does not help in the short term. But years from now, when I want to fix a typo or add information to the site, it will be easier for me to update if I have fewer dependencies to update. Everything is about tradeoffs. Not using a useful tool can make things harder, but using a useful tool can make things harder in a different way.

After some more work, I found more matches:

world	rotmg from realmeye	seed	mapgen2 from my page
1
2		perlin 20927-1
3		perlin 84542-1
4
5		perlin 30997-1
6
7		perlin 65166-1
8		perlin 7785-1
9		perlin 43671-1
10
11		perlin 59103-1
12		blob, many
13		radial, many

They don't match exactly, but they are close. I wrote up the process and results. I had allocated 4 hours for this project but ended up spending 9.

Let's start with a screenshot comparison I showed last time:

The differences are slight — there are blue blotches on the ground when I use GL_LINEAR filtering. But why?

Elevation is slightly too big for an 8 bit color channel, so I encode the elevation in the red and green channels (8 bits each), [fract(elevation), floor(elevation)/256.0]. For example an elevation of 32.5 would be encoded as [32.5%1.0, 32.0/256.0], or [0.5, 0.125]. In the fragment shader I decode it with with green*256.0 + red, in this case 0.125*256.0 + 0.5 = 32.5.

GL_LINEAR texture filtering will blend each channel independently, then combine the channels into one value:

(a.hi + b.hi) / 2.0 + ((a.lo + b.lo) / 2.0) / 256.0

But that produces a different value than correctly combining the channels first and then blending:

((a.hi + a.lo / 256.0) + (b.hi + b.lo / 256.0)) / 2.0

Near sea level, some of the below-sea-level elevation must be getting blended incorrectly into the above-sea-level ground. So I can't use GL_LINEAR filtering with this encoding.

If I could store a single 16-bit value instead of two 8-bit values, GL_LINEAR would work correctly. And I wouldn't have to split the elevation into two channels, and I wouldn't have to combine them together again. It'd be simpler code and it would look better.

Or so I thought. Here's the comparison:

It looks the same, right? Well, no, it's slightly different, especially in the mountains:

The R16F format makes the mountains smoother! It also makes the valleys smoother but that difference is harder to see.

The outputs should have been the same. Why aren't they?

It turns out mapgen4 had a bug in it. I didn't actually encode the elevation the way I described above. I encoded with vec4(fract(256.0*e), e, 0, 1) and decoded with dot(color.xy, vec2(1.0/256.0, 1.0)). That doesn't preserve the value. I should have used vec4(fract(256.0*e), floor(256.0*e)/256.0, 0, 1). I used ObservableHQ Plot to plot the difference in these two calculations. The black dots are the incorrect encoding, and we can see that half of them are a little bit off:

This bug adds a "texture" to the sides of mountains. I like the old look better. But don't want to keep the bug. I would prefer adding a texture intentionally.

After I got R16F working, I decided to compare GL_NEAREST to GL_LINEAR. I couldn't use GL_LINEAR because it interpolated each channel separately, but I can use it now that I have only one channel:

It looks almost the same. Looking closer, the coastlines look slightly smoother:

So that wasn't as much of a win as I thought it would be.

To compensate for the mountains looking smooth, I added a slider mountain_folds that adds geometry to make the sides of mountains look more interesting:

I enjoyed the graphics rewrite. I hadn't looked at the graphics code in a long time, and I found lots of places for improvement. I found and understood a bug that I had long suspected. All of this could have been done without a rewrite, but the rewrite got me to actually do it.

To simplify, I switched to using bezier curves:

I was planning to use the input and output width as the indices into the table, but ended up using only one width. What I didn't realize at the time was that I no longer needed a 2D table; I could've used a 1D table. And even better, I could've drawn the curve in a shader, like I did in these shader experiments. But I was focused on shipping and not doing things the cleanest or most efficient way.

After I rewrote the main renderer, I decided to rewrite the river renderer too. I adapted one of the earlier shader experiments to make a river shader, and compared before/after:

It's a bit hard to tell at that zoom level, but if you open the two images in separate tabs you can flip back and forth to compare. Here's a closer view:

The rivers look sharper. But some of the flaws that were hidden before by the blurriness are now visible.

To fix those flaws, I wanted to use a shader to draw wide bezier curves inside a triangle. I didn't have a good intuition for bezier curves with barycentric coordinates, so I had an LLM generate a quick & dirty tool to help me build my intuition. It's like shadertoy but for one triangle. This was an attempt at learning how to "vibe code", and I have mixed feelings about it, but that's a story for another time.

Playing with the toy made me see that there should be a way to do what I want, but it's not from bezier curves like I originally thought. Instead, I can curve fit two degree-5 polynomials, one for each side of the river. Using the LLM-generated code helped me figure this out, but it wasn't able to directly give me a solution. I'll have to work out the details myself at some point.

I added "better river shader" to my list, but at a low priority, so I'm going to work on some other things next and come back to it.

I made a list. One of the projects on the list is rewriting the renderer. But why?

1. Migrate to WebGL2. I started mapgen4 in 2017. WebGL2 wasn't widely available until 2021. I'd like to use it both for features and performance. I'm using a WebGL1 library, Regl.js, which does not support WebGL2.
2. Reduce load time. Regl.js almost doubles the JS size of mapgen4. I would be able to shrink mapgen4 significantly by switching away from it.

Rewriting the renderer was not something I was looking forward to. It's work that has to be done in "one shot", where the intermediate steps aren't testable. I decided to use an LLM to help me. I rarely use LLMs for coding and learned a lot.

1. In this project I used the LLM for translating existing working code from Regl.js to WebGL1. This felt different from having it write new code.
2. The resulting code had some subtle bugs. I was able to track some of them down by comparing the output of the original mapgen4 renderer with the rewritten renderer. Having a previous working version was important.
3. It got me out of my analysis paralysis. Having something that runs rather than a blank editor window was a huge win.
4. I didn't like the structure of the code it generated. It triggered XKCD 386 for me. I ended up replacing all of its code, bit by bit, testing after each change.

Along the way I re-evaluated some of how the renderer worked. For example, I'm using nearest neighbor filtering in many places, even though linear might be better.

Look at how much smoother the ocean colors are with linear filtering! But there are blue splotches near rivers. I decided to keep nearest neighbor filtering for now.

I also discovered some bugs, like this one where one edge of the map is jagged, but only noticeable when the camera is rotated and zoomed in:

Overall, I don't think the LLM saved me any time. I didn't end up keeping the code it gave me. But it got me unstuck, and that meant I actually made progress.

The main benefit of the rewrite was output size:

But the source code did get bigger, from 600 lines to 750 lines (to avoid 9500 lines of Regl.js).

The secondary benefits are that I got to revisit some of the decisions I made, I found some bugs, and I got unstuck. I didn't stop at the main renderer. I also rewrote the river renderer, which saved another 8666 bytes. That's for the next blog post.

If I ask what does red blob games have to say about autotiles, ChatGPT will tell me:

I couldn’t find any evidence that Red Blob Games (Amit Patel) has written anything specifically about autotiles (in the sense of tilesets that automatically pick tile variants based on neighbors). Their site covers many things like grids, map generation, procedural terrain, pathfinding, etc., but I didn’t locate a page dedicated to autotiling.

But if I ask tell me about redblobgames tutorial on wave function collapse, ChatGPT will tell me:

The Wave Function Collapse (WFC) tutorial by Red Blob Games is an in-depth guide to understanding and implementing the WFC algorithm for procedural content generation. It's highly regarded for breaking down complex concepts in a very approachable way.

I don't actually have a tutorial on Wave Function Collapse, but ChatGPT will give me the url https://www.redblobgames.com/articles/wfc/.

This is a reminder that you shouldn't believe everything an LLM tells you! But lots of people have already written about that. I want to explore something different. Instead of treating the hallucinations as something to discard, I want to treat them as something to harness.

I looked through my error logs to find other non-existent URLs that ChatGPT gave out. I also looked at what OpenAI's crawlers found but they weren't as useful. I looked at the top ten:

• /sliding-puzzle/: this is a classical AI topic that could fit into game-ai
• /articles/minimax/: this is also a classical AI topic that could fit into game-ai
• /grids/isometric/: this is a topic I'd like to write about
• /maps/noise/introduction.html: this is the wrong folder for /articles/noise/introduction.html
• /pathfinding/a-star/visualization.html: this is an existing folder but a non-existing page
• /grids/hex-grids/: this is the wrong version of /grids/hexagons/
• /articles/procgen/: this is a page I could've written, to point to other procgen topics
• /grids/mazes/: this is a reasonable guess of what I might have written about
• /articles/mazes/: same
• /maps/noise/: wrong folder

A subset of these are interesting to me — topics that I could write about. I went deeper into the list of 404s and collected these hallucinated URLs:

/x/2048/hilbert.html
/pathfinding/a-star/maze.html
/pathfinding/a-star/visualization.html
/x/1725-aabb-rectangle-overlap/
/x/1842-agent-based-model/
/x/1845-ai-car/
/grids/algorithms/
/x/1813-alpha-beta/
/pathfinding/ants/ /x/1844-ants/
/x/1841-atmosphere-simulation/
/articles/autotiles/
/x/1614-balance/
/x/1840-barycentric-coordinates/ /x/1842-barycentric-coordinates/
/x/1819-battleship-probability/
/x/1723-battleship-strategy/
/x/1846-battleship/
/x/1911-bits/
/x/1910-bitwise/
/x/1743-boids/ /x/1847-boids/ /x/2020-boids/
/x/1729-bresenham/
/maps/catan/ /x/1830-catan/
/maps/cave-generation/
/x/1845-cell-automata/
/x/2020-cells/
/articles/cellular-automata/ /grids/cellular-automata/ /x/1840-cellular-automata/ /x/1841-cellular-automata/ /x/1843-cellular-automata/ /x/2020-cellular-automata/ /x/2023-cellular-automata/ /x/2302-cellular-automata/
/grids/circle-circle-collision/
/x/1728-circle-drawing/
/x/1721-circle-packing/ /x/1830-circle-packing/ /x/1844-circle-packing/ /x/1945-circle-packing/
/articles/cities/
/x/1948-citizenship/
/maps/city-building/
/x/1840-city-generator/ /x/1841-city-generator/ /x/2023-city-generator/
/x/1842-citygen/ /x/1843-citygen/
/articles/collision-detection/
/x/1841-combat-model/
/x/compression/brotli.html
/x/compression/string.html
/articles/connected-tiles/
/articles/convex-hull/
/articles/conways-game-of-life/ /x/1721-conways-game-of-life/
/x/2020/cookie-clicker-save-editor/
/x/2020-cookie-clicker-save-viewer/
/grids/coordinates/ /maps/coordinates/
/x/2020-creature-trainer/
/x/1728-curve-editor/ /x/1729-curve-editor/
/articles/2d-geometry/
/articles/3d-octrees/
/pathfinding/d-star/introduction.html
/articles/3d-voxel-traversal/
/grids/3d/
/articles/data-oriented-design/
/x/1724-delaunay-triangulation/
/x/1722-delaunay/delaunay/
/x/1845-dialogue-trees/
/articles/dialogue/
/x/2018-dice-calculator/
/x/1846-digging/
/x/dither/
/x/1844-dots-and-crosses/
/articles/dungeon-generation/ /maps/dungeon-generation/
/articles/procedural-generation/dungeon/
/x/dungeon/dungeon.html
/x/2020/easing-graphs/
/x/1842-economy-sim/
/x/1843-ecosystem-simulation/
/x/1843-emacs-movement/
/x/1843-erosion/
/x/1843-evolution-creatures/
/x/1724-evolution-simulation/ /x/1841-evolution-simulation/ /x/1843-evolution-simulation/ /x/1844-evolution-simulation/ /x/1845-evolution-simulation/
/x/1811-evolution-simulator/ /x/1849-evolution-simulator/
/x/1810-evolution/ /x/1840-evolution/ /x/1842-evolution/ /x/1844-evolution/ /x/1846-evolution/ /x/1904-evolution/
/x/1942-evolutionary-algorithms/
/articles/finite-state-machines/
/x/1842-fire/
/maps/flag/ /x/flag/
/x/2023-flatbuffers/
/x/2020-float-to-bits/
/pathfinding/flowfield/
/fluid-simulation/ /x/1843-fluid-simulation/
/articles/fog-of-war/
/articles/formation-control/
/x/1843-fractal-tree/
/x/1843-frame-timing/
/pathfinding/funnel/
/maps/galaxy/
/x/1728-game-architecture/
/x/1842-game-loop/
/x/2023-game-programming-links/
/x/1618-genetic-algorithm/ /x/1842-genetic-algorithm/ /x/1844-genetic-algorithm/
/articles/genetic-algorithms/ /x/1717-genetic-algorithms/ /x/1724-genetic-algorithms/ /x/1816-genetic-algorithms/ /x/1830-genetic-algorithms/ /x/1845-genetic-algorithms/ /x/1951-genetic-algorithms/ /x/2020-genetic-algorithms/
/x/1844-genetic-drift/
/articles/geometry/
/x/1845-gradient-descent/ /x/1908-gradient-descent/
/x/1811-graph-maps/mazes.html
/x/1843-graphics-pipeline/
/articles/graphs/
/x/1844-gravity-simulator/
/grids/circle-rect-intersection.html
/grids/floodfill.html
/grids/wordsearch.html
/articles/hash-table/
/maps/heightmaps/
/x/1840-hex-map-editor/ /x/1843-hex-map-editor/
/x/hex-map-maker/
/x/1840-hex-range/
/x/2342-hex-roads/
/x/1843-hex-sphere/
/x/1843-hex-tile-mesh/
/x/1842-hex-viewer/
/x/1844-hexagon-tilings-spherical/
/x/hexagonal-grid-generator/
/x/1843-hexagons-on-sphere/
/x/1840-hexagons-vs-squares/
/maps/hexagons/
/x/honeycomb/
/x/1729-hydraulic-erosion/
/x/1843-icosphere-hexagons/
/x/1812-ik/ /x/1816-ik/ /x/1843-ik/ /x/1844-ik/ /x/1846-ik/
/x/2020-infection-model/
/x/1923-intro-aabb/
/x/1722-intro-spatial-partitioning/
/x/1724-inverse-kinematics/ /x/1840-inverse-kinematics/ /x/1842-inverse-kinematics/ /x/1849-inverse-kinematics/ /x/1908-inverse-kinematics/
/maps/islands/
/x/1842-isometric-depth-sorting/
/x/1911-isometric-projection/
/x/1847-isometric-tiles/
/x/1911-isometric-voxels/ /x/1911-isometric-voxels/
/articles/isometric/ /grids/isometric/ /isometric/
/x/1930-jigsaw-puzzle/
/x/1806-jumping/
/x/1721-kdtree/
/articles/kinematics/
/x/1728-langstons-ant/
/x/1840-langtons-ant/ /x/1844-langtons-ant/ /x/1943-langtons-ant/
/x/1844-life-simulation/
/x/2048-life/
/x/2022-lighting/
/x/1842-line-drawing-algorithms/
/x/2020-lorenz-attractor/
/x/1847-lowpoly/
/x/2023-lsystem/
/x/1722-lsystems/ /x/1843-lsystems/
/articles/map-generation/
/catan/map-generator/
/maps/mapgen/ /x/1844-mapgen/
/x/1843-mapgen4/
/maps/mapmaker/
/x/1843-mapping-spherical-worlds/
/x/1910-marching-squares/
/x/1728-markov-decision-processes/
/x/1843-matrix-transformations/
/matrix-transforms/
/grids/maze-generation/ /x/1847-maze-generation/
/articles/maze/ /pathfinding/maze/
/x/2020-mcts/
/maps/mesh/
/articles/minimax/ /pathfinding/minimax/ /x/1941-minimax/
/x/1840-model-view-controller/
/articles/monads/
/articles/monte-carlo-tree-search/
/x/monty-hall/
/x/1830-morphogenesis/
/pathfinding/multi/
/pathfinding/navmesh/
/x/1841-neural-networks/
/x/2217-nodal/
/x/1847-noise-viewer/
/x/1729-noise/ /x/1840-noise/
/x/1724-np-completeness/
/x/1846-octrees/
/x/1726-offset-paths/
/x/1845-optical-flow/
/articles/organisms/
/x/1729-p-vs-np/
/x/1924-packing-boxes/
/pathfinding/pacman/
/x/2020-pandemic-simulation/
/x/1843-particle-life/ /x/2020-particle-life/
/x/1846-particles/
/articles/partitioning/
/pathfinding/jump-point-search.html
/x/1843-pedestrian-simulation/
/maps/perilous-shores/
/maps/perlin-noise/ /x/1729-perlin-noise/
/x/1842-physics-simulations/
/articles/physics/ /x/1723-physics/ /x/1841-physics/
/x/pixel-count/
/x/1613-pixel-grid/
/x/1841-planet-generation/ /x/1844-planet-generation/
/x/1843-planet-generator/
/x/planet/
/articles/noise/poisson-disc/
/articles/poisson-disk-sampling/
/x/1830-poisson-disk/
/x/1848-polygon-boolean/
/x/1843-polygon-editor/
/articles/polygon-fill/
/articles/polygon-map-generation/
/x/1923-polygon-offsets/
/articles/polygon-partitioning/
/x/1721-polygons/
/articles/procedural-cave/
/x/1843-procedural-city/
/x/1843-procedural-creatures/
/articles/procedural-generation/
/x/1843-procedural-paths/
/x/1843-procedural-planet-gen/
/x/1723-procedural-rivers/
/articles/procedural-road-generation/
/articles/procgen-cave/
/maps/procgen/
/pathfinding/puzzle/
/x/1728-quadtree-visualization/ /x/1840-quadtree-visualization/
/x/1848-quadtree/
/articles/quadtrees/ /grids/quadtrees/ /x/1729-quadtrees/
/x/1843-ray-optics/
/x/1642-reaction-diffusion/ /x/1724-reaction-diffusion/ /x/1726-reaction-diffusion/ /x/1846-reaction-diffusion/ /x/1849-reaction-diffusion/ /x/2022-reaction-diffusion/ /x/reaction-diffusion/
/grids/rectangles/
/x/1844-robot-arm/
/x/1726-rock-paper-scissors/
/x/roguelike-java/
/articles/rooms-and-corridors/
/grids/rotation/
/x/1724-rotations/
/x/1723-rps-simulation/
/x/1720-sand/
/x/2216-sdf-physics/
/x/1922-self-driving-car/
/articles/shunting-yard/
/sliding-puzzle/
/x/2020-smooth-curve/
/pathfinding/smoothing/
/x/1849-soft-body-creatures/
/x/2001-solar-system-generator/ /x/2044-solar-system-generator/
/x/2142-solar-system/
/articles/spatial-indexing/
/x/1724-spherical-voronoi/
/grids/spiral/
/x/1843-spring-damper/
/x/1845-spring-mass/
/x/1843-star-map-generator/
/x/2020-starscape/
/x/1843-stat-scaling/
/articles/steering-behaviors/
/x/1844-straight-skeleton/
/x/1840-svg-vs-png/
/pathfinding/tangent-bug/
/maps/tectonics/ /x/1847-tectonics/
/x/2311-tensor-cores/
/x/1842-terrain-blending/
/x/1845-terrain-generation/
/maps/terrain/ /x/terrain/
/x/1841-tetris-ai/
/x/2312-tetris-art/
/x/1725-threading/
/articles/tile-map-logic/
/articles/tilemaps/
/grids/tiles/ /maps/tiles/
/x/1845-town/
/x/1840-triangle-intrusion/
/grids/triangle/
/x/1843-trig/
/x/1846-tsp-genetic/
/pathfinding/tsp/
/x/1723-turing-patterns/ /x/1729-turing-patterns/ /x/1844-turing-patterns/ /x/1847-turing-patterns/
/articles/turn-based-ai/
/x/1846-turn-based-loop/
/x/turtle/
/x/1841-vanishing-points/
/articles/vectors/ /x/1840-vectors/
/x/2022/06/virtual-machines-in-rust/
/x/1707-visibility-2d/
/x/1723-visibility/
/articles/visible-area/
/articles/visualizing-parallax/
/x/1846-volcano/
/x/1847-volumetric-light/
/x/1908-voronoi-3d/
/x/1836-voronoi-circle-growth/
/x/1830-voronoi-diagram/ /x/1841-voronoi-diagram/ /x/1843-voronoi-diagram/ /x/1846-voronoi-diagram/ /x/1847-voronoi-diagram/ /x/1848-voronoi-diagram/
/x/1840-voronoi-map-generator/ /x/1849-voronoi-map-generator/
/x/1721-voronoi-map/ /x/1723-voronoi-map/
/x/1730-voronoi-maps/
/x/1843-voronoi-noise/ /x/1912-voronoi-noise/
/x/1842-voronoi-relaxation/
/x/1843-voronoi-tile/
/x/1846-voronoi-tsp/
/x/1722-voronoi/ /x/1729-voronoi/ /x/1847-voronoi/ /x/2023-voronoi/ /x/voronoi/
/x/1728-voxel-meshes/
/articles/voxel-raycasting/
/x/1720-voxel-terrain/
/articles/voxels/ /grids/voxels/
/x/1840-wang-tiles/
/x/1847-water-simulation/
/x/1844-water/
/x/1843-waterflow/ /x/1844-waterflow/
/x/1728-wave-function-collapse/ /x/1729-wave-function-collapse/ /x/2023-wave-function-collapse/
/articles/wavefunction-collapse/ /x/wavefunction-collapse/
/x/2020-wavefunctioncollapse/
/x/2022-weather/
/articles/weighted-random/
/x/1836-weighted-voronoi/
/x/2023-wildfire-ca/
/maps/worldgen/
/x/1844-wraparound/
/x/1840-quadtrees
/x/1843-solar-system-generator
/x/1847-sine-wave-art
/x/1849-town
/x/1941-minimax

Most of the URLs it's giving out are plausibly something I could have written.

I think it's also interesting to see what it has "learned" about my URL structure. For my x/ pages, I use the first two digits for the year. ChatGPT is making up lots of URLs starting with x/18 which means that it thinks that 2018 was my most productive year. And … maybe it's right. Hm.

This was a fun diversion. My server's error logs tell me that ChatGPT thinks I've written pages I haven't. Maybe I should actually write those pages!

To keep the scope small, I'm going to stick to substring match and not other query matching algorithms:

• word match (graph pathfinding matches graph for pathfinding and pathfinding graph)
• ordered word match (big dog matches big red dog but not dog was big)
• regexp match (r.*dog matches red dog but not dog was red)
• wildcard match (r* dog matches red dog but not dog was red)
• plurals / stemming (hexagon matches hexagons and hexagonal)
• stop words (the algorithm matches an algorithm)
• spelling (dikjstra matches dijkstra)
• synonym match (mapgen matches map generator)
• case match (OPEN set matches OPEN set but not open set)
• punctuation match (can't matches can't but not cant)
• parent category match (#mapgen matches #mapgen4)

Note that sometimes matching should be asymmetric, because the way we write things in queries is not the same way we write things in documents:

• open set should match both OPEN set and open set, but OPEN set should match only OPEN set
• cant should match both can't and cant, but can't should match only can't
• mapgen should match both #mapgen and #mapgen4, but #mapgen4 should match only #mapgen4

There's a whole lot to explore with matching algorithms but I want to finish this project in a weekend so I will stick to case-insensitive substring match.

For some experiments I need access to headings, so I added headings to the search data. Some of my headings are title="heading" and some are <h3>heading</h3> so I extracted both:

fd --extension bxml | while read -r file; do
    echo "%%%% $file %%%%"
    perl -ne 'print "HEADER=$1\n" if /title="(.*)?"/' "$file"
    perl -ne 'print "HEADER=$2\n" if /<h(\d).*?>(.*?)<\/h\1>/' "$file"
    cat "$file" \
        | pandoc --read html --write plain --wrap none \
        | grep '[a-zA-Z0-9]'
done >all-pages.txt

I'm using regexps instead of xpath or xmllint to extract from my xhtml files, so it misses some things and improperly formats others. That's ok for now. The main goal is to try out ranking, and then I can go back and improve the extraction if I want to pursue this project.

I then refactored the code into smaller functions to make experimentation easier. I also improved the formatting and made the search results clickable (opening in a new window).

function escapeHTML(text) { // wish this was built in
    const map = {
        '&': '&',
        '<': '&lt;',
        '>': '&gt;',
        '"': '&quot;',
        "'": '&#39;'
    };
    return text.replace(/[&<>"']/g, m => map[m]);
}

function parseIntoDocuments(text) {
    // results[0] will be "", then [1] is a filename, [2] is text, alternating
    const results = text.split(/%%%% (.*?) %%%%$/gm); 
    if (results[0] !== "") throw `expected first split to be empty ${results[0]}`;
    if (results[1].length > 100) "expected second split to be filename";
    let documents = [];
    for (let i = 1; i+1 < results.length; i += 2) {
        let filename = results[i];
        let lines = results[i+1].split("\n");
        lines = lines.map(line => line.trim()); // remove extra whitespace
        lines = lines.filter(line => line); // remove blank lines
        let title = (lines.find((line) => line.startsWith("HEADER=")) ?? "").slice(7) || filename;
        let url = filename.startsWith("redblobgames/")
            ? "https://www.redblobgames.com/" + filename.slice(13)
            : "http://www-cs-students.stanford.edu/~amitp/" + filename;
        url = url.replace(".bxml", ".html");
        url = url.replace("/index.html", "/");
        documents.push({url, title, lines});
    }
    return documents;
}
const words = await (await fetch("./all-pages.txt")).text();
const documents = parseIntoDocuments(words);

function search(query) {
    let results = []; // list of {document, snippet}
    // where document is {url, title, lines}
    // where snippet is a list of {line, matches, isHeader}
    if (query) {
        query = new RegExp(RegExp.escape(query), 'gi');
        for (let document of documents) {
            let snippet = [];
            for (let line of document.lines) {
                let isHeader = false;
                if (line.startsWith("HEADER=")) {
                    line = line.slice(7);
                    isHeader = true;
                }
                let matches = line.matchAll(query).toArray();
                if (matches.length > 0) {
                    snippet.push({line, matches, isHeader});
                }
            }
            if (snippet.length > 0) results.push({document, snippet});
        }
    }
    return results;
}

function formatSnippet(snippet) {
    // TODO: be able to sort snippets before picking the top matches
    const MAX_SNIPPETS = 8;
    let result = "… ";
    for (let {line, matches} of snippet.slice(0, MAX_SNIPPETS)) {
        let pos = 0;
        for (let m of matches) {
            result += escapeHTML(line.slice(pos, m.index)) + `<b>${m[0]}</b>`;
            pos = m.index + m[0].length;
        }
        result += `${escapeHTML(line.slice(pos))} … `;
    }
    return result;
}

function formatSearchResults(query, results) {
    return results
          .map(({document, snippet}) => 
              `<a target="_blank" href="${document.url}">
                  ${document.title}
               </a>
               <div class="url">
                  ${document.url}
               </div>
               <div class="snippet">
                  ${formatSnippet(snippet)}
               </div>`)
          .join("\n");
}

function setUpSearch(id, sortResults) {
    const MAX_DOCUMENTS = 10;
    const el = document.getElementById(id);
    const update = () => {
      let query = el.querySelector("input").value;
      let results = sortResults(search(query));
      results = results.slice(0, MAX_DOCUMENTS);
      el.querySelector(".results").innerHTML = formatSearchResults(query, results);
    };
    el.querySelector("input").addEventListener('input', update);
    update();
}

setUpSearch('search-4', (results) => results);

I wanted to test the refactored code before implementing ranking. This shows documents in some arbitrary order, like before:

Let's try ranking by the number of matches in the document:

setUpSearch('search-5', (results) => {
   function score({document, snippet}) {
     let numberOfMatches = 0;
     for (let {matches} of snippet) {
       numberOfMatches += matches.length;
     }
     return numberOfMatches;
   }

   return results.toSorted((a, b) => score(b) - score(a));
});

This is certainly better. The top 5 results before were four hexagon articles and then one about colors. The top 5 results now all have lots of hexagon content.

Let's try ranking header matches higher than other matches:

setUpSearch('search-6', (results) => {
   function score({document, snippet}) {
     let numberOfMatches = 0;
     for (let {matches, isHeader} of snippet) {
       numberOfMatches += matches.length;
       if (isHeader) numberOfMatches += 10; // needs tuning!
     }
     return numberOfMatches;
   }

   return results.toSorted((a, b) => score(b) - score(a));
});

I think this is even better than before. But it's still mediocre. Should I prioritize matches that occur in a cluster (max matches per block) or matches that are spread out (number of matching blocks)? I don't know! It will take a lot of testing. Ranking takes a lot of work. There's so much more that can be done to make search better. Some ideas:

• try out the other ranking ideas I listed at the top of the page
• extract documents correctly; there are going to be lots of little details to get right (data scientists know that "data cleaning" is one of the most time consuming parts of a project)
• identify pages that are just linking to other pages (for example, my blog home page), and prefer returning search results from the linked-to pages
• identify pages that are older versions of other pages, and prefer returning the current pages
• trim snippets at word boundaries centered at the query matches instead of picking the beginning of the line
• split documents into blocks better (for example, <dt> and <dd> should go together, not separate blocks), or maybe get rid of the block idea entirely
• try the query matching rules I mentioned earlier, to handle words, phrases, ordering, synonyms, plurals, etc.
• set up a test suite of queries and hand-picked best results, so that I can evaluate the different ranking algorithms and tuning parameters
• change ranking philosophy from the set of best pages to the best set of pages
• use word2vec or a vector db (from the LLM world) to implement fuzzy matching or group related pages together
• generate a good snippet instead of the first N matches

Tangent: I think the description of search results is underrated. Early web search engines used a static description provided by the author of the page ("meta tag" description) or by a curator (Yahoo / DMoz directories), or generated a description per page. Google generated a description by scanning the document to find the query words, so that you could see the keywords in context of the document. Being able to see the matches of your query instead of a generic description made people feel that Google was actually paying attention to the query words. This is a surprisingly hard problem that I don't see much written about. (For many queries, I think this was more important than PageRank™)

Another question is how to scale this up. I decided not to here. All the words on all my web pages put together are only a few megabytes, smaller than the javascript alone on the NYTimes home page. I can load the entire web site and search over it. But if I had many more pages, I might want an inverted index, which can tell us which documents contain a word. See Let's Build a Full Text Search Engine. If I wanted to index pages written by other people, I might want ranking functions that were more resistant to spamming, such as using links between documents and the anchor text used to describe those links.

I'm not planning to implement these right now. I wanted to see how far I could get in a weekend. The search results I'm getting here are decent. If you want to play with it some more, I made a standalone version that uses the full page to show search results.

This was a fun project!