Juan Rodriguez Donado - sjdonado

Self-Hosted Password Manager with Dokku

Wed, 19 Apr 2023 00:00:00 GMT

Why?

I relied on 1Password for the last few years, it is robust, and very user-friendly. However, these types of services are prime targets for attacks, one popular example is the Lastpass case. It is definitely not cool to have your sensitive info spread in another data breach. Also, privacy is precious nowadays, whoever has the data, has the power.

While exploring Open Source alternatives, I found vaultwarden, an unofficial Bitwarden-compatible server written in Rust.

Migrating data manually is a tedious process, so I first checked how it would be like with Bitwarden. Surprise, surprise, they have it all! You don’t have to do it by hand, it already supports the most popular password managers.

Web - Import Data Page

PS: It was a relief to discover a well-done user interface, here’s a spoiler:

macOS app

To make it cost effective, I decided to use my current Dokku setup, hosted on Google Cloud and powered by Ubuntu 20.04. If you haven’t heard of Dokku, this is your chance to try it out https://dokku.com/docs/getting-started/installation/.

So, having in mind the advantages and the infra required, here’s how I did it.

Prerequisites

Before we get started, check that you already have the following:

A server with Dokku installed.
A domain name (letsencrypt will check it).
Local machine with Docker, Git and any SSH client.

Step by Step

1. Create Dokku App

Set your app name and link to the domain (the Dokku Proxy plugin is built in since version 0.5.0)

dokku apps:create bitwarden

# example: dokku domains:set bitwarden mypasswords.sjdonado.de
dokku domains:set bitwarden yourdomain.com

2. Set up TLS certificates

The Let’s Encrypt plugin automates the generation and renewal of our certificates

# plugin installation requires root, hence the user change
sudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git

dokku letsencrypt:enable bitwarden
# enable the automatic renewal of certificates
dokku letsencrypt:cron-job --add

# replace proxy port, default: 5000
dokku config:set bitwarden DOKKU_PROXY_PORT_MAP="http:80:80 https:443:80"

3. Set up database

We’ll use the PostgreSQL plugin

# setup plugin
sudo dokku plugin:install https://github.com/dokku/dokku-postgres.git

# create a new database for our app
dokku postgres:create bitwarden

# link the postgres service to the app, DATABASE_URL will be attached to the ENV variables automatically.
dokku postgres:link bitwarden bitwarden

4. Set up persistent storage

We can use the built-in Dokku storage plugin

dokku storage:ensure-directory bitwarden

dokku storage:mount bitwarden /var/lib/dokku/data/storage/bitwarden:/data

5. Set up ENV variables

By default, Bitwarden is open to public signups, although we can restrict it to invitation-only by setting SIGNUPS_ALLOWED to false and providing a valid ADMIN_TOKEN.

We’re gonna use argon2 to generate a PHC string for the ADMIN_TOKEN ENV variable

# skip this line if you are a linux user
docker run --rm -it ubuntu

$ apt update && apt install openssl argon2
$ echo -n "MySecretPassword" | argon2 "$(openssl rand -base64 32)" -e -id -k 65540 -t 3 -p 4
# output: $argon2id$v=19$m=65540,t=3,p=4$bXBGMENBZUVzT3VUSFErTzQzK25Jck1BN2Z0amFuWjdSdVlIQVZqYzAzYz0$T9m73OdD2mz9+aJKLuOAdbvoARdaKxtOZ+jZcSL9/N0

For the SMTP_ variables, feel free to use your own SMTP server or comercial solutions with free-tier such as sendgrid

dokku config:set bitwarden \
  DOMAIN=https://mypasswords.sjdonado.de \
  SIGNUPS_ALLOWED=false \
  ADMIN_TOKEN='$argon2id...' \
  SMTP_HOST=smtp.sjdonado.de \
  SMTP_FROM=vaultwarden@sjdonado.de \
  SMTP_PORT=587 \
  SMTP_SECURITY=starttls \
  SMTP_USERNAME= \
  SMTP_PASSWORD=

6. Create the Dockerfile

Let’s create a new folder for the project, the Dockerfile and initialise git

mkdir yourfolder && cd yourfolder
git init
touch Dockerfile

yourfolder/Dockerfile

FROM vaultwarden/server:latest

EXPOSE 80

ENV DB=postgresql

7. Deploy

We are almost there! Before to push the changes, we have to set the dokku remote url

git remote add dokku-bitwarden dokku@yourdomain.com:bitwarden

And finally

git add -A && git commit -m "My bitwarden Dockerfile"
git push dokku master

# output:
git push using:  dokku master
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (6/6), 551 bytes | 551.00 KiB/s, done.
Total 6 (delta 0), reused 0 (delta 0), pack-reused 0
-----> Cleaning up...
-----> Building bitwarden from Dockerfile
remote: #1 [internal] load build definition from Dockerfile
remote: #1 transferring dockerfile: 65B done
remote: #1 DONE 0.0s
remote:
remote: #2 [internal] load .dockerignore
remote: #2 transferring context: 2B done
remote: #2 DONE 0.1s
...

8. That’s it

Now you can access the admin panel via https://yourdomain.com/admin with your MySecretPassword (from step 5) and send invitations by email 😎.

NOTE: Be sure to add your domain when installing the official Bitwarden apps/extensions.

Chrome Extension

Wrapping up

We’ve explored the process of setting up a self-hosted password manager using Dokku. This can be easily replicated on any IaaS provider.

Now, you can have peace of mind knowing that your passwords are securely stored and easily accessible whenever and wherever you need them.

Source code available on Github.

Happy hacking!

HTMX with Bun: A Real World App

Tue, 05 Mar 2024 00:00:00 GMT

Let’s build a real-world app with HTMX + Tailwind CSS + Bun. If you are already familiar with these tools, feel free to skip the context part.

Context

HTMX, HTMZ, HTMY…

All we need to achieve decent smooth reactivity is already out there, built-in by the major browsers. As an example, here is one interesting front-end framework:

<iframe
  hidden
  name=htmz onload="setTimeout(()=>document.querySelector(contentWindow.location.hash||null)?.replaceWith(...contentDocument.body.childNodes))"
></iframe>

That’s it, just an iframe, try it out yourself on https://leanrada.com/htmz/.

The philosophy behind this is the same as HTMX and the same one that drives this post: Don’t reinvent rendering, just use HTML.

The blazingly fast JS server

With HTMX in mind, the next step is to get up and running a a simple web server.

Bun has gained a lot of popularity in the last few months. It is fast, and there is a framework called ElysiaJS that resembles Express.js and promises to be 22 times faster than Express.

Additionally, Bun understands TypeScript and JSX out of the box, so we can structure our app with components, server-side render them, and rely on HTMX for client-side actions.

Tailwind CSS Unlike traditional CSS frameworks that come with predefined components, Tailwind CSS emphasizes atomic classes that represent individual CSS properties. The bundle size could be lighter in comparison to other tools, but the strength here, IMHO, is rapid prototyping.

A Real World App

The problem: I’m an Apple Music user. Someone shares a Spotify link with me, and I don’t have a way to know what’s inside or listen to it.

The solution: I Don’t Have Spotify scrapes the links (songs/albums/artists etc) from all your favorite streaming services, and it even provides a listening preview if you just want a quick look.

The approach

Given a Spotify link, we need to extract its metadata: Fetch request + HTML parsing.
Given the metadata, we need to search for the resource on each supported streaming service: Multiple API/HTML requests + parsing (adapters).
Given the results of (1) and (2), we need to render and return the data to the client.
Additionally, we list a JSON endpoint for a cool Raycast extension.

There are two major design decisions:

The folder structure / separation of concerns
Tooling + build process, how to live reload the JSX components?

For the first one, since we have multiple streaming services, we’ll inevitably encounter shared logic between them. Drawing inspiration from the adapter pattern but without the boilerplate, I organized the adapters folder with pure functions. These functions all receive the same arguments: query: string and metadata: SpotifyMetadata, and they return a SpotifyContentLink.

The adapters are called by the spotifySearch function in the search service, which is also responsible for caching and updating the statistics. Why caching? In short: these calls are quite time-consuming, and we can easily hit a rate limit if we send many of them at the same time.

For the second aspect, running Bun is very straightforward: bun run --watch www/bin.ts. However, as a real-world app, we need some JavaScript on the client side, apart from sending AJAX requests. There is an audio player that has to be rendered on the client side (in order to append listeners to the DOM). Additionally, we would like to access the Clipboard API to improve the user experience when searching.

To bundle and minify the required JavaScript, I used Vite with the rollup-plugin-copy plugin:

import path from 'path';

import { defineConfig } from 'vite';
import copy from 'rollup-plugin-copy';

export default defineConfig({
  plugins: [
    copy({
      targets: [
        {
          src: 'dist/*',
          dest: 'public',
          copyOnce: false,
        },
      ],
      hook: 'writeBundle',
    }),
  ],
  resolve: {
    alias: {
      '~/config/constants': path.resolve(__dirname, './src/config/constants.ts'),
    },
  },
  build: {
    outDir: './dist',
    target: 'esnext',
    rollupOptions: {
      input: {
        'assets/js/audio-preview': './src/views/js/audio-preview.js',
        'assets/js/search-bar': './src/views/js/search-bar.js',
        'assets/css/index': './src/views/css/index.css',
      },
      output: {
        entryFileNames: '[name].min.js',
        chunkFileNames: `[name].min.js`,
        assetFileNames: `[name].min.[ext]`,
      },
    },
  },
});

Showcase

The search bar component (where HTMX shines)

export default function SearchBar() {
  return (
    <>
      <form
        id="search-form"
        hx-post="/search"
        hx-target="#search-results"
        hx-swap="innerHTML"
        hx-indicator="#loading-indicator"
        hx-request='\"timeout\":24000'
        class="flex w-full max-w-3xl items-center justify-center"
      >
        <label for="song-link" class="sr-only">
          Search
        </label>
        <input
          type="text"
          id="song-link"
          name="spotifyLink"
          class="flex-1 rounded-lg border bg-white p-2.5 text-sm font-normal text-black placeholder:text-gray-400 lg:text-base"
          placeholder="https://open.spotify.com/track/7A8MwSsu9efJXP6xvZfRN3?si=d4f1e2eb324c43df"
          pattern={SPOTIFY_LINK_REGEX.source}
        />
        <button
          type="submit"
          class="ml-2 rounded-lg border border-green-500 bg-green-500 p-2.5 text-sm font-medium text-white focus:outline-none focus:ring-1 focus:ring-white"
        >
          <i class="fas fa-search p-1 text-black" />
          <span class="sr-only">Search</span>
        </button>
      </form>
      <div class="my-4">
        <div id="search-results"></div>
      </div>
    </>
  );
}

The web server

import { Elysia } from 'elysia';
import { html } from '@elysiajs/html';
import { staticPlugin } from '@elysiajs/static';

import { logger } from './utils/logger';

import { apiRouter } from './routes/api';
import { pageRouter } from './routes/page';

export const app = new Elysia()
  .use(html())
  .use(
    staticPlugin({
      prefix: '',
      headers: {
        'Cache-Control': 'public, max-age=86400',
      },
    })
  )
  .on('beforeHandle', async ({ request }) => {
    logger.info(
      `${request.method} ${request.url} - ${request.headers.get('user-agent')}`
    );
  })
  .use(apiRouter)
  .use(pageRouter);

Testing

In our case, testing involves mocks and more mocks. They run on push changes to master, thanks to GitHub Actions:

The setup relies on bun:test and AxiosMockAdapter. It is separated into integration and unit tests, and the requests are injected using two helpers:

export const JSONRequest = (endpoint: string, body: object) => {
  return new Request(endpoint, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(body),
  });
};

export const formDataRequest = (endpoint: string, body: object) => {
  const formData = new FormData();

  Object.entries(body).forEach(([key, value]) => {
    formData.append(key, value);
  });

  return new Request(endpoint, {
    method: 'POST',
    body: formData,
  });
};

Incorporating e2e shouldn’t be complex. An example of running the dev server with Playwright:

  webServer: {    command: 'bun run dev',    port: 3333,    reuseExistingServer: !process.env.CI,  }

Wrapping up

The source code is available on GitHub: https://github.com/sjdonado/idonthavespotify.
The app is up and running thanks to Dokku: https://idonthavespotify.sjdonado.com.
The Raycast extension will be addressed in a future post, stay tuned: https://www.raycast.com/sjdonado/idonthavespotify.

Fast, Intuitive, Smart: Restaurant search engine with Cloudflare AI

Sat, 13 Apr 2024 00:00:00 GMT

import culefiloSearch from ’../images/kjp6ltwap8ithxp8b97w.gif?url’;

This project was submitted to the Cloudflare AI Challenge.

The Inspiration Behind “Cule filo”

Imagine you’re craving your favorite meal—a local specialty or a unique dish you tried once and can’t forget. Now, imagine finding the best spot for that meal near you, simply by typing the dish name and your location. That’s where “Cule filo” comes in, our AI-powered restaurant search engine that guides you to the top 3 restaurants serving exactly what you’re in the mood for. Built with powerful AI models and designed for simplicity, this free app is your smart companion to uncover culinary gems in your area.

Our Development Journey

Building “Cule filo” was an exhilarating journey that started with a brainstorm on how to leverage Cloudflare’s AI tools to bring real value to users. After exploring various ideas, we zeroed in on the concept of a restaurant search engine that uses AI to make dining discovery intuitive and exciting.

To make it happen, we brought together multiple AI models, each serving a distinct role in shaping a smooth user experience:

Suggestions & Thumbnails: We used the llama-2-13b-chat-awq model to suggest alternative restaurants when fewer than 3 results matched the search, ensuring users always find relevant options. The same model selects thumbnails for the results, making them visually appealing.
Summarized Descriptions: For quick overviews of each restaurant, we turned to the bart-large-cnn model, which condenses user reviews into concise descriptions that capture each place’s unique qualities.
Image-to-Text Descriptions: Lastly, we used the uform-gen2-qwen-500m model to analyze restaurant photos and generate text descriptions, giving users a sense of the restaurant ambiance before they even step inside.

Throughout development, we prioritized creating an interface that feels both responsive and intuitive. Real-time search logs keep users engaged, and a search history feature allows easy access to previous explorations.

AI Models in Action

Our project is particularly innovative in its use of multiple AI models per task and the integration of three distinct AI task types, making it a standout submission in the challenge:

Multiple Models per Task:

The llama-2-13b-chat-awq model is used both for generating suggestions when the initial search yields limited results and for selecting suitable thumbnails.
The bart-large-cnn model generates summaries of user reviews, giving a clear snapshot of what to expect at each restaurant.
The uform-gen2-qwen-500m model converts restaurant images into textual descriptions, adding depth and context to the search results.

Triple Task Types:

Text Generation: The llama-2-13b-chat-awq model generates recommendations and captions.
Text Summarization: The bart-large-cnn model condenses user reviews into summaries.
Image-to-Text: The uform-gen2-qwen-500m model creates descriptions based on restaurant images.

To wrap up

This project was a true team effort, and I couldn’t have done it without the creativity and support of my awesome friends and teammates: gjhernandez and krthr. Building this together was a blast, and we’re proud of what we’ve accomplished.

If you’re curious about the inner workings or just want to poke around the code, you’re more than welcome to check out the repository: https://github.com/sjdonado/culefilo.

Validated forms with useFetcher in Remix

Wed, 10 Jul 2024 00:00:00 GMT

Interacting with the server without using window.navigation significantly improves the user experience. E.g: Login forms within a dialog box or modal, optimistic UI forms or submitting multiple forms within a complex view.

If you are not familiar with Remix or the useFetcher hook, please refer to:

The FetcherForm Provider

The Remix fetcher object contains three primary attributes: fetcher.state, fetcher.data and the method fetcher.submit. To interact with them we will use React.useEffect.

This post will show how to create a provider that manages the state of a fetcher submitted form, along with a minimal custom hook:

const { onChange, submitForm, isSubmitted, error } = useFetcherForm();

Let’s break down the FetcherFormProvider props one by one.

1) onChange

External inputs will notify through this method to change the internal state of the provider, it receives a FormData argument:

export default function FetcherFormProvider({
  action,
  method,
  children,
}: {
  action: string;
  method: SubmitOptions['method'];
  children: React.ReactNode;
}) {
  const [formData, setFormData] = useState<FormData>();
  [...]
  return (
    <FetcherFormContext.Provider
      value={[
        (formData: FormData) => {
          setFormData(formData);
        },
        [...]
        ]}
    >
      {children}
    </FetcherFormContext.Provider>
  );
}

2) submitForm

Since the formData was already captured by the onChange method, the request can be send by calling fetcher.submit:

export default function FetcherFormProvider({
  action,
  method,
  children,
}: {
  action: string;
  method: SubmitOptions['method'];
  children: React.ReactNode;
}) {
  const fetcher = useFetcher();
  const [formData, setFormData] = useState<FormData>();

  [...]
  return (
    <FetcherFormContext.Provider
      value={[
        (formData: FormData) => {
          setFormData(formData);
        },
        () => {
          if (formData) {
            fetcher.submit(formData, {
              method,
              action,
            });
          }
        },
        [...]
      ]}
    >
      {children}
    </FetcherFormContext.Provider>
  );
}

3) isSubmitted

Submitting the form is an asynchronous operation. There is a separate variable to listen to the form submitted event: isSubmitted. This is helpful in the following cases:

To close the Dialog or Modal when the form is successfully submitted.
To check from outside that the form was submitted and/or the request returned an OK status.

export default function FetcherFormProvider({
  action,
  method,
  children,
}: {
  action: string;
  method: SubmitOptions['method'];
  children: React.ReactNode;
}) {
  const fetcher = useFetcher();

+ const [isSubmitted, setIsSubmitted] = useState(false);
  [...]
  const [formData, setFormData] = useState<FormData>();

  useEffect(() => {
    const response = fetcher.data as { error: string } | undefined;

+   if (isSubmitted || error) return;

    if (fetcher.state === 'loading' && response) {
      [...]
+     setIsSubmitted(true);
    }
  }, [fetcher, action, formData, isSubmitted, error]);

  return (
    <FetcherFormContext.Provider
      value={[
        (formData: FormData) => {
          setFormData(formData);
        },
        () => {
          if (formData) {
            fetcher.submit(formData, {
              method,
              action,
            });
          }
        },
+       isSubmitted,
        [...]
      ]}
    >
      {children}
    </FetcherFormContext.Provider>
  );
}

4) error

One drawback of using Remix useFetcher is the lack of a straightforward error handling method. There are proposals in progress to provide a more streamlined error handling approach:

As a workaround we can rely on fetcher.state to check if the request is complete and get the message with fetcher.data by defining a common structure between the client and server actions.

The highlights

const response = fetcher.data as { error: string } | undefined; -> defines the JSON response structure to be received form the server return json({ error: new Error() }, { status: 400 } );
if (fetcher.state === 'loading' && response) { -> checks if the request is completed and a response is available.

export default function FetcherFormProvider({
  action,
  method,
  children,
}: {
  action: string;
  method: SubmitOptions['method'];
  children: React.ReactNode;
}) {
  const fetcher = useFetcher();

  const [isSubmitted, setIsSubmitted] = useState(false);
+ const [error, setError] = useState<string>();

  const [formData, setFormData] = useState<FormData>();

  useEffect(() => {
    const response = fetcher.data as { error: string } | undefined;

    if (isSubmitted || error) return;

    if (fetcher.state === 'loading' && response) {
+     if (response.error) {
+       setError(response.error);
+       return;
+     }
      setIsSubmitted(true);
    }
  }, [fetcher, action, formData, isSubmitted, error]);

  return (
    <FetcherFormContext.Provider
      value={[
        (formData: FormData) => {
          setFormData(formData);
+         setError(undefined);
        },
        () => {
          if (formData) {
            fetcher.submit(formData, {
              method,
              action,
            });
          }
        },
        isSubmitted,
+       error,
      ]}
    >
      {children}
    </FetcherFormContext.Provider>
  );
}

Optionally, the error state management can be sent as a callback. This can be received in the onSubmit function and registered as state. For example, using a registeredCallback:

- if (response.error) {
- setError(response.error);
- return;
- }
+ registeredCallback?.(response.error);

The useFetcherForm hook

The values sent to FetcherFormContext.Provider are defined in the FetcherFormContext

const FetcherFormContext = createContext<
  [(formData: FormData) => void, (callback?: () => void) => void, boolean, string?]
>([() => null, () => null, false]);

Then the hook exposes them in an object structure:

export const useFetcherForm = () => {
  const [onChange, submitForm, isSubmitted, error] = useContext(FetcherFormContext);
  return {
    onChange,
    submitForm,
    isSubmitted,
    error,
  };
};

An example of how it can be defined:

export function AssignmentUpdateStatusDialogButton({
  assignmentId,
  status,
}: {
  assignmentId: string;
  status: AssignmentStatus;
}) {
  const [isAttached, setIsAttached] = useState(false);
  const dialog = useRef<HTMLDialogElement>();

  useEffect(() => {
    if (isAttached) {
      dialog.current?.showModal();
    }
  }, [isAttached, dialog]);

  return (
    <ClientOnly>
      {() => (
        <>
          {isAttached &&
            createPortal(
              <FetcherFormProvider
                action={`/assignments/${assignmentId}/status`}
                method="post"
              >
                <AssignmentUpdateStatusDialog
                  ref={dialog}
                  [...]
                  setIsAttached={setIsAttached}
                />
              </FetcherFormProvider>,
              document.body
            )}
          <button
            type="button"
            className="cursor-pointer leading-none"
            onClick={() => setIsAttached(true)}
          >
            <AssignmentStatusBadge status={status} />
          </button>
        </>
      )}
    </ClientOnly>
  );
}

Demo

Real-world example available on GitHub

Building a Fast and Compact SQLite Cache Store

Wed, 24 Jul 2024 00:00:00 GMT

When working on applications, caching is crucial for enhancing performance by reducing the need for repeated database fetches. Among the various SQLite libraries available, Bun’s native integration is optimized for performance and takes advantage of its non-blocking I/O capabilities.

The bun:sqlite module is roughly 3-6x faster than better-sqlite3 and 8-9x faster than deno.land/x/sqlite for read queries. Each driver was benchmarked against the Northwind Traders dataset. View and run the benchmark source. Source

In addition to efficient caching, serializing JavaScript objects can be slow when using JSON. This is why it makes sense to opt for efficient binary encoding alternatives like Msgpackr or CBOR. These formats are faster to parse, support complex data types, require less CPU usage, and store data more compactly, further enhancing overall application performance.

With this in mind, let’s explore how to build a cache manager using bun:sqlite along with efficient binary encoding.

A cache-manager Store

cache-manager provides a straightforward and intuitive API for caching, abstracting away the complexity of managing different cache stores and their configurations. With support for multiple stores, built-in expiration and TTL management, and robust error handling and fallback mechanisms, it ensures data integrity and freshness.

Additionally, cache-manager is highly customizable and extensible, allowing you to create custom cache stores tailored to your needs. This flexibility means you can set up and use caching with minimal code, allowing you to focus on your application’s core logic.

The required interface to fulfill as a cache-manager store is as follows:

export type Store = {
    get<T>(key: string): Promise<T | undefined>;
    set<T>(key: string, data: T, ttl?: Milliseconds): Promise<void>;
    del(key: string): Promise<void>;
    reset(): Promise<void>;
    mset(arguments_: Array<[string, unknown]>, ttl?: Milliseconds): Promise<void>;
    mget(...arguments_: string[]): Promise<unknown[]>;
    mdel(...arguments_: string[]): Promise<void>;
    keys(pattern?: string): Promise<string[]>;
    ttl(key: string): Promise<number>;
};

See more here.

The Queries

Configuring SQLite for Optimal Performance

PRAGMA main.synchronous = NORMAL;: Ensures that SQLite writes are fast while still maintaining a reasonable level of data safety. It does not guarantee as much durability as FULL, but it is sufficient for many use cases.
PRAGMA main.journal_mode = WAL2;: Improves concurrency by allowing readers to access the database while a write operation is ongoing.
PRAGMA main.auto_vacuum = INCREMENTAL;: Allows SQLite to reclaim unused space incrementally, rather than all at once.

1. Creating the Cache Table

CREATE TABLE IF NOT EXISTS {table} (
    key TEXT PRIMARY KEY,
    val BLOB,
    created_at INTEGER,
    expire_at INTEGER
);
CREATE INDEX IF NOT EXISTS index_expire_{table} ON {table}(expire_at);

The val column stores the cached value in a binary large object (BLOB) format, allowing it to handle various data types depending on the chosen serializer.

2. Inserting or Updating Cache Entries

INSERT OR REPLACE INTO ${name}(key, val, created_at, expire_at) VALUES (?, ?, ?, ?)

expire_at is pre-calculated based on the ttl value in milliseconds, and val is pre-checked by a function isCacheable: (value: unknown) => boolean;.

3. Retrieving Cache Entries

SELECT * FROM ${name} WHERE key = ? AND expire_at > ? LIMIT 1

Returns one record that has not expired.

Auto Purge and Batch Operations

1. purgeExpired

const purgeExpired = async () => {
  const now = Date.now();
  if (now - lastPurgeTime >= 60 * 60 * 1000) {
    const statement = db.prepare(`DELETE FROM ${name} WHERE expire_at < ?`);
    statement.run(now);
    lastPurgeTime = now;
  }
};

Keeps the cache clean and efficient by regularly removing stale entries, ensuring that the cache does not grow indefinitely and affect performance.

2. Batch Set (mset)

const mset = async (pairs: [string, unknown][], ttl?: number) => {
  const ttlValue = ttl !== undefined ? ttl * 1000 : defaultTtl;
  if (ttlValue < 0) {
    return;
  }
  const expireAt = Date.now() + ttlValue;

  const stmt = `INSERT OR REPLACE INTO ${name} (key, val, created_at, expire_at) VALUES ${pairs.map(() => '(?, ?, ?, ?)').join(', ')}`;
  const bindings = pairs.flatMap(([key, value]) => {
    if (!isCacheable(value)) {
      throw new NoCacheableError(`"${value}" is not a cacheable value`);
    }
    return [key, serializerAdapter.serialize(value), Date.now(), expireAt];
  });

  const statement = db.prepare(stmt);
  statement.run(...bindings);
};

Improves efficiency by reducing the number of individual database operations. In the same way mget executes one single query to returns valid records with the query:

SELECT * FROM ${name} WHERE key IN (${placeholders}) AND expire_at > ?

Conclusion

Bun’s SQLite implementation combined with efficient binary encoding formats like Msgpackr provides a powerful solution for building fast and compact cache stores.

For the complete source code and implementation details, visitt: https://github.com/sjdonado/cache-manager-bun-sqlite3

You can also find the npm package cache-manager-bun-sqlite3.

Happy hacking!

Should we leave behind the joy of curating our own content?

Fri, 02 Aug 2024 00:00:00 GMT

For a while, I’ve been reading blog posts, articles, and comment threads about various tech and finance topics. I initially found it convenient to have a Medium subscription, as I thought it would allow me to have a feed automatically shaped by my interests. However, I ended up reading articles prioritized by clickbait, echo chambers and bias.

So, I canceled my subscription and switched to Hacker News. I have discovered amazing independent writers discussing fascinating topics. Eventually, I started collecting these RSS feeds in one place, creating my own “Medium”.

I believe that’s the true purpose of the internet: to be free to decide where to go and what to explore. There is joy in the journey more than the destination.

If you’re curious about the blogs I follow, you can download and import the OPML file into your RSS reader of choice.

Tower of Hanoi in P5.js + WASM

Mon, 08 May 2023 00:00:00 GMT

import towerAnimation from ’../images/60ltbfz5s9nm9ejj07bf.gif?url’; import wasmPackOutput from ’../images/j5sg6n8rehm3s1pegu66.gif?url’;

In 2018, a professor at the Uni asked me to build a series of computer science games; he was looking for a very visual way to show students how the solution algorithms of NP problems (like the Knapsack Problem, Tower of Hanoi, Tic-Tac-Toe, etc.) behave and grow with their inputs.

Tower of Hanoi is a good exercise when students are getting started with recursion and vectors. The pegs are usually fixed at 3, so it is easy to define the base case, and the only variable is the number of disks. It is worth noting that there are other more efficient approaches, such as The Binary solution or the Gray-code solution, but this was a Complexity course and we wanted to show how exponential time can break your computer while keeping it as minimal as possible.

Vanilla JS Implementation

There was only one design requirement: to show the animated solutions of the problems in a web browser. At that time I was familiar with HTML5 but not with any modern UI frameworks. Frankly, I couldn’t imagine myself rendering objects from scratch in a plain HTML canvas. There had to be something out there that had already invented that wheel, and there was.

I found P5.js, it is sort of the JS version of Processing and it has everything I was looking for, a complete API for rendering, painting images and text, calling DOM elements and much more.

The main core of the animation system was the draw function provided by P5. It works as an infinite loop, so if we want to show the disks moving from x1, y1 to x2, y2, we have to update the current position x += speed y += speed where 0 < speed < 120. There are two types of motion, vertical and horizontal. An example of the former is:

if (isMovingHorizontallly) {
  if (animation.currentMove.start < animation.currentMove.end) {
    refreshCanvas(p5);
    drawDiskByCoordinates(
      p5,
      animation.currentDisk,
      animation.currentMove.start,
      draw.topMargin,
    );

    animation.currentMove.start += game.speed;

    if (animation.currentMove.start >= animation.currentMove.end) {
      animation.currentMove.start = animation.currentMove.end;
      animation.currentMove.down = draw.topMargin;
    }
  }

  if (animation.currentMove.start > animation.currentMove.end) {
    refreshCanvas(p5);
    drawDiskByCoordinates(
      p5,
      animation.currentDisk,
      animation.currentMove.start,
      draw.topMargin,
    );

    animation.currentMove.start -= game.speed;

    if (animation.currentMove.start <= animation.currentMove.end) {
      animation.currentMove.start = animation.currentMove.end;
      animation.currentMove.down = draw.topMargin;
    }
  }
}

Note that the refreshCanvas function refreshes the canvas and redraws the towers filled by the disks, but not the one being animated.

You may wonder how the disks know where to move, do they move at the same time as the Hanoi calculations? That would be cool, but unfortunately they don’t. We are limited by the recursive algorithm, so we have to wait until all calculations are done (until the call stack is empty).

Therefore, the draw function constantly checks if the array of moves [towerFrom:towerTo, ...] generated by recursiveHanoi is not empty. If it is, the animation starts.

function recursiveHanoi(A, C, B, n){
  if (n == 1) {
    moves.push(A + ":" + C);
  } else {
    recursiveHanoi(A, B, C, n - 1);
    moves.push(A + ":" + C);
    recursiveHanoi(B, C, A, n - 1);
  }
}
function generateHanoiArray(disksNumber) {
  var moves = [];
  recursiveHanoi(0, 2, 1, disksNumber);
  return moves;
};

recursiveHanoi did the job, but it was very inefficient: it couldn’t handle more than 14 disks (16K steps). The root cause was not in the algorithm itself, but in its execution in the V8 interpreter. Anyway, the same recursive function in C++ could run perfectly for more than 20 disks (aprox 16M of steps). So, running compiled bytecode in the browser was clearly the next step.

wasm-pack

After four years, I found some time to pay that deb-tech (yes, quite a long time, eh). To make it fun I rewrote everything from scratch in SolidJS, which went smooth thanks to this amazing library p5js-wrapper. For WASM, C++ is still a good choice, but what about Rust? I did some research and found wasm-pack. A few lines in the cargo.toml file and we were ready to generate compiled + ready to import bytecode!

Let’s dive into the configuration:

The crate-type is set to indicate that the crate will be compiled as a dynamic library: wasm-bindgen.
gloo-utils and serde are included separately to avoid circular dependencies (history).

...

[lib]
path = "logic/lib.rs"
crate-type = ["cdylib"]

[dependencies]
wasm-bindgen = "0.2"
getrandom = { version = "0.2.8", features = ["js"] }
gloo-utils = { version = "0.1", features = ["serde"] }
serde = { version = "1.0", features = ["derive"] }

Once the configuration is ready, we run wasm-pack build -d wasm --no-typescript to compile. There is no advantage in generating ts files because we won’t use those definitions since the get_moves function is gonna be called using a Web Worker.

Then, to make the wasm folder accessible to the solid app, we need to update the vite.config.js file following the recommended setup from vite-plugin-wasm:

import { defineConfig } from 'vite';

import solidPlugin from 'vite-plugin-solid';
+ import wasm from 'vite-plugin-wasm';
+ import topLevelAwait from 'vite-plugin-top-level-await';

import path from 'path';

export default defineConfig({
  plugins: [
    solidPlugin(),
+   wasm(),
+   topLevelAwait(),
  ],
  server: {
    port: 3000,
  },
  build: {
    target: 'esnext',
  },
  resolve: {
    alias: {
      '@src': path.resolve(__dirname, './src'),
+     '@wasm': path.resolve(__dirname, './wasm'),
    },
  },
+  worker: {
+    plugins: [
+      wasm(),
+      topLevelAwait(),
+    ],
+  },
});

The compiled output is now accessible as follows:

import { get_moves } from '@wasm/games';

But wait a second, we haven’t talked about this function yet, and why do we need a Web Worker? Let’s move on to the final implementation.

Hanoi Algorithm in Rust

The tricky part of writing WASM code is how to interact with the outside world (I/O operations). Fortunately, our function is quite small, its conditions are mapped as follows:

The function takes a Number -> i32 argument: The number of disks.
The function returns an array of strings serialised by the serde_json crate (which allows you to convert a Rust data structure that implements the serde::Serialize trait to a serde_json::Value type, in this case a JsValue).

And that’s it, inside the public fn, we can use regular Rust types like Vect.


use gloo_utils::format::JsValueSerdeExt;
use wasm_bindgen::prelude::*;

#[wasm_bindgen]
pub fn get_moves(n: i32) -> JsValue {
    fn move_tower(n: i32, from: i32, to: i32, aux: i32, moves: &mut Vec<String>) {
        if n == 1 {
            moves.push(format!("{}:{}", from, to));
        } else {
            move_tower(n - 1, from, aux, to, moves);
            moves.push(format!("{}:{}", from, to));
            move_tower(n - 1, aux, to, from, moves);
        }
    }

    let mut moves = Vec::new();
    move_tower(n, 0, 2, 1, &mut moves);
    JsValue::from_serde(&moves).unwrap()
}

Web Worker

As mentioned above, the number of disks is exponentially related to the number of steps required in the solution. 20 disks are about 1M, but 24 are more than 16M (which is quite a lot 😰). In fact, the execution time of the recursive function exceeds the Doherty threshold (300ms). So, it becomes mandatory to execute it inside a non-blocking thread, for which we can use the Web Workers API.

A new .js file must be created for the worker anywhere in the src folder.

import { get_moves } from '@wasm/games';

onmessage = (event) => {
  const moves = get_moves(event.data.n);

  postMessage(moves);
};

Then, since the worker instance has nothing to do with the solid component lifecycle, we can instance it outside.

const wasmWorker = new Worker(new URL('../workers/hanoi.js', import.meta.url), {
  type: 'module',
});

Finally, when the user clicks on the play button, the async function is called:

const getHanoiMoves = async (n) => new Promise((resolve) => {
  wasmWorker.onmessage = (event) => resolve(event.data);
  wasmWorker.postMessage({ n });
});

In Summary

NP problems are highly CPU-intensive, and wasm-pack makes it easier to reduce the execution time. In addition, wrapping these operations in a Web Worker improves the user experience.

A picture is worth a thousand words: for 20 disks the Vanilla JS implementation takes more than 312000ms, while Rust does it in under 200ms (tested on a Macbook Air M2).

Vanilla JS	WASM (Rust)

Source code available on Github: https://github.com/sjdonado/cs-games.

Sitemap generation by streaming from WordPress headless

Sun, 03 Nov 2024 00:00:00 GMT

The sitemap.xml file is very useful for search engines to understand which URLs are available for crawling. Therefore, when building a blog, generating this file can help boost SEO.

Taking this website as an example, the sitemap includes static URLs like /home, /digital-garden, as well as blog posts at /posts/:slug. This is the output we need to generate:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>http://localhost:5173/</loc>
    <lastmod>2024-11-04T06:32:35.319Z</lastmod>
    <changefreq>monthly</changefreq>
    <priority>1</priority>
  </url>
  <url>
    <loc>http://localhost:5173/home</loc>
    <lastmod>2024-11-04T06:32:35.319Z</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>http://localhost:5173/posts</loc>
    <lastmod>2024-11-04T06:32:35.319Z</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>http://localhost:5173/digital-garden</loc>
    <lastmod>2024-11-04T06:32:35.319Z</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>http://localhost:5173/social</loc>
    <lastmod>2024-11-04T06:32:35.319Z</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.6</priority>
  </url>
  <url>
    <loc>http://localhost:5173/slides</loc>
    <lastmod>2024-11-04T06:32:35.319Z</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.6</priority>
  </url>
  <url>
    <loc>https://sjdonado.com/posts/should-we-leave-behind-the-joy-of-curating-our-own-content</loc>
    <lastmod>2024-11-02T12:38:11.000Z</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://sjdonado.com/posts/building-a-fast-and-compact-sqlite-cache-store</loc>
    <lastmod>2024-10-31T17:06:21.000Z</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://sjdonado.com/posts/validated-forms-with-usefetcher-in-remix</loc>
    <lastmod>2024-10-31T15:15:14.000Z</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://sjdonado.com/posts/fast-intuitive-smart-restaurant-search-engine-with-cloudflare-ai</loc>
    <lastmod>2024-10-31T15:31:28.000Z</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://sjdonado.com/posts/htmx-with-bun-a-real-world-app</loc>
    <lastmod>2024-11-03T14:02:46.000Z</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://sjdonado.com/posts/tower-of-hanoi-in-p5-js-wasm</loc>
    <lastmod>2024-11-03T12:58:11.000Z</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://sjdonado.com/posts/self-hosted-password-manager-with-dokku</loc>
    <lastmod>2024-10-31T15:55:16.000Z</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
</urlset>

As you might guess, the challenge of generating this file lies in how to fetch the posts using the WordPress REST API. Initially, we could opt to fetch all posts with GET /wp-json/wp/v2/posts and iterate over the response in a separate backend:

  const response = await fetch(`${siteUrl}/wp-json/wp/v2/posts?per_page=100&_embed`);
  const posts = await response.json();

  let sitemap = '<?xml version="1.0" encoding="UTF-8"?>\n';
  sitemap += '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\n';

  for (const url of staticUrls) {
    sitemap += `
      <url>
        <loc>${url.loc}</loc>
        <lastmod>${url.lastmod}</lastmod>
        <changefreq>${url.changefreq}</changefreq>
        <priority>${url.priority}</priority>
      </url>
    `;
  }

  for (const post of posts) {
    sitemap += `
      <url>
        <loc>${siteUrl}/posts/${post.slug}</loc>
        <lastmod>${new Date(post.modified || post.date).toISOString()}</lastmod>
        <changefreq>weekly</changefreq>
        <priority>0.8</priority>
      </url>
    `;
  }

  sitemap += '</urlset>';

  return new Response(sitemap, {
    headers: {
      'Content-Type': 'application/xml',
    },
  });

That approach works fine, and in the end, the most time-consuming operation is waiting for the API response, which tends to increase linearly as more posts are fetched. The question now is, can this be faster?

The initial answer has to be yes, since we are adding overhead by not rendering this file directly from the WordPress environment, which is closest to the database. Moving from a headless approach to creating a WP action would be the ideal, fastest scenario. However, we don’t want to do that, we want to maintain a separation of concerns, where WP is our CMS, and the blog routes are rendered elsewhere.

This approach leads us to consider rendering the sitemap file immediately as data is received from the database. Instead of waiting for the entire dataset, we can process posts in chunks as they are sent from the database. By setting up a new REST endpoint that returns posts as chunked data, a ReadableStream can handle each chunk, appending it to the XML sitemap while streaming the file directly to the browser. This way, the browser starts receiving the sitemap file almost instantly, and the XML is built dynamically as new data arrives.

It may sound a bit complicated, but think of it this way: We are streaming from the database to the browser by converting each post into a urlset entry in the final XML file, each post goes through two streams:

WP (stream 1, chunk post) -> Backend (stream 2, chunk urlset) -> Browser

With this in mind, let’s dive into the code.

Stream 1: WordPress endpoint

Chunked transfer encoding allows chunks to be sent and received independently of one another. Therefore, we query the posts in batches, collect their information, and send them one by one in chunks. We need to ensure that each JSON-encoded chunk includes its size appended at the end.

function stream_posts() {
    // Set the HTTP headers to indicate a stream
    header('Content-Type: application/json');
    header('Transfer-Encoding: chunked');
    header('Connection: keep-alive');
    header('X-Accel-Buffering: no'); // Disable buffering in Nginx if applicable

    // Start output buffering and flush
    ob_start();

    // Fetch posts in batches
    $query = new WP_Query([
        'post_type' => 'post',
        'posts_per_page' => 10,
    ]);

    while ($query->have_posts()) {
        $query->the_post();

        $post_data = [
            'title'       => get_the_title(),
            'link'        => get_permalink(),
            'excerpt'     => get_the_excerpt(),
            'date'        => get_the_date('c'),
            'modified'    => get_the_modified_date('c'),
            'content'     => apply_filters('the_content', get_the_content()),
        ];

        $json_chunk = json_encode($post_data) . "\n";

        // Output the chunk size in hexadecimal, followed by \r\n
        echo dechex(strlen($json_chunk)) . "\r\n";

        // Output the chunk data
        echo $json_chunk . "\r\n";

        // Flush the output buffer to send data immediately
        ob_flush();
        flush();
    }

    // Send the final chunk to indicate the end of the response
    echo "0\r\n\r\n";

    // Clean up
    wp_reset_postdata();
    ob_end_flush();

    exit; // Terminate the script to prevent WordPress from adding any further output
}

Stream 2: XML file

We do the same as the WP endpoint, but on the blog rendering side. We open the second stream to the client while keeping the first stream connection open:

const sitemapStream = new ReadableStream({
    async start(controller) {
      controller.enqueue(
        `<?xml version="1.0" encoding="UTF-8"?>\n` +
          `<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\n`
      );

      for (const url of staticUrls) {
        const staticUrlItem = `
          <url>
            <loc>${url.loc}</loc>
            <lastmod>${url.lastmod}</lastmod>
            <changefreq>${url.changefreq}</changefreq>
            <priority>${url.priority}</priority>
          </url>
        `;
        controller.enqueue(staticUrlItem);
      }

      for await (const post of streamFromWordpress('/stream-posts')) {
        const sitemapItem = `
          <url>
            <loc>${post.link}</loc>
            <lastmod>${new Date(post.modified || post.date).toISOString()}</lastmod>
            <changefreq>weekly</changefreq>
            <priority>0.8</priority>
          </url>
        `;
        controller.enqueue(sitemapItem);
      }

      controller.enqueue('</urlset>');
      controller.close();
    },
  });

  return new Response(sitemapStream, {
    headers: {
      'Content-Type': 'application/xml',
    },
  });

And that’s it. The result is an average reduction of more than 100ms.

Fetching All Posts	Streaming Posts

Table: Fetching All Posts vs. Streaming Posts.

Managing 10k+ Visitors Overnight with Bare Metal

Wed, 13 Nov 2024 00:00:00 GMT

Yesterday, I Don’t Have Spotify reached #2 on Hacker News, getting 700+ points https://news.ycombinator.com/item?id=42110877. This brought in 13k views and over 10k unique visitors, pretty intense for a single day!

What really stood out to me was how seamlessly the stack: HTMX, Bun, SQLite, and Dokku handled this traffic spike. No downtime, no slowdowns, no surprises. Just reliable performance.

This highlights how far efficient, lightweight tools can take you. Instead of over-engineering with costly solutions, this setup handled a massive traffic spike overnight while keeping hosting costs under the price of a coffee per month.

Sometimes, simplicity outshines the hype around “scale-ready” architectures.

For anyone learning about infrastructure or preparing to launch a product, I thought this was worth sharing: scaling isn’t always about Kubernetes or microservices. You don’t need to invest heavily in infrastructure to achieve great results. The bare-metal approach can effectively handle real-world challenges, enabling quick and efficient scaling while keeping costs low. It’s a practical and accessible path to foster innovation.

I Don’t Have Spotify is open source and available here: https://github.com/sjdonado/idonthavespotify. Feel free to explore, contribute, or use it as inspiration for your own projects.

Migrate remix eslint-config to ESLint v9

Sun, 02 Mar 2025 00:00:00 GMT

This weekend I finally found time to upgrade the old deprecated @remix-run/eslint-config in my Remix projects (not yet migrated to React Router v7).

I first attempted to remove the package entirely and adopt the recommended configuration from the official Remix template (https://github.com/remix-run/remix/blob/main/templates/remix/.eslintrc.cjs). However, after running the ESLint migration tool (https://eslint.org/docs/latest/use/getting-started) npm init @eslint/migrate-config, the output was problematic. This was likely because some dependencies weren’t yet compatible with ESLint’s new flat configuration system.

Instead, I opted for a manual approach, installing and configuring each plugin individually. Here’s the outcome:

Install new dependencies

npm install --save-dev @eslint/js @remix-run/dev @typescript-eslint/eslint-plugin @typescript-eslint/parser eslint eslint-plugin-react eslint-plugin-simple-import-sort eslint-plugin-tailwindcss globals tailwindcss typescript-eslint

Changes in the dev dependencies should look similar to these:

   "devDependencies": {
+    "@eslint/js": "^9.20.0",
- "@remix-run/dev": "^2.13.1",
- "@remix-run/eslint-config": "^2.13.1",
+    "@remix-run/dev": "^2.15.3",
@@ -89,16 +89,22 @@
+    "@typescript-eslint/eslint-plugin": "^8.23.0",
+    "@typescript-eslint/parser": "^8.23.0",
- "eslint": "^9.3.0",
+    "eslint": "^9.20.0",
+    "eslint-plugin-react": "^7.37.4",
+    "globals": "^15.14.0",
+    "typescript-eslint": "^8.23.0",
   },

Migrate old config file

Now we can remove .eslintrc.cjs and replace it with the new flat config eslint.config.js:

import jseslint from "@eslint/js";
import reactPlugin from "eslint-plugin-react";
import globals from "globals";
import tseslint from "typescript-eslint";

/** @type {import('eslint').Linter.Config[]} */
export default [
  { files: ["**/*.{js,mjs,cjs,ts,jsx,tsx}"] },
  { languageOptions: { globals: { ...globals.browser, ...globals.node } } },
  jseslint.configs.recommended,
  ...tseslint.configs.recommended,
  reactPlugin.configs.flat.recommended,
  reactPlugin.configs.flat["jsx-runtime"]
];

Optionally, this is how it looks like if you would like to include the eslint-plugin-tailwindcss and eslint-plugin-simple-import-sort plugins:

import jseslint from "@eslint/js";
import reactPlugin from "eslint-plugin-react";
import simpleImportSort from "eslint-plugin-simple-import-sort";
import globals from "globals";
import tseslint from "typescript-eslint";
import tailwind from "eslint-plugin-tailwindcss";

/** @type {import('eslint').Linter.Config[]} */
export default [
  { files: ["**/*.{js,mjs,cjs,ts,jsx,tsx}"] },
  { languageOptions: { globals: { ...globals.browser, ...globals.node } } },
  jseslint.configs.recommended,
  ...tseslint.configs.recommended,
  reactPlugin.configs.flat.recommended,
  reactPlugin.configs.flat["jsx-runtime"],
  {
    plugins: {
      "simple-import-sort": simpleImportSort,
    },
    rules: {
      "simple-import-sort/imports": "error",
      "simple-import-sort/exports": "error",
    },
  },
  ...tailwind.configs["flat/recommended"],
];

You are the easiest person to fool

Sun, 09 Mar 2025 00:00:00 GMT

The first principle is that you must not fool yourself—and you are the easiest person to fool. — Richard Feynman, Cargo Cult Science, Caltech commencement address, 1974.¹

At the heart of Feynman’s warning lies a simple yet profound truth: we humans are remarkably good at deceiving ourselves. Our brains didn’t evolve primarily to discover objective truth, but to help us survive and sometimes that means prioritizing comfortable beliefs over accurate ones.

This self-deception works through everyday mental shortcuts: we naturally seek out information that confirms what we already believe; we expertly weave narratives that make our self-interested decisions seem logical; and perhaps most dangerously, we convince ourselves we’re being objective when we’re anything but. What makes these processes so powerful is that they operate largely outside our awareness.

Feynman recognized that this presents a special challenge to knowledge because we apply different standards to different sources. We’re quick to question claims from others while giving our own thinking process a free pass. This happens because we have insider access to our own motivations; we can craft justifications perfectly tailored to soothe our psychological defenses.

Today, these cognitive blind spots are systematically amplified. Digital platforms optimize for engagement rather than accuracy, creating spaces where our biases are reinforced rather than challenged. Political divisions intensify as we interpret opposing viewpoints through increasingly hostile filters. Scientific questions become matters of identity rather than evidence.

Science itself developed its methods: peer review, replication requirements, statistical controls; precisely as defenses against these human tendencies. Yet these safeguards only work when we acknowledge how easily we can fool ourselves.

Feynman’s insight is not just about good science, it offers a fundamental approach to understanding reality: true knowledge begins with humility about our own mental limitations.

Some remarks on science, pseudoscience, and learning how to not fool yourself. Caltech’s 1974 commencement address. ↩

The Covering Index Paradox

Thu, 10 Apr 2025 00:00:00 GMT

While recently tuning Bit’s performance, I encountered a bottleneck in one highly-selective query SELECT id, url FROM links WHERE slug = (?) LIMIT 1. Initial optimizations, such as replacing UUIDs with rowid and reducing the size of the slug column, significantly improved throughput. However, when I tested a covering index on (slug, id, url), I was surprised to find that it actually slowed down the queries, despite containing all the required columns and theoretically eliminating the need for one extra table lookups. Let’s dive into SQLite’s cost-based optimizer priorities to understand this counterintuitive behavior.

For further context and reproducibility, here it is a separated database with a links table mirroring the original schema, populated with 10000 rows:

CREATE TABLE IF NOT EXISTS links (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    slug TEXT NOT NULL UNIQUE,
    url TEXT NOT NULL
);

WITH RECURSIVE generate_links AS (
    SELECT 1 AS id, 'slug-' || 1 AS slug, 'https://sjdonado.com/link-' || 1 AS url
    UNION ALL
    SELECT id + 1, 'slug-' || (id + 1), 'https://sjdonado.com/link-' || (id + 1)
    FROM generate_links
    WHERE id < 10000
)
INSERT INTO links (slug, url)
SELECT slug, url
FROM generate_links;

SQLite automatically creates sqlite_autoindex_links_1 for the UNIQUE constraint on slug:

test.db> SELECT * FROM pragma_index_list('links') AS il JOIN pragma_index_info(il.name) ON 1 = 1;
+-----+--------------------------+--------+--------+---------+-------+-----+------+
| seq | name                     | unique | origin | partial | seqno | cid | name |
+-----+--------------------------+--------+--------+---------+-------+-----+------+
| 0   | sqlite_autoindex_links_1 | 1      | u      | 0       | 0     | 1   | slug |
+-----+--------------------------+--------+--------+---------+-------+-----+------+
1 row in set
Time: 0.004s

SQLite uses a default page size of 4KB, though this can be configured to any power of two between 512 bytes and 64KB. With the default configuration, each internal B-tree node can store approximately 500 child pointers, creating a substantial branching factor. This branching factor directly determines how many levels (the tree height) are needed to index a given number of rows, approximately log₅₀₀(N) levels for N rows. When querying data, each level traversed requires a separate page read operation, so a lower tree height dramatically improves performance. SQLite’s pager module caches recently accessed pages in memory, but the initial read of each required page still imposes I/O costs that scale with tree height.

Covering Indexes

To execute our query, several sequential operations are performed. SQLite will perform a binary search on the index table using SeekGE, looking for matches with the slug. It will stop there (because of the LIMIT clause, which sets the IdxGT counter) and then access the corresponding table using DeferredSeek with the rowid (IdxRowid) as a pointer:

test.db> EXPLAIN QUERY PLAN SELECT id, url FROM links WHERE slug = 'slug-9345' LIMIT 1;
+----+--------+---------+------------------------------------------------------------+
| id | parent | notused | detail                                                     |
+----+--------+---------+------------------------------------------------------------+
| 4  | 0      | 39      | SEARCH links USING INDEX sqlite_autoindex_links_1 (slug=?) |
+----+--------+---------+------------------------------------------------------------+
1 row in set
Time: 0.005s
test.db> EXPLAIN SELECT id, url FROM links WHERE slug = 'slug-9345' LIMIT 1;
+------+--------------+----+----+----+-----------+----+---------+
| addr | opcode       | p1 | p2 | p3 | p4        | p5 | comment |
+------+--------------+----+----+----+-----------+----+---------+
| 0    | Init         | 0  | 13 | 0  | <null>    | 0  | <null>  |
| 1    | Integer      | 1  | 1  | 0  | <null>    | 0  | <null>  |
| 2    | OpenRead     | 0  | 2  | 0  | 3         | 0  | <null>  |
| 3    | OpenRead     | 1  | 3  | 0  | k(1,)     | 2  | <null>  |
| 4    | String8      | 0  | 2  | 0  | slug-9345 | 0  | <null>  |
| 5    | SeekGE       | 1  | 12 | 2  | 1         | 0  | <null>  |
| 6    | IdxGT        | 1  | 12 | 2  | 1         | 0  | <null>  |
| 7    | DeferredSeek | 1  | 0  | 0  | <null>    | 0  | <null>  |
| 8    | IdxRowid     | 1  | 3  | 0  | <null>    | 0  | <null>  |
| 9    | Column       | 0  | 2  | 4  | <null>    | 0  | <null>  |
| 10   | ResultRow    | 3  | 2  | 0  | <null>    | 0  | <null>  |
| 11   | DecrJumpZero | 1  | 12 | 0  | <null>    | 0  | <null>  |
| 12   | Halt         | 0  | 0  | 0  | <null>    | 0  | <null>  |
| 13   | Transaction  | 0  | 0  | 6  | 0         | 1  | <null>  |
| 14   | Goto         | 0  | 1  | 0  | <null>    | 0  | <null>  |
+------+--------------+----+----+----+-----------+----+---------+
15 rows in set
Time: 0.005s

One can optimize query operations by avoiding table lookups through storing the url value in a single concatenated key structure. This is possible due to all columns of a composite index are stored in both B-tree internal nodes and leaf nodes, ensuring the key is sorted on a left-to-right basis. This allows efficient handling of various data distributions without performance degradation, as long as the data follows a deterministic sorting order.

Outcome

Following that optimization idea, let’s introduce a covering index for our table:

 CREATE UNIQUE INDEX IF NOT EXISTS idx_links_slug_covering ON links(slug, id, url);

test.db> SELECT * FROM pragma_index_list('links') AS il JOIN pragma_index_info(il.name) ON 1 = 1;
+-----+--------------------------+--------+--------+---------+-------+-----+------+
| seq | name                     | unique | origin | partial | seqno | cid | name |
+-----+--------------------------+--------+--------+---------+-------+-----+------+
| 0   | idx_links_slug_covering  | 1      | c      | 0       | 0     | 1   | slug |
| 0   | idx_links_slug_covering  | 1      | c      | 0       | 1     | 0   | id   |
| 0   | idx_links_slug_covering  | 1      | c      | 0       | 2     | 2   | url  |
| 1   | sqlite_autoindex_links_1 | 1      | u      | 0       | 0     | 1   | slug |
+-----+--------------------------+--------+--------+---------+-------+-----+------+
4 rows in set
Time: 0.002s

Then, with the new index available, and after performing the same query again, we notice that it was not chosen by SQLite’s query planner:

test.db> EXPLAIN QUERY PLAN SELECT id, url FROM links WHERE slug = 'slug-9345' LIMIT 1;
+----+--------+---------+------------------------------------------------------------+
| id | parent | notused | detail                                                     |
+----+--------+---------+------------------------------------------------------------+
| 4  | 0      | 39      | SEARCH links USING INDEX sqlite_autoindex_links_1 (slug=?) |
+----+--------+---------+------------------------------------------------------------+
1 row in set
Time: 0.005s

We can force the query to use our covering index, but there is no notable improvement:

test.db> EXPLAIN QUERY PLAN SELECT id, url FROM links INDEXED BY idx_links_slug_covering WHERE slug = 'slug-9345'
          LIMIT 1;
+----+--------+---------+--------------------------------------------------------------------+
| id | parent | notused | detail                                                             |
+----+--------+---------+--------------------------------------------------------------------+
| 3  | 0      | 40      | SEARCH links USING COVERING INDEX idx_links_slug_covering (slug=?) |
+----+--------+---------+--------------------------------------------------------------------+
1 row in set
Time: 0.005s
test.db> EXPLAIN SELECT id, url FROM links INDEXED BY idx_links_slug_covering WHERE slug = 'slug-9345' LIMIT 1;
+------+--------------+----+-----+----+-----------+----+---------+
| addr | opcode       | p1 | p2  | p3 | p4        | p5 | comment |
+------+--------------+----+-----+----+-----------+----+---------+
| 0    | Init         | 0  | 12  | 0  | <null>    | 0  | <null>  |
| 1    | Integer      | 1  | 1   | 0  | <null>    | 0  | <null>  |
| 2    | OpenRead     | 1  | 931 | 0  | k(3,,,)   | 2  | <null>  |
| 3    | String8      | 0  | 2   | 0  | slug-9345 | 0  | <null>  |
| 4    | SeekGE       | 1  | 11  | 2  | 1         | 0  | <null>  |
| 5    | IdxGT        | 1  | 11  | 2  | 1         | 0  | <null>  |
| 6    | IdxRowid     | 1  | 3   | 0  | <null>    | 0  | <null>  |
| 7    | Column       | 1  | 2   | 4  | <null>    | 0  | <null>  |
| 8    | ResultRow    | 3  | 2   | 0  | <null>    | 0  | <null>  |
| 9    | DecrJumpZero | 1  | 11  | 0  | <null>    | 0  | <null>  |
| 10   | Next         | 1  | 5   | 0  | <null>    | 0  | <null>  |
| 11   | Halt         | 0  | 0   | 0  | <null>    | 0  | <null>  |
| 12   | Transaction  | 0  | 0   | 6  | 0         | 1  | <null>  |
| 13   | Goto         | 0  | 1   | 0  | <null>    | 0  | <null>  |
+------+--------------+----+-----+----+-----------+----+---------+
14 rows in set
Time: 0.007s

The main difference in this last query is that instruction 7, Column retrieves the slug value directly from the index. Since DeferredSeek is no longer required, there is one operation less. However, the query is still slower. Why?

The answer lies in the page size and density difference between these index types. Despite both indexes having the same B-tree height for our 1000-row dataset, the covering index requires triple the storage space (18 pages vs 6 pages). This directly translates to increased I/O operations during query execution.

Index	Entry Size	Pages (N=10000)	B-tree Height	Entries/Page	Complexity	Page Reads
`rowid`	-	-	1	-	O(1)	1
`sqlite_autoindex_links_1`	24 bytes	60	3	~166	O(log N)	2-3
`idx_links_slug_covering`	72 bytes	182	4	~55	O(log N)	4-5

For larger datasets, this density disparity would likely increase the B-tree height of the covering index before the autoindex, creating an even larger performance gap. The covering index approach only becomes advantageous when the cost of the additional table lookup DeferredSeek exceeds the cost of traversing a taller, less dense B-tree structure, typically with very large tables or when retrieving multiple wide columns that would make table lookups particularly expensive.

What a paradox, one expect a denser index is expected to be more efficient, yet it can lead to a taller B-tree and potentially worse performance.

Consistency isn’t repetition

Sun, 13 Apr 2025 00:00:00 GMT

“Tomorrow, I’ll wake up at 06:00, meditate, work out, and finally crush those overdues in my Todo list”. That’s what I often tell myself, but when tomorrow arrives, reality throws a wrench into my well-laid plans, and I’m left wondering why I can’t stick to my script.

The mismatch between plans and reality lies in the way that life operates, much more like a relay than a solo run, where each day, a new runner (“today-you”) takes the baton, you are not the same person you were yesterday, but a subtly different version with fresh energy, priorities, and patience.

This isn’t about laziness. It’s about our evolutionary wiring for instant gratification, our brains prioritize short-term rewards over long-term gains¹. We binge Netflix tonight, aware future-us will pay the price, but tomorrow’s discomfort feels abstract today. This isn’t a moral failure, it’s biology.

Consistency isn’t repetition, it’s continuity.

Think of momentum not as a linear path, but as a game where every small effort unlocks future advantages. Instead of demanding identical effort daily, focus on leaving momentum for the next iteration of you. Skipped the gym? Do five push-ups. Overwhelmed by work? Write one sentence. These aren’t compromises, they’re lifelines. Like saving progress in a video game, they ensure tomorrow’s player doesn’t reboot at level one.

If you feel disconnected from your future self, consider this: small investments build trust in your capacity to grow. Reframe future-you as a family member, you wouldn’t abandon a sibling to chaos. Meal prepping, clearing your desk, or evening journaling aren’t tedious chores. They’re strategic passes to a teammate you’re rooting for, even if you’re still building a relationship with them.

You don’t have to win the whole game today, just set the next player up for an easier round. A failed attempt teaches tomorrow’s self what not to repeat. A half-finished task is a foothold for progress. Action, even messy action, is data, not the final verdict.

Play your turn fully, savor the moment, and then hand off the controller. Momentum compounds when today’s moves, no matter how small, lay the groundwork for tomorrow’s success.

Uncovering the Neural Basis of Resisting Immediate Gratification while Pursuing Long-Term Goals ↩

No Freemium. Not for Free. Free

Sat, 02 Aug 2025 00:00:00 GMT

The freemium model is marketing, a clever funnel. Users get to play with the product until they hit the walls: a storage limit, an export button grayed out, or a 14‑day timer ticking down. That frustration is intentional, it turns “just curious” into “just paid”. And that revenue does not just cover the cost of those extra features, it props up an MVP weighed down with tech debt, growing infrastructure bills, and the payroll needed to keep chasing growth.

That is the real game, growth, not craft. New features to attract new users, even if the foundation cracks. The goal is not to make it sustainable, it is to make it bigger, to chase the next‑Google dream, or at least look valuable enough for an exit.

Free software comes from a different impulse. You build it for yourself first. You polish it because you rely on it. You craft it because creative minds hate to leave rough edges unshaped. And when you open it up, you are not giving away charity, you are inviting the world to help make it even better.

If you are proud of your work, you want to see it used, you want to see it free. There is nothing better than watching your solution live in the wild. More users bring more edge cases, and each one nudges the software into something great. That kind of refinement only happens when the backbone is built carefully, without the rush of quarterly deadlines, when there is room to focus on the invisible parts, the ones that make a tool last.

Quieter Social Networking

Wed, 29 Oct 2025 00:00:00 GMT

I’ve been gradually stepping away from doomscrolling for a while now. I got rid off my Facebook, Instagram, TikTok, and Twitter accounts. Plus, their domains are blacklisted through Screen Time’s web content restrictions, so it’s impossible for me to open any related links in the browser or via web viewers.

However, I kept YouTube and LinkedIn because they offer content I intentionally want to see. The problem is that whenever I access them, I’m at risk of falling into recommendation algorithm traps, and I’m left with nothing but my willpower.

Until last week, when I finally found a solution by reading Herman’s post, Smartphones and being present. The post shows how to use uBlock Origin filters to remove elements from the DOM, which means I can access a clean version of the app through the web browser. For example, after uninstalling the YouTube app, when I go into Safari, I see a customized version: No home feed, no short videos, and no suggested videos.

The ruleset I currently have on quieter-social-networking-custom-filters.txt looks like this:

For more filters to declutter the web with uBlock, I highly recommend this repo: github.com/yokoffing/filterlists.

Besides that, I thought about replacing Twitter with Bluesky but realized it wouldn’t change the habit. It still wouldn’t let me choose when to be informed instead of being flooded by constant headlines. So I picked the recently launched Kagi News, the old friend Hacker News and a few RSS blogs curated by myself. Together, they offer a focused, manageable view of the world without hijacking curiosity. When I want to explore, I do it deliberately, not through a feed.

With this minimal setup of content restrictions, browser-only use, and custom uBlock filters, I’ve found a quieter way to be present online. I hope it helps others do the same.

The tools changed because the job changed

Thu, 12 Feb 2026 00:00:00 GMT

By the end of 2022, I had one clear goal: be faster. I had lots of ideas but struggled to quickly prototype them; my code editor was a big part of the problem, tediously slow and eating all my RAM. Neovim was the answer, typing as fast as thinking. Plus, my laptop was flying, just one terminal tab, a handful of TUIs, nothing else.

I spent the next years writing Lua scripts to shape my workflow. It was fun, and I valued customization more than the maintenance cost. It worked, until the job outgrew it. Growing in my career meant more code reviews, more discussions, more jumping between branches. git stash falls short when you are juggling three features and a hotfix, it introduces the same kind of barrier as leaving the terminal. No amount of keymaps can fix a workflow problem.

Git worktrees solved it. One worktree per branch, each pinned to a tmux session. Switching context became switching panes. The friction dropped and I could finally collaborate at the pace the role demanded.

[!NOTE] 2022 setup: tmux, Neovim

Somewhere along the way AI went from an assistant to the thing that writes most of the code. That changed the job more than any tool swap ever did. I spend less time in my editor and more time in the planning phase, whiteboarding, sketching interfaces, writing specs before touching a file. Reviewing other people’s work still takes the same effort, maybe more, because AI-generated code is fluent but not always thoughtful. The bottleneck moved upstream, and I think that is true for everyone, not just me. The kind of thinking that used to be reserved for staff engineers, system design, trade-off analysis, deciding what not to build, is quietly becoming the baseline expectation at every level.

So I adjusted my tooling. Zed won me over with its first-class vim motions, and it stuck. It is the fastest editor experience I have found outside a TUI. Their ACP simplifies working across multiple worktrees. My workflow now looks like this: manage sessions in tmux, open Zed and inside it launch the agent chat and the integrated terminal. Once I am done with a branch, I close the tab. Multiple project tabs in one window means no switching between editor instances.

I also mapped every app I use daily to a hotkey in Raycast, all following the same pattern: <leader> + one letter. Zed: <leader>;. Helium: <leader>h. Terminal: <leader>/. Slack: <leader>s. No transition animations, just jump where I need to. Raycast’s clipboard manager is a delightful tool, I save entries as snippets and rely on them constantly.

[!NOTE] 2026 setup: workspace, Zed, Raycast

Cannot deny I feel nostalgia when I open my dotfiles, I still use Neovim to edit them. But that trade-off no longer holds. I do not need to fine-tune keymaps, chase LSP edge cases, configure Treesitter grammars, or wire up DAP. The tools I care about now are the ones that let me think, not the ones that let me type.

Looking back, the pattern is simple. First I optimized for typing speed, then for context-switching speed, now for decision-making speed. Each time, the tools changed because the work changed first. AI did not make the coding part disappear, it just made the engineering part around it much bigger. The tools followed, as they always do.

Juan Rodriguez Donado - sjdonado

Self-Hosted Password Manager with Dokku

Why?

Prerequisites

Step by Step

Wrapping up

HTMX with Bun: A Real World App

Context

A Real World App

Wrapping up

Fast, Intuitive, Smart: Restaurant search engine with Cloudflare AI

The Inspiration Behind “Cule filo”

Our Development Journey

AI Models in Action

To wrap up

Validated forms with useFetcher in Remix

The FetcherForm Provider

The useFetcherForm hook

Demo

Building a Fast and Compact SQLite Cache Store

A cache-manager Store

The Queries

Auto Purge and Batch Operations

Conclusion

Should we leave behind the joy of curating our own content?

Tower of Hanoi in P5.js + WASM

Vanilla JS Implementation

wasm-pack

Hanoi Algorithm in Rust

Web Worker

In Summary

Sitemap generation by streaming from WordPress headless

Managing 10k+ Visitors Overnight with Bare Metal

Migrate remix eslint-config to ESLint v9

You are the easiest person to fool

Footnotes

The Covering Index Paradox

Covering Indexes

Outcome

Consistency isn’t repetition

Footnotes

No Freemium. Not for Free. Free

Quieter Social Networking

The tools changed because the job changed