Nick Winans

Attention Span August

Thu, 31 Jul 2025 00:00:00 GMT

This August, I'm taking a month-long break from social media and recommendation algorithms^[1]. This is a personal challenge to see how it affects my attention span, productivity, and overall mental health.

Why?

We all know that social media and their recommendation algorithms are made to keep us engaged for as long as possible. They do a really good job at it, and I often find myself scrolling mindlessly, every once in a while becoming self-aware and wondering why I just wasted a minute watching a clip of a show I've never even heard of.

Six months ago, I installed ScreenZen, an app designed to stop the doomscrolling and give you back your time and attention. I created a 15 minute limit per app per day, and it helped me reduce my screen time immensely. At the same time it's created a new habit and feeling that somehow might feel even worse.

Now that I have a set limit per day, I need to use every minute to its fullest every day. During an app session there's no going to search up something that piqued my interest; I can't waste my precious seconds on that. Video or post not interesting enough? Swipe away fast to get to the next thing that may be more worth it. It's become a chore of maximizing as much entertainment as I can. And I feel anxious going through this ritual every day.

Why should I continue this?

What will I do instead?

I want to spend more time creating things. I find more fulfillment in the things I've made. When I look back on years prior, I don't think I could tell you about one truly meaningful TikTok or Twitter post I saw. What I create is what comes to mind, and it defines my own self-image. I'm going to write more and build more things to nurture that.

Of course, life can't be all output. My consumption needs a change of pace, too. Embarrassingly, I'll admit that I finally read a book in full for the first time since middle school this month. And it was great! I want to read more books and watch more shows (yes, even The Wire). I'll actually have a chance at remembering them in a few years.

Join me!

If you've ever felt that same anxiety, that sense of time wasted on digital chores, consider this your sign.

I'll likely keep YouTube, but I'll disable the home page and recommendation bar. I'll only watch a few subscriptions and videos I find externally or specifically search for. ↩︎

Building My Perfect Keyboard — The Plan

Tue, 29 Jul 2025 00:00:00 GMT

I use a wireless Lily58 (a split, ergonomic keyboard) at work, and it's great. I love the layout, the slim profile, and the long battery life. Recently, though, I built a Photon 65% keyboard from CannonKeys, and it's made me realize how hollow my work board sounds and feels.

It's time to change that!

Sound Comparison

Before I get into the details of my plan, I want to share a sound comparison to illustrate where I'm starting from and what I'm aiming for. I've recorded the sound of my current Lily58, a Corne with an aluminum case, and the Photon.

Here's how they sound:

Board	Sound Test
Lily58 (Current)	<audio controls preload="metadata" src="./lily58.mp3"></audio>
Corne Premium	<audio controls preload="metadata" src="./corne.mp3"></audio>
Photon (Target)^[1]	<audio controls preload="metadata" src="./photon.mp3"></audio>

I hope you can hear a big difference. The Lily58 is hollow and the switch stabilizers rattle a lot. The Corne certainly sounds a bit more solid, but the switches are still shallow in sound profile. Then we have the Photon, wow! The sound is deep, crisp, and satisfying. It's got thock!

The Plan

The first step is going to be picking the switches. The existing Kailh Choc v1 switches just don't sound or feel that great. This isn't surprising to anyone who's tried them. At the same time, the full-size MX-style switches in the Photon are too tall for a small ergo board like the Lily58, and they would make for a disproportionately tall build.

As a middle ground, both the Choc v2 and the Gateron KS-33 (or Low Profile 2.0) switches are the clear options. Both are decently low-profile, but they should provide a better sound and feel than the Choc v1 switches. While I don't think we'll get to sounding as good as the Photon, we'll do our best.

The Choc v2 switches can sound pretty good. I know a couple of people with the Lofree Flow, and I'm pretty impressed with how the all POM Choc v2 switches sound. The KS-33 on the other hand are recommended more often, so I'm willing to give them a try, too.

Great switches are only one part of the equation. After picking my switches, I'll be designing a new aluminum chassis, which will give us a solid feel and deeper sound than the existing thin plastic case.

A new PCB will also be part of the design. No working around a separate microcontroller board like the nice!nano. By integrating the wireless controller directly onto the board, the final build can be slimmer and more cohesive.

I'm excited.

What's Next?

I've gone ahead and ordered a Gateron KS-33 switch tester and every Choc v2 switch I could find. In the next post, I'll share my thoughts on the switches and which ones I choose.

I want to share every step of the process of building this keyboard. My goal isn't just to show off the final product, but to help you understand how each part fits together to create the best possible keyboard (for me, at least!).

It's pretty important to note that this is using TTC Venus switches, GMK keycaps, TX "AP" Stabilizers Rev. 4, and case foam. All of these make a difference to the sound profile. ↩︎

I Made a Million Dollar Product from My Dorm Room

Sun, 23 Mar 2025 00:00:00 GMT

This post shares the story of the nice!nano; a wireless, Pro Micro-compatible microcontroller board I made in my freshman year of college. The nice!nano powers tens of thousands of keyboards, has inspired many, and changed my life.

Over my first winter break in college, I created what I called the Dissatisfaction65, a wireless 65% keyboard inspired by the Satisfaction75. I don't remember exactly why, but I wanted to try making a DIY wireless keyboard after having made a few wired ones. The Adafruit 32u4 Bluefruit LE microcontroller was used to accomplish wireless since the open-source QMK keyboard firmware supported Bluetooth with this specific board. The project looked great in the end, but its performance was awful. The typing latency was nearly unusable, and it only lasted a few days on battery even with a huge battery inside.

Seeing all the low-latency, long battery-life wireless products from companies like Logitech and Apple, I knew that something better was possible. In the next two months I dove into the world of wireless microcontrollers and DIY keyboards. I quickly learned that Nordic microchips were the hobbyist's choice and the Pro Micro format reigned as king for DIY keyboards. In my search I discovered three microcontrollers trying to fill the gap between the two: the BlueMicro, the nRFMicro, and the BLE-Micro-Pro.^[1]

Board	Retail Cost	Form Factor	Open Source
BlueMicro	N/A	Too Large	Yes
nRFMicro	N/A	Yes	Yes
BLE-Micro-Pro	~$40	Yes	No

The BlueMicro's form factor meant that I couldn't build most Pro Micro keyboards since it would interfere. The BLE-Micro-Pro was pretty expensive, locked down, and only sold in Japan. The nRFMicro was pretty close. At first, I decided to modify the nRFMicro to fit my needs, but I soon realized my goals were a bit too ambitious, so I restarted from scratch.

The nice!nano was born

The weekend (yes, the whole thing was designed in a weekend) I created the nice!nano, I don't think I left my desk for more than sleeping and getting food from the dining hall maybe three times. It was just me, KiCad, Nordic's Infocenter^[2], nRFMicro wiki, and the Adafruit nRF52840 Feather schematic. I put together the schematic and BOM, laid out the PCB, and routed (and re-routed) the connections. On the other side I came out with the thinnest Pro Micro compatible nRF52840 based board.

Over the next week I created a name and found my PCB assembler. The name is based on my online username, "Nicell". I wanted to continue the spirit of metric naming of the Pro Micro and came up with "nice!nano". The stylized lower-case pixel font mark was created to sit atop the antenna. After reaching out to a few assemblers, the cheapest option for producing five was about $100. That was a lot to spend on what could've easily been a broken design, but after a few days of meticulously re-reviewing my designs, I paid.^[3]

A few weeks later the boards showed up at my door. I was both ecstatic and terrified they wouldn't work. As I plugged in my first one I closed my eyes, tensed up, and peeked. To my surprise and relief, they worked! Over the next couple of weeks I built a Lily58 with them and got a modified version of QMK working on it. In my testing I found the board could last a few weeks on a 110mAh battery. When comparing to the Dissatisfaction65 that lasted a few days on a 2,500mAh battery, we were looking at over a 100x improvement in power efficiency. I was elated, and I posted my fully wireless Lily58 on Reddit and it got quite a bit of interest.

Over the next few weeks my tiny Discord grew into a sizable community focused on wireless keyboard innovation. I launched an interest check for a group buy, made a few more refinements of the nice!nano, and then I was ready to launch the group buy in mid June.

Group buys are awful

As a college student, I didn't have the money to bank roll a purchase of 1,000 nice!nanos, so I ran a group buy pre-purchase. At the time I had set a minimum purchase amount of 200 pieces for the order to go through and a maximum of 1,000 because I didn't think I could handle more than that. I set the end date for a month later. In the end, it wasn't even open for a day.

The sale went live on June 20th at 11am central. Within the first few minutes I had met my minimum purchase amount. I remember sitting in my childhood bedroom (thanks covid) on the Shopify dashboard watching orders pour in. It was an incredible feeling. Within just seven hours all 1,000 nice!nanos had been sold, ending the group buy. In the next two months I got all the product in and shipped out the 400+ unique orders with the help of my family.

My mom posted about the fulfillment process on Facebook. It was a family effort!

With the success of the group buy, you might wonder, what's so awful? Well, it was extremely stressful holding on to so many people's money without a physical product to back it yet. Along with PayPal holding half the funds of the group buy for a while, it was a bit terrifying. At the same time, group buys have caused the mechanical keyboard community a lot of strife with stolen funds and extremely delayed projects. When I see well-established stores that should have capital running group buys, I can't help but shake my head. I decided shortly after this that I will never run a group buy again.

ZMK

Rewinding back a couple of months, as I was waiting for group buy product to come in, there was still a fairly major part of the ecosystem missing: decent firmware. I bounced between different existing options unsatisfied with the result. That was until I was connected with Pete Johanson who coincidentally had started working on a wireless keyboard firmware powered by the modern Zephyr RTOS.

I quickly sent some pre-production units to Pete to mess around with. Shortly after, he got an early version of ZMK working on the nice!nano, and we hit the ground running building a new wireless-first firmware with a low-power focus. By early 2021 a small community led by Pete had created an extremely performant and feature-full wireless firmware.

Settling in

In 2021 I really settled in to my new small business. My vendor network was growing around the world, nice!nanos were flying off the shelves and hard to keep in stock, and the ZMK community continued to grow and strengthen. Other popular ZMK boards started to pop up around here with lots of them inspired by the nice!nano, or they were at least using my schematic, which I released publicly.

Everything was looking great, but I noticed that most of my vendors did not carry all the parts needed for a wireless build, or that their builds focused on wired microcontrollers. I figured I could bring something new to this area.

Becoming the vendor

As a full time student, I knew I couldn't run a whole ecommerce store easily by myself. Luckily, my parents decided to retire at the end of 2021, and my dad was saying he needed something to keep himself busy. Together in 2022 we started Typeractive, a keyboard store focused on the wireless keyboard experience.

I created a 3D interactive configuration tool for people to get all the parts they needed and kits specially designed for wireless boards. This low friction experience was a huge success, and now in 2025 we're one of the largest split keyboard stores. There's a lot more that happened with Typeractive, but I can tell that story another time.

Cloned, twice!

In 2023 the nice!nano was cloned, not once, but twice. Two different designed copies popped up on Taobao, and it wasn't long before they ended up on AliExpress and even on my existing vendors' stores. I was a bit shocked by this, and in the end I found I couldn't do much about this.

To be clear, these are clones. I think competition is fair, but both of these new boards that popped up are advertised as nice!nanos and are shipped with the exact same firmware I use on the nice!nano, so when someone plugs it in, it says it's a nice!nano. If the manufacturers would have just built their own firmware (it's open source!) and not used the nice!nano in the title of their listings, I would say it's fair game.

Seeing my product get cloned gave me mixed feelings. As everyone knows, imitation is the greatest form of flattery, but seeing them ride the coattails of my work was frustrating. At the end of the day though, their product is subpar, and nice!nanos continue to sell at a consistent rate. Some of that is likely due to the largest DIY wireless keyboard store not stocking them. Thanks, Typeractive!

The million dollar product

Ok, so the title is a bit of click-bait, but if you made it this far, I suppose it worked. The nice!nano might have been designed in my dorm room, but this was a multi-year journey. To date, over 50,000 nice!nanos have been sold at various online retailers around the world representing over a million dollars in sales. It's hard to wrap my head around still, and I'm extremely grateful. While I put in a lot of hard work, I also recognize that timing and luck played a significant role. The growing interest in wireless keyboards and the lack of available options in the DIY space created the perfect environment for the nice!nano to thrive.

Creating this post has been an incredible trip down memory lane. The nice!nano has had immeasurable impact on my life, and it only happened thanks to so many people that helped along the way. In semi-chronological order, I'd like to shout out individuals that helped me immensely:

Joric (creator of the nRFMicro)
Pierre Constantineau (creator of the BlueMicro board and firmware)
Pete Johanson (creator of ZMK)
Mike and Pam (my parents)

Thank you. It's been incredibly rewarding to see all the custom keyboards built with or derived from the nice!nano. The community is still growing, and I'm glad that the nice!nano gets to be a big part of it.

I've purposefully left links to each repository in the state I would have been reading them back in early 2020. ↩︎
As I wrote this, I found out that Nordic's Infocenter was shut down, RIP. ↩︎
I sometimes laugh at how scary that $100 purchase was for me at the time. All things considered, this is an extremely cheap R&D investment. ↩︎

eCommerce Automations

Sun, 23 Feb 2025 00:00:00 GMT

I'm the owner of Typeractive.xyz, a DIY ergonomic keyboard kit store. We currently average 30 orders a day with 10 items per order. This is relatively high volume considering we only have one person full-time working on the store.^[1]

As opposed to other stores in the ergonomic keyboard kit space, we are rarely out of stock of our 250+ product variants, dispatch >99% of eligible orders in one working day, and respond to all customer requests in an average of five hours.^[2]

These achievements are only possible thanks to heavy automation of our daily operations, and after three years of running our store, I'd like to share more details about which small automations have created the biggest impact.

Disclaimer: Our store runs on Shopify, so lots of the solutions are centered around some tools that are only available with Shopify.^[3]

Staying in Stock

After expanding the store to more variants, we grew quickly and immediately ran into a problem every store owner wishes for: how do we keep our products in stock?

Reorder Notifications

If I had to get rid of every other automation and keep just one, it's this. Every day a Cloudflare Worker calculates how many days of stock remain based on past sales, creates a list of products that need to be reordered, and posts it to our team's internal Discord.

Something like this can be implemented in many different ways; here's some more details on ours.

Different products take different amounts of time to order and receive. We use Shopify Metafields to assign a "Days to Reorder" number to every product.
Quantity sold per day is found using Shopify's "Days of inventory remaining" report.
Once ordered, we create a Purchase Order in Shopify, which causes that product to fall off our reorder notification keeping it neat and tidy.
Neither Shopify's reports nor purchase orders are supported via their API. We access them programmatically by forging a Shopify admin client GraphQL request. You can learn more about forging requests from my friend David. Shopify, please add these to your API.

I've considered packaging this up as a simple Shopify App, but I haven't gotten around to it. If this might interest you, let me know!

3D Printing Dashboard

Our store also sells some 3D printed products we produce in-house. These products don't have significant lead time, but we can only produce a small amount each day using our three BambuLab printers.

Our existing reorder notification didn't work as well since we should be continually printing and it would cause significant clutter considering we have over 100 3D printed variants. Instead, we have a dashboard showing which variants to prioritize printing.

Once again, we use Metafields to define a per-variant "Desired Stock" value. We know some colors are more popular, so we want to have more ready to go. With this dashboard, instead of blindly scrolling through >100 variants figuring out what to print, we can just find which variants have the lowest desired stock level (percent). This reduces the amount of time it takes to start another print, which requires stepping away from fulfilling orders.

This dashboard in particular is generated using Retool and Shopify's Admin GraphQL API. Retool works well, but I don't think it's incredible, and there's probably lots of other options out there. It's a shame Shopify's new "Explorations" in Reports don't appear to support inventory queries like this in any significant capacity.

Back in Stock Notifications

As a last resort, if we can't keep items in stock, we have back in stock notifications. These are pretty standard. The customer signs up for notifications and receives an email when they're back in stock.

Except I'm a software engineer, so of course I looked at existing solutions and thought, "I can do that better and cheaper". As is usually the case, that was only true if my time wasn't taken into account.

This was all built in a weekend and has cost $0. Here's how it's built.

A Firebase Firestore stores the requests with a timestamp, product variant ID, and an email.
When a variant is back in stock, Shopify Flow sends a request to a Firebase functions.
The function finds all unsent requests in Firestore and batch submits email documents using Firebase's Trigger Email extension.
Those emails are sent using our Google Workspace SMTP server.
Templates are generated using MJML and stored in Firestore as well. JSX Email looks like another good option.

If I were rebuilding this today, I'd probably use Cloudflare and Amazon SES. Firebase functions are unbelievably slow to deploy. At the time I built this, Cloudflare Email Workers weren't yet available.

Dispatching on Time

To dispatch our 30 orders a day on time, we need to save as much of Jack's (our full-time warehouse worker) time as possible.

Packing Slip Ordering

Our warehouse is laid out with our products along the walls in cubbies, stackable bins, and drawers. Normally, when a Shopify packing slip is printed out, the order is based on the order that the customer added the items to the cart. This isn't very helpful.

For the final time in this post, I will once again point to Metafields. For every product variant, we set a five-digit integer, which orders it in our packing slip. Note that the sorting is alphabetical, so make sure to use all five digits.

After setting up the Metafields we can add a few lines to our packing slip liquid template that will order everything how we want it.

+{% assign packing_orders = order.line_items | map: 'variant' | map: 'metafields' | map: 'custom' | map: 'packing_order' | compact | sort | uniq %}
+{% for packing_order in packing_orders %}
{% for line_item in order.line_items %}
+{% if line_item.variant.metafields.custom.packing_order == packing_order %}

Finally, some code! We first get every unique line item packing order, and then we iterate through each packing order and each line item and only print out when the packing order matches. This is definitely inefficient, but it's fast enough and works well with liquid.

Packing Slip Notes

When filling an order, it's helpful to know the shipping method and box size. This informs both what box to use (size and whether we use a USPS Priority Mail box) and how things should be packed (international orders may require things like bubble wrap).

To add the shipping method and box size to the order, I use Shopify Flow to update the note.

Whenever an order is created, we bucket the order into a box size by total shipment weight and then extract the shipping method.^[4] Then the order note is set.

Box Inventory

I'm really surprised that Shopify nor our current shipping platform support box inventories. Keeping track of box inventory is tedious and can cause huge delays in shipping if you forget to order or check on stock levels. We have all the data, yet it's not tracked!

To track this, when we complete a shipment on our shipping platform, a webhook hits a Firebase function endpoint with the shipment payload. Using this payload, we use the box name to log a use of a box and update our inventory in a Firestore.

Once more we create a Retool dashboard that calculates the average box usage per day and tells us the amount of days left. At the end of every day, Jack takes a look at this dashboard and if any is under 30 days, we order more boxes and leave a note.

Maybe at some point we can integrate this with our reorder notifications, but this works for now.

Multiple Order Flagging

Often times, customers forget an item and end up creating multiple orders in one day. We can save a lot on shipping by combining these orders, but manually going through the orders every day is time consuming.

Amazingly, Shopify has a Flow template exactly for this, and it works great! Not much else to say on this besides it's awesome.

Quality Support, Fast

Admittedly, automating customer support is challenging and can result in a poor customer experience. Because of that, we don't actually automate much of our customer service. Instead, we try to automate everything else as much as possible to give Jack and Mike the time they need to properly handle requests.

We also try to offer our customers as much documentation as possible to prevent unnecessary customer service requests. There's still a few things we do to lighten the burden.

Discord and an AI Bot

We have an incredible Discord community. Members help each other, share their builds, and have created a true community. When we ship our kits, we include a business card with a link to both our extensive documentation as well as our community Discord.

Once a customer gets to Discord, they can create a post in our help forum. When they do this, they're greeted by an AI bot that has our entire documentation as a system prompt.^[5] While the AI bot does not resolve all submissions, it provides instant acknowledgment and starts the support process.

Mike and Jack monitor the Discord and will help how they can, but often the technical questions aren't in their wheelhouse. Other community members graciously help out, otherwise the questions will be forwarded to me, and I'll help as soon as I can.

NotebookLM

This is a recent addition to our toolbox, but NotebookLM acts similarly to our Discord AI bot while being very flexible. We can create new notebooks, drop in documentation, data, customer feedback, etc and then get insights or ask questions about that.

I used this to analyze the results of our recent customer feedback giveaway, and the results were really impressive. I read every response myself, but the summary was spot on and I was able to share it with the team.

Closing Thoughts

It's been fun to be a storefront owner and a software engineer. There's so many opportunities to spend a little time to make things efficient for our exact use case. It's felt like a superpower that's put us ahead of our competition.

This just scratches the surface of the technology that powers our store. There's some other interesting pieces like our the 3D customizer tool (our main differentiator), our shipping integrations, and our B2B vendor order automations. I may write about these in the future, but they don't feel immediately useful to others.

I hope you found this peek into our operations useful or at least interesting! What automations have you found most helpful in your business? I'd love to hear about them.

I also have a full-time software engineering job, so I only spend a few hours a week on the storefront. Jack is our only full-time, in-person employee. We have a few part-time people working ~5 hours a week for support and item pre-packaging. ↩︎
We actually achieve a P25 response time of just twelve minutes. Jack and Mike are incredible. ↩︎
I often see stores spend a lot of time setting up alternative self-hosted storefronts like WooCommerce, Vendure, Medusa, etc. While I'm a huge advocate for self-hosting software (lots of our internal tools are self-hosted), I would highly recommend sticking to Shopify. Your main store expense, by far, will end up being payment processing. Shopify, at least in the US, is extremely competitive on payment processing, and the small monthly fee easily pays for itself in terms of stability, global speed, API and custom app capabilities, Shopify Flow, and more. ↩︎
Thank you Shopify for adding the "Run code" block! This used to be a massive if/then/else flow tree. I can see it potentially replacing even more of my Firebase functions and Cloudflare Workers if they continue to expand its capabilities. ↩︎
Google Gemini is slept on. I'm using the Flash 2.0 model, and it's one of the few models that actually seems to take the entire long context into account and doesn't miss instructions. It does all of this while being incredible cost effective. ↩︎

InkLink

Sun, 26 Jan 2025 00:00:00 GMT

This week, I built a collaborative e-paper display as a gift that I’m calling InkLink.

This fun conversation piece updates every three minutes based on what people have drawn on a web interface.

The web interface supports real-time multi-user canvas drawing. Try it out at inklink.winans.io! (This isn't connected to the actual display, feel free to draw whatever you want)

How it works

Hardware

The hardware side of things is pretty simple. The display is a 7.5-inch tri-color e-paper display from Waveshare. It's connected to an ESP32 microcontroller running CircuitPython that fetches the image from a server every three minutes.

Software

The website is a React app and a Bun server communicating real-time over WebSockets. The Bun server has its own server-side canvas thanks to @napi-rs/canvas. Each action is applied to the server canvas and then sent to all clients in real-time. The server also exposes a /drawing.bmp endpoint that the ESP32 fetches every three minutes.

Side note: This was my first time using Bun, and I’m impressed. Its built-in WebSocket support is fantastic, and the server uses just 10MB of RAM at idle.

Build your own

This project is simple to make and makes a great gift. It's open source and available on GitHub at github.com/Nicell/InkLink including a full build guide. All the parts are readily available and the total cost is around ~$45-$85 depending on where you source the parts. Feel free to make your own or contribute to the project!

Making My Blog 3x Faster

Tue, 27 Jun 2023 00:00:00 GMT

Custom fonts are a great way to make your website stand out, but for simple static websites, they can be a huge performance bottleneck. Let me show you how I reduced the First Contentful Paint of my blog from 2.5s to 0.9s, a nearly 3x improvement, by optimizing fonts.

Font overview

On my blog, I use three fonts: Lexend Deca for headings, Geologica for the body text, and JetBrains Mono for code blocks. Then, I use the Google Fonts CDN to load them.

Google Fonts offers a simple way to load fonts on your website. It generates a few link tags that you add to your head element. Here's the snippet my blog used:

<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Geologica&family=JetBrains+Mono:wght@400;600&family=Lexend+Deca:wght@400;600;700;800;900&display=swap" rel="stylesheet">

Why is Google Fonts slow?

Above you can see the results of running Google PageSpeed Insights on my blog. It gives me a score of 88 for performance and a First Contentful Paint of 2.5s. That's not great for a basic static website.

While Google Fonts is extremely easy to use, that simplicity comes at a cost. Because the fonts are hosted on a separate domain, the browser has to:

Make a DNS request to resolve the domain
Make an HTTP request to fetch the CSS definitions for the fonts
Make another request to download the fonts from the CSS definitions

That's a lot of work! The DNS request can be slow, and the chained request of CSS and then font is very slow on mobile networks that have high latency.

Hosting Fonts Locally

By hosting the font locally, we avoid the DNS request and the chained request. This way, we're just doing the font download step, which is much faster.

To host the fonts locally, we need to download the font files and add them to our project. I downloaded the base variable fonts from Google Fonts and added them to my project in the public fonts directory.

Then I added inline CSS to my head element in the layout file that's used for every page:

  <style>
    @font-face {
      font-family: "Geologica";
      src: url("/fonts/Geologica.ttf") format("truetype");
      font-display: fallback;
    }
    @font-face {
      font-family: "Lexend Deca";
      src: url("/fonts/LexendDeca.ttf") format("truetype");
      font-display: fallback;
    }
    @font-face {
      font-family: "JetBrains Mono";
      src: url("/fonts/JetBrainsMono.ttf") format("truetype");
      font-display: fallback;
    }
  </style>

Quick Aside on `font-display`

I also added font-display: fallback to the CSS to make sure that the fallback font is used while the custom font is loading. This is slightly different from font-display: swap. fallback gives the custom font 100ms of blank loading time for the custom font to load before switching to the fallback font. swap switches to the fallback font immediately and then switches to the custom font when it's loaded. I prefer fallback because it usually avoids the flash of unstyled text that swap can cause.

Local Font Results

As we can see, the results are much better! We get a score of 98 for performance and a First Contentful Paint of 0.9s. That's a nearly 3x improvement in First Contentful Paint already!

There are still some improvements to be done, though. The default variable fonts we downloaded from Google Fonts are huge.

The total download size of the page is 305 KB, which is pretty good. However, the fonts are 231 KB, which is over 75% of the total download size! This is resulting in a Largest Contentful Paint that hasn't changed much, and there's still layout shift as the fonts load.

Optimizing Local Fonts

As you might suspect, Google Fonts doesn't serve these original variable fonts to the browser. Instead, it serves a subset of the font, optimizes it, and compresses it. We can do the same thing to our local fonts to get the same benefits.

We'll be using the pyftsubset tool from fonttools to subset and optimize the fonts. First, we need to install fonttools:

pip install fonttools brotli zopfli

Then, we can run pyftsubset on our fonts to subset and optimize them:

pyftsubset\
  LexendDeca.ttf \
  --output-file="LexendDeca.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,\
  U+02C6,U+02DA,U+02DC,U+2000-206F,U+2074,U+20AC,\
  U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD"

Let's break down what each of these options are doing:

--flavor=woff2 tells pyftsubset to output a WOFF2 file, which is a compressed font format that's supported by all modern browsers.
--layout-features lets us select the OpenType features we want to include in the font. We're selecting all the ones we should need.
--unicodes lets us select the Unicode characters we want to include in the font. We're selecting the same range that Google Fonts uses for their Latin subset. You can change this depending on if you need to support other languages.

For further reading on why these options are being chosen, please check out the amazing article from Markos Konstantopoulos that gives much more detail to this process.

Now that we have our subsetted and optimized font, we can update our CSS to use the WOFF2 file instead of the TTF file:

@font-face {
  font-family: "Lexend Deca";
  src: url("/fonts/LexendDeca.woff2") format("woff2");
  font-display: fallback;
}

After completing this, the home page fonts download size was reduced from 231 KB to 82 KB and the total download size was reduced from 305 KB to 153 KB.

Optimized Local Font Results

And now we've achieved the coveted 100 performance score! The reduction in the font download size helped us reduce the Largest Contentful Paint by a whole second and eliminated the layout shift entirely.

Optimize Your Fonts!

Fonts are often overlooked when optimizing the speed of a website, but they can have a huge impact, particularly on simple static websites. By taking the simple step of hosting the fonts locally and optimizing them, we're able to dramatically improve the performance of our website.

Give it a try on your own website, and hopefully you'll see some impressive results!

Exploiting GitHub Actions

Thu, 17 Mar 2022 00:00:00 GMT

I'm a contributor to the open-source keyboard firmware ZMK, which is a firmware that runs on custom, usually wireless keyboards. It turns electrical keypresses into real output to your computer with a bunch of cool features and customizations. In our project, we house widely available custom "boards" and "shields". "Boards" are defined as the part of the keyboard that has the microprocessor running the firmware, and "shields" are the keyboard looking circuit board that a "board" plugs into. With every commit to the main repository and pull requests we build firmware for every single board and shield combination using GitHub Actions.

For those unfamiliar, GitHub Actions is a Continuous Integration/Delivery. More simply, it allows us to run build scripts on every commit to make sure everything still works with the new changes committed. This is free to open source projects with some limitations, which we'll see soon.

The Problem

ZMK has grown in popularity, and today we're sitting at 13 boards and 44 shields, which is awesome. For our GitHub Actions workflows, we utilize what's called a build matrix to generate all of the different firmware files required. This allows us to set all the boards as one dimension and all the shields as the other, and then it builds every single combination possible, which is perfect... except for one thing. GitHub Actions limits matrix builds to 256 maximum jobs. Considering today we have 528 combinations, we exceed this limit, and it's not going to ever get better. We need a solution to be able to continue building every combination.

Analyzing Possible Solutions

The first thing you might think is, why not just loop through every combination in one job instead of running a different job for each firmware file? This isn't a bad train of thought, but we have to consider our limitations.

We have to use basic bash scripting to complete these loops, and feeding these combinations via JSON/YAML is not an easy task. The other major issue is that we need to upload each build file as an artifact so users can find and download them. Unfortunately, there isn't an easy way to upload unique artifacts in a loop since you can't call the artifact upload action from inside a raw bash script.

When I looked into this problem I first tried to remove the build matrix portion entirely. In my mind, there was no way of getting around the 256 limit. I turned to the GitHub Actions API, which has a way to invoke workflow runs with parameters. I thought this was perfect to run an unlimited amount of jobs since we're working outside of the matrix job limit and we can queue however many workflows as we want.

This was looking like a good solution until I realized one flaw. To call these API endpoints we needed to pass a privileged API key into the GitHub Action script. While we can hide the key fairly effectively, the project is open source, and we run these scripts on every pull request making it an easy target to do other nefarious activities.

With direct API calls off the table, I turned back to looping through every combination as a possible solution. This time I was trying to figure out how I can more easily walk through all the combinations and upload every file individually. I ultimately couldn't figure out a way to do it in bash scripting, and then I stumbled upon a GitHub action that solved all my problems by misusing it; GitHub Script.

GitHub Script

The GitHub Script action's tagline is "Write workflows scripting the GitHub API in JavaScript". This action is supposed to be used to interact with GitHub issues, pull requests, and other GitHub API elements. Crucially, it gives a sandboxed environment that's still privileged and contains the current workflow's context. This allows us to upload file artifacts arbitrarily whenever we want, including in loops.

Instead of using it to interact with GitHub API elements, I use it to run the same scripts we ran in the bash script to build each combination and then upload the file artifact, perfect!

One issue with this solution though, is that we lost all parallelization. This means we have to build all 528 files one at a time, which is pretty slow. To improve upon this I decided to return the job matrix, but use it differently this time. I now use it to create a new job for each board, not for each combination. So we now create 13 board jobs that run all the possible combinations for that specific board using GitHub Script.

With this solution, we can build 528 firmware files just as fast as we used to build only 256 before. I'd consider this a huge success that should last for years to come. That is until we get 256 boards...

Other Improvements

While solving this major problem for our build script, I also overhauled a couple of other parts of our build script that may interest you.

We used to manually write every board/shield in a list inside the build script to keep track of what combinations to build. This was an extra file to maintain and caused tons of merge issues as people were creating pull requests simultaneously. We've transitioned to a system that reads in what boards and shields there are dynamically by walking the board/shield directory of the project.
We used to build around 256 firmware files on every single commit no matter what the change was. It covered everything, but it was inefficient and caused hours of pileups of queued jobs. The new script checks whether the changes change a board/shield or change something core in the firmware. If there is a board/shield change, it builds every board/shield combination that's affected by the change. If there's a core change, it builds a short list of boards/shields that cover nearly every use case. This has cut down on almost all of our queued job issues and made our checks complete in a fraction of the time. We also build every combination nightly to make sure every combination has fresh firmware with all the new features.

If you're curious to see what these changes looked like you can see the pull request here.

Mysterious Broken Bootloader

Sat, 03 Oct 2020 00:00:00 GMT

This post was originally published on the ZMK Firmware Blog.

Recently I was able to fix the "stuck in the bootloader" issue in #322 that had been plaguing us for quite some time. I want to go over what the issue was, how the issue was diagnosed, and how it was fixed.

Background

What exactly is the "stuck in the bootloader" issue? Seemingly randomly, users' keyboards would suddenly stop working and when they would reset their keyboard they would get put into the bootloader instead of back into the firmware. This would require the user to re-flash the firmware again to get into the firmware. That wouldn't be so bad except for the fact that once this occurs, every reset would require the user to re-flash the firmware again. The only way to really fix this issue was to re-flash the bootloader itself, which is a huge pain.

Going into this, all we knew was that this issue was most likely introduced somewhere in the #133, which added Bluetooth profile management. We've had quite a few attempts at trying to recreate the issue, but we never were able to get it to happen consistently.

Diagnosing the issue

This issue had been happening sporadically for the past month, and I finally decided to dig in to see what was going on. We started in the Discord and discussed what was common between all of the people who have experienced this issue. Everyone who had this issue reported that they did quite a bit of profile switching. This lined up with the possible connection to the Bluetooth profile management pull request.

Pinpointing the cause

I had a hunch that this was related to the settings system. The settings system is used by profile Bluetooth switching, and the settings system works directly with the system flash. Based on this hunch, I tried spamming the RGB underglow cycle behavior on my main keyboard. Sure enough after a couple minutes, I got stuck in the bootloader. I was even able to reproduce it again.

This was an important discovery for two reasons. First, I was able to recreate the issue consistently, which meant I could set up logging and more closely monitor what the board was doing. Second, this more or less proved that it was specifically the settings system at fault. Both Bluetooth profile switching and RGB underglow cycling trigger it, and the one common piece is they save their state to settings.

Settings system overview

To understand what's going wrong, we first need to understand how the settings system works. Here's a diagram to explain the flash space that the settings system holds for our nRF52840 based boards (nice!nano, nRFMicro, BlueMicro).

The settings flash space lives at the end of the flash of the chip. In this case it starts at 0xF8000 and is 0x8000 bytes long, which is 32KB in more comprehensible units. Then due to the chip's architecture, this flash space is broken into pages, which are 0x1000 bytes in size (4KB).

The backend that carries out the settings save and read operation in ZMK is called NVS. NVS calls these pages sectors. Due to how flash works, you can't write to the same bytes multiple times without erasing them first, and to erase bytes, you need to erase the entire sector of flash. This means when NVS writes to the settings flash if there's no erased space available for the new value, it will need to erase a sector.

Logging discoveries

So first I enabled logging of the NVS module by adding CONFIG_NVS_LOG_LEVEL_DBG=y to my .conf file. I repeated the same test of spamming RGB underglow effect cycle and the resulting logs I got were this:

[00:00:00.000,671] <inf> fs_nvs: 8 Sectors of 4096 bytes
[00:00:00.000,671] <inf> fs_nvs: alloc wra: 3, f70
[00:00:00.000,671] <inf> fs_nvs: data wra: 3, f40
// A bunch of effect cycle spam
[00:02:34.781,188] <dbg> fs_nvs: Erasing flash at fd000, len 4096
// A bunch more effect cycle spam
[00:06:42.219,970] <dbg> fs_nvs: Erasing flash at ff000, len 4096
// A bunch more effect cycle spam
// KABOOM - bootloader issue

So at start up, we can see that the 8 sectors of 4KB are found by NVS properly, however, I wasn't sure what the second and third lines meant, but we'll get back to that. Nonetheless the next two logs from NVS showed erasing the sector at 0xFD000 and then erasing the 0xFF000 sector.

It's really odd that the third to last sector and the last sector are erased, and then shortly after the bootloader issue is hit. I really had no explanation for this behavior.

Reaching out to Zephyr

At this point, I nor anyone else working on the ZMK project knew enough about NVS to explain what was going on here. Pete Johanson, project founder, reached out on the Zephyr Project's Slack (ZMK is built on top of Zephyr if you weren't aware). Justin B and Laczen assisted by first explaining that those alloc wra and data wra logs from earlier are showing what data NVS found at startup.

More specifically, data wra should be 0 when it first starts up on a clean flash. As we can see from my earlier logging on a clean flash I was instead getting f40. NVS is finding data in our settings sectors when they should be blank! We were then given the advice to double check our bootloader.

The Adafruit nRF52 Bootloader

Most of the boards the contributors of ZMK use have the Adafruit nRF52 Bootloader, which allows for extremely easy flashing by dragging and dropping .uf2 files onto the board as a USB drive. Every bootloader takes up a portion of the flash, and in the README explains that the first 0x26000 is reserved for the bootloader with the nRF52840, and we've properly allocated that.

However, there isn't a full explanation of the flash allocation of the bootloader in the README. There's a possibility that the bootloader is using part of the same flash area we're using. I reached out on the Adafruit Discord, and Dan Halbert pointed me towards the linker map of the nRF52840. Let's take a look.

FLASH (rx) : ORIGIN = 0xF4000, LENGTH = 0xFE000-0xF4000-2048 /* 38 KB */

BOOTLOADER_CONFIG (r): ORIGIN = 0xFE000 - 2048, LENGTH = 2048

/** Location of mbr params page in flash. */
MBR_PARAMS_PAGE (rw) : ORIGIN = 0xFE000, LENGTH = 0x1000

/** Location of bootloader setting in flash. */
BOOTLOADER_SETTINGS (rw) : ORIGIN = 0xFF000, LENGTH = 0x1000

Here's a diagram to show this a bit better.

We've found the issue! As you can see from the red bar (representing our settings flash area), we've put the settings flash area right on top of the Adafruit bootloader's flash space. Oops!

This also shines some light on why NVS erased 0xFD000 and 0xFF000 sectors. It's possible there was no flash written to 0xFD000 because the bootloader didn't use up all of that space it has, and then there possibly weren't any bootloader settings set yet, so 0xFF000 could be used and erased by NVS too.

After erasing 0xFF000, NVS probably next erased a rather important part of the bootloader that resulted in this issue at hand. In my opinion, we're pretty lucky that it didn't delete an even more vital part of the bootloader. At least we could still get to it, so that we could re-flash the bootloader easily!

The solution

Now that we've found the issue, we can pretty easily fix this. We'll need to move the settings flash area back so that it doesn't overlap with the bootloader. First we calculate the size of the of flash area the bootloader is using.

0x100000 (end of flash) - 0x0F4000 (start of bootloader) = 0xC000 (48KB)

So the bootloader is using the last 48KB of the flash, this means all we need to do is shift back the settings area and code space 0xC000 bytes. We'll apply this to all of the .dts files for the boards that were affected by this issue.

		code_partition: partition@26000 {
			label = "code_partition";
-			reg = <0x00026000 0x000d2000>;
+			reg = <0x00026000 0x000c6000>;
		};


-		storage_partition: partition@f8000 {
+		storage_partition: partition@ec000 {
			label = "storage";
-			reg = <0x000f8000 0x00008000>;
+			reg = <0x000ec000 0x00008000>;
		};

And with those changes, we should no longer run into this issue! In the process of these changes, we lost 48KB of space for application code, but we're only using around 20% of it anyways. 🎉