{ "version": "https://jsonfeed.org/version/1.1", "title": "Posts — msfjarvis.dev", "description": "Posts by Harsh Shandilya", "home_page_url": "https://msfjarvis.dev/", "feed_url": "https://msfjarvis.dev/posts/feed.json", "authors": [ { "name": "Harsh Shandilya", "url": "https://msfjarvis.dev" } ], "items": [ { "id": "https://msfjarvis.dev/posts/rewriting-claw-html-parser-for-over-the-air-updates/", "url": "https://msfjarvis.dev/posts/rewriting-claw-html-parser-for-over-the-air-updates/", "title": "Rewriting Claw's HTML parser for over-the-air updates", "date_published": "2026-07-17T13:34:00.000Z", "summary": "Struggling through a brand new framework so I can hotfix my app faster", "content_html": "
\n

NOTE

\n

I used LLMs to help debug some of the code explained below, and ended up throwing away a lot of it while re-reviewing that work as part of writing this post. People have strong opinions on the use of LLMs and I don’t want them to feel misled by discovering partway through that this work involved an LLM, so here’s your PSA.

\n
\n

First, some background. Claw is an Android app for browsing the linkblogging community known as Lobsters. For years, the app relied on a JSON API to interact with the site. Some months ago, I came across a GitHub comment from the site admin explaining that the JSON API is considered a Ruby on Rails misfeature and not something he supports for external integrations. I don’t expect it to ever fully disappear since many other third-party integrations rely on it, but I decided I should probably look into options anyway. I filed an issue with my initial findings then pretty much forgot about it for over half a year.

\n
\n

The problems with HTML parsing

\n

In May 2026 I decided to revisit the problem and make some progress on this front, since I had also added the ability to log into the site through the app now and wanted to start offering more interactive features that would need parsing HTML anyway. The result of it was this massive PR where my ability to comprehend CSS selectors was tested repeatedly and I did not exactly score passing marks. Unsurprisingly, I broke some things[1] [2] in the process, but it all came together in the end, and I shipped these changes towards the end of that month.

\n

Anybody who has had to write a scraper can already see the next line coming: Lobsters made a change to the site markup exactly 2 days after I released that code. I scrambled to release a fix (shoutout to Firefox DevTools for introducing me to this horror: div.byline > a[href^=/~]:not([tabindex]):not([aria-hidden=true])), which made me think there had to be a better solution to this problem that didn’t involve waiting hours or days for Google Play Store.

\n

Looking for a faster deployment model

\n

The React Native folks have had this solved for years through Expo, which offers a cloud-based service called EAS Update that can push updated JavaScript, assets and images to an existing build of a React Native app without needing to go through an app store. How do we bring this to native Android apps?

\n

This is the question that Cash App seems to have asked themselves as well, and their answer is called Zipline. It’s a Gradle plugin and Kotlin Multiplatform library that enables you to run Kotlin/JS libraries on Kotlin/JVM and Kotlin/Native apps, using the QuickJS embeddable JavaScript engine. If you can compile your code to JavaScript, you can leverage Zipline to update it over the air in a reliable and secure manner.

\n

Zipline offered everything I was looking for, and since I was predominantly using Kotlin/JS-capable libraries for the HTML parsing it seemed like a no-brainer to give this a go. What follows is a deep dive through false leads about code size, stack overflows, and cache invalidation.

\n

The great Zipline migration

\n
\n

WARNING

\n

This is a novice stumbling his way through an unfamiliar framework and paradigm, who will unsurprisingly be misled by a clanker and make a heroic recovery. Please exercise caution/patience.

\n
\n

My dependencies were all ready for Kotlin/JS, but my own code was woefully Android-specific and needed some refactoring. Before I started on this, I had to first decide on how I wanted to split the responsibilities between the K/JS “guest” and the Android “host”. This is what I ended up with:

\n\"Mermaid\n

Step 1: Converted the data model layer to Kotlin Multiplatform, which went relatively fine.

\n

Step 2: Added the actual Zipline-backed parsers. I replaced Kspoon with the underlying Ksoup library and called it directly instead of through a kotlinx.serialization based parser.

\n

Step 3: Migrated the Android code to using the new Zipline-backed parser. I’ve put together a terrible kludge to determine when to call which parser method, and I am sure there’s a nicer way to do this that evaded me at the time.

\n

Step 4: Embedded the latest copy of the Zipline assets automatically in the APK to speed up the first-launch experience. Since my app really only works with the internet it’s not a dealbreaker if I skip this, but I felt it’s better to have a fallback in case Cloudflare or I have done something to cause my remote manifest to become inaccessible. I later discovered this logic wasn’t actually falling back to embedded assets like I assumed, which I fixed by explicitly loading the embedded assets manually in case of errors.

\n

And that should’ve been it! Except…

\n

The troubles begin

\n

After the mechanical refactors I tried running the app and immediately ran into stack overflow errors. Zipline helpfully includes a sampling profiler to identify bottlenecks, but I wasn’t able to figure out how to make it help with this specific problem. I also didn’t exactly understand what was causing this stack overflow. I flailed around for a fair bit but that’s not really interesting, so here’s a quick summary.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Thing triedImpact
Cache Regex objects and replace functional chains with simple loopsNothing whatsoever
Simplifying parser models to drop generated serializable code [1][2][3]Made the code more annoying to deal with
Replaced recursive code with a simpler loopNative crash instead of stack overflow!
Lazy initializationNothing
De-duplicating fieldsNothing
Removing data classesNothing
\n

At this point it had already been almost two weeks, and I was pretty tired of going from one confusing problem to another and gave up on this adventure for a few days. After having some success at work with giving GPT-5.5 a harness based on the android CLI to let it debug a complex issue by itself, I decided to bring that hammer back to this problem. I wrote out a handoff doc, cloned a local copy of Zipline for the clanker to refer to, and let it attack the problem overnight.

\n

In the morning it claimed to have solved the crash, which I was able to verify myself. However, as is often the case with LLMs, the code was completely inscrutable. I asked it to commit the changes with a proper commit message but that shed no real light. There were tests, but it was clear the clanker was too lazy and had chosen to invent a separate code path just for tests so it could get away with writing mostly useless fluff. I deleted the tests and decided I’d add some manually written ones when I understood the code.

\n

GPT-5.5 had also found another source of problems, the stack size. It raised the stack size limit from the default 6 MB, and changed the code to ensure every Zipline operation ran on the same thread/dispatcher that we initialized Zipline with. I had to go look that up, and apparently the bigger stack size is an upstream recommendation buried in a code comment. Also, TIL that Thread has a four-argument constructor!

\n

I immediately understood the reason for the increased stack size, but it took a bit to realize that since JavaScript is single-threaded, it only makes sense we confine our reads and writes of JavaScript-exposed memory to that same thread.

\n

Once I verified everything was working, I shipped it to the internal testing track.

\n

That bit of code

\n
\n

NOTE

\n

This serializer was ultimately unnecessary and no longer exists. I’m including it because understanding and then removing it was an important part of validating the LLM-generated changes.

\n
\n

While I did merge the branch without really understanding what the ParserSerializers code was doing, I was determined to figure it out. To improve my understanding of it, I talked my way through the code to Mayank at work, and eventually put in some handwritten tests to validate my mental model.

\n

The clanker invented a very rudimentary serialization format that creates a flat string that is smaller than a comparative JSON object but trades that with forcing you to know ahead of time what field names to expect and in what order.

\n
private class PacketWriter {
  private val parts = StringBuilder()
\n
  fun string(value: String): PacketWriter {
    parts.append(value.length).append(':').append(value)
    return this
  }
\n
  // Other types skipped for brevity
\n
  fun build(): String = parts.toString()
}
\n
\n
// To go from object to String, you call methods on
// the PacketWriter builder to feed in the fields of your values.
fun encode(value: Tag): String {
  return PacketWriter()
    .string(value.tag)
    .string(value.description)
    .boolean(value.privileged)
    .boolean(value.active)
    .string(value.category)
    .boolean(value.isMedia)
    .double(value.hotnessMod)
    .build()
}
\n

Effectively, what you’re seeing is a classic accumulator pattern but in a String form. Instead of building up a JSON object, we construct a “flat” string that is essentially a concatenation of many len(content):content pairs. Null values are encoded with a length of -1; the colon is retained to keep parsing unambiguous.

\n

The deserialization code will be self-explanatory if you were able to follow the serializer.

\n
private class PacketReader(private val payload: String) {
  private var index = 0
\n
  // This code has to be more defensive about invalid
  // inputs, and hence most of the method is just validation.
  fun string(): String {
    val separatorIndex = payload.indexOf(':', index)
    if (separatorIndex == -1) throw SerializationException(\"Malformed payload\")
    val length = payload.substring(index, separatorIndex).toInt()
    index = separatorIndex + 1
    if (length < 0) throw SerializationException(\"Expected non-null string\")
    val endIndex = index + length
    if (endIndex > payload.length) throw SerializationException(\"Malformed payload\")
    return payload.substring(index, endIndex).also { index = endIndex }
  }
\n
  // Other types skipped for brevity
}
\n
// Due to the nature of this scheme, deserialization is sensitive
// to the order in which serialization occurred. Your field ordering
// was already public API for public types, it is now also required
// to be stable for this serialization codec.
fun decodeObject(payload: String): Tag {
  val reader = PacketReader(payload)
  return Tag(
    tag = reader.string(),
    description = reader.string(),
    privileged = reader.boolean(),
    active = reader.boolean(),
    category = reader.string(),
    isMedia = reader.boolean(),
    hotnessMod = reader.double(),
  )
}
\n

To decode individual fields, we hold a top-level index variable and slice out a substring between index and the first : character to determine the length of data to be read. Then we read off that much after the colon, and move index to the first unread position. This is repeated till the decoder has read all fields it wants, and the object is constructed.

\n

This is obviously extremely naive: The decoder reads the fields it expects; it does not ensure that the entire payload was traversed or reject trailing data. Invalid booleans become false, any negative nullable length becomes null, and numeric conversion exceptions are not normalized.

\n

Deployment

\n

Once I had the code working, the remaining part was deployment. I didn’t want to make this complicated or rely on GitHub Pages as a CDN which they discourage, so I opted to host it myself. The directory layout I chose is intentionally quite simplified and idiot-proofed against yours truly, who did fuck it up many times. The directory structure is hopefully self-explanatory:

\n
/var/lib/claw-deploy
├── current -> /var/lib/claw-deploy/releases/20260712T113110Z
└── releases
    ├── 20260710T065949Z
    │   ├── Claw-zipline-parser.zipline
    │   ├── fleeksoft-io-charset.zipline
    │   ├── fleeksoft-io-io-core.zipline
    │   ├── kotlin-kotlin-stdlib.zipline
    │   ├── kotlin_org_jetbrains_kotlin_kotlin_dom_api_compat.zipline
    │   ├── kotlinx-atomicfu.zipline
    │   ├── kotlinx-coroutines-core.zipline
    │   ├── kotlinx-serialization-kotlinx-serialization-core.zipline
    │   ├── kotlinx-serialization-kotlinx-serialization-json.zipline
    │   ├── ksoup.zipline
    │   ├── manifest.zipline.json
    │   ├── Stately-stately-concurrency.zipline
    │   ├── Stately-stately-strict.zipline
    │   └── zipline-root-zipline.zipline
    └── 20260712T113110Z
    ...
\n

current is a symlink to a timestamped directory containing the compiled Zipline artifacts. The size of individual releases is relatively tiny so I keep all previous releases around, making it easy to roll back during one of those aforementioned fuck ups. The app only ever fetches the /current path, so to the client this bundle is effectively unversioned.

\n

The documented trust model for Zipline states that the host app and the build server are implicitly trusted, so I stuck with the official path of signing the manifest with an Ed25519 key. The app fetches the manifest, validates it is signed with the trusted key that is embedded in the app, then downloads the modules listed in it. Debug builds do not fetch remote assets, only loading the embedded bundle to ease testing. Failing to fetch the remote manifest also gracefully degrades to loading the embedded copy, which means 100% uptime is not a strict requirement. The K/JS bundle is currently only the parser code, but even more functionality could end up in there at some point. Exciting stuff! Unless…

\n

When I went to make my first over-the-air release, I ran into caching problems :) While Zipline takes care of disabling the HTTP client cache when downloading .zipline files, it doesn’t do so for manifest.zipline.json. Thus, a server with caching enabled can cause OkHttp to load its cached and outdated manifest, which results in new .zipline modules being downloaded without an updated manifest, ultimately causing checksum verification to fail. This was a failure state I wasn’t aware of when I first implemented Zipline in the app, and had to patch it on both ends: setting the right Cache-Control headers on the server, and deleting the manifest URL cache entry in the app. That seems to have done the job, and I have been able to roll out two updates since making these changes without any problems.

\n

What changes for users

\n

Since I didn’t want to involve any complex server-side logic, my deployment setup is extremely rudimentary which makes some aspects of the in-app behavior suboptimal. The way things are configured right now, the app re-fetches the manifest on each launch, which slows down the initial content load. Every app launch sends at least one network request to my server, which means I was now accidentally collecting user IPs. To mitigate that on the origin server, I have disabled logging in my webserver and deleted the logs that were already recorded.

\n

The release bundles are also unversioned, which means I do not yet know how I would roll out a breaking change, which I’ve already made as you’ll read in the very next section. This is all very new and exciting to figure out, even if it’s making my app somewhat worse in the short term.

\n

Google Play’s policy on remote code loading exempts code that is run in interpreters, which means the Zipline approach is compliant.

\n

On debugging with LLMs

\n

I generally take pride in being meticulous and having a good system-level understanding of the code I work on, but here I gave up and deferred to an LLM which felt like admitting defeat. When it solved the problem overnight in a problem space I barely understood I was pretty glad in the moment to be done with it, but eventually that made way for just disappointment. I’d have probably felt better if I didn’t give up at that point and just continued investigating and researching, after all that was the entire reason Claw even exists. I’ve used it for nearly 5 years to learn new things the hard way and I gave up on that this time just because I had a clanker handy.

\n

The real kicker? All that inscrutable code is also completely useless. The stack size change and confining operations to the shared thread were the actual fixes. As I mentioned at the start, as part of writing this post over the past month I reviewed and validated every bit of that PR again which surfaced these hallucinations. As of writing, the custom serializer is gone, and so is the zipline-parser-api module. Removing the custom serializer and its model classes is a breaking API change, and my internal testing track is now broken because I haven’t updated the remote bundle with these changes. Another thing to figure out, sans LLMs this time :)

\n

I’ll try harder to hold off on the LLMs going forward. Massive thanks to Jesse Wilson whose writing and public speaking have helped me rediscover the fun of programming. Go check out his talk Deconstructing OkHttp, it’s a great use of your time even if you do not know or care about OkHttp! You’ll learn a good deal about evolving a complex library over a period of years and how the benefits of thoughtful API design stack up.

\n

Closing notes

\n

The effort both taught me more about Zipline and solved a problem I fully expected to encounter. I was able to roll out a tag parsing fix via Zipline before publishing it to Google Play. I then extended the fix to search, and added OP markers to submitters — things I missed before the release.

\n

All in all, this has been quite the learning experience and I’m excited to once again have fun while programming.

" }, { "id": "https://msfjarvis.dev/posts/apologies-for-the-rss-mishap/", "url": "https://msfjarvis.dev/posts/apologies-for-the-rss-mishap/", "title": "Apologies for the RSS mishap!", "date_published": "2026-07-10T19:34:00.000Z", "summary": "Defeated yet again by an entirely too complicated build pipeline", "content_html": "

Just a quick apology to all subscribers of the RSS feed, whom I accidentally sent a lot of duplicate posts to.

\n

I was working on the site and pushed to the wrong deployment, which caused the main site to start behaving like it was the preview site which includes my draft posts. Lots of duplicates and incomplete posts have now been dumped into your feed readers due to this, since it took me over an hour to figure out I had the mistake.

\n

Your best course of action is to just delete and re-add the feeds instead of trying to hunt down the duplicates.

\n

Sorry again!

" }, { "id": "https://msfjarvis.dev/posts/julyreply-re-stuck-in-draft/", "url": "https://msfjarvis.dev/posts/julyreply-re-stuck-in-draft/", "title": "JulyReply - Re: Stuck in draft", "date_published": "2026-07-09T18:52:00.000Z", "summary": "On drafts and being okay with them", "content_html": "
\n

NOTE

\n

This post is part of the JulyReply blogging event

\n
\n

Gordon wrote about his blog posts being stuck in draft, which is a problem I also have to some extent – my drafts site has 75 posts right now compared to the 67 on the main site. At least one of those drafts carries a date from 2020!

\n

I agree with him on urgency being a strong motivator. I’ve turned around similarly sized posts in under an hour and also in months. My interest in specific things ebbs and flows quite frequently, and often it will be a very long time before I can summon the requisite curiosity and motivation to wrap up a draft. Many times, it will be someone chancing across that draft and posting some feedback which motivates me to finish it off. Other times I’ll simply post about having an incomplete draft and let someone express interest, which then becomes my driving force to deliver that bit of writing.

\n

And yet, in MOST cases, I just no longer care about the topic to finish that draft ¯\\(ツ)/¯

\n

I like having ideas written down anywhere, so my blogging process will almost always start with writing down either broad bullet points or a stream of consciousness dump into a draft. Most of these drafts will become something real in a few weeks time, and some of them will sink to the bottom to be forgotten forever. I think I’m okay with that.

\n

That said, I’ll take a page out of Gordon’s metaphorical book and also list some of my (comparatively uninteresting) drafts that I would like to finish. Reach out on the Fediverse, via email, or whatever communication channel you can grab a hold of me through and let me know what looks interesting. I do apologize that this is all technical content since I’m still learning to be a human on this blog outside my weeknotes 😅

\n

I’ll do a count on 17th July, which will give me around 2 weeks to finish up the post.

\n\n

P.S. Abhinav built a cool page on his site serving this exact purpose, go cast your vote there too if you enjoy quality technical writing! I wanted to steal the idea and build a voting UI but that would make this post get stuck in draft!

" }, { "id": "https://msfjarvis.dev/posts/my-first-spam-webmention/", "url": "https://msfjarvis.dev/posts/my-first-spam-webmention/", "title": "My first spam Webmention", "date_published": "2026-07-07T10:39:00.000Z", "summary": "Finally being forced to build moderation capabilities into my Webmentions server", "content_html": "

I finally got my first spam Webmention on this site

\n
\"ScreenshotExpand image
A spam post from Bluesky that bridged over as a Webmention
\n

I chanced across this on one of my weeknotes, which seems to have already been deleted by Bluesky moderation or the poster by the time I got around to seeing it. I have been coasting along for a few months now without any moderation set up to easily moderate Webmentions, but I clearly should be paying more attention. This is far from being bad as far as spam goes, but I’d like to be prepared for the eventuality of something really terrible showing up in one of these.

\n

So far I had been resorting to manually running queries in the Cloudflare dashboard’s D1 SQL Explorer, but that’s too slow and complex to serve as an actual moderation tool.

\n

To that effect I now have a very brutalist dashboard being served by the Webmentions server (code here) which has a list of all pages that have received Webmentions. To mask my inability to do good web design, it is styled using the Oat CSS library like all my non-blog web projects. I find the minimalist aesthetic of Oat quite nice and it rarely needs any significant changes to a semantically laid out page when you add it.

\n
\"WebmentionsExpand image
\n

Clicking on one of these links takes you to this equally terrible UI which could be reduced down to a list with delete buttons and still serve exactly the same purpose. Not exactly rocket science here.

\n
\"AdminExpand image
\n

I didn’t want to figure out passwords and bearer auth, so the server does not attempt to do any authentication and just relies on Cloudflare Access to guard access to the UI and the endpoints this uses.

\n

Hopefully I never have to use it!

" }, { "id": "https://msfjarvis.dev/posts/keeping-ai-crawlers-off-my-forgejo-server/", "url": "https://msfjarvis.dev/posts/keeping-ai-crawlers-off-my-forgejo-server/", "title": "Keeping AI crawlers off my Forgejo server", "date_published": "2026-06-29T04:53:18.854Z", "summary": "The short and bumbling journey to finally giving my tiny VPS some respite", "content_html": "
\n

NOTE

\n

This post will mention Cloudflare. People have a lot of strong opinions about the company’s effective monopoly, myself included, but it was the fastest way for me to solve my immediate problem.\nI’m also employed at Cloudflare, and get a free Enterprise account from them, but I believe nothing I have set up here requires one. If someone tries this out and runs into any free-tier limitations, please let me know and I will make a correction.

\n
\n

Beginning 2026-06-18 I was starting to see a lot of slowness in my Forgejo instance, which I hoped would eventually subside, so I chose to just watch the load dashboards and wait.

\n

The crawlers obviously did not relent, so I started earnestly mitigating it on 2026-06-25 by enabling fail2ban which unfortunately did not do much to help the situation. I had to take a hammer to the problem and put in a Cloudflare security rule banning the countries of China and Vietnam where most of the requests were originating. The rule also carried a specific ban for accesses to the oneplus3 repository because it was surprisingly a “fan” favorite. The Cloudflare block immediately got to work and within 16 hours mitigated over 130K hits.

\n

However, the site’s availability started dropping over the next few hours and I started seeing a near constant stream of HTTP 521 responses which indicates Cloudflare couldn’t reach my server. Turns out the log transformer I enabled for fail2ban to work wasn’t logging the real client IPs, it was logging Cloudflare IPs, so fail2ban had just been banning Cloudflare 😬

\n

Solving that wasn’t too difficult, Caddy has the necessary config knobs to mark reverse proxies as trusted and forward the client IP headers correctly. I unbanned all the IPs in fail2ban, and after that I followed this example to make fail2ban directly block offenders via Cloudflare’s IP Access Rules.

\n

Within 3 hours I was up to 2200 individual IPs banned, so I decided to do some analysis of banned IPs to figure out if there’s a broader block I can instate and prevent running into the 50,000 limit. Most of them were popular consumer providers like Comcast and Verizon, so I banned the ASNs south of them:

\n
- AS4229 Zenlayer
- AS150436 Byteplus
- AS62240 Clouvider
- AS62651 Strong Technology
- AS32934 Facebook
\n

This helped to some degree but not quite, and I was still getting a sustained 50 to a 100 RPS which wasn’t impossible to serve, just very wasteful and causing other things to slow down on this rather weak VPS. After around 12 hours, I realized fail2ban had now banned 31K individual IPs which was coming dangerously close to the 50K IP Access Rules limit from Cloudflare, and was admittedly a rather naive way of addressing the problem.

\n

I turned off fail2ban, and started poring through my Forgejo logs to find patterns I could let Cloudflare handle via a custom ruleset. This took a bit to figure out, but ultimately I landed on these, in order:

\n
    \n
  1. Block China and Vietnam. These still have the worst of the residential proxy botnet problem.
    2. \n
  2. Block the known-bad ASNs from before.
    3. \n
  3. Block anyone accessing the /oneplus3 and /revanced-patches repositories since they had become super attractive scraper honeypots for some reason, I don’t really need either to be available so this was fine.
    4. \n
  4. Block fetches of .bundle archives, since I don’t use them and they’re expensive to generate.
    5. \n
  5. Block (downgraded to Managed Challenge later) paginated history crawling:\n
    http.request.method eq \"GET\" and
    http.request.uri.path strict wildcard \"/*/*/commits/commit/*\" and
    http.request.uri.query contains \"page=\"
    \n
    6. \n
  6. Managed Challenge for all other anonymous history crawling\n
    http.request.method eq \"GET\" and
    not cf.client.bot and
    (
      http.request.uri.path strict wildcard \"/*/*/blame/commit/*\" or
      http.request.uri.path strict wildcard \"/*/*/commits/commit/*\" or
      http.request.uri.path strict wildcard \"/*/*/src/commit/*\" or
      http.request.uri.path strict wildcard \"/*/*/blame/tag/*\" or
      http.request.uri.path strict wildcard \"/*/*/commits/tag/*\" or
      http.request.uri.path strict wildcard \"/*/*/src/tag/*\" or
      http.request.uri.path strict wildcard \"/*/*/find/tag/*\" or
      http.request.uri.path strict wildcard \"/*/*/commit/*.diff\" or
      http.request.uri.path strict wildcard \"/*/*/commit/*.patch\" or
      (
        http.request.uri.path strict wildcard \"/*/*/commit/*\" and
        (
          http.request.uri.query contains \"files=\" or
          http.request.uri.query contains \"show-outdated=\" or
          http.request.uri.query contains \"whitespace=\"
        )
      )
    )
    \n
    7. \n
  7. Managed Challenge for reading historical copies of files\n
    http.request.method eq \"GET\" and
    not cf.client.bot and
    (
      http.request.uri.path strict wildcard \"/*/*/raw/commit/*\" or
      http.request.uri.path strict wildcard \"/*/*/raw/tag/*\"
    )
    \n
    8. \n
\n

With all these in place the Git server is now chilling. I can’t remember the last time it was serving so few spam hits that I could see the Prometheus /metrics query in the journal logs.

\n

According to Cloudflare’s logs it has mitigated 489.65K requests for me since I deployed this rule set, and of the 100K Managed Challenge that have been issued, only 3 have been solved. Go BOTS team!

\n

Of course, I still had the problem of the now-useless IP Access Rules that fail2ban had deployed, which needed to be removed. Here I have some criticism of Cloudflare and the way the whole account/zone authentication situation works, because this took me over an hour to figure out and none of it was documented.

\n

I tried using an account API token to fetch IP Access Rules, and consistently got 0 results from the API. I tried a user API token and got the same result. I switched from the /account API shape to the /zone one, and started getting the completely useless code: 10000 message: Authorization error error. No matter how I tried to call the API, I would either get 0 or an error, while I was staring at 31K of them in the dashboard. Finally, I decided to try the Global API key whose docs has a big limitations section but no mention that it’s the only way to access the supposed legacy IP Access Rules.

\n

So now that I had wasted an hour and could finally get back these 31K rules, I do a cursory analysis on them with MaxMind and obviously find no country or ASN I can do a simple ban on. To delete all these IP Access Rules I use the Cloudflare Python SDK to write a small script that would remove these serially since I didn’t wanna go for max speed and then have to handle being rate limited. This took a whole 8 hours and 49 minutes to chew through all those IP Access Rules, but given I was effectively doing 1 RPS off my residential Internet connection I’m not mad about it.

\n

Here’s some graphs to spruce up this word salad of a post.

\n
\"CloudflareExpand image
The 0% there means nobody hitting the endpoint is solving the challenge, a.k.a that's all bots
\n
\"GrafanaExpand image
No points for guessing when the ruleset was deployed
" }, { "id": "https://msfjarvis.dev/posts/taming-my-intimidating-rss-reader-queue/", "url": "https://msfjarvis.dev/posts/taming-my-intimidating-rss-reader-queue/", "title": "Taming my intimidating RSS reader queue", "date_published": "2026-06-18T07:22:00.000Z", "summary": "How I regained control of my reading habits", "content_html": "

I talked in last week’s weeknote about how I was struggling with getting my Miniflux reader situation under control, and what I was attempting to identify the problem. Today, I came across this blog post from Michał Sapka (via this Fediverse post) talking about the exact feeling, and choosing to entirely drop RSS and rely on manually checking up on bookmarks instead.

\n

I think that was quite saddening to read, that the thing they had switched to as a way to get out of mindless algorithmic scrolling had ended up feeling like that itself. I’m personally not quite at the giving up stage, and over the weekend ended up making a bunch of changes within Miniflux that have significantly improved my experience already.

\n

I’m going to preempt the weeknote and say this: the whole data driven analysis approach I was trying out last week was a complete waste of time and mental faculties. I had misdiagnosed my problem of not reading things as being tied to individual feeds which wasn’t quite accurate. “Bad” feeds do contribute to and exacerbate the problem, but ultimately the thing I found to be the real issue was perception.

\n
\n

The feeds I subscribe to can broadly be categorized under the following labels

\n
    \n
  1. News outlets: high volume, mix of intriguing and uninteresting content, leaning towards the latter on most days
    2. \n
  2. Prolific individuals posting multiple times a week, sometimes multiple times a day. Unsung is a perfect fit here.
    3. \n
  3. Slower paced and longer length posters.
    4. \n
  4. Ed Zitron. He writes long insightful newsletters at frightening pace, and I am scared to put a label on this madman.
    5. \n
\n

The overwhelming feeling for me comes from the first category. It crowds the sorted-by-new unread queue and makes it look like the whole pile is just uninteresting things so you either close the tab/app or mark all as read and move on. The second problem is that cheeky unread count at the top of the screen which keeps reminding you that you’re a failure for not keeping up with this free knowledge you asked for.

\n

At least on Miniflux, these are both solvable problems, and it has helped me immensely in just the 3-4 days since I made these changes.

\n

For the first: Exclude the high volume/low signal feeds from the global unread list. Head over to Feeds > Click ‘Edit’ on the one you want > Check ‘Hide entries in global unread list’

\n
\"FeedExpand image
Hiding high volume feeds does wonders for your unread page
\n

You can also hide entire categories if your feeds are already categorized

\n
\"CategoryExpand image
Gone, just like that
\n

This removes the feed/category’s entries from the homepage while still letting me browse the posts individually in the feeds/category page when I am ready for my daily dose of Biz & IT.

\n

Now the second problem: the ever looming unread counter haunting your dreams can simply be hidden with a bit of CSS. Click over to the ‘Settings’ option at the top and then paste snippet in the ‘Custom CSS’ section on that page, then click the ‘Update’ button.

\n
.unread-counter-wrapper,
#page-header-title > span:nth-child(1) {
  display: none;
}
\n

With these changes you should find your default view of Miniflux a much more approachable space

\n
\"UnreadExpand image
The unread feed before changes, a big looming number over your head and a lot of relatively uninteresting things taking up most of your screen
\"UnreadExpand image
A much cleaner experience with more focus on writing by individuals, and more attuned to what you'd like to naturally discover in a casual browsing
\n

No unread counter screaming at you, and your default view now has a significantly better signal-to-noise ratio. Hope this helps someone out in creating a more approachable feed reader experience for themselves!

" }, { "id": "https://msfjarvis.dev/posts/i-guess-i-cook-now/", "url": "https://msfjarvis.dev/posts/i-guess-i-cook-now/", "title": "I guess I cook now", "date_published": "2026-04-01T09:26:00.000Z", "summary": "Learning to cook for myself went from necessity to hobby almost overnight", "content_html": "
\n

This is a post for April Cools, encouraging people to break out of their standard fare and write something new.

\n
\n

For the longest time I’ve not had a particularly great inclination for cooking. I had lived at or very close to my home for practically all my life, I was pampered (spoiled) and was never really encouraged to pick it up. That had to change when I moved to Bengaluru as part of being acqui-hired by Cloudflare in early 2025.

\n

I moved into a 2 bedroom apartment with old friend and current colleague Yash, and we decided to get a home cook. Having been in a hotel for a month or so while house-hunting we had landed pretty squarely on not wanting to be dependent on food delivery. Through some neighbourly drama that is not worth unpacking, we had to fire our home cook pretty quickly and we found ourselves in the kitchen with a strong desire to not starve. We started with the basics: simple curries with chapattis. Chickpeas, black beans, paneer (cottage cheese), and potatoes were pretty common.

\n

The progress was honestly pretty slow. We started with using precooked chapattis sold in the supermarket for a few weeks before shame really set in. We bought a food processor (neither of us had the temperament for kneading dough by hand), and started making our own chapattis at home. We made a very early experimental foray into a Taro root (called “Arbi” in Hindi) based curry, a favorite of Yash’, which was an absolute nightmare to prepare and really killed the vibe on gastronomical experimentation for a while. Eventually we started doing pastas, fried rice, pulao, even a chicken curry where we learned the hard way why you’re supposed to thaw chicken 6 hours in advance (it was edible).

\n

My mother, who had kept me out of the kitchen all this time, found herself quite concerned for the well being of her pampered buffoon. For maybe 6 months I had to send a picture of my dinner every night so she could rest assured that I had a home cooked meal and wasn’t pounding McChickens three times a day. My photo gallery in the past year is probably 40% dinner plates by count. I’m not gonna pretend I took all of those to satisfy her and not to gloat to myself, but she definitely is the reason it started :)

\n

I was already part of the air fryer cult back at home, so when my friend Subhrajyoti was moving out to the Netherlands I happily snapped up his air fryer. This unlocked a whole new dimension in the kitchen, and since then we haven’t gone a week without some type of frozen food in our freezer. The convenience is simply unbeatable, though Yash has not been a very big user of it so far. Some day he’ll see the light.

\n

Yash and I had been big fans of a hummus bowl sold by a food delivery service that was our staple breakfast at office for a while, and when they discontinued it I decided to finally look into how much effort it was to just make hummus myself. Turns out, not much! Hummus is just cooked chickpeas blended into a paste with maybe 5 ingredients added to it. I used the Christmas <> New Year’s week to try it out and ended up with a genuinely delicious batch.

\n
\"AExpand image
This was a bit of an arm workout not gonna lie
\n
\"AExpand image
4/10 plating but 8/10 flavour, I'll take it.
\n

The success of this experiment made me try this 3 more times to increasing levels of success, and also make my own falafels a couple of times. Still haven’t nailed that one, but got very close the last time!

\n

Yash also got us into making burritos ourselves, a repeat of the story with hummus but more driven by California Burrito being expensive rather than it disappearing altogether. It makes for something we can prep easily, lasts a couple of days, is quite healthy and very filling. Some highlights from the first burrito misadventures, before I resigned to just doing burrito bowls and skipping the embarrassment of not being able to wrap a tortilla very well.

\n
\"TwoExpand image
The internet told me this is how you make a burrito with two wraps.
\n
\"AExpand image
One of the rare few burritos that stayed in one piece when we grilled them.
\n

I bought a veggie chopper that lets me just make big piles of nutritious salads, topped with some air fried chicken sausages or whatever and just have an exceptionally filling dinner in under 20 minutes. Completely game changer for the days when I’m feeling lazy, has helped immensely with my struggle to lose weight.

\n
\n

When I started going to the IndieWebClub events late last year, I picked up the habit of publishing weeknotes. These have gradually started featuring food more prominently as I’ve gotten comfortable with simply trying things in the kitchen. Being able to navigate the kitchen with confidence has given me so much freedom that I previously thought to be out of my reach. Excited to see what other cuisines I’ll pick up this year!

" }, { "id": "https://msfjarvis.dev/posts/setting-up-forward-auth-with-caddy-and-pocket-id/", "url": "https://msfjarvis.dev/posts/setting-up-forward-auth-with-caddy-and-pocket-id/", "title": "Setting up forward auth with Caddy and Pocket ID", "date_published": "2026-03-21T10:03:00.000Z", "summary": "Using Pocket ID to secure services being proxied by Caddy", "content_html": "

As I mentioned in my last weeknote, I set up Calibre-Web last week which necessitated the use of a forward authentication setup to work with my existing SSO provider. It was rather non-trivial to get it all to work, so I’m documenting it here in hopes of helping others.

\n
\n

WARNING

\n

05/05/2026: Since writing this post I’ve come to realize that caddy-security very frequently makes subtle breaking changes, and you only figure out the migration by creating an issue yourself or reading the GitHub issue someone else created. That’s far too much churn than I wish to have in security-sensitive software. I will personally be moving to Authelia and LLDAP in the near future to leverage its first party support for forward and reverse auth, and a more stable release cycle.

\n
\n

Requirements

\n\n

Pocket ID and caddy-security setup

\n

Follow the Caddy guide here to set up an OIDC client and the caddy-security configuration in your Caddyfile. This gets you 90% of the way, but due to recent regressions in caddy-security you’ll need to make some tweaks.

\n

First of all, in the oauth identity provider block, add this line:

\n
trust login redirect uri domain exact ${app.domain} path prefix /
\n

Replace ${app.domain} with the domain to the service you are securing.

\n

The guide also assumes you will re-use the same caddy-security authentication portal for all your services which is fine, but I prefer to have each OIDC client be isolated on a service level instead of just having a generic caddy-security one. I’ll explain the basic changes first then dive into the NixOS-specific stuff I did for my own deployment.

\n
security {
  # Rename the provider from generic to the service name
  oauth identity provider calibreweb {
    delay_start 3
    # Give this its own realm
    realm calibreweb
    # This is the OIDC provider type, make sure this stays `generic`
    driver generic
    # Service-scoped secrets from the Pocket ID OIDC client setup
    client_id {$CALIBRE_WEB_POCKET_ID_CLIENT_ID}
    client_secret {$CALIBRE_WEB_POCKET_ID_CLIENT_SECRET}
    scopes openid email profile
    base_auth_url https://auth.msfjarvis.dev
    metadata_url https://auth.msfjarvis.dev/.well-known/openid-configuration
  }
\n
   # Rename the authentication portal as well
  authentication portal calibreweb_portal {
    crypto default token lifetime 3600
    # Use the new name for the identity provider defined above
    enable identity provider calibreweb
    trust login redirect uri domain exact books.msfjarvis.dev path prefix /
    cookie insecure off
    cookie domain books.msfjarvis.dev
    transform user {
      # The new realm name
      match realm calibreweb
      action add role user
    }
  }
\n
  # Per-service configuration
  authorization policy calibreweb_policy {
    set auth url /caddy-security/oauth2/calibreweb
    allow roles user
    inject headers with claims
  }
}
\n

With this setup you should be able to then copy-paste the code block above for a new service and just replace the name and URLs. Do note that the security block is globally unique, and only the contents inside it are meant to be duplicated.

\n

Sidebar: NixOS version

\n
\n

People not using NixOS can ignore this section

\n
\n

On NixOS I already have a Caddy module that applies a set of defaults, so I retrofitted generating the security block into it. I added an option to let individual services configure the inputs via module options:

\n
pocketIdApplications = mkOption {
  type = types.attrsOf (
    types.submodule {
      options = {
        domain = mkOption {
          type = types.str;
          description = \"Domain of the proxied service\";
        };
        clientIdEnvVar = mkOption {
          type = types.str;
          description = \"Environment variable name containing the client ID\";
        };
        clientSecretEnvVar = mkOption {
          type = types.str;
          description = \"Environment variable name containing the client secret\";
        };
      };
    }
  );
  default = { };
  description = \"Applications to protect with Pocket ID OIDC via caddy-security\";
};
\n

Then in the services.caddy.globalConfig option, you loop over the value of this to generate the config we wrote above.

\n
${lib.optionalString (config.services.caddy.pocketIdApplications != { }) ''
  security {
    ${lib.concatStringsSep \"\\n\" (
      lib.mapAttrsToList (name: app: ''
        oauth identity provider ${name} {
          delay_start 3
          realm ${name}
          driver generic
          client_id {${app.clientIdEnvVar}}
          client_secret {${app.clientSecretEnvVar}}
          scopes openid email profile
          base_auth_url https://auth.msfjarvis.dev
          metadata_url https://auth.msfjarvis.dev/.well-known/openid-configuration
        }
\n
        authentication portal ${name}_portal {
          crypto default token lifetime 3600
          enable identity provider ${name}
          trust login redirect uri domain exact ${app.domain} path prefix /
          cookie insecure off
          cookie domain ${app.domain}
          transform user {
            match realm ${name}
            action add role user
          }
        }
\n
        authorization policy ${name}_policy {
          set auth url /caddy-security/oauth2/${name}
          allow roles user
          inject headers with claims
        }
      '') config.services.caddy.pocketIdApplications
    )}
  }
''}
\n

With all this in place, you can then configure this alongside (in my case) Calibre-Web and have all the stuff above get generated without having to think too much about it:

\n
sops.secrets.calibre-web-caddy-env = {
  sopsFile = lib.snowfall.fs.get-file \"secrets/calibre-web.env\";
  format = \"dotenv\";
  owner = config.services.caddy.user;
  restartUnits = [ \"caddy.service\" ];
};
services.caddy.pocketIdApplications.\"calibreweb\" = {
  domain = cfg.domain;
  clientIdEnvVar = \"$CALIBRE_WEB_POCKET_ID_CLIENT_ID\";
  clientSecretEnvVar = \"$CALIBRE_WEB_POCKET_ID_CLIENT_SECRET\";
};
systemd.services.caddy.serviceConfig.EnvironmentFile = [
  config.sops.secrets.calibre-web-caddy-env.path
];
\n

Caddy routing setup

\n

Usually just throwing a reverse_proxy localhost:<port> gets you all the way with Caddy, but Calibre-Web specifically requires some extra configuration that serves for a nice demonstration. We need to treat different paths in different ways, so we’ll use the Caddy handle directive to create mutually exclusive routing paths for them.

\n

caddy-security adds a /caddy-security path in your sites that should always require authentication, so we simply route it to our portal without any changes.

\n
handle /caddy-security/* {
  route {
    authenticate with calibreweb_portal
  }
}
\n

Caddy lets you label a set of paths or with the @name syntax, this is just here for convenience. Calibre Web exposes OPDS and Kobo-specific endpoints for use with your devices that likely can’t do OIDC, so we allow those to bypass the authentication requirements. The transport buffer size was taken from https://github.com/janeczku/calibre-web/issues/1891 after I faced the same sync issues.

\n
@integrations {
  path /opds /opds/* /kobo /kobo/*
}
handle @integrations {
  reverse_proxy localhost:8080 {
    header_up X-Scheme https
    transport http {
      read_buffer 1024k
      write_buffer 1024k
    }
  }
}
\n

This is the catch-all route for everything else that doesn’t need special treatment. This only differs from the one above in requiring authorization.

\n
handle {
  route {
    authorize with calibreweb_policy
    reverse_proxy localhost:8080 {
      header_up X-Scheme https
      transport http {
        read_buffer 1024k
        write_buffer 1024k
      }
    }
  }
}
\n

The full configuration then becomes this:

\n
handle /caddy-security/* {
  route {
    authenticate with calibreweb_portal
  }
}
@integrations {
  path /opds /opds/* /kobo /kobo/*
}
handle @integrations {
  reverse_proxy localhost:8080 {
    header_up X-Scheme https
    transport http {
      read_buffer 1024k
      write_buffer 1024k
    }
  }
}
handle {
  route {
    authorize with calibreweb_policy
    reverse_proxy localhost:8080 {
      header_up X-Scheme https
      transport http {
        read_buffer 1024k
        write_buffer 1024k
      }
    }
  }
}
\n

Sidebar: Authentication vs Authorization

\n

This is something that took too long for me to hammer into my brain, so I’ll reproduce it here as well for others who may have had the same question but didn’t feel confident asking it.

\n\n

We require authentication for the caddy-security portal because you should be able to at least see that you have an account. The rest of the service requires authorization so only people who should be allowed access to the Calibre-Web instance are able to log in.

\n
\n

And that’s the long and short of it! There was a lot of one-time effort in getting this working and I had to dig through a lot of GitHub issues to find bits and pieces that had to be tweaked, but I’m quite happy with how this all came together. I now have Calibre-Web syncing to my Kobo eReader, and no longer have to struggle with a split desktop library back at home and here in Bengaluru.

" }, { "id": "https://msfjarvis.dev/posts/migrating-my-website-s-analytics-from-plausible-to-umami/", "url": "https://msfjarvis.dev/posts/migrating-my-website-s-analytics-from-plausible-to-umami/", "title": "Migrating my website's analytics from Plausible to Umami", "date_published": "2026-01-09T19:10:00.000Z", "summary": "Frustrated by ClickHouse's high disk usage, I ditched Plausible for Umami", "content_html": "

On January 5th I had another minor outage on one of my servers due to a full disk, caused in part by misconfigured Forgejo data dumps. In the process of fixing that I discovered that most of my server disk was actually being used by ClickHouse, which is holding data for my Plausible instance that records page views for this website.

\n

Why does it need 108 gigabytes to store 102K page views? I wish I could tell you. One of the developers of Plausible replied to my grumbling explaining that it was likely ClickHouse collecting a ton of logs which was the real storage hog rather than the data itself. By that point I had already finished migrating off to Umami and deleted the offending ClickHouse data so I couldn’t verify the hypothesis.

\n

The migration itself was quite simple so I’ll reproduce the steps below for any interested people looking for migration steps.

\n\n

And that’s kind of it! After this I turned off Plausible and was able to delete the ClickHouse directory and reclaim the nearly half of my 256 GB disk it had been keeping hostage. Umami’s lack of OIDC and easy import options is definitely a sore point, but with how easy the whole thing ended up being for me I can live with it for the time being. Interestingly, Plausible only lasted 8 months before I moved out of it. I should’ve been more wary of ClickHouse than I was of MySQL when I chose Plausible over Matomo :)

" }, { "id": "https://msfjarvis.dev/posts/coming-around-on-the-utility-of-llms/", "url": "https://msfjarvis.dev/posts/coming-around-on-the-utility-of-llms/", "title": "Coming around on the utility of LLMs", "date_published": "2026-01-02T16:53:00.000Z", "summary": "Having unsuccessfully tried to use Windsurf and GitHub Copilot over the years, OpenCode finally made LLMs useful to me", "content_html": "
\n

LLMs are an extremely divisive topic so I hesitated for a while before writing this post but reading this blog from Joan Westenberg convinced me to not worry too much about what I put on my own blog.

\n
\n

For the longest time I’ve found LLMs to be quite underwhelming. I tried all the new and fancy models on and off, first via GitHub Copilot which I get for free for being an active open source maintainer, and then with Windsurf at work. The sycophancy was annoying, the training data was always outdated, and the models were yet to learn how to accept they had no fucking idea about the thing I asked.

\n

That only came to change very recently, with OpenCode. We have access to it at work via the Cloudflare AI Gateway and the latest Anthropic models.

\n

OpenCode is quite capable by default, with sensible defaults and a functional UI to go with it. What really unlocked its capabilities for me was oh-my-opencode, a plugin that supercharges OpenCode by giving it the capabilities to launch background tasks via “agents” that enhance its capabilities across specific dimensions such as online research, UI design and system design. It adds integrations with the Language Server Protocol to ensure edits don’t introduce diagnostic errors. It includes a comment checker to prevent the typical annoyance of over-commenting code and giving it the distinct LLM smell. It bundles a set of curated MCP servers to enhance its ability to do research: Exa to do web searches, Context7 to find up-to-date documentation, and Grep to efficiently search through code on GitHub. All of these complicated bells and whistles come together to become a cohesive tool that can tackle both easy and difficult programming tasks with a very high degree of reliability.

\n

I’ve only started using OpenCode outside work in the past week or two, but the value add is undeniable. Here’s a non-exhaustive list of things I used OpenCode for in Claw, with varying degrees of complexity and impact.

\n\n

None of these are things that would be impossible for me to figure out, but they sure as hell wouldn’t have been done in the span of two weeks. I don’t really subscribe to the idea of fully phoning it in, the so called “vibecoding” aspect of using an LLM as a coding assistant. In every case where the LLM has solved a bug, I’ve learned something new from the fix and even went beyond in many cases because of the leg up I got by using the LLM.

\n

With the Sentry MCP server, I can spin up an OpenCode session, say “Investigate the root cause for Sentry issue <id> and include references in the summary” and go off to do something else. When I get back, I have a better understanding of the issue in question, I can use the references to do my own research and tell the LLM to implement one of potentially multiple ways to fix the bug. In many situations, writing the actual code is just not the more fun part of the problem.

\n

That said, I do want to clarify that I still find writing code to be enjoyable or fulfilling in most cases even with an LLM in the picture. This pull request was mostly me doing a bulk find -> replace across the codebase, something an LLM would excel at. However, I believe I am more capable of learning about Metro’s capabilities ahead of time and spotting opportunities to replace existing constructs with Metro-native implementations, so I did it by hand and manually reviewed each file.

\n

Just a few months ago, I would not have believed that I could use LLMs to write any code I intended to actually maintain. Similar to Simon Willison’s tools.simonwillison.net, I had also set myself up with a small repo and a public page that hosted those tools. These things still have the default LLM sheen to them, and I gave up on even having them follow a personalized code style. For all definitions of the phrase, it is vibe coding. They solve simple problems for me, and will likely never break or need an update.

\n

Despite my changed opinions about LLMs as programming assistants, I am vehemently against the anthropomorphizing of LLM-based chatbots. They do very real harm, and should be very aggressively regulated. I hold an even stronger belief that image and video generation models should not be available to the general public at all. The fact that humans can no longer trust even their eyes is deeply disturbing to me. It has been proven that these technologies will actively be misused for deceit and in the case of Elon Musk’s xAI, sexual harassment of minors.

\n

This is close to devolving into rambling, so I’ll end it here. I don’t intend to form a crippling reliance on LLMs anytime soon, nor start paying for them. As long as GitHub is willing to continue wasting money on giving me access to LLMs for using their website, I’ll use them for my personal projects. When they stop, I’ll just go back to the good old elbow grease that has got me this far. If I can write Java over SSH into nano, I think I can still use a search engine by hand.

" }, { "id": "https://msfjarvis.dev/posts/my-experience-at-droidcon-india-2025/", "url": "https://msfjarvis.dev/posts/my-experience-at-droidcon-india-2025/", "title": "My experience at droidcon India 2025", "date_published": "2025-12-14T06:47:00.000Z", "summary": "I finally stepped out of my house! And met people! So cool!", "content_html": "
\"MyExpand image
My droidcon 2025 badge
\n

I was at droidcon India this Saturday and got the chance to meet a bunch of people I hadn’t seen in a while and attend some really great talks. It was a great experience and definitely motivated me to come out for meetups and events a bit more and be less of a loner :P

\n

Before I start walking through my day, I would like to extend an apology to my friends who accompanied me to the conference and the Android developer community at large. I might be the singular dumbass who would be at an event where both Adit Lal and Jitin Sharma were speaking and plan an itinerary that included neither of their talks. I am extremely sorry and will never live this down.

\n

We arrived at the Conrad Bengaluru a little before 9 AM to an already bustling space and settled into the Fireside Chat with Google’s DevRel Team. Lots of talk about AI as expected, but also the topic of passkeys and Credential Manager potentially doing away with the ridiculous OTP authentication flows used by banks here in India (🤞 one can hope).

\n

After a short break we attended our first talk, Suresh Maidaragi speaking about Building Mobile Apps at Scale with Kotlin Multiplatform. It was a case study on their experience at Physics Wallah of unifying a lagging iOS app with their higher development velocity Android app via KMP. I was interested in learning some new things here as my work has involved a Kotlin Multiplatform powered SDK for ~2 years now, but was left a little disappointed as the talk was rather shallow on technical details. I was rather surprised horrified to hear that their app takes 2 hours to build in their release CI with their current setup!

\n

Another short break and then one of my highlights for the day: Rahul Ravikumar’s “A Busy Android App Developer’s Guide to Perfetto”. Inspired by Lalit Maganti’s recent writing I had been teaching myself how to use Perfetto to debug some performance issues at work, so this was perfect for me and the colleagues I had dragged along to the conference (I joke, they came willingly. I think). This talk was very information-dense, I learned a lot about navigating Perfetto and making sense of its seemingly endless UI components. We got a sneak peek at the in-development Tracing 2.0 library and all the problems it’s solving from Tracing 1. Especially exciting was context propagation, which can connect parent coroutines to their children in the Perfetto UI allowing you to effectively trace multistep async operations.

\n

The next talk I attended was Himanshu Singh speaking about “Building Android Open Source Libraries: Managing Public APIs with Intention”. Another talk I found to be rather underwhelming, as the synopsis in the agenda made me assume this would carry a little more technical content than it ended up with. Himanshu mostly spoke about the social aspects of starting and managing open source projects in general before mentions of Kotlin BCV and release automation towards the end.

\n

After the talk we had a lunch break, and I got to catch up with Pratul Kalia who used to be my CTO at Obvious. He’s now building Tramline to make mobile releases as streamlined as deploying to the web. I’m a happy solo user of the product, but am otherwise not paid to shill it here :P

\n

We skipped the 1:30 PM to 2:00 PM time slot since and just hung out since none of the talks particularly appealed to our small group of 4.

\n

At 2:00 PM we headed to Sid Patil and Rutvik Bhatt’s talk: “The Art of Re-architecture: Lessons from the Trenches at Foodpanda & Delivery Hero”. As someone currently in the weeds of planning a re-architecture myself, their deep insights on how to evaluate the desired outcomes and engaging stakeholders of an engineering investment like this were very helpful. The mantra of “Refactor for today, Re-architect for tomorrow” was particularly inspiring as the resident refactorman of my team 👌

\n

Next up was Rajesh Hadiya with “Build Kotlin Compiler Plugins for Production level Android Apps”. Despite the extremely dense topic he did a great job at making it fun and approachable. I’ve tried to solve a logging-related problem earlier last year with a Kotlin compiler plugin so I was pretty familiar with the pain involved in dealing with the Kotlin IR and the various compiler backends. Rajesh was able to explain the compiler fundamentals that enable the various frontend/backend compiler plugins work the way they do in a succinct manner.

\n

We skipped the 3:45 PM to 04:05 PM slot as well due to nothing particularly appealing to the group.

\n

Omkar Tenkale’s talk titled “Building the Coroutine Framework from Scratch” fully delivered on the premise. We started by talking about the distinctions between kotlinx.coroutines and kotlin.coroutines, then fully throwing away the former and just relying on the standard library + compiler intrinsics to rebuild the primitives we’ve come to rely on from kotlinx.coroutines such as runBlocking, launch, Dispatchers , GlobalScope and such. The talk was packed to the gills with low level details yet was very approachable and I left the room a lot more knowledgeable and appreciative of kotlinx.coroutines! Omkar’s was easily my second favorite talk here after Rahul’s Perfetto deep dive.

\n

The last talk we attended was Nikheel Vishwas Savant from Meta talking about their experience building up a “Cross-Platform Bluetooth Architecture for Android, iOS, and Embedded Devices” for the Meta RayBan glasses. It was expected yet disappointing that we didn’t see any code in this talk, but there were a lot of great nuggets of information that I expect to take back to my work on RealtimeKit.

\n

And that was it for droidcon! By 5:30 PM we were all pretty exhausted so I bid my goodbyes to the people staying back and explored the nearby Church Street area with a couple of friends before heading home.

\n

This was my first proper con-style event in a long time (the last time I was at one was Google Developer Days back in 2017!) so I was somewhat nervous but it ended up being a great time. I missed a few people who were also there which was sad, but now that I’m in Bengaluru full time I hope to take more opportunities to show up at meetups and see people more than once every 3 years :)

" }, { "id": "https://msfjarvis.dev/posts/creating-private-services-on-nixos-using-tailscale-and-caddy/", "url": "https://msfjarvis.dev/posts/creating-private-services-on-nixos-using-tailscale-and-caddy/", "title": "Creating private services on NixOS using Tailscale and Caddy", "date_published": "2025-09-13T15:48:00.000Z", "summary": "A simple guide to setting up private services on NixOS using Tailscale and Caddy with authentication.", "content_html": "

Tailscale is a mesh VPN that makes it dead simple to connect almost any device together in a private network. Caddy is a web server that focuses on ease of use and automatic HTTPS. I am a fan of both of these, and I was very excited to discover that Tailscale has an experimental integration with Caddy that leverages their tsnet library to allow creating unique Tailscale addresses for individual virtual hosts in your Caddy configuration. Here’s a quick run down of how to set this up on NixOS.

\n

Installing caddy-tailscale for NixOS

\n

As of the latest NixOS release, the Caddy package in nixpkgs supports including plugins directly which greatly simplifies this step. As an example, here’s how I include the caddy-tailscale plugin with the excellent caddy-defender that lets me drop traffic from AI scrapers more safely.

\n
\n

You can also keep the package definition in a separate file like this, and use this script to keep it up-to-date. DISCLAIMER: The script was vibe coded by Claude, all I’ve done is checked it for obviously silly shit.

\n
\n
services.caddy = {
  enable = true;
  package =
    pkgs.caddy.withPlugins {
      plugins = [
        \"github.com/jasonlovesdoggo/caddy-defender@v0.8.5\"
        \"github.com/tailscale/caddy-tailscale@v0.0.0-20250207163903-69a970c84556\"
      ];
      hash = \"sha256-z+zj3rfXbyxldRjO1yoLD77ACRWEAofzMDiZe/bHAqw=\";
   }
  globalConfig = ''
    servers {
\n

This will swap out the caddy binary in your NixOS configuration with one that has the caddy-tailscale module patched in. You will also need to provide a Tailscale authkey to allow the caddy-tailscale module to interact with the Tailscale API and provision new nodes in your Tailnet. This can be done by setting the TS_AUTHKEY environment variable in your Caddy service configuration. Here’s an example of how to do so using sops-nix:

\n
sops.secrets.services-tsauthkey-env = {
  sopsFile = ./secrets/tailscale.env;
  owner = config.services.caddy.user;
  format = \"dotenv\";
};
services.caddy.environmentFile = config.sops.secrets.services-tsauthkey-env.path;
\n

Create the secrets/tailscale.env file with sops, the contents should look something like this:

\n
TS_AUTHKEY=tskey-auth-something-something
\n

Setting up your first private service

\n

Now we’re ready to rumble. To showcase the full capabilities of caddy-tailscale, we’re going to set up Grafana with proxy authentication and let Tailscale handle the authentication for us. Here’s a simple example (N.B. $TAILNET_NAME should be replaced with your Tailscale network name.):

\n
services.caddy.virtualHosts = {
  \"https://${config.services.grafana.settings.server.domain}\" = {
    extraConfig = ''
      # This will create a new node at grafana.$TAILNET_NAME.ts.net
      bind tailscale/grafana
      # Enables the Tailscale authentication provider
      tailscale_auth
      # Forwards your Tailscale user ID to Grafana. Usually your email address.
      reverse_proxy ${config.services.grafana.settings.server.http_addr}:${toString config.services.grafana.settings.server.http_port} {
        header_up X-Webauth-User {http.auth.user.tailscale_user}
      }
    '';
  };
};
\n
services.grafana = {
  enable = true;
  settings = {
    \"auth.proxy\" = {
      # Enable proxy-based authentication
      enabled = true;
      # Automatically create new accounts for people who access the service
      auto_sign_up = true;
      # Do not store login cookies, instead always relying on proxy authentication
      enable_login_token = false;
    };
    server = {
      domain = \"grafana.$TAILNET_NAME.ts.net\";
      http_addr = \"127.0.0.1\";
      http_port = 2342;
    };
  };
};
\n

This configuration will set up a new Grafana instance at https://grafana.$TAILNET_NAME.ts.net that is only accessible to members of your Tailscale network. When you visit the URL, your Tailscale identity will be used to log you into your existing Grafana account, or sign you up if this is your first time visiting the service.

\n
\"TheExpand image
Grafana profile showing that user identification is being done via the Tailscale proxy
\n

Caveats

\n

While this approach works great for most needs, there are a few caveats to be aware of:

\n\n

Conclusion

\n

Despite its experimental status, caddy-tailscale is pretty powerful and exposes enough functionality to cover most basic use cases. I’ve been pretty happy running it in my homelab for the past couple months and I’m excited to see how it evolves in the future. If you have any questions or corrections, please let me know either in the comments below or on Mastodon.

" }, { "id": "https://msfjarvis.dev/posts/migrating-from-gitea-to-forgejo-the-long-way/", "url": "https://msfjarvis.dev/posts/migrating-from-gitea-to-forgejo-the-long-way/", "title": "Migrating from Gitea to Forgejo the long way", "date_published": "2025-09-01T14:02:00.000Z", "summary": "With the renewed interest in Forgejo I decided to finally pull the plug on moving out of Gitea, and this is how it went.", "content_html": "

I’ve had a Git server at https://git.msfjarvis.dev for a while now, running Gitea but my faith in the project’s open core model has steadily been going down. When Codeberg announced they were forking Gitea as Forgejo I quietly put down a line item to switch over in my overflowing TODO list and promptly forgot about it.

\n

GitHub has been getting increasingly unpleasant for the past few years, but them officially announcing its demise as an independent entity and joining Microsoft’s “CoreAI” division was the straw that broke the camel’s back and made me want to use my own Git forge more actively. Thus the Forgejo migration shot up in my TODO list, and ended up being a rather involved process.

\n
\n

This post is gonna have some rambling and going around in circles, but the TL;DR is that I ended up having to migrate individual accounts instead of the whole instance data by using a tool that I ironically generated using GitHub Copilot (the code is available on my Git server).

\n
\n

Attempt 1: The naive way

\n

Forgejo initially used to support migrating data from Gitea pretty easily, but the increased friction of having to maintain a full fork against a moving target forced them to drop support for these migration paths. I of course am stupid and was not aware that I had missed my window, so I had to make the unpleasant discovery that simply swapping the latest Gitea for latest Forgejo will not work anymore. The NixOS manual’s migration section suggested that this was possible, but it’s actually missing crucial information about version requirements that was added later.

\n

My naive attempt almost caused me to irreversibly corrupt my instance data, so I was more careful with future attempts and used a backup each time.

\n

Attempt 2: The gitea dump command

\n

This looked like it had some potential until I realized the lack of a companion gitea restore functionality. The generated dump is a big ZIP file of your entire state directory, along with the SQL commands to recreate your database with whatever database engine you were using. The recommended way to use this dump is to unzip it and then replay the SQL commands into sqlite (my current database) to build up the state directory. This approach kind of felt like a dead-end already but I gave it a good try anyway.

\n

I had also chosen to run Forgejo using PostgreSQL instead of SQLite, since I was already using it with a bunch of other software running on this machine and had proper backup strategies in place. This meant I had to reach out for pgloader to import the SQLite-compatible dump into Postgres which worked out pretty well. Running pgloader sqlite:///var/lib/gitea/data/gitea.db 'postgresql://forgejo:@/forgejo?host=/run/postgresql' as root was able to successfully migrate the database to Postgres.

\n

Unfortunately this didn’t work due to the previously discussed version incompatibilities, and Forgejo rejected this database I had created for it. At this point I had lost a couple hours on this and constant electricity problems at my place had additionally caused my patience to run thin. I decided to cut my losses and re-deployed Forgejo with the same settings as my Gitea server, deciding that I would figure out a way to just copy my account data over since the instance configuration was managed by NixOS anyway (for the most part).

\n

Attempt 3: Vibe coding deployed somewhat effectively

\n
\n

I’m not generally a big believer in the AI hype. I have yet to pay for any of these tools, between GitHub and my employer I get plenty of access to top of the line models that are supposedly reinventing my field of work every 3 months. LLMs have yet to meaningfully help me at work, and the only times I’ve gotten real value out of them is by getting them to write one-off things like this that I am glad to have but would likely never invest the time to upskill into and build myself.

\n
\n

I hooked up the free license to GitHub Copilot that I get for satisfying some criteria of “popular open source maintainer” into Zed and wrote out a simple README file describing what I wanted out of the tool and had it go to town. The end result of this was an unnecessarily abstracted Go project (I doubt anybody would use that directory structure for a project this small) that looked like it would do the job. You can find the source code for this here.

\n
\n

According to a comment on this post, organizations may not be migrated correctly by this script. I do not wish to change the code to an untested version so be prepared to do some manual fixes to address that deficiency. I would also accept a patch from someone who does end up fixing the script for themselves.

\n
\n

Now onto actually running this tool. For this to work, it would require both my new Forgejo server and my old Gitea server to be up at the same time. First hurdle: conflicting ports. This was solved pretty easily.

\n

I screwed up here by running Forgejo on the primary domain and deploying the old Gitea server into a new Tailscale-based service. This caused multiple failures:

\n

Inability to log into the Gitea instance.

\n

I could no longer sign into Gitea since I had 2FA enabled using Passkeys which require the domains to match up.

\n

To make up for the inability to log into my Gitea server to create an access token, I just used the gitea CLI instead.

\n
sudo -i su - gitea
# To get the actual path to the Gitea binary, which weirdly isn't installed for the gitea user?
systemctl cat gitea.service | grep ExecStart=
/nix/store/foo-bar-baz-gitea-1.23.1/bin/gitea admin -w $(pwd) user generate-access-token --token-name forgejo-migration --scopes \"read:repository,read:user\" --raw
\n

On the forgejo side, a similar dance ensued but this time to actually create my account since this was a fresh start.

\n
sudo -i su - forgejo
systemctl cat forgejo.service | grep ExecStart=
# yes the actual CLI seems to still be available as gitea
/nix/store/foo-bar-baz-forgejo-10.0.0/bin/gitea admin user create -w $(pwd) --username msfjarvis --email me@msfjarvis.dev --admin --access-token --access-token-name forgejo-migration --access-token-scopes \"read:user,write:repository\"
\n

With access tokens in hand, I ran the tool and hit my second problem.

\n

Tailscale ACLs bite me in the ass

\n

The way Tailscale’s networking shebang works meant that the server running this isolated Gitea service couldn’t connect to it via the network without some tweaks to my ACL policies. I was not feeling like I had this extra debugging in me, so I opted to just temporarily hijack another subdomain pointing to this server for the purpose and resumed from there.

\n

Mirrors of GitHub private repos did not work

\n

Back when I first created mirrors of all my stuff I had also done so for my private repos and just forgot about it. When the tool tried to migrate them they obviously failed to mirror since I wasn’t providing any access tokens for GitHub. I went back to Zed and had Claude add a retry for this and pull an access token from the $GITHUB_TOKEN environment variable.

\n

When I ran the tool again it was able to successfully recreate all the repositories from my Gitea server.

\n

Conclusion

\n

Overall this migration was worthwhile for me, both for documenting it for others in my position as well as for finally being free of Gitea. I trust the people in charge significantly more, and they continue to deliver great work driven in part by them actually using their own software at scale.

" }, { "id": "https://msfjarvis.dev/posts/mildly-overengineering-my-glance-configuration/", "url": "https://msfjarvis.dev/posts/mildly-overengineering-my-glance-configuration/", "title": "Mildly overengineering my Glance configuration", "date_published": "2025-05-07T17:35:00.000Z", "summary": "The story of setting up a live environment for configuring my Glance dashboard", "content_html": "
\n

May 10th update: Powered by a lack of sleep and extreme fatigue I cooked up a far simpler solution that is mentioned at the end of the post.

\n
\n

Setting the scene

\n

I’m a very happy user of the Glance dashboard, and make use of it multiple times a day.

\n

As a NixOS user, I have a very popular problem of wanting to iterate on my Glance configuration without having to rebuild my whole system every time to see the change.

\n

One way I’ve seen this addressed (please message me on @msfjarvis@fantastic.earth if you can find the actual blog post) is to have some sort of “dev” switch that generates a writable version of your configuration using some home-manager tricks that you can poke at before encoding it into Nix. Turning off the aforementioned “dev” mode will make the configuration go through the more usual Nix ways and give you your declarative system back.

\n

I think that idea is cool! I also think that idea kinda sucks when the configuration is in a “language” you despise, like YAML. Which is where I am!

\n

Coming up with a solution

\n

At the end of the day, the problem I have is that there are too many intermediate steps between me writing Nix and it being converted into YAML. So let’s break it down!

\n

Phase 1: Pull out the Nix bits I need to a standalone file that I can muck around with: done

\n

Now, we need a way to convert this Nix code into a YAML file that Glance can ingest. I came across a relatively new project that solves exactly this: nix-converter.

\n

Running nix-converter --from-nix -f path/to/file.nix -l yaml outputs YAML to stdout, ready to be shoved wherever you desire. This worked almost flawlessly, except an easily worked-around issue:

\n
{
  # This generates the following (incorrect) YAML
  # foo: \"baz\"
  foo.bar = \"baz\";
  # This way of declaraing the attrsets works around the
  # issue and generates the following:
  # foo:
  #   bar: baz
  foo = {
    bar = \"baz\";
  };
}
\n

Glance will automatically reload itself if the config file changes, which\ngreatly simplifies the setup. The only remaining bit we have is to automatically\ngenerate this YAML every time we make a change to the Nix file, so that we can freely\nswitch between editor and web browser to see changes without taking a trip to the\nterminal.

\n

I solved this using watchexec and a tiny Bash script that I stuffed\ninto a devshell command:

\n
Terminal window
out_conf=$(mktemp)
${lib.getExe pkgs.watchexec} -r -w \"${glanceConf}\" -- \"${lib.getExe pkgs.nix-converter} --from-nix -f ${glanceConf} -l yaml > $out_conf\" &
${lib.getExe pkgs.glance} -config \"$out_conf\"
\n

This is a mix of regular Bash and Nix which will make sense to Nix users, but\nfor everyone else: ${lib.getExe pkgs.watchexec} gives you a path of the form\n/nix/store/store-hash-watchexec-version/bin/watchexec which is the binary you\ncan execute. $glanceConf is a Nix variable containing a string path to the\nglance.nix file we pulled out earlier.

\n

All put together it looks like\nthis. If I need to edit my\nGlance config, I will go into the glance devShell, boot up\nZed, run the dev command in my terminal and enjoy a fully\nautomated iteration experience.

\n

Caveats

\n

This setup is not even remotely perfect, nix-generator has bugs and discrepancies compared to the Nix to YAML implementation inside Nixpkgs. Not all constructs that work fine with Nixpkgs work that well with nix-generator, complicating development.

\n

May 10th update

\n

Out of the blue I had a minor crisis looking at the kludge I had previously put together and realized there was actually a simpler way to be generating this YAML code without having to forego the niceties of Nixpkgs. The end result of that is this commit, which I will explain below.

\n

The bulk of the work is now handled by a bash script, which I have annotated as comments below to explain what’s going on.

\n
#!/usr/bin/env bash
\n
# EXPR is a Nix expression that we will be evaluating
# and building to obtain our final YAML
EXPR=\"
let
  # Load nixpkgs from the search path because it's the easiest way to do it.
  pkgs = import <nixpkgs> { };
  # This is the same code we use in our Glance module to generate the YAML
  settingsFormat = pkgs.formats.yaml { };
  settingsFile = settingsFormat.generate \\\"glance.yaml\\\" (import ./${1:?});
in
# settingsFile returns a derivation that the `nix build` below will, well, build.
\\\"\\${settingsFile}\\\"
\"
\n
nix build \\
  # Prevent Nix from trying to check binary caches each time we build
  --option substitute false \\
  # Feed the nixpkgs input from your flakes registry to the search path
  --impure -I nixpkgs=flake:nixpkgs \\
  # Pass in our expression from above that will emit the derivation to be built
  --expr \"${EXPR}\"
\n
# ./result is the default nix3-build output path
glance -config ./result
" }, { "id": "https://msfjarvis.dev/posts/migrating-from-simple-analytics-to-self-hosted-plausible/", "url": "https://msfjarvis.dev/posts/migrating-from-simple-analytics-to-self-hosted-plausible/", "title": "Migrating from Simple Analytics to self-hosted Plausible", "date_published": "2025-04-22T11:13:00.000Z", "summary": "Documenting the misadventures of self-hosting my site analytics", "content_html": "

For the longest time (nearly 6 years according to the data export), I have been a user of Simple Analytics and their dead simple web analytics offering. I believe I got access to it via GitHub’s big bundle of freemium services provided to students and they just didn’t bother checking that I had gone to and dropped out of college twice since they first offered me free access until this month.

\n

So while I was rather satisfied with the free ride I had gotten from Simple Analytics till now, having to pay a rather eye-watering 15 USD every month for it was stretching the otherwise $0 USD budget for my blog. The site itself is hosted by Netlify, but I have a small server from netcup which was suitable to take on the load of running a self-hosted analytics server that I could switch to.

\n

Choosing the analytics service

\n

Like the absolute chump I am, my first visit was to awesome-selfhosted’s analytics page and from there, the only options that stuck out were Plausible and Matomo.

\n

I will be honest here, the reason to not choose Matomo was very superficial - I didn’t want to be running PHP and MySQL. Plausible is an Elixir app and thus not much better as far as me understanding the language goes, but I was reasonably more confident about its choice of Clickhouse and Postgres as the backing data stores.

\n

Getting set up

\n

My server runs NixOS, and thankfully there is a module for Plausible already so setup was very simple and all I had to add was this:

\n
services.caddy.virtualHosts = {
  \"https://stats.msfjarvis.dev\" = let
    p = config.services.plausible.server;
  in
  {
    extraConfig = ''
      reverse_proxy ${p.listenAddress}:${toString p.port}
    '';
  };
};
\n
sops.secrets.plausible-secret = {
  sopsFile = lib.snowfall.fs.get-file \"secrets/plausible.yaml\";
};
services.plausible = {
  enable = true;
  server = {
    baseUrl = \"https://stats.msfjarvis.dev\";
    secretKeybaseFile = config.sops.secrets.plausible-secret.path;
  };
};
\n

The only hiccup here was me apparently not knowing what the hell is 64 bytes supposed to mean and failing to fill in a long enough string into the secretKeybaseFile, which is how it came to end with brother-man-please-work-what-the-fuck-man during testing after which I swapped it out for a 20 word passphrase generated by Bitwarden.

\n

Once everything was running, I set up an admin account and replaced the SimpleAnalytics JS snippet on my website with the one Plausible gave me. Notable difference here was a lack of tracking for users who disable JS, so that’s a net improvement to user privacy that I was too naive to think about in 2020.

\n

Migrating existing data from Simple Analytics

\n

To their credit, Simple Analytics made this very easy and the export option is at the top of their settings page. My only complaint is about their range selection feature, I found the calendar widget very un-intuitive and the input field didn’t let me just fill in the date manually.

\n

Once I had the 45 mB CSV in hand, I looked around for someone else’s homework to copy so I didn’t have to write the migration tool myself. I came across this repo which looked like it would do the job but I had no plans to install NodeJS, so I had gpt-4o (via my free, GitHub-provided Copilot license) rewrite it as a single-file Python script that I could hack on more comfortably. That version is available here, with some additional changes described below.

\n

I ran the script and got a handful of CSVs that Plausible was happy to import, after which I realized that the location data was completely missing. Indeed, my hopes of not having to write any code were going to remain so.

\n

I pulled up the Plausible CSV import docs, got the headers it was expecting for the location CSV and was able to piece together the data relatively easily. One big omission was Simple Analytics not recording cities for page visits, which Plausible had a field for. After figuring out what it was expecting, I filled in zeroes for that column.

\n
\n

I have a big gripe with Plausible’s non-existent error reporting here. When my newly generated CSV failed to import, Plausible simply put up an ⚠️ sign in the list of imports and offered no logs. After digging through their GitHub issues I discovered that the errors had to be pulled out from a Postgres table which I was not very impressed by. For others who come by here with the same problem: it’s in the errors column in the oban_jobs table of the plausible database.

\n
\n

The final step was making the data public like it was on Simple Analytics, so my humble blog’s popularity or lack thereof can continue to be scrutinized by anyone.

\n

Wrapping up

\n

All things considered this took me approximately an hour to get done, most of which was spent digging through Plausible’s GitHub issues and re-learning how to use the psql CLI. I’m quite pleased with how it turned out in the end!

\n

Thanks to Tanvi and Abhinav for answering some questions regarding Plausible.

" }, { "id": "https://msfjarvis.dev/posts/overengineering-an-obsidian-dashboard-to-get-better-at-marvel-snap/", "url": "https://msfjarvis.dev/posts/overengineering-an-obsidian-dashboard-to-get-better-at-marvel-snap/", "title": "Overengineering an Obsidian dashboard to get better at Marvel Snap", "date_published": "2025-04-01T05:16:52.000Z", "summary": "Using data to answer the ultimate gamer question - why am I so bad?", "content_html": "

Marvel Snap at its core is a card battler set in the Marvel Universe, where you play a deck of 12 cards and are trying to win\nat least 2 of 3 randomly generated locations on the board in 6 turns.

\n

You start with 3 cards in your hand, and draw one each turn. Each card has an energy that decides what it costs to play, and a power that adds to the total\nto determine which player wins that location. You start at 1 energy and gain +1 energy every turn.

\n

The game has a competitive mode, and I am horrendous at it. Over the course of the 2 years or so I’ve played this game I’ve made multiple excuses as to why I am so unlucky on the ranked ladder but ultimately I just play the game wrong and then blame my opponents. Like a true gamer.

\n

Last month I migrated my personal journal and knowledge base to Obsidian. On a whim, I decided that it might be a fun project to build a system to track and visualize statistics about my Snap games to try and objectively answer the question - why am I so bad?

\n

Laying down the foundation

\n

To get started with analysis, I first need to figure out what data to collect and how.

\n

For a typical game, the stuff you want to record is the following:

\n\n

Taking all this into account I settled on the following template that I would use for each individual game I recorded.

\n
---
date: { { date } }
time: { { time } }
type: snap-game
result: win/loss/retreat
opponent_name:
opponent_deck:
my_deck:
locations:
  - location1
  - location2
  - location3
cubes: # Use positive for gains (e.g., +4), negative for losses (e.g., -2)
notes:
---
\n
## Game Details
\n
### Key Moments
\n
### Lessons Learned
\n

The YAML frontmatter is what I use to power my dashboards, and the Markdown underneath is unstructured data to record how the game actually progressed so I can also have a play-by-play if I fumbled a good play or managed to cook something clever.

\n

Obsidian provides a very serviceable GUI to edit frontmatter fields which makes the job of filling it in pretty easy.

\n
\"Obsidian'sExpand image
\n

I fed this page into the Obsidian Templates plugin which takes care of filling in the date and time automatically when I create a new page based on this template.

\n

Building the visualization

\n

For this purpose I leveraged the excellent Obsidian DataView plugin which provides an SQL-like query language that can introspect the Markdown documents in a standard Obsidian Vault and surface structured data from it.

\n

As an example, here’s the code for the dashboard that is in my daily note template to see the current day’s games.

\n
TABLE WITHOUT ID
  length(rows) as \"Total Games\",
  length(filter(rows, (x) => x.result = \"win\")) as Wins,
  length(filter(rows, (x) => x.result = \"loss\")) as Losses,
  length(filter(rows, (x) => x.result = \"retreat\")) as Retreats,
  sum(map(rows.cubes, (x) => number(x))) as \"Net Cubes\"
FROM \"snap-games\"
WHERE type = \"snap-game\"
  AND date = date(\"{{date}}\")
FLATTEN file.link as FileLink
GROUP BY null
\n

It is pretty self-explanatory, but essentially this walks through every page in my Vault that has its type field set to snap-game and today’s date, then collates the win/loss record from the other YAML frontmatter fields and makes a nice and simple table out of it.

\n
\"AExpand image
\n

This can also be extended to cover the past week

\n
\"AExpand image
\n
DataView code
TABLE
  date as Date,
  time as Time,
  result as Result,
  my_deck as \"My Deck\",
  opponent_deck as \"Opponent's Deck\",
  cubes as \"Cubes Δ\"
FROM \"snap-games\"
WHERE type = \"snap-game\"
  AND date >= date(today) - dur(7 days)
SORT date desc, time desc
\n
\n

I can see what decks I played and how they performed relative to each other

\n
\"AExpand image
\n
DataView code
TABLE
  length(rows) as Games,
  round(length(filter(rows, (x) => x.result = \"win\")) / length(rows) * 100, 1) + \"%\" as \"Win Rate\",
  sum(map(rows.cubes, (x) => number(x))) as \"Net Cubes\"
FROM \"snap-games\"
WHERE type = \"snap-game\"
  AND date >= date(today) - dur(7 days)
GROUP BY my_deck
SORT length(rows) desc
\n
\n

I can also see which locations favor me the most

\n
\"AExpand image
\n
DataView code
TABLE
  length(rows) as \"Times Encountered\",
  round(length(filter(rows, (x) => x.result = \"win\")) / length(rows) * 100, 1) + \"%\" as \"Win Rate\",
  sum(map(rows.cubes, (x) => number(x))) as \"Net Cubes\"
FROM \"snap-games\"
WHERE type = \"snap-game\"
  AND date >= date(today) - dur(7 days)
FLATTEN locations
GROUP BY locations
SORT sum(map(rows.cubes, (x) => number(x))) desc
\n

Did it help?

\n

I don’t know!

\n

Work getting hectic meant I had little time to actually play and the re-launch of the High Voltage mode took up most of that.

\n

I did play a tiny amount of games when I first built this out, and did pick up on some of my tendencies that make me lose games but ultimately it’s a very small sample size. The new season of Snap will start tomorrow so hopefully I’ll be able to get back into it and actually figure out if this whole shebang was worth it.

" }, { "id": "https://msfjarvis.dev/posts/the-obsidian-migration--one-week-later/", "url": "https://msfjarvis.dev/posts/the-obsidian-migration--one-week-later/", "title": "The Obsidian Migration - One Week Later", "date_published": "2025-03-17T05:38:37.000Z", "summary": "An experience report of using Obsidian for a week, coming from a year of Logseq", "content_html": "

In my last post I had a list of to-dos to fully finish up the migration, which I’ll reproduce below:

\n
\n\n
\n

Migrating dates and tags to Obsidian

\n

I haven’t found a solution for automated tagging yet, but inspired by Simon Willison’s post about how he uses LLMs for code I decided to have GitHub Copilot write out a Python script to perform the migration of Logseq-isms to the appropriate Obsidian-native patterns.

\n

The choice of Python was for multiple reasons: it’s fast enough for the job, I understand it well, and it can get much farther without needing third-party dependencies.

\n

There wasn’t admittedly a whole lot to this process, since all I was doing is reviewing and running some 100 odd lines of Python then hard resetting my repo after it did the wrong thing. My initial prompt was this:

\n
Write a Python script using only the standard library to process an Obsidian
vault and migrate Logseq patterns to Obsidian's systems in the markdown files.
Below is a list of changes to make:
\n
1. Convert any `[[Thing]]` style links to use Obsidian-native tags by adding a
`#` prefix, resulting in `#Thing`. If there are spaces in the link text,
convert them to underscores.
\n
2. If a link's text matches the date format `May 8th, 2024`, replace its text
by changing the date format to YYYY-MM-DD. In the given example, the output
should be `[[2024-05-08]]`.
\n

The first output from this was unsurprisingly buggy, the script converted [[May 13th, 2024]] to #2024-05-13 instead of [[2024-05-13]]. The second iteration did not resolve the bug either, but in the third one progress was made.

\n

This was where Copilot impressed me a bit. After running the third iteration and finding a bug in the results, I told the chatbot that the date Jun 8th, 2024 was not being converted and it was able to quickly clock that this was caused by my usage of May in the initial prompt which made it look only for full month names, while Jun is obviously abbreviated. I’m not sure if I would’ve figured that one out right away, but it was a good way to also learn more about the Python standard library’s date formatting facilities.

\n

The final script is available here, with individual revisions so you can step through the changes made by Copilot as I prompted it.

\n

Getting set up on mobile

\n

This was the bit I was more excited for, because one major advantage of using Obsidian on mobile that I didn’t realize earlier was that the same plugins Just Work™️. As far as I could tell, Logseq did not have support for plugins on mobile which mas a little annoying because how I interacted with the app changed across platforms even though there was no technical reason for it to. In Obsidian everything works out-of-the-box since plugins get vendored into your vault so you can actually modify and review the plugin code you’re running.

\n

My existing workflow on Android involved manually committing and pushing changes to my graph/vault via MGit which was too many clicks in the MGit app and was getting a little frustrating over time. I asked on the Fediverse if someone was aware of a newer Git client for Android and very quickly discovered PuppyGit.

\n

Once I had removed Logseq and imported my existing vault into Obsidian (I’ve opted to re-use my existing Git repository), I followed PuppyGit’s guide to setting up automatic syncing with Obsidian which worked flawlessly. It works by mapping a list of apps to a list of repositories that they interact with, and using an accessibility service monitors when apps are launched or closed, upon which it would pull remote changes to the repos or commit and push respectively.

\n

The Git plugin I was using for my desktop didn’t work on Android since it required a git binary, but they helpfully include an option to disable it only for the current device which let me delegate to PuppyGit on mobile with no issues.

\n
\n

A week into it I find myself much happier using Obsidian than I ever was with Logseq, and I’ve even started using it for more things beyond journaling such as taking meeting notes and organizing my tasks which I found too unintuitive to achieve with Logseq. Logseq excels at a daily journaling tool but once my needs started expanding it was pretty clear that Obsidian really is the all-rounder I needed.

" }, { "id": "https://msfjarvis.dev/posts/migrating-from-logseq-to-obsidian/", "url": "https://msfjarvis.dev/posts/migrating-from-logseq-to-obsidian/", "title": "Migrating from Logseq to Obsidian", "date_published": "2025-03-09T10:29:00.000Z", "summary": "Logseq's clunky apps and glacial development pace finally motivated me to migrate my journaling to Obsidian, here's how I did it.", "content_html": "

I have been using Logseq for just over a year to maintain a daily journal, but I’ve been eyeing Obsidian for the last couple months as my note-keeping needs have expanded past daily journaling.

\n

Logseq has had a long-standing problem with the version of Electron used by it being EOL, which finally prompted its removal from Nixpkgs on security grounds this week and accelerating my migration to Obsidian. The below is a short list of things I did to make Obsidian more comfortable as a daily journaling system.

\n

Moving over content

\n

I made the following changes to the files in my Logseq graph when moving to the new Obsidian Vault

\n\n

Plugins

\n

I enabled the following core plugins:

\n\n

Looking around other people’s experience with this migration path I also found the following community plugins:

\n\n
\n

Side note: I greatly appreciate the fact that Obsidian defaults to vendoring plugins in the Vault itself so if you’re tracking the .obsidian folder in Git, you get a fully reproducible Vault regardless of what machine you open it on.

\n
\n

Configuration changes

\n

In Settings → Daily Notes:

\n\n

In Settings → Hotkeys:

\n\n

Pending changes

\n

What I’ve done so far is just the result of 30 minutes of looking around, and there is still a bunch of currently broken stuff that needs resolving.

\n\n

Despite these issues, I think the move to Obsidian will end up being a net positive to my note taking process. Please let me know on Mastodon or the comments here if you have any tips to improve things.

" }, { "id": "https://msfjarvis.dev/posts/assorted-nixos-things/", "url": "https://msfjarvis.dev/posts/assorted-nixos-things/", "title": "Assorted NixOS things", "date_published": "2024-12-07T20:19:00.000Z", "summary": "Running log of random things I've learned about Nix/NixOS/Nixpkgs", "content_html": "

I run NixOS on my desktop and servers which results in coming across a bunch of generally random things, this is a running log of those discoveries which I’ll attempt to keep up-to-date with my personal Logseq graph where I’ve kept the record of them.

\n

nix3-flake-metadata

\n

The “new” (read: experimental) Nix CLI has a bunch of useful commands one of which is nix flake metadata that dumps a bunch of information about a flake, most notably, a tree-like representation of your inputs. It makes it significantly easier to find duplicate inputs and avoid the problem of “1000 instances of Nixpkgs”. nix-melt can also assist with this, though for this purpose I find the output from the Nix CLI to be easier to grok than navigating the nix-melt TUI.

\n

NixOS module system

\n

The module system is behind a lot of what makes NixOS and Nixpkgs so powerful for composing reproducible systems. The nix.dev reference includes a great tutorial that is a great read regardless of the complexity level of what you are trying to achieve.

\n

Updating your custom packages from a Git branch

\n
\n

This isn’t directly related to Nix itself but to nix-update which is a very popular tool for keeping Nix packages up-to-date’

\n
\n

You can pass the --version=branch argument to update the package from the default branch instead of the latest tag/release for an Arch-style -git package. Specify --version=branch=branch_name if you want to use a non-default branch instead.

\n

The Nix language

\n

It is possible to replace a call to builtins.map with nixpkgs.lib.catAttrs if the transformation is just pulling out a field. For example: builtins.map (item: item.field) list can instead just be catAttrs \"field\" list.

\n

Building packages from a REPL

\n

In a Nix REPL you can run :bl inputs.nixpkgs.legacyPackages.aarch64-linux.pkgs.attic-server to build the derivation and create a repl-result-out symlink in your current working directory. You can also run :bl nixosConfigurations.ryzenbox.pkgs.attic-client if you wish to build the package from your NixOS configuration’s instance of Nixpkgs.

\n

Avoiding mysterious errors with fixed-output derivations

\n

For many languages Nixpkgs’ builders vendor dependencies in a fixed-output derivation (FOD) which will give seemingly random errors if the contents change but its name doesn’t, since the old copy of the FOD will continue to be used in the later build stages. For example, in a package using buildGoModule, updating the version but not changing the vendorHash will cause a new FOD to be created but the build will continue with the FOD of the previous version.

\n

To avoid these problems when doing manual updates, make sure to change the relevant cargoHash/vendorHash to lib.fakeHash to cause the new FOD to be picked up which will also give you the correct hash to fill in the field.

\n

Tailscale exit nodes on NixOS

\n

Tailscale is a mesh VPN that includes a feature to run certain endpoints as exit nodes, allowing traffic from other devices in the VPN to be routed from that machine.

\n

On NixOS, the relevant firewall knobs for it are exposed under the services.tailscale.useRoutingFeatures option which is required to run an exit node on a machine as well as allow a machine to use a different exit node. Set this to server if you want to run the specific machine as an exit node, and to client if you want it to be able to use exit nodes in your tailnet.

\n

Dual booting Windows using systemd-boot

\n

NixOS includes options within its systemd-boot module that allow configuring boot entries for Windows without requiring manual intervention of copying over EFIs and what not. There are an initial few steps involved to get the relevant values, which are documented in the option reference.

\n

Nix’s builtins.toString

\n

The toString builtin in Nix handles Booleans rather unintuitively (even if it makes sense for the shell interpolation use-case) so here’s the docs for you to go read just in case something else is a surprise.

" }, { "id": "https://msfjarvis.dev/posts/deploying-applications-to-flyio-without-docker/", "url": "https://msfjarvis.dev/posts/deploying-applications-to-flyio-without-docker/", "title": "Deploying applications to Fly.io without Docker", "date_published": "2024-10-05T06:30:00.000Z", "summary": "A quick guide to deploying apps to Fly.io without a local Docker installation", "content_html": "
\n

This post assumes basic familiarity with Fly.io, the flyctl CLI and Nix.

\n
\n
\n

Fly.io is one of the many players in the “We run your app close to your users” space. I’ve used their services for just over a couple years to run my personal instance of linkleaner. Recently I ran into a combination of a platform outage and localized issue that necessitated recreating the app from scratch, and I decided to try doing so locally rather than my usual CI-based deployment system.

\n

Deployments with Fly.io usually start with a Docker/OCI image. Fly.io requires this image to either exist in their own OCI registry at registry.fly.io, or any other publicly accessible registry. The common way to do this is to use a Dockerfile and build the image locally, then use their flyctl CLI to upload that image from your local Docker registry to registry.fly.io and then deploy the app.

\n

Since I use Nix as my Docker image builder (check out this talk that explains why one would prefer it over Docker itself), I do not run Docker locally and had no plans to start now. This guide is mostly the result of that stubbornness so I intend to keep it short.

\n

Setting up

\n

The main ingredient of this is the skopeo CLI that allows one to interact with remote OCI registries. We’ll replace the Fly.io Docker integration steps with skopeo copy that allows copying Docker archives directly to a registry.

\n

You’re also going to need a Fly.io token to authenticate with registry.fly.io. With flyctl installed you can run fly tokens create deploy to get an auth token usable for the purpose. I stored mine in a FLY_AUTH_TOKEN environment variable that will be referenced later.

\n

Building and deploying

\n

First we need to build the image. With Nix it is as easy as running nix build .#container (since I declare the OCI image as a package in my Nix configuration) which will generate a symlink at ./result pointing to a Docker image tarball.

\n
Terminal window
➜ ls -l result
result ⇒ /nix/store/h0jgv00fg68d7gsaadd11ydpk6wxxn8w-docker-image-linkleaner.tar.gz
\n

Now to the actual deployment: assuming your app is already set up (fly apps create), you just need to copy this image over to their registry and trigger a new deployment. skopeo makes this rather easy.

\n
Terminal window
skopeo \\
  # Remove the need for a policy file since we only need this for a one off thing
  --insecure-policy \\
  copy \\
  # The source argument, which is a docker image archive at `./result`
  docker-archive:./result \\
  # The destination argument, the `build.image` value from your fly.toml
  docker://registry.fly.io/linkleaner:latest \\
  # Injecting the previously generated credentials
  --dest-creds x:\"$FLY_AUTH_TOKEN\" \\
  # Docker v2.2 manifest format: https://containers.gitbook.io/build-containers-the-hard-way#registry-format-oci-image-manifest
  # Probably not necessary but I just wanted to be sure
  --format v2s2
\n

This will print a bunch of stuff while it’s doing its thing and upon success you have now placed your organically built, Docker-free OCI image into the Fly.io registry.

\n
\n

And that’s more or less it! You can now proceed with the usual Fly.io deployment process of running flyctl deploy and it’ll be able to find the image and run your application:

\n
Terminal window
➜ flyctl deploy
==> Verifying app config
Validating /home/msfjarvis/code/linkleaner/fly.toml
✓ Configuration is valid
--> Verified app config
==> Building image
Searching for image 'registry.fly.io/linkleaner:latest' remotely...
image found: img_3mno4wk809624k18
\n
Watch your deployment at https://fly.io/apps/linkleaner/monitoring
\n
-------
Updating existing machines in 'linkleaner' with rolling strategy
\n
-------
 ✔ [1/2] Cleared lease for 0801937b607358
 ✔ [2/2] Cleared lease for 9080155dfe0de8
-------
" }, { "id": "https://msfjarvis.dev/posts/assorted-git-things/", "url": "https://msfjarvis.dev/posts/assorted-git-things/", "title": "Assorted Git things", "date_published": "2024-10-01T08:27:00.000Z", "summary": "A running log of Git concepts I've learned since I started journaling", "content_html": "

When I asked for suggestions about note-taking apps back in May I wasn’t fully convinced I’d be able to stick with it, but nearly 6 months later it has probably been the thing I’ve been most consistent about. Anyways, here’s a rather short list of Git things off the ‘TIL’ list in my Logseq graph. I’ll keep adding new things to the top as they come up.

\n

GIT_EDITOR variable

\n

While Git will usually pick the $EDITOR environment variable to present an edit UI for commit messages and rebase to-dos, it also supports its own $GIT_EDITOR variable that can be used to set a Git-specific editor. This functionality can be leveraged from integrated terminals inside IDEs to allow writing commit message from the IDE without having to use their entire Git UI.

\n

git add

\n\n

Git submodules

\n

Probably the biggest pain in the ass in all of Git land. Every time I’ve had to use a submodule, I’ve disliked the experience. Here’s how to properly get rid of a submodule in case you’re switching to git-subrepo or a different solution.

\n
git submodule deinit -f path/to/module
rm -rf .git/modules/path/to/module
git config -f .gitmodules --remove-section submodule.path/to/module
git add .gitmodules
git rm --cached path/to/module
\n

In order, this does the following:

\n
    \n
  1. deinit removes the checkout of the submodule and removes it from .git/config
    2. \n
  2. Deletes the bare repository Git creates of the submodule which is used to generate the actual checkout
    3. \n
  3. Edits your .gitmodules file to remove the submodule entry
    4. \n
  4. Stages the changes to .gitmodules
    5. \n
  5. Deletes the submodule fully from the index
    6. \n
\n

At this point you can commit the changes and hopefully never deal with a submodule again :D

\n

Git credential helpers

\n

While trying to configure pass-git-helper to authenticate with my personal Git forge I ended up learning a bit about how Git interacts with credential helpers

\n

Git credential helpers work via writing and reading from stdin and stdout which helped me debug an issue I was having with pass-git-helper. To get a credential helper to spit out the password, it needs to be called with get as the first argument and pass in some basic key=value data. The interaction looks somewhat like given below, with < indicating the stdin of the credential helper and > indicating its stdout.

\n
# for URL https://git.msfjarvis.dev/msfjarvis/super-private
$ pass-git-helper get
< protocol=https
< host=git.msfjarvis.dev
< path=msfjarvis/super-private
> protocol=https
> host=git.msfjarvis.dev
> username=msfjarvis
> password=hunter2
\n

Maintaining a fork in the presence of fetch.pruneTags

\n

Enabling fetch.pruneTags causes my Nixpkgs Git checkout to constantly delete and fetch tags, since my fork is missing most tags that upstream has so they keep getting pruned when updating from origin and get re-created when fetching upstream.

\n

One solution suggested for this was setting remotes.<name>.tagOpt = \"--no-tags\" but that didn’t do the job for me.

\n

The thing that worked was to conditionally disable pruneTags for just the origin remote so it would not try to clear out the tags pulled from upstream. Achieved by running git config --local remotes.origin.pruneTags false.

" }, { "id": "https://msfjarvis.dev/posts/a-tour-of-my-screenshots-folder/", "url": "https://msfjarvis.dev/posts/a-tour-of-my-screenshots-folder/", "title": "A tour of my screenshots folder", "date_published": "2024-04-01T06:30:00.000Z", "summary": "The history of my Minecraft adventures as told by my screenshots folder", "content_html": "

Preface

\n
\n

This is a post for the April Cools Club which encourages people to break away from the typical cringiness of April Fools and do things you don’t normally do.

\n
\n

To set the stage, every screenshot you will see going forward is gonna be Minecraft. I just love the game and I have had such fun with it for the past 5 years that it feels remiss to not share every so often (which I do these days at @msfjarvis@mstdn.games). These are gonna be in order from oldest to newest, and I’ll try to annotate each screenshot with dates, alt text and the relevant anecdote as I remember them but honestly a bunch of this is just goofy shit I happened to capture.

\n

Just a bridge, really

\n
\"AExpand image
Date taken: August 2, 2022
\n

This bridge is in a Minecraft world that Sasikanth and I started in the second half of 2022, and built entirely by him next to our starter base. After Sasi kinda moved on from playing on the server (as Minecraft players inevitably do, myself included) I copied the world and started using it as my singleplayer world and I still play on it to this day.

\n

An unlikely friendship

\n
\"AExpand image
Date taken: August 13, 2022
\n

I don’t think I really remember where this is from, but if I had to guess it was the village I and Sasikanth discovered and promptly lay ruin to which today happens to be my full time base.

\n

The start of the storage room

\n
\"AnExpand image
Date taken: August 20, 2022
\n

After commandeering the aforementioned village I decided to lay roots next door, and this is basically the start of my storage room. The basic design is still the same, but it has like, walls and stuff now.

\n

I am a dwarf, and I’m digging a hole

\n
\"AExpand image
Date taken: December 29, 2022
\n

Honestly not much to say, I was in a bit of a slump with my mental health and decided the best use of my mushy brain was to dig down a whole chunk to eventually build a slime farm.

\n

Did I drain this Ocean Monument or did it drain me?

\n
\"CinematicExpand image
Date taken: February 27, 2023
\n

I didn’t play much for a month or two, so I decided to pick up a somewhat involved project to get me back in the swing of things, which happened to be a Guardian farm for the prismarine family of blocks. I’ll let the other screenshots paint the picture, but suffice to say I had under-estimated the scope of this 😬

\n
\"OverheadExpand image
Date taken: March 7, 2023
\n
\"TheExpand image
Date taken: March 12, 2023
\n
\"TheExpand image
Date taken: March 15, 2023
\n
\"MeExpand image
Date taken: March 20, 2023
\n

When this was finally done I genuinely used it like 4 times total, turns out I’m not really a prismarine guy so that’s a couple of weeks I am not getting back.

\n

Getting real personal with a Warden

\n
\"AExpand image
Date taken: March 12, 2023
\n

To break up the monotony of placing sand for the Guardian farm I paid a visit to a nearby Ancient City and ended up a little too close to a Warden, which did eventually kill me.

\n

The sea shanty era

\n
\"TwoExpand image
Date taken: June 27, 2023
\n

I took another couple months off and then set up a small server with a handful of friends to mess around on a fresh world with relative newbies to the game. A lot of chaos ensued, but a large chunk of the time I spent on the server ended up being me and Yash just boating across oceans looking for anything mildly interesting while in a voice call with our friends. It was fun times, but as always people’s interest dwindled down and I shut the server down a couple of weeks later.

\n

Excavating the Nether

\n
\"AExpand image
Date taken: July 11, 2023
\n

I wanted Ancient Debris for a project and decided to just get a whole load of it at once, which resulted in this set of TNT-filled tunnels. Here’s the damage all the TNT did:

\n
\"TheExpand image
Date taken: July 11, 2023
\n

My first Sniffers

\n
\"AExpand image
Date taken: July 12, 2023
\n

When the Sniffers were added to the game I just had to get them, and obviously then I created a farm for the seeds they “sniff” up.

\n
\"ElevatedExpand image
Date taken: September 13, 2023
\n

The End Ring Project

\n
\"TopExpand image
Date taken: March 28, 2024
\n

I envisioned a continuous ring of Prismarine walkways around all the End gateways to be a cool way to get to them rather than just pillaring up, but I biffed the “circle” so many times that it’s kinda stalled at the moment.

\n

All Trimmed Up

\n
\"AExpand image
Date taken: March 16, 2024
\n

I went on a quest to obtain every single armor trim and enough Netherite to create armor to put them on, which in total probably took 20-odd hours including all the Diamond and Ancient Debris mining as well as actually locating all the trims. My Minecraft closet has more variety than my IRL one which feels like cause for concern.

\n

Bonus randomness

\n

The Minecraft x FaZe collab

\n

For some reason Minecraft likes to spawn Nether Fossils that look way too much like the [FaZe Clan] logo and I apparently have a bunch of them screenshotted, so here they are:

\n
\"TheExpand image
\n
\"AnotherExpand image
\n
\"YetExpand image
\n
\"WouldExpand image
\n
\"ToExpand image
\n

The impossible portal

\n
\"AExpand image
Date taken: June 12, 2023
\n

I manually prune unused chunks before Minecraft updates in order to let them regenerate with the updated terrain and accidentally sliced off this portal as it happened to be on a chunk boundary. It’s all the way back at my starter base so it does not see much use, but it’s there.

\n

The end…?

\n

I don’t know why I picked this to do of all the things I could have for April Cools. However, reliving all the memories of fun times I had with my friends and even by myself were worth the pain it was to go through 800 odd screenshots and find moderately interesting things :)

\n

Hopefully I have something cooler for next year and that I actually give myself more than 4 hours to write it up.

" }, { "id": "https://msfjarvis.dev/posts/improving-dependency-sync-speeds-for-your-gradle-project/", "url": "https://msfjarvis.dev/posts/improving-dependency-sync-speeds-for-your-gradle-project/", "title": "Improving dependency sync speeds for your Gradle project", "date_published": "2024-03-30T21:43:07.000Z", "summary": "Waiting for Gradle to download dependencies is so 2023", "content_html": "

Android developers are intimately familiar with the ritual of staring at your IDE for tens of minutes while Gradle imports a new project before they can start working on it. While not fully avoidable, there are many ways to improve the situation. For small to medium projects, the time spent on this import phase can be largely dominated by dependency downloads.

\n

Preface

\n

This post is going to assume some things about you and your project, but you should be fine even if these aren’t true for you.

\n\n

If you’re stuck on a lower version of Gradle, you will hit this bug with the code samples in the post. Replacing all calls to includeGroupAndSubgroups with includeGroupByRegex can let you work around it temporarily (Note the addition of the .* at the end):

\n
includeGroupAndSubgroups(\"com.example\")
includeGroupByRegex(\"com.example.*\")
\n

Obtaining a baseline

\n

To get an idea for how long it actually takes your project to fetch its dependencies and to establish a baseline to compare improvements again, we can leverage Android Studio’s relatively new Downloads Info view to see how many network requests are being made and how many of those are failing and contributing to our slower build. Gradle has a --refresh-dependencies flag which ignores its existing cache of downloaded dependencies and redownloads them from the remote repositories which will allow us to get consistent results, barring network and disk fluctuations.

\n

In Android Studio, create a new run configuration for Gradle’s in-built dependencies task that will resolve all configurations and give us a more representative number. The --refresh-dependencies flag will force a full re-download to ensure caches do not affect our benchmarks:

\n
\"\"Expand image
The Android Studio Run configuration window configured with the task ':android:dependencies --refresh-dependencies'
\n

When you run this task, you’ll see the Build tool window at the bottom get populated with logs of Gradle downloading dependencies and the Downloads info tab will start accumulating the statistics for it.

\n
\"\"Expand image
Download info window showing a list of network requests and their sources while the task is running
\n

How Gradle fetches your dependencies

\n

The Gradle documentation for dependency resolution explains in some depth how the version conflict resolution and caching systems work, but that’s pretty jargon-heavy and too much of a detour from what we’re really here to do so I’ll just Spark Notes™️ it and move on to more fun stuff.

\n\n

A quick attempt at optimisation

\n

In the previous section I started off by mentioning repositories, which define where Gradle will look for dependencies. The repositories setup for a typical Android project might look something like this:

\n
settings.gradle.kts
pluginManagement {
  repositories {
    mavenCentral()
    google()
  }
}
\n
dependencyResolutionManagement {
  repositories {
    maven(\"https://jitpack.io\") { name = \"JitPack\" }
    google()
    mavenCentral()
    maven(\"https://oss.sonatype.org/content/repositories/snapshots/\") {
      name = \"Sonatype snapshots
    }
  }
}
\n

This tells Gradle that you want plugin dependencies to be looked up from Maven Central and gMaven, and for all other dependencies to come from JitPack, gMaven, Maven Central, or the Maven Central snapshots repository; in that order. The first simple tweak you can make is to reorder these based on how many of your dependencies you expect to come from what repository.

\n

For example, in a typical Android build most of your plugin classpath will be dominated by the Android Gradle Plugin (AGP), so you’d want gMaven to be come first so that Gradle does not waste time trying to find AGP on Maven Central.

\n
pluginManagement {
  repositories {
    mavenCentral()
    google()
    mavenCentral()
  }
}
\n

For the dependencyResolutionManagement block, JitPack is very likely to only host one or two of the dependencies you require, while most of them will be coming from gMaven and Maven Central, so shifting JitPack to the very end will significantly reduce the number of failed requests.

\n
dependencyResolutionManagement {
  repositories {
    maven(\"https://jitpack.io\") { name = \"JitPack\" }
    google()
    mavenCentral()
    maven(\"https://jitpack.io\") { name = \"JitPack\" }
    maven(\"https://oss.sonatype.org/content/repositories/snapshots/\") {
      name = \"Sonatype snapshots
    }
  }
}
\n

Going for zero

\n

With the minor changes made above we have already significantly improved on our failed requests metric, but why stop at good when we can have perfect.

\n

Gradle’s repositories APIs also support the notion of specifying the expected “contents” of individual repositories, which tells Gradle what groups of dependencies are supposed to be available in what repositories. This allows it to prevent redundant network requests and significantly boosts sync performance.

\n

These filters can be of two types:

\n\n

The difference is subtle, but should become clearer shortly as we start hacking on our setup.

\n

For our plugins block, we want only gMaven to supply AGP and everything else can come from Maven Central. Here’s how to achieve that:

\n
settings.gradle.kts
pluginManagement {
  repositories {
    google()
    exclusiveContent { // First type of filter
      forRepository { google() } // Specify the repository this applies to
      filter { // Start specifying what dependencies are *only* found in this repo
        includeGroupAndSubgroups(\"androidx\")
        includeGroupAndSubgroups(\"com.android\")
        includeGroup(\"com.google.testing.platform\")
      }
    }
    mavenCentral()
  }
}
\n

For other dependencies that are governed by the dependencyResolutionManagement block, the setup is similar. To demonstrate the usage of the second kind of filter, we’re introducing an additional constraint: assume the build relies on the Jetpack Compose Compiler, and we go back and forth between stable and pre-release builds of it. The pre-release builds can only be obtained from androidx.dev, while the stable builds only exist on gMaven. If we tried to use exclusiveContent here, it would make Gradle only check one of the declared repositories for the artifact and fail if it doesn’t find it there. To allow this fallback, we instead use a content filter as follows.

\n
dependencyResolutionManagement {
  repositories {
    google()
    google {
      content {
        includeGroupAndSubgroups(\"androidx\")
        includeGroupAndSubgroups(\"com.android\")
        includeGroupAndSubgroups(\"com.google\")
      }
    }
    maven(\"https://androidx.dev/storage/compose-compiler/repository\") {
      name = \"Compose Compiler Snapshots\"
      content { includeGroup(\"androidx.compose.compiler\") }
    }
    mavenCentral()
    maven(\"https://jitpack.io\") { name = \"JitPack\" }
    maven(\"https://oss.sonatype.org/content/repositories/snapshots/\") {
      name = \"Sonatype snapshots
    }
  }
}
\n

This setup tells Gradle the specific artifacts present in these repositories but does not enforce any restrictions on which repository said artifacts can come from. Now, if I use a pre-release version of the Compose Compiler, Gradle will first try to look it up in gMaven and then fall back to the androidx.dev repository.

\n

In the above example we also see JitPack being mentioned, which we only wish to use for a specific dependency that’s unavailable elsewhere. The exclusiveContent filter is precisely for this use case:

\n
dependencyResolutionManagement {
  repositories {
    google {
      content {
        includeGroupAndSubgroups(\"androidx\")
        includeGroupAndSubgroups(\"com.android\")
        includeGroupAndSubgroups(\"com.google\")
      }
    }
    maven(\"https://androidx.dev/storage/compose-compiler/repository\") {
      name = \"Compose Compiler Snapshots\"
      content { includeGroup(\"androidx.compose.compiler\") }
    }
    mavenCentral()
    maven(\"https://jitpack.io\") { name = \"JitPack\" }
    exclusiveContent {
      forRepository { maven(\"https://jitpack.io\") { name = \"JitPack\" } }
      filter { includeGroup(\"com.github.requery\") }
    }
    maven(\"https://oss.sonatype.org/content/repositories/snapshots/\")
  }
}
\n

The Sonatype OSS snapshots repository is only intended to be used for snapshot releases of dependencies we’d otherwise source from Maven Central, so we can indicate to Gradle to only search for snapshots in there with a mavenContent directive:

\n
dependencyResolutionManagement {
  repositories {
    google {
      content {
        includeGroupAndSubgroups(\"androidx\")
        includeGroupAndSubgroups(\"com.android\")
        includeGroupAndSubgroups(\"com.google\")
      }
    }
    maven(\"https://androidx.dev/storage/compose-compiler/repository\") {
      name = \"Compose Compiler Snapshots\"
      content { includeGroup(\"androidx.compose.compiler\") }
    }
    mavenCentral()
    maven(\"https://jitpack.io\")
    exclusiveContent {
      forRepository { maven(\"https://jitpack.io\") { name = \"JitPack\" } }
      filter { includeGroup(\"com.github.requery\") }
    }
    maven(\"https://oss.sonatype.org/content/repositories/snapshots/\") {
      name = \"Sonatype Snapshots\"
     mavenContent {
       snapshotsOnly()
     }
    }
  }
}
\n

Bonus section: Kotlin Multiplatform

\n

If you’re working with Kotlin Multiplatform, these directions will sadly not cover all the dependencies being fetched during your build. There’s a YouTrack issue (KT-51379) that you can subscribe to for updates on this, but in the mean time here’s the missing bits:

\n
dependencyResolutionManagement {
  repositories {
    // workaround for https://youtrack.jetbrains.com/issue/KT-51379
    exclusiveContent {
      forRepository {
        ivy(\"https://download.jetbrains.com/kotlin/native/builds\") {
          name = \"Kotlin Native\"
          patternLayout {
            listOf(
                \"macos-x86_64\",
                \"macos-aarch64\",
                \"osx-x86_64\",
                \"osx-aarch64\",
                \"linux-x86_64\",
                \"windows-x86_64\",
              )
              .forEach { os ->
                listOf(\"dev\", \"releases\").forEach { stage ->
                  artifact(\"$stage/[revision]/$os/[artifact]-[revision].[ext]\")
                }
              }
          }
          metadataSources { artifact() }
        }
      }
      filter { includeModuleByRegex(\".*\", \".*kotlin-native-prebuilt.*\") }
    }
    exclusiveContent {
      forRepository {
        ivy(\"https://nodejs.org/dist/\") {
          name = \"Node Distributions at $url\"
          patternLayout { artifact(\"v[revision]/[artifact](-v[revision]-[classifier]).[ext]\") }
          metadataSources { artifact() }
          content { includeModule(\"org.nodejs\", \"node\") }
        }
      }
      filter { includeGroup(\"org.nodejs\") }
    }
    exclusiveContent {
      forRepository {
        ivy(\"https://github.com/yarnpkg/yarn/releases/download\") {
          name = \"Yarn Distributions at $url\"
          patternLayout { artifact(\"v[revision]/[artifact](-v[revision]).[ext]\") }
          metadataSources { artifact() }
          content { includeModule(\"com.yarnpkg\", \"yarn\") }
        }
      }
      filter { includeGroup(\"com.yarnpkg\") }
    }
  }
}
\n

If you’re not using a JavaScript target in your project, it should be safe to skip the NodeJS and Yarn repositories but it’s probably easier to keep it configured ahead of time in case you adopt JavaScript in the future.

\n

Conclusion

\n

The exact percentage improvement you can expect can vary depending on how many dependencies you have as well as how many repositories were previously declared and in what order, but you should most definitely see a noticeable difference consistently. These are the before and after numbers for a project I optimised for my day job.

\n

Before

\n
\"\"Expand image
The Android Studio Dependency Sync window, showing a total sync duration of 5 minutes and 56 seconds of which 1 minute and 30 seconds went into failed network requests
\n

After

\n
\"\"Expand image
The Android Studio Dependency Sync window, now showing the total sync taking only 3 minutes and 17 seconds with 0 failed requests
\n

Like and subscribe, and hit that notification bell so you don’t miss my next post some time within this decade (hopefully).

" }, { "id": "https://msfjarvis.dev/posts/tips-and-tricks-for-using-renovate/", "url": "https://msfjarvis.dev/posts/tips-and-tricks-for-using-renovate/", "title": "Tips and tricks for using Renovate", "date_published": "2023-01-17T19:32:18.000Z", "summary": "Renovate is an extremely powerful tool for keeping your dependencies up-to-date, and its flexibility is often left unexplored. I'm hoping to change that", "content_html": "

Mend Renovate is a free to use dependency update management service powered by the open-source renovate, and is a compelling alternative to GitHub’s blessed solution for this problem space: Dependabot. Renovate offers a significantly larger suite of supported language ecosystems compared to Dependabot as well as fine-grained control over where it finds dependencies, how it chooses updated versions, and a lot more. TL;DR: Renovate is a massive upgrade over Dependabot and you should evaluate it if any aspect of Dependabot has caused you grief, there’s a good chance Renovate does it better.

\n

I’m collecting some tips here about “fancy” things I’ve done using Renovate that may be helpful to other folks. You’ll be able to find more details about all of these in their very high quality docs at docs.renovatebot.com.

\n

Disabling updates for individual packages

\n

There are times where you’re sticking with an older version of a package (temporarily or otherwise) and you just don’t want to see PRs bumping it, wasting CI resources for an upgrade that will probably fail and is definitely not going to be merged. Renovate offers a convenient way to do this:

\n
{
  \"packageRules\": [
    {
      \"managers\": [\"gradle\"],
      \"packagePatterns\": [\"^com.squareup.okhttp3\"],
      \"enabled\": false
    }
  ]
}
\n

Grouping updates together

\n

Renovate already includes preset configurations for monorepos that publish multiple packages with identical versions, but you can also easily add more of your own. As an example, here’s how you can combine updates of the serde crate and its derive macro.

\n
{
  \"packageRules\": [
    {
      \"managers\": [\"cargo\"],
      \"matchPackagePatterns\": [\"serde\", \"serde_derive\"],
      \"groupName\": \"serde\"
    }
  ]
}
\n

Set a semver range for upgrades

\n

Sometimes there are cases where you may need to set an upper bound on a package dependency to avoid breaking changes or regressions. Renovate offers intuitive support for the same.

\n
{
  \"packageRules\": [
    {
      \"matchPackageNames\": [\"com.android.tools.build:gradle\"],
      \"allowedVersions\": \"<=7.4.0\"
    }
  ]
}
\n

Supporting non-standard dependency declarations

\n

Dependency versions are sometimes specified without their package names, for example in config files. These cannot be automatically detected by Renovate, but you can use a regular expression to teach it how to identify these dependencies.

\n

For example, you can specify the version of Hugo to build your Netlify site with in the netlify.toml file in your repository.

\n
[build.environment]
HUGO_VERSION = \"0.109.0\"
\n

This is how the relevant configuration might look like with Renovate

\n
{
  \"regexManagers\": [
    {
      \"description\": \"Update Hugo version in Netlify config\",
      \"fileMatch\": [\".toml$\"],
      \"matchStrings\": [\"HUGO_VERSION = \\\"(?<currentValue>.*?)\\\"\"],
      \"depNameTemplate\": \"gohugoio/hugo\",
      \"datasourceTemplate\": \"github-releases\"
    }
  ]
}
\n

You can read more about Regex Managers here.

\n

Making your GitHub Actions usage more secure

\n

According to GitHub’s official recommendations, you should be using exact commit SHAs instead of tags for third-party actions. However, this is a pain to do manually. Instead, allow Renovate to manage it for you!

\n
{
  \"extends\": [
    \"config:base\",
    \":dependencyDashboard\",
    \"helpers:pinGitHubActionDigests\"
  ]
}
\n

Automatically merging compatible updates

\n

Every person with a JavaScript project has definitely loved getting 20 PRs from Dependabot about arbitrary transitive dependencies that they didn’t even realise they had. With Renovate, that pain can also be automated away if you have a robust enough test suite to permit automatic merging of minor updates.

\n
{
  \"automergeType\": \"branch\",
  \"packageRules\": [
    {
      \"description\": \"Automerge non-major updates\",
      \"matchUpdateTypes\": [\"minor\", \"patch\", \"digest\", \"lockFileMaintenance\"],
      \"automerge\": true
    }
  ]
}
\n

With this configuration, Renovate will push compatible updates to renovate/$depName branches and merge it automatically to your main branch if CI runs on the branch and passes. To make that happen, you will also need to update your GitHub Actions workflows.

\n
name: Run tests
on:
  pull_request:
    branches:
      - main
  push:
    branches:
      - renovate/**
\n

Closing notes

\n

This list currently consists exclusively of things I’ve used in my own projects. There is way more you can achieve with Renovate, and I recommend going through the docs at docs.renovatebot.com to find any useful knobs for the language ecosystem you wish to use it with. If you come across something interesting not covered here, let me know either below or on Mastodon at @msfjarvis@fantastic.earth!

" }, { "id": "https://msfjarvis.dev/posts/writing-your-own-nix-flake-checks/", "url": "https://msfjarvis.dev/posts/writing-your-own-nix-flake-checks/", "title": "Writing your own Nix Flake checks", "date_published": "2022-12-18T06:30:00.000Z", "summary": "Quick how-to for writing ad-hoc checks for your own Nix Flakes", "content_html": "

Preface

\n

Ever since discovering nix(3) flake check from crane (wonderful tool btw, highly recommend it if you’re building Rust things), I’ve wanted to be able to quickly write my own flake checks. Unfortunately, as with everything Nix, dummy-friendly documentation was hard to come by so I started trying out a bunch of things until I ended up with something that worked, which I’ll share below.

\n

The premise

\n

I had been using a basic shell script with a nix-shell shebang for a while to run formatters on my scripts repo and while it worked, nix-shell startup is fairly slow and it just wasn’t cutting it for me. So I decided to try porting it to nix flake check which would benefit from evaluation caching and be faster while removing the overhead of nix-shell from the utility script.

\n

The thing you’re here for

\n

Like everything in Nix, the checks needed to be derivations that Nix will build and run the respective checkPhase of. So naively, I put together this to run the alejandra Nix formatter, shfmt to format shell scripts and shellcheck to lint them:

\n
outputs = {
  self,
  nixpkgs,
  flake-utils,
}:
  flake-utils.lib.eachDefaultSystem (system: let
    pkgs = import nixpkgs {inherit system;};
    files = pkgs.lib.concatStringsSep \" \" [
      # Individual shell scripts from the repository
    ];
    fmt-check = pkgs.stdenv.mkDerivation {
      name = \"fmt-check\";
      src = ./.;
      doCheck = true;
      nativeBuildInputs = with pkgs; [alejandra shellcheck shfmt];
      checkPhase = ''
        shfmt -d -s -i 2 -ci ${files}
        alejandra -c .
        shellcheck -x ${files}
      '';
    };
  in {
    checks = {inherit fmt-check;};
  });
\n

I needed a space separated list of my shell scripts to pass to shfmt and shellcheck, so I used a library function from nixpkgs called concatStringsSep that takes a list, and concatenates it together with the given separator. That’s the files binding declared in the snippet above.

\n

Here I ran into my first problem: Nix expects every derivation to generate an output which meant this doesn’t actually build.

\n
➜ nix flake check
error: flake attribute 'checks.fmt-check.outPath' is not a derivation
\n

There’s been some discussion about this but the TL;DR is that mkDerivation must produce an output. So I tried to cheat around this requirement by faking an output.

\n
diff --git flake.nix flake.nix
index b7fef3b99110..a531a30ad88e 100644
--- flake.nix
+++ flake.nix
@@ -18,6 +18,7 @@
       ];
       fmt-check = pkgs.stdenv.mkDerivation {
         name = \"fmt-check\";
+        dontBuild = true;
         src = ./.;
         doCheck = true;
         nativeBuildInputs = with pkgs; [alejandra shellcheck shfmt];
         checkPhase = ''
@@ -25,6 +26,11 @@
           alejandra -c .
           shellcheck -x ${files}
         '';
+        installPhase = ''
+          mkdir \"$out\"
+        '';
       };
     in {
       checks = {inherit fmt-check;};
\n

dontBuild does exactly what you’d think and makes Nix not execute the buildPhase of the derivation, and the mkdir $out in the installPhase generates the output directory Nix was looking for which is still valid even if completely empty.

\n

You can make this slightly faster by using a smaller stdenv that won’t pull in a compiler toolchain or be rebuilt when said toolchain is updated:

\n
diff --git flake.nix flake.nix
index 7ce7a2ba80f8..b69db13fbc6d 100644
--- flake.nix
+++ flake.nix
@@ -16,7 +16,7 @@
       files = pkgs.lib.concatStringsSep \" \" [
         # bunch of shell scripts since I didn't have an extension I could glob against
       ];
-      fmt-check = pkgs.stdenv.mkDerivation {
+      fmt-check = pkgs.stdenvNoCC.mkDerivation {
         name = \"fmt-check\";
         dontBuild = true;
         src = ./.;
\n

The final result

\n

This is what the flake looked like for me after all this

\n
{
  description = \"A very basic flake\";
\n
  inputs = {
    nixpkgs.url = \"github:nixos/nixpkgs/nixpkgs-unstable\";
    flake-utils.url = \"github:numtide/flake-utils/main\";
  };
\n
  outputs = {
    self,
    nixpkgs,
    flake-utils,
  }:
    flake-utils.lib.eachDefaultSystem (system: let
      pkgs = import nixpkgs {inherit system;};
      files = pkgs.lib.concatStringsSep \" \" [
        # Individual shell scripts from the repository
      ];
      fmt-check = pkgs.stdenvNoCC.mkDerivation {
        name = \"fmt-check\";
        dontBuild = true;
        src = ./.;
        doCheck = true;
        nativeBuildInputs = with pkgs; [alejandra shellcheck shfmt];
        checkPhase = ''
          shfmt -d -s -i 2 -ci ${files}
          alejandra -c .
          shellcheck -x ${files}
        '';
        installPhase = ''
          mkdir \"$out\"
        '';
      };
    in {
      checks = {inherit fmt-check;};
    });
}
\n

It’s probably not idiomatic Nix (for some definition of idiomatic) but the entire thing has been a trial and error anyway so :shrug:

\n

I’m very much a noob when it comes to Nix so any feedback is very welcome and appreciated!

\n

Alternative solution (June 2024 update)

\n

The above ‘hack’ can also be changed to use pkgs.runCommand which I recently learned about.

\n
{
  description = \"A very basic flake\";
\n
  inputs = {
    nixpkgs.url = \"github:nixos/nixpkgs/nixpkgs-unstable\";
    flake-utils.url = \"github:numtide/flake-utils/main\";
  };
\n
  outputs = {
    self,
    nixpkgs,
    flake-utils,
  }:
    flake-utils.lib.eachDefaultSystem (system: let
      pkgs = import nixpkgs {inherit system;};
      files = pkgs.lib.concatStringsSep \" \" [
        # Individual shell scripts from the repository
      ];
      # Variant of runCommand intended for commands
      # that run quickly and will be slowed down by
      # the network round-trip.
      fmt-check = pkgs.runCommandLocal \"fmt-check\" {
        src = ./.;
        nativeBuildInputs = with pkgs; [alejandra shellcheck shfmt];
      } ''
          shfmt -d -s -i 2 -ci ${files}
          alejandra -c .
          shellcheck -x ${files}
          mkdir \"$out\"
        '';
    in {
      checks = {inherit fmt-check;};
    });
}
" }, { "id": "https://msfjarvis.dev/posts/mastodon-on-your-own-domain-without-hosting-a-server-netlify-edition/", "url": "https://msfjarvis.dev/posts/mastodon-on-your-own-domain-without-hosting-a-server-netlify-edition/", "title": "Mastodon on your own domain without hosting a server, Netlify edition", "date_published": "2022-11-16T06:30:00.000Z", "summary": "A quick and easy way of creating a Fediverse identity on your own domain without an ActivityPub server", "content_html": "

Preface

\n

I recently came across a blog post from Maarten Balliauw that explained how they had managed to create an ActivityPub compatible identity for themselves, without hosting Mastodon or any other ActivityPub server.

\n

I recommend going to their blog and reading the whole thing, but here’s a TL;DR

\n\n

For all practical purposes, WebFinger is essentially a JSON document that is served at /.well-known/webfinger from a domain and is used to identify “actors” across the Fediverse.

\n

Maarten’s approach to implementing this was to simply place the JSON document at /.well-known/webfinger on their domain balliauw.be, which allowed @maarten@balliauw.be to become a WebFinger-compatible identity that can be searched for on Mastodon and will return their actual @maartenballiauw.be@mastodon.online profile.

\n

Maarten did however note that since they’re relying on static hosting, they’re unable to restrict what identities they can enforce as valid, and thus a search for @anything@balliauw.be will also return their mastodon.online identity.

\n

The implementation

\n

I wanted to also set up something like this, but without the limitation Maarten had run into. Since my website runs on Netlify, I decided to try out using an Edge Function to build this up.

\n

Similar to Maarten, I first obtained my current Fediverse identity from the Mastodon server I am on: fantastic.earth.

\n
➜ curl -s https://fantastic.earth/.well-known/webfinger?resource=acct:msfjarvis@fantastic.earth | jq .
{
  \"subject\": \"acct:msfjarvis@fantastic.earth\",
  \"aliases\": [
    \"https://fantastic.earth/@msfjarvis\",
    \"https://fantastic.earth/users/msfjarvis\"
  ],
  \"links\": [
    {
      \"rel\": \"http://webfinger.net/rel/profile-page\",
      \"type\": \"text/html\",
      \"href\": \"https://fantastic.earth/@msfjarvis\"
    },
    {
      \"rel\": \"self\",
      \"type\": \"application/activity+json\",
      \"href\": \"https://fantastic.earth/users/msfjarvis\"
    },
    {
      \"rel\": \"http://ostatus.org/schema/1.0/subscribe\",
      \"template\": \"https://fantastic.earth/authorize_interaction?uri={uri}\"
    }
  ]
}
\n

With this in hand, now we can get started on wiring this up into our website.

\n

First, create an Edge Function using the Netlify CLI. Here’s the options I chose.

\n
➜ yarn exec ntl functions:create --name webfinger
? Select the type of function you'd like to create: Edge function (Deno)
? Select the language of your function: TypeScript
? Pick a template: typescript-json
? Name your function: webfinger
◈ Creating function webfinger
◈ Created netlify/edge-functions/webfinger/webfinger.ts
? What route do you want your edge function to be invoked on?: /.well-known/webfinger
◈ Function 'webfinger' registered for route `/.well-known/webfinger`. To change, edit your `netlify.toml` file.
\n

Next, add the following code to the TypeScript file just created for you. I’ve added comments inline to explain what each part of the code does so you can customize it according to your needs.

\n
// Netlify Edge Functions run on Deno (https://deno.land), so imports use URLs rather than package names.
import { Status } from \"https://deno.land/std@0.136.0/http/http_status.ts\";
import type { Context } from \"https://edge.netlify.com\";
\n
export default async (request: Request, context: Context) => {
  // We obtain the value of the 'resource' query parameter so that we
  // can ensure a response is only sent for the identity we want.
  const url = new URL(request.url);
  const resourceParam = url.searchParams.get(\"resource\");
  if (resourceParam === null) {
    return context.json(
      {
        error: \"No 'resource' query parameter was provided\",
      },
      {
        status: Status.BadRequest,
      },
    );
    // I want to be searchable as `@harsh@msfjarvis.dev`, so I only
    // allow requests that set the resource query param to this value.
  } else if (resourceParam !== \"acct:harsh@msfjarvis.dev\") {
    return context.json(
      {
        error: \"An invalid identity was requested\",
      },
      {
        status: Status.BadRequest,
      },
    );
  } else {
    // Here's the JSON object we got earlier
    return context.json({
      subject: \"acct:msfjarvis@fantastic.earth\",
      aliases: [
        \"https://fantastic.earth/@msfjarvis\",
        \"https://fantastic.earth/users/msfjarvis\",
      ],
      links: [
        {
          rel: \"http://webfinger.net/rel/profile-page\",
          type: \"text/html\",
          href: \"https://fantastic.earth/@msfjarvis\",
        },
        {
          rel: \"self\",
          type: \"application/activity+json\",
          href: \"https://fantastic.earth/users/msfjarvis\",
        },
        {
          rel: \"http://ostatus.org/schema/1.0/subscribe\",
          template: \"https://fantastic.earth/authorize_interaction?uri={uri}\",
        },
      ],
    });
  }
};
\n

And that’s it! You can test it out as below to verify things work as expected.

\n
➜ curl -s https://msfjarvis.dev/.well-known/webfinger | jq .
{
  \"error\": \"No 'resource' query parameter was provided\"
}
\n
➜ curl -s https://msfjarvis.dev/.well-known/webfinger?resource=acct:anything@msfjarvis.dev | jq .
{
  \"error\": \"An invalid identity was requested\"
}
\n
➜ curl -s https://msfjarvis.dev/.well-known/webfinger?resource=acct:harsh@msfjarvis.dev | jq .
{
  \"subject\": \"acct:msfjarvis@fantastic.earth\",
  \"aliases\": [
    \"https://fantastic.earth/@msfjarvis\",
    \"https://fantastic.earth/users/msfjarvis\"
  ],
  \"links\": [
    {
      \"rel\": \"http://webfinger.net/rel/profile-page\",
      \"type\": \"text/html\",
      \"href\": \"https://fantastic.earth/@msfjarvis\"
    },
    {
      \"rel\": \"self\",
      \"type\": \"application/activity+json\",
      \"href\": \"https://fantastic.earth/users/msfjarvis\"
    },
    {
      \"rel\": \"http://ostatus.org/schema/1.0/subscribe\",
      \"template\": \"https://fantastic.earth/authorize_interaction?uri={uri}\"
    }
  ]
}
\n

Thanks again to Maarten for doing the initial research for this and writing about it!

" }, { "id": "https://msfjarvis.dev/posts/writing-paparazzi-tests-for-your-kotlin-multiplatform-projects/", "url": "https://msfjarvis.dev/posts/writing-paparazzi-tests-for-your-kotlin-multiplatform-projects/", "title": "Writing Paparazzi tests for your Kotlin Multiplatform projects", "date_published": "2022-06-26T06:30:00.000Z", "summary": "Paparazzi enables a radically faster and improved UI testing workflow, and using a small workaround we can bring that to our multiplatform Compose projects", "content_html": "

Introduction

\n

Paparazzi is a Gradle plugin and library that enables writing UI tests for Android screens that run entirely on the JVM, without needing a physical device or emulator. This is massive, since it significantly increases the speed of UI tests as well as allows them to run on any CI system, not just ones using macOS or Linux with KVM enabled.

\n

Unfortunately, Paparazzi does not directly work with Kotlin Multiplatform projects so you cannot apply it to a KMP + Android module and start putting your tests in the androidTest source set (not to be confused with androidAndroidTest. Yes, I know). Why would you want to do this in the first place? Like everything cool and new in Android land, Compose! Specifically, compose-jb, JetBrains’ redistribution of Jetpack Compose optimised for Kotlin Multiplatform.

\n

I’ve sent a PR to Paparazzi that will resolve this issue, and in the mean time we can workaround this limitation.

\n

Setting things up

\n

To begin, we’ll need a new Gradle module for our Paparazzi tests. Since Paparazzi doesn’t understand Kotlin Multiplatform yet, we’re gonna hide that aspect of our project and present it a pure Android library project. Set up the module like so:

\n
paparazzi-tests/build.gradle.kts
plugins {
  id(\"com.android.library\")
  id(\"app.cash.paparazzi\")
}
\n
android {
  buildFeatures { compose = true }
}
\n

Now, add dependencies in this module to the modules that contain the composables you’d like to test. As you might have guessed, this approach currently limits you to only being able to test public composables. However, if you’re trying to test the UI exposed by a “common” module like I am, that might not be such a big deal.

\n
paparazzi-tests/build.gradle.kts
dependencies {
  testImplementation(projects.common)
}
\n

And that’s pretty much it! You can now be off to the races and start writing your tests:

\n
paparazzi-tests/src/test/kotlin/UserProfileTest.kt
class UserProfileTest {
  @get:Rule val paparazzi = Paparazzi()
\n
  @Test
  fun light_mode() {
    paparazzi.snapshot {
      MaterialTheme(colorScheme = LightThemeColors) { UserProfile() }
    }
  }
\n
  @Test
  fun dark_mode() {
    paparazzi.snapshot {
      MaterialTheme(colorScheme = DarkThemeColors) { UserProfile() }
    }
  }
}
\n

Consult the Paparazzi documentation for the Gradle tasks reference and customization options.

\n

Recipes

\n

Disable release build type for test module

\n

If you use ./gradlew check in your CI, our new module will be tested in both release and debug build types. This is fairly redundant, so you can disable the release build type altogether:

\n
paparazzi-tests/build.gradle.kts
androidComponents {
  beforeVariants { variant ->
    variant.enable = variant.buildType == \"debug\"
  }
}
\n

Running with JDK 12+

\n

You will run into this issue if you use JDK 12 or above to run Paparazzi-backed tests. I’ve started working on a fix for it upstream, in the mean time it can be worked around by forcing the test tasks to run with JDK 11.

\n
paparazzi-tests/build.gradle.kts
tasks.withType<Test>().configureEach {
  javaLauncher.set(javaToolchains.launcherFor {
    languageVersion.set(JavaLanguageVersion.of(11))
  })
}
\n

Testing with multiple themes easily

\n

Using an enum and Google’s TestParameterInjector you can write a single test and have it run against all your themes.

\n
paparazzi-tests/src/test/kotlin/Theme.kt
import androidx.compose.material3.ColorScheme
\n
enum class Theme(val colors: ColorScheme) {
  Light(LightThemeColors),
  Dark(DarkThemeColors),
}
\n
paparazzi-tests/src/test/kotlin/UserProfileTest.kt
@RunWith(TestParameterInjector::class)
class UserProfileTest {
  @get:Rule val paparazzi = Paparazzi()
\n
  @Test
  fun verify(@TestParameter theme: Theme) {
    paparazzi.snapshot(name = theme.name) {
      MaterialTheme(colorScheme = theme.colors) { UserProfile() }
    }
  }
}
" }, { "id": "https://msfjarvis.dev/posts/converting-gradle-convention-plugins-to-binary-plugins/", "url": "https://msfjarvis.dev/posts/converting-gradle-convention-plugins-to-binary-plugins/", "title": "Converting Gradle convention plugins to binary plugins", "date_published": "2022-04-17T06:30:00.000Z", "summary": "Gradle's convention plugins are a fantastic way to share common build configuration, why not take them a step further?", "content_html": "

Introduction

\n

Gradle’s convention plugins are a powerful feature that allow creating simple, reusable Gradle plugins that can be used across your multi-module projects to ensure all modules of a certain type are configured the same way. As an example, if you want to enforce that none of your Android library projects contain a BuildConfig class then the convention plugin for it could look something like this:

\n
\n

com.example.android-library.gradle.kts

\n
plugins {
  id(\"com.android.library\")
}
\n
android {
  buildFeatures {
    buildConfig = false
  }
}
\n
\n

Then in your modules, you can use this plugin like so:

\n
\n

library-module/build.gradle.kts

\n
plugins {
  id (\"com.example.android-library\")
}
\n
\n

Setting up convention plugins in your project

\n

Gradle’s official sample linked above mentions buildSrc as the location for your convention plugins. I’m inclined to disagree, buildSrc has historically had issues with IDE support and it’s special status within Gradle’s project handling means any change within buildSrc invalidates caches for your entire project resulting in incredible amounts of time lost during incremental builds.

\n

The solution to all of these problems is composite builds, and Josef Raska has a fantastic article that thoroughly explains the shortcomings of buildSrc and how composite builds solve them.

\n

A full explainer on the topic is slightly out of scope for this post, but I can wholeheartedly endorse Jendrik Johannes’ idiomatic-gradle repository as an example of setting up the Gradle build of a real-world project while leveraging features introduced in recent versions of Gradle. I highly recommend also checking out their ‘Understanding Gradle’ video series.

\n

Why would you want to make binary plugins out of convention plugins

\n

First, let’s answer this: what is a binary plugin?

\n

A Gradle plugin that is resolved as a dependency rather than compiled from source is a binary plugin. Binary plugins are cool because the next best thing after a cached compilation task is one that doesn’t exist in the first place.

\n

For most use cases, convention plugins will need to be updated very infrequently. This means that having each developer execute the plugin build as part of their development process is needlessly wasteful, and we can instead just distribute them as maven dependencies.

\n

This also makes it significantly easier to share convention plugins between projects without resorting to historically painful solutions like Git submodules or just straight up copy-pasting.

\n

Publishing your convention plugins

\n

To their credit, Gradle supports this ability very well and you can actually publish all plugins within a build/project with minimal configuration. The changes required to publish Android Password Store’s convention plugins for Android are:

\n
\n

build-logic/android-plugins/build.gradle.kts

\n
plugins { `kotlin-dsl` }
plugins {
  `kotlin-dsl`
  id(\"maven-publish\")
}
\n
group = \"com.github.android-password-store\"
\n
version = \"1.0.0\"
\n
\n

After that you can run gradle -p build-logic publishToMavenLocal and it will Just Work:tm:. You can configure additional publishing repositories in similar fashion to how you’d do it for a library project.

\n

If like me you need to publish these to Maven Central, you’ll need slightly more setup since it enforces multiple security and publishing related best practices. Here’s how I use gradle-maven-publish-plugin to configure the same (gradle.properties changes omitted for brevity, the GitHub repository explains what you need):

\n
\n

build-logic/settings.gradle.kts

\n
pluginManagement {
  repositories {
    mavenCentral()
    gradlePluginPortal()
  }
  plugins {
    id(\"com.vanniktech.maven.publish.base\") version \"0.19.0\"
  }
}
\n
\n
\n
\n

build-logic/android-plugins/build.gradle.kts

\n
import com.vanniktech.maven.publish.JavadocJar
import com.vanniktech.maven.publish.JavaLibrary
import com.vanniktech.maven.publish.MavenPublishBaseExtension
import com.vanniktech.maven.publish.SonatypeHost
import org.gradle.kotlin.dsl.provideDelegate
\n
plugins { `kotlin-dsl` }
plugins {
  `kotlin-dsl`
  id(\"com.vanniktech.maven.publish.base\")
  id(\"signing\")
}
\n
configure<MavenPublishBaseExtension> {
  group = requireNotNull(project.findProperty(\"GROUP\"))
  version = requireNotNull(project.findProperty(\"VERSION_NAME\"))
  publishToMavenCentral(SonatypeHost.DEFAULT)
  signAllPublications()
  configure(JavaLibrary(JavadocJar.Empty()))
  pomFromGradleProperties()
}
\n
 afterEvaluate {
  signing {
    val signingKey: String? by project
    val signingPassword: String? by project
    useInMemoryPgpKeys(signingKey, signingPassword)
  }
 }
\n
\n

This will populate your POM files with the properties required by Maven Central and sign all artifacts with PGP.

\n

Consuming your new binary plugins

\n

With your convention plugins converted to shiny new binary plugins, you might be inclined to start using them like so:

\n
\n

autofill-parser/build.gradle.kts

\n
plugins {
  id(\"com.github.android-password-store.published-android-library\")
  id(\"com.github.android-password-store.kotlin-android\")
  id(\"com.github.android-password-store.kotlin-library\")
  id(\"com.github.android-password-store.psl-plugin\")
  id(\"com.github.android-password-store.published-android-library\") version \"1.0.0\"
  id(\"com.github.android-password-store.kotlin-android\") version \"1.0.0\"
  id(\"com.github.android-password-store.kotlin-library\") version \"1.0.0\"
  id(\"com.github.android-password-store.psl-plugin\") version \"1.0.0\"
}
\n
\n

However, this fails because kotlin-android and kotlin-library plugins resolve to the same binary JAR that encompasses all plugins from the build-logic/kotlin-plugins module and results in a classpath conflict. To better understand how this resolution works, check out the docs on plugin markers.

\n

The way to resolve this problem is to define the plugin versions in your settings.gradle.kts file, where these classpath conflicts will be resolved automatically by Gradle:

\n
\n

settings.gradle.kts

\n
@@ -14,6 +14,25 @@ pluginManagement {
     mavenCentral()
     gradlePluginPortal()
   }
+  plugins {
+    id(\"com.github.android-password-store.kotlin-android\") version \"1.0.0\"
+    id(\"com.github.android-password-store.kotlin-library\") version \"1.0.0\"
+    id(\"com.github.android-password-store.psl-plugin\") version \"1.0.0\"
+    id(\"com.github.android-password-store.published-android-library\") version \"1.0.0\"
+  }
 }
\n
\n

And you’re off to the races!

\n

Closing notes

\n

This post was motivated by my goal of sharing a common set of Gradle configurations across my projects such as Android Password Store and Claw, which maintain a nearly identical set of convention plugins shared between the projects that I manually copy-paste back and forth. I’ve extracted the build-logic subproject of APS to a separate aps-build-logic repository, set it up for standalone development and configured publishing support. My goal is to supplement this with a continuous deployment workflow where an automatic version bump + release happens after each commit to the main, after which I can migrate my projects to it.

" }, { "id": "https://msfjarvis.dev/posts/backing-up-your-content-from-google-photos/", "url": "https://msfjarvis.dev/posts/backing-up-your-content-from-google-photos/", "title": "Backing up your content from Google Photos", "date_published": "2022-04-04T06:30:00.000Z", "summary": "Putting your media into Google Photos is easy, taking it out, not as much.", "content_html": "

Google Photos has established itself as one of the most popular photo storage services, and like a typical Google service, it makes it impressively difficult to get your data back out of it :D

\n

There are many good reasons why you’d want to archive your pictures outside of Google Photos, having an extra backup never hurts, maybe you want an offline copy for reasons, or you just want to get your stuff out so you can switch away from Google Photos entirely.

\n

How to archive your images from Google Photos

\n
    \n
  1. \n

    You can use Takeout, except it always strips the EXIF metadata of your images, and often generates incomplete archives. Losing EXIF metadata is a deal-breaker, because you can no longer organize images automatically based on properties like date, location, and camera type.

    \n
    2. \n
  2. \n

    You can download directly from photos.google.com which preserves metadata, but is embarassingly manual and basically impossible to use if you’re trying archive a few years of history.

    \n
    3. \n
\n

So, what’s the solution?

\n

gphotos-cdp

\n

gphotos-cdp is a tool that uses the nearly-perfect method number 2 and makes it automated. It does so by using the Chrome DevTools Protocol to drive an instance of the Google Chrome browser, and emulates all the manual actions you’d take as a human to ensure you get copies of your pictures with all the EXIF metadata retained.

\n

Setting up gphotos-cdp

\n
\n

Disclaimer: I’ve only tested this on Linux. This should be doable on other platforms, but it’s not relevant to my needs so I will not be investigating that.

\n
\n

Ideally you’d want to run this tool on a schedule on a NAS or a server to keep archiving images automatically as they get added to your Google Photos. I personally run this inside a hosted VM on a daily schedule.

\n

For gphotos-cdp to run in a non-interactive manner, it requires your browser data directory with your Google login cookies. You can easily create this with the following command:

\n
Terminal window
google-chrome \\
  --user-data-dir=gphotos-cdp \\
  --no-first-run  \\
  --password-store=basic \\
  --use-mock-keychain \\
  https://photos.google.com/
\n

This will launch Google Chrome with a brand new profile. Login to Photos, and then close the browser. Optionally, re-run the command to ensure that you do not need to login again.

\n
\n

The flags passed to google-chrome are extracted from the default set of parameters used by gphotos-cdp. I wish I could explain why each flag is necessary, but all I know is that it does the trick. I got them from this GitHub comment on the issue tracker for gphotos-cdp.

\n
\n

Once done, you’ll have a gphotos-cdp directory that you’ll need to move to the /tmp directory of whichever machine you wish to run gphotos-cdp on.

\n

gphotos-cdp is written in Golang so you’ll need to install it first. Once done, run the following command to install the latest version of gphotos-cdp

\n
Terminal window
go install github.com/perkeep/gphotos-cdp@latest
\n

Then you can go ahead and start using gphotos-cdp, as given below

\n
Terminal window
~/go/bin/gphotos-cdp \\ # go install puts things in ~/go/bin by default
  -v \\ # Enable verbose logging
  -dev \\ # Enable dev mode which always uses /tmp/gphotos-cdp as the profile directory
  -headless \\ # Run Chrome in headless mode so it works on servers and such
  -dldir ~/photos # Download everything to ~/photos
\n

The automation techniques used are not completely reliable and can often fail. You’ll want to implement some kind of retry-on-failure logic to ensure this is run a few times every day.

\n

Monitoring

\n

With anything built on such a brittle foundation, it’s useful to be able to constantly monitor that things are working as they should.

\n

Using healthchecks.io you can easily set up alerts that notify you of failures running the tool or unintentional gaps in the schedule you run the gphotos-cdp on. I use my healthchecks-monitor CLI in a cron job to run gphotos-cdp every day, and healthchecks.io notifies me via Telegram when it fails. The script running in cron looks like this

\n
#!/usr/bin/env bash
\n
HEALTHCHECKS_CHECK_ID=<UUID for check as given on healthchecks> \\
HEALTHCHECKS_USERAGENT=crontab \\
~/bin/healthchecks-monitor --retries 3 \\ # Try running the command thrice before giving up
  --timer \\ # Start off a server-side timer on healthchecks
  --logs \\ # Record execution logs on healthchecks in case of failure, to help with debugging
  --exec \"~/go/bin/gphotos-cdp -v -dev -headless -dldir ~/photos\"
\n

Conclusion

\n

As evident, it’s not an easy task to automatically archive your pictures from Google Photos. The setup is tedious and prone to breakage when any authentication related change happens, such as you accidentally logging out the “device” being used by gphotos-cdp or changing your password, in which case you will need to create the gphotos-cdp directory with Chrome again.

\n

Also, the technique in this post could easily stop working at any time if Google chooses to break it. That being said, gphotos-cdp was last updated in 2020 and still continues to function as-is so there is some degree of hope that it can be used for quite a bit more.

\n

Hopefully this setup causes you minimal grief and allows you to back up your precious memories without relying only on Google :)

" }, { "id": "https://msfjarvis.dev/posts/migrating-aps-to-material-you/", "url": "https://msfjarvis.dev/posts/migrating-aps-to-material-you/", "title": "Migrating APS to Material You", "date_published": "2021-11-06T06:30:00.000Z", "summary": "I recently migrated Password Store to Material You, Google's latest iteration of Material Design. Here's how it went.", "content_html": "

With much fanfare, Google released the next iteration of Material Design: Material You. It’s received mixed reviews, but I found it extremely pleasant to use and the homogeneity of Google apps following the platform colors felt great. That’s what prompted me to update APS to Material You and join in :)

\n

As expected, the library ecosystem (specifically the material-components-android library) took a while to support Material You but with the 1.5.0-alpha05 release of the MDC Android library, things are finally at a place where migration to Material You (henceforth referred to as M3) is viable for simpler apps like APS.

\n

APS has had some design work done to it but for the most part remains the culmination of ad-hoc choices (often bad) over a period of several years. With this migration I sought to change that and make things a bit more cohesive as well as give the app some much needed oomph.

\n

Getting the basics in

\n

I began with scouring through the resources in the conveniently isolated M3 website, where the Material Design team has helpfully created a lot of great tools and content to help developers and designers through the process. There’s the “Migration to Material 3” blog post, and the Material Theme Builder to generate palettes/styles/themes for apps, for both Jetpack Compose and Android’s View-based system. These were extremely helpful in getting a headstart on the whole process. It’s all documented in the commit history of the migration PR but I figure some additional context can’t hurt.

\n

Once I had the themes in, I decided to take the opportunity to also introduce a custom typeface. The app has been using Roboto since forever and it felt like it was time to spice things up. I decided to go with Manrope since it is a font I’ve previously used and found to be excellent for visual appeal and accessibility. I’m still not a 100% confident in my choice, so if people have better options in mind I’d love to know down in the comments.

\n

Once the new font face was in, I opted to enable dynamic colors. Admittedly not the right choice, since I should’ve validated the “default” palette first but that’s what I did ¯\\(ツ)/¯.

\n

Bugfixes and improvements™️

\n

Once the M3 themes were all prepped, it was time to actually start migrating.

\n

I switched our activities to use the M3 themes, and immediately started noticing bugs from non-idiomatic and straight up incorrect theming we’ve been lugging around for the past couple years.

\n

First step was to update our iconography, which was using inconsistent tints throughout. I updated all of them to use ?attr/colorControlNormal which made them blend in correctly with the rest of the updated UI.

\n

Earlier this year we had migrated a selection UI in one of our screens to use Chips, but we never got the theming right so it always looked kind of wrong. With M3, we were able to revert back to MaterialButtonToggleGroup without regressing on the accessibility issue which made us do it in the first place.

\n

There were a lot more smaller changes that were made to address the remaining visual bugs

\n\n

The final stretch

\n

With the visual fixes out of the way, I went in and cleaned up the themes and styles. I commonized shared attributes such as fonts and widget styles, created M3 variants of other special-purpose themes we had, and got rid of all the now unused M2 theming. Overall, the PR touched 60+ separate files and generated a final diff of +603,-314 lines. The PR can be seen here.

\n

We use a third-party library by Max Rumpf called ModernAndroidPreferences for our settings UI, and it hard-coded the use of AppCompat dialogs. Max was extremely helpful and made that customisable for us over the weekend which allowed us to use the appropriate Material You dialogs consistently. Huge thanks to Max, and check out his library! <3

\n

Screenshots!

\n

Before

\n
\"ScreenshotExpand image
\n

After

\n
\"ScreenshotExpand image
\n

Closing notes

\n

APS is a very low-effort app when it comes to UI work. We do not have a custom design system, everything follows Material to a T, and we try to stay in that lane. Our migration took me around 9 hours of work over two days, most of which was really spent on menial work such as manually checking all layouts for hard-coded styles and replacing them with attributes. This isn’t representative of what this process would look like for any project which rolls its own design system on top of Material, since they have a lot more to do before they can even begin the migration of their screens.

\n

I’d like to thank the Material Design team once more for the fabulous work they have done both in creating Material You as well as the technical documentation around it. Material Theme Builder was an extremely crucial tool for me that set the tone of the whole process, and I would have certainly repeated the same mistakes I did with Material 2 if it wasn’t for the tooling and guidance from the team.

" }, { "id": "https://msfjarvis.dev/posts/building-static-rust-binaries-for-linux/", "url": "https://msfjarvis.dev/posts/building-static-rust-binaries-for-linux/", "title": "Building static Rust binaries for Linux", "date_published": "2021-10-17T06:30:00.000Z", "summary": "Some tips on building static binaries of Rust projects targeting Linux", "content_html": "

Rust has supported producing statically linked binaries since RFC #1721 which proposed the target-feature=+crt-static flag to statically link the platform C library into the final binary. This was initially only supported for Windows MSVC and the MUSL C library. While MUSL works for most people, it\nhas many problems by virtue of being a work-in-progress such as unpredictable performance and many unimplemented features which programs tend to assume are present due to glibc being ubiquitous. In lieu of these concerns, support was added to Rust in 2019 to be able to statically link against glibc.

\n

Unfortunately, if you try to directly use it with RUSTFLAGS='-C target-feature=+crt-static' cargo build there is a good chance you’ll run into an error similar to this:

\n
cannot produce proc-macro for `async-trait v0.1.51` as the target `x86_64-unknown-linux-gnu` does not support these crate types
\n

This is a bit of a head scratcher, because the target (your host machine) definitely supports proc-macro crates. Turns out, even Rust contributors were confused by this. The “fix” for this is apparently to pass in the --target explicitly. The reason behind this seems to be a bug with cargo, where the RUSTFLAGS are applied to the target platform only when --target is explicitly provided. Without it, RUSTFLAGS values are set for the host only which results in the errors we see. More details are available Rust issue #78210

\n

Therefore, the correct way to build a statically linked glibc executable for an x86_64 machine is this:

\n
Terminal window
RUSTFLAGS='-C target-feature=+crt-static' cargo build --release --target x86_64-unknown-linux-gnu
\n

Other potential problems

\n

You may be unable to statically link your binary even after all this, due to dependencies that mandate dynamic linking. In some cases this is avoidable, such as using rustls in place of OpenSSL for cryptography, and reqwest or ureq in place of bindings to cURL for HTTP, not so much in others. Thanks to the convention of native-linking crates using the -sys suffix in their name it is fairly simple to find if your build has dependencies that dynamically link to libraries. Using cargo’s native tree subcommand and greping (or ripgreping for me), you can locate native dependencies. Running cargo tree | rg -- -sys against androidx-release-watcher’s v4.1.0 release gives us this:

\n
Terminal window
$ cargo tree | rg -- -sys
│   │   │   │   ├── curl-sys v0.4.45+curl-7.78.0
│   │   │   │   │   ├── libnghttp2-sys v0.1.6+1.43.0
│   │   │   │   │   ├── libz-sys v1.1.3
│   │   │   │   │   └── openssl-sys v0.9.66
│   │   │   │   ├── openssl-sys v0.9.66 (*)
│   │   │   ├── curl-sys v0.4.45+curl-7.78.0 (*)
│   └── web-sys v0.3.53
│       ├── js-sys v0.3.53
\n

This indicates curl, zlib, openssl, and libnghttp2 as well as a bunch of WASM-related things are being dynamically linked into my executable. To resolve this, I looked at the build features exposed by surf and found that it selects the \"curl_client\" feature by default, which can be turned off and replaced with \"h1-client-rustls\" which uses an HTTP client backed by rustls and async-std and no dynamically linked libraries. Enabling this build feature removed all -sys dependencies from androidx-release-watcher, allowing me to build static executables of it.

" }, { "id": "https://msfjarvis.dev/posts/learning-zig--day-4/", "url": "https://msfjarvis.dev/posts/learning-zig--day-4/", "title": "Learning Zig - Day 4", "date_published": "2021-05-18T06:30:00.000Z", "summary": "Brushing up on standards", "content_html": "

Today I’ll be getting familiar with common patterns in Zig like providing explicit allocators and learn about more of the standard library. This is of course, chapter 2 of the ZigLearn.org curriculum.

\n

Thoughts™️

\n

Allocators

\n

Rust’s standard library performs allocations as and when necessary, automatically, and provides a separate no_std mode that eliminates the standard library from the build and only retains the libcore which is suitable for use in bare-metal deployments (more on that in The Embedded Rust Book). Zig takes an alternative approach, where the convention is that all functions in the the standard library that require allocations take in an explicit Allocator which can either be implemented by the user, or you can pick one of the many options available in the standard library itself.

\n

I’ve never given an extreme amount of thought to the allocations my programs perform, so I’m rather unopinionated on this. That being said, whenever I start working on the healthchecks library I’m definitely going to follow the ecosystem’s practices and make allocations explicit for consumers.

\n

Filesystem

\n

The FS APIs appear to be quite expansive, covering everything that I can think of, and I finally discovered a practical use-case for defer! I’m pretty sure I’ll end up passing the incorrect CreateFlags on more than one occasion, but at least they’re documented and discoverable, and I have some memory of doing the same with Python back when I still believed in my country’s education system :P

\n

Formatting

\n

Finally an answer to my “how to format an array” mystery! It’s fascinating that formatting also requires allocations, but then Rust does return an owned String rather than a &str when you format things so that should have been obvious in hindsight.

\n

JSON

\n

The native JSON support looks pretty great, though I am left to wonder how much of serde’s flexibility is available here. Guess we’ll find out soon enough :)

\n

Random numbers and crypto

\n

Great to see native crypto in Zig! I’m not the target audience, but lack of crypto primitives in Rust seems to come up often so it’s a nice plus for Zig.

\n

Formatting specifiers and Advanced Formatting

\n

Zig’s formatting system seems about as powerful as Rust’s, maybe more, so it’s definitely a plus for me since I’ve ended up debugging a lot of code with println!(\"{:?}\", $field) in Rust 😅

\n

Conclusion

\n

I skipped through the parts about HashMaps, Stacks, sorting and iterators since those are fairly straightforward concepts that Zig does not appear to reinvent in any way.

\n

Overall, I’m liking everything I’m seeing. Very excited to start building things in Zig!

" }, { "id": "https://msfjarvis.dev/posts/learning-zig--day-3/", "url": "https://msfjarvis.dev/posts/learning-zig--day-3/", "title": "Learning Zig - Day 3", "date_published": "2021-05-16T06:30:00.000Z", "summary": "Finishing up the basics", "content_html": "

Yesterday’s post was a bit shorter than I planned, since I didn’t manage to go through as much of the ZigLearn chapter 1 as I thought I would. Today we’ll be wrapping it up.

\n

Thoughts™️

\n

Same as yesterday, this section will be a brain dump of what I think of the things I learn today.

\n

Pointers

\n

Rust made pointers a very friendly concept thanks to the borrow checker and amazing compiler diagnostics, and Zig seems to follow the same path with keeping them straightforward. I’m not the biggest fan of the variable.* syntax for dereferencing a pointer, since it breaks my existing muscle memory in a major way but I’m sure I’ll get used to it in no time.

\n

Just like Rust, mutable and immutable pointers are explicitly distinct which is a ✅ in my book.

\n

There wasn’t a lot of content on ZigLearn about many-item pointers so I’m still not sure I understand any of it. That’s probably just me though.

\n

I’ve known and used Rust’s usize in my programs, but only after reading about pointer-sized integers on ZigLearn did I actually make the connection that the size of a usize is that of a pointer. 💡

\n

Enums

\n

On a syntactic level, Zig enums are closer to Kotlin than to Rust w.r.t. declaring functions in them, which is very nice.

\n

Structs

\n

The syntax note from enums applies here as well, with an additional nicety about pointers. Specifically, a struct function that accepts a pointer value will automatically dereference the value inside the function body. This only goes one level deep though, so keep that in mind.

\n
const Rectangle = struct {
    length: i32,
    width: i32,
\n
    pub fn swap(self: *Rectangle) void {
        // No explicit dereferencing needed!
        const tmp = self.length;
        self.length = self.width;
        self.width = tmp;
    }
};
\n

Unions

\n

I have never worked with unions, but tagged unions gave me awful ideas about matching Kotlin’s sealed classes functionality so I’m looking forward to writing some cursed code :D

\n

Integer rules and Floats

\n

Zig’s type coercion syntax is nicer than Rust’s, though the lack of a runtime error concerned me initially. Rust’s TryInto trait is explicit about the fact that the conversion is fallible, and thus returns a Result. Zig on the other hand attempts to validate these conversions at compile-time. Given this code:

\n
fn returnsNum() u32 {
    return 1_00_000;
}
\n
test \"typecast\" {
    testing.expect(@TypeOf(@as(u8, returnsNum())) == u8);
}
\n

The build will fail with:

\n
./src/main.zig:182:46: error: expected type 'u8', found 'u32'
    testing.expect(@TypeOf(@as(u8, returnsNum())) == u8);
                                             ^
./src/main.zig:182:46: note: unsigned 8-bit int cannot represent all possible unsigned 32-bit values
    testing.expect(@TypeOf(@as(u8, returnsNum())) == u8);
\n

This is good, but I haven’t yet found a way to force the conversion to go through in instances where I can confirm that the incoming u32 definitely fits in a u8.

\n

Optionals

\n

Zig’s Optionals are a very good parallel for Rust’s Option, though Zig provides a lot more syntactic niceties.

\n

The fact that you can use a while loop to capture values until they become null is pretty damn sweet.

\n
var numbers_left: u32 = 4;
fn eventuallyNullSequence() ?u32 {
    if (numbers_left == 0) return null;
    numbers_left -= 1;
    return numbers_left;
}
\n
test \"while null capture\" {
    var sum: u32 = 0;
    while (eventuallyNullSequence()) |value| {
        sum += value;
    }
    expect(sum == 6); // 3 + 2 + 1
}
\n

Conclusion

\n

Everything following optionals was either uncontroversial or too powerful/low level for my current interests, so I admittedly glossed over some of the gory details.

\n

Chapter 2 introduces JSON, which we’ll need for our eventual healthchecks.io library, so I’m looking forward to it! I do have work tomorrow, so we’ll have to see if I can keep up the daily streak :)

" }, { "id": "https://msfjarvis.dev/posts/learning-zig--day-2/", "url": "https://msfjarvis.dev/posts/learning-zig--day-2/", "title": "Learning Zig - Day 2", "date_published": "2021-05-15T06:30:00.000Z", "summary": "Onwards in our quest to learn Zig", "content_html": "

In the previous post I documented how I went about setting up my Zig environment, and it’s now time to start learning things.

\n

My preferred method of learning new languages is rebuilding an existing project in them, like I did when going from Python to Kotlin to Rust. For Zig, I’ve elected to rebuild my healthchecks-rs library. It’s something I use on a day-to-day basis for keeping an eye on my backup jobs, and it would be a great addition to the healthchecks.io ecosystem.

\n

Getting the basics down

\n

Among the resources enlisted on the Zig getting started page, I opted to go with ziglearn.org for learning the ropes of the language. It is concise yet detailed, and the chapter-wise breakdown makes for great mental “checkpoints”, much like the Rust book.

\n

For this post I’m going through chapter 1.

\n

Thoughts™️

\n

I’m going to use this section to jot down my thoughts about Zig, broken down by the sections on ZigLearn. I’ll skip the parts that I don’t have anything to say on.

\n

Assignment

\n

The presence of undefined is very interesting to me. It appears to be functionally identical to Rust’s Default trait, as shown in this snippet (had to skip to structs for this since I was so curious about it).

\n
const std = @import(\"std\");
\n
const Vec3 = struct {
    x: f32,
    y: f32,
    z: f32,
};
\n
pub fn main() void {
    const inferred_constant: Vec3 = undefined;
    std.debug.print(\"Hello, {d}!\\n\", .{inferred_constant});
}
\n

which prints:

\n
➜ zig build run
Hello, Vec3{ .x = 0, .y = 0, .z = 0 }!
\n

I have clearly not gotten very far yet, but initial thoughts here: Rust’s trait-based implementation means I can customize the “default” values for my structs, which I’m not seeing in this implicit coercion yet. Guess we’ll find out soon whether or not this can be handled explicitly :D

\n

Arrays

\n

The syntax for array declarations is quite clear and explicit, which I like. Notably, while you can do this:

\n
const implicitly_sized_array = [_]u8{}; // _ means \"infer the size\"
\n

you cannot have an inferred size reference as the type:

\n
const implicitly_sized_array[_]u8 = {};
\n

Rust also disallows this, but the error is surprisingly worse than with Zig. Rust’s resident diagnostics magician Esteban assures me this is a regression and is being tracked.

\n

The only problem I encountered here was that I can’t figure out how to print an array!

\n
const implicitly_sized_array = [_]u8{0, 1, 2, 3};
std.debug.print(\"This is an array: {}\\n\", .{implicitly_sized_array});
\n
// Outputs: \"This is an array: \"
\n

I found a PR that overhauls formatting but nothing there gave me any pointers on why my code doesn’t work. Hopefully this’ll get cleared up later.

\n

If

\n

Nothing special here, aside from the early introduction to testing, which is slightly more pleasant than with Rust. I do however have qualms about the test output, which is unnecessarily noisy:

\n
Terminal window
Test [2/2] test \"while with continue expression\"... expected 2080, found 10
/nix/store/nhd75c4sr3l9wlaspilkwawx5ixkn74w-zig-0.7.1/lib/zig/std/testing.zig:74:32: 0x206f9a in std.testing.expectEqual (test)
                std.debug.panic(\"expected {}, found {}\", .{ expected, actual });
                               ^
/home/msfjarvis/git-repos/zig-playground/src/main.zig:29:24: 0x205abd in test \"while with continue expression\" (test)
    testing.expectEqual(sum, 10);
                       ^
/nix/store/nhd75c4sr3l9wlaspilkwawx5ixkn74w-zig-0.7.1/lib/zig/std/special/test_runner.zig:61:28: 0x22e161 in std.special.main (test)
        } else test_fn.func();
                           ^
/nix/store/nhd75c4sr3l9wlaspilkwawx5ixkn74w-zig-0.7.1/lib/zig/std/start.zig:334:37: 0x20749d in std.start.posixCallMainAndExit (test)
            const result = root.main() catch |err| {
                                    ^
/nix/store/nhd75c4sr3l9wlaspilkwawx5ixkn74w-zig-0.7.1/lib/zig/std/start.zig:162:5: 0x2071d2 in std.start._start (test)
    @call(.{ .modifier = .never_inline }, posixCallMainAndExit, .{});
    ^
error: the following test command crashed:
./src/zig-cache/o/5a647f3d3394214b30b3861a6b0ffbbb/test
\n

Defer

\n

The defer language feature is something I’ve always been curious about since seeing it in Go, so I’m excited to discover use-cases for it when I finally have it available. Through ZigLearn I discovered that defer calls can be stacked to be executed in LIFO order, and Golang does it in the exact same fashion.

\n

Errors

\n

I like how easy it is to define errors, but the syntax feels kinda icky. Having each error enum I declare ‘magically’ become a property on the error keyword doesn’t sit right with me :(

\n
const NumericError = error{};
\n
fn mayError(shouldError: bool) anyerror!u32 {
    return if (shouldError)
        // This is different from what I'm accustomed to as a user of
        // Either/Result type monads.
        error.NumericError
    else
        10;
}
\n

Runtime Safety

\n

Being able to turn off runtime safety features (like bounds checking) in specific blocks is pretty interesting! Not sure if I’ll ever have a valid use for it though…

\n

Conclusion

\n

I like most of what I’ve seen so far in Zig. Aside from the issues I mentioned above, the lack of string type is a very confusing thing for me. I’ve kinda come to expect it everywhere based on my previous experiences with Python, Java, Kotlin, and Rust; but maybe I’ll now learn to appreciate how every character on my screen is just numbers :D.

\n

I was very easily distracted today, so I only made it a third of the way in 3 hours for a chapter that is supposed to take 1 hour for the whole thing. Hoping to finish it tomorrow!

" }, { "id": "https://msfjarvis.dev/posts/first-steps-with-zig/", "url": "https://msfjarvis.dev/posts/first-steps-with-zig/", "title": "First steps with Zig", "date_published": "2021-05-14T06:30:00.000Z", "summary": "I've decided to learn Zig, and here's how I'm preparing for it.", "content_html": "

Zig is a systems programming language much akin to Rust and C, and has been showing up in my feeds a lot as of late. Many Zig programmers have documented their experience with Zig as much better than with Rust, which I have been programming in for the last year or so, citing simplicity and ease. I tend to agree that Rust can often be complex to enforce the guarantee of being correct, so I set out to finally buy into the promise of Zig and give it a shot.

\n

Compiler and IDE setup

\n

The installing Zig page recommends that while using the Zig stable releases is fine for evaluating it, their stable release cadence matches LLVM’s ~6 months which means they are often rendered outdated by the fast pace of Zig development.

\n

Since I wanted to stick with using Nix to manage my (currently) temporary Zig environment, I went with the stable 0.7.1 release available on nixpkgs.

\n

A quick nix-shell -p zig later, I now had access to the Zig compiler.

\n
Terminal window
➜ nix-shell -p zig
➜ zig version
0.7.1
\n

To be able to use VSCode for writing Zig, I also installed the official zls language server for Zig. This did get me go-to-declaration support for the standard library, but not syntax highlighting. I’m not sure if that’s intended, or a bug with my local setup. Syntax highlighting is also present, thanks to Lewis Gaul for his suggestion of using the tiehuis.zig extension.

\n

Learning resources

\n

The Zig team frankly admits that they do not yet have the resources to maintain extensive learning resources, but the Zig community has stepped forward to fill in those gaps. ziglearn.org is a great jumping off point for people who prefer to learn language basics directly, and there is a rustlings counterpart in ziglings for learning by looking at code.

\n

On the official side of things, you get the standard library reference as one would expect, as well as a fairly detailed language reference.

\n

This is in contrast with Rust, which has an officially maintained book and maintains rustlings as a first-party learning resource. They do however are a significantly larger and older team, so maybe with sufficient funding we’ll see Zig be able to devote effort towards this as well.

\n

Your first program

\n

The zig CLI contains commands to generate new projects easily, so let’s create a new binary project.

\n
➜ zig init-exe
info: Created build.zig
info: Created src/main.zig
info: Next, try `zig build --help` or `zig build run`
\n

The build.zig file appears to describe to the zig CLI how to build this program, and src/main.zig is our application code. Here’s what zig init-exe gives you for a “hello world” program:

\n
src/main.zig
const std = @import(\"std\");
\n
pub fn main() anyerror!void {
    std.log.info(\"All your codebase are belong to us.\", .{});
}
\n

Cheeky.

\n

This post is just a brief overview of how I went about setting things up for learning Zig. I intend to post more detailed blogs as I progress :)

" }, { "id": "https://msfjarvis.dev/posts/github-actions-tips-tricks/", "url": "https://msfjarvis.dev/posts/github-actions-tips-tricks/", "title": "Tips and Tricks for GitHub Actions", "date_published": "2021-01-02T00:00:00.000Z", "summary": "GitHub Actions is a power CI/CD platform that can do a lot more than your traditional CI systems. Here's some tips to get you started with exploring its true potential.", "content_html": "

GitHub Actions has grown at a rapid pace, and has become the CI platform of choice for most open source projects. The recent changes to Travis CI’s pricing for open source is certainly bound to accelerate this even more.

\n

Due to it being a first-party addition to GitHub, Actions has nearly infinite potential to run jobs in reaction to changes on GitHub. You can automatically set labels to newly opened pull requests, greet first time contributors, and more.

\n

Let’s go over some things that you can do with Actions, and we’ll end it with some safety related tips to ensure that your workflows are secure from both rogue action authors as well as rogue pull requests.

\n

Running workflows based on a cron trigger

\n

GitHub Actions can trigger the execution of a workflow in response to a large list of events as given here, one of them being a cron schedule. Let’s see how we can use the schedule feature to automate repetitive tasks.

\n

For Android Password Store, we maintain a list of known public suffixes to be able efficiently detect the ‘base’ domain of the website we’re autofilling into. This list changes frequently, and we typically sync our repository with the latest copy on a weekly basis. Actions enables us to do this automatically:

\n
name: Update Publix Suffix List data
on:
  schedule:
    - cron: \"0 0 * * 6\"
\n
jobs:
  update-publicsuffix-data:
  # The actual workflow doing the update job
\n

Putting the cron expression into crontab guru, you can see that it executes at 12AM on every Saturday. Going through the merged pull requests in APS, you will also notice that the publicsuffixlist pull requests indeed happen no sooner than 7 days apart.

\n

Mine is a very naive example of how you can use cron triggers to automate parts of your workflow. The Rust project uses these same triggers to implement a significantly more important aspect of their daily workings. Rust maintains a repository called glacier which contains a list of internal compiler errors (ICEs) and code fragments to reproduce each of them. Using a similar cron trigger, this repository checks each new nightly release of Rust to see if any of these compiler crashes were resolved silently by a refactor. When it comes across a ICE that was fixed (compiles correctly or fails with errors rather than crashing the compiler), it files a pull request moving the reproduction file to the fixed pile.

\n

Running jobs based on commit message

\n

Continuous delivery is great, but sometimes you want slightly more control. Rather than run a deployment task on each push to your repository, what if you want it to only run when a specific keyword is in the commit message? Actions has support for this natively, and the deployment pipeline of this very site relies on this feature:

\n
name: Deploy to Cloudflare Workers Sites
\n
on:
  push:
    branches:
      - main
\n
jobs:
  deploy-main:
    if: \"contains(github.event.head_commit.message, '[deploy]')\"
    # Set up wrangler and push to the production environment
\n
  deploy-staging:
    if: \"contains(github.event.head_commit.message, '[staging]')\"
    # Set up wrangler and push to the staging environment
\n

This snippet defines a job that is only executed when the top commit of the push contains the text [deploy] in its message, and another that only runs when the commit message contains [staging]. Together, these let me control if I want a change to not be immediately deployed, deployed to either the main or staging site, or to both at the same time. So now I can update a draft post without a full re-deployment of the main site, or make a quick edit to a published post that doesn’t need to be reflected in the staging environment.

\n

The core logic of this operation is composed of three parts. The github context, the if conditional and the contains method. The linked documentation for each does a great job at explaining them, and has further references to allow you to fulfill even more advanced use cases.

\n

Testing across multiple configurations in parallel

\n

Jobs in a workflow run in parallel by default, and GitHub comes with an amazing matrix functionality that can automatically generate multiple jobs for you from a single definition. Take this specific example:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
WindowsMacOSUbuntu
StableWindows + StableMacOS + StableUbuntu + Stable
BetaWindows + BetaMacOS + BetaUbuntu + Beta
NightlyWindows + NightlyMacOS + NightlyUbuntu + Nightly
\n
\n

This particular matrix is for Rust, to test a codebase across Windows,\nUbuntu, and macOS using the Rust stable, beta, and nightly toolchains.

\n
\n

In GitHub Actions, we can simply provide the platforms (Windows, MacOS and, Ubuntu) and the Rust channels (Stable, Beta, and Nightly) inside a single job and let it figure out how to make the permutations and create separate jobs for them. To configure such a matrix, we write something like this:

\n
jobs:
  check-rust-code:
    strategy:
      # Defines a matrix strategy
      matrix:
        # Sets the OSes we want to run jobs on
        os: [ubuntu-latest, windows-latest, macOS-latest]
        # Sets the Rust channels we want to test against
        rust: [stable, beta, nightly]
    # Make the job run on the OS picked by the matrix
    runs-on: ${{ matrix.os }}
\n
    steps:
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          components: rustfmt, clippy
          # Installs the Rust toolchain for the channel picked by the matrix
          toolchain: ${{ matrix.rust }}
\n

This will automatically generate 9 (3 platforms * 3 Rust channels) parallel jobs to test this entire configuration, without requiring us to manually define each of them. DRY at its finest :)

\n

Make a job run after another

\n

By default, jobs defined in a workflow file run in parallel. However, we might need a more sequential order of execution for some cases, and GHA does include support for this case. Let’s try another real world example!

\n

LeakCanary has a checks job that runs on each push to the main branch and on each pull request. They wanted to add support for snapshot deployment, in order to finally retire Travis CI. To make this happen, I simply added a new job to the same workflow, having it run only on push events and have a dependency on the checks job. This ensures that there won’t be a snapshot deployment until all tests are passing on the main branch. The relevant parts of the workflow configuration are here:

\n
on:
  pull_request:
  push:
    branches:
      - main
\n
jobs:
  checks:
  # Runs automated unit and instrumentation tests
\n
  snapshot-deployment:
  # Only run if the push event triggered this workflow run
  if: \"github.event_name == 'push'\"
  # Run after the 'checks' job has passed
  needs: [checks]
\n

Mitigating security concerns with Actions

\n

GitHub Actions benefits from a vibrant ecosystem of user-authored actions, which opens it up to equal opportunities for abuse. It is relatively easy to work around the common ones, and I’m going to outline them here. I’m no authority on security, and these recommendations are based on a combination of my reading and understanding. These should be helpful, but this list is not exhaustive, and you should exercise all the caution you can.

\n

Use exact commit hashes rather than tags

\n

Tags are moving qualifiers, and can be force pushed at any moment. If the repository for an Action you use in your workflows is compromised, the tag you use could be force pushed with a malicious version that can send your repository secrets to a third-party server. Auditing the source of a repository at a given tag, then using the SHA1 commit hash it currently points to as the version addresses that concern due to it being nearly impossible to fake a new commit with the exact hash.

\n

To get the commit hash for a specific tag, head to the Releases page of the repository, then click the short SHA1 hash below the tag name and copy the full hash from the URL.

\n
\"AExpand image
\n
\n

Here, the commit hash is feb985e. Ideally, you want to click that link and\ncopy the full hash from the URL

\n
\n
job:
  checks:
   - uses: burrunan/gradle-cache-actions@v1.6
 +  - uses: burrunan/gradle-cache-actions@feb985ecf49f57f54f31920821a50d0394faf122
\n

Alternate solution

\n

A more extreme fix for this problem is to vendor each third-party action you use into your own repository, and then use the local copy as the source. This puts you in charge of manually syncing the source to each version, but allows you to restrict the allowed Actions to ones in your repository thereby greatly increasing security. However, having to manually sync can get tedious if your workflows involve a lot of third-party actions. However, the same manual sync also gives you slightly better visibility into the changes between versions since they’d be available in a single PR diff.

\n

To use an Action from a local directory, replace the uses: line with the relative path to the local copy in the repository.

\n
job:
  checks:
    - name: Checkout repository
    # Assuming the copy of actions/checkout is at .github/actions/checkout
   - uses: actions/checkout@v2
   - uses: ./.github/actions/checkout
\n

Replace pull_request_target with pull_request

\n

pull_request_target grants a PR access to a github token that can write to your repository, exposing your code to modification by a malicious third-party who simply needs to open a PR against your repository. Most people will already be using the safe pull_request event, but if you are not, audit your requirements for pull_request_target and make the switch.

\n
on: [push, pull_request_target]
on: [push, pull_request]
\n
\n

I’m still learning about Actions, and there is a lot that I did not cover here. I highly encourage readers to refer the GitHub docs for Workflow syntax and Context and expressions syntax to gain more knowledge of the workflow configuration capabilities. Let me know if you find something cool that I did not cover here!

" }, { "id": "https://msfjarvis.dev/posts/manually-parsing-json-with-moshi/", "url": "https://msfjarvis.dev/posts/manually-parsing-json-with-moshi/", "title": "Manually parsing JSON with Moshi", "date_published": "2020-12-21T06:30:00.000Z", "summary": "Moshi is a fast and powerful JSON parsing library for the JVM and Android. Today we look into manually parsing JSON to and from Java/Kotlin classes", "content_html": "

What is Moshi?

\n

Moshi is a fast and powerful JSON parsing library for the JVM and Android, built by the former creators of Google’s Gson to address some of its shortcomings and to have an alternative that was actively maintained.

\n

Unlike Gson, Moshi has excellent Kotlin support and supports both reflection based parsing and a kapt-backed codegen backend that eliminates the runtime performance cost in favor of generating adapters during build time. The kotlin-reflect dependency required for doing reflection-based parsing can add up to 1.8 mB to the final binary, so it’s recommended to use the codegen method if possible.

\n

What is an adapter?

\n

An adapter is Moshi-speak for a class that can convert JSON into an object and an instance of that object into JSON. There are multiple types of JSON adapters supported by Moshi. The first is the one demonstrated in their README which contains two methods annotated @ToJson and @FromJson. The former takes an instance of the object and returns a String, and the latter takes a String and returns an instance of the object. This is the simplest type, and should be used for non-complex types that typically can be represented in simpler forms. Here’s the example Moshi uses, and should be all the introduction you need for this particular type.

\n

The other type is similar to what Moshi generates for its kapt-generated adapters, but leverages the @ToJson/@FromJson annotations. The method signatures here are a bit verbose, and these are the ones we’re going to try to build.

\n

Why write your own adapters?

\n

Good question. Consider this example class:

\n
@JsonClass(generateAdapter = true)
class TextParts(val heading: String, val body: String? = null)
\n

Pretty straightforward. The JsonClass annotation with generateAdapter = true will attempt to use the codegen backend to write an adapter automatically for this. Let’s try converting this to JSON.

\n
val text = TextParts(\"This is the heading\", \"And this is the body\")
val moshi = Moshi.Builder().build()
// TextPartsJsonAdapter was generated by the codegen backend
println(TextPartsJsonAdapter(moshi).toJson(text))
{\"heading\":\"This is the heading\",\"body\":\"And this is the body\"}
\n

What this means is, given a JSON object that looks like this

\n
{ \"heading\": \"This is the heading\", \"body\": \"And this is the body\" }
\n

We can get an instance of TextParts that looks like this

\n
val text = TextParts(\"This is the heading\", \"And this is the body\")
\n

Cool! Now, let’s make things unfortunate. Imagine your backend team is stretched thin, and due to a limitation with how they initially built their database schema, you can only get the above JSON in this form

\n
{
  \"heading\": \"This is the heading\",
  \"extras\": { \"body\": \"And this is the body\" }
}
\n

If you try to parse this with the old TextPartsJsonAdapter, your app is going to crash, because the JSON and its Kotlin representation have diverged. The equivalent Kotlin for this new JSON is going to be something like this:

\n
@JsonClass(generateAdapter = true)
class Extras(val body: String? = null)
\n
@JsonClass(generateAdapter = true)
class TextParts(val heading: String, val extras: Extras? = null)
\n

Many things changed here. Your direct access to the body field now needs to go through extras, which just isn’t that nice. You’re also now incurring the (albeit miniscule) overhead of generating two adapters rather than one. Wouldn’t it be great if we could continue to have a flat object like before? Let’s try to make that happen.

\n

How to write your own Moshi adapter?

\n

With less effort than one might think! Let’s put down the basic building blocks.

\n
class TextPartsJsonAdapter {
  // Moshi is flexible about the parameters of these two methods, and for simpler types
  // you will find it easier to follow the example from the Moshi README which does not
  // use JsonReader/JsonWriter and instead directly converts items to and from their String
  // representations. The method names are also not enforced, as Moshi only uses the
  // annotations to find relevant methods. The internal implementation of how they do it
  // can be found here: https://git.io/JLwnb
\n
  @FromJson
  fun fromJson(reader: JsonReader): TextParts? {
    TODO(\"Not implemented\")
  }
\n
  @ToJson
  fun toJson(writer: JsonWriter: value: TextParts?) {
    TODO(\"Not implemented\")
  }
}
\n

Now we’re ready to start parsing. First, let’s implement the toJson part, where we take an instance of the object and then try to write the equivalent JSON for it. Since this is comparatively easier, I’m going to do it in one go and leave comments inline to explain what’s happening.

\n
@ToJson
fun toJson(writer: JsonWriter: value: TextParts?) {
  // Null values shouldn't arrive to the adapter, this error lets callers know
  // what builder options need to be passed to the Moshi.Builder() instance
  // to avoid this particular situation.
  if (value == null) {
    throw NullPointerException(\"value was null! Wrap in .nullSafe() to write nullable values.\")
  }
  // Use the Kotlin `with` scoping method so we don't need to call
  // all methods with the `writer.` prefix.
  with(writer) {
    // Start the JSON object.
    beginObject()
\n
    // Since our `extras` field is nullable, and our backend will send
    // it as a literal null rather than skip it, we want null values to
    // be written into the final JSON.
    serializeNulls = true
\n
    // Create a JSON field with the name 'heading'
    name(\"heading\")
\n
    // Set the value of the 'heading' field to the actual heading
    value(value.heading)
\n
    // Create the 'extras' field
    name(\"extras\")
    if (value.body != null) {
      // If the body text exists, then start a new object and add a
      // body field
      beginObject()
      name(\"body\")
      value(value.bodyText)
      endObject()
    } else {
      // Otherwise we put down a literal null
      nullValue()
    }
\n
    // End the top-level object.
    endObject()
  }
}
\n

Parsing JSON manually is relatively easy to screw up and Moshi will let you know if you get nesting wrong (missed a closing endObject() or endArray()) and other easily detectable problems, but you should definitely have tests for all possible cases. I’ll let the readers do that on their own, but if you really need to see an example then let me know below.

\n

Anyways, that’s the object -> JSON part sorted. Now let’s try to do the reverse. Here’s where we are as of now.

\n
fun fromJson(reader: JsonReader): TextParts? {
  TODO(\"Not implemented\")
}
\n

Same as writing JSON, we need to start by making an object.

\n
fun fromJson(reader: JsonReader): TextParts? {
  // We'll be constructing the object at the end so these
  // will store the values we read.
  var heading: String? = null
  var body: String? = null
  with(reader) {
    beginObject()
    endObject()
  }
  TODO(\"Not implemented\")
}
\n

We have a fixed set of keys that we expect to read, so go ahead and configure a couple instances of JsonReader.Options that we will use to find the keys in this JSON.

\n
val topLevelKeys = JsonReader.Options.of(\"heading\", \"extras\")
val extrasKeys = JsonReader.Options.of(\"body\")
\n
fun fromJson(reader: JsonReader): TextParts? {
  // We'll be constructing the object at the end so these
  // will store the values we read.
\n

And we’re set. You’ll see the significance of the Options objects now.

\n
var body: String? = null
with(reader) {
  beginObject()
  while(hasNext()) {
    when(selectName(topLevelKeys)) {
      0 -> heading = nextString() ?: throw Util.unexpectedNull(
        \"heading\",
        \"text\",
        this
      )
    }
  }
  endObject()
}
TODO(\"Not implemented\")
\n

reader.hasNext() is going to continue iterating through the document’s tokens until it’s completed, which lets us look through the entire document for the parts we need. The selectName(JsonReader.Options) method will return the index of a matched key, so 0 there means that the heading key was found. In response to that, we want to read it as a string and throw if it is null (since it’s non-nullable in TextParts). The Util.unexpectedNull method is a little nicety that is part of Moshi’s internals and is used by its kapt-generated adapters to provide better error messages and we’re going to do the same.

\n
    0 -> heading = nextString() ?: throw Util.unexpectedNull(
      \"heading\",
      \"text\",
      this
    )
    -1 -> {
      // Skip unknown values
      skipName()
      skipValue()
    }
  }
}
endObject()
\n

When I said that selectName returns the index of the matched key, I didn’t mention that it returns -1 when it comes across a key that isn’t in the Options object. Since we don’t care about them, we’re going to skip both their name and value and continue right on ahead. Now, we’re going to try and parse that inner extras object. A lot is about to happen quickly, but bear with me as I explain things.

\n
  \"text\",
  this
)
1 -> {
  // \"extras\" is nullable, so we first try to see if it is null.
  // If it isn't, this will throw and we can then safely assume
  // a non-null value and proceed.
  try {
    nextNull<Any>()
  } catch (_: JsonDataException) {
    beginObject()
    while (hasNext()) {
      when (selectName(extrasKeys)) {
        0 -> body = nextString()
        -1 -> {
          // Skip unknown values
          skipName()
          skipValue()
        }
      }
    }
    endObject()
  }
}
-1 -> {
  // Skip unknown values
  skipName()
  skipValue()
\n

Now that you look at it, not really that different from what we did above. The only new thing here is the nextNull method, which simply tries to find a null value and throws the JsonDataException if the value wasn’t null.

\n
    }
    endObject()
  }
  TODO(\"Not implemented\")
  // Satisfy the typechecker and throw in case the JSON body
  // didn't contain the 'heading' field at all
  require(heading != null) { \"heading must not be null\" }
  return TextParts(heading, body)
}
\n

And that’s it! The final adapter is going to look like this

\n
class TextPartsJsonAdapter {
  val topLevelKeys = JsonReader.Options.of(\"heading\", \"extras\")
  val extrasKeys = JsonReader.Options.of(\"body\")
\n
  @FromJson
  fun fromJson(reader: JsonReader): TextParts? {
    // We'll be constructing the object at the end so these
    // will store the values we read.
    var heading: String? = null
    var body: String? = null
    with(reader) {
      beginObject()
      while(hasNext()) {
        when(selectName(topLevelKeys)) {
          0 -> heading = nextString() ?: throw Util.unexpectedNull(
            \"heading\",
            \"text\",
            this
          )
          1 -> {
            // \"extras\" is nullable, so we first try to see if it is null.
            // If it isn't, this will throw and we can then safely assume
            // a non-null value and proceed.
            try {
              nextNull<Any>()
            } catch (_: JsonDataException) {
              beginObject()
              while (hasNext()) {
                when (selectName(extrasKeys)) {
                  0 -> body = nextString()
                  else -> {
                    // Skip unknown
                    skipName()
                    skipValue()
                  }
                }
              }
              endObject()
            }
          }
          -1 -> {
            skipName()
            skipValue()
          }
        }
      }
      endObject()
    }
    // Satisfy the typechecker and throw in case the JSON body
    // didn't contain the 'heading' field at all
    require(heading != null) { \"heading must not be null\" }
    return TextParts(heading, body)
  }
\n
  @ToJson
  fun toJson(writer: JsonWriter: value: TextParts?) {
    // Null values shouldn't arrive to the adapter, this error lets callers know
    // what builder options need to be passed to the Moshi.Builder() instance
    // to avoid this particular situation.
    if (value == null) {
      throw NullPointerException(\"value was null! Wrap in .nullSafe() to write nullable values.\")
    }
    // Use the Kotlin `with` scoping method so we don't need to call
    // all methods with the `writer.` prefix.
    with(writer) {
      // Start the JSON object.
      beginObject()
\n
      // Since our `extras` field is nullable, and our backend will send
      // it as a literal null rather than skip it, we want null values to
      // be written into the final JSON.
      serializeNulls = true
\n
      // Create a JSON field with the name 'heading'
      name(\"heading\")
\n
      // Set the value of the 'heading' field to the actual heading
      value(value.heading)
\n
      // Create the 'extras' field
      name(\"extras\")
      if (value.body != null) {
        // If the body text exists, then start a new object and add a body field
        beginObject()
        name(\"body\")
        value(value.bodyText)
        endObject()
      } else {
        // Otherwise we put down a literal null
        nullValue()
      }
\n
      // End the top-level object.
      endObject()
    }
  }
}
\n

This is certainly a lengthy job to do, and this blog post is a result of nearly 8 hours I spent writing JSON adapters by hand. Certainly not recommended if avoidable, but sometimes you just need to. When it comes to it, now you hopefully know how :)

" }, { "id": "https://msfjarvis.dev/posts/aps-october-release/", "url": "https://msfjarvis.dev/posts/aps-october-release/", "title": "Android Password Store October release", "date_published": "2020-10-22T06:30:00.000Z", "summary": "Long form release notes for the Android Password Store October release", "content_html": "

We’re back with yet another release! As I shared earlier this month, this is going to our last release for a while. There’s a lot of work left to be done, and we’re simply not big enough a team to have these larger changes be done separately from our main development. We’ll still be doing bugfix releases if and when required, so please do file bug reports as and when you encounter issues.

\n

New features

\n

GPG key selection added to onboarding

\n

Creating a new store from the app previously created an unusable store, because we never configured a GPG key in the .gpg-id file. This has now been remedied in two ways: empty .gpg-id files are correctly handled as invalid and included in our quickfix solution, and creating a new store will now request you to select a key and then write it into the .gpg-id file. Here’s what the key selection screen looks like:

\n
\"GPGExpand image
\n

Allow configuring an HTTPS proxy

\n

Before we close the gates on our regularly scheduled releases, our focus has been to address most longstanding issues and one of the major ones there has been Proxy support. This has now been added, and can be accessed from the settings screen. Unfortunately, there are still a few caveats with this current implementation that may or may not change in a future patch release:

\n\n

Add option to automatically sync repository

\n

This too, has been a consistent request in the past. While our implementation does not exactly match what was requested, we feel it’s good enough to be shipped. You now have the option to sync your repository on every launch to ensure things are always up-to-date when you get in the app.

\n

Due to multiple bugs, this feature has been rolled back in v1.13.1.

\n\n

Fixes

\n

Improved error messaging

\n

For a large set of connection related errors, the failure message would simply be ‘Invalid remote: origin’. That is exactly as unhelpful as one might think, and now we try harder to extract the actual, more meaningful error message.

\n

Use Git’s default user and email when none are configured

\n

We don’t force users to set a name and email before they make any changes requiring Git commits, but somewhere in the last couple releases we regressed our behavior around this. Rather than the root <root@localhost> committer, we were incorrectly using empty strings resulting in all commits being authored by &lt;&gt;. This has now been resolved, and your commit history will now be adorned by root@localhost once more (but seriously, just set your name and email already).

\n

Improvements around phishing detection UX

\n

APS has had comprehensive phishing detection built into our Autofill since day one. Our phishing-resistant search will not show your google.com passwords when you try to fill into goggle.com, and if the signature of an application changes after you first filled a password into it, we will warn you about the change. There were a couple issues with the way this was happening.

\n

First, the phishing detection UI was a bit complicated, and also had some unreadable, black-on-dark text. Since this was never reported to us, I believe none of our users are being phished by their apps which is great news :) Regardless, it is now fixed.

\n

Secondly, some complexity with how Android’s Autofill APIs work resulted in the “no I’m not being phished, accept this new signature” case to not work correctly. This caused the user to be continually shown the phishing detection prompt until they force closed the target app and started it again. That’s cumbersome, so we’ve fixed it now. Cheers to Fabian for his stellar work as always!

\n

Conclusion

\n

As you can notice, this is a bit of a small release by our standards. Fabian’s been busy with his Ph.D. (!!) and the new job he’s starting at soon (!!), and me and Aditya have been busy with our day jobs as well. This doesn’t spell doom for the project (yet), but your financial contributions over on GitHub Sponsors and OpenCollective are now much more important than ever to sustain the project during this time via bountied issues and simply compensating the current crop of developers for their time.

" }, { "id": "https://msfjarvis.dev/posts/aps-september-release/", "url": "https://msfjarvis.dev/posts/aps-september-release/", "title": "Android Password Store September release", "date_published": "2020-09-21T06:30:00.000Z", "summary": "Long form release notes for the Android Password Store September release", "content_html": "

Continuing with this new-ish tradition we have going, here are the detailed release notes for the v1.12.0 release.

\n
\n

Multiple important announcements at the end of the page, make sure to read the whole thing!

\n
\n

New features

\n

Extend Autofill support to more browsers

\n

Devin J. Pohly and Rounak Dutta collectively contributed support for 3 new Chromium-based browsers: Bromite, Ungoogled Chromium and Kiwi.

\n

Allow sorting by recently used

\n

This feature was requested a while ago and was implemented by Alex Molinares early in the cycle. The database that keeps track of the recently used passwords is always active, so if and when you switch to this sorting mode you’ll see everything already sorted based on your old usage patterns. Neat!

\n

Add ability to view Git commit log

\n

Another, even older feature request has finally been addressed. This too, came from an external contributor and was one of the best pull requests I have ever seen. It’s a great feature, and I thoroughly enjoyed the entire process of its inclusion.

\n

SSH key generation and handling improvements

\n

The old SSH key generation has been scrapped and rewritten to use a set of safer cryptographic curve options that span the distance between widely supported and very secure. The wiki page has been updated for these changes with information on how we’re securing access to the actual SSH keys, like storing the key file in the Android Keystore and requiring screen lock authentication before the key can be used.

\n

Fallback authentication for SSH

\n

SSH servers are often configured to have multiple authentication methods, where you first attempt to authenticate with private keys and if that fails, fall back to passwords. This wasn’t previously supported in APS, which would quit after the first failure. We’ve changed that to now offer the option of entering a password if the server is configured to fall back to it.

\n

Rewritten and redesigned onboarding flow

\n

In a multi-step refactoring process, the initial flow of setting up the app has been completely revamped. The internals were completely overhauled to improve stability, weed out some gnarly hacks, and make the whole thing easier to test and understand. Maintainer Aditya Wasan did a fabulous job giving the UI a facelift. It’s real pretty now ✨

\n

Show hidden folders now also shows hidden directories

\n

Our old ‘Show hidden folders’ feature has now been simplified to show all hidden files and folders in the repository. It is intended to make it easier to perform trivial maintenance tasks that would normally require access to a PC.

\n

Bugfixes

\n

SSH connection problems with Bitbucket

\n

In our last major release, we included a change to re-use SSH connections to speed up Git operations. This had an unfortunate side effect: Bitbucket users were unable to use SSH to connect to their repositories. Atlassian has been aware of this problem for quite some time now and did nothing about it, so we now include a helpful message and an internal workaround when this particular type of error is encountered.

\n\n

While still potentially finicky, we’re now confident that this is ready to be shipped to all users without the risk of crashes.

\n

Assorted UX improvements

\n

As always, there are a handful of Quality of Life changes to make the app more enjoyable to use:

\n\n

There’s definitely more fixes here, but we ended up rewriting, breaking and fixing so many things for this release that it’s hard to tell what was actually broken in the previous release and what is just us fixing regressions during refactoring. We’ve been busy :)

\n

Important announcements

\n

Autofill parser is now a standalone library!

\n

Our excellent Autofill capabilities are now bundled as a separate Android library and can be used by other password managers to improve their Autofill experiences. Detailed documentation will be coming over the next few days, keep an eye out here if it’s something you’re interested in.

\n

RFC for removal of Git support in external repos

\n

Based on the issues raised in the repository and the support emails I’ve received, the maintainers have come to the conclusion that nearly all users who choose to store their pass repositories in their device storage or external SD card as opposed to the app’s private, hidden directory are not users of Git and rely on solutions like Syncthing and Nextcloud to keep the repository in sync with their other devices.

\n

As such, we are now in the process of removing Git support from these repositories. We’ve carefully evaluated how we want to do this, and have started with removing the ability to clone repositories to public storage in this release. If this doesn’t blow up in our faces, we will be completing the transition in v1.13.0. If you believe the change adversely affects your usage of the app, we wanna know! Drop a comment on GitHub and we will do our best to either propose an alternative for your use case or entirely scrap our plans if we discover that our initial inferences were misguided.

" }, { "id": "https://msfjarvis.dev/posts/tools-of-the-trade-sdkman/", "url": "https://msfjarvis.dev/posts/tools-of-the-trade-sdkman/", "title": "Tools of the trade: SDKMAN!", "date_published": "2020-09-02T06:30:00.000Z", "summary": "Bringing this series back on popular demand, we're here to talk about SDKMAN!", "content_html": "

In the fourth post of this series, we’re talking about SDKMAN!.

\n

What is SDKMAN?

\n

SDKMAN is a SDK Manager. By employing its CLI, you can install a plethora of JVM-related tools and SDKs to your computer, without ever needing root access.

\n

Why do I use it?

\n

Since I primarily work with Kotlin, having an up-to-date copy of the Kotlin compiler becomes helpful for quickly testing code samples with its inbuilt REPL (Read Evaluate Print Loop). I also use Gradle as my build tool of choice, and tend to stay up-to-date with their releases in my projects. Finally, to run all these Java-based tools, you’re gonna need Java itself. Linux distros tend to package very outdated versions of the JDK which becomes a hindrance when building standalone JVM apps that I want to use the latest Java APIs in. SDKMAN allows you to install Java from multiple sources including AdoptOpenJDK, Azulsystems’ Zulu and many more.

\n

The real kicker here is the fact that you can keep multiple versions of the same thing installed. Using Java 14 everywhere but a specific project breaks over Java 8? Just install it alongside!

\n

To make this side-by-side ability even more useful, SDKMAN let’s you create a .sdkmanrc file in a directory, and it will switch your currently active version of any installed tool to the version specified in the file. People following the series from the beginning might recall this sounds awfully like direnv, because it is. However, SDKMAN is noticeably slow when executing these changes, presumably because its tearing down an existing symlink and creating a new one. For that reason, SDKMAN ships with the sdk_auto_env feature (automatically parse .sdkmanrc when you change directories) off by default, requiring you to manually type sdk env each time you enter a directory where you have a .sdkmanrc.

\n

Since the auto env feature matches what direnv does, I just use it directly. So, rather than doing this:

\n
.sdkmanrc
java=8.0.262-zulu
\n

I do:

\n
.envrc
# In bash, doing `${VARIABLE/src/dest}` replaces `src` with
# `dest` in `${VARIABLE}`, but you still need to write it
# back manually (hence the `export`).
export JAVA_HOME=\"${JAVA_HOME/current/8.0.262-zulu}\"
\n

for the same end result, but a lot faster.

\n

And that’s really it! SDKMAN’s pretty neat, but for the most part it just stays out of your way and thus there’s not a lot to talk about it. The next post’s gonna be much more hands-on :-)

\n

This was part 4 of the Tools of the trade series.

" } ] }