tlduck

Setting up Java on macOS

Ben Butterworth — Thu, 31 Dec 2020 00:00:00 GMT

Motivation

Setting up Java/ JAVA_HOME happens infrequently enough that I forgot how to do it efficiently, but frequent enough that it is annoying.

Problem?

I noticed MKYong's method uses brew to install java JDK versions (recommended by OpenJDK) and then sets JAVA_HOME explicitly. He doesn't use jenv because of "simplicity and no black box magic". MKYong modifies JAVA_HOME to change the java version, and this will hide all other JAVA versions from being listed in /usr/libexec/java_home -V. Also, if that path is broken, no versions show up. I suggest not modifying JAVA_HOME, because in a few months when you want to switch your java version, java_home will still show you which versions you have on your machine.

jenv does 2 things

It allows switching between Java versions and sets JAVA_HOME for you. You can just read jenv's Getting Started Guide now, but I've written the rest for my convenience, since I've set up too many machines this year (e.g. M1 Mac Mini, Intel i7 Hackintosh, MacBook Pro 16" from Popsa and my personal Intel MacBook Pro 13").

Steps

Ensure Homebrew is installed
Run brew install --cask adoptopenjdk. Optional: read/ understand the AdoptOpenJDK homebrew tap
Add the tap so you can also install other versions of Java: brew tap AdoptOpenJDK/openjdk
Install the version of Java you want: brew install --cask adoptopenjdk11 or with any other version you want.
Warning: DO NOT modify JAVA_HOME yourself, because that is what jenv will do. If you're updating an older system, get rid of your old code, e.g. export JAVA_HOME=... from your .bash_profile, .zprofile, .zshrc, or wherever you might have put it.
Install jenv: brew install jenv, and follow its output: it should say copy some lines into a configuration file.
mkdir -p ~/.jenv/versions, to prepare the folders for jenv to work properly. This is a workaround for a bug.
- The bug: When trying to add a JAVA version to jenv using e.g. jenv add adoptopenjdk-16.jdk/Contents/Home, you'll get ln: /Users/zen/.jenv/versions/openjdk64-16.0.1: No such file or directory.
jenv add /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home, remember to modify the path for your specific version. Notice the Contents/Home subdirectory.
Add more JDKs, for example for JDK15:jenv v
brew install --cask adoptopenjdk15
jenv add /Library/Java/JavaVirtualMachines/adoptopenjdk-15.jdk/Contents/Home/
jenv versions to list all JDKs
jenv global jdk_name to set a specific JDK, which also sets the JAVA_HOME variable for you. Check it with echo $JAVA_HOME. I prefer to run jenv global 15.

java --version should show the version you just set:

openjdk 15.0.2 2021-01-19
OpenJDK Runtime Environment AdoptOpenJDK (build 15.0.2+7)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15.0.2+7, mixed mode, sharing)

Conclusion

After finding jenv, I know I won't be running into setting up Java anymore. I highly recommend it.

How to delete many events in one go in Google Calendar

Ben Butterworth — Sun, 26 Apr 2020 00:00:00 GMT

CalendarMD has one sole purpose of existence: deleting events from google calendar in bulk, so I called it CalendarMD, where the MD stands for mass deleting. You search for the events in a specific calendar, and a list shows up. Scroll through the list to make sure you want to delete those events, and press delete.

I made a Free Android app with react-native. The code is on GitHub. Neither the UI nor the code look pretty, but it was the first app I made on React Native. Thinking back, this shouldn't have been a mobile app, and would have done better as a web app, because people are not going to install an app just to delete events from their calendar. I've since built another app for tracking the carbon footprint of food, Foodprint.

Markdown for convenient academic writing

Ben Butterworth — Thu, 28 May 2020 00:00:00 GMT

Read this if you think LaTex is too complicated, and you want to focus on writing content instead of formatting your the LaTex document.

Markdown `.md`

Markdown is a standard text format that many text editors support. It's actually the format that README files use in repositories websites, like GitHub and GitLab. I use Markdown to write notes (revision notes for courses at Imperial, notes about books/ articles I read), and project work, including my Master's thesis, because it can generate LaTex documents. Below is a screenshot of what it looks like when I'm writing my notes in Typora.

When I'm thinking about LaTex or Overleaf syntax, I'm not thinking about the content. In I don't want to think about back-ticks to get quotation marks correctly, copying references into a .bib file, and other tedious tasks I don't want to force out from the depths of my slender Hippocampus.

Benefits

Easy to move to a different editor when an editor becomes unpopular or a new editor is released with more features. Because Markdown is standard format, you can easily switch editors.
- Notion and OneNote are popular, but there are issues with it. What happens when the Notion app ceases to exist, or its subscription fee is increased, or a better app gets created? You might lose all your documents, manually copy all your notes over, or be forced to pay a higher price.
Many beautiful editors available. I use Typora because it has the cleanest UI which lets you focus on thinking about what you're writing, but Zettlr is also good. Compare this to installing the huge applications like LaTex or Microsoft Word. I've tried Overleaf, but this is a paid software that has reportedly had inconvenient downtime before a deadline, according to a lecturer at Imperial College.
- For example, putting quotes around a word requires you ``word"`. Start with a backtick and end with a double or single quote. That is extremely weird.
- Uploading files to Overleaf is tedious: you need to upload the file manually, and then write an LaTex figure which uses that image. It takes at least a minute, if you're good! It's not the time, its the context switching cost. With a Typora plugin it takes one drag of the mouse. Dragging an image into Typora will upload it to online storage that you configure. I have it storing images to Imgur, you can set it to store to AWS S3 and more. You'll need the UPic plugin for Typora though.
Widely used: A few blogging systems use Mardown or MDX (Markdown + JSX) for blog articles. JSX is the a markup language for React which looks similar to html. I'm using MDX for this blog. If I want to publish something I was writing in Typora, I just paste it into my blog .mdx file, set the publish date, and npm run build. I can also add some ReactJS components too.

Drawbacks

Syncing and collaboration: It would be great to collaborate on Markdown with others. And I still can't edit Markdown on Mobile, there doesn't seem to an application that can open sync . I might try to build a cloud service for Markdown files, with collaboration + mobile app 🙂.

LaTex

If I'm writing a real report with references, equation and Figure numbers/ captions, then I use something called Pandoc (a CLI tool & library) to generate the nicely formatted LaTex output without touching LaTex. The entire LaTex ecosystem is complicated. I don't want to think about formatting or even see the formatting when I'm reading research papers and writing content. It breaks my train of thought. There is still a learning curve with using Pandoc. The benefit is that you can write in peace, and worry about formatting much less.

I write on the document on the left, and through a shell script, the PDF on the right is generated. In summary, Pandoc takes the markdown file, images, and reference style file (e.g. Harvard, IEEE reference style) and creates a PDF file. I don't even have the PDF open until I'm proof reading and making sure the formatting looks good. When I focussing, I will also disable the sidebar. So this is what I see when I'm writing this blog post right now.

Challenges I faced & a guide for others

What referencing system do I use?

I use Zotero, including the browser connector, which allows me to add PDFs, websites and articles to my Zotero library with one click. After using lots of open source projects in the past few years, I've decided to use the open source instead of Mendeley, because they often work well with other open source software, for example, editors and Pandoc.

What referencing style do I use?

First I tried to find the citation style that the conference enforces. After a while of 🤔, I realised there was no standard, when I compared a few papers from the conference that I am reading papers from. I noticed IEEE and ACM was popular from StackOverflow and reading articles. ~~I went with IEEE because it puts quotation marks around the title of the paper, which I prefer.~~ I now use transactions-on-intelligent-systems-and-technology available on Zotero Styles.

How do I export Markdown to LaTex, including references?

This was annoying to setup because there's a learning curve, but now it works blissfully. This guide is only for 🍏 users (Windows/ Linux users need to find an alternative to Typora). In summary, you need Pandoc, Zotero, your Zotero library exported as a .bib file, the BetterBibTex extension for Zotero, and style file .csl.

Install Zotero (reference manager)
Install the Zotero connector extension in your browser, to add documents to your Zotero library easily, but also install Zotero things like style files
Create a folder where your report will live such as images and the markdown file
Export your Zotero library (remember the path/ directory to this, because you'll need it when generating the PDF with Pandoc, specifically the --bibliography). (File → Export → Better BibTex) your Zotero library somewhere, into the .bib file.
To get the citations formatted the way I wanted it, I had to install a new style I liked and that matches the conferences I am reading from to Zotero. I used Zotero Style Repository to find the correct style. You could try searching for IEEE or ACM, and either install it to Zotero or download the .csl file. If you installed it to Zotero, you need to export the style from Zotero into a folder somewhere, for Pandoc to use it.
Install the BetterBibTex extension for Zotero
Get the referencing style you want to be used on your document.
Install Pandoc
Install a markdown editor, I suggest Typora
Write your content!
- In your markdown file, all you need to do is use the Citation Key, which is provided in Zotero. Every time you want to add a reference in your document, just use [@citationKey], for example Kolotouros et al. mentioned blah blah blah [@kolotourosConvolutionMeshRegression2019]. You can shorten the citation key in the Zotero settings, and I have done that now, so it looks like [@kolotouros2019] instead.
Generate your PDF using this command. Remember to change the paths to match the files you exported/ downloaded previously.

pandoc --filter pandoc-citeproc "report.md" --bibliography /Users/user/ZoteroLibrary.bib --csl /Users/user/ieee.csl -o "report.pdf"
open -a preview "report.pdf" # This will open the PDF with macOS Preview everytime you run the script

Repeat the previous 2 steps as many times as you want. To have links in the PDF, to link from the reference in your text to the bibliography section, you should add a .YAML section to the top of your markdown file. So now when you click the reference it will jump to the references section. I also added the command I used to generate the PDF so I don't have to save it somewhere else.

link-citations: true
# More useful YAML attributes:
numbersections: true
secnumdepth: 2
number-sections: true
toc: true
lof: true
strip-comments: true
xnos-number-by-section: true
documentclass: report
---

How do I reference figure numbers in the text?

This is helpful if you don't want to manually update figure numbers them in caption and all the references in your text, every time a new image gets added before it. StackOverflow is your friend. This question/ answer shows you. However, you'll soon find out that figures are not numbered per section, and are just ascending from 1, 2, 3 and onwards. It would be nicer to have Figure 1.1.3 instead of Figure 24. To fix this, you can use a Pandoc extension called Pandoc Fignos, by Thomas Duck on GitHub. There is a similar Pandoc extension, called Pandoc Eqnos, by Thomas Duck on GitHub.

How do I actually customise the LaTex output?

Give Pandoc a custom theme (a .tex file), and it will generate the report using that theme instead of the default LaTex layout. This allows for separation of concerns (a software engineering term, aka. SoC): you can create your own custom looking LaTex files to follow a standard template someone has given you, but you don't have to even look this complex file when trying to work on the content of the report. Pandoc's documentation on theming will come in handy as you explore this topic. It can get complicated. Your LaTex file has access to all the YAML attributes at the top of the markdown file. For example, my theme uses a special front page, so I use a title: "This is the tile of my report" in my YAML, and the theme can these use this in the LaTex file, with $title$ .

For Imperial College computing, students are required to use LaTex and are recommended to use a college-provided theme for many of their projects. I have modified the theme Imperial provided us with to work with Pandoc, and it is available on GitHub. There is a starter report you can use, so just set up the environment following the guide above, and don't worry about LaTex or Overleaf.

Cool plugins for Typora

UPic is a plugin that allows you to upload images to the internet purely by dragging images into the Editor. This means you don't have to worry about file directories. For example, once you have an image you like, just drag it into Typora and it will be uploaded to your online account on Imgur. UPic requires configuration to know which account to upload, so follow the documentation for this. UPic allows you to upload to Imgur, AWS S3 buckets, and more.

Alfred: Alfred is a replacement for spotlight on macOS. Its a free software but the paid version has a lot more functionality. With the paid version, if I want to find a reference or a citation key in Zotero, zot ResearchPaper will do that that for me.

fin

I hope you've learnt something, and that your life is not consumed by LaTex, but by your research instead. I was worried about LaTex formatting a few months ago, but I'm happy to say I won't be worrying ever again.

Original photo by Neil Thomas on Unsplash.

Computer sciences courses I loved at Imperial College

Ben Butterworth — Wed, 20 May 2020 00:00:00 GMT

👋 Hello! This page might help you if you're deciding on doing a degree in computing, especially if you're considering Imperial College. You might be a student about to start a Computing Science Masters at Imperial, the same course I'm doing right now.

I'm documenting what I learnt and the people that taught/ challenged me, and also reflect on my decisions so far. I've just finished all exams for this degree, and so am concluding the 'taught' part of the course. Onwards, to the individual project!

Courses I loved

I reserve this spot for courses with both amazing lecturers & content. Its no surprise all 3 lecturers below have been at Imperial for many years.

C++ by Professor Will Knottenbelt & Dr Fidelis Perkonigg: these lecturers did not just teach C++, but also debugging, problem solving, communication, hard work and grit! The most valuable part of the degree. Dr Perkonigg is there with you in the computer labs, and students (including me) cannot get enough of him. Note: You get what you put in though. His guidance will be more valuable if you put more effort in programming yourself.
Operating Systems by Dr Anandha Gopalan: These lectures are interesting and Anandha puts them into context for beginners. Useful algorithms and concepts that show up here, repeatedly came up at developer events I attended and technical videos online: a few months ago, I would've closed that tab. Now, I smile and think of Anandha. Anandha also edited his lecture recordings to make them more convenient to students during revision. He knows the struggle.
Logic by Dr Fariba Sadri: This course made my brain think in weird & interesting ways. I intellectually enjoyed the course, and hope it will come in useful in the future. I've gained confidence in tackling weird looking things in the future. And... truth tables ARE used in industry! However, lectures aren't recorded for Logic. I'll never know what happened in that one lecture I missed. There are weekly sessions for practicing the logic questions. If you want to do Prolog, that can be done optionally.
Group project supervised by Anandha Gopalan: You can find more details about Foodprint here.

The other courses I did

These were all definitely useful & interesting to attend the lectures for. Some are required, and some are optional.

Machine Learning
Computer Vision
Object oriented programming (OOP)
Network & Web Security: I wish the tutorials weren't run in VMs.
Computer Architecture
Networks & Distributed systems (course is split in 2: Networks makes me 😻; Distributed systems part makes me 😕 because it is extremely outdated in comparison with the equivalent course at MIT).
🥺 Algorithms: way too theoretical, it did not include any popular algorithms, and did barely any implementation.

The other courses

There are more courses shown here. For me, the available optional courses were:

Systems Verification, Logic-Based Learning, Computer Vision, Graphics, Network and Web Security, Performance Engineering, Introduction to Machine Learning, Advanced Computer Graphics, Probabilistic Inference, Computer Networks and Distributed Systems, Introduction to Symbolic Artificial Intelligence (MSc), Software Engineering, Algorithms

Courses I wish I could do

There still is time. To the interwebs! I have the fundamentals now. I can learn anything with time & tea.

Rust
Concurrency programming
Distributed Algorithms
Performance engineering
More operating systems
Compilers

Early mistakes as a self-taught developer

Ben Butterworth — Sun, 19 Apr 2020 00:00:00 GMT

Read this if you want to know what mistakes I made, so you can decide for yourself if you want to make them too.

Someone told me earlier today to get a good job, you can't just apply (he meant, you need to network/ have existing connections). Then I thought, he's right: I've never gotten a job by applying, except the one at an Ice Rink. My first software job was when I was randomly called by a recruiter, and they pitched a job to me. I didn't even do Computer Science, and graduated from Engineering a few months before their call. I was spending my days, weeks and months coding, learning to become a software engineer! A few weeks later, I was a Backend Software Engineer at a really cool company. Don't worry, I didn't get fired, but I made some mistakes and this article helps me reflect on what I should do now & beyond.

In summary

Watching too many tutorials & not doing enough of my own project ideas
Learning too much & not coding enough
Irrelevant side projects and not focusing on work outside of work (maybe I should've chosen a job that tied well with my side projects)
Being distracted by everything interesting (might be a good thing?)

Watching tutorials.

Problem: I didn't know how important it is to just build your own project, instead of following tutorials. I confidently tell people now, tutorials suck my motivation. That is the best way I can put it. There is no reason to learn how to implement authentication if you don't even have a project that needs it. Over time, I've noticed other people find this too.

Solution: Never do online courses or watch tutorials just because they sound interesting. If you hear a new technology is interesting or even programming is fun. Do it to see for yourself. Immediately go to the 'Getting started section' of the technology or programming language. Don't watch someone else or follow along in their tutorial. Its either filled with stuff you don't know (overwhelming) or too easy (underwhelming). The more pain you're facing in your project, the more motivation you'll have to watch a few videos or read articles, and thats okay. I am problem & project driven. I don't know about you. PS: I still enjoy short videos like Fireship, but I know I don't get nearly as much out of it compared to when I design and code programs myself.

Learning too much. Coding too little.

Problem: The codebase was overwhelming: it was a huge monolith Java application which used the publish-subscribe pattern for different components of the application to communicate. The network of these messages and communication was drawn by plugin, and it literally looked like Spaghetti 🍝. Also, there was a lot of domain specific knowledge, and I knew none of it. I also didn't know Java that well, not like I know Javascript now ❤️. Luckily for me, there was a lot of material to read: years worth of cutting edge material, in slides & docs. So I started there, and never really finished 😅. Thinking back, there was a lot of opportunity to just code and break more things.

My fix:

I think technical debt is an important balance in software that gets used vs one that is too slow to get developed, so I shouldn't complain at problems if I see them, but instead try to solve them. At the same time, I am relatively new, and the responsibility for keeping the developer experience good should not be on me. So in cases like this, the most aggressive thing I can do is probably speak to others to see how we can work together to improve the developer experience.
I can't just decide to 'learn faster'. Learning takes time and effort, and spending time on learning takes away time from coding. Its a balance, and I probably overdid the learning part. If I'm met with a situation like this, I will try to write more code, and spend less time understanding the architecture/ system when its not important for the task. I should spend more time on the individual components I need. Easier said that done though, because adding a feature in one component often requires you to understand a whole list of other components in the application. My first task was to decouple one component from everything else. Looking back, it might have been a more optimal learning curve to have given a task in one section of the code.

Irrelevant side projects?

Problem: I love side projects because they teach you things a lot quicker than following tutorials. You can also get a feeling for the latest tech and libraries that are being written. Often though, it did not tie in with what I did in my software engineering job, and so it wasn't being augmented by my efforts outside of work. Because of this, I often thought to myself, the developer experience of so-and-so language was much better than the one we have here, and it sometimes got frustrating. Till now, Java has dissapointed me: there are way too many annotations, its slow & I found it generally tiring to work with. I feel like the learning curve for other languages are more interesting. However, I see Java still being used by one or two startups probably because of their employees' past-experience, although I don't see it very often at all.

My fix: In the future, I will find a role that makes me work on a technology I already know or like. When I first got that job, I didn't know many technologies, so I didn't even think about this. Focusing just on the technology the company had would have limited my understanding of computer science, especially as a 'self-taught/ non-computer-science graduate'. Now, I've spent some time in Python, C++ and Javascript (with React/ React Native), I've gotten a better sense of the ecosystems. But as I learn more, I realize its less about the programming language, but more about the problem you're solving. I'm still looking for companies that are solving interesting problems, like the environmental crisis, human crises and other societal problems. However, I am careful not to work on old technologies. Personally, I more inclined to get better at making apps individually than to practice cracking the coding interview.

Being distracted

Problem: So many things interest me: the CI/CD pipeline, the security vulnerabilities, and any of the applications running alongside the one I work on. I argue that spending time on these other applications give me context, and help me understand my application better.

Solution: This 'problem' only existed because I was relatively new: its a very interesting new world I've entered, and its natural to want to absorb all of it. There's a benefit to understanding more things, but focus is also important. As Xun Zi said, "The person attempting to travel two roads at once will get nowhere." Over time, my productivity would increase as the puzzle pieces together.

macOS CLIs: Building, Signing & Notarizing

Ben Butterworth — Sat, 27 Mar 2021 00:00:00 GMT

Once you've built your CLI tool and you're happy with the functionality, you want to share it with others. This is called distribution. You'll need the Apple Developer Program membership ($99) to perform "Software distribution outside the Mac App Store" as per Apple. This post goes over making a CLI tool in Xcode, including Building, Signing and Notarizing, on macOS Big Sur.

When users download your app from outside the App Store, macOS (GateKeeper) adds an attribute to the file (the Quarantine flag). When users try to launch the downloaded application (or cli tool), GateKeeper checks that the files meet certain requirements, and either allows the user to launch the app, or restrict the launch and shows the following UI instead:

When running in the terminal, you'll get:

> ./Example\ CLI\ Tool
zsh: permission denied: ./Example CLI Tool

That's not a great user experience.

Tip: Right click > Open to bypass GateKeeper app to launch the app anyway, unfortunately we can't expect users to do this, so thats why we notarize our app. You can check this quarantine flag in the terminal using the xattr (extended attribute CLI) tool:

> xattr -p com.apple.quarantine Example\ CLI\ Tool
0081;605f4bb1;Brave;39D0D1E9-5786-4D0E-9773-9EDB45F08C69

The above output states the flag is 0081, the date is 605f4bb1 which translates to GMT: Saturday, 27 March 2021 15:13:53 and is displayed on the UI alert, it was downloaded using the Brave browser and has a specific UUID. You can convert that Hex date into a human readable one using EpochConverter.

These requirements for a successful launch ~~are~~ seem to be:

Application installers (.pkg) and their binaries (inside the .pkg) need to be signed by the developer team to ensure they come from a specific developer team and not just created by some random person pretending to be. You sign things with certificates, not a pen.
Applications files (e.g. installers, apps) need to be notarized by Apple to ensure they are trusted. This process involves uploading it to the server through your developer account. The binaries (inside the .pkg) don't need to be.
Applications need to have the hardened capability (this is a flag set on the built executable), which limits the access of the executable, which guarantees that the application doesn't have access to certain things without permission. On Xcode, this is an opt-in feature (capability) as opposed to opt-out.
Info.plist must be present with the executable, either in a parent directory or embedded in the executable. Just having notarization and signed apps will not prevent your app from being quarantined by GateKeeper. If your installer places a folder in the users machine, then you can just put the Info.plist in that folder. However, in this project, we currently just place 1 file in the user machine, in /usr/local/bin so there is no place to put the Info.plist. We can embed Info.plist inside the executable instead, done through Xcode project settings.
- Note for future development: If going to make the installer install a directory on the users machine, you'll need to add a symbolic/ soft link in /usr/local/bin pointing to executables in your directory. Otherwise the user won't be able to use it from their terminal since the executables aren't in the path.

XCode vs. Swift Package ❌

There are 2 proper ways of creating a CLI tool for macOS. I prefer using a Xcode project because of the extra GUI features, all those Project settings tabs: General, Signing and Capabilities, Build Settings, Build Phases and more. You get No Editor in a Swift package in Xcode. Some trade-offs:

Xcode project. Xcode can be helpful in learning and understanding the steps in the entire process, as opposed to typing everything in the terminal. Unfortunately, an Xcode project doesn't build very well with xcodebuild clean build, so you'll need to use Xcode for building the application.
- xcodebuild doesn't build Xcode projects very well? What's it for then?? I don't know, I've tried to build a few applications using xcodebuild and either I don't know how to use it, or its an extremely neglected tool lacking in documentation.
Swift package: If we use the swift CLI to create a swift package. We won't do this, but these are the getting started steps:mkdir projectName, then cd projectName, then swift package init --type executable, then double click package.swift (Open it with Xcode). This is nice in that the folder is more organised, e.g. it has Sources/ and Tests/. I needed to restart Xcode because no files were showing up. This does work well with swift build, so building the package is easy on the mac terminal. Unfortunately, Xcode is much less helpful.

Create Xcode project 👷‍♀️

Use Xcode > File > New > Project... > macOS > Command Line Tool template to create an Xcode project.

Swift Tools Support

What's Swift Tools Support? In the readme, it says "Contains common infrastructural code for both SwiftPM and llbuild.", however I prefer Federico Zanetello's description of it:

these modules offer powerful abstractions for common [CLI related] operations.

Add Swift Tools Support

In the search bar (package repository URL), add the github URL: https://github.com/apple/swift-tools-support-core
In the next window, leave them to the default Rules/ settings, press Next
Pick SwiftToolsSupport-auto, this is important for the binary since we want a Library, not a non Dynamic library. This allows the binary to be standalone, without needing extra files.
- Optional: What's this dynamic and static linking?
  - If you look in SwiftToolsSupport's package.swift, you'll see these package products described. The difference between -auto and without, SwiftToolsSupport has type = .dynamic, which means this dependency will be dynamically linked. -auto means let the swift compiler decide what to do: it conveniently chooses the static linking, we want this. Find more here. You can change this later in the project settings.
  - ```
      .library(
          name: "SwiftToolsSupport",
          type: .dynamic,
          targets: ["TSCBasic", "TSCUtility"]),
      .library(
          name: "SwiftToolsSupport-auto",
        targets: ["TSCBasic", "TSCUtility"]),
```
- In static linking, all code, including shared libraries are bundled in your executable, when dynamically linked, they have to be added (linked) at runtime. I'll prefer static linking because I want my compiled CLI tool to be standalone, even though it might make the binary bigger.
- Stack Overflow question: Static linking vs dynamic linking
- And also try man dyld in the terminal to get the man page for the macOS dynamic linker.
```
/// A library's product can either be statically or dynamically linked. It
/// is recommended to not declare the type of library explicitly to let the
/// Swift Package Manager choose between static or dynamic linking depending
/// on the consumer of the package.

taken from https://docs.swift.org/package-manager/PackageDescription/PackageDescription.html
```
Now that you've added the repo, you can expand the dropdown menu in the file list (Project Navigator) and read the package.swift file for yourself. I recommend cross referencing with PackageDescription docs.

⌨️🎨 Write some code 🚀🌈

Federico's post has a great section called "Common Patterns" here, including exit codes, system modules, launch arguments, iterative scripts, environment variables, pipeline messages, async calls, input parsing, and progress animations. Please have a read there, to make your CLI tool conform to expected CLI practices.

Signing and Notarization ✍️

Set up your machine/ Xcode

Generate Developer ID certificates

You need the account holder of the Apple developer team to create 2 certificates for you, Developer ID Installer and Developer ID Application.

What is a Developer ID certificate?: They are certificates for distribution (not developers...) to identify themselves (sign their products) for delivery outside the App Store. More info here. This is distinct to Development Certificate. There are 2 types:
- You need the Developer ID Application certificate to sign the application/ executable
- You need the Developer ID Installer certificate to sign the installer
- One Apple account (e.g. An account which costs $100/year) can create a maximum of 5 Developer ID installer certificates and 5 Developer ID application certificates. Personally, I would have named them Distribution ID, not Developer ID. To summarize, all developers get a Development Certificate, but there's no way all developers get a Developer ID certificate.
Create the certificate, account holders usually already know how to do this on either Xcode Guide or developer.apple.com Guide

Add certificate to keychain

Double-click/ open the 2 files provided by the account holder. They've had to password protect the certificate when they generated it, so you'll need it to unlock this file. This password is not needed anymore once its in the keychain. You can delete the downloaded file they gave you too.

Configure the Xcode project for release

Embed Info.plist: You need Info.plist embedded in the executable. In the target or project Build Settings,
- Search for Info.plist, and you'll see Create Info.plist Section in Binary. Set this to Yes.
- Then, create an Info.plist somewhere in your project (I recommend the root of the project) which has at least 3 items, CFBundleIdentifier, CFBundleName, and CFBundleShortVersionString. I took this requirement from Howard Oakley's blog post:
```
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>CFBundleIdentifier</key>
	<string>com.example.example-cli-tool</string>
	<key>CFBundleName</key>
	<string>Example CLI Tool</string>
	<key>CFBundleShortVersionString</key>
	<string>1</string>
</dict>
</plist>
```
  - Howard called them Bundle Identifier, Bundle Name and Bundle version string (short). Those are the human readable strings which Xcode shows you when it sees the "raw strings" in the .plist file. Which is actually just an .xml file. Nothing special.
- Set Info.plist File to the path, for example $(PROJECT_DIR)/Info.plist if you placed this in the project root directory. Xcode will convert this into its absolute path in the UI, you don't write the absolute path yourself.
Signing configuration: In Signing and Capabilities
- Don't 'automatically manage signing'. Uncheck this.
- Set the correct team (your team)
- Update the correct bundle identifier
Hardened Runtime: In Signing & Capabilities, Click + Capability and select Hardened Runtime. Leave it to default settings.

Create an Archive

Click XCode menubar > Product > Archive. Wait for the archive to complete, and the Organizer will open. You can also open it with Xcode menubar > Window > Organizer > Archives
Click Distribute Content > Built Products > Export it as a build folder.

Quick check: Is your executable signed?

Make sure executable was signed: codesign -dv --verbose=4 "build/Products/usr/local/bin/Example Cli Tool"

Build package/ installer

pkgbuild --root build/Products \
           --identifier "com.example.example-cli-tool" \
           --version "1.0" \
           --install-location "/" \
           --sign "Developer ID Installer: Team Name (Team ID)" \
           "Example CLI Tool.pkg"

The --root folder needs to have the files arranged in the way you want it to be installed on your users device.
Notice the install location is /. This is because /usr/local/bin is already set as an Installation Directory in the Xcode project. If we did them in both places, you'll get /usr/local/bin/usr/local/bin/ROOT_DIRECTORY_CONTENTS.
The --sign value (i.e. "Developer ID Installer: Team Name (Team ID)") can be found in Keychain.app: but look specifically for the certificate with Installer, which you should've created (or the account holder gave you).
You can rust pre- and post- install scripts if you use --scripts build/Scripts. Read man pkgbuild for more details. This is not necessary for the most basic app, since we bundle all files needed into this one binary, and it sits inside /usr/local/bin, which is already on your macOS PATH.
Alternative: Build package/ installer: use Packages
- Using the Packages app, we can configure the destination of each file in our program through a GUI. We might want to put some files in /usr/local/bin, a directory in /usr/local, etc. It also has a CLI, but if we can get away with not using third party app, we should.
- Then sign the .pkg file: productsign --sign "Developer ID Installer: Team Name (Team ID)" "Example CLI Tool.pkg" "~/Desktop/Signed-Example CLI Tool.pkg"

Quick Tests

Make sure installer works: Lets see the installer does its job, before playing around with GateKeeper and Notarization. Run the .pkg file generated. You should be able to run `` from any directory. You'll need a fresh terminal (restart it).

Notarization hasn't completed: I've got three ways to check:

> spctl --assess -vvv --type install "Example CLI Tool.pkg"
Example CLI Tool.pkg: accepted
source=Notarized Developer ID
origin=Developer ID Installer: Team Name (Team ID)

> xcrun stapler validate "Example CLI Tool.pkg"
Processing: Example CLI Tool.pkg
Example CLI Tool.pkg does not have a ticket stapled to it.

My favorite: Drag the .pkg file into a new browser window, and re-download it. Now try to open it, it should say: “Example CLI Tool.pkg” can’t be opened because Apple cannot check it for malicious software. Even the Apple don't know about this, they suggest one of 2 ways to trigger quarantine:
- Email the app to yourself and use the copy that Mail downloads.
- Host the app on your own local or remote server and use the copy that Safari downloads.

Generate an app-specific password on Apple ID

To notarize the app, we could use, for example: xcrun altool --notarize-app ... --password "SENSITIVE_APPLE_ID_PASSWORD" ..., but this reveals the password. Instead, we can add the password to the Apple keychain and refer to it as xcrun altool --notarize-app ... --password "@keychain:ITEM_NAME" ....
Create an app specific password on https://appleid.apple.com/ > Security > APP-SPECIFIC PASSWORDS > Click Generate Password…. Give it a label that makes sense to you, e.g. MacBook Pro 16" or Mac M1 Mini. This name is just for labelling it in your Apple ID account. You might want to revoke in the future when you're not using that password anymore.
Open Keychain.app
Click File > New Password Item
- Keychain Item Name: Apple ID (or any name you prefer). You'll refer to this password item with e.g. @keychain:Apple ID.
- Account Name: $APPLE_ID_EMAIL, set this to your Apple ID email.
- Password: The app password generated from https://appleid.apple.com/ mentioned before.
Note: If editing an existing password item, its the Where field which matters, not the Name field.

Notarize package

To notarize the installer (.pkg). A previous blog post mentioned that you need a Developer ID Distribution signature but this is not necessary.

xcrun altool --notarize-app \
             --primary-bundle-id "com.example.example-cli-tool" \
             --username "apple_id_email@your_domain.com" \
             --password "@keychain:Apple ID" \
             --asc-provider "APPLE_TEAM_ID" \
             --file "Example CLI Tool.pkg"

Where do I get asc-provider value? It's the team ID you're in, you can find it in the developer.apple.com website, or on your certificate, or run the following command: xcrun altool --list-providers --password "@keychain:Apple ID" --username "apple_id_email@your_domain.com"
Check for completion: Get your request uuid from the output of the above command. You have to wait until the status message is: Status: success, you get Status message: Package Approved. You also get an email about the success. I am pretty sure you need to get the success response before you try doing the below stapling step.

REQUEST_ID=SET_YOUR_REQUEST_ID_HERE
xcrun altool --notarization-info "$REQUEST_ID" \
             --username "apple_id_email@your_domain.com" \
             --password "@keychain:Apple ID"

Staple the notarization ticket to the pkg file with xcrun stapler staple "Example CLI Tool.pkg"
Quick tests: You can verify Notarization ticket was stapled onto your installer or executable:
- xcrun stapler validate example_cli.pkg
- or spctl --assess -vvv --type install example_cli.pkg

Release

You can distribute this app through Github, your personal website or other places. Users will enjoy the lack of GateKeeper UIs.

Other resources

Have a read of these resources for more context:

Building and delivering command tools for Catalina was useful in that it mentioned Info.plist was a requirement, and demonstarted that CLI tools can be build through Xcode. However, it uses the Packages.app application instead of using the command line. My post will show both ways: using Xcode with First-party CLI tools, and also briefly on Packages.app
I found "notarize a command line tool" very useful, but unfortunately it creates a swift package instead of an Xcode project. Therefore, a lot of other Apple and Xcode guides are not relevant. You're in the dark if things go wrong, especially since swift package manager hasn't been around as long as Xcode. This article also goes to say certain things in Howard Oakley's post are not needed, but I disagree.

Questions

Feel free to comment on this page, I'll get a notification about it and do my best to reply. I found working through notarization has been challenging and interesting, though I am not sure everyone will enjoy it.

Using Cloudflare Products for 2 years

Ben Butterworth — Thu, 03 Jun 2021 00:00:00 GMT

Cloudflare provides a generous free tier for many of their useful products. I list them below (and will keep them updated as I use more). Some products do have some downsides during development, even when working at small-scale. I'm happy to say I overcame them, and want you to help you do the same.

⚠️ Warning ⚠️: This page also includes random technical details to help you solve issues. ⚠️

Cloudflare DNS

The DNS records propagate globally within seconds, so you won't second guess your DNS configuration.
The UI is much better than competing services for simple tasks like mine (e.g. AWS Route53, and other common registrars like GoDaddy, Namecheap).
- Though, the Cloudflare DNS UI will probably get quite hectic if I had any more DNS records.
- PS: I recommend nuttyabouthosting.co.uk if want a .uk or .co.uk domain), instead of GoDaddy. Great domain prices, and it just works.

Cloudflare Pages vs Netlify

Switching from Netlify to Cloudflare Pages brings 2 main benefits:

Cloudflare pages provides free web analytics, which allows you to see where your users are coming and if they are GoogleBot or not.
Better integration with other products. Its nice to reduce the number of companies I use (and will potentially pay for): I can use Cloudflare Workers, Cloudflare DNS/ Registrar, Cloudflare Workers, and I generally find more Cloudflare feature more valuable, so I'll be using more of them in the future.

Cloudflare Workers vs Firebase Cloud Functions (or AWS Lambda)

Cloudflare workers might seem like the normal serverless functions we've heard about, like Vercel, Google/ Firebase Cloud Functions, AWS Lambda and Azure Functions are selling.
However, Cloudflare Workers' runtime is more restrictive. This might mean Cloudflare Workers are much cheaper and faster to run at scale, but I won't dig into performance today.
- They are Javascript only, to run other languages you need to use WebAssembly. Other platforms allow you to run pretty much run anything: Google offers Cloud Run to run serverless containers, and Lambda allows custom runtimes to allow you to run anything you want.
- Cloudflare workers runs as a Service Worker (in a V8 isolate), not as a NodeJS application or a browser application. Some libraries don't support this very well, and so this limits the use of Cloudflare Workers. For example, XMLHttpRequest doesn't work. This is different to other serverless products, which run in NodeJS. The reason why Cloudflare uses Service Workers and not NodeJS is to run in V8 isolates for "scale, security and speed". See Kenton Varda's talk for more.
  - Service Workers are specialised typed of Web Workers, specifically designed to allow intercepting network requests to act as a cache/ proxy. This allows offline use. Web Workers only have a smaller set of APIs, which won't allow you to intercept network requests, and is more designed to run Javascript off the main thread.
  - Whats the difference between a Service Worker and a Web Worker?
    - Web Worker: General work done off the main thread.
    - Service Worker: A special purpose runtime which is a proxy between the browser and the server. Service Workers have extra APIs for intercepting network requests. Because Cloudflare Workers intercept network requests that are incoming to a domain, they kind of fit the service worker model.
    - Webpack uses the same target (webworker) for both of them
    - From client/ browser apps, you pass messages to it using window.postMessage()
Billing experience: Cloudflare Functions do not bill when the CPU is not running: i.e. It is free when you're making a network request in the cloudflare worker and waiting for the promise to return. This is different to Google Cloud/ Firebase pricing, which will definitely be more expensive:

Compute time is measured from the time your function receives a request to the time it completes, either through you signaling completion, or through a timeout, other failure or any other termination.
The debugging experience is relatively poor: you cannot set breakpoints, run it locally, and the error messages are not that helpful. I recommend cfworker to improve the normal Cloudflare debugging experience though. My example repo which uses Ably in a Cloudflare Worker, including helpful ways to debug (e.g. cfworker) exists here.
- e.g. Running wrangler dev might not give you any errors. You should first try to publish the worker using wrangler publish, fix the errors and confirm it works. Then wrangler dev will work.
- Cloudflare's free TLS/SSL certs only cover the apex (orth.uk) and 1 subdomain (club2d.orth.uk). So api.club2d.orth.uk doesn't get a valid SSL cert. Thanks The Freelancer ;) on Discord again. So I solved this by using the default worker domain, i.e. worker_name.your_worker_subdomain.workers.dev, which doesn't have this problem.
- Environment variables specified in wrangler.toml or the cloudflare website will be accessible as global variables, not through process.env.VAR_NAME, but just VAR_NAME. This is confusing, because if you're using typescript, you need to ignore the warning like so:

// @ts-ignore
console.warn(`api key is ${ABLY_PRIVATE_API_KEY}`);

You need a DNS record pointing to the cloudflare worker route. i.e. If you want your worker to work at call.orth.uk, you need to explicitly set call.orth.uk to have a DNS record with Cloudflare proxying it. Erisa on the Cloudflare Workers discord suggested setting a AAAA record to 100::, and that worked great. The Workers dashboard does not notify you of any of this, but the documentation does.
Conclusion: To speed up development I moved my project's serverless function (I only had 1 😅) to firebase functions, mainly because of the poor development experience/ difficult debugging. I did get around to fixing my Cloudflare Worker, so there is no fundamental flaw with Cloudflare.

Cloudflare Community

Join the cloudflare discord to get for support from really helpful Cloudflare people. The community is very helpful, and they've saved me a lot of time in debugging my Cloudflare Workers. However, with other platforms, I hardly needed to debug anything.

Question for you

Which Cloudflare products are you using or considering using? What other options are you looking at?

Anonymous video calls in React, using Ably, Mediapipe, Three.js and Next.js

Ben Butterworth — Tue, 08 Jun 2021 00:00:00 GMT

You can try it at Club2D.orth.uk, but for now, you'll need to use a Desktop browser running Chrome or Firefox. The repo is available here, which is a fork of my original repo available here.

Club2D is an anonymous video calling app inspired by the Memoji feature available on FaceTime, and allows emotions and face gestures to be shared with others without revealing someone's identity, background, disability or ethnicity. You should try building random projects which interest you because it will help you learn a lot faster than reading books or watching tutorials. Before this project, I did not know how to do 3D rendering (three.js), machine learning in the browser (MediaPipe/ TensorFlowJS), static site generation (NextJS), realtime/ event driven behaviour (Ably), nice CSS styling (TailwindCSS), and only knew a bit of ReactJS, CSS, Cloudflare and Firebase. I'm quite proud of what I made, and was delighted to see many my colleagues impressed by it as well. In doing this, I managed to ask and answer questions on StackOverflow, and am also proud to see my reputation reach 6.1k.

A short demo

<div style={{ textAlign: "center" }}> <iframe width="900" height="506" src="https://www.youtube.com/embed/QUqt0xbigOc" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen

</iframe> </div>

Benefits over conventional video calling

Protect identity: By using a virtual face instead of your real face, you can be anonymous in video calls
Avoid bias: Your disability, ethnicity. Many conferencing apps have added background-blur.
Comfort: not everyone is comfortable being in front of a camera. When in a video call, we don't know if someone has us zoomed in or is recording. It might prevent distracting other colleagues.

Technologies used

Ably Javascript/ Typescript SDK: Sending 3D faces between users https://ably.com/
MediaPipe: Getting 3D face mesh coordinates https://www.mediapipe.dev/
Three.js: Rendering the 3D faces and text https://threejs.org/
NextJS: Building a fast react app efficiently http://nextjs.org/
React: UI Rendering and state management https://reactjs.org/

What ML model does Apple Vision Framework use?

Ben Butterworth — Sat, 17 Apr 2021 00:00:00 GMT

Developers can perform computer vision tasks conveniently using the Vision framework, which is a convenient wrapper around coreML, coreML models and associated logic provided by Apple.

Optional extra reading: If you haven't used coreML before, have a go at the tutorials on the Apple documentation or build a computer vision project for yourself. This is a nice deep dive into coreML, which doesn't help you use it, but allows you to compare coreML with TensorFlow and TensorFlow lite, and see how models are stored. I recommend it to people who have used coreML a few times in the past.

Discovering where the files are

Skip to here to avoid the detail about discovering where the files are.
If you're curious about what syscalls there are on macOS: Run sudo dtrace -ln 'syscall:::entry' to see the system calls available on macOS, there seems to be 546.
To use dtrace, we need to enable it. Restart into recovery mode.
- Normal macs: Shutdown. Turn on, and hold ⌘ + R
- Hackintosh macs: Enter the boot picker, and select Recovery, usually 2nd or 3rd in the list.
Disable system integrity protection, and optionally re-enable it without dtrace. So either:
- csrutil disable, then csrutil enable --without dtrace
I ran a built-in script in one terminal window: sudo ./usr/bin/filebyproc.d, and then ran my CLI tool
In a separate terminal, run an application which uses Vision: e.g. sudo dtrace -c image_analysis. A load of paths will show up: search for relevant strings: scene if you're using labeling, face if you're using face detection, etc. Or just search for vision.
I found that /System/Library/Frameworks/Vision.framework/Resources was being accessed

Resources directory

Lets cd /System/Library/Frameworks/Vision.framework/Resources
This directory contains espresso models, which is a "A minimal high performance parallel neural network framework running on iOS".
There are csv, txt and plist files which describe labels, relationships/ mappings

berry -> fruit
bib -> clothing
bichon -> dog
bicycle -> vehicle
CVML_UNKNOWN_70 -> sign
billiards -> sport
CVML_UNKNOWN_239 -> games
binoculars -> eyeglasses
bird -> animal
birdhouse -> structure
birthday_cake -> holiday
camel -> ungulates
camera -> equipment
camping -> recreation
candle -> light
candlestick -> decoration
candy -> food
candy_cane -> holiday
candy_cane -> candy

Unimportant: To optimize for specific GPUs, there are Wisdom files Wisdom/$HARDWARE_NAME/wisdom.json which contain recommended parameters for specific layers in specific models.

Here are some takeaways:

I learn that Apple uses (a fork of) https://codinfox.github.io/espresso/ for their models. I guess Apple just took this CMU project internally, it hasn't been updated in 5 years.
The actual model files are split into 3 files, .net, .shape and .weight files, just like any coreML model. Except in this case, its not packaged into an .mlmodel file or .mlmodelc directory, and its missing its .bin file (metadata).
- - coremldata.bin: this appears to be the model’s metadata (name of author etc) and the classification labels
  - model.espresso.net: describes the structure of the model, i.e. which layers it uses and how they are connected to each other
  - model.espresso.shape: the output sizes of the layers in the neural network (the same thing you saw in the output of the build step above)
  - model.espresso.weights: the learned parameters of the model (this is usually a big file, 96MB for Inception-v3)
  - model/coremldata.bin: ?
  source
An optimisation: Also, for most files, they’re not using json, and just text files (you can append to it without parsing the entire json file, this would avoid keeping the results of the CLI in memory until the json is saved). Since a json needs to be parsed in its entirety. (and so do protobufs, which is why FlatBuffers were created for tflite).

The rest of the details are in coreML and not necessarily Vision, which is described in Matthijs Holleman's blog post about coreML here. I might edit this page with more details as I get learn more about Vision.

FlatBuffers for TensorFlow Lite, with a bonus about quantization

Ben Butterworth — Fri, 16 Apr 2021 00:00:00 GMT

TensorFlow models are saved in the protocol buffer format, which is an efficient way of storing and transporting data. I first heard and used it when I worked at Popsa where we used them to speed network requests between mobile apps and the backend. I then realised TensorFlow also uses this format to store models, including the weights and metadata. Popsa had a github repo specifically used to design and agree upon the interface between the backend and the mobile applications. gRPC also uses protocol buffers as the format. gRPC also uses protocol buffers as its format, by default. You can generate the classes to create, serialize and deserialize these objects in your preferred language with the protobuf compiler, protoc.

Instead of protocol buffers, TensorFlow Lite uses FlatBuffers. It has many of the benefits of protocol buffers, but without the cost of needing to deserialise the entire data file. In network requests, we don't pack that much information into a protocol buffer anyway, where a file might be 100KB. We can deserialise the entire data object very quickly. However, for ML model, we might store anywhere between 500KB to 1GB. For example, the CenterNet Object and Keypoints detection model with the Hourglass backbone is ~695MB. It might be important to get some information out of the model without loading the entire model in memory.

I wanted to read the tensorflow lite model (flatbuffers) in Python without using TensorFlow lite. There are easier ways to do this, but I wanted to use flatc!

Getting `flatc`

We had protoc for protobufs, but I couldn't find the FlatBuffer equivalent, flatc anywhere. Unfortunately, the FlatBuffers repo doesn't provide an obvious way of getting the flatc binary so you can generate your class files. You can either download it from a CI job, or build it yourself.

Download it from CI: Instead of building it, you can download it from most CI jobs' "Artifacts" section. For example, here, but this likely to be outdated for you.
Building it yourself:
- git clone git@github.com:google/flatbuffers.git
- cd flatbuffers
- cmake -G "Xcode" -DCMAKE_BUILD_TYPE=Release
- Open the project with Xcode (xed .) and build it, or just run xcodebuild clean build: build all the binaries. Find the flatc binary and use it, or
- Optional: Copy the generated binary into a directory which is already on path: cp ./Debug/flatc /usr/local/bin. This means you can use flatc anywhere

Compiling TensorFlow Lite schema in Python

Download the schema by visiting the following link, or running wget https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/schema/schema.fbs. I found it by just searching for it.
``flatc --python schema.fbs . Code is generated in a new directory:tflite/`
Create a python environment and install flatbuffers and numpy:
- python3 -m venv ./venv
- . ./venv/bin/activate
- pip install --upgrade flatbuffers numpy
Optional stretch goal for you: Can you modify the following script to

Using the library

import flatbuffers
# import numpy # Not used directly, but needed for AsNumpy function calls
from tflite.Model import Model

# Update the path to your model
model_path = "./mobilenet_v2_1.0_224_quantized_1_metadata_1.tflite"
with open(model_path, "rb") as model_file:
  buffer = model_file.read()

# Read output tensor
model = Model.GetRootAs(buffer)

print(dir(model))

print(model.Metadata(0).Name())

output_index = model.Subgraphs(0).Outputs(0)
output_tensor = model.Subgraphs(0).Tensors(output_index)
quantization = output_tensor.Quantization()

# read its quantizationParameters
print(f"Quantization: {quantization}")
q_min = quantization.MinAsNumpy()
q_max = quantization.MaxAsNumpy()
q_scale = quantization.ScaleAsNumpy()
q_zero_point = quantization.ZeroPointAsNumpy()
print(f"q_min: {q_min}")
print(f"q_max: {q_max}")
print(f"q_scale: {q_scale}")
print(f"q_zero_point: {q_zero_point}")

An easier path

There is an easier way to get the quantization parameters in Python. Just load the model, and read the input or output details. If quantization parameters are set, they will be in the dictionary.

So lets download a quantized model from tfhub.dev, called mobilenet_v1_0.25_128_quantized

Then lets prepare the python environment. TensorFlow now supports Python3.9, so this is easy.

First, create a virtual environment in a directory called venv: python3 -m venv ./venv
Load the environment: . ./venv/bin/activate
Install tensorflow: pip install tensorflow
run this script:

import tensorflow as tf

interpreter = tf.lite.Interpreter(model_path=model_path)
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

An even easier path

Open it in Netron, and the quantization parameters are right there:

The syntax is: min <= scale * (q - zero_point) <= max. I wrote more about the syntax on StackOverflow.

At least now, you know where they sit in the TensorFlow model file, which is actually flatbuffer file with a .tflite extension.

How to debug Android apps (including Flutter) without launching from the debugger?

Ben Butterworth — Sun, 12 Sep 2021 00:00:00 GMT

Motivation

You may want to observe/ debug behaviour related to your app's Activity, Broadcast Receiver or Service being launched by an Intent, without any side effects of the application already running. For example, I was testing push notifications, and wanted to debug the case where no other application components was running (Activity, Service), and only the Broadcast Receiver declared in my AndroidManifest.xml was launched in response to the push notification message. This is only possible if the application hasn't been launched yet by Android Studio.

The trick is prevent your app from launching until the debugger is attached, and then to attach the debugger. This means you won't miss anything between the app launching and you attaching the debugger.

Steps

Ensure your app is installed
Enable and configure Developer options. In Developer options,
- Enable USB debugging. This allows you t debug your apps, and to use adb to copy files, check processes running on your device, etc.
- Press Select debug app, and select your application.
- Then enable Wait for debugger
- You screen should look something like:
- Optional: enable Stay Awake. This is nice to avoid your Android from locking while you are trying to run and test apps on your device. This could get annoying though, for example when you say "Ok Google", the screen will turn off but won't turn off, so you should remember to turn this off at the end of the day.

Trigger your app to launch

If you are trying to test push notifications, send a push notification to your device. For example, I have an iOS application which can send a push notification to other devices. Your app should not actually launch, but it would be waiting for the debugger:

Flutter only

If you are working on Flutter, you should have 2 Android Studio windows open:

One window for the project (root project directory, project_dir)
One window for the android project (project_dir/android, or for flutter package plugins, project_dir/example/android)

At this point, you should press Flutter attach in your project_dir Android Studio window first. Nothing will happen until after you attach your Android app debugger, below. You should currently only see, for example "Waiting for a connection from Flutter on SM G991B...". SM G991B is my physical device, and Samsung S21.

Attach your debugger

Click the Attach debugger to Android Process, and in the window, select your application by its application ID. Your application should launch, and your debugger breakpoints will work. If running Flutter, your Flutter debugger in Android Studio will also

Tips/ warnings

🐛 If Wait for debugger is enabled, sometimes you need to detach or close Android Studio's debugger and re-attach it if you want to handle a subsequent application launch successfully. Otherwise, the app would never launch.

Other nice tools

scrcpy: A great tool to show your Android screen on your computer.
pidcat: A logcat which is easier to read and use, maybe.

A case against Symlinks in Xcode

Ben Butterworth — Mon, 21 Jun 2021 00:00:00 GMT

I've seen a few complaints 😈 related to symlinks in Xcode dotted around the internet, but nothing put things into perspective as much as me trying to use symlinks in Xcode, myself.

A small price to pay to avoid having to mess with symlinks which cause their own issues in Xcode _ - Joe Susnick_

The package no longer appears as a package within Xcode (icon is like a terminal-stamped file), and it's not built anymore. - Raphael Sebbe

I wondered what Joe meant 🤔...

this might be why:

You cannot save the file in Xcode when you opened the file via its symlink. Developers might open the file in Xcode by clicking the symlink, since it appears to be a normal file. It's possible to open the file, edit it, but not to save it, you'll get the popup shown below. Once this happens, you also cannot edit the original file (via its original hardlink), because Xcode is storing the modified version of the symlink file. To resolve this, you need to force quit Xcode. It won't close normal (pressing the red/ close menubar item won't work).

Annoying popups: Upon every rebuild after trying to save the file, you'll get this popups twice, one after the other.
**Xcode incremental builds fail: **Xcode doesn't know if the file (the one that the symlink points to) is changed. So the build won't work properly, because it won't recompile that file. It fools Xcode build system. Even after cleaning the project, I can't get it to recompile (even after restarting Xcode). The solution is to delete the symlink from not just the Xcode project, but the actual symlink, and re-create it. So when a user modified the header (even via the original file), the target which relies on the symlink version of the file will not notice, and so this file won't get recompiled. You have to clean the project/ build files Product > Clean.
Mental overhead: An Xcode project already has its own meaning of files: instead of directly using the directory structure, it has references, so you can put your files in any folder structure in Xcode. It's only visible inside Xcode. Putting another layer of indirection via symlinks adds another layer of indirection that developers have to think about. Keep your project structure simple.

TLDR: Xcode gets very confused by symlinks, and if you're not convinced, I suggest you play around with symlinks in Xcode. For those who understand how to use Xcode with symlinks, its a pain to clean and rebuild (losing incremental builds) and a small bit of mental overhead to ensure you check you're editing a hardlink. For someone joining your project, its a booby trap. May Tim Cook help you 😅.

Why I was considering Symlinks

I wanted to share code/ files between modules (targets) in a Swift Package, a question I posted on the Swift forums, and Swift doesn't compile if it detects 2 files being shared across 2 targets. Fortunately:

I think you can trick SwiftPM using symlinks. - Jeremy David Giesbrechts

Though I was able to use symlinks in Xcode, Xcode did not treat me very well, with the problems I listed in this page. The other alternative is to save the shared file in module, and share that module between the modules that want to use it.

Another reason you might use Symlinks

You want to move your header files into the include/ in preparation for Swift Package Manager support for your Objective-C library. I recommend moving the files into the include/ directory, instead of creating symlinks, to avoid the issues listed above. If you're looking for a great Swift Package to model your project structure on, look at SGDCornerstone.

Why you shouldn't use hardlinks

hard links cannot be represented in git - Stack Overflow's Jakub Narębski

That should be enough. 😅

PS: I used Xcode 13 beta.

The issue with standardisation

Ben Butterworth — Wed, 15 Sep 2021 00:00:00 GMT

I work in a team that manages far more software libraries than there are people. I wanted to raise my concern about standardisation across projects/ libraries (e.g. readmes, tools, github workflow file naming conventions). For example, synchronizing all the projects to have:

the same README structure/ template.
using ASDF, a new runtime manager tool for as many libraries as possible, instead of using rbenv, nvm.
Calling any CI workflow check.yaml if they are related to tests, static analysis, unit tests or integrations tests.
Sharing the same features between all libraries.
Sharing the same API/ interface between all libraries.
Sharing an implementation guide/ document, where all developers are required to implement features in the exact same way, including testing their applications in the same way.

Benefits/ costs

What are the benefits (and costs) of doing these things? An engineer did create these, so there's is actually a benefit to considering them. Generally, it avoids "duplicate" thinking (dummy-proofing).

The README will be more familiar to people (developers, customers) as they move between projects.
- Cost: No README is perfect, and structure is part of that. Developers should feel free to structure the README. Additionally, if a documentation site is created for one library in particular, we might nuke the README completely. A developer will second guess adding a section that they believe is important. If they want to improve the structure by removing a useless section or make structural changes to the README, they might choose to seek permission from the standard setter, or I predict, make no improvements to the docs (accept the fate, and give up).
Using asdf: Developers can easily drop in and out of SDKs (mercenary style) and run the runtimes.
- Cost: Of course, using an uncommon lowest common denominator tool might bring use outdated runtimes/ have bugs/ worse developer experience. Node version manager, and Ruby are tools designed specifically for that runtime, are already established.
Calling any CI workflow file: check.yaml:
- Cost: Each workflow should have a meaningful name. If there are 2 workflows, one for unit tests, and the other for integration tests, it would ideally be clear from the filename itself.

It can be detrimental and restrict the improvement/ growth than each repo/ project can make, compared to if repos didn’t have this standardisation cost. Developers have to “argue” their case for breaking the standard, when the standard was never justified very well or discussed. IMHO there are much more important things to think about.

Once the decision is made to create a standard, there might still be a cost to design a standard that is suitable. For example, if creating a consistent README structure, work has to be done to create a suitable README that fits all libraries. Problem alert?: This structure might be the lowest common denominator or contain unnecessary sections.

When a standard is put in place, you'll find lots of things that don't conform to the standard. I've noticed my team lead who has many years of development experience working on this type of non-technical work. Apart from this synchronising cost between libraries when the standard is first enacted/ discovered, I think there is a hidden, but much greater cost of maintaining this synchronicity. Developers are restricted in choosing their tools or an approach in their project, which makes the process dummy-proof. Reed Hasting's (Netflix founder) says "if you dummy-proof the process, you only get dummies to work there". Importantly, when a new tool or technology comes out, we’d be stuck with a standard, and the onus is on the developer to argue the case. A huge amount of work would be on us to decide to upgrade the standard, and make a new standards leap (upgrade to the new tool). This is unnecessary friction, not all developers voice their concerns like I do, and defaults are powerful.

I believe avoiding this type of cost is actually in some ways a Netflix principle. Read no rules rules, by Reed Hastings :wink:

Solution

So I say:

Bad standards are easy to add, but expensive to maintain. Good standards are hard to add, but saves money.

This is part of my Engineering principle to just keep things simple. Synchronisation between projects is not simple.

For every instance of standardisation, there should be a clear argument for (benefits vs. costs) and a discussion about it, before it gets made a standard. And along this standard document, a justification should live, where future developers can question and remove the justifications if they are no longer necessary. Make it harder for the dummy-proof process designers to create their processes, so developers can keep our autonomy and productivity.

What is the difference between a clang (C++, Objective-C, C) module and a Swift module?

Ben Butterworth — Tue, 21 Sep 2021 00:00:00 GMT

They are different. At the end of the build process though, they both need to be linked to your application/ library's other .o and .dylib files for it to run.

Swift modules

From Swift Serialization.md docs:

The fundamental unit of distribution for Swift code is a module. A module contains declarations as an interface for clients to write code against.
Swift acccess control docs:

A module is a single unit of code distribution: a framework or application that’s built and shipped as a single unit and that can be imported by another module with Swift’s import keyword.
Configured by .target()'s in Package.swift
Cannot have submodules, so users cannot import Module.Submodule in Swift. Users can still import specific entities, import struct PackageModel.Manifest, but this is a lot more verbose than importing submodules.
Its interface exists as a .swiftmodule. What is a .swiftmodule?. The documentation says:

Conceptually, the file containing the interface for a module serves much the same purpose as the collection of C header files for a particular library.
The compiler produces this .swiftmodule file a lot, like a generated objective-C header, but instead of text, its a binary repesentation. It includes the bodies of inlinable functions, much like static inline functions in objective-C or header implementations in C++. However, Swift modules does include the names and types of private declarations. This allows you to refer to them in the debugger, but it does mean you shouldn't name a private variable after your deepest darkest secret. from WWDC 2018: Behind the Scenes of the Xcode Build Process
- So private declarations are exposed in your .swiftmodule (Swift module interface).
When importing pure Objective-C frameworks into Swift, the Swift compiler uses its built-in clang compiler to import an Objective-C header.

The importer finds declarations in the headers exposed in Clangs .modulemap for that framework. (again, from WWDC2018)
When importing Objective-C + Swift frameworks into Swift, the Swift compiler uses the Umbrella header.

Clang modules

Configured by YourModuleName.modulemap file (previously module.map, but this is deprecated), formatted like this
Can have submodules, e.g. std module has std.io and std.complex.
A clang module exposes header files specified in the module map. Private details (in .m) are not exposed at all.
Is an improvement of the original #include or #import style imports to improve the build process (This is a big topic, read the Clang module docs).

I originally posted this question and answer on Stack Overflow here. I had to do some digging and reading for this one.

Debugging Push Notifications on iOS

Ben Butterworth — Wed, 22 Sep 2021 00:00:00 GMT

The real value/uniqueness on this page is showing how to use Console.app for push notifications. I also include some more basic guidance to help you avoid some common problems.

Use the debugger

Launch the application from Xcode (run, but ensure your scheme uses the Debug build configuration, not release or profile) or AppCode (debug mode). Place breakpoints in methods that should be called to confirm that your methods are (not) being called. Some useful methods:

Alert notification:
- Note: If you send a message without data, you won't be able to handle the notification in didReceiveRemoteNotification below. Instead, you can handle notification taps by the user, and decide to even show it if the app is in the foreground.
- Notification tap: didReceiveNotificationResponse:completionHandler:
- Foreground alert notifications: userNotificationCenter(_:willPresent:withCompletionHandler:)
Background notifications: application(_:didReceiveRemoteNotification:fetchCompletionHandler:)
- Don't use application(_:didReceiveRemoteNotification:), the deprecated version.

Request authorization from the user

If showing an alert notification to the user, be sure to requestAuthorization to show notifications from the user. This is not required if you just want background messages, handling messages in didReceive(_:completionHandler:). It's really easy to forget this.

Try sending it directly to APNs

You should do this when first debugging. This allows you to test push notifications without having to structure a message correctly, since this file does it for you. This also reduces the complexity/ layers in the problem when first debugging, since 3rd party services (Ably's Channels or Firebase Cloud Messaging). If the device is registered correctly, it should receive this message. Make sure to update the TEAMID, KEYID, SECRET, BUNDLEID and DEVICETOKEN. My colleague shared this script with me:

Usage: Get direct_apns.sh from this GitHub Gist. Run chmod +x direct_apns.sh and execute it: ./direct_apns.sh.

If the script failed with an error, check the error which is returned by APNs.
If the script was successful but it doesn't appear to arrive on the device, time to read the logs. (See below)
If this arrives on the device, but your own messages don't, this means you are not structuring your message correctly. For example, I've seen users who set apns-headers in Firebase at the wrong level of the JSON being sent to Firebase.

Reading device logs

Open Console.app (this is different to Terminal.app or iTerm2.app) installed on your mac. To confirm your application received the push message/ check for errors related to push notifications:

Be sure to start logging for the correct dvice. Then find relevant logs by:

search for the following log messages:
- Both failures and success: com.apple.pushLaunch
- Failures only: CANCELED: com.apple.pushLaunch. For example, this may show the log line: CANCELED: com.apple.pushLaunch.com.example.app:DBA43D at priority 10
- Success only: COMPLETED com.apple.pushLaunch. For example, this may show the log line COMPLETED com.apple.pushLaunch.package_name:XXXXXX at priority 5
filter for dasd process either by right clicking a log line with dasd and click Show Process 'dasd'.

If you are sending a background notification, it may be throttled by iOS. In this case, an error will be shown in Console.app, but will eventually arrive to your application, where didReceiveRemoteNotification delegate method will be called, often within a few minutes. If you look in the Console.app logs, you may find sending the exact same message gives different outcomes:

ThunderingHerdPolicy error (What really is the ThunderingHerdPolicy?):

{name: ThunderingHerdPolicy, policyWeight: 1.000, response: {Decision: Must Not Proceed, Score: 0.00, Rationale: [{deviceInUse == 1 AND timeSinceThunderingHerdTriggerEvent < 900}]}}
  ], FinalDecision: Must Not Proceed}

cameraIsActive error:

com.apple.pushLaunch.io.ably.flutter.plugin-example:4935F4:[
{name: MemoryPressurePolicy, policyWeight: 5.000, response: {Decision: Must Not Proceed, Score: 0.00, Rationale: [{cameraIsActive == 1}]}}
], FinalDecision: Must Not Proceed}

A successful delivery:

{name: ApplicationPolicy, policyWeight: 50.000, response: {Decision: Absolutely Must Proceed, Score: 1.00, Rationale: [{[appIsForeground]: Required:1.00, Observed:1.00},]}}
  ], FinalDecision: Absolutely Must Proceed}

Another successful delivery:

com.apple.pushLaunch.io.ably.flutter.plugin-example:5E1C66:[
{name: DeviceActivityPolicy, policyWeight: 5.000, response: {Decision: Can Proceed, Score: 0.25}}
] sumScores:93.270000, denominator:97.020000, FinalDecision: Can Proceed FinalScore: 0.961348}

I initially wrote this to add Push Notifications support to a Flutter library. There might be more useful content there.

A note about environments (APNs sandbox vs production)

There are 2 environments, sandbox and production.

Sandbox: When running your application via Xcode or your machine (Android Studio, command line), your application runs in either debug, profile or release mode. In all cases, your application will use the sandbox/development APNs environment. Also, If distributing through Development, the sandbox / development APNs environment is used.
Production: When distributing your app in the App Store, Ad Hoc or through App Store Connect, it will always use the production environment.

How/when Flutter runs on iOS and Android

Ben Butterworth — Thu, 23 Sep 2021 00:00:00 GMT

Being cross-platform, Flutter abstracts away Platform APIs, such as iOS, Android, Web, macOS, Windows and Linux. However, there are platform specific differences which will affect how you implement features or develop package plugins.

For example, on iOS, you need to implement didReceive(_:completionHandler:) in your app's main entrypoint (AppDelegate), where as on Android, you need to declare a Service or Broadcast Receiver in the AndroidManifest.xml file, and override FirebaseMessagingService onMessageReceived method.

Flutter always runs on iOS, but not on Android

iOS

On iOS, your Flutter application always runs if a push notification (or background process) is running. This is because the FlutterViewController is initialized when the application launches, and this creates a FlutterEngine.

The FlutterViewController is declared in the Main.storyboard file, so technically Flutter apps are Storyboard apps (very basic ones) 🤓. A Flutter Engine is created in the Objective-C++ file, FlutterViewController.mm:

auto engine = fml::scoped_nsobject<FlutterEngine>{[[FlutterEngine alloc]
              initWithName:@"io.flutter"
                    project:project
    allowHeadlessExecution:self.engineAllowHeadlessExecution
        restorationEnabled:[self restorationIdentifier] != nil]};

Note: this isn't Objective-C, it's Objective-C++. It has both verbose allocation/initiazation syntax and also uses C++ style class: ``.

template <typename NST>
class scoped_nsobject : public scoped_nsprotocol<NST*> {
 public:
  explicit scoped_nsobject(NST* object = nil) : scoped_nsprotocol<NST*>(object) {}

  scoped_nsobject(const scoped_nsobject<NST>& that) : scoped_nsprotocol<NST*>(that) {}

  template <typename NSU>
  scoped_nsobject(const scoped_nsobject<NSU>& that) : scoped_nsprotocol<NST*>(that) {}

  scoped_nsobject& operator=(const scoped_nsobject<NST>& that) {
    scoped_nsprotocol<NST*>::operator=(that);
    return *this;
  }
};

Android

On Android, only the component (e.g. Activity, Broadcast Receiver, Service) you declared runs, and the Flutter application doesn't necessarily run. The FlutterEngine is only automatically run if a FlutterActivity or FlutterFragment is used. For example, in FlutterActivity, it creates a FlutterEngine in the onCreate method. This means on Android, we need to launch some kind of dart/ flutter code in the component if the FlutterEngine doesn't yet exist. There are 2 options:

Define. This was described in Mahdi-Malv's answer on Stack Overflow. This is also the approach of firebase_messaging, a Flutter library for push notifications. Take a look at those if you want to understand the approach. Or, you could:
Launch the original Flutter application. This is what I chose to do. You can do this in relatively few lines:

flutterEngine = new FlutterEngine(context, null);
final MethodChannel methodChannel = new MethodChannel(executor.getBinaryMessenger(), methodChannelName, new StandardMethodCodec(new AblyMessageCodec()));
methodChannel.setMethodCallHandler(this);
// Get and launch the users app isolate manually:
flutterEngine.getDartExecutor().executeDartEntrypoint(DartExecutor.DartEntrypoint.createDefault());
// Even though lifecycle parameter is @NonNull, the implementation `FlutterEngineConnectionRegistry`
// does not use it, because it was only meant to be exposed for testing framework. See https://github.com/flutter/flutter/issues/90316
flutterEngine.getBroadcastReceiverControlSurface().attachToBroadcastReceiver(receiver, null);

Once you do this, you'll need the Flutter application to inform the Android side that the dart side is ready to receive data being sent the Android side. This is because there is no way for the Android side to determine when the FlutterEngine/Isolate/Flutter app is ready.

I'll stop there. Try to understand the following things by reading code, preferably cloning the repo and navigating the code using an IDE (Android Studio, Xcode and AppCode, rather than VSCode for this kind of code):

Understand implementation details for launching the Flutter application: Invoking methods between Dart and Platform side, and passing data. Drawing a diagram on paper would help. You can start with Firebase Messaging's FlutterFirebaseMessagingBackgroundExecutor.java. Then you could take a look at my simpler implementation in ably_flutter.
How FlutterEngine works internally with the Platform shell: FlutterViewController, FlutterJNI, etc. You should clone flutter/engine.
- On Android, start with: FlutterActivity and then FlutterActivityAndFragmentDelegate.
- On iOS, start with FlutterViewController.mm.

Feel free to ask questions in the comments below :)

SSH into your private machines from anywhere, for free, using Cloudflare Tunnel

Ben Butterworth — Thu, 10 Feb 2022 00:00:00 GMT

Outcome

By the end of this post, you'll be able to run: ssh $machine_name from anywhere in the internet-connected planet, using SSH keys. It is free and requires no future maintainance. This guide uses Cloudflare Tunnel, a service by Cloudflare with a free-tier. It will filter traffic to your machines through Cloudflare's network, including authenticating you. Because of this, your machines won't directly be exposed to threat actors and "1337 haxors".

This was discussed on Hacker News.

Motivation

You might have a machine running in your local network and want to access from anywhere in the world. Alternative solutions include:

configuring your router to port forward to your specific machine or setting your machine as a demilitarized zone (DMZ). Disadvantages: your home router's IP address might change, so you'll need to use the new IP address. Each port is also limited to a single machine, so you'd have to choose a different port for a different machine. Your DNS record also publishes the IP address of your router, which can be used to locate you - somewhat unreliably, depending on who you are.
Many alternatives are listed in anderspitman/awesome-tunnelling, including Cloudflare Tunnel. Many of these require configuration, and the last time I tried ngrok, I couldn't choose the domain name on the free tier.

I already use Cloudflare for this website (Cloudflare Pages), web analytics, configuring DNS records, and registering my domains (with no additional fee on top of the Administrator fee, e.g. Verisign for .com and Nominet for .uk). Hey UK domain owners, Cloudflare Registrar supports uk and co.uk TLDs now 😉.

So I used cloudflared to do this, but realised there was no guide written that takes me step by step, to success. There was a Cloudflare official guide, but some of the steps are not necessary, confusing (e.g. limiting the "application" to 1 month) and missing the config.yml schema.

Pre-requisites 🦑

A free Cloudflare account
Understand Cloudflare Tunnel, by following Set up your first tunnel. You'll end up with a website added to Cloudflare, cloudflared installed and logged in on your machine, and a high level understanding of Cloudflare Tunnel. You don't have to do step 4 (create configuration file) or later.

Setup SSH server on your target machine 🎯

Skip if you've done this already, for example, you can already SSH into your machine.
For Ubuntu/Debian,
- Install service: run sudo apt install openssh-server
- Start service: run sudo systemctl start ssh
- Schedule service (to start on machine start-up): sudo systemctl enable ssh
For other operating systems, try using search engines.

Quick check: setup a tunnel and SSH 🧪

This step is just a quick check that you can SSH into the machine, before you waste time configuring stuff.
Start a cloudflare tunnel: run cloudflared tunnel --hostname machine.example.com --url ssh://localhost:22
- Reminder: edit example.com to one you have added on Cloudflare, and update machine to whatever you prefer. cloudflared will create a CNAME record called machine.example.com.
SSH into your machine using password based authentication: run ssh username@machine.example.com
- Reminder: Update username and host.

Configure the `cloudflared` service 🧑‍💻

We want this service to be running all the time (daemon), not just when I run the command. Also, some of the steps in Cloudflare: Run as a service are broken.
Create a tunnel: cloudflared tunnel create $tunnel_name, where $tunnel_name is a name you can use to reference it.
Update a DNS record: cloudflared tunnel route dns $tunnel_id phanteks.orth.uk
Create a config file: ~/.cloudflare/config.yml, containing:

tunnel: $tunnel_id
credentials-file: /home/$user/.cloudflared/$tunnel_id.json
url: ssh://localhost:22

Run the tunnel: cloudflared tunnel run --url=ssh://localhost:22 $tunnel_name
Install service: sudo cloudflared --config /home/ben/.cloudflared/config.yml service install
Start service: run sudo systemctl start cloudflared
Schedule service (to start on machine start-up): sudo systemctl enable cloudflared

Configure SSH client 💻

Create a keypair. The private key is for your client and the public key should be given to anyone who wants to authenticate you (the machine you want to SSH to, Phanteks). My keypair is called phanteks(private key) and phanteks.pub (public key):
- Run ssh-keygen. Before that, preferably cd ~/.ssh, so that your generated keys are stored there.
Copy over the public SSH key ($KEYNAME.pub): Run ssh-copy-id -i ~/.ssh/KEYNAME.pub $machine_domain
Add the following to ~/.ssh/config. Create the file if it doesn't exist.

# This only works when you are on the same local network
Host phanteksLocal
        HostName Phanteks.local
        IdentityFile ~/.ssh/phanteks
        User ben
        Port 22

# Cloudflare Tunnel
Host phanteks
        HostName phanteks.orth.uk
        IdentityFile ~/.ssh/phanteks
        ProxyCommand cloudflared access ssh --hostname %h
        User ben

Secure SSH server 🚔

Optional: Change the port used by SSH, since the default port is well known (22). You'll also have to update ~/.ssh/config and ~/.cloudflared/config.yml with this new port.
- Warning ‼️: cloudflared actually duplicates this file into /etc/cloudflared/config.yml first when you install the service. If you change ~/.cloudflared/config.yml, be sure to:
  - manually update the file, or
  - delete the file (run sudo rm /etc/cloudflared/config.yml) and reinstall the service (run sudo cloudflared --config /home/$USER/.cloudflared/config.yml service install).
Optional: Disable password authentication
- This is usually done to improve security: mitigate random password attacks, but this is already mitigated with Cloudflare Tunnels.
- But in the spirit of "defense in depth", we should still disable it. We should assume attackers might still come from within your local network, including successfully authenticating with Cloudflare Tunnel
Restart the services: run sudo systemctl restart ssh cloudflared

Usage: connect to machine 🤞

Run ssh phanteks

Conclusion 💌

Maybe there is a way to automate this so all machines I use in the future will be set up automatically. Thankfully, I am not building a server farm at home or moving homes very often, so this is a very low priority. I was just taking a quick look at Ansible yesterday.
Question: do you use a different tool which require no maintenance or cost to run?

Playing Android device audio on macOS

Ben Butterworth — Sat, 11 Dec 2021 00:00:00 GMT

Why?

Update: scrcpy 2.0+ supports Audio forwarding. You don't need sndcpy anymore. Thanks Romain Vimont.

I want to listen to the instructor playing on an Android app (Fiit) on my speakers which are connected to my MacBook. I am already sharing the screen of my Android on my MacBook by using scrcpy, a tool by Romain Vimont. sndcpy lets us to do this, also a tool made by Romain Vimont. The naming of these tools were inspired by C library function, strcpy, which copies strings between memory locations.

You want to play music or app sounds on your Android device on your macOS, and

You are displaying your Android screen on the mac already, but you want sound too
Your Android device doesn't have an audio jack
Your speakers aren't bluetooth
You don't want to use your headphones
- I do not want to sweat onto my headphones, or
- I want to share my music with others
Your macOS is connected via AUX cable (or USB-C/HDMI) to speakers, so let's use them instead

Pre-requisites

Command line experience, or just being resourceful.
adb installed and usable. To install, run brew install android-platform-tools
VLC installed in your /Applications folder: https://www.videolan.org/vlc/
Open Terminal.app (already installed at /Applications/Utilities.Terminal.app) or iterm2

Installation steps

Download the sndcpy APK and script from rom1v/sndcpy on GitHub, and place them in /usr/local/bin
- Unzip the file downloaded from Github and move them using
  - sudo mv ~/Downloads/sndcpy/sndcpy /usr/local/bin, and
  - sudo mv ~/Downloads/sndcpy/sndcpy.apk /usr/local/bin
- This APK is installed to your android device whenever you run sndcpy
Create a script called vlc in /usr/local/bin by running sudo vim /usr/local/bin/vlc, and passing the following content:
- ```
/Applications/VLC.app/Contents/MacOS/VLC -I rc "$@"
```
- The reason we put it in /usr/local/bin is that your mac will instantly be able to use it like a command, by calling vlc (once you restart the shell 🐚).
- Optional: Confirm the file contains those contents, by running cat /usr/local/bin/vlc
Make the file executable to you and the group: sudo chmod 755 /usr/local/bin/vlc
Add the following to your ~/.zshrc so that sndcpy can get the APK whenever it wants to install it onto your device:
- ```
export SNDCPY_APK=/usr/local/bin/sndcpy.apk
```
Restart your shell, or reload that file: source ~/.zshrc

Usage steps

Connect android device to computer (or attach it remotely via adb)
Run sndcpy in terminal

Syncing Obsidian Vault on macOS and Android, for free, using git

Ben Butterworth — Mon, 18 Apr 2022 00:00:00 GMT

Motivation 💪

You have been using Obsidian to store your notes. I highly recommend it if you don't - in that case, come back to this page later.
Your current solution to write notes is to use a separate app or separate Obsidian Vault on your phone. After you reach your computer, you manually copy and paste data into your primary Obsidian Vault. I previously used Google Keep for temporary notes, which were moved into Obsidian Vault when I found the time.
Update: I no longer use git to sync my Obsidian Vault. This is because manually resolving conflicts between my notes and .obsidian files using git and Termux is tedious on a small screen and a touch keyboard. Now, I pay for Obsidian Sync. I am planning on trying Syncthing soon. I also now trust GitHub (Microsoft) less with my private notes. See Copilot regurgitating Quake code, including sweary comments for some discussion.

Warning 🌶

This guide requires the use of command line and git, and storing your data privately online, like on GitHub. Therefore, it is not end-to-end encrypted, and Github developers could have access to your data.
If you are unwilling, there are other ways to sync Obsidian Vaults, e.g. using Google Drive, Dropbox or paying for the Sync Service from Obsidian directly.
If you want end-to-end encryption for your files, use the Obsidian Sync Add-On Service (currently $8/mo).

Brief notes about my note taking 📝

Markdown is an open file format to store your information, and it was created by Aaron Swartz and John Gruber.
- It's readable even in its raw format (text) without an app, eg (MS Word), but you can make it look even better with apps like Typora and Obsidian.
- Open file format: This means if the app you're using shuts down, changes it's pricing or is unethical, you can take your data to another app without much cost.
I started using Typora when it was free, and it was great. As they previously warned, it became paid software, and that is fair - it's a great app.
I found Obsidian on a variety of websites and from a colleague. Over time, I realised it would organise all the notes I made in the past, and make them searchable.
Eventually, I abandoned Typora to use Obsidian completely since I could do everything in Obsidian.

Example 😃

Here's a picture of a file from my Obsidian Vault I wrote on macOS, but it's viewable and editable from Android. My Android device screen is captured using scrcpy

Steps 🗺

Put your notes in a `git` repo hosted online. 🌎

Basically, use Github, Gitlab or other cloud provider which hosts git repositories.
You probably want a private repo so that other people can't read your files.
Github has a guide for this: Create a repo

Clone the repo on your android phone 📲

Install GitJournal from the Play Store. GitJournal is an app that allows cloning repos, and syncing files back to the cloud.
Configure GitJournal to store repo files in the local directory, not in the internal app data:
- Open GitJournal, Press Hamburger button > Settings > Storage & File Formats > Enable "Store Repo Externally"
- Create a new folder (e.g. called repos). I chose Documents/repos/. Then, select the repos folder to the be the directory. GitJournal will store all the repos, 1 folder each.
  - Some notes about paths:
    - GitJournal will show the true path for the directory (ie /storage/emulated/0/Documents), whereas
    - the Files app only shows files inside of /storage/emulated/0 as the root aka. Internal storage folder.
  - Warning 🌶: You might get some permissions issues if you did not create the folder yourself, or you are using a restricted directory. In that case, GitJournal will show an error related to permissions. Delete the previous folder (using the Files app) and find/create a different folder.

Open the Obsidian Vault 📖

Install Obisidian from the playstore
Open vault in Obsidian

Usage 🔁

Make changes to your Vault in Obsidian
sync with the Cloud using GitJournal

Ending 👋

I wrote this document (you are reading it) on Obsidian, and in fact, Obsidian's new WYSIWIG editor is what inspired me to write SSH into your private machines from anywhere, for free, using Cloudflare Tunnel I posted on HN earlier this year. With the extensions installed, my Obsidian looks like:
Let's chat in the comments: Was I not clear on some of the steps? Do you have any ideas or suggestions on improving my Obsidian workflow?
Alternatives:
- I also tried Termux and cloned my repo there. However, I stopped using it because it didn't have an easy button to press, unlike the sync button on GitJournal. You have to a run a few commands in the terminal to set up the file permissions and update the package repositories so you can install git, vim and other useful tools.
- If you install adb, you can enter the command line on the Android by running adb shell. You can explore the Android filesystem and use adb to manually copy files using scp or rsync from your machine.
Not for iOS: This guide doesn't work for iOS, but there is a way to use git to sync files on iOS, but the software is paid (Working Copy, $19.99) if you want to make changes on iOS.

Extra Hint: git conflicts

If you do use git to sync files, you might find some merge conflicts between devices caused by the .obsidian/workspace file. To fix this, in your Obsidian Vault, create a .gitignore file and add .obisidian/workspace in the file. If you have already previously committed the file, run git rm --cached .obsidian/workspace to untrack the file.

Replacing SSH over Cloudflare with Tailscale

Ben Butterworth — Tue, 05 Jul 2022 00:00:00 GMT

I wrote SSH into your private machines from anywhere, for free, using Cloudflare Tunnel in April which led to a lot of discussion on Hacker News. A lot of people started mentioning alternatives (Tailscale, ZeroTier and issues with Cloudflare).

If you like this, you’re gonna love Tailscale https://tailscale.com/ - mmettler

Tailscale has a beta product, Tailscale SSH with 2 significant differences:

Easier setup: Tailscale SSH is easy to configure (I don't really need to write the setup steps).
Less security: Cloudflare Tunnel didn't replace authentication, but Tailscale SSH does. This means Tailscale could SSH (and access files) into my machine if they wanted to, but Cloudflare can't. With Cloudflare, I still had to authenticate with my private key.

Steps 🛼

Install tailscale on client machine (e.g. macOS) and login
Install tailscale on SSH server (e.g. raspberry pi)
On SSH server, set up Tailscale to use SSH: sudo tailscale up --ssh
Get the IP address of the SSH server machine from client machine or Tailscale website
SSH into the machine:
- ssh username@$IP_ADDRESS
- If you have Cloudflare WARP: You need to disable it, because that prevented me connecting to my local machines using Tailscale - even though I could still use e.g. ssh zen@raspberrypi.local -i ~/.ssh/raspberrypi though.
Optional: shorten SSH command into ssh pi
- On Tailscale's DNS admin page,
  - select a global nameserver and
  - enable MagicDNS

To SSH into your machine using fewer keystrokes, e.g. ssh pi, go into Tailscale's Machines admin page, and "edit machine name". Disable "auto-generate from OS hostname" and set the machine name.

Benefits 🤙

No machine passwords or SSH keys needed. Everything is replaced by your Tailscale account or SSO. I can move between laptops by just installing and logging into Tailscale.
No SSH configuration needed (e.g. ~/.ssh/config and /etc/ssh/sshd_config)
The configuration is so easy, that we can save time and avoid frustration at work. I work at a company with quite a few test devices, and each new person or test device would mean more configuration/setup time. We can:
- see a list of available test devices on the Tailscale website,
- setup developer machines to access these devices using SSO, and
- setup new test devices with 2 commands (install and launch Tailscale).
- We would do all of that whilst avoiding the poor experience with using VPNs and password managers.

Conclusion 🏁

Overall, I think I'm going to stick to Tailscale instead of Cloudflare Tunnel for accessing machines on my home network when away.

Question: what would you do? 😇

Alternative Cargo Registry authentication workaround

Ben Butterworth — Thu, 22 Sep 2022 00:00:00 GMT

This is a workaround to authenticating with an alternative cargo registry like Artifactory, a feature that is not yet available in Cargo.

Introduction 👋

There has been a Cargo MR: Implement RFC 3139: alternative registry authentication support, to add support to Cargo to authenticate with Registries when downloading crates. Currently, Cargo only sends authentication tokens for publish and yank network requests. Unfortunately, this MR has been open quite a while - since April 2022.
Workaround: This workaround works by injecting a basic authentication token to requests to the specific cargo registry.
Background: I noticed a colleague implementing a workaround using proxy.py. However, this tool might be more versatile, so it requires more config for MITM proxying, including generating certificate authorities and programatically mutating config files. I knew of mitmproxy which could potentially do things a bit more cleanly, and was a bit curious how it would work with Cargo. I also knew this attempt would be more open-source-able.

Pause: easier solution 🐣

There is an even easier solution.
If your crates are not sensitive, you could just disable authentication when downloading crates. There is an option to do this for Jfrog Artifactory repositories (for artifactory admins).

Setup ⚙️

These are setup instructions. You only do this once per machine. They are written for macOS so adjust them for other platforms if necessary.

Pre-requisites: Rust and Cargo: brew install rust
Install mitmproxy: run brew install mitmproxy
- This installs mitmproxy, mitmdump and more.
- Note, you can also python3 -m pip install mitmproxy, which installs binaries in e.g. /Users/$USER/Library/Python/3.9/lib. Be sure to add this to your path. For example, in ~/.zshrc, add export PATH=/Users/$USER/Library/Python/3.9/lib:$PATH
Run mitmproxy, and close it: run mitmproxy and then quit: Ctrl + C, type yes
- This will generate the certificate authority certificates in ~/.mitmproxy.
Create a file: add_cargo_auth_header.py is:

from mitmproxy import http
import base64
import os

REGISTRY_DOMAIN = "example.jfrog.io"

def request(flow: http.HTTPFlow):
    if flow.request.pretty_host == REGISTRY_DOMAIN:
        # If you prefer hardcoding your email and token in your script. This is okay if you don't commit your file
        email = "user@example.com"
        token = "replace this with your token"

        # Or if you prefer environment variables.
        email = os.environ["EMAIL"]
        token = os.environ["TOKEN"]

        encoded_token = base64.b64encode(f"{email}:{token}".encode())
        flow.request.headers["Authorization"] = b"Basic " + encoded_token

    # Optional: Handle more registries here.

Configure cargo to use your registry and this proxy
- Adjust your desired config file, to use the mitmproxy certificate authority.
  - To affect a single project, adjust .cargo/config.toml. For more information, see The Cargo Book: Configuration.
  - For example, update your .cargo/config.yaml to look like:
    - This example uses an "Jfrog Artifactory" Cargo registry. Be sure to update the cainfo file path

[registries.example]
index = https://example.jfrog.io/artifactory/git/cargo-repository.git

[http]
proxy = "localhost:9001"
# on macOS
cainfo = "/Users/ben/.mitmproxy/mitmproxy-ca.pem"
# in a Docker container
# cainfo = "/root/.mitmproxy/mitmproxy-ca.pem"

[net]
git-fetch-with-cli = true

Authenticate with the Registry index
- The index contains the information about all crates
- Warning: update EMAIL and INDEX_DOMAIN.

# Replace @ with %40
EMAIL=user@example.com
REGISTRY_DOMAIN=example.jfrog.io/
SAFE_EMAIL=${EMAIL/@/%40}
git config --global url."https://${SAFE_EMAIL}:${TOKEN}@/${REGISTRY_DOMAIN}".insteadOf "https://${REGISTRY_DOMAIN}"

Usage 🎮

Start the local proxy server: either
- for interactive: run mitmproxy --listen-port 9001 -s add_cargo_auth_header.py
- for no interactivity: run mitmdump --listen-port 9001 -s add_cargo_auth_header.py
  - You can replace 9001 with whichever port you prefer.
Launch cargo as normal in a separate terminal. e.g. cargo build, cargo run, etc.

Bonus: Usage in CI 🎁

To use it programatically (e.g. in CI), you could run:

# Start proxy server
mitmdump --listen-port 9001 -s add_cargo_auth_header.py &
# Use cargo as normal
cargo build # or cargo update, cargo run, etc.
# Kill proxy server
kill %1

Conclusion 🏁

I look forward to deleting this local proxy server when this feature is available in Cargo. In the mean time, I've got a consistent way to access other registries securely.
If you have a suggestion or found a problem, comment below. That would probably be useful to other readers, and also helps me learn.

Docker Compose, quickly

Ben Butterworth — Sat, 17 Sep 2022 00:00:00 GMT

This is a quick, practical guide to get you started. A common issue I notice in projects that use Dockerfile's is that they don't easily run on developer machines and don't work across operating systems and IDEs (Visual Studio Code, vs. PyCharm, vs. Command line).

Why Docker? 😋

Consistent environment:
- Some binaries and applications might not be able to run your platform natively. Fortunately, Apple ARM macs (e.g. M1/M2) already allow you to run amd64/x86_64 binaries thanks to Rosetta 2. However, you still can't run linux/arm64 or linux/amd64 on macOS. You still can't run ARM binaries on older, x86_64 macs. With Docker, you can run binaries from other architectures more easily, thanks to QEMU emulation.
- Environment variables are not automatically passed to the docker image or container.
Clean environment:
- whenever I want a clean environment to test in, I find myself spawning a docker container and entering it's shell: docker run -it python:3.9-bullseye bash. This makes me more confident when helping colleagues or writing articles.
Powerful build: you can compile docker images for other platforms on your machine. Without Docker, how can you build applications or binaries for Linux?
Many more benefits. This is a quick guide. 😉

Steps 🚶🏻

Do all the steps in your project directory. See https://github.com/uxduck/docker-compose-guide for an example.

Create a `Dockerfile` for your application

Writing a Dockerfile is very specific to the application you want to deploy, so I won't include that here. It can also get quite complex. In the example, we use a basic `Dockerfile.
You'll need to copy over your application.

Create a `docker-compose.yml` file

This should contain:

services:
  development:
    build:
      context: "./"
      dockerfile: Dockerfile
      args:
        # Optional credentials used during image build.
        - USERNAME
        - TOKEN
    env_file:
      - .env
    command: "tail -f /dev/null"
    # Specify a platform if some of your binaries only work on certain platforms. e.g. linux/amd64, linux/arm64
    # platform: linux/amd64
    volumes:
      - .:/workspace

If you need credentials when building your image:

Create a `.env`

The .env file should contain all the credentials in your args

Add `.env` to `.gitignore`

Ignore this file, because it will contain your credentials.

Create a `.env.example`

Copy your .env and remove the credentials.
Keep this up to date whenever your .env changes

Create VSCode devcontainer configuration

Even if you don't use VSCode, another developer might want in the future. Create one now so you configure it correctly 😆.
- We don't want developer environment to drift or become inconsistent between Visual Studio Code users and others. Unfortunately, devcontainers defaults to configuring things inside devcontainers.json this can easily happen.
install "Remote - Containers" extension from Microsoft.
Open Command Palette, run "Remote-Container: Add Development Container Configuration Files..."
Delete devcontainer/docker-compose.yml
Inside .devcontainer/devcontainer.json,
- use only 1 docker-compose file:

{
    ...
    "dockerComposeFile": ["../docker-compose.yml"],
}

Add a warning to the top of .devcontainer/devcontainer.json:

// WARNING: developers should put most configuration in docker-compose.yml instead of VSCode specific configuration to ensure all developer environments work, not just VSCode.

Try launching the application

From the command line, run docker-compose up -d --build
Enter the shell inside the container: docker exec -it docker-compose-guide_development_1 bash
- This command won't need to change everything you build the image or restart your computer.
- That name should be consistent to the folder name and the service inside docker-compose.yml. If you change those, you'll need to update this command.
Run a command, you're in Linux: e.g. run uname -a
- This should return Linux eed70de56bc9 5.10.104-linuxkit #1 SMP PREEMPT aarch64 GNU/Linux

Write the documentation

Describe what you need to do to launch the application locally. Take a look at the example Contributing guide.

Conclusion 🏁

That was a quick guide to using Docker compose locally. I'll be happy to clarify things and point to more resources. Some tricky things for me were to:

setup Gitlab authenticate with a private Container Registry so that it would use the image when running CI commands. You need to generate a DOCKER_AUTH_CONFIG Gitlab environment variable.
Building complex existing tools together with my application (e.g. Ardupilot simulator/SITL) in the same Docker image. This is useful for running integration tests of your application communicating with the drone simulator.

Why you shouldn't pay into a pension 💀

Ben Butterworth — Sun, 19 Mar 2023 00:00:00 GMT

I've been opting out of my pension within a few months of starting a job at any company. Every time, it's taken considerable thought and confusion. It seems like every few months a ridiculous event occurs, related to pensions. I wanted to share why I stopped contributing to pensions so others can make a more informed decision. I wish I had seen such an article in the past.

Caveat 🥦

The following is based on my experience of having 3 pensions in the UK (different companies I worked for use different providers).
I am not selling anything.
This content is free.
This is not financial advice, which is a regulated activity. Financial advice is usually paid for, and the seller of the advice often makes money from you even when you don't - their incentives are not aligned with yours.

Background: What is a pension? 🤯

A pension is a monthly bill you automatically pay as a percentage of your salary. For example, if you earn £1000 a month, you can pay £50 (5%) every month until you retire. This money cannot be accessed until you retire. It is opt-in, which means everyone who works gets this, by default. In some cases, your employer will pay more money every month, e.g. £25.

ChatGPT: A pension is a long-term savings plan, where you contribute a percentage of your salary every month until you retire. These contributions are usually made automatically, and you cannot access the money until you reach retirement age.

Why use a pension 🧟‍♂️

There's a lot of information about this online. For example, a course by MoneySavingExpert, MSE’s Academy of Money.

Why avoid pensions 🚨

Potential death: You might not make it to the end. You might die before you retire. And if you do make it to retirement, the pension fund has an incentive to make it harder for you to get your money - they might not even have all your money. They might lobby to change the date, reduce the payments, etc.
Law changes: The government and pension providers have a lot of power over your money. They can raise the age you can take your money or limit how you can use that money. See Proposed new timetable for State Pension age increases. Also, the government organisation to handle pension complaints is not well trusted on trustpilot - currently rated 1.7 stars. Your retirement plans will be affected by the governments changes. As governments get squeezed - they need to squeeze their people, with tax and pensions law changes.
- Examples:
  - UK: Jeremy Hunt plans to tax pensioners in November 2022
  - France: Macron pushing through pension changes without vote
Higher taxes and extra cost: Although you don't get taxed now, you will get taxed when you retire. By then, taxation will probably be higher, not lower, as governments go into more debt. Overall, you might get taxed more than just getting the money now. See UK tax burden to hit highest levels since the 60s. Also, the pension providers charge you a fee (percentage of your pot).
Get the money now:
- Contribute to a house deposit
- Invest in yourself: education, training, holidays?
- Feed people: bills have been going up, and perhaps you actually want to spend that money now to feed your children today.
- Invest your money anywhere else
Rigidity: currency might be devalued in the future. Sure, you can invest in index or mutual funds on there instead, but have you seen the limitations of the pension websites.
Irrelevance: for some people who are financially well-off, they would have investments which make them more money, consistently. This might be a business, or a mortgage on a flat they own but rent out. Every month, they get more money than they spend (including mortgage, bills, food, holidays). The cost of managing multiple pensions would be unnecessary work in their lives. They'd have to monitor a separate app and potentially types of investments.
Incentives and competition: pension funds/managers are not very good at investing: the UI is not very good, unlike other dedicated-investment apps. They have no incentive to build good UX, since you have no choice in taking your money out. They are not very good companies, just look on the complaints filed on the pensions ombudsman (for example, this complaint on the NHS Pension Scheme).
Fragmentation of pension: over the years, you might have opened more than 1 pension. It hurts my brain to remember the pension providers that I might have used.
- Switching cost: Choosing and moving-between them is a hassle. Different jobs you have will get different pension providers. There are many pension providers, and they do not want to talk nicely with each other. See trustpilot reviews of nestpensions.
  - The company has no incentive to be a good investor. They have your money already, and have the ultimate customer on long term subscription.
  - Can you even move your pension to another country? What happens if you don't like Brexit or are struggling in the UK economy?
- Forget: You might forget about some pensions. The company has no incentive to remind you.
Less control: You have very little control over the money:
- simply, you can't use the money until the term ends (48 years for me). Your buying power is less. When you find a flat to buy and it is 5% higher than you can buy, remember your pension took that money away from you.
- you can't control how it is invested. If you bought that 5% house, you could rent out some room to lodgers, and save money.
Philosophical:
- It is opt-out: Facebook was annoyed that the iOS default switched away from opt-out to "ask the user" (not even as harsh as opt-in), they even put adverts on the news to target Apple. They set it to opt-out so most people are automatically in it - if it's so good, why not just let people know and let them opt-in.
- The wrong solution: What is the argument for a pension? That you don't save enough for retirement, you overspend before retirement and/or you overspend when you're retired. Surely a good solution would be education.
- Agility: Those who move fast can exploit an opportunity for success. Conversely, if you move slowly, you have a lot to lose. Pensions is immobile money, which means you get the worst deals.

Summary 🏁

So that's it - an uncommon view on pensions. Reply with your view points in the comment section - tell me I'm nuts - you might be helping me 🤓. I wonder how the recent and future AI technology will affect the economy and governments, which in turn affect taxes and pensions.

Warning: Do not opt-out of the pension to spend the extra money on unnecessary things. Default to saving money, not spending it. When I got my first raise, I thought to myself, "Oh wow, I've got £N extra cash, every month. What can I spend it on?". Then I remembered, this will help build my safety buffer.

Reminder: do you know where all your pensions are, and how they are growing over time? Keep track of them.

tRPC, Astro and Cloudflare Workers

Ben Butterworth — Sun, 19 Mar 2023 00:00:00 GMT

I haven't seen if Astro, tRPC and Cloudflare Workers go well together. I found myself asking: must tRPC be in a separate package/deployment (e.g. packages/ in a monorepo, or separate Cloudflare Worker). I also managed to get Cloudflare bindings to work (e.g. D1, environment variables, R2). I'm pleased to say it's relatively straight forward - there are a few options.

Update: I last updated this page on 6th April 2023, as I use tRPC in my job, and Astro in my free time.

Background:

If you haven't heard of Cloudflare Workers, tRPC or Astro, read this. Otherwise, skip to options.

Cloudflare Workers: cheap but fast serverless platform that integrates Cloudflare's other services: SQLite on the edge (D1), S3-alternative with zero egress fees (R2) and WebSockets (durable objects, also used by DriftDB).
tRPC lets you call your backend like you call functions. It's similar to gRPC, but without the code generation headache that comes with it. When using gRPC/protobufs, I've found the codegen to be a very complex area:
- There are multiple code generation tools for each language, with different features and bugs. They're not consistent. Not all features are implemented for each language.
- gRPC/protobuf code generation tools seem to be neglected.
- Code generation can be configured in different ways. This means the generated code is different and incompatible. Switching codegen tools to get their features (e.g. google-protobuf/grpc-web to protobuf-ts, or in Python: from betterproto to grpcio_tools) are major breaking changes.
- Warning: Should tRPC even be used in Cloudflare Workers? It might have cold-start issues.

I don't have these issues, and another discord user mentioned:

Cold starts definitely play a part but it just sounds like the codes own initialisation
Astro is a framework for developing web applications with any Javascript/Typescript framework. You can make static or server-side rendered (SSG) websites. You can write code that executes in the browser and in your backend, in the same file (when using SSG). It's like NextJS but it supports React, Svelte, Vue, Solid and more, in the same app (and UI component).

---
import Layout from "../layouts/Layout.astro";
import { trpcClient } from "../ts/trpcClient";

// Server-side: Call rest of backend APIs, e.g. createUser
const userId = await trpcClient.createUser.mutate({
	name: "benbutterworth",
	bio: "gardener",
});
const user = await trpcClient.getUserById.query(userId);
// Logged server-side (in cloudflare)
console.log({ user });
---

<Layout title="Welcome to Astro.">
	<main>
    <p>Generated server-side</p>
		<h1>Welcome, {user.name}</h1>
		<h2>User ID: {userId}</h2>
		<h2>User bio: {user.bio}</h2>
		<p>Generated client-side</p>
		<h1 class="user2"></h1>
		<h2 class="user2-id"></h2>
		<h2 class="user2-bio"></h2>
	</main>
	<script>
		// Client-side: Call backend APIs, e.g. createUser
		import { trpcClient } from "../ts/trpcClient";

		const userId = await trpcClient.createUser.mutate({
			name: "Anonymous",
			bio: "Interwebs",
		});
		const user = await trpcClient.getUserById.query(userId)
		// Logged client-side (browser console).
		console.log({ user });

		document.querySelector("h1.user2")!.textContent = `Hello, ${user.name}`;
		document.querySelector("h2.user2-id")!.textContent = userId;
		document.querySelector("h2.user2-bio")!.textContent = user.bio;
	</script>
</Layout>

Zapp.run: is a cool online Flutter IDE that makes use of Cloudflare Workers and Astro, and judging by Astro + tRPC v10, perhaps they use tRPC as well.

Options:

Simplest option: Avoid tRPC completely
Simpler option: Cloudflare Pages
Complex option: Cloudflare Pages + Cloudflare Worker

Simplest option: Avoid tRPC completely

In practice, this isn't really an option.

Astro already supports server-side rendering. You can run code server-side above --- in your components. You can already use your environment variables securely, and connect directly to private APIs/databases. You can already call end-to-end typesafe APIs, because each Astro page can be rendered server side.

However, Astro doesn't give you a way to call your APIs from the browser in a typesafe way, after the page loads.

Simpler option: Cloudflare Pages

Both the backend code (tRPC, database access, R2 access) lives in the same "package" as the frontend.
Example
- website: https://trpc-astro-cloudflare.pages.dev/
- repo: https://github.com/uxduck/trpc-astro-cloudflare
Positives:
- Less configuration and code
- Simpler deployment. Your server-side code in .astro files run on the same service.
- Single command to deploy
- Hot-restart works well. Saving a file leads to the backend rebuilding, and the frontend refreshing. In the example, run pnpm dev to start the dev server.
Negatives:
- Larger bundle size. Your frontend and backend application (or part of it) is in the same cloudflare pages project.
Deployed using Cloudflare Pages: wrangler pages publish (internally uses Cloudflare Workers)
Separation of concerns: You can call other Cloudflare Workers from your Cloudflare Pages Function using "Cloudflare Pages Functions Service Bindings". This does not incur network latency because tRPC running on Cloudflare Workers uses the fetch API, which does not go over the network.

Complex option: Cloudflare Pages + Backend

This is approach is probably good if you've already got a backend. This backend can be Cloudflare Workers running tRPC, Hono, Express, any combination of these, or anything else (e.g. NodeJS). Consider tRPC's Express.js adapter or Hono adapter (discussion). This allows you to avoid Cloudflare Worker platform for server-side rendering.
Positives:
- Deploy frontend and backend separately. Choose different services for frontend and backend.
- Separation: the code will not get entangled, because they're in separate packages. Forces you to be more strict.
- Cloudflare Workers separate from frontend: Cloudflare Pages (which uses Workers internally)
  - This separation means they are separate workers:
    - more control over worker / features
Negatives:
- More complex maintenance. e.g. You need to setup monorepo tool to build backend code before frontend is built. When deploying a preview environment, you need to deploy the backend. The frontend needs to be pointed to the correct backend. Preview environments are not self-contained.
Deployed as 1 Cloudflare Pages deployment, and 1 Cloudflare worker: wrangler publish
Separation of concerns: You can call other Cloudflare Workers using Service Bindings. This also does not incur network latency because tRPC running on Cloudflare Workers uses the fetch API, which does not go over the network.

Summary 📒

I'm currently trying tRPC with Astro on Cloudflare Pages, with a single Cloudflare Pages Function. That is to say, I only have 1 worker: a fat worker. Let me know if you have any questions.

Other resources

tRPC Pages Plugin: for both options, you can use this plugin for some convenience (to get Cloudflare bindings to D1, R2, etc.). I don't use it because it doesn't seem to support Astro.
Warning: When calling another worker (service bindings), you need to use the Cloudflare API, which means you lose end-to-end typing / benefit of tRPC. tRPC has server-side calls, however that will not call another worker: it does not allow you to call workers with different implementations. This means you'd only have 1 worker, and it would get large. You lose "separation of concerns" between multiple workers.
- I initially thought: "Surely there's a way to get the benefits of both tRPC and Cloudflare Service bindings".
- Edit: oh, see tRPC with no latency or network errors. Thanks arjunyel.com - they did this to turn on Node compatibility just for the backend. However, IMHO if you don't need node compatibility, it doesn't make sense to do this unless you need to split your code across multiple workers - your worker bundle size might be too big (>1MB on free tier, >5MB on paid tier, or the size is affecting latency/start-up time). Or perhaps you have multiple teams and they want to deploy independently. It might get messy having multiple tRPC backends running and connecting to each other. I think tRPC being a backend framework where you can define all APIs has tension with splitting your backend into multiple workers/bundles. It's possible, but I'd recommend not doing it until performance or functionality is limited because of it. Also, I got an answer from the blog author on Discord.

createTRPCProxyClient<AppRouter>({
  links: [
    httpBatchLink({
      // Any URL will work since we are communicating with the worker directly
      url: "https://www.example.com/trpc",
      // Pass in the Service binding after binding it (lol)
      fetch: env.BACKEND.fetch.bind(env.BACKEND),
    }),
  ],
});

Warning: Even though Jetbrain's Webstorm IDE has an Astro plugin, it looks like it doesn't support tRPC. For example, you cannot navigate to the implementation of a query/mutation (getUserById: t.procedure.input(z.string())...) from the astro file (const user = await client.getUserById.query(userId);).

Example tRPC Backend

`pages/trpc/api/[trpc].ts`

import { initTRPC } from "@trpc/server";
import { fetchRequestHandler } from "@trpc/server/adapters/fetch";
import { Context, createContext } from "../context";
import { z } from "zod";
import type { APIRoute } from "astro";
import { trpcApiPath } from "../trpcPath";
type User = {
  id: string;
  name: string;
  bio?: string;
};

// This has to match quite strongly to the schema. Considering using an ORM?
type UserDb = {
  UserID: string;
  name: string;
  bio?: string;
};

// Inpsired by https://invertase.io/blog/astro-trpc-v10/ and https://trpc.io/docs/fetch
export const t = initTRPC.context<Context>().create();

const appRouter = t.router({
  getUsers: t.procedure.query(async ({ ctx }) => {
    try {
      // When running locally, it seems to use the wrong database. My local database has a Users table
      // See `wrangler d1 execute trpc-astro-cloudflare-template --command="SELECT name FROM sqlite_master WHERE type='table'" --local`.
      // However, error is
      // {
      //     message: 'D1_ERROR',
      //     cause: 'Error: SqliteError: no such table: Users'
      //   }
      const stmt = ctx.db.prepare("SELECT * FROM Users LIMIT 10");
      const { results } = await stmt.all<UserDb>();

      return results?.map((user) => ({
        id: user.UserID,
        name: user.name,
        bio: user.bio,
      }));
    } catch (e: any) {
      console.log({
        message: e.message,
        cause: e.cause.message,
      });
    }
  }),
  getUserById: t.procedure.input(z.string()).query(async ({ input, ctx }) => {
    const result = await ctx.db
      .prepare("SELECT * FROM Users where UserID = ?1")
      .bind(input)
      .first();
    return { id: result.UserID, name: result.name, bio: result.bio };
  }),
  deleteUsers: t.procedure.mutation(async ({ ctx }) => {
    await ctx.db.prepare("DELETE FROM Users").run();
  }),
  createUser: t.procedure
    // validate input with Zod
    .input(
      z.object({
        name: z.string().min(3),
        bio: z.string().max(142).optional(),
      }),
    )
    .mutation(async ({ input, ctx }) => {
      const id = Date.now().toString();
      const user: User = { id, ...input };
      await ctx.db
        .prepare("INSERT INTO Users (UserID, name, bio) VALUES (?1, ?2, ?3);")
        .bind(id, user.name, user.bio)
        .run();
      return id;
    }),
});

// The Astro API route, handling all incoming HTTP requests.
export const all: APIRoute = ({ request }) => {
  return fetchRequestHandler({
    req: request,
    endpoint: trpcApiPath,
    router: appRouter,
    createContext,
  });
};

export type AppRouter = typeof appRouter;

`src/ts/context.ts`

import type { inferAsyncReturnType } from "@trpc/server";
import type { FetchCreateContextFnOptions } from "@trpc/server/adapters/fetch";
import type { Env } from "./worker-configuration";
import { getRuntime } from "@astrojs/cloudflare/runtime";

export function createContext({
  req,
  resHeaders,
}: FetchCreateContextFnOptions) {
  // Get cloudflare bindings, as per https://docs.astro.build/en/guides/integrations-guide/cloudflare/#access-to-the-cloudflare-runtime
  const runtime = getRuntime<Env>(req);
  // console.log({runtime});
  // Now use a binding, for example: `runtime.env.SERVICE.fetch()`
  // Alternatively, get an environment variable with: `import.meta.env.SERVER_URL`
  // You can read custom or pre-defined environmment variables with e.g. import.meta.env.MODE, .BASE_URL, .CUSTOM_VAR, etc.

  const user = { name: req.headers.get("username") ?? "anonymous" };
  const db = runtime.env.DB;
  return { req, resHeaders, user, db };
}
export type Context = inferAsyncReturnType<typeof createContext>;

Cloudflare Worker's Bindings in tRPC 🧙‍♀️

Ben Butterworth — Sun, 14 May 2023 00:00:00 GMT

When using Cloudflare Workers, the main selling point is you can integrate with a lot of other Cloudflare APIs, like D1 (database), R2 (storage) and Durable Objects (websockets), and more. However, there isn't any documentation on how to do that.

The up-to-date way to implement Cloudflare Workers is by using the module workers syntax, as follows:

export default {
  async fetch(
    request: Request,
    env: Env, // 👈 not shown on the tRPC docs
    ctx: ExecutionContext, // 👈 not shown on the tRPC docs
  ): Promise<Response> {
    return fetchRequestHandler({
      endpoint: trpcApiPath,
      req: request,
      router: appRouter,
      createContext,
    });
  },
};

You get the env object here, allowing you to access cloudflare bindings. Often, examples won't include the 2nd and 3rd parameters of fetch, showing only async fetch(request: Request): Promise<Response> {...}. For example, the tRPC docs.

Actually, we just need to use env in createContext. Pretty simple. So let's try.

Use `env` in `createContext` 🐛

The easiest option is to "inline" the createContext argument. However, this won't actually work because we need the createContext function to instantiate tRPC / appRouter: const t = initTRPC.context<Context>().create();, in another file. You might also get into a circular dependency hell.

import { FetchCreateContextFnOptions, fetchRequestHandler } from '@trpc/server/adapters/fetch';
import { appRouter } from './ts/trpc';
import { trpcApiPath } from './ts/trpcPath';
import { Env } from './ts/worker-configuration';
import { drizzle } from 'drizzle-orm/d1';

export default {
    async fetch(
        request: Request,
        env: Env,
        ctx: ExecutionContext
    ): Promise<Response> {
        return fetchRequestHandler({
            endpoint: trpcApiPath,
            req: request,
            router: appRouter,
            createContext: ({ req, resHeaders }: FetchCreateContextFnOptions) =>
          			// 🎸 You have access to `env` here, so
                // use a binding, for example: `env.SERVICE.fetch()`
                const user = { name: req.headers.get('username') ?? 'anonymous' };
                const db = drizzle(env.DB);
                return { req, resHeaders, user, db };
            },
        });
    },
};

Refactoring... it works. 😍

// src/index.ts
import {
  FetchCreateContextFnOptions,
  fetchRequestHandler,
} from "@trpc/server/adapters/fetch";
import { appRouter } from "./trpc/appRouter";
import { trpcApiPath } from "./trpc/trpcPath";
import { Env } from "./worker-configuration";
import { createContext } from "./trpc/context";

export default {
  async fetch(
    request: Request,
    env: Env,
    ctx: ExecutionContext,
  ): Promise<Response> {
    if (request.method === "OPTIONS") {
      const response = new Response(null, {
        status: 200,
        headers: {
          "Access-Control-Allow-Origin": "*",
          "Access-Control-Allow-Headers": "*",
        },
      });
      return response;
    }

    return fetchRequestHandler({
      endpoint: trpcApiPath,
      req: request,
      router: appRouter,
      createContext: (options: FetchCreateContextFnOptions) =>
        createContext({ ...options, env, ctx }),
    });
  },
};

// src/trpc/context.ts
import { FetchCreateContextFnOptions } from "@trpc/server/adapters/fetch";
import { Env } from "./ts/worker-configuration";
import { drizzle } from "drizzle-orm/d1";

const createContext = async ({
  req,
  env,
  resHeaders,
}: FetchCreateContextFnOptions & { env: Env; ctx: ExecutionContext }) => {
  console.log(`Gotttt itt: ${env.MY_ENV_VAR}`);
  // Now use a binding, for example: `env.SERVICE.fetch()`
  const user = { name: req.headers.get("username") ?? "anonymous" };
  const db = drizzle(env.DB);
  return { req, resHeaders, user, db };
};

export type Context = inferAsyncReturnType<typeof createContext>;

Benefits 📈

This approach:

avoids a third party dependency
explains how to do it, so you can change it or extend it
so when the Cloudflare Worker's API changes in the future, your understanding can help you upgrade to the new API.
Avoids hijacking the Cloudflare "entry point", so you can still use the official way of defining a Cloudflare worker. For example, to use static assets,
There is an alternative (a plugin, cloudflare-pages-plugin-trpc) which doesn't always work, for example when you're using Astro + tRPC + Cloudflare Workers. See it's internals.

Conclusion

I'd be interested to know what you're building, and how you're using tRPC and Cloudflare!

Running SeamlessM4T locally for translating text and speech

Ben Butterworth — Fri, 05 Jan 2024 00:00:00 GMT

I found a YouTube video spoken in french, and I wanted to translate it to English. This was an opportunity to try SeamlessM4T. However, when I tried to use it, it ended up taking more than an hour. After finding the path of least resistance that actually worked, I wanted to share this with others.

These steps are for the terminal. I used an ARM mac (M1 Pro). Intel macs are a bit more complicated because some dependencies need to be manually built locally (fairseq2).

SeamlessM4T is a family of models that can translate:

text to speech
speech to text
text to text
speech to speech (my use case today, translating French videos into English)

Set up SeamlessM4T locally

Use Python 3.11, because 3.12 is not supported by torch or fairseq2
- I already use pyenv, so I'll run: pyenv local 3.11
- You can install it too, but a simpler option is to uninstall python and download and install version 3.11 from https://www.python.org/downloads/
Clone the repo: git clone git@github.com:facebookresearch/seamless_communication.git
Change directory: cd seamless_communication
python -m venv venv
source venv/bin/activate
pip install torch fairseq2 -r dev_requirements.txt
pip install -e .
- this downloads dependencies from pyproject.toml and makes the current directory an editable package
brew install libsndfile
- When running m4t_predict later, if you get an error OSError: libsndfile is not found! Use your system package manager to install it , reinstall it by running brew reinstall libsndfile

my extra step (for YouTube)

Download the YouTube video, using yt-dlp
Convert the video into wav, using ffmpeg, I ran ffmpeg -i path/to/video.webm path/to/output-video.wav
Run it through seamless (in my case I'm running the speech to speech task - s2st):

m4t_predict --task s2st --model_name seamlessM4T_medium --tgt_lang eng 'path/to/input.wav' --output_path output`date +%Y-%m-%d.%H:%M:%S`.wav

Lag/crash warning

When running the default model (SeamlessM4T-Large v2) on my MacBook Pro with 16GB memory and M1 pro chip, it started lagging for a few minutes, "your system has run out of application memory" constantly popped up, and then my computer froze. Now, I use the SeamlessM4T-Medium model, and it works.

ValueError: The input sequence length must be less than or equal to the maximum sequence length (4096), but is 11800 instead.

If this happens, your input audio is too long. You can manually split the a few clips, depending on how many words are spoken. I'll be writing a python script to split the audio clip into chunks of 4096 max, and then run seamless on each chunk, and merge the audio back together.

A quick test is to just trim the input audio to 60 seconds chunks with ffmpeg, run ffmpeg - i input.wav -t 60 output-60s-1.wav for the first clip, then start at 60s and end at 60s ffmpeg - i input.wav -ss 60 -t 60 output-60s-2.wav, and so on.

To delete the downloaded models

The v2 large model (seamlessM4T_v2_large.pt) is 8.4GB. You might not have enough space to keep all models locally.

On macOS, models are stored in your home folder: /$HOME/.cache/fairseq2/assets, for example, /Users/yourusername/.cache/fairseq2/assets. Open it in finder and delete files you don't want: open $HOME/.cache/fairseq2/assets and delete them.

More resources

docs: https://github.com/facebookresearch/seamless_communication/tree/main/docs/m4t
Seamless Tutorial: https://github.com/facebookresearch/seamless_communication/blob/main/Seamless_Tutorial.ipynb
Languages supported: https://github.com/facebookresearch/seamless_communication/blob/main/docs/m4t/README.md#supported-languages

Serverless functions and edge databases

Ben Butterworth — Sun, 21 May 2023 00:00:00 GMT

Here are things I learnt when building applications using serverless functions (primarily Cloudflare Workers, previously GCP Functions) and edge/distributed databases (Turso, Cloudflare D1, Neon).

This article isn't polished because I'm focused on building, not writing. Also, in the spirit of start ups, ship things and get feedback! I'll keep this article updated as I build more stateful personal projects.

Why did I even bother with serverless? 🧑‍🎓

Low cost, and often generous free tier
- Cloudflare even provides serverless WebSockets (using Websockets over Workers and Durable Objects). Unfortunately, this is expensive, up to $5 a month per shared state - for example each chat room can cost you up to $5 a month).
Low latency / high performance: if the runtime and database is managed, the service providers (e.g. Cloudflare, Vercel, AWS, etc.) can run code and keep data close to your users.
DX Hype: Lots of hype around new products, all screaming "great developer experience". Everything is managed...
Curiosity: It's exciting to learn about unconventional, futuristic infrastructure made by companies like Fly, Neon, Turso and Cloudflare.
Run locally: I assumed all these services would be easy to run locally, like any conventional architecture. Unfortunately, this is not the case. Ironic, given that these services are meant to run everywhere/on-the-edge.

What went wrong 🐩💩

Almost everything I tried seemed to have rough edges.

Why not Cloudflare D1

Not open:
- API access: I don't want to put my data into D1 because it is closed to Cloudflare. You cannot access it from Fly.io, GCP or others directly.
- Closed source
Not distributed or replicated: It's a single instance. During the alpha, the database was a single instance of SQLite in a specific datacenter in the US. Now, you can choose where this instance lives. However, there is no replication or sharding. Even in the future, I expect writes will all go to a single instance.
Doesn't run locally: For a while, this was not possible. There is no documentation about it and it is not listed in the Miniflare features. However, the older miniflare 2's help pages mentioned --d1 and --d1-persist. Unfortunately, miniflare CLI is deprecated, and it's only for wrangler to use now. Now it's not clear. After configuring miniflare and running a quick check, it looks like the D1Database binding tries to connect to the cloud D1 instance, and freezes forever - it doesn't time out.
- I turned the wifi on my laptop 5 minutes later and it connected and made the query. 😅
- If someone has a way of running D1 locally with the latest wrangler and miniflare (v3), let me know!
Lagging DX: In general, I find Cloudflare are really fast at releases new product lines (e.g. D1, R2, Queues, Constellation) but the developer experience is quite bad.
Bad, unintuitive error handling: To read the error message, you need to log it in a specific, documented way, otherwise you'll only see an unhelpful D1_ERROR.

try {
  await db.exec("INSERTZ INTO my_table (name, employees) VALUES ()");
} catch (e: any) {
  console.log({
    message: e.message,
    cause: e.cause.message,
  });
}

Why not Turso 🤖

Can't run Turso locally with Cloudflare Workers.
- turso dev is not maintained, so I needed to install sqld and use that instead. That also didn't work.
- "For now the best will be to just create a new database for development in the actual service." - CEO of ChiselStrike, who develop Turso.
Fork of SQLite means I hit some issues that wouldn't have been the case for SQLite, since there are protocols used for communication.
Can't export/dump data: https://github.com/chiselstrike/turso-cli/issues/433
Thought: I'll be willing to try Turso again once I can run it locally and behaves like a normal database.

Why not serverless functions, in general? 📈

APIs are platform specific. Examples:
- The serverless functions you wrote for AWS Lambda cannot just be run on Cloudflare Workers, since Workers doesn't use NodeJS.
- This is extremely annoying when you find a bug and you want to switch to another service provider, but your code or your dependencies do not support another provider.
- This also means you have to learn how the service works.
Restrictions and limitations:
- for a long time, Cloudflare Workers had CPU limits of 50ms, even for paid users. This meant you couldn't even hash passwords with bcrypt. Workers still have a low memory limit, so I still struggled with image resizing in Cloudflare Workers recently.
- NextJS functions don't support WebSockets.
- Some have bad developer experience. For a long time, the only way to debug Cloudflare Workers was console.log, and this is described below. FWIW: this has improved with Miniflare, though I have had a lot of issues getting that to work.
  - Inserting console.log lines throughout your code can help you understand the state of your application in various stages until you reach the desired output. - How to put breakpoints and debug Cloudflare Workers?, originally Debugging Workers
Cold start time: to be efficient, serverless services don't keep your application running all the time. This means it pays for the cost for launching your application on every request. It is only launched when a user makes a request, and might stay alive for some time, like a cache.
- Example article: Cal.com Cold Start Resolution - HN comments
- Cloudflare tries to avoid this by using a more efficient runtime that starts up quicker.
- In a conventional, non serverless system, you pay the startup cost once, and it is invisible to users, since requests will be routed to the older instance until the new one is ready. You might also have multiple instances.
Expensive: Often serverless functions are good for low and unpredictable loads. If it's predictable, you can autoscale to handle the load, and if the load is high, it's cheaper to have dedicated machines.
Bad design: passing data between serverless functions means copying data, and sometimes even running the same code multiple times (e.g. authorization checks). This takes time, and time is money for serverless functions (and users). If you had your logic in one application, your total execution time, CPU utilisation and memory utilisation is lower. Your bill will likely be higher if these are higher. AWS have shown they reduced cost by 10x moving distributed serverless functions to a monolith one.
Overall, you trade old, solved problems for new problems that can be platform specific.

Neon Postgres 🌟

Currently experimenting with Postgres, Neon.

Vercel Postgres

Doesn't support drizzle yet.
Probably uses Neon under the hood.

My chosen tech stack 🤔

Still in progress... 🔨

Currently experimenting with Postgres, Neon.
If I want to connect multiple users with state and websockets, then use Fly.io/containers and use websockets as normal, as opposed to serverless and platform specific APIs.

Conclusion 👋

Offline last: Many serverless products have relatively bad developer experience even when the marketing sounds like you can "get X done in 5 minutes". Running things locally is pretty important for me, because it means I can work completely offline when travelling, and tests run very quickly.
- I wanted to run it locally since I've been taking flights and trains journeys with no internet, and sometimes am located in areas with really bad internet.
Continuous improvement: All the products I mentioned before I continuously improving, I look forward to building something using boring technology. See you in 2025, serverless SQLite databases.

1 Question 👋

Which serverless services would you recommend?

Bonus: Serverless WebSockets 💸

on Cloudflare:
- It is expensive. See discord message: "Its $4.22/mo for each DO (if active all month) not $5 per hour. But the team has been working on a hibernation API that will sleep the durable object if it isnt actively handling a message from a websocket. No clue when that will actually come out but its been in talks the last few months."
- It hasn't gotten cheaper over time. "They're working on improving pricing" has been suggested for more than a year.
  - See discord message: "in the past they have talked about making websockets not count as active time in order to make websocket pricing better which would mean it can sleep. But I assume if/when that happens it will be a new API specific to allowing the object to sleep with websocket connections but we will find out whenever that happens if it does"
- It's not supported by popular backend frameworks in Node. e.g. tRPC, Fastify, express, hono, or itty-router.
- It's a pretty confusing API. To actually get multiple clients connected together, there is only 1 paragraph of documentation on Cloudflare:
  - If your application needs to coordinate among multiple WebSocket connections, such as a chat room or game match, you will need to create a Durable Object so clients send messages to a single-point-of-coordination. Durable Objects are a coordinated state tool for Cloudflare Workers, which are often used in parallel with WebSockets to persist state over multiple clients and connections. Refer to Durable Objects to get started.
  - To optimise the cost, you could consider sharing a single durable objects between different "channels", but this adds to the mess. For example, the first approach is to create a durable object, and all clients connect to a worker which then connects to a durable object managing a single group chat. However, if you're a chat application, this doesn't scale with group chats, since each group chat would be ~$5/month. Instead, your durable object should handle multiple group chats. Oh, and keep in mind that if it does too much, it will drop requests: . Doesn't sound like it scales nicely.
on AWS Lambda
- It's possible
- Lock in: AWS only API
- I have not used it, so I can't say too much. I'm not interested in serious-lock-in products, especially the AWS ones.

Bonus: Why not Vitess

Vitess isn't really an edge database, but it is a sharded MySQL database used by Stripe, YouTube, Slack, GitHub and more.
Expensive: The first database is free, but the second is $29 a month. That's a sudden increase, that is not tied to usage. 🤔
I also don't have the problems of YouTube scale.

Application configuration, a simple approach.

Ben Butterworth — Tue, 21 Jan 2025 00:00:00 GMT

A quick post about setting up app configuration using environment variables (or files).

Why?

There are the the standard reasons of avoiding committing production secrets in code, like Rabbit R1. However, I've come across more projects that do rely on .env files, but simply haven't documented what the .env should look like, or where to get API keys, database connection strings, etc. Because it's gitignored, it's a puzzle to figure out what goes in it.

I'm writing this so I can signpost developers to a simple, maintainable way to set up configuration in their projects. I'd love your feedback would gladly update it if you've got better ideas.

Benefits of my approach

Keeps project configuration maintainable. You get a clear error immediately when the app starts, instead of getting confused by weird application or library behaviour because a string was empty later. You have up to date configuration documentation.

Bonus: You can also add additional error messages and documentation when the configuration is wrong or out of date, to help migrate existing developers local .env files when you make changes to it.

Environment variables

We can configure variables in .env files, and use libraries to automatically read and load text from them. Production applications would have the environment variables setup through a secrets manager or a web UI. Local application development would rely on local, .env files.

Git ignored `.env`

You have 2 files: .env (git ignored) and example.env (not gitignored), which have identical structure. Every variable in the file contains a comment / documentation on how to set it up. For example:

# REMINDER: Update the `example.env` file when you update `.env`
# Database
# These credentials already match the local development database in docker (docker-compose.yml)
DATABASE_URL=postgres://user:password@127.0.0.1:5432/appname
## Staging (Supabase) - SSL/TLS is required by configuring this in the Supabase project settings
#DATABASE_URL=
# Production (Another DB service) - add some useful links here
#DATABASE_URL=

In the README, one of the setup steps would be: "Create a .env file from the template cp example.env .env and update according to the comments". This keeps documentation of environment variables in the same place where it gets changed, and avoids duplication of documentation in README files (easily gets out of date).

Committed `.env` + git ignored `.env.local`

This is a similar approach, but I don't like it because IDEs don't automatically syntax highlight this file, since it ends with .local.

Validation

I validate environmnet variables, and I do that at application start-up time because I prefer the peace of mind that all the required API keys, backend URLs, database connection strings, etc. are already set correctly anywhere in my application.

I'll share some simplified examples.

Typescript example

I use zod types. I can import env anywhere else in the application.

import { z } from "zod";

const envSchema = z.object({
  LOG_DATABASE: z.coerce.boolean().default(false),
  OPENAI_API_KEY: z.string(),
  DATABASE_URL: z
    .string()
    .describe(
      [
        "Connection string to postgres database, starting with `postgres://`.",
        "Can be local or remote. Get the remote database from your project.",
      ].join(" "),
    ),
  PORT: z
    .string()
    .describe(
      [
        "The port the backend application should listen on.",
        "Useful when hosting on services that want to configure which",
        "port is listened on, like Fly.io. Defaults to 4000.",
      ].join(" "),
    )
    .optional(),
});

// Load environment variables just before reading and validating them
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const dotenvPath = path.resolve(__dirname, "../.env");
dotenv.config({ path: dotenvPath });
export const env = envSchema.parse(process.env);

Python example

I use pydantic.

from pydantic import BaseModel
from dotenv import load_dotenv
import os

load_dotenv()

class Config(BaseModel):
    LOG_DATABASE: bool = False
    DATABASE_URL: str
    PORT: int

config = Config(**os.environ)

Bonus Python recommendation: I use uv by Astral, instead of Poetry, pdm, pyenv or anything else. It seems to have simplified python and python package management.

Configuration files

For more complicated projects and special cases, I use yaml config files. This isn't an option if you can't easily swap out the config files. For example, on Fly.io, it's really easy to change environment variables and restart the application, but replacing the config file requires a new deploy. However, if you're using Kubernetes/Helm, then yaml files fit in well.

The validation above can be used directly on the data generated from config files, including easily adding nested objects. I prefer yaml because it's easier to read and edit, but you can use JSON or XML too.

Summary

I usually go with the environment variables + validation in my side projects because I don't need complex structures available in yaml/json files.

Should I Close Source my Company? Replying to Supabase

Ben Butterworth — Fri, 16 Aug 2024 00:00:00 GMT

Overview

I've been thinking about open source in the defence tech industry. I was reading Supabase's Should I Open Source my Company?, which highlights the benefits of building an open source company like Supabase. I wanted to add some nuance which might be useful for certain companies and industries. All things being equal, I would strongly prefer working on open source or running an open source company. Unfortunately, not all things are equal. For example, in the context of Russian invasion of Ukraine, innovations in Ukraine last between 3 to 6 months and we need to constantly innovate just to keep up. l'd hate to open source key technologies and reduce that window of advantage to adversaries who invade countries.

Benefits

Developers get to use their code even after they leave the company.
It commoditises your complement. Some of the functionality that your competitors would have to reimplement are now commodities, where customers can more easily and cheaply get them. The value capture for a competitor to implement those features is now lower.
Marketing to customers. If you were a consulting company and published some popular open source devtools, you'll be top of mind when the users of the devtools need extra help.
You get to show your code to others, which helps with employment, branding, starting-a-startup and I think, developer street-cred (only between developers and only if the code is good 😂️).
The supabase article has a lot more benefits, I'm keeping it DRY

Challenges

There are some circumstances where open source might be more difficult. Supabase avoids some challenges because:

PaaS technology is iterative, not ground breaking: Supabase is a PaaS where there is a lot of competition and improvements are iterative. There is not much code that is "novel" (intellectual property someone wants to steal), it's an iteration on existing software. Extremely novel systems (e.g. algorithms, e.g. path planning/scheduling algorithms) are unique/novel, and open sourcing them could mean a competitor improves their efficiency. Please tell me if I am wrong here.
open source is a marketing strategy that works when selling to developers. If Supabase was the first PaaS in the world, they'd probably get customers even if they weren't open source. Closed-source Firebase already exists, and plenty of other open source ones too. This works doubly well when developer-focused articles/memes are written about open source. Preaching to the choir.
it operates in a meritocratic industry. Developers are knowledgeable and decide which platforms to use. Some industries are extremely sketchy, where customers have no idea which product is better. They may just choose the biggest brand name. If that brand name can tick a box (implements novel algorithm), many customers will choose that big brand name over the original creators of the tool.
Their industry respects licensing. PaaS competitors have highly paid, respectful engineers who would not accept code theft or license violations. Other companies like Elastic, Terraform, Meta have licenses that treat large companies differently. This would likely be respected by competitors who can compete effectively, such as AWS and Azure. In defence, someone told me "don't work with X or Y (billion dollar companies in defence) because they have a bad reputation of stealing ideas from startups or acquiring them to effectively shut them down." Supabase's license is very open, so this point is more for other companies.
the (self-)hosting and scaling aspect of Supabase is actually difficult, and can change at any time. This is not because of Supabase's "evil" intentions. Supabase does not open source everything. Even then, documentation about DevOps might be on a different system, not GitHub. I'll let the users or maintainers of Supabase add more detail here if possible (they're an incredibly honest, knowledgeable and open team)

Summary

TLDR: think about what type of industry you're in, the customers you have and how that affects theft, license violation or simply legal copying. If open source, consider broadcasting your status by writing many articles about it. Also, don't prioritise self hosting documentation, just focus on your users - that is the priority (this is not malicious, it's business). Think about the value that a competitor will get by reading or copying your code, and hide some important bits. Then you can decide which parts of your company to open source without turning your for-profit into a charity or bankruptcy.

This wasn't meant to be a dig at Supabase. I respect their product and the major bits they've open sourced. I know people who have worked there. I just want the other side of the story to be written down, from the perspective of someone who's worked in closed source, open source and then closed source again. I still try to open source certain things.

Disagree? Agree? Distasteful? Leave a comment!

tlduck

Setting up Java on macOS

Motivation

Problem?

jenv does 2 things

Steps

Conclusion

How to delete many events in one go in Google Calendar

Markdown for convenient academic writing

Markdown .md

Benefits

Drawbacks

LaTex

Challenges I faced & a guide for others

What referencing system do I use?

What referencing style do I use?

How do I export Markdown to LaTex, including references?

How do I reference figure numbers in the text?

How do I actually customise the LaTex output?

Cool plugins for Typora

More documentation

fin

Computer sciences courses I loved at Imperial College

Courses I loved

The other courses I did

The other courses

Courses I wish I could do

Early mistakes as a self-taught developer

In summary

Watching tutorials.

Learning too much. Coding too little.

Irrelevant side projects?

Being distracted

macOS CLIs: Building, Signing & Notarizing

XCode vs. Swift Package ❌

Create Xcode project 👷‍♀️

Swift Tools Support

Add Swift Tools Support

⌨️🎨 Write some code 🚀🌈

Signing and Notarization ✍️

Set up your machine/ Xcode

Generate Developer ID certificates

Add certificate to keychain

Configure the Xcode project for release

Create an Archive

Quick check: Is your executable signed?

Build package/ installer

Quick Tests

Generate an app-specific password on Apple ID

Notarize package

Release

Other resources

Questions

Using Cloudflare Products for 2 years

Cloudflare DNS

Cloudflare Pages vs Netlify

Cloudflare Workers vs Firebase Cloud Functions (or AWS Lambda)

Cloudflare Community

Question for you

Anonymous video calls in React, using Ably, Mediapipe, Three.js and Next.js

A short demo

Benefits over conventional video calling

Technologies used

What ML model does Apple Vision Framework use?

Discovering where the files are

Resources directory

FlatBuffers for TensorFlow Lite, with a bonus about quantization

Getting flatc

Compiling TensorFlow Lite schema in Python

Using the library

An easier path

An even easier path

How to debug Android apps (including Flutter) without launching from the debugger?

Motivation

Steps

Trigger your app to launch

Flutter only

Attach your debugger

Tips/ warnings

Other nice tools

Markdown `.md`

Getting `flatc`

Configure the `cloudflared` service 🧑‍💻

Put your notes in a `git` repo hosted online. 🌎

Create a `Dockerfile` for your application

Create a `docker-compose.yml` file

Create a `.env`

Add `.env` to `.gitignore`

Create a `.env.example`