Background

The app I work on uses Snowplow for event tracking. When you send events, the payload has to adhere to strict JSON schemas. When you send an event with the right structure then everything is good, but if you get the structure wrong then the validation kicks in and events are lost to a bad queue.

Snowplow provides a debug collector that you can run locally and it will give you good information about whether the events you are sending are valid or not. The issue I was toying with is that the app I want to use to observe the debug collector doesn’t know when new data is available in order to update its UI.

The service structure looks something like this:

+-----+   sends events   +-----------+    needs to update   +-----------+
| iOS |----------------->| Collector |<---------------------| Debug GUI |
+-----+                  +-----------+                      +-----------+

The Debug GUI is a small local tool we plan to write to inspect what the collector has received, but it currently has no way of knowing when new data arrives. For the sake of my experiment I had the limitation that I cannot change the code in iOS or the Collector. It’s not cheating but the iOS client already has the ability to change the location of the Collector so I can point traffic where I want.

A couple of less fun solutions that would work would be:

• Manual - add a refresh button to the Debug GUI
• Noisy - add polling to the Debug GUI

I opted to explore option 3 as noted at the top of the post, which is to create a reverse proxy to sit in the middle and be able to observe when events occur and push them to the Debug GUI. This looks something like this

+-----+   sends events   +-------+   forwards   +-----------+
| iOS |----------------->| Proxy |------------->| Collector |
+-----+                  +-------+              +-----------+
                             |
                             |  sends messages  +-----------+
                             '----------------->| Debug GUI |
                                                +-----------+

With a rough structure to aim for I started experimenting.

Creating a basic proxy

I’m not aiming for production readiness as this is only for a local debugging tool but I started with a new empty project and manually added all the ktor bits I would need. As a start I know I’ll need both client and server libraries for Ktor and as I know I’ll be using WebSockets I looked ahead at the docs and noticed they were suggesting to use Netty for WebSocket support. The basic configuration then looks like this:

gradle/libs.versions.toml

[versions]
ktor = "3.4.0"

[libraries]
ktor-client-cio = { module = "io.ktor:ktor-client-cio", version.ref = "ktor" }
ktor-client-core = { module = "io.ktor:ktor-client-core", version.ref = "ktor" }
ktor-server-core = { module = "io.ktor:ktor-server-core", version.ref = "ktor" }
ktor-server-netty = { module = "io.ktor:ktor-server-netty", version.ref = "ktor" }

[bundles]
ktor-client = [
    "ktor-client-cio",
    "ktor-client-core",
]
ktor-server = [
    "ktor-server-core",
    "ktor-server-netty",
]

build.gradle.kts

...

dependencies {
    implementation(libs.bundles.ktor.client)
    implementation(libs.bundles.ktor.server)
}

...

With all the dependencies in place I can implement a main function that starts a server that will receive requests on port 9090 and forward them to 9091 before returning the result.

src/main/kotlin/Main.kt

fun main() {
    embeddedServer(Netty, port = 9090, host = "0.0.0.0") {
        val client = HttpClient(CIO)

        routing {
            route("{...}") {
                handle {
                    val path = call.request.uri

                    val response = client.request("http://localhost:9091$path") {
                        method = call.request.httpMethod

                        call.request.headers.forEach { key, values ->
                            values.forEach { header(key, it) }
                        }

                        setBody(call.receiveChannel())
                    }

                    call.respondBytes(
                        bytes = response.bodyAsBytes(),
                        status = response.status,
                        contentType = response.contentType()
                    )
                }
            }
        }
    }.start(wait = true)
}

The code isn’t super exciting but the high level idea is to copy the path, headers and body and construct an identical request to the service I am wrapping. The results of calling the wrapped service are then just sent back to the original caller. Because I forward the request body as a channel, large payloads can stream through without being buffered in memory.

After firing this up and changing my iOS client to point from port 9091 to 9090 I could see everything was working as if I’d done nothing. This was a bit worrying as I wasn’t sure I’d actually done anything so I placed a few breakpoints to confirm my code was actually being executed.

The proxy works, but it doesn’t observe anything yet - so next I added a WebSocket endpoint that interested tools can subscribe to.

Adding WebSockets

The general idea now is to continue the existing proxying logic but instead of just returning the result I’m also going to expose a WebSocket endpoint that I will publish new data to. To publish new data I’m going to need to be able to parse the data coming from the collector which means my dependency shopping list now includes WebSockets and json serialization.

gradle/libs.versions.toml

  [libraries]
  ktor-client-cio = { module = "io.ktor:ktor-client-cio", version.ref = "ktor" }
+ ktor-client-content-negotiation = { module = "io.ktor:ktor-client-content-negotiation", version.ref = "ktor" }
  ktor-client-core = { module = "io.ktor:ktor-client-core", version.ref = "ktor" }
+ ktor-client-serialization = { module = "io.ktor:ktor-serialization-kotlinx-json", version.ref = "ktor" }
  ktor-server-core = { module = "io.ktor:ktor-server-core", version.ref = "ktor" }
+ ktor-server-content-negotiation = { module = "io.ktor:ktor-server-content-negotiation", version.ref = "ktor" }
  ktor-server-netty = { module = "io.ktor:ktor-server-netty", version.ref = "ktor" }
+ ktor-server-websockets = { module = "io.ktor:ktor-server-websockets", version.ref = "ktor" }

+ [plugins]
+ kotlinx-serialization = { id = "org.jetbrains.kotlin.plugin.serialization", version = "2.3.0" }

  [bundles]
  ktor-client = [
      "ktor-client-cio",
+     "ktor-client-content-negotiation",
      "ktor-client-core",
+     "ktor-client-serialization"
  ]
  ktor-server = [
      "ktor-server-core",
+     "ktor-server-content-negotiation",
      "ktor-server-netty",
+     "ktor-server-websockets"
  ]

build.gradle.kts

  plugins {
      kotlin("jvm") version "2.2.21"
+     alias(libs.plugins.kotlinx.serialization)
  }

  ...

With all the dependencies installed I looked at the docs for guidance and came up with this just to verify the general setup

src/main/kotlin/Main.kt

suspend fun main() {
    embeddedServer(Netty, port = 9090, host = "0.0.0.0") {
        install(WebSockets)

        val subscribers = Collections.synchronizedList<WebSocketServerSession>(mutableListOf())

        routing {
            webSocket("/ws") {
                println("Web socket connected")
                subscribers += this

                repeat(10) {
                    send("Message $it")
                    delay(1.seconds)
                }

                try {
                    for (frame in incoming) {
                        // ignore client messages
                    }
                } finally {
                    subscribers -= this
                }

                close(CloseReason(CloseReason.Codes.NORMAL, "All done"))
            }
        }
    }.start(wait = true)
}

When this server is run and I connect to the WebSocket (I used websocat for testing) the server will print that a connection was made and then start sending Message 0-Message 9 with a second delay between each message. This works nicely and the socket is kept open by the infinite read loop that waits for input and just throws it away on repeat.

Now I know I can do the proxying and establish WebSockets it’s time to stitch things together.

Consume and emit events

The actual collector doesn’t actually return any useful data when we inspect the traffic so in reality I will need to make a further network request to fetch data. For the sake of this example, let’s pretend the collector returns the data directly in the response body rather than requiring a follow-up request.

I started by setting up the JSON parsing side of things. The reason for parsing the JSON is that the payload from the collector contains a lot of stuff that the Debug GUI doesn’t need, so it makes sense to return a filtered view.

val json = Json {
    ignoreUnknownKeys = true
    explicitNulls = false
}

embeddedServer(Netty, port = 9090, host = "0.0.0.0") {
    install(WebSockets) {
        contentConverter = KotlinxWebsocketSerializationConverter(json)
    }

    install(ServerContentNegotiation) {
        json(json)
    }

    val client = HttpClient(CIO) {
        install(ContentNegotiation) {
            json(json)
        }
    }

    ...
}

The above code creates a more lenient Json instance that will ignore unknown keys as I’m not going to reconstruct the shape of the entire response. With this created it’s wired into the ContentNegotiation plugins for the client/server. Annoyingly because both the client and server call the plugin ContentNegotiation you need to fully qualify to have them both in one project - I opted for aliasing on the import.

Next I updated the routing to wire things together

routing {
    webSocket("/ws") {
        println("Web socket connected")
        subscribers += this
        try {
            for (frame in incoming) {
                // ignore client messages
            }
        } finally {
            subscribers -= this
        }
    }

    route("{...}") {
        handle {
            val path = call.request.uri

            val response = client.request("http://localhost:9091$path") {
                method = call.request.httpMethod

                call.request.headers.forEach { key, values ->
                    values.forEach { header(key, it) }
                }

                setBody(call.receiveChannel())
            }

            val bodyText = response.bodyAsText()
            val items = json.decodeFromString<List<Item>>(bodyText)

            val targets = synchronized(subscribers) { subscribers.toList() }
            for (subscriber in targets) {
                subscriber.sendSerialized(items)
            }

            call.respondBytes(
                bytes = bodyText.toByteArray(),
                status = response.status,
                contentType = response.contentType()
            )
        }
    }
}

With this in place I spun everything up and connected to the endpoint with websocat and started sending events and then… Nothing, absolutely nothing was happening.

There was nothing in the logs so I slapped a break point in and stepped through things. It turns out there was an exception being thrown on the val items = json.decodeFromString<List<Item>>(bodyText) line but because there is no configuration for logging this error was just swallowed 🤦🏼‍♂️.

As it happens I’d messed up the structure of my @Serializable types but it was an annoying lesson that logging was not installed.

Having to drop down to stepping through the code line by line was a tedius, so the obvious fix was to install CallLogging and an slf4j backend.

In my case that was

ktor-server-call-logging = { module = "io.ktor:ktor-server-call-logging", version.ref = "ktor" }
logback = { module = "ch.qos.logback:logback-classic", version.ref = "logback" }

With the corrected data structures I spun it up again and everything worked perfectly.

Wrap up

In the end I had a tiny Ktor service that transparently proxied Snowplow traffic and broadcast filtered events over WebSockets to any connected UI.

This was actually a fun exploration and I reckon I’ll be able to take the learnings forward. The main take away on this one was that I should just try stuff - in the past I’ve wanted to do similar things with wrapping services but I’d assumed it would be too tricky. The whole thing took a couple of hours of experimenting and then it was all just clicking, which just reminds me that the things I put off aren’t ever that bad if you just start.

I ran into some code that captured work in a closure and ensured it was only invoked once. The pattern for this one-shot closure looked like this:

class Example {
    private var delayedWork: (() -> Void)?

    func functionThatQueuesWork() {
        delayedWork = { print("some work") }
    }

    func functionThatInvokesTheWork() {
        if let delayedWork {
            self.delayedWork = nil
            delayedWork()
        }
    }
}

Upon seeing this I wondered a) if we had a helper for this pattern b) if not what could the API look like. Basically I was nerd sniped into exploring what a reusable API could look like.

A Basic Property Wrapper

As always I find it best to start with concrete types then try and make things generic later on. The closure above is pretty simple - it takes no arguments and returns nothing so it’s a good place to start.

A rough scaffold to get some types lined up would look like this:

@propertyWrapper class CallOnce {
    var wrappedValue: () -> Void

    init(wrappedValue: @escaping () -> Void) {
        self.wrappedValue = wrappedValue
    }
}

This compiles but doesn’t really do anything useful for us as it’s literally just wrapping a closure with no additional behaviour. We need to mirror the previously mentioned logic, which was to store the closure then ensure it’s only called once. We currently have storage with var wrappedValue but we need to arrange for this to be some kind of no-op after it’s been invoked.

The simplest thing I can think of is to override the setter for wrappedValue and wrap the incoming closure with the additional logic. In order to do this we’ll need to define some backing storage.

Let’s start with storing the closure by adding some storage and just delegating to that

@propertyWrapper class CallOnce {
    private var function: () -> Void = {}

    var wrappedValue: () -> Void {
        get { function }
        set { function = newValue }
    }

    init(wrappedValue: @escaping () -> Void) {
        self.wrappedValue = wrappedValue
    }
}

Here I’ve opted to keep the wrappedValue as non optional and instead I’m just going to use a no-op instead of assigning function = nil.

To handle the functionality of changing the passed in closure to a no-op I’ll add a helper method that both init and the setter can use.

@propertyWrapper class CallOnce {
    private var function: () -> Void = {}

    var wrappedValue: () -> Void {
        get { function }
        set { decorate(newValue) }
    }

    init(wrappedValue: @escaping () -> Void) {
        decorate(wrappedValue)
    }

    private func decorate(_ function: @escaping () -> Void) {
        self.function = { [weak self] in
            defer { self?.function = {} }
            function()
        }
    }
}

I capture self weakly so the wrapper doesn’t participate in a retain cycle with the stored closure if it’s already gone, the no-op behavior is fine.

You’ll probably have noticed by now that this is making no attempt to be thread safe and accessing this property wrapper from the correct isolation context is left to the caller.

This all works great now but only for simple closure types.

Making it slightly more generic

To make this slightly more reusable we could open it up to allow any return type. This is fairly simple to do as we just need to introduce one new generic and slot it in to any place where we currently have a return type of Void.

- @propertyWrapper class CallOnce {
-     private var function: () -> Void = { }
+ @propertyWrapper class CallOnce<Output> {
+     private var function: () -> Output? = { nil }

-     var wrappedValue: () -> Void {
+     var wrappedValue: () -> Output? {
          get { function }
          set { decorate(newValue) }
      }

-     init(wrappedValue: @escaping () -> Void) {
+     init(wrappedValue: @escaping () -> Output?) {
          decorate(wrappedValue)
      }

-     private func decorate(_ function: @escaping () -> Void) {
+     private func decorate(_ function: @escaping () -> Output?) {
          self.function = { [weak self] in
-             defer { self?.function = {} }
+             defer { self?.function = { nil } }
              return function()
          }
      }
  }

Again this is working but wouldn’t it be nice if we could support any shape of function?

Parameter packs to the rescue

The way this is going to work is that we are going to add a parameter pack in a similar way to how we added the Output generic. Swift’s parameter packs let us abstract over an arbitrary number of parameters in a function signature. When we introduce the generic we use the each syntax then anytime after that when we want to use them again we use repeat each.

- @propertyWrapper class CallOnce<Output> {
-     private var function: () -> Output? = { nil }
+ @propertyWrapper class CallOnce<each Argument, Output> {
+     private var function: (repeat each Argument) -> Output? = { (_: repeat each Argument) in nil }

-     var wrappedValue: () -> Output? {
+     var wrappedValue: (repeat each Argument) -> Output? {
          get { function }
          set { decorate(newValue) }
      }

-     init(wrappedValue: @escaping () -> Output?) {
+     init(wrappedValue: @escaping (repeat each Argument) -> Output?) {
          decorate(wrappedValue)
      }

-     private func decorate(_ function: @escaping () -> Output?) {
-         self.function = { [weak self] in
-             defer { self?.function = { nil } }
-             return function()
-         }
+     private func decorate(_ function: @escaping (repeat each Argument) -> Output?) {
+         self.function = { [weak self] (argument: repeat each Argument) in
+             defer { self?.function = { (_: repeat each Argument) in nil } }
+             return function(repeat each argument)
+         }
     }
  }

Without the diff markup that looks like this

@propertyWrapper class CallOnce<each Argument, Output> {
    private var function: (repeat each Argument) -> Output? = { (_: repeat each Argument) in nil }

    var wrappedValue: (repeat each Argument) -> Output? {
        get { function }
        set { decorate(newValue) }
    }

    init(wrappedValue: @escaping (repeat each Argument) -> Output?) {
        decorate(wrappedValue)
    }

    private func decorate(_ function: @escaping (repeat each Argument) -> Output?) {
        self.function = { [weak self] (argument: repeat each Argument) in
            defer { self?.function = { (_: repeat each Argument) in nil } }
            return function(repeat each argument)
        }
    }
}

This is pretty handy and can handle any shape of function now. If we applied this to the original code we’d have this

class Example {
    @CallOnce private var delayedWork: () -> Void? = { nil }

    func functionThatQueuesWork() {
        delayedWork = { print("some work") }
    }

    func functionThatInvokesTheWork() {
        delayedWork()
    }
}

Now if we instantiate this type, enqueue some work and repeatedly invoke it we’d only see one log

The initializer syntax is a little clunky because the wrapper expects an initial closure, even if it’s just a no-op.

let example = Example()
example.functionThatQueuesWork()
example.functionThatInvokesTheWork() //=> "some work"
example.functionThatInvokesTheWork() //=> nil
example.functionThatInvokesTheWork() //=> nil

Conclusion

I’m not sure this is actually useful but it’s always a fun exercise to try and see how you can utilise different features to craft the functionality you need. I find knowing the available primitives and having the hands on experience of piecing them together is really powerful for helping me solve problems and just generally not be phased when a challenge comes in that I’m unfamiliar with. Even if this exact wrapper never ships, the exercise paid off by forcing me to combine property wrappers and parameter packs into something concrete.

This isn’t a post about Docker, SwiftUI or Compose. It’s about what I learned building a developer tool that people actually used and what I got wrong along the way.

The Problem

For my day job I work in the native apps team at Autotrader on the iOS/platform side of things. As a team we don’t only own the Android/iOS code bases, we also own a few backend services that power the apps, the number of backend services has grown over time. This means that if you want to make changes to a backend service and test it locally in a simulator you need to spin up all the related services locally.

As a simplified example imagine this setup

+-----+     +-----------+     +-----------+
| iOS |-----| Service A |-----| Service B |
+-----+     +-----------+     +-----------+

To run this locally there are many ways I can configure things. One scenario might be that I want to make changes to Service B, which means I still need to spin up Service A to allow the communication.

In the beginning this was all done manually - you’d load up the projects for Service A and Service B in IntelliJ and run them both locally. This worked but it pushed both cognitive and operational complexity onto individual developers, which is exactly where DX debt hurts most.

First Solution

Some people will have been screaming “use docker compose” and you’d be right that’s what I did. All of our services were already containerised so there was nothing to change there. I just needed to write the docker-compose.yml to configure everything, which I did and it worked fine.

There were a few issues that made me feel uncomfortable stopping here

• Not everyone is comfortable on the command line so this can be intimidating
• Docker is possibly not a tech a lot of Android/iOS devs delve into
• The UI isn’t great

To elaborate on that last point the UI for interacting with this new setup would be one of the following:

docker compose up
docker compose up service-a
docker compose up service-b

Not everyone knows about reverse history search in bash (ctrl + r) so scraping around for these commands or pressing up a million times at the prompt isn’t the best experience. There’s also an issue with discoverability as you’d need to know the correct spelling of each service you want to run.

It seemed pretty clear that some kind of simple UI would really reduce the barrier to entry for using this tool.

Put Some UI on it

I’d dabbled in macOS apps before and honestly not enjoyed the experience too much as all my previous experience was in UIKit. Luckily SwiftUI exists so I thought I’d use it as a learning experience to make a menu bar app that essentially wraps the docker compose.

This is what that first version looked like

Yup it’s not going to win any design awards but the power of this abstraction can’t be overstated.

• We are completely abstracting away docker from the mobile app devs
• All available services are visible and just a button tap away from being started/stopped
• We show useful debug information like what ports things are running on
• We show the running status of each service and detect if running via docker or as .jar (read: most likely in IntelliJ)

This wasn’t a quick task and I ended up burning a lot of the midnight oil as it was an interesting task. There were lots of gotchas with handling subprocesses and their environments and coordinating lots of state events to the UI.

I was pretty happy with this step until I realised my first mistake…

How do I install it?

The first lesson was that building a tool isn’t good enough; you need to make it accessible. I’m not sure if it’s common knowledge but developers tend to be very lazy energy efficient and having a multi step install is just a recipe for support pain.

It wasn’t long before I added an install script so people could simply run a curl | bash and enjoy a nicely installed app without manually

• Finding the repo
• Navigating to releases
• Downloading the binary
• Doing some gatekeeper to get it out of quarantine
• Finally be able to launch the app

Obviously whenever I shared installation instructions I did the good citizen thing and issued a disclaimer that people shouldn’t blindly trust me and pipe my remotely hosted bash script into an interpreter without reading it first.

Success and Growing Pains

The tool in the form of a macOS menu bar app written in SwiftUI had a good 4 year run. New services were added and bugs were fixed and it ended up looking more like this

One fun lesson that you can see played out in the screen shot is that I added { {commit-sha} }, which is populated on a release build with the git sha. This was because people have a tendency to not update things, especially when it involved manually running a curl | bash. Although adding some version identifier was helpful for support, it was just a plaster and the actual requirement I wish I noticed earlier is that this really needed to automatically update itself.

As you can see from the growth of services the tool is popular enough to have been updated multiple times but it wasn’t popular enough to help people overcome the thought of a steep learning curve required to contribute. The project was mostly maintained by myself and in all honesty I’d made some questionable architectural decisions early on and Combine heavy state management made the learning curve steeper than it needed to be.

Another detail that I’d started to notice with the way this had grown is that a long list of services is only good if the user knows how they relate. In an evolving estate where many teams contribute it may not be very obvious what services you need to run to enable you to work. You could spin up all the services but that’s sometimes overkill and hard on our poor little CPUs.

The project needed change but needed some inspiration…

A Reimagining

So here I was thinking

• This codebase is a pain to maintain
• I want to better visualise how services hang together
• I want more people to be able to contribute
• I want the tool to autoupdate

My colleague (who doesn’t like being named) was working on a Compose Desktop app to help debug our apps. As they’d done all the hard work of getting a project scaffolded and off the ground I decided to see how hard it would be to port the SwiftUI tool to Compose, update the UI and incorporate it into this debug app.

This was actually a perfect opportunity to rethink architectural choices as it was an entirely different language and although I’d consider myself proficient in Kotlin I’d never done Compose so it would be a fun experience. Doing the work in this codebase also opened up the contributor pool considerably as now any Android dev could contribute and conveniently all of our iOS devs are solid Kotlin devs already. At this point, the limiting factor wasn’t UX polish - it was who felt capable of contributing.

After some fun learning and porting all the process management over I ended up with something like this

Personally I think I nailed the visualisation requirement as you can now at a glance see what services you’d need running to access different parts of the estate. The red connecting lines do actually go black to show that the apps can access services but I hastily made changes to get the anonymised screenshot and messed that detection up.

The other main thing with this tool was the autoupdate ability. I leant on having more contributors and got a colleague to write the autoupdate process logic and it works great. We even held back giving anyone access to the tool until the autoupdate was available as we just didn’t want to deal with the support.

I was feeling pretty good about this developer experience…

Arrgghh SSO

Then a new requirement came in in the form of services needing to get tokens to communicate to our preprod environment. Actually this wasn’t a new thing as I’d just been avoiding doing anything about it for a few years but it was being more broadly rolled out so I couldn’t ignore it anymore.

As with most things I put off, the programming part didn’t actually end up being super complicated. Essentially when a service requires an auth token we need to invoke an external helper tool that does all the SSO magic and then inject the returned token into the relevant service when it is started. The more complicated part was figuring out how the UI would educate users that they might need to get permission to do this for each service and then signpost how/where they do that.

Once a user is setup they just press the play button like normal and everything is automatic. If the user isn’t configured then they get an ⚠️ on the service, which tells them how to resolve the misconfiguration.

Make sure you are listening

Another lesson that I completely missed was tuning into the frequency of support requests. Behind the scenes to get all of the networking setup correctly to allow docker to communicate with localhost seamlessly people need to add an entry to /etc/hosts. This was constantly missed but quickly resolved with a trip to the docs. Unfortunately no one knew where the docs were so I’d just be dishing out the link constantly.

Instead of redirecting people to the docs and then helping them through the steps I decided to add a “Doctor” command that will diagnose common misconfigurations/issues and give remediation steps.

Build it Remove friction and they will come

I think there’s a few good victory stories on getting more contributors. Some that come to mind are:

A massive eye saver from one of the iOS devs was adding dark mode. A pretty impressive feat considering he’d never done Compose and actually didn’t just tweak colours he went full hog and made various components much more pleasing on the eyes.

Another capability added by a few iOS devs was the ability to install the latest development build into your simulator and launch it. This is a massive productivity boost especially for web developers who now don’t need to learn how to get the project running and interact with Xcode. There is still a requirement to have Xcode installed but after that it’s zero knowledge.

One of the QA engineers added hot reloading to make the development experience on the tool that improves developer experience better.

Building a platform

Another observation that we made was that we now have a platform to build out tools that are automatically updating and installed by many people. This has lead to me sitting with another QA engineer to help them port their command line tool for finding all kinds of difficult to find adverts that people need for testing/development.

I think in total we spent about an hour getting something working together and then he’s been a one man feature factory improving things.

Wrap up

I really think developer experience is important. Making and evolving tools is a nice way to show your colleagues that you value their time and hear their frustrations. It’s also been highly rewarding seeing other people get involved and seeing people interested in building a DX culture. I have no metrics on how much time this tool has saved others but I personally use it multiple times a day and the time I’ve saved alone has more than paid off for the personal investment I’ve put into this.

This post is about one of those loops: a small problem in some of our KSP generated code. I’ll walk through a few iterations of the solution and end with a much simpler approach that let the compiler do the hard work instead of us.

The problem

We have some KSP (Kotlin Symbol Processor) code that generates helper functions for routing within our application. Inside the generated functions we were blindly taking in arguments and attempting to serialize them with kotlinx.serialization e.g.

fun routeToCheckout(cart: Cart) {
    val encodedCart = Json.encodeToString(cart)
    ...
}

The issue here is that if Cart is not annotated with @Serializable then this code will explode at runtime, which is less than ideal.

Solutions

Your friend and mine (your LLM of choice) suggested explicitly taking a serializer at the call site. This would force the caller to guarantee that a serializer exists.

In practice though, it felt wrong. Requiring consumers of this generated API to manually thread serializers through their code makes the API harder to use and leaks an implementation detail that callers shouldn’t need to care about.

The other suggestion from the LLM was to use a reified inline function but that was not an option based on the code setup.

Attempt 1

Using the suggestions from the LLM I thought maybe we could just blindly generate the serializer into the body of the function and then the compiler would reject our code if the serializer didn’t exist e.g.

fun routeToCheckout(cart: Cart) {
    val encodedCart = Json.encodeToString(Cart.serializer(), cart)
    ...
}

This works as the compiler will now error if Cart.serializer() does not exist, which is much better than a runtime exception. Granted this does have a less than ideal failure mode as the consumer of this KSP processor could end up with their code not compiling and being pointed to generated code. Whilst not great I was happy with the compromise and we can generate a comment as well to help steer anyone who does end up here e.g.

fun routeToCheckout(cart: Cart) {
    /*
     * If you find yourself here with a compilation error, ensure that the relevant
     * type has a serializer defined.
     */
    val encodedCart = Json.encodeToString(Cart.serializer(), cart)
    ...
}

Never smooth sailing

I applied this latest code to a larger code base and it immediately flagged the code that caused the original issue that prompted this investigation in the first place, which was reassuring. It also highlighted a few other call sites that would exhibit the same breakage but luckily those code paths weren’t executed. More annoyingly the new code found that we sometimes pass more complex types that use Kotlin collections e.g. we had types like List<String> or Map<String, CustomType>. Obviously I didn’t see the wood for the trees and started “making it work” but it got real ugly real quick with all the potential nesting. The serializers in the above cases would be

ListSerializer(String.serializer())
MapSerializer(String.serializer(), CustomType.serializer())

To do this I started thinking about writing a recursive function that would keep traversing through the types building up these serializers and special casing the various Kotlin collection types.

Could we do it simpler?

Luckily at this point I’d already asked my colleague JackMack for his thoughts, which was could we do it simpler?. The key insight he’d had after hearing me ramble on about my current progress was that we don’t actually need to reproduce the exact serializer to pass to Json.encodeToString; the root of the problem is that we want to prove that each type mentioned has a serializer.

The new idea was to simply list out all the types outside of the Json.encodeToString function and let the compiler just do its thing. So essentially for the Map<String, CustomType> example the target is something like

fun routeToPage(customTypes: Map<String, CustomType>) {
    /*
     * If you find yourself here with a compilation error, ensure that the relevant
     * type has a serializer defined.
     */
    CustomType.serializer()

    val encodedCustomTypes = Json.encodeToString(customTypes)
    ...
}

We don’t need to worry about checking String.serializer() or MapSerializer exist because we know the library provides those.

Getting our prompt on

Once the target shape was clear, the remaining work became much more mechanical. We needed a way to walk the types involved in a function signature, including any type parameters and extract the set of domain types we cared about.

This was ideal for a quick human/LLM pairing session.

Conclusion

This problem went through several iterations, each one technically “working” but increasingly complex. The turning point came from stepping back and questioning what we were really trying to achieve.

We didn’t need to construct the correct serializer. We didn’t need to mirror kotlinx.serialization’s internal rules. We just needed proof at compile time that the types flowing through our generated APIs were @Serializable.

By narrowing the problem to that single requirement, the solution became smaller, clearer and more robust. It also produced better failures: early, explicit and enforced by the compiler rather than discovered at runtime.

It’s a useful reminder that when a solution starts to grow unwieldy, the answer is often not more code, but a better question.

The best part is that this is now another pattern my brain can store away and recognise more quickly in the future, reducing the number of iterations when I hit similar problems again.

Types

class Example {
    val title: String? = null
}

data class Id(val value: String)

Usage

val example = Example()

example.title?.let {
    Id(example.title)
}

At first glance, this should compile - and it does. But we can make small, seemingly harmless tweaks that suddenly break it.

Failing to Compile

The first breaking change would be to make title a computed property

  class Example {
-     val title: String? = null
+     val title: String? get() = null
  }

With this change, we get the following error:

Smart cast to ‘String’ is impossible, because ‘title’ is a property that has an open or custom getter.

This error rather cunningly suggests another change that also causes it to fail.

-  class Example {
-     val title: String? = null
+  open class Example {
+     open val title: String? = null
  }

One final change I can think of that breaks for the same underlying reason but is achieved in a different way is to declare Example in a different module from the usage code. This gives the error

Smart cast to ‘String’ is impossible, because ‘title’ is a public API property declared in different module.

So what’s going on here?

Failing to Smart Cast

We’ve seen Smart cast mentioned in both errors but what does that mean? A smart cast is when the Kotlin compiler automatically treats a variable as a non-null or a more specific type after it’s checked - but only if it can guarantee the value won’t change in the meantime.

In the original working code, the compiler can see that Example.title is declared as val and cannot be reassigned. So inside the scope of the ?.let the compiler is able to prove that the value cannot change.

All these breaking changes are just different ways of preventing the compiler from making that guarantee.

There’s another subtle language feature that allows us to write this code without realising the mistake.

Ignoring Closure Arguments

Kotlin allows you to silently ignore closure arguments. This contrasts with languages like Swift, which require you to explicitly mark when you’re ignoring one. For example in Swift we cannot implicitly ignore closure arguments so the compiler would force us to make this change

- example.title?.let {
+ example.title?.let { _ ->
      Id(example.title)
  }

Thinking about this actually points to the root issue, we should use the argument passed to the closure rather than doing the example.title access again.

Recommendation

Now that we understand why, the fix should be clearer. Instead of relying on a Smart cast and ignoring the lambda argument, we should just use the closure argument it directly: This means our call site would change like this

  example.title?.let {
-     Id(example.title)
+     Id(it)
  }

Personally, I would go a step further and favour the more concise call site:

- example.title?.let {
-     Id(it)
- }
+ example.title?.let(::Id)

This is an example of point-free style.

Using Point Free Style

Not everyone is a fan of method references because :: looks odd and scary at first. Once you get used to it, it’s really handy for writing super concise code.

It’s subtle but the second line here packs a punch.

example.title?.let { Id(it) }
example.title?.let(::Id)

In essence we are taking the data from let and plumbing it straight into the Id constructor. In the first line we are doing this manually by providing a closure and then invoking the Id constructor. In the second line we are just composing these functions together.

It’s subtle, especially in such a short example. With the first line, I have to mentally parse the closure, verify how it is used, and ensure it isn’t modified before being passed to Id. With the second line I don’t have to do any of that - I just know that the let will feed the value straight into Id.

Conclusion

Keeping in mind that the original code listing worked, we could arguably just not worry about any of this. I personally think the end code is simpler and more descriptive about the intent but that can be debated. Also the original code has the issue that unrelated changes can start breaking things, which is something that I think we should always try to avoid. This is definitely one of those changes that I would suggest in a pull request but feel guilty about not providing a thorough explanation of the reasoning. This post gives me extended notes I can point to when explaining the change.

This post explores how to remove duplication when two functions share logic but act on unrelated types with no shared interface - using Kotlin and Swift as examples.

Let’s look at a more challenging case.

The Problem

The algorithm itself isn’t important here, but consider the following starting point:

struct InputA { let name: String; let tag: String? }
struct InputB { let name: String; let tag: String? }

struct Output {
    let name: String
    let tag: String

    init(inputA: InputA, tag: String) {
        self.name = inputA.name
        self.tag = tag
    }

    init(inputB: InputB, tag: String) {
        self.name = inputB.name
        self.tag = tag
    }
}

func processInputAs(inputs: [InputA]) -> [Output] {
    inputs
        .compactMap { input in
            input.tag.map { Output(inputA: input, tag: $0) }
        }
}

func processInputBs(inputBs: [InputB]) -> [Output] {
    inputBs
        .compactMap { input in
            input.tag.map { Output(inputB: input, tag: $0) }
        }
}

data class InputA(val name: String, val tag: String?)
data class InputB(val name: String, val tag: String?)

data class Output private constructor(val name: String, val tag: String) {
    constructor(inputA: InputA, tag: String) : this(inputA.name, tag)
    constructor(inputB: InputB, tag: String) : this(inputB.name, tag)
}

fun processInputAs(inputs: List<InputA>): List<Output> {
    return inputs
        .mapNotNull {
            it.tag?.let { tag -> Output(inputA = it, tag = tag) }
        }
}

fun processInputBs(inputs: List<InputB>): List<Output> {
    return inputs
        .mapNotNull {
            it.tag?.let { tag -> Output(inputB = it, tag = tag) }
        }
}

As shown above, both processInputAs and processInputBs are nearly identical. The issue is that these two functions operate on different input types with no common supertype. This means we can’t write a simple generic function with a constrained type parameter, because we have no common supertype to constrain against.

So how might we solve this duplication if our types did share a common shape?

When a Shared Constraint Exists

If InputA and InputB did have a common supertype we could trivially create a single function that handles them both.

protocol Input {
    var name: String { get }
    var tag: String? { get }
}

struct InputA: Input {
    let name: String
    let tag: String?
}

struct InputB: Input {
    let name: String
    let tag: String?
}

struct Output {
    let name: String
    let tag: String

    init(input: Input, tag: String) {
        self.name = input.name
        self.tag = tag
    }
}

func processInputs(inputs: [Input]) -> [Output] {
    inputs
        .compactMap { input in
            input.tag.map { Output(input: input, tag: $0) }
        }
}

interface Input {
    val name: String
    val tag: String?
}

data class InputA(override val name: String, override val tag: String?): Input
data class InputB(override val name: String, override val tag: String?): Input

data class Output private constructor(val name: String, val tag: String) {
    companion object {
        operator fun <T: Input> invoke(input: T, tag: String) = Output(input.name, tag)
    }
}

fun processInputs(inputs: List<Input>): List<Output> {
    return inputs
        .mapNotNull {
            it.tag?.let { tag -> Output(input = it, tag = tag) }
        }
}

Unfortunately, in our original framing, InputA and InputB share no common supertype. This can occur in situations where you don’t own the types as they come from a third-party library or maybe the types are generated with something like OpenAPI.

When I hit this situation, I tend to look at the two functions side by side and highlight the differences.

Reveal The Differences

For these functions the differences are going to be:

• The type of the argument
• How we access tag (keep in mind the code looks the same but InputA and InputB are distinct types)
• How we construct Output.

In the listing below I’ve highlighted these areas with multiple ?s.

func processInputs(inputs: [??????]) -> [Output] {
    inputs
        .compactMap { input in
            input.???.map { ??????(input: input, tag: $0) }
        }
}

fun processInputs(inputs: List<?????>): List<Output> {
    return inputs
        .mapNotNull {
            it.????.let { tag -> ??????(input = it, tag = tag) }
        }
}

Solving the Problem

For the first point, we can’t avoid not knowing the type - there’s nothing common to constrain to. So we’ll have to just introduce a type parameter.

func processInputs<T>(inputs: [T]) -> [Output] {
    inputs
        .compactMap { input in
            input.???.map { ??????(input: input, tag: $0) }
        }
}

fun <T> processInputs(inputs: List<T>): List<Output> {
    return inputs
        .mapNotNull {
            it.????.let { tag -> ??????(input = it, tag = tag) }
        }
}

Next, we need to teach the function how to read the tag value. As we only know we have a T we can provide a function that takes a T and returns the String? we expect.

func processInputs<T>(
    inputs: [T],
    tag: (T) -> String?
) -> [Output] {
    inputs
        .compactMap { input in
            tag(input).map { ??????(input: input, tag: $0) }
        }
}

fun <T> processInputs(
    inputs: List<T>,
    tag: (T) -> String?
): List<Output> {
    return inputs
        .mapNotNull {
            tag(it)?.let { tag -> ??????(input = it, tag = tag) }
        }
}

Finally, we need to teach the function how to create the new outputs.

func processInputs<T>(
    inputs: [T],
    tag: (T) -> String?,
    makeOutput: (T, String) -> Output
) -> [Output] {
    inputs
        .compactMap { input in
            tag(input).map { makeOutput(input, $0) }
        }
}

fun <T> processInputs(
    inputs: List<T>,
    tag: (T) -> String?,
    makeOutput: (T, String) -> Output
): List<Output> {
    return inputs
        .mapNotNull {
            tag(it)?.let { tag -> makeOutput(it, tag) }
        }
}

With this helper function in place we can update our original processInputAs and processInputBs functions to call through to the shared code.

Pulling it Together

func processInputAs(inputs: [InputA]) -> [Output] {
    processInputs(inputs: inputs, tag: \.tag, makeOutput: Output.init)
}

func processInputBs(inputs: [InputB]) -> [Output] {
    processInputs(inputs: inputs, tag: \.tag, makeOutput: Output.init)
}

private func processInputs<T>(
    inputs: [T],
    tag: (T) -> String?,
    makeOutput: (T, String) -> Output
) -> [Output] {
    inputs
        .compactMap { input in
            tag(input).map { makeOutput(input, $0) }
        }
}

fun processInputAs(inputs: List<InputA>): List<Output> {
    return processInputs(inputs, InputA::tag, Output.Companion::invoke)
}

fun processInputBs(inputs: List<InputB>): List<Output> {
    return processInputs(inputs, InputB::tag, Output.Companion::invoke)
}

fun <T> processInputs(
    inputs: List<T>,
    tag: (T) -> String?,
    makeOutput: (T, String) -> Output
): List<Output> {
    return inputs
        .mapNotNull {
            tag(it)?.let { tag -> makeOutput(it, tag) }
        }
}

Conclusion

We don’t always need shared supertypes to make algorithms generic. With a little upfront work teaching the algorithm how to access and create data, we can operate on unrelated types just as effectively. Both Kotlin and Swift make this especially clean through their support for passing method and initializer references - keeping our higher-order functions readable and expressive.

Ruthless Simplification with DYSNI

At the day job, my colleagues Adam and Ellen (aka the only two who foolishly raised their hands when I asked for volunteers to help remove some Objective-C) and I were doing some Objective-C cleanup.

We still have a portion of Objective-C in our codebase, but we’re aiming to remove it. The team’s Objective-C skills aren’t being exercised and risk atrophying, new hires rarely have experience with it, and its presence complicates adopting modern technologies such as Swift Concurrency.

The first instinct when migrating from Objective-C is usually to just port the code like for like. There’s often an attempt to make things feel more Swifty, but ultimately the original structure tends to linger.

By asking DYSNI on repeat, we were able to take a pretty gnarly abstraction and simplify it right down.

The Problem

The starting point we had was this lovely structure

     +------------------------+
     |      CommandBlock      |
     +------------------------+
     | +executeWithCompletion |
     +------------------------+
        ^                  ^
       /                    \
      /                      \
+----------+            +----------+
| CommandA |            | CommandB |
+----------+            +----------+

The signature of the completion handler was void (^)(NSDictionary *userInfo, BOOL success, NSError *error) (don’t we all just miss the Objective-C syntax?). The way this was being used is that there was a view controller that held a reference to the current command

@property (nonatomic, strong) CommandBlock *currentCommandBlock;

The currently executing block would be stored in this ivar until it completed its work and then it was nil‘d out.

The first DYSNI

We started in possibly the worst place but hey that’s how reality is sometimes. We asked Do Ya Still Need It whilst looking at this hierarchy. The thinking was that if you have a class with a single method, it could be represented as a single anonymous function. So instead of holding onto the CommandBlock instance itself we could just hold on to the function reference execute(completion:) with a view to removing the base class at some point.

Why was this a bad starting point? Because we had to remember Objective-C block syntax… on the second attempt, we got it working and made the following change

- @property (nonatomic, strong) CommandBlock *currentCommandBlock;
+ @property (nonatomic, strong) void (^action)(void (^)(NSDictionary *userInfo, BOOL success, NSError *error));

Just as we were about to update everything, we paused and asked DYSNI again…

The second DYSNI

Before updating all the code, we wondered if we should check all the callers to see what userInfo is even being used for. Upon tracing things through for a bit we found that precisely zero callers were using userInfo, so we made the change

- @property (nonatomic, strong) void (^action)(void (^)(NSDictionary *userInfo, BOOL success, NSError *error));
+ @property (nonatomic, strong) void (^action)(void (^)(BOOL success, NSError *error));

That worked well and we kind of assumed that success and error would be used. Thankfully that assumption didn’t stop us asking DYSNI.

The third DYSNI

We questioned who’s using error so we did the same dance but this time we found one caller was actually reading the error. Out of curiosity, we followed that caller through and found they didn’t actually do anything with error so

- @property (nonatomic, strong) void (^action)(void (^)(BOOL success, NSError *error));
+ @property (nonatomic, strong) void (^action)(void (^)(BOOL success));

We had a flow going so you guessed it…

The fourth DYSNI

We had to ask is anyone using success or do callers just want to understand when the operation has finished? In this case success is required so we couldn’t delete it 😢 but never fear because we made things much simpler already. Thanks to those small DYSNI checks, no future readers of the code would need to ask the same questions we just have.

The fifth DYSNI

At this point we’d spent a fair bit of time in the weeds looking at this function so we zoomed out a bit and asked are these commands even required at all? What we found was that at some point all uses of CommandA had been removed from the codebase but this was never tidied up. This unlocked many more opportunities to simplify, for starters we got our delete on and went from

     +------------------------+
     |      CommandBlock      |
     +------------------------+
     | +executeWithCompletion |
     +------------------------+
        ^                  ^
       /                    \
      /                      \
+----------+            +----------+
| CommandA |            | CommandB |
+----------+            +----------+

to

+----------+
| CommandB |
+----------+

The sixth DYSNI

With this simplification done we went back and asked the question we didn’t want to ask but knew we should, do we still need the function signature we spent ages messing around with? The answer was no we do not - it turns out that we don’t know why there was an instance variable in the first place. Our best guess is that the original developers assumed they needed to keep a reference to the command whilst it was executing and then nil it out on completion. In reality, the command is a one shot deal that keeps itself alive until the work is completed then it would naturally go out of scope.

With this knowledge we said goodbye to the ivar

- @property (nonatomic, strong) void (^action)(void (^)(BOOL success));

Instead we just new up an instance of the command and invoke it

[CommandB.new executeWithCompletion:^(BOOL success) {
    // do some stuff
}];

*Controversial use of dot syntax on the new there 👀.

Conclusion

Asking the question “Do Ya Still Need It?” can be a surprisingly powerful tool. The story above resulted in a diff of 126 insertions(+), 928 deletions(-), which is a great result. Not only did we reduce line count, but we also encoded our new understanding of the problem into more modern approaches, which will hopefully be easier for future readers to pick up.

Sometimes, when I ask DYSNI, I get the sense people think I’m being lazy or annoying on pull requests but ruthlessly simplifying things down means less work now and less complexity to unpick later.

The Basic Command

I use this formulation of the git rebase command

             (1) The commit where we branched from
              |
              |           (2) The branch we want to rebase on top of
              |            |
              |            |              (3) The flag to keep merge bubbles
              |            |               |
           .------. .-------------. .-------------.
git rebase old_base --onto new_base --rebase-merges

1. By providing the old_base explicitly we avoid scenarios where git gets confused¹.
2. The branch we want to replay all of our commits on top of.
3. This keeps the empty commits that create merge bubbles.

Keeping merge bubbles is seemingly another contentious topic but I find them valuable. For example with this listing using git lol² I can see that feature1 was potentially less complicated than feature2 because it required less commits.

*   cb08e97c1a Merge pull request #2 from feature2
|\
| * a5c310e392 Implement feature2
| * e07178d052 Refactor to make feature possible
|/
*   3fe7557433 Merge pull request #1 from feature1
|\
| * 07b845a110 Implement feature1
|/
*

* The branch names/commit messages in these examples are not good examples of naming/describing but I’ve kept them short to keep the example small.

This view also allows me to know what commits would need reverting if I want to back out a feature.

Verifying the Result

Clean Rebase

If the rebase was clean and there were no conflicts that I had to resolve, I tend to verify that the result is good by diffing between my local branch and the remote. For this I have another alias git dfr³ (short for diff remote). A successful result would essentially just contain a diff showing the changes that went into the base branch after the point the current branch was forked.

This breaks down when rebasing a branch that has gotten quite out of date with the new base. Keep in mind that the diff includes all the changes that went into the new base branch. This can produce a lot of output, and if you weren’t the one who made those changes, it can be tricky to reason about.

Rebase that had merge conflicts

When I’ve had to resolve merge conflicts during the rebase the above diff isn’t very helpful because the changes dealing with the merge conflicts are mixed in with the changes that went into the new base branch. To get a better view of things I reach for git range-diff old_base..old_head new_base..new_head.

What this command does is it tries to find the same commit in both ranges using heuristics like commit message. It then creates a diff between each pair of commits. The output of this command is a little hard to read because there are potentially two levels of ± indicators in the gutter. Persevere and it will make sense especially if you have coloured output in your terminal.

Fixing when it goes wrong

Using the verification steps above, I sometimes discover that I’ve messed up a merge conflict. I’d rather try and fix the broken commit itself over adding a new commit. To achieve this I reach for an interactive rebase following these steps:

• Find the SHA of the parent for the broken commit
• Run git rebase --interactive parent_sha --rebase-merges
• In the text editor find the sha I want to edit and change its option to e/edit
• Follow the usual process of a rebase to step through the reapplication of commits

If you’ve ever used git rebase -i before you’ll notice that adding the --rebase-merges flag really steps up the difficulty level. For simple edits it’s easy enough to just ignore the commands like label, reset, merge etc and concentrate on the normal options you may be used to.

Starting again

Sometimes stuff really goes wrong and it’s time to admit defeat (for now) and call git rebase --abort. Even after years of rebasing, I still end up here a lot. It’s not a sign of failure - usually it just means the first attempt went wrong and I’ve learned what to do differently next time.

Pushing changes

I always use the --force-with-lease flag when pushing changes as it’s slightly safer than plain --force. Essentially --force-with-lease bails out if git notices that your copy of the remote is not up to date. This reduces the chances of you clobbering someone else’s work because you’d need to do a git fetch to get the latest changes and then resolve any conflicts locally.

Preventing pain

To reduce painful rebases it’s good practice to rebase early and often. The longer you leave branches to diverge the more chances you have of getting conflicts so integrating as early as possible is beneficial.

Conclusion

The tips and tricks above have taken years to figure out. As well as knowing the commands I think practicing and trial/error are really the only way you get better at this stuff so don’t be afraid to get stuck in.

The Problem

I work on a mobile app team that owns parts of the backend estate. We often need spin up multiple web services locally and point our simulator at them. We’ve built a GUI that reduces all the complexity of configuring these services down to a few play buttons.

This worked well for years until we started needing to talk to various services using OAuth. To boot these services we now need to obtain a JWT and inject it into the container.

Obtaining the JWTs is easy as our delivery platform team provide helpful tooling. Where we hit pain is that developers are used to starting these services and forgetting about them, potentially having them run for days. This doesn’t work so well now with the JWTs having short expiry times meaning that services start to fail in interesting ways that may not be super obvious to debug.

The Solution

I solved this by adding a global pop up that shows which service’s token has expired and provides a restart button. I could have automatically restarted services but it feels more user friendly to let developers decide when this is going to happen. It looks like this:

* The image capture software I used makes this pop up look novelty sized, it’s not this big in reality but it made me chuckle so I’ve left it in.

Design notes on the behaviour of the pop up

• It sits on top of all windows
• It can be dismissed but comes back if you activate the application again
• If more than one service needs restarting it offers a single button to restart them all

How does it work technically?

I’m glad you asked… as the services are all running in docker we can run docker inspect <container-id>, which returns a load of helpful information about the running container. Nestled away under Config > Env is all the environment variables that were provided to run the docker container. We can grab the environment variable that contains the JWT and then decode the payload to get access to the expiry information.

Our tooling is built using Kotlin Compose Desktop so these code snippets are for Kotlin. We start by declaring a @Serializable type that represents the payload and use a handy custom serializer to get the expiresAt value in the right format.

@Serializable
private data class OAuthPayload(
    @Serializable(with = DateSerializer::class) val expiresAt: LocalDateTime,
)

With this type declared we need to unpack the JWT, decode the base64 value and then parse the JSON body.

val (_, payload, _) = it.split(".")
json.decodeFromString<OAuthPayload>(payload.decodeBase64String()).expiresAt

At this point we know when the token expires and we can just schedule for the UI to appear at the right time.

The Result

As a developer I always get a sense of satisfaction when I tap the button and it automagically gets me a new token and restarts my service. I knew the pain of manually getting a token and then trying to remember the right docker commands to stop the running service, export my fresh new JWT then restart the service so I enjoy seeing the friction removed.

As someone supporting other developers, I love that I can’t remember the last time someone came to me that had been tripped up by expired tokens.

Conclusion

Not everything has to be an engineering master piece it just has to solve a problem. These OAuth tokens were a real usability issue for developers who are mainly focussed on the front end, where running backend services locally is more of a secondary concern. The best developer tools don’t just automate they anticipate where developers might trip up and quietly save the day.

A common pattern I keep seeing in various codebases I work on is that data transfer objects are being defined using default arguments in their constructors. I think this leads to a few issues that I’ll explore in this post.

A Simple Example

Here’s a typical example of a class with a default argument on tags.

struct BlogPost {
    let title: String
    let tags: [String]

    init(title: String, tags: [String] = []) {
        self.title = title
        self.tags = tags
    }
}

data class BlogPost(
    val title: String,
    val tags: List<String> = emptyList()
)

It’s not always clear why this is done. I suspect it’s often out of habit or convenience for testing. My suspicion that this is related to testing comes from seeing this pattern repeatedly even though the production code is explicitly providing every argument and therefore never makes use of the defaults but the tests do. It gets my spidey senses tingling when it feels like we are weakening our production code in the service of adding tests.

The unintended consequence of these defaults is that the compiler can no longer be as helpful.

Exhaustivity Tangent

Just to make sure we are on the same page let’s talk about exhaustivity checking with enums as hopefully people will have experience with this. If we declare an enum and later switch over its cases the compiler can check to make sure we cover every case (assuming we don’t add a default case.)

For example let’s start with a PostStatus with two cases

enum PostStatus {
    case draft
    case published
}

func outputFolder(status: PostStatus) -> String {
    switch status {
        case .draft: "/dev/null"
        case .published: "blog"
    }
}

enum class PostStatus {
    Draft,
    Published
}

fun outputFolder(status: PostStatus): String {
    return when (status) {
        PostStatus.Draft -> "/dev/null"
        PostStatus.Published -> "blog"
    }
}

If we add a third case of archived then the compiler will force us to revisit our outputFolder function as it’s no longer exhaustive:

enum PostStatus {
    case archived
    case draft
    case published
}

func outputFolder(status: PostStatus) -> String {
    switch status { // Error -> Switch must be exhaustive
        case .draft: "/dev/null"
        case .published: "blog"
    }
}

enum class PostStatus {
    Archived,
    Draft,
    Published
}

fun outputFolder(status: PostStatus): String {
    return when (status) { // Error -> 'when' expression must be exhaustive. Add the 'Archived' branch or an 'else' branch.
        PostStatus.Draft -> "/dev/null"
        PostStatus.Published -> "blog"
    }
}

This is great because it means the compiler will guide us step by step through every callsite so we can decide what the appropriate action to take is.

Missing Exhaustivity 😢

If we agree that exhaustivity checking is a good thing then we can extend the same logic to the first example. Let’s say we create instances of our BlogPost type in a few places in our codebase and we want to add a new property of isBehindPaywall. If we add a default value then the compiler doesn’t help us by highlighting all the callsites that we should reconsider. If we are lucky then we make the change and all is fine, if we aren’t so lucky then we could accidentally have blog posts being hidden/shown when they are not supposed to be. In this case I’d much rather the compiler makes me check every callsite so I can make the correct decision.

In practice all this means is that I have to explicitly specify the isBehindPaywall argument and accept that there might be some duplication:

// Explicit
BlogPost(title: "Some blog post", isBehindPaywall: false)

// Implicit
BlogPost(title: "Some blog post")

// Explicit
BlogPost(title = "Some blog post", isBehindPaywall = false)

// Implicit
BlogPost(title = "Some blog post")

Local Reasoning

The explicit version above has another strength to it that is due to the improved local reasoning. If I want to know how the isBehindPaywall state was decided I can simply look at the callsite that instantiated the instance. In the defaults case this isn’t as simple - first I need to look at the callsite then if no value was provided I need to look up the declaration of BlogPost. With IDEs that click through this might not seem like a hardship but it also means we are vulnerable to changes being made at a distance e.g. someone could change the default value and it could have wide ranging side effects to any callsite that didn’t explicitly add a value.

Discoverability

You might think it’s fine when I add the property I’ll go and check every callsite myself manually. This is all well and good if you are the sole owner of the codebase and if you aren’t publishing your code as a library. But bear in mind it’s not a permanent fix as anyone can come along and create an instance of BlogPost and they may or may not see the isBehindPaywall option, which means they could get it wrong.

The issue is when people come to create instances of our BlogPost type they can get away without providing a value for isBehindPaywall and be blissfully unaware it even exists or its impact e.g.

BlogPost(title: "My Blog post")

BlogPost(title = "My Blog post")

Respecting Boundaries

Another subtle issue is whether something like a data transfer object should even have this knowledge bestowed upon it or if some code that has the business rules should be in charge.

Consider this scenario I had recently:

I have a backend service that supplies data for Android and iOS clients. The backend uses kotlinx.serialization, iOS uses some legacy JSONSerialization code and Android is using Gson.

       +---------+
       | Backend |
       | kotlinx |
       +---------+

+---------+  +-------------------+
| Android |  |        iOS        |
|   Gson  |  | JSONSerialization |
+---------+  +-------------------+

With this setup we have 3 different code bases that are bound by an informal contract of what the JSON should look like. Each platform is using different libraries to encode/decode and could have subtle differences in how this is done.

We also have a Kotlin Multiplatform library so it makes sense to refactor like this:

• Extract the code from the backend service to the Kotlin Multiplatform module
• Utilise this kotlinx.serialization based code from all 3 places

       +-----------------------+
       | kotlinx.serialization |
       +-----------------------+
            ^      ^      ^
           /       |       \
          /   +---------+   \
         |    | Backend |    |
         |    +---------+    |
         |                   |
       +---------+    +---------+
       | Android |    |   iOS   |
       +---------+    +---------+

We still have 3 code bases that have to agree on how the JSON is structured but now that is handled in a more concrete way by providing the type and the encoding/decoding logic in a shared library.

With this refactor originally the type that lived in the backend service had default values encoded into it but with this new split it doesn’t really make sense. The Android/iOS clients are supposed to be totally dumb and just trust the data handed to them but the original type knows too much with defaults being baked in. It makes much more sense to strip the defaults and keep the business rules on the server populate these values, which means that the type is a simple as possible.

Conclusion

It may seem like I’m beating on default arguments but in reality I use them all the time. My main point is before adding defaults, ask what you might lose. Sometimes, explicit arguments add a little duplication but make your code safer, more discoverable and easier to reason about.

Something that I’ve been doing for a while but hadn’t really thought about was using Xcode’s Refactor > Convert to Regex Builder as a way to explain a regular expression without having data leave my machine. Yes, it’s a shocker but one of Xcode’s refactoring tools actually works for me. It works surprisingly well and as a bonus gets those endorphins flowing knowing that my data is safe from some AI drivel.

Here’s what this looks like in practice using an example of decoding a regex for UK postcodes taken from this Stackoverflow post.

let postcodeRegex = /^([a-zA-Z]{1,2}[a-zA-Z\d]{1,2})\s(\d[a-zA-Z]{2})$/

After running Refactor > Convert to Regex Builder we get

Regex {
    /^/
    Capture {
        Regex {
            Repeat(1...2) {
                CharacterClass(
                    ("a"..."z"),
                    ("A"..."Z")
                )
            }
            Repeat(1...2) {
                CharacterClass(
                    ("a"..."z"),
                    ("A"..."Z"),
                    .digit
                )
            }
        }
    }
    One(.whitespace)
    Capture {
        Regex {
            One(.digit)
            Repeat(count: 2) {
                CharacterClass(
                    ("a"..."z"),
                    ("A"..."Z")
                )
            }
        }
    }
    /$/
}

Depending on your level of experience with regex you might think this is actually more wordy/overkill (in this case I’d agree) but the point is each part of the regex is broken out into smaller parts that have names explaining what they are doing. Normally concepts like Capture, Repeat and CharacterClass require you to understand regex syntax but the builder makes them much more approachable. With the builder style a dev with no experience with regex could make sense of this without needing to look up regex rules and how they are applied.

Wrap up

Even if it wasn’t designed as an “explain this regex” feature, that’s exactly how I use it and it’s a lifesaver when revisiting old code.

What if instead, your build just emitted special log lines, and a wrapper tool noticed them and took action? That way your CI stays simple, projects don’t need extra secrets, and you get rich behaviour like PR comments or Slack alerts “for free.”

One way around this is to do something similar to what Xcode does where if I emit a log like warning: This is a warning it will furnish the UI with a nice warning triangle. So in our case if we provide an executable that knows all about pinging slack, GitHub and other services we care about we can have this program wrap our build script and look for special logs. When it sees a log it knows how to handle it can perform the right action. With this new parent process being responsible for executing our build script we can choose to strip out any environment variables we don’t want to share, meaning the parent process can be auth’d to talk to slack and the child will know nothing about it.

Less writing more writing code

Thankfully swift-subprocess has us pretty nicely set up for running the child process and parsing logs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import Foundation
import Subprocess

let terminationStatus = try await run(
    .path("/bin/bash"),
    arguments: ["-c", ProcessInfo.processInfo.arguments.dropFirst().joined(separator: " ")],
    error: .standardError
) { execution, standardOutput in
    for try await line in standardOutput.lines() {
        // Parse the log text
        print(line, terminator: "")
    }
}.terminationStatus

switch terminationStatus {
case let .exited(code):
    exit(code)
case let .unhandledException(code):
    exit(code)
}

Let’s unpack the interesting bits:

• Lines 4-8 are going to execute the command we pass in a bash subprocess.
• Line 6 is dropping the first argument as this will be the path to the executable. The rest is the command we want to execute.
• Line 7 ensures that stderr isn’t dropped on the floor as we want our command to be as transparent as possible.
• Lines 9-12 are where the magic is going to happen soon.
• Line 11 is just printing stdout otherwise our program will appear to just swallow all output.
• Lines 15-20 are just taking the result of the subprocess and making it the final result.

I haven’t explored the full API surface yet but this approach seems reasonable.

At a high level, we will invoke our new executable that I’ll call log-commander with our normal build script something like this

log-commander bundle exec fastlane build

Under the hood log-commander will essentially execute /bin/bash -c "bundle exec fastlane build" and then proxy all output and the final result.

Let’s make it do something interesting

As it stands we’ve not achieved much so let’s teach log-commander to comment on a PR. We’ll assume that log-commander will be provided the GITHUB_AUTH, GITHUB_ISSUE_ID, GITHUB_OWNER and GITHUB_REPO as environment variables. With this we can create a simple client to post a comment to a PR on GitHub

class GitHubClient {
    private let baseURL: URL
    private let session: URLSession

    init() {
        let sessionConfiguration = URLSessionConfiguration.default
        sessionConfiguration.httpAdditionalHeaders = [
            "Accept": "application/vnd.github.v3+json",
            "Authorization": "Bearer \(getOrFatalError("TOKEN"))",
            "X-GitHub-Api-Version": "2022-11-28",
        ]
        self.session = .init(configuration: sessionConfiguration)
        self.baseURL = URL(string: getOrFatalError("URL"))!
            .appending(components: "repos", getOrFatalError("OWNER"), getOrFatalError("REPO"))
    }

    func postComment(_ message: String) async throws {
        let url = baseURL.appending(components: "issues", getOrFatalError("ISSUE_ID"), "comments")
        var urlRequest = URLRequest(url: url)
        urlRequest.httpMethod = "POST"
        urlRequest.httpBody = try JSONEncoder().encode(["body": message])
        let (_, response) = try await session.data(for: urlRequest)

        if (response as? HTTPURLResponse)?.statusCode != 201 {
            throw NSError(domain: "GitHubClient", code: (response as? HTTPURLResponse)?.statusCode ?? -1, userInfo: nil)
        }
    }
}

private func getOrFatalError(_ key: String) -> String {
    if let value = ProcessInfo.processInfo.environment["GITHUB_\(key)"] {
        value
    } else {
        fatalError("Required 'GITHUB_\(key)' environment variable not set")
    }
}

* Proper error handling is left as an exercise for the reader.

Connecting the dots

Now we have the top level code to wrap our normal build process and a client to communicate with GitHub we just need to link them together. I’m going to go with the idea that if a line contains an opening and closing curly brace then I’ll attempt to parse it as a JSON payload. If it parses correctly then the log-commander will post the comment to GitHub. Any unsuccessful decoding will just be ignored.

To achieve this we’ll introduce a structure that represents the command we care about parsing

struct Action: Decodable {
    let prComment: PRComment?

    struct PRComment: Decodable {
        let message: String
    }
}

I’m nesting the PRComment under an Action struct so I can build up different actions that will require different data.

With this we can update our line parsing to something like this

let decoder = JSONDecoder()
decoder.keyDecodingStrategy = .convertFromSnakeCase

for try await line in standardOutput.lines() {
    print(line, terminator: "")

    guard
        let openingBrace = line.firstIndex(of: "{"),
        let closingBrace = line.lastIndex(of: "}"),
        let message = try? decoder.decode(Action.self, from: Data(line[openingBrace...closingBrace].utf8))
    else { continue }

    if let prComment = message.prComment?.message {
        try await client.postComment(prComment)
    }
}

Now if we just emit a log statement in our normal build tool that looks like this

{"pr_comment":{"message":"Example"}}

The log-commander will parse this out and post a comment on the PR with the message Example.

Tidying up

I mentioned that we don’t want the child process to know about GitHub auth and this can be taken care of by manipulating the environment in the initial run invocation something like this:

environment: .inherit.updating(["GITHUB_TOKEN": ""])

Wrap up

I like the idea of writing this logic once and then sharing it among teams to make debugging life easier. Instead of scrolling endlessly through CI logs or wiring up ad-hoc scripts for every project, you get a single wrapper that turns structured logs into actions. This blog showed GitHub PR comments but you could extend to do Slack alerts, build metrics or whatever if you can dream up. It also makes it super low friction to get rich behaviour - e.g. wrap your normal invocation to your build command with this tool and then create some specially formatted logs.

A blocker

After building an app I was then stuck with where do I actually deploy this? I’ll admit I looked around and got stuck because:

• I’m not making any money from this so didn’t want to pay for hosting
• Even if I pay there’s complexity with setting up accounts for docker repositories to host images and handing deployment etc

I gave up and resorted to just having my brother tell me when he needed the app, then I’d run it locally on my machine and then use ngrok to give him access over the public internet. Not ideal at all.

Some light

I was talking to my colleague Jack and he was telling me about how he uses a few Raspberry Pis for home automation and that he can access various bits over the public internet. This sounded cool so I bought a Raspberry Pi and then subsequently filed all this information under “I can’t do that” and moved on. Apparently Jack is on some kind of commission and kept asking how I was getting on with my Raspberry Pi, to the point where it was getting embarrassing that I hadn’t done anymore than turn the thing on. I explained that the thing really putting me off was opening ports and managing iptables etc, to which Jack said don’t do that use Cloudflare tunnels instead. A Cloudflare tunnel avoids the pain of opening ports because you run an app locally that makes an outbound connection to Cloudflare. Cloudflare then routes traffic from a public URL back down that secure tunnel to your app.

One late evening

I sat down and set myself the target of deploying this Vapor app so my brother could use it with having to have me fire it up on my mac. Here’s the set up steps:

• Install an OS on the Raspberry Pi and get it connected to my network (you use the Raspberry Pi Imager for this)
• Install Docker on the Pi - Raspbian is based off Debian so I used the instructions for debian
• Run a docker registry on the Pi so I can build on my mac but push the image to the Pi
  I did originally try building on the Pi and it didn’t pan out well. I reckon it was having power spike issues based on the fact I was using a noddy phone charger and nothing beefier.
• On my mac allow to push to my pi repository using insecure http (YOLO it’s only on my private network anyway)
  This involved adding some configuration like the following to the Docker Engine config

  "insecure-registries": [
    "raspberrypi.local:5000"
  ]
  

• Build the app on my mac, tag it and push to the registry on my Pi
• Configure a Cloudflare tunnel
  Using the config in the compose file in the next step I essentially need to configure Cloudflare to map a public url to http://app:8080. The odd url is taking advantage of the fact that the Cloudflare tunnel is running in Docker as well so we can use the app name to resolve to the app’s address
• Create a compose file that will spin up my app and a Cloudflare tunnel
  I found a really good blog post that helped me write

  services:
    app:
      restart: unless-stopped
      container_name: app
      image: localhost:5000/app:0.0.1
      ports:
        - '8080:8080'
      command: ["serve", "--env", "production", "--hostname", "0.0.0.0", "--port", "8080"]
  
    tunnel:
      restart: unless-stopped
      container_name: tunnel
      image: cloudflare/cloudflared:latest
      command: tunnel run
      environment:
        - TUNNEL_TOKEN=
  

  This is going to pull my app’s image from the registry and spin it up on port 8080, whilst also starting a Cloudflare tunnel (the TUNNEL_TOKEN environment variable can be retrieved from your Cloudflare Zero Trust dashboard)

• Spin up the containers and prosper

Wrap up

As per usual the hardest part of this project was actually getting started. Thanks to Jack for the initial idea and then pestering I managed to get this all set up and I’m pretty happy with how repeatable it all is. In fact if you are reading this blog post then boom, this is also running on the same Raspberry Pi via a different tunnel exposing a static webserver.

The problem

Here’s a simplified reproduction of the issue

class Example {
    var task: Task<Void, Never>?

    init() {
        task = Task { [weak self] in
            guard let self else { return }

            repeat {
                performSomeWork()
            } while !Task.isCancelled
        }
    }

    func performSomeWork() { }

    deinit {
        print("deinit")
        task?.cancel()
    }
}

Let’s not focus too much on the exact code as it doesn’t do anything except illustrate the issue. When running this code the deinit will never run because the Task is creating a retain cycle keeping Example alive indefinitely.

At first glance this looks like fairly standard code - the part of interest is

task = Task { [weak self] in
    guard let self else { return }

    repeat {
        performSomeWork()
    } while !Task.isCancelled
}

In the above we see the common weak/strong dance that we are all used to but we still have a cycle so what gives?

We are spinning a loop in the task that only stops when the task is cancelled. The only place we currently call cancel is in the deinit of the Example class so this loop is partly responsible for the cycle. The key thing to look for is who is taking a strong reference and what is the scope of that reference?

task = Task { [weak self] in        //
    guard let self else { return }  // - strong reference taken here
                                    //
    repeat {                        //
        performSomeWork()           //
    } while !Task.isCancelled       //
}                                   // - goes out of scope here

The problem we have looking at the scope is that the strong reference is in scope until the end of the function, but we have our repeat loop before the end of the function so we will never get to the end.

Breaking the cycle

There’s many ways to break the cycle - let’s look at a few

Change the scope of the strong reference

task = Task { [weak self] in            //
    repeat {                            //
        guard let self else { return }  // - strong reference taken here
        performSomeWork()               //
    } while !Task.isCancelled           // - goes out of scope here
}                                       //

If we move the guard inside the repeat then it will only take a strong reference for the body of repeat. This means that the strong reference is released and retaken each time through the loop. Due to the guard being evaluated fresh each time this allows the cycle to be broken.

Use the weak reference everywhere

task = Task { [weak self] in
    repeat {
        self?.performSomeWork()
    } while !Task.isCancelled
}

In this example it looks pretty clean to do this but in real code you might not be able to have the nullability in which case you’d end up using guard or if let to unwrap things (just be careful on scope).

Manually break the cycle

task?.cancel()

For this you’d have to have some other code get a reference to the task and call cancel() at the appropriate time.

Be careful

Another thing you might try to break the cycle is using the capture groups.

task = Task { [performSomeWork] in
    repeat {
        performSomeWork()
    } while !Task.isCancelled
}

For this example we are back to retain cycle city. The issue is instance methods have an implicit reference to self so this won’t do the job.

The capture group would indeed work if we are getting a reference to something that doesn’t have a self reference for example instance properties.

You could write a unit to verify that the reference does not leak something like this. In this example though you’d need to add a short delay before you set the system under test to nil to ensure that the Task has had enough time to start working and take any strong reference you want to validate is held correctly.

Conclusion

Retain cycles are a pain and the ol’ chuck a weak on things doesn’t always work so it’s worth writing tests and using instruments to hunt things down.

First things First

Let’s start by recognising if you are new to KSP it is hard to get up to speed, it’s not impossible but it will require some graft to really get stuck in. Many of the blog posts I read when I was starting were very good at helping you get something compiling but then pretty much finished there. Without someone holding my hand or giving me cues of where to look I was kind of stuck not really knowing the potential of the tool I was learning.

Don’t treat what you read on the internet as gospel

Many of the blog posts I read when starting out had a similar pattern of suggesting you should use the visitor pattern and KotlinPoet, without really saying why you’d want to use them. I’ve read the Gang of Four book many moons ago but had all but forgotten the visitor pattern and I’d never heard of KotlinPoet so that’s two things I was expected to learn just to follow an introductory tutorial.

Thankfully I’m a few years in and I’ve mostly managed to avoid using the visitor pattern for my use cases. My coding style these days leans more towards a functional style so less common OO patterns just feel alien and slow me down.

For example to get all of a class’ nested children I could use the visitor pattern something like this:

class MyVisitor: KSDefaultVisitor<Unit, Klass?>() {
    override fun visitClassDeclaration(classDeclaration: KSClassDeclaration, data: Unit) = Klass(
        classDeclaration.simpleName.asString(),
        classDeclaration.declarations.mapNotNull { it.accept(this, Unit) }.toList()
    )

    override fun defaultHandler(node: KSNode, data: Unit): Klass? = null
}

Tangent: None of the examples I read at the time actually accumulated results in this functional style using the second type parameter but instead opted but having an instance variable that you accessed after parsing was completed.

Or I could use a more functional style like this:

fun KSClassDeclaration.nestedClasses(): Klass = Klass(
    simpleName.asString(),
    declarations.filterIsInstance<KSClassDeclaration>().map(KSClassDeclaration::nestedClasses).toList()
)

The functional style I personally find more direct and I can see the recursion happening I’m not relying on learning what the various conformances to visitor are and which is right for my use case and the methods I need to use/implement (accept, defaultHandler) and why.

Anyway I’m not trying to sell one approach over the other because that’s for you and your team to thrash out. I’m mostly just saying if it works then use it, you don’t have to feel like I did that I was somehow holding it wrong because my code didn’t look like all the blog posts I was reading.

The other good thing to report is that I haven’t needed to learn KotlinPoet, again for the things I’ve worked on multiline string literals have been more than adequate. I mean I know what the Kotlin code I want to generate should look like so having an extra layer in the middle doesn’t add much for me personally.

Separate parsing and generating

When I started I kept trying to build up the final String of what the Kotlin source code should be whilst parsing the code. This is not a great idea as you soon tie yourself in knots. What compilers tend to do, which is the pattern I follow now is

• Parse the input into some intermediate representation
• Process the intermediate representation
• Render the output

For step 1 I like to take the types provided by KSP such as KSClassDeclaration and extract out the information I need into simple data class types. That way the processing logic I write next doesn’t need to know about KSP and the task is more focussed on gathering all the data that my processor thinks is interesting.

Once I have the data I’ll then do any validation, filtering or mapping to new representations. At this point I’m working with simple immutable data classs with well named properties, which is much preferred to having all my business logic calling all combinations of deeply nested resolve(), declaration, asString(), etc.

The final step is rendering, which is very often now just a collection of string templates that interpolate in the nicely structured data from the previous step.

I think there are a few great advantages to separating things out:

• You can generate different code for different targets (e.g. Kotlin/JS vs Kotlin/JVM) in a much simpler way
• Future readers don’t have to follow a potential mess of building a string whilst parsing
• More easily add unit tests around the business rules in the processor

Validate before you generate

Linked to the mistake mentioned in the section above about trying to do things in one go I would fully recommend writing out the code you want to generate manually and checking it works. I’ve found that I was constantly starting off with a simple picture of what I needed to generate and it seemed so obvious what was needed that I started writing the generation code. The issue is I’m not a very good developer and the simple code I imagine never really works and often requires changes. It’s much simpler to edit, compile and run code directly rather than trying to change the code to generate new code so that you can run and validate it.

Example use cases

The biggest pain point for me was not having that spark of inspiration for what I could be doing with KSP. Here’s a few things that me and my team have used KSP for:

• Validation tasks
  • Ensuring that a module correctly uses functions or computed properties, this is a bit niche but this module houses UI strings and if we use properties then every single string would end up in the JS artifact even if it wasn’t referenced. To avoid every JS artifact having every possible string we rely on the dead code elimination you get when the compiler notices you don’t invoke a function.
  • Ensuring that certain types conform to Serializable to support Android. If we forget the conformance then we could crash at runtime if an activity tries to serialize state.
• Generate type aliases
  • KMP doesn’t export typealises to iOS and the naming rules for types can be a little funky. Some times subtypes have dot separators (Parent.Child) and other times the symbol names are just smashed together (ParentChild). This is super confusing and we want to alias the most recent versions of some generated types so iOS developers never know about the actual versioning. The processor for this outputs Swift code, which is then packaged via SKIE.
• Generate type safe routing helpers
  • A colleague wrote a processor that will read various spring boot annotations to calculate the path for an endpoint and what arguments are required. This is then all used to generate typesafe helper functions that allow people to do routing in a much safer manor.
• Generate DSL versions
  • Me and a colleague wrote a pretty comprehensive processor that generates type safe versioned DSLs allowing us to migrate away from a system that meant adding a new version of our DSL required fairly specialised knowledge of our versioning system + many hours-days of work and resulted in inconsistent results to now mostly just bumping a number.
• Generating observability wrappers
  • Me and various colleagues wrote a processor that takes a class with some annotations sprinkled on it and generates a wrapper class that knows when properties are being written and will require us to recompute a new state. This processor also generates type safe bindings that allow us to bind our UI to these properties.
• Generate per request caches in webflux
  • I’m still learning webflux but a requirement came up to have a per request cache, this would normally be done with an @Cacheable annotation on a method in a normal thread based spring boot application. What I ended up spiking was having KSP look for an annotation of @Memoize which then generated a CoWebFilter to create a typed caffeine cache and slap it in the coroutine context. Then the KSP would generate a wrapper class that delegates to the original after trying the cache. This generated delegate wrapper would have a @Primary annotation so spring would wire it in rather than original.

There’s plenty more example uses out there these days if you look around but all of the above are either in live active projects or hopefully will be soon.

Conclusion

I think it’s great to have good documentation on how to use a library but sometimes the thing that is missing is the little bit of inspiration that get you thinking about how you could apply a technology to your project. I’m glad we embraced KSP and we have done away with so much boilerplate code and all the opportunities for mistakes and inconsistencies to sneak in that makes maintenance harder.

Let’s take the contrived example of testing an enum initialiser that allows the enum to be constructed with a case insensitive string.

Attempt 1

As a first stab we might go for

struct SoftwareTests {
    @Test("Lowercase names can be used")
    func lowercaseNamesCanBeUsed() {
        #expect(Software("macos") == .macOS)
    }

    @Test("Uppercase names can be used")
    func uppercaseNamesCanBeUsed() {
        #expect(Software("MACOS") == .macOS)
    }

    @Test("Unknown names are mapped to unknown")
    func unknownNamesAreMappedToUnknown() {
        #expect(Software("??") == .unknown)
    }
}

This is fine and results in the test navigator showing our tests like this:

└── SoftwareTests
    ├── Lowercase names can be used
    ├── Uppercase names can be used
    └── Unknown names are mapped to unknown

This all looks fairly reasonable but even in this simple example we can see duplication. Each test repeats the exact same pattern in the expectation. Full disclosure I probably wouldn’t bother changing this code as it’s already fairly concise but let’s imagine that the test bodies were a little longer and there was duplicated set up happening in the bodies.

Attempt 2

In this case you’d want to jump to some parameterised tests which might look like this:

struct SoftwareTests {
    @Test(
        "Init is case insensitive and handles unknown cases",
        arguments: [
            (input: "macos", expected: Software.macOS),
            (input: "MACOS", expected: Software.macOS),
            (input: "??", expected: Software.unknown),
        ]
    )
    func initIsCaseInsensitiveAndHandlesUnknownCases(input: String, expected: Software) {
        #expect(Software(input) == expected)
    }
}

The duplication is gone and the different permutations are run in parallel which is great for test performance. The issue is that we’ve made the test navigator view a little less useful as it now looks like this:

└── SoftwareTests
    └── Init is case insensitive and handles unknown cases
        ├── "macos", .macOS
        ├── "MACOS", .macOS
        └── "??", .unknown

Those labels don’t really mean much unless you read the test implementation. Something to note is that the actual arguments declaration in the @Test annotation is using labels to make it easier to read the test set up to know which field is the input vs the expected. Although the code source is enhanced with these labels the test navigator is not so clear.

Attempt 3

Let’s fix the previous issue using the CustomTestStringConvertible protocol

struct SoftwareTests {
    @Test(
        "Init is case insensitive and handles unknown cases",
        arguments: CaseInsensitiveInit.allCases
    )
    func initIsCaseInsensitiveAndHandlesUnknownCases(fixture: CaseInsensitiveInit) {
        #expect(Software(fixture.input) == fixture.expected)
    }

    struct CaseInsensitiveInit: CustomTestStringConvertible, CaseIterable {
        let input: String
        let expected: Software
        let testDescription: String

        static let allCases: [CaseInsensitiveInit] = [
            .init(input: "macos", expected: .macOS, testDescription: "Lowercase names can be used"),
            .init(input: "MACOS", expected: .macOS, testDescription: "Uppercase names can be used"),
            .init(input: "??", expected: .unknown, testDescription: "Unknown names are mapped to unknown"),
        ]
    }
}

With this set up the test navigator is looking much nicer:

└── SoftwareTests
    └── Init is case insensitive and handles unknown cases
        ├── Lowercase names can be used
        ├── Uppercase names can be used
        └── Unknown names are mapped to unknown

We’ve restored the handy naming whilst keeping the ability for the framework to optimise and call all the cases in parallel.

Going further

With the above we have to add boiler plate but the benefits are quite useful. There are common types where we can provide helper functions to make this process a little smoother like booleans. If we create a little helper like this:

struct DescribedBool: CustomTestStringConvertible {
    let testDescription: String
    let value: Bool
}

func boolTestStringConvertible(label: (Bool) -> String) -> [DescribedBool] {
   [
    .init(testDescription: label(false), value: false),
    .init(testDescription: label(true), value: true),
   ]
}

Then we can write tests that can be described a lot easier

@Test("Display an appropriate install state", arguments: boolTestStringConvertible { "installed = \($0)"})
func displayAnAppropriateInstallState(fixture: DescribedBool) {
    // ...
}

NB: The above will hopefully work in future but due to way the macro currently works it doesn’t like there being a closure in the arguments position. We can work around this by adding a little shim

@Test("Display an appropriate install state", arguments: installedCases())
func displayAnAppropriateInstallState(fixture: DescribedBool) {
    // ...
}

private static func installedCases() -> [DescribedBool] {
    boolTestStringConvertible { "installed = \($0)" }
}

With this in place we get nice descriptions like this:

└── SoftwareTests
    └── Display an appropriate install state
        ├── installed = false
        └── installed = true

Conclusion

SwiftTesting parameterised tests are great. It’s also very easy just to slap a load of test cases in simple tuples and exercise a lot of things but maybe lose some clarity in the test navigator around what the tests are doing. Using CustomTestStringConvertible is a nice way to bring some order back and help yourself and other travellers of your code base to navigate some hopefully extensive tests suites.

Here’s a retelling of a journey I went on to build a macOS virtualised CI solution. This is not a how to guide but in theory if you follow along you could build out your own working solution.

It Began

Our CI set up at work is a few Mac Studios that each have two CI runners installed per physical machine. The machines themselves had all been configured with a well crafted script that my colleague Sam made that installed all required tooling and got the environment into a good shape. The problem that we kept facing is that on first set up the machines were in a known good state but after that it was the wild west. Anyone could remote onto the machine and run anything they liked, whilst no one would ever do anything malicious, over time the machines drift or just generally become less healthy and builds become less repeatable.

I’d been thinking for a while about virtualisation and repeatable builds. All of our backend is set up around docker which gives you this but wouldn’t it be nice to have it for our builds that required macOS.

When Apple brought out the M1 chips they made virtualisation easier and they even had sample code for creating virtual machines. Although this was cool I didn’t get much further than having a play around as it felt like a lot more work would be required for me to use the technology.

I knew that people had gotten virtualisation to work to the point where they were offering cloud based CI services and after a bit of research I stumbled upon tart, which I bookmarked and then didn’t look at for months. It wasn’t until I saw some Tweets/Toots/Skeets (whatever the bloody platform was at the time) from Simon B. Støvring talking about his CI work that I started investigating properly.

The first virtual machine

Despite my hesitation on getting started it actually was quite simple to create a virtual machine and begin to mess around (I kicked myself for putting off trying sooner).

After installing tart from homebrew

brew install cirruslabs/cli/tart

It’s then a case of providing the url for an IPSW to a tart create invocation, that will then download the IPSW and create a virtual machine.

tart create example \
  --disk-size=120 \
  --from-ipsw=https://updates.cdn-apple.com/2024FallFCS/fullrestores/072-30094/44BD016F-6EE3-4EE5-8890-6F9AA008C537/UniversalMac_15.1.1_24B91_Restore.ipsw

The IPSW link above will no doubt go out of date pretty quickly but you can find the download urls at https://ipsw.me. I know the website looks a bit scammy and has adverts everywhere but it’s providing links to Apple domains so ¯_(ツ)_/¯.

After running the above (which will take a small eternity as the IPSW is a big download, don’t worry the download is cached for future invocations) you end up with… your terminal prompt back. To actually run the machine you just created you need to run

tart run example

example is the name I passed to the create command - if you create a machine with a different name make sure to run that instead.

This will boot the machine and allow you to start configuring the OS the same as if you fired up new macOS hardware for the first time. This was pretty exciting and brought back memories of the Dave Verwer course where I first learnt iOS and specifically the cool feeling from the first time deploying code I’d written onto a physical device.

At this point I was still a long way from the end goal but I’d made a start so the momentum kept me moving. The first thing I wanted to prove was could I actually get projects to build inside the virtual machine and was the performance alright. I think I discovered that I could get our project to build but for some reason the tests just would not run. When I hit this road block I moved things to the back burner again.

Asking for help

Weeks had passed and I was still keen to make this work and from following Simon B. Støvring on Mastodon I’d seen more posts he had made and discovered all the documentation that had been written for some tooling he’d made called Tartelet (see tartelet docs). Sure that I could make this work I tried again but kept getting the same result, I eventually just reached out on slack and got helpful responses from Simon and someone with the handle biscuit, which suggested two things to try

• prewarming simulators using something like https://github.com/biscuitehh/yeetd/blob/main/Resources/prewarm_simulators.sh
• bumping up the available memory on the virtual machines

I can’t specifically remember which one of the above it was but I now had proved to myself that I could indeed run our project inside virtual machines including the tests and the performance wasn’t noticeably worse than our existing set up.

Repeatable Configuration

The next thing I wanted to achieve was creating machines repeatably - for this I turned to the Tartelet docs mentioned before. The Tartelet docs talk you through sensible configuration to use for a CI runner but the configuration is done using the GUI. The problem I had is that I’m the type of person who locks my front door, walks to my car then turns around to check the front door is locked. This means that I just didn’t feel comfortable having to manually configure machines incase I mess a step up or mistype something.

After a bit of research I found the company that created tart (remember they provide a CI service) also has a repo that contains their configuration for building machines here. The first take away is that they are using Packer to provision machines. In this case Packer is using a tart plugin, which is using the tart tool under the hood. So after installing packer with mise and then the tart plugin I was set to explore building machines from scratch using just code.

mise install packer
packer plugins install github.com/cirruslabs/tart

Using the cirrus labs templates as a starting point I ended up with something like this to build a Sequoia machine

packer {
  required_plugins {
    tart = {
      version = ">= 1.12.0"
      source = "github.com/cirruslabs/tart"
    }
  }
}

source "tart-cli" "tart" {
  from_ipsw = "https://updates.cdn-apple.com/2024FallFCS/fullrestores/072-30094/44BD016F-6EE3-4EE5-8890-6F9AA008C537/UniversalMac_15.1.1_24B91_Restore.ipsw"
  vm_name = "base-sequoia"
  cpu_count = 8
  memory_gb = 8
  disk_size_gb = 100
  headless = false
  ssh_password = "runner"
  ssh_username = "runner"
  ssh_timeout = "120s"
  boot_command = [
    // hello, hola, bonjour, etc.
    // > Tap get started
    "<wait60s><spacebar>",

    // Language
    // > Typing english gets us to English UK
    "<wait10s>english<enter>",

    // Select Your Country or Region
    "<wait20s>united kingdom<leftShiftOn><tab><leftShiftOff><spacebar>",

    // Written and Spoken Languages
    // > Tap continue
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",

    // Accessibility
    // > Tap Not now
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",

    // Data & Privacy
    // > Tap continue
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",

    // Migration Assistant
    // > Tap Not now
    "<wait10s><tab><tab><tab><spacebar>",

    // Sign In with Your Apple ID
    // > Tap Set up later
    "<wait10s><leftShiftOn><tab><leftShiftOff><leftShiftOn><tab><leftShiftOff><spacebar>",

    // Are you sure you want to skip signing in with an Apple ID?
    // > Tap Skip
    "<wait10s><tab><spacebar>",

    // Terms and Conditions
    // > Tap Agree
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",

    // I have read and agree to the macOS Software License Agreement
    // > Tap Agree
    "<wait10s><tab><spacebar>",

    // Create a Computer Account
    // > Set username, account name, password all to runner
    "<wait10s>runner<tab><tab>runner<tab>runner<tab><tab><tab><spacebar>",

    // Enable Location Services
    // > Deselect and tap continue
    "<wait30s><leftShiftOn><tab><leftShiftOff><spacebar>",

    // Are you sure you don't want to use Location Services?
    // > Tap continue
    "<wait10s><tab><spacebar>",

    // Select Your Time Zone
    // > Type UTC and tap continue
    "<wait10s><tab><tab>UTC<enter><leftShiftOn><tab><tab><leftShiftOff><spacebar>",

    // Analytics
    // > Tap continue
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",

    // Screen Time
    // > Tap Set up later
    "<wait10s><tab><spacebar>",

    // Siri
    // > Deselect and tap continue
    "<wait10s><tab><spacebar><leftShiftOn><tab><leftShiftOff><spacebar>",

    // Choose your look
    // > Select light mode
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",

    // Welcome to Mac
    "<spacebar>",

    // Open terminal
    "<wait10s><leftAltOn>n<leftAltOff><wait3s><leftAltOn><leftShiftOn>g<leftShiftOff><leftAltOff>/Applications/Utilities/Terminal.app<enter><wait3s><leftAltOn>o<leftAltOff><wait3s>defaults write NSGlobalDomain AppleKeyboardUIMode -int 3<enter><wait5s><leftAltOn>q<leftAltOff>",

    // Open system settings
    "<wait10s><leftAltOn>n<leftAltOff><wait3s><leftAltOn><leftShiftOn>g<leftShiftOff><leftAltOff>/Applications/System Settings.app<enter><wait3s><leftAltOn>o<leftAltOff>",

    // Search for 'sharing'
    "<wait10s><leftAltOn>f<leftAltOff>sharing<enter>",

    // Tab to 'Screen Sharing' and enable it
    "<wait10s><tab><tab><tab><tab><tab><spacebar>",

    // Navigate to 'Remote Login' and enable it
    "<wait10s><tab><tab><tab><tab><tab><tab><tab><tab><tab><tab><tab><tab><spacebar>",

    // Close settings
    "<wait5s><leftAltOn>q<leftAltOff>"
  ]

  // A workaround for Virtualization.Framework's installation process not fully finishing in a timely manner
  create_grace_time = "30s"
  run_extra_args = []
}

build {
  sources = ["source.tart-cli.tart"]

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mEnable passwordless sudo\\033[0m'",
      "echo '└── echo runner | sudo -S sh -c \"mkdir -p /etc/sudoers.d/; echo \\047runner ALL=(ALL) NOPASSWD: ALL\\047 | EDITOR=tee visudo /etc/sudoers.d/runner-nopasswd\"'",
      "echo runner | sudo -S sh -c \"mkdir -p /etc/sudoers.d/; echo 'runner ALL=(ALL) NOPASSWD: ALL' | EDITOR=tee visudo /etc/sudoers.d/runner-nopasswd\""
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mEnable autologin\\033[0m'",
      "echo '├── printf \\047\\x0f\\xfc\\x3c\\x4d\\xb7\\xce\\xdd\\x8d\\x65\\xd0\\x6c\\x2c\\047 > /tmp/kcpassword'",
      "printf '\\x0f\\xfc\\x3c\\x4d\\xb7\\xce\\xdd\\x8d\\x65\\xd0\\x6c\\x2c' > /tmp/kcpassword",
      "echo '├── sudo mv /tmp/kcpassword /etc/kcpassword'",
      "sudo mv /tmp/kcpassword /etc/kcpassword",
      "echo '├── sudo chmod 600 /etc/kcpassword'",
      "sudo chmod 600 /etc/kcpassword",
      "echo '├── sudo chown root:wheel /etc/kcpassword'",
      "sudo chown root:wheel /etc/kcpassword",
      "echo '└── sudo defaults write /Library/Preferences/com.apple.loginwindow autoLoginUser runner'",
      "sudo defaults write /Library/Preferences/com.apple.loginwindow autoLoginUser runner"
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mDisable screensaver\\033[0m'",
      "echo '├── sudo defaults write /Library/Preferences/com.apple.screensaver loginWindowIdleTime 0'",
      "sudo defaults write /Library/Preferences/com.apple.screensaver loginWindowIdleTime 0",
      "echo '└── defaults -currentHost write com.apple.screensaver idleTime 0'",
      "defaults -currentHost write com.apple.screensaver idleTime 0"
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mDisable sleeping\\033[0m'",
      "echo '├── sudo systemsetup -setdisplaysleep Off 2> /dev/null'",
      "sudo systemsetup -setdisplaysleep Off 2> /dev/null",
      "echo '├── sudo systemsetup -setsleep Off 2> /dev/null'",
      "sudo systemsetup -setsleep Off 2> /dev/null",
      "echo '└── sudo systemsetup -setcomputersleep Off 2> /dev/null'",
      "sudo systemsetup -setcomputersleep Off 2> /dev/null"
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mDisable screen lock\\033[0m'",
      "echo '└── sysadminctl -screenLock off -password runner'",
      "sysadminctl -screenLock off -password runner"
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mDisable spotlight indexing\\033[0m'",
      "echo '└── sudo mdutil -a -i off'",
      "sudo mdutil -a -i off"
    ]
  }
}

If this is stored in a file called sequoia.pkr.hcl and you run packer build sequoia.pkr.hcl and then wait you’ll end up with a new machine in tart called base-sequoia that has various basic things configured like passwordless login, turning off screensaver etc. Arriving at the above configuration took a short eternity with a lot of trial and error tweaking things and rerunning.

Some notes on the hcl file above:

hcl is a configuration language that has all kinds of features, which I did start using but then in the spirit of not wanting future maintainers of this tooling having to learn yet another thing I opted to entirely wrap it. The above is generated by calling a thin kotlin dsl I wrote, which has a few advantages in my mind

• Other team mates can more easily help maintain things by calling the Kotlin dsl and not having to worry about learning hcl
• With a dsl I can make things like <leftShiftOn><tab><leftShiftOff> safer by managing the on/off state
• IDEs are going to be miles better at supporting Kotlin as opposed to a custom markup language
• I’m in control of the dsl output, which allowed me to do pretty printing of the commands

That last bullet deserves some expansion. By Default invoking shell commands in packer won’t actually tell you what is being invoked. In the output all you see is lines like Provisioning with shell script: /var/folders/wl/92q0hw051vnbwncp7lxsp3m80000gq/T/packer-shell2461613400. When you are debugging configuration or even just want to see the progress it’s super helpful to be able to see where you are up to especially in case of failure. With this in mind I made it so in the dsl you’d call a function like this

script("Disable sleeping") {
    """
    sudo systemsetup -setdisplaysleep Off 2> /dev/null
    sudo systemsetup -setsleep Off 2> /dev/null
    sudo systemsetup -setcomputersleep Off 2> /dev/null
    """.trimIndent()
}

This would generate this hcl configuration

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mDisable sleeping\\033[0m'",
      "echo '├── sudo systemsetup -setdisplaysleep Off 2> /dev/null'",
      "sudo systemsetup -setdisplaysleep Off 2> /dev/null",
      "echo '├── sudo systemsetup -setsleep Off 2> /dev/null'",
      "sudo systemsetup -setsleep Off 2> /dev/null",
      "echo '└── sudo systemsetup -setcomputersleep Off 2> /dev/null'",
      "sudo systemsetup -setcomputersleep Off 2> /dev/null"
    ]
  }

At run time you’d actually see this

==> tart-cli.tart: Provisioning with shell script: /var/folders/wl/92q0hw051vnbwncp7lxsp3m80000gq/T/packer-shell2461613400
    tart-cli.tart: 🟢 Disable sleeping
    tart-cli.tart: ├── sudo systemsetup -setdisplaysleep Off 2> /dev/null
    tart-cli.tart: warning: this combination of display sleep and system sleep may prevent system sleep.
    tart-cli.tart: setdisplaysleep: Never
    tart-cli.tart: ├── sudo systemsetup -setsleep Off 2> /dev/null
    tart-cli.tart: setsleep: Never (computer, display, hard disk)
    tart-cli.tart: └── sudo systemsetup -setcomputersleep Off 2> /dev/null
    tart-cli.tart: setcomputersleep: Never

It might not be pretty but I was borrowing the lines (├─, └─) from the tree command to give a bit of structure to show that the high level step is 🟢 Disable sleeping and after that you have the commands that make this process up starting with ├─, └─ and then the std{out,err} of those commands below that.

Marvel at what the above achieved

At this point I kept running tart list and tart run base-sequoia just to marvel at the fact I had indeed got a machine working and can yield the power of starting and stopping it on command.

More boring configuration

Having a machine with no software installed is pretty boring so the next step was spending ages figuring out the incantations to get things like Xcode and Ruby (for fastlane) installed. I was doing this like a cave man by starting the machine, opening the terminal and manually typing commands (as copy/paste doesn’t work across machines) until a colleague gave a disapproving look and said “why don’t you just ssh into the machine?”. Things went faster after that.

The basic config for getting Xcode installed looked something like

packer {
  required_plugins {
    tart = {
      version = ">= 1.12.0"
      source = "github.com/cirruslabs/tart"
    }
  }
}

source "tart-cli" "tart" {
  vm_base_name = "base-sequoia"
  vm_name = "xcode"
  cpu_count = 8
  memory_gb = 8
  disk_size_gb = 100
  headless = true
  ssh_password = "runner"
  ssh_username = "runner"
  ssh_timeout = "120s"
  run_extra_args = []
}

build {
  sources = ["source.tart-cli.tart"]

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mCreate directory for artifacts we want to install\\033[0m'",
      "echo '└── mkdir -p /Users/runner/Downloads/packer'",
      "mkdir -p /Users/runner/Downloads/packer"
    ]
  }

  provisioner "file" {
    sources = [
      pathexpand("~/XcodesCache/Command_Line_Tools_for_Xcode_16.1.dmg"),
      pathexpand("~/XcodesCache/Xcode_16.1.xip"),
      pathexpand("~/XcodesCache/iOS_18.1_Simulator_Runtime.dmg")
    ]
    destination = "/Users/runner/Downloads/packer/"
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mCreate ~/.zprofile\\033[0m'",
      "echo '└── echo \"export LANG=en_US.UTF-8\" >> ~/.zprofile'",
      "echo \"export LANG=en_US.UTF-8\" >> ~/.zprofile"
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mInstall command line tools\\033[0m'",
      "echo '├── hdiutil attach \"/Users/runner/Downloads/packer/Command_Line_Tools_for_Xcode_16.1.dmg\"'",
      "hdiutil attach \"/Users/runner/Downloads/packer/Command_Line_Tools_for_Xcode_16.1.dmg\"",
      "echo '├── sudo installer -pkg \"/Volumes/Command Line Developer Tools/Command Line Tools.pkg\" -target \"/Volumes/Macintosh HD\"'",
      "sudo installer -pkg \"/Volumes/Command Line Developer Tools/Command Line Tools.pkg\" -target \"/Volumes/Macintosh HD\"",
      "echo '└── hdiutil detach \"/Volumes/Command Line Developer Tools\"'",
      "hdiutil detach \"/Volumes/Command Line Developer Tools\""
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mInstall homebrew\\033[0m'",
      "echo '├── /bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\"'",
      "/bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\"",
      "echo '├── eval \"$(/opt/homebrew/bin/brew shellenv)\"'",
      "eval \"$(/opt/homebrew/bin/brew shellenv)\"",
      "echo '├── echo \\047eval \"$(/opt/homebrew/bin/brew shellenv)\"\\047 >> ~/.zprofile'",
      "echo 'eval \"$(/opt/homebrew/bin/brew shellenv)\"' >> ~/.zprofile",
      "echo '├── echo \\047export HOMEBREW_NO_AUTO_UPDATE=1\\047 >> ~/.zprofile'",
      "echo 'export HOMEBREW_NO_AUTO_UPDATE=1' >> ~/.zprofile",
      "echo '└── echo \\047export HOMEBREW_NO_INSTALL_CLEANUP=1\\047 >> ~/.zprofile'",
      "echo 'export HOMEBREW_NO_INSTALL_CLEANUP=1' >> ~/.zprofile"
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mInstall mise\\033[0m'",
      "echo '├── curl https://mise.run | sh'",
      "curl https://mise.run | sh",
      "echo '├── export PATH=\"$HOME/.local/bin:$PATH\"'",
      "export PATH=\"$HOME/.local/bin:$PATH\"",
      "echo '├── echo \\047eval \"$(~/.local/bin/mise activate zsh)\"\\047 >> ~/.zprofile'",
      "echo 'eval \"$(~/.local/bin/mise activate zsh)\"' >> ~/.zprofile",
      "echo '├── mkdir -p ~/.config || true'",
      "mkdir -p ~/.config || true",
      "echo '├── echo \\047[alias]\\047 >> ~/.config/mise.toml'",
      "echo '[alias]' >> ~/.config/mise.toml",
      "echo '└── echo \"xcodes = \\047asdf:younke/asdf-xcodes\\047\" >> ~/.config/mise.toml'",
      "echo \"xcodes = 'asdf:younke/asdf-xcodes'\" >> ~/.config/mise.toml"
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mInstall Xcode\\033[0m'",
      "echo '├── \"$HOME/.local/bin/mise\" trust --yes'",
      "\"$HOME/.local/bin/mise\" trust --yes",
      "echo '├── \"$HOME/.local/bin/mise\" use xcodes@\"1.6.0\" --yes'",
      "\"$HOME/.local/bin/mise\" use xcodes@\"1.6.0\" --yes",
      "echo '├── eval \"$(\"$HOME/.local/bin/mise\" activate --shims)\"'",
      "eval \"$(\"$HOME/.local/bin/mise\" activate --shims)\"",
      "echo '├── xcodes install \"16.1\" --path \"/Users/runner/Downloads/packer/Xcode_16.1.xip\" --experimental-unxip --empty-trash'",
      "xcodes install \"16.1\" --path \"/Users/runner/Downloads/packer/Xcode_16.1.xip\" --experimental-unxip --empty-trash",
      "echo '├── sudo xcodes select \"16.1\"'",
      "sudo xcodes select \"16.1\"",
      "echo '├── xcodebuild -runFirstLaunch'",
      "xcodebuild -runFirstLaunch",
      "echo '└── xcrun simctl runtime add \"/Users/runner/Downloads/packer/iOS_18.1_Simulator_Runtime.dmg\"'",
      "xcrun simctl runtime add \"/Users/runner/Downloads/packer/iOS_18.1_Simulator_Runtime.dmg\""
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mInstall Ruby\\033[0m'",
      "echo '├── eval \"$(/opt/homebrew/bin/brew shellenv)\"'",
      "eval \"$(/opt/homebrew/bin/brew shellenv)\"",
      "echo '├── brew install readline libyaml gmp'",
      "brew install readline libyaml gmp",
      "echo '└── \"$HOME/.local/bin/mise\" use ruby@\"3.3.0\"'",
      "\"$HOME/.local/bin/mise\" use ruby@\"3.3.0\""
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mDelete install artifacts\\033[0m'",
      "echo '└── rm -rf /Users/runner/Downloads/packer'",
      "rm -rf /Users/runner/Downloads/packer"
    ]
  }
}

There’s a few things to note with the above:

• Duplicated hardcoded version numbers - in reality these are interpolated in and not manually maintained.
• Assets related to installing Xcode are copied from the host machine to avoid needing to auth with Apple to download things.
• Homebrew is installed after Xcode command line tools - again to avoid being asked to auth with Apple to download the installer.

Marvel some more

Again it was time to reflect on how far we’d come and how it wasn’t easy. Figuring out the exact shell commands to install all the things you need was a combination of Google, ChatGPT, talking to colleagues and just trying loads of things.

Between the two packer files above I learnt a load of new things

• Various admin commands to disable screensavers and screen locks
• What on earth kcpassword was
• Mounting/unmounting volumes
• More in depth knowledge of installing/working with mise
• Ruby is still an absolute pain to install right

To even get this far though there’s a lot of foundational knowledge that I used

• Being comfortable with the command line
• Having the ability to navigate directories/files from the command line
• Understanding environment variables in shell environments
• Understanding file permissions
• Knowing which commands look safe to run that were found on random internet searches
• Knowing when to use sudo and not just using it blindly causing issues later on

What about CI?

Now we had machines running the next step was actually hooking up to the CI machinery to run some workloads. There were many misfires at this stage with trying different approaches. The most promising was to add more provisioning steps to install the CI runner inside the virtual machine and then orchestrate having 2 virtual machines booted that self registered with the CI server to run workloads. Once the virtual machine had run a workload it would then need to destroy itself and another machine be spun up in its place.

Although the above worked and seemed reasonable it was tricky and means that you always have to have virtual machines fired up even when not in use. It also felt a bit wrong having the virtual machines needing to know about CI when that information could be hidden from them - for instance I often just spin up the virtual machines to try things out in a clean environment but I don’t want to worry about it adding my personal machine to the CI work pool. Discussing this with a colleague we came to the conclusion that actually we could have a couple of CI runners on the bare metal and when they receive a work load they would clone a virtual machine and boot it, then copy the source files into the virtual machine before running the job. If you squint hard enough this feels a bit like docker where you’d have various layers being created to configure the environment then you’d copy your source code in and operate on that.

We messed around for a while trying to do this with combinations of tart commands directly and then one of us (can’t remember who) had the spark that we should just use Packer again for this as it has already massively simplified the task of copying files to/from the virtual machine as well as running shell tasks.

With this in mind we need a new packer file:

packer {
  required_plugins {
    tart = {
      version = ">= 1.12.0"
      source = "github.com/cirruslabs/tart"
    }
  }
}

source "tart-cli" "tart" {
  vm_base_name = "xcode"
  vm_name = "runner"
  cpu_count = 8
  memory_gb = 8
  disk_size_gb = 100
  headless = true
  ssh_password = "runner"
  ssh_username = "runner"
  ssh_timeout = "120s"
}

build {
  sources = ["source.tart-cli.tart"]

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mCreate workspace directory\\033[0m'",
      "echo '└── mkdir -p $HOME/workspace'",
      "mkdir -p $HOME/workspace"
    ]
  }

  provisioner "file" {
    source = "./"
    destination = "$HOME/workspace"
    direction = "upload"
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mRun the entry point\\033[0m'",
      "echo '├── cd $HOME/workspace'",
      "cd $HOME/workspace",
      "echo '└── echo Hello, World!'",
      "echo 'Hello, World!'"
    ]
  }
}

This packer file will upload the current directory into the vm, then run echo 'Hello, World!'. Although this is pretty pointless it does demonstrate that everything works and we have the following:

• A clean machine is run by cloning the xcode machine
• We can copy our source code into the virtual machine
• We can execute arbitrary code as an entry point

Most CI solutions will allow you to create a template or reuse configuration. We use GoCD at work, which is “interesting” but we was able to find a way to make it so we could have a template that will invoke something like the packer file above but allow each pipeline to provide the entry point to invoke inside the virtual machine.

Entry points and configuration

When we started I think I was tunnel visioned and made it so that in the CI template if you provided an entry point of publish it would follow a convention of looking for an executable file in your repo called .gocd/publish. Although this worked it wasn’t super discoverable and if you wanted to share configuration between pipelines you were pretty much forced to start sourcing bash files from bash files 🤮.

The final straw for this set up was when we needed to pass environment variables to the workloads. My colleague took the fun task of changing this set up to instead read a configuration file written in json that describes the entry point, environment variables and any other config you might want to pass to a pipeline.

Getting data out of the virtual machine

When CI fails it’s helpful to know why and sometimes those details don’t appear in the logs but in various artifacts scattered throughout build folders. Packer gives a mechanism for this by allowing you to register an action on failure, in these cases we opt to copy files from a known directory to the host so they can be uploaded to the CI server before the virtual machine is deleted. Having the CI always upload files it finds in a directory is handy because it means you can put reports or artifacts in this special folder on the VM and know it will be uploaded to the CI regardless of whether there was a failure or not.

Android

Although we had plans to have Android builds run in docker to save the macOS runners we still needed to support Android because we have some Kotlin Multiplatform projects that we just want to build in one place. I’m not going to pretend to know what wizardry went on here but a big problem with the Virtualization framework is that is doesn’t support nested emulation, which means no Android emulators. To get around this my colleague created an ssh tunnel between the host and the VM and runs the emulators on the bare metal but connects Adb in the virtual machine via this tunnel. This works well.

Taking stock of things

The above represents the foundational thinking blocks that underpin the solution we built. It took quite a while to build out as it was mostly stewing as an idea whilst the pieces starting drifting together slowly from different experiments. Not all effort is equal - there we some tasks where it felt like I’d made a massive leap towards the end goal and then others where the work was important but it felt like a hard slog.

Where we ended up after some more iterations was we have a Kotlin Multiplatform command line tool that controls all of the above. There’s a dsl that hides away the hcl files and instead you have a nice Kotlin interface. The tooling has the ability to build machines in “layers” where each subsequent machine depends on the machine above existing. This means that you only need to rebuild machines (which can be slow) when required. We also have a json file that each repo uses to configure various things like caching, environment variables and the entry points themselves.

Other use cases

At some point we used Github’s virtual runners and when something went wrong it was an absolute nightmare to debug without access to the machine after the run. With this set up I’m never more than a few commands away from being in the same environment as the one that produced the failure so I can explore and try things out to get to the root issue.

Being able to build virtual machines easily in a repeatable way is empowering as it means if I want to jump on the beta release trains for macOS or Xcode versions I can do it without risking my primary machine.

I’ve also found it useful having the ability to have a clean machine with no software installed to help quickly iterate install scripts to provide to colleagues. It’s amazing how many assumptions you can make about other people’s system having similar set ups to your own which all fly out the window when you use a clean machine.

Conclusion

Working on this kicked my ass and I’d picked it up and put it down a few times but as it was a nagging idea I really wanted to see it through. Fortunately once it had some legs and I was pretty confident we could get it done I spent dedicated time on this with my colleague Jack. The end solution builds on the above adding several key things that are needed to productionise it. We’ve been using it for various macOS and Kotlin Multiplatform projects for ~6 months and all the iOS pipelines for ~2 months and despite some teething issues we’ve not really seen many issues.

In the above I’ve mentioned colleagues a few times and it’s worth stating that this project was probably only doable because of being fortunate to work with people who can solve problems, brainstorm and muck in across a wide range of technologies and tolerate listening to me talk nonsense and rewrite my stuff multiple times as I realised that the me who wrote code yesterday was an idiot.

Try creating a cli executable in your project that exposes common project tasks that are written in the project’s core language. This allows better contribution and less single points of failure with pockets of knowledge in the team.

Scene setting

Over time projects accumulate helper scripts to perform various admin tasks. I’ve historically tried to avoid bash as much as possible for these scripts because the projects I work on often have teams of people unfamiliar with bash or its idiosyncrasies. With this in mind I’ve then gravitated towards Ruby because I’ve always loved the language and it’s a safer choice in my mind. Unfortunately I’ve been kidding myself because as much as I love Ruby it still has the same issue as bash with people not knowing it and also it’s a right pain to make sure people’s environment are set up.

The next logical step is to just use the main project’s language for building up tasks. This is potentially easier said than done but I’ve seen success with doing it. As a mobile developer this means using Swift with SPM to build out tasks on the iOS side and Kotlin with gradle on the backend/mulitplatform parts.

The good

In taking this approach I’ve removed myself from being the single point of failure on maintaining stuff. This not only means that I don’t have to be on hand to debug things but also opens the door for easier contribution/reuse. For example with Swift being the langauge used to write an admin script other people have contributed various tasks with the obvious plus being that the whole team can much more easily adopt and understand what is being done without trying to understand cryptic personal scripts.

Using languages like Swift/Kotlin encourages me to write more reusable code than if I was just slinging bash around. For example I’d write a Github client that can be reused rather than being lazy and copy/pasta’ing curl invocations around with duplicated configuration.

You have the full power of available libraries like type safe serialisation with Codable or by pulling in something like kotlinx.serialization. I can’t even count how many times I’ve written dodgy JSON interpolation in scripts when really I should have just not been lazy and used the right tools for the job.

Debugging is a super power for these scripts even though I might end up cave man debugging (print) I love having the option to use a debugger and inspect all the things or try changing state on the fly to see what would happen.

Types, types, types… I love types and they are really handy for helping me write safe code.

The bad

Both Swift and Kotlin just aren’t that great as scripting languages even though I really want them to be. This may be a personal lack of competence but when I’m writing scripts I’m looking for super fast feedback, which means I’ll often start just curling things on the command line or opening TextMate, setting it to bash and hitting ⌘+R. With both of these I’m running code straight away with very little ceremony, which I simply can’t reliably do for Swift or Kotlin as both pretty much require that I use an IDE to help with types and missed keywords (try, await, suspend…). This may sound contradictory to Types, types, types... but at different points in the development process I value different things. Often when I am just trying things out I’m not very professional and just want to throw code around to see what works before I put on my big developer pants and do the job properly.

Another weakness is forking. In bash or Ruby I can just slap backticks around a command to have it run in a subshell and then collect the result. It’s just not that simple in Swift/Kotlin even when pulling in libraries, which I do.

Approach

I’m not 100% sold on the exact naming/layout but as a reference this is what I set up. We have a shim at the root of the project called cli, this file’s job is to essentially cd into the project that has the tasks and call swift build followed by running the built exectuable. It’s a little bit of redirection but it’s certainly easier than expecting people to remember the calling convention. My other thinking is that if we come to some standarisation that in our projects you just call ./cli to be presented with all the various admin tasks then it’s just one thing to learn.

With this in place we currently use swift-argument-parser to build a cli that has various subcommands as an example for some inspiration here’s some top level tasks that we’ve built out

OVERVIEW: A utility for working with the ios repo.

USAGE: cli <subcommand>

OPTIONS:
  -h, --help              Show help information.

SUBCOMMANDS:
  ci                      Commands the CI pipeline uses
  code-gen                Regenerate generated code for the app.
  collect-debug-info      Print information useful for getting help with debugging.
  doctor                  Help diagnose environment set up issues.
  firewall                Add/Remove firewall rules for simulator
  kmp-doctor              Update local repository to add KMP files into Xcode
  set-marketing-value     Create a branch with a commit that updates to the passed version number.
  sim                     Utilities for working with simulators.

On the Kotlin side we’ve been using clikt to perform the role of swift-argument-parser but set up is very similar.

Misfires

I spent far too long trying to use cute tricks like #!/usr/bin/env kotlin with kts files to get the “scripting” feel with the language of choice. I personally found this a complete train wreck as I had to pull in loads of dependencies using @file:DependsOn and then very quickly hit the fact that I can’t write Kotlin without an IDE. For some reason IntelliJ was giving me no help with limited syntax highlighting and no suggestions. To actually get anything working I had to create a project, import dependencies in the normal way and then once I had code that worked and had all the right imports etc I could copy/pasta it over. At which point I sat scratching my head wondering why my brain hadn’t stopped me doing such a ridiculous thing - e.g. if I only committed the kts file I’d be committing the lossy version of my work that is hard to debug or work with.

For example on an iOS project I might have a script like this as a starting point

bin/collect-debug-info

#!/bin/bash

cat << EOF
OS: $(sw_vers --productName) $(sw_vers --productVersion) ($(sw_vers --buildVersion))
Git: $(git rev-parse --abbrev-ref HEAD) ($(git rev-parse HEAD))
Xcode: $(xcode-select -p)

Simulators:
$(xcrun simctl list devices booted)

Mise $(mise --version):
$(mise list --current)
EOF

An example output might be:

OS: macOS 14.3.1 (23D60)
Git: main (aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa)
Xcode: /Applications/Xcode-15.4.0.app/Contents/Developer

Simulators:
== Devices ==
-- iOS 16.4 --
-- iOS 17.0 --
-- iOS 17.0 --
-- iOS 17.2 --
-- iOS 17.4 --
-- iOS 17.5 --
    iPhone 11 (AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAAE) (Booted)

Mise 2024.7.0 macos-arm64 (e518900 2024-07-03):
jq              1.7.1    ~/src/ios/my-proj/.mise.toml latest
ruby            3.3.0    ~/src/ios/my-proj/.mise.toml 3.3.0
swiftformat     0.53.9   ~/src/ios/my-proj/.mise.toml 0.53.9
swiftlint       0.55.0   ~/src/ios/my-proj/.mise.toml 0.55.0
tuist           4.17.0   ~/src/ios/my-proj/.mise.toml 4.17.0
xcodes          1.4.1    ~/src/ios/my-proj/.mise.toml 1.4.1

Now when someone asks for help and I suspect there might be environment issues I can just ask for the output of bin/collect-debug-info and we’ll be up to speed debugging in no time. This is the kind of script you can build up over time and add all kinds of useful info as and when you decide it would be useful to collect.

Starting point

Let’s say I have the following simple class that I want to migrate one function at a time

MyObject.h

@interface MyObject: NSObject

- (void)doSomething1;
- (void)doSomething2;

@end

MyObject.m

@interface MyObject ()

@property (nonatomic, copy) NSString *title;

@end

@implementation MyObject

- (void)doSomething1 { ... }
- (void)doSomething2 { ... }

@end

The above is a class with two public methods and a “private” property declared in an anonymous category.

One step migration

If I wanted to migrate this in one go I can delete the .m file and create a Swift file like this

MyObject.swift

@_objcImplementation MyObject { }

At this point the compiler will complain about the missing implementations

Extension for main class interface should provide implementation for instance method ‘doSomething1()’; this will become an error before ‘@_objcImplementation’ is stabilized

Extension for main class interface should provide implementation for instance method ‘doSomething2()’; this will become an error before ‘@_objcImplementation’ is stabilized

To make the compiler happy I need to provide all the implementations like so:

MyObject.swift

@_objcImplementation MyObject {
    func doSomething1() { ... }
    func doSomething2() { ... }
}

This might be fine for small classes but my goal was to be able to break the task down into small chunks whilst keeping everything compiling and tests passing.

Create named categories

To do this in a more controlled way the best thing to do is to split the @interface into named categories and then specify the category name in the annotation. For example I called my category SwiftMigration

MyObject.h

@interface MyObject: NSObject

- (void)doSomething2;

@end

@interface MyObject (SwiftMigration)

- (void)doSomething1;

@end

The corresponding Swift file that targets the category now only needs to implement the one method and would look like this:

MyObject.swift

@_objcImplementation(SwiftMigration) extension MyObject {
    func doSomething1() { ... }
}

With this approach I can go one method at a time and it doesn’t feel like such a big risk doing the port.

Properties

In the example above I have a property declared in an anonymous category which essentially makes it private to my class implementation. Normally you cannot declare new storage in extensions but with @_objcImplementation you are allowed to declare storage on the top implementation (the unnamed one), which would look like this:

MyObject.swift

@_objcImplementation extension MyObject {
    private var title: String?
}

Final clean up

Whether migrating in one go or piece by piece it’s then worth asking if the @_objcImplementation is required at all or if you can delete the header file and make it a pure Swift class. There are cases where you might need to continue to use the new capabilities like if you still have code in Objective-C that subclasses your class.

General migration strategies

There are other ways of avoiding rewriting large amounts of code whilst still taking advantage of Swift. I use these techniques to help avoid adding any new Objective-C.

Extensions

Swift extensions are a great way for adding new Swift code to legacy Objective-C code. In the simple case where all call sites will only be Swift based you can create an extension and don’t annotate it as @objc

MyObject.swift

extension MyObject {
    func doSomethingNew() { ... }
}

I will even prefer doing this over writing too much new Objective-C in the a class. For example I might just annotate my doSomethingNew function as @objc and then call it on self. This isn’t perfect for encapsulation but I don’t generally write frameworks and I’m happy to ignore the purity for the added safety.

MyObject.m

@implementation MyObject

- (void)doSomething
{
    [self doSomethingNew];
}

@end

Shims

In cases where I know the interop between Swift and Objective-C should be fairly short lived I’ll often create small shims. The aim is to write the Swift code in the most natural style and then just write ugly bridge code in the shim with the knowledge that in future I can delete the shim and won’t need to reevaluate the interface of the underlying class. For example I might have

class MyObject {
    func doSomething(completion: (Result<String, Error>) -> Error) { ... }
}

With the above I can’t annotate the method with @objc because Objective-C can’t represent the Result type. Instead of making this less Swifty I’d write a shim like this

class MyObject {
    @objc
    func doSomething(completion: (Bool, String?, Error?) -> Error) {
        doSomething { result in
            switch result {
            case let .success(string):
                completion(true, string, nil)
            case let .failure(error):
                completion(false, nil, error)
            }
        }
    }

    func doSomething(completion: (Result<String, Error>) -> Error) { ... }
}

Wrap up

Although it may not always make sense it’s amazing how many bugs you find when you look at porting Objective-C to Swift. There’s the obvious errors that language features help you avoid writing and then there is just being forced to look at old code with fresh eyes and new patterns.