A few aspects of the shape of tea’s CLI have been bothering me.

While building out the pre-release tea began to take shape as a “universal interpreter”. I have long thought package managers were depressingly basic in operation, a strange thing for such a (potentially) powerful part of the stack. It seemed somehow backwards that if I obtain a script, I first must figure out what package runs it, and then possibly even install a bunch of dependencies in order for it to run. Why can’t the package manager figure this stuff out for you? And thus tea script.ext did just that.

While iterating around this I realized that it wasn’t laziness that was holding me back from implementing tea install , it was a desire to find something better; a better paradigm for using open source; a better paradigm than managing it. So I held off. My coworkers complained that they still needed brew even though we were building its spiritual successor.

Yet I held off because I know that if I make parts of my workflow tedious, I will come up with a solution. I suffer a little now to reap the rewards later.

I would joke with people that “tea wasn’t a package manager”. This was tongue in cheek—of course tea was and is a package manager. But it didn’t seem much like one. I wanted people to stop thinking about the package manager altogether. I wanted to make something where managing packages stops being a part of your workflow at all. You have apps to build, and I want you to build them with tools like mine getting out of your way entirely.

In pursuit of being a universal interpreter without installing packages one must build environments that create a kind of container for the software. It seemed useful to make those environments available more generally.

With tea you build an environment by adding packages with +pkg.com syntax. With the environment assembled you then execute a command. Plus package syntax was a wonderful discovery allowing entirely new categories of workflows.

What was tea? It seemed to have become a tool for assembling environments of open source software which you then apply to other tooling. If we can figure out those environments for you, we do, if we cannot we need you to tell us.

tea was a universal interpreter. So I looked to other interpreters to see if the similarities should go further.

It’s typical with interpreters for just typing them to start a REPL (a read eval print loop). You exchange your Bash prompt for a Python, Ruby, etc. prompt. Should tea do this? We tried it, so tea +deno.land would start a new shell with deno available. It was somewhat a good feature and it was idiomatic with the concept that tea was a kind of interpreter.

But something about it irked me.

tea described itself as a unified package manager, a universal interpreter, a universal environment manager. It was all these things, but you can’t sell a product if you can’t put your finger on what the product *is*.

Last week I talked about how we resolved the messaging by more clearly seeing what tea was. tea is a magical package manager, it’s invisible and powerful. Stop thinking about installing stuff—get on with your work instead. If you don’t need to even think about managing packages entirely you can use new tools and workflows effortlessly, but also other people who make tools, scripts or workflows can know you, the end user for their efforts, have the entire open source ecosystem at your fingertips too. They don’t need to tell you how to install a bunch of stuff, with a bunch of different versions, for a bunch of different platforms in their README—you have tea.

This week I realized that during the history of UNIX there was another innovation in this area. As developers for UNIX like systems exploded, new tools began to appear all over. Each UNIX had different conventions and preferences for where such tools may be installed. This would be problematic for the shebang.

Shebangs

Perl was one of the first interpreted languages. It grabbed the minds of all UNIX users since it was fast to dev like shell scripting but powerful like “proper” languages (eg. C). It supported being “shebanged”, the #! at the front of a text file that told your shell who interprets the file you want it to run:

#!/usr/bin/perl

The problem is: perl was new and not POSIX with an associated “this is where it should be installed” path. It might be installed anywhere! Shebangs were not designed to search PATH (for security reasons presumably) but this impeded the possibility of distributing a perl script to other communities.

So/usr/bin/env was built. It was designed to construct environments† and then execute commands inside those environments searching the specified PATH for the command to run. POSIX declared its path to be /usr/bin/env so now shebangs could be:

#!/usr/bin/env perl

Thus alleviating the location problem.

However the user still needed to install perl themselves.

† though not commonly used env FOO=bar baz and other environment management features are a primary part of env’s CLI.

tea is env++

#!/usr/bin/env -S tea perl

## see `man env` to understand the `-S`

Like env we construct an environment for perl and run it, but as supplemental functionality we install perl and its dependencies first.

One day perhaps we’ll be POSIX and we can reduce this to:

#!/usr/bin/tea perl

One can only dream…

Adapting tea/cli Based on this Realization

tea is a more capable env. Should this be something we document? Should this adapt how we introduce it and talk about it? I began exploring that.

Firstly, our REPL like feature tea +rust-lang.org starts a new shell that contains the environment for rust. Should instead we output the environment like env does?

I tried it and it felt right. We already use this functionality for tea’s magic, so now instead of that being a special mode it was a idiomatic use of tea as a CLI.

However the REPL functionality is neat, so do we really dare lose it?

Fortuantely an idiomatic way to do the REPL presented itself to me, simply append sh, eg. tea +rust-lang.org sh. This is all our REPL was doing underneath anyway. Now tea doesn’t have any special modes, we are communicating to the user intuitively that tea installs packages, creates environments and executes commands within them. To see the environment that will be created, stop typing (eg tea +rust-lang.org). To use the environment, add commands.

Typing tea by itself dumps the current supplemented environment. If you have our magic installed then inside a developer environment you can see what tea did to create that environment.

I’m super happy with these changes as now I feel tea is close to being very well scoped, intuitive to understand, composable, flexible and useful to many different tasks along the lines of its utility proposition. We are both more powerful than all other package managers while also being more intuitive.

I foresee a v1 in the near future.

Breaking Change

This is a breaking change, but we are pre v1. After v1 we will not break the CLI API unless we bump the major version.

dev.diary — tea is env++ was originally published in teaxyz on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is the approach I take with my products. From my childhood writing toys in BBC Basic, to working on the amaroK music player for KDE in the early 2000s, to making the Android version of TweetDeck in 2010, to how I approached making Homebrew and now with its spiritual successor: tea.

We released tea/cli a couple months ago. At that point I felt the product was “ready enough” for general consumption. We had a number of compelling features coupled with a good mission statement and together to me that seemed enough to present it to the community and start gathering wider ideas and interest.

Which is a key differentiator between developing most products and developing open source: you bring your userbase in before you’ve “found the fun”.

Before launch I jokingly would say that tea was not a package manager. I was being flippant. But also I wasn't being flippant. Package Managers as a class of tool haven’t evolved since their inception in the 90s. They are very much in the what I call “egotistical tool” category: that being: you *will* learn how it works thank you very much because the package manager is a very important tool and you are lucky to have it. I’ve not a fan of this attitude.

I believe tools are a special class of utility. A well written tool completely eradicates entire classes of tedium enabling its users to reach new heights in productivity. A well written tool enables functionality that before was either too tricky to accomplish or completely impossible.

At launch we said tea was a universal interpreter, a universal environment manager, a unified dependency manager and it could run executable markdown.

Changelog rightly called me out on this, saying “might Max and the team be thinking too big this time around?”

The truth is we hadn’t yet quite found the fun.

Two months of passionate early adopter usage later and a few things were becoming apparent. tea’s real super powers were the underlying rethinking about the nature of package management, specifically that with tea the package manager disappeared. So I decided to experiment with doubling down on that.

At launch you could type tea -X npm startand tea would look for npm and run start , but if it didn’t find npm it would fetch it first. This was in fact a very near launch addition by me but it was a truly wonderful reduction in friction for using the open source ecosystem. And fundamentally what this feature said was: the package manager needs to get out the way.

tea doesn’t install packages—at least not in a conventional sense. In the above example npm is not available unless you type tea -X npm otherwise it is “stowed” in ~/.tea . This is important since I want using tea to complement your system. tea is not an operating system, it is a complement to whatever system (Mac, Linux, Windows, etc.) you are using.

Soon after a passionate early adopter of tea suggested that symlinks to tea should behave as though tea -X foo bar was called. So we implemented that and immediately it was apparent that this was a seriously cool feature. I used this to create a symlink from tea to bpb (a tool that makes signing your git commits trivially easy) and immediately knew that forever more I would not have to piss about for an hour getting my git-signing set up whenever it stopped working or a had to set myself up on a new machine, remote server or when clean installing my OS.

A developer at tea realized that you could make the shell’s command-not-found handler call tea -X . Command not found handlers have been used by the package manager before. For example, Ubuntu uses it to suggest packages that may satisfy whatever you typed. The idea made me laugh because it seemed so ill-advised! But we added it to the README as a hack that users may want to try out—for fun.

But I got to wondering… had we in fact found the fun? I decided to add the command-not-found handler to my shell and see how I liked it.

After a few days I stopped laughing. I started getting excited. This hack wasn’t a hack. It was awesome. It completely delivered on tea’s promise and significantly simplified our narrative.

How do you use tea? You don’t. You use the tools that the open source ecosystem provides. If you read a blog post about a new tool then just type the commands you read into your Terminal and try it out. There’s no “figure out the package name and then install it step”. There’s no “will installing this break other stuff on my system” concerns. If you don’t like it then don’t worry about it—it’s already gone.

# want to see if bun will run your node project?
$ bun run start
tea: stowing bun^0.4
bun: start

$ which bun
bun not found
# ^^ it’s not in your PATH!

$ bun --version
0.4.0
# ^^ but tea surfaces it for you

tea still is a “universal interpreter” but that just falls under the general banner; we don’t have to explicitly name this feature; tea just knows how to make what you throw at it work. Be that a command or a script, tea makes it possible for you to get on with what matters: making your app.

And so I dropped executable markdown completely. Part of the reason I added exe/md was the desire for tea to be a tool that makes development a dream. And we have achieved this because now if you want to use executable markdown to run the script build you just type xc build and tea installs xc transparently. Whatever “task runner” your project needs is transparently available. Having this feature in tea itself no longer made sense.

When it comes to making great products you need to be able to drop entire features without ego or the sunk-cost fallacy holding you back. What matters is the entire product vision no matter how cool you think something is. I still think exe/md is a cool idea, and didn’t know about xc before writing it. I’m very glad someone else thought the same and built it. UNIX tools should do one thing and one thing well.

tea 0.19 lives up to the UNIX philosophy and thanks to Changelog for reminding me about that. I revisited every part of the CLI and retooled it towards this purpose. Check out the new README for the deets, the caveats and what comes next:

https://github.com/teaxyz/cli

dev.diary—Finding the Fun was originally published in teaxyz on Medium, where people are continuing the conversation by highlighting and responding to this story.

From the start, I designed it to be delightful where other tools were not. I have always firmly believed when building anything, especially open source, that the tooling should get out of your way because you have much more important work to be doing.

I also intentionally designed it to go viral. I was lazy, you see, in the way all the best developers are. I needed you — all of you — to help me build it. I did this so well that Homebrew became one of the biggest open source projects of all time.

Not long after starting, I quit my job to work on this project I felt I was put on the world to make. However… within a few months, the task was long from complete, and I found myself skint. I still remember going to the bank and asking what would happen if I overdrew the next day to pay for rent. They didn’t care much.

Thus began a steady rotation between jobs and open source. I wanted to work on open source full time, believing it would help the world become a better place. But that remained an elusive dream.

Eventually I burned out and quit maintaining Homebrew. I handed the reins to the community that had emerged from the trail I had laid out. I’m proud to say the project has done very well without me.

A few times a year someone would ask me if I would make a brew2. I always said no. What would be the point? brew is good, and I’m not going to reinvent the wheel.

A Wild Timothy Appears

I met Timothy in Chicago in late 2013. We clicked and worked together on many projects. We were both passionate about what people can build in open environments that encourage and incentivize contribution. Eventually I moved away but the friendship stuck. Timothy caught the wave of crypto while I moved on to become a top mobile developer.

Every year or so he’d hit me up and talk to me about what was new in crypto, trying to get me to bite. I always declined. I admired the bitcoin white paper but I felt crypto was just money and, for me, money was the least interesting consequence of work.

A Change of Heart

My partner and I were trying for a baby last year and in September, she showed me the positive test. We were overjoyed, but the next morning I woke up in a cold sweat. How was I going to provide for my family? Open source was not paying the bills.

I went back over all my old ideas looking for a startup idea I could turn into a business. Eventually I came back to my ream of notes about “brew2”. Despite previously snubbing the idea of making another, I am an obsessive note taker and had been recording my thoughts as they came to me for years.

I moved on, at the time still convinced open source could not pay the bills.

Timothy phoned me to catch up and once more pointed me to what he was now calling “web3”.

I sniffed around the top of the rabbit hole.

While learning about web3 I bought and sold a few NFTs. The process was mostly uninteresting except for when I sold one and saw the automated, unavoidable 10% royalty enforced by a digital contract (with no need for a legal structure) that compensated the original creator for secondary sales. I felt the sting of inspiration.

web3 enables indirect compensation.

I wondered if we could apply this concept to helping distribute value to open source.

I dove down the rabbit hole.

The Nebraska Problem

It’s a well-known problem. Developers make fundamental improvements to the nature of the Internet. Their contributions are grabbed by the community with both hands and inserted into the tower:

Once a block is placed it’s rarely removed. The maintainers become charitable volunteers.

Solutions like sponsorship and bounties exist. Both unfortunately reward only the top of the tower, the visible packages. Beneath them lie tens of thousands of vital packages that most people don’t even know exist.

Recently Log4J hit the news with a zero day that impacted most of the Internet. They fixed the bug and endured abuse while pointing out that they receive no funding from the myriad of unicorns that use them. They probably still don’t. Most of these hyper profitable corporations didn’t even know that Log4J was a vital brick in their tower of open source built by people they don’t even know and have never even paid.

CoreJS is another famous example, downloaded 30 million times a week. Their README is depressingly laced with genuine need for funding to help them keep every Node.js application that exists afloat.

Let’s Brew a Fresh Pot

Tools like Homebrew lie beneath all development tools, assisting developers to actually get development done. We know the graph of all open source, which means we’re uniquely placed to innovate in interesting and exciting ways. This is exactly what tea will do. We’re taking our knowledge of how to make development more efficient and throwing innovations nobody has ever really considered before. Package managers haven’t been sexy. Until now.

Most importantly, we’re moving the package registry on-chain (relax, we’ll use a low-energy proof of stake chain). This has numerous benefits due to the inherent benefits of blockchain technology:

• Packages will be immutable (no more left-pad incidents)
• Packages will always be available (we’ll use decentralized storage)
• Releases will be signed by the maintainers themselves (rather than a middleman you are told you can trust)
• Tools can be built to fundamentally verify the integrity of your app’s open source constitution
• Token can flow through the graph

Token flowing is where things get really interesting, but first let’s set things straight: we’re not changing the nature of open source. It’s still free. web3 has enabled novel new ways to distribute value, and with our system people who care about the health of the open source ecosystem buy some token and stake it. Periodically, we reward this staking because it is securing our token network. We give a portion of these rewards to the staker and a portion to packages of their choice along with all the dependencies of those packages.

Note that no portion goes to us. We’re not like the other app stores.

What is tea?

Founded by Max Howell, the creator of Homebrew and Timothy Lewis, famous web3 evangelist:

tea is a delightful product suite that you will adore.

tea is the home to a DAO that will ensure the open source maintainers that keep the Internet running are rewarded as they deserve.

tea is our revolution against a failing system.

Stay in touch: https://tea.xyz & https://linktr.ee/teaxyz

Something new is brewing was originally published in teaxyz on Medium, where people are continuing the conversation by highlighting and responding to this story.

I read somebody recently complaining that installing Swift projects was hard because SwiftPM has no install command. Avoiding going into my opinion on that (I don’t want SwiftPM to go there), I disagree that maintaining a Homebrew tap is hard. I do it for swift-sh, and I even have Travis set up to automatically update the tap for me so I don’t have to do anything.

For example installing swift-sh is this easy:

brew install mxcl/made/swift-sh

And Homebrew will keep it up-to-date.

Why Should I Do This?

I maintain that a super large hindrance to the success of your work is adoption pain. You need to reduce adoption pain as much as possible if you want your work to be successful, it’s that simple.

How To Make a Tap

1. Tag a release for your project ✻
2. Create a new repository on GitHub, name it homebrew-tap†
3. Add a file called your-project.rb, adapting the following:

class ProjectName < Formula
  desc "Describe your project"
  homepage "https://github.com/your/project"
  url "https://github.com/your/project/archive/1.2.3.tar.gz"

  def install
    system "swift", "build",
        "--configuration", "release",
        "--disable-sandbox"

    bin.install '.build/release/your-project'
  end
end

4. Install it:brew install ./your-project.rb, you do this to test your formula, but also because brew will helpfully tell you the SHA256, which you then must add to the formula:

class ProjectName < Formula
  desc "…"
  homepage "…"
  url "…"

  sha256 "INSERT-SHA256-HERE"

  def install
    # …
  end
end

5. Commit and push

6. Try it: brew install your-github-username/tap/your-project --force ‡

That’s it.

✻ SwiftPM requires that projects be tagged with a semantic version, eg.1.2.3 or v1.2.3.

† You can pick a different name than homebrew-tap, the homebrew prefix is required though.

‡ --force is only required for you because you already installed your formula in step 4.

Automating Formula Updates

Maintaining metadata like this is tedious for every release, so automate it with the following Travis configuration (in your Swift project, not the tap):

os: osx
language: swift
osx_image: xcode10.2
env:
  - HOMEBREW_NO_AUTO_UPDATE=1
  - HOMEBREW_NO_INSTALL_CLEANUP=1

deploy:
  provider: script
  on:
    all_branches: true
    condition: $TRAVIS_BRANCH =~ ^v?\d+\.\d+\.\d+$
  script:
    - USER=$(dirname $TRAVIS_REPO_SLUG)
    - PROJ=$(basename $TRAVIS_REPO_SLUG)
    - URL="https://github.com/$TRAVIS_SLUG/archive/$TRAVIS_TAG.tar.gz"
    - ORIGIN="https://$GITHUB_TOKEN@github.com/$USER/homebrew-tap.git"

    # grab the tap
    - brew tap $USER/homebrew-tap
    - cd /usr/local/Homebrew/Library/Taps/$USER/homebrew-tap
        
    # update the formula URL
    - sed -E -i '' 's~^  url ".+"~  url "'$URL\"~ ./swift-sh.rb
    - brew fetch $PROJ || true
    # ^^ fails because SHA is wrong, hence || true
    
    # figure out the sha
    - SHA256=$(shasum --algorithm 256 $(brew --cache --build-from-source $PROJ) | awk '{print $1}')
    
    # update the formula SHA
    - sed -E -i '' 's/^  sha256 ".+"/  sha256 "'$SHA256\"/ $PROJ.rb

    # commit to the tap
    - git remote set-url origin "$ORIGIN"
    - git add $PROJ.rb
    - git config user.name "Travis"
    - git config user.email "bot@travis-ci.com"
    - git commit -m "$PROJ $TRAVIS_TAG"
    - git push origin master

You may need to adapt the variables at the top depending on how you named everything.

You also need to add a GitHub token to your Travis account (consumed above with $GITHUB_TOKEN).

Automatic Bottling

Bottles are brew’s binary downloads, which makes your users happier, but this is entirely optional for you.

Automating this took a lot of trial and error since the documentation for bottling is somewhat lacking. So please adapt what I did:

• Formula
• .travis.yml

Also notably the bottles only work for Swift ≥ 5 and macOS ≥ 10.14.4 due to ABI being a thing. My formula detects if the bottle is ok to pour.

Optional Extras

You’ll see I do a little more work in my formula in order to statically link the standard library on macOS < 10.14.4, but this is optional if you only support Swift ≥ 5, and even then it’s optional (your users will need to reinstall the formula if they upgrade to Xcode 11, and probably only a small minority of people are not on the latest Xcode already).

Add Travis for Your Tap

If you’re feeling keen you can verify that your tap actually builds and that the bottle works. Here’s mine. You can see if your package works on Linux for example (yes, Homebrew supports Linux).

An Idea

Still, it’s work, and not everyone can be bothered. One idea would be for someone to do the above for all Swift projects that create binaries:

1. Scan GitHub periodically for new Package.swift which build binaries
2. Create formula for those repos (via a template obv.)
3. Periodically scan for new releases for those repos and update the formula

Maybe you should build this idea?

Workbench is (currently) a small app that monitors changes in your “dotfiles”, uploading them to your private iCloud account when changed.

There are many “dotfile” backup solutions out there, but I found they all were limited or tedious in some manner. Hence I made my own.

Use of iCloud for services like Workbench is something I’d like to see more of from other developers. It is truly zero‐conf, giving your apps sign‐in‐less access to secure and private cloud‐data storage.

I have already tested it for clean-installs, and it works a treat.

Check it out!

I’m already at five projects this year:

• swift-sh: trivially run Swift scripts with third party dependencies.
• Path.swift a delightful, robust, cross-platform and chainable file-pathing library.
• LegibleError a tool for displaying Swift errors to the user that are less The operation couldn’t be completed. (ThirdPartyModule.(unknown context at 0xx10d6b4a44).SomeError error 0.) and more legible and useful.
• Version a µ‐framework for representing, comparing, encoding and utilizing semantic versions.
• AppUpdater a µ‐framework for updating your open source macOS app based on your repo’s GitHub releases.

Update, now eight:

Workbench, Chalk, Cake.

Who are you exactly?

I’m Max Howell, the creator of Homebrew and an Apple fanboy since I switched from working on Linux open source 13 years ago. I am a prolific and famous creator of open source, but also a renowned developer of high profile apps. I love making developer tools because:

Developer tools are the foundation of the modern world

Nothing makes me happier than working on vital tooling for our community and helping Swift take over the world.

Will you help?

Will you help me fill in the gaps in the Swift toolkit with 52 modular, well-designed, high-quality Swift packages this year? Please support my patreon:

https://patreon.com/mxcl

Max Howell is creating Open Source | Patreon

All my projects are announced on my Patreon first, giving you the opportunity to talk about them before anyone else. Higher tiers also give access to my invite only Discord where you can ask me for help with API design or Swift in general.

Tell Me More!

I have a long list of ideas that I’ve been assembling for years, and I want to give them to the Swift community. All my projects have 95%+ code coverage, 100% documentation, excellent READMEs and are under liberal licenses (I favor the Unlicense nowadays).

I am meticulous with my API design; at the forefront of my mind is always the trade-offs of simplicity, elegance and power. A tool should be easy to learn, easy to master and provide power to those who master it. I care about your developer experience, supporting all the package managers in the ecosystem, updating documentation automatically and thoroughly testing in CI for all the platforms Swift supports (yes, especially Linux). I adhere religiously to semantic versioning, I never, ever want my libraries to break when you update them, or when you update Swift (or Xcode) itself.

I am careful with pull requests, I care about my products and resist feature bloat: libraries should be lean and tight and know their responsibility boundaries.

I have been working with Apple tooling for 15 years, am an accomplished mobile app developer for all platforms and worked at Apple on the beginnings of the Swift Package Manager. I maintain the top-100 pod PromiseKit used by many of the worlds top apps. Oh, and I also created Homebrew.

I understand the intricacies of developing Swift applications, the process, the difficulties, the hurdles and the areas developers like you need improvements.

Community

I am a prolific creator of open source, have managed some of the biggest communities open source has ever had and contributed to hundreds of projects. If I’m full‐time on open source I can be a real boon to our community. Managing open source is like nothing else, being that it is composed of volunteers, but developers are like nobody else, we build our own tooling and we revel in trying out new ideas and ways of working. Producing diamonds from this environment is something I excel at.

What will I make?

I intend to provide:

• Tools. I have three, one of which is mostly finished that aid and facilitate development.
• Libraries. Micro, small, large, model-layer and UI-layer.
• Organizations. I have two ideas for community programs, formed to facilitate and moderate aspects of the Swift packaging community.

Ask mxcl anything

Questions? I have an AmA, go wild!

A real version object is essential as soon as you have to parse one. And this is often enough, I mean, every app you make has one in its Info.plist.

I have for years used the Version.swift that is part of the Swift Package Manager, and this library is that code, extracted, documented, tested and enhanced. Relative to the original implementation it has:

1. A more compact Codable implementation †
2. It implements LosslessStringConvertible ‡
3. Is not a massive-single-source-file (MSSF)
4. Has online documentation (100% documented)
5. Extensions for Bundle and ProcessInfo
6. Removal of the potentially fatal ExpressibleByStringLiteral conformance
7. A “tolerant” initializer for user input like 10.0 or 3
8. Idiomatic implementations for Range<Version>
9. 99% code coverage

I feel ok releasing this as my own, partly because I wrote the original implementation when I was at Apple 😏. And I maintained the license.

The version library has implementations for Equatable, Comparable, Hashable, Codable and Range. It fully conforms to the Semantic Version RFC, which includes support for prerelease identifiers and build identifiers, and we know how to sort and compare these properly (2.0.0-alpha is less than 2.0.0 and 1.0.0..<2.0.0 does not contain it).

Hey there!

I’m Max Howell and I write open source all day, every day. I’ve been doing open source for more than 15 years, and you probably already use some of it (Homebrew anyone?). But, I need your help, currently I’m earning less than minimum wage. Don’t let me need apply to Google again!

https://patreon.com/mxcl

The operation couldn’t be completed. (ThirdPartyModule.(unknown context at 0xx10d6b4a44).SomeError error 0.)

That string is the default localizedDescription for a Swift Error. Instead use LegibleError and you’ll get something more like this:

The operation couldn’t be completed. (ThirdPartyModule.SomeError.networkFailure)

This way both you and the user are less confused and more informed.

LegibleError also provides a method to give better strings for logging and debug, providing all the information we can get from the runtime.

Check it out.

A pain point in Swift’s early life for sure, but for now it’s just extra boilerplate. Well almost, since you have to re-run that command every time you add new tests, it’s easy to forget and then you aren’t testing on Linux properly.

So let’s add a CI hook to do it. Using Travis’s build stages it’s pretty easy to add a single stage that all tests depend on:

# these defaults are used unless we override them in individual jobs
os: osx
language: swift
osx_image: xcode10.1
  
jobs:
  include:
    - stage: pretest
      name: Check Linux tests are sync’d
      install: swift test --generate-linuxmain
      script: git diff --exit-code

    - stage: test
      name: macOS / Swift 4.2.1
      script: swift test --parallel

    - env: SWIFT_VERSION=4.2.1
      os: linux
      name: Linux / Swift 4.2.1
      language: generic
      install: |
        eval "$(curl -sL https://swiftenv.fuller.li/install.sh)"
      script: swift test --parallel

The pretest is simple, we run swift test --generate-linuxmain and then ask git if there’s a diff. If there’s any differences then we (or a contributor) forgot to run swift test --generate-linuxmain before committing.

It would be neat to automate running swift test --generate-linuxmain and then committing those changes, and perhaps GitHub Actions (when final) will get us there, but until then having our CI commit to the repo it is testing against is a really bad idea (infinite CI cycles are not a future I want for you). So adding a check is the best we can get.

We are using Kyle Fuller’s excellent Swift Version Manager to install a specific version of Swift on Linux. The image Travis provides does not have Swift available via APT .

This gives us a matrix that will look similar to this:

If the pretest fails, the tests don’t run and the pull-request will be blocked.

Hey there!

I’m Max Howell and I want to be full-time making, writing and being all about open source. I’ve been doing open source for more than 15 years, and you probably already use some of it (Homebrew anyone?). I need your help to continue, any contribution is most welcome. Thanks so much.

https://patreon.com/mxcl

What is Canopy?

Canopy has apps for macOS and iOS, and it’s available here.

APNs

The Apple Push Notification service. Why is the “s” lowercase? Nobody knows.

The service is a black box on the Internet. Your server connects to it securely and sends small JSON payloads, each for a specific device; Apple then send that to the device. If the device isn’t powered they will “hold” it for a little, if it has power it arrives pretty much instantly.

The connection is a persistent HTTP2 connection, HTTP2 is still pretty new and not much of the Internet is using it. Strictly URLSession supports it, but it will not keep the connection open. While some people do use URLSession for APNs, you should be careful: Apple promise that they consider this (opening and closing many connections in succession) a literal attack and will ban you for it.

Implementing APNs

First thing, go to the Apple provisioning portal and get yourself an APNs key. It comes as a file, transfer it to your server. Make sure it’s not exposed to the web somehow, if anyone received it they could send your users notifications as your app, which could be embarrassing for you.

Secondly you need to send your server the device tokens that Apple gives users of your client-apps. You’ve seen this code before, and now you understand better what it’s for.

However unless you plan to send the same notifications to everybody you also need some manner of associating that token with the user-id system you are using on your server. So send this information as well. Don’t forget you can use the same Codable struct on both ends! (See Part 1)

Perfect Notifications

Perfect provides Perfect-Notifications, so to start with I used that. It was easy to setup and use:

import PerfectNotifications

let configurationName = "configurationName"
let apnsKeyIdentifier = "AB90CD56XY"   // from provisioning portal
let apnsTeamIdentifier = "YX65DC09BA"  // from provisioning portal
let apnsPrivateKeyFilePath = "./APNsAuthKey_AB90CD56XY.p8"

NotificationPusher.addConfigurationAPNS(
    name: configurationName, 
    production: false,  // set to `true` when you go live!
    keyId: apnsKeyIdentifier, 
    teamId: apnsTeamIdentifier, 
    privateKeyPath: apnsPrivateKeyFilePath)

Then you can send a notification:

NotificationPusher(apnsTopic: "your.bundle.id").pushAPNS(
    configurationName: configurationName, 
    deviceTokens: [deviceToken], 
    notificationItems: [.alertBody("Hello!"), .sound("default")]) {
        responses in
        print("\(responses)")
}

If this is a test, grab your token from your phone app with a print statement and hardcode it. After testing though you will need to start storing and grabbing your tokens from your database. You’ll need to add a route to your server (see part 2) and send the token to that endpoint where you then store that token for that user in your database (see part 4).

Troubles

After a couple of weeks I noticed that if many events happened on GitHub in quick succession the Perfect notification engine would continuously send push notifications without end and I’d have to kill the server.

I reported the bug and the Perfect maintainer was very fast to commit a fix, unfortunately it was not a complete fix and I could still experience it. After that the maintainer stopped talking to me. I looked into fixing it myself but figuring out the cause was beyond me, so I started looking into other options.

Vapor have a APNs layer, but using it would mean pulling in most of Vapor. By this point I knew how communicating with APNs worked quite well and I realized I could just write my own because I only needed to implement a subset of the full requirements. For example APNs supports having multiple “pipes” inside the same HTTP2 connection, this number is limited and you have to support knowing how many to use at a time. Canopy doesn’t need this yet and probably won’t for some time, instead I can only send one notification at a time, but my tests showed I can still send almost 600 per second this way. (Apple claim you can send more than 5,000 per second if your code is working efficiently). Also advanced APNs engines will make new connections to Apple as needed, I wouldn’t need more than one.

Reinventing The Wheel

RtW is not that bad; you learn the most when you do it yourself.

Firstly I needed a HTTP2 library. Perfect does it themselves. Vapor uses libcurl. The libcurl I needed though was a newer version than comes with Ubuntu 16.04, so I had to configure, make and make install my own to /usr/local.

Perfect already uses libcurl for other things, so importing this newer version was in fact automatic (also because Linux has no rpath so it will use whichever library is in the library-path first, and /usr/local is there).

Here’s how we prepare the connection:

let curlHandle: UnsafeMutableRawPointer
var url: String
if production {
    url = "https://api.push.apple.com/3/device/"
} else {
    url = "https://api.sandbox.push.apple.com/3/device/"
}
curlHandle = curl_easy_init()

url.withCString {
    var str = UnsafeMutablePointer(mutating: $0)
    curlHelperSetOptString(curlHandle, CURLOPT_URL, str)
}

curlHelperSetOptInt(curlHandle, CURLOPT_HTTP_VERSION, 
    CURL_HTTP_VERSION_2_0)
curlHelperSetOptInt(curlHandle, CURLOPT_PORT, 443)
curlHelperSetOptBool(curlHandle, CURLOPT_FOLLOWLOCATION, CURL_TRUE)
curlHelperSetOptBool(curlHandle, CURLOPT_POST, CURL_TRUE)
curlHelperSetOptBool(curlHandle, CURLOPT_HEADER, CURL_TRUE)

Then to send something:

var json: [String: Any] = //…
json.append(0)
json.withUnsafeMutableBytes {
    _ = curlHelperSetOptString(curlHandle, CURLOPT_POSTFIELDS, $0)
}
curlHelperSetOptInt(curlHandle, CURLOPT_POSTFIELDSIZE, json.count - 1)

// headers
var curlHeaders: UnsafeMutablePointer<curl_slist>?
curlHeaders = curl_slist_append(curlHeaders, "Authorization: bearer \(jwt)")
curlHeaders = curl_slist_append(curlHeaders, "User-Agent: Canopy, Codebase LLC")
curlHeaders = curl_slist_append(curlHeaders, "apns-topic: \(topic)")
curlHeaders = curl_slist_append(curlHeaders, "Accept: application/json")
curlHeaders = curl_slist_append(curlHeaders, "Content-Type: application/json; charset=utf-8")
curlHelperSetOptHeaders(curlHandle, curlHeaders)
defer {
    curlHeaders.map(curl_slist_free_all)
}

// send
curl_easy_perform(curlHandle)

Yikes. And that’s not all. The topic is your app’s bundle id, which for Canopy varies (macOS and iOS), but you may be able to hardcode it if you have one. The json is weird, but documented. The jwt is a unique key that you have to generate once every hour. Here’s my code:

let now = Date()
let payload: [String: Any] = ["iss": teamId, "iat": Int(now.timeIntervalSince1970)]
let jwt = JWTCreator(payload: payload)!
let pem = try! PEMKey(pemPath: "./AuthKey_5354D789X6.p8")
return try! jwt.sign(alg: JWT.Alg.es256, key: pem, headers: ["kid": "5354D789X6"])

Perfect provides the JWTCreator and PEMKey objects. The bangs are safe since any nil here would be a development error.

Now be sure that you don’t create new jwts too often, Apple will close your connection. Also keep the same curl_handle, don’t close it or reopen it. You want a persistent connection. libcurl will automatically reopen it for you should Apple close it which is very handy.

You also have to do more, you need to listen to the responses from Apple, if they send certain HTTP response codes you need to delete those tokens from your database. The Internet thinks that if you ignore these messages Apple start to throttle your connection or worse.

Having done all this my APNs system has stayed up for weeks. I was surprised, but really I have the libcurl people to thank for it all, they did the hard work.

Don’t Write Your Own

If you need APNs pick Vapor.

Hey there!

I’m Max Howell and I want to be full-time making, writing and being all about open source. I’ve been doing open source for more than 15 years, and you probably already use some of it (Homebrew anyone?). I need your help to continue, any contribution is most welcome. Thanks so much.

https://patreon.com/mxcl

Part 6 Coming Soon!

Next week: final thoughts, conclusions and hindsight.