But if you and your team struggle with testing and you think it’s because you lack the discipline to “do things right,” I’d like to offer another way of thinking about testing that has worked very well for many years, for several teams, and in multiple languages. In this series I’m going to focus specifically on iOS development working in Swift. Some topics are different in other environments. I might touch on those eventually, but to keep this already sprawling topic bounded, I’m going to stick to client-side Swift for now.

So none of this may apply to you. But if you feel that your current testing approach isn’t working, I’d like to offer another option:

Mock as little as you can. Mock only at the edges. Minimize dependency injection. Test real code.

This is the point in the conversation at which many a good and wise friend has, with great kindness, called me an idiot. And before I’m done, you might too. And that’s fine. If your fancy DI framework is working for you, you shouldn’t abandon it. Maybe listen to some of my philosophical points and ignore the rest. Or ignore it all. I’m warning you up front where this is headed.

But the best thing I’ve done for testing in multiple code bases has been to delete nearly all the mocks and mostly remove dependency injection. In the process, I’ve also made production code better. I’ve spent the last few months removing mocks from my current project while also improving test code coverage by tens of thousands of lines. Tests are simpler to write and they actually test real things. It’s possible to do a lot of testing with very little mocking.

I want to start this series with a concrete example: a Keychain wrapper. We’ve all used them. I’ve worked with many on several teams (and sometimes several on the same team), and this example is based on lessons learned from all of them.

You’re going to look at this wrapper and think “that’s not an ideal design,” and you’re right. But it’s the kind of design you’ll find in real code bases. We need to be able to test things that aren’t perfect. You don’t need to read it all now. The comments at the top are enough, and I’ll explain the API along the way.

Source code

/// A Keychain wrapper that offers key/value storage with the following features:
///
/// * Takes an identifier to maintain separate stores
/// * Each Keychain offers independent key/value storage
/// * Caches values in memory for performance
/// * Allows reading and writing Data directly
/// * Encodes Strings as UTF-8
/// * Encodes JSONSerialization-compatible types (except String and Data) as JSON
/// * Supports "reset" to delete all keys for this identifier
///
public actor Keychain {
  public let identifier: String
  private var cache: [String: Data] = [:]

  public init(identifier: String) { self.identifier = identifier }

  // MARK: - Data Operations

  public func data(for key: String) throws -> Data? {
    // First check the cache
    if let data = cache[key] {
      return data
    }

    // That terrible `SecItemCopyMatching` call you all know...
    let data = try _data(for: key)

    // And cache it for later
    cache[key] = data
    return data
  }

  public func set(data: Data, for key: String) throws {
    // Set it to the cache and to system keychain
    cache[key] = data
    try _set(data: data, for: key)
  }

  public func removeData(for key: String) throws {
    // Remove it from the cache and the system keychain
    cache[key] = nil
    try _removeData(for: key)
  }

  public func reset() throws {
    // Clear the cache and delete all keys for this identifier
    cache = [:]
    try _reset()
  }

  // MARK: - String Operations -- Encode as UTF-8

  public func string(for key: String) throws -> String? {
    guard let data = try data(for: key) else { return nil }
    return String(data: data, encoding: .utf8)
  }

  public func set(string: String, for key: String) throws {
    try set(data: Data(string.utf8), for: key)
  }

  // MARK: - JSONSerialization Operations

  public func value(for key: String) throws -> Any? {
    guard let data = try data(for: key) else { return nil }
    return try JSONSerialization.jsonObject(with: data, options: [.fragmentsAllowed])
  }

  public func set(value: Any, for key: String) throws {
    let data = try JSONSerialization.data(withJSONObject: value, options: [.fragmentsAllowed])
    try set(data: data, for: key)
  }

  // MARK: - Public Extensions for Common Types (Bool, Int)

  public func bool(for key: String) throws -> Bool? { try value(for: key) as? Bool }
  public func set(bool: Bool, for key: String) throws { try set(value: bool, for: key) }

  public func int(for key: String) throws -> Int? { try value(for: key) as? Int }
  public func set(int: Int, for key: String) throws { try set(value: int, for: key) }

  // MARK: - All those horrible low-level SecItem... wrappers that I'm not going to bore you with

  private func _data(for key: String) throws -> Data? { 
    // ...
  }
  private func _set(data: Data, for key: String) throws {
    // ...
  }
  private func _removeData(for key: String) throws {
    // ...
  }
  private func _reset() throws {
    // ...
  }
}

Some key points to this code that will come up later:

• Calling set(string:forKey:) encodes the value differently than set(value:forKey:) when passing a String. In some cases, a mismatch will lead to returning the wrong value (quoted vs not-quoted). In other cases it may return nil. “Well that’s awkward. Let’s fix it!” But remember, this code has shipped. Millions of keys have already been written to user keychains. If you change the encoding, you’ll need to write a migrator. There may be other code that has hacked around the current behavior and will break if you change it (ask me why I think that might happen…). Before you go redesigning a critical piece of persistent storage, it sure would be nice to have tests, right? As much as we can, we want a testing approach that can deal with things as they are, not just how they should be.¹

• This code cannot be called from a unit test on any platform but macOS if the test lives in an SPM package (rather than a hosted application). To access the system keychain, even on Simulator, requires an entitlements file, and SPM can’t provide it. That means that some kind of “mocking” solution is absolutely required for any non-macOS SPM code that relies on Keychain. It’s literally impossible to test otherwise. The point of this series isn’t “never mock.” It’s to reduce mocking as much as we can.

For the moment, I want to ignore how we test Keychain itself. Maybe it’s been around for years and you’ve never had any bugs from it. Should you even write tests at that point? Meh? We do not write tests for virtue’s sake. They are work, and we need to have a reason to write them. In a later post I’ll discuss the many different reasons to write tests, but chasing ever-higher code coverage isn’t one of them.

I want to start by testing things that use Keychain. Since using Keychain completely breaks unit tests for iOS, we need to do something.

So, I’m going to reach for the tool you’re all thinking of. It’s the wrong tool, but it’s where we all start. I’m going to make a protocol!

Abusing Protocols for Tests and (Dubious) Profit

Here we go. Each public method becomes a protocol requirement:

Source code

public protocol KeychainProtocol: Actor {
  func data(for key: String) throws -> Data?
  func set(data: Data, for key: String) throws
  func removeData(for key: String) throws
  func reset() throws

  func string(for key: String) throws -> String?
  func set(string: String, for key: String) throws

  func value(for key: String) throws -> Any?
  func set(value: Any, for key: String) throws

  func bool(for key: String) throws -> Bool?
  func set(bool: Bool, for key: String) throws

  func int(for key: String) throws -> Int?
  func set(int: Int, for key: String) throws
}

And we build a mock. Because we always need a mock that implements every single part of the subject:

Source code

public actor MockKeychain: KeychainProtocol {
  private var storage: [String: Data] = [:]

  //
  // Data accessors
  //
  public func data(for key: String) throws -> Data? { storage[key] }
  public func set(data: Data, for key: String) throws { storage[key] = data }
  public func removeData(for key: String) throws { storage[key] = nil }
  public func reset() throws { storage.removeAll() }

  //
  // Codable accessors
  //
  public func string(for key: String) throws -> String? { try decode(key: key) }
  public func set(string: String, for key: String) throws { try store(string, for: key)}
  public func bool(for key: String) throws -> Bool? { try decode(key: key) }
  public func set(bool: Bool, for key: String) throws { try store(bool, for: key) }
  public func int(for key: String) throws -> Int? { try decode(key: key) }
  public func set(int: Int, for key: String) throws { try store(int, for: key) }

  //
  // JSONSerialization accessors
  //
  public func value(for key: String) throws -> Any? {
    guard let data = storage[key] else { return nil }
    return try JSONSerialization.jsonObject(with: data, options: [.fragmentsAllowed])
  }

  public func set(value: Any, for key: String) throws {
    storage[key] = try JSONSerialization.data(withJSONObject: data, options: [.fragmentsAllowed])
  }

  //
  // Private helpers
  //
  private func decode<T: Codable>(key: String, as: T.Type = T.self) throws -> T? {
    guard let data = storage[key] else { return nil }
    return try JSONDecoder().decode(T.self, from: data)
  }

  private func store<T: Codable>(_ value: T, for key: String) throws {
    storage[key] = try JSONEncoder().encode(value)
  }
}

Make your mocks small

This is exactly how I’ve seen this problem solved so many times. And now that I’ve written it out myself, I must cry a little. Give me a moment. Please… stop doing this. You can have your mocks. Mock if it makes you happy, but not like this.

If a protocol is large and has basically the same interface as its implementations, then it’s not a protocol, it’s an abstract class. And Swift protocols are not abstract classes. The point of protocols is to adapt types to algorithms. It is not to recreate the pImpl pattern. You shouldn’t have to modify the protocol and the mock every single time you touch the subject’s API.

A protocol, even for a mock, should capture the parts that vary between implementations, not the parts that are the same. Here’s what actually varies, the parts that read and write data to storage:

Source code

public protocol KeychainStorage: Actor {
  func data(for key: String) throws -> Data?
  func set(data: Data, for key: String) throws
  func removeData(for key: String) throws
  func reset() throws
}

The vast majority of Keychain has nothing to do with the system keychain. Most of the methods implement encoding logic, “business” logic. There is nothing about encoding logic that needs mocking, so it shouldn’t be mocked. You want to test it! Instead, that logic should be moved to a protocol extension and shared between Keychain and its mock.

extension KeychainStorage {
  // MARK: - String Operations -- Encode as UTF-8

  public func string(for key: String) throws -> String? {
    guard let data = try data(for: key) else { return nil }
    return String(data: data, encoding: .utf8)
  }

  public func set(string: String, for key: String) throws {
    try set(data: Data(string.utf8), for: key)
  }

  // MARK: - JSONSerialization Operations

  public func value(for key: String) throws -> Any? {
    guard let data = try data(for: key) else { return nil }
    return try JSONSerialization.jsonObject(with: data, options: [.fragmentsAllowed])
  }

  public func set(value: Any, for key: String) throws {
    let data = try JSONSerialization.data(withJSONObject: value, options: [.fragmentsAllowed])
    try set(data: data, for: key)
  }

  // MARK: - Public Extensions for Common Types (Bool, Int)

  public func bool(for key: String) throws -> Bool? { try value(for: key) as? Bool }
  public func set(bool: Bool, for key: String) throws { try set(value: bool, for: key) }

  public func int(for key: String) throws -> Int? { try value(for: key) as? Int }
  public func set(int: Int, for key: String) throws { try set(value: int, for: key) }
}

Now the entire mock looks like this:

Source code

public actor MockKeychain: KeychainStorage {
  private var cache: [String: Data] = [:]

  public func data(for key: String) throws -> Data? { cache[key] }
  public func set(data: Data, for key: String) throws { cache[key] = data }
  public func removeData(for key: String) throws { cache[key] = nil }
  public func reset() throws { cache.removeAll() }
}

Two roads diverged… and that’s bad

Mocks that duplicate a large subject’s API do it one of two ways:

• They copy a lot of the code from the subject.
• They don’t copy a lot of the code from the subject.

Above is an example where I didn’t copy a lot of code from the subject. I reimplemented it in an easy way, which is the most common way I see mocks implemented. I used JSONEncoder for everything rather than having custom logic for String. Since it’s symmetrical, it “works,” but the mock has diverged from the thing it’s pretending to be. Consider this code:

Source code

@Test func testCrossStorage() async throws {
  // Write it as String
  try await keychain.set(string: "abc", for: "key")

  // Read it as data and decode manually
  let data = try #require(try await keychain.data(for: "key"))
  let result = String(decoding: data, as: UTF8.self)

  #expect(result == "abc")  // testCrossStorage(): Expectation failed: (result → ""abc"") == "abc"
}

This test works with the real Keychain, but it fails with the mock that uses Codable internally. Nothing here is invalid. It’s a weird way to use the system, but it’s legal and even useful. If the caller wants the value as Data in order to write it to a file, there’s no reason to round-trip it through String. They may rely on the fact that they know how it’s encoded.

That’s bad, but the test will fail, so at least it would be caught. The real problem is the reverse. Someone designs their system based on the mock’s behavior rather than what you ship. All the tests will pass, but it will fail in the field. That’s the worst case. You spend all this time building mocks, writing tests, running tests, and what do you get for all that? A bug that’s a real pain to figure out because your tests are lying to you. Test your product, not your mocks.

Of course, instead of re-implementing everything for the mock, I could have copied a lot more code from Keychain into MockKeychain, but that’s not any better. It means that today they’re aligned, but as Keychain evolves the two will almost certainly diverge. If the tests don’t verify all observable behaviors (and that takes a lot more than just “100% code coverage”), then you’ll have the same situation.

Instead, I made the mock protocol small. Just 4 methods:

public protocol KeychainStorage: Actor {
  func data(for key: String) throws -> Data?
  func set(data: Data, for key: String) throws
  func removeData(for key: String) throws
  func reset() throws
}

Get, set, delete, reset. Those are what the rest of the code needs to operate. Those are exactly the methods I pulled out into some private helpers because they’re kind of ugly.

No, smaller than that

But now that we’ve pulled these methods into a protocol, why don’t we go further. We could pull them into their own type. What if Keychain HAS-A storage rather than IS-A storage?

So the storage is just a struct:

Source code

struct KeychainStorage {
  let identifier: String

  init(identifier: String) { self.identifier = identifier}

  // MARK: - All those horrible low-level SecItem... wrappers that I'm not going to bore you with

  func data(for key: String) throws -> Data? { 
    // ...
  }
  func set(data: Data, for key: String) throws {
    // ...
  }
  func removeData(for key: String) throws {
    // ...
  }
  func reset() throws {
    // ...
  }
}

And the Keychain uses it.

Source code

public actor Keychain {
  private var cache: [String: Data] = [:]
  private let storage: KeychainStorage?

  public init(identifier: String) {
    self.storage = KeychainStorage(identifier: identifier)
  }

  // For tests, but we'll talk about this much more.
  init(storage: KeychainStorage?) {
    self.storage = storage
  }

  // MARK: - Data Operations

  public func data(for key: String) throws -> Data? {
    if let data = cache[key] {
      return data
    }

    let data = try storage?.data(for: key)
    cache[key] = data
    return data
  }

  public func set(data: Data, for key: String) throws {
    cache[key] = data
    try storage?.set(data: data, for: key)
  }

  public func removeData(for key: String) throws {
    cache[key] = nil
    try storage?.removeData(for: key)
  }

  public func reset() throws {
    cache = [:]
    try storage?.reset()
  }

  // ...
}

Wait, where did the protocol go? What about mocking? Look closely:

private let storage: KeychainStorage?

The storage is optional. What happens in this code if it’s nil? Well, it reads and writes to a local dictionary, and doesn’t do anything with the system keychain. Isn’t that exactly what KeychainMock does?

Keychain is its own mock! All you need to do is set storage = nil. And if you modify Keychain, its mock automatically matches because it’s the same thing. The only thing we’ve removed is the little bit that we didn’t want during testing: writing to the real keychain.

This is a shockingly powerful pattern. I call it “self-mocking” and I use it a lot. A whole lot. Keychain is possibly the most perfect example of it, but many types can benefit from this approach, particularly when you add a few more techniques.

And that’s what this series looks like. This is the simplest form of it, and I’m skipping a few very important details that I’ll discuss soon, like “how do you set storage = nil without a DI system?” Yeah, that’s an important question.² And there will be philosophy. So much philosophy. Stay tuned.

The repository

In 1961 the American Standards Association began developing a new character encoding to replace the dozens of existing systems. In the end they created the 7-bit system that we call the American Standard Code for Information Interchange. We take ASCII for granted today, almost obvious, but it’s actually very carefully designed and surprisingly clever. ASCII is inspired by automated telegraph codes that evolved to drive teletypes.

To simplify hardware implementations, ASCII splits the characters into 8 vertical “sticks,” or columns, based on the first three bits of its encoding. The first two sticks are control characters which are intended to interact with the teletype itself. Things like starting a record or moving the carriage head. And that means, in hardware, you can look at just those first two bits and decide where to route the rest of the information for decoding. Very fast. Very easy to implement.

In stick 3 (0x30-0x3F) come the digits zero through nine. That means you can just stick a “3” on the front of any digit and you get the character for that digit. That’s not an accident. Notice how uppercase and lowercase letters differ by just one bit. Makes case-insensitive search really easy, just clear one bit. Also not an accident.

Before ASCII: (E)BCDIC

IBM, who didn’t invent ASCII but was part of the committee, had decades of experience evolving what eventually became their 8-bit EBCDIC, Extended Binary Coded Decimal Interchange Code. In EBCDIC, the decimal digits start at 0xF0 rather than 0x30, but the concept is the same.

As the name suggests, it extends the previous 6-bit BCDIC format, which was designed to work with punch card equipment and mechanical tabulators, which is why it has what seem like odd “gaps” in its layout. This approach made it more compatible with the existing 6-bit systems and the related equipment. It isn’t particularly well-suited to electrical or magnetic encoding, but it taught IBM the value of lining up digits this way for BCD conversions.¹

If you’re working on business-focused computers, rather than scientific computing, the math generally isn’t very complicated. There aren’t a lot of square roots or logarithms in accounting. You mostly need add, subtract, multiply, and the occasional divide. But you spend a lot of time inputting and outputting numeric results in base 10. So it makes sense to do all your math in decimal so that converting back and forth between digits and their characters is really cheap and you avoid base-conversion rounding errors. And that’s exactly what they did. A lot of IBM computers did all their math in binary coded decimal, and (E)BCDIC and ASCII reflect that.

Math!

Now at the same time, there were completely different computers being built for scientific and engineering computing. While a lot of business computers used 6-, 8-, or 9-bit characters, or “bytes,” scientific computers often worked with 36-bit “words.” They did their math in binary because it’s more efficient. It made it more expensive to convert between numbers and characters for I/O, but scientific computers do a lot more computing than they do I/O-ing.

Sidebar: It’s always the System/360 or the PDP-11

Wait, why 36 bits? Who does that? Well, 36 binary digits gives the same accuracy as 10 decimal digits.

OK, why 10 decimal digits? To match the accuracy of the mechanical calculators they competed against.

Notice the lack of powers of 2? Yeah, our obsession with powers of 2 came later.

And by “later,” I mean 1964 and the IBM System/360.

So many things about modern computers can be traced to the System/360, but one of the most ubiquitous is it used 8-bit characters, and IBM was powerful enough to make that the norm that we use to this day. Before the System/360, “bytes” came in a variety of sizes. After the System/360, a byte was 8 bits.

Despite IBM being a massive supporter of ASCII, the System/360 used EBCDIC because it was more compatible with existing IBM peripherals. So it’s kind of a distraction on our discussion of ASCII, but it’ll come back up later. History isn’t always a simple matter of “this caused that.” Sometimes a lot of things are going on at the same time. EBCDIC technically post-dates ASCII by a few months. Sometimes history doesn’t quite follow the narrative.

The many *SCIIs

Getting back to 7-bit ASCII. ASCII was pretty clever, but it’s also very US-centric. That wasn’t because no one was thinking about other countries and cultures. ASCII is the American Standard Code. It is just one of numerous national standards under the ISO/IEC 646 umbrella. For example, there’s YUSCII, the Yugoslav Standard Code. There’s a Chinese national standard, an Icelandic standard, two Japanese, a couple of Canadian. There’s a ton of them.

The idea was that there were certain invariant characters across standards, the ones in white, and then countries could assign whatever they wanted to certain code points, the ones in gray. And they could get combining characters using a backspace with diacritics. But you’ll notice there are only 9 customizable characters. Not a lot to work with. And building accented characters with backspace makes many text algorithms very hard.

But after the System/360 (I told you it’d come back up), more computers moved to 8-bit characters. And with the rise of microcomputers, most of which were 8-bit, there was this extra bit to work with. Some systems used the extra bit to carry extra information, or used it as a parity bit to detect data corruption. But it could also be used to get another 128 characters. That’s a lot, but it’s still nowhere close to enough to cover everything you’d want, even just for Latin-based writing systems.

So all those ISO/IEC 646 national standards morphed into a menagerie of ISO/IEC 8859 “extended ASCII” encodings, establishing the American national standard as the base that others built on, and thankfully getting rid of the “backspace for diacritics” system. The most popular was Latin-1, based mostly on an older DEC standard called the Multinational Character Set. It does a pretty good job of covering Western European languages, along with several others.

But on the other hand, consider:

• Język Polski (Polish)
• Latviešu Valoda (Latvian)
• Hànyǔ Pīnyīn (Chinese Pinyin)

They’re all “Latin-based,” but they all need characters, mostly in the form of diacritics or tone marks, that aren’t in Latin-1. And if you like diacritics and tone marks on your Latin, you can have that, too:

• Tiếng Việt (Vietnamese)

Oh dear….

So Latin-1 only does a “pretty good job” for Western European languages, but there are still a lot more languages out there using Latin that need their own extended ASCII. Non-Latin alphabets like Cyrillic, Greek, Arabic, Hebrew, and Thai are all encoded using different forms of extended ASCII, too. Even Asian-language encodings like Big5 and Shift JIS are based on ASCII, or at least compatible with it. The number 65 is the encoding of the uppercase Latin A almost everywhere, even in non-Latin encodings. Not an accident.

But this consistency can also be a problem. The encodings are “almost compatible.” They’re so compatible that it is very difficult to look at an arbitrary text file and determine what encoding it’s in. Extended ASCII is also very “dense.” Every possible sequence of bytes is a valid extended ASCII string. If you encounter the string “Tiêìng Viêòt”, have you accidentally treated the CP1258 encoding of “Tiếng Việt” as if it were Latin-1? Or is this text intentional, like it is in this document?

Even worse, there’s no easy way to mix encodings in the same document. Even if you add markup to indicate the encoding of each section, algorithms like searching becomes incredibly hard. And if you jump into the middle of a document, or worse, a stream, you have to rewind to find out what the current encoding is.

It’s…bad.

What if we “unified” all of this? 🤔

It took a few decades to finally build a better way. But that’s for another post.

It’s something that wraps a string value, that may also wrap an int value. That’s it.

public protocol CodingKey : CustomDebugStringConvertible, CustomStringConvertible, Sendable {
    var stringValue: String { get }
    init?(stringValue: String)
    var intValue: Int? { get }
    init?(intValue: Int)
}

You can see how the compiler generates automatic coding keys by running it through swiftc -print-ast. That will output the Abstract Syntax Tree as Swift code, after automatic conformances are injected.

% echo 'struct Person:Decodable { var name: String }' | swiftc -print-ast -

// Output of `-print-ast`
internal struct Person : Decodable {
  internal var name: String
  private enum CodingKeys : CodingKey {
    case name
    @_implements(Equatable, ==(_:_:)) fileprivate static func __derived_enum_equals(_ a: Person.CodingKeys, _ b: Person.CodingKeys) -> Bool {
      private var index_a: Int
      switch a {
      case .name:
        index_a = 0
      }
      private var index_b: Int
      switch b {
      case .name:
        index_b = 0
      }
      return index_a == index_b
    }
    fileprivate func hash(into hasher: inout Hasher) { ... }
    ...
}

It’ll generate an enum, and a few dozen or a few hundred lines worth of conformances. You’ll notice that Equatable implementation is based on a switch statement and numeric values rather than the string comparison you might have expected. Comparing integers is a lot faster than strings.

But the important part is that it generates a very simple stringValue initializer and property. For int values, it just returns nil. It doesn’t support int values.

private init?(stringValue: String) {
  switch stringValue {
  case "name":
    self = Person.CodingKeys.name
  default:
    return nil
  }
}

private init?(intValue: Int) {
  return nil
}

fileprivate var intValue: Int? {
  get { return nil }
}

fileprivate var stringValue: String {
  get {
    switch self {
    case .name:
      return "name"
    }
  }
}

What if we made a struct that did the same thing? Glad you asked:

public struct AnyCodingKey: CodingKey {
    public let stringValue: String

    public var intValue: Int?

    public init(stringValue: String) {
        self.stringValue = stringValue
    }

    public init?(intValue: Int) { return nil }
}

This is my absolute favorite custom type and I use it all the time. This is its most minimal form, and the way you’ll see it pretty often in the wild. Lots of people have invented this under different names.

The form I use is a little more fancy. It supports Int keys, and most importantly it conforms to ExpressibleByStringLiteral so that I can use quoted strings as keys. But at its heart, it’s just this, a coding key that can wrap any String and so can be the key of any JSON object.

Why do I love this struct so much? Well, for one, it gets rid of the need for defining CodingKeys.

struct Person: Decodable {
    var name: String
    var age: Int
    var children: [Person]?
}

init(from decoder: Decoder) throws {
    let c = try decoder.container(keyedBy: AnyCodingKey.self)

    name     = try c.decode(String.self, forKey: "name")
    age      = try c.decode(Int.self, forKey: "age")
    children = try c.decodeIfPresent([Person].self, forKey: "children")
}

The hard-coded string literals may cause you to freak out a little. But here’s the thing. If you’re only implementing Decodable or only implementing Encodable, that string will occur in exactly one place which is exactly the place you use it. That’s better than creating a hand-written constant somewhere else. And I generally recommend that you only implement Encodable or Decodable unless you need them both. Unneeded conformances just add headaches and overhead.

And with this tool some really interesting, and ultimately quite simple, syntax is possible.

extension Decoder {
    public func anyKeyedContainer() throws -> KeyedDecodingContainer<AnyCodingKey> {
        try container(keyedBy: AnyCodingKey.self)
    }
}

extension KeyedDecodingContainer {
    public subscript<T: Decodable>(key: Key) -> T {
        get throws { try self.decode(T.self, forKey: key) }
    }
    public subscript<T: Decodable>(ifPresent key: Key) -> T? {
        get throws { try self.decodeIfPresent(T.self, forKey: key) }
    }
}

And now, custom decoding looks like this:

init(from decoder: Decoder) throws {
    let c = try decoder.anyKeyedContainer()
    name     = try c["name"]
    age      = try c["age"]
    children = try c[ifPresent: "children"]
}

See my advanced-codable repo for lots of examples that handle optionals, default values, robust error handling, and more. But the point isn’t to build a fancy library. The point is that with just a few lines of code, you can implement the things you need for your specific problem. And that the starting point is AnyCodingKey.

func addOne(_ x: Int) -> Int {
    fatalError("Haha! No Int for you!")
}

That’s legal Swift. I don’t think many would find that surprising. Of course that’s legal, right? But it is surprising. addOne claims to be a function that accepts an Int and returns an Int. It does accept an Int, but…it doesn’t return an Int.

“Don’t be silly, Rob. It crashes. It can’t return an Int if it crashes.”

Well, yeah. But it promised to return an Int. It didn’t promise to “return an Int or crash,” did it? The whole point of strong types was that the compiler would enforce our promises, and the compiler doesn’t bat an eye at this code.

“The compiler can’t possibly know everything that might crash.”

I’m not ready to concede that, but even so, the compiler clearly can see that this function doesn’t return an Int. There’s no return intValue anywhere. This should prick our ears a bit. Something is strange here. Is it just compiler magic, or is there something deeper?

Hint: There’s something deeper.

Staring into the Void

Let’s take a step back and think about another thing that should surprise us.

let printSquid: () -> Void = { print("🦑") }

So, printSquid is a closure that takes no parameters and returns…what?

You might be tempted to say “nothing.” But that’s not what the code says. It says it returns Void. What’s Void?

public typealias Void = ()

Void is a real thing. It’s the type of the empty tuple. It’s not “nothing.” In Swift, the type () and the value () happen to have the same spelling. When we mean () as a return type, we traditionally write it using the typealias Void. But () -> () is exactly the same thing as () -> Void. Why “Void?” Because Swift has ObjC roots, and ObjC is C-like, and in C there are void functions that literally return nothing.

In Swift, every function returns something. There are functions that return (), and there’s some syntactic sugar that inserts -> Void and return () for you in some cases. But you’re free to include them if you like (it’s not particularly idiomatic Swift, but it’s legal).

func f() -> Void {}
func g() { return () }

And functions like print that seem to have no return value, really do return Void:

let printResult: Void = print("🦑") // 🦑
print(printResult)                  // ()

If we don’t add the : Void here, the compiler will emit a warning because you usually don’t want Void values, but they’re totally legal.

The measure of a type

Each type has a set of values that are part of that type. Some types have an enormous number of values. Int has around 18 quintillion values. String has even more. And some types have very few values. Most enums have just a handful of possible values. Bool has just 2. A lot of the power of strong types is that we can create small types; types that can only hold exactly the values that are legal for our program. That’s why we prefer enums to String and Int unless we mean “arbitrary text” or “an arbitrary number,” and we prefer structs to Dictionary unless we mean “an arbitrary mapping of keys to values.” Smaller, more constrained types help the compiler help us.

So how small can a type get?

Void has just one value. So that’s pretty small. In most languages that have this type, it’s called Unit. The one-value type.

Can they get smaller?

Not a value in sight

It’s time to get back to the original example:

func addOne(_ x: Int) -> Int {
    fatalError("Haha! No Int for you!")
}

Looking at fatalError, it feels a lot like print. The syntax is similar. But we can’t just throw print into places fatalError goes:

func addOne(_ x: Int) -> Int {
    print("Haha! No Int for you!") // Cannot convert return expression of type '()' to return type 'Int'
    ^^^^^
}

How does the compiler know we’re allowed to use fatalError here, but not print? Maybe the compiler just knows that fatalError is special. That’s what was done prior to Swift 3:

@noreturn public func fatalError(...)
^^^^^^^^^

The @noreturn attribute told the compiler that this function doesn’t return, and then the compiler had logic to handle that case and let us skip returning the Int we promised. But…bleh. I hate that solution. It opens up weird corner cases. For example, what happens if we add @noreturn to a function that we claim does return something, or something that throws:

@noreturn func addOne(_ x: Int) -> Int { ... }
@noreturn func runForever() throws { ... }

The first one is probably an error, and maybe the compiler should forbid it. But what about the second one? Is throwing a “return?” At the implementation level, it actually is. But should this attribute allow it? There’s not a really obvious answer. Is it possible for a function to require a @noreturn parameter? How does this impact function overloading?

In Swift 3 they got rid of the attribute hack, and solved the problem with a type: Never. The signature of fatalError is now:

public func fatalError(...) -> Never
                               ^^^^^

So what’s Never? Is it some new compiler trick? Nope. It’s just a type, an enum with no cases:

public enum Never {}

How many Never values are there? Well, none. You can’t construct one. That has all kinds of interesting implications.

func f() -> Never { ... }   // f never returns
func g(_: Never) { ... }    // g can never be called

struct S {                  // S can never be constructed
    let nope: Never
    ...
} 

enum E {
    case ok(Int)       // E.ok can be constructed
    case nope(Never)   // E.nope can never be constructed
}

// But also interesting:

struct G<Element> {}
let ok = G<Never>()     // This is fine. Never can be a phantom type.

Another interesting implication is that [Never] is an empty array.

Never is the smallest possible type. We call it an “uninhabited type.” There’s nothing special about the name “Never.” You can create your own no-case enum, and it’ll work the same.

// Our own custom uninhabited type
enum NeverReturn {}

func neverReturn() -> NeverReturn {
    // We could call fatalError() here, since generating any uninhabited type
    // is sufficient to create any other. But we can also use an infinite loop.
    // The compiler can prove this will never return. Never isn't just for crashing!
    while true {}
}

func addOne(_ x: Int) -> Int {
    neverReturn()   // It's fine not to return Int, because the compiler knows this doesn't return
}

// While Never can be used to create a NeverReturn, they're not the same type
let never: Never = neverReturn()           // Cannot convert value of type 'NeverReturn' to specified type 'Never'
let neverEver: NeverReturn = fatalError()  // Cannot convert value of type 'Never' to specified type 'NeverReturn'

While it’s possible to create your own uninhabited types, I don’t really recommend it. The Swift team considered having different types for things like “exit” vs “abort” and intentionally chose not to. One uninhabited type is probably plenty. But it’s nice that it’s not some magical name.

You never hit bottom

In type theory, an uninhabited type is often called a bottom type, and written as ⊥. A bottom type is a subtype of every other type. So Never would be an Int and a String and a UIViewController and every other type. The opposite is the top type (⊤), the supertype of every other type. In Swift, that’s Any.

But in Swift, Never isn’t actually a bottom type. If it were, you could write this, and you can’t:

let x: Int = fatalError()   // Cannot convert value of type 'Never' to specified type 'Int'

It’s easy to fix this with a little extra syntax if you need it:

let x: Int = { fatalError() }() 

So Never acts like a bottom type when it’s being returned from a function, but not when it’s being passed to a function or assigned to a variable. It would be more consistent for Never to be a true bottom type, and for it to conform to every non-static protocol (i.e. protocols without static or init requirements). Whether that’s worth the weird corner cases it might create, I’m not sure. But maybe.

Some things just don’t happen

Never is my favorite type in stdlib. It’s been my favorite type since it was introduced in Swift 3, and the Combine framework has completely justified my love of it by applying it to generics.

Publishers generate a series of values or a typed failure:

public protocol Publisher {
    associatedtype Output
    associatedtype Failure : Error
    func receive<S>(subscriber: S) where S : Subscriber, Self.Failure == S.Failure, Self.Output == S.Input
}

But what should Failure be if the Publisher never generates errors? Never, of course. No magic needed. It just works. What if it only can generate a Failure? Well, then Output is Never (see IgnoreOutput). When there’s a proper type, the special cases disappear, and it just works.

So Never has gone from a little-known type solving a little-known problem around fatalError to something Swift developers will probably use every day without even thinking about it. And that makes me very happy.

final class APIClient {
    static let shared = APIClient()
    let baseURL = URL(string: "https://www.example.com")!
    let transport: Transport

    init(transport: Transport = URLSession.shared) { self.transport = transport }

    // Fetch any Fetchable type given an ID, and return it asynchronously
    func fetch<Model: Fetchable>(_ id: Model.ID,
                                 completion: @escaping (Result<Model, Error>) -> Void)
    {
        // Construct the URLRequest
        let url = baseURL
            .appendingPathComponent(Model.apiBase)
            .appendingPathComponent("\(id)")
        let urlRequest = URLRequest(url: url)

        // Send it to the transport
        transport.send(request: urlRequest) { data in
            let result = Result { try JSONDecoder().decode(Model.self, from: data.get()) }
            completion(result)
        }
    }
}

This fetch method is great for getting a model by ID, but I have other things I want to do. For example, I’d like to periodically POST to /keepalive and return if there’s an error. That’s really similar, but kind of different.

// GET /<model>/<id> -> Model
func fetch<Model: Fetchable>(
    _ id: Model.ID,
    completion: @escaping (Result<Model, Error>) -> Void)
{
    let urlRequest = URLRequest(url: baseURL
        .appendingPathComponent(Model.apiBase)
        .appendingPathComponent("\(id)")
    )

    transport.fetch(request: urlRequest) {
        data in
        completion(Result {
            let decoder = JSONDecoder()
            return try decoder.decode(
                Model.self,
                from: data.get())
        })
    }
}

// POST /keepalive -> Error?
func keepAlive(
    completion: @escaping (Error?) -> Void)
{
    var urlRequest = URLRequest(url: baseURL
        .appendingPathComponent("keepalive")
    )
    urlRequest.httpMethod = "POST"

    transport.send(request: urlRequest) {
        switch $0 {
        case .success: completion(nil)
        case .failure(let error): 
             completion(error)
        }
    }
}

Both basically follow this pattern of build an URL request, pass it to transport, and then deal with the result. I know it’s just one line that exactly duplicates, but the structure is still really similar, and it feels we could pull this apart. The problem is that fetch is doing too much.

So maybe we pull out the part that changes and call it Request. But what should Request be? So often, I see people jump to a PAT (protocol with associated type) like this:

// This is a bad idea
protocol Request {
    var urlRequest: URLRequest { get }
    associatedtype Response
    var completion: (Result<Response, Error>) -> Void { get }
}

So what’s the question we ask whenever we make a PAT? Would I ever want an array of these? I think we would definitely want an array of requests. A list of pending requests. Chaining requests together. Requests that should be retried. We definitely want an array of requests. This is a great example where someone might come along as say, if only we had generalized existentials then everything would be wonderful. No. That wouldn’t fix anything. The problem is this treats a PAT like a generic, which isn’t the right way to think about it.

PATs are not “generic protocols” (or at least not in the way you’re thinking)

Generics and PATs are very different things that solve very different problems. Generics are type-parameterization. That means that the types are being passed as parameters to the function. They’re passed at compile time, but they’re still passed by the caller. When you say Array<Int>, you, the caller, get to decide what kinds of elements Array holds. In Array<Int>(repeating: 0, count: 10) , Int is just as much a parameter as 0 and 10. It’s just a different kind of parameter.

PATs aren’t like that. Their associated types are not parameters passed by the caller. They’re hooks provided by the implementor of the conforming type (or whoever wrote the conforming extension). When you conform a type to a PAT, you have to provide a mapping of stuff that algorithms need to stuff this type has. Collection requires an Index type in order to implement subscripts (among other things). Set says “here’s my Set.Index type that Collection algorithms should use when you need an Index type.” Array says “please use Int as my Index for those algorithms.” As the consumer of Set or Array, you can’t change those choices. You can’t say “I want an Array indexed by Character.” That’s not up to you. It’s not a type parameter.

The point of a PAT is to allow algorithms to use the type. If you’re thinking about storage (like putting things in an Array) rather than algorithms, you probably do not want a PAT.

First use it, then build it

Rather than focusing first on how to construct a Request, let’s focus on how we’d like to use one. I wish something would just know all the stuff I needed to send to the transport….

class APIClient {
    func send(_ request: Request) {
        transport.send(request: request.urlRequest,
                       completion: request.completion)
    }
}

This is a kind of “wish driven development.” We “wish” there were some type that could handle the URLRequest and completion handler for us, we pretend it exists, write the code that uses it, and then make it a reality. And the reality couldn’t be simpler:

struct Request {
    let urlRequest: URLRequest
    let completion: (Result<Data, Error>) -> Void
}

OK, that’s simple, but that’s still not quite what we want. There’s no model information in there. I want to create Requests that know about model types, like this:

client.send(Request.fetching(id: User.ID(1), completion: { print($0)} ))

So I want to put User.ID into a system and get User back out in the completion handler, but the system (Request) only understands Data. That means we’re making a type eraser. We’re hiding a type (User) inside Request. How? With one of the simplest type erasers you can have: a generic function or closure. Basically, we just take fetch and wrap it into a closure. Here’s fetch:

class APIClient {
    func fetch<Model: Fetchable>(_ id: Model.ID,
                                 completion: @escaping (Result<Model, Error>) -> Void)
    {
        // Construct the URLRequest
        let url = baseURL
            .appendingPathComponent(Model.apiBase)
            .appendingPathComponent("\(id)")
        let urlRequest = URLRequest(url: url)

        // Send it to the transport
        transport.send(request: urlRequest) { data in
            let result = Result { try JSONDecoder().decode(Model.self, from: data.get()) }
            completion(result)
        }
    }
}

And here’s fetching:

extension Request {
    static var baseURL: URL { URL(string: "https://www.example.com")! }

    // GET /<model>/<id> -> Model
    static func fetching<Model: Fetchable>(id: Model.ID,
                                           completion: @escaping (Result<Model, Error>) -> Void) -> Request
    {
        // Construct the URLRequest
        let url = baseURL
            .appendingPathComponent(Model.apiBase)
            .appendingPathComponent("\(id)")
        let urlRequest = URLRequest(url: url)

        return self.init(urlRequest: urlRequest) {  // Here's the closure that hides (erases) Model
            data in
            completion(Result {
                let decoder = JSONDecoder()
                return try decoder.decode(Model.self, from: data.get())
            })
        }
    }
}

fetching is a generic method, but it returns a non-generic Request struct. This kind of generic->non-generic conversion is an incredibly powerful way to simplify your system and keep generics from spiraling out of control.

You may ask “why a static fetching method rather than creating an init(fetching:completion) extension” For this one, init would probably be fine, but as you think about other kinds of Requests, especially ones with no parameters, it would get messy. For example, it’s hard to build a nice init for /keepalive. (This isn’t a deep design point; it’s just a stylistic choice. You might prefer init(keepAliveWithCompletion:), and that’s up to you.)

In any case, this is how I’d build the /keepalive handler:

extension Request {
    // POST /keepalive -> Error?
    static func keepAlive(completion: @escaping (Error?) -> Void) -> Request
    {
        var urlRequest = URLRequest(url: baseURL
            .appendingPathComponent("keepalive")
        )
        urlRequest.httpMethod = "POST"

        return self.init(urlRequest: urlRequest) {
            switch $0 {
            case .success: completion(nil)
            case .failure(let error): completion(error)
            }
        }
    }
}

Wrapping up the network stack

This is the end of my discussion of this little network stack (though not the end of my discussion of generics). It’s not designed to be a “real” network stack. I don’t expect anyone to use this directly as described. I build stacks based on these principles all the time, but I’ve never had one look exactly like this. They’re each quite specialized to their particular API, and the particular needs of the app. The goal here wasn’t to create a general purpose library to solve all networking problems. The goal was to show how you would extract generic code tailored to a problem. Your API is probably different, and you’ll probably build your solution in a different way. Don’t feel you have to use a Transport and an APIClient and a Request. (Though maybe you should use Transport… :D)

If you want to build a general purpose library around this, I suggest you first build small, custom libraries around several APIs that are very different from each other. Then look for the abstractions. Abstracting too soon, before you really understand the problem, is the most common cause of generic headaches.

There is no such thing as “as generic as possible.” Generic code is abstraction, and abstraction is choices. Choosing to make things flexible in one direction often makes it harder to be flexible in another. I have a long list of code bases where we needed more flexibility, and the first step was to rip out all the “flexible” code that made the wrong assumptions about what kind of flexibility would be needed and was never actually used.

Write concrete code first. Then work out the generics.

If you stick to that, it’ll probably work out ok.

Swift Playground

The current models are User and Document:

struct User: Codable, Hashable {
    let id: Int
    let name: String
}

struct Document: Codable, Hashable {
    let id: Int
    let title: String
}

But now the server API is changing. Document IDs will be Strings, not Ints. (True story.) But really, IDs never really were Ints. I mean, IDs aren’t numbers. What would it mean to add two IDs together? Or divide them? How can I pretend that an ID is a kind of number if most number-like operations would be nonsense? The current design allows me to pass document IDs when I mean user IDs. It even lets me pass random integers when I mean an ID. That can’t be right. IDs are their own thing. They want a type.

As usual, I’ll start very concretely with User and see if anything generic develops. The first step is to lift the ID into its own type.

struct User: Codable, Hashable {
    struct ID: Codable, Hashable { 
        let value: Int 
    }
    let id: ID
    let name: String
}

So now creating a User looks like this:

let user = User(id: User.ID(value: 1), name: "Alice")

That’s ok, but I don’t like the value: label. It violates one of the principles of the API Design Guidelines:

In initializers that perform value preserving type conversions, omit the first argument label, e.g. Int64(someUInt32).

To comply, I should I add another initializer.

struct User: Codable, Hashable {
    struct ID: Codable, Hashable { 
        let value: Int 
        init(_ value: Int) { self.value = value }
    }
    let id: ID
    let name: String
}

let user = User(id: User.ID(1), name: "Alice")

Much better. Document will be almost exactly the same.

struct Document: Codable, Hashable {
    struct ID: Codable, Hashable {
        let value: String
        init(_ value: String) { self.value = value }
    }
    let id: ID
    let title: String
}

It’s not a lot of code, but anytime I’m tempted to cut and paste, it’s time to wonder if there’s generic code hiding in there. After all, most of the model types in this system will probably have an ID.

Maybe a protocol?

When I see code duplication, I often reach first for a protocol so I can extract a generic algorithm. That’s something protocols are very good at. In the case of ID, there are two duplicated concepts: identifiers conform to Codable and Hashable, and identifiers have a “no label” initializer.

It’s important to focus on the duplication of concepts, not keystrokes. DRY doesn’t mean “never type the same letters twice.” The point is to extract things that will vary together. I don’t want to capture “types that include the characters : Codable, Hashable and init(_....” I want to capture “things that behave as identifiers.” So I’m going to capture that concept as Identifier:

protocol Identifier: Codable, Hashable {
    associatedtype Value: Codable, Hashable
    var value: Value { get }
    init(value: Value)
}

extension Identifier {
    init(_ value: Value) { self.init(value: value) }
}

With that, User.ID is simplified to:

struct User: Codable, Hashable {
    struct ID: Identifier { let value: Int }
    let id: ID
    let name: String
}

To use it, APIClient.fetch needs to accept an ID type rather than an Int:

func fetch<Model: Fetchable>(_ model: Model.Type, id: Model.ID,
                             completion: @escaping (Result<Model, Error>) -> Void)

And of course Fetchable needs to add an ID type:

protocol Fetchable: Decodable {
    associatedtype ID: Identifier
    static var apiBase: String { get }  // The part of the URL for this fetchable type
}

Wait a minute… There’s nothing “of course” about that last change. Fetchable used to be a simple protocol. Now it’s a PAT (protocol with associated type). That’s a big change in Swift. Whenever you find yourself typing associatedtype, you need to stop for a moment and think “would I ever want to put this in an Array?” Once you put an associated type on a protocol in Swift today, it’s no longer a “thing.” It’s only a constraint that can be used for extensions and generic functions. It can’t put put in a variable, or be passed to a function, or in any other way be treated as a value.

Yes, someday generalized existentials will improve this in some cases. But before you pine for those days, or reach for a type-eraser, it’s time to think harder about the protocol.

What’s the next line of code?

I want to roll back to the Identifier protocol and ask that question, “would I ever want to put an Identifier in an Array?” I’ve used this protocol in production projects for a long time now, and the answer so far has been no. It just hasn’t come up. As I wrote this article, I tried to invent use cases that needed an Array of Identifiers, and each time the example kind of fell apart. I was always forcing it. But it’s worth walking through the thought process anyway.

If I try to create an Array of Identifiers today, it spits out that infamous error:

let ids: [Identifier] = [User.ID(1), Document.ID("password")]
// Protocol 'Identifier' can only be used as a generic constraint because it has Self or associated type requirements

And this it the point where you cry out “generalized existential!” But that wouldn’t actually change anything. Let’s just imagine that we have a generalized existential or I’ve written an AnyIdentifier type-eraser. Eventually I’m going to wind up with some loop over ids:

for id in ids {
    // ??? the only property is .value, which is an unknown type ???
}

I call this the “what now?” problem. The only thing I can do with id is get its value, because that’s the only thing in the protocol. But each ID can have a different value type. So what can I do with it? Even with the fabled generalized existential, the type of .value would have to be Any. What else could it be? I can’t call fetch with that. I don’t even know the Model type.

“I don’t even know the Model type.” As I said, I’ve used this protocol in several projects and I’ve never needed a list of Identifiers, but as soon as I started writing this article, I realized how weird it is that an Identifier doesn’t know what type it identifies. Originally I was going to just rewrite this article to ignore it, but these kinds of…mistakes?…are important to explore. I hesitate to call it a mistake, because it’s never mattered in any shipping software I’ve worked on. If a type is solving your problems, it’s not wrong. But maybe it could be better.

When you think about it, everything’s a function.

Before I make it better, I want to show how to solve a “what now?” problem without changing Identifier. I know that sounds a little strange, but sometimes you inherit types that you can’t easily change, and it’s good to have lots of tools in your belt that don’t require rewriting half your project every time something is less than ideal. So let me walk through an example where you think you want to use an Array of Identifiers, but don’t.

Let’s say that once an hour I want to refresh all the model objects by re-fetching them. So I build a list of Identifiers to refresh, and get the “can only be used as a generic constraint” error, and now have to decide what to do. The answer is to look again at what I really want. I don’t want a list of Identifiers. I want a list of refresh requests. A refresh request is a future action, and a future action is closure. I typically like to wrap that closure into a type. Maybe something specialized to this problem like:

struct RefreshRequest {
    // The delayed action to perform.
    let perform: () -> Void

    init<Model: Fetchable>(id: Model.ID,
                           with client: APIClient = APIClient.shared,
                           updateHandler: @escaping (Model) -> Void,    // On success
                           errorHandler: @escaping (Model.ID, Error) -> Void = logError) // On failure, with a default
    {
        // Smash together updateHandler and errorHandler into a single () -> Void.
        perform = {
            client.fetch(Model.self, id: id) {
                switch $0 {
                case .success(let model): updateHandler(model)
                case .failure(let error): errorHandler(id, error)
                }
            }
        }
    }

    // Just a helper so errorHandler can have a default value
    static func logError<ID: Identifier>(id: ID, error: Error) {
        print("Failure fetching \(id): \(error)")
    }
}

let requests = [
    RefreshRequest(id: userID, updateHandler: { users[$0.id] = $0 }),
    RefreshRequest(id: documentID, updateHandler: { documents[$0.id] = $0 }),
]

The point of all of this isn’t this specific data structure. It’s that () -> Void is an incredibly powerful and flexible type, and you can construct it from all kinds of other functions. It’s another case of “common currency.” If you want a delayed action, that’s just a function. A lot of complicated generic code comes from trying to keep track of all the parameters to a generic function you intend to call later. You don’t need to keep track of parameters (and their types) if all you need is the function itself. This is the heart of type-erasure rather than focusing on type-erasers. It’s hiding types I don’t care about any more, like Model. Note in this example how two generic closures that rely on Model (updateHandler and errorHandler) are combined into a single () -> Void, non-generic closure that relies on nothing. This is very common technique, and it’ll come up again in this series.

There are more improvements I could make here. The basic closure { someModel[$0.id] = $0 } is going to be duplicated a lot and I could fix that. But I’m going to leave it for now and focus on a better identifier.

A Real Identifier

What I really want is the model type to know its ID type, and the ID type to know its model type. If you remember the APIClient.fetch method, it takes both a type and an identifier:

func fetch<Model: Fetchable>(_ model: Model.Type, id: Model.ID,
                             completion: @escaping (Result<Model, Error>) -> Void)

This creates awkward repetition in the API:

client.fetch(User.self, id: User.ID(1), completion: { print($0)} )

I could add an extra “Model” associated type to the Identifier protocol, but it gets a bit messy. Some of it is just Swift syntax (where clauses and the like), but it really comes down to Identifier not being a very good protocol. Look at the implementations:

struct User.ID: Identifier { let value: Int }
struct Document.ID: Identifier { let value: String }

If you think about any other implementations, they’re going to be almost identical: a struct with a single property called value. It’s hard to imagine any other way you’d want to implement this protocol. If every instance of a protocol conforms in exactly the same way, it should probably be a generic struct.

// An identifier (of some Value type) that applies to a specific Model type
struct Identifier<Model, Value> where Value: Codable & Hashable {
    let value: Value
    init(_ value: Value) { self.value = value }
}

extension Identifier: Codable, Hashable {
    init(from decoder: Decoder) throws {
        self.init(try decoder.singleValueContainer().decode(Value.self))
    }

    func encode(to encoder: Encoder) throws {
        var container = encoder.singleValueContainer()
        try container.encode(value)
    }
}

Identifier has two type parameters. The Model is the type this identifier applies to. The Value is the kind of identifier it requires (Int, UInt64, String, etc). The Model isn’t actually used anywhere, but it means that Identifier<User, Int> and Identifier<Document, Int> are completely different types and can’t be mixed up.

So User becomes:

struct User: Codable, Hashable {
    let id: Identifier<User, Int>
    let name: String
}

That’s ok, but it’d be nicer to typealias it so I can refer to User.ID as a type:

struct User: Codable, Hashable {
    typealias ID = Identifier<User, Int>
    let id: ID
    let name: String
}

And it can be even a little nicer if I extract a protocol, and apply it to Fetchable:

// Something identified with an Identifier
protocol Identified: Codable {
    associatedtype IDType: Codable & Hashable
    typealias ID = Identifier<Self, IDType>
    var id: ID { get }
}

// Something that can be fetched from the API by ID
protocol Fetchable: Identified {
    static var apiBase: String { get }  // The part of the URL for this fetchable type
}

// User model object
struct User: Identified {
    typealias IDType = Int
    let id: ID
    let name: String
}

extension User: Fetchable {
    static var apiBase: String { return "user" }
}

And finally, fetch doesn’t need any type parameters. The only thing that could be fetched with a User.ID is a User:

func fetch<Model: Fetchable>(_ id: Model.ID,
                             completion: @escaping (Result<Model, Error>) -> Void)

client.fetch(User.ID(1), completion: { print($0)} )

Trials and Witnesses

There are a lot of people who are a lot better than I am at this, and I’m sure they would have built this (or something better!) all at once on the first try. But I’m not bad at this stuff, and this is how it usually works for me. I want to stress that I’ve shipped the protocol version of Identifier successfully in several products, and have never run into a case where I actually wanted a more powerful Identifier that knew its Model. It’s just that by playing around (and thinking a lot about Brandon Williams’ excellent Protocol Witnesses talk) I discovered another approach.

Of course I’ve never actually shipped this Identified protocol. Maybe I’m wrong. Maybe it has quirks when you try to use it in real code. Maybe it turns out to awkward or limited for some reason. I won’t know until I ship it in a production project.

I’ll now remind you that stdlib’s Collection protocol required a pretty major overhaul in Swift 3, and tweaks in Swift 4.1 and Swift 5. The stdlib team is definitely better at this than I am, and Collection is probably the most foundational and carefully considered protocol in Swift. And still, it’s hard to get it right on the first try. (For another major example, see the four iterations of Protocol-oriented integers.)

Generic code is hard. There are trade-offs. Some things are hard because Swift is still evolving. And some things are hard because generic code is just hard. Build simply and concretely, and extract solutions as you discover problems. Don’t invent problems for yourself. “It isn’t generic enough” is not a problem. Make sure your generic code is solving a problem you really have, and put it off as long as you can get away with. You’ll probably have to redesign it anyway.

Next time I’ll move beyond fetching models. There are so many other things an API can do. What would that look like?

Swift Playground

If you’re interested in the future of generics in Swift, Joe Groff has a must-read post called Improving the UI of generics. (You should also read the linked Generics Manifesto for background.) In it, he touches on a common confusion in Swift. If you don’t understand what he’s talking about here, don’t worry. Explaining this paragraph is the point of this article.

We gave existential types an extremely lightweight spelling, just the bare protocol name, partially following the example of other languages like Java and C# where interfaces also serve as value-abstracted types, and partially out of a hope that they would “just work” the way people expect; if you want a type that can hold any type conforming to a protocol, just use the protocol as a type, and you don’t have to know what “existential” means or anything like that. In practice, for a number of reasons, this hasn’t worked out as smoothly as we had originally hoped. Although the syntax strongly suggests that the protocol as a constraint and the protocol as a type are one thing, in practice, they’re related but different things, and this manifests most confusingly in the “Protocol (the type) does not to conform to Protocol (the constraint)” error.

Spelling?

In programming languages, the “spelling” of something is the sequence of characters a programmer would type to represent a concept. This is often the most visible and argued-over part of a language. It’s also often a fairly shallow concern to the design, which is why it’s common to use intentionally bad “straw man” names to discuss a concept without getting bogged down in spelling. Consider the concept “true if x or y, otherwise false.” Swift spells that x || y. In SML the same concept is spelled x orelse y. But the spelling difference, the difference between the characters || and orelse, isn’t very important. It doesn’t tell you much about how the language works. A more interesting difference, at least to me, is that || is a stdlib function in Swift, while orelse is hard-coded into the SML compiler, which would likely be true no matter how they were spelled.

In English, some spellings have multiple meanings. The same thing happens in programming languages, and it happened in the last article:

final class AddHeaders: Transport
{
    let base: Transport
    ...
}

The spelling “Transport” has two related, but distinct, meanings. The first refers to the protocol Transport. The second refers to the existential of Transport.

Existential?

The “existential of a protocol” can mean several things, but here it refers to a compiler-generated box that holds a value that conforms to the protocol. To see why Swift needs this box, consider an Array of Transports:

// URLSession and TestTransport both conform to Transport
var transports: [Transport] = [URLSession.shared, TestTransport(...)]

Swift would like to store Arrays contiguously in memory. So for an Array of Ints, the storage looks like this:

+---+---+---+
| 0 | 1 | 2 |
+---+---+---+

There are no pointers or indirection. The Ints are stored one after the other. To find the offset of index 2, you just have to multiply the size of an Int times two. That’s really fast and how you probably expect Arrays to work. Swift does the same thing for structs. It just lays them out field after field (there might be some padding, but that’s not important here).

struct S {
    let a: Int
    let b: Int
}

++--------+--------++--------+--------++--------+--------++
|| S[0].a | S[0].b || S[1].a | S[1].b || S[2].a | S[2].b ||
++--------+--------++--------+--------++--------+--------++

Again, to find the offset of S[2], Swift just has to multiply two times the size of S (which is the same as two Ints). But what happens in a “protocol-typed” Array like [Transport]? Each element might be a different size. What can Swift do?

It makes a box that’s a fixed size (currently five machine words, with three for storage). If the type can fit in the box, then it’s stored in the box. If it can’t fit, then the compiler allocates some space, copies the data there, and puts a pointer in the box. Reference types are already pointers, so it just puts the pointer in the box. In Swift, that box is called an existential container. The thing in the box is called a witness.

See WWDC 2016: Understanding Swift Performance for more on the implementation details.

Sure, but “existential?”

This section is a bit more technical; feel free to skip it if you like.

Why “existential?” Because the Transport protocol asserts that there exists some type that satisfies its requirements. By “some type,” I mean “in the universe of all possible types,” not “types that happen to be in your program.” That assertion may be wrong. It’s possible to define a protocol that nothing could ever conform to. For example:

protocol Impossible {
    func make<A>() -> A 
}

(If you don’t believe me, spend some time trying to implement make. You need to return an instance of whatever the caller requests.)

An existential container is a placeholder box for some unknown type that satisfies the protocol. It’s possible there is no such type, or there may not be any such type in your program. Nothing can be done with it at runtime until a real, concrete value, a witness, is put in the box. The existence of a witness proves that such a type really does exist.

This implicit box isn’t the only example of an existential in Swift. The “Any” types like AnySequence, AnyHashable, and AnyKeyPath often get called “type-erasers” because they hide the concrete type, but they’re also explicit existentials. In future Swift, we may spell implicit existentials as any Transport to parallel the explicit spelling.

While protocols create existential (“there exists”) types, generics create universal (“for all”) types. When you write struct Array<Element> {...}, that’s an assertion that “for all types (Element), there is another type (Array<Element>) with the following attributes….”

Existentials and universals are “duals,” which means that one can be transformed into the other without losing its structure. So AnySequence is a universal type (generic) that’s equivalent to an explicit existential of Sequence (protocol). That’s why when you run into problems with protocols, your solution may be to convert it into generic structs (or vice versa). They solve the same problems in different ways with different trade-offs.

Copy values or code?

If you have a function with a parameter whose type is a protocol, that really means it requires an existential of that protocol.

protocol Transport { ... }
func transmit(data: Data, over transport: Transport) { ... }

In order to call transmit with URLSession, Swift needs to copy the URLSession into an existential, and then pass that to transmit.

What if you used a generic function instead?

func transmit<T: Transport>(data: data, over transport: T) { ... }

This says that the caller gets to decide the type of T. If they pass URLSession, then the compiler creates an implicit overload:

func transmit(data: Data, over transport: URLSession) { ... }

If somewhere else in the code they pass TestTransport, then the compiler creates another overload:

func transmit(data: Data, over transport: TestTransport) { ... }

The entire transmit function is (in principle) copied, just as if you’d written an overload transmit for each type. This is an over-simplification, and the compiler may not actually make all the copies, or it may generate an existential version instead (or in addition). It depends on a lot of things, including the optimization flags. But when you call a generic function, you should think of it as creating a new version of the function written specifically for the type you called it with.¹

• In an existential (protocol parameter) function, the parameters may need to be copied into an existential at run-time.
• In a generic function, duplicate copies of the code may be generated at compile-time.

This run-time/compile-time distinction is a key difference between existentials and generics. Existentials are containers that are filled at run-time. Generics are compile-time functions for generating new code.

Existentials are used when you need to store heterogeneous values that are only known at run-time, for example in a heterogeneous collection.

Generics are used to apply algorithms to types that are known at compile-time. Protocols constrain what types can be used with those generics.

You don’t pass “a protocol value” to a function. You pass the existential of the protocol. Because Swift often converts concrete types into existentials for you, it’s easy to forget that they’re not the same thing. So when Swift doesn’t perform the conversion, it comes as a surprise, and we get the “can only be used as a generic constraint” (i.e. “as a protocol”) error.

What if we made things more generally…um…existential?

So couldn’t Swift just create an existential all the time, even for protocols with associated types (PATs)? Yes, but…it’s complicated. For the most common cases, yes, Swift could automatically create an any Collection<.Element == T>² implicit existential just like it currently has an AnyCollection<T> explicit existential. That idea is called generalized existentials, and I’m pretty certain Swift will add it eventually (maybe even soon). That’ll knock off several of protocols’ sharp edges for some of the most common cases.

But it probably won’t solve as many problems as people expect. Many protocol problems I see in the wild are really just design problems that have little to do with missing Swift features. A generalized existential will get you past the compiler error, but in the process it may let you go much further down a wrong road.

And there are many kinds of types that don’t lend themselves to automatically-generated existentials. The compiler can’t fulfill an init requirement or any static requirements on its own. It needs help from the programmer to determine what the default implementations are. It’s similar for protocols with a Self requirement. It may not always be possible to create a sensible default implementation. For protocols like Decodable that have no instance methods, an existential may not make sense at all.

Why you gotta talk about math so much?

As Joe said, the hope was that existentials wouldn’t really matter. They’re created by the compiler, you can’t access them, and you can’t even refer to them directly in the language today. You’d think they’d be an implementation detail. But sometimes when you type the name of a protocol you mean the protocol and sometimes you mean the box, and sometimes that matters. We’d like to ignore reference counting, too, and mostly we can…except when we can’t.

The point of a protocol is algorithms. Protocols express what a type must be able to do in order to be used in certain ways. Ideally, protocols should have a very small number of requirements, and enable a large number of extensions and generic functions. A good protocol is short, but shows up in a lot of where clauses and extensions. They’re fundamentally about compile-time type concerns. “I want to apply this algorithm to many different concrete types.”

The point of an existential is heterogeneous collections, or “type-erasure” where you want to know less about the specific type and just use it according to an interface. They’re fundamentally about run-time values. “I want to assign values of many different concrete types to this variable.”

They’re not unrelated, but they’re not the same thing. When I say “protocols do not (generally) conform to protocols,” I really mean “existentials do not (generally) conform to protocols.” And when you see “can only be used as a generic constraint,” what the compiler is really telling you is that protocols with associated types (PATs) don’t have an existential.

So what about these opaque result things?

My hope is that after reading all this, you’ll feel more comfortable reading SE-244, which adds opaque return types in Swift 5.1. I don’t expect opaque return types to be an important feature for most developers. Please don’t assume you need to rewrite your code to use them. The problems they solve impact stdlib much more than day-to-day app development in my opinion. Looking over my code, I haven’t found a single place I want to use one.

The importance of SE-244 isn’t opaque return types. It’s that it lays the groundwork for the future of Swift generic code. If that interests you, then you definitely want to study it, and in particular get comfortable with any P (an existential) versus some P (an unknown but concrete type that conforms). Hopefully this article demystifies some of the terminology.

Next time, back to the networking stack and hopefully some more practical concerns.

// Something that can be fetched from the API
protocol Fetchable: Decodable {
    static var apiBase: String { get }  // The part of the URL for this type
}

// A client that fetches things from the API
final class APIClient {
    let baseURL = URL(string: "https://www.example.com")!
    let session: URLSession = URLSession.shared

    // Fetch any Fetchable type given an ID, and return it asynchronously
    func fetch<Model: Fetchable>(_ model: Model.Type, id: Int,
                                 completion: @escaping (Result<Model, Error>) -> Void)
    {
        // Construct the URLRequest
        let url = baseURL
            .appendingPathComponent(Model.apiBase)
            .appendingPathComponent("\(id)")
        let urlRequest = URLRequest(url: url)

        // Send it to URLSession
        let task = session.dataTask(with: urlRequest) { (data, _, error) in
            if let error = error {
                completion(.failure(error))
            } else if let data = data {
                let result = Result { try JSONDecoder().decode(Model.self, from: data) } 
                completion(result)
            }
        }
        task.resume()
    }
}

This can decode any Fetchable model from an API endpoint that has a URL something like https://<base>/<model>/<id>. That’s pretty good, but we can do a lot better. A natural first question is “how do I test it?” It relies explicitly on URLSession, which is very hard to test against. A natural approach would be to create a protocol to mock URLSession.

I hope by the time you’re done with this series, hearing “create a protocol to mock” makes you flinch just a little.

The basic premise of a mock is to build a test object that mimics some other object you want to replace. That encourages you to design a protocol that very closely matches the existing interface, and then your “mock object” will also closely match that interface. This makes your “real” object, the protocol, and the mock evolve tightly in lockstep, and it cuts off opportunities for more powerful protocols that aren’t tied to one implementation. If the only reason you can imagine using a protocol is for testing, then you’re not getting all you could out of it. Protocols can be so much more.

So my goal isn’t to “mock” URLSession, but to abstract the functionality I need. What I want is to map a URLRequest to Data, asynchronously:¹

// A transport maps a URLRequest to Data, asynchronously
protocol Transport {
    func send(request: URLRequest, completion: @escaping (Result<Data, Error>) -> Void)
}

Notice that nothing about that says “HTTP server over the network.” Anything that can map a URLRequest to Data asynchronously is fine. It could be a database. It could be static unit test data. It could be flat files. It could be different routes depending on the scheme.

Now comes the power of retroactive modeling. I can extend URLSession to be a Transport:

extension URLSession: Transport {
    func send(request: URLRequest, completion: @escaping (Result<Data, Error>) -> Void)
    {
        let task = self.dataTask(with: request) { (data, _, error) in
            if let error = error { completion(.failure(error)) }
            else if let data = data { completion(.success(data)) }
        }
        task.resume()
    }
}

And then anything that requires a Transport can use a URLSession directly. No need for wrappers or adapters. It just works, even though URLSession is a Foundation type and Apple doesn’t know anything about my Transport protocol. A few lines of code and it just works, without giving up any of the power of URLSession.

With that in place, APIClient can use Transport rather than URLSession.

final class APIClient {
    let baseURL = URL(string: "https://www.example.com")!
    let transport: Transport   

    init(transport: Transport = URLSession.shared) { self.transport = transport }

    // Fetch any Fetchable type given an ID, and return it asynchronously
    func fetch<Model: Fetchable>(_ model: Model.Type, id: Int,
                                 completion: @escaping (Result<Model, Error>) -> Void)
    {
        // Construct the URLRequest
        let url = baseURL
            .appendingPathComponent(Model.apiBase)
            .appendingPathComponent("\(id)")
        let urlRequest = URLRequest(url: url)

        // Send it to the transport
        transport.send(request: urlRequest) { data in
            let result = Result { try JSONDecoder().decode(Model.self, from: data.get()) }
            completion(result)
        }
    }
}

By using a default value in init, callers can still use the APIClient() syntax if they want the standard network transport.

Transport is a lot more powerful than just “a URLSession mock.” It’s a function that converts URLRequests into Data. That means it can be composed. I can build a Transport that wraps other Transports. For example, I can build a Transport that adds headers to every request.

// Add headers to an existing transport
final class AddHeaders: Transport
{
    let base: Transport
    var headers: [String: String]

    init(base: Transport, headers: [String: String]) {
        self.base = base
        self.headers = headers
    }

    func send(request: URLRequest, completion: @escaping (Result<Data, Error>) -> Void)
    {
        var newRequest = request
        for (key, value) in headers { newRequest.addValue(value, forHTTPHeaderField: key) }
        base.send(request: newRequest, completion: completion)
    }
}

let transport = AddHeaders(base: URLSession.shared,
                           headers: ["Authorization": "..."])

Now, rather than having every request deal with authorization, that can be centralized to a single Transport transparently. If the authorization token changes, then I can update a single object, and all future requests will get the right headers. But this is still unit testable (even the AddHeaders part). I can swap in whatever lower-level Transport I want.

This means I can extend existing systems in a really flexible way. I can add encryption or logging or caching or priority queues or automatic retries or whatever without intermingling that with the actual network layer. I can tunnel all the network traffic over a custom VPN protocol (I’ve done exactly that with a system like this), all without losing the ability to unit test. So yes, I get mocks, yes, I get unit testing, but I get so much more.

For completeness, here’s a “mock” Transport, but it’s probably the least interesting thing we can do with this protocol.

// A transport that returns static values for tests
enum TestTransportError: Swift.Error { case tooManyRequests }

final class TestTransport: Transport {
    var history: [URLRequest] = []
    var responseData: [Data]

    init(responseData: [Data]) { self.responseData = responseData }

    func send(request: URLRequest, completion: @escaping (Result<Data, Error>) -> Void) {
        history.append(request)
        if !responseData.isEmpty {
            completion(.success(responseData.removeFirst()))
        } else {
            completion(.failure(TestTransportError.tooManyRequests))
        }
    }
}

And I still haven’t used an associated type or a Self requirement. Transport doesn’t need any of that. It’s not even generic.

Common currency

The split between a APIClient.fetch, which is generic, and Transport.send, which is not, is a common structure that I look for. Transport.send operates on a small set of concrete types: URLRequest in, Data out. When you’re working with a small set of concrete types, then composition is easy. Anything that can generate a URLRequest or can consume Data can participate. APIClient.fetch converts Data into any kind of generic Fetchable. When angle-brackets and associated types start creeping in, the code becomes more expressive, but harder to compose because you have to make sure all the types line up.

The power of the Internet is that it mostly operates on just one type: the packet. It doesn’t care what’s in the packet or what the packet “means.” It just moves packets from one place to another; packets in, packets out. And that’s why the Internet is so flexible, and the equipment that makes it work can be implemented by numerous vendors in wildly different ways, and they can all work together.

At each layer above the network layer, additional context and meaning is applied to the information. It’s interpreted as user information or commands to execute or video to display. That’s composition, gluing together independent layers, each with their own concerns. When designing protocols, I try to employ the same approach. Particularly at the lowest layers I look for common, concrete types to work with. URL and URLRequest. Data and Int. Simple functions like () -> Void. As I move up the stack, then greater meaning is applied to the data in the form of model types and the like. That means it’s easy to write Transports and many different things can use Transports. And that’s the goal.

This network stack still is nowhere near as flexible and powerful as I want. But now it can fetch a wide variety of model types from a particular type of API in a very composable and testable way. That’s great progress. For some very simple APIs, it might even be done. There’s no need to make it more flexible for its own sake. But I think we’ll quickly find more features we need to add.

Next time, I’ll jump back up to the very top of the stack, to the models, and show where a PAT (protocol with associated type) can really shine.

Swift Playground

I need a new protocol.

protocol Fetchable: Decodable {
    static var apiBase: String { get }
}

I need a protocol that requires that the type be Decodable, and also requires that it provide this extra string, apiBase.

Read that again. It requires that the type be Decodable and also requires other things. I didn’t say that Fetchable is Decodable. It isn’t.

Protocols do not conform

Protocols (with a few exceptions) do not conform to protocols, not even to themselves. A type that conforms to Fetchable, must also conform to Decodable, but Fetchable is not Decodable. Fetchable is not Fetchable. Decodable is not Decodable. Why do I keep repeating this. Because you will forget, and it will bite you. What would it mean if Decodable were Decodable?

func decode<T>(_ type: T.Type, from data: Data) throws -> T where T : Decodable

Well, remember that JSONDecoder’s decode method requires a type that conforms to Decodable. If Decodable (or Fetchable) conformed to Decodable, I could write:

let result = try JSONDecoder().decode(Decodable.self, from: data)

And in fact, I see people try to write that all the time. But how could that possibly work? How can JSONDecoder know which of an unbounded number of possible types you want this JSON to be decoded into? Even if you did it, what could you possibly do with result? It’s only known method would be init(from: Decoder). There are no instance methods on Decodable.

And so again: Protocols do not conform to protocols, not even to themselves.

When will it bite me?

When I say this bites people all the time, here’s a common example: Say you have a protocol and an extension on Array:

protocol MyProtocol {...}

extension Array where Element: MyProtocol {
    func doThing() { ... }
}

And then you have some array of MyProtocol:

let things: [MyProtocol] = ...

You might imagine that you could call things.doThing(). After all, doThing() applies to any array of MyProtocol, and what’s more MyProtocol than MyProtocol? But that’s exactly what you can’t do. The syntax where Element: MyProtocol means “Element conforms to MyProtocol.” And as I’ve repeated many times now: MyProtocol does not conform to itself. In order to add an extension on [MyProtocol], you would need to write an extension with == rather than :.

extension Array where Element == MyProtocol {
    func doThing() { ... }
}

This says that Element is exactly MyProtocol. That doesn’t include types that conform to MyProtocol. It only applies exactly to [MyProtocol].

There’s always usually an exception

OK, what about the exceptions? There are a some protocols that do conform to themselves. @objc protocols do unless they have “static” requirements such as init, or static properties or methods. And in Swift 5, Error conforms to itself so that you can have “untyped error” Results like Result<T, Error>. If Error didn’t conform to itself, you’d have to use a concrete type for the error. But these are compiler-enforced special cases. You can’t make your protocol conform to itself.

Impossible? Or just not implemented?

But could they? Yes, some could in principle. The rule is pretty straightforward: if a protocol includes an init or static requirement, or includes a Self method parameter, then self-conformance is tricky. If there is no such requirement, then it it’s much more straightforward (basically the same as for @objc). There’s no deep reason that Encodable can’t be Encodable. The following could work, and I think would be both sensible and useful, it just doesn’t today:

let encodables: [Encodable] = ...
let json = try JSONEncoder().encode(encodables)

Will this ever work? I don’t know. It’s been brought up a few times on Swift Evolution, and hasn’t been rejected outright. One concern is that adding an init requirement to an existing protocol could break existing usage (possibly in downstream code) in ways that might surprise developers. I haven’t found a clear statement, but it seems the team wants to make this work someday.

It’s even possible that “challenging” protocols could self-conform if there were default implementations. One could imagine a Swift where Collection(1, 2, 3) would return an Array in a Collection existential. (I’m not suggesting that would be a good idea; I really don’t know. It’s just that it’s the kind of thing one could imagine.)

In this series I’m generally going to talk about things I know from experience using today’s Swift or can predict about likely-near-term Swift (i.e. there’s an SE in the works). So any time I say something like “that won’t work,” I mean “without adding a significant feature to Swift that I don’t know is planned.” (Hopefully folks will continue to correct me if I’m misleading about how hard something would be.)

I of course mean “existentials”

I want to talk about this more later, but when I say “a protocol doesn’t conform to itself,” it’s more accurate to say “the existential of a protocol doesn’t conform to that protocol.” But again, that’s for a later sidebar…. The thing to keep in mind is that these two things are different:

func f<T: P>(t: T)    // This requires a concrete T that conforms to P
func f(p: P)          // This requires a variable of type P (pedantically: "a P existential")

So that’s just a quick side-bar. Next time, I’ll continue expanding the network stack.

In 2015, at WWDC, Dave Abrahams gave what I believe is still the greatest Swift talk ever given, and certainly the most influential. ”Protocol-Oriented Programming in Swift,” or as it is more affectionately known, “The Crusty Talk.”

This is the talk that introduced the phrase “protocol oriented programming.” The first time I watched it, I took away just one key phrase:

Start with a protocol.

And so, dutifully, I started with a protocol. I made a UserProtocol and a DocumentProtocol and a ShapeProtocol and on and on, and then started implementing all those protocols with generic subclasses and eventually I found myself in a corner.

Protocol 'P' can only be used as a generic constraint because it has Self or associated type requirements

And then I started throwing things.

For a couple of years, I was rather annoyed at the phrase “protocol-oriented programming.” If by “protocol” you just mean “interface,” then Go is much more “protocol oriented” than Swift. But the more I’ve wrestled with this new paradigm, the more I realized that protocols are more than just interfaces, and POP isn’t deeply about the protocols anyway. It’s about the extensions. But “extension-oriented programming” would be an even worse name. And more than extensions, it’s really, deeply, about generic algorithms. And “algorithm-oriented programming,” well, aren’t we all?

Naming a paradigm is always fraught with trouble. Most modern “object-oriented” languages aren’t object-oriented at all. They’re class-oriented (vs Smalltalk and JavaScript). And most “functional programming” languages are mostly value-oriented (vs FP and point-free). But the point of the names is shorthand for concepts bigger than a word, so let’s not get too caught up on the “protocol” in protocol-oriented programming. The Holy Roman Empire was in no way holy, nor Roman, nor an empire. Discuss.

Beware quotes traveling sans context

The famous “start with a protocol” quote is actually the end of a longer paragraph:

For example, if you want to write a generalized sort or binary search…Don’t start with a class. Start with a protocol.

Or as Dave clarified on Twitter:

Use value types, then if you need polymorphism, make them conform to protocols. Avoid classes.

If you’re reaching for class inheritance, try a protocol and value type instead. That’s very different from “start with a protocol for every problem.” Ben Cohen covered this in much more detail in the WWDC 2018 talk Swift Generics (Expanded).

So notice that we considered a varied number of concrete types first. And now, we’re thinking about a kind of protocol that could join them all together. And, it’s important to think of things as this way around. To start with some concrete types, and then try and unify them with a protocol.

If you take away just one thing from this series, I want it to be this: Write concrete code first. Then work out the generics. Start with concrete types and clear use cases, and find the places that duplication happens. Then find abstractions. The power of protocol-oriented programming is that you don’t have to decide when you create a type exactly how it will be used. When you use class inheritance, you have to design your class hierarchy very early. But with protocols, you can wait until later.

When I most get into trouble with protocols is when I try to write code “as generically as possible.” That doesn’t really mean anything. Abstractions are choices, and when you make a choice to be flexible in one direction, you generally make it harder to be flexible in other directions. Without some clear use cases, you don’t know what abstractions make sense.

So today, I want to come to protocol-oriented programming fresh, with a focus on very every-day problems we face when developing iOS apps in Swift.

Setting the stage

Over the next several articles I’ll be developing a very common system, a general-purpose networking stack that can fetch data asynchronously and decode arbitrary types. You may have built a system like this yourself in Swift. You may have used a framework that does it. The point of this exercise isn’t really the end result (though I think it’s quite useful code), but the process. What questions should you ask, and when, and how do you know what good answers look like? And most importantly, how does this “protocol-oriented” thing guide us? How is it different than other approaches?

I expect that you’re somewhat familiar with Swift, and particularly that you understand the syntax of generic functions and types, and have at least seen an associatedtype before. If you’re just getting started in Swift, maybe bookmark this series for later.

So to get started, I want to show a common starting point that never goes well for me. I’ve made this mistake many times, and I always find myself in a corner eventually. I see a lot of other people make this mistake, too.

// A network Request knows the URLRequest to fetch some data, and then can parse it.
// This will not go well.
protocol Request {
    associatedtype Response
    func parse(data: Data) throws -> Response
    var urlRequest: URLRequest { get }
}

How do I know this won’t go well? I’ll discuss it much more in depth later, but Request is a protocol with associated type (PAT). Any time you create a PAT, you should ask yourself “will I ever want to put this in an Array?” If the answer is yes, you don’t want a PAT. Requests are certainly something you’d want to put in an Array. Lists of pending requests, lists of requests that need to be retried, request priority queues. There are lots of reasons to put a Request in an Array.

You might be tempted to look for a work-around, but don’t. Type-eraser? No. Generalized Existential?!?! …no… Even if you find some “work-around” to the problem at hand you’ll run into other walls very quickly (and I’ve seen that again and again). That “can only be used as a generic constraint” is telling you something important. This isn’t a problem with Swift. This just isn’t what PATs are for. There are other tools for this problem. In later articles I’ll explain why you don’t want these work-arounds, but the basic problem is starting with a protocol before we even know what algorithm we want to write.

So what does “know the algorithm” look like in practice?

Start concrete

A good way to find a generic algorithm is to start with several concrete algorithms, and then make a parameter out of what varies. In this case, I want to fetch several model types from an API and decode them. In order to start concretely, I’ll make some actual types.

struct User: Codable, Hashable {
    let id: Int
    let name: String
}

struct Document: Codable, Hashable {
    let id: Int
    let title: String
}

This may not be our final implementations, but they’re good enough to get started. They’re pretty similar, but not identical, and that’s good for the first concrete types. I’ll want to push the envelope a bit more later, but this is good enough for now.

I also want a client to manage my connection to the server. I’m marking classes “final” to remind you that there’s no class inheritance here. I’m not suggesting you need to include “final” on all your class definitions. It’s not usually necessary. I’m making it a reference type because the client might eventually have some shared state. For example, if a login step were required, I’d want all references to the client to be logged in together.

// A client that fetches things from the API
final class APIClient {
    let baseURL = URL(string: "https://www.example.com")!
    let session = URLSession.shared

    // ... methods to come ...
}

And now I want the code to fetch and decode a User, as a method on APIClient.

func fetchUser(id: Int, completion: @escaping (Result<User, Error>) -> Void)
{
    // Construct the URLRequest
    let url = baseURL
        .appendingPathComponent("user")
        .appendingPathComponent("\(id)")
    let urlRequest = URLRequest(url: url)

    // Send it to the URLSession
    let task = session.dataTask(with: urlRequest) { (data, _, error) in
        if let error = error { 
            completion(.failure(error))
        } else if let data = data {
            let result = Result { try JSONDecoder().decode(Model.self, from: data) }
            completion(result)
        }
    }
    task.resume()
}

I’m sure many of you have written code kind of like this many times. Construct a URLRequest. Fetch it. Parse it. Pass it to the completion handler. Now, what does the code for fetchDocument look like?

func fetchDocument(id: Int, completion: @escaping (Result<Document, Error>) -> Void)
{
    // Construct the URLRequest
    let url = baseURL
        .appendingPathComponent("document")
        .appendingPathComponent("\(id)")
    let urlRequest = URLRequest(url: url)

    // Send it to the URLSession
    let task = session.dataTask(with: urlRequest) { (data, _, error) in
        if let error = error { 
            completion(.failure(error)) 
        } else if let data = data {
            let result = Result { try JSONDecoder().decode(Document.self, from: data) }
            completion(result)
        }
    }
    task.resume()
}

Unsurprisingly, fetchDocument is almost identical except for four changes: the function name, the type to pass to the closure, the URL path, and the type to decode. It’s so similar because I copied and pasted it. And when you find yourself copying and pasting, that’s where I know there’s probably some reusable code. So I extract that into a generic function:

func fetch<Model: Decodable>(_: Model.Type, id: Int, 
                             completion: @escaping (Result<Model, Error>) -> Void)
{
   ...
}

Where should type parameters go?

Before going on, it’s worth exploring the signature. Notice that I pass the type of Model as a parameter. It doesn’t even need a name, because the value won’t be used. It’s just there to nail down the type parameter in the function’s parameters rather than in completion handler’s parameters. I’m mostly doing this to show a technique, and because fetch(2) { ... } is a bit ambiguous to the reader (since all ID types are Int currently). Sometimes this makes sense, sometimes it doesn’t.

A good example where I think it makes a lot of sense is JSONDecoder’s decode method. It’s called this way:

let value = try JSONDecoder().decode(Int.self, from: data)

It could have been designed this way instead:

let value: Int = try JSONDecoder().decode(data)

It would have even been a little shorter that way. But it forces the caller to add a type annotation on the variable, which is a little ugly, and unusual in Swift. If the only place the type parameter shows up is in the return value, I usually recommend passing it as a parameter. But in any case, try writing some code with it, and focus on making things clear at the call-site. ¹

Making fetch generic

Implementing fetch is pretty straightforward, except for one small problem:

func fetch<Model>(_ model: Model.Type, id: Int,
                  completion: @escaping (Result<Model, Error>) -> Void)
    where Model: Fetchable
{
    // Construct the URLRequest
    let url =  baseURL
        .appendingPathComponent("??? user | document ???")
        .appendingPathComponent("\(id)")
    let urlRequest = URLRequest(url: url)

    // Send it to the URLSession
    let task = session.dataTask(with: urlRequest) { (data, _, error) in
        if let error = error { 
           completion(.failure(error)) 
        } else if let data = data {
            let result = Result { try JSONDecoder().decode(Model.self, from: data) }
            completion(result)
        }
    }
    task.resume()
}

There’s this string that’s either “user” or “document”. That’s something that this algorithm requires, but isn’t part of Decodable. So Decodable isn’t powerful enough to implement this. I need a new protocol.

// Something that can be fetched from the API
protocol Fetchable: Decodable {
    static var apiBase: String { get }
}

I need a protocol that requires that the type be Decodable, and also requires that it provide this extra string, apiBase. (See Protocols are nonconformists for more on the difference between “requires Decodable” and “is Decodable.”) With that, I can finish writing fetch:

// Fetch any Fetchable type given an ID, and return it asynchronously
func fetch<Model>(_ model: Model.Type, id: Int,
                  completion: @escaping (Result<Model, Error>) -> Void)
    where Model: Fetchable
{
    let url = baseURL
        .appendingPathComponent(Model.apiBase)
        .appendingPathComponent("\(id)")
    let urlRequest = URLRequest(url: url)

    let task = session.dataTask(with: urlRequest) { (data, _, error) in
        if let error = error { completion(.failure(error)) }
        else if let data = data {
            let result = Result { try JSONDecoder().decode(Model.self, from: data) }
            completion(result)
        }
    }
    task.resume()
}

Retroactive modeling

Now to use it, I need to make User and Document conform to Fetchable.

extension User: Fetchable {
    static var apiBase: String { return "user" }
}

extension Document: Fetchable {
    static var apiBase: String { return "document" }
}

These tiny extensions represent one of the most powerful, and easiest to overlook, aspects of protocol-oriented programming: retroactive modeling. It is quite non-obvious that I can take a type like User that wasn’t designed to be Fetchable, and make it Fetchable in an extension. And that extension doesn’t even have to be in the same module. That’s not something you can typically do with class inheritance. You need to choose a superclass when you define a type.

I can take any type I want and conform it to my own protocols to use it in new and more powerful ways that the original type creator may never have thought of. There’s no need to tie User to this one use case and this one API. That’s why this protocol is called Fetchable rather than something like Model. It isn’t a “model.” It’s “something that can be fetched” and it only provides the methods and properties that allow that. I’m not suggesting that you should create a protocol for every use case, just the opposite. Really good protocols are usable by many algorithms. But you want most uses of the protocol to need most of the requirements. If the protocol is just a copy of the type’s entire API, it’s not doing its job. I’ll talk about that more in later articles.

First checkpoint

I know this has been basic so far. I know many of you “know all this.” This article is a warm-up, and the point of the exercise is not what was built, but how it was built. I started with simple, concrete code, and extracted first a generic function, and then a simple (no associated type) protocol. This is exactly the opposite of starting with a Request PAT and then trying to figure out the callers. This was just the first step. This system is nowhere near as flexible and powerful as it could be, but already it’s meeting the goal I set at the beginning: “fetch a several model types from an API and decode them.” Keep the current goal in mind and don’t let the protocols get out in front of you.

Next time, I’ll push this example further, and start seeing what protocol oriented programming can really accomplish. Eventually I’ll even need a PAT!

Swift Playground

(I don’t have comments on this site, but if you’re interested in any conversations about it, follow the thread on Twitter.)

Originally, I wasn’t comfortable being a full “Conditional.” I’m not much of a guitarist. I can play along reasonably if there’s someone to cover my mistakes, and a group probably sounds slightly better with me than without. But listening to me play alone is an act of love and friendship, not something you’d do on purpose. So I kind of wanted to be called a “Provisional Breakpoint” instead. But that was wrong. Someone who’s played at a Breakpoint Jam is a Conditional Breakpoint. That’s what it means. If you’ve done it, you’ve earned it.

Everyone who played guitar that night was a Conditional, but Ellen Shapiro was clearly our leader. She plays with an energy and style that I want to emulate. She’s much better than I am, but what I’ve played in years, she’s played in decades, so maybe that’s natural. Choosing the right people to copy is a worthwhile skill in itself. There’s nothing wrong with being a beginner. There’s nothing wrong with learning and flailing and trying again. There’s nothing wrong with letting someone mentor you. And there are a lot of ways to mentor. You might not even know when you’re doing it.

When I first asked to play with the Breakpoints a few weeks ago, James said yes, and I immediately panicked a little on the inside, and I asked for the whole setlist so I could practice. And he said sure and sent them. But then he said, you know, it’s fine to just play some of the songs, or even drop out of parts if you’re not up for it. It’s better to have a few songs you’re good at than a bunch you stumble through. And I said yeah, yeah, yeah, I know that. I’m cool. And I did kind of know. But not really. I’d been planning to practice them all and just beat myself up a lot everywhere I stumbled. I needed someone who knows more than me to give me permission to be a beginner, but still let me play. Sometimes you’ll never know the impact of a small kindness.

After the show, Eric Knapp asked me a very useful question. “So, that was your dream, and now you’ve done it. What’s your next dream?” It’s easy to lose your direction when you get to a destination. If you want to keep growing, your goals have to grow with you. My next goal is to be good enough that I can play alone and you’d rather stay than leave, even if you’re not my friend. It’s what I call “a mediocre guitarist.” Eric thought it was a good goal, but suggested a more concrete one: Play one song at an open mic night. And he passed on some wisdom he’d received and I needed. “Don’t wait until you’re ready. Go play before you’re ready. There are lessons you can only learn by doing.” Eric has forgotten more about playing than I know (and I don’t believe I’m exaggerating). It’s good advice.

I talk a bit about learning guitar and make a bit of noise about being a beginner. Some of that is because I love to tell stories, and I hope my stories can help other people. But of course saying “I’m a beginner” lowers expectations and overdone is a cheap way of getting praise. It’s a dangerous thing to make too big a deal about. For all my “I’m a beginner and I’m scared,” Laura Savino has been playing guitar for less time than I have, and this wasn’t her first Breakpoint Jam. So sometimes the brave ones don’t make as much noise. You have to look or you’ll never notice.

I may just have been lucky so far, but I’ve found the guitar community to be incredibly welcoming. James isn’t alone in inviting beginners to play, but the Breakpoints has a special history of it. Except for Ellen, the rest of us (Laura, myself, and Josh Smith) all made our first public performances at Breakpoint Jams. Creating that kind of opportunity is a gift and how you create and sustain a community. If we all “only hire the best,” where do you think the next generation will grow? Thankfully, James is willing to play with folks who are just trying their best. Are we as willing to develop software the same way?

There are senior developers and there are junior developers, and there are developers in between. Different people have different experience and skill. You might be a senior developer in one language or platform or style, and just learning another. But there’s no point talking about “real” developers. If you write software and it runs, you’re a real developer. That’s what it means. You develop software. If you’ve done it, you’ve earned it.

And if you develop software in Cocoa, then you really want to listen to Backtrace. Hope to see you at the next show to benefit App Camp For Girls!

The conversation started by referencing the classic Joel piece, Things You should Never Do, Part I. Leading to my thoughts:

Just finished some major refactoring work, moving ObjC to Swift and completely redesigning its state machine. I absolutely stand behind the pieces I rewrote (which were a constant source of subtle race conditions and bugs, with every fix causing two new problems). I absolutely stand behind the pieces that I have delayed rewriting (which are a spaghetti mess, and incredibly difficult to safely modify, but after some minor tweaks are stable enough to leave alone).

I’m a big fan of “radical refactoring.” I’ve refactored several code bases until there was almost nothing left of the original code. But it was done steadily, only doing major rewrites to individual pieces after painstakingly detangling them from the rest of the code (usually over the course of several releases). And at the end, there was always some “ball of mud” part that was a bit crazy, but just worked and didn’t need to be touched that often, so we let it be.

I’ve even refactored a C program into a Go program, by turning it into two independent processes that communicated over sockets, and moving bits from one side of the API to the other.

(So even “we need to switch languages/platforms entirely” doesn’t stop you from evolving towards a goal.)

But there’s an exception that Joel doesn’t mention (but I think Martin Fowler does): if you have incredibly buggy code, that is, if you don’t have working code, then that’s the time to consider a rewrite. Not ugly code. Not badly designed or horrible to work with code. But code that doesn’t actually work, and several attempts to make it work have failed. That’s when a rewrite (at least of those portions) is likely appropriate.

The discussion then turned to unit testing, and particluarly Forgotten Refactorings.

Having had some very successful radical refactors on code without solid unit test coverage, I think it’s worth discussing how that can be done.

First, unit test coverage is absolutely the best first step. That said, sometimes it is impossible in any meaningful way. When all the most likely and common bugs in your system are race conditions and corner cases involving things outside the program (non-trivial networking, bluetooth, version-specific OS interactions, complex animations, etc), I’ve found unit tests rapidly become tests of mocks, and not tests of the system. We can debate whether or not it is possible or profitable to redesign your system so it is more testable. I’ll even concede that it is and leave arguments about TDD for another day (I’m actually a fan of TDD). But redesigning for testablity will itself require massive refactoring without unit tests (because you can’t unit test until you make it testable). Even if you have lots of tests, refactoring often means changing the tests dramatically (which means you’re not really testing the same thing). So at some point, you’re going to find yourself needing to refactor without perfect (or even barely sufficient) unit tests. How do you do it?

Slow down.

I cannot stress this enough. Slow. Down. Expect your refactor to take many releases. Do a small piece of refactoring, and run it through a full QA cycle (whatever that means for you) and ship it. Do it again and again. My “convert a C project to Go” project included a release where we just shipped the Go code alongside the C code, without even calling the Go code, just to prove it would install and not break anything. Then we built one, tiny, new feature in the Go code. It was so minor and impacted so few users, we were ready to declare it unsupported if it didn’t work. We’d been working on the Go code for almost two years before we cut over to it “for real” (and the vast majority of the code was still in C at that point). But at each step along the way, the system was better, and saner, and more reliable. And at each step along the way, it shipped, and got real field exercise. And we built a lot of tests for it, and we still found bugs that we were unable to build automated tests for. “Fails to determine domain on Mac previously joined to AD domain, but then removed, only on OS X prior to 10.8” or “SMB connection fails to Window 2000 server if username contains space” or “fails to determine correct IP address on Mac with case-sensitive file system if on Cisco VPN.” That kind of stuff.

Second point that goes along with this is to keep your refactor steps contained. I’ve had so many experimental refactor branches that I threw away because they spiraled out of control and touched too many pieces of the system in non-trivial ways. Don’t be afraid to throw away several attempts at refactoring until you can get your change focused enough that the risk is contained. Sometimes that means creating “firebreaks,” an object that wraps the thing you’re refactoring and provides the old API for code you don’t want to touch yet. Creating a firebreak often starts as just a pass-through that does nothing but call methods on the original. Tedious, but often invaluable. They make it possible to move to your new API piece by piece rather than having to touch half the system in one go.

I strongly recommend keeping your commits very focused. “Rename FooAdapter to Foo” should be its own commit. Don’t mix it with changes to API. “Rename X to Y” commits are really easy to code review, even if they touch hundreds of files. But if you also changed logic in there, then it’s a monster. Similarly, anything that is an easy win with little risk (like naming things sanely, or moving some duplicated code into a function), do those first and get them into the main code base. That way, when you discover that your ambitious new design is out of control and have to start over, you don’t lose your easy wins.

Testing is great. Testing is critical. Testing is necessary. But unit testing is not sufficient. And when there are hundreds of test cases that need to be rewritten, they can be a hindrance to refactoring. The more important rule in my experience is go slow and steady and keep shipping.

And yes. Write your unit tests. We’re professionals here.

There’s a huge difference when you compare a talk from someone that has been working a lot on the topic and from someone that studied the topic for giving a talk. Why do people do it then? Talks with a lot of value usually come from unknown people. From these people that from the anonymity worked on a topic and they achieved something that they were willing to share. … People don’t care about the company that person had worked for, or the newsletter that the person had written, but instead, what that person wants to share.

While I agree with Pedro’s concern, I disagree that this is the proper ideal.

I’ve learned more about programming from listening to Daniel Steinberg talk about baking cookies than I have from a dozen talks from intelligent, highly experienced people who don’t have his skill for teaching. There are a number of speakers whose sessions I will attend no matter their topic because it’ll always be worth the time. I get inspired every time I listen to Jaimee Newberry, even though almost everything about her life and style is different than mine. If I watch Chris Eidhof live-code, I know I’m going to see some amazing idea that forces me to rethink something in my code.

I absolutely care who’s giving the talk. And if we’re going to ask people to spend hours or days of their time and hundreds or thousands of dollars to attend conferences, I think we owe the best we can offer.

But while I disagree with some of Pedro’s analysis, he raises a very important point. A community with many teachers and broad sharing is better than one with few teachers and a hierarchical flow of knowledge. How do we improve?¹

We should recognize that there’s nothing wrong with having a core group of known, skilled speakers who draw a crowd. We should celebrate that. We should grow that group, and we should use them to make things even better.

At dotSwift, Daniel Steinberg reached out to all the speakers and offered assistance refining their talks. Conference organizers should encourage and facilitate that kind of mentorship, and use it to reduce the risks of inexperienced speakers. I hope conference organizers speak to each other and share names of promising speakers whom they didn’t have room for. CocoaHeads and other local groups should strive to video their local talks and make them available. Conference organizers should use those to find and contact promising speakers outside the usual suspects. Local lightning talks have a very low barrier to entry and make an excellent way to get into speaking. Not everyone has a local group, but it’s a start.

For those without a local group to video them, I always recommend blogging. Learning to write well is a major part of learning to speak well, and the barrier to entry for blogs is lower. A blog doesn’t have to be constantly updated to be helpful. A single, well-written article can be a huge value. This is another important place for conference organizers to search for new speakers, and I think those who are already well known have a duty to amplify lesser-known blogs that are well written and insightful.

To those who want to speak, I’d like to offer a little advice. I’m not a top-tier speaker. I don’t get invited to many different conferences (thanks to CocoaConf for making room for me so often), and most of my CfP submissions are rejected, but I’ve spoken at 11 conferences over the last 4 years and some of my talks have been very well received, so take my advice for what it’s worth, remembering that it comes from someone with a lot of privilege. Not all of it applies easily to marginalized groups, but hopefully it can be of some help.

First, I want to quote Pedro again:

New announcements from Apple are the perfect source of topics for talks, grabbing it quickly is crucial: Protocol Oriented, Swift Open Source, Extension, Swift in the server… You can build your developer brand around the topic. After a few conferences talking about it, the community will tag you as the expert in the topic X. You might not have worked in a production environment with that new thing, you might not have faced the real use cases and issues, but documentation is perfect to prepare a talk based on it, isn’t it?

Yes! I totally agree with most of this, except that this is positive. I take exception to a few points, though. First, “grabbing it quickly” is not crucial. Few people talk about Bluetooth or CoreMotion despite them being around for years. If those interest you, there’s plenty of room for new talks. (I sure would love some more on Bluetooth!) Would you be a better speaker if you shipped many production products with them? Of course. But you can still help a lot of people understand what’s possible by spending a few months going a little further than most and coming back and teaching. Don’t think you have to know everything before you’re allowed to say something! Just be honest and don’t pretend to know more than you do. There are many topics to explore. Natalia Berdys gives a brilliant talk on random number generation. You can’t get much more niche than that. You don’t have to chase the “current cool thing.”

I don’t know how it is for most speakers, but for me, preparing a talk is very difficult. It took me nearly five months to develop my talk for try! Swift. I completely rewrote it four times and practiced it for weeks. I think many speakers are much faster at this than I am, so it may not be so hard for you. But if you find it challenging and find yourself throwing away draft after draft because you can’t figure out what you’re trying to say, just know you’re not alone. If it matters to you, keep at it, and don’t be afraid to throw away a draft that isn’t working.²

If you want to give a talk, and think I can help you make it better, please reach out. I’ve critiqued talks before, and I’m happy to keep doing it. (I will start by asking you to evaluate the talk using Goethe’s Three Questions. You have been warned.)

And if you read all this and say “hey, I don’t even want to give a talk,” that is absolutely fine. Most people don’t. As anyone who’s scheduled CocoaHeads talks before knows, public speaking isn’t for everyone and one reason there’s a small group who speaks so often is because they’re the ones willing to do it. It’s hard and it’s scary and it’s rewarding and it’s valuable. And sometimes it’s even fun. I hope we can include everyone who wants to be part of it. And I hope there’s always a venue for those who want to listen.

I travel pretty well, but sometimes I make mistakes, and this was one of those times. The deodorant I thought I’d packed turned out to be body wash. Now there are a dozen reason that this shouldn’t really matter, and wouldn’t really matter given the A/C and the weather, etc., but I’m a product of my culture, and it was a bit stressful. I tried to find a drug store on the way to the conference, but I was afraid of being late and finally resigned myself to accepting things as they are and moving on.

And then, in the conference rest room, I discovered a small cache of toiletries under a try! Swift sign saying “if you need one, please take one.” I was dumbfounded. It was a very small kindness, but it mattered to me. I spoke to the conference organizer, Natasha, to thank her. She immediately told me it wasn’t her idea, she’d just copied it from Erik Romijn. I went to thank Erik, and he assured me it wasn’t his idea, he’d just copied it from Django conferences he’d been a part of. So I just wanted to say thanks to whomever came up with this very kind idea.

To me, the best part of try! Swift was chatting with people during the breaks and in office hours. Hopefully the videos will be up soon, since I missed a few presentations due to conversations that ran long. So if you only “attend” by watching the videos online (like I did for the first one in Tokyo), you’ll unfortunately miss the best parts. I wish I could change that for you, but I can’t. I’m sorry.

One of the more interesting talks I had was about cut and pasting from Stack Overflow. This practice gets a lot of shaming, and we were discussing that. I can’t promise that this conversation actually took the form of a Platonic dialog, but I can’t promise it didn’t, either. You’ll have to see me at a conference to find out.

Anubis: I’ve only been programming for a little while. Swift is my first language, and I really don’t know anything yet. I find myself just searching Stack Overflow and cutting and pasting code. I know that’s a horrible way to program, but I don’t know what else to do. Before the Internet, I guess people had to figure it out themselves, but we’ve all gotten so lazy.

Kakophonis: Most of my first years programming were just copying BASIC out of Nibble magazine.

Anubis: You mean reading articles and implementing what they taught?

Kakophonis: Oh, no. I mean typing hundreds and hundreds of lines of code for full programs that they included.

Anubis: So you didn’t really learn much those first few years.

Kakophonis: I learned a lot. I learned to use my tools. I learned to debug because I made a lot of typing mistakes. But most of all, I learned what good programs looked like. These were working programs written by good programmers. And I didn’t just read their code, I started changing it. Not a lot. I didn’t know a lot. But I learned to change the colors, or add a trivial feature, or just make random changes that made me happier with it.

Anubis: When did you stop copying other people’s code and become a real programmer?

Kakophonis: I copy code all the time.

Anubis: As a short-cut, right? To make a deadline?

Kakophonis: Not at all. I use others’ code whenever it’s beautiful and useful and they’ve shared it. But you’re right that I copy code a little differently than a beginner does.

Anubis: How so?

Kakophonis: I rarely cut and paste. Maybe it’s a habit from my magazine days, but I usually retype the code by hand. Stack Overflow answers aren’t very long, and retyping gives me a chance to really think about what the code is doing. And then, if the code works and solves my problem, I usually restyle it to match my preferences and make it a little more my own. I almost always rename things. Sometimes I rewrite it from scratch. I make sure I know what each line does and why it’s there.

Anubis: Is that all?

Kakophonis: If the code is more than a line or two, I generally add a comment crediting the source.

Anubis: For politeness?

Kakophonis: Yes, but more importantly, I want future maintainers, which is sometimes me, to know the context around the code, why it does things this way. That keeps them from creating regression bugs if they ever need to rewrite it further.

Anubis: It sounds like you might learn more about programming from copying that way than from doing it yourself.

Kakophonis: Exactly. But copying only improves your code and your understanding if you choose the right things to copy. Many answers on Stack Overflow are incorrect, or they’re presented as magic incantations that may work, but are fragile. A mark of a good programmer is the ability to distinguish what they should copy from what they shouldn’t. Stack Overflow provides beginners some clues about quality, like votes and reputation and comments, but ultimately it’s experience that will help you recognize good code.

Anubis: So Stack Overflow doesn’t make us weak programmers?

Kakophonis: It’s a tool. It is what you make of it. Programmers today have many more resources than they did before the Internet. It’s easy to think that the previous generations had it much harder. But computers and programs were also much, much simpler. No networking. No threading. Not even a GUI. I’m amazed that anyone is able to jump straight to building an iOS app with no programming experience. I had 25 years of experience before I encountered my first multi-threaded program. No, I don’t think beginning programmers today are weak.

Sometimes life is not unlike programming. Who and what we choose to copy says a lot about us. Choosing to use someone’s idea is still a choice. Give credit to your inspirations, but don’t discount the wisdom in picking something beautiful to copy.

This isn’t to say we’re close. We see each other at conferences. We email and tweet. We’re members of a community. So of course I have some idealized picture of him in my mind, without all the this and that of a real person. Even so, he inspires me. When my wife, Janet, and I talk about what’s ahead for our future, we always talk about Daniel and his wife, Kim. I’ve seen them at conferences, traveling together, independent but a team, and I think, hey, we could pull that off when the kids go to school. And Janet and I talk about how to make that work. And we know we only have the shallowest understanding of their real lives, but they inspire us.

Kim died this week. I didn’t know her well. We’d met, and she was nice to me. She and Daniel always seemed so “together” even when they were apart most of the day. Hearing the news unmoored me and scared me and made me think about myself and my family and my plans. And then my heart broke for Daniel.

I’m not kind by nature. But I’m part of a community that is, and part of that is Daniel’s influence on us. Ellen Shapiro has been kind and set up a fund to let us support SmileTrain, where Daniel and his daughter Maggie have asked us to give in Kim’s name. I had never heard of SmileTrain, but it is so perfect. It is kind. And it is practical. It is literally the gift of a smile. It is Daniel and Kim.

Forgive me, NSData. I was running around with that flashy [UInt8], acting like you didn’t have everything I need. I’ve learned my lesson.

— Rob Napier (@cocoaphony) September 28, 2015

I did a lot of writing and rewriting of the Swift version of RNCryptor. I struggled especially with what type to use for data. I gravitated quickly to [UInt8] with all its apparent Swiftiness. But in the end, after many iterations, I refactored back to NSData, and I’m really glad I did.

This is the story of why.

In fair frameworks, where we lay our scene

First, I want to be clear that I don’t think [UInt8] is bad. In some places it’s better than NSData, but there are tradeoffs, and ultimately I found the tradeoffs favored NSData today. Some of those will improve in Future Swift, and I suspect something more Swifty than NSData will be the way of the future. But today, in the kinds of projects I work on, there’s a lot going for NSData.

“In the kinds of projects I work on” is an important caveat. I mostly build things that are used by other developers; frameworks, engines, services, even just snippets of code. I don’t build a lot of full applications that an end user would see. And the systems I build typically have a very small API surface. They typically do just one thing, and they’re built to be easily clicked together with things that I didn’t write.

To achieve that, I try to make most of my code as self-contained as possible, with minimal dependencies. I try to make it easy to plug into whatever system you prefer. That usually means sticking as much as possible to the types provided by the system. Much as I love Result, none of my systems use it externally (and only a few use it internally). I try to avoid exposing my caller to any cleverness. If they’re familiar with the platform, I want them to find my API obvious, even boring. If they have a preferred error handling system, they probably have a way to convert throws to it, since that’s what Cocoa generates. So I use throws.

The elephant in the room

I see a lot of Swift devs behaving as though Cocoa has somehow disappeared. Cocoa has become the embarrassing uncle that no one wants to acknowledge, even though he’s sitting right there at Thanksgiving dinner passing you the potatoes. And this is crazy. First, Cocoa is a great framework, filled with all kinds of tools that we use every day, implemented well and refined for years. And second, Cocoa is a required framework, filled with tools that we have to use every day if we want to write apps.

Trying to cordon off Cocoa means constantly converting your types and patterns. That’s horrible for programs, and it’s very unswifty. Swift is all about integrating cleanly with Cocoa.

If you’re writing Cocoa apps you wind up with NSData all the time. You get it when you read or write files, when you download things from the network, when you create PNGs, when you serialize. You can’t escape NSData. Swift automatically bridges NSString and String, NSArray and Array, NSError and ErrorType. Some day I hope NSData gets a bridge, but it doesn’t have one today. So the question is what to do in the meantime?

There are basically two options: build the bridge or use NSData. Building the bridge (without modifying stdlib) is tricky if you want to avoid copying. It’s not hard if you’re not worried about performance, but an NSData can easily be multiple megabytes and that’s both time and memory. Even temporary copies raise your high water mark, which hurts the whole system. Yes, I know all about premature optimization, but when you’re building frameworks you need to avoid patterns that are reasonably likely to cause performance problems. Even when you’re building just one app, there’s a difference between “build simply, then optimize” and “throw performance out the window until Apple rejects your app, then optimize.” If it were just one copy, and it made everything else really simple, that might be worth discussing. But making a copy every time you move from one part of the system to another is a problem.

With enough work you can solve this problem. It’s not tons of code, but it is a little bit tricky to be certain you’ve done it exactly right and won’t leak or crash (and much trickier to do from outside of stdlib). But is that work and complexity worth it? What problems were we really solving converting NSData to [UInt8]?

The magic that is Array

Even if you don’t care about NSData interop, using [UInt8] doesn’t give you a magical unicorn API. Array has all kinds of little sharp edges that surprise and confuse if you want to very careful of making copies.

Let’s start with a simple function using NSData and see what happens with [UInt8]. This function takes a CCCryptorRef and updates it with some data, writes the resulting encrypted data to a buffer and returns the buffer.

func updateCryptor(cryptor: CCCryptorRef, data: NSData) -> NSData {
    let outputLength = CCCryptorGetOutputLength(cryptor, data.length, false)
    let buffer = NSMutableData(length: outputLength)!
    var dataOutMoved: Int = 0

    let result = CCCryptorUpdate(cryptor,
        data.bytes, data.length,
        buffer.mutableBytes, buffer.length,
        &dataOutMoved)
    guard result == 0 else { fatalError() }

    buffer.length = dataOutMoved
    return buffer
}

No problems there IMO. That’s a fine implementation. Easy to read and understand (if you understand CCCryptorRef). The [UInt8] implementation is about the same. I don’t think you could say one is really much cleaner than the other.

func updateCryptor_(cryptor: CCCryptorRef, data: [UInt8]) -> [UInt8] {
    let outputLength = CCCryptorGetOutputLength(cryptor, data.length, false)
    var buffer = [UInt8](count: outputLength, repeatedValue: 0)
    var dataOutMoved: Int = 0

    let result = CCCryptorUpdate(cryptor,
        data, data.count,
        &buffer, buffer.count,
        &dataOutMoved)
    guard result == 0 else { fatalError() }

    buffer[dataOutMoved..<buffer.endIndex] = []
    return buffer
}

But is it correct? Can we be certain that data is contiguous memory and isn’t really an NSArray<NSNumber> under the covers? If it is an NSArray, will this work or will we get the wrong data? Should we use .withUnsafePointer here? I studied the docs, and talked to several devs (including Apple devs), and in the end am pretty sure that this will always work. But I’m only “pretty sure.” And that’s only because of kind people at Apple (especially @jckarter) taking time to walk through it with me. Not everyone has that.

This “but is it correct?” came up all over the place. Would this operation cause a copy? Exactly how long is an UnsafeBufferPointer valid? There’s a lot of bridging magic in Array, and it’s not always clear what is promised. Testing only gets you so far if the current implementation just happens to work. Sometimes behaviors change just by importing Foundation.

I thought I might avoid the Cocoa-bridging ambiguities of Array by using ContiguousArray instead. That way I could be very precise about my expectations. But it turns out that passing ContiguousArray to C behaves very differently than passing Array. Array gets turned into a pointer to the first element, but ContiguousArray gets turned into a pointer to the struct. So the ContiguousArray gets corrupted and you crash. Array is more magical than you think. Magic is wonderful until your program crashes and you don’t know why.

I struggled with copy-on-write behavior. How do I know if an array’s buffer is shared so that a copy will happen on mutation? Will this code allocate 10MB or 20MB?

func makeArray() -> [UInt8] {
    return Array(count: 10_000_000, repeatedValue: 0)
}
var array = makeArray() + [1]

What tests would prove that? Is it promised or just the current implementation? Does optimization level matter? Is it the same if makeArray() is in another module than the caller? Would small changes in my code lead to dramatic and surprising performance changes in apps that use my framework? This was a common problem in Scala before the @tailrec annotation was added. Very small tweaks to a recursive function could cause your stack to explode because you quietly broke tail call optimization. All your unit tests still pass, but the program crashes.

In the end, I spent hours trying to be certain of the precise behaviors of Array bridging and copying. And all that to replace NSData code that is perfectly fine.

Slice and dice

When updating the cryptor, it is common that you’ll only want some of the data you were passed. You might want to slice off a header, or you might want to chunk the data up to reduce your encryption buffer size. In either case, you want to pass updateCryptor() a slice.

For an immutable NSData that’s easy. Call .subdataWithRange() and you get another NSData back with no copying.

But the SubSequence of Array is ArraySlice, and updateCryptor() doesn’t accept that. Of course you can copy your slice into a new Array, but unnecessary copying was what we wanted to avoid.

We could make all the functions take ArraySlice and overload all the functions with an Array interface that forwards to the ArraySlice interface. But it’s a lot of duplication.

So I decided to just convert everything to UnsafeBufferPointer and then pass that around internally. Easier semantics after a one-time conversion. No bridging worries. No unexpected copies. It seemed like a good idea at the time.

The problem is that using UnsafeBufferPointer everywhere tends to turn your code inside out. Where you used to say:

updateCryptor(cryptor, data: data)

You now have to say:

data.withUnsafeBufferPointer { updateCryptor(cryptor, data: $0) }

Two solutions present themselves. First you decide that you are very clever, and use the UnsafeBufferPointer constructor:

// Never do this
updateCryptor(cryptor, data: UnsafeBufferPointer(start: data, count: data.count))

Then @jckarter points out that by the time updateCryptor runs, there’s no promise that the UnsafeBufferPointer is still valid. ARC could destroy data before the statement even completes. (If you know that data is life-extended, then it is possible to know this will work, but it’s very unsafe, fragile, and hard to audit. Coding that way breaks everything Swift was trying to fix.)

So then you start creating function overloads to accept Array and ArraySlice and convert them into UnsafeBufferPointer, and you have even more duplicated code. And then you realize you want to accept NSData here, too, so you write an extension that adds .withUnsafeBufferPointer() to NSData, and now you have four versions of every function, and you realize you really should use a protocol instead. Brilliant!

protocol BufferType {
    func withUnsafeBufferPointer<R>(body: (UnsafeBufferPointer<UInt8>) throws -> R) rethrows -> R
}

This really feels like it’ll solve all these problems very elegantly. Except for this one problem. You want [UInt8] to be a BufferType, but you don’t want [String] to be a BufferType. And then you discover that while you can write extensions that only apply to [UInt8], you can’t use those extensions to conform to a protocol. And that’s when the screaming starts. And then the barginning, and then the drinking.

When you get to the muttering, you came back and start building a Buffer class to wrap Array, ArraySlice, NSData, and even CollectionType to give it all a consistent interface. It’s ok, but it creates another “thing” for callers to deal with. In almost all cases, they have an NSData. There is almost no chance they had a [UInt8]. This is all just an extra layer for callers to deal with and to get in the way of the optimizer.

I want to remind you that all of this, all these many, many hours of struggle, were to avoid the simple NSData code that took two minutes to write, works great, and is pretty darn Swifty as long as you don’t define “Swifty” as “does not import Foundation.”

What’s the matter with NSData?

So why did I resist using NSData anyway? Well, even though I believe the Foundation is absolutely a part of Swift, some of it isn’t great Swift. Notably NSData isn’t a CollectionType. But fixing that is pretty easy.

End-to-end NSData also opened up some other opportunities for me, namely dispatch_data, which had threatened to be another can of worms with [UInt8].

For some, none of this will matter. The vast majority of my problems come from trying to dodge unnecessary copies. Much of this is very simple if you’re willing to just copy the data all over the place. For many kinds of problems, that’s fine. Use whatever you like. The copy-on-write system is actually pretty awesome and for most problems you can certainly trust it.

But for those in my situation, where performance is a serious consideration in much of your code, you’re looking for predictability as much as speed, and data can be huge, my hope is we get a Buffer type (or Data or whatever) that acts as a bridge to NSData, supports dispatch_data, and plays nicely with stdlib. But until that comes, I think NSData is just fine.

(While this works exactly like AnySequence, this isn’t how AnySequence is implemented. In my next post I’ll discuss why and how to implement type erasers like stdlib does.)

At the time I thought I’d pretty well nailed it down, but every time I dug into it I found another little thing I’d missed, and it never seemed to end. And with stdlib open sourcing soon, you’ll all just be able to read this yourselves, so why embarrass myself getting it all wrong? Over time I kind of hoped you all had forgotten that comment and planned to move on to other things. But then I was busted by Michael Welch, and so I had to finish the spelunking and here you go.

So, here is my annotated implementation of AnyGenerator and AnySequence that I believe pretty closely matches Apple’s implementation in stdlib (minus some low-level optimizations in AnySequence that I’ll call out when we get there). I’ve named my versions of public symbols with a trailing underscore to differentiate from the Swift version. For private symbols (with leading underscore), I’ve used the same name Swift does.

All type information was determined using :type lookup in the Swift REPL. In a few places I checked the implementation with Hopper, but in most cases, once you have the types, the implementation is obvious. (There’s a deep lesson in there if you pay attention.)

RNCryptor 4 is a complete rewrite of RNCryptor for Swift 2 with full bridging support to Objective-C. It has a streamlined API, simpler installation, and improved internals. It continues to use the v3 data format and is fully interoperable with other RNCryptor implementations.

For users desiring a fully Objective-C solution, v3 is still available. I don’t currently plan to do significant new work on v3, but will consider it if there is strong interest. Going forward, I expect most OS X and iOS projects to be able to accept a mix of ObjC and Swift code. Objective-C continues to be a fully supported language in RNCryptor 4.

I plan to convert this to a final release in a week or so if no major issues are discovered.

I now move onto:

• Continuing to prepare for dotSwift.
• Getting back to the blog.
• RNCryptor v5, which will finally update the data format to improve performance, security, and robustness. Much of the v4 design was specifically to make v5 easier to write.

In Swift 2, our little type erasers got a rename and some friends. Now they’re all named “Any”-something. So SequenceOf became AnySequence and GeneratorOf became AnyGenerator and there are a gaggle of indexes and collections from AnyForwardIndex to AnyRandomAccessCollection.

So what are these type erasers? Let’s start with how to use one and we’ll work backwards to why.

let seq = AnySequence([1,2,3])

This creates an AnySequence<Int>. It’s just a sequence of Ints that we can iterate over. Isn’t [1,2,3] also a sequence of Ints we can iterate over? Well, yeah. But it’s also explicitly an Array. And sometimes you don’t want to have to deal with that kind of implementation detail.

Who Needs Types Like That?

Let’s consider a little more complicated case:

let xs = [1,2,3]
let ys = ["A","B","C"]
let zs = zip(xs.reverse(), ys.reverse())
// Zip2Sequence<ReverseRandomAccessCollection<Array<Int>>, ReverseRandomAccessCollection<Array<String>>>

That’s quite a type. Imagine it as the return type of a function:

func reverseZip<T,U>(xs: [T], _ ys: [U]) -> Zip2Sequence<ReverseRandomAccessCollection<[T]>, ReverseRandomAccessCollection<[U]>> {
  return zip(xs.reverse(), ys.reverse())
}

That’s insane. Let’s not do that. Not only is the type overwhelming, but it ties us to this particular implementation. We might want to refactor the code like this:

  return zip(xs, ys).reverse()

Then the return type would change to [(T,U)] and all the callers would have to be updated. Clearly we’re leaking too much information about our implementation. What’s the point of reverseZip? Is it to return a Zip2Sequence<...>? No. It’s to return a sequence of tuples. We want a type that means “a sequence of tuples.” Often we use Array for that, but there’s an even less restrictive way that doesn’t require making an extra copy: AnySequence.

func reverseZip<T,U>(xs: [T], _ ys: [U]) -> AnySequence<(T,U)> {
    return AnySequence(zip(xs, ys).reverse())
}

Now we can keep our implementation details private. If we have some internal sequence type, we don’t have to share it with our callers. We just give them what they need and no more.

Notice that AnySequence is not a protocol. It’s a generic struct that wraps another sequence. You can’t use an [Int] in a place that expects an AnySequence<Int>. You still want to use SequenceType for parameters in most cases.

These “Any” type erasers also aren’t like Any and AnyObject, which are protocols that just “hide” the type. You can still as! an AnyObject back to its original type. AnySequence and its kin completely encapsulate the underlying data. You can’t get the original back. This creates a very strong abstraction layer and strengthens type safety by making as! casting impossible.

The new names worry me a little because they make it look like AnyObject and AnySequence are the same kind of thing when they’re not. But the new naming convention is definitely more flexible. You couldn’t have named the AnyIndex types using the old ...Of convention. So, I’m getting used to the new names.

Chains of Association

Hopefully by now you’re sold on why you’d want to use a type eraser. But would you ever want to build one? Let’s look at an example that comes up pretty often around associated types in protocols.

// An Animal can eat
protocol Animal {
    typealias Food
    func feed(food: Food)
}

// Kinds of Food
struct Grass {}
struct Worm {}

struct Cow: Animal {
    func feed(food: Grass) { print("moo") }
}

struct Goat: Animal {
    func feed(food: Grass) { print("bah") }
}

struct Bird: Animal {
    func feed(food: Worm) { print("chirp") }
}

So now let’s say we have a bunch of grass available and we’d like to feed it to some grass eaters. Seems easy:

for animal in grassEaters {
    animal.feed(Grass())
}

Now we just have to create this array of grass eaters. Should be simple, right? Hmmm…

let grassEaters = [Cow(), Goat()] // error: '_' is not convertible to 'Goat'

That’s a weird error. We probably just need to be explicit about the the type.

let grassEaters: [Animal] = [Cow(), Goat()]
// error: protocol 'Animal' can only be used as a generic constraint because it has Self or associated type requirements

We all know that error, don’t we? OK, let’s try generics.

let grassEaters: [Animal<Grass>] = [Cow(), Goat()]
// error: protocol 'Animal' can only be used as a generic constraint because it has Self or associated type requirements

Still? Oh right. You can’t specialize an associated type using generic syntax. That’s fine, we’ll just make the protocol generic.

protocol Animal<Food> {
    func feed(food: Food)
}
// error: Statement cannot begin with a closure expression

Right, protocols can’t be generic. Type-safety is for chumps. Let’s go back to Objective-C.

…Or maybe type erasure is what we need. Let’s build AnyAnimal. There are several ways to do this, but the easiest in my opinion is with closures.

struct AnyAnimal<Food>: Animal {
    private let _feed: (Food) -> Void
    init<Base: Animal where Food == Base.Food>(_ base: Base) {
        _feed = base.feed
    }
    func feed(food: Food) { _feed(food) }
}

(While this works exactly like AnySequence, this isn’t how AnySequence is implemented. In my next post I’ll discuss why and how to implement type erasers like stdlib does.)

Now we can make grassEaters:

let grassEaters = [AnyAnimal(Cow()), AnyAnimal(Goat())] // Type is [AnyAnimal<Grass>]

But we still get type safety if we try to incorrectly mix our animals:

let mixedEaters = [AnyAnimal(Cow()), AnyAnimal(Bird())]
// error: type of expression is ambiguous without more context

This kind of type eraser lets us convert a protocol with associated types into a generic type. That means we can put it in properties and return values and other places that we can’t use protocols directly. As you use more protocols in your Swift (and you should be), I think this will become an important tool in your toolbelt.

So get out there and erase some over-specific types. Focus on the protocol, hide the implementation.

And here’s the code for your amusement.

A friend of mine is an arborist. He takes care of a large forest, trimming and culling trees. He’s quite good at it and enjoys it, but he’s worried about job security. He thinks cabinet making would be a good career move. He likes to work with wood, and high-end cabinets are very expensive so there’s clearly a lot of money there. I’m a hobbyist woodworker, so we were talking about it.

Me: So, what saws are you looking at?

Him: I have a Husqvarna 3120. That should be all I need.

Me: Husqvarna? Isn’t that a…chainsaw?

Him: It’s the best! I use it every day. I plan to do everything with it.

Me: It’s a good saw, sure, but… um… How are you going to join the pieces?

Him: Dovetail joints. I’ve heard those are best and you just have to cut notches. My chainsaw will make quick work of that.

Me: Dovetails have really tight tolerances. I can barely get them to work with a dovetail saw. Maybe you should start with pocket screws? They’re much easier and make great joints. A chainsaw…

Him: I’ve worked it all out. I modded this router table to hold my chainsaw upright so that’s just like a table saw. And I can make the other cuts freehand. I’m really good with a chainsaw. See, I already have a proof-of-concept.

Me: Um… that’s just a board with a notch in it.

Him: Yeah, but a cabinet is just a bunch of boards with notches. I’ll make some more and they should snap together.

Me: Cabinet tolerances are really tight. I mean, it’s not that hard to make a box, but making a full cabinet is difficult, even with proper tools. Why would you want to use some cobbled-together thing? That table looks terrifying.

Him: Look, I know chainsaws. I’m not going to go learn a bunch of new tools just to build cabinets. Wood is wood. Saws are saws. I’m using the saw I’m comfortable with. If I use the tools I know, I’m sure I can make really great cabinets.

Me: …

In unrelated news, a lot of folks have asked if I’m excited that Go 1.5 can build iOS apps.

My wife’s background is art education. When you’re teaching art, she tells me, it’s important to remember whether you’re doing process art or product art in order to judge its success. Is the end result the goal, or how you got there? Am I trying to make the most beautiful painting I can, or am I trying to make a painting using only recycled materials? Does the audience need to know the process in order to appreciate the art, or does the end result stand on its own?

A lot of great hacks are process art. Can you run VMS on Linux? Sure you can! Is that awesome? It is absolutely awesome. Is there any commercial reason to develop new VMS apps to run on Linux? Of course not. That would be ridiculous.

Is it fun to get the Go runtime working on surprising platforms? Sure, and the process may improve Go. That’s why process art is so common in art education. You put unnecessary restrictions on students so that they have to develop new skills that will be useful when they work on product art. “Draw without looking at your work” is an important exercise, but not usually how you create a masterpiece.

So let’s get to the “unnecessary restrictions” I’m talking about if you want to write excellent iOS apps in Go (or C# or JavaScript or….) I don’t want to spend too much time on the great memory management debate. Do modern concurrent garbage collection engines work well on dual-core, 1GB iPhones? I don’t know. I’m still hearing a lot of complaints about GC pauses on Lollipop with 2x the memory. It’s hard to imagine that Go GC on iOS is going to be faster and more memory efficient than Java GC on Android. But let’s put that question aside for the moment. Let’s assume there is a large genre of apps for which Go performance and memory usage is adequate, even excellent, on iOS devices. (This might even be true.)

The real question is “what’s the hard part of iOS development?”

If you think the answer is “Objective-C” then you don’t understand iOS development at all. Learning Objective-C is the easiest part of iOS development, doubly-so for anyone who is comfortable in a wacky (and awesome) language like Go. And when it finally settles down a bit, Swift will likely be easier to learn than Objective-C for Java-esque developers.

The sort-of hard piece of iOS development is UIKit, CloudKit, HomeKit, SpriteKit, SceneKit, HealthKit, WebKit, MapKit, StoreKit, GameplayKit, and another new Kit every release. It’s power management and backgrounding and handoff and watch integration and multitasking and adaptive fonts and auto layout and animations and app thinning. You can sometimes ignore those pieces, but they’re what separate barely acceptable apps from great apps. If your only goal is something that kind of works, fine. Anything can do that. Web apps can do that. But if you want to make a really great app, integrating with the platform is a must. And the platform is designed for Cocoa. And Cocoa is designed for ObjC and Swift. This stuff is hard when you’re using the tools Apple intends you to use. Working through translation layers is surgery with welding gloves.

And that brings us around to the really hard piece of iOS development. It changes. Constantly. Apple moves forward fast and they deprecate old APIs mercilessly. Clever work-arounds have short half-lives. You have to adapt every release, and find ways to be backward compatible while still compiling against the latest SDK. Once you’ve lived through a few releases of iOS, you discover how hard this piece can be even in ObjC, even when you’ve followed all the rules. I don’t envy anyone trying to deal with it through an unsupported layer. By the time anyone knows what the next iOS version breaks, the clock is already ticking. Whoever supports your extra layer has to fix their piece, and then you have to fix your piece. And with open iOS betas, your customers are already yelling.

So I may celebrate your chainsaw router table as a glorious (and terrifying) hack. But I’ll stick with my table saw for cabinets and my chainsaw for cutting up trees. A good crafter is capable with many tools, not just their favorite. And most iOS apps are about product, not process.

Swift on Android? I’ll take Java, thanks.

Lee Whitney has an interesting take on this subject from the C# side.