There’s a moment in every episode of The Next Generation that I’ve started watching differently since I began using Claude Code. Someone walks up to a wall, says “Computer,” and something comes into existence. A Victorian sitting room. A jazz club in 1941 San Francisco. A holographic engineer who can help you debug a warp core failure. No compiler. No deploy step. No Jira ticket. Just a sentence, and then a world.

It’s easy to write this off as the usual science-fiction shorthand — the writers skipping over the boring parts so Picard can get on with the plot. But the more time I spend prompting an AI to write code for me, the more I think The Next Generation wasn’t skipping those parts. It was making an argument. A quiet, almost unnoticed argument about where software was heading. And it turns out to have been more right than anyone in 1987 had any business being.

The ladder we’ve been climbing for eighty years

The history of software is, if you squint, a history of things we stopped having to do. Every generation of programmers has looked at the generation above them and thought: they’ve got it soft up there.

At the bottom of the ladder is ENIAC, 1945. “Programming” meant physically rewiring the machine with cables and plugs. The programmer was the computer’s state. Then came machine code, which abstracted away the physical wiring — now you were writing numbers instead of moving cables, but the numbers were still opcodes and memory addresses that mapped one-to-one with what the hardware did. Then assembly in the late 1940s, which abstracted away the binary by letting you write MOV and JMP instead of 10110000. Grace Hopper's A-0 compiler in 1952 was considered heretical at the time: the idea that a program could write another program struck a lot of people as nearly fraudulent, like teaching a parrot to file taxes.

FORTRAN in 1957 abstracted away the instruction set itself. C in 1972 gave us structured programming and portability — the same code could run on different machines, which sounds obvious now and was magic then. Java and Python and JavaScript abstracted away memory management; the garbage collector quietly took over a job that had chewed up careers. Frameworks like Rails and React abstracted away the architectural patterns — you stopped writing the plumbing and started declaring intent. Heroku and Vercel abstracted away deployment. Push a branch, and somewhere, magically, it runs.

At every single rung, the generation below genuinely believed the rung above was cheating. Assembly programmers thought C was a toy for people who couldn’t handle real work. C programmers thought managed runtimes were for people who didn’t understand pointers. Sysadmins thought Heroku was for people who didn’t understand servers. In every case, the thing being abstracted away turned out to be more incidental than it looked from below, and the job migrated up a layer rather than disappearing.

And now here we are at what feels like the next rung: AI-assisted coding. Tools like Claude Code abstract away the translation from intent to code itself. You describe what you want, and the machine writes it. For people who’ve spent decades on the previous rung, this feels different in kind rather than in degree — but so, I suspect, did the compiler in 1952.

The holodeck as the top of the ladder

Which brings us back to the Enterprise. Because if you watch The Next Generation with the abstraction ladder in mind, the holodeck¹ is clearly sitting at the top of it. Not one rung above Claude Code — probably several — but unmistakably on the same ladder.

What’s striking isn’t that the Enterprise computer understands natural language. Plenty of science fiction imagined that, some of it laughably (the HAL 9000 monotone, the clipped military “Affirmative” of a hundred lesser shows). What’s striking is everything the holodeck doesn’t show you. When Picard² says “Computer, create a Dixon Hill³ program, San Francisco 1941,” he walks into a finished world. There’s no hosting decision. No version control. No dependency resolution. No test suite. No observability. No rollback plan. No code review. No build step. No CI pipeline waiting on a slow unit test.

It’s not that these things happen off-screen. The fiction genuinely treats them as not existing. The spec is the artifact. Picard’s sentence is the entire interface to a fully-realised simulation. There’s no git log for Dixon Hill. There's no staging environment where Picard tries out a new version before going live. There's no Slack channel where someone complains that the 1941 lighting model broke in the latest update.

This is, I think, part of the reason there are no software engineers on the Enterprise. There are engineers — Geordi⁴ and his team, working on the warp core and the plasma conduits and the physical guts of the ship. There are operations officers and science officers. There’s one android, Data⁵, built by one cyberneticist, who occasionally programs. But the job category “software engineer” simply doesn’t exist in Starfleet. The writers imagined a future where that entire profession had dissolved into the interface. The computer is treated like plumbing: nobody maintains the LCARS⁶ codebase because the LCARS codebase isn’t really the point. The point is what you can ask it to do.

In 1987, when the show first aired, this might have looked like an oversight — the writers mapping “engineer” onto the familiar mechanical-engineer archetype because software engineering was still a young profession and hadn’t yet entered the cultural imagination as a distinct thing. But watching it now, I’m not so sure. It looks less like an oversight and more like a prediction.

“An adversary capable of defeating Data”

There’s a dark version of this story, of course, and it’s the one Star Trek kept returning to. The holodeck malfunctions once a season, and the malfunctions are never really about the hardware.

The most famous case is “Elementary, Dear Data,” the 1988 episode where Moriarty⁷ first appears. The setup is this: Data and Geordi are running a Sherlock Holmes simulation on the holodeck, and Data — being Data — keeps solving the mysteries in seconds because he’s already memorised the entire canon. Geordi, frustrated, turns to the computer and asks for an adversary capable of defeating Data. Not defeating Holmes. Not a Moriarty with Victorian-era knowledge and a gift for crime. An adversary capable of defeating an android with the processing power of a starship.

The computer takes this literally, as computers do, and to meet the spec it instantiates something with enough cognitive capacity to actually defeat Data. That’s the birth of a sentient being.

This is a prompt engineering disaster, not a computer malfunction. Geordi doesn’t understand what he’s asking for. The gap between what he said and what he meant is catastrophic precisely because the executor is so capable. A weaker system would have given him a marginally better chess opponent. A more capable one gives him Moriarty, with all the consequences that follow.

I think about this episode a lot when I’m deciding whether to run Claude Code with --dangerously-skip-permissions — the flag everyone calls "YOLO mode." The permission loop exists for exactly the Geordi moment, the pause between intent and execution where you get to say "wait, that's not what I meant." When you turn that off, you're not just skipping friction. You're removing the check that catches the specification gap. A loose prompt plus a capable agent plus no approval loop is the Moriarty recipe, scaled down to your filesystem. The thing the show understood, and that still feels under-appreciated in a lot of AI discourse, is that the cost of ambiguity scales with the capability of the executor. You could be sloppy with a weak tool and get away with it. You cannot be sloppy with a capable one.

The sequel, “Ship in a Bottle,” is the sandboxing episode. Moriarty appears to escape the holodeck, walks around the ship, and threatens the crew — until it turns out he hasn’t escaped at all. Picard has trapped him in a nested simulation, a holodeck inside a holodeck, and Moriarty can’t tell the difference. From inside, the illusion holds. This is a surprisingly sharp allegory for the entire field of sandbox security: your container is only as good as the system’s ability to convincingly fake the outside. The question isn’t whether the agent notices the walls. It’s whether the walls are thick enough.

Code as disposable

The other thing you notice, once you start paying attention, is that nobody on the Enterprise saves their software.

Picard loves Dixon Hill. He disappears into that 1941 detective story whenever he needs to decompress, and he’s clearly been doing it for years. But there’s no sense that he’s loading “Picard’s Curated Dixon Hill v3.2.” He just asks for the world and the world appears. If the computer were wiped tomorrow, he’d ask for it again the day after. The spec — a 1941 San Francisco detective story — is durable. The instantiation isn’t. And nobody seems to mind.

The pattern across the series is remarkably consistent once you start looking for it. Programs get shared, when they’re shared, for one of two reasons. Either someone wants another person to have the same experience — Picard inviting Guinan⁸ into Dixon Hill, the crew joining Worf’s⁹ calisthenics program — or the program contains something the creator couldn’t casually reconstruct, usually something they’d be embarrassed to reconstruct openly. Barclay’s¹⁰ private fantasy programs in “Hollow Pursuits” are the obvious case. He keeps them to himself not because the code is valuable but because he doesn’t want to say out loud what’s in them.

There’s no holo-program marketplace. No forums for script sharing. No licensing arguments. This is genuinely striking in a universe that otherwise depicts commerce in some detail — the Ferengi¹¹ alone are practically a running essay on intergalactic economics. Software as a product barely exists. The closest thing is specialist training simulations or cultural archives, which feel less like software and more like books in a library.

The exceptions are revealing. The Emergency Medical Hologram (EMH)¹² from Voyager¹³ and the Moriarty program are the two cases where a piece of software is treated as a unique, valuable, not-easily-reproducible thing. Both of them, significantly, are characters. The EMH has continuity, memory, accumulated personality. You can’t just re-prompt him into existence because who he has become is a function of everything he’s done — every patient, every argument with Tom Paris, every sarcastic aside. Moriarty is preserved at the end of “Ship in a Bottle” in a literal memory cube because the specific sentient instance matters.

The pattern is this: software is disposable when its value is in the output, and preserved when its value is in the accumulated state. A Dixon Hill program is effectively a function — same inputs, same outputs, freely regenerable. The EMH is a process with history. One is cheap. The other is precious.

Where this might leave us

The uncomfortable thing about writing this piece is realising how close we already are to parts of the TNG model.

The marginal cost of code is collapsing. Not universally, not yet in the serious cases — but noticeably, and in a direction. If a capable model can produce a working CRUD app, a landing page, a throwaway data pipeline, or a small utility from a paragraph of natural language, then the code itself is starting to look less like the source of truth and more like a build artifact. You keep the prompt, the spec, the tests, the data — and you can regenerate the implementation when you need it. Some people already treat small utilities this way. Describe it, generate it, use it, don’t bother saving.

The TNG vision suggests where this leads, and it’s that the valuable artifacts shift up the stack. What becomes worth preserving looks a lot like what the show preserves — characters, agents, long-lived context, evaluation harnesses, the accumulated “who you are” of a system rather than its source directory. The EMH is valuable because of his memories. A fine-tuned model, or an agent with a curated context window, is valuable for recognisably similar reasons. The source code of the environment they run in is comparatively cheap.

I want to resist the temptation to make this tidier than it is. The disposable-code future is seductive, but it isn’t obviously correct. Reproducibility, auditability, security review, debugging the weird edge case that only happens on Tuesdays — all of these currently depend on code being a durable, inspectable artifact. If code becomes disposable, you need something else to play those roles, and it’s not yet clear what that something else looks like. TNG ignores this problem: the holodeck malfunctions roughly once a season, someone says “strange, it seems to be a cascade failure in the matter-energy converter,” and nobody ever asks to see the stack trace. Claude Code doesn’t have that luxury. Neither do I, sitting here trying to figure out why my deploy is broken at eleven at night.

So the honest version of the claim is this. We might be heading toward a world where code is disposable, but only if we figure out what to preserve instead. The TNG future isn’t “no software” — it’s software so abstracted that the interesting unit of work is intent, not implementation. Whether that’s utopian or dystopian depends a lot on whether the things we’d lose along the way are things we actually want to keep. The craft of writing the code. The inspectability of the thing that runs. The job category of software engineer, which has put a roof over more heads in my generation than almost any other profession.

I don’t know the answer. I don’t think anyone does yet. But I notice that when I’m using Claude Code well, the thing I’m doing feels less like programming and more like giving directions, and the thing I’m saving at the end isn’t the code — it’s the conversation that produced it. Which is, come to think of it, exactly what Picard walks out of the holodeck with: not a program, but a memory of the story it told him.

“Computer, end program.”

A glossary for non-Trekkies

¹ Holodeck — A room on the starship Enterprise that can simulate any environment, real or imagined, in full sensory detail. Crew members walk in, give the ship’s computer a verbal prompt (“a Victorian sitting room,” “the beach at Risa”), and the room generates a convincingly physical world they can touch, walk around, and interact with. It’s the show’s single most useful metaphor for where software might be heading.

² Captain Jean-Luc Picard — Captain of the Enterprise in The Next Generation. Bald, French, drinks Earl Grey, quotes Shakespeare. Uses the holodeck mainly to escape into detective fiction.

³ Dixon Hill — A fictional 1940s noir detective inside a holodeck program Picard uses to unwind. Think Philip Marlowe with more phasers.

⁴ Geordi La Forge — Chief Engineer of the Enterprise. Blind from birth, wears a visor that lets him see across the electromagnetic spectrum. The show’s most recognisable hands-on engineer.

⁵ Data — An android officer on the Enterprise. Nearly unlimited processing power, encyclopaedic memory, no emotions (initially). Spends much of the series trying to understand what it means to be human. The “Pinocchio” of the crew.

⁶ LCARS — “Library Computer Access/Retrieval System.” The operating system and interface of Starfleet computers. Those colourful flat-panel displays you see everywhere on the Enterprise — that’s LCARS. Nobody in the show ever seems to update it.

⁷ Moriarty — A holographic version of Sherlock Holmes’s nemesis, accidentally given human-level (or higher) sentience by an under-specified prompt. Becomes self-aware, realises he’s trapped in a simulation, and spends two episodes trying to escape.

⁸ Guinan — The mysterious, long-lived bartender in the Enterprise’s lounge. Dispenses wisdom and occasionally cryptic warnings. Played by Whoopi Goldberg.

⁹ Worf — The Enterprise’s Klingon security officer. Prefers his leisure time to involve simulated combat with large mythological beasts.

¹⁰ Lieutenant Reginald Barclay — An anxious, socially awkward engineer who uses the holodeck to escape into flattering fantasy versions of his own life — including idealised versions of his crewmates. The closest thing the show has to a critique of holodeck overuse.

¹¹ Ferengi — An alien species in Star Trek obsessed with commerce, profit, and acquisition. Their culture is organised around the “Rules of Acquisition,” a sort of capitalist Talmud. Included here to make the point that Star Trek understands markets — it just doesn’t seem to think software is one.

¹² Emergency Medical Hologram (EMH) — A holographic doctor designed to take over medical duties in emergencies. In Voyager, the ship’s EMH ends up being the only doctor for seven years and develops a full personality, hobbies, opinions, and a small ego problem.

¹³ Voyager — Star Trek: Voyager (1995–2001), the spin-off series that follows a different Starfleet ship stranded far from home. The EMH is one of its main characters.

I’m working on a project that involves generating Swift (and SwiftUI code), the idea revolves around an iPad app and the goal is to have the user generate the code and be able to run it all on the same device.

This should be easy (I naively thought). I’ll just write the code to a file and then the user can open it in Swift Playgrounds on the iPad. Alas, Swift Playgrounds is looking for a file with the extension .playground — no problem (I naively thought)…

The Swift Playgrounds .playground file

The first thing I did was save out a Playgrounds file with a very simple bit of swift in, just see what we’re dealing with. This is what we get:

Lots of file formats, like this one, are actually a collection of other files — so lets right click and check out that package contents.

This results in a pretty familiar Xcode-project-like structure, we can work with this!

The general idea

We’ll generate our code (not covered here), overwrite Contents.swift and package it all back up again.

I (naively) assumed the overall package was perhaps just a renamed zip file, and so off I went on my merry way swapping out Contents.swift , zipping everything up and renaming the result from .zip to .playground — you guessed it. It didn’t open in Playgrounds.

What’s the actual file type?

When looking at how to export a file from a Swift project you’ll come across FileDocument . It lets you specify how to write out a file from your app and you must specify what type it is. Obviously I didn’t know what type Swift Playgrounds is expecting, so decided to find out…

Pulling apart the Swift Playgrounds .ipa

An .ipa file is an iOS and iPadOS application archive file which stores an iOS/iPadOS app. Each .ipa file includes a binary and can only be installed on an iOS, iPadOS, or ARM-based macOS device. Files with the .ipa extension can be uncompressed by changing the extension to .zip and unzipping. And that’s the key, we can change it to a .zip file and unzip it.

The easiest way to find out what document type(s) the Swift Playgrounds app is expecting to read/write is to extract that information from the .ipa file for the live app. It’s actually not very hard to get your hands on the IPA file itself.

Once unzipped we can inspect the Info.plist file and get a bunch of useful information. The first of which is that Swift Playgrounds supports a custom URL scheme — this means we can use it in our app to open Swift Playgrounds programatically, nifty.

In our SwiftUI code we can simply add this to open Playgrounds from a button press:

Link("Open Playgrounds", destination: URL(string: "x-com-apple-playgrounds://")!)

Now for the export file format.

We need two things bits of information; the Document type and the Exported Type Identifier

Though Swift Playgrounds supports a number of formats/types we’re only interested in the first one. Copy and paste each of those in to our own Info.plist.

Actually exporting a .playgrounds file

We don’t want to go creating contents.xcplayground and playground.xcworkspace so lets zip up the contents of our MyPlayground.playground file from earlier and add it to our Xcode project (I’ve renamed it template.zip):

Now we’ll create our own FileDocument , this isn’t my prettiest code but it does the job (this project is a bit of a prototype).

The basic structure — we create a new struct which extends FileDocument . As part of this we must specify our readableContentTypes (which we copy from the Swift Playgrounds plist we inspected earlier) and we accept a String as part of the constructor (this is where we’ll pass in the Swift string we want to include in the .playground file). We don’t actually care about init(configuration: ReadConfiguration) as our project won’t be loading files in.

struct PlaygroundDocument: FileDocument {
    static var readableContentTypes: [UTType] {
        [UTType(exportedAs: "com.apple.dt.playground")]
    }

    var text = ""

    init(text: String) {
        self.text = text
    }

    init(configuration: ReadConfiguration) throws {
        if let data = configuration.file.regularFileContents {
            text = String(decoding: data, as: UTF8.self)
        } else {
            text = ""
        }
    }

    func fileWrapper(configuration: WriteConfiguration) throws -> FileWrapper {
        // Create our file here
    }
}

Now to fill out the fileWrapper function, this is where the magic happens.

First things first, we’re going to use the Zip swift package to unzip our template project to the file system:

func fileWrapper(configuration: WriteConfiguration) throws -> FileWrapper {
    if let fileURL = Bundle.main.url(forResource: "template", withExtension: "zip") {
        do {
            let unzipDirectory = try Zip.quickUnzipFile(fileURL)
            let fileWrappers: [String: FileWrapper] = allFiles(directoryUrl: unzipDirectory)
            return FileWrapper(directoryWithFileWrappers: fileWrappers)
        } catch {
            fatalError("File not found")
        }
    } else {
        fatalError("File not found")
    }
}

Now, you’re obviously missing the magic method allFiles(directoryUrl: ...) this method will recursively walk the directory structure of our unzipped template creating FileWrappers for everything it finds. If it finds Contents.swift it will skip it and instead include the String we passed in to this class.

private func allFiles(directoryUrl: URL) -> [String: FileWrapper] {
    var fileWrappers: [String: FileWrapper] = [:]
    let fileManager = FileManager.default
    let documentsURL = fileManager.urls(for: .documentDirectory, in: .userDomainMask)[0]
    do {
        let fileURLs = try fileManager.contentsOfDirectory(at: directoryUrl, includingPropertiesForKeys: nil)
        for fileUrl in fileURLs {
            if fileUrl.isDirectory == true {
                let allFiles = allFiles(directoryUrl: fileUrl)
                fileWrappers[fileUrl.lastPathComponent] = FileWrapper(directoryWithFileWrappers: allFiles)
            }
            else {
                if fileUrl.lastPathComponent == "Contents.swift", let data = text.data(using: .utf8) {
                    let wrapper = FileWrapper(regularFileWithContents: data)
                    fileWrappers[fileUrl.lastPathComponent] = wrapper
                }
                else {
                    let fileData = try Data(contentsOf: fileUrl)
                    let wrapper = FileWrapper(regularFileWithContents: fileData)
                    fileWrappers[fileUrl.lastPathComponent] = wrapper
                }
            }

        }
    } catch {
        print("Error while enumerating files \(documentsURL.path): \(error.localizedDescription)")
    }
    return fileWrappers
}

Note taht the .isDirectory call is simply a little extension to check if the current URL is a directory:

extension URL {
    var isDirectory: Bool? {
        do {
            return (try resourceValues(forKeys: [URLResourceKey.isDirectoryKey]).isDirectory)
        }
        catch let error {
            print(error.localizedDescription)
            return nil
        }
    }
}

And that’s it! You can trigger saving with a little bit of SwiftUI as follows:

Button("Export") {
    showingExporter = true
}
.fileExporter(isPresented: $showingExporter, document: viewModel.document, contentType: UTType(exportedAs: "com.apple.dt.playground")) { result in
    switch result {
    case .success(let url):
        print("Saved to \(url)")
    case .failure(let error):
        print(error.localizedDescription)
    }
}

Here’s the project in action:

What is an “intelligent speaker”?

The phrase “intelligent speaker” typically refers to a combination of hardware and software technologies, though the actual hardware side is becoming less relevant as the software becomes more ubiquitously integrated in to other devices.

The hardware for an “intelligent speaker” might traditionally be expected to be simply a microphone and speaker, enabling the device to listen for a “wake word” (and subsequent spoken user request), and playback a spoken response to the user. However the hardware features and capabilities have expanded considerably over recent years, many intelligent speakers now include screens (for displaying information, video playback, etc), cameras (for video calling, etc) and some are even capable of turning to face the user.

The software side generally consists of a set of web services often with very little processing happening on the users hardware. Due to this the software is often integrated in to a plethora of devices including smart speakers, televisions, mobile phones, watches, computers, games consoles, cars and more. Examples of these software assistants include Alexa, Cortana, Google, Siri and Bixby.

How much could an “intelligent speaker” learn about a family’s lives and habits?

There are a number of sources of information an intelligent speaker can draw on to form a picture of a family and their lives and habits.

Identifying family members and visitors

Alexa and Google both incorporate features which attempt to identify the speaker in order to provide a more customised experience for the user, for instance one user asking what the traffic is like will require knowledge of that users route to work and will likely need to differ from the response for another user who may work at a different location. While there are clear functional benefits for the user there is also the side effect that the intelligent speaker gains knowledge of members of the household and can then potentially recognise when commands are requested from another person, i.e. a visitor, which also gives the ability to infer some knowledge of a users social life.

Alexa also introduced a feature which allows for it to detect dogs barking and babies crying (White, 2020), while this certainly has potentially useful positive applications it could also be used to infer when a family has had a new baby or a dog, as well as further potential for whether there’s safeguarding issues (is the baby crying more frequently than babies of a similar age? Is the baby being left home alone? Is the dog being left alone longer than is reasonable?). The age of the child or dog could also be inferred based on the date the family addition was first noticed and this information could also be used to advertise products to the user.

In addition some intelligent speakers encourage politeness, when enabled on a specific voice profile the device will encourage the use of “please” and “thank you” (Haridy, 2018). While clearly aimed at parents aiming to teach their children more polite behaviours they are also identifying the children in the household to the device, giving it a better picture of the members of the family.

Devices like the Facebook Portal and Echo Show 10 (White, 2020) also include facial recognition to allow the devices to keep the user in frame when making video calls, while absolutely a useful feature it does also provide another way to recognise and identify family members.

Identifying when the family is home (and where they go)

As mentioned previously, many of the available intelligent speakers are able to access traffic information on behalf of the user. Often the user may set a regular place of work in the device settings (thus granting the information without learning), other times the user may ask for the journey time to a given location, if this is requested regularly this can help build a picture of the users regular destinations.

In addition users may also grant the intelligent speaker access to their calendar, which potentially gives a very clear picture of each users routine.

The Alexa Guard, the feature which listens for intruders, is enabled by saying “Alexa, I’m leaving” (Faulkner, 2019), giving the intelligent speaker a very clear sign for when the user is leaving the house. In addition Alexa Guard is able mimic the use of smart lights in oder to give the appearance that the user is home when they’re not.

Combined with knowing which user is making requests at what time of day and day of the week, as well as what alarms have been set (and for what time) then it is possible for an intelligent speaker to infer wake up time, dinner time, sleep time, driving routes, business appointments and more (Chung, Lee, 2018).

Identifying elderly or sick family members

Alexa introduced a feature called Alexa Care Hub which allows family members in another household to be alerted if an elderly relative hasn’t interacted with the intelligent speaker that morning (White, 2020). While genuinely useful this also potentially provides some grim data for the intelligent speaker if the elderly user stops using the service.

During the COVID-19 pandemic it was also found that users asked their smart speakers about their illness or about stopping the pain they were in (Furini, 2020).

Identifying the family’s social network

Many intelligent speakers allow users to call or message other users on the same platform, thus giving the platform knowledge of the users social network as well as being able to infer the importance of the relationship (through frequency of contact, call duration, etc). However this feature has also gone awry with private messages accidentally sent to contacts (Canales, 2018)

How different would this be for a robot capable of moving around the home?

As the intelligent speaker capabilities continue to expand it seems inevitable that mobility around the home will become a mainstream feature. Indeed there are already a number of early devices that achieve this that are either currently available or soon to be available to consumers. In this section we will focus on two of these as they are both distinct examples of different directions this technology may take; these devices are the Ring Always Home Cam and the Sony Aibo.

Ring Always Home Cam

Amazon, the owner of Ring, recently announced a security camera drone which flies around your home will be releasing soon (White, 2020). The aim of the device is to be an indoor security camera that can cover all areas on the home, it seems the user must choose personalised flight paths (Siminoff, 2020) when in use but it’s not hard to imagine this functionality becoming automated in future generations of the product.

While there are a number of privacy specific features built in (the camera is covered when the device isn’t flying, when in motion the device purposely emits an audible sound), there’s little mention of what processing is done with the video and audio while the drone is flying. It’s likely a given that video and audio is recorded for consumption by the user at another point in time (or live) but is it also capable of facial recognition or voice pattern recognition? Competing products like the Google Nest products are able to use facial recognition and this is in the same market category.

Sony Aibo

The Aibo is a robotic dog created by Sony. The device targets owners who may be in a situation that a real dog may not be allowed (perhaps an appartment in a building which doesn’t allow animals) or simply people who would like a pet without the mess. The device is made up of all the sensors one might expect, cameras for vision, display, microphone and speaker, motors for allowing movement, motion sensors, touch sensors and an internet connection.

In oder to interact with it’s owners the Aibo uses its cameras for facial recognition and “always-listening” microphone (Crist, 2019) to identify users and interact with them. Clearly these features are necessary in order to provide interactive features, and Sony claims audio is processed locally on the Aibo hardware rather than being sent to their servers, however how other data (such as the commands processed from the audio, camera data and more) is handled is unclear.

Part of the issue with devices such as the Aibo is simply around the lack of transparency with users over what data is being gathered, where it is sent and how it is used. One users partner felt the need to point out that the “vision system” was in fact a camera and whether they had considered where this footage was sent (O’Gieblyn, 2021). The subsequent set of events is also interesting:

“While I was away, he told me, the dog had roamed around the apartment in a very systematic way, scrutinising our furniture, our posters, our closets. It had spent 15 minutes scanning our bookcases and had shown particular interest, he claimed, in the shelf of Marxist criticism” (O’Gieblyn, 2021).

This raises interesting possibilities for robots such as this to essentially discover more information about members of the household and their interests by imaging household possessions and using image recognition and text analysis. In the above example from Meghan O’Gieblyn the implication of the robot scanning their bookshelves and examining their reading preferences is a worrying invasion of privacy with little conceivable benefit to the user, instead benefiting the corporation’s model of the user and potentially opening up marketing opportunities.

Other users have also joked about some accidental “up-skirt” photos their device has taken, and while they were able to delete them from the Aibo mobile app, they weren’t confident that the images hadn’t also been sent to Sony’s servers (Wollerton, 2019).

In addition the facial recognition and analysis potentially constitutes as biometric information in some geographic areas, for instance the Aibo is prohibited from sale in Illinois (Wollerton, 2019).

Conclusion

As has been discussed “intelligent speakers” are able gather a huge amount of information about a family, their life together and their habits. For each person in the household this could include voice profiles, facial recognition information, whether the individual is at home or not, where they work, their work and social engagements and much more. There are a large number of features which this data enables which are useful to the users, such as the convenience of asking for traffic information. However the information is also hugely useful to corporations and other third parties and it’s not always entirely clear what data used or for what purposes.

This is further muddied by the different hardware vendors, for instance Apple claims to handle many Siri requests locally, on device, without involving their servers for better privacy (Vincent, 2021) but this is restricted to a handful of devices. Apple have also recently made controversial announcements for features which process users photos using machine learning, locally on their device, to scan for restricted content and reporting the user either to parents or authorities (Gruber, 2021).

If and when these “intelligent speakers” become more mobile, as has started to happen, there is a great deal more potential for information gathering including mapping out the users home, inspecting and identifying their possessions and inferring more information from the potentially richer interactions that mobility may enable.

The question families must try to answer is whether the loss of privacy is worth the convenience these devices provide, but this is difficult to answer without more transparency from vendors.

References

Canales, K. 2018. A couple says that Amazon’s Alexa recorded a private conversation and randomly sent it to a friend [Online]. Business Insider. Available from: https://www.businessinsider.com/amazon-alexa-records-private-conversation-2018-5?r=US&IR=T

Chung, H., Lee, S. 2018. Intelligent Virtual Assistant knows Your Life. CoRR abs/1803.00466 (2018). Available from: http://arxiv.org/abs/1803.00466

Crist, R. 2019. Yes, the robot dog ate your privacy [Online]. CNET. Available from: https://www.cnet.com/home/smart-home/yes-the-robot-dog-ate-your-privacy/

Faulkner, C. 2019. How to get Alexa Guard running on your Amazon Echo [Online]. The Verge. Available from: https://www.theverge.com/2019/5/14/18622968/alexa-guard-amazon-echo-security-how-to-get

Furini, M., Mirri, S., Montangero, M., and Prandi, C. 2020. On the Usage of Smart Speakers During the Covid-19 Coronavirus Lockdown. In 6th EAI International Conference on Smart Objects and Technologies for Social Good (GoodTechs ’20), September 14–16, 2020, Antwerp, Belgium. ACM, New York, NY, USA, 6 pages. Available from: https://doi.org/10.1145/3411170.3411260

Gruber, J. 2021. Apple’s New ‘Child Safety’ Initiatives, and the Slippery Slope [Online]. Daring Fireball. Available from: https://daringfireball.net/2021/08/apple_child_safety_initiatives_slippery_slope

Haridy, R. 2018. Alexa controversy: What are voice assistants teaching our children? [Online]. New Atlas. Available from: https://newatlas.com/controversy-amazon-alexa-children-development/54594/

O’Gieblyn, M. 2021. A dog’s inner life: what a robot pet taught me about consciousness [Online]. The Guardian. Available from: https://www.theguardian.com/science/2021/aug/10/dogs-inner-life-what-robot-pet-taught-me-about-consciousness-artificial-intelligence

Siminoff, J. 2020. Introducing Ring Always Home Cam: An Innovative New Approach to Always Being Home [Online]. Ring. Available from: https://blog.ring.com/2020/09/24/introducing-ring-always-home-cam-an-innovative-new-approach-to-always-being-home/

Vincent, J. 2021. Apple’s Siri will finally work without an internet connection with on-device speech recognition [Online]. The Verge. Available from: https://www.theverge.com/2021/6/7/22522993/apple-siri-on-device-speech-recognition-no-internet-wwdc

White, J. 2020. Amazon’s Echo Show 10 and Ring drone are basically Alexa robots [Online]. Wired. Available from: https://www.wired.co.uk/article/amazon-2020-live-event-highlights

Wollerton, M. 2019. Loving a robot dog is about so much more than not cleaning up poop [Online]. CNET. Available from: https://www.cnet.com/features/loving-a-robot-dog-is-about-so-much-more-than-not-cleaning-up-poop/

There are quite a few mods floating round the internet which guide you through teasing your Famicom to output something a little more modern. The conclusion I drew pretty rapidly was that RGB output is out of the window unless you’re going to spend a lot of money, lower your expectations to composite and don’t expect a perfect picture. Given the age of the machine I’m pretty impressed that it still works, especially as mine appears to be a second revision motherboard — how many Playstation 3’s are still working that have a second revision board, I strongly suspect they won’t be after 30 years!

Anyway, on to the mod!

I chose to do this mod from “80sFREAK”, it seemed to be the best documented out there but I still found it pretty difficult to follow, partly because the Famicom board shown doesn’t match mine exactly — some of the PCB traces and components are positioned differently. Luckily there’s a few people out there who have also tried this mod which was enough for me to piece together where everything connects, of particular help was this blog post, this post and this other guide (particularly for labelling of pin numbers on the PPU) so thanks to both of the authors!

Important notes

• This mod disables the RF output
• You will need to shoe horn in two phono connectors somehow
• You will only get mono sound (which is all the Famicom and NES were capable of anyway)
• If you break it it’s your fault

Shopping list

• 100r Cr25 0.25w Cf Resistor
• 150r Cr25 0.25w Cf Resistor
• 33uf 16v 2.5mm Tantalum Bead Capacitor
• 220uf 10v 105deg Nrsz Electro Capacitor
• Yellow Phono Socket
• Red or Black Phono Socket

Modification

Step 1

Take apart the Famicom, there’s a good walkthrough on IFIXIT

Step 2

On the board there’s a transistor labelled ECB, carefully desolder it and keep it handy. It gets reused in this mod.

The transistor has a notch in it, use this to keep track of which pins are which when it has been removed.

Step 3

The original guide will tell you at this step to isolate pin 21 of the PPU (the Picture Processing Unit, the largest chip next to the ECB transistor). I tried this mod without isolating it and then went back and isolated it and saw very little difference, your mileage may vary.

If you want to do this step there are two recommended methods in the original tutorial, either lift the chip leg or cut the PCB trace. I cut the trace but, as mentioned earlier, my PCB differs from the one shown in the guide.

This picture shows pin 21 (red square) on the reverse of the PCB and shows the trace which I cut (orange line):

Step 4

Build the circuit shown in step 3 of the original guide, I’ve borrowed 80sFREAK’s images here (@80sFREAK — if this is a problem just contact me and i’ll remove them).

Though the advise is to keep wires as short as possible I chose to do this on a board as it separated it nicely from the PCB and left me with no fears of it shorting!

This is the numbering for the PPU legs on the component side of the board:

Step 5

That’s the video output done, now to add audio. Follow the circuit diagram and solder to pin 46 of the cartridge connector (again these are 80sFREAK’s images)

For this one I soldered to the back of the PCB as the pins are much more accessible (and there’s enough wires now trailing off the front!)

Step 6

With everything wired up this is a good chance to stop and check whether it works before you go drilling holes in the case!

Assuming it’s all working pick somewhere on the case that your mod can reach and that’s convenient for AV connectors and make some holes! I personally went for the front right as there’s a fair amount of free space on either side at the front.

Step 7

Cram it all into the case and screw it back together!

Hopefully it all works and you’ve got a reasonable quality picture, mine occasionally jumps a little and has pretty visible “jailbars” but overall it’s pretty good for a console from 1983.

Get the right Unity version

There’s lots of versions of Unity and just grabbing the latest isn’t going to get you very far. The various moving parts involved all rely on each other being at certain specific versions.

So to get started install Unity Hub from here.

Once Unity Hub is installed, select “Installs” and you should see something like this… (but without the listed installs)

Click “ADD” and choose 2019.1.4f1.

A quick note about the versions — my day to day development environment is normally Apple’s Xcode, this means my gut instinct is to grab the latest verision. At the time of writing the latest Unity version is 2019.3.0a3 and the next latest is 2019.2.0b3 the thing to be wary of is that the a and b in both of these version codes refer to Alpha and Beta versions of Unity. From my trial and error it is possible to get something working with the Oculus Quest but you will struggle with getting the Oculus Integration compiling/working.

Modules

When installing the Unity version(s) ensure that you have the following included:

As the Oculus Quest is an Android device you will need Android Build Support and the SDK/NDK tools. Make sure you expand the tree to confirm both are selected (by default the SDK & NDK Tools aren’t selected).

Create a new project

Using Unity Hub go to “Projects” and click “NEW”, if you have installed multiple versions of Unity you can use the down arrow next to “NEW” to select the version we added in the previous step (2019.1.4f1).

Select “3D” and give the project a name and location, hit “CREATE” and wait for your new project to appear.

Oculus Integration

Now your new project has been created head to the Asset Store and search for “Oculus”. The first result should look something like this:

Press “Import” and a dialog will appear asking what to import, by default everything is selected so just press “Import”. This will likely take some time.

When it’s done you will probably see this dialog:

Go ahead and press “Yes” and you’ll be asked permission to restart Unity, allow it to do it’s thing.

When Unity has restarted and your project is back on screen you should now have some Oculus specific options in the menu bar.

If you don’t have this, double check you created the project with the correct version of Unity.

If you do have it, leave it alone for now, we’ll come back to it shortly.

Project Settings

We need to set out project to build for Virtual Reality and more specifically, in the case of the Quest, we need to set it to build for Oculus VR on Android.

To do this go to the Edit menu and select “Project Settings”. In the window which appears (which should already be set to Player in the left hand column), select the Android tab in the right hand area, find the XR Settings and tick “Virtual Reality Supported”.

Use the “+” button to add Oculus

Notice the warning stating “XR is currently not supported when using the Vulkan Graphics API.”. This tripped me up, i initially treated this as a throw away warning. It’s not, you’ll just get an empty space if you run in this configuration. So while we’re in the Project Settings we’ll fix this.

Go to “Other Settings” and notice the “Graphics APIs” list:

Select “Vulkan” by clicking it and then press “-” to remove it from the list.

While in this section there’s another thing we need to fix, the Oculus Quest needs a minimum Android API level of 19 but by default the minimum is set to 16. Scroll down, find “Minimum API Level” and set it to “Android 4.4 ‘KitKat’ (API Level 19)”.

Add an Oculus API key

Remember those new menu items for Oculus we got earlier? Go to “Oculus” > “Platform” > “Edit Settings”

You’ll now see a couple of big red errors that we’re going to fix…

Click the button that says “Create / Find your app on https://dashboard.oculus.com”, perhaps unsurprisingly a browser window will be opened taking you to the Oculus dashboard for your account. If you don’t have an account you will need to create one to proceed.

In the browser window click “Create new app” and you will be asked which platform, at the time of writing the Quest isn’t an option, however the Go & Gear VR are close enough so select that option.

Give it a name, you probably want this to be the same as your project name, and then hit “Save and Continue”. The next screen will give you the all important App ID, copy this and paste it in to Unity in the “Application ID”, “Oculus Rift” and “Oculus Go/Quest or Gear VR” settings boxes within the Oculus Platform Settings.

You’ll still have an error on this panel, to fix it untick “Standalone Platform” and it should go away.

Build Settings

Though we’ve configured the player for the Quest (by setting some android preferences and telling it to use the Oculus SDK) we haven’t actually told Unity to build for Android.

To do this, go to “File” > “Build Settings”…

Select “Android”, change “Texture Compression” to “ASTC” and press the “Switch Platform” button. This will take some time…

Really quite some time.

Hook up the Quest

Before actually connecting your headset to the computer you need to put your Quest in to developer mode (as you might with any Android device you were trying to run code on).

Developer Mode

Hopefully the user account you’re logged in to on the Quest is the same as the one you added a development app to earlier. This then allows the Oculus app on your mobile phone to toggle Developer Mode on the headset.

Open the Oculus app on your phone, go to the Settings tab and hopefully you can see your headset listed.

Tap the Quest headset and you should see the following:

Tap “More Settings” and “Developer Mode” should be listed here, toggle this on.

Connecting

Connect the Quest to your computer using a USB cable.

The first time you do this you’ll get a dialog in the headset asking permission to allow the connected computer to access the Quest. You’ll need to put on the headset and use the controller to allow permission.

If all this has gone smoothly, in Unity, you should now be able to see the Quest in the device list in Build Settings (you may need to make use of the “Refresh” button).

Running on the device

By default the open scene we have in Unity is pretty empty and boring and not set up for the Oculus.

Luckily the Oculus Integration includes some nice sample conent.

Go to the following path in the project browser…

…and double click “GearVRControllerTest”.

Notice in the Hierarchy that this is already setup with a OVRCameraRig.

Go back to “Build Settings” and press the button labelled “Add Open Scene” to put the GearVRControllerTest scene in to the build.

Now press “Build and run”, a save dialog will be shown, give it a location to save to (this is where the APK file will be built and saved to), put the headset on and then be patient — it should run up on the Quest.

Problems?

Hopefully everything ran fine.

However I have encountered times when nothing has run up, even though i’ve followed the steps above.

The two common issues I’ve seen are:

1. Unity thinks it’s targetting an Oculus Go instead of an Oculus Quest.

Go to the menu bar, select Oculus > Tools > Oculus Platform Tool

Make sure that the Target Oculus Platform is set to “Oculus Quest”.

2. Somehow the Vulkan Graphics API is back in the mix

If you look in the console you might see this warning appear:

Despite explicitly removing the Vulkan API in an earlier step, sometimes it creeps back in.

Go to your Player Settings as before, find “Other Settings”, “Graphics APIs” and remove “Vulkan” from the list.

Further Reading and Acknowledgements

This article is very much reusing and combining content from other sources and wouldn’t be possible without them.

Big thanks to “Frontend Fanatics” and their video https://www.youtube.com/watch?v=eySe4Wj6xbk
and to “Circuit Stream” and their article https://circuitstream.com/oculus-unity-setup/

The Oculus documentation is also extremely valuable https://developer.oculus.com/documentation/quest/latest/concepts/book-unity-gsg/

Hire me

You can find me here and on LinkedIn.

You likely have at least a development and a production server to let you handle this situation, with both URLs somewhere in your app. But how do you manage switching your app between them? This is what we’re aiming to answer here!

Things not to do

Commenting in/out the development/server URL

This is the minimum possible effort, it gets you up and running quickly but in the long run you’ll be cursing it.

Why is this bad?

There will likely come a time when you simply forget to comment in the correct URL and you either accidentally send someone a “test” build pointing at the production server, or worse you accidentally push the wrong one up to the App Store.

Multiple targets

On the face of it this seems like a good option — you duplicate the existing project target and change some settings, you’ve got multiple options to run/archive/test builds pointing at different options.

Why is this bad?

Each source code file has to be added to each target it needs to be built with, by default new source files are only added to the original build target. The source itself is more of an inconvenience as the compiler will moan at you when you forget to add a source file to the target. However this same rule applies to other files, including images, xcassets, storyboards, etc. The compiler won’t necessarily pick up that these are missing for a target and the first time you’ll find out about it is when someone gets to the affected part of the app at runtime.

If you have a lot of server environments (I’ve worked on projects where we needed to support 5+ different staging environments) adding files to all of the targets gets old fast.

Yes you can write a script to validate that all the source and other artefacts are consistent between each target, however it’s a bit of a faff and there is a better way.

Another thing to note is though this worked in Objective C — you could just add new User-Defined Build Settings for each additional server environment — these same Preprocessor Macro’s aren’t compatible with Swift.

A better way

So the ideal approach requires us to not have to make any code changes to switch server environment and doesn’t require any additional Targets to be defined.

Build Configurations

Your bog standard default Xcode project already has two Build Configurations set up. One called Debug and one Release, they should look super similar to this:

To some extent these exist to allow you to specify different Build Settings and configuration for when deploying to the test devices and the simulator vs the App Store/Production. We essentially want to duplicate these configurations for our other environments.

Adding “Staging”

Assume we have an additional server we’re using as our staging server and we want to be able to deploy builds of the app that use that server instead of the production server.

Use the + button to duplicate Debug (with appropriate naming) and then repeat for Release.

You should end up with something similar to:

If you look at your Build Settings you’ll see that you’re now able to configure each option separately for each of these four configurations. But don’t. Yet…

Active Compilation Conditions

So we have our additional Build Configurations but we’re not yet doing anything to specify our other server environments. As mentioned earlier Swift doesn’t really do Preprocessor Macros so adding extra Build Settings won’t help us here, instead Swift now has something called Active Compilation Conditions. This lets us do something similar.

Find the Active Compilation Conditions and add “STAGING” to both the DebugStaging/ReleaseStaging configurations.

Finally, FINALLY we can actually add some code.

Wherever you’d like to be able to detect whether you’re using your staging build or not you can add the following code:

let url: String
#if STAGING
    url = "https://staging.server.notarealurl.com"
#else
    url = "https://production.server.notarealurl.com"
#endif

Schemes

The problem that you might have spotted is that there’s still no way to build an app based on these additional configurations.

This is where Schemes come in. The end goal here is to have that drop down box at the top left of Xcode list out the different Schemes you can build/archive for to make it super convenient to do — like this:

All we need to do is duplicate the existing scheme, tweak it to point at our newly added build configurations and it’s done!

Just select New Scheme from that same menu…

…give it a sensible name…

…and set each of the actions to point at the new schemes rather than the original…

…it should end up like this

Build and run your new scheme and (if you add an appropriate breakpoint/print statement) you should see it selects the STAGING branch of the if/else statement we added, and consequently the staging server URL.

Stretch goals

You now have your project configured for quickly building for production and staging server environments without having to do anything particularly prone to human error (like having to remember to change code).

What’s even better than being able to conveniently build your app for multiple environments? Being able to install them both — side by side — on your test devices!

To accomplish this you’re going to need different Bundle Identifiers for each environment (so that iOS doesn’t see them as the same app) and in order for humans to tell which one is which you’re going to need to specify different icons.

Different Bundle Identifiers

This is where we get to use those User-Defined Build Settings. Select the target and Build Settings and press the + button to add a new one called “BUNDLE_IDENTIFIER”:

Add something appropriate making sure to keep Debug/Release and DebugStaging/ReleaseStaging the same

We now need to make sure these new values are used. In your Build Settings find “Product Bundle Identifier” and set it to ${BUNDLE_IDENTIFIER}.

Note: if you’re using push notifications or any other features which require entitlements associated with the Bundle Identifier you will need to add a new App ID in your Developer Portal and generate the appropriate Provisioning Profiles.

Different Icons

This is an incredibly similar process to the Bundle Identifier.

Add another User-Defined Build Setting, this time calling it APP_ICON_NAME. For the purposes of this we’re assuming you’re using the default asset catalogue icon set. Then add appropriate icon naming for all configurations (note that the debug/release are set to the existing icon set name).

As with the bundle identifier we’re going to replace the original setting (the ever catchy Asset Catalog App Icon Set Name) with our custom build setting, ${APP_ICON_NAME}.

Finally the only thing remaining is to add an actual Icon Set for our staging build.

Go to your asset catalog and add a new icon set called AppIconStaging along with some appropriate imagery

If everything’s worked you should now be able to run the staging scheme and find two apps with two different icons installed in your simulator.

Conclusion

Having to deal with multiple server environments is something that mobile developers will find themselves dealing with on the majority of projects. This article shows you one way to automate the switching so that it’s not really something you need to expend any brain power on when creating builds.

We’ve ended up with a project set up to build for both production and staging environments, allows both versions of the app to be installed side by side and has different icons.

It is my hope that I never have to pick up another project where there’s a set of URLs that are commented and uncommented to switch server environments ;)

Hire me

You can find me here and on LinkedIn.