In 2015, I wanted to help with AI research, but taking the first steps felt daunting. I’d graduated from MIT then spent eight years building web startups. I’d put in my 10,000 hours, gotten funding from Y Combinator and grown a company to thirty people. Moving to research felt like starting over in my career. Was it really a good idea to throw away years of work?

A friend told me about South Park Commons (SPC), a new space for people who were taking the first steps on a new path, and introduced me to Ruchi, the founder. Ruchi is super impressive, she was one of the earliest Facebook engineers, and had founded and sold a successful company. She also has a high-bandwidth and disarmingly direct communication style that I found refreshing.

Over lunch, Ruchi described South Park Commons as a community in whicheveryone is starting over. Starting over is in fact the main thing that unifies the group. For example, two current Commons members are Jason, the maintainer of a popular open-source project Quill, who’s been learning to do enterprise sales, and Malcolm, a successful infrastructure engineer who’s starting a fund Strong Atomics to invest in nuclear fusion companies.

I joined South Park Commons, blocked off three months to see if I could make progress, and made a plan to teach myself machine learning. Several other SPC members were interested in the space, so we started going through a curriculum of courses and organized a paper reading group. As soon as I got over the fear and took the plunge, things got vastly easier. Six months of focused work later, I had a position at OpenAI.

If you’re thinking about switching into the field of AI, here are some resources that were useful to me in getting started

• Stanford’s CS231n — Convolutional Neural Networks for Visual Recognition
• Greg Brockman’s Quora post on picking up Deep Learning skills as an engineer
• 80000 Hours AI Safety Syllabus

How I made the switch to AI Research was originally published in Tom B Brown on Medium, where people are continuing the conversation by highlighting and responding to this story.

In RL, we usually run algorithms on the CPU because we’re usually data-starved (limited by the speed at which the environment generates data). In image classification, everything is on the GPU, so I wrote a basic algorithm, then tried to set things up for determinism. No such luck — It turns out there’s no easy way to get determinism on the GPU.

For example, doing backprop through a simple reduction operation is nondeterministic.

Running it outputs the following:

CPU (deterministic)
23.066511
23.066511
23.066511
23.066511

GPU (nondeterministic)
23.066513
23.066511
23.066509
23.066513

This isn’t a bug that will be fixed. There’s a fundamental tradeoff between speed and determinism. From @yaroslavb:

Floating point math is non-associative, and if you want speed, you use intermediate results as soon as they arrive on multiple cores, so you get slightly different answers, which get blown up later

When you’re using GPUs, you can’t get determinism without making your code 10x slower, defeating the whole point of GPUs.

So I’m giving up on determinism for image classification now. Best alternative so far: Run multiple seeds of the algorithm and overlay them:

ML Tidbits: Nondeterminism on the GPU was originally published in Tom B Brown on Medium, where people are continuing the conversation by highlighting and responding to this story.

He started slow, but soon had her trust, and together they began to weave their dreams. Their dreams were of adventures and far off lands. They would be Egyptian lovers in ancient times, or sworn enemies fighting great battles. The girl was a horse and the boy was her keeper. The girl was a bumblebee and the boy was a flower. They were siblings and rivals and predators and prey.

In a night’s sleep together they’d each experience a lifetime. Sometimes the lives were short and vivid, sometimes they were long and sprawling. During their dream, they’d forget that they were dreaming. And sometimes they’d be pulled apart. The boy would die in an accident. The girl would be killed by a jealous lover. But no matter how long each life took, they would always live the whole life, and awaken at the same moment, holding each other in a tight embrace.

And together, they lived a million million lives, outlandish ones and plain ones. They learned more about each other than anyone had ever learned about anyone else before. And then one day, the girl suggested something that they’d both been thinking for a long time. The dream that they had seen in their periphery, but had always avoided. The boy was scared. There was more that they still hadn’t experienced together.

And yet, stronger than his fear, the boy still wanted to answer the question of her laughter. So they laid down side-by-side, and dreamt of being each other.