Adam Paszke

A quick tour of Torch internals

Sun, 18 Oct 2015 20:05:24 +0000

Recently, I have been kind of confused. I couldn’t find myself anything to work on and had no ideas for new projects (apparently, I just had to wait for the new academic year to start - I have plenty of ideas now, but no time for them).

Anyway, I often get the impression that many people are using Machine Learning libraries as a kind of black-boxes with only a high-level API. It’s as if they weren’t interested at all in how they work, but solely in the output (this is why I like Torch so much - it’s hackable to the bone). I’ve been using Torch for a few months now and I’ve always been curious how it’s built. This is why I decided to get down to it and browse the code of TH library, which is at the core of Torch.

It’s really a great thing to do. I’ll write more about it in the end of this post, but you should seriously consider doing it with your favourite library or framework too.

Torch’s source is written in plain C, which was very pleasing to me. I don’t really like many C++ features and although I find it very powerful and flexible, it often seems confusing. C’s extremely minimal syntax allows you to read and quickly grasp what exactly happens at any moment. However, if C++ is the way to go for you, there is also a wrapper around TH called thpp.

Where can you get it?

You can find the TH library in two places:

In its standalone repository (git subtree of torch7; outdated at the time of writing)
In torch7 repository in lib/TH folder (always up to date)

The folder structure is very simple. There are some cmake tests and definitions in cmake directory while the code is located both in generic directory and at the repo root.

Interesting findings

Before going into the details and describing functionality implemented in individual files I’d like to point out some really cool techniques that I’ve found in the implementation.

Code generation

First thing that appeared really strange to me was that many files existed both in the root folder as well as in generic. If you opened them, you would quickly notice that copies in generic contain the actual code, while at the root they all look very similar. Here is THStorage.c for example:

#include "THAtomic.h"
#include "THStorage.h"

#include "generic/THStorage.c"
#include "THGenerateAllTypes.h"

#include "generic/THStorageCopy.c"
#include "THGenerateAllTypes.h"

Quite unusual for a .c file, right?

THGenerateAllTypes sounds interesting so i looked it up and this if what I’ve found:

#ifndef TH_GENERIC_FILE
#error "You must define TH_GENERIC_FILE before including THGenerateAllTypes.h"
#endif

#define real unsigned char
#define accreal long
#define Real Byte
#define TH_REAL_IS_BYTE
#line 1 TH_GENERIC_FILE
#include TH_GENERIC_FILE
#undef real
#undef accreal
#undef Real
#undef TH_REAL_IS_BYTE

#define real char
#define accreal long
#define Real Char
#define TH_REAL_IS_CHAR
#line 1 TH_GENERIC_FILE
#include TH_GENERIC_FILE
#undef real
#undef accreal
#undef Real
#undef TH_REAL_IS_CHAR
...

Which continued for a few more types. At first I was puzzled, but then I suddenly realized what it does and how brilliant this is! There are no templates in C, but objects like THStorage should be available for different types. It would be a terrible waste to repeat the same implementation with just a few words replaced and this is what this piece achieves! In generic files you can see variables of type real all over the place. At first it was obvious to me that it’s probably a matter of some compile time optimizations whether it was chosen to be a float or a double, but apparently it’s different - it allows code generation for many other types too!

Clever usage of macros also makes the generic files more readable. Take this example from generic/THStorage.c:

real* THStorage_(data)(const THStorage *self)

It looks nice, but what about name conflicts for different types? It can’t be THStorage and THStorage_data all the time! Worry not, macros take care of that as well:

#define THStorage        TH_CONCAT_3(TH,Real,Storage)
#define THStorage_(NAME) TH_CONCAT_4(TH,Real,Storage_,NAME)

During preprocessing this function name will be expanded to something like THByteStorage_data and THStorage will be replaced with THByteStorage. Super cool x2!

It’s also smart to use a #line 1 TH_GENERIC_FILE directive, because if there would be any errors they will appear in the compiler as if they were in the original generic file - not in the middle of the implementation pasted over and over.

I think that these are some awesome ways to make C code more type-agnostic.

OOP & Virtual tables

TH also implements a file API, where you can find good examples of how you could implement some basic OOP patterns in C. There are four files that I’ll be talking about here:

THFilePrivate.h - defines basic structs
THFile.c - contains some generic implementation
THDiskFile.c - code for handling disk files
THMemoryFile.c - implementation of in-memory files

Let’s start with the private header file.

struct THFile__
{
    struct THFileVTable *vtable;

    int isQuiet;
    ...
};

struct THFileVTable
{
    int (*isOpened)(THFile *self);

    long (*readByte)(THFile *self, unsigned char *data, long n);
    long (*readChar)(THFile *self, char *data, long n);
    long (*readShort)(THFile *self, short *data, long n);
    long (*readInt)(THFile *self, int *data, long n);
    ...
};

You can see that it defines a virtual method table with pointers to functions that THFile subclasses will have to implement (THFile is an abstract class - it has no constructors). Other structs are defined as such:

typedef struct THDiskFile__
{
    THFile file;

    FILE *handle;
    char *name;
    int isNativeEncoding;

} THDiskFile;

What makes this struct interesting is that because it’s first member is of type THFile it’s actually valid to cast struct THDiskFile * to struct THFile * and use it normally. What’s more, because THDiskFile’s constructor fills in the function pointers in file field’s virtual table, it will behave as THDiskFile object even when casted to THFile!

Shared memory

I had little knowledge about UNIX process management and threading until now, when I took up an operating systems course at my university, so it was really interesting to learn about mmap (maps a file to memory, so you can use it like an array) and to see how memory can be shared between processes with shm_open. I even wrote a piece of code to try it out. You can find it here.

SIMD

Another cool thing you can find in TH are vector instructions. There are some cmake tests that check if they are available on your CPU (cmake/FindSSE.cmake) and several files implementing convolution operations using them (generic/simd/*). I can’t understand it yet - function that takes 10 lines of code is expanded to unrolled vectorized loop taking more than 120 lines and using APIs with unreadable function names for a SSE beginner. This code spans 134 lines after macro expansion:

void convolve_5x5_1_avx(float* output, float* image, float* weight, long count, long outputStride, long inputStride) {
  long i = 0;
  long alignedCount = count & 0xFFFFFFF8;
  DECLARE_OUTPUT_1()
  for (; i < alignedCount; i+=8) {
    CONVOLVE_8COLS_XROWS(1, i)
  }
}

Anyway, it’s definitely a thing worth learning so I will probably write more about it soon!

Allocators

TH declares it’s own function for memory allocation called THAlloc. It tries to allocate a properly aligned chunks if you allocate big blocks and handles out-of-memory errors. Before reading Torch’s source I didn’t know about the concept of allocators. They are just small virtual tables providing their own memory management API (alloc, realloc, free). It’s cool that you can pass an Allocator to THStorage or THTensor and construct it not only in the regular heap region, but also allocate it in the shared memory.

Random module

It’s natural to have a pseudorandom number generator in all programming languages, but I’ve never read an implementation of one (ok, except the linear congruential generator). In THRandom.c you can find a full implementation of Mersenne twister, which (according to Wikipedia) is a default implementation for R, Python, Ruby, PHP, CMU Common Lisp, GLib, MATLAB and some more. There are also several methods which convert returned uniform distribution into other shapes.

Quick library overview

In this section I will briefly describe most of the functionalities provided by TH.

THAllocator
- creates a default default allocator, which just calls TH memory management functions and, if possible, a THMapAllocator that can map files or shared memory objects into memory.
THAtomic
- multiplatform implementation of atomic operations
THTensor
- defines a general Tensor type
- supports lots of indexing, linear algebra and math operations
- available for all primitive datatypes (TH<type>Tensor, e.g. THFloatTensor)
THBlas
- wraps BLAS library for use in THTensor
- provides a general implementation as a fallback
THLapack
- wraps LAPACK library for use in THTensor
- DOESN’T provide fallbacks - throws errors if called
THFile
- abstract file class
- only creates wrappers for calling methods contained in virtual table
THDiskFile
- concrete file class
- wraps disk file APIs
THMemoryFile
- concrete file class
- operates on an in-memory buffer and fakes file operations
THGeneral
- implements general utilities
- contains memory management routines
- can notify external GCs
THRandom
- implements a random number generator
- can sample from many distributions
THStorage
- defines a general storage object
- contains mainly bookkeeping code
- available for all primitive datatypes (TH<type>Storage, e.g. THFloatStorage)

How to use it

If you want to install TH you can either perform a full Torch installation or you can follow these steps:

# clone Torch repository
git clone https://github.com/torch/torch7
mkdir th_build
cd th_build
# configure TH build
cmake ../torch7/lib/TH
# compile library
make
# install shared library and header files
make install

Then, you only have to #include <TH/TH.h> in your program and link the library during the compilation process (-lTH).

Example program

To wrap up I just wanted to show you an example program using TH. It will simply load 10 floats from two files into tensors, compute their dot product and add to it a sum of all values in one of them. This is the code:

#include "TH/TH.h"

int main()
{
    THFile *x_file = THDiskFile_new("x", "r", 0);
    THFile *y_file = THDiskFile_new("y", "r", 0);

    THFloatTensor *x = THFloatTensor_newWithSize1d(10);
    THFloatTensor *y = THFloatTensor_newWithSize1d(10);

    THFile_readFloat(x_file, x->storage);
    THFile_readFloat(y_file, y->storage);

    double result = THFloatTensor_dot(x, y) + THFloatTensor_sumall(x);

    printf("%f\n", result);

    THFloatTensor_free(x);
    THFloatTensor_free(y);
    THFile_free(x_file);
    THFile_free(y_file);
    return 0;
}

All input parsing and possible errors are handled by Torch. Convenient, isn’t it?

Afterthoughts

I actually enjoy reading other’s source code - especially if it’s well written. If you have some spare time, then seriously, consider picking your favourite library or framework, and try to understand how it works - even the tiniest bits of it. I guarantee that you will find many fascinating things and learn many concepts and ways of structuring your code that you had no idea existed. I haven’t learned that much in such short period of time for a while. I liked it so that I’m thinking about doing this on a more regular basis.

TH has no documentation at the moment. Since I’ve already studied most of it’s code, I’ll probably try to write at least a bit. I’ve used Torch for so long that it’s time to make some contribution myself.

Thanks for reading! I hope that you liked it!

LSTM implementation explained

Sun, 30 Aug 2015 20:04:15 +0000

Preface

For a long time I’ve been looking for a good tutorial on implementing LSTM networks. They seemed to be complicated and I’ve never done anything with them before. Quick googling didn’t help, as all I’ve found were some slides.

Fortunately, I took part in Kaggle EEG Competition and thought that it might be fun to use LSTMs and finally learn how they work. I based my solution and this post’s code on char-rnn by Andrej Karpathy, which I highly recommend you to check out.

RNN misconception

There is one important thing that as I feel hasn’t been emphasized strongly enough (and is the main reason why I couldn’t get myself to do anything with RNNs). There isn’t much difference between an RNN and feedforward network implementation. It’s the easiest to implement an RNN just as a feedforward network with some parts of the input feeding into the middle of the stack, and a bunch of outputs coming out from there as well. There is no magic internal state kept in the network. It’s provided as a part of the input!

The overall structure of RNNs is very similar to that of feedforward networks.

LSTM refresher

This section will cover only the formal definition of LSTMs. There are lots of other nice blog posts describing in detail how can you imagine and think of these equations.

LSTMs have many variations, but we’ll stick to a simple one. One cell consists of three gates (input, forget, output), and a cell unit. Gates use a sigmoid activation, while input and cell state is often transformed with tanh. LSTM cell can be defined with a following set of equations:

Gates:

\[i_{t} = g(W_{xi}x_{t} + W_{hi}h_{t-1} + b_{i})\] \[f_{t} = g(W_{xf}x_{t} + W_{hf}h_{t-1} + b_{f})\] \[o_{t} = g(W_{xo}x_{t} + W_{ho}h_{t-1} + b_{o})\]

Input transform:

\[c\_in_{t} = tanh(W_{xc}x_{t} + W_{hc}h_{t-1} + b_{c\_in})\]

State update:

\[c_{t} = f_{t} \cdot c_{t-1} + i_{t} \cdot c\_in_{t}\] \[h_{t} = o_{t} \cdot tanh(c_{t})\]

It can be pictured like this:

Because of the gating mechanism the cell can keep a piece of information for long periods of time during work and protect the gradient inside the cell from harmful changes during the training. Vanilla LSTMs don’t have a forget gate and add unchanged cell state during the update (it can be seen as a recurrent connection with a constant weight of 1), what is often referred to as a Constant Error Carousel (CEC). It’s called like that, because it solves a serious RNN training problem of vanishing and exploding gradients, which in turn makes it possible to learn long-term relationships.

Building your own LSTM layer

The code for this tutorial will be using Torch7. Don’t worry if you don’t know it. I’ll explain everything, so you’ll be able to implement the same algorithm in your favorite framework.

The network will be implemented as a nngraph.gModule, which basically means that we’ll define a computation graph consisting of standard nn modules. We will need the following layers:

nn.Identity() - passes on the input (used as a placeholder for input)
nn.Dropout(p) - standard dropout module (drops with probability 1 - p)
nn.Linear(in, out) - an affine transform from in dimensions to out dims
nn.Narrow(dim, start, len) - selects a subvector along dim dimension having len elements starting from start index
nn.Sigmoid() - applies sigmoid element-wise
nn.Tanh() - applies tanh element-wise
nn.CMulTable() - outputs the product of tensors in forwarded table
nn.CAddTable() - outputs the sum of tensors in forwarded table

Inputs

First, let’s define the input structure. The array-like objects in lua are called tables. This network will accept a table of tensors like the one below:

local inputs = {}
table.insert(inputs, nn.Identity()())   -- network input
table.insert(inputs, nn.Identity()())   -- c at time t-1
table.insert(inputs, nn.Identity()())   -- h at time t-1
local input = inputs[1]
local prev_c = inputs[2]
local prev_h = inputs[3]

Identity modules will just copy whatever we provide to the network into the graph.

Computing gate values

To make our implementation faster we will be applying the transformations of the whole LSTM layer simultaneously.

local i2h = nn.Linear(input_size, 4 * rnn_size)(input)  -- input to hidden
local h2h = nn.Linear(rnn_size, 4 * rnn_size)(prev_h)   -- hidden to hidden
local preactivations = nn.CAddTable()({i2h, h2h})       -- i2h + h2h

If you’re unfamiliar with nngraph it probably seems strange that we’re constructing a module and already calling it once more with a graph node. What actually happens is that the second call converts the nn.Module to nngraph.gModule and the argument specifies it’s parent in the graph.

preactivations outputs a vector created by a linear transform of input and previous hidden state. These are raw values which will be used to compute the gate activations and the cell input. This vector is divided into 4 parts, each of size rnn_size. The first will be used for in gates, second for forget gates, third for out gates and the last one as a cell input (so the indices of respective gates and input of a cell number \(i\) are \(\left\{i,\ \text{rnn_size}+i,\ 2\cdot\text{rnn_size}+i,\ 3\cdot\text{rnn_size}+i\right\}\)).

Next, we have to apply a nonlinearity, but while all the gates use the sigmoid, we will use a tanh for the input preactivation. Because of this, we will place two nn.Narrow modules, which will select appropriate parts of the preactivation vector.

-- gates
local pre_sigmoid_chunk = nn.Narrow(2, 1, 3 * rnn_size)(preactivations)
local all_gates = nn.Sigmoid()(pre_sigmoid_chunk)

-- input
local in_chunk = nn.Narrow(2, 3 * rnn_size + 1, rnn_size)(preactivations)
local in_transform = nn.Tanh()(in_chunk)

After the nonlinearities we have to place a couple more nn.Narrows and we have the gates done!

local in_gate = nn.Narrow(2, 1, rnn_size)(all_gates)
local forget_gate = nn.Narrow(2, rnn_size + 1, rnn_size)(all_gates)
local out_gate = nn.Narrow(2, 2 * rnn_size + 1, rnn_size)(all_gates)

Cell and hidden state

Having computed the gate values we can now calculate the current cell state. All that’s required are just two nn.CMulTable modules (one for \(f \cdot c_{t-1}^{l}\) and one for \(i \cdot x\)), and a nn.CAddTable to sum them up to a current cell state.

-- previous cell state contribution
local c_forget = nn.CMulTable()({forget_gate, prev_c})
-- input contribution
local c_input = nn.CMulTable()({in_gate, in_transform})
-- next cell state
local next_c = nn.CAddTable()({
  c_forget,
  c_input
})

It’s finally time to implement hidden state calculation. It’s the simplest part, because it just involves applying tanh to current cell state (nn.Tanh) and multiplying it with an output gate (nn.CMulTable).

local c_transform = nn.Tanh()(next_c)
local next_h = nn.CMulTable()({out_gate, c_transform})

Defining the module

Now, if you want to export the whole graph as a standalone module you can wrap it like that:

-- module outputs
outputs = {}
table.insert(outputs, next_c)
table.insert(outputs, next_h)

-- packs the graph into a convenient module with standard API (:forward(), :backward())
return nn.gModule(inputs, outputs)

Examples

LSTM layer implementation is available here. You can use it like that:

th> LSTM = require 'LSTM.lua'
                                                                      [0.0224s]
th> layer = LSTM.create(3, 2)
                                                                      [0.0019s]
th> layer:forward({torch.randn(1,3), torch.randn(1,2), torch.randn(1,2)})
{
  1 : DoubleTensor - size: 1x2
  2 : DoubleTensor - size: 1x2
}
                                                                      [0.0005s]

To make a multi-layer LSTM network you can forward subsequent layers in a for loop, taking next_h from previous layer as next layer’s input. You can check this example.

Training

If you’re interested please leave a comment and I’ll try to expand this post!

That’s it!

That’s it. It’s quite easy to implement any RNN when you understand how to deal with the hidden state. After connecting several layers just put a regular MLP on top and connect it to last layer’s hidden state and you’re done!

Here are some nice papers on RNNs if you’re interested: