David Barber

Generative Neural Machine Translation

Wed, 12 Sep 2018 00:00:00 +0000

Machine Learning models are still largely superficial – the models don’t really ‘understand’ the meaning of the sentences they are translating. If we want increasingly ‘intelligent’ machines, it’s important that models begin to incorporate more knowledge of the world.

One approach to achieve this is to require models to be good, not only at translation, but additional tasks, such as question answering. A parallel direction is to encourage models to internally focus on the meaning of the sentence. This post summarises our approach to this latter direction, published at NIPS 2018¹ (arxiv version).

Learning meaningful representations of data

One possible method to learn meaningful representations of sentences is to use a latent variable model. Latent variable models are based on the hypothesis that for each sentence, there is a ‘hidden’ (or ‘latent’) vector which represents the sentence’s meaning, and that the sentence itself is a textual manifestation of that meaning.

Latent variable models in natural language processing typically posit the following generative process for sentences:

The ‘hidden’ or ‘latent’ representation of the sentence’s meaning is randomly generated according to a prior distribution.
The sentence itself is then generated conditioned on this latent representation.

Given a latent variable model and a sentence, the posterior distribution of the latent representation (i.e. the values of the representation that are likely to have generated that sentence) can be inferred. This posterior distribution can then be used for downstream tasks, e.g. the inferred representation could be used for answering questions about that sentence. Intuitively, the more information about the meaning of the sentence that the representation contains, the better it should be at performing at downstream tasks. Therefore, we would like to design a model which can use its latent representation to better represent the semantics of the text.

Most latent variable models use one latent representation per sentence in the data set. The problem with this approach is that there are no guarantees that the representations will learn semantically meaningful information about the text. For example, consider the two sentences: “she walked across the road” and “the woman crossed the street” - a basic latent variable model does not know a priori that walking across a road and crossing a street are similar actions. Therefore, the typical model would not be able to guarantee that the latent representations of these two sentences are similar.

Instead, if we were able to encode into the model that two sentences are semantically similar, we may be able to learn representations which better understand the meaning of the text. Unfortunately, large corpora of sentences with similar meanings in a single language are rare. However in the machine translation context, the same sentence expressed in different languages offers the potential to learn a latent variable model which better represents the sentence’s meaning. For example, a model which knows that the English sentence “the woman crossed the street” and the French sentence “la femme a traversé la rue” have the same meaning should be able to learn a representation with better semantic understanding.

Generative Neural Machine Translation (GNMT)

With Generative Neural Machine Translation (GNMT)¹, we use a single shared latent representation to model the same sentence in multiple languages. The latent variable is a language agnostic representation of the sentence; by giving it the responsiblity for modelling the same sentence in multiple languages, it is encouraged to learn the semantic meaning.

For each data point \(n\) in a data set, typical latent variable architectures model the joint distribution of the latent representation \(\mathbf{z}^{(n)}\) and the observed sentence \(\mathbf{x}^{(n)}\) as:

\[p(\mathbf{z}^{(n)},\mathbf{x}^{(n)}) = p(\mathbf{z}^{(n)}) p_{\theta}(\mathbf{x}^{(n)}\vert \mathbf{z}^{(n)})\]

where \(\theta\) are trainable parameters of the model.

Instead of modelling a single sentence per latent representation, GNMT uses a shared latent representation to model the same sentence both in the source and target languages. GNMT models the joint distribution of the latent representation, source sentence \(\mathbf{x}^{(n)}\) and target sentence \(\mathbf{y}^{(n)}\) as:

\[p(\mathbf{z}^{(n)},\mathbf{x}^{(n)},\mathbf{y}^{(n)}) = p(\mathbf{z}^{(n)}) p_{\theta}(\mathbf{x}^{(n)}\vert \mathbf{z}^{(n)}) p_{\theta}(\mathbf{y}^{(n)}\vert \mathbf{z}^{(n)},\mathbf{x}^{(n)})\]

This set up means that \(\mathbf{z}^{(n)}\) models the commonality between the source and target sentences, which is the semantic meaning.

For full details on the neural networks used to model the distributions of the source \(p_{\theta}(\mathbf{x}^{(n)}\vert\mathbf{z}^{(n)})\) and target \(p_{\theta}(\mathbf{y}^{(n)}\vert \mathbf{z}^{(n)},\mathbf{x}^{(n)})\), see ¹.

One may argue that it would be better if the target sentence were dependent only on the latent representation and not directly on the source sentence, giving the model \(p(\mathbf{x}\vert \mathbf{z})p(\mathbf{y}\vert \mathbf{z})p(\mathbf{z})\) – forcing the latent representation to be fully responsible for generating both the source and target sentence. However, we found that the generated translations weren’t sufficiently syntactically coherent.

Training the model

We use the Stochastic Gradient Variational Bayes (SGVB)²³ algorithm to train the model described above. SGVB introduces a ‘recognition’ model \(q_{\phi}(\mathbf{z}^{(n)}\vert \mathbf{x}^{(n)})\) which acts as an approximation to the true but intractable posterior \(p(\mathbf{z}^{(n)}\vert \mathbf{x}^{(n)})\), thus forming the following lower bound on the log likelihood of the observed data:

\[\mathcal{L}(\mathbf{x}^{(n)}) = \mathbb{E}_{q_{\phi}(\mathbf{z}^{(n)}\vert \mathbf{x}^{(n)})} [\log p(\mathbf{z}^{(n)},\mathbf{x}^{(n)}) - \log q_{\phi}(\mathbf{z}^{(n)}\vert \mathbf{x}^{(n)})]\]

The model parameters \(\theta\) and recognition parameters \(\phi\) are then jointly learned by performing gradient ascent on this lower bound.

Generating translations - the ‘banana trick’

Suppose the model has been trained, and then we are given a sentence in the source language (\(\mathbf{x}\)) and asked to find a translation of that sentence. In this scenario, we want to find the most likely target sentence conditioned on the given source sentence. The natural objective is therefore:

\[\mathbf{y}^{*} = \arg \max_{y} p(\mathbf{y}\vert \mathbf{x}) = \arg \max_{y} \int p_{\theta}(\mathbf{y}\vert \mathbf{z},\mathbf{x}) p(\mathbf{z}\vert \mathbf{x}) d\mathbf{z}\]

However this integral is intractable, and so we cannot perform this maximisation exactly. Instead, we perform approximate maximisation by iteratively refining a ‘guess’ for the target sentence. We first make a random guess for the target sentence, and then iterate between the following two steps:

Draw samples of the latent representation from the approximate posterior, using the source sentence and the latest guess for the target sentence.
Update the guess for the target sentence based on the latent representation samples from step 1. This update is done by choosing \(\mathbf{y}\) to maximise \(p_{\theta}(\mathbf{y}\vert \mathbf{z},\mathbf{x})\).

Intuitively, this procedure computes the values of the latent representation that are likely to have generated both the source sentence and the latest guess for the target sentence. It then improves the guess based on those values of the latent variable. Mathematically, this iteratively increases a lower bound on \(\log p(\mathbf{y}\vert \mathbf{x})\) until convergence and is an application of the Expectation Maximisation algorithm, here \(\mathbf{y}\) playing the role of parameters.

Note: in our group we refer to this as the ‘banana trick’ because we are first aware of its usage in exercise 5.7 in ⁴, discussing a fictitious protein sequence in bananas.

Below, we show an example of a long sentence translated from French to English by GNMT. The long range coherence of the translation is a good indicator of the model’s ability to capture semantic information about the sentence within the latent representation.

Source: Dans ce décret, il met en lumière les principales réalisations de la République d’Ouzbékistan dans le domaine de la protection et de la promotion des droits de l’homme et approuve le programme d’activités marquant le soixantième anniversaire de la déclaration universelle des droits de l’homme.

Target: The decree highlights major achievements by the Republic of Uzbekistan in the field of protection and promotion of human rights and approves the programme of activities devoted to the sixtieth anniversary of the universal declaration of human rights.

GNMT: In this decree, it highlights the main achievements of the Republic of Uzbekistan on the protection and promotion of human rights and approves the activities of the sixtieth anniversary of the universal declaration of human rights.

Dealing with missing words

Because GNMT’s latent representation captures information about the meaning of the sentence rather than just the syntax, it is able to produce good translations even when there are missing words in the source sentence. The procedure for generating translations is similar to that described above – however in this scenario we also have to refine a guess for the missing words in the source sentence.

We first make random guesses for the missing words in the source sentence and for the target sentence, and then iterate between the following three steps:

Draw samples of the latent representation from the approximate posterior, using the latest guesses for the source and target sentences.
Update the guess for the source sentence based on the latent representation samples from step 1. This update is done by choosing \(\mathbf{x}\) to maximise \(p_{\theta}(\mathbf{x}\vert \mathbf{z})\).
Update the guess for the target sentence based on the latent representation samples from step 1 and on the updated guess for the source sentence from step 2. This update is done by choosing \(\mathbf{y}\) to maximise \(p_{\theta}(\mathbf{y}\vert \mathbf{z},\mathbf{x})\).

Below is an example of a sentence translated from Spanish to English, where the struck through words in the source sentence are considered missing. Using its latent representation, the model does remarkably well at imputing what the missing words may be and translating them accordingly.

Source: Expresando su ~~satisfacción~~ por la asistencia que han ~~prestado~~ a los territorios no autónomos algunos ~~organismos~~ especializados y ~~otras~~ organizaciones del sistema de las naciones ~~unidas~~, especialmente el ~~programa~~ de las naciones unidas ~~para~~ el desarrollo.

Target: Welcoming the assistance extended to non-self-governing territories by certain specialized agencies and other organizations of the United Nations system, in particular the United Nations development programme.

GNMT: Expressing its gratitude for the assistance given to non-self-governing territories by some specialized agencies and other organizations of the United Nations system, in particular from the development programmes of the United Nations.

With the architecture described above, if we wanted to translate between, say, English (EN), Spanish (ES) and French (FR), we would have to train 6 separate models for EN → ES, ES → EN, EN → FR, etc. However because all three of these languages share somewhat similar structures, we may not lose much performance by sharing parameters. We therefore add two indicator variables to the model, one for the input language (\(l_{x}\)) and another for the output language (\(l_{y}\)). By doing this, we only have to train a single model to translate between three languages, instead of having 6 separate models.

The joint distribution of the latent representation, source sentence and target sentence becomes:

\[p(\mathbf{z}^{(n)},\mathbf{x}^{(n)},\mathbf{y}^{(n)}\vert l_{x},l_{y}) = p(\mathbf{z}^{(n)}) p_{\theta}(\mathbf{x}^{(n)}\vert \mathbf{z}^{(n)},l_{x}) p_{\theta}(\mathbf{y}^{(n)}\vert \mathbf{z}^{(n)},\mathbf{x}^{(n)},l_{x},l_{y})\]

We refer to this version of the model as GNMT-Multi.

Overfitting is a phenomenon whereby a model is too closely fit to a particular set of data points. In machine translation, this often occurs when there aren’t enough paired sentences for the model to learn from. However, the cross language parameter sharing used for GNMT-Multi helps to mitigate this issue. This is because 6 separate models are essentially condensed into a single model, meaning that there aren’t enough parameters to allow the model to memorise the training data.

Below, we plot the BLEU scores comparing GNMT-Multi against 6 separate GNMT models, trained with only 400,000 pairs of translated sentences. BLEU is a measure of how well the generated translations match the true target sentences; higher is better. Clearly GNMT-Multi with a limited amount of available training data performs significantly better than each of the 6 separate GNMT models.

When there is a large amount of training data available, we find that there is very little difference in the performance of GNMT and GNMT-Multi. Below, we plot the BLEU scores for the models trained with 4,000,000 sentence pairs; we find that there is no degradation in performance due to sharing parameters across languages!

Semi-supervised learning

Suppose, as in the previous section, that we only have access to a limited number of paired sentences, but that we now have available lots of untranslated sentences in each language. To learn from untranslated sentences, we can set the input language \(l_{x}\) and output language \(l_{y}\) to the same value, so that the model learns to reconstruct the sentence instead of translating it. Intuitively, this should further help the model to learn the structure and style of each language and thus produce more coherent translations at test time. We refer to this version of the model as GNMT-Multi-SSL.

Below we plot the BLEU scores of GNMT-Multi-SSL, GNMT-Multi and the 6 separate GNMT models. They are trained first with 400,000 then with 4,000,000 pairs of translated sentences. In both cases, they are also trained with approximately 20,900,000 untranslated English sentences, 2,700,000 untranslated Spanish sentences and 4,500,000 untranslated French sentences. GNMT-Multi-SSL clearly helps to mitigate overfitting when there are limited paired sentences available. In fact, GNMT-Multi-SSL trained with only 400,000 paired sentences performs about as well as each of the 6 separate GNMT models trained with 4,000,000 sentences! GNMT-Multi-SSL also produces higher BLEU scores even when there is lots of paired data; this verifies our intuition that adding monolingual data helps the model to develop a better understanding of each language individually, and output more coherent sentences accordingly.

Summary

We introduce Generative Neural Machine Translation (GNMT), which is a latent variable model that uses sentences with the same meaning in multiple languages to learn representations which better understand the semantics of the text. It can be used to translate a source sentence by iteratively refining a guess for the target sentence and updating the latent representation accordingly. Because it captures the meaning of the sentence, GNMT is particularly effective at producing translations when there are missing words in the source sentence. We also introduce GNMT-Multi, which is a single unified model (instead of one per language pair) to mitigate overfitting when there is limited paired data available. Finally, we leverage large amounts of untranslated sentences to help the model to further learn the structure and style of each language and produce more coherent translations.

References

H. Shah and D. Barber. Generative Neural Machine Translation. In Advances in Neural Information Processing Systems, 2018. ↩ ↩² ↩³
D. Kingma and M. Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations, 2014. ↩
D. Rezende et al. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In Proceedings of the 31st International Conference on Machine Learning, PMLR 32, pages 1278–1286, 2014. ↩
D. Barber. Bayesian Reasoning and Machine Learning. Cambridge University Press, 2012. ↩

Learning From Scratch by Thinking Fast and Slow with Deep Learning and Tree Search

Tue, 07 Nov 2017 00:00:00 +0000

Training powerful reinforcement learning agents from scratch by Thinking Fast and Slow.

Dual Process Theory

According to dual process theory human reasoning consists of two different kinds of thinking. System 1 is a fast, unconscious and automatic mode of thought, also known as intuition. System 2 is a slow, conscious, explicit and rule-based mode of reasoning that is believed to be an evolutionarily recent process.

image credit

When learning to complete a challenging planning task, such as playing a board game, humans exploit both processes: strong intuitions allow for more effective analytic reasoning by rapidly selecting interesting lines of play for consideration. Repeated deep study gradually improves intuitions. Stronger intuitions feedback to stronger analysis, creating a closed learning loop. In other words, humans learn by thinking fast and slow¹.

What’s wrong with current Deep RL?

In current Deep Reinforcement Learning (RL) algorithms such as Policy Gradients² and DQN³, neural networks make action selections with no lookahead; this is analogous to System 1. Unlike human intuition, their training does not benefit from a ‘System 2’ to suggest strong policies.

A criticism of some AI algorithms such as AlphaGo⁴ is that they use a database of human expert play⁴. In the initial phase of training the RL agent mimics the moves of a human expert – only after this initial phase does it begin to learn potentially more powerful super-human play. This is somewhat unsatisfactory since the resulting algorithm may be heavily biased toward a human style of playing, blind to potentially more powerful lines of play. Whilst, in areas such as game playing, it may be natural to assume that there will be a database of human expert play available, in other settings in which we wish to train an AI machine, no such database may be available. Therefore, showing how to train a state-of-the-art board game player ex nihilo is a major challenge for AI.

Expert Iteration (ExIt)

Expert Iteration⁵ (ExIt) is a general framework for learning that we introduced in May 2017 and can result in powerful AI machines, without needing to mimic human strategies.

ExIt can be viewed as an extension of Imitation Learning (IL) methods to domains where the best known experts are unable to achieve satisfactory performance. In standard IL an apprentice is trained to imitate the behaviour of an expert. In ExIt, we extend this to an iterative learning process. Between each iteration, we perform an Expert Improvement step, where we bootstrap the (fast) apprentice policy to increase the performance of the (comparatively slow) expert.

To give some intuition around this idea, consider playing a board game such as chess. Here the expert is analogous to chess player playing on slow time controls (having lots of time to decide on her move), and the apprentice is playing on blitz time controls (having little time to decide which move to make).

During independent study, the player considers multiple possible moves from a position, thinking deeply (and slowly) about each possible move. She discovers which moves are successful and which are not in this position. When she encounters a similar board state in the future, her study will have given her an intuitive understanding of what moves are likely to be good, allowing her to play well, even under blitz time controls. Her intuition is imitating the strong play she calculated via deep thinking. Humans do not become become excellent chess players by only playing blitz matches, deeper study is an essential part of the learning process.

For an AI game playing machine, this imitation could be achieved, for example, by fitting a neural network to the move made by another `machine expert’ from a game position. The apprentice learns a fast policy that is able to quickly imitate the play of the expert on the moves seen so far. A key point here is that, assuming that there is structure underlying the game, Machine Learning enables the apprentice to generalise their intuition to take quick decisions on positions not previously seen. That is, the apprentice isn’t just a creating a look-up-table of moves made by the human from a fixed database of positions. The neural network thus plays the role of both generalising and imitating the play of the expert.

Now that the apprentice has learned a fast imitation of the expert (on the moves seen so far), it can try to be of use to the expert. When the expert now wishes to make a move, a small set of candidate moves are suggested very quickly by the apprentice which the expert can then consider in depth, possibly also guided during this slow thought process by other quick insights from the apprentice.

At the end of this phase, the expert will have made a set of apprentice-aided moves, with each move being typically much stronger than either the apprentice or expert could have made alone.

The above process now repeats, with the apprentice retraining on the moves suggested by the expert. This completes one full iteration of the learning phase and we iterate this process until the apprentice converges.

From a Dual Process perspective, the Imitation Learning step is analogous to a human improving their intuition for the task by studying example problems, while the Expert Improvement step is analogous to a human using their improved intuition to guide future analysis.

Tree Search and Deep Learning

Exit is a general strategy for learning and the apprentice and expert can be specified in a variety of ways. In board games Monte Carlo Tree Search (MCTS) is a strong playing strategy⁶ and is a natural candidate to play the role of the expert. Deep Learning has been shown to be a successful method to imitate the play of strong players⁴ which we therefore use as the apprentice.

At the Expert Improvement phase we use the apprentice to direct the MCTS algorithm toward promising moves, effectively reducing the game tree search breadth and depth. In this way, we bootstrap the knowledge acquired by Imitation Learning back into the planning algorithm.

The board game Hex

Hex is a classic two-player board game played on a \(n\times n\) hexagonal grid. The players, denoted by colours black and white, alternate placing stones of their colour in empty cells. The black player wins if there is a sequence of adjacent black stones connecting the North edge of the board to the South edge. White wins if he achieves a sequence of adjacent white stones running from the West edge to the East edge.

An example game on a \(5\times 5\) Hex board.

The above represents play on a \(5\times 5\) board, with white winning (reproduced from ⁷). Hex has deep strategy, making it challenging for machines to play and its large action set and connection-based rules means it shares similar challenges for AI to Go. Compared to Go, however, the rules are simpler and there can be no draws.

Because the rules of Hex are so simple, the game is relatively amenable to mathematical analysis (compared for example to Go) and the current best machine player MoHex⁷ uses a combination of MCTS and smart mathematical insights. MoHex has won every Computer Games Olympiad Hex tournament since 2009. It is noteworthy, that MoHex uses a rollout policy trained on datasets of human expert play.

We wanted to see if we can use our ExIt training strategy to learn an AI player than can outperform MoHex, without using any game-specific knowledge or human example play (beside the rules of the game). To do this, our expert is a MCTS player that is guided by the apprentice neural network. Our neural network is a form of deep convolutional network with two output policies – ones for black play and one for white (see ⁵ for details).

Expert Improvement is achieved by using the modified MCTS formula⁸

\[UCT(s,a) + w \frac{\hat{\pi}(a|s)}{n(s,a)+1} \tag{1}\label{eq:uct}\]

Here \(s\) is the state of the Hex board, \(a\) is a possible action (i.e. move) from \(s\). The term \(UCT(s,a)\) represents the classical Upper Confidence Bound for Trees⁶ used in MCTS. The additional term helps the neural network apprentice guide the search to more promising moves. In this term \(\hat{\pi}\) is the policy (suggested relative strength of each possible action \(a\) from the board state \(s\)) of the apprentice and \(n(s,a)\) the number of visits currently made by the search algorithm through state \(s\) and taking action \(a\); \(w\) is an empirically chosen weighting factor that balances the slow thinking of the expert with the fast intuition of the apprentice. Through the additional term, the neural network apprentice guides the search to more promising moves, and rejects weak moves more quickly.

To generate the data for training the apprentice (during each Imitation Learning phase), the batch approach generates data afresh, discarding all data from previous iterations. We also consider a online version in which we instead keep a running buffer of the most recent moves generated; we further consider an online version that retains all data, but with exponentially more data from more recent experts (which correspond to the strongest play). A comparison of these different approaches is given below in which we compare the strength (measured in terms of the ELO score) of each learned policy network against a measure of training time.

We also show the result of using a more traditional Reinforcement Learning approach in which a policy \(\hat{\pi}(a\vert s)\) is learned only through self play (i.e. no MCTS). This is essentially the method used within AlphaGo⁴ to train their policy network. The figure shows that the ExIt training approach is considerably more effective than more classical approaches. It is worth noting that in this example training has not yet fully converged and the apprentice would be expected to improve in ability further given additional training time.

In our paper⁵ we include an additional mechanism to improve play, namely a value network \(V^{\hat{\pi}}(s)\) that approximates the probability that the apprentice (alone) would win the game from position \(s\). Both the policy and value network are then used in combination to help guide the final apprentice-aided MCTS player. The policy network and value networks guide the final MCTS player using an equation similar \eqref{eq:uct}, but modified to include the apprentice’s value of the state \(s\) (see ⁵ for details).

Our final MCTS player outperforms the best known machine Hex player, MoHex, beating it in 75% of games played on a \(9\times 9\) board. These results are even more remarkable considering that training has not fully converged. A couple of examples of game-play of our ExIt trained player versus the state-of-the-art MoHex player are shown below⁹. We contrast the play of each algorithm when started at the same position. See the paper⁵ for more examples.

ExIt (black) versus MoHex (white)

MoHex (black) versus ExIt (white)

Why does ExIt work so well?

Imitation Learning is generally appreciated to be easier than Reinforcement Learning, and this partly explains why ExIt is more successful than model-free methods like REINFORCE.

Furthermore, for MCTS to recommend a move, it must be unable to find any weakness with its search. Effectively, therefore, a move played by MCTS is good against a large selection of possible opponents. In contrast, in regular self play (in which the opponent move is made by the network playing as the opposite colour), moves are recommended if they beat only this single opponent under consideration. This is, we believe, a key insight into why ExIt works well (when using MCTS as the expert) — the apprentice effectively learns to play well against many opponents.

Relation to AlphaGo Zero

AlphaGo Zero¹⁰ (developed independently of our work¹¹) also implements an ExIt style algorithm and shows that it is possible to achieve state-of-the-art performance in Go without the use of human expert play. A detailed comparison of the approaches is given in our paper⁵.

Summary

Expert Iteration is a new Reinforcement Learning algorithm, motivated by the dual process theory of human thought. ExIt decomposes the Reinforcement Learning into the separate subproblems of generalisation and planning. Planning is performed on a case-by-case basis, and only once a strong plan is found is the resultant policy generalised. This allows for long-term planning and results in faster learning and state-of-the-art final performance, particularly for challenging problems. This training strategy is powerful enough to learn state-of-the-art board game AI players without requiring any examples of expert human play.

References

D. Kahneman. Thinking, Fast and Slow. Macmillan, 2011. ↩
R. J. Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning, 8(3-4):229–256, 1992. ↩
V. Mnih et al. Human-Level Control through Deep Reinforcement Learning. Nature, 518(7540):529–533, 2015. ↩
D. Silver et al. Mastering the Game of Go with Deep Neural Networks and Tree Search. Nature, 529(7587):484–489, 2016. ↩ ↩² ↩³ ↩⁴
T. Anthony, Z. Tian and D. Barber. Thinking Fast and Slow with Deep Learning and Tree Search, Neural Information Processing Systems (NIPS 2017). In Press. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶
L. Kocsis and C. Szepesvári. Bandit Based Monte-Carlo Planning. In European Conference on Machine Learning, pages 282–293. Springer, 2006. ↩ ↩²
S.-C. Huang, B. Arneson, R. Hayward, M. Müller, and J. Pawlewicz. MoHex 2.0: A Pattern-Based MCTS Hex Player. In International Conference on Computers and Games, pages 60–71. Springer, 2013. ↩ ↩²
S. Gelly and D. Silver. Combining Online and Offline Knowledge in UCT. In Proceedings of the 24th International Conference on Machine learning, pages 273–280. ACM, 2007. ↩
Thanks to Ryan Hayward for providing a tool to draw Hex positions. ↩
D. Silver, et al. Mastering the game of Go without human knowledge. Nature 550:354–359, October 2017. ↩
T. Anthony, Z. Tian and D. Barber. Thinking Fast and Slow with Deep Learning and Tree Search. arXiv CoRR:abs/1705.08439, May 2017. ↩

Some modest insights into the error surface of Neural Nets

Sun, 30 Jul 2017 00:00:00 +0000

Did you know that feedforward Neural Nets (with piecewise linear transfer functions) have no smooth local maxima?

In our recent ICML paper Practical Gauss-Newton Optimisation for Deep Learning) we discuss a second order method that can be applied successfully to accelerate training of Neural Networks. However, here I want to discuss some of the fairly straightforward, but perhaps interesting, insights into the geometry of the error surface that that work gives.

\(\newcommand{\br}[1]{\left(#1\right)}\) \(\newcommand{\sq}[1]{\left[#1\right]}\) \(\newcommand{\ave}[1]{\mathbb{E}\sq{#1}}\)

Feedforward Neural Networks

In our description, a feedforward NN takes an input vector \(x\) and produces a vector \(h_L\) on the final \(L^{th}\) layer. We write \(h_\lambda\) to be the vector of pre-activation values for layer \(\lambda\) and \(a_\lambda\) to denote the vector of activation values after passing through the transfer function \(f_\lambda\).

Starting with setting \(a_0\) to the input \(x\), a feedforward NN is defined by the recursion

\[h_\lambda = W_\lambda a_{\lambda-1}\]

where \(W_\lambda\) is the weight matrix of layer \(\lambda\) (we use a sub or superscript \(\lambda\) wherever most convenient) and the activation vector is given by

\[a_\lambda = f_\lambda(h_\lambda)\]

We define a loss \(E(h_L,y)\) between the final output layer \(h_L\) and a desired training output \(y\). For example, we might use a squared loss

\[E(h_L,y) = (h_L-y)^2\]

where the loss is summed over all elements of the vector. For a training dataset the total error function is the summed loss over individual training points

\[\bar{E}(\theta) = \sum_{n=1}^N E(h_L(n),y(n))\]

where \(\theta\) represents the stacked vector of all parameters of the network. For simplicity we will write \(E(\theta)\) for the error for a single generic datapoint.

The Gradient

For training a NN, a key quantity is the gradient of the error

\[g_{i} = \frac{\partial}{\partial\theta_i} E(\theta)\]

We use this for example in gradient descent training algorithms. An important issue is how to compute the gradient efficiently. Thanks to the layered structure of the network, it’s intuitive that there is an efficient scheme (backprop which is a special case of Reverse Mode AutoDiff) that propagates information from layer to layer.

The Hessian

One aspect of the structure of the error surface is the local curvature, defined by the Hessian matrix with elements

\[H_{ij} = \frac{\partial^2}{\partial\theta_i\partial\theta_j} E(\theta)\]

The Hessian matrix itself is typically very large. To make this more manageable, we’ll focus here on the Hessian of the parameters of a given layer \(\lambda\). That is

\[[H_\lambda]_{(a,b),(c,d)} = \frac{\partial^2 E}{\partial W^\lambda_{a,b}\partial W^\lambda_{c,d}}\]

The Hessians \(H_\lambda\) then form the diagonal block matrices of the full Hessian \(H\).

A recursion for the Hessian

Similar to the gradient, it’s perhaps intuitive that a recursion exists to calculate this layerwise Hessian. Starting from

\[\frac{\partial E}{\partial W^\lambda_{a,b}}=\sum_i \frac{\partial h^\lambda_i}{W^\lambda_{a,b}}\frac{\partial E}{\partial h^\lambda_i} = a^{\lambda-1}_b\frac{\partial E}{\partial h^\lambda_a}\]

and differentiating again we obtain

\[[H_\lambda]_{(a,b),(c,d)} = a^{\lambda-1}_b a^{\lambda-1}_d [{\cal{H}}_\lambda]_{a,c} \tag{1}\label{eq:H}\]

where we define the pre-activation Hessian for layer \(\lambda\) as

\[[{\cal{H}}_\lambda]_{a,c} = \frac{\partial^2 E}{\partial h^\lambda_a\partial h^\lambda_c}\]

We show in Practical Gauss-Newton Optimisation for Deep Learning that one can derive a simple backwards recursion for this pre-activation Hessian (the recursion is for a single datapoint – the total Hessian \(\bar{H}\) is a sum over the individual datapoint Hessians):

\[{\cal{H}}_\lambda = B_\lambda W_{\lambda+1}^\top {\cal{H}}_{\lambda+1}W_{\lambda+1}B_{\lambda}+D_\lambda \tag{2}\label{eq:recursion}\]

where we define the diagonal matrices

\[B_\lambda = \text{diag}(f'_\lambda(h_\lambda))\]

and

\[D_\lambda = \text{diag}\br{f''_\lambda(h_\lambda)\frac{\partial E}{\partial a_\lambda}}\]

Here \(f'\) is the first derivative of the transfer function and \(f''\) is the second derivative.

The recursion is initialised with \({\cal{H}}_L\) which depends on the objective \(E(h_L,y)\) and is easily calculated for the usual loss functions. For example, for the square loss \((y-h_L)^2/2\) we have \({\cal{H}}_L=I\), namely the identity matrix. We use this recursion in our paper to build an approximate Gauss-Newton optimisation method.

Consequences

Piecewise linear transfer functions, such as the ReLU \(f(x) = \max(x,0)\) are currently popular due to both their speed of evaluation (compared to more traditional transfer functions such as \(\tanh(x)\)) and also the empirical observation that, under gradient based training, they tend to get trapped less often in local optima. Note that if the transfer functions are piecewise linear, this does not necessarily mean that the objective will be piecewise linear (since the loss is usually itself not piecewise linear).

For a piecewise linear transfer function, apart from the `nodes’ where the linear sections meet, the function is differentiable and has zero second derivative, \(f''(x)=0\). This means that the matrices \(D_\lambda\) in the above Hessian recursion will be zero (away from nodes).

For many common loss functions, such as squared loss (for regression) and cross entropy loss (for classification) the Hessian \({\cal{H}}_L\) is Positive Semi-Definite (PSD).

Note that, according to \eqref{eq:recursion}, for transfer functions that contain zero gradient points \(f'(x)=0\) then the Hessian \(H_\lambda\) can have lower rank than \(H_{\lambda+1}\), reducing the curvature information propagating back from layers close to the output towards layers closer to the input. This has the effect of creating flat plateaus in the surface and makes gradient based training potentially more problematic. Conversely, provided the gradient of the transfer function is never zero \(f'\neq 0\), then according to \eqref{eq:recursion} each layer pre-activation Hessian is Positive Definite, helping preserve the propagation of surface curvature back through the network.

Structure within a layer

For such loss functions, it follows that the pre-activation Hessian \({\cal{H}}_\lambda\) for all layers is PSD as well (away from nodes). It immediately follows from \eqref{eq:H} that the Hessian \(H_\lambda\) for each layer \(\lambda\) is PSD. This means that, if we fix all the parameters of the network, and vary only the parameters in a layer \(W^\lambda\), then the objective \(E\) can exhibit no smooth local maxima or smooth saddle points. Note that this does not imply that the objective is convex everywhere with respect to \(W_\lambda\) as the surface will contain ridges corresponding to the non-differentiable nodes.

No differentiable local maxima

The trace of the full Hessian \(H\) is the sum of the traces of each of the layerwise blocks \(H_\lambda\). Since (as usual away from nodes) by the above argument each matrix \(H_\lambda\) is PSD, it follows that the trace of the full Hessian is non-negative. This means that it is not possible for all eigenvalues of the Hessian to be simultaneously negative, with the immediate consequence that feedforward networks (with piecewise linear transfer functions) have no differentiable local maxima. The picture below illustrates the kind of situtation therefore that can happen in terms of local maxima (test):

whereas the image below depicts the kind of smooth local maxima that cannot happen:

Visualisation for a simple two layer net

We consider a simple network with two layers, ReLU transfer functions and square loss error. The network thus has two weight matrices \(W^1\) and \(W^2\). Below we choose two fixed matrices \(U\) and \(V\) and parameterise the weight matrix \(W^1\) as a function of two scalars \(u\) and \(v\), so that \(W^1(u,v)=uU + vV\). As we vary \(u\) and \(v\) we then plot the objective function \(E(u,v)\), keeping all other parameters of the network fixed.

As we can see the surface contains no local differentiable local maxima as we vary the parameters in the layer.

Below we show an analogous plot for varying the parameters of the second layer weights \(W^2(u,v)\), which has the same predicted property that there are no differentiable local maxima.

Finally, below we plot \(E(u,v)\) using \(W^1=uU\) and \(W^2=vV\), showing how the objective function changes as we simultaneously change the parameters in different layers. As we can see, there are no differentiable maxima.

Summary

A simple consequence of using piecewise linear transfer functions and a convex loss, is that feedforward networks cannot have any differentiable maxima (or saddle points) as parameters are varied within a layer. Furthermore, the objective cannot contain any differentiable maxima, even as we vary parameters across layers. Note that the objective \(E(u,v)\) though can (and empirically does) have smooth saddle points as one varies parameters \(u\) and \(v\) across different layers.

It’s unclear how practically significant these modest insights are. However, they do potentially partially support the use of piecewise linear transfer functions (particularly those with no zero gradient regions) since for such transfer functions gradient based training algorithms cannot easily dawdle on local maxima (anywhere), or idle around saddle points (within a layer) since such regions correspond to sharp slopes in the objective.

These results are part of a more detailed study of second order methods for optimisation in feedforward Neural Nets which will appear in ICML 2017.

Evolutionary Optimization as a Variational Method

Mon, 03 Apr 2017 00:00:00 +0000

A simple connection between evolutionary optimisation and variational methods.

\(\newcommand{\sq}[1]{\left[#1\right]}\) \(\newcommand{\ave}[1]{\mathbb{E}\sq{#1}}\)

Variational Optimization

Variational Optimization is based on the bound

\[min_x f(x) \leq \ave{f(x)}_{p(x|\theta)}\]

That is, the minimum of a collection of values is always less than their average. By defining

\[U(\theta) = \ave{f(x)}_{p(x|\theta)}\]

instead of minimising \(f\) with respect to \(x\), we can minimise the upper bound \(U\) with respect to \(\theta\). Provided the distribution \(p(x\vert \theta)\) is rich enough, this will be equivalent to minimising \(f(x)\).

The gradient of the upper bound is then given by

\[\frac{\partial U}{\partial \theta} = \ave{f(x)\frac{\partial}{\partial \theta}\log p(x|\theta)}_{p(x|\theta)}\]

which is reminiscent of the REINFORCE (Williams 1992) policy gradient approach in Reinforcement Learning.

In the original VO report and paper this idea was used to form a differentiable upper bound for non-differentiable \(f\) and also discrete \(x\).

Sampling Approximation

There is an interesting connection to evolutionary computation (more precisely Estimation of Distribution Algorithms) if the expectation with respect to \(p(x\vert \theta)\) is performed using sampling. In this case one can draw samples \(x^1,\ldots,x^S\) from \(p(x\vert\theta)\) and form an unbiased approximation to the upper bound gradient

\[\frac{\partial U}{\partial \theta} \approx \frac{1}{S} \sum_{s}f(x^s)\frac{\partial}{\partial \theta}\log p(x^s|\theta)\]

The “evolutionary” connection is that the samples \(x^s\) can be thought of as “particles” or “swarm members” that are used to estimate the gradient. Based on the approximate gradient, simple Stochastic Gradient Descent (SGD) would then perform the parameter update (for learning rate \(\eta\))

\[\theta^{new} = \theta-\frac{\eta}{S} \sum_{s}f(x^s)\frac{\partial}{\partial \theta}\log p(x^s|\theta)\]

The “swarm” then disperses and draws a new set of members from \(p(x\vert \theta^{new})\) and the process repeats.

A special case of VO is to use a Gaussian so that (for the scalar case – the multivariate setting follows similarly)

\[U(\theta) = \frac{1}{\sqrt{2\pi\sigma^2}}\int e^{-\frac{1}{2\sigma^2}(x-\theta)^2}f(x)dx\]

Then the gradient of this upper bound is given by

\[U'(\theta) = \frac{1}{\sigma^2}\ave{(x-\theta)f(x)}_{x\sim N(\theta,\sigma^2)}\]

By changing variable \(\epsilon=x-\theta\) this is equivalent to

\[U'(\theta) = \frac{1}{\sigma^2}\ave{\epsilon f(\theta+\epsilon)}_{\epsilon \sim N(0,\sigma^2)} \label{eq:grad}\tag{1}\]

Fixing \(\sigma=5\) and using \(S=10\) samples, we show below the trajectory (for 150 steps of SGD with fixed learning rate \(\eta=0.1\)) of \(\theta\) based on Stochastic VO and compare this to the underlying function \(f(x)\) (which in this case is a simple quadratic). Note that we only plot below the parameter \(\theta\) trajectory (each red dot represents a parameter \(\theta\), with the initial parameter in the bottom right) and not the samples from \(p(x\vert \theta)\). As we see, despite the noisy gradient estimate, the parameter values \(\theta\) move toward the minimum of the objective \(f(x)\). The matlab code is available if you’d like to play with this.

One can also consider the bound as a function of both the mean \(\theta\) and variance \(\sigma^2\):

\[U(\theta,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}\int e^{-\frac{1}{2\sigma^2}(x-\theta)^2}f(x)dx\]

and minimise the bound with respect to both \(\theta\) and \(\sigma^2\) (which we will parameterise using \(\sigma^2=e^\beta\) to ensure a positive variance). More generally, one can consider parameterising the Gaussian covariance matrix for example using factor analysis and minimsing the bound with respect to the factor loadings.

Using a Gaussian with covariance \(e^\beta I\) and performing gradient descent on both \(\beta\) and \(\theta\), for the same objective function, learning rate \(\eta=0.1\) and initial \(\sigma=5\), we obtain the trajectory below for \(\theta\)

As we can see, by learning \(\sigma\), the trajectory is much less noisy and more quickly homes in on the optimum. The trajectory of the learned standard deviation \(\sigma\) is given below, showing how the variance reduces as we home in on the optimum.

In the context of more general optimisation problems (such as in deep learning and reinforcement learning), VO is potentially interesting since the sampling process can be distributed across different machines.

Gradient Approximation by Gaussian Perturbation

Ferenc Huszar‏ has a nice post Evolution Strategies: Almost Embarrassingly Parallel Optimization summarising recent work by Salimans etal on Evolution Strategies as a Scalable Alternative to Reinforcement Learning.

The aim is to minimise a function \(f(x)\) by using gradient based approaches, without explicitly calculating the gradient. The first observation is that the gradient can be approximated by considering the Taylor expansion

\[f(x+\epsilon) = f(x)+\epsilon f'(x) + \frac{\epsilon^2}{2} f''(x) + O(\epsilon^3)\]

Multiplying both sides by \(\epsilon\)

\[\epsilon f(x+\epsilon) = \epsilon f(x)+\epsilon^2 f'(x) +\frac{\epsilon^3}{2}f''(x)+ O(\epsilon^4)\]

Finally, taking the expectation with respect to \(\epsilon\) drawn from a Gaussian distribution with zero mean and variance \(\sigma^2\) we have

\[\ave{\epsilon f(x+\epsilon)} = \sigma^2 f'(x) + O(\epsilon^4)\]

Hence, we have the approximation

\[f'(x) \approx \frac{1}{\sigma^2}\ave{\epsilon f(x+\epsilon)} \label{eq:grad2}\tag{2}\]

Based on the above discussion of VO, and comparing equations (1) and (2) we see that this Gaussian perturbation approach is related to VO in which we use a Gaussian \(p(x\vert \theta)\), with the understanding that in the VO case the optimisation is over \(\theta\) rather than \(x\). An advantage of the VO approach, however, is that it provides a principled way to adjust parameters such as the variance \(\sigma^2\) (based on minimising the upper bound).

Whilst this was derived for the scalar setting, the vector derivative is obtained by applying the same method, where the \(\epsilon\) vector is drawn from the zero mean multivariate Gaussian with covariance \(\sigma^2 I\) for identity matrix \(I\).

Efficient Communication

A key insight in Evolution Strategies as a Scalable Alternative to Reinforcement Learning is that the sampling process can be distributed across multiple machines, \(i\in\{1,\ldots,S\}\) so that

\[f'(x) \approx \frac{1}{S\sigma^2}\sum_{i=1}^S {\epsilon^i f(x+\epsilon^i)}\]

where \(\epsilon^i\) is a vector sample and \(i\) is the sample index. Each machine \(i\) can then calculate \(f(x+\epsilon^i)\). The Stochastic Gradient parameter update with learning rate \(\eta\) is

\[x^{new} = x - \frac{\eta}{S\sigma^2}\sum_{i=1}^S {\epsilon^i f(x+\epsilon^i)}\]

Provided each machine \(i\) also knows the random seed used to generate the \(\epsilon^j\) of each other machine, it therefore knows what all the \(\epsilon^j\) are (by sampling according to the known seeds) and can thus calculate \(x^{new}\) based on only the \(S\) scalar values calculated by each machine. The basic point here is that, thanks to seed sharing, there is no requirement to send the vectors \(\epsilon^i\) between the machines (only the scalar values \(f(x+\epsilon^i)\) need be sent), keeping the transmission costs very low.

Based on the insight that the Parallel Gaussian Perturbation approach is a special case of VO, it would be natural to apply VO using seed sharing to efficiently parallelise the sampling. This has the benefit that other parameters such as the variance can also be efficiently communicated, potentially significantly speeding up convergence.

Training with a large number of classes

Wed, 15 Mar 2017 00:00:00 +0000

In machine learning we often face the issue of a very large number of classes in a classification problem. This causes a bottleneck in the computation. There’s though a simple and effective way to deal with this.

Probabilistic Classification

In areas like Natural Language Processing (NLP) a common task is to predict the next word in sequence (like in preditictive text on a smartphone or in learning word embeddings). For input \(x\) and class label \(c\), the probability of predicting class \(c\) is

\[p_\theta(c|x) = \frac{u_\theta(c,x)}{Z_\theta(x)}\]

where \(u_\theta(c,x)\) is some defined function with parameters \(\theta\). For example, \(u_\theta(c,x)=\exp(w_c'x)\), where \(w_c\) is a parameter vector for class \(c\) and \(x\) is the vector input. The normalising term is

\[Z_\theta(x) = \sum_{c=1}^C u_\theta(c,x)\]

The task is then to adjust the parameters \(\theta\) to maximise the probability of the correct class for each of the training points.

However, if there are \(C=100,000\) words in the dictionary, this means calculating the normalisation \(Z\) for each datapoint is going to be expensive. There have been a variety of approaches suggested over the years to make computationally efficient approximations, many based on importance sampling.

Why plain Importance Sampling doesn’t work

A standard approach to approximating \(Z_\theta(x)\) is to use

\[Z_\theta(x) = \sum_{c=1}^C q(c) \frac{u_\theta(c,x)}{q(c)}\]

where \(q\) is an importance distribution over all \(C\) classes. We can then form an approximation by sampling from \(q\) a small number \(S\) of classes to form a sample bag \({\cal{S}}\) and using

\[Z_\theta(x) \approx \tilde{Z}_\theta(x) = \frac{1}{S}\sum_{s\in{\cal{S}}} \frac{u_\theta(s,x)}{q(s)}\]

The problem with this approach is that it results in a potentially catastrophic under-estimate of \(Z_\theta(x)\). If the classifier is working well, we want that \(u_\theta(c,x)\) is much higher than \(u_\theta(d,x)\) for any incorrect class \(d\). Hence, unless the importance sample bag \({\cal{S}}\) includes class \(c\), then the normalisation approximation will miss this significant mass and the probability approximation

\[\frac{u_\theta(c,x)}{\tilde{Z}_\theta(x)}\]

will be wildly inaccurate, see figure (a) below. This is the source of the historically well-documented instabilities in training large-scale classifiers.

Making Importance Sampling work

However, there is an easy fix for this – simply ensure that \({\cal{S}}\) includes the correct class \(c\).

On the left above we show for \(C=10,000\) classes the ratio \(u_\theta(c,x)/Z_\theta(x)\) on the \(x\)-axis against its approximation \(u_\theta(c,x)/\tilde{Z}_\theta(x)\) on the \(y\)-axis. Each dot represents a different randomly drawn set of \(u\) values. Red, green and blue represent 10, 20 and 50 importance samples respectively. The ideal estimation would be such that all points are along the line \(y=x\). Note the vertical scale – these values are supposed to be probabilities and lie between 0 and 1. Even as we increase the number of importance samples, this remains a wildly incorrect estimation of the probability.

On the right above we show the same probability estimate but now simply also include the correct class in the set \({\cal{S}}\). The vertical scale is now sensible and the estimated probabiliy is close to the true value.

Deep Learning Recurrent NLP models

We applied this method to learning word embeddings for a deep recurrent network. The training objective was standard maximum likelihood, but with the normalisation approximation above. Below we plot the exact log likelihood (\(y\)-axis) against the optimisation gradient ascent iteration (\(x\)-axis). We also plot the exact log likelihood for some alternative training approaches. As we see, standard Importance Sampling becomes unstable as learning progresses. However our simple modification stabilizes learning and is competitive against a range of alternatives including Noise Contrastive Estimation, Ranking approaches, Negative Sampling and BlackOut.

This is so simple and works so well that we use this in all our NLP deep learning training experiments.

This forms the basis for our paper Complementary Sum Sampling for Likelihood Approximation in Large Scale Classification which will appear in AISTATS 2017.

David Barber

Generative Neural Machine Translation

Learning meaningful representations of data

Generative Neural Machine Translation (GNMT)

Training the model

Generating translations - the ‘banana trick’

Dealing with missing words

Cross-language parameter sharing

Semi-supervised learning

Summary

References

Learning From Scratch by Thinking Fast and Slow with Deep Learning and Tree Search

Dual Process Theory

What’s wrong with current Deep RL?

Expert Iteration (ExIt)

Tree Search and Deep Learning

The board game Hex

Why does ExIt work so well?

Relation to AlphaGo Zero

Summary

References

Some modest insights into the error surface of Neural Nets

Feedforward Neural Networks

The Gradient

The Hessian

A recursion for the Hessian

Consequences

Structure within a layer

No differentiable local maxima

Visualisation for a simple two layer net

Summary

Evolutionary Optimization as a Variational Method

Variational Optimization

Sampling Approximation

Gradient Approximation by Gaussian Perturbation

Efficient Communication

Training with a large number of classes

Probabilistic Classification

Why plain Importance Sampling doesn’t work

Making Importance Sampling work

Deep Learning Recurrent NLP models