Course Notes: Idempotent Productions

 Abstract 

This document provides an introduction to the concepts, methods and technologies covered in this course. The research and engineering disciplines spanned by this introduction are evolving so fast that by this time next year much of the content will be obsolete. Why would anyone bother to learn this material? The answer is that the ideas explored in this course will transform how we think about machine intelligence and make possible the next generation of smart systems. This class will prepare students to contribute to the development of this technology and help shape the way it is employed in society¹.

Contents

    1  Artificial Neural Networks & Machine Learning
        1.1  Basic Terminology
        1.2  Network Components
        1.3  Vectors and Matrices
        1.4  Activation Functions
        1.5  Perceptron Models
        1.6  Connection Matrices
        1.7  Differentiable Models
        1.8  Learning the Weights
        1.9  Convolutional Layers
        1.10  Multilayer Networks
        1.11  Energy Landscapes
        1.12  Accelerating Returns
        1.13  Recurrent Networks
        1.14  Signifying Meaning
        1.15  The History of Ideas
        1.16  Key Ideas Summary

    2  Accelerated Evolution of Artificial Intelligence
        2.1  Technology Predictions
        2.2  Thinking About Thinking
        2.3  Programmer's Apprentice
        2.4  Memory and Computation
        2.5  Cognitive Neuroscience
        2.6  Anatomy and Physiology
        2.7  Models of Human Memory
        2.8  Neural Net Terminology
        2.9  Reinforcement Learning
        2.10  Attentional Networks
        2.11  Machine Memory Models
        2.12  Embodied Cognition
        2.13  Planning and Behaving
        2.14  Language and Thinking
        2.15  The History of Ideas
        2.16  Key Ideas Summary

1  Artificial Neural Networks & Machine Learning

This chapter describes the basic the machine learning and artificial neural network concepts assumed in this class. The class encourages multi-disciplinary teams working on final projects that combine expertise in computer science and neuroscience and so it is not necessary that everyone taking the class has mastered these concepts; however, familiarity with the basic ideas is strongly encouraged, since the concepts presented here will provide a foundation for understanding and applying the more advanced concepts covered in the next chapter.

The human brain is incredibly complicated. The more we learn about it, the more we appreciate the intricacy of its structure and the breadth of its function. Scientists think of it as a network of neurons, but it is so much more complicated than any engineer's model or artist's conception of a neural network as to beggar the imagination. There are hundreds of specialized cell types classified as neurons that employ a bewildering array of chemical, electrical, mechanical and genetic signaling pathways in order to communicate with one another². Hardly a month goes by when there isn't some new experimental finding published in a respected journal that unsettles the prevailing dogma or calls into question some cherished theory³. As an example with profound consequences for how we comprehend and treat brain disorders, there are thought to be as many or more glial cells as there are neurons in the mammalian brain and we've only begun to understand their function — it is not, as was once believed and is still taught in some textbooks, simply a means of structural and housekeeping support for and insulation between neurons⁴.

Artificial neural networks (ANNs) are simple by comparison. The sort of ANNs now an integral part of machine learning were originally inspired by biological networks and continue to exploit new insights from neuroscience, but they are now rapidly evolving as researchers incorporate ideas from computer science and machine learning to improve their performance and reliability. Artificial neural networks provide interesting abstractions useful in studying the computational properties of brains — they were originally conceived for this purpose, but provide little to advance our clinical or scientific understanding of healthy and diseased brains. There is, however, a level of abstraction on which they largely agree, and it is this level of abstraction that has played such an important role in the success of so-called deep neural networks in outperforming humans on tasks once thought to be out of reach for AI technology.

We take a closer look at biological brains in subsequent chapters, but, for the remainder of this chapter, we focus on ANNs, including a nuts-and-bolts introduction to ANN technology to demonstrate just how simple are the basic components, and set the stage for an explanation of how we can assemble these components into powerful architectures that combine ideas from biological and conventional computing. I promise that this is the only chapter in which you will see an equation and there are only a few included here. The objective is to convey to you in as direct a way possible just how simple are the bones of this technology in order to give you a better idea of how neural networks can be compactly described, functionally extended, scaled and adapted to run on commodity desktop computers, commercial cloud computing services, phones and laptops and specialized parallel processing hardware.

---

1.1  Basic Terminology

The terminology surrounding artificial neural networks can be confusing to the uninitiated. Part of the confusion arises due to the seemingly inconsistent uses of the word "layer". The network shown in Figure 1.A is characterized in the text as a single-layer perceptron⁵, whereas the network shown in Figure 1.C is called a multilayer perceptron⁶. The term deep neural network⁷ entered the lexicon much later and is generally used to refer to any artificial neural network with multiple layers between the input and output layer. We only briefly mention single-layer perceptrons in this chapter in order to make the point that these neural networks are less powerful⁸ than multilayer networks⁹ in a specific technical sense that sheds light on what it means to learn.

---

1.2  Network Components

In the following, we describe classes of specialized neural networks that can be used like Lego bricks to build more complicated architectures. We start by looking at three basic artificial neural network components. These three components are illustrated in Figure 1 and are examples of Lego-brick-like networks that can be used to construct more complicated networks. Networks are built of layers that are depicted as rectangular boxes with rounded corners.

A neural network has two or more layers, an input layer, an output layer and zero or more hidden layers. Layers are stacked to construct networks. Each layer is composed of one or more units and selected units in one layer can be linked to selected units in an adjacent layer by connections. The units in adjacent layers and the connections between them constitute a graph in the terminology of graph theory where units are referred to as nodes in the graph and the connections between them are called edges. Graph theory¹⁰ is a sub-discipline of discrete mathematics, and won't figure prominently in our discussion except to characterize different patterns of connectivity. Nodes and edges have been associated with neurons and synapses in biological networks but the analogy can be misleading and is generally discouraged here and elsewhere in the scientific literature.

As a simple example of how graph theory is applied to neural networks, the connections plus the combined units in the two layers define a directed bipartite graph in which connections correspond to the edges in the graph and the units to nodes¹¹. If each unit in the input layer — say there are \({n}\) of them — is connected to every unit in the output layer — say there are \({m}\) of them, then the connections form a complete bipartite graph and there are n × m connections — a number that can become rather large in practice. In our discussion of convolutional neural networks, we consider a powerful technique that ameliorates some of the computational problems that arise from working with very large input or output layers.

---

1.3  Vectors and Matrices

Neural networks represent both mathematical and computational objects. The nodes in a layer correspond to variables in equations describing the mathematical functions that neural networks implement as computable objects. It is often appropriate to think of the variables within a layer as defining vectors in a vector space and the edges as defining linear transformations from one vector space to another¹². A vector space is an \({n}\)-dimensional generalization of the familiar three-dimensional Euclidean space you might have encountered in high-school calculus. A vector represents the coordinates of a point in that space — see Figure 4.A. This abstraction frees us from having to deal with individual units in layers and allows us to bring to bear the full power and beauty of linear algebra¹³ and multivariate calculus¹⁴ in order to share models and exploit powerful software packages and specialized hardware to expedite engineering and accelerate the computations required to develop, train and deploy neural networks in various applications.

From this perspective, each layer represents a sequence — ordered list — of variables that can be used to assign numeric values to the units that comprise a layer. This sequence is called a vector in algebraic matrix theory which is the underlying branch of mathematics associated with neural networks. A vector variable represents a sequence of scalar variables — meaning that each variable corresponds to single real number¹⁵, often referred to as vector components each of which can be assigned a numeric value. We display vector variables in bold font and scalar variables in italics using subscripts to refer to the individual units in a layer / components in the vector. For example, the expression \({\mathbf{u} = \langle{}u_1, u_2, \ldots, u_n\rangle{}}\) defines \({\mathbf{u}}\) to be a vector variable consisting of \({n}\) components shown as subscripted scalar variables delimited by angle brackets. A neural network defines a mathematical function that takes as input a vector of real numbers and produces as output a second vector of real numbers. The term function is used loosely to apply to both the abstract mathematical object and to any computable instantiation of that object corresponding to a program that runs on a conventional computer or specialized hardware designed specifically for neural networks.

In specifying functions represented by a network we often iterate over all the components in a vector using an indexed variable \({u_i}\) as in the expression \({\sum_{i-1}^{n} = u_1 + u_2 + \cdots+ u_n}\) representing the sum of the components of the vector \({\mathbf{u}}\). Algebraic matrix theory allows us to add two vectors component wise as in \({\langle{}1.3, 4.0, 7.5\rangle{} \oplus\langle{}0.5, 1.9, 5.2\rangle{} = \langle{}1.8, 5.9, 12.7\rangle{}}\) or normalize a vector \({\mathbf{u} = \langle{}4.0, 1.0, 5.0\rangle{}}\) by dividing each component by the sum of all the components as in the equation, \({\mathbf{u}\,/\,\sum_{i-1}^{i} u_i}\), which in this case yields the vector \({\langle{}0.4, 0.1, 0.5\rangle{}}\) in which all of the components are between \({0}\) and \({1}\) and together sum to \({1.0}\). Note that this calculation involves three scalar divisions, \({(4.0\,/\,10.0)}\), \({(1.0\,/\,10.0)}\) and \({(5.0\,/\,10.0)}\). These operations can be carried out in parallel given a processor such as the graphics processing unit (GPU) found in most laptops and cell phones.

---

1.4  Activation Functions

In the sequel, we define our neural network building blocks as having two or more layers where the required two correspond to the input and output layers. Where it aids exposition, we depict the individual units within layers and draw the connections between the units in adjacent layers. Otherwise, to render the diagrams less busy graphically, we hide the individual units using just the iconic round-edged rectangles as a visual proxy for the corresponding vectors and draw a single edge between the two layers to represent the linear transformation between the relevant vector spaces. Each layer is also characterized by an activation function — typically nonlinear — that is described in the text but generally not graphically depicted in the network architecture diagrams.

The artificial neural networks discussed here are said to be discrete-time¹⁶ rather than continuous-time dynamical systems¹⁷. The state of an artificial neural network at any point in time corresponds to the values assigned to the variables that constitute the vectors associated with each layer. For our purposes, each tick of the clock corresponds to a pass through the entire network during which all of the variables in the network are updated. Conceptually, it is easiest to think of the input layer as being on the bottom of a stack of layers culminating in the output layer at the top of the stack. A forward pass starts at the bottom with an assignment to the input layer, computing each layer in turn and ending up at the top with the final assignment to the output layer, ignoring for now the details of how we handle recurrent connections.

When we speak of a single layer \({L}\) as one of many in a stack of layers, think of the input to \({L}\) as being the vector of variables corresponding to all of the units that connect to the units in \({L}\). The update for a specified layer \({L}\) at time \({t}\) is defined as follows: given assignments to the variables in the input to \({L}\) at \({t}\), compute the linear transformation associated with \({L}\) and pass the result to the activation function associated with \({L}\) to obtain the assigments to the vector of variables associated with output of \({L}\) at \({t}\). The corresponding vector of values associated with the output of \({L}\) at time \({t}\) is called its activation state. The technical term thought vector is also used to refer to the activation state of a layer, but we generally reserve the term for hidden layers and, in particular, hidden layers that encode abstract concepts — though admittedly the meaning of the word "abstract" in this setting is a bit murky.

While we discourage equating patterns of spiking neurons in biological networks with the activation states of hidden layers, we will draw upon work in cognitive and systems neuroscience to make the case that both representations are examples of highly-parallel distributed codes capable of representing the sort of context sensitivity that one expects from connectionist systems [93]. Such representations are in stark contrast with the systematic, combinatorial nature of largely-serial symbolic systems that account for traditional computing paradigms such as those based on the computer architectures of Alan Turing [15] and John von Neumann [87]. There are also significant differences in that biological networks are continuous-time while most current artificial networks are discrete time.

Biological networks are inherently subject to the problems described by von Neumann in his treatise on the synthesis of reliable machines, including living organisms, from unreliable components [132], whereas artificial neural networks run on hardware that finesses the problems that von Neumann alludes to by enforcing the digital abstraction [37]¹⁸. If the spiking activity of biological neurons is primarily in service to achieving some degree of stability in the behavior of such neurons — whether by managing the cell's metabolic equilibrium to maintain internal state or by employing some method of coding¹⁹ to enable the reliable transfer of digital signals between neurons, then perhaps biological neurons enforce their own digital abstraction.

---

1.5  Perceptron Models

Layers sandwiched between the input and output layers are called hidden layers and play a key role in learning. Figure 1.A shows an historically important network called a perceptron or, more specifically, a single-layer perceptron²⁰. A single-layer perceptron has no hidden layers and is restricted in what it can learn to the set of binary classifiers said to be linearly separable and perhaps best understood by an example²¹.

The Boolean function for conjunction denoted AND is defined as follow: for any Boolean propositions \({A}\) and \({B}\), if \({A}\) and \({B}\) are both true, then the expression \({A}\) AND \({B}\) is true and otherwise, i.e., for any other assignments of truth values to propositions, it is false. Perceptrons have no problem learning the AND function. The Boolean function for exclusive disjunction denoted XOR is defined as follows: if \({A}\) is true and \({B}\) is false or \({B}\) is true and \({A}\) is false, then \({A}\) XOR \({B}\) is true, and otherwise it is false. Figure 2 illustrates how the property of linear separability distinguishes these two functions.

---

1.6  Connection Matrices

While the values of the units in a layer are determined by the input and the function defined by the network and connection weights, the connection weights are initially assigned randomly and then learned from data to complete the definition of the network function. The connection weights for a given layer are represented by a mathematical object called a matrix²³. that stores the weights in a tabular format so that the row indices refer to the units in the bottom layer and the column indices refer to the units in the top layer. A matrix with \({n}\) rows and \({m}\) columns is called an \({n \times{} m}\) or \({n}\)-by-\({m}\) matrix where \({n}\) and \({m}\) are referred to as the dimensions of the matrix.

By convention, a matrix is generally denoted by an upper case letter in bold font, e.g., \({\mathbf{W}}\), and the individual matrix components are displayed in italics as done with vector components, except that they have two indices, e.g., \({w_{i,j}}\), one index to reference the row (\({i}\)) and the other to reference the column (\({j}\)) in which the entry is stored as illustrated below: $${\mathbf{W} = \begin{pmatrix} w_{1,1} & w_{1,2} & \cdots& w_{1,m} \\ w_{2,1} & w_{2,2} & \cdots& w_{2,m} \\ \vdots& \vdots& \ddots& \vdots\\ w_{n,1} & w_{n,2} & \cdots& w_{n,m} \\ \end{pmatrix} }$$ Matrices and vectors are related in that a \({1 \times{} n}\) matrix is called a row vector and a \({n \times{} 1}\) matrix is called a column vector.

The function definition corresponding to the perceptron network shown in Figure 1 can be written concisely as the equation \({\mathbf{v} \mathbf{W} = \mathbf{u}}\) where \({\mathbf{v} \mathbf{W}}\) denotes the matrix multiplication of \({\mathbf{v}}\) and \({\mathbf{W}}\) that determines the output vector \({\mathbf{u} = \langle{}u_1, u_2, \ldots{}, u_m\rangle{}}\) such that \({u_{i} = v_{i}w_{1,i} + \cdots{} + v_{i}w_{n,i} = \sum_{j=1}^{n} v_{i}w_{j,i}}\) for \({i = 1,\ldots{}m}\). Such compact notation considerably simplifies both describing neural networks and writing programs that implement neural networks since there exist programming languages that allow the same succinct representation thereby freeing the programmer from having to read and write the low-level code that implements operations on vectors and matrices.

---

1.7  Differentiable Models

As described so far, the function associated with a network is simply a linear transformation mapping from one vector space to another²⁴. Moreover, since the composition of two linear transformations is itself a linear transformation, any neural network consisting of a stack of such layers defines a linear transformation. This means that, if you are trying to learn a nonlinear function — and much of what we want to learn is nonlinear, the best you can hope for is to find a linear function that closely approximates the nonlinear one. Of course, it could be that a good linear approximation does not exist. For this reason, we apply a nonlinear activation function to the result of performing the linear transformation, thereby enabling the network to learn nonlinear functions as well.

Learning inevitably involves search. In the case of learning a model corresponding to an artificial neural network, it involves searching in the high-dimensional space of possible assignments to the connection weights. Each weight assignment corresponds to a fully instantiated network function, and the learning algorithm is restricted to find a model within the space of all such functions. This restriction is referred to as an inductive bias²⁵ in learning theory and it plays an important role in guiding search to predict outputs, given inputs that it hasn't encountered in the training data. Bias in the context of machine learning is a good thing since without bias there can be no generalization. The selection of activation functions and network topologies also serves to define the set of possible network functions and further bias the search. Here we consider two of the most common activation functions.

It is often handy to have a binary switch or threshold function. Whenever you have to make a choice between two options, e.g., between two competing chess moves or whether a photo includes a cat, you are making a binary decision. For example, the threshold function, \({f : \mathbf{R} \mapsto{} \{0,1\}}\), defined: if \({x \geq{} 0}\) then \({f(x) = 1}\) otherwise \({f(x) = 0}\), maps a scalar value on the real number line to the Boolean values \({0}\) or \({1}\). We can also use this scalar function to implement a component-wise vector version, \({\mathbf{f} : \mathbf{R}^m \mapsto{} \{0,1\}^m}\), defined: \({\mathbf{f}(\mathbf{v}) = \langle{}f(u_1), f(u_2), \ldots{}, f(u_m)\rangle{}}\). Another useful function is the max function \({f : \mathbf{R}^m \mapsto{} \mathbf{R}}\), defined: \({f(\langle{}u_1, u_2, \ldots{}, u_m\rangle{}) = u_i}\) such that for all \({j\,\neq{}\, i, u_j \leq{}u_i}\).

The problem is that these functions are discontinuous and hence they are not differentiable²⁶. The threshold function has a discontinuity at \({0}\) — or its vector equivalent \({\mathbf{0}}\) in the case of the vector function, and the max function is also discontinuous though the multivariate case is a bit trickier to explain²⁷.

To get around this problem, we substitute the continuous sigmoid and softmax functions for, respectively, the discontinuous threshold and max functions. These two continuous activation functions are among the most common such functions used in the design of artificial neural networks. A scalar sigmoid function, \({L : \mathbf{R} \mapsto{} \mathbf{R}}\), can be implemented in terms of the logistic function²⁸ as: $${L(x) = \frac{1}{1+e^{-x}} = \frac{e^{x}}{e^{x}+1}}$$ where \({e}\) is the natural logarithm base — also known as Euler's constant \({2.71828 \ldots{}\,}\), and the most common corresponding vector function is the obvious component-wise \({\mathbf{L} : \mathbf{R}^{K} \mapsto{} \mathbf{R}^{K}}\). The softmax function, \({\mathbf{S} : \mathbf{R}^{K} \mapsto{} \mathbf{R}^{K}}\) is defined as a generalization of the logistic function: $${{\mathbf{S}}({\langle{} u_{1},\,\ldots{},\,u_{K} \rangle{}) = \langle{} z_{1},\,\ldots{},\,z_{K} \rangle{} {\rm{\;such\;that\;}} {z_j = \frac{e^{z_{j}}}{\sum_{k=1}^{n}e^{z_{k}}}} {\rm{\;for\;all\;}} k = 1 \ldots{} K}}$$ that squashes and normalizes a \({K}\)-dimensional vector \({\mathbf{u}}\) of arbitrary real values to a \({K}\)-dimensional vector \({\mathbf{z}}\) of real values such that each entry, \({z_i {\rm{\;for\;all\;}} i = 1 \ldots{} K}\), is in the interval \({(0, 1)}\) and all the entries sum to \({1.0}\). In order to understand why it is important that the objective function be differentiable, we need to consider how neural networks are trained.

---

1.8  Learning the Weights

Once the architecture of the network is completely configured — including the number of units in each layer, the connections between layers and the activation function for each layer, the only step remaining to fully define the network function is to assign values to the connection weights. This is referred to as training a neural network and, as mentioned earlier, it involves searching in the high-dimensional space of possible assignments to the connection weights. One of the most powerful algorithms for training neural networks is called stochastic gradient descent²⁹. While it is possible to train a large network by breaking it up into more tractable pieces and training each piece separately, this is generally undesirable since it often means optimizing the pieces in isolation resulting in a global solution that is suboptimal. Training the entire network simultaneously is called end-to-end training³⁰.

Training relies on an objective function — also called a loss function — to evaluate the expected performance of the network for a given assignment of connection weights and guide search for an optimal assignment³¹. SGD attempts to minimize the network loss by following the gradient which is the multi-variate generalization³² of the derivative — the derivative for a real-valued function \({f(x) = y}\) measures the sensitivity to change of the output value \({y}\) with respect to a change in its input value \({x}\) and plays an important role in machine learning by way of the calculus of variations³³.

Developing an artificial neural network that can be trained end-to-end with gradient descent involves the design of a differentiable objective function. This can be challenging given that many of the supporting functions routinely used in defining objective functions for related optimization problems are not differentiable. These basic support functions include the threshold and max functions mentioned earlier Since the objective function incorporates the network function, we can't compute the gradient of the objective-function and apply SGD unless the network function is differentiable, and if the network function uses the discontinuous threshold function mentioned earlier then the network function will not be differentiable. When used to minimize the above function, the standard³⁴ gradient descent method performs the following weight update on each iteration: $${\mathbf{w}_{t+1}\,\colon{=}\,\mathbf{w}_t -\eta{} \nabla{} L(\mathbf{w}_t)}$$ where \({\mathbf{w}_t}\) is the vector of all the weights in the network prior to the update, \({L}\) is the objective or "loss" function, \({\eta{}}\) is the learning rate that determines how much to change the weights on each iteration, \({\nabla{}}\) is the gradient operator and \({\nabla{} L(\mathbf{w}_{t})}\) is the gradient of the loss, and \({\mathbf{w}_{t+1}}\) is the updated vector of all the weights.

---

1.9  Convolutional Layers

Fully connected neural network layers don't scale well to large inputs such as high-resolution images. If we have an input vector corresponding to a million pixel image, e.g., \({1000 \times{} 1000}\), and an output vector of the same size, then the fully connected weight matrix for the resulting bipartite graph would have a trillion (\({10^{12}}\)) entries. The class of models called convolutional neural networks (CNNs) solve this problem by recoding large input layers as the composition of features that span relatively small contiguous regions of the input³⁵. In the case of an image, these features might correspond to small patches that represent abrupt changes in contrast — often called edge detectors — and thereby emphasize the boundaries of objects.

In the terminology of CNNs, such a feature is called a filter — specifically a linear filter³⁶, and the contiguous region to which it applies is its receptive field. A collection of filters is called a filter bank, and a typical filter bank could easily include hundreds of filters, e.g., a set of edge detectors that discriminate edge orientation in increments of \({5^{\circ}}\). A fully configured convolutional layer consists of a set of slices, one slice for each filter in the filter bank. Figure 1.B represents a single CNN slice and the connections highlighted in red depict a \({1 \times{} 3}\) filter consisting of just three weights. Let \({\mathbf{v} = \langle{}v_1, v_2, v_3, v_4, v_5, v_6, v_7, v_8\rangle{}}\) represent the vector corresponding to the input layer and \({\mathbf{u} = \langle{}u_1, u_2, u_3, u_4, u_5, u_6\rangle{}}\) represent the vector corresponding to the slice output layer.

The filter as shown has a receptive field corresponding to \({\langle{}v_4, v_5, v_6\rangle{}}\) and is poised to compute the response of the filter at \({u_4}\) in the output layer of the slice. The response value is intended to measure how well the filter accounts for the part of the image covered by its receptive field. If you think of the filter and the receptive field as vectors — if they happen to be some other shape, e.g., a 2-D array as in the case of images, then just flatten them to make a vector, then one of the most common methods is to use cosine similarity³⁷ as a proxy for the angle between the two vectors. In the case of vectors in positive space, i.e., all of the vector components are positive, cosine similarity is bounded by \({1.0}\) for an angle of \({0^{\circ}}\) — so the two vectors are parallel and hence similar — and \({0.0}\) for an angle of \({\pi}\) radians — the two vectors are perpendicular and hence dissimilar.

Often the inner — or "dot" — product of two vectors is used as a more easily computed but numerically less stable measure of similarity that must treated with mathematical care. If the filter vector is \({\mathbf{w} = \langle{}w_1, w_2, w_3\rangle{}}\) and the receptive field vector is \({\mathbf{u} = \langle{}u_4, u_5, u_6\rangle{}}\) then the response value at unit \({v_4}\) in the output layer of the slice is the dot product of filter weight vector and the receptive field input vector \({\mathbf{w} \cdot{} \mathbf{u} = (w_1 * v_4) + (w_2 * v_5) + (w_3 * v_6)}\). The filter response is computed for each unit in the output layer of the slice. This is done for all slices and the slices are stacked in column vectors such that the \({i}\)th element of this vector corresponds to the computed response of the \({i}\)th filter. If we started with a square — two-dimensional — matrix of pixel values we would now have a three-dimensional output layer. The activation function for this layer is typically max pooling³⁸ in which the output of pooling collapses the extra dimension added in order accommodate the filter responses by selecting the maximum value in each column vector.

Note that if we have a bank of 100 \({10 \times{} 10}\) filters, each slice has 100 weights and there is a total of one thousand weights for the entire filter bank. That's quite a difference from the trillion weights required for the fully connected case. The downside of this saving is that we can only learn relatively small features. However, if we stack convolutional layers — as is typically done in practice — we can learn a hierarchy of features that account for increasingly larger patterns in the image at each successive stage in the hierarchy.

Convolutional networks are not biologically plausible in the particular way that they share variables and compute filter responses. They are, however, extraordinarily powerful neural network components and have shown off their advantages in diverse applications far from computer vision including natural language processing. Moreover the required computations can be efficiently carried out on parallel processing hardware such as are used in rendering graphics³⁹.

---

1.10  Multilayer Networks

Figure 1.C depicts a multilayer perceptron — commonly called a deep neural network — with one hidden layer. The MLP network is intended to illustrate a connection matrix corresponding to a complete bipartite graph though only two units are shown to avoid an overly cluttered graphic obscuring the pattern of connections. The output layer of (C) has a recurrent connection highlighted in red, making the point that a recurrent network is just a non-recurrent network with recurrent connections. In the case shown here, we might update the output layer by taking the average of the current activity with that of previous, e.g., \({\mathbf{v}_{t} = ( \mathbf{v'}_{t} + \mathbf{v}_{t-1} )\,/\,2}\) where \({\mathbf{v'}_{t}}\) is intended to represent the layer output prior to taking the recurrent connection into account.

In the example shown here, the hidden layer of the MLP has fewer units than either the input or the output layers which are of the same size. This is not required in general and only serves in the present circumstances to demonstrate a special case of MLP called an autoencoder that is worth emphasizing here⁴⁰. An autoencoder is an MLP that facilitates learning a compact encoding of the input that reconstructs the input and in so doing assists generalization — the ability to generalize from the training data to account for novel input. For this reason, the resulting encoding is often referred to as a generative model. The narrowing in the hidden layer is sometimes referred to as a bottleneck in that it prevents training from simply learning the identity function⁴¹.

Generative learning is contrasted with discriminative learning in that the latter emphasizes discriminating between similar examples and needn't construct an explicit model of the data, whereas the latter emphasizes learning a general model that can be used for multiple purposes by downstream components in an MLP with many layers. An important feature of autoencoders is that they can be trained by unsupervised learning in which the objective function compares the original encoding represented by the input with the encoding produced by the learned model represented in the output layer and the objective function measures the fidelity of the reconstruction. More sophisticated objectives penalize reconstruction error while at the same time rewarding the facilitation of downstream components.

While we seldom learn models just for the sake of learning them, in some cases, a discriminative model is just what you need, as in the case where you want to sort manufactured parts into two categories: defective versus certified. Generative models are often useful in cases in which their ultimate purpose is not known or clearly understood, or when it is important to explain why the system made a decision or arrived at a particular conclusion. There is some evidence to suggest that humans employ some form of Bayesian statistical inference⁴² to construct generative models of their environment that explain how humans can make reliable generalizations from so little data — what are commonly referred to as inductive leaps in learning.

---

1.11  Energy Landscapes

The notion of an energy landscape⁴³ arises in many disciplines where optimizing a function is seen as search in a combinatorial space of possible parameter settings of a complex system. In the case of computational chemistry, the parameter settings correspond to the possible conformations of a molecular entity and the goal is to find a conformation that minimizes the free energy of the system⁴⁴ so that we can build a model of molecular structure to use in synthesizing new materials or drug discovery [133, 74]. In the case of training an artificial neural network, our goal is to search for a weight assignment (conformation) that minimizes the loss function (free energy).

Figure 4.(B) illustrates the energy landscape for a network consisting of two weights, \({w_1}\) and \({w_2}\), assigned to what are conventionally called the \({x}\) and \({y}\) axes, and the value of the objective function on the \({z}\) axis computed at each point on the \({x\,y}\) plane. The resulting rugged surface exhibits multiple local maxima and minima. Ideally the surface would be convex but this is rarely the case in practice⁴⁶. In training networks with millions of weights, the energy surface is likely to be pockmarked with local minima as well as saddle points⁴⁷ that can mislead gradient descent. More sophisticated methods using second-order partial derivatives can test if a critical point is a local maximum, local minimum or saddle point, but they don't scale and can't be expected to point the way to global minimum of the objective function. Remarkably, in many cases, having more parameters results in a rougher landscape — more potholes to get stuck in, but also produces more local minima that are closer to the global minimum. Suffice it to say that a great deal of effort has been put into designing better solvers.

---

1.12  Accelerating Returns

Researchers developing neural network solutions typically have to train and test hundreds if not thousands of experimental networks. It's one thing if running a test on a network, takes no more time than it takes to grab coffee, and quite another if it takes days or even weeks. It helps if you're working in industry and have access to unlimited computing resources. Unfortunately, there is no such thing as unlimited computing and engineers are always competing with one another and with the needs of customers to get their work done. It is perhaps no surprise then that engineers have taken to applying neural networks to automatically and efficiently search in the space of network architectures for solutions that satisfy their needs.

This kind of technology is part of a trend in applying machine learning technology recursively to accelerating the development machine learning technology. Throughout the last few decades, researchers working on neural networks often expressed frustration with progress in the development of better analytical tools, mathematical foundations, software and scalable infrastructure to run it on. At times, the state-of-the-art in developing neural-network technology seemed more like alchemy than science or engineering.

The wave of excitement around the beginning of the new millennium as neural networks were shown to outperform more traditional methods of machine learning was tempered by a realization that we were in uncharted territory. There was time when, miraculously and contrary to received wisdom, adding more layers or layers with more units and more connections often seemed to help rather than hinder finding good if not optimal solutions. Researcher puzzled over why this should happen, and the mathematically adept resurrected old or developed new sophisticated regularization methods to understand such phenomena⁴⁸.

Neural network technology is flourishing in large part because it has been so successful and there are so many smart individuals contributing to its development and savvy companies investing in a growing ecosystem of talent and supporting technologies. Artificial neural networks have received the lions share of the attention but the dividends from its success are being spread across a much wider spectrum of technologies including robotics, computer vision, imaging, advanced biosensors and neural prostheses.

An entire generation is growing up with possibility of living decades longer than their parents, conquering the major causes of mental and physical decline and developing technology to make us smarter and better able to address problems on global scale. Machine learning, artificial intelligence, synthetic biology, genetic engineering and materials science are advancing at an incredible, some would say alarming pace, reinforcing and accelerating one another. It may sound trite to say so given all the hype surrounding these technologies, but we will have evolve along with our technologies if we are to continue to play our enhanced and artificially intelligent friends.

---

1.13  Recurrent Networks

In his book I Am a Strange Loop⁴⁹ Douglas Hofstadter explores the idea of consciousness and the concept of "self" in terms of a recurrent process thereby dismissing the philosopher's confusion about homuncular regress⁵⁰ and the viewer in the Cartesian theatre⁵¹. In a subsequent chapter, we consider how one might go about implementing a digital assistant that exhibits many if not all of the properties of human consciousness worth preserving in an artificial intelligence. The artificial neural network implementation makes use of recurrent connections to explain and provide a compelling use case for the emergent phenomena of inner speech⁵² and self-reflection. In the following, we explore more practical, less controversial applications of recurrent networks.

All three of the networks shown in Figure 5 are said to be recurrent. In contrast, the networks shown in Figures 1.A and 1.B include only feedforward connections. Recurrent connections run in the opposite direction from feedforward connections and can span multiple layers. A recurrent connection can also feed back into the same layer it originated from as illustrated in Figure 1.C or it can connect two units within a layer. Recurrent neural networks — networks that have recurrent connections — will play an important role in our understanding human cognition and figure prominently in our discussion of neural network architectures⁵³.

Even the simplest biological networks make use of feedback. Without feedback, networks couldn't keep state — they couldn't remember past events, they couldn't measure progress or adjust their responses to deal with errors in perception or motor control. Norbert Wiener⁵⁵ introduced the term cybernetics⁵⁶ to describe the study of systems that employ feedback to control their behavior. Biological systems employ feedback to regulate everything from hormone balance to consciousness. The three networks shown in Figure 5 illustrate the use of feedback to support three basic functions: attention, memory and recursion. In subsequent chapters we'll apply these basic functions to support a wide range of activity. Here we take a closer look at how they are implemented since, the detail provide insight into how they work and how they might adapted for other purposes.

Three specialized recurrent neural networks components are illustrated in Figure 5: (A) is an attentional network that learns how to shape the context for performing inference as in the case of machine translation or continuous dialogue, (B) is a short-term memory module that learns how to anticipate information required for inference and set it aside until needed, and (C) is a network architecture consisting of a pair of recurrent networks used to transform one sequence into another as in the case of question answering.

Figure 5.A shows a schematic description of an attentional selection module that learns what part of our recent experience is most important for our current purposes and makes that information available to guide inference. The network shows four (feedforward) connections to the bottom layer each of which corresponds to a thought vector of the same size. The red (recurrent) connection corresponds to a single thought vector summarizing the current state of information, i.e., the context for current decision making. Each of the feedforward vectors is concatenated with the current context vector and fed into a softmax layer, the output of which is used to decide (gate) which of the four (feedforward) thought vectors is most relevant to the current context.

The attentional selection module is useful in machine translation (MT) to determine which parts of the original (input) sentence are relevant in deciding the next word in the translated (output) sentence. For example, in deciding the gender or number — singular or plural — of a pronoun to use in the translation it is often useful to refer back to any previous information in the input sentence that mentions the reference of the pronoun under consideration. The network in Figure 5.C is designed to support MT as well as other problems that involve transforming one sequence — of words in the case of MT — to another sequence.

---

Figure 5.B shows a schematic description of a gated memory module that can be used for any application in which it is useful to learn what information to keep in short-term memory in performing a task. This module makes use of three trainable gates that control the contents of a memory cell. The three gates determine (a) when the cell content should be updated with the current input, (b) when the current content of memory should be made available as output and (c) when the current content should be erased.

Each gate consists of a learnable network that takes as input both feedforward and feedback connections, computes the weighted sum of these inputs and then applies a sigmoid activation function to produce a (differentiable) approximation to a binary switch. The scalar output of the sigmoid function can be used to — depending on the gate — add the block input to the content of the memory cell, make the content of the memory cell available as the block output, or control whether the current content of the memory cell is fed back to cell by the gated recurrent connection or not, i.e., it is forgotten.

As an example, if the output value of the forget-gate sigmoid is close to one and the output value of the input-gate sigmoid is also close to one, but the output-gate sigmoid is close to zero, then the cell memory will contain the sum of its previous content and current block input vector and the block output will be (approximately) a vector of all zeros the same dimensionality as the memory cell content — which is the same size as input block vector. This affords the rest of network using the gated cell block a great deal of learnable control over the information required to perform its assigned duties.

---

The autoencoder described in Figure 1.C works well for learning useful representations of fixed size data like still images or video from a given camera⁵⁸. In principle, you could make it work with text data in the form of sentences for machine translation or documents for summarization, but processing text sequentially is convenient for many applications including, for example, continuous conversation and sentiment analysis⁵⁹. Recurrent models are particularly useful for such applications and there is a variation on the autoencoder architecture we described earlier that has proved especially versatile for working with structured data including but not limited to text sequences.

The Long Short-Term Memory (LSTM) model was invented by Sepp Hochreiter and Jürgen Schmidhuber [57]. It was originally developed to deal with a problem in training recurrent models in which the gradient either grows without bound or shrinks to zero. The initial version of an LSTM block included memory cells plus input and output gates as described in our discussion of Figure 1.B. Forget gates were added later by Felix Gers [45]. A fully configured LSTM includes one or more memory cell blocks along with conventional multilayer network components that control the memory cell gates in order to anticipate the need for and remember state information for as long as is required.

Figure 5.C shows the LSTM model used as the recurrent module in an encoder-decoder architecture that can be trained for machine translation, dialogue management or code synthesis. Unlike the stacked encoder-decoder architecture of the autoencoder, here we have a separate encoder and decoder each of which is implemented as a recurrent neural network. The encoder ingests the input sentence, \({\langle{}w_{1},w_{2},w_{3},w_{4}\rangle{}}\), one word at a time starting with some initial thought vector encoding previously processed text and providing a context for understanding the next sentence in a document or conversation.

When the sentence is completely ingested, the encoder hands off the resulting thought vector to the decoder that generates the translation, \({\langle{}u_{1},u_{2},u_{3},u_{4}\rangle{}}\), one word or phrase at a time. The decoder enriches the thought vector it has been provided as its input in order to account for its extension of the ongoing translation. The decoder continues in this fashion until the translation is complete. In Figure 5.C there is just one LSTM network for the encoder and one for the decoder. The graphic shows the recurrent process as it unfolds over time one word at a time. The lengths of the two sentences needn't be the same.

In production, the output of the decoder is sent to a softmax layer that generates a probability distribution over possible next words that is used to select the next word in the translation. Training data consists of source-language target-language sentence pairs, e.g., \({\langle{}\,\langle{}w_{1},w_{2},w_{3},w_{4}\rangle{},\,\langle{}v_{1},v_{2},v_{3},v_{4}\rangle{}\,\rangle{}}\). During training, the encoder-decoder is provided with such a pair and the objective function takes into account the difference between the supplied target, \({\langle{}v_{1},v_{2},v_{3},v_{4}\rangle{}}\), and the generated translation, \({\langle{}u_{1},u_{2},u_{3},u_{4}\rangle{}}\).

---

Knowing the exact technical details regarding how LSTM technology works is not our goal here. The LSTM technology is now nearly two decades old and several related competing technologies have been developed in the meantime. The important take away is that the LSTM technology provided us with new capabilities and did so by using existing components and relying on fully differentiable networks that can be trained with gradient descent. An LSTM can learn to anticipate information that it will need later, store that information and maintain it until it is required. The information is stored in the form of a high dimensional vector, what we have been calling the thought vector, that allows us to encode abstract concepts in a native format that is completely compatible with our existing neural network technology. We can further enhance these capabilities by incorporating the attentional module in Figure 5.C.

In subsequent chapters will look at network architectures that allow us to make complex predictions about the future and use those predictions to construct and execute plans to solve all sorts of practical problems from driving a car to writing a computer program. These architectures will depend upon the same basic components and training methods in much the same way that designing new computer architectures depends upon the same logic gates and memory devices as the computers we use today. Many of the recent innovations in the development of artificial neural networks have come about by figuring out better ways of training such networks that don't rely upon labeled data and that can be trained in stages by exposure to increasingly complicated learning environments much as a child learns by building new skills on top of existing ones throughout the decades long developmental stages that set humans apart from any other organism that we know about.

As we follow this thread throughout the subsequent chapters, we will also be looking for opportunities to accelerate skill acquisition by artificial means in both humans and machines. The path to Matrix style uploading of new skills is probably either technically infeasible or simply too risky to perform on humans anytime soon. But it may be possible to develop AI systems dedicated to the education of each individual child in a way that was never practical for either public school systems or working parents. The implications of this for society could herald new hope for a broadly educated, well informed self-governing society.

---

1.14  Signifying Meaning

Words are signs and signs signify meaning. According to Charles Sanders Peirce⁶⁰, one of the founders of semiotics⁶¹, there are three types of sign: icons, indexes and symbols. Words are symbols. This turns out to be important in communicating ideas that have a precise technical meaning, but not so much for communicating about more fluid concepts.

What about images? Images are also signs and signify meaning. We know how to represent images in a computer. Images are composed of pixels arranged in rectangular grids and pixels are represented as numbers encoded as bit vectors. Unfortunately, this representation isn't particularly useful for working with visual information in artificial neural networks. Let's consider the meaning of images, and we'll get to words and the fact that words are symbols in due course.

The meaning of words and images is all about context⁶². If we're looking at a photo of a barnyard with cows and pigs and a farmer sitting on a tractor, and we notice something the size of a small dog wearing what looks like a red hat, we might reasonably jump to the conclusion that it's a rooster. However, if someone were to cut out and then present us with just the pixels corresponding to a pig or the farmer's upper torso, we probably wouldn't have clue what we were looking at.

From linguistic standpoint, individual words have little meaning without some context in which to interpret their meaning. In conversation, the context of a spoken word has many levels: the words that immediately proceed and follow, the last few statements, our previous conversations with the person we are speaking, etc. We rely on context to imbue words with meaning in developing neural networks to handle natural language. Here is a good example of how, by taking inspiration from human cognition in the design of neural network architectures, we achieve a high degree of competency. This observation is not surprising given that we invented natural language to serve our purposes.

We represent words, phrases and larger fragments of language by embedding⁶³ them as points in high-dimensional vector spaces so that nearby points are semantically related and different dimensions reflect different comparative characteristics along which words differ such that relatively simple operations, e.g., subtracting one vector from another, expose these differences and reveal these relationships. In this approach, the boundaries between words are not sharply defined and meaning has a tendency to drift as usage evolves and exposure to language changes over time, e.g. you start reading more Tolstoy and Dostoyevsky and fewer superhero comics.

We can create embeddings of all sorts of abstract entities including relational graphs, computer programs, taxonomies, family trees, organizational charts, social networks, dynamical system models, etc. We can use such models to answer questions, "Who was Sophie's grandmother her father's side?, make predictions "What if the pressure relief valve failed?" and keep track of and reason about personal interactions "Does Alan know Sherry has the keys?". AI systems can't automatically decide when to learn these model but that is largely because the current generation of AI personal assistants aren't smart enough [...] humans do this sort of thing all the time. Creating models and making use of them is an integral part of how we [manage our complex lives | go about our every day lives | conduct business ].

---

1.15  The History of Ideas

The limitation to linearly-separable classifiers was considered so restrictive that it effectively discouraged a generation of computer scientists from pursuing research in this area. Academic advisers would often speak disparagingly about neural network, joking that the second best approach to solving any problem was either an artificial neural network or genetic algorithms — proponents of which were likened to alchemists in their single-mined pursuit of a method for turning dross into gold, thereby sending a clear signal to their students not to waste time pursuing such solutions.

A number of determined researchers persisted and were able to demonstrate that networks with one or more hidden layers were able to learn much more complicated functions. These networks — often referred to as multilayer perceptrons would resurface decades later as so-called deep neural networks. The reason for the delay can be attributed to several factors. Multilayer networks required more connections and hence more parameters; with more parameters to fit it was assumed that it would be easy to overfit, and, as a consequence, generalization and therefore performance on unseen data would suffer. There were also numerical problems with the existing optimization algorithms and surface of the energy landscape was riddled with potholes corresponding to local minima for gradient methods to fall into or saddle points where the slopes (derivatives) in orthogonal directions are all zero, but which is not a local extremum of the function⁶⁴.

Among those who kept the faith a small group of researchers who referred to themselves as the PDP group were able to make considerable theoretical progress even without the benefit of more powerful computing resources. Some of the most prolific included David Rumelhart, Jay McClelland, Geoffrey Hinton, Terrence Sejnowski, David Touretzky and Peter Dayan. Hinton in particular developed a huge number of sophisticated models, useful algorithms and clever insights. Together they created a wealth of science and technology that would serve as a springboard when the tide turned in the late 20th and early 21st century and neural networks saw a reversal in fortune and a renaissance in the underlying theory and practical applications of neural networks.

It is perhaps worth pointing out that despite the contributions of the PDP group and those few stalwarts who followed in their footsteps, much of the opposition came from researchers in the United States who were convinced that symbolic methods and logic-based artificial intelligence was the best way to pursue AI. The 1980s saw the rise of expert systems and rule-based technologies promulgated by researchers at Stanford University including John McCarthy and Ed Feigenbaum. During the same time members of the PDP group and their colleagues in cognitive science were discovering how symbolic and connectionist approaches complemented one another, explaining how human beings solve complex problems.

During the 1980s and the decades following, interest in connectionist models remained relatively strong in Canada, Europe and Asia with Yoshua Bengio, Jürgen Schmidhuber and Kunihiko Fukushima among those making foundational contributions. During the same period, the PDP group produced a two-volume compilation of their research and that of their students and colleagues that consolidated their research and has provided a wonderful resource for the current generation of connectionists⁶⁵.

The computational resources available to most researchers at the time were such that training was prohibitive or at the very least tedious — it is difficult to make progress when running one experiment takes days to complete with little or no intermediate indication that it will produce a good solution. In the intervening years the picture has changed dramatically. Increasingly personal computers and small servers were being networked and networking hardware was becoming faster even relatively small academic departments could afford to build clusters consisting cheap commodity computers.

Practical parallel computing was still in its infancy but as innovations in semiconductor lithography increased the number and density of transistors you could fit on an integrated circuits, chip manufacturers started designing chips with multiple cores and dedicating silicon to SIMD⁶⁶ hardware (implementing so called single-instruction, multiple-data parallelism) and a few intrepid hackers were figuring out how to write parallel programs that ran on dedicated graphics processors [...] eventually better graphics, cell phone technology and the Internet would create an insatiable demand for computing resources [...] leading to the rapid deployment of cloud computing services and data warehouses sprinkled around the globe [...]

---

1.16  Key Ideas Summary

Here is a summary of the most important ideas covered in this chapter:

1. Networks with one or more hidden layers are more powerful than those with none;

2. Biological and artificial neural networks rely on different design principles;

3. ANNs exploit different opportunities for parallelism than biological networks;

4. Biological and artificial neural networks use similar computational strategies;

5. ANN vectors play a similar role to spiking neurons in biological networks;

6. Gradient descent relies on the network objective function being differentiable;

7. Convolutional neural networks work on large input layers by sharing weights;

8. An hierarchy of convolutional layers can learn features at multiple scales;

9. Autoencoders use unlabeled data to learn representations without supervision;

10. Recurrent networks extend feedforward networks using feedback to retain state⁶⁷

11. Attentional networks learn how to create an appropriate context for inference;

12. Encoder-decoder architectures make it possible to work with structured data;

---

Nonlinear Functions

We make a rather big deal about linear systems in these pages. Linear functions are generally more tractable to manipulate mathematically. It's as if you were to draw a Venn diagram, linear functions would look like a small circle surrounded by everything else — meaning all the nonlinear functions. Like a tiny kernel in middle of a giant peach. Classifiers that have certain linear properties are easier to learn than those that don't. Linear systems are easier to control than nonlinear systems. Linear systems theory is pretty much a closed book — current textbooks look a lot like the textbooks that were printed 40 years ago.

Often complex systems include a combination of linear and nonlinear components simplifying analysis somewhat, but there is always the concern that the nonlinear parts are the most important understand⁶⁸. I can't remember just now who first said it — might have been Carver Meade, but I seem to recall I heard it from Bruno Olshausen: "It isn't surprising that neurons behave nonlinearly. What's surprising is that in combination they can manage to behave linearly." John von Neumann's ideas concerning the synthesis of reliable organisms from unreliable components were motivated by dealing with unreliability of vacuum tube components for building computers, but apply equally well to biological systems.

One important consideration is that solving systems of linear equations can be done in polynomial time⁶⁹. Linear systems theory⁷⁰ is enormously useful in a wide range of applications from seismology to circuit design. Linear programming is often used to generate approximate answers to intractable problems. For example, many important optimization problems in operations research can be expressed as a (mixed) integer linear program⁷¹ which are notoriously hard to solve given that integer programming is NP-complete.

One of the easiest approximations is to reduce the problem to a linear program by relaxing the integral requirement, solve the resulting linear system and then round the original integer variables to satisfy the integral requirement⁷². This approximation runs in polynomial time and often produces solution that are competitive than those arrived at by (mixed) integer linear program solvers.

---

No Magic Required

I've said that this chapter is an attempt to make sure that you understand exactly what an artificial neural network consists of. If you follow the presentation here, you should be able to look at an artificial neural network and break it down into simpler components until everything can be written down as an equation and that equation consists entirely of basic arithmetic operations, including additions, subtractions, and multiplications plus a few mathematical functions such as sines and cosines that are either familiar to you from high school math or easy to understand intuitively when plotted as a graph.

I want you to contrast that level of understanding with our current understanding of biological neural networks composed of neurons. These are living cells that are constantly directing their genetic machinery to manufacture the molecules required not only to perform the complicated signaling that goes on when two neurons communicate with one another, but also produce the proteins and lipids that are required to maintain its molecular integrity and generate the energy required for every operation carried out in the cell. And then there are the poorly understood glia, the dark matter of the brain. We do know a lot about neurons but I would hazard a guess that (a) there is much more that we don't know than we do, and (b) we are not even sure that what we think we know is true.

Anybody who tells you that artificial neural networks are nothing like biological neural networks is very likely to be right, but their confidence is unwarranted since at best they dimly perceive the true function of biological networks. Simply because no one does. Moreover, there may very well be a strong case to be made that at some level of description artificial and biological networks are more similar functionally than they are different, and that there is a great deal of value to be had by studying artificial neural networks if you want to understand biological ones. Later we will talk about different levels of abstraction and the overused and often abused concept of emergence. At that point, I hope we can make even more clear the benefits of studying artificial networks.

---

2  Accelerated Evolution of Artificial Intelligence

In this chapter, we investigate the prospects for building digital assistants that meet or exceed human performance in specialized technical areas and are capable of interacting with humans using natural language to collaborate and share knowledge. We draw upon research from cognitive and systems neuroscience for insight into the neural basis of human cognitive function. The sections on automated planning and natural language processing are a work in progress and will be fleshed out with help from our invited speakers and the students taking the class.

2.1  Technology Predictions

In the previous chapter we looked at some of the basic neural-network components from which more complicated network architectures are constructed. These components, referred to as Lego-block like networks, are the foundation for the current generation of artificial neural networks. The intention is to make it clear that there is no magic and demonstrate that the basic ideas are accessible to anyone with high-school math. In this chapter, we use those basic building blocks to predict the next generation of personal assistants. This exercise sets the stage for thinking about how the technology might help us deal with the problems facing us as a species.

To focus discussion, we consider a personal assistant that works with a software engineer in the role of an apprentice learning on the job, as was common in the guilds and trade associations of medieval cities and continues to this day in some crafts such as glassworking, albeit with better laws to protect the rights of employees. The programmer's apprentice we consider here is a novice programmer but has the intuitive skills of an idiot savant, given that the apprentice has a suite of powerful programming tools as an integral part of its brain. The assistant application allows us to explore the important role of language in solving problems and the technical challenges in exploiting cognitive enhancements.

While the next generation of AI systems will include personal assistants familiar to those of us who have sampled the current generation of consumer products, we can also expect systems that exhibit greater autonomy, improved communication skills and specialized expertise on a par with or exceeding that of humans lacking augmentation. We can also expect technology that will make it possible for us to seamlessly integrate advanced capabilities with our innate human cognitive apparatus utilizing increasingly powerful and invasive brain computer interfaces. It is inevitable that consumers will crave and engineers will invent technologies that provide an edge — an edge that will quickly become the new norm.

This will be just the beginning of how intelligent machines become more deeply integrated into human society, both as cognitive enhancements and as autonomous agents that will no doubt become increasingly demanding of rights on a par with those accorded humans. In the near term we have an opportunity to solve a wide range of human problems even as we come to grips with the possibility of AI systems that deserve to be treated with the same respect we currently reserve for one another. The goal here is not to anticipate cross-species conflict or suggest how we might resolve such conflicts. The goal is to explore the options for our own evolution even as we invent the possibilities for own creations to evolve.

Making predictions even a few years out is difficult. Making predictions a few decades out is best left to science fiction writers. Accurate prediction is made much harder as a consequence of technologies accelerating the pace of change to such an extent it is almost impossible for us to really comprehend. In any case, long-term prediction is a fools errand and we are generally better off thinking about how to apply ourselves to solving today's problems given that they aren't likely to disappear without our intervention. In this and subsequent chapters, we focus on how to make the best of what is likely to transpire technologically in the ten years or so.

2.2  Thinking About Thinking

This chapter provides the technical basis for making reasonable predictions about the development of AI technology over the next decade. I think it's important the reader understand the technology well enough to form their own opinion about how this technology might make a difference within our lifetimes. The story is complex in large part because it involves a merging of human and machine intelligence. The focus in this chapter is on how our understanding of one informs the design of the other.

But there is another advantage of the way this chapter and the book as a whole are organized. Artificial neural networks are simpler than biological neural networks. They were developed to understand biological networks as computing machines. In much the same way as the digital abstraction makes it possible to design digital computers with analog components, the artificial neural network abstraction (ANNA) allows us to ignore the details of how neurons function while retaining a level of description that provides insight into how human think.

Cognitive neuroscientists routinely employ concepts from artificial intelligence in order to explain complex phenomena that would be difficult to explain in terms of genes, molecules and basic chemistry. ANNA serves to bridge the conceptual gap between neurons and behavior. The goal is to use ANNA to explain how human brains accomplish basic cognitive skills, to demonstrate how machines can emulate such skills, and to describe how we can engineer autonomous systems and powerful cognitive prostheses that enhance our innate cognitive abilities.

The reader is led to imagine a recurring cycle of activity that begins at the periphery of our bodies as we gather information through our senses and engage with the world through our limbs. Information from multiple sensory modalities is combined to create abstract representations that we consciously choose to attend to and by doing so conjure up memories of related experience that we then apply to engaging both the complex physical world that we inhabit and the equally complex intellectual world that defines our social and cultural milieu.

This cycle is explored further by considering how we can take what we have learned about human cognition to build artificial systems of the sort alluded to above. In this case, we consider the design of a highly capable digital apprentice that would work with and learn from a human software engineer in writing computer programs as an initial step in bootstrapping the next generation of autonomous agents and ultimately creating a world in which intelligent machines and augmented and baseline humans live and work together toward shared goals.

2.3  Programmer's Apprentice

The original programmer's apprentice was the name of project initiated at MIT by Chuck Rich and Dick Waters and Howie Shrobe to build an intelligent assistant that would help a programmer to write, debug and evolve software. Our version of the programmer's apprentice is implemented as an instance of an hierarchical neural network architecture. It has a variety of conventional inputs that include speech and vision, as well as output modalities including speech and text. In these respects, it operates like most existing commercial personal assistants.

It differs substantially, however, in terms of the way in which the apprentice interacts with the programmer. It is useful to think of the programmer and apprentice as pair programming, with the caveat that the programmer is in charge, knows more than the apprentice does — at least initially, and is invested in training the apprentice to become a competent software engineer. One aspect of their joint attention is manifest in the fact that they share a browser window. The programmer interacts with the browser in a conventional manner while the apprentice interacts with it as though it is part of its body directly reading and manipulating the HTML using the browser API. The browser serves both programmer and apprentice as and encyclopedic source of useful knowledge as well as another mode of interaction and teaching.

Suppose you could merely imagine a computation, and a digital prostheses, an extension of your biological brain, would turn it into code that instantly realizes what you had in mind. Imagine looking at an image, dataset or set of equations and wanting to analyze and explore its meaning as an artistic whim or part of a scientific investigation. I don't mean you would use an existing software suite to produce a standard visualization, but rather you would make use of an extensive repository of existing code to assemble a new program analogous to how a composer draws upon a repertoire of musical motifs, themes and styles to construct new works, and tantamount to having a talented musical amanuensis who, in addition to copying your scores, takes liberties with your prior work, making small alterations here and there and occasionally adding new works of its own invention, novel but consistent with your taste and sensibilities.

Perhaps the interaction would be wordless and you would express your objective by simply focusing your attention and guiding your imagination, the prostheses operating directly on patterns of activation arising in your primary sensory, proprioceptive and associative cortex that have become part of an extensive vocabulary that you now share with your personal digital amanuensis. Or perhaps it would involve a conversation conducted in subvocal, unarticulated speech in which you specify what it is you want to compute and your assistant asks questions to clarify your intention and the two of you share examples of input and output to ground your internal conversation in concrete terms.

More than thirty years ago, Charles Rich and Richard Waters published an MIT AI Lab technical report [102] entitled The Programmer's Apprentice: A Research Overview. Whether they intended it or not, it would have been easy in those days for someone to misremember the title and inadvertently refer to it as "The Sorcerer's Apprentice" since computer programmers at the time were often characterized as wizards and most children were familiar with the Walt Disney movie Fantasia, featuring music written by Paul Dukas inspired by Goethe's poem of the same name⁷³. The Rich and Waters conception of an apprentice was certainly more prosaic than the idea described above⁷⁴, but they might have had trouble imagining the amount of code available in open-source repositories and the considerable computational power we carry about on our persons or can access through the cloud.

In any case, you might find it easier to imagine describing programs in natural language and supplementing your descriptions with input-output pairs. The programs could be as simple as short shell scripts used to extract text fragments from large numbers of files or as complicated as powerful simulators and optimization algorithms. The point is that there is a set of use cases that are within our reach now and that set will grow as we improve our natural language understanding and machine learning tools. The scope of applications within reach today is probably larger than you think and our growing understanding of human cognition is helping to substantially broaden that scope and significantly improve the means by which we interact with computers in general and a new generation of digital prostheses in particular. Here are just a few of the implications that might follow from pursuing a very practical and actionable modern version of The Programmer's Apprentice:

Develop systems that enable human-machine collaboration on challenging design problems including software engineering:

The objective of this effort is to develop digital assistants that learn from continuous dialog with an expert software engineer while providing initial value as powerful analytical, computational and mathematical savants. Over time these savants learn cognitive strategies (domain-relevant problem solving skills) and develop intuitions (heuristics and the experience necessary for applying them) by learning from their expert associates. By doing so these savants elevate their innate analytical skills allowing them to partner on an equal footing as versatile collaborators — effectively serving as cognitive extensions and digital prostheses, thereby amplifying and emulating their human partner's conceptually-flexible thinking patterns and enabling improved access to and control over powerful computing resources.

Leverage and extend the current state of the art in machine learning by integrating human and machine intelligence:

Current methods for training neural networks typically require substantial amounts of carefully labeled and curated data. Moreover the environments in which many learning systems are expected to perform are partially observable and non-stationary. The distributions that govern the presentation of examples change over time requiring constant effort to collect new data and retrain. The ability to solicit and incorporate knowledge gleaned from new experience to modify subsequent expectations and adapt behavior is particularly important for systems such as digital assistants with whom we interact and routinely share experience. Effective planning and decision making rely on counterfactual reasoning in which we imagine future states in which propositions not currently true are accommodated or steps taken to make them true [53]. The ability for digital assistants to construct predictive models of other agents — so-called theory-of-mind modeling — is critically important for collaboration [101].

Draw insight from cognitive and systems neuroscience to implement hybrid connectionist and symbolic reasoning systems:

Many state-of-the-art machine learning systems now combine differentiable and non-differentiable computational models. The former consist of fully-differentiable connectionist artificial neural networks. They achieve their competence by leveraging a combination of distributed representations facilitating context-sensitive, noise-tolerant pattern-recognition and end-to-end training via backpropagation. The latter, non-differentiable models, excel at manipulating representations that exhibit combinatorial syntax and semantics, are said to be full systematic and compositional, and can directly and efficiently exploit the advantages of traditional von Neumann computing architectures. The differences between the two models are at the heart of the connectionist versus symbolic systems debate that dominated cognitive science in 80's and continues to this day [93, 40]. Rather than simulate symbolic reasoning within connectionist models or vice a versa, we simply acknowledge their strengths and build systems that enable efficient integration of both types of reasoning.

Take advantage of advances in natural language processing to implement systems capable of continuous focused dialog:

Language is arguably the most important technical innovation in the history of humanity. Not only does it make possible our advanced social skills, but it allows us to pass knowledge from one generation to the next and provides the foundation for mathematical and logical reasoning. Natural language is our native programming language. It is the way we communicate plans and coordinate their execution. In terms of expressiveness, it surpasses modern computer programming languages, but its capability for communicating imprecisely and even incoherently, and our tendency for utilizing that capability makes it a poor tool for programming conventional computers. That said it serves us well in training scientists and engineers to develop and apply more precise languages, and its expressiveness along with our facility using it make it an ideal means for humans and AI systems to collaborate. The consolidation and subsequent recall and management of episodic memory is a key part of what makes us human and enables our diverse social behaviors. Episodic memory makes it possible to create and maintain long-term relationships and collaborations [100, 83, 92].

Think seriously about how such technology might ultimately be employed to build brain-computer-interfaced prostheses:

This exercise primarily relies on the use of natural language to facilitate communication between the expert programmer and apprentice AI system. The AI system learns to use natural language in much the same way as a human apprentice would — as a flexible and expressive tool to capture and convey understanding and recognize and resolve misunderstanding and ambiguity. The AI system interacts with computing hardware through a highly instrumented integrated development environment. Essentially, the AI system can read, write, execute and debug code by simply thinking — reading and writing to a differentiable neural computing interface [48]. It can also directly sense code running by reading messages generated by the code, parsing output from the debugger and collecting and analyzing program traces. The same principles could be applied to develop digital prostheses employed for a wide range of intelligence-enhancing human-computer interfaces.

2.4  Memory and Computation

The fields of cognitive and systems neuroscience⁷⁵ are playing an important role in directing and accelerating research on artificial neural network systems. Much of this work predates and helped give rise to the especially exciting work on connectionist models in the 1980s. However, in the nearly 40 intervening years, a great deal of progress has been made, much of it due to improved methods for studying the behavior of awake behaving animal subjects and human beings in particular. Indeed, this work is undergoing a renaissance fueled by even more powerful methods for observing brain activity in human beings in the midst of solving complex cognitive tasks.

The field of automated programming, after decades of steady, often quite practical research on using symbolic methods — much of it originating in labs outside the United States, is seeing a renewed interest in artificial neural networks. It remains to be seen whether artificial neural networks will have a significant impact on code synthesis, however there appear to be opportunities to leverage what we know about both natural and artificial neural networks to make progress, and hybrid systems that combine both connectionist and traditional symbolic methods may have the best chance of pushing the state-of-the-art significantly beyond its present level.

Understanding human intelligence, advancing artificial intelligence and automating the field software engineering so as to accelerate progress on the first two! Critics can't complain we haven't set ourselves lofty goals. They may however argue that we have stretched ourselves too thin. I maintain that these three areas of science and engineering offer a confluence of ideas that are leading to new solutions and in the process fueling their own advancement. Already artificial intelligence is being used to construct models from brain data and automated programming is being used to learn new artificial neural network architectures. In the following, we apply lessons from neuroscience to the design of digital assistants.

Memory and computation in biological neural networks are collocated in the sense that neurons store information (data) and perform computations (calculations) involving that information. There is no direct analog of a computer program since the calculations are determined by the current internal state of the neuron and the information provided by its neighbors. This is in contrast with conventional computing on machines based on the von Neumann architecture in which memory and computation are kept separate. To perform computations, instructions and data stored in random access memory (RAM) are copied into special registers in the arithmetic and logic unit (ALU) where instructions are carried out on data using ALU hardware and the results copied back into RAM. In von Neumann machines, programs are also stored in RAM as sequences of instructions. The operations performed by the ALU are general in that any program written can be reduced to ALU operations.

In the case of the programmer's apprentice, much of the relevant information is about computing and software and includes the type of items that software engineers routinely think about in plying their trade such as algorithms, data structures, interfaces, programs, subroutines and tools such as assemblers, compilers, debuggers, interpreters, parsers and syntax checkers. Then there are the things that programmers generally do not think about explicitly but that concern how they solve problems and organize their thoughts, including, for example, the design strategies we learn in computer science courses such as divide-and-conquer, dynamic-programming and recursion. Finally, there is strategic organizational information of a sort that plays a role in any complex individual or collaborative effort including plans, tasks, subtasks, specifications and requirements.

All of this information has to be encoded in memory and made accessible when required to perform cognitive tasks. Information, whether in a computer or a brain, tends to move around depending on what is to be done with it, and, at least in biological brains, it is constantly changing. In our biological brains, it is difficult if not impossible to think about something without changing it. In building systems inspired by biological brains we have somewhat more control over such changes, but control comes at a cost. We make no distinction between concrete and abstract thoughts — all thoughts are abstract whether they represent atoms or bits. We will on occasion refer to memories as being short- or long-term but the distinction doesn't begin to address real issues. When we talk about episodic memory, it may seem that we are referring to some sort of permanent or archival memory, but that's not the case in biological systems for reasons that make sense from an evolutionary standpoint and we may decide to follow suit in building artificial systems.

2.5  Cognitive Neuroscience

The study of human cognitive function at the systems level primarily consists of human subjects performing cognitive tasks in a functional Magnetic Resonance Imaging (fMRI) scanner that measures brain activity as reflected in changes in local blood oxygen levels associated neural activity. These studies localize functional activity in space and time to construct cognitive models by measuring the correlation between regions of observed brain activity and the steps carried out by subjects in solving problems. While these studies have played an important role in our understanding of human cognition, they are limited by their relatively poor spatial (several millimeters) and temporal (hundreds of milliseconds) resolution when compared with more invasive methods. That said, fMRI is currently the best method for whole brain imaging of awake, behaving human subjects. This may change with recent improvements to magnetoencephalography (MET) technology which as better comparable spatial and substantially better temporal (milliseconds).

In addition to localizing brain activity correlated with cognitive functions, Diffusion Tensor Imaging (DTI) can be used for tractographic reconstructions to infer white-matter connections between putative functional regions. White matter corresponds to myelinated axons and serves to connect neurons that are relatively far apart — myelin increases the velocity at which electrical impulses propagate along myelinated fibers in both the central and peripheral nervous system and is especially important in large brains where faster signal conduction speeds neural processing. By using anatomical landmarks to localize functional regions and tractography to trace the white-matter connections between functional regions, researchers can analyze time-series consisting of the activity patterns recorded at fixed intervals while subjects perform standardized cognitive tests, for the same subject on multiple trials, between subjects or averaged over multiple trials.

For example, in one study, economists investigate how neural activity patterns vary as subjects decide how to bet in two different scenarios. In first scenario, subjects select from a deck of 20 cards composed of 10 red and 10 blue cards — this is referred to as the risky bet, and, in the second scenario, the deck has 20 red or blue cards, but the composition of the red and blue cards is completely unknown — referred to as the ambiguous bet. A successful bet on a color wins a certain amount of money if a card with the chosen color is drawn. Otherwise the participant loses a certain amount of money.

The details aren't particularly relevant to our discussion; however, when asked to choose between a risky bet and an ambiguous bet, subjects prefer the risky option. What is relevant in terms our understanding the parts of the brain figure most prominently in this choice, is that Xu et al [140] found that two adjacent brain regions in the medial prefrontal cortex (MPFC) conveyed distinct decision signals, specifically, the dorsal MPFC was more activated when individuals experienced higher level of risk. In contrast with this finding, the ventral MPFC was parametrically modulated by the received gain or loss. This sort of study contributes to our understanding of how functional regions play a role in decision making under uncertainty.

If you are interested in how humans make decisions, fMRI studies can provide insight into the role that different brain regions play at different stages in the decision-making process, but it is much harder to drill down to the cellular or molecular level in order to understand what's going on in circuits comprised of individual neurons or in more diffuse signaling pathways such as the neurotransmitter systems originating in the brain stem that control the release of dopamine, seratonin and acetylcholine⁷⁶. Studies that involve mice on the other hand allow much more freedom in studying behavior in general and decision making in particular. Researchers can use optogenetic probes to systematically dissect the neural circuits. For example, there are genetically-engineered — or transgenic — mice that allow photostimulation or photoinhibition of genetically defined populations of neurons, e.g., specific cell types, with high temporal and spatial resolution [2]. As an example of the power of these tools, researchers have discovered a mode of inhibitory control in which inhibitory neurons specifically suppress the firing of other inhibitory neurons thereby inhibiting inhibition and by so doing effectively facilitating exitation [97].

More relevant to our interest in the logical and mathematical thinking that goes into programming, researchers have made progress uncovering the neural basis of numerical cognition. Evidence from human neuroimaging, primate neurophysiology and developmental neuropsychology reveal that humans and animals share a system of approximate numerical processing [30]. Stanislas Dehaene, first author on the cited paper, has written extensively on what he refers to as the number sense [31] and we look at his model of consciousness later in this chapter [32]. Krueger et al [69] builds on the work of Dehaene and his colleagues to provide evidence that a network of regions in the prefrontal cortex and adjacent sulci are implicated in solving abstract mathematical problems exhibits similar patterns of activity. At this point, I think it will help if we take a closer look at the anatomy (structure) and physiology (function) of the human brain and, in particular, the functional regions and how these regions communicate with one another. It's a useful orientation exercise for all of us and in particular for those of us who benefit from visual aids in understanding complicated processes that are physically realized in structural configurations that reveal aspects of their collective function.

2.6  Anatomy and Physiology

The human brain is a highly evolved survival machine. Its purpose is to enable us to survive long enough to reproduce and pass on our genes. Given that it is a product of natural selection forced to rely on its existing design in order to live long enough to improve upon that design under selective pressure, it's not surprising that it is quite unlike the engineered products that humans manufacture. It's also not surprising that human beings have been curious about the brain for as long as we knew we had one and that the language we use to describe the structure and function of the brain reflects the long history of its study. Even the very basic anatomy of the brain is still described using a vocabulary that goes back centuries.

On the one hand, anterior, posterior, superior and inferior are used to identify the front, back, top and bottom of the brain for some purposes. Whereas rostral, caudal, dorsal and ventral are employed to specify the same coordinates for other purposes. Medial can be used to refine location within the brain as a whole or within a specified subcomponent. Terms like telencephalon, diencephalon, etc. describe subdivisions of the embryonic vertebrate brain and can be used to refer collectively to the adult regions they mature into, e.g., the telencephalon develops into the two cerebral hemispheres, including the cerebral cortex and a number of smaller subcortical structures.

In our discussions, we'll largely ignore the fact that the much of the brain and the cortex in particular is divided into two hemispheres with similar structure though somewhat different function. In talking about the cortex, we will often refer to the lobes that comprise each hemisphere, including the occipital lobe near the back of the brain encompassing the early visual system, the temporal lobe near the temple that is responsible for hearing and implicated in language, the parietal lobe including the somatosensory cortex, and the frontal lobe including the primary motor and premotor cortex as well as circuits involved in executive control, problem solving and consciousness.

Of course, the brain is two-dimensional and highly structured. The cortex can be thought of as a sheet of neural tissue consisting of several layers such that each layer plays a different role and often involves different types of neurons in different patterns of connectivity. If you stretched it out flat on a table, the cortex would be about the size of a large dinner napkin, but when folded up to fit within the human cranium it is divided into ridges referred to as gyri — singular gyrus — separated by fissures called sulci — singular sulcus — so that areas that appear near to one another on the surface may be at some distance from one another.

All of this can be enormously confusing to the uninitiated. Fortunately, we won't spend a lot of time talking about anatomical details. Our objective is to design artificial neural network architectures that emulate parts of the brain that humans employ in solving practical problems, and combine these component architectures to design end-to-end systems to assist human engineers and scientists in their research. We aren't interested in slavishly simulating human behavior with all its idiosyncrasies, but rather exploit what we know about the brain in designing these assistants and re-engineer human capabilities when reasonable to do so.

We'll spend a good deal of our time talking about structures in the cerebral cortex including the so-called neocortex in the mammalian brain which is regarded as the most recently evolved part of the cortex. We'll also spend some time talking about the cerebellar cortex generally referred to as the cerebellum, especially with respect to its recently-evolved circuitry complementing that found in the neocortex.

The primate cortex appears to be tiled with columnar structures referred to as cortical columns. Some neuroscientists believe that all of these columns compute the same basic function. However, there is considerable variation in cell type, thickness of the cortical layers, and the size of the dendritic arbors to question this hypothesis. The prefrontal cortex is populated with a type of neuron, called a spindle neuron, similar in some respects to the pyramidal cells found throughout the cortex, that allow rapid communication across the large brains of great apes, elephants, and cetaceans. Spindle neurons are abundant in humans and play an important role in consciousness and attentional networks.

Figure 6 presents a stylized drawing of a human brain shown from the side so that only one hemisphere is visible. The drawing shows the major functional areas of the human cortex. In the following we provide a rough characterization of their function and how they contribute to the realization of human cognition in general and the programmer's apprentice in particular. When necessary, we introduce relevant subcortical circuits such as the basal ganglia and hippocampal nuclei and provide additional anatomical detail as required to explain their function.

Figure 6 shows the four lobes of the primate brain plus several functional areas that will figure prominently in this chapter. Here are several additional terms that appear frequently enough that it makes sense to give each one a short description and convenient abbreviation:

• prefrontal cortex (PFC) including attention, conscious access, reward-based-learning and executive control [134, 68];

• entorhinal-hippocampal complex (EHC) in its role as primary interface between the hippocampus and neocortex [92, 89];

• global workspace (GW) broadly distributed cortical circuits connected through long-range excitatory axons⁷⁷ [33, 4];

• basal ganglia (BG) for its role in action selection and dynamic gating to direct input to the prefrontal cortex[91, 68];

• semantic memory system (SMS) including areas of the brain responsible for mathematical and abstract thought [126, 9];

Your primary peripheral sensory systems, e.g., auditory, visual, touch, etc, tend to produce representations that are similar from one individual to another. The brains of most animals are genetically predisposed to organize themselves using an incredible array of chemotactic and cellular-scaffolding techniques that materials scientists would love to have for building self-assembling nanoscale machines. From a very early stage in fetal development the structures are in place to ensure that the primary auditory and visual areas of the cortex will end up in their respective locations in the temporal and occipital lobes and connect to their respective sensory apparatus.

It makes sense that our genes are programmed to enforce this regularity but it only goes so deep, in part because there's not enough information in our genome to precisely determine every neuron and every synapse in the adult brain, but more so because the body needs some flexibility to deal with unforeseen consequences. For example, there is some evidence to suggest that, if one were to have a mutation resulting in the auditory and visual nerve fibers being reversed, the brain would utilize its plasticity to accommodate and see with neural circuits that were designed to hear and hear with those designed to see [63, 118].

If such a reversal were to happen, the result is certainly not likely to be optimal, but it could mean the difference between living long enough to procreate and dying without passing on your genes. The main point here is that we seem to have enough flexibility in how and what we can represent in our brains to deal with the fact that natural selection hasn't had enough time to redesign our brains so that we can manage to learn how to perform mathematics in our heads and understand quantum electrodynamics well enough to design the next generation of semiconductors. In particular, the further we stray from the periphery, the more distinctive / idiosyncratic our encoding of personal events and multi-modal sensor experiences.

Generally when we use the term "cortex", we mean the cerebral cortex. The control system involving the cerebellar cortex or cerebellum, the cerebral cortex and number of subcortical areas including the basal ganglia is one of the most interesting and neglected systems in the mammalian brain and primates in particular. It is often described in terms of a control theoretic block diagram consisting of separate transfer functions as shown here. Contrary to our original understanding of the cerebellum as primarily involved in orchestrating the motor activity controlling fine movement, we now know that the primate neocortex in concert with recently-evolved augmentation of the cerebellum enable human beings to transfer abstract thinking such as that involved in solving algebraic equations from the cerebral to the cerebellar cortex allowing us to carry out such calculations with significantly greater speed and accuracy. Imagine how an experienced software engineer might take advantage of this capability to facilitate thinking about programs, and then think about how we might develop an architecture for the programmer's apprentice that employs a similar strategy.

2.7  Models of Human Memory

There are two major memory systems in the human brain. One is called explicit, or declarative, memory, and it enables us to remember the people, places, objects and events in our lives. In keeping with its name, we are generally conscious when accessing declarative memory and so can report on what we remember. We distinguish between two types of declarative memory, episodic and semantic, in discussing how different types of memory support cognitive functions⁷⁸.

The second type of memory is implicit, or non-declarative, memory, and serves to encode motor and perceptual skills that we perform automatically. We access non-declarative memory when we perform such skills or practice them to improve our performance. We generally are not conscious of what we are doing when performing these skills. Indeed, attempting to think about what we are doing often impairs performance.

Explicit memories are encoded by the hippocampus, entorhinal cortex and perirhinal cortex, but are consolidated and stored in the temporal cortex and elsewhere. In the case of implicit memory, the prefrontal cortex, parietal cortex, and cerebellum play a role in learning motor skills. The cerebellum is important, as it is required to coordinate sequences of movement required for skilled motion and timing, but also plays a role in cognitive function [13].

We also distinguish between short-term and long-term memory. The former lasts at most a few minutes unless skillfully rehearsed, while the latter can remain available for days, weeks or even years. In the human brain, short-term memory results from strengthening existing connections, while long-term memory results in the growth of new synapses and, in some cases, the birth of new neurons. We discuss several novel ANN memory systems in this chapter.

Human episodic memory is one of the most important elements in determining who we are and how we relate to others. It is critical in reasoning about what's going on in other minds and plays a central role in our social interactions. Not surprisingly episodic memory engages many cortical and sub cortical regions of the primate brain. If one were to pick out a single locus of activity relating to episodic memory it would probably be the entorhinal area corresponding to Brodmann areas 28 and 34 shown here highlighted in yellow. The entorhinal cortex is located in the temporal lobe in close proximity to the hippocampus, but this area is merely the hub of a great deal of activity spread throughout the brain.

The hippocampus is perhaps best known for its role in supporting spatial reasoning. A type of pyramidal neuron called a place cell has been shown to become active when an experimental animal enters an area of a maze that it has visited before. However, the hippocampus plays a much larger role in memory by representing not just the "where" of experience but also the "when". The manner in which we employ short- and long-term memory is very different. We might construct a representation of our current situation in short-term memory, drawing upon our long-term memory to provide detail.

The two memory systems are said to be complementary in that they serve different purposes, one provides a record of the past while the other serves as a scratchpad for planning purposes. In retrieving a memory it is possible that we "corrupt" the long-term memory in the process of subsequent planning. This isn't simply an academic question, it is at the heart of how we learn from experience and employ what we've learned to think about the future. Our memory systems enable us to imagine solutions to problems we have never before encountered, and they account for a good deal of our incredible adaptivity.

2.8  Neural Net Terminology

This chapter is intended to be self contained. For those of you who skipped over the previous chapter, we begin by reviewing the terminology for talking about neural networks, starting with basic units that can be assigned numerical values and are organized in stacks of layers, along with connections between units in adjacent layers that are weighted and stored in tables that represent the strength of the connections.

Connections are said to be directed originating from a unit in one layer and terminating at a unit in another. Units can have more than one outgoing connection as well as multiple incoming connections and generally units in a given layer don't connect to units in the same layer except in the case of recurrent connections where information stored in a unit at time \({t}\) is said to feed back to units at time \({t+1}\).

A neural network model consists of multiple layers arranged in an architecture. Mathematically, the units correspond to scalar variables, the layers to vectors, the tables of connection weights to matrices and the neural network model to a function from inputs — the units in one or more layers that have no incoming connections — to outputs — the units in layers with no outgoing connections.

Though generally discouraged, since the analogy doesn't begin to do justice to the complexity of biological neural networks, practitioners sometimes refer to the units as neurons and the connections as synapses. The connection weights are determined by training a model with data that consists of pairs of input and output assignments, and exploiting the fact that the network functions are differentiable and hence amenable to standard optimization techniques.

The property of being differentiable allows us to adjust all of the weights in a model at once using an objective function that defines what it means to have a good model. We can evaluate how each of the weights in a model contributes to the changes in the objective function when evaluated with respect to one pair of input and output assignments or a batch of such pairs, and then adjust the weights so as to improve the model performance as measured by the objective function.

Training a model involves by randomly selecting individual pairs or batches until we can no longer improve performance. Supervised learning requires labeled data that is often hard to come by. Unsupervised learning doesn't need labels but requires a method of evaluating performance without explicitly being given the correct answer. Later in this chapter, we also explore reinforcement learning in which the model receives intermittent rewards for making progress toward a goal, e.g., winning at chess, and has to figure which of its earlier actions actually contributed to the win, e.g., taking the Queen's knight early in the game.

The basic artificial neural network architecture consists of a hierarchy of specialized networks with a relatively dense collection of feedforward and feedback connections that enable recurrent state, attentional focus and the management of specialized memory systems that persists across different temporal scales. Individual networks are specialized to serve different types of representation, employing convolutional networks, gated-feedback recurrent networks and other network architectures introduced in the previous chapter as components for constructing more complicated architectures.

All of these networks are distributed representations of one sort or another, many of which were developed by the early pioneers in the nascent field of connectionism and featured in the PDP — Parallel Distributed Processing — books that we discussed in the previous class. Typically, they encode information in high-dimensional vector spaces such that different dimensions can be trained to represent different features allowing attentional mechanisms to emphasize or modify encodings so as to alter their meaning in a manner analogous to variable substitution in traditional symbolic systems.

There are a lot of details that engineers have to think about in designing neural network architectures training a model with billions of weights, but this summary account should suffice for understanding the concepts explored in this chapter and you can always return to the previous chapter if you want to dig a little deeper. Part of our goal in developing artificial neural networks is to gain insight into how biological networks — and our brains in particular — actually function.

Semantic Memory

Our objective in developing systems that incorporate characteristics of human intelligence is three fold: First, humans provide a compelling solution to the problem of building intelligent systems that we can use as a basic blueprint and then improve upon. Second, the resulting AI systems are likely to be well suited to developing assistants that complement and extend human intelligence while operating in a manner comprehensible to human understanding. Finally, cognitive and systems neuroscience provide clues to engineers interested in exploiting what we know concerning how humans think about and solve problems. In this appendix, we demonstrate one attempt to concretely realize what we've learned from these disciplines in an architecture constructed from off-the-shelf neural networks.

The programmer's apprentice relies on multiple sources of input, including dialogue in the form of text utterances, visual information from an editor buffer shared by the programmer and apprentice and information from a specially instrumented integrated development environment designed for analyzing, writing and debugging code adapted to interface seamlessly with the apprentice. This input is processed by a collection of neural networks modeled after the primary sensory areas in the primate brain. The outputs of these networks feed into a hierarchy of additional networks corresponding to uni-modal secondary and multi-modal association areas that produce increasingly abstract representations as one ascends the hierarchy — see Figure 7.

2.9  Reinforcement Learning

In the previous chapter we focused on supervised learning depending on labeled data consisting of input-output samples where the objective function is a measure of how well the system can predict the output given the input, or unsupervised learning depending on unlabeled consisting of input-only samples where the objective function is a measure of fitness for a given purpose that applies to any input sample, e.g., the size of the output in bits for the purpose of compressing image data.

In most current applications, all of the learning occurs prior to the use of the fully trained model in a production setting, but this approach is changing as we tackle more complicated problems in dynamic environments that require altering what we learned in light of new information — with or without supervision — and learning entirely new skills and knowledge, possibly with the benefit of intermittent feedback, as we take on new challenges "learning on the job" as it were.

Reinforcement learning is a method of learning that lends itself well to continued learning on the job during which the learning system often only receives feedback upon completing a particular task or achieving a goal and then generally without any detail concerning what exactly the system did to deserve that feedback, as in the case where, having played an entire game of chess, the novice receives little more than is obvious from looking at the game board, i.e., a win, loss or draw.

There are several sources of uncertainty in reinforcement learning that make learning hard. Feedback could be provided precisely when the agent performs a specific action that warrants some approbation, but that is rare. Feedback is sometimes ambiguous or graded using an obscure metric. Understanding feedback — its valence — positive or negative — and weight — just how seriously should one interpret it — requiring a form of sentiment analysis not unlike what is required for ranking movies by reading reviews and comments on web pages. Learning to understand feedback provided in natural language would probably qualify for passing the Turing test.

Another source of uncertainty arises due to the timing of feedback. If the only feedback you get for, say, winning at chess, is at the end of the match when the winner is announced, then it's difficult to understand what you did that warranted praise, sympathy or derision. A reinforcement learning system needs some form of intermediate rewards in order to become a chess champion or talented software engineer. Judging progress in solving complex problems is itself a difficult learning problem and can be characterized as a prediction problem. It's probably instinctual but requires practice and coaching to become good at it.

The objective of reinforcement learning (RL) is to learn a policy that maps from an agent's representation of the current situation or state to the best action to take in that situation. In service to that goal, the agent learns a value function that given a state \({s}\) and action \({\alpha}\) returns a number that represents the expected value (reward) of performing the action in that state. In chess, a state corresponds to an arrangement of pieces on the board and an action to a legal move.

A little more precisely, the neural network for an RL system defines a value function \({\varphi}\) that maps each state-action pair \({(s, \alpha)}\) to a real number \({v}\) where \({s \in{} S}\), \({\alpha\in{} A}\) and \({v \in{} \mathbf{R}}\) and implements a policy \({\pi: S \mapsto{} A}\) such that \({\pi(s) = \alpha}\) for \({\alpha}\) maximizing \({\varphi(\alpha, s)}\).

Realistically, the state of an RL-based chess playing system is likely to be much more than just an arrangement of pieces on chess board. In practice, the state also includes a representation of the moves made in the current game, possibly information relating to the current opponent and likely information relating to previous games and to the systems current stragegy and assessment of strengths and weaknesses. Indeed, the context for any action will include a good deal of information gathered together from multiple memory systems.

This book emphasizes how our understanding of the human brain has informed our efforts to build artificial brains and so, in the following section and later in Section IGNORE, we introduce two examples illustrating how our study of biological neural networks has guided us in designing artificial neural networks.

The example biological networks support capabilities associated with human cognition that are uniquely human, at least in the degree to which we have these capabilities when compared with other animals and the extent to which we have applied them in service to the advancement of human welfare, culture and the pursuit of knowledge.

Action Selection

In this section, we consider the biological basis for action selection and higher order executive control in human beings. In particular we looks at the neural circuits generally considered to be our best guess about how reinforcement learning in implemented in the brain. These circuits assist in formulating and carrying out plans to achieve our goals and solving complex problems that require a high degree of attention, the capacity to hold several concepts in mind at a time and the ability to think about how these concepts relate to one another so as to draw new conclusions.

In each case, we start with a simplified introduction to the relevant anatomy, followed by an explanation of how the biological networks support the target capability. Later, we provide an example artificial neural network that supports a related, if not precisely equivalent capability. The two capabilities that we consider are also noteworthy for the fact that each relies upon subcortical regions that every mammal comes equipped with. However, these functional areas have evolved considerably in humans over the last few million years as they have adapted to take advantage of and support the extended capabilities of the mammalian human neocortex.

In contrast with relative simplicity of the neocortical architecture, the basal ganglia consist of several subcortical nuclei that are related by their evolved function, but not so neatly organized structurally, or at least not to the untrained eye used to human engineered artifacts. In the following, we emphasize and simplify some of those nuclei and ignore others to focus the discussion and simplify the biology. The basal ganglia provide the basis for motor activity controlled by circuits in the brainstem and conserved throughout vertebrate evolution for nearly half a billion years. The cerebral cortex has been around in the form of a six-layer sheet tiled with a repeating columnar structure since the early mammals came on the scene in the Jurassic period about 200 million years ago. Our lineage separated from mice around 100 million and from macaques and other old world monkeys around 25 million years ago. The modern human neocortex owes much to these earlier evolutionary innovations but is different in ways that make possible our facility with language, complex social organization and sophisticated abstract thinking. Compared with the basal ganglia, the neocortex is structurally elegant and functionally general.

The basal ganglia have evolved along with our neocortex to provide us with a powerful thinking machine, while at the same time leaving us to make do with some less-than-ideal adaptations components. We can simulate a conventional computer in our heads but are limited to fewer than a dozen memory registers. Most of us can't perform long division in our heads even though we might know the algorithm and aided with paper and pencil carry out the necessary computations to produce an answer. We rely on the same basic cognitive machinery we use to list a few names in alphabetical order to perform all sorts of more complicated cognitive tasks. Even simpler, however, is the basic operation of choosing one several actions to perform next. The basal ganglia play a key role in supporting action selection and it is worth looking at in a little more detail in order get a handle on some of the parts of the brain that figure prominently in human cognition. Recall that the cortex is a sheet of neural tissue more or less homogeneous in terms of its local structure quite unlike almost any other part of the brain except for the cerebellar cortex. The cortex sits on top of a structure called the thalamus which among other things serves as a relay in passing information back and forth between the cortex and various subcortical nuclei.

The basal ganglia consist of a bunch of circuits, of varied size, sometimes but not always consisting primarily of one cell type, sometimes but not necessarily compactly clustered together, sporting projections that seem to wander off aimlessly, but more or less located above the brainstem and below the cortex. As a general principle, if a signal sets off along some path exiting from a circuit, then expect some derivative of that signal to appear later reentering the circuit to serve as feedback. Everything about the brain, and your entire body for that matter, has to be carefully regulated to maintain a dynamic state of the equilibrium, and unlike human designs, evolution is generally not able to cleanly separate the parts of the circuit that, say, perform some computation from the parts that deal with respiration, immune response, waste removal, cell repair, death and regeneration, etc.

The basal ganglia are depicted in Figure 8.A taking some artistic license to keep things simple. The thalamus along with another structure called the striatum provide the interface between the cortex and basal ganglia. The striatum is a combination of a number of smaller nuclei that are anatomically and functionally related; they include the Globus Pallidus (GP), Putamen and Caudate Nucleus and aside from their function as part of the striatum, only the GP will figure prominently in our discussion and only one part of it — referred to as the internal GP and identified with the "i" subscript to distinguish it from the external part with "e" subscript.

The other players include the Substantia Nigra (SN) which is at one end of the striatum nestled close to the amygdala which is part of the limbic system involved with memory, decision-making and modulating emotional responses, and the Subthalamic Nucleus (STN). You can think of the cortex as integrating sensory and motor information and making suggestions for what action to take next and the amygdala as supplying information pertaining to the possible emotional consequences of taking different actions to be used as input to action selection. Figure 8.B reconfigures these component nuclei into a smaller number of functionally motivated blocks that control two pathways — the direct pathway associated primarily with inhibition and consisting of the internal GP and SN and the indirect pathway playing an excitatory role and consisting of the external GP and STN⁷⁹.

The lines connecting the functional blocks shown in Figure 8.B imply neural connectivity, with arrows indicating the direction of influence and colors indicating the valence of the influence, green for excitatory and red for inhibitory. In the action selection cycle, the cortex forwards activations that you can think of as suggestions for what action to take next. These suggestions are propagated through the striatum and forwarded along the direct pathway where two stages of inhibitory neurons initially suppress all of the suggestions and propagate signals back the cortex to activate inhibitory neurons that suppress activity at the source. As this cycle continues, an additional process takes place in the indirect path — identified with dashed lines — that weighs the advantages and disadvantages of the proposed actions taking in information from throughout the cortex and adjusting the inhibitory bias accordingly.

Eventually, one proposal wins out and all of the others are suppressed allowing a single preferred action to be executed. This cycle of exploring the options for acting and then selecting a single action to execute is constantly repeated during your waking hours. Additional machinery in the thalamus and brain stem regulates whether or not to forward suggestions for acting during sleep when your cortex receives no sensory input and hence any suggestions for acting uninformed by sensory input are ill-advised if not outright dangerous. The above description doesn't begin to convey the complexity of what's going on at the level of individual neurons. Suffice it to say that the usual perfunctory summary consisting of "the winner takes all" doesn't begin to do it justice. The subtleties arise from the way in which the evidence for and against an action proposal is combined, how ties are broken and deciding when enough evaluation is determined sufficient to make a final choice.

2.10  Attentional Networks

While biological and artificial networks can carry out a lot computations in parallel, there are resource limitations and many activities like reading, speaking and controlled movement are typically, if not fundamentally, serial. In the previous chapter, we discussed the concept of attentional networks that learn to identify what part of our recent experience is most important for our current purposes and make that information available to guide subsequent processing.

Attentional networks take as input activation patterns from multiple sources plus additional input summarizing the current context for the task at hand, e.g., coming up with next word or phrase to speak in a conversation. Based on prior training for the task, the network computes a weighted sum of the input activation patterns and combines it with the context summary to produce an enriched context, e.g., incorporating an earlier comment from the same conversation, in which to decide what do do.

The applications we consider in this chapter require a combination of intensive early training followed by a lifetime of learning on the job. The early training can be thought of in terms of naturally paced development stages from prenatal, through early childhood and extending into early adulthood, along with targetted curriculum-based learning in kindergarten through high school combined with extensive extracurricular training in effective communication and social interaction.

These skills require several specialized memory systems and those memory systems rely on specialized mechanisms for focussing attention and learning what to attend to, and so to start and keep our promise to make this chapter self contained, we review the relevant details and elaborate on the discussion of attention in the previous chapter. While any number of weighty tomes have been written on the subject, we emphasize a general mechanism and focus on its simple neural-network realization.

Start with a multilayer network having two or more separate input layers that represent different information sources originating in the sensory cortex or one of several short- or long-term memory systems, along with an input layer that represents the context for the deliberative function this attentional system serves. For example, the context might represent the state of a conversation and the input sources to memories that might enrich the context with a name, clever comment or change of topic.

Train the network to do a good job of improving conversations by simply rewarding the system if the conversation continues in a satisfying manner. Now that last bit about "conversations continuing in a satisfying manner" might seem a bit of a stretch, but I'm assuming that learning how to carry on a conversation doesn't become important until long after you've learned how to read other people well enough to determine whether or not they are pleased with you. Babies are innately able to do this.

And so, while understanding the neural correlates for, say human visual attention, is still an active area of research important in understanding human cognition, software engineers implementing an attentional component of a larger network needn't concern themselves with the details of how attention is handled in humans, and need only to design a network architecture that accommodates the particular set of inputs and outputs required of the application at hand — see Figure 5.

Long Short-Term Memory (LSTM) models were one of the first really practical recurrent neural-network architectures. Now there are quite a few variations including Gated-Feedback Recurrent Networks (GF-RNN) and convolutional neural networks have been shown to work as well as recurrent models for some problems. There have been several important extensions to RNN models that have considerably extended their range. Here we mention three extensions that are applicable to the programmer's apprentice. All three of these extensions exploit attention in some manner, and they can be combined to design even more capable architectures.

The basic Attentional Interface that we've already encountered allows an RNN model to selectively focus on parts of its input. Neural Turing Machine (NTM) — also called Differentiable Neural Computer (DNC) — models employ external dynamic memory that can be read from and written to under attentional control. Finally, Neural Programmer models learn how to write programs as a means of accomplishing a target task. These programs need not be conventional computer programs and learning to program doesn't require examples of correct programs since the objective function depends on the target task and gradient descent does the rest.

Single-modality sensory information feeds into multi-modal association areas to create rich abstract representations. Attentional networks in the prefrontal cortex take as input activations occurring throughout the posterior cortex. These networks are trained by reinforcement learning to identify areas worth attending to and the resulting policy selects areas to attend to and sustain. This attentional process is guided by a prior that prefers low-dimensional thought vectors corresponding to statements about the world that are either true, highly probable or very useful for making decisions. Humans can sustain only a few such activations at a time. The apprentice need not be so constrained.

To understand attentional networks, think about an encoder-decoder network for machine translation. As the encoder digests each word in the sequence of words that constitute the input sentence, it produces a representation — Geoff Hinton refers to these as thought clouds in analogy to the iconic clouds that you see in comic strips — of the sentence fragment or prefix that it has seen so far. Because the sentence is ingested one word at a time — generally proceeding from left to right — the resulting thought cloud will tend to emphasize the meaning of the most recently ingested words in each prefix.

You could encode the entire input sentence and then pass the resulting representation on to the decoder, but earlier words in the sentence will receive less attention that later words. Alternatively, you could introduce a new network layer that takes as input all of the encodings of the sentence prefixes seen so far and trains the new layer — thereby taking advantage of the power of gradient descent — to produce a composite representation that emphasizes those parts of the input that are most relevant in decoding / generating the next word in the output. Check out this tutorial for more detail.

Conscious Memory

Stanislas Dehaene and his colleagues at the Collège de France in Paris developed a computational model of consciousness that provides a practical framework for thinking about consciousness that is sufficiently detailed for much of what an engineer might care about in designing digital assistants [32]. Dehaene’s work extends the Global Workspace Theory of Bernard Baars [4]. Dehaene’s version of the theory combined with Yoshua Bengio’s concept of a consciousness prior and deep reinforcement learning [81, 84] suggest a model for constructing and maintaining the cognitive states that arise and persist during complex problem solving [6].

Global Workspace Theory accounts for both conscious and unconscious thought with the primary distinction for our purpose being that the former has been selected for attention and the latter has not been so selected. Sensory data arrives at the periphery of the organism. The data is initially processed in the primary sensory areas located in posterior cortex, propagates forward and is further processed in increasingly-abstract multi-modal association areas. Even as information flows forward toward the front of the brain, the results of abstract computations performed in the association areas are fed back toward the primary sensory cortex. This basic pattern of activity is common in all mammals.

The human brain has evolved to handle language. In particular, humans have a large frontal cortex that includes machinery responsible for conscious awareness and that depends on an extensive network of specialized neurons called spindle cells that span a large portion of the posterior cortex allowing circuits in the frontal cortex to sense relevant activity throughout this area and then manage this activity by creating and maintaining the persistent state vectors that are necessary when inventing extended narratives or working on complex problems that require juggling many component concepts at once. Figure 10 suggests a neural architecture combining the idea of a global workspace with that of an attentional system for identifying relevant input.

In Dehaene's theory, information about the external world enters the cortex through the thalamus and, with input from subcortical structures responsible for arousal and sleep, generates a signal marking the information available for examination. The raw sensory input is propagated to higher-level processing systems eventually reaching the prefrontal cortex where it is evaluated and selectively made available in short-term memory. The computations required for evaluation and selection can be modeled by attentional networks initialized with a prior that selects for suitably abstract features that maximize executive performance.

The global workspace enables us to construct and maintain a complex composite representation in a serial fashion guided by attention. This can be accomplished using a form of persistent memory enabled by fast weights — originally referred to as dynamic links in the work of Christoph von der Malsburg — to manage a working memory in which information can be added, modified and removed, that supports associative retrieval and is implemented as a fully differentiable model and so can be trained end-to-end by gradient descent. Fast weights are used to store temporary memories of the recent past and provide a neurally plausible method of implementing the type of attention to the past that has recently proved helpful in sequence-to-sequence models. In Section IGNORE, we study how this capability plays a key role in long-term / episodic memory.

You can think of the episodic memory encoded in the hippocampus and entorhinal cortex as RAM and the actively maintained memories in the prefrontal cortex as the contents of registers in a conventional von Neumann architecture. Since the activated memories have different temporal characteristics and functional relationships with the contents of the global workspace, we implement them as two separate NTM memory systems each with its own special-purpose controller.

The cortical sheet, in addition to its layered structure, appears to be tiled with columnar structures referred to as cortical columns. Some neuroscientists believe that all of these columns compute the same basic function. However, there is considerable variation in cell type, thickness of the cortical layers, and the size of the dendritic arbors to question this hypothesis. The prefrontal cortex is populated with a type of neuron, called a spindle neuron, similar in some respects to the pyramidal cells found throughout the cortex, that allow rapid communication across the large brains of great apes, elephants, and cetaceans. Although rare in comparison to other neurons, spindle neurons are abundant and quite large in humans and apparently play an important role in consciousness and attentional networks.

These attentional networks are connected to regions throughout the cortex and are trained via reinforcement learning to recognize events worth attending to according to the learned value function. Using extensive networks of connections — both incoming and outgoing, attentional networks are able to create a composite representation of the current situation that can serve a wide range of executive cognitive functions, including decision making and imagining possible futures. The basic idea of a neural network trained to attend to relevant parts of the input is key to a number of the systems that we'll be looking at.

2.11  Machine Memory Models

In the artificial neural networks discussed in the previous chapter, short-term memories reside for seconds in the transient activation patterns of the units in hidden layers and for longer — indefinitely if actively maintained — in the case of recurrent networks. Long term memories are encoded in the connection-weight matrices of each layer as a result of batch training with stochastic gradient descent.

In this chapter, we discuss artificial neural network components that borrow from both biological and conventional computing technologies. These hybrid memory architectures have homologous structures in biological systems. However, they were not designed for the purpose and hence are ill-suited to many of the tasks we attempt to apply them to, e.g., remembering lists of more than a few words or doing long division in your head [48].

This section is about machine models of memory. However, this book is about how our understanding of the human brain has informed our efforts to build artificial brains and so, here, following up on the example presented in Section IGNORE, we introduce the second of our two examples illustrating how our study of biological neural networks has guided us in designing artificial neural networks. This example involves our ability to encode vivid memories of our experience and recall specific memories that help us to imagine alternative futures to solve our immediate problems and plan for the future.

Episodic Memory

Figure 11 provides a glimpse of how we construct memories of our experience and subsequently retrieve those memories to support a diverse range of cognitive strategies. In this case, the hypothalamus will play a central role as did the basal ganglia in the previous example. Later in this book we explore how the basal ganglia work in concert with the hypothalamus to support reinforcement learning. For now, our goal is simply to describe the process whereby we consolidate and then encode experience. In doing so we will also take the opportunity to talk about the process whereby we retrieve memories, reconstruct a version of that past experience to perform counterfactual inference and imagine possibilities that we have never actually experienced.

The name hippocampus like so many biological terms has obscure origins, generally in Latin or Greek and in this case the latter, relating to its shape that looked like a seahorse to some early anatomists. As shown in Figure 11.A it is primarily comprised of four subnuclei referred to as CA1, CA2, CA3 and CA4, the first two characters in each abbreviation recalling a previous Latin name, Cornu Ammanonis associated with a ram's horn, apparently preferred by even earlier anatomists. These nuclei are capped by the dentate gyrus (DG) at one end and the fornix at the other⁸⁰.

The hippocampus consists of two nearly identical structures, one in each hemisphere, connected where the parallel tracts of the fornix come together at the midline of the brain. The hippocampus is tightly coupled with the entorhinal cortex (EHC) that plays an important role in memory, navigation and our perception of time. Information flows from the EHC to the hippocampus by one of two pathways: either through the DG to CA3 or via reciprocal connections to and from CA1. The EHC also has reciprocal connections to many cortical areas⁸¹.

In the process of creating a new memory, the hippocampus receives input from multiple cortical areas relevant to current experience, consolidates this information in a condensed format that will enable subsequent retrieval and stores the resulting encoding in memory. In retrieving an existing memory, The EHC starts with cortical activity, typically from motor and sensory association areas, and uses this information to reconstruct a previous memory by activating cortical areas corresponding to the original memory. Before describing how we think such creative consolidation and subsequent reconstruction works, a word about why this process is beneficial might be in order.

Almost every stage of memory is fraught with opportunities to distort the truth. Reconstruction is a creative process in which we are more often than not forced to fill in some details that we might think we observed at the time but actually didn't. In the formation of new memories, consolidation can only make do with whatever information about the experience we have gleaned from observation and committed to short-term memory. If you don't rehearse what you've stored in short-term memory then it will quickly fade, losing detail and potentially introducing errors of omission and commission.

You might purposely or unconsciously alter what you think you remember to conveniently leave the impression that you behaved better than you actually did. There are all kinds of reasons to alter your memories to suit your purposes just as there all sorts of reasons for lying to ourselves [125]. Self-deception can offer interpersonal benefits that offset its costs in whatever currency you value most. There is evidence to suggest that people lie to convince themselves of the truth of their persuasive goal, and by doing so, are able to argue their case more persuasively to others [115].

This sort of imaginative reconstruction has called into question the accuracy of first-person accounts and expert witnesses. But it also explains our eminently useful commonsense application of believable counterfactual reasoning, e.g., if I hadn't had that second cup of coffee, I would have slept better last night [94, 26]. And, considered in the context of avoiding dangerous situations, planning for contingencies, or imagining a world very different from the one in which we live, one that can inspire and motivate us to act to achieve such a future, it underscores the amazing strength, resilience and determination of the human mind.

The basic mechanism employed by the hippocampus and entorhinal cortex working together is illustrated in Figure 11.B. There are two basic processes that we consider here: encoding new memories and retrieving of old memories. Encoding involves collecting information gleaned from diverse neural activity originating in multiple cortical regions and consolidating [82] this information to construct a compact encoding that serves as a key or index that will enable subsequent stable — meaning reliably consistent even in the presence of distracting information [36] — retrieval and reconstruction⁸².

EHC receives input from all cortical regions in a condensed form and the axons of EHC pyramidal neurons project primarily to the DG but also to CA1. DG then projects to CA3 which plays a particularly important role in encoding and retrieving memories. CA3 is thought to behave as an autoassociative memory shown here as a recursive neural network. The crucial property of an autoassociative memory is that it is able to retrieve an item from memory using only a portion of the information associated with that item⁸³.

For example, this property allows us to retrieve information about a party that was held years ago just by catching a glimpse of a person on the street who happens to look like someone whom we first met at that party. You can think of your observation of the person from the party as a probe or key that enables you to access and unlock that earlier experience. Of course, you could have simply mistaken that person you saw on the street for the person you met at the party.

Hippocampal indexing theory [123] hypothesizes that when we have a conscious experience, different areas of the cortex are activated related to that experience such as activity in the auditory cortex resulting from our listening to a recording of Yo-Yo Ma playing Bach or activity in the visual cortex produced by a particularly vivid sunset we witnessing. When we remember the experience later, similar areas of the cortex are reactivated allowing us to re-experience the event.

The hippocampus serves as an index, storing different patterns of neocortex activity and allowing us to retrieve our memories using only a fragment of what we can recall. The recurrent connections of CA3 are thought to enable this sort of creative reconstruction and play a role in both encoding and retrieving memories. CA3 then projects to CA1 and from there back to the EHC completing the loop and thereby providing recurrent activity involving a much larger circuit.

There are a couple of details that are worth pointing out here as they demonstrate both the strengths and weaknesses of human episodic memory. The first concerns the issue of retrieving a complete memory given a partial index and the second concerns how to retrieve a memory when that memory is similar to one or more other memories, at least in the sense that their respective indices are similar to one another. In the model described here, the first issue is handled by the autoassociative network.

The triptych shown in Figure 12 illustrates how the autoassociative network solves this problem. The panel on the far left is meant to indicate the autoassociative network and its initial state. In the middle panel, we assume the input from the dentate gyrus consist of the two sub patterns A and C and illustrates the reciprocal connections that would be strengthened were we to train the network with this composite pattern of activity. CA3 is responsible for encoding these memory specific patterns of activity for all of our memories. The panel on the far right is intended to illustrate how given a partial pattern, in this case just one of the two representative patterns that comprise the composite pattern shown in the middle panel, is able to reconstruct the other representative pattern by using the trained autoassociative network to first identify and then strengthen the connections in the original composite.

The second detail concerns the possibility that the encodings for two memories are alike enough to be mistaken with one another. A full account of any of the theories explaining how the human brain solves this problem is beyond the scope of what we can go into here but one theory — first articulated by David Marr [73, 139] — posits that, since the dentate gyrus has a larger number of cells than the EHC, its forward projection will tend to produces an expansion recoding in the DG leading to an increase in the separation between the patterns in CA3. Evidence from recordings in mice and computational modeling using artificial neural network and dynamical system models offer support for the theory [88].

To complete our account of memory retrieval, we look at how the path that started in the EHC loops back to complete a feedback loop that stabilizes the encoding of memories. So far we've seen how an experience represented by a pattern of activations in the cortex is compressed and represented in the entorhinal cortex which projects this pattern onto the cells in the dentate gyrus thereby increasing the separation between competing patterns the results of which are bound together to generate an index. This index is fed to CA3 where it is incorporated in an autoassociative recursive network so that subsequently when a feature of the original stimulus is present in our conscious experience it activates a subset of the original neurons activated in CA3 and the recurrent connections in the autoassociative network reactivate the remaining neurons completing the pattern that was incorporated when the experience was initially encoded in memory.

The remaining step involves explaining how the representation in CA3 reactivates the original stimulus. As shown in Figure 11.B, the entorhinal cortex projects to CA1 in addition to the dentate gyrus. When neurons are projected forward to DG and activated in CA3 they are also activated in CA. Since they are activated at the same time, the connections between the neurons in CA3 and CA1 are strengthened by long-term potentiation⁸⁴. The result is a stable, sparse, invertible mapping that allows the hippocampus to recreate the original cortical activity patterns during retrieval [92, 78]. Reactivating the same combination of cortical areas as the original stimulus and causing us to reexperience the event as a memory. While still speculative, an additional process called reconsolidation is thought to allow previously-consolidated memories become labile again as a consequence of reactivation⁸⁵.

Neural Turing Machines

For the most part, human memory and the artificial neural networks developed to emulate human cognition are based on the sort of distributed representations that are the hallmark of connectionist models. This sort of memory has a number of characteristics that complement the sort of memory employed in conventional computing hardware and software. Memory in connectionist models is contextual in terms of how it encodes meaning and fluid in terms of allowing us to represent concepts with flexible, semi-permeable boundaries. Memory in conventional computing hardware is said to be compositional — complex representations can be composed of other complex representations or simpler ones and these compositions behave in a straightforward manner in terms of not altering the meaning of the constituents — and systematic — exhibit a clear separation between form and content / syntax and semantics⁸⁶.

Conventional computing is well suited to maniplating symbolic representations such as word sequences, documents and highly structured data such as labeled graphs and tables with named rows and columns. Conventional computing technologies include high-performance hardware for calculations involving logic and mathematics, and high-capacity memory supporting low-latency reads and writes. Conventional and connectionist computing models complement one another, and, while humans are able to perform symbolic reasoning by essentially simulating it on a connectionist architecture, the inference is slow and error-prone. To compensate, neural network researchers have developed hybrid computing models that provide the benefits of conventional computing models, by employing differentiable interfaces compatible with connectionist architectures that can be trained to serve different purposes.

Neural Turing Machines (NTM) — also known as Differentiable Neural Computers (DNC) — can be added to connectionist models to provide conventional computing and memory services [48, 47]. Here we describe one such component supporting content-addressable random-access key-value memory. In a traditional key-value memory, entries are stored in a table as pairs (key, value). To add a new entry you simply insert a new entry in the table, if there already exists an entry with the same key, then the system either signals an error or updates the value in the existing entry. To look up a value given a query key you simply find the corresponding entry and return the value or signal an error if there is no entry with the given key. This is not quite the same as conventional random access memory (RAM) since the keys need not be ordered in memory as they are in conventional RAM using sequential binary addresses.

The keys in a conventional key-value memory are typically strings, symbols or integers, none which is much use in a connectionist model. As one might expect, DNC keys and values correspond to activation patterns in network layers and so are realized as vectors of real numbers. Considered as a function that given a key returns a value, traditional key-value memory is not differentiable and hence the corresponding DNC interface function can't be learned by an artificial neural network. First of all, given that the keys are real valued vectors the probability that exactly the same vector will be computed and used as a key is vanishingly small. We can compensate for this by finding the entry with the key closest to the query key, where by closest we mean the closest in the corresponding vector space typically determined by the obvious \({n}\)-dimensional generalization of the Euclidean distance metric.

Unfortunately picking the entry with the closest key is still a non-differentiable function, since given a small change in the query key the corresponding closest entry could change, resulting in a discontinuous change in the value returned by the function. To remedy this problem, we use the same trick that we employed in the previous chapter by using the continuous softmax function as an alternative to the conventional discontinuous max function. What this means in the case of a differentiable key-value memory, is that we compute the distance from the query key to each key corresponding to an entry in memory, we then normalize the distances so that each normalized distance is a number between 0 and 1 and together they sum to 1, resulting in a vector we can interpret as a probability distribution. Finally, we compute the weighted sum of all the values in the entries stored in the memory, where the weights correspond to the normalized distances.

The keys and values are all generated by the neural network implementing the target application. Exactly how the keys and values are distributed in their corresponding vector spaces is largely determined by the objective function and the training data. Recall from the previous chapter that each layer constitutes a separate vector space and the learned weight matrix associated with the connections between adjacent layers composed with the corresponding activation function defines a transformation from one vector space to another. The only model parameters that serve to influence the associated vector spaces are those that determine the number of units in each layer and so the dimensionality of each vector space. The dictates of the problem — implicit in the objective function and the training data — will determine how the values are distributed in the corresponding vector spaces.

The vectors corresponding to the activation patterns in a given layer could be sparsely distributed and hence well separated and unlikely to be confused with one another, or they could be clumped with each clump corresponding to some class of roughly similar vectors. The keys will have their related distributional characteristics. The point is that these distributions will be shaped by the objective function and the data and so we don't have to constrain them any further — indeed we would be foolish to try to second guess training. The original paper describing the DNC model notes that their model exhibits usage patterns that exhibit parallels between the DNC and mammalian hippocampus⁸⁷, and we explore this connection in Section IGNORE on the neural correlates of episodic memory.

The learnable weights that govern the behavior of a DNC model are organized in a controller module illustrated in Figure 13. The models shown in (A) and (B) have essentially the same controller but differ in terms of the conventional memory component. The model labeled (A) has a fixed-size memory of the sort you might find in a microprocessor central processing unit (CPU) consisting of a set of registers used to temporarily store operands when executing machine instructions. The model labeled (B) has an arbitrarily large memory such as that provided by a conventional electromechanical hard disk drive or solid-state RAM disk. The controller uses three differentiable attention networks to define distributions over the \({N}\) rows or locations in the \({N \times{} W}\) memory matrix \({M}\).

The attention networks are used to generate location addresses to control a read head and a write head borrowing from the terminology of conventional disk drives. For some applications, a DNC model can be configured to operate multiple read and write heads to facilitate parallel reads and writes. The first attention network assists with content addressing for reads and writes by using the method described earlier in which reads and writes are distributed over multiple locations depending on their similarity to the key vector emitted by the controller. The second attention network assists in traversing consecutively written locations and the third allocates memory locations for writing. Together these three attention networks plus controller provide a versatile alternative to using recurrent networks for maintaining state information.

Episodic Memory

Fundamental to our understanding of human cognition is the essential tradeoff between fast, highly-parallel, context-sensitive, distributed connectionist-style computations and slow, serial, systematic, combinatorial symbolic computations. In developing the programmer's apprentice, symbolic computations of the sort common in conventional computing are realized using extensions that provide a differentiable interface to conventional memory and information processing hardware and software. Such interfaces include the Neural Turing Machine [47] (NTM), Memory Network Model [138, 116] and Differentiable Neural Computer [48] (DNC).

The global workspace summarizes recent experience in terms of sensory input, its integration, abstraction and inferred relevance to the context in which the underlying information was acquired. To exploit the knowledge encapsulated in such experience, the apprentice must identify and make available relevant experience. The apprentice’s experiential knowledge is encoded as tuples in a Neural Turing Machine (NTM) memory that supports associative recall. We’ll ignore the details of the encoding process to focus on how episodic memory is organized, searched and applied to solving problems.

In the biological analog of an NTM the hippocampus and entorhinal region of the frontal cortex play the role of episodic memory and several subcortical circuits including the basal ganglia comprise the controller [91, 89]. The controller employs associative keys in the form of low-dimensional vectors generated from activations highlighted in the global workspace to access related memories that are then actively maintained in the prefrontal cortex and serve to bias processing throughout the brain but particularly in those circuits highlighted in the global workspace. Figure 14 provides a sketch of how this is accomplished in the apprentice architecture.

Figure 15 combines the components that we've introduced so far in a single neural network architecture. The empty box on the far right includes both the language processing and dialogue management systems as well the networks that interface with FIDE and the other components involved in code synthesis. There are several classes of programming tasks that we might tackle in order to show off the apprentice, including commenting, extending, refactoring and repairing programs. We could focus on functional languages like Scheme or Haskell, strongly typed languages like Pascal and Java or domain specific languages like HTML or SQL.

However, rather than emphasize any particular programming language or task, in the remainder of this appendix we focus on how one might represent structured programs consisting of one or more procedures in a distributed connectionist framework so as to exploit the advantages of this computational paradigm. We believe the highly-parallel, contextual, connectionist computations that dominate in human information processing will complement the primarily-serial, combinatorial, symbolic computations that characterize conventional information processing and will have a considerable positive impact on the development of practical automatic programming methods.

2.12  Embodied Cognition

There is a very real sense in which the apprentice is embodied in the same sense the Deep Recurrent Attentive Writer neural network architecture — the acronym is DRAW — developed by researchers at DeepMind embodies a digital slate and drawing program. DRAW networks employ a spatial attention mechanism that mimics foveation of the human eye, allowing it to iteratively construct complex images. The browser interface serves as a special type of prostheses enabling the assistant to interface more directly with the programmer by directly apprehending every key stroke and mouse movement initiated by the programmer.

The most efficient mode of interaction between the apprentice and programmer is likely to be through the use of natural language. Existing personal assistants are not particularly talented at continuous dialogue. The job of the apprentice is simplified somewhat by the relatively narrow scope of the task of programming. That said, verbal communication is complicated by the inherent ambiguity of natural language, the importance of reading another person's body language and facial cues, the subtlety required to notice when our conversational partner is confused and then easily recover from misunderstanding, and, finally, the fact that human beings share a great deal of basic knowledge and common sense.

We imagine the programmer's apprentice with a body part consisting of an instrumented integrated development environment (IDE). Alternatively you might think of it as a prosthetic device. It is not, however, something that you can simply remove or replace with an alternative device outfitted with a different interface or supporting different functions. Like the legs you were born with or the prosthesis replacing an amputee's severed arm, you have to learn how to use these devices.

Architecturally, the apprentice's prosthetic IDE is an instance of a differentiable neural computer (DNC) introduced by Alex Graves and his colleagues at DeepMind. The assistant combined with its prosthetic IDE is neural network that can read from and write to an external memory matrix, combining the characteristics of a random-access memory and set of memory-mapped device drivers and programmable interrupt controllers. The interface supports a fixed number of commands and channels that provide feedback. You can think of it as roughly similar to an Atari game console.

The spatial relationships among the ganglion cells in the retina are preserved in the activity of neurons found in the primary visual — or striate — cortex. Most sensory and motor areas maintain similar modality-specific topographic relationships. Shown here, for example, are Wilder Penfield's famous motor and somatosensory homunculi depicting the areas and proportions of the human brain dedicated to processing motor and sensory functions. Scientists have observed that the area devoted to the hands tend to be larger among pianists, while the relevant areas in the brains of amputees typically become significantly smaller.

In the mammalian brain, information pertaining to sensing and motor control is topographically mapped to reflect the intrinsic structure of that information required for interpretation. This was early recognized in the work of Hubel and Wiesel on the visual cortex and Wilder Penfield in his work developing the notion of a cortical homunculus in the primary motor and somatosensory areas of the cortex as shown here. Such maps relate to the idea of embodied intelligence.

The integrated development environment and its associated software engineering tools constitute an extension of the apprentice’s capabilities in much the same way that a piano or violin extends a musician. The extension becomes an integral part of the person possessing it and over time their brain creates a topographic map that facilitates interacting with the extension.

As engineers designing the apprentice, part of our job is to create tools that enable the apprentice to learn its trade and eventually become an expert. Conventional, IDE tools simplify the job of software engineers in designing software. The IDE that we build for the apprentice (FIDE) will be integrated into the apprentice’s cognitive architecture so that tasks like stepping a debugger or setting breakpoints are as easy for the apprentice as balancing parentheses and checking for spelling errors in a text editor is for us.

Procedure Model

The integrated development environment and its associated software engineering tools constitute an extension of the apprentice’s capabilities in much the same way that a piano or violin extends a musician or a prosthetic limb extends someone who has lost an arm or leg. The extension becomes an integral part of the person possessing it and over time their brain creates a topographic map that facilitates interacting with the extension⁸⁸.

As engineers designing the apprentice, part of our job is to create tools that enable the apprentice to learn its trade and eventually become an expert. Conventional IDE tools simplify the job of software engineers in designing software. The fully instrumented IDE (FIDE) that we engineer for the apprentice will be integrated into the apprentice’s cognitive architecture so that tasks like stepping a debugger or setting breakpoints are as easy for the apprentice as balancing parentheses and checking for spelling errors in a text editor is for us.

As a first step in simplifying the use of FIDE for coding, the apprentice is designed to manipulate programs as abstract syntax trees (AST) and easily move back and forth between the AST representation and the original source code in collaborating with the programmer. Both the apprentice and the programmer can modify or make references to text appearing in the FIDE window by pointing to items or highlighting regions of the source code. The text and AST versions of the programs represented in the FIDE are automatically synchronized so that the program under development is forced to adhere to certain syntactic invariants.

To support this hypothesis, we are developing distributed representations for programs that enable the apprentice to efficiently search for solutions to programming problems by allowing the apprentice to easily move back and forth between the two paradigms, exploiting both conventional approaches to program synthesis and recent work on machine learning and inference in artificial neural networks. Neural Turing Machines coupled with reinforcement learning are capable of learning simple programs. We are interested in representing structured programs expressed in modern programming languages. Our approach is to alter the NTM controller and impose additional structure on the NTM memory designed to support procedural abstraction.

Execution Model

What could we do with such a representation? It is important to understand why we don’t work with some intermediate representation like bytecodes. By working in the target programming language, we can take advantage of both the abstractions afforded by the language and the expert knowledge of the programmer about how to exploit those abstractions. The apprentice is bootstrapped with several statistical language models: one trained on a natural language corpus and the other on a large code repository. Using these resources and the means of representing and manipulating program embeddings, we intend to train the apprentice to predict the next expression in a partially constructed program by using a variant of imagination-based planning [95]. As another example, we will attempt to leverage NLP methods to generate proposals for substituting one program fragment for another as the basis for code completion.

The Differentiable Neural Program (DNP) representation and associated NTM controller for managing the call stack and single-stepping through such programs allow us to exploit the advantages of distributed vector representations to predict the next statement in a program under construction. This model makes it easy to take advantage of supplied natural language descriptions and example input / output pairs plus incorporate semantic information in the form of execution traces generated by utilizing the FIDE to evaluate each statement and encoding information about local variables on the stack.

Embodied Brains

There are science fiction books and movies about brains separated from bodies that are artificially kept alive and are able to interact with the world through various brain-computer interfaces. The interfaces allow these disembodied brains to apprehend what's going on around them and possibly control robotic arms to touch, feel and manipulate physical objects. There are also patients with a medical condition called locked in syndrome resulting from a stroke that damages part of the brain stem so that the body, including most of the facial muscles, is paralyzed but consciousness remains and the ability to perform certain eye movements is preserved.

It is hard to imagine what it would be like not to have a body or to have a body but not feel it. The fictional accounts of brains without bodies give the impression that, assuming form of advanced medicine, brains could carry on quite happily relieved of their bodies. There is no insurmountable technical barrier to keeping a brain alive on life support indefinitely. However the human body and brain are so interdependent that to have never experienced what it's like to have one would result in our being as much different from other humans as naturally embodied humans are different from dolphins, whales, elephants or any other intelligent species.

The physical basis that determines how we think extends well beyond the central nervous system. What you see, hear and feel, the way in which you express yourself, how you deal with challenging situations and relate to others are inextricably dependent on the many chemical, electrical and genomic pathways that govern how you feel and filter what you perceive. Your interaction with other others is largely determined by factors beyond your control including not only your behavior and current physical state but a much larger ensemble of social and emotional factors that create the context for everything that you experience [66, 128].

One important consequence of how your body and the environment that you inhabit influence your thinking is that your experience serves to ground your use of language. The meaning of visual signs and spoken words are determined by what they refer to or signify. While you can introduce several steps of indirection, eventually signs and words are grounded in experience and the richer that experience the richer your repertoire of analogies, abstractions, relationships, etc. Being able to ground concepts in a physically coherent reality allows you to draw upon your experience to check what you think against a ruthlessly honest account of what is really possible.

This ability to test our naïve theories against the physics of the real world provides a self-correcting feedback loop — a recurrent process that feeds back on itself. The system composed of the brain, body and environment is common in nature and fundamental in the field of cybernetics — the study of communication and control in animals and machines. Human beings routinely extend their reach by creating self-consistent physical systems as well as abstract theoretical systems governed by the uncompromising laws of logic and mathematics. Human language is permeated with analogies that provide additional means for extending and checking our theories.

The same principle applies in communication. Dialogue is not a sequence of unrelated utterances, but rather it is a collaborative act in which speaker and hearer must constantly establish common ground to make it clear that the hearer has understood the speaker's meaning and intention⁸⁹ . This is challenging when speaker and hearer have different grounding but made possible through analogy. The programmer tells the assistant that he has to use the toilet. Puzzled, the assistant asks, "What is a toilet?" After some tortured back and forth, the assistant proudly says, "That's like when my program uses up all the available memory and I have to run my garbage collection routine to free up some space."

Einstein grounded his nascent theory of light and gravity in terms of trains traveling along parallel tracks. He had to make leap beyond his physical experience, but thinking about trains provided a vantage point from which it was possible to make the leap to curved spacetime. Our experience of physically interacting with the world around us provides us a system we can use to draw commonsense conclusions about the physics that governs our world. If two people inhabit very different environments they can share their experience through analogy. Language is the most powerful technology humans invented and, while language is not necessary to survive, it enables us to rise above mere existence.

The programmer's assistant lacks a physical body but is immersed in several engineered systems having their own internal consistency and sources of inspiration. First of all, the assistant is constantly engaged in conversation with the programmer with whom it shares the context of their joint efforts to produce software. In addition, the assistant lives in a world of code (software) and computing hardware. The assistant can think and communicate in programming language, execute lines of code as you would swat a fly or scratch an itch and interact directly with running code as though the hardware and the programming tools that run on the hardware are natural extensions of its mind.

As engineers designing the apprentice, part of our job is to create tools that enable the apprentice to learn its trade and eventually become an expert. Conventional integrated development environment (IDE) tools simplify the job of software engineers in designing software. The fully instrumented IDE (FIDE) that we engineer for the apprentice is integrated into the apprentice's cognitive architecture so that tasks like stepping a program in a debugger or setting breakpoints are as easy for the apprentice as balancing parentheses and checking for spelling errors in a text editor is for us. The FIDE is both cognitive prosthesis and gateway into a world the assistant inhabits.

2.13  Planning and Behaving

The basal ganglia in cognitive models such as the one described by Randall O'Reilly's in his presentation in class, play a central role in action selection. This seems like a good opportunity to review how actions are represented in deep-neural-network implementations of reinforcement learning. Returning to our default representation for the simplest sort of episodic memory, \({(s_{t}, a_{t}, r_{t}, s_{t+1})}\), it’s easy to think of a state \({s}\) as a vector \({s \in{} \mathbf{R}^{n}}\) and a reward \({r}\) as a scalar value, \({r \in{} \mathbf{R}}\), but how are actions represented?

Most approaches to deep reinforcement learning employ a tabular model of the policy implying a finite — and generally rather small — repertoire of actions. For example, most of the experiments described in Wayne et al [136] (MERLIN) six-dimensional one-hot binary vector that maps a set of six actions: move forward, move backward, rotate left with rotation rate of 30, rotate right with rotation rate of 30, move forward and turn left, move forward and turn right. The action space for the grid-world problems described in Rabinowitz et al [101] (ToMnets) consists of four movement actions: up, down, left, right and stay.

Rather than immediately focusing on how biological and artificial brains learn and apply hierarchical models, we start by considering the simpler problem of how we might represent a subroutine, the smallest fungible unit of activity for our purposes. Subroutines can be used to kick a soccer ball or implement simple program transformations in a neural-network architecture.

We begin with the simplifying assumption that subroutines can be represented as tuples consisting of a set of operands represented as high-dimensional embedding vectors, a weight matrix representing the transformation and a product vector space in which to embed the result. ... arbitrary source code is unwieldy and designed for the convenience of someone writing code ... an abstract syntax tree (AST) is alternative canonical representation for representing the abstract syntactic structure of a source-code program⁹⁰ [35, 135] in a manner that is more convenient for computer program to manipulate ... In applying this idea to program transformations, assume that each operand corresponds to the embedding of an abstract-syntax-tree representation of a code fragment, w.l.o.g., any non-terminal node in the AST of a syntactically well-formed program.

The programmer's apprentice operates on programs represented as trees, where the set of actions includes basic operations for traversing and editing trees — or more generally directed-graphs with cycles if you assume edges in abstract syntax trees corresponding to loops, recursion and nested procedure calls, i.e., features common to nearly all the programs we actually care about. We still have a finite number of actions since for any given project we can represent the code base as a directed-acyclic graph with annotations to accommodate procedure calls and recursion, and use attention to direct and contextualize a finite set of edit operations⁹¹.

In a collaboration, figuring out what to say requires planning and a certain degree of imagination. Suppose you are the apprentice and you want to tell the programmer with whom you're collaborating that you don't understand what a particular expression does. You want to understand what role it plays in the program you are jointly working on. How do you convey this message? What do you need to say explicitly and what can be assumed common knowledge? What does the programmer know and what does she need to be told in order to provide you with assistance?

Somehow you need to model what the programmer knows. In planning what to say, you might turn this around and imagine that you're the programmer and ask how you would respond to an apprentice's effort to solicit help, but in imagining this role reversal you have be careful that you don't assume the programmer knows everything that you do. You need a model of what you know as well as a model of what the programmer knows. This is called Theory of Mind (ToM) reasoning and learning how to carry out such reasoning occurs in a critical stage of child development.

Shared knowledge includes general knowledge about programming, knowledge about the current state of a particular program you are working on, as well as specific details concerning what you are attending to at the moment, including program fragments and variable names that have been mentioned recently in the discussion or can be inferred from context. This sort of reasoning can be applied recursively if, for example, the apprentice wants to know what the programmer thinks it knows about what the apprentice knows. To a large extent we can finesse the problem of reasoning about other minds by practicing transparency, redundancy and simplicity so that both parties can depend on not having to work hard to figure out what the other means. However, there are some opportunities in the programmer's apprentice problem for applying ToM reasoning to parts of the problem that cannot be so easily finessed.

Suppose that the apprentice has started a new program using an existing program P following a suggestion by the expert programmer. Realizing that the body of a loop in P is irrelevant to the task at hand, the apprentice replaces the useless body B with a fragment from another program that does more or less what is required and then makes local changes to the fragment to create a new body B′ so that it works with the extant loop variables, e.g., loop counter, termination flag, etc. When the assistant has completed these local changes, the programmer intervenes and changes the name of a variable in B′. What induced the programmer to make this change?

The programmer noticed that the variable in B′ was not initialized or referenced in P but that another variable that was initialized in P and is no longer referenced — it only appeared in the original loop body B, is perfectly suited for the purposes of the new program. Assume for the sake of this discussion, that the programmer does not explain her action. How might the assistant learn from this intervention or, at the very least, understand why it was made? A reasonable theory of mind might assume that agents perform actions for reasons and those reasons often have to do with preconditions for acting in the world, and, moreover, that determining if action-enabling preconditions are true often requires effort. A useful ToM also depends on having a model allowing an agent to infer how preconditions enable actions by working backward from actions to enabling preconditions.

The phrase "Theory of Mind" refers to the idea that humans and possibly other organisms construct models of how other agents reason in order to understand and anticipate the behavior of such agents. Developmental psychologists have conducted fascinating research about how and when children learn such models, and what goes wrong when children fail to learn such models during a crucial developmental window.

Theory of Mind Reasoning refers to the mental processing required to employ such models in various collaborative and strategic circumstances. Here we consider what it would mean for a machine to learn such a theory and employ it in practical applications such as the programmer's apprentice. I've left this topic for last for a number of reasons. It might seem presumptive to suppose that machines could learn such models. It might not at first blush appear to be useful, but on reflection I hope you will agree that when two programmers collaborate in designing or debugging a computer program, it helps a lot to have some idea of what the other programmer knows or might be mistaken about.

Imagine the following scene, there's a man holding the reins of a donkey harnessed to a two-wheeled cart — often called a dray and its owner referred to as a drayman — carrying a load of rocks. He makes the donkey rear up and by so doing the surface of the cart tilts, dumping the rocks onto the road which was clearly his intention given the appreciative nods from the onlooking pedestrians. This short video illustrates that, while this might seem an unusual way of delivering a load of rocks, most people think they understand exactly how it was done. Not so!

The fact is that, as with so many other perceptual and conceptual tasks, people feel strongly that they perceived or understood much more than in fact they did. For example, most people would be hard-pressed to induce a donkey to rear up and, if you asked them to draw the donkey harnessed to the cart with its load of stone, they would very likely misrepresent the geometric relationships involving the height of the donkey, how the harness is attached, how far off the ground the axle is located, the diameter of the wheels and the level of the cart surface and center of gravity of the load with respect to the axle's frame of reference. In other words, they would not have — and possibly never could have — designed a working version of the system used by the drayman.

Now imagine that the drayman has a new apprentice who was watching the entire scene with some concentration, anticipating that he might want to do the very same thing before the first week of his apprenticeship is complete. Sure enough, the next day the drayman tells the apprentice to take a load of bricks to a building site in town where they are constructing a chimney on a new house. He stacks the bricks in a pile that looks something like how he remembers the rocks were arranged on the dray the day before. Unfortunately the load isn't balanced over the axle and almost lifts the donkey off its feet. After some experimentation he discovers how to balance the weight so the donkey can pull the load of bricks without too much effort.

When he finally gets to the building site, he nearly gets trampled by the donkey in the process of repeatedly trying to induce the distressed animal to rear up on its hind legs. Finally, one of the brick masons intervenes and demonstrates the trick. Unfortunately, the bricks don’t slide neatly off the dray as the rocks did for the experienced drayman the day before, but instead the bricks on the top of the stack tumble to the pavement and several break into pieces. The helpful brick mason suggests that in the future the assistant should prepare the dray by sprinkling a layer of sand on the surface of cart so that the bricks will slide more freely and that he should also dump the bricks on a softer surface to mitigate possible breakage. He then helps the assistant to unload the rest of the bricks but refuses to pay for the broken ones, telling the assistant he will probably have to pay the drayman to make up for the difference.

An interesting challenge is to develop a model based on what is known about the human brain explaining how memories of the events depicted in the video and extended in the above story might be formed, consolidated, and, subsequently, retrieved, altered, applied and finally assigned a value taking into account the possible negative implications of damaged goods and destroyed property. In the story above, the assistant initially uses his innate "physics engine" to convince himself that he understands the lesson from the master drayman, he then uses a combination of his physical intuitions and trial-and-error to load the cart, but runs up against a wall due to his unfamiliarity with handling reluctant beasts of burden. Finally, he gets into trouble with laws of friction and the quite-reasonable expectations of consumers unwilling to pay for damaged goods.

We don't propose to solve the general problems of theory-of-mind and physics-based reasoning in developing a programmer's apprentice, though the application provides an interesting opportunity to address particular special cases. As mentioned earlier, the stream of conversation between the assistant an expert programmer will inevitably relate to many different topics and specialized areas of expertise. It will include specific and general advice, reasons for acting, suggestions for what to attend to and a wide range of comments and criticisms. Several recent approaches for combining planning and prediction, especially in the case of partially observable Markov decision processes, are particularly promising for this application [50, 39, 41, 136, 114].

The apprentice will want to separate this information into different categories to construct solutions to problems that arise at multiple levels of abstraction and complexity during code synthesis. Or will it? We like to think of knowledge neatly packaged into modules that result in textbooks, courses, monographs, tutorials, etc. The apparent order in which activities appear in a stream of activities is largely a consequence of the context in which those activities are carried out. They may seem to arise in accord with some plan, as if assembled and orchestrated with a particular purpose in mind, but, even if there was plan at the outset, we tend to make up things on the fly to accommodate the sort of unpredictable circumstances that characterize most of our evolutionary history.

In some cases that context or purpose is used to assign a name, but that name or contextual handle is seldom used to initiate or identify the activity except in academic circumstances where divisions and boundaries are highly prized and made much of. The point of this is that in a diverse stream of activities — or utterances intended to instigate activities — credit assignment can be difficult. Proximity in terms of the length of time or number of intervening activities between a action and a reward is not necessarily a good measure of its value. We suggest it is possible to build a programer's apprentice or other sort of digital assistant that performs its essential services primarily by learning to predict actions, their consequences and their value from observing such a diverse stream of dialog intermixed with actions and observations.

Harnessed Imagination

The imagination-based planning (IBP) for reinforcement learning framework [95] serves as an example for how the code synthesis module might be implemented. The IBP architecture combines three separate adaptive components: (a) the CONTROLLER + MEMORY system which maps a state s ∈ S and history h ∈ H to an action a ∈ A; (b) the MANAGER maps a history h ∈ H to a route u ∈ U that determines whether the system performs an action in the COMPUTE environment, e.g., single-step the program in the FIDE, or performs an imagination step, e.g., generates a proposal for modifying the existing code under construction; the IMAGINATION MODEL is a form of dynamical systems model that maps a pair consisting of a state s ∈ S and an action a ∈ A to an imagined next state s′ ∈ S and scalar-valued reward r ∈ R.

The IMAGINATION MODEL is implemented as an interaction network [5] that could also be represented using the graph-networks framework introduced here. The three components are trained by three distinct, concurrent, on-policy training loops. The IBP framework shown in Figure 19 allows code synthesis to alternate between exploiting by modifying and running code, and exploring by using the model to investigate and analyze what would happen if you actually did act. The MANAGER chooses whether to execute a command or predict (imagine) its result and can generate any number of trajectories to produce a tree h_t of imagined results. The CONTROLLER takes this tree plus the compiled history and chooses an action (command) to carry out in the FIDE.

2.14  Language and Thinking

This section is heavily influenced by my reading three books on language all of which take some form of evolutionary stance [24, 122, 34]. Add to that Chomsky, Chater, Deacon and Pinker [20, 16, 27, 98], and you have some idea of the tempest I've ginned up for the purpose of tearing down my outdated conceptions of language and constructing a synthesis that accounts for what I've learned over the last few years⁹². Is it "ginned up" — enlisting an engine of industrial manufacture, or "jinned up" " — invoking a devil of some demonic species? Perhaps both apply here, but the evolutionary tale of each word is complex as only a linguist could revel in. From my small sample, one might guess that many linguists are channeling the ghosts of William Safire, Samuel Johnson or their ilk with varying success. For the most part, I've avoided these academic spiritualists to focus on work that furthers our objective of building autonomous agents. Guy Deutscher's The Unfolding of Language is somewhat of a compromise, but his was the first I read and he helped to reshape my understanding of the origins and uses of language [34].

The subtitle of Deutscher's book "An Evolutionary Tour of Mankind's Greatest Invention", is intentionally misleading. Deutscher begins by lauding our invention of language only to subsequently deflate our collective ego by pointing out that no individual institution or governing body can be held responsible for human language — it evolved, and the way in which it evolved and the corresponding selective pressures that directed its evolution are the focus of Deutscher's book. Deacon makes the case that humans and language coevolved and that language is an emergent phenomena dependent on the complexity of the life forms that employ language and the environment in which their lives play out. There are consequences of Deacon's theory that become apparent in reading these three evolutionary accounts, and that make reading my summary of Deacon's Incomplete Nature worth the effort. Here is Deutscher stroking our high opinion of ourselves, only to reveal that whatever role we might have had in the evolution of language it is more along the lines of a worker ant than a Boswell, Diderot or Hume:

Of all mankind's manifold creations, language must take pride of place. Other inventions — the wheel, agriculture, sliced bread — may have transformed our material existence, but the advent of language is what made us human compared to language, all other inventions pale in significance since everything we have ever achieved depends on language and originates from it. Without language we could never have embarked on our ascent to unparalleled power over all other animals, and even over nature itself. [ ... ] Even so, there is just one flaw in all these hymns of praise, for the homage to language's unique accomplishments conceals a simple yet critical incongruity. Language is mankind's greatest invention — except, of course, that it was never invented. Excerpt from Page 1 in The Unfolding of Language by Deutscher [34].

To explain the evolution of linguistic norms, Duetcher begins by describing the different forces acting on the initial selection, subsequent adoption, creative amendment and inevitable corruption and wholesale rejection of linguistic variations. He begins with destruction:

First to come under the magnifying glass will be the forces of destruction, for the devastation they wreak is perhaps the most conspicuous aspects of language's volatility. And strangely enough, it will also emerge that these forces of destruction are instrumental in understanding linguistic creation and regeneration. Above all they will be indispensable for solving a key question: the origin of the "raw materials" for the structure of language. [ . . . ] One thing is certain: in language, as in anything else, nothing comes from nothing. Only very rarely are words "invented" out of the blue (the English word "blurb" is reputedly one of the exceptions). Certainly, grammatical elements were not devised at a prehistoric assembly one summer afternoon, nor did they rise from the brew of some alchemist's cauldron. So they must've developed out of something that was already at hand. But what?

The answer may come as rather a surprise. The ultimate source of grammatical elements is nothing other than the most mundane everyday words, unassuming nouns and verbs like "head" or "go". Somehow, over the course of time, plain words like these can undergo drastic surgery, and turned into quite different beings altogether: case endings, prepositions, tense markers and the like. To discover how these metamorphoses take place, we'll have to dig beneath the surface of language and expose some of its familiar aspects in an unfamiliar light. But for the moment just to give a flavor of the sort of transformations we'll encounter, think of the verb "go" — surely one of the plainest and most unpretentious of words. In phrases such as "go away!" or "she's going to Basingstoke", "go" simply denotes movement from one place to another. But now take a look at these sentences:

Is the rain ever going to stop?
She's going to think about it.

Here, "go" has little to do with movement of any kind: the rain is not literally going to anywhere to stop, in fact it has no plans to go anywhere at all, nor is anyone really "going" anywhere to think. The phrase "going to" merely indicates that the event "be going to" can be replaced with "will" in these examples without changing the basic meaning in any way:

Will the rain ever stop?
She will think about it.

So what exactly is going on here? "Go" started out in life as an entirely ordinary verb, with the straightforward meaning of movement. But somehow, the phrase "going to" has acquired a completely different function, and has come to be used as a grammatical element, a marker of the future tense. In this role, the phrase can even be shortened to "gonna", at least in informal spoken language:

Is the rain ever gonna stop?
She's gonna think about it.

but if you try the same contraction when "go" is still used in the original meaning of movement, you're going to be disappointed. No matter how colloquial the style or how jazzy the setting, you simply cannot say "I'm gonna Basingstoke". So "going to" seems to have developed a kind of schizophrenic existence, since on the one hand it is still used in its original "normal" sense ("she's going to Basingstoke"), but on the other it has acquired an alter ego, one that has been transformed into an element of grammar. It has a different function, a different meaning, and has even acquired the possibility of a different pronunciation. Excerpt from Pages 10-11 in The Unfolding of Language by Deutscher [34].

Some of the earliest work in AI on natural language processing (NLP) both absorbed from and contributed to the fields of cognitive science and computational linguistics. Unfortunately, many AI researchers attacked the dual problems of production and understanding as if they only had write down the rules of grammar (syntax), the definitions of words (semantics), and encode any relevant knowledge as symbolic structures using propositional or first-order logic (logical forms), frame- or script-based representations (slot-filler notations) and relational graphs (hierarchies and taxonomies) and we'd be done with it, just like some thought that computer vision was a summer's work for an undergraduate MIT student⁹³ .

For example, Conceptual Dependency Theory [111] developed by Roger Schank built on the work of Sydney Lamb who developed his Relational Network Theory of Language as a significant challenge to Chomsky's transformational grammar. Script Theory is a psychological theory developed by Silvan Tomkins that posits human behavior largely falls into patterns called "scripts" because they function analogously to the way a written script does, by providing a program for action. Schank and Abelson [112] extended Tomkins' scripts in developing AI applications. Marvin Minsky's Frames predate scripts and are closely related in terms of representational power.

There was some effort to create internal representations that serve to ground linguistic forms and functionally group concepts. It was thought that by doing so one could exploit generalizations to simplify understanding and exhibit the sort of varied production we observe in human speech. Here, for example, is a partial taxonomy of primitive actions provided in conceptual dependency theory [111]:

State Changes physical and abstract transfers:

• PTRANS: to change the location of a physical object.

• ATRANS: to change an object's abstract relationship.

Mental and Cognitive actions and activities:

• SPEAK: to produce a sound intended for communication.

• ATTEND: to direct a sense organ or focus on stimulus.

Instruments for other actions and activities:

• MTRANS: to transfer information mentally.

• MBUILD: to create or combine thoughts.

Unfortunately, these approaches — now collectively referred to as "good old-fashioned AI" (GOFAI) — proved to be brittle and not amenable to the machine learning technology of the time. Douglas Hofstadter pointed out the inflexible nature of such representations and characterized a possible solution in terms of what he referred to as fluid concepts and creative analogies [61, 60, 59]. The connectionists focused on a biologically-inspired approach to exploiting distributed processing, context sensitivity and continuous (smoothly varying and differentiable) adaptation / responsiveness to environmental change, but were stymied by the seeming intractability of learning artificial neural networks due, in part, to the so-called vanishing gradient problem.

Modern ANN technology has demonstrated multiple viable approaches to learning complex networks and shown how ANN representations can effectively solve interesting computer vision and natural language processing problems. So why don't we all have personal assistants capable of continuous conversation across a wide range of topics and working collaboratively with us to solve practical problems? There are a number of reasons typically cited, ranging from the lack of physical grounding or semantics of any sort — McDermott's "no notation without denotation" was a challenge [79] to the AI community to adopt Tarskian semantics — to the difficulty of common sense reasoning. The attempt to encode a "naïve physics" model of the world tailored to everyday speech was part of the — largely failed — effort to formalize the common knowledge of human beings.

On Page 133 of Deutscher's book [34], he quotes the Encyclopedia Britannica article on the concept of 'space-time', "In physical science, 'space-time' is a single concept that recognizes the union of space and time, posited by Albert Einstein in the theories of relativity (1905, 1915). Common intuition previously supposed no connection between space and time." To this, Deutscher responds "everyday language proves that 'common intuition' has in fact recognized this link for many thousands of years (even if not exactly an Einstein's sense)." There is now evidence that hippocampal structures in both rodents and humans support a form of sequence memory that combines both temporal and spatial information⁹⁴ . Deutscher goes on to provide many examples to support his observation, and includes this interesting table:

Later he mentions that "like most of the metaphors already encountered, the images here are ultimately grounded in experience — think of a sentence like 'the travelers got typhoid from the contaminated water'. The physical origin of the disease is also its cause: the disease started because of the water, but it also came — physically — from it. But in generalizing the metaphor, we have unshackled the image from that basis in experience, and can now talk freely about one thing coming 'from' another, 'out of' another, or happening 'through' another, to express abstract chains of cause and event."

Deutscher makes many interesting observations about the everyday use of language. I'll mention two additional examples before turning to summarize what I've learned from reading the book. On Page 31, he notes that language with only words, with no structure — hierarchy, nesting, connectives, truth functionals, recursion — to prop them up would be a poor instrument of communication, and, on Page 45, he observes the obvious that "the names we use for things bear no inherent relationship to the things themselves. The names are entirely arbitrary, and this is why [the German word for spoon] 'löffel' or [the French word for spoon] 'cuillère' is just as good a designation for a [drawing or photograph of a spoon] as [the English word] 'spoon'" — a comment that apparently needs repeating as we read contemporary commentary on the role of language in everyday life.

We started this inquiry trying to understand human language both as a technology for communicating ideas and as a framework for exploring, incubating, and creating new ideas. Now, more than six books and several dozen journal articles later, I have pretty much come full circle. Yes, brains compute but they are also memory, indeed they integrate multiple memory systems each of which has its own neural mechanisms and natural use cases. Language production and understanding are built upon and constrained by the brain's various memory systems; however, in developing advanced AI systems that borrow from what we've learned about human brains, we need not be slavishly bound by such constraints.

Alternatives to the notion of a universal grammar have received more attention lately as the number of linguists under the thrall of Chomsky's theory has waned in recent years. While we can thank Chomsky for freeing language from the stifling influence of behaviorism, present day computational and cognitive linguists offer compelling alternative explanations for the seemingly infinite variety of human language suggesting that simple phrasal templates can account for a good deal of everyday language [24]. Moreover a full account of human linguistic competence based on natural selection follows from an understanding of general learning and processing biases deriving from the structure of natural thought processes [21, 23, 22].

A grammar is just one way of describing the syntactic structure of a language — a prescribed, collectively acquired set of constraints, but human language users can communicate using what most linguists would consider a very simple grammar. We have suggested that we begin with the most basic methods for signaling and referring. We should look carefully at the work in developmental psychology relating to how children acquire such basic communicative tools, considering them as an essential foundation for more sophisticated language use. Given what we know about how various memory systems mature in the developing brain, it is no surprise that sophisticated language skills develop later.

Historically, psychologists and neuroscientists group different memory systems as being either short-term or long-term with short-term memory⁹⁵ handled by neural mechanisms involving the basal ganglia and prefrontal cortex requiring some form of active maintenance, whereas long-term memory⁹⁶ is handled by the hippocampal-entorhinal complex and doesn't require active maintenance. This model works when we consider simple tasks like remembering a sequence of words and repeating at some later time. It doesn't appear to work well when subjects are interrupted in the midst of performing a task.

That some human subjects are able to handle so-called distractor tasks suggests that some subjects have more short-term storage capacity or are more adept at actively maintaining what short-term memory they have, or perhaps there is some yet-to-be discovered form of intermediate-duration storage that plays a similar role to that of a buffer, stack or local cache in modern computer architectures. The term working memory is sometimes used synonymously with short-term memory but some accounts ascribe working memory a larger role in decision making and problem solving⁹⁷.

In my experience, if you're interested in deeply understanding anything reasonably complex, it helps to read or listen to several different authors and speakers, for reasons of completeness, multiple perspectives and different ways of thinking — preferably those who are both clear and reasonably succinct. Neil Burgess is particularly articulate and economical in his exposition of a number of topics relating to our discussion of human memory, including both short-term memory, long-term (episodic) memory and spatial navigation — see also his Neuron article on spatial and episodic memory with Eleanor Maguire and John O'Keefe [14].

Michael Corballis [24] follows William James in treating stories and storytelling as a mode of "narrative, descriptive, contemplative" thinking, and links it to our "capacity to travel mentally in time and space and into the minds of others, along with a dose of imaginative construction." Regarding the role of episodic memory, he quotes Catherine Nelson [86] in writing about storied thoughts, "narrative models and early influences in early childhood help to transform the episodic memory system into a long-lasting autobiographical memory for one's own life, and thus a self history, which to a large extent underlies our concept of self" and notes the natural tendency of children to engage in playful fantasy, often blending fact and fantasy and delighting in the telling of frightening events and disasters. Corballis also mentions the natural tendency of children to act out the scripts of life, such as feeding a baby, doing a medical checkup and cooking a meal. The repeating of old stories and reliving old memories, trotting them out when appropriate to teach a lesson as in the case of telling epic stories.

In the third and final part of his book, Corballis explains how language evolved as a communication system for sharing thoughts and experiences. Some of what he has to say repeats what we have heard from other authors, including how expressive language originated in bodily gestures, as opposed to vocally articulated calls and warnings, and how vocalizing gradually evolved, eventually coming to dominate as the mode of choice in communicating intent. he weaves in the idea of motor neurons providing detail about the cell types and their locus in human brains he writes

It is now clear that our brains house a mirror system. A meta-analysis of 125 studies using functional magnetic resonance imaging revealed 14 clusters of neurons with mirror properties. These were located in areas homologous to those identified in the monkey brain, including the inferior parietal lobule, inferior frontal gyrus, the ventral prefrontal cortex, but also included regions in the primary visual cortex, the cerebellum, and the limbic system. In another remarkable study a group of investigators recorded activity directly in over a thousand single neurons in the brains of patients about to undergo surgery. A significant portion of these neurons responded to both observation and execution of movements of the hand and face. The areas included area F5, along with other areas associated with language and gesture in the frontal and temporal lobes. And not surprisingly, the human mirror system does indeed overlap with Broca's and Wernicke's area. Excerpt from Page 130 in The Truth About Language by Corballis [24]

Corballis mentions Hickok and Poeppel [56] on their dual pathways theory of speech perception. His summary should provide motivation to read his full account or read Hickok and Poeppel if you want more detail:

Hickok and Poeppel have suggested what they call a "dual stream" theory of speech perception. One stream is lower in the brain: the so-called ventral stream, which progresses from the temporal lobe to the frontal lobe of the brain. This stream is responsible for the understanding of speech, and it is presumably this stream, or its homologue, that is responsible for the limited understanding of human speech and Kansi or in dogs like Rico and Chaser — and perhaps even your pet cat, not to mention inarticulate teenagers. The other stream is higher: the so-called dorsal stream, which flows from the parietal lobe and is responsible for the actual production of articulate speech. It is this component that is lacking in dogs and apes.

The dorsal stream is part of the mirror system and may indeed play a role in perceiving as well as in producing action, as implied by the motor theory of speech perception. In an extensive critique of mirror-neuron theory, Greg Haycock suggests that the mirror system is not so much a device for tuning into speech sounds as a "feed forward" mechanism for calibrating the production of speech sounds. We shape our speech in terms of how we hear them — this, perhaps is why babies babble before they learn to speak. They are calibrating. The ability to speak may well modify the way we perceive speech, but, as the examples of dogs and apes show, perception of speech can occur without any influence of speech production. Hickok gives the analogy of observing a person playing the saxophone. A non-saxophone player may well have adequate perception of the player's performance, but a fellow saxophonist is likely to have an enriched perception and understanding by relating her perception to her own experience. Excerpt from Pages 151-152 in The Truth About Language by Corballis [24]

In the chapter entitled "How Language is Structured", mentions John O'Keefe's observations that "most prepositions in English, such as, about, across, against, among, along, around, between, from, in, through, and to, apply to both space and time and apply even to logical expression that are symbolic rather than spatial, as in "A follows from B" or the "argument against A is B. [...] O'Keefe also suggest that the hippocampus and adjacent structures gained an asymmetry [noting that] damage to the right mesial temporal lobe (which includes the hippocampus) results in amnesia for episodes coded in visuo-spatial terms while left-sided damage results in amnesia for episodes coded verbally⁹⁸.

I was particularly interested in Corballis's discussion — Pages 184-187 — of how manufacture and the use of tools may have influenced the structure of language. He notes a grammar-like feature of toolmaking in the form of its "hierarchical nature, with subgoals that must be met within overarching goals. For example, in the removal of stone flakes, various 'subroutines' such as edging, thinning, and shaping need to be carried out before a flake is removed. One consequence of this is a separation in time between related operations — a subroutine must be completed before the toolmaker can return to the original goal. Likewise in language, one must retain information from the beginning of a sentence while a subordinate clause is processed before the rest of the sentence completes the meaning."

Assuming — as we have here — that you are a self-replicating, recurrent process, a thread of conscious awareness that continuously reimagines itself through its interaction with the environment and its internal mutable record of the past mediated by scores of unconscious threads of computation governed in part by the complex interplay of resident microbial and viral influences, ambient gases and airborne particulate matter, sustaining nutrients, neuromodulatory hormones and neurotransmitters largely under the control of primitive instincts that have persisted pretty much unchanged for millions of years [...] If you can read and understand the last sentence does that serve as evidence that you are human ... or that you are a Turing machine? [Note that "both" is also a legitimate answer.]

It is interesting to note that we are suggesting here that human language could be as simple as template matching and table lookup. The encoder-decoder model that we encountered in Figure 5 can used to generate often quite plausible responses when trained on utterance pairs extracted from conversation or call-center customer-operator sessions. Of course, to get it right often enough that customers don't just immediately hang up the phone, it's necessary to adapt the previously recorded response to suit the current circumstances. It's possible that much of what we say is generated by recalling something we or someone else said, modifying it superficially and producing it as if generated de novo. Perhaps the most eloquent and erudite among us are just better at disguising what would otherwise be considered as plagiarism or, more charitably, imitation.

From this perspective, communication and problem solving are all about bringing context to bear. What to say in a given situation and how to interpret what someone else is saying. What is relevant to think about in solving a problem and what can be discarded so as not to clutter the mind. The context sensitivity of connectionist models and the fluid properties of continuous vector representations are the friend of recall and the enemy of precision. For most children, their initial exposure to language and early development of a facility for its day-to-day application is full of opportunities to learn. Our immediate goal is to learn what's required to design, write and debug programs and not to challenge the likes of William Safire. Learning how to attend and refer to relevant information, following simple instructions such as evaluating a code fragment, asking for help ... these are the sort of activities the assistant needs to master in order to become useful.

Nick Chater [16] has written an interesting and provocative book hypothesizing that human cognition is "flat" in the sense that, apart from what is apparent in our conscious thoughts, there is very little else going on (unconsciously) in the background except for the sort of sensory processing that occurs in the visual and auditory areas of the cortex. My short summary of his book would be something like: human cognition consists primarily of our conscious thoughts substantially enhanced by imaginative (unconscious) recall of related memories stimulated by a combination of our recently activated thoughts and our cu-rent perceptions which are influenced by our emotional and cognitive state; all the rest of our thoughts are the result of unconscious, highly-parallel processing that are similar to what's going on in the visual cortex. You can read more of my thoughts on Chater's book in this footnote. Chater, Morten Christiansen and their various co-conspirators have also written a good deal about the origins of human language and the biological plausibility of our having an innate language facility such as Chomsky's universal grammar [21, 19, 17].

Communicating Code

The brain didn't evolve to accommodate language, rather, language evolved to accommodate the brain [17]. Biological and cognitive constraints determine what types of linguistic structure are learned, processed and transmitted from person to person and generation to generation. Language acquisition has comparatively little to do with linguistics and is probably best viewed as a form of skill acquisition. Indeed, we are constantly processing streams of sensory information into successively more abstract representations while simultaneously learning to recode the compressed information into hierarchies of skills that serve our diverse purposes [18, 19].

Contrary to what some textbook authors might think, students learn to code by writing programs, a process that can be considerably accelerated by timely communication with peers and invested collaborators. In the case of unequal skill levels, communication tends to be on the terms of the more capable interlocutor, and the onus of understanding on the less capable partner in the collaboration. To sustain the collaboration, we need to bootstrap the apprentice to achieve a reasonble threshold level of competence in both language and in working with computers so as to compensate for expert programmer's investment in effort. From a value-proposition perspective, the apprentice has to provide net positive benefit to the programmer from day one.

It is important to keep in mind that any program specification whether in the form of input/output pairs or natural language descriptions when communicated between individuals of differing expertise is just the beginning of a conversation. Experts are often careless in specifying computations and assume too much of the student. Students, on the other hand, can surprise experts with their logical thinking while frustrate and disappoint with their difficulty in handling ambiguity and analogy, particularly of the esoteric sort familiar to professional programmers. Input from an expert will typically consist of a stream of facts, suggestions, heuristics, shortcuts, etc., peppered with clarifications, interruptions and other miscellany.

We propose a hybrid system for achieving competence in communicating programming knowledge and collaboratively generating software by creating a non-differentiable conventional dialogue management system that works in tandem with a differentiable neural-network dialogue system (NDS) that will become more competent as it gains experience coding and collaborating with its expert partner. The deployment of these two language systems will be controlled on a sentence-by-sentence basis by a meta-reinforcement learning (MRL) system that will depend less and less on the more conventional system but likely never entirely eclipse its utility. The MRL system subsumes the role of a beam search or softmax layer in an encoder-decoder dialogue model.

The conventional system will be built as a hierarchical planner⁹⁹ following in the footsteps of the CMU Ravenclaw Dialogue System [11] and will come equipped with a relatively sophisticated suite of hierarchical dialogue plans for dealing with communication problems arising due to ambiguity and misunderstanding. While clumsy compared to how humans handle ambiguity and misunderstanding, these dialogue plans are designed to resolve the ambiguity and mitigate the consequences of misunderstanding quickly and get the conversation back on track by attempting various repairs involving requests for definition, clarification, repetition and restatement in as inconspicuous manner as possible [10].

The conventional dialogue system will also include a collection of hierarchical plans for interpreting requests made by the expert programmer to alter programs in the shared editor space, execute programs on specified inputs and perform analyses on the output generated by the program, debugger and other tools provided in the integrated development environment (IDE) accessible through a set of commands implemented as either primitive tasks in the non-differentiable hierarchical planner or through a differentiable neural computer (DNC) interface using the NDS system that will ultimately replace most of the functions of the hierarchical-planner-based dialogue management system.

This dual mode dialogue system and its MRL controller allows the apprentice to practice on its own and rewards it for learning to emulate the less-flexible hierarchical-planner implementation. Indeed there is quite a bit that we can do to preload the apprentice’s basic language competence and facility using the instrumented IDE and related compiler chain. A parallel dialogue system implemented using the same hierarchical planner can be designed to carry out the programmer’s side of the conversation so as to train the NDS system and the meta-reinforcement learning system that controls its deployment utterance by utterance. We can also train domain-specific language models using n-gram corpora gleaned from discussions between pair programmers engaged in writing code for projects requiring the same programming language and working on similar programming tasks.

Cognitive and systems neuroscience provide a wide range of insights into how to design better algorithms and better artificial neural network architectures. From our study of the fly olfactory system we have learned new algorithms for locality sensitive hashing, one of the most important algorithms used for scalable nearest-neighbor search. The mammalian visual system is the most well studied area of the brain. Early on we discovered that the architecture of the visual system involved two separate pathways referred to as the dorsal and the ventral pathways and roughly characterized as being responsible for the "where" and "what" dimensions of visual stimuli.

Subsequently it was argued that other sensory modalities employed dual pathways and that the production and understanding of language were organized as such with Broca's and Wernicke's areas playing key roles in this architecture. The engineering insights we have gained studying the brain are cashed out in the technical language of different disciplines. For example, the dual loop language model as it's called is often presented as a block diagram consisting of conceptually separate neural networks. The Hodgkin-Huxley model of action potential propagation in the giant squid axon was described both as a system of differential equations and as an electrical circuit diagram.

This is not the place for a lecture on dialogue management, but I'll make just a few comments about how we might finesse the problem either by brute force approaches or by resorting to a highly structured subset of natural language sufficient for most of the communication that goes on between two programmers focused intently in working on a specific problem.

The CMU RavenClaw Dialogue Management Architecture recast dialogue management as hierarchical task-based planning, demonstrating how error recovery, new topic introduction, recursive structure, and other characteristics of conversational dialogue could be handled within this framework. We've experimented with this approach and it might be a good fit for a prototype version of the programmer's apprentice. A more ambitious approach might leverage the idea of value iteration networks to create a dialogue management system trained on real dialog. A hybrid alternative would be to combine the two approaches.

In 2013 we created a dialogue dataset from the logs of chat sessions between software engineers and technical support staff. We borrowed the idea that dialogue management as a specialized form of language translation in which both speakers use the same basic language but employ different word choices, and created a sequence-to-sequence encoder-decoder translation model. The dataset was relative small compared with most of our language translation datasets, but the performance was better than we had any right to expect. More sophisticated hierarchical architectures using various attentional mechanisms would no doubt outperform our earlier models.

Contextual Integration

Context is everything in language and problem solving. When we converse with someone or read a book we keep in mind what was said or written previously. When we attempt to understand what was said in a conversation or formulate what to say next we draw upon our short-term memories of earlier mentioned people and events, but we also draw upon our long-term episodic memories involving the people, places and events related to those explicitly mentioned in the conversation. In solving complex design problems, it is often necessary to keep in mind a large number of specific facts about the different components that go into the design as well as general knowledge pertaining to how those components might be adapted and assembled to produce the final product.

Much of a programmer’s procedural knowledge about how to write code is baked into various cognitive subroutines that can be executed with minimal thinking. For example, writing a simple FOR loop in Python to iterate through a list is effortless for an experienced Python programmer, but may require careful thought for an analogous code block in a less familiar programming language like C++. In thinking about how the apprentice’s knowledge of programming is organized in memory, routine tasks would likely be baked into value functions trained by reinforcement learning. When faced with a new challenge involving unfamiliar concepts or seldom used syntax, we often draw upon less structured knowledge stored in episodic memory. The apprentice uses this same strategy.

The neural network architecture for managing dialogue and writing code involves encoder-decoder pairs comprised of gated recurrent networks that are augmented with attention networks. We’ll focus on dialogue to illustrate how context is handled in the process of ingesting (encoding) fragments of an ongoing conversation and generating (decoding) appropriate responses, but the basic architecture is similar for ingesting fragments of code and generating modified fragments that more closely match a specification. The basic architecture employs three attention networks, each of which is associated with a separate encoder network specialized to handle a different type of context. The outputs of the three attention networks are combined and then fed to a single decoder.

The (user response) encoder ingests the most recent utterance produced by the programmer and corresponds to the encoder associated with the encoder-decoder architectures used in machine translation and dialogue management. The (dialogue context) encoder ingests the N words prior to the last utterance. The (episodic memory) encoder ingests older dialogue selected from episodic memory. The attentional machinery responsible for the selection and active maintenance of relevant circuits in the global workspace (GWS) will likely notice and attend to every utterance produced by the programmer. Attentional focus and active maintenance of such circuits in the GWS will result in the corresponding thought vector added to NTM the partition responsible for short-term memory.

The controller for the NTM partition responsible for short-term (active) memory then generates keys from the newly added thought vectors and transmits these keys to the controller of the NTM partition responsible for long-term (episodic) memory. The episodic memory controller uses these keys to select episodic memories relevant to the current discourse, combining the selected memories into a fixed-length composite thought vector that serves as input for the corresponding encoder. Figure 20 depicts the basic architecture showing only two of the three encoders and their associated attention networks, illustrating how the outputs of the attention networks are combined prior to being used by the decoder to generate the next word or words in the assistant’s next utterance.

Memory Consolidation

One step of the decoder could add zero, one, or more words, i.e., a phrase, to the current utterance under construction. Memories — both short- and long-term — are in the form of thought vectors or word sequences that could be used to reconstruct the original thought vectors for embedding or constructing composites by adding context or conditioning to emphasize relevant dimensions. The dialogue manager — a Q-function network trained by reinforcement learning — can also choose not to respond at all or could respond at some length perhaps incorporating references to code, explanations for design choices and demonstrations showing the results of executing code in the IDE.

To control generation, we adapt the pointer-generator network framework developed by See et al for document summarization [113]. In the standard sequence-to-sequence machine-translation model a weighted average of encoder states becomes the decoder state and attention is just the distribution of weights. In See et al attention is simpler: instead of weighting input elements, it points at them probabilistically. It isn't necessary to use all the pointers; such networks can mark excerpts by pointing to their start and end constituents. We apply their approach here to digest and integrate contextual information originating from multiple sources.

In humans, memory formation and consolidation involves several systems, multiple stages and can span hours, weeks or months depending on the stage and associated neural circuitry¹⁰⁰. Our primary interest relates to the earliest stages of memory formation and role of the hippocampus and entorhinal region of the frontal cortex along with several ancillary subcortical circuits including the basal ganglia (BG). Influenced by the work of O'Reilly and Frank [90], we focus on the function of the dentate gyrus (DG) in the hippocampal formation and encode thought vectors using a sparse, invertible mapping thereby providing a high degree of pattern separation in encoding new information while avoiding interference with existing memories.

We finesse the details of what gets stored and when by simply storing everything. We could store the sparse representation provided by the DG, but prefer to use this probe as the key in a key-value pair in the NTM partition dedicated to episodic memory and store the raw data as the value. This means we have to reconstruct the original encoding produced when initially ingesting the text of an utterance. This is preferable for two reasons: (i) we need the words — or tokens of an abstract syntax tree in the case of ingesting code fragments — in order for the decoder to generate the apprentice's response, and (ii) the embeddings of the symbolic entities that constitute their meaning are likely to drift during ongoing training.

Hierarchical Memories

Language, programs and plans are all usefully thought of as having hierarchical, recursive structure. It also makes sense to think of brains as being organized as such, and the human brain employs hierarchical models to make sense of the world in which it evolved. Sensory information enters central nervous system in separate streams for each of the different senses, vision, hearing, taste, smell, etc., but is quickly processed to extract useful features and combinef to construct abstract concepts that allow us to recognize similarities and exploit differences.

To the untutored mind, the world is essentially flat. We impose hierarchical structure to make understanding it more tractable. We ingest sequences of observations as input and execute sequences of actions as output. What goes on between is complicated. We start by consulting a map of the human brain and relating different anatomical regions with the functional roles we believe they play in cognition. There is a great deal we still don't know about the brain and much of what is written here is provisional. That said, for the most part we stick to what is generally agreed upon, and will alert you if we deviate substantially from current consensus. Enamored as we are with our highly evolved cortex, much of the discussion will focus on this part of the brain. However, the cortex is built on top of a highly conserved foundation of subcortical regions without which we would be reduced to gibbering idiots.

Humans tend to organize information in hierarchies that are based on different methods of cataloguing data. Sometimes they correspond to class-subclass hierarchies as are used in taxonomic classifications, e.g., species, genus, family. Class hierarchies are different from what are commonly referred to as compositional hierarchies, with features at one level in the hierarchy constructed from the features in the subordinate level as in the case of part-whole hierarchies, e.g., the carburetor is part of the engine and the engine is part of the car, or feature hierarchies in object recognition, e.g., lines, vertices, edges and surfaces.

The human brain seems to naturally use all of these strategies and more, sometimes in combination, but hardly ever in the clean tidy way that scientists create taxonomies and engineers catalog engineered artifacts. In the visual system, we see examples of what appear to be compositional models in which small features that represent little fragments of lines are composed to create longer lines which in turn become the boundaries of objects. There are more than a dozen topographically mapped representations in the primate visual system and while they roughly constitute an hierarchy of features, that may have more to do with what we expect to see.

Machine learning researchers have come to realize that it's a bad idea to impose what you might think are the right features for representing something. It is generally better to let the data speak for itself in generating appropriate features for representation. One drawback of this is that the representations the learning system builds are often said to be uninterpretable which can be a problem if, for example, you are learning to identify tumors in tissue samples for performing biopsies and want a human expert to pass judgment on how well the system is working. Fortunately, it is often possible to learn generative models that allow researchers to visualize learned features.

Some search engines still depend on hierarchical models in which documents are composed of sections, that are composed of paragraphs that are composed of sentences, phrases, words and letters. Nowadays machine learning experts are more likely learn generative models of text and not impose biases that are more often than not misguided [1, 141]. Your brain routinely constructs complex representations of events that encode a wide range of relevant information — relevant, that is, to your particular interests. Seeing an old photo might cause you to recall the person who took the photo, the location where it was taken, what you had for dinner that night, etc.

Bootstrapped Language

In this section, we consider how we might design an end-to-end training protocol for bootstrapping a variant of the programmer's apprentice application. We begin with the analogous stages in early child development. Each of the following four stages is briefly introduced with additional technical details provided in the accompanying footnotes:

• Basic cognitive bootstrapping and linguistic grounding¹⁰¹:

  • modeling language: statistical \({n}\)-gram language model trained on programming corpus;

  • hierarchical planning: automated tutor generates lessons using curriculum training;

• Simple interactive behavior for signaling and editing¹⁰³:

  • following instruction: learning to carry out simple plans one instruction at a time;

  • explaining behavior: providing short explanations of behavior, acknowledging failure;

• Mixed dialogue interleaving instruction and mirroring¹⁰⁴:

  • classifying intention: learning to categorize tasks and summarize intentions to act;

  • confirming comprehension: conveying practical understanding of specific instructions;

• Composite behaviors corresponding to simple repairs¹⁰⁵:

  • executing complex plans: generating and executing multi-step plans with contingencies;

  • recovering from failure: backtracking, recovering, retracting steps on failed branches;

Chimps are apparently better than humans in tasks that require rapidly acquiring the details of a visual scene. In the wild, their near photographic memory allows them to quickly ascertain possible threats at a glance. In an experimental setting, they exhibit better recall than humans in trials where they are shown, say, an image showing single digits scattered across a background of contrasting color for a few milliseconds, and then tested on whether than can point to where each digit was in numerical order. In this sense, chimps may be likened to humans with autism who display weak central coherence, i.e., an eye for detail, but without the corresponding big-picture [77, 76]. One hypothesis is that humans are slower due, in part, to their more developed linguistic capacity that causes them to attend to the meaning of the individual stimuli as well as the relationships between them et al [107]¹⁰⁶.

2.15  The History of Ideas

Hebb, Lashley, Pribram

Donald O. Hebb (1904-1985) was an extraordinary scientist and educator. He is probably best known for his unforgettable aphorism "neurons that fire together wire together", but there is so much more he contributed to psychology, neuroscience and our understanding of the human brain. His 1949 "The Organization of Behavior" had a profound impact on the direction of psychology and the creation of neuroscience as a scientific discipline [54]. I've included a short biography highlighting his contributions to science and his lifelong passion for teaching and scholarship while on the faculty of McGill University. I highly recommend this wonderfully scripted, narrated and graphically animated video produced at the University of British Columbia illustrating Hebb's postulates as applied to learning in the hippocampus.

The history of neuroscience and the study of human brain makes fascinating reading, both for the history of ideas and for insights into the fascinating people who were drawn to this field and persevered despite the technical and intellectual challenges of trying to understand one of the most complicated systems we know of. The Society for Neuroscience has created an archive of interviews with some of the most influential scientists who have contributed to our understanding of the many different areas of scientific inquiry relating to the study of neuroscience. Transcripts of the interviews are available from SfN for download as PDF files and there are video recordings for many of the interviews available from the SfN website or on the SfN YouTube channel. Hebb didn't live long enough to be included in the SfN archives, but the Canadian Association for Neuroscience has a short biography covering his major contributions.

William Maxwell Cowan (1931-2002) was a South African neuroscientist known for his work on developmental plasticity and neural connectivity. He is credited with helping to contribute to the growth of modern neuroanatomy through his use of novel anterograde tracing techniques which fundamentally transformed the field in the 1970s. His video interview for Society for Neuroscience History of Neuroscience archive is fascinating both for Cowan's choice of animal models and neural circuits to work on, e.g., connections involving the striatum, thalamus and other basal ganglia, the development of connections in the dentate gyrus and hippocampus, and the tools that he either employed or invented in his study of the retina and visual pathways.

Relative to our discussion of episodic memory, I highly recommend that you watch the interview with Mortimer Mishkin, former senior investigator at NIH and current chief of the section on cognitive neuroscience in the laboratory of neuropsychology at NIMH. The interview took place April 5-6, 2006. This video is part of the Society for Neuroscience's autobiography series, "The History of Neuroscience in Autobiography," detailing the lives and discoveries of eminent senior neuroscientists. Mortimer studied with two pioneers in neuroscience Donald Hebb and Karl Pribram whose ideas continue to influence the field to this day. Mortimer and his colleagues challenged the prevailing theory concerning organization of the brain due to Karl Lashley and provided evidence supporting a new understanding of how we process and remember visual information. Mishkin speaks in detail about the experiments he conducted, what they thought they knew at the time and what they learned as a result of their studies.

Karl Pribram worked with Karl Lashley at the Yerkes National Primate Research Center which is part of Emery University in Atlanta, Georgia. Pribram challenged Lashley's theory that any particular memory — or engram — did not exist in any specific part of the rat's brain, but that memory was widely distributed throughout the cortex. "In Lashley's experiments, rats were trained to run a maze. Tissue was removed from their cerebral cortices before re-introducing them to the maze, to see how their memory was affected. Increasing the amount of tissue removed degraded memory, but more remarkably, where the tissue was removed from made no difference". On the other hand, Pribram removed large areas of cortex and smaller areas that respected areal — possibly functional — boundaries formed by gyri and sulci, leading to multiple serious deficits in memory and cognitive function. In this interview commissioned by the American Association for Neurological Surgeons, Pribram describes his work at Yerkes and his realization that many functions that appeared to be carried out entirely within the cerebral cortex actually involved processes that descend into subcortical regions including the hippocampus and basal ganglia and even extending into the brain stem. Pribram comments on his epiphany around 00:21:00 minutes into the interview and mentions Mortimer Mishkin's contributions relating to the hippocampus and entorhinal cortex.

On the Hard Problems

I feel I should probably say something about consciousness, but believe the computational account provided in this chapter speaks for itself, and have read too much to think I could change any minds. However, there are a few voices I respect and so I will channel their enthusiasm by quoting excerpts from their writing and speaking. I strongly agree with Stanislas Dehaene's view of consciousness as expressed in this twenty minute video and summarized: "Stanislas Dehaene argues against the philosophical dualism of mind and brain and believes that consciousness is not a magical property but evolved for flexibility and adaptation." I agree with Daniel Dennett though I wish he didn't feel the need to address the concerns of every philosopher and scientist who ever expressed an opinion about consciousness. He has much to say and a good deal of it worth reading. I enjoyed Felipe de Brigard's compelling comments concerning David Chalmers' "hard problem of consciousness" at 01:03:00 into this discussion on NEURO.tv [70] produced by Dianna Xie and Jean François Garièpy. While researching Douglas Hofstadter's work on analogy, I ran across his book "I Am A Strange Loop" [62] in which he explores the idea of consciousness and the concept of "self". In comparing his account of consciousness with that of Dehaene, Graziano, Dennett and others we've discussed in these notes, I find Hofstadter's liberating in the way he avoids many of the earlier philosophical, psychological, biophysical and metaphysical conundrums that make this topic so confusing and fraught with controversy for the uninitiated. That said, I think some of you may find that this video retelling by Will Schoder entitled You Are A Strange Loop does Hofstadter's account one better and achieves its weight and clarity in a scant twenty minutes.

Emergent Phenomena

There are many disciplines that study the brain and for the most part they have different criteria for what it might mean to understand the brain. For example, there are neuroscientists who study the brain from the perspective of molecular biology. This could involve understanding the dynamics of how proteins, sugars, lipids and other types of molecules interact with one another to produce behavior at the molecular level e.g., how the proteins and lipids that make up neural cell membranes enable the selective passage of ions required to initiate and propagate neural signals along dendrites and axons. Some neuroscientists studied the genetic pathways that are ultimately responsible for most everything that happens within the brain. These disciplines tend to have a reductionist approach to doing science and are primarily interested in theories that explain how behavior arises at the most basic level.

Artificial and biological networks have in common the characteristic that it is difficult to map activity patterns at the level of individual neurons to the complex behaviors produced by the system as a whole. Explanations of how neural firing patterns give rise to observable behavior often tail off with an unsatisfying appeal to the idea of emergence in which the complex interactions between individual elements of a complex dynamical system give rise to surprisingly ordered behavior. The kinetic theory of gases allows us to predict the behavior of individual particles but is mathematically cumbersome if you want to understand how the curve of a wing provides lift for a bird in flight or predict the path of a hurricane. The tools of fluid dynamics provide a more tractable framework for studying aerodynamics and predicting weather. In the case of neuroscience, not only do we lack the models necessary to extrapolate from the activity of individual neurons to the composite behavior of ensembles of neurons whereby we solve simple word problems, we don't even have suitable technology for accurately measuring the related physical processes at multiple scales so as to provide the necessary data to construct and evaluate such models.

In contrast with cellular and molecular neuroscience, the cognitive neurosciences [44] are concerned with more complex behavior of the sort that give rise to the macroscale activities by which we interact with the physical world. Ultimately, they too would like explanations that connect observable behavior with the neural circuitry that gives rise to it, but the phenomena that they are concerned with is said to be an emergent property of the complex behavior of molecules. A single theory that extended from, say, the interaction of molecules to the production of speech acts is likely to be cumbersome and limited in its explanatory value, for example, in diagnosing behavioral problems in young children. Ultimately, what we need is a hierarchy of models each of which designed for a particular purpose so that the gap between any two layers of this hierarchy is small enough that it is possible for the corresponding models to bridge that gap in a manner that yields explanations suitable for some predictive or diagnostic purposes¹⁰⁷.

The primary tools of cognitive neuroscience allow us to observe proxies for the gross behavior of such ensembles as in the case of using functional MRI to measure average blood oxygen levels within voxels on the order of a few millimeters on a side. Blood oxygen level is a lagging indicator for metabolic processes that are necessary in order for neurons to fire and for information to flow across the synaptic cleft from one neuron to another. The advantage of this technology is that it allows us to study the neural correlates of complex problem-solving behavior in human beings without surgery or other invasive methods of collecting data. Even if we could record from every neuron in the brain of an awake behaving human being without endangering the subject, it is not clear that we could make sense of the resulting tsunami of data — on the order of 100 billion neurons and well over 100 trillion synapses at millisecond temporal resolution, but there are certainly those of us who would like to try. Currently the best we can hope for is to record such data in a more tractable subject such as a fruit fly or juvenile zebrafish both of which have something on the order of 100,000 neurons, and even this will pose a considerable challenge to existing technologies.

Imagine you are walking in a forest in late Fall. The native birches and oaks have lost most of their foliage and it hasn't rained in weeks. A light breeze has been rustling the dry leaves, but now the wind is picking up and the few leaves left on the trees are taking flight, swirling through the forest like a flock of starlings. Up ahead is a clearing warmed by the sun. In the resulting thermal updraft, the leaves are swept up and disperse as if starlings are fleeing in disarray from a predatory hawk or falcon. How would you model this common but relatively complex behavior?

A physicist could choose from several theoretical frameworks depending upon his or her target level of explanation, e.g., quantum electrodynamics (QED), the kinetic molecular theory of gases, fluid dynamics (FD) or start by describing a computational finite-element model and then use one of the earlier frameworks to define the boundary equations as is done in weather and climate predictions. Each option has its own specific practical consequences, and a good scientist or engineer understands the limitations and advantages of the available modeling tools.

In the case of our walk in the woods, the behavior we observe arises out of interactions between familiar objects that are themselves complex physical systems that have their own dynamics and explanatory frameworks that may or may not be relevant if a detailed accounting does not substantially alter the predictions we might wish to make. It is reasonable expect that an explanation at the level of QED or even FD provides more detail than required for most purposes and that carrying out the necessary calculations or establishing the appropriate boundary conditions is prohibitive in any case.

Your body exhibits emergent properties that can be partially explained at the molecular level but are made clearer by appeal to their gross (motor) function. You are an emergent phenomenon. You learn and use language. You make predictions and construct plans. You have feelings and experience emotion. The society that you were born into has emergent properties that can't easily or succinctly be described at the molecular or cellular level. It is difficult if not impossible to adequately explain your behavior without taking into account the physical and social environment you inhabit.

Sir John Eccles who shared 1963 Nobel Prize in Physiology or Medicine for his work on the synapse with Andrew Huxley and Alan Lloyd Hodgkin believed that synaptic transmission was primarily electrical rather than chemical. He believed this so strongly that he discouraged those who challenged his theory. The fact is he was partially right. Models that strongly adhere to his bias have turned out to be extraordinarily useful in accounting for some of the neural activity we observe, but they have proved inadequate when it comes to explaining many brain disorders that arise from biochemical signaling¹⁰⁸ .

The point is that explanations based on faulty or inadequate theories are inevitable and that, moreover, they often provide value in terms of adequately explaining phenomena. Theories that abstract from the details in order to account for emergent phenomena play an essential role in science. In either case, we have to remain reasonably skeptical. Today most biomedical students know that protein phosphorylation mediates a wide range of neural phenomena, and, in particular, gives rise to many debilitating brain disorders. It would be naïve to believe that there are no remaining significant deficits in our basic understanding of neurons¹⁰⁹.

In this chapter, we have attempted to explain how interactions between neurons on a microscopic scale — that we are just beginning to understand — produce behavior on a macroscopic scale using the technology of artificial neural networks — that we understand insofar as we can build such systems, control every characteristic of their design and observe every nuance of their behavior. Artificial neural networks serve as a bridge across multiple scales by sacrificing some features in order to reduce complexity and gain insight into other features.

---

2.16  Key Ideas Summary

References

[1]	Tameem Adel, Zoubin Ghahramani, and Adrian Weller. Discovering interpretable representations for both deep generative and discriminative models. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, pages 50--59. PMLR, 2018.
[2]	Brent Asrican, George J. Augustine, Ken Berglund, Susu Chen, Nick Chow, Karl Deisseroth, Guoping Feng, Bernd Gloss, Riichiro Hira, Carolin Hoffmann, Haruo Kasai, Malvika Katarya, Jinsook Kim, John Kudolo, Li Ming Lee, Shun Qiang Lo, James Mancuso, Masanori Matsuzaki, Ryuichi Nakajima, Li Qiu, Gregory Tan, Yanxia Tang, Jonathan T. Ting, Sachiko Tsuda, Lei Wen, Xuying Zhang, and Shengli Zhao. Next-generation transgenic mice for optogenetic analysis of neural circuits. Frontiers in Neural Circuits, 7:160--160, 2013.
[3]	Frederico A.C. Azevedo, Ludmila R.B. Carvalho, Lea T. Grinberg, José Marcelo Farfel, Renata E.L. Ferretti, Renata E.P. Leite, Wilson Jacob Filho, Roberto Lent, and Suzana Herculano-Houzel. Equal numbers of neuronal and nonneuronal cells make the human brain an isometrically scaled-up primate brain. The Journal of Comparative Neurology, 513(5):532--541, 2009.
[4]	B. J. Baars. A cognitive theory of consciousness. Cambridge University Press, New York, NY, 1988.
[5]	Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 4509--4517. Curran Associates Inc., 2016.
[6]	Yoshua Bengio. The consciousness prior. CoRR, arXiv:1709.08568, 2017.
[7]	Staci D. Bilbo. Early-life infection is a vulnerability factor for aging-related glial alterations and cognitive decline. Neurobiology of Learning and Memory, 94(1):57--64, 2010.
[8]	Staci D. Bilbo and J.M. Schwarz. The immune system and developmental programming of brain and behavior. Frontiers in Neuroendocrinology, 33(3):267--286, 2012.
[9]	Jeffrey R. Binder and Rutvik H. Desai. The neurobiology of semantic memory. Trends in Cognitive Science, 15:527--536, 2011.
[10]	Dan Bohus. Error Awareness and Recovery in Conversational Spoken Language Interfaces. PhD thesis, Carnegie Mellon University, 2007.
[11]	Dan Bohus and Alexander I. Rudnicky. The RavenClaw dialogue management framework: architecture and systems. Computer Speech & Language, 23:332--361, 2009.
[12]	Dan Bohus and Alexander I. Rudnicky. The RavenClaw Dialogue Management Framework: Architecture and Systems. Computer Speech and Language, 23:332--361, 2009.
[13]	R. L. Buckner. The cerebellum and cognitive function: 25 years of insight from anatomy and neuroimaging. Neuron, 80(3):807--815, 2013.
[14]	Neil Burgess, Eleanor A Maguire, and John O'Keefe. The human hippocampus and spatial and episodic memory. Neuron, 35(4):625--641, 2002.
[15]	B.E. Carpenter, R.W. Doran, A.M. Turing, and M. Woodger. A.M. Turing's ACE report of 1946 and other papers. Charles Babbage Institute reprint series for the history of computing. MIT Press, 1986.
[16]	Nick Chater. The Mind is Flat: The Illusion of Mental Depth and The Improvised Mind. Penguin Books Limited, 2018.
[17]	Nick Chater and Morten H. Christiansen. A solution to the logical problem of language evolution: language as an adaptation to the human brain. In Kathleen R. Gibson and Maggie Tallerman, editors, The Oxford Handbook of Language Evolution. Oxford University Press, 2011.
[18]	Nick Chater and Morten H. Christiansen. Language acquisition as skill learning. Current Opinion in Behavioral Sciences, 2018.
[19]	Nick Chater, Stewart M. McCauley, and Morten H. Christiansen. Language as skill: Intertwining comprehension and production. Journal of Memory and Language, 89:244--254, 2016.
[20]	Noam Chomsky. Knowledge of Language: Its Nature, Origin and Use. Praeger, Santa Barbara, CA, 1986.
[21]	M. H. Christiansen and N. Chater. Language as shaped by the brain. Behavior Brain Science, 31(5):489--508, 2008.
[22]	M. H. Christiansen and Nick Chater. Generalization and connectionist language learning. Mind and Language, 9(3):273--87, 1994.
[23]	Morten H. Christiansen and Nick Chater. Toward a connectionist model of recursion in human linguistic performance. Cognitive Science, 23:157--205, 1999.
[24]	Michael C. Corballis. The Truth about Language: What It Is and Where It Came From. University of Chicago Press, 2017.
[25]	Andrea Crotti and Richard M. Ransohoff. Microglial physiology and pathophysiology: Insights from genome-wide transcriptional profiling. Immunity, 44:505--515, 2018.
[26]	Felipe De Brigard, R. Nathan Spreng, Jason P. Mitchell, and Daniel L. Schacter. Neural activity associated with self, other, and object-based counterfactual thinking. Neuroimage, 109:12--26, 2015.
[27]	Terrence W. Deacon. The Symbolic Species: The Co-evolution of Language and the Brain. W. W. Norton, 1998.
[28]	Terrence W. Deacon. Incomplete Nature: How Mind Emerged from Matter. W. W. Norton, 2012.
[29]	Thomas Dean. Interaction and negotiation in learning and understanding dialog. https://web.stanford.edu/class/cs379c/resources/dialogical/zanax_DOC.dir/index.html, 2014.
[30]	S. Dehaene, M. Piazza, P. Pinel, and L. Cohen. Three parietal circuits for number processing. Cognitive Neuropsychology, 20(3):487--506, 2003.
[31]	Stanislas Dehaene. The Number Sense: How the Mind Creates Mathematics. Oxford University Press, 1999.
[32]	Stanislas Dehaene. Consciousness and the Brain: Deciphering How the Brain Codes Our Thoughts. Viking Press, 2014.
[33]	Stanislas Dehaene, Michel Kerszberg, and Jean-Pierre Changeux. A neuronal model of a global workspace in effortful cognitive tasks. Proceedings of the National Academy of Sciences, 95:14529--14534, 1998.
[34]	Guy Deutscher. The Unfolding of Language: An Evolutionary Tour of Mankind's Greatest Invention. Henry Holt and Company, New York City, NY, 2006.
[35]	Jacob Devlin, Jonathan Uesato, Rishabh Singh, and Pushmeet Kohli. Semantic code repair using neuro-symbolic transformation networks. In International Conference on Learning Representations, 2018.
[36]	Mark Eisenberg, Tali Kobilo, Diego E. Berman, and Yadin Dudai. Stability of retrieved memory: Inverse correlation with trace dominance. Science, 301(5636):1102--1104, 2003.
[37]	Guy Even. Chapter 1: Digital abstraction. In Guy Even, editor, Computer Structure & Introduction to Digital Computers, pages 1--10. Department of Electrical Engineering & Systems, Tel-Aviv University, 2003.
[38]	Guy Even. Computer Structure & Introduction to Digital Computers. Department of Electrical Engineering & Systems, Tel-Aviv University, 2003.
[39]	Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. CoRR, arXiv:1703.03400, 2017.
[40]	Jerry A. Fodor and Zenon W. Pylyshyn. Connectionism and cognitive architecture. Cognition, 28(1-2):3--71, 1988.
[41]	Jakob N. Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. CoRR, abs/1709.04326, 2017.
[42]	C.R. Gallistel and A.P. King. Memory and the Computational Brain: Why Cognitive Science will Transform Neuroscience. Wiley, 2009.
[43]	H. Gärtner, M. Minnerop, P. Pieperhoff, A. Schleicher, K. Zilles, E. Altenmüller, and K. Amunts. Brain morphometry shows effects of long-term musical practice in middle-aged keyboard players. Frontiers in Psychology, 4:636--636, 2013.
[44]	Michael S. Gazzaniga. The Cognitive Neurosciences (Third Edition). Bradford Books. MIT Press, Cambridge, MA, 2009.
[45]	Felix Gers. Long short-term memory in recurrent neural networks. Diploma thesis. Écolepolytechnique Fédẃrale de Lausanne, 2001.
[46]	James J. Gibson. The Ecological Approach to Visual Perception. Houghton Mifflin, Boston, 1979.
[47]	Alex Graves, Greg Wayne, and Ivo Danihelka. Neural Turing machines. CoRR, arXiv:1410.5401, 2014.
[48]	Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdoménech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. Hybrid computing using a neural network with dynamic external memory. Nature, 538:471--476, 2016.
[49]	Paul Greengard and Eric J. Nestler. A conversation with paul greengard. Annual Review of Pharmacology and Toxicology, 53(1):1--16, 2013.
[50]	Edward Groshev, Aviv Tamar, Siddharth Srivastava, and Pieter Abbeel. Learning generalized reactive policies using deep neural networks. CoRR, arXiv:1708.07280, 2017.
[51]	Arthur Guez, Théophane Weber, Ioannis Antonoglou, Karen Simonyan, Oriol Vinyals, Daan Wierstra, Rémi Munos, and David Silver. Learning to search with MCTSnets. CoRR, arXiv:1802.04697, 2018.
[52]	Jessica B. Hamrick, Andrew J. Ballard, Razvan Pascanu, Oriol Vinyals, Nicolas Heess, and Peter W. Battaglia. Metacontrol for adaptive imagination-based optimization. CoRR, arXiv:1705.02670, 2017.
[53]	Demis Hassabis and Eleanor A. Maguire. Deconstructing episodic memory with construction. Trends in Cognitive Science, 11:299--306, 2007.
[54]	Donald O. Hebb. The organization of behavior: A neuropsychological theory. Wiley, New York, 1949.
[55]	Suzana Herculano-Houzel, Bruno Mota, Peiyan Wong, and Jon H. Kaas. Connectivity-driven white matter scaling and folding in primate cerebral cortex. Proceedings of the National Academy of Sciences, 107:19008--19013, 2010.
[56]	Gregory Hickok and David Poeppel. The cortical organization of speech processing. Nature Reviews Neuroscience, 8:393, 2007.
[57]	Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computing, 9:1735--1780, 1997.
[58]	Sepp Hochreiter, Arthur Younger, and Peter R. Conwell. Learning to learn using gradient descent. In Hornik K. Dorffner G., Bischof H., editor, ICANN 2001. Lecture Notes in Computer Science, volume 2130, pages 87--94. Springer, Berlin, Heidelberg, 2001.
[59]	Douglas Hofstadter. Gödel, Escher, Bach: An Eternal Golden Braid. Basic Books, New York, NY, 1979.
[60]	Douglas Hofstadter. Fluid Concepts and Creative Analogies: Computer Models of the Fundamental Mechanisms of Thought. Basic Books, 1996.
[61]	Douglas Hofstadter. Epilogue: Analogy as the core of cognition. In Dedre Gentner, Keith J. Holyoak, and Boicho N. Kokinov, editors, The Analogical Mind: Perspectives from Cognitive Science, pages 499--538. MIT Press, 2001.
[62]	Douglas Hofstadter. I Am a Strange Loop. Basic Books, 2007.
[63]	S. H. Horng and M. Sur. Visual activity and cortical rewiring: activity-dependent plasticity of cortical networks. Progress in Brain Research, 157:3--11, 2006.
[64]	D. H. Hubel and T. N Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. Journal of Physiology, 160:106--154, 1962.
[65]	D. H. Hubel and T. N Wiesel. Receptive fields and functional architecture of monkey striate cortex. Journal of Physiology, 195:215--243, 1968.
[66]	Alan Jasanoff. The Biological Mind: How Brain, Body, and Environment Collaborate to Make Us Who We Are. Basic Books, 2018.
[67]	D. Jurafsky and J.H. Martin. Speech and Language Processing. Pearson, 2014.
[68]	Trenton Kriete, David C. Noelle, Jonathan D. Cohen, and Randall C. O'Reilly. Indirection and symbol-like processing in the prefrontal cortex and basal ganglia. Proceedings of the National Academy of Sciences, 2013.
[69]	Frank Krueger, Maria Vittoria Spampinato, Matteo Pardini, Sinisa Pajevic, Jacqueline N. Wood, George H. Weiss, Steffen Landgraf, and Jordan Grafman. Integral calculus problem solving: an fmri investigation. Neuroreport, 19:1095--1099, 2008.
[70]	Xie Diana L, Miller Steven L, Boucher Leanne, Kubie John L, and Garièpy Jean François. NEURO.TV: Neuroscience education on the internet. Malaysian Journal of Medical Science, 21:78--80, 2014.
[71]	Shane A. Liddelow, Kevin A. Guttenplan, Laura E. Clarke, Frederick C. Bennett, Christopher J. Bohlen, Lucas Schirmer, Mariko L. Bennett, Alexandra E. Munch, Won-Suk Chung, Todd C. Peterson, Daniel K. Wilton, Arnaud Frouin, Brooke A. Napier, Nikhil Panicker, Manoj Kumar, Marion S. Buckwalter, David H. Rowitch, Valina L. Dawson, Ted M. Dawson, Beth Stevens, and Ben A. Barres. Neurotoxic reactive astrocytes are induced by activated microglia. Nature, 541:481--487, 2017.
[72]	C. J. MacDonald, K. Q. Lepage, U. T. Eden, and H. Eichenbaum. Hippocampal 'time cells' bridge the gap in memory for discontiguous events. Neuron, 71(4):737--749, 2011.
[73]	D. Marr and Giles Skey Brindley. Simple memory: a theory for archicortex. Philosophical Transactions of the Royal Society of London. B, Biological Sciences, 262(841):23--81, 1971.
[74]	Jansen Martin. The energy landscape concept and its implications for synthesis planning. Pure and Applied Chemistry, 86, 2014.
[75]	Richard Masland. Neuronal cell types. Current Biology, 14, 2004.
[76]	Tetsuro Matsuzawa. Comparative cognitive development. Developmental Science, 10(1):97–103, 2007.
[77]	Tetsuro Matsuzawa. Cognitive development in chimpanzees: A trade-off between memory and abstraction? In D. Mareschal, P. C. Quinn, and S. E. G. Lea, editors, Oxford series in developmental cognitive neuroscience. The making of human concepts, pages 227--244. Oxford University Press, New York, NY, US, 2010.
[78]	J. L. McClelland and N. H. Goddard. Considerations arising from a complementary learning systems perspective on hippocampus and neocortex. Hippocampus, 6(6):654--665, 1996.
[79]	Drew McDermott. Tarskian semantics, or no notation without denotation! Cognitive Science, 2(3):277--282, 1978.
[80]	Marvin Minsky and Seymour Papert. Perceptrons. MIT Press, Cambridge, Massachusetts, 1969.
[81]	Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. CoRR, arXiv:1312.5602, 2013.
[82]	Richard G.M. Morris, Jennifer Inglis, James A. Ainge, Henry J. Olverman, Jane Tulloch, Yadin Dudai, and Paul A.T. Kelly. Memory reconsolidation: Sensitivity of spatial memory to inhibition of protein synthesis in dorsal hippocampus during encoding and retrieval. Neuron, 50(3):479--489, 2006.
[83]	M. Moscovitch, R. Cabeza, G. Winocur, and L. Nadel. Episodic memory and beyond: The hippocampus and neocortex in transformation. Annual Review of Psychology, 67:105--134, 2016.
[84]	Arun Naira, Praveen Srinivasana, Sam Blackwella, Cagdas Alciceka, Rory Fearona, Alessandro De Mariaa, Vedavyas Panneershelvama, Mustafa Suleymana, Charles Beattiea, Stig Petersena, Shane Legga, Volodymyr Mniha, Koray Kavukcuoglua, and David Silver. Massively parallel methods for deep reinforcement learning. CoRR, arXiv:1507.04296, 2015.
[85]	Dana Nau, Tsz-Chiu Au, Okhtay Ilghami, Ugur Kuter, J. William Murdock, Dan Wu, and Fusun Yaman. SHOP2: An HTN planning system. Journal of Artificial Intelligence Research, 20:379--404, 2003.
[86]	K. Nelson. Language in Cognitive Development: The Emergence of the Mediated Mind. Cambridge University Press, 1998.
[87]	John von Neumann. First draft of a report on the EDVAC. Technical report, Institute for Advanced Study, 1945.
[88]	Joshua P. Neunuebel and James J. Knierim. Ca3 retrieves coherent representations from degraded input: Direct evidence for ca3 pattern completion and dentate gyrus pattern separation. Neuron, 81(2):416--427, 2014.
[89]	Randall C. O'Reilly. Biologically based computational models of high-level cognition. Science, 314:91--94, 2006.
[90]	Randall C. O'Reilly and Michael J. Frank. Making working memory work: A computational model of learning in the prefrontal cortex and basal ganglia. Neural Computation, 18:283--328, 2006.
[91]	Randall C. O'Reilly, Thomas E. Hazy, and Seth A. Herd. The Leabra cognitive architecture: How to play 20 principles with nature and win! In Susan E. F. Chipman, editor, The Oxford Handbook of Cognitive Science, Oxford Handbooks, pages 91--115. Oxford University Press, 2016.
[92]	Randall C. O’Reilly, Rajan Bhattacharyya, Michael D. Howard, and Nicholas Ketz. Complementary learning systems. Cognitive Science, 38(6):1229--1248, 2014.
[93]	Randall C. O’Reilly, Alex A. Petrov, Jonathan D. Cohen, Christian J. Lebiere, Seth A. Herd, and Trent Kriete. How limited systematicity emerges: A computational cognitive neuroscience approach. In Paco Calvo and John Symons, editors, The Architecture of Cognition, pages 191--224. MIT Press, Cambridge, Massachusetts, 2014.
[94]	N. Parikh, L. Ruzic, G. W. Stewart, R. N. Spreng, and F. De Brigard. What if? Neural activity underlying semantic and episodic counterfactual thinking. Neuroimage, 178:332--345, 2018.
[95]	Razvan Pascanu, Yujia Li, Oriol Vinyals, Nicolas Heess, Lars Buesing, Sébastien Racanière, David P. Reichert, Theophane Weber, Daan Wierstra, and Peter Battaglia. Learning model-based planning from scratch. CoRR, arXiv:1707.06170, 2017.
[96]	Wilder Penfield and Edwin Boldrey. Somatic motor and sensory representation in the cerebral cortex of man as studied by electrical stimulation. Brain, 60(4):389--443, 1937.
[97]	Hyun-Jae Pi, Balàzs Hangya, Duda Kvitsiani, Joshua I. Sanders, Z. Josh Huang, and Adam Kepecs. Cortical interneurons that specialize in disinhibitory control. Nature, 503:521–524, 2013.
[98]	Steven Pinker. The Language Instinct: How the Mind Creates Language. HarperCollins Publishers, New York, NY, 1994.
[99]	Marco Prinz, Daniel Erny, and Nora Hagemeyer. Ontogeny and homeostasis of CNS myeloid cells. Nature Immunology, 18:385--392, 2017.
[100]	Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adrià Puigdomènech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2827--2836, International Convention Centre, Sydney, Australia, 2017. PMLR.
[101]	Neil C. Rabinowitz, Frank Perbet, H. Francis Song, Chiyuan Zhang, S.M. Ali Eslami, and Matthew Botvinick. Machine theory of mind. CoRR, arXiv:1802.07740, 2018.
[102]	Charles Rich and Richard C. Waters. The programmer's apprentice: A research overview. Computer, 21(11):10--25, 1988.
[103]	Charlies Rich and Richard C. Waters. Automatic programming: myths and prospects. IEEE Computer, 21, 1988.
[104]	Martin A. Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing - solving sparse reward tasks from scratch. CoRR, arXiv:1802.10567, 2018.
[105]	D. E. Rumelhart and J. L. McClelland, editors. Parallel Distributed Processing, Volume 1 - Explorations in the Microstructure of Cognition: Foundations. MIT Press, Cambridge, MA, 1986.
[106]	D. E. Rumelhart and J. L. McClelland, editors. Parallel Distributed Processing, Volume 2 - Explorations in the Microstructure of Cognition: Psychological and Biological Models. MIT Press, Cambridge, MA, 1986.
[107]	T. Sakai, A. Mikami, J. Suzuki, T. Miyabe-Nishiwaki, M. Matsui, M. Tomonaga, Y. Hamada, T. Matsuzawa, H. Okano, and K. Oishi. Developmental trajectory of the corpus callosum from infancy to the juvenile stage: Comparative MRI between chimpanzees and humans. PLoS ONE, 12(6):e0179624, 2017.
[108]	Michael W. Salter and Beth Stevens. Microglia emerge as central players in brain disease. Nature Medicine, 23:1018--1027, 2017.
[109]	Daniel M. Salz, Zoran Tiganj, Srijesa Khasnabish, Annalyse Kohley, Daniel Sheehan, Marc W. Howard, and Howard Eichenbaum. Time cells in hippocampal area ca3. Journal of Neuroscience, 36(28):7476--7484, 2016.
[110]	Anton Maximilian Schäfer and Hans Georg Zimmermann. Recurrent neural networks are universal approximators. In Proceedings of the 16th International Conference on Artificial Neural Networks - Volume Part I, pages 632--640. Springer-Verlag, 2006.
[111]	Roger C. Schank. Conceptual dependency: A theory of natural language understanding. Cognitive Psychology, 3(4):552--631, 1972.
[112]	Roger C. Schank and Robert P. Abelson. Scripts, Plans, Goals and Understanding: An Inquiry into Human Knowledge Structures. Lawrence Erlbaum, Hillsdale, New Jersey, 1977.
[113]	Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, volume arXiv:1704.04368, 2017.
[114]	David Silver, Hado van Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnold, David P. Reichert, Neil C. Rabinowitz, André Barreto, and Thomas Degris. The predictron: End-to-end learning and planning. Proceedings of the 34th International Conference on Machine Learning, 2017.
[115]	Megan K. Smith, Robert Trivers, and William von Hippel. Self-deception facilitates interpersonal persuasion. Journal of Economic Psychology, 2017.
[116]	Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. Weakly supervised memory networks. CoRR, arXiv:1503.08895, 2015.
[117]	Uygar Sümbül, Aleksandar Zlateski, Ashwin Vishwanathan, Richard H. Masland, and H. Sebastian Seung. Automated computation of arbor densities: a step toward identifying neuronal cell types. Frontiers in Neuroanatomy, 8, 2014.
[118]	M. Sur, A. Angelucci, and J. Sharma. Rewiring cortex: the role of patterned activity in development and plasticity of neocortical circuits. Journal of Neurobiology, 41(1):33--43, 1999.
[119]	I. Sutskever and G. E. Hinton. Deep, narrow sigmoid belief networks are universal approximators. Neural Computing, 20(11):2629--2636, 2008.
[120]	Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial. Intelligence, 112(1-2):181--211, 1999.
[121]	Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value iteration networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems. Curran Associates Inc., 2016.
[122]	Charles Taylor. The Language Animal: The Full Shape of the Human Linguistic Capacity. Harvard University Press, Cambridge, MA, 2016.
[123]	T. J. Teyler and P. DiScenna. The hippocampal memory indexing theory. Behavioral Neuroscience, 100(2):147--154, 1986.
[124]	Sathesan Thavabalasingam, Edward B. O`Neil, Jonathan Tay, Adrian Nestor, and Andy C. H. Lee. Evidence for the incorporation of temporal duration information in human hippocampal long-term memory sequence representations. Proceedings of the National Academy of Sciences, 2019.
[125]	Robert Trivers. The Folly of Fools: The Logic of Deceit and Self-Deception in Human Life. Basic Books, New York, NY, 2011.
[126]	E. Tulving, W. Donaldson, and G.H. Bower. Organization of memory. Academic Press, 1972.
[127]	Endel Tulving. Elements of Episodic Memory. Oxford Psychology Series. Oxford University Press, 1985.
[128]	F.J. Varela, E. Rosch, and E. Thompson. The Embodied Mind: Cognitive Science and Human Experience. MIT Press, 1991.
[129]	Oriol Vinyals and Quoc V. Le. A neural conversational model. In ICML Deep Learning Workshop, 2015.
[130]	J. Volkmann, C. Daniels, and K. Witt. Neuropsychiatric effects of subthalamic neurostimulation in Parkinson disease. Nature Reviews Neurology, 6(9):487--498, 2010.
[131]	Christopher S. von Bartheld, Jami Bahney, and Suzana Herculano-Houzel. The search for true numbers of neurons and glial cells in the human brain: A review of 150 years of cell counting. J Comp Neurol, 524:3865--3895, 2016.
[132]	John von Neumann. Probabilistic logics and the synthesis of reliable organisms from unreliable components. In Claude E. Shannon and John McCarthy, editors, Automata Studies, pages 329--378. Princeton University Press, Princeton, NJ, 1956.
[133]	David J. Wales. Exploring energy landscapes. Annual Review of Physical Chemistry, 69(1):401--425, 2018.
[134]	Jane X. Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as a meta-reinforcement learning system. Nature Neuroscience, 21:860--868, 2018.
[135]	Ke Wang, Rishabh Singh, and Zhendong Su. Dynamic neural program embedding for program repair. CoRR, arXiv:1711.07163, 2017.
[136]	Greg Wayne, Chia-Chun Hung, David Amos, Mehdi Mirza, Arun Ahuja, Agnieszka Grabska-Barwinska, Jack Rae, Piotr Mirowski, Joel Z. Leibo, Adam Santoro, Mevlana Gemici, Malcolm Reynolds, Tim Harley, Josh Abramson, Shakir Mohamed, Danilo Rezende, David Saxton, Adam Cain, Chloe Hillier, David Silver, Koray Kavukcuoglu, Matt Botvinick, Demis Hassabis, and Timothy Lillicrap. Unsupervised predictive memory in a goal-directed agent. CoRR, arXiv:1803.10760, 2018.
[137]	Theophane Weber, Sébastien Racanière, David P. Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adrià Puigdomènech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, Razvan Pascanu, Peter Battaglia, David Silver, and Daan Wierstra. Imagination-augmented agents for deep reinforcement learning. CoRR, arXiv:1707.06203, 2017.
[138]	Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. CoRR, arXiv:1410.3916, 2014.
[139]	D. J. Willshaw, P. Dayan, and R. G. M. Morris. Memory, modelling and marr: a commentary on marr (1971) simple memory: a theory of archicortex. Philosophical Transactions of the Royal Society B: Biological Sciences, 370(1666):20140383, 2015.
[140]	G. Xue, Z. Lu, I. P. Levin, J. A. Weller, X. Li, and A. Bechara. Functional dissociations of risk and reward processing in the medial prefrontal cortex. Cerebral Cortex, 19(5):1019--1027, 2009.
[141]	Yao Zhu, Saksham Suri, Pranav Kulkarni, Yueru Chen, Jiali Duan, and C.-C. Jay Kuo. An interpretable generative model for handwritten digit image synthesis. CoRR, arXiv:1811.04507, 2018.