How to Run a Student Discussion Section

This is a quick primer for how to run a discussion section. Each of you will be responsible for leading one in-class discussion. This along with your participation in the discussions led by other students will account for 30% of grade. This shouldn't be onerous. It is intended to make it easy for you to get the most out of what this course has to offer, and hopefully it will give you an idea of how similar student-guided discussions are routinely conducted in research labs that include undergraduates, graduate students, postdocs and faculty from multiple disciplines all of whom have different backgrounds and specialties.

Outline of the Basic Selection and Execution Process

  1. Sign up for a topic. We will send you a link to a spreadsheet where you can enter your preferences.

  2. No more than four students will be assigned any given topic. Early topics will have at least three.

  3. If there is more than one person assigned your topic, get in touch with them and start planning.

  4. I'll send you examples of what you're expected to do and suggestions for reading and watching1.

  5. A week before the date for your discussion, send out links to your required reading & viewing.

  6. Familiarize yourself with how to use Zoom to lead your discussion, and coordinate with Rafael.

Example Topic Preparation, Materials and Readings

The topics for discussion sections are listed below. In this section, we provide some sample material that you could use if you are assigned either the first topic, actor-critic learning, or the third, action selection. The action-selection section builds on actor-critic learning and together they provide the basis for understanding the complex role of the basal ganglia in governing behavior in mammals and humans in particular.

The five topics listed under the heading "Functional Components" are ordered taking into account the dependencies between them, and together they provide the foundation for the rest of the class. I'll ask students to sign up for these five first and after we have completed the first two or three, we can select additional topics from the "Basic Representations" and "Fundamental Problems" lists in any order, since they have few if any dependencies constraining the order in which we discuss them.

In the 2018 instantiation of the class, Randy O'Reilly gave a great introduction on computational cognitive neuroscience and distributed representations. Randy worked with Jay McClelland and other members of the was part of the PDP (Parallel Distributed Processing) group at CMU where completed his PhD. He co-authored several influential papers on complementary learning systems [12, 13, 15].

In addition to his earlier talk, we met with Randy this summer for nearly two hours discussing the finer points of his PBWM (Prefrontal Cortex Basal Ganglia Working Memory —) theory [16]. We'll go over that meeting in covering the executive control topic in a subsequent discussion section, and you can find Randy's 2018 class discussion here.

Matt Botvinick's talk on the prefrontal cortex as meta reinforcement learning [23] is at the heart of the actor-critic topic and covered well in Matt's talk in class. For the PBWM, discussion it would make sense for the students to read the section entitled "Executive Function: Prefrontal Cortex and Basal Ganglia" in O'Reilly et al [18] — see Figure 2.

Take a look at these short — less than ten minutes each — videos on the Brain Inspired YouTube Channel:

There are similar videos covering the same material on the Khan Academy website. Here are two examples:

Figure 1:  In the paper [3] that Chaofei, Gene, Meg and I wrote last year, we fleshed out what we learned from the class in the Spring quarter and then expanded the material during the Fall to include four content boxes of the sort used in survey articles to focus on key concepts and provide additional detail. Each of Chaofei, Gene and Meg did a literature search on a general topic and then compressed what they learned to highlight specific technical content relevant to the rest of the paper. In developing presentation material for your discussion sections, you should try to do the same albeit in a substantially shorter span of time than we had to write the paper. Your task is made easier since your goal is to convey a basic understanding of the neuroscience and related technology, and there are plenty of online tutorials, lectures and papers that you can use.

The student-guided discussions last year were a success in that students learned more and learned it more quickly and thoroughly than if they had passively listened to an invited speaker. In addition to providing an overview and channeling expert content by alternating between showing video and providing their own explanations, student teams set the agenda by specifying papers to read prior to class and exchanged email with the experts providing content to answer questions related to their work.

The team consisting of Tyler Benster, Julia Gong, Jerry Meng, John Mern, Megumi Sano and Paul Warren reviewed Oriol Vinyal's presentation from 2018, elicited questions from the students and Oriol provided answers in return. In addition to student guided-discussions, last year we also had new invited talks from: Jessica Hamrick, Adam Marblestone, Loren Frank, Michael Frank and Peter Battaglia, there were several new speakers.

An alternative approach to finding speakers that is less intrusive than asking someone to go out of their way to produce a new talk tailored precisely to our interests, is for us to find interviews, public talks at conferences or invited speakers at Google, Stanford and other venues. For example, Alex Graves gave an excellent talk on neural Turing machines at a symposium on deep learning held at Microsoft Research that was recorded and posted on YouTube, and there is a more recent, condensed overview at the NIPS 2016 Symposium on Recurrent Neural Networks and Other Machines that Learn Algorithms.

Figure 2:  The text Computational Cognitive Neuroscience [18] covers most of the topics that we have selected for class this year and serves as a primary reference to simplify explanations in the discussion sections. Here we see a sequence of graphics taken from the O'Reilly text that would work well for the discussion section on the Prefrontal Cortex Basal Ganglia Working Memory (PBWM) model. For example, in explaining this model to the rest of the class, you could sequence through the graphics using them as visual talking points, starting from the upper center and continuing in a clockwise direction around the figure shown above.

List of Topics in Order of Presentation

Functional Components:

  1. actor-critic learning (basal ganglia and reinforcement learning) — April 14, Tuesday

  2. episodic memory (hippocampus and differentiable neural computers) — April 16, Thursday

  3. action selection (frontal cortex, striatum, thalamus and working memory) — April 21, Tuesday

  4. executive control (prefrontal cortex, planning complex cognitive behavior) — April 23, Thursday

  5. prediction-action cycle (reciprocally connected motor and sensory systems) — April 28, Tuesday

Basic Representations:

  1. relational learning (recognizing, learning and using relational models) — April 30, Thursday

  2. graph networks (learning, reasoning about connections between entities) — May 5, Tuesday

  3. interaction networks (learning and reasoning about dynamical systems) — May 7, Thursday

  4. analogical reasoning (combining existing models to create new models) — May 12, Tuesday

  5. perception as memory (attention, affordances, anticipatory behaviors) — May 14, Thursday

Fundamental Problems:

  1. algorithmic abstraction (when we generalize skills creating hierarchies) — May 19, Tuesday

  2. catastrophic forgetting (when we forget old skills in learning new ones) — May 21, Thursday

  3. partial observability (when the system is a higher-order Markov process) — May 26, Tuesday:

List of Representative Papers

O'Reilly [17] — An Introduction to Distributed Processing in the Brain

Botvinick [24] — The Prefrontal Cortex as Meta Reinforcement Learning

O'Reilly [16] — Prefrontal Cortex Basal Ganglia Working Memory (PBWM)

Daw, Frank, Gershman [4, 6] — Revisiting Reinforcement Learning and Episodic Memory

Graves, Weston [7, 25] — Memory Networks and Differentiable Neural Computers

Fuster, Botvinick [5, 2] — Fuster's Hierarchy and the Perception-Action Cycle

Stachenfeld et al [21] — Hippocampus as Predictive Model in Higher-level Cognition

Hamrick [8] — Relational Inductive Bias for Physical Constructions

Battaglia [1, 11] — Relational Inductive Biases, Learning and Graph Networks

Sanchez [19] — Graph Networks as Learnable Physics Engines for Inference

Hill et al [9] — Analogies by Contrasting Abstract Relational Structure

Mnih, Kaiser, Silver, Vinyalls [10, 22, 20, 14] — What have Atari, Go and StarCraft taught us about AI?

References

[1]   Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. CoRR, arXiv:1806.01261, 2018.

[2]   Matthew M. Botvinick. Multilevel structure in behaviour and in the brain: a model of fuster's hierarchy. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 362:1615--1626, 2007.

[3]   Thomas Dean, Chaofei Fan, Francis E. Lewis, and Megumi Sano. Biological blueprints for next generation AI systems. CoRR, arXiv:1912.00421, 2019.

[4]   Nicholas T. Franklin and Michael J. Frank. Compositional clustering in task structure learning. PLoS Computational Biology, 4:e1006116, 2018.

[5]   Joaquín M. Fuster. Chapter 8: An Overview of Prefrontal Functions, pages 375--425. Elsevier, London, 2015.

[6]   Samuel J. Gershman and Nathaniel D. Daw. Reinforcement learning and episodic memory in humans and animals: An integrative framework. Annual Reviews of Psychology, 68:101--128, 2017.

[7]   Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdoménech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. Hybrid computing using a neural network with dynamic external memory. Nature, 538:471--476, 2016.

[8]   Jessica B. Hamrick, Kelsey R. Allen, Victor Bapst, Tina Zhu, Kevin R. McKee, Joshua B. Tenenbaum, and Peter W. Battaglia. Relational inductive bias for physical construction in humans and machines. In Proceedings of the 40th Annual Conference of the Cognitive Science Society, pages 1773--1779, 2018.

[9]   Felix Hill, Adam Santoro, David G. T. Barrett, Ari S. Morcos, and Timothy P. Lillicrap. Learning to make analogies by contrasting abstract relational structure. In International Conference on Learning Representations, 2019.

[10]   Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model-based reinforcement learning for atari. CoRR, arXiv:1903.00374, 2019.

[11]   Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning deep generative models of graphs. In International Conference on Learning Representations, 2018.

[12]   James L. McClelland, Bruce L. McNaughton, and Randall C. O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102:419--457, 1995.

[13]   James L. McClelland, Bruce L. McNaughton, and Randall C. O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3):419--457, 1995.

[14]   Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529--533, 2015.

[15]   Randall C. O'Reilly, Rajan Bhattacharyya, Michael D. Howard, and Nicholas Ketz. Complementary learning systems. Cognitive Science, 38(6):1229--1248, 2014.

[16]   Randall C. O'Reilly and Michael J. Frank. Making working memory work: A computational model of learning in the prefrontal cortex and basal ganglia. Neural Computation, 18:283--328, 2006.

[17]   Randall C. O'Reilly, Yuko Munakata, Michael J. Frank, Thomas E. Hazy, and Contributors. Computational Cognitive Neuroscience. Wiki Book, 1st Edition, 2012.

[18]   Randall C. O'Reilly, Yuko Munakata, Michael J. Frank, Thomas E. Hazy, and Contributors. Computational Cognitive Neuroscience. Wiki Book, Third Edition, 2016.

[19]   Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin A. Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. CoRR, arXiv:1806.01242, 2018.

[20]   David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484--489, 2016.

[21]   Kimberly L. Stachenfeld, Matthew M. Botvinick, and Samuel J. Gershman. The hippocampus as a predictive map. Nature Neuroscience, 20:1643, 2017.

[22]   Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michal Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wunsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575:350--354, 2019.

[23]   Jane X. Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as a meta-reinforcement learning system. Nature Neuroscience, 21:860--868, 2018.

[24]   Jane X Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as a meta-reinforcement learning system. bioRxiv, 2018.

[25]   Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. CoRR, arXiv:1410.3916, 2014.

1 Raphael and I will help you find a suitable video lecture featuring one or more of the researchers working on your assigned topic. Everyone in the class will be expected to watch / review this prior to class and / or read one or more of the suggested relevant research papers.