# Bayesian Statistics for Psychologists (Psych 201S) ## Summer 2016 [Link to class repo](https://github.com/mhtess/psych201s) ### Overview Learning statistics is like learning pottery. With pottery, you can learn how to make different shapes (e.g. a bowl, a vase, a spoon) without understanding general principles. The other way is to learn the basic strokes of forming pottery (e.g. how to mold a curved surface, a flat surface, long pointy things). In this course, we are going to learn the basic strokes of statistics, and compose those strokes to make shapes you've seen before (e.g. a t-test), some shapes you've probably never seen before, and develop ideas how you would make new shapes if you needed to. We won't learn *what tests apply to what data types* but instead foster the ability to reason through data analysis. We will do this through the lens of Bayesian statistics, though the basic ideas will aid your understanding of classical (frequentist) statistics as well. Bayesian data analysis is a general purpose data analysis approach for making explicit hypotheses about where the data came from (e.g. the hypothesis that data from 2 experimental conditions came from two different distributions). In this course, we will explore and learn how to use Bayesian data analytic tools for analyzing data from psychology experiments. The course will focus on the practicalities of running Bayesian analyses, of describing analyses for purposes of publication, and of making inferences about data and design decisions for subsequent experiments. This course is ideal for graduate or advanced-undergraduate students in Psychology, Linguistics, and related fields, who conduct experiments on human behavior; appropriate for students with limited statistical background but who actively think about the connection between experimental data and scientific hypotheses. This course is designed to engage with empirical data. Though we will often engage with example data sets from the textbook, the final project will have you look at some of your own data afresh through the Bayesian lens. If you have not run an experiment before or do not have your own data, come talk to me and I will set you up with a data set. + Instructor: Michael Henry (MH) Tessler (`mtessler` at stanford dot edu) + TA: Robert Hawkins (`rxdh` at stanford dot edu) + Meeting time: Mon, Wed 1:30 - 3:20 + Meeting place: 200-219 (History Corner) + Office hours (MHT): [A la carte](https://mht.youcanbook.me) + Office hours (RXDH): Tues 2-4pm + Textbook: *Bayesian Cognitive Modeling: A Practice Course*, Lee & Wagenmakers (2014) [below, LW]. [Available online](http://ebooks.cambridge.org/ebook.jsf?bid=CBO9781139087759) through Cambridge University Press, via Stanford. We will also read some excerpts from *Introduction to Applied Bayesian Statistics and Estimation for Social Scientists* by Scott M. Lynch (2007) [below, Lynch]. **If you have a question during the quarter**, probably somebody else either currently has or has had the same question. Hence, go to [Piazza](http://piazza.com/stanford/summer2016/psych201s/), look for your question, ask your question! And get answers! Actually, it's a forum for the whole class. So you as well can post answers! (And it will give you bonus participation credit!) Your answers don't necessarily have to be right. We will verify whatever you post. It's basically just an extension of class discussion. [Piazza signup](http://piazza.com/stanford/summer2016/psych201s/) ### Learning goals. By the end of the course, students will be able to: 0. Feel comfortable with statistics, both with the analysis you already do and with understanding analyses in general 1. Build Bayesian statistical models for simple and complex problems in your data using a probabilisitic programming language (e.g. [WebPPL](http://webppl.org), BUGS/[JAGS](http://mcmc-jags.sourceforge.net), [STAN](http://mc-stan.org)) 2. Interpret the various components of such a model in terms of your scientific hypotheses 3. Relate this model to more familiar, statistical tests (e.g. t-test, linear model) you may run 4. Defend a particular model specification (priors, likelihood) in a way that other psychologists will understand 5. Explain (3)-(5) in the context of a manuscript (ideally, a manuscript you are currently working on) ### Class structure. Our class meets for 1h50m twice a week. The class will be divided into 2 x 50m chunks, with a 10 minute break in between. During the first 50 minutes, I will lecture and/or the group will discuss the relevant topics of the day. During the second 50 minutes, you will engage with the practicalities of Bayesian analyses. This will include practical coding exercises and writing exercises. Most of this will be done with another student (pair work). ### Final project. The goal of this course is to make you more comfortable with the basic strokes of statistics. To that end, you will apply the Bayesian statistical approach outlined in the first few weeks of class to your own experimental data. You will be asked to justify your choice of model specification (or, introduce a set of alternative specifications corresponding to alternative hypotheses), possibly by drawing analogies to some of the types of models we've seen in the course. Tell us about your experiment, the task, the form of the data, your hypotheses, and what assumptions are in your model. Describe the output in a way that directly relates to your hypotheses. You will tell the class about your project and analysis in a lightning talk (~ 5 min + Q&A) on 8/3. You will also write this up in an "analysis section" that could be included in a manuscript you plan to write up for your project (i.e., for peer reviewed publication; you do not need to write up a manuscript for this class). Be sure to take whatever feedback you may get during your lightning talk and incorporate it into your write-up. If you are taking this class Credit/No-credit, you are only required to do one of the above two assignments (lightning talk vs write-up). ### Course grade. + 25% in-class practicum participation + 25% discussion participation (in class and/or piazza) + 25% final project: lightning talk (due 8/3, in class) + 25% final project: "Analysis and results section" (due 8/8, by 5pm) ### Resources + [R Data Wrangling cheatsheet](http://www.rstudio.com/wp-content/uploads/2015/02/data-wrangling-cheatsheet.pdf) + [R ggplot2 visualization cheatsheet](http://www.rstudio.com/wp-content/uploads/2015/12/ggplot2-cheatsheet-2.0.pdf) + [Intro to WebPPL language](http://dippl.org/chapters/02-webppl.html) + [WebPPL documentation: Distributions](http://webppl.readthedocs.io/en/master/distributions.html) + [WebPPL documentation: Array functions](http://webppl.readthedocs.io/en/master/arrays.html) + [WebPPL documentation: Other functions](http://webppl.readthedocs.io/en/master/header.html) + [ggplot2 book Ed.2](https://www.dropbox.com/s/0lap7tr7zoryyjk/ggplot2-book.pdf?dl=0), with stuff on tidyr, dplyr ### Practicum Doing Bayesian data analysis is made simpler with programming languages with built-in modes for sampling from distributions and conditioning on data. This course uses the programming language [WebPPL](http://webppl.org/), a light-weight probabilistic language embedded in JavaScript. WebPPL can called out to from R using the R package [RWebPPL](https://github.com/mhtess/rwebppl). At this time, RWebPPL is only supported for Mac and Linux systems. (WebPPL, however, can be run on the command-line in Windows.) All of the practica can be found in the [class git repo](https://github.com/mhtess/psych201s), and are also linked to in the topic column of the schedule. The practica use a custom WebPPL package `utils`, which can be downloaded in a .zip from [here (last update: 8/6/2016)](http://web.stanford.edu/class/psych201s/psych201s/practicums/utils.zip) or found in the repo. ### Day by day. | Week | Date | Meta-topic | Topic / Practicum | Readings / Home assignment | |------ |------ |------------ |------- |---------------------------- | | 1 | 6/20 |Basic probability| Course overview. A brief history of probability. What is probability? <ul><li>Practicum: [Distributions in R.](https://raw.githubusercontent.com/mhtess/psych201s/master/practicums/distributions.Rmd)</li></ul> | <ul><li>[Download JASP](https://jasp-stats.org/download/)</li><li>Watch tutorial on [Bayesian Binomial Test](https://www.youtube.com/watch?v=rchMvOGOW1k) and apply it to [this data set](http://web.stanford.edu/class/psych201s/psych201s/data/)</li><li>Lindley (1993). [The Analysis of Experimental Data: The Appreciation of Tea and Wine.](http://www2.isye.gatech.edu/~brani/isyebayes/bank/lindleybayeslady.pdf)</li></ul> | | | 6/22 |Generative models are hypotheses | Probability as rational degrees of belief. Rules of probability. <ul><li>Practicum: [Generative models and basic functional programming in WebPPL.](https://raw.githubusercontent.com/mhtess/psych201s/master/practicums/generative-models.Rmd) </li></ul>| <ul><li>Finish [Distributions in R.](https://raw.githubusercontent.com/mhtess/psych201s/master/practicums/distributions.Rmd)</li><li>Brewer, Brenden. (2015) [The Great Statistical Schism.](http://quillette.com/2015/11/13/the-great-statistical-schism/) [blogpost]</li></ul> | | | 6/24 | Updating beliefs with data| Bayes' Theorem. <ul><li>Practicum: [Bayes' Rule.](https://raw.githubusercontent.com/mhtess/psych201s/master/practicums/bayes-rule.Rmd) </li><li>Practicum:</li>[Inferences with Binomials (Part 1)](https://raw.githubusercontent.com/mhtess/psych201s/master/practicums/ch3.Rmd)<li>Associated text: [Chapter 3](http://ebooks.cambridge.org/pdf_viewer.jsf?cid=CBO9781139087759A022&ref=false&pubCode=CUP&urlPrefix=cambridge&productCode=cbo)</li></ul>|<ul><li>[Chapter 1: Basics of Bayesian analysis](http://ebooks.cambridge.org/pdf_viewer.jsf?cid=CBO9781139087759A010&ref=false&pubCode=CUP&urlPrefix=cambridge&productCode=cbo): Read pages 3-6; 11-12 (incl., Boxes 1.1, 1.2); Think about exercises on pg 5</li><li>[Chapter 2: Getting started](http://ebooks.cambridge.org/pdf_viewer.jsf?cid=CBO9781139087759A017&ref=false&pubCode=CUP&urlPrefix=cambridge&productCode=cbo) Read only: section 2.2.1 (Pages 17-19; including Box 2.1)</li></ul>| | 2 | 6/27 | Estimation 1/2 | Bayesian benefits. Bayes in 3 ways. Distributions over parameters vs. predicted outcomes. <ul><li>Practicum: [Inferences with Binomials (Part 2)](https://raw.githubusercontent.com/mhtess/psych201s/master/practicums/ch3_part2.Rmd) </li> <li>Associated text: [Chapter 3](http://ebooks.cambridge.org/pdf_viewer.jsf?cid=CBO9781139087759A022&ref=false&pubCode=CUP&urlPrefix=cambridge&productCode=cbo)</li></ul>| <ul><li>Wagenmakers, Morey, Lee (in press). [Bayesian benefits for the pragmatic researcher.](https://osf.io/3tdh9/) *Current directions in Psychological Science*</li><li>Optional: Lee (2014). [The "new statistics" are built on fundamentally flawed foundations](https://webfiles.uci.edu/mdlee/Lee2014_NewStatistics.pdf)</li></ul>| | | 6/29 |Estimation 2/2 | Frequentist interpretation of probability. Confidence vs. credible intervals. <ul><li> Practicum: [Inferences with Gaussians](https://raw.githubusercontent.com/mhtess/psych201s/master/practicums/ch4.Rmd) </li><li> Associated text: [Chapter 4](http://ebooks.cambridge.org/pdf_viewer.jsf?cid=CBO9781139087759A029&ref=false&pubCode=CUP&urlPrefix=cambridge&productCode=cbo)</li> </ul>| <ul>Choose one: <li>Morey, Hoekstra, Rouder, Lee, Wagenmakers (2015). [The fallacy of placing confidence in confidence intervals.](https://learnbayes.org/papers/confidenceIntervalsFallacy/fundamentalError_PBR.pdf) *Psychonomic Bulletin & Review*</li><li>Hoekstra, Morey, Rouder, Wagenmakers (2014). [Robust misinterpretation of confidence intervals.](http://www.ejwagenmakers.com/inpress/HoekstraEtAlPBR.pdf) *Psychonomic Bulletin & Review*</li>Optional readings: <li>Miller and Ulrich (2015). [Interpreting confidence intervals: A comment on Hoekstra, Morey, Rouder, and Wagenmakers (2014)](http://link.springer.com/article/10.3758%2Fs13423-015-0859-7)</li><li>Hoekstra, Morey, Rouder, Wagenmakers (2015). [Continued misinterpretation of confidence intervals: response to Miller and Ulrich.](http://orca.cf.ac.uk/84471/1/IOA-20152016-72.pdf)</li></ul>| | 3 | 7/4 |---NO CLASS--- | ---NO CLASS--- |-----Independence day----| | | 7/6 | Linear regression | Linear and logistic regression. Random effects. [Bayesian regression tutorial repo](https://github.com/vasishth/BayesLMMTutorial) <ul><li>Practicum: [Regression](https://raw.githubusercontent.com/mhtess/psych201s/master/practicums/linear-regression.Rmd)</li><li>Model: [Linear regression](https://raw.githubusercontent.com/mhtess/psych201s/master/practicums/webppl_models/linearRegression.wppl)</li><li>Model: [Logistic regression](https://raw.githubusercontent.com/mhtess/psych201s/master/practicums/webppl_models/logisticRegression.wppl)</li><li>Model: [Gibson & Wu (2013): Fixed effects](https://raw.githubusercontent.com/mhtess/psych201s/master/practicums/webppl_models/fixedEffects.wppl)</li><li>Model: [Gibson & Wu (2013) Varying intercepts](https://raw.githubusercontent.com/mhtess/psych201s/master/practicums/webppl_models/varyingIntercepts.wppl)</li></ul> | <ul><li>Sorensen, Hohenstein, Vasishth. (under review) [Bayesian linear mixed models using Stan: A tutorial for psychologists, linguists, and cognitive scientists.](http://www.stanford.edu/class/psych201s/psych201s/papers/SorensenEtAl.pdf) </li></ul> | | | 7/8 | Building interesting models 1/2 | Plate notation vs. probabilistic programs. Patterns of inferences. Modeling different behaviors, individual differences <ul><li> Practicum: [Mixture Models](https://raw.githubusercontent.com/mhtess/psych201s/master/practicums/ch6_part1.Rmd) </li><li> Associated text: [Chapter 6: Latent Mixture Models (sections 6.1 - 6.4)](http://ebooks.cambridge.org/pdf_viewer.jsf?cid=CBO9781139087759A040&ref=false&pubCode=CUP&urlPrefix=cambridge&productCode=cbo)</li> </ul>| | | 4 | 7/11 | Building interesting models 2/2 |Individual differences. Mixture models (part 2) <ul><li> Practicum: [Mixture Models Pt. 2](https://raw.githubusercontent.com/mhtess/psych201s/master/practicums/ch6_part2.Rmd) </li><li> Associated text: [Chapter 6: Latent Mixture Models (sections 6.5 - 6.7)](http://ebooks.cambridge.org/pdf_viewer.jsf?cid=CBO9781139087759A040&ref=false&pubCode=CUP&urlPrefix=cambridge&productCode=cbo)</li> </ul>| | | | 7/13 | Hypothesis testing 1/2 | Model checking. Model comparison: Marginal likelihood and [Bayes Occam's Razor](https://probmods.org/occam's-razor.html); Bayes Factors; Posterior model probabilities. [An interactive visualization](http://rpsychologist.com/d3/bayes/) of Bayesian null hypothesis testing. <br> <ul><li>Practicum: [Ch. 7 exercises (Sections 7.1, 7.2, 7.3)](http://ebooks.cambridge.org/pdf_viewer.jsf?cid=CBO9781139087759A049&ref=false&pubCode=CUP&urlPrefix=cambridge&productCode=cbo) </li></ul>|**Project proposals due** <br>Read: <ul><li> [Chapter 7: Bayesian model comparison](http://ebooks.cambridge.org/pdf_viewer.jsf?cid=CBO9781139087759A049&ref=false&pubCode=CUP&urlPrefix=cambridge&productCode=cbo)</li></ul>| | 5 | 7/18 |Hypothesis testing 2/2 | Bayes Factors, p-values, effect sizes. More model checking. Issues with Bayes Factors. <ul><li>Practicum: [Selected exercises from Ch. 8 & 9](https://raw.githubusercontent.com/mhtess/psych201s/master/practicums/ch8_and_9.Rmd) </li><li> Associated text: [Chapter 8: Comparing Gaussian means](http://ebooks.cambridge.org/pdf_viewer.jsf?cid=CBO9781139087759A057&ref=false&pubCode=CUP&urlPrefix=cambridge&productCode=cbo)</li><li>Associated text: [Chapter 9: Comparing binomial probabilities](http://ebooks.cambridge.org/pdf_viewer.jsf?cid=CBO9781139087759A061&ref=false&pubCode=CUP&urlPrefix=cambridge&productCode=cbo)</li></ul> |<ul><li>Dienes and Mclatchie (2016). [Four reasons to prefer Bayesian over orthodox statistical analyses](http://web.stanford.edu/class/psych201s/psych201s/papers/Dienes,%20Mclatchie_2016.pdf)</li><li>Optional: Wetzels et al. (2011) [Statistical Evidence in Experimental Psychology](http://www.ejwagenmakers.com/2011/WetzelsEtAl2011_855.pdf). *Perspectives on Psychological Science*</li></ul>| | | 7/20 | There be dragons | Bem (2011). [Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect](http://psycnet.apa.org/journals/psp/100/3/407.pdf) | <ul><li>Etz (2016, blogpost). [Understanding Bayes: How to cheat to get the maximum Bayes factor for a given p-value](https://alexanderetz.com/2016/06/19/understanding-bayes-how-to-cheat-to-get-the-maximum-bayes-factor-for-a-given-p-value/)</li><li>Wagenmakers et al. (2011) [Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi: Comment on Bem (2011)](http://web.stanford.edu/class/psych201s/psych201s/papers/Wagenmakers-etal-2011-bemComment.pdf)</li><li>Bem, Utts, and Johnson (2011). [Must Psychologists Change the Way They Analyze Their Data?](http://web.stanford.edu/class/psych201s/psych201s/papers/Bem2011.pdf)</li><li>Wagenmakers et al. (2011) [Yes, Psychologists Must Change the Way They Analyze Their Data: Clarifications for Bem, Utts, and Johnson (2011)](http://web.stanford.edu/class/psych201s/psych201s/papers/ClarificationsForBemUttsJohnson.pdf)</li><li>[Chapter 13: Extrasensory Perception](http://ebooks.cambridge.org/pdf_viewer.jsf?cid=CBO9781139087759A079&ref=false&pubCode=CUP&urlPrefix=cambridge&productCode=cbo)</li></ul> | | 6 | 7/25 | | **Lightning Talks** / Bring your bayes to school day || | | 7/27 | The role and future of Bayesian data analysis in Psychology|Panel discussion with Mike Frank and Russ Poldrack|<ul><li>Gigerenzer, G. and Marewski, J. N. (2015). [Surrogate Science: The Idol of a Universal Method for Scientific Inference](http://www.dcscience.net/Gigerenzer-Journal-of-Management-2015.pdf)</li></ul>| | | 7/29 |Catch-up and an introduction to Bayesian Cognitive Modeling | | <ul><li>Gershman, S. J. (2016) [On the blessing of abstraction.](http://gershmanlab.webfactional.com/pubs/abstraction.pdf)</li></ul>| Unsorted readings: Scheibehenne, Jamil, Wagenmakers (in press). [Bayesian evidence synthesis can reconcile seemingly inconsistent results: The case of hotel towel reuse.](https://osf.io/nptv8/) *Psychological Science* Wagenmakers et al. (2015). [A power fallacy.](http://www.ejwagenmakers.com/2015/WagenmakersEtAlAPowerFallacy2015.pdf)