{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Lecture 28: Outro\n", "\n", "STATS 60 / STATS 160 / PSYCH 10\n", "\n", "
\n", "\n", "
\n", "\n", "**Announcements**\n", "\n", "\n", "\n", "- Additional practice final.\n", "- Review sessions\n", " - Skyler: Thursday 1:00 -- 2:30 pm in CoDA E401\n", " - Michael: Friday 11:00 am -- 12:00 pm in CoDA B40\n", "\n", "\n", "
\n", "\n", "
\n", "\n", "\n", "
\n", "
\n", "\n", "\n", "## Today's lecture\n", "\n", "\n", "\n", "\n", "\n", "- Overview of the quarter\n", "- Three themes for the class\n", "- What\u2019s next if you\u2019re stats-curious?\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "# Remembering our journey\n", "\n", "# Unit 1: Thinking about scale\n", "\n", "## Numbers in statistics\n", "\n", "In statistics and data science, numbers are used to\n", "\n", "1. Describe observations\n", "\n", "2. Quantify how confident we are\n", "\n", "\n", "But, Numbers are only meaningful *in context.*\n", "\n", "- Is $10 billion a lot of money?\n", "\n", "
\n", "\n", "
\n", "\n", "
\"\"
\n", "\n", "\n", "
\n", "\n", "
\n", "\n", "
\"\"
\n", "\n", "
\n", "
\n", "\n", "\n", "## Three questions\n", "\n", "\n", "1) What type of number is this?\n", "\n", "2) What can I compare this number to? Is it large or small compared to other similar values?\n", "\n", "3) What would I have expected this number to be? \n", "\n", "## Ballpark estimates\n", "\n", "\n", "1) Set up a simple model to compute the quantity approximately by break up the estimate into small parts\n", "\n", " - How many visitors go on tours at Stanford per year?\n", "\n", " $$\\frac{\\text{# visitors}}{\\text{year}} = \\frac{\\text{# days}}{\\text{year}} \\times \\frac{\\text{# tours}}{\\text{day}} \\times \\frac{\\text{# visitors}}{\\text{day}}$$\n", "\n", "2) Approximate parts up to a factor of 10\n", "\n", "## Cost benefit analysis\n", "\n", "Ballpark estimates can be used to make important decisions.\n", "\n", "
\n", "\n", "
\n", "\n", "
\"\"
An insecticide treated bed net.
\n", "\n", "\n", "
\n", "\n", "
\n", "\n", "
\"\"
A malaria vaccine.
\n", "\n", "
\n", "
\n", "\n", "# Unit 2: Exploratory data analysis\n", "\n", "## Putting data in context\n", "\n", "It is hard to find insight from looking at raw data.\n", "\n", "
\"\"
\n", "\n", "\n", "Data visualization and data summaries let us choose what to communicate and focus on.\n", "\n", "## Data visualization \n", "\n", "Common graphic representations:\n", "\n", "- Bar chart and pie chart (categorical variables)\n", "- Time series (seeing how a variable changes over time)\n", "- Histogram (one quantitative variable)\n", "- Scatter plot (two quantitative variables).\n", "\n", "## Effective data visualization \n", "\n", "\n", "
\"\"
Our World in Data: Top of the Charts 2025.
\n", "\n", "\n", "\n", "\n", "\n", "## Deceptive data visualization\n", "\n", "Watch out for misleading data visualizations.\n", "\n", "
\n", "\n", "
\n", "\n", "
\"\"
\n", "\n", "
\n", "
\n", "\n", "
\"\"
\n", "\n", "
\n", "
\n", "\n", "## Summaries of center and variability\n", "\n", "Summaries of center: what is the *one number* that best summarizes the data?\n", "\n", "- Mean, median, mode.\n", "\n", "\n", "Variability: how similar are the different datapoints in the dataset?\n", "\n", "- Would you rather be given $\\$150$ or flip a coin for $\\$300$?\n", "- Variance, standard deviation, quantiles and gaps between quantiles.\n", "\n", "## Correlation and correlation coefficient\n", "\n", "\n", "What is the *direction* and *strength* of the association between two quantitative variables?\n", "\n", "
\"\"
\n", "\n", "## Misleading means\n", "\n", "The usefulness of a summary statistic depends on the data!\n", "\n", "
\n", "\n", "
\n", "\n", "\n", "Outliers/skew\n", "\n", "
\"\"
\n", "\n", "
\n", "\n", "
\n", "\n", "\n", "\n", "Multi-modal data\n", "\n", " \n", "
\"\"
\n", "\n", "
\n", "\n", "
\n", "\n", "\n", "\n", "Subgroups\n", "\n", "
\"\"
\n", "\n", "
\n", "\n", "
\n", "\n", " \n", "\n", "\n", "# Unit 3: Probability\n", "\n", "## Probability\n", "\n", "- The mathematics of uncertainty.\n", "- One of the foundations of statistics.\n", "- Probability helped us:\n", " - Assess how likely/unlikely coincidences are.\n", " - Update our beliefs based on new information (conditional probability).\n", " - Generalize findings from data to a broader group of people (hypothesis testing and confidence intervals).\n", "\n", "\n", "## Sample space and outcomes\n", "\n", "\n", "
\n", "\n", "
\n", "\n", "- The set of possible outcomes is called the **sample space**.\n", "- An **event** is a collection of some of the possible outcomes.\n", "- If all outcomes are equally likely, then the **probability** of an event is equal to the number of outcomes in the event divided by the total number of possible outcomes.\n", "\n", "
\n", "
\n", "\n", "
\"\"
\n", "\n", "
\n", "
\n", "\n", "## Computing probabilities\n", "\n", "
\n", "\n", "
\n", "\n", "The multiplication rule\n", "\n", "
\"\"
From The Art of Chance
\n", "\n", "
\n", "
\n", "\n", "\n", "Complement rule\n", "\n", "\n", "\n", "
\"\"
\n", "\n", "\n", "
\n", "
\n", "\n", "## Coincidences\n", "\n", "- Even if an event is rare, it is likely to happen when there are many opportunities for the rare event to take place.\n", "- Examples: \n", " - The birthday paradox\n", "\n", " - Winning streaks\n", "\n", " - Unit 4: multiple testing!\n", "\n", "## Conditional probability\n", "\n", "\n", "- Updating probabilities based on partial information\n", "\n", "- Bayes' rule\n", "\n", "- Common mistakes in conditional probability:\n", "\n", " - Base rate fallacy: the conditional probability is not informative by itself (librarians vs. farmers)\n", "\n", " - $\\Pr[A \\mid B] \\neq \\Pr[B \\mid A]$ (distracted driving, gateway drugs)\n", "\n", " - Failing to condition on important information (OJ Simpson)\n", "\n", " - Generalizing from a biased sample or failing to realize you have conditioned (hot guys are jerks, selection bias)\n", "\n", "\n", "\n", "\n", "\n", "\n", "# Unit 4: Estimates, hypothesis testing and experiments\n", "\n", "## Statistical significance\n", "\n", "- **Statistically significant** means that the results are unlikely to have occurred by random chance alone.\n", "- A **p-value** is the probability of finding a result *at least* as extreme/surprising, if outcomes happened by random chance alone.\n", "- The **null hypothesis** corresponds to \"just chance\" or \"no effect.\"\n", "- The **alternative hypothesis** corresponds to \"better than chance\" or \"an effect.\"\n", "\n", "## Computing p-value by simulation\n", "\n", "
\n", "\n", "
\n", "\n", "- A *simulation* shows what the results would have looked like if the null hypothesis was true.\n", "- Computing the proportion of repetitions that were *at least as extreme* as the observed data gives the p-value.\n", "- Can be one-sided or two-sided.\n", "\n", "
\n", "
\n", "\n", "
\"\"
\n", "\n", "
\n", "
\n", "\n", "## Experiments\n", "\n", "- Drawbacks of observational studies (marshmallow experiment)\n", "- Correlation vs. causation and confounding/hidden variables\n", "- Effect of selection bias\n", "- Potential outcomes model\n", "- Computing p-values with simulations and permutation tests\n", "\n", "## Estimates\n", "\n", "\n", "- Sample vs. population\n", "\n", "- Sample size matters for estimation!\n", " \n", " - The standard deviation of the sample mean is $\\frac{\\sigma}{\\sqrt{n}}$\n", " - To get $10$ times more accurate, you need $100$ times more samples.\n", "\n", "## Confidence intervals\n", "\n", "\n", "For large samples, the distribution of the sample mean is described by the \"normal distribution.\"\n", "\n", "\n", "\n", "This can be used to make confidence intervals:\n", "\n", "\n", "\n", "- $\\hat{\\mu}_n \\pm \\frac{\\hat{\\sigma}_x}{\\sqrt{n}}$ is a 68% confidence interval.\n", "- $\\hat{\\mu}_n \\pm 2 \\times \\frac{\\hat{\\sigma}_x}{\\sqrt{n}}$ is a 95% confidence interval.\n", "- $\\hat{\\mu}_n \\pm 3 \\times \\frac{\\hat{\\sigma}_x}{\\sqrt{n}}$ is a 99% confidence interval.\n", "\n", "\n", "\n", "\n", "For proportions, $\\hat{\\sigma}_x = \\sqrt{\\hat{\\pi}_n(1-\\hat{\\pi}_n)}$ where $\\hat{\\pi}_n$ is the sample proportion.\n", "\n", "\n", "\n", "\n", "# Unit 5: Machine Learning and Regression\n", "\n", "## Predictions\n", "\n", "Statistics is often concerned with making *predictions*.\n", "\n", "On observation $x$, predict outcome $y$.\n", "\n", "- $x$ is symptoms/test results, $y$ is diagnosis\n", "- $x$ is SAT score, $y$ is first-year GPA\n", "- $x$ is weather now, $y$ is weather later\n", "\n", "\n", "We construct a simple *model* $f$ so that $f(x) = \\hat{y}$, with the goal that $\\hat{y}$ is as close to $y$ as possible.\n", "\n", "## Building models\n", "\n", "\n", "- It is easier to learn from examples than build a model by hand\n", "- Use \"training\" data to build the model and \"testing\" data to evaluate the model.\n", "- Types of prediction problems \n", "\n", " - Regression (predicting a quantitative $y$).\n", " - Classification (predicting a categorical $y$).\n", " - Text generation (predicting the next word in a sentence).\n", "\n", "## Examples of models\n", "\n", "\n", "
\n", "\n", "
\n", "\n", "\n", "\n", "Linear and quadratic regression\n", "\n", "
\"\"
\n", "\n", "\n", "
\n", "\n", "
\n", "\n", "\n", "\n", "$k$-nearest neighbors\n", "\n", " \n", "
\"\"
\n", "\n", "
\n", "\n", "
\n", "\n", "\n", "\n", "Markov text generators\n", "\n", "
\"\"
\n", "\n", "
\n", "\n", "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "## Training data is everything!\n", "\n", "- Selection bias in training data leads to biased models.\n", "- If $x$ is far from all training examples, $f(x)$ is probably not that accurate for predicting $y$.\n", "- More (good) data and better coverage improves performance.\n", "\n", "# tl;dr: three themes\n", "\n", "The three major ideas that I want you to take away from this class.\n", "\n", "## Theme 1: Insight from simple models\n", "\n", "The world is complicated. \n", "\n", "Answering a question exactly is overwhelming and often impossible.\n", "\n", "\n", "Strategy: construct **a simple model** of the situation.\n", "\n", "\n", "At least within the simple model, we have the **power** to answer questions precisely and often quantitatively.\n", "\n", "## Theme 1: examples\n", "\n", "1) Ballpark estimates and cost-benefit analysis.\n", "\n", "2) Hypothesis testing.\n", "\n", "3) Machine learning and prediction.\n", "\n", "4) Decision-making in sports\n", "\n", "\n", "With great power comes great responsibility. \n", "\n", "\n", "**Know the strengths and limitations of your model.**\n", "\n", "## Theme 2: Conditioning matters\n", "\n", "We might understand an uncertain situation well, but everything can change if we condition!\n", "\n", "1) Common mistakes in conditional probability\n", "\n", "2) Selection bias and sampling bias \n", " - Hot guys are jerks\n", " - Biased estimates and biased ML predictions from biased training data\n", "\n", "3) Multi-modal data affects interpretation of summary statistics\n", " - Male vs. female penguin body mass\n", " - Does generic medical advice apply to you? \n", "\n", "## Theme 3: Critical thinking is essential\n", "\n", "Once you specify the model, statistics can give precise answers.\n", "\n", "\n", "*Is our model good? Does it fit the situation?*\n", "\n", "\n", "**Think critically! Don't calculate blindly.**\n", "\n", "## Theme 3: examples\n", "\n", "1) \"When means mislead\" \n", " - Usefulness of fundamental summary statistics (mean, median, standard deviation) depends on data (outliers, skew)\n", "\n", "2) Correlation vs. causation\n", " - Confounding variables\n", " - Experimental design\n", "\n", "3) Misleading graphs and figures\n", "\n", "4) Multiple testing and $p$-hacking\n", "\n", "## Thinking critically\n", "\n", "We did a lot of \"what does this mean in plain English?\" exercises. \n", "\n", "\n", "Thinking like this is important---I do this in my research and in my daily life constantly.\n", "\n", "Even though a concept is formal and/or technical, we can and should try to really understand.\n", "\n", "# Feedback\n", "\n", "I want to make STATS60 great!\n", "\n", "Please take a couple of minutes to give some feedback on the course this quarter.\n", "\n", "# I'm stats-curious. What's next?\n", " \n", "## If you like exploratory data analysis\n", "\n", "\n", "- DATASCI 112: Principles of Data Science\n", " - Deeper dive into data visualization and data analysis\n", " - More machine learning: how to train and evaluate ML models\n", " - Learn some programming essentials\n", "\n", "## If you like probability\n", "\n", "* STATS 117: Introduction to probability theory\n", " - Dive into probability theory\n", " - Simple discrete models (coinflips, bags of marbles)\n", " - Continuous models (Normal)\n", "\n", "* STATS 118: Probability theory for statistical inference\n", " - Deeper dive into probability theory\n", " - The theory behind the normal approximation\n", " - Math behind other hypothesis tests\n", "\n", "\n", "\n", "## If you like experiments and hypothesis testing\n", "\n", "* STATS 191: Introduction to Applied Statistics\n", " - Deeper dive into methods for data analysis and prediction\n", " - Applications to biology and social sciences\n", "\n", "\n", "After taking probability theory,\n", "\n", "* STATS 200: Introduction to Theoretical Statistics\n", " - Hypothesis testing\n", " - Estimation and confidence intervals\n", " - Bayesian methods\n", " - Some theory of machine learning\n", "\n", "\n", "## If you like machine learning\n", "\n", "* CS 106EA: Exploring artificial intelligence\n", " - Training and evaluating ML models\n", " - How do neural networks work?\n", " - Challenges in ML: over-fitting, bias, distribution shift\n", "\n", "\n", "After taking MATH 51 and CS 106:\n", "\n", "* CS 129: Applied Machine Learning\n", " - More ML models: \n", " - logistic regression\n", " - support vector machines\n", " - deep learning\n", " - \"Unsupervised learning\": clustering and feature discovery\n", "\n", "# Thanks for a great quarter!\n" ] } ], "metadata": { "kernelspec": { "display_name": "Phython (JB)", "language": "python", "name": "jb-python" } }, "nbformat": 4, "nbformat_minor": 4 }