How Do You Know What They Know?
Day 2 activity — design an assessment experience for Probability for AI
The task
Probability for AI (PAI) is a free, online probability class. Thousands of learners around the world work through it at their own pace, and each one belongs to a small section — a dozen or so people who meet weekly over Zoom with a volunteer section leader. Nobody is in a room with us. Nobody is proctored. Every learner has a frontier model one tab away, and in one part of the class we hand them a coding agent on purpose.
Your job today: design an experience that tells us what a learner actually knows. Not "design a test" — a test is one possible answer, and probably not the best one. The deliverable is a design, and the real skill we're practising is the second half: deciding which of your ideas are any good. Ideas are cheap. Almost all of them are bad. Today is about getting fast at telling the difference.
First: who is the answer for?
"Knowing what someone knows" is not one problem. It's at least four, and they want different things. A design that doesn't name its consumer will drift into measuring everything badly. Pick one before you start.
| Consumer | What they need | What "good" means to them |
|---|---|---|
| The learner | Am I getting this? What should I go back to? | Honest, immediate, not discouraging enough to make them quit |
| The platform | A per-concept belief, updated after every interaction, to pick what to show next | Calibrated probabilities — this is Day 1's knowledge tracing, now with the data collection as your design choice |
| The section leader | Which three of my twelve people are quietly drowning, before Thursday | Arrives in time, points at a person and a topic, takes under five minutes to read |
| The institution | Does this certificate mean anything to someone who wasn't here? | Defensible to a skeptic, comparable across learners, hard to buy |
The surfaces you already have
You are not designing on a blank page. PAI is a real product with real surfaces, and each one already throws off data. Before you invent a new assessment, ask what the existing surfaces would tell you if you actually looked. The best designs today will probably be small changes to these, not new things bolted on the side.
The lesson page. Free-response questions inside the lesson, plus a "top answers" feed of what other learners wrote. Already collects: the text they wrote, how long they took, and which of their peers' answers they stopped to read.
The agent project. Learners build a real tool with a code agent, against staged milestones. Already collects: the entire conversation. Every question they asked it, every suggestion they accepted, and every wrong suggestion they did or didn't catch.
The classwide forum. Thousands of learners asking and answering each other. Already collects: what confused people and when, and — more interesting — who is able to explain it to someone else.
The section page. A dozen people, a named human, a weekly rhythm. Already collects: who showed up, and who stopped showing up.
The roster. Twelve people, small enough that a leader could genuinely know each one. Also a reminder of how thin the signal is by default: most of them are a single letter on a coloured square.
The live section. The only synchronous, human, hard-to-outsource channel in the entire class — and it costs a volunteer's hour. The scarcest resource you have. Spend it carefully.
What "knowing" looks like here. Worth staring at. The final number is the least of it — the skill is defining the right event. Any design that only checks the number is measuring the easy part, and the part a model is best at.
Every design is three pieces
When you write your idea down, write it as three separate things. Most bad ideas are bad because one of the three was never specified, and the gap got filled in by wishful thinking.
| Piece | The question it answers | Where it usually breaks |
|---|---|---|
| Evidence | What does the learner do, and what artifact does that leave behind? | The task is fun but the artifact it leaves is uninformative |
| Inference | What turns that artifact into a claim about what they know? A rubric, a model, a human, a threshold? | "...and then we look at it" — for 10,000 learners |
| Decision | What happens differently because of the claim? For whom, by when? | Nothing happens. You built a dashboard nobody opens |
If your idea has no decision, it is not an assessment, it's a measurement hobby. Cut it or find its decision.
How to tell a good idea from a bad one
Here is the scoring card. Take an idea, walk the seven rows, and be specific: the row is only useful if you can name the concrete scenario where it fails. "Somewhat scalable" is not an answer. "Twelve thousand learners × two minutes = 400 volunteer hours a week" is.
| Criterion | The question to ask out loud | Fails when… |
|---|---|---|
| 1. Validity | Is this measuring probability, or is it measuring English fluency, typing speed, free time, or willingness to play along? | You can name a learner who understands the material and would score badly |
| 2. Attributability | If the learner pastes the whole thing into a frontier model, what survives? Is the evidence cheap for someone who knows and expensive for someone who doesn't? | The cheapest path to a good score doesn't pass through understanding |
| 3. Cost at scale | What is the marginal cost per learner, in staff-minutes and in dollars? Multiply by 12,000. Who exactly does that work? | The answer is "a TA" and there are no TAs |
| 4. Experience | This class is free and voluntary. Does doing this make someone more likely to finish the course, or less? Would you do it on a Tuesday night after work? | It measures knowledge accurately right up until people quit rather than take it |
| 5. Actionability | Who reads the result, and what do they do that they wouldn't have done otherwise — within the window where it still matters? | The signal arrives after the moment it could have changed anything |
| 6. Equity | Who does this quietly exclude? Phone-only learners, bad bandwidth, hostile timezones, second-language learners, people who can't be on camera. | It works beautifully for learners who look like us |
| 7. Falsifiability | How would you find out this measure is wrong? What would you correlate it against? What result would make you kill it? | There is no experiment that could embarrass it |
Two of these do most of the killing. Attributability is the one this era forces on us, and cost at scale is the one that quietly murders every design that assumed a grader. Run those two first — they're fast, and they'll clear the table.