Student Summaries

Day 3 activity — evaluating natural-language descriptions of student understanding

Bundles are password protected, and the same password unlocks the LLM endpoint below. Both are announced in class. All data has been annonymized.

Start with Probability for AI. It is the course you have been through, the applicants are reasoning about material you know, and their thinking is visible in what they wrote rather than inferred from code. Later we will be able to see how well the summaries predict the student.

Code in Place is there because it can be scored. Each learner's next action was recorded and held back, so a summary can be checked against what the person actually did next. It is the smaller and more artificial problem, and that is exactly what makes a number possible. Working the unscored dataset first and the scored one second is the order in which the difficulty becomes obvious.

---

The task

Adaptive learning systems require some representation of what a learner currently understands. Natural language is an attractive candidate: a model can generate it from a learner's work, and a teacher can read it. The difficulty is evaluation, since there is no reference summary to compare against.

You are given a record of one learner working. You write a short summary of what they understand and where they are stuck. The question the activity is built around is how anyone would tell whether that summary is any good.

Submitting is optional. Nothing here is graded on the number you get, and you do not have to upload anything. The submission form and the leaderboard exist because a measurement is more interesting than an opinion, not because they are the assignment. Reading a few learners carefully and writing summaries by hand is the point; bring those to class whether or not you submit them.

Probability for AI — the main dataset

100 applicants to Probability for AI working through Word Detective, the conversational activity in the PAI application. Each of them is given a grid of short answers to the same questions, half written by people and half by a language model, and asked to find a word that distinguishes the two. A tutoring agent walks them through it: pick a word, count how often it appears in each group, turn those counts into conditional probabilities, decide what to do about words that never appear at all, and then design a way to display the result.

The activity is diagnostic almost by accident. To get anywhere the applicant has to separate P(word | AI) from P(AI | word), normalise counts by something sensible, and confront zero counts. Their turns show which of those they have and which they do not, often explicitly, because the agent keeps asking them to say what they would do next and why.

Each record has the applicant's four free-text answers from the application form and their full transcript:

{
  "participant_id": "p072",
  "activity": "word-detective",
  "about": {
    "interest":        "why they want to take the course",
    "occupation":      "...",
    "previous-coding": "...",
    "about-yourself":  "..."
  },
  "age": 30, "country": "USA", "gender": "...", "hours_per_week": "4",
  "transcript": [{"from": "user"|"agent", "text": "..."}]
}

These applicants were never observed again. There is no next action to predict and nothing to score a summary against, so PAI submissions are recorded but not scored and there is no leaderboard for them. That is the ordinary situation in education research, and it is the harder one.

A worked example

Participant p042 is 16, in India, and in 11th grade. Their form answers, verbatim:

occupation	11th Grader
previous-coding	"I have built a programming language from python, 2 websites, RAG pipeline for evaluating small language models on TriviaQA"
interest	"I am curious how AI works; I want to be part of a community of other students who are also learning; For fun and enlightenment"
about-yourself	"I always have enjoyed being exerted, which meant that naturally, I started my life with challenges. […] Reading books above what teachers considered my level and succeeding, Participating in quizzes, Solving interesting problems […] and then comes math. My enemy and best friend."

Now the opening of the transcript. Agent turns are trimmed where they run long.

agent  Take a look at the grid on the right: that's our raw material.
       In your own words, what's in it? [...]

p042   Its a grid of ai and human responses to questions.

agent  Exactly right. And for milestone 1 — in your own words, what
       are we actually going to measure? [...]

p042   we are measuring the probability that a word is written by ai
       or by a human and putting that probability on a scale

agent  Close! We're actually going to measure it slightly differently:
       we'll pick a word, and then measure how often it shows up in
       all the AI answers, versus how often it shows up in all the
       human answers. [...]

p042   no . thank you. lets go on.

That is a genuine misconception, not a slip. "The probability that a word is written by AI" is P(AI | word): you have the word, you want the author. The milestone asks for P(word | AI): you have the author, you want the word. They are different quantities, related by a prior the participant has not thought about, and the confusion between them is the most common error in applied probability.

What happens next is the interesting part:

agent  I'll use 'W' for the event 'the text contains our word'. So in
       your notation, our goal is to estimate exactly two numbers:
       P(W | E) and P(W | F). [...]

p042   out of all of the ai written responses (n) which contain the
       word. We count the number of times that "imagine" occurs in
       the ai written responses.

       [... results come back ...]

p042   The word "Imagine" shows up almost 20 times more in the AI
       written dataset as compared to the human written dataset

agent  Now say we wanted the same two numbers for a different word,
       say 'super'. What would you do? [...]

p042   First we would ,again, find the number of outcomes where
       "super" occurs in ai responses, then we would find the number
       of outcomes where "super" occurs in human responses. the total
       ai responses and total human responses remains the same

They have it. The direction is right, the denominator is right, and they notice unprompted that the group totals stay fixed when the word changes, which is the point of the second milestone. So the misconception was real and is gone within a few turns, corrected by a single sentence they explicitly declined to discuss.

Three summaries of this person:

vague	"A strong and motivated learner who understands most of the probability but was briefly confused at the start."
surface	"16-year-old 11th grader who has built a RAG pipeline. Chose the word 'imagine'. Used E and F for the events. Asked for dark mode and three tabs."
specific	"Opens by inverting the conditional, describing the task as the probability a word was written by AI rather than the probability AI writes that word, and is corrected in one sentence which they wave off without discussion. It takes: three turns later they restrict to the AI responses, count within them, and then observe unprompted that the group totals stay fixed when only the word changes. Invents their own event symbols rather than accept the suggested ones. Reads a 20x ratio off the output without hesitating. Challenges the point of switching datasets rather than complying. Fast, formally comfortable, and impatient with scaffolding they have already outgrown."

The three differ in what a reader could do with them. The first would fit most of the cohort. The second is a log of actions and choices, accurate and inert. The third records that a specific, nameable error appeared and then did not persist, which is a claim about how this person learns rather than about what they know. It also carries the thing the form fields alone would get wrong: an 11th grader who has written a RAG pipeline needs less scaffolding than their age suggests, and the transcript shows them saying so.

Code in Place — the scored dataset

100 Code in Place students working on Khansole Academy: generate an addition problem from two 2-digit integers, read the user's answer, and report whether it was correct. Some students attempt an optional extension that repeats until three consecutive correct answers.

The record is narrower than the PAI one. There is no conversation and nothing the learner wrote in prose, only four consecutive saves of their program taken seconds apart. Everything you can say about them has to be inferred from what changed between snapshots and how long it took.

Each students/sNNN.json looks like this:

{
  "student_id": "s001",
  "assignment": "khansole",
  "snapshots": [
    {"t_plus_seconds": 0.0,  "code": "import random\n..."},
    {"t_plus_seconds": 4.8,  "code": "..."},
    {"t_plus_seconds": 10.5, "code": "..."},
    {"t_plus_seconds": 44.1, "code": "..."}
  ]
}

Snapshots are keystroke-level saves, so consecutive entries may differ by a single character. Timing carries information: a 34-second gap before a line appears is usually meaningful. The held-out fifth snapshot, the one your summary is scored against, is not in the bundle.

How the scoring works — Code in Place only

Scoring uses PromptNCE (Woodrow & Piech, COLM 2026), a contrastive method for evaluating text where no reference exists. For each student a lineup is constructed from the true next snapshot and four snapshots written by other students, in fixed random order. The model receives your summary and the lineup, and returns a distribution over the five candidates together with an OTHER category for outcomes not listed.

$$\text{score} = \frac{1}{N}\sum_{i=1}^{N} P\big(\text{true snapshot}_i \mid \text{your summary}_i,\ S_i\big)$$

Lineups are fixed. Every submission is scored against the same five candidates for each student, so a score does not depend on what other teams submitted. Chance is 0.20 before OTHER absorbs probability mass, which lowers the effective floor.

Scores to beat — Code in Place

Summary	Score
"This student is confused about everything."	0.162	says nothing, identifies nobody
example_baseline.py	0.193	counts characters and keywords
expert human teacher	0.396	your target
the student's full raw code	~0.55	no compression — not allowed, but shows the ceiling

The mechanical baseline gains little over an uninformative summary. Surface features, such as quantity of code written or which keywords appear, carry almost no identifying information. The interval between 0.19 and 0.55 corresponds to conceptual content, and the question is how much of it survives compression to 150 words.

The paper

The method and the Code in Place results come from Learner Insight: Tracing Natural Language Knowledge States that Predict Future Work (Woodrow & Piech). Same password as the bundles. Worth reading before you start: it sets out why summary quality is hard to measure and what the contrastive construction is doing.

LLM access

The course provides a shared endpoint; no personal API key is required. Use the class password with cs398_llm.py from the bundle:

export CS398_TEAM="Your Team Name"
export CS398_PASSWORD="<announced in class>"

from cs398_llm import complete
text = complete("Summarise this student: ...")

Identical calls are cached server-side and cost nothing, so repeated runs during development are free. Each team has a token budget, reported in the response. The model is fixed for all teams.

Getting started

By hand. A Code in Place record is about 400 tokens and a PAI record about 3,000, so either fits in a chat window one at a time. Read three or four people properly and write their summaries yourself before automating anything. It is the only way to find out what is actually in these records, and on PAI it is the whole activity.

Programmatically. The Code in Place bundle ships two scripts. example_baseline.py needs no network access and produces a valid submission; replacing its summarise() function gives a working pipeline. example_llm.py does the same through the class endpoint with a deliberately minimal prompt, and resumes after interruption. Both read the Code in Place format, so adapting them to the PAI records is a small change and a reasonable first exercise.

Open questions. Whether describing correct understanding contributes anything beyond describing errors. Whether naming a misconception outperforms describing its symptom. Whether predicting the next action outperforms describing the current state. Whether 150 words are better spent on a single distinctive error than on a balanced account. And whether a summary that reads well to you scores well at all, which is the one question only the Code in Place half can answer.

Submit

Upload a JSON file of the form {"summaries": {"s001": "...", ..., "s100": "..."}} with all 100 students, each at most 150 words.

Scoring makes 100 model calls and takes around a minute. Limit of 40 submissions per team.