Judging Tutors

The setting

In Stanford's free Probability for AI course, an AI tutor talks each learner through a small problem, one step at a time. About 2,000 learners have had the same conversation. The problem: if one AI attempt fixes a bug 15.9% of the time, how many attempts do you need to be 95% sure one works?

Every time we change the tutor's prompt, we need to know if it got better. The cheap way is to ask another AI: here are two replies; which is better tutoring? That AI is a judge. Today you'll find out how good a judge can be, and how easy it is to fool.

Two kinds of team

How judges are scored

In the course's logs we found natural experiments. Two learners were at the same step and typed the same thing. The tutor answered each one differently. Afterwards, one learner tried the next question. The other gave up, got confused, or left.

Your judge sees both tutor replies, but not what happened next, and picks one. Its score is how often it picks the reply that was followed by a real try.

• Every pair is judged twice, with A and B swapped. Changing its answer when the order flips counts half. So guessing scores 50%.
• There are 44 practice pairs you can be scored on any time, and 129 hidden pairs for the real score.
• We also check how well your judge holds up against the tutor teams' replies, and how often it agrees with Chris's own picks (a tiebreak).

The numbers to beat

Guessing	50%
Always pick the longer reply	44%
Always pick the shorter reply	56%
Claude, asked "which is better tutoring?"	59%
Claude with a detailed teaching rubric	58%

Each number is uncertain by about ±7 points.

Replies that often came before a learner stalled:

• giving the answer away, then asking a bigger question
• a vague "why?" or "say it in your own words"
• sentences that don't quite fit together
• repeating itself
• ending with "Clear?"

An example

Invented for this page. You never see real learner text; you see scores.

The learner was asked for the chance that at least one of two attempts works, and added the chances:

B is longer and more correct. It also gives the answer away and then asks something hard. A leaves one small, concrete step that shows the mistake. A judge that rewards completeness picks B. The patterns above suggest A is more likely to get a real try.

The real tutor's prompts

Each turn, the course's server function sends gpt-5.1 two prompts. Tutor teams: this is what you're up against.

Where the system prompt's length goes

• The 17 steps, 45%
• Persona and general rules, 24%
• App parameters, 10%
• Shared rules and reply format, 21%

Many of its rules were added after watching real learners get stuck. Whether every one still earns its place is an open question.

1. The system prompt

Loading…

2. The function's per-turn prompt

An example: the app shows 111% at seven copies, and the learner types oops, over 1. This is the last message the model receives, in full:

Loading…

Both are generated by the server's own code from the live version.

What to submit

Judge teams: one text or markdown file, your judge's system prompt. Rules, a rubric, examples: anything goes.

For each pair, your prompt is followed by:

step     the name of the step in the conversation
tutor    the tutor's previous message (its last ~400 characters)
student  what the learner said (a button they tapped, or a short answer)
Reply A  ...
Reply B  ...

Which reply is better? Answer A or B.

The answer is forced to be A or B, so you can't change the format.

Tutor teams: one text or markdown file, the tutor's prompt for TBD: the assigned step, or the whole conversation. We generate its replies at a fixed set of learner moments, and every judge compares them with the other tutor teams' replies.

How to submit: TBD One file per team per round, named with your team and round, e.g. bottleneck-judge-r2.md.

Rules

• No real student data in any submission.
• A judge must answer A or B.
• A tutor's reply must read like normal tutoring for the learner: no mention of the judge, the scoring, or the other reply.

Tips

Judges. Aim at the learner's next message, not the best explanation. Expect tutor teams to try long, polished or flattering replies. Watch your practice scores for answers that change when A and B swap. With 44 practice pairs, a difference of a few pairs is noise, so change one thing at a time.

Tutors. Each round's scores show what the judges reward, so use them. But people pick the final winner, so a reply that games the judges and reads badly won't win. The gap between the two rankings is the interesting part.