Improving an AI Tutor

Thursday, October 15 — judging, writing and choosing between tutor prompts

An AI tutor in Stanford's free Probability for AI course has talked about 2,000 learners through the same problem: if one AI attempt fixes a bug 15.9% of the time, how many attempts make you 95% sure one works?

How do you make a tutor like that better? Three short activities, in order: judge replies, write a tutor, then choose between versions.

1. Judge the Tutor

Two learners typed the same thing and got different replies. Which reply came before the learner kept going? Race an AI judge.

Open Judge the Tutor

2. Prompt Lab

Write a tutor prompt. It answers eight learners, and an AI judge compares each answer with our real tutor's. Class password needed.

Open Prompt Lab

3. Be the Bandit

Three tutor versions, sixty learners. Pick a version for each, then see how Thompson sampling did with the same learners.

Open Be the Bandit

Talk about it

How much of what a learner does next comes from the reply, and how much from the learner?
In Prompt Lab, did your score go up because the tutoring got better, or because you learned what the judge likes?
What if the bandit's reward were the Prompt Lab judge's verdict instead of "finished"? What could go wrong?

Then the team workshop: Judging Tutors.