Knowledge Tracing

Day 1 activity — predict whether a learner gets the next problem right

Download starter bundle

---

The task

A huge amount of algorithms for education comes down to one question: given everything you know about a learner so far, will they get the next problem right? Get that right, and you can decide what to show them next, when to move on, when to intervene — this is the engine under adaptive learning platforms everywhere.

Today's learners happen to be chess players, and the "problems" are chess puzzles — but the modeling question is exactly the same one behind any adaptive learning system. Chess.com already tries to answer it with its own rating system which gets ~59% accuracy on this data. Your job, working with an AI coding agent, is to recreate Chess.com's results (or even to beat them).

The starter bundle above has training data, a test set to predict on, and a working example script showing the exact submission format. Direct your agent, build a model, and upload your predictions below to see your score. The true answers live on a server, never in your browser — there's nothing for you or your agent to go find. The whole point is to see how well your model actually does.

Columns

train.csv / test.csv

Column	Meaning
user_id	Which player
tactics_problem_id	Which puzzle
create_date	When the attempt happened
pre_rating	The player's Chess.com rating before this attempt
puzzle_rating_original	Chess.com's own difficulty rating for the puzzle (same scale as pre_rating)
puzzle_rating_ours	Our own corrected difficulty rating for the same puzzle — use this for IRT/DKT
puzzle_move_count	How many moves the puzzle's solution has
target_time	Expected time to solve (seconds)
solve_time*	How long this player actually took (seconds)
is_correct	1 if solved correctly, 0 if not — only in train.csv

* worth thinking about when this would actually be known.

puzzles.csv

Column	Meaning
tactics_problem_id	Which puzzle (one row per unique puzzle) — join key back to train.csv/test.csv
fen	The board position the player was shown, in FEN notation
rating_is_provisional	1 if Chess.com hadn't yet settled on a stable rating for this puzzle
times_played	How many times this puzzle has been attempted, across all of Chess.com
themes	Chess.com's tactical theme tags for the puzzle (e.g. Fork / Double Attack), semicolon-separated

A puzzle's position (fen) and tactical tags (themes) are raw material your agent can turn into features — e.g. are some themes harder than their rating implies?

Three ways to model this

Chess.com's baseline gives every player and every puzzle a single rating on the same scale, and predicts success with an Elo-family formula (they actually run Glicko-2, which also tracks how confident it is in each rating). Simple and interpretable, but it trusts those ratings, even though they're noisy and (as you'll find) not always well-calibrated. The example_baseline.py in the starter bundle uses the simpler textbook Elo formula as an easy first step to check your submission format — it actually does worse than a coin flip here (~47%), since the test set was chosen around what a well-calibrated model finds easiest, not what this simpler formula finds easiest.

Item Response Theory (IRT) takes the same shape — one ability number per learner, one difficulty number per item — but instead of trusting a reported rating, it estimates both from the data itself, fitting them so the predicted probabilities best match everyone's actual outcomes at once. Same structure as the baseline, better calibrated numbers.

$$P(\text{correct}) = \frac{1}{1 + e^{-(\theta_{\text{player}} - b_{\text{puzzle}})}}$$

where θ_player (ability) and b_puzzle (difficulty) are fit from the training data, instead of read off a reported rating.

Deep Knowledge Tracing (DKT) drops the idea of a single fixed ability altogether. It uses a recurrent neural network to maintain a hidden "belief" about a learner that updates after every single attempt, and predicts the next outcome from that evolving belief rather than one static score — letting it capture learning, forgetting, and momentum that a fixed rating never could.

$$h_t = \text{RNN}(h_{t-1}, x_t) \qquad P(\text{correct}_t) = \sigma(W h_t + b)$$

where x_t encodes the puzzle attempted (and whether it was solved) at step t, h_t is the updated hidden belief, and σ squashes the output into a probability.

One catch for both IRT and DKT: Chess.com's own puzzle_rating_original isn't fully trustworthy — it's measurably miscalibrated for some puzzles. Use puzzle_rating_ours instead (see the columns table below) as your difficulty signal for those two models.

Submit your predictions

Upload a CSV with columns row_id, is_correct (a probability between 0 and 1, or a hard 0/1 guess) for every row in test.csv.

Leaderboard

Rank	Team	Best model	Best accuracy	Submissions
Loading…