A team has months of logs of its AI agent at work and wants to fine-tune or run RL on them. Finding out whether that will work by training costs money and weeks. I built trace2wm telemetry to answer from the logs alone, before anything is trained: can a training environment and a reward be built from these recordings; what data to collect next; roughly what each post-training method can reach; and, after training, whether a claimed improvement is real.
Drop in the logs you already have: OpenAI chat transcripts, OpenTelemetry GenAI traces, Arize Phoenix span tables, ToolTalk or ATIF trajectories, or a spreadsheet with one tool call per row. Emails, phone numbers and card numbers are hidden as soon as the log is read. It runs on your own computer, where nothing is sent anywhere, or on a hosted site for invited teams, where the uploaded log is deleted as soon as the report is made.
A minute later you get two pages. The first is for the people who run the agent: it opens with one verdict — “Fix the logging before training the AI”, or “Trying again already works well” — and then the checks behind it, in plain words. The second is for the engineer, with every number, the rule that produced it and the files it was read from. Another agent can ask the same questions through MCP.
Every check in it exists because the mistake behind it happened in my own trace2wm project and was found late:
None of these needs a GPU to find. Each is a count over the logs, the configuration or the evaluation files, and each blocks a specific next step until it is fixed.
The report reads the corpus in seven stages — intake, corpus health, coverage, environment fidelity, learnability, training health, outcome. Each stage has metrics, gates that say what they block, and actions computed by rules over the metrics. It is read-only (every input file is hashed before and after), runs on a CPU, and makes no API calls. The output is a JSON document with the hash of every file it read, so anyone can later ask whether a saved report still describes the data and the code.
One question is whether a simulator behaves like the real system, task by task; if it does not, RL in the simulator teaches the wrong thing. The obvious test compares success rates of the same policy in both environments. It mostly cannot decide: with 8 runs on each side, a task needs at least 5 successful runs before any arrangement of them could be significant, and most tasks never succeed at all.
So the report compares steps instead. It replays the saved trajectories through both environments and, for each task, counts the tool calls one environment ran and the other refused (an exact paired test, corrected for the number of tasks). On trace2wm's corpus that gave 20,814 paired calls and a verdict on 23 of 23 tasks, where the success-rate test left 9 undecided. Each disagreement also says why — an argument the real tool does not accept, a refusal the simulator has no check for, a state the real scorer will not read — which points at the file where the fix belongs.
The report says what each method — fine-tuning, rejection sampling, RL in the simulator, test-time methods — might reach, always as a range and never as a single number, with a tag saying whether it was measured, extrapolated or taken from published work. An end that cannot be estimated is drawn as unknown, never as zero.
The page for the people who run the agent goes further: no sentence on it is written by hand or by a model. Each one is a phrase with numbers from their log filled in, chosen by written rules whose limits they can see and change. And their report never carries my project's conclusions: a customer is owed readings of their own files.
Platforms that check a training run compare the trained model with the one in production. I have not found one that compares it with the untrained agent simply trying again at the same cost — and retrying is often what you should do. Today the page shows two cards side by side: if the agent tries again, measured on the customer's own log (tasks done after one try and after k tries, and what the extra tries cost), and if you train a small model now, a rough estimate from published results, marked low-reliability because those results go both ways — on the same ~2,000 starting examples, one study's 8B model got worse while its 14B model got better. Measuring the comparison directly, from a log before training and a log after, at the same spend, is the next piece; it is designed and not yet built.
Limits: the checks were built on trace2wm's own corpora and mistakes; another team's logs will break assumptions mine did not. The forecasts from published work are deliberately labelled low-reliability.