From agent logs to training: what the papers measure, and what they skip

A team running an AI agent in production accumulates logs: conversations, tool calls, outcomes. Sooner or later someone asks whether to train a smaller model on them. The literature has a lot to say about how to train agents from trajectories, and much less about the question the team actually has: should we, on these logs, and will it beat simply letting the agent try again? This post reviews what is known — trajectory data and fine-tuning, simulators built from traces, reinforcement learning, training versus retrying, and whether measured gains are real — and where my tool trace2wm telemetry differs. (For what the tool does, see this post.)

Terms used below: SFT (supervised fine-tuning) trains a model to copy example trajectories; RL (reinforcement learning) trains it by trial and a reward; pass@k is the share of tasks solved at least once in k attempts.

Trajectory data and fine-tuning

The standard recipe: collect trajectories of a strong model (or of people) doing the task, keep the ones that succeeded, convert them into a training format, and fine-tune a smaller model. It works across three orders of magnitude of data. 500 GPT-4 trajectories raised Llama2-7B's HotpotQA performance by 77 % (Chen et al. 2023); 1,866 verified trajectories mixed with general data made a 70B model comparable to GPT-3.5-turbo on unseen agent tasks (Zeng et al. 2023); 491 trajectories in executable software tasks gave up to 19 % absolute gains (Pan et al. 2024); unifying 13 datasets into 1.3 million trajectories gave an average gain of about 20 % (Song et al. 2025). The papers agree that quality and format matter more than volume (see Fig. 2 of AgentTuning for a typical construction pipeline).

When training hurts

The cases where training made an agent worse are documented, but scattered:

• Fine-tuning Qwen2.5-32B on AppWorld's own ground-truth programs dropped its task completion from 39.2 to 6.2; an RL variant trained on only 24 scenarios reached 71.3 (Chen et al. 2025, LOOP).
• The same roughly 2,000 cold-start trajectories lowered Qwen3-8B on τ²-Retail from 41.0 to 31.4 but raised Qwen3-14B from 43.1 to 53.7, and lowered Telecom scores at all three model sizes (Zhao et al. 2025, MUA-RL).
• Agent corpora entangle format-following with reasoning, different capabilities are learned at different speeds, and agent tuning adds hallucinations (Chen et al. 2024, Agent-FLAN).
• RL needs a reward that varies: tasks solved by every attempt, or by none, provide “no meaningful training signal” (Foster et al. 2025), and multi-turn RL can fall into an “Echo Trap” where reward variance collapses (Wang et al. 2025, RAGEN).

Simulators built from traces

RL and large-scale data generation need an environment the agent can act in. Three families build one: a language model that simulates the tools and the user; hand-built executable environments; and environment configurations generated by a model. ToolEmu emulates tool execution with a language model and evaluates safety automatically; humans validated that 68.8 % of the failures it found were real (Ruan et al. 2023). τ-bench simulates the user with a language model and scores the final database state against a goal state (Yao et al. 2024); in its successor, annotators found user-simulator errors in 16 % of telecom conversations, against 40 % in the original retail domain (Barres et al. 2025). Simia amplifies about 5,000 seed trajectories into 90,000 simulated ones and lifts Qwen2.5-3B on τ²-Bench from 9.5 to 31.7, without reporting how often a simulated tool response matches a real one (Li et al. 2025).

The common thread: simulator fidelity is judged by humans on small pooled samples, or indirectly through gains after moving to the real environment. We did not find a paper that decides, task by task and with a statistical test, whether a simulator behaves like the real system.

Train, or try again?

Before training, there is a cheaper option: sample the agent several times and keep an attempt that works. Repeated sampling scales surprisingly far — on SWE-bench Lite, coverage rose from 15.9 % with one sample to 56 % with 250 (Brown et al. 2024) — and compute-optimal test-time search beat a 14× larger model in a FLOPs-matched comparison on problems of moderate difficulty (Snell et al. 2024).

Training can erode exactly this advantage. Standard fine-tuning made Llama-3-8B more confident: over four epochs its pass@1 on MATH rose from 4.4 % to 7.4 %, while its pass@4k fell from 82.5 % to 63.0 % (Chen et al. 2025). RL with verifiable rewards shows the same shape: trained models beat their base models at small k and lose at large k (Yue et al. 2025). And retrying has its own ceiling: with an imperfect verifier, more attempts cannot lower the chance of accepting a wrong answer, and the best number of attempts is often under ten (Stroebl et al. 2024).

All of this evidence comes from math, code and question answering, where attempts are independent and cheap to repeat. We did not find a study that compares training against retrying at equal dollar cost on a team's own deployment logs.

Is the measured gain real?

Evaluation-validity work keeps finding the same mistakes. τ-bench counted empty responses as successes on impossible tasks, so a trivial agent scored 38 % and beat a GPT-4o agent; flaws like this mis-estimate agent performance by up to 100 % in relative terms (Zhu et al. 2025). Many agent benchmarks “have inadequate holdout sets, and sometimes none at all”, and ignore cost (Kapoor et al. 2024). Training on the test task confounds comparisons unless every arm is trained on the same task data (Dominguez-Olmedo et al. 2024). The mistakes trace2wm telemetry was built to catch — each of which happened in my own trace2wm project — line up with published ones:

Forecasting before training

Forecasting what training will reach exists, but at the level of scale: emergence laws fitted over amounts of fine-tuning data (Snell et al. 2024), and observational scaling laws built from about 100 public models (Ruan et al. 2024). Both work from model checkpoints or leaderboards. Learnability is estimated online, from the policy's own samples during training (Foster et al. 2025). None of it works read-only from a team's logs.

Where trace2wm telemetry differs

Where the literature is ahead

• Real training, measured. The papers above train end to end and report gains — LOOP on AppWorld, DreamGym, MUA-RL, SWE-Gym. trace2wm telemetry forecasts; it does not train.
• Scale. Corpora of 1.3 million trajectories, 90,000 synthetic trajectories, dual-control simulators with annotated error rates. trace2wm was built on one project's corpora and mistakes; another team's logs will break assumptions ours did not.
• Theory of retrying versus training. The literature has one for math and code; trace2wm's own equal-spend comparison is not built yet, so its training side still rests on published ranges.
• Learnability from someone else's logs. We found no method that estimates learnability offline from another policy's logs — which is what trace2wm's learnability stage attempts, and so remains unvalidated.

References

1. Chen, B. et al. “FireAct: Toward Language Agent Fine-tuning.” 2023. arXiv:2310.05915
2. Zeng, A. et al. “AgentTuning: Enabling Generalized Agent Abilities for LLMs.” 2023. arXiv:2310.12823
3. Pan, J. et al. “Training Software Engineering Agents and Verifiers with SWE-Gym.” 2024. arXiv:2412.21139
4. Song et al. “Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents.” 2025. arXiv:2510.24702
5. Chen, K. et al. “Reinforcement Learning for Long-Horizon Interactive LLM Agents” (LOOP). 2025. arXiv:2502.01600
6. Zhao, W. et al. “MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use.” 2025. arXiv:2508.18669
7. Chen, Z. et al. “Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models.” 2024. arXiv:2403.12881
8. Foster, T. et al. “Learning to Reason at the Frontier of Learnability.” 2025. arXiv:2502.12272
9. Wang, Z. et al. RAGEN. 2025. arXiv:2504.20073
10. Ruan, Y. et al. “Identifying the Risks of LM Agents with an LM-Emulated Sandbox” (ToolEmu). 2023. arXiv:2309.15817
11. Yao, S. et al. “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.” 2024. arXiv:2406.12045
12. Barres, V. et al. “τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment.” 2025. arXiv:2506.07982
13. Li, Y. et al. “Simulating Environments with Reasoning Models for Agent Training” (Simia). 2025. arXiv:2511.01824
14. Brown, B. et al. “Large Language Monkeys: Scaling Inference Compute with Repeated Sampling.” 2024. arXiv:2407.21787
15. Snell, C. et al. “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters.” 2024. arXiv:2408.03314
16. Chen, F., Raventos, A., Cheng, N., Ganguli, S. & Druckmann, S. “Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning.” 2025. arXiv:2502.07154
17. Yue, Y. et al. “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?” 2025. arXiv:2504.13837
18. Stroebl, B. et al. “Inference Scaling fLaws: The Limits of LLM Resampling with Imperfect Verifiers.” 2024. arXiv:2411.17501
19. Zhu, Y. et al. “Establishing Best Practices for Building Rigorous Agentic Benchmarks” (ABC). 2025. arXiv:2507.02825
20. Kapoor, S. et al. “AI Agents That Matter.” 2024. arXiv:2407.01502
21. Dominguez-Olmedo, R. et al. “Training on the Test Task Confounds Evaluation and Emergence.” 2024. arXiv:2407.07890
22. Snell, C. et al. “Predicting Emergent Capabilities by Finetuning.” 2024. arXiv:2411.16035
23. Ruan, Y. et al. “Observational Scaling Laws and the Predictability of Language Model Performance.” 2024. arXiv:2405.10938
24. Farn, N. & Shin, R. “ToolTalk: Evaluating Tool-Usage in a Conversational Setting.” 2023. arXiv:2311.10775

Citation

Cited as: Qu, Shuhui. (Oct 2026). From agent logs to training: what the papers measure, and what they skip. Shuhui Qu's blog. https://shuhuiq.github.io/blog/review-agent-logs-training.html

@article{qu2026agentlogstraining,
  title   = "From agent logs to training: what the papers measure, and what they skip",
  author  = "Qu, Shuhui",
  journal = "shuhuiq.github.io",
  year    = "2026",
  month   = "Oct",
  url     = "https://shuhuiq.github.io/blog/review-agent-logs-training.html"
}