Auditing Optimization Papers with Agentic AI

Warmup — what are you using AI for?

Idea generation

Proofs or mathematical reasoning

Code

Writing and revision

Literature review

Not yet—or something else

Compared with two years ago…

A

I am writing more papers, faster than ever before.

B

I am reviewing more papers, faster than ever before.

The goal of peer review is more good research

The standard peer review process judges distinct properties:

Correctthe claims survive appropriate checks

Reproduciblethe evidence can be inspected and rebuilt

Novelthe result changes what the literature contains

Significanta human community decides what matters

Which require humans? Which can be done (possibly better) by AI?

Will the AI flywheel increase knowledge, or degenerate to slop?

Better AImore capable research partner

Faster researchideas, proofs, code, prose

More artifactspapers become future inputs

Verification capacityCan checks keep pace?

Research production and verification are moving at different speeds

$A combine harvesting winter wheat across a broad field$

Production

AI accelerates

ideas · proofs · code · prose

Two workers harvesting wheat by hand with a cradle scythe and binding sheaves

Verification

Humans still inspect

assumptions · citations · experiments

Production is becoming faster. Verification still proceeds by hand.

USDA combine harvesting winter wheat · Dorothea Lange, hand-harvesting wheat

We built 50 traceable audit packages

50optimization-paper audit packages25 MOR · 25 MPC

What this establishes

AI agents can assemble inspectable evidence at scale

Each package links claims to code, calculations, reruns, and candidate findings.

What it does not establish—yetprecision, recall, error prevalence, or journal quality

Next: authors and experts adjudicate the findings.

AI agents are entering the full research workflow

Pokutta and colleagues' agentic research workflow: a researcher prompts an instruction-governed CLI agent in a sandbox, which iterates through explore, plan, implement, evaluate, analyze, record, commit, and iterate

The trajectory

Research loops persist, execute, and record
Research benchmarks test project-level autonomy
Replication agents rerun conference experiments

August 202610reported advancesmathematics + theoretical computer science

Zimmer et al., arXiv:2603.15914 (2026), Fig. 3 · Zhou et al., arXiv:2608.17271 (2026) · SAI, research report (2026) · Abid, Hugging Face Blog (2026) · OpenAI, technical report (2026)

Timing: 90 seconds. First show the workflow alone. It is a documented framework, not evidence that autonomous agents reliably produce correct science. The researcher supplies persistent instructions; the CLI agent works in a sandbox, records and commits work, and reports back. The important shift is from one-shot answers to an iterative, instrumented workflow.

Briefly name the broader trajectory: ASI-Bench evaluates project-level research tasks; SAI and the Hugging Face Open Reproductions effort report agent-run replication and public audit trails. These systems and reports measure different things, so do not compare their headline percentages.

Then reveal Ten Advances in Mathematics and Theoretical Computer Science. It documents a fast-moving frontier but does not make every case equally autonomous or equally verified. The transition is the same: producing a high-stakes claim and establishing that it is correct are different achievements.

[Sources] - Zimmer et al., arXiv:2603.15914 (2026): https://arxiv.org/abs/2603.15914 - Zhou et al., arXiv:2608.17271 (2026): https://arxiv.org/abs/2608.17271 - SAI report, 22 July 2026: https://sai.science/blog/how-much-science-is-verifiable - Nature coverage of the SAI exercise: https://www.nature.com/articles/d41586-026-02235-8 - Hugging Face Open Reproductions report, 13 August 2026: https://huggingface.co/blog/icml-2026-open-reproductions - OpenAI, Ten Advances in Mathematics and Theoretical Computer Science, August 2026: https://cdn.openai.com/pdf/ten-proofs-oai.pdf

Optimization researchers are inside the capability loop

Ernest Ryu

UCLA, on leave · OpenAI

Ryu checked GPT-5’s mixed-quality proof steps and completed the argument.

Sébastien Bubeck

OpenAI

GPT-5 proposed a sharper step-size condition. Bubeck verified it independently.

Optimization experts are building and scrutinizing these systems.

Ryu portrait and role · Bubeck portrait · Ryu case · Bubeck case

Four kinds of claims require different adjudication

TheoryDoes the theorem follow?

assumptions · proof steps · counterexamples · certificates

meaning · scope · unresolved gaps

NumericsDo experiments support the claim?

paper/code consistency · reruns · budgets · stress tests

design adequacy · consequence

Literature-relativeIs it supported or first?

bounded search · candidate antecedents · quoted evidence

equivalence · priority norms · search adequacy

SignificanceDoes this contribution matter?

organize context—no score

Human judgment · importance · beauty · taste

Machine-produced evidence

Human responsibility

Let’s keep optimization verifiable

Universe A

Truth is sovereign

Claims live or die by decisive proof, falsifiable experiment, or high-fidelity replication.

Humans allocate attention; evidence settles checkable claims.

Universe B

Proxies are sovereign

Claims outpace verification, so prestige, checklists, citations, and polish dominate.

Researchers rationally optimize the signals review can afford to see.

Review-mimicking AI moves claims faster. Verification-first AI expands our capacity to test them.

Lei You, Lele Cao & Iryna Gurevych, Preventing the Collapse of Peer Review Requires Verification-First AI, arXiv:2601.16909 (2026)

Check what is checkable; do not automate taste

Machines

Gather auditable evidence

run checks and reproductions
expose discrepancies
state scope and uncertainty
identify unresolved questions

People

Exercise scientific judgment

significance and conceptual relevance
novelty in context and severity
evidentiary standards and venue fit
publication decisions

An audit should change what experts can see, not decide what they must value.

The cautious mathematics positionRetain human responsibility; use formal verification, cross-checks, and external pre-submission review.

Leiden Declaration on Artificial Intelligence and Mathematics (2026) · endorsed by the IMU

This division of responsibility is the thesis. Novelty has both a checkable literature-relative component and a human contextual judgment. Significance remains human.

The Leiden Declaration is evidence of alignment with a cautious mathematical position, not an endorsement of this particular system. It states that humans retain responsibility for correctness; it asks mathematical organizations to maintain rigor through measures that may include formal verification, cross-checking theoretical and computational results, or external pre-submission review. The International Mathematical Union has endorsed the declaration.

[Sources] - Leiden Declaration on Artificial Intelligence and Mathematics, 2 June 2026: https://leidendeclaration.ai/ - IMU endorsement, Circular Letter 8/2026: https://www.mathunion.org/fileadmin/documents/2026-06/IMU_AO_CL_8_2026.pdf

Discuss — what evidence would actually save you time?

Think of a paper you’ve reviewed recently.

1. What evidence could a machine gather to save you time—or make your review better?

2. What part would you still need to judge yourself?

Think, then discuss with a neighbor

AI review is already entering formal workflows

Supporting authorsInside formal review

NeurIPS ’24

Author checklistOptional feedback before submission

ICLR ’25

Reviewer coachFeedback on 20,000 human reviews

AAAI-26

Extra AI review22,977 papers in the full-review stage

TMLR announced

Soundness pilotAdvisory AI review; August rollout expected

SciCoQA46.7%best recall on 92 real paper–code discrepancies

What has not scaled

Execution-backed review of numerical experiments

No venue-scale system in this literature reruns experimental evidence across an entire conference.

Reviewing papers has scaled. Re-running their evidence has not.

Goldberg et al., arXiv:2411.03417 (2024) · Thakkar et al., Nature Machine Intelligence (2026) · Biswas et al., arXiv:2604.13940 (2026) · Rao et al., CSPaper OpenPrint (2026) · Baumgärtner et al., ACL (2026)

The roles matter. NeurIPS tested an optional checklist assistant for authors. ICLR randomized AI feedback on more than 20,000 human-written reviews; the AI was coaching reviewers, not reviewing papers. AAAI-26 was the first full-scale major-conference pilot to add an official AI-generated review alongside human reviews for every main-track paper that reached full review: 22,977 papers, with no score or recommendation from the AI. TMLR selected a soundness-focused system for a planned pilot in which every submission would receive one advisory AI review alongside the human reviews. As of 23 August 2026, CSPaper’s announcement still described deployment as expected in August; do not call it live without a fresh check.

SciCoQA is not a numerical-rerun benchmark. It tests whether models can identify meaningful discrepancies between a paper and its code. Its best-performing models detected 46.7% of the 92 real-world discrepancies. In a pooled analysis of 129 discrepancies, recall ranged from 41.1% to 55.8%, so recall—not precision—remains the bottleneck.

The statement that venue-scale numerical reruns have not appeared is a conclusion from this related-work audit, not a formal impossibility result. A likely explanation is operational cost and heterogeneity: GPU time, large or proprietary data, fragile dependencies, and specialized hardware. This is the transition to the next two slides: optimization may be a comparatively tractable test bed for execution-backed review.

[Sources] - Goldberg et al., arXiv:2411.03417 (2024): https://arxiv.org/abs/2411.03417 - Thakkar et al., A large-scale randomized study of large language model feedback in peer review, Nature Machine Intelligence 8 (2026): https://www.nature.com/articles/s42256-026-01188-x - Biswas et al., arXiv:2604.13940 (2026): https://arxiv.org/abs/2604.13940 - Rao et al., CSPaper OpenPrint (2026): https://cspaper.org/openprint/20260706.0002v1 - TMLR pilot announcement and expected rollout: https://cspaper.org/articles/tmlr-selects-cspaper - Baumgärtner et al., ACL (2026): https://aclanthology.org/2026.acl-long.1795/ - Starace et al., ICML (2025): https://proceedings.mlr.press/v267/starace25a.html

AI review is complementary—not interchangeable

AI adds coverage

46.6%

of reviewer and chair respondents agreed the AI found concerns a human reviewer would have difficulty catching.

+Both needed

People add coverage

49.4%

agreed the AI overlooked points a human reviewer would likely have caught.

AI advantage: finds technical errors · raises overlooked points · reads thoroughlyAI weakness: overemphasizes minor issues · misreads details · lacks big-picture judgment

Use AI as a second reader. Keep judgment human.

J. Biswas, S. Schoepp, G. Vasan, A. Opipari, A. Zhang, Z. Hu, S. Joseph, M. Lease, J. J. Li, P. Stone, K. L. Wagstaff, M. E. Taylor & O. C. Jenkins, AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot, arXiv:2604.13940 (2026) · 5,834 optional survey responses overall; percentages shown are reviewer/chair responses

Timing: 75 seconds. Present the two percentages together. Among program committee members, senior program committee members, and area chairs, 46.6% agreed that the AI review caught concerns a human reviewer would have difficulty catching, while 49.4% agreed that it missed points a human reviewer would likely catch. These are not contradictory findings; together they support complementarity.

The voluntary survey is subject to self-selection. Its 5,834 responses came from authors, program committee members, senior program committee members, and area chairs evaluating both AI and human reviews. Do not call these percentages population estimates. Respondents rated AI reviews more highly for identifying technical errors and raising unconsidered points; they also rated them more likely to overemphasize minor issues and make technical errors.

[Sources] - Joydeep Biswas, Sheila Schoepp, Gautham Vasan, Anthony Opipari, Arthur Zhang, Zichao Hu, Sebastian Joseph, Matthew Lease, Junyi Jessy Li, Peter Stone, Kiri L. Wagstaff, Matthew E. Taylor, and Odest Chadwicke Jenkins, AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot, arXiv:2604.13940 (2026), Sections 3.1–3.3: https://arxiv.org/html/2604.13940v1#S3

Evidence-backed audits can find real mistakes

Human-adjudicated sample263 / 316candidate findings confirmed83.2% precision

Checking claims, mathematics, and reasoning

Human experts confirmed 263 of 316 sampled findings: 83.2% precision in that adjudicated sample.

LessonSpecialized audits can surface real mistakes at scale.

LimitSampled precision is not the prevalence of errors across 2,500 papers.

Federico Bianchi, Yongchan Kwon, Zachary Izzo, Linjun Zhang & James Zou, To Err Is Human: Systematic Quantification of Errors in Published AI Papers via LLM Analysis, arXiv:2512.05925 (2025)

Numerical verification remains the hard frontier

PaperBench20ICML papers reproduced from scratch8,316 graded tasks

Code and reproducibility

PaperBench asks agents to reproduce research from scratch—a harder task than reviewing prose.

underspecifiedpaper ≠ codeartifact failsrun ≠ claim

The opportunity: structured claims and affordable experiments make optimization a tractable test bed.

G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, J. Heidecke, A. Glaese & T. Patwardhan, PaperBench: Evaluating AI’s Ability to Replicate AI Research, ICML (2025)

Do not compare PaperBench scores with PAT, SPOT, FLAWS, or the Bianchi audit. They measure different tasks. Refresh all time-sensitive claims immediately before delivery.

[Sources] - Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan, PaperBench: Evaluating AI’s Ability to Replicate AI Research, ICML (2025): https://proceedings.mlr.press/v267/starace25a.html - paperreview.ai: https://paperreview.ai/tech-overview - AI feedback on ICLR reviews: https://www.nature.com/articles/s42256-026-01188-x - Jayaram et al., arXiv:2606.28277 (2026): https://arxiv.org/abs/2606.28277 - NeurIPS PAT announcement: https://blog.neurips.cc/2026/04/21/neurips-supports-authors-with-googles-paper-assistant-tool-pat/ - Biswas et al., arXiv:2604.13940 (2026): https://arxiv.org/abs/2604.13940 - Rao et al., CSPaper OpenPrint (2026): https://www.cs.cmu.edu/~nihars/preprints/TMLR_AI_reviewer_evals.pdf - Son et al., arXiv:2505.11855 (2025): https://arxiv.org/abs/2505.11855 - Xi et al., arXiv:2511.21843 (2025): https://arxiv.org/abs/2511.21843 - Baumgärtner et al., ACL (2026): https://aclanthology.org/2026.acl-long.1795/ - Siegel et al., TMLR (2025): https://openreview.net/forum?id=BsMMc4MEGS

AI changes an old optimization bargain

Mathematical Programming · 1991

The 1991 paper's three principles: results justify claims, details allow reproducibility, and referees need not reproduce the results

AI can supply patient technical labor without making reproduction the referee’s personal duty.

Mathematical Programming Computation · today

2general referees

+

1technical editor

The technical editor examines the software and data, tests their usability, and assesses whether scientific results appear qualitatively reproducible.

The standard is old. What changes is who can afford to perform the check.

Jackson et al., Mathematical Programming (1991) · MPC reviewing guidelines (2026)

Timing: 90 seconds. Read the three principles quickly, then emphasize the boundary. In 1991, the field required enough evidence and detail for reproduction while explicitly saying that reproduction was not the referee’s duty. The authors also distinguished plausibility from significance: even a paper satisfying the three principles might still lack scientific value.

MPC now institutionalizes technical review. Two general referees and one technical editor typically handle each manuscript. The technical editor concentrates on software and data and assesses usability and whether the claimed scientific results appear qualitatively reproducible. This staffing model is an existence proof for separating technical checking from broader scientific judgment, not evidence that AI can replace the technical editor.

[Sources] - Jackson et al., Mathematical Programming (1991): https://doi.org/10.1007/BF01588801 - Mathematical Programming Computation submission and reviewing guidelines, accessed 23 August 2026: https://link.springer.com/journal/12532/submission-guidelines - Local principles excerpt: talks/figs/jackson-1991.jpg, cropped from refs/1991_Jackson_Guidelines_for_Reporting_Results_of_Computational_Experiments.pdf

Begin with the consequential claim—not the code

What does the paper promise?Start with the abstract, conclusions, and named contributions.

What evidence supports it?Proof, citation, table, figure, code, or computation.

What check could change the conclusion?Prioritize consequential tests over local nit-picking.

What would a person need to decide?Consequence, severity, and significance remain distinct.

Route each commitment to the right kind of audit

Theoryproof audit · counterexample search · formal certificatebroader architecture

Literature-relativecitation support · bounded prior-art searchbroader architecture

Numericspaper/code consistency · execution · stress testscurrent prototype

Significanceimportance · beauty · relevance · venue fithuman judgment

The numerical audit

From scientific claims to inspectable evidence. Rebuild experiments; compare released code.

Reimplement the paper, and compare with authors’ code

Four audit phases: read and specify, rebuild from the paper, run released materials, then verify and report

Witness and challenge simulations remain separate through independent review.

Evidence determines the next check

New evidence can trigger a new test

Suspicious equationDerive a small instance

Table or figure mismatchReconstruct the arithmetic

Missing comparisonSearch for a baseline or predecessor

The result changes the next action

Assumption violatedReject the counterexample

Test was faultyRepair it and rerun

Evidence is partial or disputedNarrow the claim or request a separate reviewer

Agentic means using new evidence to choose the next finite check.

We audited our own work first

Benchmark assumption

labels ∈ {−1, +1}

Observed instead{-1, 1}{0, 1}{1, 2}

10 / 66degenerate LIBSVM benchmark instances

5 major+3 minor

Algorithm mismatchAlgorithm 3 in code uses adaptive tuning omitted from the paper.

Benchmark constructionA label-encoding error makes 10 of 66 LIBSVM instances degenerate.

Unequal accountingGradient calls are counted differently between methods.

Inspect all eight findings →

Ya-Chi Chu, Wenzhi Gao, Yinyu Ye & Madeleine Udell, Provable and Practical Online Learning Rate Adaptation with Hypergradient Descent, ICML (2025) · audit packet · code commit f848bca25910

Timing: three minutes. Exactly one main results slide is devoted to HDM; the later app invitation reuses this public case. Use the packet as the source of truth. Pause on “Benchmark construction” and ask the three reviewer-vote questions separately. The exercise demonstrates that technical correctness and claim consequence are distinct. The audit inspected 31 of 33 datasets; ijcnn1 and leu were unavailable. Do not collapse all findings into the label issue or claim that every plotted result is false.

[Sources] - HDM packet: https://www.optimend.app/cases/chu_2025_hypergradient_descent_6 - Ya-Chi Chu, Wenzhi Gao, Yinyu Ye, and Madeleine Udell, Provable and Practical Online Learning Rate Adaptation with Hypergradient Descent, ICML (2025): https://proceedings.mlr.press/v267/chu25a.html - Code: https://github.com/udellgroup/osgm-best-hypergrad/tree/f848bca25910

Candidate findings span the whole research stack

1,105 candidate findingsacross 50 audit packages

Mathematical claims264

Algorithms and code206

Agreement between paper and code162

Files needed to reproduce results107

Numerical error and tolerances64

Reported results and metrics62

Time, memory, and compute57

Experiment design and uncertainty54

Baselines and test coverage54

Citations and prior work38

Reporting and claim scope37

Failures in claims and calculations

MPCMathematical claim

On a three-variable example, the theorem’s conclusion becomes 0 ≥ 2.

MPCAlgorithm

A minimization routine advertised as exact returns 3, although exhaustive search gives 2.

MORPaper versus code

The proof uses Euclidean distance; the code sums absolute values. A predicted decrease becomes an increase.

MORNumerical tolerance

A fixed threshold treats unequal eigenvalues as equal. The resulting derivative is more than a million times too large.

These are anonymized candidate findings from the 50-paper audit. MPC means Mathematical Programming Computation; MOR means Mathematics of Operations Research.

The first theorem and its corollary fail on the stated fractional-budget domain; the example does not establish failure for integer budgets. The exact-method example shows that the released implementation can certify a suboptimal answer, but it does not identify every affected published result. In the norm mismatch, the sign reversal can change step acceptance; the paper’s proof does not cover the released code. In the tolerance example, the symbolic formula is unaffected and no published table row has been shown to contain the ill-conditioned case.

[Sources] - Student public-example deck: results/deck.pdf - Private evidence and scope dossier: results/dossier.pdf

Failures in computational evidence

MORMissing files

All 48 runs for one comparison method are absent, so the comparison figure cannot be rebuilt.

MPCReported result

A benchmark checker accepts a worse solution—and even NaNs—as optimal.

MPCTime and compute

The paper claims a 2.5× speedup; the released wall-clock data show 1.32×.

MORUncertainty

Numerical results divide by the wrong sample count, making every confidence interval 41% too wide.

Failures in comparisons and conclusions

MORTest coverage

A matrix-completion method is tested only on the entries used to fit it—not on the entries it must recover.

MORPrior work

A method described as the first of its kind has an earlier published predecessor for the same class of problems.

MORClaim scope

The paper says “consistently outperforms,” but its own tables show losses.

Precision and recall require different answer keys

Known-error set E

Recall

Of the registered mistakes, how many did the audit find?

registered errors found÷registered errors

FLAWS: 39.1% of 713 planted paper–error pairs localized within 10 ranked excerpts by the best tested model.

Reported set A_τ

Precision

Of the reported findings, how many are real?

confirmed real findings÷findings reported

Optimend: authors add content and review errors found by AI.

Estimating error prevalence requires representative sampling and a more exhaustive reference audit.

Xi et al., arXiv:2511.21843 (2025) · Optimend prospective adjudication study (2026)

Recall and precision require different answer keys. For recall, register a known or planted error set, fix the output budget and matching rule before running the audit, and report recall against that set. FLAWS calls its metric identification accuracy at ten. Because each benchmark instance contains one inserted target error, it supplies the recall-like denominator needed here: 713 planted paper–error pairs and at most ten ranked excerpts. The best tested model localized 39.1%. This is not natural-paper recall or field-wide error prevalence.

For precision, start from every finding the system reports above a stated threshold, or a prespecified probability sample. Authors are uniquely valuable for intent, version history, and missing artifacts, but are not automatically independent ground truth. Send disputed or uncertain cases to independent expert adjudication. Nonresponse and lack of contest remain unadjudicated, not confirmed.

Fix the reporting threshold, error taxonomy, and matching rule in advance. Otherwise recall can rise merely because the system emits more candidates, while precision changes with the threshold. Keep correctness separate from usefulness and severity; a finding can be real but minor. Findings cluster within papers, so use paper-level uncertainty and report precision by type and severity when sample sizes allow. The webapp on the next slide begins the precision and usefulness study; it does not measure recall.

[Sources] - Xi et al., arXiv:2511.21843 (2025): https://arxiv.org/abs/2511.21843 - Optimend: https://www.optimend.app/ - Current measurement boundary: talks/audit-results-brief.md

Where should a verification packet live?

AAuthor precheckbefore submission

BJournal technical checkbefore human review

CReviewer / AE toolinvoked when useful

DPost-publication auditafter the record is public

Existing policies constrain reviewer-run AI audits

Policy pattern	Representative journals	What a reviewer may provide to AI
Prohibited	MPC · MP SIOPT · SIMODS IJOC · EJOR	No confidential manuscript text, figures, or supplements
Controlled-system exception	MOR · Management Science · IJOO	Only systems that do not retain, share, or train on manuscript content
Public-version exception	Operations Research	Public preprints may go to public AI; confidential material requires a protected tool

Across this survey: no unrestricted upload of confidential submissions to public consumer AI.

Policy snapshot: 23 August 2026 · Springer Nature · SIAM · INFORMS · Elsevier

This is a policy snapshot, not legal advice. The meaningful distinction is not simply “AI allowed: yes or no.” It is which material may be supplied to which tool under what confidentiality conditions.

Springer Nature asks reviewers not to upload manuscripts to generative AI. SIAM prohibits referees from sharing papers under review with generative AI in any form. INFORMS allows review use only when the tool does not retain, share, or train on manuscript content and fully satisfies confidentiality rules. Operations Research adds a public-version exception: an openly posted preprint may be uploaded to public AI, while the confidential submission may not. Elsevier does not permit manuscript upload even to a nominally private tool, though private tools may support background search or improve the reviewer’s own prose without receiving manuscript content.

All surveyed policies preserve human accountability. Most are more permissive about authors using AI, usually with disclosure, than about reviewers uploading confidential submissions. This policy landscape currently makes a reviewer-run audit difficult and non-uniform; a journal- or AE-provided system could specify the tool, data handling, disclosure, and author-response process consistently. Refresh immediately before delivery.

[Sources] - Local survey and journal-level links: journal-ai-policy-survey.md - Springer Nature AI guidance: https://group.springernature.com/gp/group/ai/ai-guidance-for-our-researchers-and-communities - SIAM Editorial Policy on Artificial Intelligence, version 2.0: https://epubs.siam.org/artificial-intelligence - INFORMS Guidelines for Ethical Behavior in Publishing: https://pubsonline.informs.org/authorportal/guidelines-for-ethical-behavior-in-publishing - Operations Research Generative AI Guidelines: https://pubsonline.informs.org/page/opre/generative-ai-guidelines - IJOC reviewing and generative AI policy: https://pubsonline.informs.org/page/ijoc/aipolicy - Elsevier generative-AI policy for journals: https://www.elsevier.com/about/policies-and-standards/generative-ai-policies-for-journals

Start with an opt-in, fail-open pilot

InputPublic preprint or a manuscript supplied with explicit author consent

OutputEvidence packet and unresolved questions—never an automatic score

ResponseAuthors add context or corrections before any finding circulates

EvaluationFix the reporting threshold; adjudicate correctness and usefulness

Earn authority with evidence. Do not grant it by default.

Try the audit yourself

Public evidence packet

Hypergradient Descent

5 major · 3 minor

Claim → check → evidence → bounded finding

Inspect the case →

optimend.app

Submit your paper for an audit

Optimend · public HDM case · QR → https://www.optimend.app/

Make research easier to check

AI will make optimization research easier to produce.

AI should also make it easier to check.

Machines gather evidence. People decide what matters.

optimend.app

FAQ — who should run the audit?

Won’t it nitpick?

Target consequential claims, not issue counts. Minor errors can still obstruct reproduction; authors can explain or repair them.

Should authors run it?

Yes—but many do not. Strong audits still require specialized workflows.

Should reviewers run it?

Private tools vary. A journal or AE can commission one common audit.

Isn’t it too expensive?

In our current runs, each audit takes about 40 minutes. A journal would need one API subscription plus computation proportional to its volume.

Nitpicking is partly a prompting and workflow-design problem, which is one reason for the community to develop a shared standard rather than many hidden private prompts. Even a minor real mistake can cause major headaches for a student trying to reproduce or understand a paper. The paper should be correct, and authors are best equipped to supply context and a repair.

So far, the candidate findings surfaced to authors have seemed minor enough that no one appears alarmed. That does not suggest a field in crisis, which is good. Keep the inference narrow: many authors have not formally adjudicated individual findings. Also emphasize that the distribution is moving as AI-assisted paper production scales; future papers may contain more errors or different kinds of errors.

Authors can and should run prechecks, but many do not. Choosing a model, writing a good prompt, granting artifact access, and checking numerical experiments remains challenging. Reviewer-run audits are also costly in attention: every reviewer may use a different tool, and someone must sort through the output. Madeleine’s AE anecdote: two reviews of a mostly theoretical paper did not mention errors in the numerics; an AI check found several serious but reparable issues, including flipped inequalities, incorrect references, and missing assumptions. The AE is already synthesizing evidence, so a standardized AE- or journal-solicited audit is a more natural institutional location.

The runtime is a speaker-supplied planning estimate, not a measurement from the current result package. “40 minutes” means approximate wall-clock audit time, not 40 minutes of human labor. Do not quote a per-journal cost until model calls, tokens, compute, and staff time have been measured.

[Sources] - Speaker notes and operating estimates supplied by Madeleine Udell, 23 August 2026.

AI can already be a useful second reader

First page of Liang and colleagues' study of AI feedback on research papers

Feedback on papers and reviews

AI–human overlap was comparable to human–human overlap; many authors found the feedback useful.

LessonA useful second reader is already plausible.

LimitPlausible feedback is not the same as detecting a consequential error.

W. Liang, Y. Zhang, H. Cao, B. Wang, D. Ding, X. Yang, K. Vodrahalli, S. He, D. Smith, Y. Yin, D. McFarland & J. Zou, Can large language models provide useful feedback on research papers? A large-scale empirical analysis, NEJM AI (2024)

Related-work map

Feedback and review assistance
Liang et al., NEJM AI (2024)
paperreview.ai (2026)
Thakkar et al., Nature Machine Intelligence (2026)
Biswas et al., arXiv:2604.13940 (2026)

Claim and correctness checks
Bianchi et al., arXiv:2512.05925 (2025)
Jayaram et al., arXiv:2606.28277 (2026)
Son et al., arXiv:2505.11855 (2025)
Xi et al., arXiv:2511.21843 (2025)

Code and reproducibility
Baumgärtner et al., ACL (2026)
Siegel et al., TMLR (2025)
Starace et al., ICML (2025)
Rao et al., CSPaper OpenPrint (2026)

Do not compare headline percentages across tasks with different targets, selection rules, and grading procedures.

FAQ — what about privacy, security, and wariness of AI?

ConfidentialitySpecify provider, retention, training use, access control, and incident response.

Deployment choicesUse a contract-protected private API or an improving local model; public and confidential papers need different rules.

Author responseLet authors correct scope before circulation; define opt-out consequences.

Institutional compactName the tool, data terms, disclosure, accountability, and audit meaning.

Ask whether AI review is more like opening a PDF in a cloud-backed viewer or discussing the manuscript with a colleague down the hall. Neither analogy settles the policy; it exposes the choices about third-party processing, retention, training, human access, and accountability. Do not claim all APIs have the same terms. Contract-protected private APIs and local models can reduce risk, but the actual policy must be checked.

Many papers are already public on arXiv and many researchers already synchronize local files to services such as Dropbox. Those facts make some objections weaker, but do not erase the confidentiality of an unpublished submission. Opt-in may be necessary. If so, the journal must decide how to indicate that one paper received a deeper technical audit than another, and whether equal decision speed is compatible with declining an optional service.

Policy disagreement may motivate new journals or author compacts that state AI-review terms in advance. Present that as an institutional possibility, not an immediate mandate. The journal-policy slide in the main talk shows that existing policies currently vary and generally restrict confidential manuscript upload.

[Sources] - Journal-policy snapshot and links: journal-ai-policy-survey.md - Speaker framing supplied by Madeleine Udell, 23 August 2026.

Two packets show different failure modes

PEPit

Mathematical Programming Computation

Certificate acceptanceA finite bound can return after a non-accepted solver status; certificate claims need a fail-closed gate.

State mismatchThe printed Point-SAGA potential omits memory state represented in code.

WMMSE

Mathematics of Operations Research

Figure interpretationFigure 2’s one-radius labels and surface axes are interchanged.

Feasibility witnessA rectangular SVD path can return a covariance outside the stated Gelbrich ball.

Goujaud et al., Mathematical Programming Computation (2024) · Nguyen et al., Mathematics of Operations Research (2023) · exact source revisions and witnesses in packets

These remain candidate findings with inspectable evidence. The authors have not contested the findings surfaced to them, but most have also not completed formal finding-by-finding adjudication. Nonresponse or lack of contest is not certification. For PEPit, do not claim that all bounds or published figures are false. For WMMSE, do not claim that the central Gaussian equilibrium theory fails or that every historical figure used the affected path.

[Sources] - Goujaud et al., Mathematical Programming Computation (2024): https://doi.org/10.1007/s12532-024-00259-7 - PEPit source: https://github.com/PerformanceEstimation/PEPit/tree/ce6242347ed64f463e36ac3fb09704189e71a452 - Local PEPit packet: talks/audit-author-packets/pepit_candidate_findings.pdf - Nguyen et al., Mathematics of Operations Research (2023): https://doi.org/10.1287/moor.2021.1176 - WMMSE source: https://github.com/sorooshafiee/WMMSE/tree/fcf52dcd0232fdd3753a4a88f43101f886c37829 - Local WMMSE packet: talks/audit-author-packets/wmmse_candidate_findings.pdf - Author-response status supplied by Madeleine Udell, 2026-08-23.

One inspectable witness: a number is not automatically a certificate

PEPit certificate acceptance

In the selected release, the wrapper records solver status but the return path tests only whether a numerical value exists.

adverse or inaccurate status→finite value→warning→value returned

Required boundary: define accepted status and feasibility, PSD, reconstruction, and primal–dual-gap tolerances; otherwise label the result as non-certified.

Baptiste Goujaud, Céline Moucer, François Glineur, Julien M. Hendrickx, Adrien B. Taylor & Aymeric Dieuleveut, PEPit: computer-assisted worst-case analyses of first-order optimization methods in Python, Mathematical Programming Computation (2024) · source tag 0.3.2, commit ce6242347