What Agentic AI Can Prove and What It Still Fabricates: The Real Frontier of Machine Science

Agentic AI beats humans at narrow, verifiable tasks
Citation fabrication has grown twelvefold since 2023
Trust checkable answers; verify everything else

In 2023, roughly 1 in every 2,828 published papers contained a fabricated citation. By 2025, that rate had climbed to 1 in 458. In the first seven weeks of 2026, it reached 1 in 277, a sixfold jump in under three years, tracked by a Columbia University team publishing in The Lancet. Around the same time, a DeepMind system called AlphaEvolve found a new 48-multiplication algorithm for multiplying two 4x4 complex-valued matrices, beating a record that had stood since 1969. Agentic AI in scientific research is delivering both outcomes at once. Real, checkable mathematical discovery sits on one side. A rising tide of invented references sits on the other. That contradiction, one system proving a theorem and inventing a source in the same year, says more about where machine science actually stands than any single benchmark.

Where Agentic Search Actually Wins

AlphaEvolve didn't guess at a better algorithm, it paired a large language model with an evolutionary search loop, proposing code, testing it against a strict mathematical objective and refining whatever failed, a loop that matters more than the model driving it. A matrix-multiplication algorithm either uses fewer multiplications than the current record or it doesn't, with no ambiguity to resolve and no judgment call about whether the result counts. The system could run millions of attempts and keep only the ones that passed a hard, mechanical check.

The same pattern shows up elsewhere. Multi-agent systems built for physics-informed and operator learning, where separate agents propose, critique and refine candidate solutions, have cut error rates by up to 10,000-fold against human-designed baselines on some benchmarks, though typical gains are far more modest. Sakana AI's autonomous research system went further, drafting and submitting three complete manuscripts to an ICLR workshop without a human author and one scored above the average acceptance threshold given to human-written submissions.

Just how widely this influence is spreading through the literature shows up in the numbers below: a Tübingen re-analysis found excess LLM-associated vocabulary in 89 percent of December 2025 biomedical full texts, up sharply from the 13.5 percent estimated for abstracts alone a year earlier. In every one of these cases, the task offered something a machine can check, whether a number to beat, a workshop rubric to satisfy or a simulation that either matches known physics or doesn't. Agentic AI performs best exactly where success can be verified automatically, at scale, without a person reading the fine print.

The Line Agentic Systems Cannot Cross

Move away from that narrow category and performance falls off fast. Researchers building benchmarks for data-driven scientific discovery- the kind of work that asks a system to explore a real dataset, choose an appropriate statistical method, and draw a defensible conclusion- have repeatedly found current models unreliable at the task. The models struggle to judge which statistical method fits a given dataset in the first place and causal reasoning, the kind of inference that separates a genuine finding from a coincidental pattern, gives them even more trouble. This is a different kind of problem than matrix multiplication because the right answer here depends on context and messy data rather than a single number to search toward.

Novelty assessment shows the same gap from another angle. Several research teams have tried using large language models themselves as judges of whether a proposed research idea is genuinely new, largely because human expert review doesn't scale to the volume of ideas these systems can generate. Results have been mixed at best. Judging novelty requires comparing a new idea against the full, unwritten context of a field, including work that never made it into a clean dataset the model can search. A system trained to sound confident doesn't necessarily know what it doesn't know and scientific novelty punishes exactly that kind of false confidence.

The Verification Gap: Why Citations Keep Breaking

This gap between narrow, verified tasks and open-ended judgment shows up most starkly in citations because a citation is a small, checkable claim that current systems still get wrong at scale. An audit of 2.5 million papers across arXiv, bioRxiv, SSRN and PubMed Central found close to 146,900 AI-generated fake citations in 2025 alone, with a sharp rise beginning in mid-2024. Researchers at Deakin University ran a controlled study, asking GPT-4o to write short literature reviews on mental health topics and found that 19.9 percent of the citations it generated were entirely fabricated, referring to papers that don't exist. Looking at 100 hallucinated citations found in NeurIPS 2025 papers, a separate analysis found that 66 percent were total fabrications rather than corrupted versions of real sources and every single one involved more than one kind of error at once: wrong author, wrong title, wrong venue, stacked together in a single fake reference.

A citation looks, structurally, exactly like the rest of a model's fluent output: a plausible string of words shaped like an author name, a title and a year. Nothing in how these systems generate text distinguishes a claim grounded in a real, retrievable source from a claim that merely sounds like one. Correct matrix multiplication is easy for a model to prove because a checker confirms the answer instantly and rejects anything that fails. No equivalent check sits inside the model when it writes a bibliography, which helps explain why fabrication rates keep climbing even as the underlying models keep improving at nearly everything else.

Matching the Tool to the Task

None of this makes agentic AI a poor fit for science. What it lacks is judgment on unverifiable tasks and what it has in abundance is speed on tasks with a clear right answer. Algorithm search, simulation-driven optimization and code generation against a fixed test suite now sit solidly in expert territory. Literature synthesis, statistical method selection on real-world data and judgments of novelty or significance remain the weak spot, where reliability stays low and where convincingly worded prose can mislead a reader who has no reason to doubt it.

The productive path forward treats this as an engineering problem rather than a reason for blanket suspicion. Wrapping agentic systems in external verification closes much of the gap that raw model output leaves open: citation-checking tools that confirm a reference actually exists before it reaches a draft, symbolic verifiers for statistical claims and mandatory human review at the judgment-heavy stages. Systems like AlphaEvolve already prove this works when the check is built into the search loop itself. The unresolved question isn't whether agentic AI belongs in a laboratory, since it already does. It's whether the parts of research that still require judgment get the same rigorous screening that matrix multiplication already receives or whether they get left to a model that can't yet tell a real citation from an invented one.

A system capable of beating a 57-year-old mathematical record is, at the very same moment, capable of inventing a reference to a paper that was never written. Agentic AI has earned real trust in the domains where its answers can be checked in an instant and hasn't yet earned that trust anywhere else. Treating every output the same, whether it comes from a verified search loop or an unverified paragraph of prose, is the actual risk. The technology's capabilities have already been demonstrated. What research now needs is the discipline to stop asking it to do the rest without anything watching for mistakes.

This article reflects the analytical judgment of The SIAI Editorial Board and does not constitute policy advice or the official position of any affiliated institution.

References

Ansari, N. (2026) 'Compound deception in elite peer review: a failure mode taxonomy of 100 fabricated citations at NeurIPS 2025', arXiv preprint.
Bao, H., Sun, M. and Teplitskiy, M. (2025) 'Where there's a will there's a way: ChatGPT is used more for science in countries where it is prohibited', Quantitative Science Studies.
Gibney, E. (2025) 'DeepMind AI algorithm beats 56-year-old record for matrix multiplication speed', Nature. London: Springer Nature.
Holzwarth et al. (2026) LLM Usage in Biomedical Full Texts. Tübingen: University of Tübingen, arXiv:2608.10715.
Jiang, X. and Karniadakis, G.E. (2025) AgenticSciML: Multi-Agent Systems for Physics-Informed and Operator Learning. Providence: Brown University.
Linardon, J., Jarman, H.K., McClure, Z., Anderson, C., Liu, C. and Messer, M. (2025) 'Influence of topic familiarity and prompt specificity on citation fabrication in mental health research using large language models: experimental study', JMIR Mental Health. Geelong: Deakin University.
Topaz, M., Roguin, N., Gupta, P., Zhang, Z. and Peltonen, L-M. (2026) 'Fabricated citations: an audit across 2.5 million biomedical papers', The Lancet, 407.
Yamada, Y., Lange, R.T. and Lu, C. (2025) The AI Scientist-v2: Workshop-Level Automated Scientific Discovery. Tokyo: Sakana AI.