More papers do not prove more discovery.

A research administrator's in-tray holds twice as many draft abstracts this term as it did last term. The tray, the administrator and the doubling are invented for this dispatch; the doubling is not anyone's measured effect. Twenty-four drafts are easy to count, and the count is easy to report. What the count cannot say is what is in the tray: more reliable answers to questions that matter, more versions of answers we already had, or more claims that someone will now have to check.

Last Wednesday, May 27, Anthropic published early results from a survey of 1,260 quantitative social scientists, fielded in late February and March. Among respondents, 81 percent said they had used generative AI to aid their research, but only 20 percent regularly used coding agents, tools that write and run analysis code on their own. Those who did reported starting more empirical projects, posting more working papers and sending out more grant proposals over the previous six months than respondents in the same discipline and career stage: around 10 percent more projects started and around 75 percent more working papers posted. The authors found no evidence that the same researchers were submitting more papers to journals.¹

The authors are careful about what this means, and the care is the useful part. The comparison is descriptive, they write, and "differences should not be interpreted as causal." The survey did not measure the quality of any of the work.¹ My argument is that research leaders should hold onto that distinction as these tools spread. Research output volume counts what was produced: projects started, analyses run, drafts posted, proposals sent. Credible contribution is what the work adds to what is known once someone has checked it: a finding that holds up, a method others can use, a careful negative result. More output is a promising signal. It is not by itself evidence that discovery has increased, or that the tool caused the increase. Count the work, and also measure what it adds.

The opportunity is real

It is worth starting with what is promising, because it is a good deal. Much empirical research is held up by the work between an idea and a result: cleaning data, writing analysis code, debugging it, rerunning it when a variable turns out to be miscoded. The survey found that the most common research use of AI, among coding agent users and other AI users alike, was generating code to analyze quantitative data, reported by 97 percent of coding agent users and 77 percent of other AI users.¹ If that work gets cheaper, a researcher can pursue questions that used to cost too much time to try, and can abandon a dead end sooner. The authors note the hope that fast execution could make research cheaper, more plentiful and more daring.¹

Nothing in this dispatch argues against that hope. The survey does not show that it has come true, and its authors do not claim it does. The randomized experiment they are running alongside it, which gives some researchers access to Claude Code and compares them with others, is designed to answer exactly that question, and its results are still to come.¹

Who adopts is not what the tool does

The difficulty with comparing adopters to everyone else is that people choose whether to adopt. Imagine that, before any coding agent existed, some researchers were already starting more projects than their colleagues: they had more energy, more data, fewer teaching duties or more urgent career pressure. Those are also the people most likely to try a new tool that promises to speed up the work. Compare them with their colleagues a year later and they will be starting more projects, whether or not the tool did anything at all.

The survey's own description of who adopts makes this concrete. Adoption was concentrated among early-career researchers: just over a quarter of doctoral students and postdocs used coding agents at least weekly, and the rate among tenured professors was less than half that. Respondents with typically male names, classified by name, adopted at more than twice the rate of those with typically female names, and researchers at top universities were notably more likely to use the tools.¹ The authors adjust for discipline, career stage and the week of the survey, but they say plainly that the two groups probably differ in many ways the survey cannot measure, and that early adopters may simply have been more productive to begin with. The sample itself was recruited for a study that offered access to Claude accounts, so respondents probably lean toward researchers who are curious about these tools.¹

Selection can also change under a study's feet. In February, METR, a nonprofit that studies AI capabilities, described why it was redesigning an experiment on developer productivity. Its earlier study, with data from early 2025, had found that experienced open-source developers took longer on tasks when allowed to use AI. When it repeated the experiment later in 2025, an increasing share of developers declined to take part because they did not want to work without AI, and between 30 and 50 percent of surveyed participants said they had held back some tasks they did not want to do without it. METR concluded that its new data gave "an unreliable signal of the current productivity effect of AI tools," and that the true effect was probably larger than its estimate.² That is a different population and a different design, not a replication of the survey. It is a reminder that who shows up in the comparison is part of the result.

Output is a chain, not a number

The survey has a second feature that deserves more attention than the headline. The differences between users and nonusers appeared only early in the research pipeline: projects started, working papers posted, grant proposals sent. At the stage of submitting new papers to journals or resubmitting revised ones, the authors found no evidence of a difference. They offer two readings. It may simply be too soon, since coding agents are new and journal submission takes time. Or the tools may be more useful for getting projects started than for the last mile of finishing them.¹

Either way, the finding is a useful warning against treating research output as one number. A research result passes through several distinct states, and each supports a different claim. An analysis can be generated. It can be executed, end to end, from the raw data. A specific claim in it can be checked by someone other than its author. Its main result can be reproduced, by a fresh analysis of the same data or by a new study. And it can be used, to inform a decision or later research. These are different kinds of evidence, not a ladder that every study must climb in order. A theoretical paper may never be executed in this sense, and a valuable exploratory analysis may never be reproduced. But a count of the first state says nothing about the others.

An evidence record for assisted work

If coding agents make it cheap to produce analyses, the scarce thing becomes the ability to tell which analyses to trust. That suggests a simple practice for any research group adopting these tools: attach an evidence record to each consequential result. What follows is my own recommendation, not a checklist the survey tested.

The record would hold five things. The question, and, where the field uses them, the analysis plan written before the results were seen. The provenance of the data: where it came from, which version, and what was done to it. The methods in executable form, so that someone else can run them. A list of deviations from the plan, including the analysis choices that an agent made rather than the researcher, since an agent can make many small choices quickly and without comment. And, for the results that matter most, an independent check by someone other than the author.

The deviations deserve emphasis. When running an analysis takes an afternoon, a researcher runs a few. When it takes a minute, it is easy to run dozens of variations and report the one that looks best, without ever deciding to do anything questionable. The survey's authors mention respondents' fears that AI tools could aggravate existing problems in social science, such as selective reporting and incremental research.¹ A record of what was run, not only of what was reported, is the cheapest defense. The authors also note that the way their fields study the economy and politics is increasingly shaped by analysis decisions made partly by coding agents.¹ Those decisions should be visible.

Reproducible code is not the same as a correct finding. A script that runs perfectly can implement the wrong idea, and a reproduced result can share its original's mistaken assumption. The record does not settle whether a hypothesis is true. It settles whether anyone can find out.

Fund the checking, not only the starting

For people who fund or manage research, the practical choice is about where capacity goes. If tools make starting projects cheaper, more projects will start. The work that follows them, checking, reproducing, maintaining code and data so that others can use them, does not get cheaper at the same rate, and it is rarely what gets counted. A department that rewards new starts and working papers will get more of both. Whether it gets more knowledge depends on whether anyone is paid to check.

That suggests reserving some capacity for replication, verification and maintenance alongside new projects, and recognizing it when it is done. How much depends on the field and the stakes. A group whose results inform medical or financial decisions should spend more on checking than one doing early exploratory work, and there is no universal ratio I would defend. The point is that the share should be a decision rather than whatever is left over after the new starts.

Objections, considered

More output can mean more discovery. It can. Discovery is partly a numbers game: more questions tried means more chances of finding something, and cheaper tools can open research to people with fewer resources. Nothing here argues for producing less. The argument is against treating the count as if it already measured the value.

Quality is slow and contested to measure. True. Citations take years and reward fashion; peer review is inconsistent and does not guarantee that a result is right. But the difficulty of measuring quality is a reason to record the evidence a result has, not a reason to count something easier instead. A tray annotated with specific checks is more informative than a tray that is merely fuller.

Some research cannot be fully reproduced. Sensitive data, one-time field conditions and confidential sources can make full reproduction impossible. Methods, code and the procedure for gaining access to the data can still be documented, and an authorized colleague can still check a result under the same restrictions. The aim is a record that matches the work, not a single standard imposed on every kind of study.

The survey might be right about productivity. It might. The early-pipeline differences could be the first sign of a real gain, as the authors themselves suggest is possible.¹ If so, the planned experiment should show it. Until then, a leader should neither dismiss the signal nor plan budgets as if it were a measured effect.

Exploratory work should not need a docket. Agreed. A quick look at a dataset to see whether a question is worth asking should stay quick. The evidence record matters for results that someone will rely on, and an exploratory analysis that produces a careful negative result can be valuable even if it attracts little attention. The record is for claims, not for curiosity.

Count the work, then ask what it added to what we know.

Before the next research review

• Which outputs received independent checks? For each consequential result, record which check it passed, who did it and what was examined.
• What comparison supports the productivity claim? Ask whether it compares the same researchers before and after, a randomized group, or simply adopters with everyone else.
• What evidence of quality is still missing? Name it, rather than letting a count of drafts stand in for it.
• Who is paid to check? Decide how much capacity goes to replication, verification and maintenance, and recognize that work when it is done.

Back to the tray. The twenty-four drafts may be the best term the department has had. The way to find out is not to count them again. It is to clip a record to each one that matters: where the data came from, what was run, what changed along the way, and who checked it. Then the tray can be counted twice, once for what was written and once for what is known.