The wrong diagnosis

There is a growing and structurally important disparity in how people perceive AI. Regardless of where you might sit on the "out of the loop vs vibecoding every day" spectrum, perception of AI and its current and future capabilities is a main driver of its development, expansion and adoption, with profound consequences for the future of society in all axes.

The prospect of achieving AGI is a powerful lobbying strategy, and one AI leaders use to shape their narrative to best sell their product to whichever audience they're addressing. As Karen Hao¹, author of the best-selling book "Empire of AI", put it: "This idea that we can replicate human intelligence is pegged to this idea that we understand human intelligence, but there's actually no scientific consensus around what human intelligence is. (...) And so the effect of that lack of clarity is that these companies get to just make AGI whatever they need it to be."

There is one particular ideology that, as a neuroscientist, has me concerned. I first came across it in Anthropic CEO Dario Amodei's "Machines of Loving Grace" essay². In it, Amodei provides a vision of what AI could deliver for humanity in the next decade. He also provides extensive reasoning on how AI has the potential to become a "virtual biologist".

"If our core hypothesis about AI progress is correct, then the right way to think of AI is not as a method of data analysis, but as a virtual biologist who performs all the tasks biologists do, including designing and running experiments in the real world (by controlling lab robots or simply telling humans which experiments to run – as a Principal Investigator would to their graduate students), inventing new biological methods or measurement techniques, and so on."

Suffice to say, he is not alone in his vision: OpenAI CEO Sam Altman³ says this will be achievable by 2028, and companies like SF-based Periodic Labs⁴ (backed by Jeff Bezos and valued at $7.5B) and Lila Sciences⁵ are trying to build just that.

As someone driven by the quest for knowledge, I am eagerly looking forward to the next decade of AI development, and seeing whether it will bring what has been promised. The problem with this motivation is not that it doesn't have potential, it is that I believe it rests on a misdiagnosis of why science is often slow.

"(...) large fraction of the progress in biology has come from a truly tiny number of discoveries, often related to broad measurement tools or techniques (...) I think their rate of discovery could be increased by 10x or more if there were a lot more talented, creative researchers. (…)

First, these discoveries are generally made by a tiny number of researchers, often the same people repeatedly, suggesting skill and not random search (…)

Second, they often "could have been made" years earlier than they were: for example, CRISPR was a naturally occurring component of the immune system in bacteria that's been known since the 1980s, but it took another 25 years for people to realize it could be repurposed for general gene editing. They also are often delayed many years by lack of support from the scientific community for promising directions.

Third, successful projects are often scrappy or were afterthoughts that people didn't initially think were promising, rather than massively funded efforts. This suggests that it's not just massive resource concentration that drives discoveries, but ingenuity. Finally, although some of these discoveries have "serial dependence" (...) many, perhaps most, are independent, meaning many at once can be worked on in parallel.

Both these facts, and my general experience as a biologist, strongly suggest to me that there are hundreds of these discoveries waiting to be made if scientists were smarter and better at making connections between the vast amount of biological knowledge humanity possesses (again consider the CRISPR example)."

— From Machines of Loving Grace²

The implicit logic behind Amodei's argument is that a large bottleneck to scientific development is researcher quality or quantity. It is much easier to defend the need for AI by transferring the blame of slow progress from the system to the people; as a society, we should proceed with caution when amplifying such visions. If the dominant narrative is "researchers are the bottleneck, AI will solve this", then the incentives to invest in more science and scientists and to reform the academic system weaken. Science funding decisions, public science policy, and career incentives for young researchers crucially depend on it.

I am not arguing that Amodei is malicious or that AI is useless. I am arguing that we should be very careful about which diagnoses we amplify, because the wrong diagnosis produces the wrong treatment. In this essay I'd like to provide some evidence that actual bottlenecks to scientific progress are structural.

Why the academic system hinders science

Science has become progressively more capitalistic since World War II⁶. Not in the sense that it's profitable per se, but in the sense that it has absorbed the metrics and incentive structures of market economies: output volume, competition, performance measurement, and precarity⁷.

Post-war economic and societal shifts allowed for more people, over decades, to be able to reach higher education, and, consequently, research. The expansion of PhD programs⁸ without a corresponding expansion in permanent research positions has been central to this shift, producing a large pool of highly trained, precarious researchers competing for a number of positions that did not accompany such growth.

Alongside the position stall, funding rates for research projects have kept declining. In the US, National Institutes of Health (NIH) R01 grant success rates⁹ have dropped from 44% in 1975 to 13% in 2025. Similarly, in Europe, European Research Council grants¹⁰ have a success rate of around 12%. This means the majority of submitted science, even if reviewed as competitive by expert panels, goes unfunded. Regarding the creativity issue mentioned by Amodei, this consequence is not neutral — low success rates incentivize conservative proposals. Scientists propose what they know will work, not the side-project out of the box idea that might be transformative.

Under such monetary constraints, intellectual freedom in research becomes limited. It is true for almost all areas of research, in particular in biology, that certain topics become highly studied and "trendy", leaving many others dormant. This produces a selective pressure to publish new results in said topics from both the scientific magazines and the funding institutions in a cyclic manner. Looking for instance at Neuroscience, an analysis of the last five decades of abstracts in leading neuroscience journals¹¹ shows such trends. For instance migraine, which affects more than 1 billion people worldwide¹² and is one of the top causes of years lived with disability, ranks at 168th place. Multiple sclerosis, which affects 2.9 million people globally¹³, ranks 30th. Both are neurological, disabling and neither immediately fatal diseases, but the contrast of research effort does not equal their impact in society. And such examples are never-ending.

The CRISPR technology development is an example of such dynamics. Decades of research were poured into understanding bacterial antiviral defense mechanisms by a small community, scattered all over the world, before Jennifer Doudna and Emmanuelle Charpentier ultimately received the Nobel prize for the application of this knowledge into a gene editing technology. The conceptual jump from "how does bacterial immunity work" to "can we make a programmable nuclease" was only possible in the first place because the field had already gathered substantial amounts of knowledge. The facts were not lying dormant in the literature waiting for a sufficiently brilliant mind to connect them, but rather, they were produced one at a time over decades, by a community dedicated to such niche question despite the challenges. Several of the researchers involved struggled to publish landmark results¹⁴, and Doudna herself has stated¹⁵ that at the time she had funding for other projects from multiple sources, and that this was in fact a crazier, harder-to-justify side project.

The funding crisis also eventually compounds into a productivity issue. Team leaders and researchers in general spend a substantial fraction of their working time writing and reviewing grants, rather than actually doing research. From a financial standpoint, one study examining grant applications¹⁶ from the NIH equivalent in Australia found that, each year, 550 years of researchers' time is spent writing grants, equating to a cost of AU$66M.

But this funding crisis doesn't affect all researchers equally. The "Matthew effect" in science funding¹⁷ is a term used to describe the fact that those who have received grants are more likely to receive future grants. In the US, NIH research project grants are heavily skewed¹⁸ regarding distribution of funding, with 1% of awardees getting about 11% of all grant dollars, and 10% of awardees getting about 40% of the dollars. Beyond its unfairness, this concentrates intellectual influence and narrows the question space of funded science.

With lower success rates comes a higher structural pressure to excel. The answer? An academic prestige economy that ties career survival to publication count and journal prestige. The well known "publish-or-perish" system. At first glance, it might look just like the pattern with any other job — if you are great and succeed, then you'll be able to land better jobs, at better companies, paying you a higher salary. The issue is, once this ideology and vicious cycle is tied to research, you end up damaging science.

The consequences are well-documented and severe: a 2014 UK report¹⁹ found that 1 in 5 researchers over 35 had considered compromising their own research integrity, and this rose to 1 in 3 for those under 35. This phenomenon has produced a global market in fraudulent academic work, where so-called "paper mills" generate fake peer-reviewed papers for sale (well demonstrated in this video²⁰). Not surprisingly, AI tools have dramatically lowered the cost and increased the sophistication of fabricated research. Therefore as AI keeps advancing, this problem keeps worsening. A 2025 study identified 1,720 articles²¹ from paper mills infiltrating IEEE conference proceedings alone, affecting up to 23.5% of individual conferences and involving over 6,500 researchers across 55 countries.

Two by-products of this issue arise. The first is, again, a cut at the level of scientific creativity, both at the level of proposals, but also at the level of those who succeed. Once retired, Peter Higgs²² notably talked about how the current academic system would have never allowed the type of research (and researcher) that led up to the discovery of the Higgs boson to occur. The second, and perhaps the bigger issue, is likely one we will never be able to fully grasp — how much of published research is in fact a lie. In 2023 alone, more than 10,000 articles were retracted, according to a Nature analysis²³. But how much more is out there still? How much of it, alongside other confounding factors such as methodological differences and individual variances, gives rise to unexpected negative results and reproducibility issues?

The failure of published findings to replicate²⁴ when independently tested is an issue all scientific disciplines have been tackling long before the advent of AI, and one whose full extent is hard to assess. One case-study area is the field of fMRI brain research, in which common neuroimaging software packages had inflated false-positive rates of up to 70%²⁵ under certain analysis choices, and where overall reproducibility rate has been estimated at around 63%²⁶. The field is actively responding with better methods and transparency, but the literature carries substantial "legacy" unreliability.

Many of the new heavily AI-generated articles are easy to spot, even by a non-expert. However, beyond the bluntly obvious AI scientific slop, the worry is in the small yet story-changing data manipulations that go unnoticed. You'd be surprised to know that such artifacts can resist months of peer-review at competitive journals. One influential article²⁷ that got caught, and only retracted 18 years after publishing, was one of the most cited articles in Alzheimer's disease of the last decades, published in the influential journal Nature. "The immediate, obvious damage is wasted NIH funding and wasted thinking in the field because people are using these results as a starting point for their own experiments." said Nobel prize winner Thomas Südhof (who, funnily enough, has several retracted papers himself²⁸).

There are other sources of constraints that make science slow. One of them is the increasing ethical and societal pressure to stop animal experimentation. Whether and how researchers should rely on animal use and to which extent is a long debate with polarising arguments. The fact is, much of today's research does rely on animal research, but the researchers who do it spend increasing amounts of their time writing and waiting for ethical approval²⁹ of their projects. Likewise, researchers working on human patients spend hefty amounts of their time dealing with long bureaucratic matters and timelines³⁰.

My argument is that perhaps the bottleneck preventing higher rates of important and revolutionary discoveries is not researchers, their lack of talent, creativity or skill. Rather, the academic system imposes growing constraints that limit not only scientific careers and the ones who choose them, but ultimately science. Importantly, none of these constraints are problems that AI solves. They are institutional, political, and economic problems that therefore require institutional, political, and economic solutions.

A counterargument is that even a small increase in productivity across the different tasks a researcher does should lead to increased compound productivity, which at surface level already seems to be true if we consider publication number as the gold-standard output³¹. This will likely continue to be true, to a certain extent, assuming that AI adoption across research will continue to expand, although some concerns have been raised about its benefits, as I will explain in the next section. Nevertheless, the idea of a "virtual biologist" as a recipe for moving science forward faster won't be the holy grail solution, because at a larger scale it will not be possible to speed up a system solely by optimizing the non-bottleneck steps.

New problems will arise — how will we deal with them?

Beyond the critical need of rethinking and changing the academic system, wide adoption of AI systems in research has surfaced a myriad of novel issues, and poses several other conceptual ones.

Before getting into technical limitations, there's a hypothetical risk worth naming: the entire enterprise of AI-assisted science currently depends on the financial viability of a small number of private companies. None of the frontier AI labs are currently profitable; rather, they operate on capital that anticipates future returns. In the unlikely but non-trivial scenario where these companies collapse³², restructure, or dramatically change pricing (already, the current economical model raises some doubts³³), scientists and science as a whole can lose access to tools they became dependent on. This is not an argument against using AI tools; rather, it's a warning about not letting them become a structural requirement in the absence of institutional alternatives for something so fundamental as scientific advancement.

Then, there are risks inherent to AI itself. We have all heard and experienced sycophancy in AI models. This trait has become perhaps more subtle and less occurring, yet it still poses significant threats to science and medicine. LLM agents are vulnerable to accepting and being persuaded by opinions that align with the framing of the conversation³⁴, and begin mirroring the user's viewpoints in extended conversations³⁵. This behavior has been shown to occur in specific sensitive contexts, such as in medical scenarios, where when presented with medically incorrect premises stated with confidence, LLMs show high rates of compliance³⁶. Additionally, medical LLMs have been shown to be particularly vulnerable to "data poisoning"³⁷, in which introduction of small quantities of misinformation into training data degrades model performance yet remains undetectable by standard benchmarks. Similar degradation can be achieved by targeting the model weights directly³⁸ and requires no access to the training data. While research in the context of scientific work is still limited, one can think of similar risks regarding the use of AI to establish novel scientific research proposals or to interpret acquired data. It also raises larger ethical questions, particularly surrounding accountability. Who will take the blame for large amounts of money being put into unviable research projects, or when life-threatening medical errors occur? The current push toward AI in science and medicine is moving faster than our ability to audit these systems; it is imperative that policymaking catches up.

Earlier I described fake and altered data as one of the bottlenecks already slowing science. It's worth considering how this issue also deeply impacts AI. Indeed, this is a compounding problem, because if we use bad literature as training data, LLMs will inherit these biases³⁹. Today's research is moving toward open data and shared protocols, but most of the literature still offers thin method sections and no access to raw data. I'm convinced this will be one of the real limits on deploying general-purpose AI in research; a model is only as trustworthy as the data it learned from, and that data is partially flawed. On the other hand, I think this could be an issue AI could greatly help researchers with. Consider specialized models built to read across different biology subfields, parsing methods, figures, protocols, and statistics rather than abstracts and conclusions, then using all of it to reconcile findings, weight them by how rigorously each study was actually described and reported, and in the end surface potential anomalies. Turning the literature's own scale against its noise would be a genuinely transformative use of these systems in the service of research.

Another worry is how AI can narrow the scope of science. LLMs are also known to distort the historical demographic and geographic concentration of knowledge, as highly cited researchers are more accurately and frequently represented⁴⁰ in LLM outputs. Additionally, a recent publication analysing over 41 million papers⁴¹ from 1980 to 2025 found that scientists who adopted AI published 3x more papers and received nearly 5x more citations, but their work covered 4.6% less scientific territory. The mechanism by which this occurs seems to be driven by a cycle of popular problems that motivate massive datasets, those datasets making AI tools appealing, and advances using AI tools attracting more scientists to the same problems. On top of that, this increase in productivity has been hypothesized to come at the cost of lower quality of the output³¹. Particularly, proficient use of language, which used to be a hallmark of research quality, is now inversely correlated. "This creates a risk for the scientific enterprise, as a deluge of superficially convincing but scientifically underwhelming research could saturate the literature. If this occurs, it will cause the community to waste valuable time separating genuine insights from a morass of unimportant and potentially misleading work" say the authors of this Science article. This way, AI collectively narrows research diversity and reduces cross-disciplinary collaboration, concentrating work in data-rich areas, and potentially decreasing its actual scientific value.

There are many other concerns over current and future AI systems and their use in research, but I'd like to touch on one last issue. As a researcher based in France, I have seen, in the past months, my use of AI tools become legally restricted to a securitized system developed by the national research agency (CNRS) and the French company Mistral⁴². This made me increasingly aware of the discussion around sovereign AI⁴³, and of how little discourse there is on how similar limitations, either from governments over safety concerns, or from international companies over their own countries' leader decisions⁴⁴, will impact scientific development. Once again, I argue for the importance of policy making on this matter, as it has the potential to contribute to widening research barriers, particularly for those in the Global South.

Moving beyond AlphaFold

Current AI models have impressive capacities across a variety of tasks such as researching, coding, writing and summarizing information. "AI naturally dovetails with science because many of the hardest scientific problems are, at their core, search and reasoning problems." says Pushmeet Kohli⁴⁵, VP of Research at DeepMind. AI systems are particularly useful as the volume and complexity of scientific data increase, making it possible to overcome scale and speed barriers. Additionally, AI can effectively bridge disciplines by integrating information across fields, potentially enabling new connections that would otherwise have been hard to obtain. "This is not to say that the current generation of AI systems is a panacea or the sudden solution to all the world's problems. They remain a tool in the hands of scientists; a powerful tool, to be certain, but a tool nonetheless." says Kohli⁴⁵.

Biology is complex, full of emergent patterns, hidden structures and exceptions. This is why many questions, regarding function and disease, remain unanswered despite years of research and funding invested. For decades, a task that was particularly complex was protein structure prediction. Proteins are responsible for almost all biological functions. Thus, to understand and fight diseases, we must know what they look like. Unlike determining the amino acid composition of a protein from its genetic code, predicting what its folded 3D shape will be is not an easy task. Each amino acid chain can fold in a large number of ways, and discovering the structure of a single protein used to take scientists sometimes years of work. For decades, scientists painstakingly collected protein structure data, one at a time, but mostly failed to accurately predict it from DNA sequences using just computation.

"As a problem for AI, protein folding has all the right ingredients: a vast search space of potential structures, a clear and measurable goal (atomic-level accuracy), and a rich supply of protein structure training data gathered through decades of painstaking experimental work." says Kohli⁴⁵. In 2018, DeepMind released its first AI model, AlphaFold, which essentially solved this decades-long challenge of protein structure prediction. Its database contains nearly every catalogued protein known to science, and is freely available to researchers worldwide. AlphaFold is a great example of how AI can expand and accelerate research, both public and private, and improve access to the tools of discovery, democratizing and accelerating science on a global scale.

In 2024, DeepMind's Demis Hassabis and John Jumper got one half of the Nobel Prize in Chemistry. Such recognition reflects the impact of this tool, but more deeply reflects a broader change underway, where AI-driven research will continue to make its presence felt across science and industry. DeepMind alone has a wide catalog of tools beyond AlphaFold, including GNoME (Graph Networks for Materials Exploration)⁴⁶ and AlphaEvolve (a coding agent for algorithms and math)⁴⁷. And they are not alone: the landscape of AI tools for science is now vast and fast moving, including recent developments from OpenAI⁴⁸ and Anthropic⁴⁹.

Several obvious challenges are still ahead before AI models can vastly replace scientists. First, as I have mentioned, a massive data reproducibility and method access issue across disciplines urgently needs to be fixed. Moving forward as it is will inevitably lead to many dead-ends, for humans (as it already does) and machines. Second, scientific publishing, which places most full-text papers behind journal subscriptions, will likely need to be rethought. Today, anything behind a paywall is largely inaccessible to LLMs beyond the abstract, meaning most of the scientific literature is out of reach. Without access to the underlying papers and data, LLMs often remain far from useful in many areas of science. And third, without large datasets, many areas of science are still very far from being able to achieve immense jumps such as protein prediction. Any model performing prediction needs comprehensive and extensive training data. In some areas, such data is so limited in comparison to the scale of the field and its questions that obtaining any AlphaFold-like breakthrough seems decades away. This is, in my opinion, the case in neuroscience, where the most recent advances include the reconstruction of a cubic millimeter of human brain tissue⁵⁰, and the reconstruction of 75k neurons in the mouse brain⁵¹. The latter took 9 years and over 1 million manual edits.

Re-thinking the value of (human) scientists

There is a last and more abstract challenge AI will need to overcome, where humans seem to greatly surpass it. AI systems perform well on problems that are already well-defined, and poorly on problems that aren't, which is precisely the zone where scientific discovery lives. It might sound contradictory, but at the very edge of knowledge, oftentimes science is abstract, and requires a set of skills that likely relies on intuition: the trained ability to recognize when something is interesting, when a question is worth asking despite the absence of a clear path to an answer.

"Once you're actually working at the cutting edge of the unknown, intuition has to play a part, because there isn't an instruction manual (...) nobody has written down the perfect rational answer for you yet." says Ratu Mataira⁵², CEO of OpenStar, a start-up developing the next generation of nuclear fusion technologies.

All the years you have spent living and interacting with the world have allowed you to become not only who you are, but also to develop a sense of 'intuitive physics' of reality. From a young age, by experimenting with the world we understand how objects and events relate to one another in our three dimensional, temporal reality. Much like that, scientists also build their own sense of their scientific fields by exploration and investigation. AI systems, however, reflect patterns of communication without the embodied context that gives human thought its texture. This is in great part an embodiment problem, and many companies are trying to tackle it across industries⁵³. Despite their sophistication, current multimodal AI systems rarely perform better than chance when estimating distance, orientation, and size. This is because language operates in one dimension, sequentially, while the physical world operates across multiple dimensions simultaneously governed by geometry, physics, and dynamics that text can describe but not encode. World Labs⁵⁴, founded by Professor Fei-Fei Li (the 'godmother of AI'), is trying to develop new generative AI models that can better perceive, reason and interact with the 3D world. Li says it will take at least a decade⁵⁵ to build truly spatially intelligent machines.

How science and research will change in this new era is something only time will settle. But in the midst of uncertainty about what the future of AI will bring, my belief is that science is actually one of the most human things we can do.

AI will not replace scientists. Instead, it will expand what they can imagine, understand, and achieve. After all, the telescope didn't make astronomers obsolete. It gave them the stars.

— From Pushmeet Kohli⁴⁵