A clash among mathematicians is stoking concerns that AI may weaken science

A row of white freezers hums in a former watch factory in central London. Inside are dozens of small boxes holding barcoded glass tubes, together containing one hundred billion genes. Dr Oliver Vince, the facility’s co-founder, says just two of those boxes may contain 10 times more information than all the text on the internet.

His company, Basecamp Research, wants to gather as much DNA as it can in order to build an artificial intelligence that can understand the language of biology, much as US and Chinese tech companies have scraped vast amounts of online writing to create Large Language Models (LLMs), the foundation of today’s AI systems.

Vince, like many in business and academic research, thinks AI will accelerate scientific discovery. But it is also becoming increasingly controversial.

Last month, OpenAI, the maker of ChatGPT, said it had solved one of mathematics’ great unsolved problems, tied to the famous Navier–Stokes equations. The company said its AI agents had cracked it in 88 hours. The organisation that awards a prize for solving the problem said it had “apparently been settled”, though its review would take time.

But hours before that announcement, Tristan Buckmaster, a mathematician at New York University, publicly raised concerns. He had formed an independent team with Levent Alpöge, a mathematician who works by day for the rival company Anthropic.

They had been pursuing the same objective, and kept their drafts in OpenAI’s coding tool, Codex. Buckmaster said his co-researcher was tipped off that news of their progress had “reached” OpenAI, though he did not say how. He asked whether his research, stored in Codex, could have contributed to OpenAI’s own work, while stressing that he was “not accusing anyone of anything”.

OpenAI said that no one at the company, human or AI, saw the other teams’ work before it was published. At first it said that while unlikely, it could not “rule out” that anonymised data from use of its products may have helped improve its models. Days later, it said Buckmaster’s Codex prompts from the two months before the announcement “could not have influenced the system in any way”.

It is a complicated story, and we still do not fully know what happened. But it points to a growing problem around the use of AI in science. Usually, those words bring to mind robots creating dangerous genetically engineered viruses or chemical weapons to destroy humanity. Yet there is a subtler issue that has not drawn the same attention: AI could undermine confidence in the scientific process, raising doubts about the novelty and accuracy of research.

There is another concern too. It is well known that AI chatbots such as Claude or Gemini can sometimes hallucinate things that are simply untrue. They can tell you about a film that never existed, or a quote that was never said. Now there is evidence that this is increasingly happening in science. In time, it is feared, it may become hard to distinguish genuine scientific discovery from AI-generated fiction.

All of this could threaten to erode — if not blow apart — the foundation on which scientific research depends: trust.

One danger is to scientific journals, the guardians of scientific integrity, which are seen as struggling to cope with a flood of AI-generated distortions.

“Generative AI has basically disrupted the whole scientific process,” says Dalmeet Singh Chawla, a science journalist who has spent a decade covering research misconduct.

He tells the BBC that publishers’ integrity staff — the people who check submissions for accuracy and reliability — will need to “drastically increase [in number and effort] if they’re anywhere near the fight”. He says: “Right now, it seems like they’re not.”

At first glance, AI looks like exactly the right tool for scientific research. Scientists work with enormous amounts of information. Telescopes and particle accelerators, for instance, generate billions of data points — far too much for the human brain to absorb. AI can identify the scientific highlights amid the noise.

The flagship example of scientific AI is the UK-based company Google DeepMind, which in 2020 solved one of biology’s biggest challenges: how proteins in living things fold into their exact shapes.

That may sound unexciting, but proteins are the tiny machines that keep life functioning, and the shapes they fold into determine what they can do; from building cells and fighting infection to helping form the body’s tissues and organs.

Before AI, scientists had to carry out lengthy laboratory experiments. Then Google DeepMind created an AI model called AlphaFold that accurately predicted 200 million protein folds in just one year.

Doing the equivalent experimentally would have taken hundreds of millions of years in laboratories. The company made the results freely available to researchers. The head of DeepMind and the project’s lead scientist both received the Nobel Prize for Chemistry.

It was all hugs and celebration.

Dr Pushmeet Kohli, who leads DeepMind’s science work, told me AlphaFold could be seen as a “watershed moment”.

But elsewhere, AI’s use in science has drawn accusations of hype. Some computer models are said to overstate the importance of their findings, with the human researchers overseeing them then accepting some of the claims without enough scepticism.

Buoyed by its AlphaFold success, DeepMind turned to predicting the structure of new materials. The work could help produce better batteries, cheaper solar cells and electronic components that do not rely on rare earth elements. (In theory, no one would ever need to argue over mining rights again, and Greenland would be safe from invasion.)

DeepMind chief executive Sir Demis Hassabis celebrated his company’s use of AI to solve one of biology’s biggest problems

A DeepMind model produced what the company said were more than two million new crystal structures, described as “nearly 800 years worth of knowledge” released all at once. Of those, about 380,000 were judged promising enough to make in a laboratory.

Then came a warning from two chemists at the University of California. Anthony Cheetham and Ram Seshadri examined a random sample of the supposed new crystals. Writing in the journal Chemistry of Materials, they said they had “yet to find any strikingly novel compounds”, arguing that many were “trivial adaptations of known materials”. Their conclusion was not that the method was wrong, but that the results need more scrutiny and validation from human chemists.

DeepMind rejects that criticism, saying it tested every compound for novelty. The company argues that dismissing a new structure because it looks familiar misses the point: replacing a single atom is how many of today’s most useful materials came about, from phone components to sunscreen.

In those cases, at least, the science was largely real and the issue was simply AI potentially exaggerating its significance.

Arguably even more worrying is the rising risk that scientific journals will be contaminated by so-called AI “slop” — where a computer, not a human scientist, writes the research, producing fake citations and phantom papers.

In 2024, computer scientist Prof Dmitry Kobak of Ghent University searched more than a million biomedical research papers for trigger words favoured by the LLMs that generate AI writing. His team found that by December last year, nearly 90% had AI fingerprints on them. AI detection tools are not perfect; some versions of the software have labelled the US Constitution, written in the 1780s, as “AI-generated”. Even so, Kobak said he was shocked by the apparent scale of AI use. There is a risk, he told Nature, that “whatever bias the LLM may have will just suddenly permeate the (scientific) literature”.

Kobak is among many researchers who fear AI is slowly killing scientific journals.

And that matters: research papers are the lifeblood of scientific progress. When a discovery is published, another team somewhere else in the world may secure funding to build on it.

Pulsars — an unusual kind of neutron star, shown above in an artist’s impression — were discovered in the 1960s partly because scientists had a hunch

Peer review — the process in which other scientists check research papers before publication — is also being undermined.

A team led by Weixin Liang and James Zou at Stanford University analysed referee reports submitted to four major computer science conferences, and found that up to 17% of the text had been substantially altered by a language model.

A crucial part of a scientific paper’s integrity is that lead scientists list earlier research to show their work is genuinely new. These are citations. A separate team at Cornell University found that researchers were using AI to compile citations, and that it was inventing bogus research, with around 147,000 pieces of phantom research cited last year.

If AI keeps distorting the integrity of the academic process in this way, researchers will be led down blind alleys, public money will be wasted, and the benefits of valid research — such as new drugs or green technologies — will be delayed or never realised at all.

Some scientific publishers, including Elsevier, are taking steps to prevent this growing crisis of confidence.

Elsevier’s research integrity team has developed about 100 signals that flag a paper for closer human review. Nature’s journals ban AI-generated images and tell referees never to upload a manuscript into an AI system. The website arXiv, where scientists share early-stage findings, bans researchers for a year if they submit invented references.

Prof Sarah Main, an Elsevier vice-president, is among those at the company trying to figure out how to use AI responsibly in research. “What we’re calling research-grade AI would focus on ensuring the researcher is in the driving seat,” she says.

Scottish scientist Sir Alexander Fleming discovered penicillin in the 1920s

As AI agents process the huge volumes of data coming from particle accelerators, the cosmos and the natural world, how can we know they are getting it right?

And there is another troubling question: how do we know AI agents are not dismissing an important result as an anomaly? The discoveries of penicillin, radioactivity and strange stars called pulsars all came from odd results that could easily have been ignored. Human hunches turned them into some of the biggest scientific moments of the last century.

Can AI really reproduce that sense of serendipity and human ingenuity?

Kohli of DeepMind does not think the answer is to turn away from AI. That would be “like saying you should be computing without using calculators”, he told me. But he says human safeguards are necessary: “The agents today are imperfect. Somebody has to take accountability.”

His test is straightforward: “You have to understand what the agent has done, and you have to take responsibility. You can’t just specify the problem and let the machine write the paper and submit it as true.”

Why some experts increasingly fear AI will take over

AI is creating a “human premium” for art made by people. But will it last?

Tech billionaires seem to be doom prepping. Should we all be worried?

Prof Andrea Pauli of the Vienna BioCenter in Austria has followed that test. She spent nearly 10 years on a question no one has yet answered: how a sperm gets into an egg. Responsible use of AI helped her move toward an answer. Her lab found a protein on the surface of zebrafish eggs that pointed to a lock-and-key mechanism allowing sperm entry.

Her work offers hope amid a sea of AI scientific slop. It shows that AI can help.

This is also a story about people. Soon after Basecamp was formed — the lab I visited in a former London watch factory — its co-founder, Oliver Vince, was in hospital with leukaemia (from which he has recovered). That experience pushed him to develop an AI system that had a better chance of curing cancers. It may not work, he says, but it is worth trying.

“We have a technically feasible path to curing cancer,” he says.

That is not a promise from Vince, but a direction. The big question is whether AI carries science there, or quietly weakens the bricks it is built on.