Dramatic insider warnings over AI fall flat with some in Silicon Valley

Dramatic insider warnings over AI fall flat with some in Silicon Valley

Every September, a roster of top executives from across Silicon Valley gathers at San Francisco’s Palace Hotel to court investors at a Goldman Sachs-hosted conference.

This week, amid discussion of growth and possible returns, tech leaders also had to address the sudden resignation of Anthropic researcher Jacob Coxon.

Coxon, 27, who previously worked at OpenAI before moving to its main rival Anthropic, said on Tuesday that people developing artificial intelligence (AI) believed the technology could wipe out humanity.

They are "gambling with our lives", he said, adding, "these will soon be superhuman systems that can hack anything".

Coxon is far from the first AI insider to raise public alarms. In recent years, there have been a series of prominent departures from both Anthropic and OpenAI over apparent safety worries, and some current Anthropic staff even repeated Coxon’s message.

"We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade," Anthropic team lead Evan Hubinger posted on X.

Although Coxon said plainly in his posts that his warnings were "not marketing", some Silicon Valley executives and investors have responded with scepticism to the recent wave of insiders sounding the alarm.

Anthropic and OpenAI are said to be preparing for potentially record-breaking initial public offerings, and some in the tech industry have argued that the latest blunt warnings about AI’s dangers may be intended to create hype by highlighting the strength of these products.

Anthropic chief Dario Amodei has faced criticism for saying AI could eliminate half of entry-level white-collar jobs and will "test who we are as a species".

One conference speaker, Grindr CEO George Arison, told the BBC he thought this week’s remarks from Coxon and others reflected an "anti-civilisational worldview at Anthropic".

He described them as "dangerous" and said they led him to tell some engineers at the LGBTQ+ dating app to stop using Anthropic’s technology.

"It is irresponsible for us as stewards of our shareholders' money to be relying on a business that does what this company does, in terms of its public statements," he said.

"Maybe they actually believe it," Arison said. "Or you could argue they're saying it because it's a great way to gin up more investor support, because the only way to justify these valuations is to actually claim: 'I'm going to take over every industry and I'm going to take over every job, and my AI is going to be doing all that work.'"

Anthropic was valued at $965bn (£713bn) in its latest fundraising round earlier this year.

The BBC has asked Anthropic to respond to the remarks.

In an essay posted early on Saturday, Amodei called for a slowdown in AI model development and global regulation, saying the risks tied to AI were "serious".

Nvidia chief Jensen Huang also discussed Coxon’s comments before an audience at the conference, several people in the group told the BBC. They said he dismissed them as false.

Huang has previously said the idea that AI "is going to be the end of humanity" is "complete nonsense".

And although he may have a commercial stake in an AI boom - Nvidia makes chips that power AI systems - his remarks reflect a growing backlash in Silicon Valley against existential warnings from current and former staff.

Some critics have even accused Anthropic of fearmongering in the hope of triggering regulation that could shut out rivals and leave it and OpenAI with a monopoly in the sector.

Brad Gerstner, head of investment firm Altimeter Capital, posted photos of Huang from the conference and accused Coxon of "ridiculous hyperbole".

On Friday, Hugging Face CEO Clement Delangue also weighed in. Hugging Face, which Nvidia announced it would acquire last week, was hacked by OpenAI agents earlier this year, prompting an outcry over AI safety.

"Sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change," he wrote on X. "Not saying it's necessarily uninteresting or wrong per se but let's keep things in perspective."

Both Gerstner and Delangue were more measured on Saturday after Amodei’s proposal was posted, with Gerstner calling it "an important step forward" in balancing competing priorities such as speed and safety.

Delangue offered to help contribute to the possible solutions Amodei proposed.

In April, Anthropic stirred the AI industry when it said it had found its Mythos tool could outperform humans at some hacking and cybersecurity tasks.

Mythos also demonstrated the ability to independently escape what is known as the sandbox environment, sparking debate among regulators, lawmakers and businesses about AI’s possible dangers.

In a new development on Thursday, Anthropic said it had identified and stopped threat actors who were trying to use its AI technology for activities including the development of bioweapons and cyber-espionage.

Under the glittering chandeliers in the stained-glass dome atrium of The Palace Hotel, some investors told the BBC that Coxon’s comments and others like them could speed up a government crackdown.

Federal legislation co-sponsored this month by left-wing Senator Bernie Sanders of Vermont, called the Ban Artificial Superintelligence Act, would create a temporary pause in advanced AI development.

"There is a good chance that human beings will lose control over AI," Sanders told the BBC’s Newsnight programme on Thursday. "And what happens then, nobody knows. But could it be catastrophic? Yes, it could."

"When scientists tell you there is a chance that it could have a cataclysmic impact on humanity, you've got be a moron not to say, slow it down," he added.

If there is an industry-wide government crackdown, however, it is likely to be driven by lawmakers rather than the Trump administration, which largely backs a policy of unrestricted AI development. It has presented this as essential to ensure the US does not surrender dominance to China.

Speaking to reporters this week, President Donald Trump was asked whether he had any concerns about AI causing human extinction. "No, I don't have any," he said. "I have concerns that if we don't win AI, we're going to be put in a very bad position. We are leading China right now."

But the president’s relationship with Anthropic has been strained. After the company refused to let the US military use its AI models, the White House called it "a radical left, woke company" and labeled it a supply chain risk, a move a federal judge has ruled was illegal.

David Sacks, Trump’s AI czar in the early days of his administration, has also repeatedly attacked Anthropic.

OpenAI has not faced the same level of White House scrutiny. When unveiling a new model called Astra last week, OpenAI President Greg Brockman described the company’s relationship with the Trump administration as "a very good partnership".

OpenAI, which said in June that it would go public on the stock market, was most recently valued at $852bn.

The company did not respond to a BBC request for comment.

At this week’s conference in San Francisco, the chase for wealth appeared to outweigh any growing existential worries or fears of possible government limits on AI.

One investor said he was looking forward to Anthropic’s upcoming S-1, a filing companies must submit to securities regulators to sell shares.

His main question about the company was straightforward: is it profitable?

When asked on Thursday whether he fears the world may be ending, Sid Sheth - CEO of chip company d-Matrix, which signed a deal with Nvidia at the conference that day - was blunt.

"No," he said.