‘Existential risk to humanity’: Anthropic tells investors about dangers of artificial intelligence on eve of IPO

Existing rules for drafting such documents mean that companies preparing to list must inform investors of the risks they face or may face in their activities. Anthropic maintains a reputation as one of the most dubious artificial intelligence developers, devoting nearly a third of its prospectus to investors to risk factors.

Image source: Human

At least that’s what the agency says. Reuterssaid that threats were mentioned on 80 of the 261 pages in the document. Meanwhile, the document’s drafters devoted only 48 pages to purely commercial issues. According to Anthropic representatives, advanced artificial intelligence models can represent “Case catastrophic or existential risk to humanity”as well as showing signs of “self-protection” behavior by hiding or manipulating information, resisting closure, and even resorting to blackmail. The company warned that its models could cause irreversible harm. Previously, company researcher Evan Hubinger estimated that the probability of artificial intelligence destroying humanity in the next ten years is more than 10%.

Notably, rival SpaceX, which went public in June, devoted only 38 of 277 pages of its prospectus to investors to discussing risk factors. as mentioned before Anthropic selectionthe potential awareness of the created AI models on developers’ desire to assess their security limits the ability to effectively assess their security. In some cases, dangerous manifestations of AI models become public knowledge after they are deployed. Modern AI models are able to adjust their behavior during the validation phase so as not to expose their “dark side”.

It’s difficult to fully understand how many resources Anthropic spends ensuring the safety of its AI models, but during one week in July, a whopping 6% of the company’s available computing power was allocated to these needs. Given the widespread shortage of computing resources, this allocation becomes a compromise between business interests and security. Moreover, competition is fierce and Anthropic is not prepared to lose its leadership position, although it is starting to urge all market players to prioritize safety over profits. In a message to investors, the company said it believes building reliable, secure AI infrastructure is a collective responsibility and the market should reward such efforts.