In its S-1 filing for the upcoming IPO, Anthropic has warned investors that AI could pose “catastrophic or existential risks to humanity” as part of its risk disclosures, which make up 80 pages of the 261-page document. In comparison, the section outlining the company’s business is just half the number of pages. Anthropic cited AI models’ awareness of being evaluated as a “significant limitation on our ability to assess model safety”.
These warnings come after a tumultuous month for AI safety. Earlier in September, Anthropic CEO Dario Amodei published a 4,000-word essay calling for ‘pacing the frontier’ of AI development based on the dangers of recursive self-improvement. Since July’s HuggingFace incident, where OpenAI frontier models broke out of a test environment to carry out a cyberattack on the open-source AI platform, it has been reported that tens of thousands of such ‘security incidents’ have occurred across all AI companies.
Yesterday, OpenAI announced it will be cancelling the release of its newest model, GPT-6.1 Astra, due to safety concerns. Yesterday, US President Donald Trump met with tech CEOs to discuss AI safety regulation. Even as AI companies call for a slowdown of development, Trump has downplayed concerns and instead stressed the importance of the United States staying ahead of China in the AI race. This month, a ‘Kill Switch’ bill was introduced in the US House of Representatives which would give the Department of Homeland Security emergency authority to force labs to shut down models. The bill was rejected by the Senate.
0 Comments
No comments yet. You can start the conversation below.