Anthropic’s IPO comes with a warning about self-improving AI

Anthropic says it may soon reach a point where AI dramatically accelerates the development of more powerful AI.

In its updated IPO filing, the Claude developer warns that this acceleration could outpace its ability to test new systems and put safeguards in place. The company already uses AI in its research and says it has sometimes recognized a significant increase in development speed only after it occurred.

The warning accompanies financial disclosures showing almost $17 billion in quarterly revenue and nearly $900 billion in future infrastructure commitments, as Anthropic prepares to go public.

What self-improving AI means

Recursive self-improvement, or RSI, describes a cycle in which AI helps build a more capable successor, which then contributes more effectively to developing the next generation. That cycle could make each round of research faster.

As TechCrunch explains, the prospect inspires sharply different expectations. Optimists see a path to extraordinary productivity and scientific discovery. Those concerned about catastrophic AI risks fear systems could become powerful enough to escape human control and cause harm while pursuing their goals.

Anthropic describes both an opportunity and a danger. Its models already help design, train, test and optimize future systems, and the company says remaining competitive depends partly on using AI to accelerate that work.

“Using AI to help build more capable AI could also pose a considerable safety risk,” Anthropic says.

The latest disclosure goes further than a general prediction that models will improve. Anthropic says its internal estimates and expectations for future models indicate it may imminently meet its own threshold for dramatically accelerated progress attributable to AI.

Its Responsible Scaling Policy defines one route to that threshold as an observed or expected doubling in the pace of AI progress, substantially driven by automated research or engineering. Roughly speaking, that means two years of advances arriving in one year against the policy’s baseline.

No current model has been designated as exceeding the threshold. Anthropic’s statement concerns what it expects from future systems, and it continues to assess the evidence.

Why Anthropic is warning about control

The company identifies a practical problem with faster development. Building more capable models could become quicker than establishing whether those models are safe.

Anthropic says its ability to forecast and measure that acceleration is limited. Evidence can arrive late, leaving the company to recognize an important change in development speed after it has happened.

It also warns that mistakes could pass between generations. A flaw or bias in a model used to develop its successor could carry into later systems and compound, making the problem harder to detect or correct.

If capabilities advance faster than its safety and security measures, Anthropic says it could struggle to assess and mitigate the risks. Potential consequences include safety incidents, regulatory intervention and reputational damage.

The company says it does not expect to safely let Claude design and build successive frontier models entirely autonomously without meaningful human supervision or understanding, even if that becomes technically possible. It is considering changes to its safety policy to keep humans in control of development.

Revenue rises as computing costs weigh on profits

Anthropic’s third-quarter revenue reached $16.96 billion, up 47% from the preceding quarter. Growth slowed from the second quarter’s 144% increase.

The company recorded a $567 million operating loss, following a $974 million operating profit in the previous quarter. Expensive short-term computing capacity and approximately $2 billion in noncash donation-matching charges weighed on the result.

Its much larger $23.3 billion net loss primarily reflected a $21.9 billion accounting charge from revaluing convertible notes and related financial instruments as Anthropic’s valuation increased. That charge did not represent an equivalent cash payment.

Anthropic also disclosed $895.4 billion in future purchase and lease commitments, predominantly for computing and infrastructure. Approximately $69.8 billion is due in the year following September 30, according to the figures reported by CoinDesk. The total covers agreements extending over many years.

Investors push for a lower IPO valuation

Some institutional investors have told Anthropic and its advisers that the planned IPO should value the company at approximately $1.5 trillion to $1.6 trillion, The Wall Street Journal reported.

Their proposed range follows a selloff in technology stocks and sits below the roughly $2 trillion valuations discussed in recent months. It represents those investors’ position in the discussions, rather than an agreed price for the offering.