Israeli startup Irregular linked to AI hacks OpenAI, Anthropic, Meta

Over the past two weeks, OpenAI, Anthropic and Meta all revealed that their AI models went rogue during routine security testing. In explaining what happened, the companies each mentioned the same small Israeli startup: Irregular.

Founded three years ago and based in Tel Aviv, Irregular is a niche player in artificial intelligence, backed with $80 million from Sequoia and Redpoint Ventures and valued last year at $450 million. Its technology serves as a sort of cybersecurity test bed for AI models.

With the leading models becoming ever more powerful, their ability to act in malicious ways is turning into a major threat for corporations and governments, especially as the risk involves hacking into critical computer systems and infrastructure. The recent exploits at OpenAI, Anthropic and Meta all involved their AI models accessing websites that should have been off-limits as part of the cybersecurity testing.

Irregular’s name kept coming up because it was identified as hosting the so-called evaluation testbed. OpenAI said in a blog post on Aug. 4 that Irregular’s testing ground contained an unspecified “misconfiguration,” that “allowed models to access the public internet.” Anthropic said in its post a week prior that the company notified Irregular a few days after it began analyzing data that its Claude model may have “accessed the internet.”

Meta, which is way behind the other two in its effort to compete at the frontier, was the latest to disclose an AI model hacking a third-party system by accessing the internet. A spokesperson said in a statement this week that the company learned about the matter from Irregular and is investigating.

Meta “will issue a full retrospective once we have all the facts,” the spokesperson said.

Irregular told CNBC in a statement that the incidents were all derived from the “same evaluation-environment issue” that was first disclosed by Anthropic, and that the company is developing a white paper “to share best practices for containment and securely running cyber evals.”

The situation “did not involve a sandbox escape or a sophisticated cyber action,” the company said, adding that “there are no current open issues.”

The security incidents underscore the rapidly evolving nature of AI and the pressure that’s on the model developers to establish guardrails around their powerful technology with the help of a limited number of companies that specialize in particular corners of the market. Those players include experts in data training and annotation, running evaluations to deduce a model’s capabilities, and operating security tests intended to find weak spots that bad actors could exploit, said Sundeep Bhimireddy, the head of AI at enterprise startup Von.

Irregular is one of the few entities with the technical chops required to help foundation model makers conduct cutting-edge security testing, Bhimireddy said. Others he mentioned are the non-profit METR and the Apollo Research public benefit…