AI’s quiet safety gatekeepers are stepping into the spotlight

Two months ago, independent evaluators occupied a relatively sleepy corner of the multitrillion-dollar artificial intelligence industry. Now they’re being asked to come to its rescue. 

While Anthropic and OpenAI are at the heart of a fierce debate over whether they can safeguard their advanced models and grow their businesses simultaneously, they’re seeking support from a handful of small third-party groups like Model Evaluation and Threat Research (METR), Apollo Research, and Transluce.

The evaluators, which mostly operate as nonprofits, are still finding their footing in an industry where capital is flowing at historic levels and new models are rolling out faster than ever. Their primary role has been to assess AI model capabilities and risks and to flag instances where the technology behaves badly.

Without a federal push for regulation, evaluators have taken on outsized importance. Anthropic CEO Dario Amodei pledged to embed independent evaluators in his company last month – a move that OpenAI CEO Sam Altman quickly endorsed. President Donald Trump supported the idea, as did most of the largest U.S. tech companies. But left unanswered are questions about how those third parties should be funded, what level of access they will have and what the reporting structure will ultimately look like.

“To a degree, the problem, as always, is money,” Suresh Venkatasubramanian, a computer science professor at Brown University, told CNBC in an interview. “Who is paying for these companies to do their work? How are they going to support them? You need an ecosystem, you need a viable business model for this.”

Right now, Anthropic, OpenAI and the infrastructure partners that are profiting from the AI boom are writing the rules. Critics say that’s like asking the biggest banks to protect us from a financial crisis or allowing pharmaceutical companies to put drugs on the market without regulatory clearance.

Trump recently lauded AI executives for their “tremendous self-policing,” and signaled that he intends to leave companies to their own devices, unwilling to impede the growth of the industry that’s driving the economy and stock market. The president encouraged AI companies to “partner with an independent external auditor or evaluator” as part of a voluntary accord he presented in late September.

Amodei kicked off the conversation in his viral essay last month, when he called for a “slower pace” in advanced model development after researchers left his company and voiced concerns about the existential threats the technology poses.

As AI labs move to put evaluators in place, friction is already emerging.

OpenAI fired three employees last week for “violating our policies on accessing and handling…