Is China’s AI progress really down to ‘distillation’?
This report is from this week’s The Tech Download newsletter. Like what you see? You can subscribe here.
In the battle for AI supremacy between the U.S. and China, “distillation” has become a hot topic.
The term broadly refers to the training of an AI model using the output or answers from another more advanced model. In some instances the process has been deemed controversial because a less advanced lab could use the product of a model that has taken millions or billions of dollars to create to design a competitive product.
As China has continued to pump out highly advanced models which compete in some areas with their American rivals, U.S. companies and Washington have accused Chinese labs of using distillation to gain ground, denouncing their efforts as theft.
But Aidan Gomez, the CEO of Cohere, says not all of China’s progress is down to this technique.
Gomez, who spoke to me on the latest episode of The Tech Download, is an authoritative figure in AI. He was a co-author of a 2017 research paper called “Attention Is All You Need,” which underpins many of the AI services we use today such as ChatGPT and Claude.
Gomez described Chinese AI models as “world class,” adding that the lead U.S. labs have is “evaporating very quickly.”
“There was like a lot of talk about the Chinese are just copying, they’re just distilling, they’re cheating. A lot of these tropes were applied, and that was definitely true to an extent,” Gomez told me.
“However, they have developed an exceptional capability independent of distillation, and it’s proven by the fact that the latest models that are coming out, actually on some benchmarks, on some axis capabilities, beat the best American models. And you can’t copy or distill to better. You can close the gap and reduce the gap by copying, but you can’t outperform.”
This runs slightly counter to what has been said by U.S. AI labs and Washington.
The rise of Chinese AI
Before I get to that however, I think it’s important to add some context about the development of the Chinese tech industry which I witnessed first hand while reporting from the country for three years.
The talent pool is huge, the pace of innovation is rapid and when an industry is backed by the central government in the way that it has been, it certainly adds a strong tailwind. China has often been labelled with the “copycat” moniker but in reality, that has faded as real innovation has come out of the tech scene there over the last few years from electric cars, to robotics and even advancements on semiconductors.
Gomez said there is some distillation taking place but that can’t explain all of China’s progress on AI models. That view runs contrary to some of the warnings being sounded by U.S. AI labs.
Anthropic, the creator of Claude, has on numerous occasions said Chinese AI labs have used distillation, saying it amounted to theft. This month, Anthropic’s head of threat intelligence, Jacob Klein, told CNBC that “there’s an entire illicit ecosystem to try to gain…
Read More: Is China’s AI progress really down to ‘distillation’?