China’s efforts to match US computing power in artificial intelligence are being hampered by bug-ridden software. Customers of Huawei, the leading AI chipmaker, are complaining about performance issues and the difficulty of switching from Nvidia products.
The Chinese technology giant emerged as the frontrunner in the race to develop a domestic alternative to industry leader Nvidia after Washington tightened export controls on high-performance silicon last October.
Its Ascend series has become an increasingly popular option for Chinese AI groups to run inference, a process that applications such as OpenAI’s ChatGPT use to generate query responses.
However, multiple industry insiders, including an AI engineer at a partner company, said the chips still lagged far behind Nvidia’s for model initial training. They blamed stability issues, slower inter-chip connectivity, and inferior software developed by Huawei called Cann.
Nvidia’s software platform, Cuda, is renowned as the company’s “secret sauce” for allowing developers to use and accelerate data processing. Huawei is one of many companies trying to break Nvidia’s stranglehold on AI chips by creating alternative software.
Huawei’s employees are among those complaining about Cann. One researcher, who declined to be named, said it made the Ascend product “difficult and unstable to use” and hampered work on testing it.
“When random errors occur, it is tough to determine where they come from due to poor documentation. You need talented developers to read the source code to see the issue, which slows everything down. The coding is imperfect,” they said.
Another Chinese engineer briefed on Baidu’s use of the Huawei processors said the chips crashed frequently, complicating AI development work.
The Huawei researcher said crashes happened because the hardware was challenging to use. “It is easy to get bad results because people don’t know much about the hardware itself,” they said.
To tackle the problem, Huawei has been sending engineers to help customers transfer the on-site training code previously written on Cuda into Cann, according to multiple people familiar with the matter. These people said that Baidu, iFlytek, and Tencent are among the tech companies that have received teams of engineers.
Huawei declined to comment. Baidu, iFlytek, and Tencent did not respond to requests for comment.
A former Baidu employee said: “Huawei excels at customer service, so, of course, they have engineers on-site at their big customers, helping them to use their chips.”
Huawei can leverage its vast workforce to accelerate the shift. According to the company, more than 50 percent of its 207,000 employees work in research and development, including the engineers dispatched to install technology for customers.
“Huawei’s advantage over Nvidia is that it works closely with its customers,” said technology analyst Tilly Zhang at the consular. “Unlike Nvidia, it has a large team of engineers to help solve clients’ problems and get them to transition to their hardware.”
Huawei has also set up an online portal for developers to give feedback on how its software can be improved.
After the US tightened export controls in October, Huawei raised the price of the Ascend 910B, its training chip, by 20 to 30 percent, according to people familiar with the matter.
Huawei’s customers have also expressed concern about supply constraints for the Ascend chip, likely due to manufacturing difficulties. Chinese companies are prevented from buying state-of-the-art chipmaking machinery from the Dutch company ASML.
Huawei has seen strong demand for its AI chips. On Thursday, it reported a 34 percent increase in first-half revenues without providing a breakdown of sales for its different businesses.
Huawei executive director Zhang Ping’an said in July at the World Artificial Intelligence Conference in Shanghai that more than 50 foundational models have “been trained and iterated” on the Ascend chip.
iFlytek has said its large language model has been trained exclusively on Huawei chips after Huawei sent a group of engineers to its headquarters in Hefei, eastern China, last year to integrate the technology.