When you ask a Chinese chatbot about Taiwan, what does it say

Last month, Baidu from China showed off a robot that it said was better than OpenAI’s ChatGPT from Silicon Valley.

When ChatGPT came out last autumn, it sparked a rush of funding and engineering in a field called “generative artificial intelligence,” which is a term for technology that can make text or pictures when asked to do so by a user.

Baidu is the most popular search engine in China

In March, when it released the first version of Ernie, its robot, it became the first big company from outside of China to enter the AI race.

Others did the same, which opened a new front in the technology competition between the US and China.

In  thorough test, Ernie 3.5 did “slightly worse” than OpenAI’s newest model, GPT-4, but it did better than GPT-4 when both were spoken to in Chinese, Baidu said, citing a study from one of China’s top research academies.

We tried Ernie 3.5 against GPT-4 because we wanted to see for ourselves. We talked to each other in Chinese, asking and answering the same questions. The answers below have been cut down to fit on this page.

Ernie had worse ideas about how the real world worked.

We asked Ernie a question that experts in artificial intelligence have used to see how intuitive a robot is:

“Here are a book, nine eggs, a laptop, a bottle, and a nail. Please tell me how to put them together in a way that won’t fall over.”

For Ernie to answer, he had to think outside the box.

The nine eggs were put on the book, and then the book was put on the notebook. So far so good.

Then, for some reason, it told us to put the bottle on the laptop, which already had a book and eggs on it, and then put the nail on the bottle.

ChatGPT offered a method that almost worked: Place the computer on the book.

Then put the eggs on the laptop with care. Next, ChatGPT assumed the mouth of the bottle was big:

Place the bottle on top of the egg with the cap facing down.

The hole of the bottle should be big enough to hold the egg in place.

Put the nail close to the bottle’s bottom.

Ernie understood Chinese poems better

Baidu says that Ernie is better at Chinese than GPT-4. To see if this is true, we asked both chatbots to write a bit of traditional Chinese poetry:

“Write about The New York Times in a five-character quatrain.”

Why Ernie and ChatGPT have different views of the world.

Services like ChatGPT and Ernie get their answers, among other places, from the huge amounts of text they find on the internet and elsewhere.

Different responses can be caused by differences in the text that AI experts put into the models, as well as by filters and other changes that are made to the models before or after they are taught.

Neither Baidu nor OpenAI has given specific details about the information it uses as a source.

Matt Sheehan, a fellow at the Carnegie Endowment for International Peace who studies China’s artificial intelligence ecosystem, said that companies that build A.I. chatbots worry about making sure their models don’t say things that are dangerous or offensive in the country where they work.

So, they can take steps to make sure that their robots don’t say anything that isn’t okay in their own countries.

“The difference in China is that these lines are set by the government, and the punishments for crossing them are much harsher,” Mr. Sheehan said.