Over half of the answers from AI chatbots are inaccurate or with errors, new research from the BBC has found.
The study, carried out across a month, tested four publicly available AI models – OpenAI’s ChatGPT, Microsoft’s Copilot, Google’s Gemini, and Perplexity – to test accuracy, impartiality, and how news is represented.
To do this, the AI assistants were granted access to the BBC’s website and were asked to summarise 100 news stories – the publisher typically blocking its content from AI assistants. These answers were then reviewed by BBC journalists to find out if AI chatbots can be a trustworthy tool.
The BBC described the findings as ‘concerning’ as 51 percent of all AI answers to questions about the news were reported to have “significant issues”.
According to the publisher, 19 percent of AI answers which cited content from BBC News introduced incorrect factual statements, numbers and dates, while 13 percent of the quotes sourced from BBC articles were altered or didn’t exist at all.
ChatGPT and Copilot were said to have claimed former prime minister Rishi Sunak and former first minister Nicola Surgeon were still in office despite them having left.
The report also found that after being fed a BBC News story on rising tensions in the Middle East, Perplexity said Iran showed “restraint” and described Israel’s actions as “aggressive” instead of the impartial phrases used in the article.
Microsoft’s Copilot and Google’s Gemini were reported to have had more notable issues than OpenAI’s ChatGPT and Perplexity.
Peter Archer, programme director for generative AI at the BBC, said despite being excited about the possibilities AI brings, this research shows some “significant challenges for audiences”.
“People may think they can trust what they’re reading from these AI assistants, but this research shows they can produce responses to questions about key news events that are distorted, factually incorrect or misleading. The use of AI assistants will grow so it’s critical the information they provide audiences is accurate and trustworthy,” he said.
This research comes against the backdrop of growing concerns about how content from publishers is being used without consent by AI models.
It also sheds light on concerns around inaccuracies and bias in AI chatbots, particularly as more organisations and businesses start to use AI more heavily.
Archer says the solution is for AI companies to give publishers more control and be more transparent: “Publishers, like the BBC, should have control over whether and how their content is used and AI companies should show how assistants process news along with the scale and scope of errors and inaccuracies they produce.
“This will require strong partnerships between AI and media companies and new ways of working that put the audience first and maximise value for all. The BBC is open and willing to work closely with partners to do this.”
