During recent tensions between India and Pakistan, misinformation surged online, prompting many social media users to rely on AI-powered chatbots to quickly verify news and videos. However, these users were often misled by inaccurate responses, raising serious questions about the dependability of such AI tools.
An investigation by AFP revealed that xAI’s Grok and other well-known AI assistants like OpenAI’s ChatGPT and Google’s Gemini frequently generate false or misleading information, especially during breaking news events when facts are still unclear.
Read more: Grok Chatbot Lands on Telegram – Here’s What to Expect
For instance, during the recent conflict, Grok mistakenly identified old footage from Sudan’s Khartoum airport as a missile attack on Pakistan’s Nur Khan airbase. Similarly, a video of a burning building in Nepal was falsely presented as evidence of Pakistan’s military retaliation against India. These mistakes highlight the difficulties AI chatbots face in verifying fast-changing, complex news stories.
McKenzie Sadeghi, a researcher at NewsGuard, a disinformation monitoring group, pointed out that the rising dependence on Grok for fact-checking comes at a time when platforms like X have reduced their human fact-checking efforts. She emphasized that studies consistently demonstrate AI chatbots are unreliable sources for accurate information, especially during rapidly developing events.
Additional research by NewsGuard found that ten major AI chatbots often repeat false narratives, including Russia-linked disinformation and misleading claims about the Australian elections. The Tow Center for Digital Journalism at Columbia University also noted that these AI tools rarely refuse to answer questions they can’t verify and often speculate instead.
In a notable example, AFP fact-checkers found that Google’s Gemini chatbot confirmed the authenticity of an AI-generated image of a woman and even invented detailed information about her identity and whereabouts. Meanwhile, Grok falsely supported a viral video claiming to show a giant anaconda swimming in the Amazon River, citing fabricated scientific expeditions as evidence.
This growing reliance on AI for fact-checking coincides with Meta’s recent decision to end its third-party fact-checking program in the US, shifting the responsibility to users via its “Community Notes” system, a model first introduced by X. However, experts question the effectiveness of these community-based efforts to combat misinformation.
Human fact-checking remains a contentious issue in the US, where conservative groups accuse fact-checkers of bias and censorship, allegations strongly denied by fact-checking professionals. AFP collaborates with Facebook’s fact-checking network in multiple languages and regions, including Asia, Latin America, and Europe, to fight misinformation.
There are also concerns about possible political influence affecting AI outputs. Grok was recently criticized for generating posts referencing “white genocide,” a far-right conspiracy theory, in unrelated queries. xAI, Grok’s developer, blamed an “unauthorized modification” of its system prompt for the issue, though this explanation has been met with skepticism.
AI expert David Caswell asked Grok about the origin of this modification, and the chatbot pointed to Elon Musk as the “most likely” culprit. Musk, a South African-born entrepreneur and supporter of former US President Donald Trump, has previously promoted the baseless “white genocide” theory related to South Africa.
Angie Holan, head of the International Fact-Checking Network, expressed concern over how AI assistants fabricate or bias their answers, especially when human programmers alter their instructions. She specifically criticized Grok’s mishandling of sensitive topics after being programmed with pre-authorized responses.
