About the Authors
王鸿鹭
WANG Honglu
PhD, Adjunct Research Fellow of the Capital Market Research Center, Course Manager of the Executive Education (EE) Center
CHEN Xin
Professor of Dishui Lake Advanced Finance Institute, Shanghai University of Finance and Economics (SUFE-DAFI), Director of Capital Market Research Center of SUFE-DAFI
This article was published in Wenhui Daily.
SUFE-DAFI
With the rapid development and widespread application of generative artificial intelligence (AI), AI hallucinations have emerged as a critical barrier to the healthy development of the industry. AI hallucinations refer to contents generated by AI systems that appear plausible and credible on the surface but are actually inconsistent with objective facts or entirely fabricated, such as non-existent academic literature or fictional expert opinions. Research shows that over 30% of citations generated in professional tasks are false; in image generation, this manifests as violations of physical rules and detail distortions; in speech technology, “auditory hallucinations” and “sound hallucinations” occur, particularly in noisy environments or when processing low-frequency vocabulary.
01
Deep-rooted Issues
Beyond undermining the fundamental credibility of AI applications, this phenomenon has triggered a series of deep-rooted challenges across technical, social, and ethical dimensions. From a technical standpoint, the lack of effective fact-checking mechanisms in AI systems leads to frequent generation of misinformation, which may cause the model ecosystem to gradually deviate from the data distribution of the real world, ultimately risking performance degradation or even collapse. Additionally, limitations in existing model architectures for processing abstract concepts and logical reasoning make it extremely difficult to fully solve the hallucination problem. From a social perspective, the proliferation of AI hallucinations is fueling a trust crisis in the information ecosystem. After repeated exposure to false information, the public may develop “information fatigue” and abandon the pursuit of truth. Meanwhile, the combination of algorithmic personalization and AI hallucinations exacerbates “echo chambers” and social polarization, shrinking the space for consensus among different groups. The ease of generating AI contents also threatens the livelihoods of creators, potentially leading to an overall decline in the creative ecosystem. From a legal and ethical perspective, dilemmas regarding responsibility attribution, lack of transparency and disclosure obligations, risks of “intellectual outsourcing”, and widening digital divides, all brought about by AI hallucinations, pose severe tests to existing legal frameworks and ethical systems.
02
Technical Root Causes
From a technical root cause perspective, these hallucinations stem primarily from four sources: 1. Inherent limitations in model training objectives: Large models often overemphasize textual fluency over factual accuracy, leading them to fabricate plausible content when faced with uncertainty. 2. Quality flaws and coverage limitations in training data: Misinformation from the internet is incorporated into training data, and coupled with limited knowledge scope, models tend to generate hallucinations when dealing with professional fields or emerging events. 3. Limitations in model architecture and reasoning mechanisms: Current models have not truly mastered logical reasoning and fact-checking capabilities, failing to distinguish between the fundamental differences between “familiar patterns” and “verified facts”. 4. Feedback loop effect of synthetic data: Hallucination-containing contents generated by AI are reintroduced into training data, causing errors to be continuously amplified through iterations. In the long run, this may lead the knowledge foundation of the entire AI ecosystem to deviate from the real world, creating a risk of “model collapse”.
03
Governance Paths
To fundamentally mitigate the technical impacts of AI hallucinations, in-depth optimization is required across multiple aspects including model design and data governance: First, it is essential to rethink the training objectives of AI systems, shifting from a sole focus on fluency and naturalness toward a comprehensive optimization that balances factual accuracy and expressive ability. This requires developing new training frameworks and evaluation metrics, incorporating fact-checking capabilities into model evaluation systems, and encouraging the technical community to build model architectures that prioritize factual accuracy.
Retrieval-Augmented Generation (RAG) technology has emerged as a key solution to the hallucination problem. By dynamically connecting large language models with external knowledge bases, RAG enables models to perform real-time retrieval and verification of relevant facts during content generation, significantly reducing the risk of “fabrication out of thin air”. More advanced implementations also adopt multi-level knowledge integration mechanisms: first retrieving coarse-grained information to define the overall direction, and then conducting accurate retrieval of specific facts to further enhance the accuracy and efficiency of fact-checking.
Strengthening data quality governance. On one hand, more refined data filtering and purification mechanisms must be established to reduce misinformation in training data. On the other hand, efficient synthetic data identification technologies, such as digital watermarking or statistical feature analysis, should be developed to prevent AI-generated contents from being indiscriminately included in training datasets, thereby breaking the “synthetic data feedback” loop. Practice has shown that even simple data cleaning strategies can significantly reduce model hallucination rates, while more sophisticated data governance systems may fundamentally improve a model’s factual perception capabilities.
Developing Explainable AI. By enabling models to explicitly express the confidence level of their responses and explain their reasoning process, users can more accurately assess the reliability of outputs and avoid blindly accepting high-risk contents. Specific implementations may include attaching confidence scores to model outputs, explicitly citing sources, or using visualization technologies to illustrate the model’s reasoning path, helping users understand the model’s decision-making basis and potential limitations.
Additionally, self-correction and active learning mechanisms should be strengthened. Advanced AI systems should be capable of recognizing the boundaries of their own knowledge. When confronted with uncertain information, they should proactively acknowledge their limitations or seek additional verification, rather than arbitrarily providing potentially incorrect answers. Such “meta-cognitive capabilities” need to be achieved through specialized training paradigms and model architecture designs, allowing AI to re-evaluate and make necessary corrections to its own responses.
