The Pitfalls of 'I Asked AI' — Authority Bias and the Hollowing Out of Knowledge
Authority bias in accepting AI output uncritically and knowledge hollowing from skill delegation. From calculators to GPS to LLMs—a recurring pattern.
What's Happening
"According to ChatGPT" and "Gemini says"—these prefixes are becoming increasingly common in both casual conversation and business settings. It's convenient. It's fast. It returns reasonably plausible answers.
But are those answers correct?
Research from Deakin University (Australia) revealed that 56% of academic citations generated by GPT-4o contain some form of error. About 20% of citations are complete fabrications—non-existent papers, non-existent authors, non-existent journals. Moreover, ChatGPT provided uncertainty disclaimers for only 15 out of 134 incorrect citations. The remaining 119 were presented confidently as if they were real sources.
A 2025 study by the Tow Center (Columbia University) reported that AI search engines failed to generate accurate citations in over 60% of tests.
The problem lies in a structure where such errors are difficult to detect. AI output is grammatically correct, logically consistent, and presented with confident tone. Humans are vulnerable to this "plausibility."
In a study targeting radiologists, when AI presented incorrect predictions, the accuracy rate of inexperienced doctors plummeted from 79.7% to 19.8%. Even veteran doctors saw their performance drop from 82.3% to 45.5%. Even professionally trained specialists bend their own judgment when AI provides its "stamp of approval."
This phenomenon is called Automation Bias—the cognitive tendency to trust machine or algorithmic output over human judgment. A structure where AI functions as "authority" and humans submit to that authority is rapidly spreading.
Background and Context
Recurring Patterns—From Calculators to LLMs
The phenomenon of technology transforming human cognitive abilities didn't begin with AI.
Calculators. LeFevre & Penner-Wilger (2005) reported that arithmetic fluency among Canadian young adults declined by over 20% from 1993 to 2005. Dependence on calculators led to the deterioration of mental arithmetic abilities.
GPS. A study published in Nature Scientific Reports (2020) showed that GPS usage has dose-dependent adverse effects on spatial memory. The more people use GPS, the more their ability to navigate independently deteriorates over time.
Search engines. The "Google Effect" reported by Sparrow et al. at Columbia University in 2011 demonstrated that when people know information is searchable, their brains stop trying to remember the information itself. Instead, they remember meta-information about "where to search"—a cognitive reallocation from information memory to search memory occurred.
And now generative AI. As analyzed in our series article Cognitive Debt, MIT Media Lab research showed that ChatGPT users experienced up to 55% reduction in brain network connectivity, with 83% unable to quote their own writing.
The pattern is clear. Each time a new tool emerges, the cognitive functions that the tool takes over begin to atrophy. But this time, there's one qualitative difference.
Calculators substituted computational ability. GPS substituted spatial cognition. Search engines substituted memory. Generative AI is attempting to substitute thinking itself.
"Substitution" vs. "Augmentation"—Two Modes of Use
When we examine AI usage structurally, it falls into two modes.
| Substitution Mode | Augmentation Mode | |
|---|---|---|
| Attitude | Delegate and done | Collaborate and deepen |
| Thinking | Accept results only | Understand the process |
| Skills | Atrophy from disuse | New capabilities develop |
| Risk | Platform dependency | Maintain autonomous judgment |
| Example | "AI said so, it must be right" | "Verify AI output, then decide" |
Substitution mode is the pattern of having AI do what you could do yourself. You stop using the brain you used for searching. You skip the effort you put into research. You stop writing the text you used to write. It's short-term efficiency, but unused abilities atrophy. This becomes the entry point to knowledge hollowing.
Augmentation mode is the pattern of using AI as a tool to expand your capabilities. This aligns with Douglas Engelbart's 1962 concept of "Augmenting Human Intellect." Using AI output as material to deepen, verify, and develop your own thinking.
A joint study by Harvard Business School and BCG (2023, involving 758 consultants) highlighted the reality of these two modes. They discovered two collaboration patterns: "Centaur type" where AI and humans clearly divide roles, and "Cyborg type" where AI is fully integrated across tasks. However, over 25% of participants exhibited "Self-Automator" behavior—fully delegating to AI while stepping back themselves.
The 25% who fell into substitution mode saw short-term productivity gains. But underneath, the foundation of their expertise was being eroded.
Knowledge Hollowing—What the Data Reveals
Gerlich's (2025) survey of 666 participants quantitatively confirmed this erosion:
- Correlation between AI usage frequency and critical thinking ability: r = -0.75 (strong negative correlation)
- Correlation between cognitive offloading and AI usage: r = +0.72
- Younger demographics showed higher AI dependence and lower critical thinking scores
Notably, higher education partially mitigated these effects. People with thicker existing knowledge foundations were better able to critically evaluate AI output. This reveals a structure where knowledge hollowing becomes more severe for those with less knowledge—amplifying inequality.
The same structure is visible in the programming world. GitHub Copilot now has tens of millions of users, and AI-assisted development is rapidly becoming standard. While junior developers enjoy the greatest productivity improvements, they also face the greatest risk of skill atrophy. According to GitClear's 2024 report, AI-generated code shows significantly higher churn rates (rate of being rewritten immediately after being written) compared to human code. Code can be written, but the understanding of why it should be written that way is missing.
The medical field is even more direct. A clinical trial in Poland found that after removing AI following 6 months of AI assistance, adenoma detection rates in colonoscopy dropped from 28% to 22%. The eyes of doctors who had grown accustomed to an environment where AI "finds things for them" had dulled.
Homogenization of Collective Intelligence—An Overlooked Problem
The problem of knowledge hollowing doesn't stop at individual-level capability decline.
Research by Doshi & Hauser (2024, Science Advances) demonstrated that while AI assistance improves individual creativity, it damages collective diversity. AI-assisted stories were rated as "more creative" and "more enjoyable." However, AI-assisted stories were similar to each other. Only 6% of AI-generated ideas were unique, contrasting with 100% uniqueness in human ideas.
While individuals become smarter, society's intellectual diversity is lost. A 2025 follow-up study reported a phenomenon called "Creative Scar"—individual creativity doesn't return to baseline even after AI is removed, and content homogenization continues to increase.
Reading the Structure
"Wizards" and the Structure of Dependence
One commenter compared AI companies to "wizards"—OpenAI, Anthropic, and above all, Google. Indeed, Google has been a "wizard" for over 20 years.
This metaphor contains structural insight. When the magic wears off, it's the moment you realize you mistook what was accomplished through platform functionality for your own ability. Affiliates, YouTubers, bloggers—countless examples exist of income disappearing with a single platform algorithm change.
Platform dependence in the AI era is qualitatively different from before. It's not dependence on income or reach, but dependence on thinking ability itself. When platforms change, you can apply skills elsewhere. But if thinking ability has hollowed out, you can't even move to the next platform.
As analyzed in ISVD's New Forms of Ignorance Created by Algorithms, algorithms automatically determine "what not to see." Dependence on AI output automatically determines "what not to think." From an agnotology perspective, this is a new phenomenon that could be called "structurally self-chosen ignorance" rather than "strategically manufactured ignorance."
"I Want to Use Magic Even if I'm Bad at It"
The same commenter described the meaning of running LLMs at home as "wanting to use magic even if I'm bad at it."
This attitude intuitively captures the essence of augmentation mode. Operating local LLMs involves direct contact with model architecture, training mechanisms, and inference processes. It's an endeavor to understand—even partially—the contents of the black box.
As discussed in our series article AI Amplification of Brandolini Asymmetry, LLM proliferation has brought the cost of lie production close to zero. But the cost of fact-checking remains unchanged. Within this asymmetry, the attitude of trying to understand AI mechanisms "even at a surface level" can serve as the most basic remedy against blind trust.
Since 2024, open-source model releases have nearly doubled those of closed-source models. Llama, Mistral, DeepSeek, Qwen—options for stepping outside the black box are broader than ever.
Structural Solutions
Addressing knowledge hollowing requires more than individual mindfulness. Structural problems need structural solutions.
1. Institutionalize "AI-free time." Some companies have introduced "Copilot-free Fridays"—days of development without AI coding assistance. Similar designs would be effective in educational settings. Like muscle strength, cognitive abilities atrophy without use.
2. Make verification habitual. The "3R Principles" proposed by Nature neuroscientists—Results (verification of results), Responses (agency in responses), Responsibility (maintaining responsibility)—are effective as individual-level guidelines. The habit of treating AI output as "material" rather than "answers" prevents the accumulation of cognitive debt.
3. Thicken existing knowledge foundations. As Gerlich's research showed, higher education mitigates the cognitive impacts of AI dependence. The deeper the understanding of mechanisms, the better one can critically evaluate output. Time to "think for yourself before asking AI" is not a decline in efficiency, but an investment in cognitive assets.
Whether AI usage expands or hollows out intellectual activity depends not on tool performance, but on user attitudes and institutional design. The nested structure—that this very judgment capacity can itself be eroded by AI—is at the core of this problem.
To borrow from the commenter who compared this to the Bitcoin bubble: "While anyone can receive the benefits, operation requires experience, especially the experience of getting burned." To truly leverage AI's benefits, we first need the resolve to accept the pain of thinking with our own heads. The mechanisms that organizationally and institutionally support that resolve will become the social infrastructure that prevents knowledge hollowing.
Related Columns
- Cognitive Debt — What Happens to Brains and Society When We Delegate Thinking to AI
- Digital Divide 2026 — The Paradox of DX Promotion Creating 'Unreached Populations'
- Vocational Training in the Generative AI Era — Can Institutional Design Keep Up with Technology?
Related Research
- New Forms of Ignorance Created by Algorithms — Filter Bubbles and Echo Chambers
- AI Amplification of Brandolini Asymmetry — When the Cost of Lie Production Approaches Zero
- The Reproduction of Authority and Ignorance — The Structure of 'No Need to Know'
References
Fabrication and errors in bibliographic citations generated by ChatGPT
Deakin University Research Team. Nature Scientific Reports, 13, 9139
Read source
AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking
Michael Gerlich. Societies, 15(1), Article 6
Read source
Generative AI enhances individual creativity but reduces the collective diversity of novel content
Anil R. Doshi, Oliver Hauser. Science Advances
Read source
Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality
Fabrizio Dell'Acqua, Edward McFowland III, Ethan Mollick et al.. Harvard Business School Working Paper
Read source
Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips
Betsy Sparrow, Jenny Liu, Daniel M. Wegner. Science, 333(6043), 776-778
Habitual use of GPS negatively impacts spatial memory during self-guided navigation
Dahmani, L., Bhatt, M., & Bherer, L.. Nature Scientific Reports, 10, 6310
Read source
