What Is Happening
In 2019, the New York Times investigative team reported on the "rabbit hole" effect, whereby YouTube's recommendation algorithm incrementally steers viewers toward increasingly extreme content. When a user watches moderately conservative content, the algorithm — seeking to maximize watch time — recommends content that is emotionally more intense and often more radical.
This phenomenon is not confined to YouTube. Google search results are personalized based on user search history and location; the same search term yields different results for different users. The "For You" tab on X (formerly Twitter) curates content based on user engagement patterns. TikTok's recommendation engine infers "interest" from scroll speed measured in hundreds of milliseconds and constructs the feed accordingly.
What these systems share is that they automatically determine what users do not see. By selecting what users "see," recommendation algorithms simultaneously determine what they "do not see." Yet users cannot perceive this selection. There is, in principle, no way to know what is absent from one's own feed.
Proctor's (2008) agnotology (無知学) has centered its analysis on the intentional production of ignorance. Algorithmic ignorance, however, is not intentionally produced by anyone. It arises structurally as a consequence of optimization. This presents a significant challenge to the theoretical framework of agnotology.
Background and Context
Pariser's Filter Bubble Thesis
Eli Pariser (2011) was the first to systematically address the fragmentation of the information environment brought about by personalization technology in The Filter Bubble: What the Internet Is Hiding from You. Pariser's central claim is that algorithmic filtering narrows the information environment without the user's awareness.
Pariser identified three characteristics. First, users within a filter bubble cannot perceive the bubble's existence. With biased television news, one can switch channels to recognize the bias. But personalized search results and feeds have no "unfiltered version." Second, users have not chosen to enter the filter bubble. What they chose was to use the service, not to have their information filtered. Third, filter bubbles are individualized. No two people inhabit the same bubble.
Pariser's analysis remains valid fifteen years after publication because the essence of the problem lies not in the technology but in the business model. In the attention economy, user dwell time and engagement translate directly into revenue. Algorithms are optimized to recommend "what users want to see," but "what they want to see" is not necessarily "what they need to know."
Connection to the Firehose of Falsehood
The "Firehose of Falsehood" model analyzed by Paul & Matthews (2016) at RAND Corporation originally targeted state propaganda strategy. Yet its four characteristics — high volume, rapidity, indifference to consistency, no requirement for truth — also apply to the information environment created by algorithms.
The critical difference between the two is the presence or absence of intent. The Firehose of Falsehood is an intentional attack with a clear actor (the state, intelligence agencies). The degradation of the information environment by algorithms, by contrast, has no attacker. Engineers at Google, Meta, and ByteDance are not attempting to "create a biased information environment." What they are optimizing is user engagement; the degradation of the information environment is a side effect.
Bergstrom & West's Analysis of Bullshit
Bergstrom & West (2020) analyzed in Calling Bullshit: The Art of Skepticism in a Data-Driven World the mechanism by which "bullshit" proliferates precisely in environments overflowing with data.
Critical to their analysis is the concept of statistical bullshit. Whereas earlier forms of bullshit relied on rhetorical technique, modern bullshit assumes a "scientific" exterior through the selective collection, manipulation, and presentation of data. Algorithms accelerate the diffusion of this statistical bullshit, for the content that maximizes engagement is often content that stirs emotions — and is frequently inaccurate in its factual claims.
McIntyre's Post-Truth Analysis
McIntyre (2018) defined post-truth in Post-Truth not as a state in which "facts do not exist," but as a state in which "facts cease to matter." This definition is indispensable for understanding algorithmic ignorance.
The problem of filter bubbles is not that users "believe lies." The problem is that both the motivation and the capacity to evaluate the reliability of sources and the accuracy of facts decline. When the information flowing through one's feed is comfortable and confirms one's beliefs, the incentive to verify whether it is factual is low.
Reading the Structure
Three Characteristics of Algorithmic Ignorance
The ignorance structured by algorithms possesses three characteristics qualitatively distinct from conventional mechanisms of ignorance production.
Characteristic 1: Intentionless Structuring (Emergent Intentionality)
The doubt manufacturing of the tobacco industry had clear intent. Memos survived; internal documents were released in court. Algorithmic distortion of the information environment, however, lacks the intent to "create distortion."
The objective function of recommendation algorithms is typically one of the following:
- Engagement maximization: Click-through rate, watch time, comments, shares
- User satisfaction: Session continuation rate, next-day retention rate
- Advertising revenue maximization: Ad impressions, cost per click
None of these objective functions contains the command "reduce information diversity." Yet when the most efficient method of maximizing engagement is "presenting content that reinforces users' existing interests and beliefs," information diversity declines as a consequence.
In this lab's coding axes, this type of intentionality is classified as emergent. It is neither intentional (strategic) nor simply structural. From the accumulation of individual system design decisions, consequences that designers did not anticipate emerge.
Characteristic 2: Orders-of-Magnitude Scale Difference
Conventional attention-control mechanisms — media agenda-setting, for instance — operated at a scale in which thousands of journalists and editors influenced tens of millions of recipients. Algorithms operate through a system designed by a few hundred engineers that simultaneously acts on billions of users.
This scale difference is not merely quantitative; it produces qualitative change. In traditional media, institutions for criticizing, monitoring, and regulating editorial judgment existed (however imperfectly). Press ethics codes, broadcasting ethics bodies (such as Japan's BPO) — each had its problems, but at least an accountability circuit existed.
For algorithms, this accountability circuit is virtually nonexistent. The internal workings of recommendation algorithms are protected as trade secrets, making external verification difficult. Users have no means of knowing how their feed is constructed. Regulators cannot keep pace with the technical complexity of algorithms.
Characteristic 3: Invisibility — Users Do Not Know What They Are Not Seeing
This is the most fundamental characteristic of algorithmic ignorance.
Bias in traditional media was, in principle, detectable. Comparing Newspaper A with Newspaper B revealed differences in coverage. Switching television channels revealed differences in topics covered. Users could at least recognize that "other perspectives exist."
With algorithmic filtering, such comparison is in principle impossible. User feeds are entirely individualized; no "unfiltered version" exists. What part of the whole one is seeing, what has been excluded — there is no way to know.
This invisibility carries theoretically important implications for Proctor's (2008) agnotology. In Proctor's framework, resistance to strategically produced ignorance was possible through "exposure" — releasing internal documents, whistleblowing, investigative journalism — thereby bringing "hidden knowledge" to light. In algorithmic ignorance, however, "hidden knowledge" does not reside in any particular location. Ignorance is produced distributively, in real time, in a different form for each user.
The Compound of Attention-Control and Complexity-Weaponization
Described using this lab's coding axes, this case constitutes a compound mechanism of attention-control and complexity-weaponization.
Attention-control is self-evident. Algorithms direct users' attention toward specific content and away from other content.
Complexity-weaponization functions more indirectly. The technical complexity of recommendation algorithms itself makes understanding and intervention by users, regulators, and researchers difficult. The state of "not understanding how the algorithm works" produces the state of "being unable to judge whether the algorithm is problematic." Complexity itself functions as a rampart against criticism and regulation.
It should be noted, however, that this "weaponization of complexity" is not intentional. Algorithms are complex because the problem cannot be solved simply. Yet the result is that this complexity impedes accountability. Here again, the structure of emergent intentionality is apparent.
The Theoretical Significance of Emergent Intentionality
The most important implication this case holds for agnotological theory is the need for the concept of emergent intentionality (創発的意図性).
Proctor's (2008) framework primarily analyzed strategic (intentional) ignorance production — the tobacco and fossil fuel industries. McGoey (2019) extended this in the structural direction, demonstrating that "pretending not to know" is a function inherent in power structures.
Algorithmic ignorance lies further along this trajectory. No one intends it, and it is not inherent in the structure. It emerges from the accumulation of individual optimization decisions. Yet its consequences — the distortion and fragmentation of the information environment — have social impact equal to or greater than strategically produced ignorance.
This is why this lab's intentionality axis includes "emergent." To analyze the production of ignorance in the digital environment, the strategic-structural binary is insufficient; a third category is theoretically necessary.
Questions for This Lab
This case analysis raises the following questions:
- Are conventional "exposure"-type resistance strategies effective against emergent ignorance production? When there is no "hidden intent" to expose, what serves as the starting point for resistance?
- Is algorithmic transparency sufficient to resolve the problem? Even if the operating principles of algorithms were disclosed, no one has the capacity to monitor billions of individualized feeds.
- Is it technically possible to incorporate "information diversity" into the objective function? If so, is it compatible with the corporate business model?
- How effective is improving individual literacy against a structural problem? Is "becoming aware of one's bubble" synonymous with escaping the bubble?
These questions are paired with the analysis of media agenda-setting and invisibilization. Traditional media's attention-control and algorithmic attention-control are manifestations of the same mechanism at different scales; their comparative analysis constitutes the core of the cross-cutting theme "Filter Structures."
References
Calling Bullshit: The Art of Skepticism in a Data-Driven World
Bergstrom, C. T. & West, J. D.. Random House
Read source
The Russian 'Firehose of Falsehood' Propaganda Model
Paul, C. & Matthews, M.. RAND Corporation, Perspectives PE-198
Read source
Post-Truth
McIntyre, L.. MIT Press
Read source
Agnotology: The Making and Unmaking of Ignorance
Proctor, R. N. & Schiebinger, L.. Stanford University Press
Read source