Aware of All Internet Traditions: Large Language Models as Information Retrieval and Synthesis

Cosma Rohilla Shalizi (CMU + SFI)

10 March 2026, Cultural AI Workshop

In conclusion

GenAI is information retrieval and synthesis
GenAI generates formulaically
Human culture is also pretty formulaic, and that is OK
Formulas are traditional
\(\therefore\) GenAI is mechanized/prosthetic tradition

GenAI is information retrieval

With the right tools + access, we can quantify the influence of each training document on every response

1. GenAI is information retrieval

From Grosse et al. (2023):

(= Arthur C. Clarke, 2010: Odyssey Two [1984])

1. GenAI is information retrieval

With the right tools + access, we can quantify the influence of each training document on every response
- Undergrads can grasp the math, the real work of Grosse et al. (2023) was doing it efficiently and at scale in the mess of real data

1. GenAI is information retrieval and synthesis

With the right tools + access, we can quantify the influence of each training document on every response
GenAI doesn’t just repeat its training corpus
- Of course there’s stuff it memorizes (Carlini et al. 2020)
- But it can’t memorize everything and it produces stuff it could never have memorized

1. GenAI is information retrieval and synthesis

etc., etc., for several more pages

2. GenAI generates formulaically

GenAI can generate because it has learned formulas
- Also tropes, stereotypes, templates, conventions, genres, all sorts of recurring patterns in the symbol-stream

2. GenAI generates formulaically

This is all wrong about me
This is 100% right about a lot of people like me

2. GenAI generates formulaically

I didn’t say “tokenizer”
Most R tokenizers do use strsplit(), but not this one

3. Much of human culture is formulaic

Formulas for oral epic

3. Much of human culture is formulaic

Formulas for oral epic (e.g., Lord (1960))
Formulas for scientific papers

3. Much of human culture is formulaic

Formulas for oral epic (Lord 1960)
Formulas for scientific papers (e.g., Harris (2002); Harris (1988), ch. 2)
Formulas for story plots

3. Much of human culture is formulaic

Formulas for oral epic (Lord 1960)
Formulas for scientific papers (Harris 2002, 1988, ch. 2)
Formulas for story plots

3. Much of human culture is formulaic

Formulas for oral epic (Lord 1960)
Formulas for scientific papers (Harris 2002, 1988, ch. 2)
Formulas for story plots (e.g., Propp (1968))

4. Formulas are traditions

Following a tradition means not having to think for oneself
- This has real evolutionary advantages (Simon 1990)

4. Formulas are traditions

Following a tradition means not having to think for oneself: Barzun (1959)

Intellect is the capitalized and communal form of live intelligence; it is intelligence stored up and made into habits of discipline, signs and symbols of meaning, chains of reasoning and spurs to emotion — a shorthand and a wireless by which the mind can skip connectives, recognize ability, and communicate truth. Intellect is at once a body of common knowledge and the channels through which the right particle of it can be brought to bear quickly, without the effort of redemonstration, on the matter in hand.

Intellect is community property and can be handed down. We all know what we mean by an intellectual tradition, localized here or there; but we do not speak of a “tradition of intelligence,” for intelligence sprouts where it will…. And though Intellect neither implies nor precludes intelligence, two of its uses are — to make up for the lack of intelligence and to amplify the force of it by giving it quick recognition and apt embodiment.

For intelligence wherever found is an individual and private possession; it dies with the owner unless he embodies it in more or less lasting form. Intellect is on the contrary a product of social effort and an acquirement…. Intellect is an institution; it stands up as it were by itself, apart from the possessors of intelligence, even though they alone could rebuild it if it should be destroyed….

The distinction becomes unmistakable if one thinks of the alphabet — a product of successive acts of intelligence which, when completed, turned into one of the indispensable furnishings of the House of Intellect.

4. Formulas are traditions

Following a tradition means not having to think for oneself
Humans internalize traditions by immersion and practice
- Bottle-washing, canvas-stretching, etc. etc.
- We have (probably) evolved to do this (Herrmann et al. 2007)

4. Formulas are traditions

Following a tradition means not having to think for oneself
Humans internalize traditions by immersion and practice
Human transmission of tradition is always selective
- There is always too much to pass on, only what continues to be relevant makes it (Hodgson 1974; Morin 2016)

5. GenAI is the mechanization of tradition

Not “geniuses in a data-center” (intelligence), but an all-access pass to the House of Intellect

5. GenAI is the mechanization of tradition

Not “geniuses in a data-center” (intelligence), but an all-access pass to the House of Intellect
Or at least the part of that House available online
- “I am aware of all Internet traditions”
- Sucks to be a small and/or unwritten language / off-line / secretive / old / genuinely novel

5. GenAI is the mechanization of tradition

Not “geniuses in a data-center” (intelligence), but an all-access pass to the House of Intellect
Or at least the part of that House available online
Or at least the external forms of online traditions, without the inner structures / human significance
- Sometimes this might be fine?
- This might be all some traditions have (Boyer 1990)
- This might be a shambles (Vafa et al. 2025)

5. GenAI is the mechanization of tradition

Not “geniuses in a data-center” (intelligence), but an all-access pass to the House of Intellect
Or at least the part of that House available online
Or at least the external forms of online traditions
Consequences for transmission: ??!?!?

“What has concluded that we might conclude in regard to it?”

GenAI is not original, creative, problem-solving intelligence
It is mechanized intellect, prosthetic access to the external formulas of many but not all traditions
This is incredible, and perhaps a disaster

I’m glad you asked that!

Making this sound like Capital

Fetishization: LLM users think they are interacting with an artificial intelligence, but really it is a social relation (to all the authors of the source texts)
Intellect:Intelligence::Labor:Capital::Dead labor:Living labor
It is no accident, comrades, that Barzun wrote “Intellect is the capitalized … form of live intelligence”
Working in citations to Vygotsky (1986), Vygotsky (1978), Luria (1976) left as an exercise for the reader

Patterns & Transformers

Transformers are higher-order Markov chains, but still finite-order Markov chains (Zekri et al. 2024)
\(\therefore\) there are many patterns they cannot learn exactly (Chomsky 1956)
But they can and do learn approximations, often short-cuts which work very badly out-of-distribution (Liu et al. 2023; Zhang et al. 2024)

References

Barzun, Jacques. 1959. The House of Intellect. New York: Harper.

Boyer, Pascal. 1990. Tradition as Truth and Communication: A Cognitive Description of Traditional Discourse. Cambridge, England: Cambridge University Press. https://doi.org/10.1017/CBO9780511521058.

Carlini, Nicholas, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, et al. 2020. “Extracting Training Data from Large Language Models.” arxiv:2012.07805. http://arxiv.org/abs/2012.07805.

Chomsky, Noam. 1956. “Three Models for the Description of Language.” IRE Transactions on Information Theory 2:113–24. https://doi.org/10.1109/TIT.1956.1056813.

Grosse, Roger, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, et al. 2023. “Studying Large Language Model Generalization with Influence Functions.” E-print, arxiv:2308.03296. http://arxiv.org/abs/2308.03296.

Harris, Zellig. 1988. Language and Information. New York: Columbia University Press.

———. 2002. “The Structure of Science Information.” Journal of Biomedical Informatics 35:215–21. https://doi.org/10.1016/S1532-0464(03)00011-X.

Hodgson, Marshall G. S. 1974. The Venture of Islam: Conscience and History in a World Civilization. Chicago: University of Chicago Press.

Liu, Bingbin, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. 2023. “Transformers Learn Shortcuts to Automata.” In The Eleventh International Conference on Learning Representations [ICLR 2023]. https://arxiv.org/abs/2210.10749.

Lord, Albert B. 1960. The Singer of Tales. Cambridge, Massachusetts: Harvard University Press.

Luria, A. R. 1976. Cognitive Development: Its Cultural and Social Foundations. Cambridge, Massachusetts: Harvard University Press.

Morin, Olivier. 2016. How Traditions Live and Die. Oxford: Oxford University Press.

Propp, Vladimir. 1968. The Morphology of the Folktale. Second. Austin: University of Texas Press. https://doi.org/10.7560/783911.

Simon, Herbert A. 1990. “A Mechanism for Social Selection and Successful Altruism.” Science 250:1665–8. https://doi.org/10.1126/science.2270480.

Vafa, Keyon, Peter G. Chang, Ashesh Rambachan, and Sendhil Mullainathan. 2025. “What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models.” http://arxiv.org/abs/2507.06952.

Vygotsky, L. S. 1978. Mind in Society: The Development of Higher Psychological Processes. Cambridge, Massachusetts: Harvard University Press.

———. 1986. Thought and Language. Cambridge, Massachusetts: MIT Press.

Zekri, Oussama, Ambroise Odonnat, Abdelhakim Benechehab, Linus Bleistein, Nicolas Boullé, and Ievgen Redko. 2024. “Large Language Models as Markov Chains.” E-print, arxiv:2410.02724. http://arxiv.org/abs/2410.02724.

Zhang, Dylan, Curt Tigges, Zory Zhang, Stella Biderman, and Talia Ringer Maxim Raginsky and. 2024. “Transformer-Based Models Are Not yet Perfect at Learning to Emulate Structural Recursion.” E-print, arxiv:2401.12947. https://arxiv.org/abs/2401.12947.