When AI Generates Everything: The Conditions for Knowledge Formation, From Authority to Verifiability

Prior to the AI wave, five 'self-evident' assumptions about knowledge formed our traditional understanding of 'knowing'. The first is the 'Possession Theory', which posits that knowledge is a stock owned by a subject, stored in human brains, documents, or databases, where 'knowing' is equivalent to 'possessing a representation'. This is closely linked to the 'Explicit Encoding-Transfer Theory', which suggests that knowledge can be externalized into symbols without loss and disseminated, as seen in Nonaka and Takeuchi's SECI model. However, Naur long ago pointed out that the true knowledge of a program lies in the 'theory' within the team's minds, with documentation being mere remnants. Concurrently, the 'Subject-Bound Theory' is also central; from Plato to Gettier, mainstream epistemology characterizes knowledge as 'justified true belief', meaning it must be believed by someone, be true, and be justified, emphasizing that knowledge must be anchored to a subject capable of belief and accountability.

Secondly, the 'Source Authority Theory' holds that the guarantee of a knowledge claim comes from its source, such as peer review, expert endorsement, or authoritative institutions. 'Who said it and where it was published' often determines whether information is considered knowledge more than the information itself, as the Stanford Encyclopedia of Philosophy (SEP) points out that Craig even suggests the function of the concept of 'knowledge' is to mark reliable information providers. Finally, the 'Scarcity and Accumulation Theory' posits that acquiring knowledge is expensive, requiring long-term education and experience accumulation. Thus, knowledge is scarce, depreciates slowly, and constitutes an individual's human capital and an organization's moat. These five assumptions collectively depict a 'warehouse-style' view of knowledge: knowledge is an 'object', stored in a 'repository', guaranteed by 'authority', acquired through 'accumulation', and attributed to a 'subject'.

The rise of Large Language Models (LLMs) is dismantling the aforementioned traditional assumptions through seven key mechanisms. Firstly, mechanism M1, 'Parametrization and Lossy Compression', means knowledge is no longer 'within documents' but distributed across the model's vast web of weighted associations. Ted Chiang likens LLMs to a 'blurry JPEG' of the internet, while the ROME research pinpointed the mid-layer feedforward modules in GPT as the storage location for factual associations, demonstrating that knowledge becomes editable, not intuitively reviewable, and probabilistic, directly challenging assumptions A1 and A2. Mechanism M2, 'Retrieval-Augmented Generation' (RAG) technology, transforms knowledge from static 'storage' into dynamic 'assembly' at runtime, shifting the core action from 'retrieval' to 'synthesis'. Mechanism M5, 'Prediction as Knowledge', exemplified by AlphaFold's prediction of over 200 million protein structures, allows knowledge to take the form of corrigible probabilistic inferences, rather than 'possessed facts' in the sense of A1, further weakening the certainty of traditional cognition.

Secondly, mechanism M3, the 'Hallucination' phenomenon, has become crucial for reshaping the conditions for knowledge formation. Stanford's legal AI research shows that Lexis+ AI and Westlaw AI-AR exhibit hallucination rates as high as 17%–33%, indicating that RAG and 'citing real documents' are insufficient guarantees of truthfulness; it is imperative to check 'whether the source genuinely supports the assertion'. This directly pierces assumption A3, making 'the act of verification itself' a condition for knowledge formation. Mechanism M4, 'Machine-Verifiable Proofs', offers an entirely new path for assurance: Anthropic's Claude 'largely autonomously' completed a machine-checkable Lean proof of Fermat's Last Theorem, while AlphaProof also achieved silver medal level in the IMO competition. This implies that 'truthfulness' can be adjudicated by algorithms, and the 'justification' logic of knowledge is rewritten as a 'machine-checkable warrant'.

Finally, mechanism M6, 'Automated Knowledge Production and Recursive Contamination', warns that while AI can scale up the generation of 'knowledge-like artifacts', it degrades when recursively trained on its own outputs, as shown by the 'model collapse' theory. A paper generated by Sakana AI's AI scientist passed blind peer review but was later retracted due to hallucinations and erroneous citations, highlighting the historical value of external real data as a 'grounding anchor' and refuting assumption A2's 'generative self-sufficiency'. Mechanism M7, 'Cognitive Offloading', reveals the separation between 'generability' and 'internalization'. MIT's study, 'Your Brain on ChatGPT', indicates that excessive reliance on LLMs can lead to decreased brain connectivity strength and a lower sense of ownership over articles, thereby challenging assumption A5 that 'knowledge must reside with a believing subject' and disaggregating the knowing subject into a hybrid 'generator-verifier-accountable agent' system.

In the face of AI's fundamental disruption of traditional views of knowledge, we must redefine 'knowledge'. Its core operational definition is: Knowledge = the executable ability to be generated on demand when needed, and to be independently verifiable as true. 'Knowing' shifts from 'possessing a representation' to 'being able to generate and guarantee an assertion'. The minimal knowledge structure is a three-piece set: a generator (model/human), a retrieval source (corpus/evidence), and a validator (evaluation/checker/expert judgment). Therefore, the essence of knowledge is no longer a simple replacement of 'possession' with 'generability', but a deep coupling of 'generability' and 'verifiability'. Generation is the capability, but verification is the criterion for knowledge formation. As the report criteria state, generation without verification is merely fluent falsehood; verification without generation is still merely retrieval from the old paradigm. The true revolution lies in their coupling: Knowledge = Generation × Verification.

On this basis, we can construct a more comprehensive definition for the new paradigm. At the asset level, knowledge assets are no longer document repositories but a combination of 'generative capability (models/weights) + corpus and retrieval sources + validators (evaluation sets, checking rules, formalization) + processes and prompts'. At the economic level, knowledge shifts from 'stock' to 'flow', with its unit cost being the sum of inference cost and verification cost, and its scarcity stemming from verification bandwidth and accountability capacity. Epistemologically, knowledge is an 'attributable generation-verification closed loop', and the knowing subject expands into a hybrid 'human-model-checker' system. These definitions collectively point to an actionable criterion: if you are still measuring knowledge by 'how many documents there are, how many experts there are', you are in the old paradigm; if you start measuring by 'how much has been generated and verified, and what is the cost of verification', you are entering the new paradigm.

Although large language models exhibit astonishing generative capabilities, their inherent 'hallucination' phenomenon has become critical evidence for the restructuring of knowledge formation conditions. For instance, Stanford University's evaluation of Lexis+ AI and Westlaw AI-AR revealed hallucination rates in legal Q&A as high as 17% to 33% [Verified·Pre-registered paper]. This not only exposes the inherent uncertainty of AI-generated information but, more importantly, explicitly states that even 'Retrieval-Augmented Generation' (RAG)—where models cite real documents—cannot fully guarantee the truthfulness of an assertion. In the old paradigm, source authority was sufficient to guarantee knowledge; but under the new paradigm, we must independently verify 'whether the source genuinely supports the assertion,' rather than merely confirming 'whether the source exists'. This shift compels us to place 'verifiability' at the core of knowledge determination, rather than solely relying on generative acts or superficial citations.

Alongside hallucinations, the rise of machine-verifiable proofs offers a new path for knowledge formation conditions. Anthropic's Claude model 'largely autonomously' completed the first full machine-checkable Lean proof of Fermat's Last Theorem in just 11 days; this proof, verified by the Lean checker, no longer requires the authoritative endorsement of human experts [Verified·Primary source]. Concurrently, over 200 million protein structure predictions in the AlphaFold database, despite most not being experimentally confirmed, are widely used as 'knowledge', significantly bridging the gap between known sequences and the limited number of experimentally determined structures [Verified·Primary source]. These cases collectively demonstrate that knowledge assurance can be algorithmized, and its form can be probabilistic inference rather than being limited to static facts confirmed by experiment. This strongly challenges the reliance on human subjects and definitive evidence in traditional epistemology.

However, mere generative capability is not omnipotent; counter-examples instead reinforce the necessity of 'generation × verification'. A paper generated by Sakana AI's AI scientist passed blind peer review but was later retracted due to erroneous citations and hallucinations [Verified·Vendor+Peer review], highlighting that automated knowledge production, if lacking verification, can fall into 'fluent falsehood'. The 'model collapse' theory further warns that if a model relies solely on its own generated content for recursive training, it will irreversibly degrade [Verified·Paper]. This means that external real data, human experimentation, and genuine interactions are upgraded from mere 'fuel' to 'grounding anchors for truth'. Furthermore, research from MIT indicates that excessive reliance on AI tools can lead to cognitive offloading, reducing the strength of brain connectivity and the sense of 'ownership' over knowledge [Verified·Primary institution/Preprint awaiting review], emphasizing that even if a system can generate knowledge, humans still need to undertake the subjective role of understanding, verifying, and being accountable.

The profound shift in the definition of knowledge will trigger a chain reaction at the business and organizational levels. Traditional business models based on 'content subscriptions' or 'expert seats' treat knowledge as 'content' that can be owned and sold. Under the new paradigm, value capture will shift from 'how many data repositories one possesses' to 'how cost-effectively one can provide verifiable results'. A company's moat will no longer be exclusive information but an organic combination of 'proprietary corpora, efficient validators, and optimized workflows'. This means 'verification' itself becomes a core, priceable component; whoever can verify the authenticity of information in a more credible and economical way will control the upper echelons of the value chain.

Organizational structures and talent strategies must also adapt accordingly. In the old paradigm, companies tended to 'buy in knowledgeable people' and build vast document repositories. Under the new paradigm, the core task becomes 'building efficient generation-verification systems' while retaining talent with critical judgment and a sense of responsibility. An organization's knowledge assets will be redefined from traditional documents or expert directories to a combination of 'generators (models/algorithms), corpora and retrieval sources, validators (evaluation sets/checking rules), and refined processes and prompts' [Inference]. This implies that enterprises will need to regularly conduct 'tacit knowledge loss drills' to ensure that their core knowledge can still be continuously generated and verified through hybrid systems, even in the absence of specific personnel or tools.

At the product level, the shift in the knowledge paradigm will directly impact user experience and feature design. Product interfaces will no longer be satisfied with merely offering 'search boxes' or 'information feeds'; instead, they must evolve into intelligent assistants capable of providing 'answers + evidence chains + confidence/uncertainty annotations'. This means that features such as citation, provenance, and fact-checking (groundedness check) will escalate from compliance requirements to core product competencies. User expectations for 'verifiability' will become a primary metric for assessing product quality. In the future, all products whose core output is knowledge must embed transparent verification mechanisms within their core experience.

In response to the fundamental changes in the definition of knowledge, research institutions must proactively adjust their internal strategies and norms. Firstly, the core task is to re-inventory and redefine 'knowledge assets'. This means institutions can no longer use the number of documents or expert qualifications as the sole measure of their knowledge reserves. Instead, every critical knowledge asset should be accompanied by a 'knowledge asset card', detailing its original source (raw data/human/literature), generation path (model, prompt, retrieval source), verification method (evaluation set, formal verification, expert review), responsible party, and the date of the most recent verification. Documents themselves should be relegated to the category of 'corpus', no longer serving as the ultimate evidence for knowledge formation [Inference].

Secondly, research institutions urgently need to implement 'verification-first' research protocols. Any assertion intended for inclusion in the institutional knowledge base must be accompanied by clear, clickable source links and indicate its verification status (e.g., verified, inferred, pending verification). More critically, researchers must explicitly perform a 'groundedness' check—'does the source genuinely support the assertion?'—rather than merely confirming the source's existence. This will effectively prevent common erroneous citations and misleading information from AI-generated content from entering the core knowledge system. Furthermore, unverified AI-generated citations should be strictly restricted from being added to the knowledge base [Inference].

Finally, institutions should establish a task matrix based on 'generability × verifiability' to optimize human-AI collaboration and resource allocation. For tasks with high generability that can be efficiently verified by automated tools (e.g., code testing, structured data delivery, formal proofs), priority should be given to AI generation and automated verification. This will not only increase efficiency but also free human researchers from repetitive work, allowing them to focus on low-generability but high-value tasks that AI struggles with, requiring creative insight, complex cross-domain connections, or bearing ultimate responsibility. Through this refined division of labor, research institutions can effectively navigate the knowledge transformation brought by AI, enhancing the quality and reliability of overall research output [Inference].