The Fog of AI Intelligence Measurement: How AI Evaluation Systems Circumvent the Core Challenges of the 'Compression is Intelligence' Hypothesis

Empirically, the assertion 'compression is intelligence' is refined into a series of operational hypotheses: under data from the same family or distribution, improvements in compression rate correlate positively with enhancements in model quality; this association can be fitted into a transferable mapping relationship; and this mapping remains stable across unseen models. A stronger version argues that compression metrics serve as sufficient proxies for model capabilities, even allowing for extrapolation across different tokenizers, model families, and application scenarios, thereby replacing traditional business performance evaluations. This perspective imbues compression rate with additional meaning and value in AI capability assessment, beyond its intrinsic measurement, positioning it as a potentially universal metric for intelligence.

However, the path of considering compression as a proxy for intelligence faces five major structural limitations. First is the **Aggregation Gap**: Byte-per-byte (BPB) average compression rates cannot directly reflect the conjunctive (all-correct-or-nothing) knowledge required for specific task success. This can lead to improved compression with stagnant task accuracy. Second is the **Identification Gap**: Common factors (Z) such as model capacity, training data, and computational power simultaneously enhance both compression capability and task performance. This implies that the observed 'compression-capability' correlation across models is not purely causal but systematically biased. Third is the **Extrapolation Gap**: The presence of limited training samples, distribution shift, and the 'No Free Lunch' theorem (Wolpert 1996) means that any compression-capability mapping learned from limited data cannot guarantee universality across held-out models, clients, or new domains. Furthermore, the **Measurement/Construct Gap** arises from the significant influence of factors like tokenizer choice, context length, and corpus distribution on BPB, and the fact that 'capability' itself is a metric-dependent proxy, potentially making observed 'correlation' an artifact of measurement granularity. Finally, the **Incremental Validity Gap**: If compression rate merely acts as a proxy for model scale, it then lacks independent incremental value for practical applications. Its 'usefulness' might just be a tautological restatement of 'larger models perform better,' failing to provide novel insights. These limitations collectively pose formidable challenges to the path of 'evaluating/predicting capability using compression.

To understand how AI evaluation systems address these challenges, we must first clarify their main components. These include: **Benchmark Suites**, such as task-oriented evaluations like MMLU, GPQA, and MATH, as well as intrinsic language modeling evaluations like Pile/WikiText; **Evaluation Frameworks and Tools**, for instance, EleutherAI's `lm-evaluation-harness`, Stanford HELM, and OpenCompass, which define metrics, aggregation methods, and error estimation; **Leaderboards and Arenas**, such as the HuggingFace Open LLM Leaderboard and Chatbot Arena, which rank and compare models; **Methodological and Statistical Standards**, covering principles like error bars, confidence intervals, significance testing, construct validity, contamination detection, and pre-registration; Finally, **Documentation and Governance Standards**, such as Model Cards, Datasheets, and BetterBench's benchmark quality checklist. Together, these components form a complex ecosystem for measuring AI model performance.

A critical observation, however, is that the use of the 'compression' concept within evaluation systems is strictly limited and 'isolated.' Compression metrics, such as bits-per-byte (BPB) and perplexity, primarily appear in the 'intrinsic language modeling' evaluation track within benchmark suites and in independent competitions like the Hutter Prize. This signifies that the compression rate is mainly regarded as a metric for a model's intrinsic language understanding efficiency or data representation capability. In stark contrast, mainstream capability-oriented benchmarks (e.g., MMLU, GPQA) and their corresponding leaderboards do not consider compression rate as a proxy metric for model capabilities at all. This 'isolation' is not active validation or falsification, but rather a strategic circumvention: by not relying on compression rate as a capability proxy, the aforementioned threats related to aggregation, identification, and incremental validity do not directly contaminate the conclusions of mainstream capability evaluations. Consequently, the 'compression is intelligence' hypothesis is effectively rendered 'not applicable' or 'not focused upon' in most AI capability evaluation practices.

Addressing the limitations of the hypothesis, current evaluation systems primarily adopt a 'direct measurement and avoidance' strategy for the **Identification Gap**. Evaluation systems do not infer model capabilities through compression rate; instead, they directly assess model performance across various tasks. This effectively avoids confusion arising from common factors (e.g., computational power, data volume) simultaneously improving both compression and capability. Evaluation conclusions merely state the model's relative performance on specific benchmarks, without implying causal attribution. Some auxiliary methods also offer weak identification: for instance, by releasing multi-tiered model families like Pythia and Llama, researchers can observe the relationship between scale and capability within a family, treating model scale as an explicit variable. Furthermore, quantified robustness ablation tests provide an experimental arm for 'endogenous compression variation.' Regarding the **Aggregation Gap**, capability evaluation has widely adopted conjunctive metrics. For example, `acc_all` in `lm-evaluation-harness` requires a set of sub-questions to all be correct to score; mathematical and coding tasks are scored using strict standards like exact match or pass@k. The HELM framework, moreover, measures up to 7 metrics (e.g., accuracy, calibration, robustness) for each of 16 core scenarios, preventing a single average from obscuring the full picture of model performance. However, on the compression side, evaluation still relies on a single average BPB value, lacking a decomposition into 'fact-carrying bits' versus 'other bits,' leaving this gap unaddressed in compression measurement.

Concerning the **Extrapolation Gap**, the evaluation system's strategies are the most mature and institutionalized. This involves the extensive use of held-out test sets and private test sets to assess a model's generalization capabilities on new data. To combat data contamination, studies like GSM1k detect model memorization of training data by renaming problem sets, while LiveBench dynamically generates new problems from recent competitions and papers, updating them monthly to continuously differentiate true model capabilities. Multi-domain, multi-lingual benchmarks and fine-grained distribution partitioning (e.g., HELM covering 42 scenarios) are employed to ascertain whether model capabilities are limited to specific distributions. Furthermore, evaluation systems increasingly emphasize statistical error bars and significance reporting, systematically treating evaluations as experiments, providing confidence intervals and sample size planning (Miller 2024) to quantify the uncertainty of evaluation results. These measures aim to enhance the robustness and credibility of evaluation results, maximizing empirical extrapolability even without formally guaranteeing universal capabilities.

Regarding the **Measurement/Construct Gap**, evaluation systems are technically prepared. For example, `lm-evaluation-harness` registers a byte-normalized `bits_per_byte` metric, and The Pile project explicitly recommends using BPB based on UTF-8 encoded bytes, responding to calls for ensuring consistent measurement standards when evaluating compression (Huang et al. 2024). However, these precise compression metrics are primarily 'isolated' within intrinsic language modeling evaluations and have not entered mainstream leaderboards as capability proxies. At the methodological level, construct validity frameworks have been clearly proposed: psychometrics (Freiesleben & Zezulka 2025) has been introduced to explore the conditions required for inferring scientific conclusions from benchmark scores. Documentary governance tools such as Model Cards and Datasheets also require disclosure of intended uses, metrics, and limitations, integrating construct validity into the release process. Despite the existence of tools and frameworks, there is a lack of institutionalized mechanisms to combine them into specific 'compression-capability' validity protocols.

In evaluating the core question of the 'compression is intelligence' hypothesis—namely, whether compression metrics provide incremental information beyond model scale—the current mainstream evaluation system exhibits the most significant gap. Leaderboards are established to rank models, not to test whether specific metrics (e.g., compression rate) possess independent incremental predictive power relative to other baselines (e.g., model scale or computational power). This implies that evaluation systems do not routinely perform nested model comparisons (such as LRT or information criterion ΔAIC) to determine whether, after controlling for factors like scale and computational power, compression metrics still significantly explain variance in model capabilities. Although some research (Huang et al. 2024) proposes using compression efficiency as an unsupervised evaluation metric, this remains at the academic discussion stage and has not yet been adopted as a standard practice by mainstream evaluation suites (e.g., HELM or Open LLM Leaderboard). Consequently, the alternative explanation—potentially most detrimental to the hypothesis—that 'compression rate is merely a proxy for model scale,' is not routinely tested in existing evaluation practices. Such questions are largely relegated to specialized academic research rather than daily benchmark evaluations.

Furthermore, evaluation systems lack institutionalized diagnostic mechanisms for 'compression arbitrage' or spuriously high compression rates caused by measurement artifacts. While projects like BetterBench recommend various benchmark anti-gaming mechanisms, such as using unique identifiers, encrypting evaluation instances, or detecting training data contamination, regular evaluation suites are almost entirely lacking in addressing issues specifically related to the 'representational geometry' degradation of compression metrics themselves. Research by Zang et al. (2025) indicates that high compression rates can sometimes originate from the degradation of representational geometry (e.g., anisotropy) and proposes geometrically refined compression metrics to diagnose such issues. However, these diagnostic methods concerning the representational layer of models have not yet been integrated into any mainstream AI model evaluation pipelines, leaving evaluation systems without effective identification and prevention capabilities for the possibility of models achieving high compression rates through non-intelligent means. This represents a critical blind spot in hypothesis validation, limiting our in-depth understanding of the true nature of model 'intelligence.