Beyond Apparent Correlations: A Rigorous Framework for Experimentally Validating the True Predictive Efficacy of Large Language Model Compression Metrics

Compression metrics in LLM capability evaluation, such as perplexity (PPL) or bits per byte (BPB), have attracted significant attention due to their strong negative correlation with model performance. However, this superficial association often conceals complex confounding factors. For instance, model scale, training data volume, and computational resources (Z) frequently enhance both a model's 'compression capability' (C) and 'task quality' (Q) simultaneously. Under this common-cause structure, simple correlation analysis systematically inflates the observed correlation beyond the true causal association, creating an 'illusion of correlation' that misleadingly suggests compression metrics directly reflect, or even cause, model quality. `【Verified - Local: OPC-103 §1.3②; Standard Omitted Variable Bias】`

Therefore, equating 'true predictive efficacy' with simple statistical correlation is a methodologically dangerous misconception. Genuine predictive efficacy not only demands that a metric can predict a criterion variable but also requires this predictive ability to be out-of-sample generalizable, causally identified (resistant to confounding), robust to distribution shifts, and capable of providing incremental information over existing strong baselines. Without rigorously designed experiments to disentangle these confounding factors, any conclusions drawn from simple correlations may be untenable and could even misguide the research, development, and application of LLMs. `【Inference】`

To accurately assess the 'true predictive efficacy' of compression metrics, one must first deconstruct it into four types of validity issues within psychometrics. The first is **construct validity**, which asks what the compression metric truly measures, and whether it accurately operationalizes the 'capability' or 'quality' construct we claim. If the metric definition is vague or suffers from measurement artifacts, subsequent predictions will be meaningless. The second is **criterion/predictive validity**, which examines whether compression metrics can effectively predict business quality on unseen samples. This requires the predictive relationship to be validated on datasets independent of the fitting process, rather than self-validated on the same data. `【Inference: Standard psychometric classification】` `【To be checked: This conversation has not individually extracted original psychometric textbook content】`

The third is **incremental validity**, which questions whether compression metrics can still provide unique and valuable information gain after accounting for strong baselines such as model scale and training steps. If compression metrics merely act as proxies for these strong baselines, they offer no incremental value in practice. Finally, **external validity** examines whether the predictive power of compression metrics holds across different domains, languages, customers, or timeframes. These four types of validity must be individually considered and determined in experimental design. Construct validity is a prerequisite for all other validities; if the construct itself is problematic, subsequent predictive validity might merely be a measurement artifact. `【Verified - Literature: Freiesleben & Zezulka 2025; Verified - Local: OPC-316 §3.4; Inference】`

Before embarking on experimental design, five major measurement calibration issues related to compression metrics must be defined and clarified; otherwise, all subsequent statistical analyses risk becoming noise. First, metric definitions must be unambiguous. For example, 'compression rate' has at least four opposing or different definitions (engineering convention, compression ratio convention, BPB/BPC, PPL), necessitating a unified agreement on direction and calculation method. Second, cross-model comparisons must use byte/character-level metrics (BPC/BPB) instead of model-specific token-level metrics, to eliminate confusion caused by different tokenizers and ensure metric comparability. `【Verified - Local: OPC-316 §1; OPC-103 §1.1】` `【Verified - Literature: Huang et al. 2024 Abstract and Main Text; Verified - Local: OPC-316 §2.2, §3.4】`

Furthermore, caution is needed regarding potential 'compression hacking' of compression metrics. Research indicates that highly compressed model representations can degrade into highly anisotropic states, yielding spuriously high compression rates. Therefore, experiments must incorporate metric variant ablation (raw vs. geometrically refined metrics) and representation geometric diagnostics to rule out the possibility of compression metrics being measurement artifacts. Simultaneously, a distinction must be made between 'model predictive compression' and 'output redundancy compression': measuring the `zlib` compression ratio of model output assesses output redundancy, not the model's predictive compression capability against a reference distribution. These are distinct constructs and should not be conflated or substituted. `【Verified - Literature: Zang et al. 2025 (arXiv:2505.17793)】` `【Inference】` `【Verified - Local: OPC-103 §3.8, Item 1】`

Finally, to accurately evaluate a model's intrinsic compression capability, experimental design must incorporate traditional compressors as baselines. Using `gzip`/`zstd` to calculate the 'inherent compressibility of the data itself' on the same text can differentiate between 'this text is easy to compress' and 'this model predicts well'. Without such a strong baseline comparison, it is difficult to ascertain whether a compression metric reflects intrinsic textual properties or the model's genuinely superior predictive and representational abilities, which is crucial for assessing its unique value as a proxy for LLM capabilities. `【Verified - Local: OPC-203 §1.5, §1.7】`

Currently, the apparent correlation between Large Language Model (LLM) compression metrics and performance is often confounded by common factors such as model scale and training data. To sever this common-cause pathway, rigorous experimental design must fix capacity and introduce exogenous compression perturbations. For example, under the same model weights, endogenous compression variability can be induced by manipulating the relevance of in-context demonstrations (relevant vs. random demonstrations) or quantization levels (fp16/int8/int4), ensuring constant capacity variables. Furthermore, the statistical unit of analysis should be at the item level, with repeated measurements for each item under various compression conditions. Employing mixed-effects models or paired tests can effectively leverage the paired structure, reduce measurement noise, and avoid the misleading conclusions drawn from macroscopic correlation analysis relying on only a few configuration points. This approach aims to identify the independent effects of compression and quality, rather than being overshadowed by common causes. `【Verified - Local: OPC-103 §3; AUTO-005 §3.2; OPC-103 §3.2, §3.5】`

Second, the aggregation gap problem of compression metrics needs to be directly addressed. The overall bits per byte (BPB) is an average, whereas task correctness is often a conjunction of multiple pieces of information; the two cannot be simply equated. Experiments should decompose total BPB into 'fact-carrying bits per byte' (`C_fact`) and 'other bits per byte' (`C_other`). By introducing annotators to identify critical fact-carrying spans, the unique contribution of `C_fact` to quality can be directly examined, avoiding the misleading performance of total BPB. Concurrently, to ensure the 'authenticity' of predictive efficacy, a true extrapolation strategy must be implemented, involving strict separation across three dimensions: models, items, and data distributions (triple hold-out). The evaluation set must include unseen models, unseen items, and samples from different distributions to prevent overfitting and test set contamination, thereby ensuring the out-of-sample generalization capability of the predictive mapping. `【Verified - Local: OPC-103 §1.3①, §3.3】` `【Verified - Local: AUTO-005 §2.3, §3.4; Verified - Literature: Hofman et al. 2023】`

Predictive ability alone is insufficient to prove value; compression metrics must demonstrate that they provide incremental information. Experimental design should embed compression metrics into nested models that include strong baselines (e.g., model scale, training steps, average benchmark scores, and `gzip`/`zstd` compression rates as data compressibility baselines). Their ability to significantly improve out-of-sample predictive performance should be assessed using likelihood ratio tests or `ΔAUC`/`ΔR²` from cross-validation. Particularly, if compression metrics can still exhibit statistically significant partial correlation after controlling for key confounding factors like model scale, then their unique predictive value is truly demonstrated, preventing them from being mistakenly considered simple proxy variables for existing strong baselines. `【Inference】` `【Verified - Local: AUTO-005 §4.4; OPC-203 §1.5】` `【Verified - Local: OPC-103 §3.4; AUTO-005 §2.2, §2.3】`

To ensure the scientific rigor and reliability of experiments, a preregistration mechanism is indispensable. Before commencing an experiment, the item set, prompt templates, annotation manual, analysis scripts, and decision thresholds must be frozen. Furthermore, a three-state judgment rule ('support,' 'falsify,' or 'insufficient evidence') and multiple comparison correction strategies must be predefined. This mechanism aims to eliminate post-hoc selective reporting bias, enhancing the reproducibility and transparency of the research. Pre-declaration of primary and secondary endpoints, along with corresponding `α` allocation, ensures that the study focuses on key hypotheses and explicitly labels exploratory analyses, thereby avoiding common `p`-hacking pitfalls and bolstering the credibility of conclusions. `【Verified - Local: OPC-103 §3.9; AUTO-005 §3.4】` `【Verified - Local: OPC-103 §3.4–3.6】`

Finally, measurement artifacts of compression metrics are excluded through embedded falsification and negative control designs. For example, randomly shuffling compression scores or replacing them with random numbers confirms whether the predictive relationship disappears accordingly, verifying the rigor of the experimental procedure. Concurrently, introducing metric variant ablation (raw vs. geometrically refined compression `【Verified - Literature: Zang et al. 2025】`) and representation geometric diagnostics helps investigate 'compression hacking' phenomena, where models might falsely report high compression rates by sacrificing representation quality. Synthetic known-ground-truth tasks and counter-example registries are used to test whether theoretical predictions hold under ideal conditions and to identify boundary conditions at known counter-examples, together forming a robust falsification system to ensure that any observed predictive efficacy is genuine and not spurious. `【Inference】` `【Verified - Local: OPC-103 §5】`

This study proposes a five-stage hierarchical experimental plan aimed at translating the aforementioned rigorous principles into actionable protocols. First is **Stage 0: Freezing and Efficacy Simulation**, where construct definitions, item sets, prompt templates, and analysis scripts are locked, and sample size and minimum detectable effect are determined through simulation. During this stage, critical preliminary checks must be performed, such as confirming whether local models support `logprobs` output, or if relevant corpus files exist and are available for annotation; if conditions are not met, the design must be adjusted or downgraded. Next is **Stage 1: Exogenous Compression Perturbation with Fixed Weights**, where compression variability is generated using context relevance or quantization levels, measuring the impact of `C_total` and `C_fact` on task quality in an item-level `within-item` manner. **Stage 2: Cross-Configuration Calibration and External Hold-out**, using multiple checkpoints from the same model family to build calibration points and holding out from external model series to validate the out-of-sample predictive capability and calibration effect of compression metrics. `【Verified - Local: OPC-103 §3.9; AUTO-005 §3.4; OPC-103 §3.1, §3.8; AUTO-005 §1.1】` `【Verified - Local: OPC-103 §3.2, §3.3】` `【Verified - Local: AUTO-005 §2.3, §3.2, §4.4】`

The subsequent **Stage 3: Incremental and Discriminant Validity**, involves comparing compression metrics with strong baselines in nested models and introducing irrelevant distractor metrics to confirm the uniqueness of their incremental contribution. **Stage 4: Robustness, External Validity, and Falsification**, tests the stability of conclusions by varying denominator conventions, sampling strategies, and domain/language hold-outs, while performing deeper falsification using compression hacking diagnostics and random pseudo-metrics. Finally, **Stage 5: Preregistered Analysis, Three-State Publication, and Adversarial Replication**, strictly analyzes and publishes results according to the frozen scripts, regardless of whether they are negative, and invites independent analysts for adversarial replication to maximize the credibility and transparency of the findings. `【Inference】` `【Verified - Local: AUTO-005 §3.4, §7】` `【Verified - Literature: Zang et al. 2025】` `【Verified - Local: OPC-103 §3.9; AUTO-005 Falsifiability Commitment】`

Evaluating 'true predictive efficacy' requires a three-state judgment standard: 'supported,' 'falsified,' or 'insufficient evidence.' Each state has clear item-level, configuration-level, and incremental validity thresholds, such as `ΔQ≥5pp`. Reporting guidelines emphasize effect sizes and bootstrap confidence intervals, rather than solely relying on p-values. Concurrently, experimental limitations must be declared, such as a limited number of hold-out configurations serving only as falsifying evidence, or that results from a single item set should not be over-extrapolated to other domains. Furthermore, even if predictive efficacy is established, it only supports its use as a screening signal and does not imply a causal relationship where 'compression leads to capability,' cautioning researchers against over-interpretation. `【Verified - Local: OPC-103 §3.6; AUTO-005 §5】` `【Verified - Local: AUTO-005 §2.3, §2.4】` `【Verified - Local: OPC-316 §3, §7; Inference】`

When evaluating the predictive efficacy of Large Language Model compression metrics, a series of common design pitfalls exist that, if not avoided, will severely compromise the reliability of research conclusions. For instance, relying solely on configuration-level cross-sectional correlation analysis will lead to upwardly biased results due to common causes (e.g., model scale); using `bits/token` instead of byte-level metrics for cross-model comparisons introduces confounding from `tokenizer` differences; and fitting and evaluating on the same data will overestimate predictive efficacy, leading to overfitting. These are among the most fundamental and pervasive errors in the evaluation process, requiring researchers to be highly vigilant.

To circumvent these pitfalls, the rigorous framework proposed in this study offers clear countermeasures: employing fixed-capacity exogenous perturbation designs and item-level `within-item` analysis to identify causal effects; consistently using byte-level metrics for cross-model comparisons; and implementing triple hold-outs (model, item, distribution) to achieve true extrapolation. Furthermore, decomposing `C_fact` and `C_other` to address the aggregation gap, introducing strong baselines and conducting nested model analysis to validate incremental validity, and ensuring research transparency and falsification resistance through preregistration and embedded falsification designs collectively form a robust defense, guaranteeing that the assessment of compression metrics' predictive efficacy is scientific, credible, and practically instructive.