Beware of Correlation Traps: The Indirect Relationship Between Large Model Capabilities and Compression Ratios

Before discussing the relationship between large model capabilities and compression rates, the primary task is to clarify the definition of 'reduced compression rate.' In the Chinese context, the term 'compression rate' carries at least two opposing meanings, directly impacting our judgment of the correlation's direction. In engineering terms, a higher compression ratio (CR = (1 - compressed_size/original_size) * 100%) indicates more aggressive compression, so a decrease in CR means weaker compression. However, this report and OPC internal research follow the convention of mainstream academia and information theory, using bits per character (BPC) or bits per byte (BPB) as compression metrics. Under this convention, a smaller BPC/BPB value indicates shorter data encoding, meaning better compression efficiency. Therefore, 'reduced compression rate' mentioned in this report always refers to a decrease in BPC/BPB, implying improved compression efficiency, which aligns with the direction of improved model capabilities.

Regardless of the definition used, our conclusion that 'correlation does not imply directness or causality' remains unaffected. The choice of definition only alters the sign of the correlation coefficient, not this core conclusion. Thus, in subsequent discussions, please always remember that 'reduced compression rate' is equivalent to 'improved compression efficiency'.

Evidence supporting a strong correlation between large model capabilities and compression efficiency primarily stems from two levels. First, there is a formal logical mathematical bridge: any excellent predictive model can be transformed into an efficient lossless compressor via arithmetic coding theory, where its codeword length is tightly linked to prediction loss; and vice versa. This indicates that 'predictive capability' and 'compression capability' are mathematically co-originating, and minimizing the log-loss of prediction is equivalent to minimizing the data's compression rate. This principle forms the solid foundation of the 'compression as intelligence' theory, but it primarily connects 'predictive capability' with 'compression capability,' rather than the comprehensive intelligence for downstream tasks.

Second, compelling evidence from empirical research is demonstrated in cross-model comparisons. Huang et al. (2024) conducted a comprehensive evaluation of the performance of 31 public large language models on 12 benchmark tasks and compared it with the compression efficiency (BPC) of these models on external raw corpora. The results showed a high, approximately linear correlation between the average downstream scores of models and compression efficiency, with Pearson correlation coefficients for various sub-domains (e.g., knowledge, code, mathematical reasoning) all reaching approximately -0.95. This finding prompted researchers to suggest that compression efficiency might serve as an unsupervised evaluation metric for large model capabilities. However, the study also emphasized the rigor of comparison, pointing out that different model Tokenizers affect BPC, thus necessitating a unified comparison using bits per character (BPC) and ensuring consistency in context length to avoid definitional contamination.

Despite the strong statistical correlation, we cannot simply interpret the improvement in large model capabilities and the reduction in compression rates as a direct causal relationship, as there are multiple underlying structural gaps. First is the 'identification gap': hidden 'capacity factors Z' such as large model capacity, training data volume, computational power, and training steps often simultaneously enhance both the model's compression capability (a>0) and its downstream task capability (b>0). This common driving effect (i.e., Z is a common cause for both compression and capability) leads to a systematic overestimation of the correlation between the two in observational studies, preventing us from independently identifying a causal path where 'compression directly leads to capability improvement.' As demonstrated by scaling laws, capabilities indeed improve according to power laws with respect to compute, data, and parameters, which precisely corroborates the role of Z in enhancing both.

Second is the 'aggregation gap': a model's compression rate (e.g., BPB) is typically a measure of the average encoding length in a text sequence, reflecting overall or average prediction efficiency. However, the correctness of many complex tasks is often 'conjunctive,' meaning that multiple critical pieces of information must be simultaneously correct for completion. For instance, the correct answer to a reasoning task might depend on the accurate prediction of specific fact-bearing bits F within N positions. Even if the average compression rate improves, there's no guarantee that the prediction accuracy of all critical fact-bearing bits will improve concurrently. OPC-103 research once constructed instances demonstrating that two models could have identical average BPB, yet exhibit vastly different task accuracies. This implies that an increase in average compression efficiency does not necessarily lead to a proportional increase in task accuracy, unless there is an additional assumption that loss reduction is uniformly distributed across all information bits.

Furthermore, there are the 'two-dimensional frontier' and 'extrapolation gap.' Under rate-distortion theory and the information bottleneck framework, there isn't a simple monotonic relationship between data compression rates and task-relevant effective information; rather, it resembles a two-dimensional trade-off frontier. Excessive compression, especially when a model compresses representations to a constant, certainly achieves maximum compression but also reduces task-relevant information (I(T;Y)) to zero. This illustrates that compression itself cannot create information; instead, it can destroy critical information required for a task. Moreover, current research primarily relies on cross-sectional analysis with limited samples, where model performance and compression rates are influenced by data distribution, model architecture, and training details. The observed correlation in specific samples and distributions cannot be unconditionally extrapolated to all situations, nor can it rule out the limitations of the 'No Free Lunch' theorem, which states that there is no universally optimal compression strategy for all tasks.

The assertion that 'improved large model capabilities are directly related to reduced compression rates' faces specific counter-examples and significant limitations in practical observations. The research by Delétang et al. (2023) introduced the concept of 'adjusted compression rate,' incorporating the model's parameter count into the final codeword length. They found that when the model size exceeds a certain critical point, the overall compression performance actually declines. This is because, while larger models might compress input data better, the vast encoding space occupied by their own massive parameter counts offsets the benefits gained from data compression. This finding directly challenges the simplistic linear hypothesis that 'the larger the model, the better it compresses and the smarter it is,' revealing that, under comprehensive consideration, capability improvement and compression rate amelioration are not unconditionally aligned.

Another strong counter-example comes from the field of model quantization. For instance, technologies like GPTQ can significantly compress the weights of large language models from traditional floating-point numbers (e.g., 16-bit) down to 3–4 bits, while reporting 'negligible' accuracy degradation compared to uncompressed baselines, with significant degradation only occurring at extreme compression levels (e.g., 2–3 bits). This phenomenon indicates that over a considerable range of compression, model capabilities can remain essentially constant, not declining proportionally with a significant reduction in 'bits per weight.' This strongly demonstrates that there is no simple, one-to-one functional relationship between model capabilities and weight compression rates; maintaining capability does not necessarily require a low compression rate, and vice versa. It also reminds us that 'capability' is not a single scalar metric; its definition often depends on specific benchmarks, evaluation metrics, and task compositions, making it difficult for a single compression rate to fully reflect its complexity.

If 'reduced compression rate' is understood as less compression applied to the model itself (e.g., higher weight bit-width, less pruning), the question then becomes, 'Does more aggressive model compression necessarily lead to a greater loss of capability?' The answer to this is also 'not directly related,' but rather a non-linear frontier curve highly dependent on the specific compression method and task nature. As previously mentioned, advanced quantization techniques (e.g., 4-bit weights) can substantially reduce model storage and computational requirements while maintaining near-lossless core capabilities. This indicates that within a certain range, model capabilities are not sensitive to changes in compression rates, and a significant 'capability-compression' decoupling can even be achieved.

However, once the compression level surpasses a certain threshold, the degradation of capabilities becomes rapid and non-uniform. Research indicates that under excessive compression, the model's generalization ability, recall of rare knowledge, and performance in long-chain reasoning are often affected first, rather than all tasks declining linearly in parallel. This highlights the characteristic of a 'non-linear, method-dependent frontier.' Therefore, in practice, we cannot rely on a single compression rate threshold to extrapolate model capabilities. Instead, we must assess the model's post-compression performance through testing on specific tasks and calibration sets, combined with particular application scenarios, to ensure it still meets expected performance requirements.

To elevate the relationship between large model capabilities and compression rates from a 'cross-sectional correlation' to a defensible 'direct causal' claim, experimental designs and causal inference methods far more rigorous than current observational studies are required. First, the core lies in introducing 'exogenous compression variation.' This means that while controlling all other variables, we would solely alter compression-related characteristics of the model—for instance, by adjusting the number and relevance of contextual examples or employing different quantization bit-widths (e.g., fp16, int8, int4) under the same model weights—to observe their impact on model capabilities. In this way, we can bypass confounding effects introduced by common causes such as model parameter size, training data, and computational power, thereby more purely identifying the role of compression itself.

Second, the granularity of analysis must descend from the macroscopic level of model configuration to the microscopic level of tasks or items. We need to conduct 'item-level within-item comparisons,' meaning paired t-tests or mixed-effects model analyses for the same task or item under different compression conditions, to capture the subtle effects of compression on specific task performance. Furthermore, one could attempt 'decomposition of compression bits,' by separating the total bits per character (BPB) into bits carrying critical factual information (C_fact) and bits carrying other non-critical information (C_other). This would directly test the 'aggregation gap' issue, which suggests that average compression improvement does not necessarily lead to an increase in the accuracy of critical information.

Finally, to ensure the robustness and credibility of research findings, strict scientific research paradigms must be followed. This includes 'pre-registration' of research hypotheses, interventions, evaluation metrics, and success/failure determination thresholds to avoid post-hoc bias. Concurrently, all findings should be tested on 'held-out models' or 'unseen distributions' independent of the training and development process to assess their 'extrapolability.' Only when these conditions are met, and after rigorous multi-dimensional, multi-level experimental validation, can we begin to discuss a potential 'direct correlation' or causal relationship between the two, rather than misinterpreting simple observational correlation as a functional, direct determination.