The Illusion and Reality of Large Language Model 'Emergent Capabilities': A Falsifiable Scientific Experiment Design

The 'emergent capabilities' exhibited by large language models are a current hot topic in AI. Proposition P9 posits that even if a model's underlying capabilities (e.g., single-step accuracy) improve continuously and smoothly, when measured by nonlinear or threshold-based metrics (e.g., end-to-end accuracy for multi-step tasks), steep, seemingly sudden leaps may appear. For instance, if single-step accuracy increases from 0.95 to 0.99, for a 100-step task, the end-to-end accuracy would surge from approximately 0.6% to 36.6%, creating a strong impression of 'suddenly being able to do it'. Research categorizes emergence into metric emergence, mechanistic emergence, and robust capability leaps, emphasizing the need to discern their nature by switching to continuous metrics, increasing sample sizes, and conducting intervention analyses. [Verified, RQ-DB3 Report Lines 556–596 (P9)]

The core test objective of this study is the falsifiable hypothesis of Q5: whether observed capability leaps primarily stem from metric nonlinearity, floor effects, or insufficient sampling. This experimental design does not directly negate the original proposition P9 but rather tests its operationalized version (P9-op/Q5-op), i.e., whether a known smooth manipulation of capability along a controlled capability axis can produce significant apparent leaps through product-type/threshold-type metrics. It is crucial to emphasize that this design has clear boundaries: due to the lack of a trainable parameter scale ladder, the capability axis is re-calibrated as a 'controllable capability axis' rather than parameter scale; mechanistic probes are also resource-limited (no local weights), primarily downgraded to behavioral ablation; and this experiment explicitly does not claim to prove mathematical discontinuity or non-analytic phase transitions. [Verified, RQ-DB3 Report Lines 1049–1077 (Q5)]

To precisely identify the source of capability leaps, the experiment adheres to a core principle: for every 'item-capability level-repeated generation' combination, the model's output text is first frozen, and then all metrics are calculated in parallel on this identical frozen output. This method ensures that any differences between metrics are not caused by changes in the output text. The experimental design comprises two tracks: the first part aims to validate the sufficiency of metric or statistical explanations through a 'known smooth capability axis', serving as the primary efficacy track; the second part explores the actual capability leaps of existing models on a 'real resource axis', serving as an exploratory track.

The core design of the first part is the 'Prompt Information Quantity Knob' experiment. For each item, the model is progressively provided with more 'verified checkpoints' from the item's scoring rubric as prompt information, with ratios ranging from 0% to 100% (11 levels in total). This method of information injection is continuous, monotonic, dose-responsive, and does not alter the model's internal weights, thus its improvement in model capability is considered smooth. If, along this axis, step-by-step accuracy (Step) increases smoothly, while exact match accuracy (EM) exhibits a steep jump at a certain level, this directly validates the P9 mechanism, meaning the metric itself can create the illusion of emergence.

The second part focuses on examining capability performance in real-world settings with existing models and budgets. This includes exploring the impact of inference budget (e.g., limiting Chain-of-Thought token maximums), self-consistency sampling iterations (e.g., 1/2/4/8 samples taking the majority vote), and different model tiers (e.g., DeepSeek and gpt-6-astra) on task performance. The efficacy of this track is constrained by available resources, primarily serving for exploratory analysis and external validity verification, complementing the main track's validation of mechanistic sufficiency.

To comprehensively capture model capability changes, the experiment employs a suite of multi-dimensional metrics, all post-processed based on the frozen output text. These metrics include: Exact Match (EM, binary aggregation), Step-by-step Accuracy (Step, scored per checkpoint by rubric), Log Score (LS, average log-prob of output tokens, or calibrated ranking proxy), Continuous Error (CE, e.g., normalized edit distance or token-F1), and Business Threshold Pass Rate (PRτ, percentage of items with Step≥τ). These metrics are introduced to characterize model capability at different granularities and continuity levels, aiming to reveal how different metrics perceive the phenomenon of 'emergence'.

The core hypotheses of this experiment are divided into two camps: the null hypothesis H0 (H_art) asserts that observed capability leaps can be sufficiently explained by metric nonlinearity/thresholding, floor effects, and sampling noise. H0 is further subdivided into: H_M1a (discretization ratio R_disc≥3), H_M1b (leap magnitude monotonically increases with the number of item checkpoints k), H_M2 (floor-overcoming ratio F≥0.40), and H_M3 (leap differences between sample sizes N=41 and N≥200). The competing hypothesis H1 (H_rob) posits the existence of robust leaps that cannot be explained by the aforementioned factors, manifesting as multiple independent continuous metrics simultaneously exhibiting non-noise leaps around the same level, with preregistered smooth model predictions failing.

To ensure the rigor of statistical judgments, all statistics will undergo robustness evaluation. The magnitude of the leap and its occurrence level will be calculated via 'clustered bootstrapping' of items (resampling items as units, B=2000 iterations) to determine 95% percentile confidence intervals. Confidence intervals for the discretization ratio will be computed simultaneously; the floor-overcoming ratio will use the Wilson 95% confidence interval. To control for multiple comparisons, the 'max-statistic permutation null distribution' method will be employed, avoiding the overly conservative flaw of Bonferroni correction. These methods collectively form the statistical cornerstone for distinguishing genuine from spurious leaps. [Design Value]

Central to this experiment are its strictly preregistered thresholds, which were frozen and unchangeable before the experiment began. For instance, criteria supporting H0 include a discretization ratio R_disc≥3.0 with its bootstrap CI lower bound >1.5, a floor-overcoming ratio F≥0.40, and a ≥50% decrease in EM leap after removing floor items, as well as a monotonic increase in leap magnitude with the number of item steps k. The conditions for refuting H0 (i.e., supporting robust emergence) are even more stringent: at least two independent continuous metrics must show a leap magnitude J_m≥0.10 with highly consistent occurrence levels, while the preregistered smooth model's held-out error NRMSE_hold>10%. Furthermore, a human annotation Cohen κ≥0.70 serves as a prerequisite gate for main adjudication. [Design Value]

Sample size and statistical power are critical considerations in experimental design. The preliminary sample of N=41 items (D0) has insufficient power for binary metrics (e.g., EM), with a Minimum Detectable Effect (MDE) as high as approximately 30.9 percentage points, making it difficult to detect adjacent leaps of magnitude 15 percentage points. Therefore, confirmatory testing will primarily rely on the expanded D1 dataset (N≥200), where EM's MDE can be reduced to approximately 14.0 percentage points. Notably, continuous metrics (e.g., Step/LS/CE) are about 2-3 times more sensitive than binary metrics with the same sample size, which is the statistical rationale behind this design's emphasis on computing multi-dimensional continuous metrics. [Calculated Value]

The number of experiment repetitions (R=5) is primarily used to reduce sampling variance for the same item at the same capability level, rather than variance between items. Therefore, the main adjudication must be based on item-level paired or mixed-effects analysis to avoid incorrectly inflating degrees of freedom. The 'robust threshold' of 10% in the preregistered thresholds aligns with the 'relative error of excess loss prediction not exceeding 10%' in Q4 of RQ-DB3, while the minimum practical effect of 0.10 references the magnitude settings in Q3 and Q8, taking a more conservative median value. [Design Value; Report Lines 1005, 1174 Verified]

The datasets for this experiment include: D0 (41 frozen Arabic business items, serving as the main business endpoints and screening set, pending manual verification), D1 (machine-adjudicable augmented set, N≥200, expanded from D0 templates, used for confirmatory testing and sample size gating), and D2 (known smooth synthetic generator, used to provide null distribution and mechanistic positive controls). Additionally, historical outputs (D3) are used solely for exploratory analysis and do not participate in main adjudication, to avoid introducing uncontrolled confounding factors. All open-ended evaluations will be conducted by two independent blind raters, with Cohen κ≥0.70 serving as the agreement gate; otherwise, that metric will be excluded from main adjudication.

The rigor of the experiment is also reflected in the strict control of multiple variables. Parameters such as temperature, top-p, seed, system prompts, and templates are globally frozen and written into a configuration hash; item order is randomized within each level to control for position effects; raters are double-blinded and level labels are masked to reduce bias; model version drift will be monitored through reproduction tests at the start and end of the run; prompt injection will not be leaked to the scorer. The entire experimental process is divided into four phases: freezing (Phase 0), pre-experiment (Phase 1), main run (Phase 2), and analysis (Phase 3), with clear phase gates at each stage to ensure quality and progress.

The final adjudication will be a clear three-state outcome: support for H0 (metric/floor explanation), refutation of H0 (robust emergence), or insufficient evidence. Support for H0 requires M1a or M1b to be true, and at least one of M2 or M3 to be true, while robust criterion R is not true. Refutation of H0 requires at least two continuous metrics to show a leap magnitude ≥0.10 with highly consistent levels, preregistered smooth model prediction failure, passing annotation consistency, and the presence of intervenable behavioral changes. If none of the above conditions are met, the judgment will be 'insufficient evidence'. It is reiterated that even if H0 is refuted, it does not equate to proving a mathematically discontinuous phase transition.