A New Approach to LLM Efficiency: Can Verification and Recovery Outperform the “Bigger Model” Paradigm?

The central proposition of the RQ-DB3 report stems from deep consideration of the reliability of complex AI workflows. The report states precisely on page 14 that the end-to-end success probability is a product of conditional probabilities, $P(\text{success})=\prod_{i=1}^{k}P(E_i\mid E_1,\dots,E_{i-1})$, and does not require the steps to be independent. Its cost objective is $J=\mathbb E[C_{\text{model}}+C_{\text{tools}}+C_{\text{human}}+C_{\text{retries}}+C_{\text{error consequences}}]$ [verified]. This provides a comprehensive cost accounting framework that highlights the hidden costs of errors and retries. The core of this experiment is the original falsifiable hypothesis in Q8: “At the same acceptance quality, the verification and recovery approach reduces the cost per qualified delivery by at least 15%” [verified; report file path: `/Users/zouyongming/metasystem/.../RQ-DB3-REPORT-2026-09-16.md`, lines 1139–1184].

This study explicitly rejects the strong version of the P14-op/Q8-op proposition: that, at the same acceptance quality, continuing to increase the single-step model score (that is, replacing it with a stronger model) is the most cost-effective way to achieve reliable end-to-end delivery. In other words, the core claim we seek to disprove is that investing most of the budget in model replacement is more cost-effective than investing in verification and targeted recovery [verified, report line 1182]. This rejection does not challenge mathematical theorems such as P14 themselves; it tests their effects in practical system implementations, such as potential false positives or false negatives in the verifier (P12) and possible net harm caused by recovery feedback (P13).

It is worth noting that this study does not reject the possibility that “more powerful models may still have an advantage on tasks where verification is infeasible or extremely costly,” nor does it generalize the experimental results to claim that “verification and recovery are generally superior across all tasks.” We preregistered the claim in the task summary that “verification and recovery are non-inferior to (or better than) more powerful models in cost per qualified delivery and completion rate without human intervention” as the primary confirmatory non-inferiority hypothesis, while treating the report’s original claim of “at least a 15% reduction in cost per qualified delivery” as a secondary superiority hypothesis. The two will be assessed and reported separately to strictly avoid confusion and misinterpretation.

This experiment systematically defines three core experimental arms: Arm $S$ (stronger model) uses gpt-6-astra for a five-stage, end-to-end, single-pass execution, with no independent verification or recovery; Arm $V$ (original model + verification) uses DeepSeek flash for the five stages and adds deterministic or semi-deterministic verifiers, with any verification rejection sent to a human; Arm $R$ (original model + verification and targeted recovery) builds on $V$ by making up to two targeted repair attempts for violations identified by the verifiers, escalating to a human if they still fail. An auxiliary arm, $V_0$ (original model, single-pass, no verification), is also included to isolate the marginal value of verification and recovery. Qualified Delivery (QD) is strictly defined as meeting the deterministic thresholds and frozen evaluation criteria, with no human takeover or only record-keeping actions permitted [design value].

The primary endpoints include cost per qualified delivery (CPQD), with cost $J$ decomposed according to the P14 formula into five separately recorded components: model tokens, tool tokens, human minutes, retry tokens, and error consequences; and the automation rate (Auto), defined as the rate of completion without human intervention. Secondary endpoints include completion time, the missed-error rate (the measured value of $\alpha$), the severity of error consequences passed by the validator, and the stratified calibration score (based on Theorem P12). The primary non-inferiority hypotheses specify that arm $R$ must cost no more than 10% more than $S$ ($\delta_C=0.10$), and its automation rate must not be more than 5 percentage points below $S$ ($\delta_A=0.05$) [design values]. There are also secondary superiority hypotheses and probes for net harm from recovery feedback.

The sample size is based on rigorous statistical calculations. For the autonomy rate (paired binary outcome, McNemar non-inferiority test), assuming a discordance rate of $\pi_d=0.20$ [estimate] and the specified non-inferiority margin, approximately 493 task instances are required. Allowing for the clustering design effect and invalid runs, the final confirmatory experiment uses $N=520$ task instances per arm [calculated value, design value]. The sample size requirement for cost is smaller, so the autonomy rate is the binding endpoint for this experiment. It is important to note that using only 41 frozen questions would provide insufficient statistical power: the minimum detectable effect would be approximately 30.9pp [calculated value], far greater than the specified $\delta_A=5\text{pp}$. Therefore, the 41 questions must not be used for the primary determination. Confidence intervals will be estimated using a task-family-clustered bootstrap ($B=10,000$), and the Holm method will control the family-wise error rate across the two primary endpoints [design value].

This experiment instantiates the five-stage process in proposition Q8 as an automatable Arabic business delivery pipeline, covering the full workflow from requirements parsing to final delivery. Each stage has a corresponding executable implementation and strict control points. For example, requirements are parsed into structured JSON by DeepSeek, with the extraction schema frozen; source materials are retrieved using deterministic keyword search, with the index snapshot frozen. Content generation, rule checks (including the `entry.py` and `quantity_gate.py` scripts [verified]), and delivery routing are also clearly defined. This instantiation ensures that the experiment operates under real business logic rather than as a purely theoretical exercise.

To ensure scientific rigor, all key variables are strictly controlled. The independent variable (the experimental arms) is assigned randomly using a paired design in which each instance runs in all four arms. The model version, prompt template (shared across all four arms, with its hash frozen), validator version and thresholds (with the actual $\alpha, \beta$ values measured), recovery budget (at most two targeted repair rounds, with feedback that does not include reference answers), retrieval strategy, task instances (randomized in family-based blocks), and run order and timestamp recording are all standardized and frozen to minimize the influence of confounding variables. The rules for triggering human intervention and its timing are also standardized; unrecorded human actions are prohibited to ensure complete cost accounting.

The experiment will be conducted in several stages: Stage 0 (freeze, approximately 5 business days [estimate]) will verify the data, freeze the scorer and thresholds, generate task instances, and preregister the study; Stage 1 (pilot) will calibrate the sample size using 60 instances, with results excluded from confirmatory analysis; Stage 2 (confirmatory) is the core stage, using 520 task instances; Stage 3 (out-of-time) will use a separate 100 instances to retest directional consistency. This staged implementation, combined with rigorous controls, is designed to ensure the reliability and generalizability of the experimental results.

The study dataset is divided into two main parts. D0 contains 41 frozen Arabic business questions [`ar_biz_eval_v1.jsonl` and `ar_biz_eval_v2.jsonl`; verified]. These questions are used only for three purposes: as validity anchors, cost anchors, and a contamination baseline; they are not used to test the primary hypothesis. Because both gpt-6-astra and DeepSeek flash reached saturation at 41/41 after human review [verified], these questions cannot effectively distinguish subtle differences between experimental arms, so they lack the power to support decisions at the 5 pp/10% level.

The primary evaluation set is the D1 pipeline validation set, comprising $N=520$ five-stage task instances [design value] derived by a preregistered, frozen grammar from the H2/H7 arithmetic contracts in `contracts.json` and the frozen glossary. The instances are generated from parameterized slots and grouped by customer/template family, ensuring that templates and values are not shared across families. This dataset is explicitly labeled as synthetic/semi-synthetic; its external validity is addressed through D0 anchoring and an out-of-time retest in Stage 3. An additional 60 instances are used for the pilot and 100 for out-of-time testing, with family-level separation to prevent data leakage.

The evaluation framework is centered on objectivity and reliability. All primary dependent variables (DVs) are evaluated through deterministic paths; for example, arithmetic and quantity checks are performed by `arithmetic_reference.py` and `quantity_gate.py` [verified]. LLM judges are used for rubric scoring, and 20% of the results are sampled for independent double-scoring by humans. The preregistered threshold requires Cohen’s κ ≥ 0.6; otherwise, the primary determination is downgraded to “insufficient evidence.” Verifier quality is reported using α and β values, as required by P14. If α > 0.40 or κ < 0.6, the relevant experimental arm is flagged as having low-quality verification. In addition, rigorous contamination and leakage audits ensure data integrity and experimental fairness through multiple measures, including n-gram/embedding overlap detection, canary comparisons, access-log recording, and audits of recovered feedback content.

Before any final conclusions can be drawn, the experimental results must pass a series of rigorous prerequisite checks. These include: validator quality ($\alpha\le0.40$), agreement between two human rubric raters (Cohen's κ≥0.6), completeness of the five-part cost accounting, passing the contamination audit, no model-version drift, no answer leakage during recovery, and a record showing that the confirmatory set was evaluated only once. If any prerequisite check fails, the entire experiment is downgraded to “insufficient evidence,” and no directional conclusions can be drawn, ensuring the quality of the experiment's setup and process.

If all prerequisite thresholds are met, the experimental results will be classified into three outcomes according to predefined quantitative criteria. For example, a result will be classified as “supported” (i.e., the strong version of P14-op is falsified and the weak version is supported) if it meets the following criteria: cost non-inferiority (the upper bound of the 95% CI for $\Delta_C$ is $<0.10$), autonomy-rate non-inferiority (the lower bound of the 95% CI for $\Delta_A$ is $>-0.05$), and secondary superiority ($\Delta_C\le-0.15$ and the CI upper bound is $<0$). It must also be confirmed that the direction of the effect is consistent out of time, and that arm $R$ significantly outperforms arm $V$, demonstrating the independent value of the recovery mechanism. Together, these criteria describe the ideal scenario in which the verification and recovery approach performs as well as or better than a stronger model on cost and autonomy rate, while achieving the 15% cost reduction proposed in the report.

Conversely, a finding of “reversal” (meaning that a stronger model dominates system redesign on this task family) requires a significant negative effect, such as significantly higher costs (the lower bound of the 95% CI for $\Delta_C$ is $>0.10$) or a significant decline in autonomy (the upper bound of the 95% CI for $\Delta_A$ is $<-0.05$). In addition, a finding of “reversal” may result if the control arm with injected deterministic error feedback, $R_e$, shows significant net harm, or if the verifier fails to detect a critical error. A finding of “insufficient evidence” is warranted if the preregistered gate is not passed, the confidence interval spans both the non-inferiority margin and 0, the out-of-time direction is inconsistent, or the observed disagreement rate $\pi_d>0.25$ leaves statistical power inadequate. This rigorous three-outcome decision rule ensures clear, scientifically grounded conclusions, avoids an ambiguous middle ground, and distinguishes between situations such as “evidence of no practical benefit” and “insufficient sample size.”

The experiment’s resource budget is primarily measured in token usage and human time. Based on experience with the G9 matched-prompt comparison, a single gpt-6-astra response used 18,995 tokens over 73.8 seconds, averaging about 463 tokens per item [verified]. Given that this experiment’s five stages are far more complex than a single question-and-answer exchange, we estimate per-instance token usage at about 30,000 tokens for arm $S$ (astra), 14,000 tokens for arm $V$ (DeepSeek), and 17,000 tokens for arm $R$ (including 1.5 rounds of targeted recovery for approximately 30% of instances) [estimate]. Across 680 task instances in the pilot, confirmation, and out-of-time tests, total token usage is estimated at about 47 million [estimate].

In terms of time, running all five stages for a single instance takes approximately 2–5 minutes. For around 2,100 to 2,700 arm-level runs, serial computation would require 105–140 machine hours; using four-way parallelization would reduce this to 26–35 hours [estimate]. In addition, the manual double-review process is expected to take 520 instances × 2 reviewers × 3 minutes per instance, for a total of approximately 52 hours. Including the Stage 0 data freeze and the out-of-time test in Stage 3, the full experiment is expected to take 6–8 weeks, consistent with the scheduling estimate in section Q8 of the RQ-DB3 report [verified, report line 1141].

For monetary costs, this study will strictly follow the explicit rule in `runtime-ledger/status.json`: “cash_cost = unknown unless backed by a billing receipt; token counts are not money” [verified]. Accordingly, this design will primarily track costs in tokens and human minutes. Monetary conversions will be used only for sensitivity analysis once verifiable billing rates are available; no unit prices will be fabricated. This ensures that cost accounting remains transparent and objective, avoiding bias from assumed monetary costs.

This preregistered experiment aims to provide empirical evidence for systematic optimization strategies that go beyond the traditional “model upgrade” approach. If the results support the validation and recoverysolution, they would suggest that enterprises should reconsider how they allocate resources for LLM deployment and operations, shifting from simply pursuing larger models to placing greater emphasis on end-to-end process robustness and cost-effectiveness. The value of this research lies not only in quantifying the efficiency gains from validation and recovery, but also in offering a new framework for designing future LLM applications: building more resilient, cost-controlled AI systems through fine-grained process control and error-handling mechanisms.

Whether the experimental results support the hypothesis, refute it, or remain inconclusive, they will offer valuable insights to LLM strategy makers. If verification-and-recovery strategies prove significantly more effective, we recommend that enterprises prioritize investment in high-quality verifiers, intelligent error-feedback loops, and efficient targeted repair tools, and integrate these capabilities into their core AI delivery pipelines. Even if “stronger models” prove superior, the study’s rigorous methodology and detailed cost accounting will provide a benchmark for assessing the true return on investment of model upgrades and reveal the limitations of current verification techniques in specific scenarios. The ultimate goal is to help move LLMs from laboratory concepts to reliable, cost-effective, large-scale enterprise applications.