G1-G10 Scoring System: A 'Bottleneck Detector' for Research Maturity, Not a 'Performance Scorecard'

The G1-G10 scoring system (rubric.json v1, 2026-09-14T11:36 UTC [verified, first-hand local source]) aims to measure the 'deliverable system maturity acceptance score,' composed of 10 equally-weighted indicators, each worth 10 points, totaling 100 points. The current system score is 30/100, indicating that G1, G2, and G3 have been met, while G4-G10 remain unverified. This design rejects the traditional assumption of treating scores as a singular measure of capability or an industry benchmark ranking.

Its core design philosophy includes: First, the adoption of 10x10 equally-weighted indicators is not statistical weighting but a strategic managerial choice, aimed at avoiding disputes over weights and clearly identifying shortcomings through a binary system for each item (met/unverified/not_met), making it more of a 'bottleneck detector' than a 'weighted total score.' [Verified, public literature] MDPI points out that the final score of composite indicators depends on weighting, normalization, and aggregation methods, with equal weighting being one of them.

Second, establishing a rigorous validation path. Each score must be linked to evidence, and the rubric's hash value (rubric_sha256=4be4d5be... [verified, first-hand local source]) is frozen to ensure immutable rules. The score.py script only checks the rule hash, ID uniqueness, item count, status enumeration, and existence of evidence fields, explicitly stating 'no independent judgment of evidence.' For example, G2/G3 validation involves hash freezing, offline comparison, and independent recalculation; G1 confirms the model and input via session metadata (rollout jsonl [verified, first-hand local source]).

Third, introducing an anti-gaming checklist, explicitly stating that actions such as 'starting an agent,' 'generating documentation,' or 'scoring full marks on synthetic problems' cannot improve G4-G10 scores. Concurrently, revised standards must be version-upgraded, and old scores preserved, to prevent retroactive changes. For instance, G7 requires 'the core research process to run stably for 7 consecutive days' and 'an additional 30 days of continuous observation,' emphasizing long-term stability over single-instance success. [verified, first-hand local source]

Despite its intricate design, the G1-G10 system still presents significant limitations. The equal weighting design easily conceals differences in upper limits; for instance, G1 (single successful call) and G6 (business revenue) both score 10 points, yet their marginal values are clearly unequal, potentially incentivizing a 'prioritization of low-difficulty, easy items' behavior. Furthermore, the low resolution of only 10 binary scores (met/unverified/not_met), lacking partial points, Minimum Detectable Effect (MDE), or confidence intervals, means the scores do not provide a continuous signal, making it difficult to capture subtle progress. [Inference] This implies that the difference between 30 and 40 points might merely stem from a single item switching status, rather than a significant leap in capability level.

The non-independence of the validation process is also a critical challenge. score.py explicitly states that it does not judge evidence; evidence is primarily provided and recalculated by the controlling party, with its 'independence' limited to programmatic hash and count. For instance, the independent recalculation for G2 is still performed by the controlling party, which limits the impartiality and third-party credibility of the validation chain. Furthermore, the rubrics and observations for 9 existing workspaces are byte-for-byte identical [verified, first-hand local source], suggesting that scores might be copied and propagated rather than independently calculated, posing a risk of 'score proliferation with repositories.'

The seven-day stability window's sample size is too small, covering only a single environment (Mac online), lacking randomization and MDE support, making it difficult to distinguish 'true stability' from 'just happened not to break.' [Verified, public literature] BMC research indicates that reliability metrics are unstable with small sample sizes. This could lead to construct redundancy between G7 and G8 (stability and recovery closed-loop), and G5/G6 (adoption and revenue). Goodhart's Law states that when a measure becomes a target, it ceases to be a good measure. Items like G1-G3, which are easily formalized and met, might become targets for internal teams to 'game the score,' while truly valuable items G4-G6 are neglected due to higher difficulty, potentially causing incentives to stagnate at 30 points.

Finally, although the rubric explicitly forbids equating 'unverified' with 'zero points,' a total score of 30/100 might still be misinterpreted during dissemination as 'only 30% capability.' This pragmatic misinterpretation risk could distort the system designers' original intent, affecting its trustworthiness and adoption within and outside the organization.

The G1-G10 system anchors its scoring to 'downstream adoption' and 'quantifiable revenue' (G5/G6), which is highly consistent with Research OPC's 'pay-for-results' business model (FR-04/FR-15 [verified, first-hand local source]). The current state of zero commercial adoption and zero cash cost honestly reveals that OPC's business closed-loop has not yet formed. This transparency helps identify actual bottlenecks in commercialization and drives research closer to practical application scenarios. [Inference] This will force researchers to re-evaluate the ultimate value of their work.

At the organizational level, the introduction of G7 (operational stability), G8 (recovery), G9 (peer comparison), and G10 (cost ledger) signifies the integration of operations, auditing, and accounting capabilities into the 'definition of capabilities' for research organizations. This marks a shift from the traditional 'productivity is king' to 'verifiability-driven' approach. For 'one-person companies' or small research teams, the bottleneck will shift from mere output capability to ensuring the long-term stability, reproducibility, and economic traceability of their results.

For individual researchers, G9's requirement for 'frozen peer comparison' and G10's requirement for a 'complete cost ledger' transform self-improvement from subjective narrative into falsifiable experimentation. Research is no longer merely about publishing papers or developing models; it also requires proving stability, cost-effectiveness, and reproducibility. This undoubtedly increases the management burden but also enhances the rigor and credibility of research, encouraging researchers to cultivate a more comprehensive engineering and business mindset.

To enhance the utility and credibility of the G1-G10 system, we propose the following recommendations: First, equally weighted scores should only serve as an internal 'bottleneck detection' threshold; external reports must clearly state the denominator and 'unassessed' items. Concurrently, 'difficulty/value weights' or stratification (baseline gate → outcome gate) should be added for critical outcome items such as G4-G6, to prevent them from being equally weighted with easily achievable items, thus forming a dual-track mechanism of 'equally-weighted gates' and 'weighted outcome scores,' which run in parallel but are not interchangeable.

Second, pre-registering MDE (Minimum Detectable Effect) and sample size for each indicator. For example, G7 should specify 'expected number of runs N within a 7-day window, maximum allowed failure rate,' and 'rules for excluding planned downtime.' G9's peer comparison should pre-register control groups, blinding sequence, and inter-rater reliability. This will significantly enhance the statistical rigor and credibility of validation results, reducing uncertainty caused by small sample sizes.

Third, separating 'evidence judgment' from the controlling party. Introduce a second independent reviewer or an automated evidence gate (e.g., blockchain receipts, downstream system receipts), and consider Brier calibration or kappa coefficient to assess consistency. Fourth, strengthen cost accounting, elevating G10 from 'complete' to 'billable' (covering token, cash, human resources, and caller costs), and explicitly marking unknown costs as null rather than zero, laying the groundwork for subsequent unit-outcome cost analysis.

Finally, strictly prohibit the replication and dissemination of scores. Any new workspace must recalculate observations and include its own evidence, rather than simply copying existing scores (a no-copy assertion should be added for the 9 identical existing observations). Concurrently, adopt and improve the change_policy, ensuring that each standard upgrade retains old scores and old rubric hashes, establishing a complete rubric version ledger to guarantee the traceability and transparency of historical data.