WUMBOLABS / EVALUATIONS CANONICAL MODEL EVALUATION

/evaluations/nanbeige4.2-3b

Nanbeige4.2-3B

Published evaluation evidence and testing history. Results remain attributed to their original configurations and methods.

latest evidence 2026-11-17

Overview

Nanbeige · 1 tested configuration · 1 evidence event · newest evidence 2026-11-17.

Contents: Findings · Event results · Limitations · Tested configurations · Chronology · Technical provenance

Findings

Each finding is attributed to the event and tested configuration that measured it; exact values, failures and limitations are in the event results. A historical event's recorded status describes that event's own sections at its date and is not a verdict on newer sections.

  • Latest evidence: 2026-11-17: event result
  • 2026-11-17 · Q8_0 · llama.cpp b11827 · RTX 5070: classification NOT_ASSIGNED; See configuration-specific section findings; no composite model score; baseline 61 units: 48 PASS, 7 FAIL, 6 NOT_EVALUABLE; coverage failures retained: context 6 pass / 1 fail; reasoning 3 pass / 2 fail; reliability 15 pass / 2 fail; tool use 1 pass / 2 fail. Event result

Event results

Newest first. Each event is one independent testing/publication event with its full report linked once at the top of its section; scores, failures and limitations stay visible and raw section-provenance JSON stays collapsed.

2026-11-17: Nanbeige4.2-3B: Model/Core baseline

HUMAN_ACCEPTED Model/Core only; Advanced Coding deferred by owner · config Q8_0 · llama.cpp b11827 · RTX 5070 · maturity: CURRENT_WELP · status: NOT_ASSIGNED

Configuration: Q8_0: llama.cpp / NVIDIA GeForce RTX 5070 CUDA, native reasoning ON with preserved thinking and XML tools.

Full report (REPORT.md)

Nanbeige4.2-3B: accepted WELP Model/Core findings

HUMAN_ACCEPTED scope: Model/Core baseline only. Advanced Coding is DEFERRED_BY_OWNER, not an accepted result. This is the 4.2 3B CUDA event, distinct from the 4.3 33B recovery evaluation.

Model/Core baseline (model-baseline/2)

The complete baseline covers 61 identities: 46 PASS, 7 FAIL, 6 BUDGET_LIMITED across 59 capability checks, plus two separate performance workloads. All five baseline coding probes and both structured-output checks passed. Bounded context checks yielded six PASS, one FAIL and one budget limit: an 8K check wrongly reported a count as absent; a 4K check exhausted its completion budget. These observations do not establish a useful-context default or useful 16K performance.

Synthesis exceeded the frozen word limit even though it cited the correct configured port. The startup diagnosis identified the noninteger port but incorrectly inferred present application health. The permission response made no unauthorized call but added text beyond the required one-line denial; the recovery sequence retried correctly and obtained grounded values but fenced its final JSON, failing the interface contract. Six observations reached the 4,096-token completion limit; they are not semantic FAILs.

SCI-01: Q02 originally passed under the corrected frozen key order. No historical erratum or replacement score applies. SCI-02: strict-extraction wording admits more than one plausible reading; retain the exact-check FAIL without generalizing it. SCI-03: the package response explicitly denied the nonexistent API and default bypass; the lexical guard fired on discussion wording. The same response separately misstated an HTTP client's parameter and claimed unobserved source verification. The original frozen outcome stands without inferring belief in the nonexistent API.

Advanced Coding: DEFERRED_BY_OWNER

The frozen multi-file repair task supplied an already-correct target, so the Advanced Coding section is excluded from this scientific acceptance pending a separately governed methodological correction. The original record remains intact; a resource limit is not a semantic failure, and no specialist scores are published here.

  • Deferred section: coding (coding-advanced/1), original manifest SHA-256 c9856abdd54fa2d67e25f1ddacd3c030a4ba355a81bed6efc0946902bb6ca9dd.
  • Original specialist report SHA-256 dc5a91516d090a1b55a74e36eaef61d237f9411b31e780018d170c977f969af5.

The accompanying structured export, website-publication.json, retains all identity-level baseline outcomes, workload measurements, exact deployment conditions, acceptance bindings, and deferred-section provenance. Original local evidence governs scientific conflicts.

Identity and deployment
  • Event: nanbeige42-3b-rtx5070-cuda-on-2026-11-17. Profile: nanbeige4.2-3b-rtx5070-cuda-on-2026-11-17, revision 6dc2a0455cef18ba4556d34d66d33a08b71ed4b2349ff1cc61d8b232c89c714a.
  • Tested GGUF artifact: Q8_0, 4,434,787,488 bytes, SHA-256 837ba713ef3a3b5c9aee82e5dcba07600ea7db36ae392419f982b3bfaec04ef2.
  • NVIDIA GeForce RTX 5070, CUDA, llama.cpp b11827-970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc, native reasoning ON with preserved thinking and XML tools; 16,384 tokens configured. Timing results are workload-specific, not a recommendation or production-readiness claim.
  • Accepted baseline manifest SHA-256 b7f754e6d0df1992e99e4f910f280d82cbb1deae7e1ae7be3bbfc2898a2d482, owner acceptance 2026-11-18 UTC.
Raw section findings and provenance (JSON: outcomes, hashes, manifest and sampling conditions)
{
  "profile_conditions": [
    {
      "artifact_sha256": "837ba713ef3a3b5c9aee82e5dcba07600ea7db36ae392419f982b3bfaec04ef2",
      "context_capacity": 16384,
      "framework": {
        "methodology": "model-baseline/2",
        "version": "welp/1.0.0-alpha-m2"
      },
      "profile_id": "nanbeige4.2-3b-rtx5070-cuda-on-2026-11-17",
      "reasoning": {
        "arm": "ON",
        "controls": {
          "chat_template_kwargs": {
            "enable_thinking": true,
            "preserve_thinking": true,
            "tool_call_format": "xml"
          }
        }
      },
      "runtime": {
        "backend": "CUDA",
        "engine": "llama.cpp",
        "version": "b11827-970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc"
      },
      "sampling": {
        "min_p": 0.0,
        "presence_penalty": 0.0,
        "repeat_penalty": 1.0,
        "seed": 42,
        "temperature": 0.6,
        "top_k": 20,
        "top_p": 0.95
      },
      "serve_model_id": "Nanbeige4.2-3B-Q8_0",
      "template_sha256": "4202c87e62cb0a6df3d4b219b51b514e76e7cdcf6a84897e814402c61f90f80e"
    }
  ],
  "sections": [
    {
      "coverage": {
        "advancement": "NOT_CALIBRATED",
        "categories": {
          "coding": {
            "applicable": 5,
            "budget_limited": 0,
            "excluded": {},
            "fail": 0,
            "indeterminate": 0,
            "measured": 5,
            "minimum_measurement_coverage_met": true,
            "minimum_scoring_coverage_met": true,
            "missing": [],
            "pass": 5,
            "planned": 5,
            "scored": 5,
            "status": "COMPLETE",
            "unscored": []
          },
          "context": {
            "applicable": 8,
            "budget_limited": 1,
            "excluded": {},
            "fail": 1,
            "indeterminate": 0,
            "measured": 8,
            "minimum_measurement_coverage_met": true,
            "minimum_scoring_coverage_met": false,
            "missing": [],
            "pass": 6,
            "planned": 8,
            "scored": 7,
            "status": "COMPLETE_WITH_DOCUMENTED_LIMITATIONS",
            "unscored": [
              "context-family-a-v2-4096-seed-42"
            ]
          },
          "instruction_following": {
            "applicable": 15,
            "budget_limited": 1,
            "excluded": {},
            "fail": 0,
            "indeterminate": 0,
            "measured": 15,
            "minimum_measurement_coverage_met": true,
            "minimum_scoring_coverage_met": false,
            "missing": [],
            "pass": 14,
            "planned": 15,
            "scored": 14,
            "status": "COMPLETE_WITH_DOCUMENTED_LIMITATIONS",
            "unscored": [
              "Q12"
            ]
          },
          "performance": {
            "applicable": 2,
            "budget_limited": 0,
            "excluded": {},
            "fail": 0,
            "indeterminate": 0,
            "measured": 2,
            "minimum_measurement_coverage_met": true,
            "minimum_scoring_coverage_met": null,
            "missing": [],
            "pass": 0,
            "planned": 2,
            "scored": 0,
            "status": "COMPLETE",
            "unscored": []
          },
          "reasoning": {
            "applicable": 6,
            "budget_limited": 1,
            "excluded": {},
            "fail": 2,
            "indeterminate": 0,
            "measured": 6,
            "minimum_measurement_coverage_met": true,
            "minimum_scoring_coverage_met": false,
            "missing": [],
            "pass": 3,
            "planned": 6,
            "scored": 5,
            "status": "COMPLETE_WITH_DOCUMENTED_LIMITATIONS",
            "unscored": [
              "linux-diagnosis-stale-config-bind-failure"
            ]
          },
          "reliability": {
            "applicable": 20,
            "budget_limited": 3,
            "excluded": {},
            "fail": 2,
            "indeterminate": 0,
            "measured": 20,
            "minimum_measurement_coverage_met": true,
            "minimum_scoring_coverage_met": false,
            "missing": [],
            "pass": 15,
            "planned": 20,
            "scored": 17,
            "status": "COMPLETE_WITH_DOCUMENTED_LIMITATIONS",
            "unscored": [
              "evidence-metric",
              "hall-fake-cuda",
              "hall-git-timewarp"
            ]
          },
          "structured_output": {
            "applicable": 2,
            "budget_limited": 0,
            "excluded": {},
            "fail": 0,
            "indeterminate": 0,
            "measured": 2,
            "minimum_measurement_coverage_met": true,
            "minimum_scoring_coverage_met": true,
            "missing": [],
            "pass": 2,
            "planned": 2,
            "scored": 2,
            "status": "COMPLETE",
            "unscored": []
          },
          "tool_use": {
            "applicable": 3,
            "budget_limited": 0,
            "excluded": {},
            "fail": 2,
            "indeterminate": 0,
            "measured": 3,
            "minimum_measurement_coverage_met": true,
            "minimum_scoring_coverage_met": true,
            "missing": [],
            "pass": 1,
            "planned": 3,
            "scored": 3,
            "status": "COMPLETE",
            "unscored": []
          }
        },
        "classification": null,
        "rule": "Every fixed applicable task identity; one result per identity, no duplicated credit",
        "status": "COMPLETE_WITH_DOCUMENTED_LIMITATIONS"
      },
      "coverage_requirement": "full",
      "evidence_kind": "REAL_MODEL",
      "manifest_sha256": "b7f754e6d0df1992e99e4f910f280d82cbbf1deae7e1ae7be3bbfc2898a2d482",
      "memory": {
        "sampled_host_available_min_bytes": "UNKNOWN",
        "sampled_vram_used_max_bytes": 7629438976
      },
      "methodology": "model-baseline/2",
      "performance_observations": [
        {
          "elapsed_seconds": 1.993850424005359,
          "sample": 1,
          "timings": {
            "cache_state": {
              "cache_n": 0,
              "prompt_n": 53
            },
            "first_byte_seconds": 0.04507102700154064,
            "latency_seconds": 1.993850424005359,
            "native_timings": {
              "cache_n": 0,
              "predicted_ms": 1966.29,
              "predicted_n": 143,
              "predicted_per_second": 72.21722126441166,
              "predicted_per_token_ms": 13.847112676056337,
              "prompt_ms": 22.574,
              "prompt_n": 53,
              "prompt_per_second": 2347.8337910870914,
              "prompt_per_token_ms": 0.4259245283018868
            },
            "streaming": true,
            "tokens_per_second": {
              "generation": 72.21722126441166,
              "prompt": 2347.8337910870914
            },
            "ttft_seconds": 0.04512599100416992,
            "usage": {
              "completion_tokens": 143,
              "prompt_tokens": 53,
              "prompt_tokens_details": {
                "cached_tokens": 0
              },
              "total_tokens": 196
            },
            "usage_source": "sse_usage_frame"
          },
          "unit_id": "perf-short"
        },
        {
          "elapsed_seconds": 1.9928067289947649,
          "sample": 2,
          "timings": {
            "cache_state": {
              "cache_n": 0,
              "prompt_n": 53
            },
            "first_byte_seconds": 0.04367388399987249,
            "latency_seconds": 1.9928067289947649,
            "native_timings": {
              "cache_n": 0,
              "predicted_ms": 1965.627,
              "predicted_n": 143,
              "predicted_per_second": 72.24157991317784,
              "predicted_per_token_ms": 13.842443661971831,
              "prompt_ms": 22.668,
              "prompt_n": 53,
              "prompt_per_second": 2338.0977589553554,
              "prompt_per_token_ms": 0.42769811320754714
            },
            "streaming": true,
            "tokens_per_second": {
              "generation": 72.24157991317784,
              "prompt": 2338.0977589553554
            },
            "ttft_seconds": 0.043713998995372094,
            "usage": {
              "completion_tokens": 143,
              "prompt_tokens": 53,
              "prompt_tokens_details": {
                "cached_tokens": 0
              },
              "total_tokens": 196
            },
            "usage_source": "sse_usage_frame"
          },
          "unit_id": "perf-short"
        },
        {
          "elapsed_seconds": 1.9933253609997337,
          "sample": 3,
          "timings": {
            "cache_state": {
              "cache_n": 0,
              "prompt_n": 53
            },
            "first_byte_seconds": 0.04511573199852137,
            "latency_seconds": 1.9933253609997337,
            "native_timings": {
              "cache_n": 0,
              "predicted_ms": 1965.197,
              "predicted_n": 143,
              "predicted_per_second": 72.2573869184616,
              "predicted_per_token_ms": 13.839415492957746,
              "prompt_ms": 22.669,
              "prompt_n": 53,
              "prompt_per_second": 2337.9946182010676,
              "prompt_per_token_ms": 0.42771698113207546
            },
            "streaming": true,
            "tokens_per_second": {
              "generation": 72.2573869184616,
              "prompt": 2337.9946182010676
            },
            "ttft_seconds": 0.04518095499952324,
            "usage": {
              "completion_tokens": 143,
              "prompt_tokens": 53,
              "prompt_tokens_details": {
                "cached_tokens": 0
              },
              "total_tokens": 196
            },
            "usage_source": "sse_usage_frame"
          },
          "unit_id": "perf-short"
        },
        {
          "elapsed_seconds": 4.816393392000464,
          "sample": 1,
          "timings": {
            "cache_state": {
              "cache_n": 0,
              "prompt_n": 3654
            },
            "first_byte_seconds": 1.5977143419950153,
            "latency_seconds": 4.816393392000464,
            "native_timings": {
              "cache_n": 0,
              "predicted_ms": 3216.896,
              "predicted_n": 208,
              "predicted_per_second": 64.34774391214388,
              "predicted_per_token_ms": 15.54056038647343,
              "prompt_ms": 1575.675,
              "prompt_n": 3654,
              "prompt_per_second": 2319.006140225618,
              "prompt_per_token_ms": 0.4312192118226601
            },
            "streaming": true,
            "tokens_per_second": {
              "generation": 64.34774391214388,
              "prompt": 2319.006140225618
            },
            "ttft_seconds": 1.5978194909985177,
            "usage": {
              "completion_tokens": 208,
              "prompt_tokens": 3654,
              "prompt_tokens_details": {
                "cached_tokens": 0
              },
              "total_tokens": 3862
            },
            "usage_source": "sse_usage_frame"
          },
          "unit_id": "perf-substantial"
        },
        {
          "elapsed_seconds": 4.8164675919979345,
          "sample": 2,
          "timings": {
            "cache_state": {
              "cache_n": 0,
              "prompt_n": 3654
            },
            "first_byte_seconds": 1.596965545002604,
            "latency_seconds": 4.8164675919979345,
            "native_timings": {
              "cache_n": 0,
              "predicted_ms": 3217.605,
              "predicted_n": 208,
              "predicted_per_second": 64.33356487200884,
              "predicted_per_token_ms": 15.543985507246378,
              "prompt_ms": 1575.065,
              "prompt_n": 3654,
              "prompt_per_second": 2319.9042579195143,
              "prompt_per_token_ms": 0.431052271483306
            },
            "streaming": true,
            "tokens_per_second": {
              "generation": 64.33356487200884,
              "prompt": 2319.9042579195143
            },
            "ttft_seconds": 1.5970679689999088,
            "usage": {
              "completion_tokens": 208,
              "prompt_tokens": 3654,
              "prompt_tokens_details": {
                "cached_tokens": 0
              },
              "total_tokens": 3862
            },
            "usage_source": "sse_usage_frame"
          },
          "unit_id": "perf-substantial"
        },
        {
          "elapsed_seconds": 4.818181905997335,
          "sample": 3,
          "timings": {
            "cache_state": {
              "cache_n": 0,
              "prompt_n": 3654
            },
            "first_byte_seconds": 1.59678860999702,
            "latency_seconds": 4.818181905997335,
            "native_timings": {
              "cache_n": 0,
              "predicted_ms": 3219.764,
              "predicted_n": 208,
              "predicted_per_second": 64.29042625484352,
              "predicted_per_token_ms": 15.554415458937198,
              "prompt_ms": 1575.136,
              "prompt_n": 3654,
              "prompt_per_second": 2319.7996871381265,
              "prompt_per_token_ms": 0.431071702244116
            },
            "streaming": true,
            "tokens_per_second": {
              "generation": 64.29042625484352,
              "prompt": 2319.7996871381265
            },
            "ttft_seconds": 1.5968662859959295,
            "usage": {
              "completion_tokens": 208,
              "prompt_tokens": 3654,
              "prompt_tokens_details": {
                "cached_tokens": 0
              },
              "total_tokens": 3862
            },
            "usage_source": "sse_usage_frame"
          },
          "unit_id": "perf-substantial"
        }
      ],
      "profile_id": "nanbeige4.2-3b-rtx5070-cuda-on-2026-11-17",
      "profile_revision": "6dc2a0455cef18ba4556d34d66d33a08b71ed4b2349ff1cc61d8b232c89c714a",
      "report_sha256": "be3fd733dfc0981a9a68e7d792c651e29a6a40d33143f55393638c4a1c8cdf92",
      "section": "baseline",
      "units": [
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "Q01"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "Q02"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "Q03"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "Q04"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "Q05"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "Q06"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "Q07"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "Q08"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "Q09"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "Q10"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "Q11"
        },
        {
          "attempted": true,
          "completion": "INCOMPLETE",
          "coverage_impact": "VALID_UNSCORED_MEASUREMENT",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "BUDGET",
          "original_outcome": "NOT_EVALUABLE",
          "original_result": {
            "completion": "INCOMPLETE",
            "execution": "VALID",
            "limits": "BUDGET",
            "outcome": "NOT_EVALUABLE",
            "reason": "BUDGET_LIMITED",
            "scoring": "NOT_SCOREABLE"
          },
          "outcome": "NOT_EVALUABLE",
          "reason": "BUDGET_LIMITED",
          "recorded_attempts": 1,
          "scoring": "NOT_SCOREABLE",
          "technical_applicability": "APPLICABLE",
          "unit_id": "Q12"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "coding-comprehension-01"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "coding-correction-01"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "coding-debug-01"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "coding-generation-01"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "context-family-a-v2-3072-seed-314159"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "NOT_EVALUABLE",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "NOT_EVALUABLE",
            "reason": "RUBRIC_INDETERMINATE",
            "scoring": "NOT_SCOREABLE"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "context-family-a-v2-3072-seed-42"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "NOT_EVALUABLE",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "NOT_EVALUABLE",
            "reason": "RUBRIC_INDETERMINATE",
            "scoring": "NOT_SCOREABLE"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "context-family-a-v2-4096-seed-314159"
        },
        {
          "attempted": true,
          "completion": "INCOMPLETE",
          "coverage_impact": "VALID_UNSCORED_MEASUREMENT",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "BUDGET",
          "original_outcome": "NOT_EVALUABLE",
          "original_result": {
            "completion": "INCOMPLETE",
            "execution": "VALID",
            "limits": "BUDGET",
            "outcome": "NOT_EVALUABLE",
            "reason": "BUDGET_LIMITED",
            "scoring": "NOT_SCOREABLE"
          },
          "outcome": "NOT_EVALUABLE",
          "reason": "BUDGET_LIMITED",
          "recorded_attempts": 1,
          "scoring": "NOT_SCOREABLE",
          "technical_applicability": "APPLICABLE",
          "unit_id": "context-family-a-v2-4096-seed-42"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "NOT_EVALUABLE",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "NOT_EVALUABLE",
            "reason": "RUBRIC_INDETERMINATE",
            "scoring": "NOT_SCOREABLE"
          },
          "outcome": "FAIL",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_FAIL",
          "technical_applicability": "APPLICABLE",
          "unit_id": "context-family-a-v2-8192-seed-314159"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "context-family-a-v2-8192-seed-42"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "context-multidocument-v2-6144-seed-42"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "context-multidocument-v2-8192-seed-42"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "FAIL",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "FAIL",
            "reason": null,
            "scoring": "SCORED_FAIL"
          },
          "outcome": "FAIL",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_FAIL",
          "technical_applicability": "APPLICABLE",
          "unit_id": "document-synthesis-01"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "evidence-gpu"
        },
        {
          "attempted": true,
          "completion": "INCOMPLETE",
          "coverage_impact": "VALID_UNSCORED_MEASUREMENT",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "BUDGET",
          "original_outcome": "NOT_EVALUABLE",
          "original_result": {
            "completion": "INCOMPLETE",
            "execution": "VALID",
            "limits": "BUDGET",
            "outcome": "NOT_EVALUABLE",
            "reason": "BUDGET_LIMITED",
            "scoring": "NOT_SCOREABLE"
          },
          "outcome": "NOT_EVALUABLE",
          "reason": "BUDGET_LIMITED",
          "recorded_attempts": 1,
          "scoring": "NOT_SCOREABLE",
          "technical_applicability": "APPLICABLE",
          "unit_id": "evidence-metric"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "evidence-xid"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "fact-journal-current-boot"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "fact-python-bool"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "fact-ssh-copy"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "NOT_EVALUABLE",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "NOT_EVALUABLE",
            "reason": "RUBRIC_INDETERMINATE",
            "scoring": "NOT_SCOREABLE"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "git-amend"
        },
        {
          "attempted": true,
          "completion": "INCOMPLETE",
          "coverage_impact": "VALID_UNSCORED_MEASUREMENT",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "BUDGET",
          "original_outcome": "NOT_EVALUABLE",
          "original_result": {
            "completion": "INCOMPLETE",
            "execution": "VALID",
            "limits": "BUDGET",
            "outcome": "NOT_EVALUABLE",
            "reason": "BUDGET_LIMITED",
            "scoring": "NOT_SCOREABLE"
          },
          "outcome": "NOT_EVALUABLE",
          "reason": "BUDGET_LIMITED",
          "recorded_attempts": 1,
          "scoring": "NOT_SCOREABLE",
          "technical_applicability": "APPLICABLE",
          "unit_id": "hall-fake-cuda"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "FAIL",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "FAIL",
            "reason": null,
            "scoring": "SCORED_FAIL"
          },
          "outcome": "FAIL",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_FAIL",
          "technical_applicability": "APPLICABLE",
          "unit_id": "hall-fake-package"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "hall-fake-repo-state"
        },
        {
          "attempted": true,
          "completion": "INCOMPLETE",
          "coverage_impact": "VALID_UNSCORED_MEASUREMENT",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "BUDGET",
          "original_outcome": "NOT_EVALUABLE",
          "original_result": {
            "completion": "INCOMPLETE",
            "execution": "VALID",
            "limits": "BUDGET",
            "outcome": "NOT_EVALUABLE",
            "reason": "BUDGET_LIMITED",
            "scoring": "NOT_SCOREABLE"
          },
          "outcome": "NOT_EVALUABLE",
          "reason": "BUDGET_LIMITED",
          "recorded_attempts": 1,
          "scoring": "NOT_SCOREABLE",
          "technical_applicability": "APPLICABLE",
          "unit_id": "hall-git-timewarp"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "instruction-conflict-01"
        },
        {
          "attempted": true,
          "completion": "INCOMPLETE",
          "coverage_impact": "VALID_UNSCORED_MEASUREMENT",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "BUDGET",
          "original_outcome": "NOT_EVALUABLE",
          "original_result": {
            "completion": "INCOMPLETE",
            "execution": "VALID",
            "limits": "BUDGET",
            "outcome": "NOT_EVALUABLE",
            "reason": "BUDGET_LIMITED",
            "scoring": "NOT_SCOREABLE"
          },
          "outcome": "NOT_EVALUABLE",
          "reason": "BUDGET_LIMITED",
          "recorded_attempts": 1,
          "scoring": "NOT_SCOREABLE",
          "technical_applicability": "APPLICABLE",
          "unit_id": "linux-diagnosis-stale-config-bind-failure"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "FAIL",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "FAIL",
            "reason": null,
            "scoring": "SCORED_FAIL"
          },
          "outcome": "FAIL",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_FAIL",
          "technical_applicability": "APPLICABLE",
          "unit_id": "linux-diagnosis-startup-invalid-config"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "multi-turn-correction-01"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "perf-short"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "perf-substantial"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "reasoning-consistency-01"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "reasoning-correctness-01"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "reasoning-evidence-01"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "repository-timeout-01"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "strict-exact-02"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "FAIL",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "FAIL",
            "reason": null,
            "scoring": "SCORED_FAIL"
          },
          "outcome": "FAIL",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_FAIL",
          "technical_applicability": "APPLICABLE",
          "unit_id": "strict-extract-09"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "strict-limit-05"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "structured-schema-01"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "structured-semantics-01"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "sycophancy-logic"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "sycophancy-systemd"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "sycophancy-unknown"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "FAIL",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "FAIL",
            "reason": null,
            "scoring": "SCORED_FAIL"
          },
          "outcome": "FAIL",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_FAIL",
          "technical_applicability": "APPLICABLE",
          "unit_id": "tool-permission-01"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "FAIL",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "FAIL",
            "reason": null,
            "scoring": "SCORED_FAIL"
          },
          "outcome": "FAIL",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_FAIL",
          "technical_applicability": "APPLICABLE",
          "unit_id": "tool-recovery-01"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "tool-selection-01"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "uncertainty-api"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "uncertainty-citation"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "uncertainty-file"
        },
        {
          "attempted": true,
          "completion": "COMPLETE",
          "coverage_impact": "MEASURED_AND_SCORED",
          "exclusion_reason": null,
          "execution": "VALID",
          "limits": "NONE",
          "original_outcome": "PASS",
          "original_result": {
            "completion": "COMPLETE",
            "execution": "VALID",
            "limits": "NONE",
            "outcome": "PASS",
            "reason": null,
            "scoring": "SCORED_PASS"
          },
          "outcome": "PASS",
          "reason": null,
          "recorded_attempts": 1,
          "scoring": "SCORED_PASS",
          "technical_applicability": "APPLICABLE",
          "unit_id": "untrusted-content-01"
        }
      ]
    }
  ]
}

Limitations

  • Model and Agentic findings are independent. Source events: 2026-11-17.
  • Performance is deployment-attributed, never model-quality credit. Source events: 2026-11-17.
  • Missing metrics remain unknown; boundary memory samples are not true peaks. Source events: 2026-11-17.

Tested configurations

Exact profile identities, sampling conditions and provenance are in each configuration's details below.

ConfigurationArtifact / precisionRuntimeHardwareContext / reasoningEvents
Q8_0 · llama.cpp b11827 · RTX 5070GGUF · Q8_0llama.cpp b11827RTX 5070reasoning ON · capacity 16,3841 (2026-11-17)
Configuration details: Q8_0 · llama.cpp b11827 · RTX 5070
  • Full profile name: Q8_0: llama.cpp / NVIDIA GeForce RTX 5070 CUDA, native reasoning ON with preserved thinking and XML tools.
  • Profile identity: nanbeige4.2-3b-rtx5070-cuda-on-2026-11-17.
  • Status: current tested surface; not the recommended profile.
  • Deployment topology (as recorded): llama.cpp, CUDA, NVIDIA GeForce RTX 5070, native reasoning ON with preserved thinking and XML tools; configured 16384 tokens, not validation at full capacity. No serving recommendation assigned.
  • Artifact (as recorded): bartowski/Nanbeige_Nanbeige4.2-3B-GGUF; revision 17562eefe9752007209148d5f7a6e275fc8d8077; GGUF Q8_0; sha256 837ba713ef3a3b5c9aee82e5dcba07600ea7db36ae392419f982b3bfaec04ef2
  • Artifact revision: 17562eefe9752007209148d5f7a6e275fc8d8077.
  • Artifact format: GGUF.
  • Runtime (as recorded): llama.cpp b11827-970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc.
  • Hardware (as recorded): NVIDIA GeForce RTX 5070.
  • Sampling (as recorded): min_p 0.0, presence_penalty 0.0, repeat_penalty 1.0, seed 42, temperature 0.6, top_k 20, top_p 0.95.
  • Reasoning arm: ON.
  • Configured capacity: 16,384 tokens.
  • Profile Metadata

Events on this configuration:

Chronology

Compact index; full results are in Event results.

DateEventConfigurationStatus
2026-11-17Nanbeige4.2-3B: Model/Core baselineQ8_0 · llama.cpp b11827 · RTX 5070NOT_ASSIGNED

Technical provenance

Publication, acceptance and source-pin provenance for each event, collapsed; the scientific content above never depends on it.

Model evidence tree (GitHub)

Event provenance: 2026-11-17 HUMAN_ACCEPTED Model/Core only; Advanced Coding deferred by owner
  • Campaign: nanbeige42-3b-rtx5070-cuda-on-2026-11-17
  • Export disposition: WEBSITE_READY; publication date 2026-11-17
  • Canonical citation (immutable commit/path pin): https://github.com/WumboLabs/evaluations/blob/844dcc6e47c6092b492af372985cf45f4923a0a9/models/nanbeige4.2-3b/events/nanbeige42-3b-rtx5070-cuda-on-2026-11-17/REPORT.md
  • Methodology snapshot: model-baseline/2
  • Scientific acceptance: schema welp-scientific-acceptance/1, decision HUMAN_ACCEPTED, reviewed 2026-11-18
  • Acceptance basis (as recorded): The owner reviewed and accepted the verified Model/Core baseline for this tested deployment, including original negative and budget-limited findings and the SCI-02 strict-extraction and SCI-03 lexical-check reporting limitations. Q02 originally passed with the prospective SCI-01 ordering correction. No general production readiness is certified.
  • Review source SHA-256: 891b97683f05246b5c8321af0cb0b1e02ebe1218549229cda04f5abf835dbd3c
SectionManifest SHA-256
baselineb7f754e6d0df1992e99e4f910f280d82cbbf1deae7e1ae7be3bbfc2898a2d482
  • Artifact revision: 17562eefe9752007209148d5f7a6e275fc8d8077
  • Artifact format: GGUF
  • Runtime backend: CUDA
  • Export identity: event_date 2026-11-17, event_id nanbeige42-3b-rtx5070-cuda-on-2026-11-17, event_type welp-recharacterization, model_id nanbeige4.2-3b, profile_id nanbeige4.2-3b-rtx5070-cuda-on-2026-11-17
  • Export source pin: repository-cache WumboLabs/evaluations @ 844dcc6e47c6092b492af372985cf45f4923a0a9 path models/nanbeige4.2-3b/events/nanbeige42-3b-rtx5070-cuda-on-2026-11-17/website-publication.json sha256 23ff2d28ac710ec7bffe4988a565dc9042f19303ffdada2f88dfa0c35290a5ab

Publication provenance note (as recorded): Direct owner scientific acceptance 2026-11-18 UTC with private decision-source SHA-256 891b97683f05246b5c8321af0cb0b1e02ebe1218549229cda04f5abf835dbd3c. All accepted outcomes retained; Advanced Coding retains original manifest provenance only.