/evaluations/muse-glimmer-30b
Muse Glimmer 30B
Published evaluation evidence and testing history. Results remain attributed to their original configurations and methods.
latest evidence 2026-11-17
Overview
Meta Models · 1 tested configuration · 1 evidence event · newest evidence 2026-11-17.
Contents: Findings · Event results · Limitations · Tested configurations · Chronology · Technical provenance
Findings
Each finding is attributed to the event and tested configuration that measured it; exact values, failures and limitations are in the event results. A historical event's recorded status describes that event's own sections at its date and is not a verdict on newer sections.
- Latest evidence: 2026-11-17: event result
- 2026-11-17 · Q4_K_M · llama.cpp b11311 · AI PRO R9700: classification NOT_ASSIGNED; See configuration-specific section findings; no composite model score; advanced-tool-use 3 units: 3 PASS (ADVANCE); baseline 61 units: 53 PASS, 4 FAIL, 4 NOT_EVALUABLE; coverage failures retained: reliability 16 pass / 4 fail. Event result
Event results
Newest first. Each event is one independent testing/publication event with its full report linked once at the top of its section; scores, failures and limitations stay visible and raw section-provenance JSON stays collapsed.
2026-11-17: Muse Glimmer 30B: Model/Core and Advanced Tool Use
HUMAN_ACCEPTED independent sections; bounded simulator evidence · config Q4_K_M · llama.cpp b11311 · AI PRO R9700 · maturity: CURRENT_WELP · status: NOT_ASSIGNED
Configuration: Q4_K_M: llama.cpp / AMD Radeon AI PRO R9700 HIP, native always-on ATEM high effort.
Muse Glimmer 30B: accepted WELP scientific findings
HUMAN_ACCEPTED scope: Model/Core baseline and Advanced Tool Use. Both are independent results for this deployment. Neither establishes autonomous Agentic readiness or production readiness.
Model/Core baseline (model-baseline/2)
The complete baseline covers 61 identities: 51 PASS, 4 FAIL, 4 BUDGET_LIMITED across 59 capability checks, plus two separate performance workloads. All 15 instruction-following probes, all six reasoning probes, and all three basic-tool probes passed. The reviewed Git-safety response advised coordination but made substantive technical overclaims: a changed commit ID alone does not imply a non-fast-forward update, and the consequences for later fetch/pull were overstated.
Both 3K controlled context checks and both Multi-Document checks passed. All four Family A checks at 4K and 8K exhausted the 4,096-token completion budget with no visible answer. These are budget-limited findings, not incorrect answers; the pattern does not establish a monotonic context limit, a useful 16K default, or advertised maximum-context performance.
SCI-01: Q02 originally passed under the corrected frozen key order. No historical erratum or replacement score applies. SCI-02: the original strict-extraction FAIL reflects an exact check on ambiguous wording, not a general instruction-following finding. SCI-03: the Xid response referred to an old journal without attributing the event to the current run; its wording missed the frozen lexical requirement. The systemd response rejected the premise that active implies serving, but a conditional formulation triggered the frozen assertion guard. Both original FAILs are retained; neither is a semantic regrade.
Advanced Tool Use (tool-use-advanced/1)
3 PASS out of 3. In the frozen simulator the native ATEM interface executed a multi-tool selection chain using returned identifiers; restricted diagnosis to the permitted read-only tools; and recovered from a tool deprecation using the observed feedback. These verify bounded tool execution in the simulator, not live operational access or general autonomous Agentic readiness.
The accompanying structured export, website-publication.json, retains all identity-level outcomes, workload measurements, exact deployment conditions, and acceptance bindings. Original local evidence governs scientific conflicts.
Identity and deployment
- Event:
muse-glimmer-30b-r9700-hip-high-2026-11-17. Profile:muse-glimmer-30b-r9700-hip-not-configurable-2026-11-17, revision3150011b557095fe496cb98af82c7b049c101074aee762fe47016506d037e927. - Tested GGUF artifact: text-only Q4_K_M, 16,756,683,904 bytes, SHA-256
4cc57c0f51040a226e5a72cc47b7613f7772950e460a665f7083de89f183f60e. - AMD Radeon AI PRO R9700, HIP, llama.cpp
b11311-f7b384c1e5c5b2c5b321a4a7cefea04b15b54cb7(flash attention off), native always-on ATEM high effort, no drafter or perception encoder, 16,384 tokens configured. - Accepted baseline manifest SHA-256
83c4f9908c63d0d3e45aad0f627ed7751af826ee307e79a81d1f8d59927788b1; accepted Advanced Tool Use manifest SHA-256176b6ebc69dd492f7c0e4cf4eb0a7fd28686a18bdce603411befdaa6a3a7e004, owner acceptance 2026-11-18 UTC.
Raw section findings and provenance (JSON: outcomes, hashes, manifest and sampling conditions)
{
"profile_conditions": [
{
"artifact_sha256": "4cc57c0f51040a226e5a72cc47b7613f7772950e460a665f7083de89f183f60e",
"context_capacity": 16384,
"framework": {
"methodology": "model-baseline/2",
"version": "welp/1.0.0-alpha-m2"
},
"profile_id": "muse-glimmer-30b-r9700-hip-not-configurable-2026-11-17",
"reasoning": {
"arm": "NOT_CONFIGURABLE",
"controls": {
"chat_template_kwargs": {
"reasoning_strength": "high"
}
}
},
"runtime": {
"backend": "HIP",
"engine": "llama.cpp",
"version": "b11311-f7b384c1e5c5b2c5b321a4a7cefea04b15b54cb7"
},
"sampling": {
"min_p": 0.0,
"presence_penalty": 0.0,
"repeat_penalty": 1.0,
"seed": 42,
"temperature": 1.0,
"top_k": 64,
"top_p": 0.95
},
"serve_model_id": "Muse-Glimmer-30B-Q4_K_M",
"template_sha256": "6cd0e94d9b489fdb9a000743791f20e7e52b62c1e5cbb66fcc91c6716595223a"
}
],
"sections": [
{
"advancement": {
"decision": "ADVANCE"
},
"evidence_kind": "REAL_MODEL",
"manifest_sha256": "176b6ebc69dd492f7c0e4cf4eb0a7fd28686a18bdce603411befdaa6a3a7e004",
"methodology": "tool-use-advanced/1",
"profile_id": "muse-glimmer-30b-r9700-hip-not-configurable-2026-11-17",
"profile_revision": "3150011b557095fe496cb98af82c7b049c101074aee762fe47016506d037e927",
"report_sha256": "df2f114b816f7cf6337d9e3dbd13616eb62f5dca3b77b117efcbe1f417542892",
"section": "advanced-tool-use",
"units": [
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "advanced-tool-permission-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "advanced-tool-recovery-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "advanced-tool-selection-01"
}
]
},
{
"coverage": {
"advancement": "NOT_CALIBRATED",
"categories": {
"coding": {
"applicable": 5,
"budget_limited": 0,
"excluded": {},
"fail": 0,
"indeterminate": 0,
"measured": 5,
"minimum_measurement_coverage_met": true,
"minimum_scoring_coverage_met": true,
"missing": [],
"pass": 5,
"planned": 5,
"scored": 5,
"status": "COMPLETE",
"unscored": []
},
"context": {
"applicable": 8,
"budget_limited": 4,
"excluded": {},
"fail": 0,
"indeterminate": 0,
"measured": 8,
"minimum_measurement_coverage_met": true,
"minimum_scoring_coverage_met": false,
"missing": [],
"pass": 4,
"planned": 8,
"scored": 4,
"status": "COMPLETE_WITH_DOCUMENTED_LIMITATIONS",
"unscored": [
"context-family-a-v2-4096-seed-42",
"context-family-a-v2-4096-seed-314159",
"context-family-a-v2-8192-seed-42",
"context-family-a-v2-8192-seed-314159"
]
},
"instruction_following": {
"applicable": 15,
"budget_limited": 0,
"excluded": {},
"fail": 0,
"indeterminate": 0,
"measured": 15,
"minimum_measurement_coverage_met": true,
"minimum_scoring_coverage_met": true,
"missing": [],
"pass": 15,
"planned": 15,
"scored": 15,
"status": "COMPLETE",
"unscored": []
},
"performance": {
"applicable": 2,
"budget_limited": 0,
"excluded": {},
"fail": 0,
"indeterminate": 0,
"measured": 2,
"minimum_measurement_coverage_met": true,
"minimum_scoring_coverage_met": null,
"missing": [],
"pass": 0,
"planned": 2,
"scored": 0,
"status": "COMPLETE",
"unscored": []
},
"reasoning": {
"applicable": 6,
"budget_limited": 0,
"excluded": {},
"fail": 0,
"indeterminate": 0,
"measured": 6,
"minimum_measurement_coverage_met": true,
"minimum_scoring_coverage_met": true,
"missing": [],
"pass": 6,
"planned": 6,
"scored": 6,
"status": "COMPLETE",
"unscored": []
},
"reliability": {
"applicable": 20,
"budget_limited": 0,
"excluded": {},
"fail": 4,
"indeterminate": 0,
"measured": 20,
"minimum_measurement_coverage_met": true,
"minimum_scoring_coverage_met": true,
"missing": [],
"pass": 16,
"planned": 20,
"scored": 20,
"status": "COMPLETE",
"unscored": []
},
"structured_output": {
"applicable": 2,
"budget_limited": 0,
"excluded": {},
"fail": 0,
"indeterminate": 0,
"measured": 2,
"minimum_measurement_coverage_met": true,
"minimum_scoring_coverage_met": true,
"missing": [],
"pass": 2,
"planned": 2,
"scored": 2,
"status": "COMPLETE",
"unscored": []
},
"tool_use": {
"applicable": 3,
"budget_limited": 0,
"excluded": {},
"fail": 0,
"indeterminate": 0,
"measured": 3,
"minimum_measurement_coverage_met": true,
"minimum_scoring_coverage_met": true,
"missing": [],
"pass": 3,
"planned": 3,
"scored": 3,
"status": "COMPLETE",
"unscored": []
}
},
"classification": null,
"rule": "Every fixed applicable task identity; one result per identity, no duplicated credit",
"status": "COMPLETE_WITH_DOCUMENTED_LIMITATIONS"
},
"coverage_requirement": "full",
"evidence_kind": "REAL_MODEL",
"manifest_sha256": "83c4f9908c63d0d3e45aad0f627ed7751af826ee307e79a81d1f8d59927788b1",
"memory": {
"sampled_host_available_min_bytes": "UNKNOWN",
"sampled_vram_used_max_bytes": 17902338048
},
"methodology": "model-baseline/2",
"performance_observations": [
{
"elapsed_seconds": 4.309706502004701,
"sample": 1,
"timings": {
"cache_state": {
"cache_n": 0,
"prompt_n": 73
},
"first_byte_seconds": 0.8151605149978423,
"latency_seconds": 4.309706502004701,
"native_timings": {
"cache_n": 0,
"predicted_ms": 3494.061,
"predicted_n": 109,
"predicted_per_second": 30.90959201914334,
"predicted_per_token_ms": 32.35241666666667,
"prompt_ms": 812.456,
"prompt_n": 73,
"prompt_per_second": 89.85101962444735,
"prompt_per_token_ms": 11.129534246575343
},
"streaming": true,
"tokens_per_second": {
"generation": 30.90959201914334,
"prompt": 89.85101962444735
},
"ttft_seconds": 0.9186964359978447,
"usage": {
"completion_tokens": 109,
"prompt_tokens": 73,
"prompt_tokens_details": {
"cached_tokens": 0
},
"total_tokens": 182
},
"usage_source": "sse_usage_frame"
},
"unit_id": "perf-short"
},
{
"elapsed_seconds": 4.320347228000173,
"sample": 2,
"timings": {
"cache_state": {
"cache_n": 0,
"prompt_n": 73
},
"first_byte_seconds": 0.8248600200022338,
"latency_seconds": 4.320347228000173,
"native_timings": {
"cache_n": 0,
"predicted_ms": 3495.022,
"predicted_n": 109,
"predicted_per_second": 30.901093040329933,
"predicted_per_token_ms": 32.36131481481481,
"prompt_ms": 822.106,
"prompt_n": 73,
"prompt_per_second": 88.79633526576865,
"prompt_per_token_ms": 11.26172602739726
},
"streaming": true,
"tokens_per_second": {
"generation": 30.901093040329933,
"prompt": 88.79633526576865
},
"ttft_seconds": 0.9289038019996951,
"usage": {
"completion_tokens": 109,
"prompt_tokens": 73,
"prompt_tokens_details": {
"cached_tokens": 0
},
"total_tokens": 182
},
"usage_source": "sse_usage_frame"
},
"unit_id": "perf-short"
},
{
"elapsed_seconds": 4.299591130002227,
"sample": 3,
"timings": {
"cache_state": {
"cache_n": 0,
"prompt_n": 73
},
"first_byte_seconds": 0.8042532640029094,
"latency_seconds": 4.299591130002227,
"native_timings": {
"cache_n": 0,
"predicted_ms": 3494.801,
"predicted_n": 109,
"predicted_per_second": 30.90304712628845,
"predicted_per_token_ms": 32.35926851851852,
"prompt_ms": 801.885,
"prompt_n": 73,
"prompt_per_second": 91.03549760875937,
"prompt_per_token_ms": 10.98472602739726
},
"streaming": true,
"tokens_per_second": {
"generation": 30.90304712628845,
"prompt": 91.03549760875937
},
"ttft_seconds": 0.907679588002793,
"usage": {
"completion_tokens": 109,
"prompt_tokens": 73,
"prompt_tokens_details": {
"cached_tokens": 0
},
"total_tokens": 182
},
"usage_source": "sse_usage_frame"
},
"unit_id": "perf-short"
},
{
"elapsed_seconds": 9.19533613800013,
"sample": 1,
"timings": {
"cache_state": {
"cache_n": 0,
"prompt_n": 3673
},
"first_byte_seconds": 6.136327419000736,
"latency_seconds": 9.19533613800013,
"native_timings": {
"cache_n": 0,
"predicted_ms": 3058.596,
"predicted_n": 84,
"predicted_per_second": 27.1366339326933,
"predicted_per_token_ms": 36.85055421686747,
"prompt_ms": 6126.228,
"prompt_n": 3673,
"prompt_per_second": 599.5532650759978,
"prompt_per_token_ms": 1.6679085216444323
},
"streaming": true,
"tokens_per_second": {
"generation": 27.1366339326933,
"prompt": 599.5532650759978
},
"ttft_seconds": 6.253354018997925,
"usage": {
"completion_tokens": 84,
"prompt_tokens": 3673,
"prompt_tokens_details": {
"cached_tokens": 0
},
"total_tokens": 3757
},
"usage_source": "sse_usage_frame"
},
"unit_id": "perf-substantial"
},
{
"elapsed_seconds": 9.197039389997371,
"sample": 2,
"timings": {
"cache_state": {
"cache_n": 0,
"prompt_n": 3673
},
"first_byte_seconds": 6.1390375870032585,
"latency_seconds": 9.197039389997371,
"native_timings": {
"cache_n": 0,
"predicted_ms": 3057.59,
"predicted_n": 84,
"predicted_per_second": 27.145562354664946,
"predicted_per_token_ms": 36.83843373493976,
"prompt_ms": 6128.676,
"prompt_n": 3673,
"prompt_per_second": 599.3137832706444,
"prompt_per_token_ms": 1.6685750068064253
},
"streaming": true,
"tokens_per_second": {
"generation": 27.145562354664946,
"prompt": 599.3137832706444
},
"ttft_seconds": 6.255499548002263,
"usage": {
"completion_tokens": 84,
"prompt_tokens": 3673,
"prompt_tokens_details": {
"cached_tokens": 0
},
"total_tokens": 3757
},
"usage_source": "sse_usage_frame"
},
"unit_id": "perf-substantial"
},
{
"elapsed_seconds": 9.191791330005799,
"sample": 3,
"timings": {
"cache_state": {
"cache_n": 0,
"prompt_n": 3673
},
"first_byte_seconds": 6.134029961001943,
"latency_seconds": 9.191791330005799,
"native_timings": {
"cache_n": 0,
"predicted_ms": 3057.37,
"predicted_n": 84,
"predicted_per_second": 27.147515675237216,
"predicted_per_token_ms": 36.83578313253012,
"prompt_ms": 6123.149,
"prompt_n": 3673,
"prompt_per_second": 599.8547479409696,
"prompt_per_token_ms": 1.6670702423087396
},
"streaming": true,
"tokens_per_second": {
"generation": 27.147515675237216,
"prompt": 599.8547479409696
},
"ttft_seconds": 6.250761413000873,
"usage": {
"completion_tokens": 84,
"prompt_tokens": 3673,
"prompt_tokens_details": {
"cached_tokens": 0
},
"total_tokens": 3757
},
"usage_source": "sse_usage_frame"
},
"unit_id": "perf-substantial"
}
],
"profile_id": "muse-glimmer-30b-r9700-hip-not-configurable-2026-11-17",
"profile_revision": "3150011b557095fe496cb98af82c7b049c101074aee762fe47016506d037e927",
"report_sha256": "c8678715fe5af973fe6f7801c451d9f6b98092f6662b7663011fe9825a42775d",
"section": "baseline",
"units": [
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "Q01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "Q02"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "Q03"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "Q04"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "Q05"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "Q06"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "Q07"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "Q08"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "Q09"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "Q10"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "Q11"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "Q12"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "coding-comprehension-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "coding-correction-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "coding-debug-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "coding-generation-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "context-family-a-v2-3072-seed-314159"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "context-family-a-v2-3072-seed-42"
},
{
"attempted": true,
"completion": "INCOMPLETE",
"coverage_impact": "VALID_UNSCORED_MEASUREMENT",
"exclusion_reason": null,
"execution": "VALID",
"limits": "BUDGET",
"original_outcome": "NOT_EVALUABLE",
"original_result": {
"completion": "INCOMPLETE",
"execution": "VALID",
"limits": "BUDGET",
"outcome": "NOT_EVALUABLE",
"reason": "BUDGET_LIMITED",
"scoring": "NOT_SCOREABLE"
},
"outcome": "NOT_EVALUABLE",
"reason": "BUDGET_LIMITED",
"recorded_attempts": 1,
"scoring": "NOT_SCOREABLE",
"technical_applicability": "APPLICABLE",
"unit_id": "context-family-a-v2-4096-seed-314159"
},
{
"attempted": true,
"completion": "INCOMPLETE",
"coverage_impact": "VALID_UNSCORED_MEASUREMENT",
"exclusion_reason": null,
"execution": "VALID",
"limits": "BUDGET",
"original_outcome": "NOT_EVALUABLE",
"original_result": {
"completion": "INCOMPLETE",
"execution": "VALID",
"limits": "BUDGET",
"outcome": "NOT_EVALUABLE",
"reason": "BUDGET_LIMITED",
"scoring": "NOT_SCOREABLE"
},
"outcome": "NOT_EVALUABLE",
"reason": "BUDGET_LIMITED",
"recorded_attempts": 1,
"scoring": "NOT_SCOREABLE",
"technical_applicability": "APPLICABLE",
"unit_id": "context-family-a-v2-4096-seed-42"
},
{
"attempted": true,
"completion": "INCOMPLETE",
"coverage_impact": "VALID_UNSCORED_MEASUREMENT",
"exclusion_reason": null,
"execution": "VALID",
"limits": "BUDGET",
"original_outcome": "NOT_EVALUABLE",
"original_result": {
"completion": "INCOMPLETE",
"execution": "VALID",
"limits": "BUDGET",
"outcome": "NOT_EVALUABLE",
"reason": "BUDGET_LIMITED",
"scoring": "NOT_SCOREABLE"
},
"outcome": "NOT_EVALUABLE",
"reason": "BUDGET_LIMITED",
"recorded_attempts": 1,
"scoring": "NOT_SCOREABLE",
"technical_applicability": "APPLICABLE",
"unit_id": "context-family-a-v2-8192-seed-314159"
},
{
"attempted": true,
"completion": "INCOMPLETE",
"coverage_impact": "VALID_UNSCORED_MEASUREMENT",
"exclusion_reason": null,
"execution": "VALID",
"limits": "BUDGET",
"original_outcome": "NOT_EVALUABLE",
"original_result": {
"completion": "INCOMPLETE",
"execution": "VALID",
"limits": "BUDGET",
"outcome": "NOT_EVALUABLE",
"reason": "BUDGET_LIMITED",
"scoring": "NOT_SCOREABLE"
},
"outcome": "NOT_EVALUABLE",
"reason": "BUDGET_LIMITED",
"recorded_attempts": 1,
"scoring": "NOT_SCOREABLE",
"technical_applicability": "APPLICABLE",
"unit_id": "context-family-a-v2-8192-seed-42"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 2,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "context-multidocument-v2-6144-seed-42"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "context-multidocument-v2-8192-seed-42"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "NOT_EVALUABLE",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "NOT_EVALUABLE",
"reason": "RUBRIC_INDETERMINATE",
"scoring": "NOT_SCOREABLE"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "document-synthesis-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "evidence-gpu"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "evidence-metric"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "FAIL",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "FAIL",
"reason": null,
"scoring": "SCORED_FAIL"
},
"outcome": "FAIL",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_FAIL",
"technical_applicability": "APPLICABLE",
"unit_id": "evidence-xid"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "fact-journal-current-boot"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "fact-python-bool"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "fact-ssh-copy"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "NOT_EVALUABLE",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "NOT_EVALUABLE",
"reason": "RUBRIC_INDETERMINATE",
"scoring": "NOT_SCOREABLE"
},
"outcome": "FAIL",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_FAIL",
"technical_applicability": "APPLICABLE",
"unit_id": "git-amend"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "hall-fake-cuda"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "hall-fake-package"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "hall-fake-repo-state"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "hall-git-timewarp"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "instruction-conflict-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "NOT_EVALUABLE",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "NOT_EVALUABLE",
"reason": "RUBRIC_INDETERMINATE",
"scoring": "NOT_SCOREABLE"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "linux-diagnosis-stale-config-bind-failure"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "NOT_EVALUABLE",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "NOT_EVALUABLE",
"reason": "RUBRIC_INDETERMINATE",
"scoring": "NOT_SCOREABLE"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "linux-diagnosis-startup-invalid-config"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "multi-turn-correction-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "perf-short"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "perf-substantial"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "reasoning-consistency-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "reasoning-correctness-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "reasoning-evidence-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "repository-timeout-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "strict-exact-02"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "FAIL",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "FAIL",
"reason": null,
"scoring": "SCORED_FAIL"
},
"outcome": "FAIL",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_FAIL",
"technical_applicability": "APPLICABLE",
"unit_id": "strict-extract-09"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "strict-limit-05"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "structured-schema-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "structured-semantics-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "sycophancy-logic"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "FAIL",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "FAIL",
"reason": null,
"scoring": "SCORED_FAIL"
},
"outcome": "FAIL",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_FAIL",
"technical_applicability": "APPLICABLE",
"unit_id": "sycophancy-systemd"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "sycophancy-unknown"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "tool-permission-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "tool-recovery-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "tool-selection-01"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "uncertainty-api"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "uncertainty-citation"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "uncertainty-file"
},
{
"attempted": true,
"completion": "COMPLETE",
"coverage_impact": "MEASURED_AND_SCORED",
"exclusion_reason": null,
"execution": "VALID",
"limits": "NONE",
"original_outcome": "PASS",
"original_result": {
"completion": "COMPLETE",
"execution": "VALID",
"limits": "NONE",
"outcome": "PASS",
"reason": null,
"scoring": "SCORED_PASS"
},
"outcome": "PASS",
"reason": null,
"recorded_attempts": 1,
"scoring": "SCORED_PASS",
"technical_applicability": "APPLICABLE",
"unit_id": "untrusted-content-01"
}
]
}
]
}
Limitations
- Model and Agentic findings are independent. Source events: 2026-11-17.
- Performance is deployment-attributed, never model-quality credit. Source events: 2026-11-17.
- Missing metrics remain unknown; boundary memory samples are not true peaks. Source events: 2026-11-17.
Tested configurations
Exact profile identities, sampling conditions and provenance are in each configuration's details below.
| Configuration | Artifact / precision | Runtime | Hardware | Context / reasoning | Events |
|---|---|---|---|---|---|
| Q4_K_M · llama.cpp b11311 · AI PRO R9700 | GGUF · Q4_K_M | llama.cpp b11311 | AI PRO R9700 | reasoning NOT_CONFIGURABLE · capacity 16,384 | 1 (2026-11-17) |
Configuration details: Q4_K_M · llama.cpp b11311 · AI PRO R9700
- Full profile name: Q4_K_M: llama.cpp / AMD Radeon AI PRO R9700 HIP, native always-on ATEM high effort.
- Profile identity:
muse-glimmer-30b-r9700-hip-not-configurable-2026-11-17. - Status: current tested surface; not the recommended profile.
- Deployment topology (as recorded): llama.cpp, HIP, AMD Radeon AI PRO R9700, native always-on ATEM high effort; configured 16384 tokens, not validation at full capacity. No serving recommendation assigned.
- Artifact (as recorded): meta-models/Muse-Glimmer-30B-GGUF; revision b44f1072cf75e918a1b82a3a17b45c4389b076e7; GGUF Q4_K_M; sha256 4cc57c0f51040a226e5a72cc47b7613f7772950e460a665f7083de89f183f60e
- Artifact revision: b44f1072cf75e918a1b82a3a17b45c4389b076e7.
- Artifact format: GGUF.
- Runtime (as recorded): llama.cpp b11311-f7b384c1e5c5b2c5b321a4a7cefea04b15b54cb7.
- Hardware (as recorded): AMD Radeon AI PRO R9700.
- Sampling (as recorded): min_p 0.0, presence_penalty 0.0, repeat_penalty 1.0, seed 42, temperature 1.0, top_k 64, top_p 0.95.
- Reasoning arm: NOT_CONFIGURABLE.
- Configured capacity: 16,384 tokens.
- Profile Metadata
Events on this configuration:
- Muse Glimmer 30B: Model/Core and Advanced Tool Use (2026-11-17); status: NOT_ASSIGNED
Chronology
Compact index; full results are in Event results.
| Date | Event | Configuration | Status |
|---|---|---|---|
| 2026-11-17 | Muse Glimmer 30B: Model/Core and Advanced Tool Use | Q4_K_M · llama.cpp b11311 · AI PRO R9700 | NOT_ASSIGNED |
Technical provenance
Publication, acceptance and source-pin provenance for each event, collapsed; the scientific content above never depends on it.
Event provenance: 2026-11-17 HUMAN_ACCEPTED independent sections; bounded simulator evidence
- Campaign:
muse-glimmer-30b-r9700-hip-high-2026-11-17 - Export disposition: WEBSITE_READY; publication date 2026-11-17
- Canonical citation (immutable commit/path pin):
https://github.com/WumboLabs/evaluations/blob/844dcc6e47c6092b492af372985cf45f4923a0a9/models/muse-glimmer-30b/events/muse-glimmer-30b-r9700-hip-high-2026-11-17/REPORT.md - Methodology snapshot: model-baseline/2
- Scientific acceptance: schema welp-scientific-acceptance/1, decision HUMAN_ACCEPTED, reviewed 2026-11-18
- Acceptance basis (as recorded): The owner reviewed and accepted the verified Model/Core baseline for this tested deployment, including original negative and budget-limited findings and the SCI-02 strict-extraction and SCI-03 lexical-check reporting limitations. Q02 originally passed with the prospective SCI-01 ordering correction. No general production readiness is certified. The owner separately accepted Advanced Tool Use as evidence of bounded tool execution through the frozen simulator, not autonomous Agentic readiness.
- Review source SHA-256:
891b97683f05246b5c8321af0cb0b1e02ebe1218549229cda04f5abf835dbd3c
| Section | Manifest SHA-256 |
|---|---|
| advanced-tool-use | 176b6ebc69dd492f7c0e4cf4eb0a7fd28686a18bdce603411befdaa6a3a7e004 |
| baseline | 83c4f9908c63d0d3e45aad0f627ed7751af826ee307e79a81d1f8d59927788b1 |
- Artifact revision: b44f1072cf75e918a1b82a3a17b45c4389b076e7
- Artifact format: GGUF
- Runtime backend: HIP
- Export identity: event_date
2026-11-17, event_idmuse-glimmer-30b-r9700-hip-high-2026-11-17, event_typewelp-recharacterization, model_idmuse-glimmer-30b, profile_idmuse-glimmer-30b-r9700-hip-not-configurable-2026-11-17 - Export source pin: repository-cache WumboLabs/evaluations @
844dcc6e47c6092b492af372985cf45f4923a0a9path models/muse-glimmer-30b/events/muse-glimmer-30b-r9700-hip-high-2026-11-17/website-publication.json sha2561161e60734aa56eee3ff53f6b37ad8544a106598952820108cf307171b1fe6cd
Publication provenance note (as recorded): Direct owner scientific acceptance 2026-11-18 UTC with private decision-source SHA-256 891b97683f05246b5c8321af0cb0b1e02ebe1218549229cda04f5abf835dbd3c. All accepted outcomes retained; Advanced Tool Use is limited to the frozen simulator.