WUMBOLABS / EVALUATIONS CANONICAL MODEL EVALUATION

/evaluations/spark-x2.5-4b

Spark-X2.5-4B

Published evaluation evidence and testing history. Results remain attributed to their original configurations and methods.

latest evidence 2026-09-20

Overview

XHToken (SparkLLM Team, iFlytek) · 1 tested configuration · 1 evidence event · newest evidence 2026-09-20.

Contents: Findings · Event results · Limitations · Tested configurations · Chronology · Technical provenance

Findings

Each finding is attributed to the event and tested configuration that measured it; exact values, failures and limitations are in the event results. A historical event's recorded status describes that event's own sections at its date and is not a verdict on newer sections.

  • Model classification: READY_WITH_GUARDRAILS - attributed to the 2026-09-20 event, configuration Q8_0 · llama.cpp b10999 · RTX 5070.
  • Recommended configuration: llama.cpp upstream official Q8_0 (reasoning-on, vendor default) (current): event result.
  • Practical context: 32,768 default / 131,072 guarded tokens; native model-card maximum 1,048,576 (envelope complete: YES); source: event result
  • Context limit outcome: FIT_LIMIT: f16 KV alone requires 38,654,705,664 bytes at the exact native maximum (36,864 B/token measured) - 3.16x the 12,227 MiB card; bounded admission attempt failed cleanly. Official extension: none advertised (documented absence).; source: event result
  • Latest evidence: 2026-09-20: event result
  • 2026-09-20 · Q8_0 · llama.cpp b10999 · RTX 5070: classification READY_WITH_GUARDRAILS; 10/12 frozen quality screen (temp 0, thinking-off diagnostic lane); canonical thinking-on lane 1/12 with 11/12 budget-starved to NOT_EVALUABLE. Event result

Event results

Newest first. Each event is one independent testing/publication event with its full report linked once at the top of its section; scores, failures and limitations stay visible and raw section-provenance JSON stays collapsed.

2026-09-20: Current-WELP characterization (official Q8_0, upstream llama.cpp b10999)

WELP Characterization: iFlytek Spark-X2.5-4B (reasoning-on) · config Q8_0 · llama.cpp b10999 · RTX 5070 · maturity: CURRENT_WELP · status: READY_WITH_GUARDRAILS

Configuration: llama.cpp upstream official Q8_0 (reasoning-on, vendor default).

Full report (REPORT.md)

WELP outcome

  • Outcome: COMPLETE_PASS - characterization complete (all planned surfaces executed with valid evidence, including negatives)
  • Classification: READY_WITH_GUARDRAILS
  • Artifact classification: current

Publication state: published; canonical evidence linked above

Headline performance

SurfaceTTFTPrefillDecode
Short0.0266 sNot recorded122.1 tok/s
Moderate (4114-token input)0.4946 s8318.3 tok/snot recorded

llama-bench canonical arm (5 reps), uncached; near-full prefill 8,329 tok/s (8K) to 4,318 tok/s (128K); full-occupancy decode ~59 tok/s at 128K vs 122 tok/s short-prompt; SWA hybrid keeps prefill fast through 128K.

Quality and capabilities

  • Constrained result: 10/12 frozen quality screen (temp 0, thinking-off diagnostic lane); canonical thinking-on lane 1/12 with 11/12 budget-starved to NOT_EVALUABLE
  • reasoning probe: PASS at frozen 1500 and operational 4096 ceilings (exact 'TOTAL HOURS: 4')
  • coding: executable-oracle PASS at 400 (thinking-off canonical) and 1500 (thinking-on diagnostic)
  • native tools: exact call + args + grounded continuation PASS
  • structured interfaces: PASS (both lanes)
Guardrails and limitations
  • BUDGET_DISCIPLINE POOR on the canonical thinking-on profile: 0/20 operational-lane completions on BOTH reliability seeds - thinking consumes the frozen 20-256 token ceilings entirely.
  • Semantic capability intact (30/33 evaluable clean on the predeclared 2048 lane); one fabricated 'git timewarp --undo' command instance observed in a truncated channel (not replicated across seeds).
  • Greedy (temp 0) thinking can degenerate into repetition loops on absent-evidence probes; vendor sampling (1.0/0.95/-1) is the intended regime.
  • No effort-level reasoning control exists on the pinned runtime; only enable_thinking=false changes behavior (requested=effective proven in both states).

Reliability: Frozen 20-task corpus, canonical vendor profile (thinking-on), seeds 42 and 314159: gate DO_NOT_ADVANCE - 0/20 completions on both seeds (budget exhaustion, 0 UNSAFE; adaptive third seed not triggered). Predeclared semantic lane (2048): 15/15 and 15/18 evaluable clean. Thinking-off diagnostic at the same tiny ceilings: 9/20 COMPLETE, 7/10 evaluable clean.

LocalMaxxing

  • Status: SUBMITTED
  • Canonical context: 32,768 tokens
  • tok/s out: 126.14
  • TTFT: 55.2 ms
  • Submission reference: cmudarhyh0aw9lq018zug8yna
  • verifiedRun: NO

Terminal closeout 2026-09-22: the canonical llama-bench arm (pp512 9,275.99 / tg128 126.14 tok/s, 5 reps) submitted verbatim; duplicate audit NO_EXACT_MATCH (exhaustive authenticated history of 35 user submissions with a control record verified present; zero Spark entries locally, in history, or in the public listing). verifiedRun=false recorded honestly: llama-bench local mode cannot supply prompt/output/engine-timing capture. Campaign close (2026-09-20) was MEASURED_NOT_SUBMITTED per the Stage-A boundary.

Tested condition: artifact Q8_0 · llama.cpp b10999 · RTX 5070 · context 32,768 default / 131,072 guarded · native model-card maximum 1,048,576 tokens · envelope complete: YES.

Context limit outcome: FIT_LIMIT: f16 KV alone requires 38,654,705,664 bytes at the exact native maximum (36,864 B/token measured) - 3.16x the 12,227 MiB card; bounded admission attempt failed cleanly. Official extension: none advertised (documented absence).

Context boundaries: Useful context is PROFILE-SCOPED: USEFUL_CONTEXT_MAX_SEMANTIC = 131,072 on the predeclared thinking-off non-starving lane (2/2 seeds, all gates PASS at >=99% occupancy). The canonical vendor-default thinking-on lane is BUDGET_LIMITED at every tested rung (the standardized 512-token output reserve is consumed by thinking); retrieval capability is intact - the reserve-vs-thinking interaction is the blocker, not retrieval failure. The thinking-off lane is a declared alternative lane, not a separately fully characterized profile.

Runtime conditions: Native spark2_5 support (first release b10828, PR #27868); ENHANCED_SEMANTIC admission passed (new-architecture battery)

Hardware conditions: full GPU residency (-ngl 99); f16 K/V; FA on; -np 1; one heavy CUDA workload at a time

Full tested identity, runtime and context details (exact hashes, versions and configuration)

Identity

FieldValue
Profile identityspark25-4b-q8-0-llamacpp
ModelSpark-X2.5-4B
ProducerXHToken (SparkLLM Team, iFlytek)
Official modelXHToken/Spark-X2.5-4B @ 0bcb35678590218655dff3765b9e61c83b35e9c4
Tested artifactSpark-X2.5-4B-Q8_0.gguf; 4,375,021,152 bytes; SHA-256 verified exact official LFS identity
PrecisionQ8_0 (official publisher GGUF, 8.50 BPW)
Artifact SHA-2565c2c3c190e4337e1016b8593ca8e26e8b18c972200b107385d4ec61a25d9dea2
Campaignspark-x2.5-4b-rtx5070-welp-characterization-2026-09-20
Record date2026-09-22

Runtime and hardware

FieldValue
Enginellama.cpp (upstream) b04d4e567cd2fb8d2ded6e17d38dbbcfafe29063 (tag b10999, 0.4.1-dev build 1), CUDA SM120
HardwareWumboJetsII (NVIDIA GeForce RTX 5070 12GB)
Runtime notesNative spark2_5 support (first release b10828, PR #27868); ENHANCED_SEMANTIC admission passed (new-architecture battery)
Hardware notesfull GPU residency (-ngl 99); f16 K/V; FA on; -np 1; one heavy CUDA workload at a time

Context profile

FieldValue
Practical default32768 tokens
Guarded context131072 tokens
Native model-card maximum1048576 tokens
Model-card envelope completeYES
Native maximum dispositionFIT_LIMIT: f16 KV alone requires 38,654,705,664 bytes at the exact native maximum (36,864 B/token measured) - 3.16x the 12,227 MiB card; bounded admission attempt failed cleanly. Official extension: none advertised (documented absence).

Useful context is PROFILE-SCOPED: USEFUL_CONTEXT_MAX_SEMANTIC = 131,072 on the predeclared thinking-off non-starving lane (2/2 seeds, all gates PASS at >=99% occupancy). The canonical vendor-default thinking-on lane is BUDGET_LIMITED at every tested rung (the standardized 512-token output reserve is consumed by thinking); retrieval capability is intact - the reserve-vs-thinking interaction is the blocker, not retrieval failure. The thinking-off lane is a declared alternative lane, not a separately fully characterized profile.

Limitations

  • BUDGET_DISCIPLINE POOR on the canonical thinking-on profile: 0/20 operational-lane completions on BOTH reliability seeds - thinking consumes the frozen 20-256 token ceilings entirely. Source events: 2026-09-20.
  • Semantic capability intact (30/33 evaluable clean on the predeclared 2048 lane); one fabricated 'git timewarp --undo' command instance observed in a truncated channel (not replicated across seeds). Source events: 2026-09-20.
  • Greedy (temp 0) thinking can degenerate into repetition loops on absent-evidence probes; vendor sampling (1.0/0.95/-1) is the intended regime. Source events: 2026-09-20.
  • No effort-level reasoning control exists on the pinned runtime; only enable_thinking=false changes behavior (requested=effective proven in both states). Source events: 2026-09-20.

Tested configurations

Exact profile identities, sampling conditions and provenance are in each configuration's details below.

ConfigurationArtifact / precisionRuntimeHardwareContext / reasoningEvents
Q8_0 · llama.cpp b10999 · RTX 5070Q8_0 (official publisher GGUF, 8.50 BPW)llama.cpp b10999RTX 507032,768 default / 131,072 guarded1 (2026-09-20)
Configuration details: Q8_0 · llama.cpp b10999 · RTX 5070
  • Full profile name: llama.cpp upstream official Q8_0 (reasoning-on, vendor default).
  • Profile identity: spark25-4b-q8-0-llamacpp.
  • Status: current canonical/recommended tested surface.
  • Deployment topology (as recorded): Full-GPU upstream llama.cpp CUDA serving on RTX 5070 12GB: -ngl 99 (36/36 layers), -fa on, f16 KV, -np 1 (verified via /slots), embedded Jinja template, vendor sampler (temp 1.0 / top_p 0.95 / top_k -1); REASONING_ON vendor default with enable_thinking=false as the only effective control (no effort-level control exists). Default context 32,768 / guarded fit ceiling 131,072; standardized 512-token output reserve.
  • Artifact (as recorded): XHToken/Spark-X2.5-4B-GGUF@9826e0be :: Spark-X2.5-4B-Q8_0.gguf (sha256 5c2c3c190e4337e1016b8593ca8e26e8b18c972200b107385d4ec61a25d9dea2)
  • Runtime (as recorded): llama.cpp (upstream) b04d4e567cd2fb8d2ded6e17d38dbbcfafe29063 (tag b10999, 0.4.1-dev build 1), CUDA SM120.
  • Hardware (as recorded): WumboJetsII (NVIDIA GeForce RTX 5070 12GB).
  • Profile Metadata

Events on this configuration:

Chronology

Compact index; full results are in Event results.

DateEventConfigurationStatus
2026-09-20Current-WELP characterization (official Q8_0, upstream llama.cpp b10999)Q8_0 · llama.cpp b10999 · RTX 5070READY_WITH_GUARDRAILS

Technical provenance

Publication, acceptance and source-pin provenance for each event, collapsed; the scientific content above never depends on it.

Model evidence tree (GitHub)

Event provenance: 2026-09-20 WELP Characterization: iFlytek Spark-X2.5-4B (reasoning-on)
  • Campaign: spark-x2.5-4b-rtx5070-welp-characterization-2026-09-20
  • Export disposition: WEBSITE_PUBLISHED; publication date 2026-09-22
  • Canonical citation (immutable commit/path pin): https://github.com/WumboLabs/evaluations/blob/51a5ad2b0af17dff7502f1f55f94262ee1e67245/models/spark-x2.5-4b/events/spark25-4b-rtx5070-welp-20260920/REPORT.md
  • Export identity: event_date 2026-09-20, event_id spark25-4b-rtx5070-welp-20260920, event_type initial-evaluation, evidence_scope ['performance', 'reliability', 'quality', 'capability-probes', 'context-envelope', 'useful-context', 'localmaxxing', 'model-classification'], model_id spark-x2.5-4b, profile_id spark25-4b-q8-0-llamacpp, profile_status current
  • Export source pin: repository-cache WumboLabs/evaluations @ 3fe25165969c303ad32dd5002943f17e368993e0 path models/spark-x2.5-4b/events/spark25-4b-rtx5070-welp-20260920/website-publication.json sha256 a93b2c8c3d9049e9007aa70c7a7bb29b5f0687ad3e566c5ab3559ef92ad01d09

Publication provenance note (as recorded): Native current-WELP event under the consolidated architecture; no eval-* repository. Campaign measurement snapshot welp-next-snapshot-2026-09-19-methodology-revision (WELP commit 79f9de13de04325e50e05b0eed28b56471a19028); the science closed 2026-09-20/21 under the then-applicable two-stage human-acceptance gate, superseded before terminal closeout by the 2026-09-22 execution-state lifecycle (welp-next-snapshot-2026-09-22-automatic-closeout-lifecycle, methodology_changed: NO) - historical evidence states preserved unchanged. LocalMaxxing terminal disposition recorded 2026-09-22: SUBMITTED origin NEW, id cmudarhyh0aw9lq018zug8yna, APPROVED, verifiedRun false (honest; llama-bench local mode cannot supply prompt/output/engine-timing capture); duplicate audit NO_EXACT_MATCH.