WUMBOLABS / EVALUATIONS CANONICAL MODEL EVALUATION

/evaluations/qwen35-4b

Qwen3.5-4B

Published evaluation evidence and testing history. Results remain attributed to their original configurations and methods.

latest evidence 2026-09-11

Overview

Alibaba (Qwen team) · 1 tested configuration · 1 evidence event · newest evidence 2026-09-11.

Contents: Findings · Event results · Limitations · Tested configurations · Chronology · Technical provenance

Findings

Each finding is attributed to the event and tested configuration that measured it; exact values, failures and limitations are in the event results. A historical event's recorded status describes that event's own sections at its date and is not a verdict on newer sections.

  • Recommended configuration: llama.cpp BF16 GGUF (current): event result.
  • Practical context: 32,768 default / 65,536 guarded tokens; native model-card maximum 262,144 (envelope complete: YES); source: event result
  • Context limit outcome: FIT_LIMIT - native maximum 262,144 on the BF16/f16 primary surface (arithmetic lower bound) and on the authorized q4_0-KV alternate surface (measured admission failure); official YaRN 1,010,000 maximum FIT_LIMIT at every representable KV precision; source: event result
  • Latest evidence: 2026-09-11: event result
  • 2026-09-11 · BF16 weights; f16 KV · llama.cpp · RTX 5070: classification READY_WITH_GUARDRAILS; inherited: 7/7 constrained checks and 20/20 bounded reliability (baseline campaign, unchanged, not rerun). Event result

Event results

Newest first. Each event is one independent testing/publication event with its full report linked once at the top of its section; scores, failures and limitations stay visible and raw section-provenance JSON stays collapsed.

2026-09-11: Context envelope completion

Context Envelope: llama.cpp BF16 · config BF16 weights; f16 KV · llama.cpp · RTX 5070 · maturity: CONTEXT_COMPLETION · status: READY_WITH_GUARDRAILS

Configuration: llama.cpp BF16 GGUF.

Full report (REPORT.md)

WELP outcome

  • Outcome: PASS: QWEN35_4B_CONTEXT_ENVELOPE_COMPLETED
  • Classification: READY_WITH_GUARDRAILS (unchanged; new occupancy-dependent instruction-retention guardrail added)
  • Artifact classification: Comfortable-fit high-precision BF16 text control

Publication state: published; canonical evidence linked above

Headline performance

SurfaceTTFTPrefillDecode
Short0.040569 sNot recorded67.91 tok/s
Moderate (3642-token input)0.691319 s5268.19 tok/s67.29 tok/s

Near-full context fixture medians per seed pair; client/transport proxies with native timings retained in campaign evidence; short/moderate surfaces inherited unchanged from the baseline campaign (67.9 tok/s short decode, 0.69 s moderate TTFT)

Quality and capabilities

  • Constrained result: inherited: 7/7 constrained checks and 20/20 bounded reliability (baseline campaign, unchanged, not rerun)
  • Perfect synthetic retrieval (20/20 values) at 99.4-99.5% occupancy through 64K on the primary surface
  • Highest measured/admitted near-full primary context 65,536 (guarded profile confirmed); 98,304 projected below the frozen operational safety floor and intentionally not launched; exact ceiling not bracketed
  • q4_0-KV alternate surface functionally correct at 32K
  • Official YaRN mechanism representable in the pinned runtime
Guardrails and limitations
  • Near-full-occupancy instruction retention defect: auxiliary JSON fields (synthesis/absent/checksum) dropped at both rungs, both seeds; do not place binding output instructions at extreme occupancy
  • Inherited free-form grounding cautions unchanged
  • Exact native maximum and YaRN maximum are unreachable on 12GB at any representable KV precision; not a model-capability claim on other hardware

Reliability: inherited bounded 20/20 reliability (profile unchanged); no new reliability run required; context-specific observations: zero CUDA/OOM/Xid events across all arms including the measured admission-failure attempt

LocalMaxxing

  • Status: SUBMITTED
  • Canonical context: 32,768 tokens
  • tok/s out: 68.0
  • TTFT: 76.92 ms
  • Submission reference: cmtwcmm3207eqps01w7fubuuw
  • verifiedRun: NO

origin VERIFIED_EXISTING (duplicate audit): canonical practical profile unchanged; no new benchmark, no duplicate submission

Tested condition: artifact BF16 weights; f16 KV · llama.cpp · RTX 5070 · context 32,768 default / 65,536 guarded · native model-card maximum 262,144 tokens · envelope complete: YES.

Context limit outcome: FIT_LIMIT - native maximum 262,144 on the BF16/f16 primary surface (arithmetic lower bound) and on the authorized q4_0-KV alternate surface (measured admission failure); official YaRN 1,010,000 maximum FIT_LIMIT at every representable KV precision

Context boundaries: 32K default and 64K guarded profile both reevaluated at the current WELP near-full standard and confirmed; occupied 32K performance 6.6-6.7 s TTFT / 60.3 tok/s decode; 64K 15.2 s TTFT / 54.1-54.4 tok/s decode

Runtime conditions: All 33/33 runtime layers GPU-resident; non-thinking text-only greedy surface; identical inherited baseline stack

Hardware conditions: Single-user workstation; AMD Ryzen 7 9800X3D; Fedora Linux

Full tested identity, runtime and context details (exact hashes, versions and configuration)

Identity

FieldValue
Profile identityqwen35-4b-llamacpp-bf16
ModelQwen3.5-4B
ProducerAlibaba (Qwen team); GGUF conversion by Unsloth
Official modelQwen/Qwen3.5-4B @ 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Tested artifactQwen3.5-4B-BF16.gguf (text-only conversion; Unsloth revision e87f176479d0855a907a41277aca2f8ee7a09523)
PrecisionBF16 weights; f16 KV (primary surface); q4_0 KV (one alternate admission surface)
Artifact SHA-2569e6e2841a75f503ccb330831832fd7861266e187e0dbf149a954219ccb8c197a
Campaignqwen35-4b-rtx5070-context-completion-2026-09-11
Record date2026-09-09

Runtime and hardware

FieldValue
Enginellama.cpp 0.1.0-dev build b10449, commit 0d9ceae1e38291035605613ab41a8f5e693d6fcd (CUDA 13.3, SM120)
HardwareWumboJetsII (NVIDIA GeForce RTX 5070 12GB)
Runtime notesAll 33/33 runtime layers GPU-resident; non-thinking text-only greedy surface; identical inherited baseline stack
Hardware notesSingle-user workstation; AMD Ryzen 7 9800X3D; Fedora Linux

Context profile

FieldValue
Practical default32768 tokens
Guarded context65536 tokens
Native model-card maximum262144 tokens
Model-card envelope completeYES
Native maximum dispositionFIT_LIMIT - native maximum 262,144 on the BF16/f16 primary surface (arithmetic lower bound) and on the authorized q4_0-KV alternate surface (measured admission failure); official YaRN 1,010,000 maximum FIT_LIMIT at every representable KV precision

32K default and 64K guarded profile both reevaluated at the current WELP near-full standard and confirmed; occupied 32K performance 6.6-6.7 s TTFT / 60.3 tok/s decode; 64K 15.2 s TTFT / 54.1-54.4 tok/s decode

Limitations

  • Near-full-occupancy instruction retention defect: auxiliary JSON fields (synthesis/absent/checksum) dropped at both rungs, both seeds; do not place binding output instructions at extreme occupancy Source events: 2026-09-11.
  • Inherited free-form grounding cautions unchanged Source events: 2026-09-11.
  • Exact native maximum and YaRN maximum are unreachable on 12GB at any representable KV precision; not a model-capability claim on other hardware Source events: 2026-09-11.

Tested configurations

Exact profile identities, sampling conditions and provenance are in each configuration's details below.

ConfigurationArtifact / precisionRuntimeHardwareContext / reasoningEvents
BF16 weights; f16 KV · llama.cpp · RTX 5070BF16 weightsllama.cppRTX 507032,768 default / 65,536 guarded1 (2026-09-11)
Configuration details: BF16 weights; f16 KV · llama.cpp · RTX 5070
  • Full profile name: llama.cpp BF16 GGUF.
  • Profile identity: qwen35-4b-llamacpp-bf16.
  • Status: current canonical/recommended tested surface.
  • Artifact (as recorded): Qwen3.5-4B-BF16.gguf (text-only conversion; Unsloth rev e87f176479d0855a907a41277aca2f8ee7a09523)
  • Runtime (as recorded): llama.cpp 0.1.0-dev build b10449, commit 0d9ceae1e38291035605613ab41a8f5e693d6fcd (CUDA 13.3, SM120).
  • Hardware (as recorded): WumboJetsII (NVIDIA GeForce RTX 5070 12GB).
  • Profile Metadata

Events on this configuration:

Chronology

Compact index; full results are in Event results.

DateEventConfigurationStatus
2026-09-11Context envelope completionBF16 weights; f16 KV · llama.cpp · RTX 5070READY_WITH_GUARDRAILS

Technical provenance

Publication, acceptance and source-pin provenance for each event, collapsed; the scientific content above never depends on it.

Model evidence tree (GitHub)

Event provenance: 2026-09-11 Context Envelope: llama.cpp BF16
  • Campaign: qwen35-4b-rtx5070-context-completion-2026-09-11
  • Export disposition: WEBSITE_READY; publication date 2026-09-11
  • Canonical citation (immutable commit/path pin): https://github.com/WumboLabs/evaluations/blob/479cb50c197f8ea0dd905a68ffd61f632f49c652/models/qwen35-4b/events/context-envelope-completion-2026-09-11-qwen35-4b/REPORT.md
  • Export source pin: repository-cache WumboLabs/evaluations @ 479cb50c197f8ea0dd905a68ffd61f632f49c652 path models/qwen35-4b/events/context-envelope-completion-2026-09-11-qwen35-4b/website-publication.json sha256 3c8fab5a1dcea73dd57a5296a00fd8238caf1b3eb8dd28be7fb2d5d56ee4bb04

Legacy provenance