WUMBOLABS / EVALUATIONS CANONICAL MODEL EVALUATION

/evaluations/gemma4-e4b

Gemma 4 E4B

Published evaluation evidence and testing history. Results remain attributed to their original configurations and methods.

latest evidence 2026-09-10

Overview

Google · 1 tested configuration · 1 evidence event · newest evidence 2026-09-10.

Contents: Findings · Event results · Limitations · Tested configurations · Chronology · Technical provenance

Findings

Each finding is attributed to the event and tested configuration that measured it; exact values, failures and limitations are in the event results. A historical event's recorded status describes that event's own sections at its date and is not a verdict on newer sections.

  • Model classification: READY_WITH_GUARDRAILS / CLOSED_WELP_CROSS_FAMILY_CONTROL - attributed to the 2026-09-10 event, configuration QAT Q4_0 weights, F16 KV cache · llama.cpp · RTX 5070.
  • Recommended configuration: llama.cpp official QAT Q4_0 (current): event result.
  • Practical context: 32,768 default / 131,072 guarded tokens; native model-card maximum 131,072 (envelope complete: YES); source: event result
  • Context limit outcome: Exact native 131,072: measured FAILED on the strict aggregate useful-context gate while near-full performance itself was valid (both seeds 99.50% occupancy, TTFT ~38 s, decode ~79 tok/s, zero errors) and five-needle target retrieval stayed perfect (10/10 across ALL rungs including the exact maximum); source: event result
  • Latest evidence: 2026-09-10: event result
  • 2026-09-10 · QAT Q4_0 weights, F16 KV cache · llama.cpp · RTX 5070: classification READY_WITH_GUARDRAILS / CLOSED_WELP_CROSS_FAMILY_CONTROL; 3/7 strict gates (markdown-fenced JSON plus a real absent-field grounding defect); content-level supplement: 3/3 needle facts exact, decoy absent, checksum correct. Event result

Event results

Newest first. Each event is one independent testing/publication event with its full report linked once at the top of its section; scores, failures and limitations stay visible and raw section-provenance JSON stays collapsed.

2026-09-10: Initial evaluation (full characterization)

Initial Evaluation: official QAT Q4_0 · config QAT Q4_0 weights, F16 KV cache · llama.cpp · RTX 5070 · maturity: FULL_EVALUATION · status: READY_WITH_GUARDRAILS / CLOSED_WELP_CROSS_FAMILY_CONTROL

Configuration: llama.cpp official QAT Q4_0.

Full report (REPORT.md)

WELP outcome

  • Outcome: PASS: GEMMA4_E4B_EVIDENCE_COMPLETION_COMPLETE
  • Classification: READY_WITH_GUARDRAILS
  • Artifact classification: Official Google QAT Q4_0 + official mmproj; the only official precision that hosts the full card range with reserve on this GPU; community Q4_K_M pair retained as evidence only (provenance unknown)

Publication state: published; canonical evidence linked above

Headline performance

SurfaceTTFTPrefillDecode
Short0.0225 sNot recorded150.4 tok/s
Moderate (2785-token input)0.504 s7217 tok/s145.7 tok/s

Medians over five measured repetitions per surface; near-full ladder values are medians of two seeds per rung

Quality and capabilities

  • Constrained result: 3/7 strict gates (markdown-fenced JSON plus a real absent-field grounding defect); content-level supplement: 3/3 needle facts exact, decoy absent, checksum correct
  • Multimodal on 12 GB: vision TESTED_PASS, multi-image TESTED_PASS, audio ASR TESTED_PASS, audio AST TESTED_PASS, bounded three-frame video TESTED_PASS
  • Reasoning (thinking mode), coding (4/4 execution cases), function calling, and bounded tool-result use TESTED_PASS
  • Perfect five-needle retrieval through the exact 131,072 maximum; 80 valid scientific requests across 9 supervised arms with zero CUDA/OOM/Xid
Guardrails and limitations
  • Strict-format fragility: fenced JSON when markdown is not explicitly forbidden; schema-tail truncation from 16K upward; absent-field (negative-space) answers unreliable at temperature 0
  • OCR/document understanding TESTED_LIMITED (one document ID digit dropped at temperature 0; broader OCR types untested)
  • Video evidence bounded to frame-sequence probes (full <=60 s @1fps producer envelope not established); MTP/speculative deferred (distinct serving surface)

Reliability: Canonical 32K profile: 20/20 runtime completion with zero errors; behavioral exactness 14/20; all six failures are the known dossier strict-format/absent defect, bit-stable across arms; bounded, not soak/endurance

LocalMaxxing

  • Status: SUBMITTED
  • Canonical context: 32,768 tokens
  • tok/s out: 152.7
  • TTFT: 59.74 ms
  • Submission reference: cmtwgcs7c097sps01sdq9ihbu
  • verifiedRun: null (not claimed)

SUBMITTED once, origin NEW, service APPROVED (in-campaign, 2026-09-11); verifiedRun null; the service returned no verification state and the client could not transmit verification fields; recorded honestly, not claimed as verified; actual prompt tokens 77 (endpoint usage); tokSPrefill 1,322.5

Tested condition: artifact QAT Q4_0 weights, F16 KV cache · llama.cpp · RTX 5070 · context 32,768 default / 131,072 guarded · native model-card maximum 131,072 tokens · envelope complete: YES.

Context limit outcome: Exact native 131,072: measured FAILED on the strict aggregate useful-context gate while near-full performance itself was valid (both seeds 99.50% occupancy, TTFT ~38 s, decode ~79 tok/s, zero errors) and five-needle target retrieval stayed perfect (10/10 across ALL rungs including the exact maximum)

Context boundaries: Strict aggregate gate FAILED at every rung; divergence is explicit: content retrieval is perfect everywhere (decoy resistance 10/10), while strict output/instruction compliance is the failure surface (absent-field fills a present value instead of NOT_SPECIFIED; JSON tail truncation from 16K upward). No official context extensions exist for E4B (source-backed absence).

Runtime conditions: All 42 layers + PLE tables + vision/audio encoders GPU-resident; official mmproj loaded for multimodal arms; greedy temperature-0 primary surface

Hardware conditions: Single-user workstation; AMD Ryzen 7 9800X3D; Fedora Linux 44

Full tested identity, runtime and context details (exact hashes, versions and configuration)

Identity

FieldValue
Profile identitygemma4-e4b-llamacpp-qat-q4-0
ModelGemma 4 E4B
ProducerGoogle
Official modelgoogle/gemma-4-E4B-it @ ee0ef6023621cff504d758262d4e04895a5af4a2
Tested artifactgemma-4-E4B_q4_0-it.gguf + official gemma-4-E4B-it-mmproj.gguf (Google official QAT release)
PrecisionQAT Q4_0 weights, F16 KV cache
Artifact hash evidence676c3507 (weights; matches pinned upstream LFS) / 7498a37c (mmproj; matches pinned upstream LFS)
Campaigngemma4-e4b-rtx5070-welp-characterization-2026-09-10
Record date2026-09-10

Runtime and hardware

FieldValue
Enginellama.cpp 0.1.0-dev build b10449, commit 0d9ceae1e38291035605613ab41a8f5e693d6fcd (campaign-local CUDA 13.3 build, SM120)
HardwareWumboJetsII (NVIDIA GeForce RTX 5070 12GB)
Runtime notesAll 42 layers + PLE tables + vision/audio encoders GPU-resident; official mmproj loaded for multimodal arms; greedy temperature-0 primary surface
Hardware notesSingle-user workstation; AMD Ryzen 7 9800X3D; Fedora Linux 44

Context profile

FieldValue
Practical default32768 tokens
Guarded context131072 tokens
Native model-card maximum131072 tokens
Model-card envelope completeYES
Native maximum dispositionExact native 131,072: measured FAILED on the strict aggregate useful-context gate while near-full performance itself was valid (both seeds 99.50% occupancy, TTFT ~38 s, decode ~79 tok/s, zero errors) and five-needle target retrieval stayed perfect (10/10 across ALL rungs including the exact maximum)

Strict aggregate gate FAILED at every rung; divergence is explicit: content retrieval is perfect everywhere (decoy resistance 10/10), while strict output/instruction compliance is the failure surface (absent-field fills a present value instead of NOT_SPECIFIED; JSON tail truncation from 16K upward). No official context extensions exist for E4B (source-backed absence).

Limitations

  • Strict-format fragility: fenced JSON when markdown is not explicitly forbidden; schema-tail truncation from 16K upward; absent-field (negative-space) answers unreliable at temperature 0 Source events: 2026-09-10.
  • OCR/document understanding TESTED_LIMITED (one document ID digit dropped at temperature 0; broader OCR types untested) Source events: 2026-09-10.
  • Video evidence bounded to frame-sequence probes (full <=60 s @1fps producer envelope not established); MTP/speculative deferred (distinct serving surface) Source events: 2026-09-10.

Tested configurations

Exact profile identities, sampling conditions and provenance are in each configuration's details below.

ConfigurationArtifact / precisionRuntimeHardwareContext / reasoningEvents
QAT Q4_0 weights, F16 KV cache · llama.cpp · RTX 5070QAT Q4_0 weights, F16 KV cachellama.cppRTX 507032,768 default / 131,072 guarded1 (2026-09-10)
Configuration details: QAT Q4_0 weights, F16 KV cache · llama.cpp · RTX 5070
  • Full profile name: llama.cpp official QAT Q4_0.
  • Profile identity: gemma4-e4b-llamacpp-qat-q4-0.
  • Status: current canonical/recommended tested surface.
  • Artifact (as recorded): gemma-4-E4B_q4_0-it.gguf + official gemma-4-E4B-it-mmproj.gguf (Google official QAT release)
  • Runtime (as recorded): llama.cpp 0.1.0-dev build b10449, commit 0d9ceae1e38291035605613ab41a8f5e693d6fcd (campaign-local CUDA 13.3 build, SM120).
  • Hardware (as recorded): WumboJetsII (NVIDIA GeForce RTX 5070 12GB).
  • Profile Metadata

Events on this configuration:

Chronology

Compact index; full results are in Event results.

DateEventConfigurationStatus
2026-09-10Initial evaluation (full characterization)QAT Q4_0 weights, F16 KV cache · llama.cpp · RTX 5070READY_WITH_GUARDRAILS / CLOSED_WELP_CROSS_FAMILY_CONTROL

Technical provenance

Publication, acceptance and source-pin provenance for each event, collapsed; the scientific content above never depends on it.

Model evidence tree (GitHub)

Event provenance: 2026-09-10 Initial Evaluation: official QAT Q4_0
  • Campaign: gemma4-e4b-rtx5070-welp-characterization-2026-09-10
  • Export disposition: WEBSITE_READY; publication date 2026-09-11
  • Canonical citation (immutable commit/path pin): https://github.com/WumboLabs/evaluations/blob/479cb50c197f8ea0dd905a68ffd61f632f49c652/models/gemma4-e4b/events/initial-evaluation-2026-09-10-gemma4/REPORT.md
  • Export source pin: repository-cache WumboLabs/evaluations @ 479cb50c197f8ea0dd905a68ffd61f632f49c652 path models/gemma4-e4b/events/initial-evaluation-2026-09-10-gemma4/website-publication.json sha256 45aca9eb0ff421280e04e6137cdcdc8571e40754a89c5f4e705b410ab7a052bb

Legacy provenance