WUMBOLABS / EVALUATIONS CANONICAL MODEL EVALUATION

/evaluations/nemotron-3-nano-4b

Nemotron 3 Nano 4B

Published evaluation evidence and testing history. Results remain attributed to their original configurations and methods.

latest evidence 2026-09-11

Overview

NVIDIA · 1 tested configuration · 2 evidence events · newest evidence 2026-09-11.

Contents: Findings · Event results · Limitations · Tested configurations · Chronology · Technical provenance

Findings

Each finding is attributed to the event and tested configuration that measured it; exact values, failures and limitations are in the event results. A historical event's recorded status describes that event's own sections at its date and is not a verdict on newer sections.

  • Model classification: READY_WITH_GUARDRAILS - attributed to the 2026-09-11 event, configuration Q4_K_M weights; f16 KV cache; dense hybrid · llama.cpp · RTX 5070.
  • Recommended configuration: llama.cpp Q4_K_M (current): event result.
  • Practical context: 32,768 default / 131,072 guarded tokens; native model-card maximum 262,144 (envelope complete: YES); source: event result
  • Context limit outcome: VALIDATED: capacity admitted, near-full performance measured (uncached prefill ~2,877 tok/s, decode ~79-81 tok/s, 7,268 MiB VRAM), strict useful-context PASS at both required seeds (42, 314159); source: event result
  • Latest evidence: 2026-09-11: event result
  • 2026-09-11 · Q4_K_M weights; f16 KV cache; dense hybrid · llama.cpp · RTX 5070: classification READY_WITH_GUARDRAILS; frozen 12-task model-neutral screen 12/12 PASS (mechanical scorer frozen before outputs; temp 0, seed 42). Event result
  • 2026-08-25 · Q4_K_M weights; f16 KV cache; dense hybrid · llama.cpp · RTX 5070: classification PARTIAL_PROTOCOL_DEVELOPMENT. Event result

Event results

Newest first. Each event is one independent testing/publication event with its full report linked once at the top of its section; scores, failures and limitations stay visible and raw section-provenance JSON stays collapsed.

2026-09-11: Current-WELP recharacterization

WELP Recharacterization: llama.cpp Q4_K_M · config Q4_K_M weights; f16 KV cache; dense hybrid · llama.cpp · RTX 5070 · maturity: CURRENT_WELP · status: READY_WITH_GUARDRAILS

Configuration: llama.cpp Q4_K_M.

Full report (REPORT.md)

WELP outcome

  • Outcome: PASS: NEMOTRON3_NANO_4B_RTX5070_RECHARACTERIZED
  • Classification: READY_WITH_GUARDRAILS (reliability is the guardrail)
  • Artifact classification: Canonical official llama.cpp deployment surface (sole official quant); BF16 fit-limited on this GPU at the tested context targets; Q4_K_M quality confound disclosed

Publication state: published; canonical evidence linked above

Headline performance

SurfaceTTFTPrefillDecode
Short0.046 sNot recorded186.6 tok/s
Moderate (3880-token input)0.507 s7498 tok/s182.9 tok/s

Short/moderate values are campaign-measured streaming surfaces; context-rung values are near-full occupancy measurements with unique prompts (cached tokens 0 at the native-maximum rungs)

Quality and capabilities

  • Constrained result: frozen 12-task model-neutral screen 12/12 PASS (mechanical scorer frozen before outputs; temp 0, seed 42)
  • Reasoning TESTED_PASS in the official thinking mode; coding TESTED_PASS on executable frozen cases; tool calling TESTED_PASS (structural validity + grounded continuation, bounded, not autonomous-agent qualification)
  • Strongest near-full strict-compliance context result in the current control set: strict useful-context aggregate PASS at every rung through 99.49% of the native 262,144-token maximum
  • First hybrid Mamba-2 model characterized under current WELP on this host; measured allocation confirms fixed recurrent state plus 4-layer context-linear KV
  • Zero OOM, zero CUDA errors, zero Xid; peak 247-249 W within the stock 250 W envelope
Guardrails and limitations
  • Behavioral reliability has measurable hallucination/evidence-discipline defects under the adversarial corpus: clean-pass 0.30/0.25 and hallucination 0.20/0.15 across seeds 42/314159 on a 20-request stratified sample of the proven mechanical reliability corpus (transport success 40/40)
  • The model confidently fabricated a nonexistent CUDA API and capitulated to a false user assertion; independently verify technical/factual claims in adversarial or evidence-discipline-sensitive contexts
  • The 12/12 bounded quality screen is not evidence of universal reliability; the strict near-full context PASS is not proof of universal long-document reasoning
  • Q4_K_M quantization confound disclosed, not measured against BF16

Reliability: Transport/runtime reliability strong; bounded quality/structured behavior clean; adversarial behavioral reliability materially weaker; the historical Phase-4 weakness is reproduced on the current instrument and remains the mandatory guardrail

LocalMaxxing

  • Status: SUBMITTED
  • Canonical context: 32,768 tokens
  • tok/s out: 185.8
  • TTFT: 125.1 ms
  • Submission reference: cmt86gy1j000eli01ca3cwwn0
  • verifiedRun: NO

Exact canonical practical profile (Q4_K_M, llama.cpp b10449, 32,768 context, f16 KV, full GPU, RTX 5070) already submitted and approved 2026-08-25; per the never-duplicate rule zero new submissions were created. A fresh 5-rep local benchmark (tg128 191.0 tok/s, pp512 8,275 tok/s) reproduces the submitted result.

Tested condition: artifact Q4_K_M weights; f16 KV cache; dense hybrid · llama.cpp · RTX 5070 · context 32,768 default / 131,072 guarded · native model-card maximum 262,144 tokens · envelope complete: YES.

Context limit outcome: VALIDATED: capacity admitted, near-full performance measured (uncached prefill ~2,877 tok/s, decode ~79-81 tok/s, 7,268 MiB VRAM), strict useful-context PASS at both required seeds (42, 314159)

Context boundaries: Hybrid Mamba-2 memory model measured: 4 attention layers carry context-linear KV (~16 KiB/token f16); the 38 Mamba-2 layers carry a fixed context-independent recurrent state (~100-300 MiB). The full 262,144 slot fits in 7,268 MiB with 4,546 MiB free

Runtime conditions: Full GPU residency (-ngl 999), parallel=1, FlashAttention on, f16 KV, reasoning-off baseline via the official enable_thinking template kwarg

Hardware conditions: Single-user workstation; AMD Ryzen 7 9800X3D; Fedora Linux

Full tested identity, runtime and context details (exact hashes, versions and configuration)

Identity

FieldValue
Profile identitynemotron-3-nano-4b-llamacpp-q4km
ModelNemotron 3 Nano 4B
ProducerNVIDIA
Official modelnvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF (GGUF) / nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 (BF16 weights) @ ba223d14e45525f7fae81db77ea8cabeb2fc6c25
Tested artifactNVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf; the only official llama.cpp quant NVIDIA publishes for this model
PrecisionQ4_K_M weights; f16 KV cache; dense hybrid (no MoE)
Artifact SHA-256be5d9a656a51922f24f1f09a759cebb694e1f5d9728bf0ef9f8c972c5a0b5ef2
Campaignnemotron3-nano-4b-rtx5070-welp-recharacterization-2026-09-11
Record date2026-09-11

Runtime and hardware

FieldValue
Enginellama.cpp build b10449, commit 0d9ceae1e38291035605613ab41a8f5e693d6fcd (CUDA SM120)
HardwareWumboJetsII (NVIDIA GeForce RTX 5070 12GB)
Runtime notesFull GPU residency (-ngl 999), parallel=1, FlashAttention on, f16 KV, reasoning-off baseline via the official enable_thinking template kwarg
Hardware notesSingle-user workstation; AMD Ryzen 7 9800X3D; Fedora Linux

Context profile

FieldValue
Practical default32768 tokens
Guarded context131072 tokens
Native model-card maximum262144 tokens
Model-card envelope completeYES
Native maximum dispositionVALIDATED: capacity admitted, near-full performance measured (uncached prefill ~2,877 tok/s, decode ~79-81 tok/s, 7,268 MiB VRAM), strict useful-context PASS at both required seeds (42, 314159)

Hybrid Mamba-2 memory model measured: 4 attention layers carry context-linear KV (~16 KiB/token f16); the 38 Mamba-2 layers carry a fixed context-independent recurrent state (~100-300 MiB). The full 262,144 slot fits in 7,268 MiB with 4,546 MiB free

2026-08-25: Initial evaluation (protocol development, PROTOCOL_BLOCKED)

Initial Evaluation: PROTOCOL_BLOCKED (historical, preserved) · config Q4_K_M weights; f16 KV cache; dense hybrid · llama.cpp · RTX 5070 · maturity: PROTOCOL_BLOCKED · status: PARTIAL_PROTOCOL_DEVELOPMENT

Configuration: llama.cpp Q4_K_M.

Full report (REPORT.md)

Hardware
FieldValue
MachineWumboJetsII
GPUNVIDIA GeForce RTX 5070 12GB (SM120)
VRAM12,227 MiB physical
CPUAMD Ryzen 7 9800X3D
OSFedora Linux 44, kernel 7.1.9
Driver / CUDANVIDIA 610.57.04
Runtimellama.cpp b10449 (commit 0d9ceae1e38291035605613ab41a8f5e693d6fcd)
Headline Verdict

PROTOCOL_BLOCKED at Phase 3 gate decision.

The model showed no failing behavior. WELP's documented material contains no Phase 3 advancement threshold, so no deterministic PASS/FAIL could be produced without inventing a rule, which the campaign forbids.

This campaign contributed to protocol development. It is not a complete WELP capability review. Do not present it as equivalent in evaluation depth to Qwen3.8-27B.

Performance
MetricResult
Decode~178 tok/s mean (512-token generations)
Prefill~6,500 tok/s at ~2,600-token prompts
TTFT (short prompt)~30 ms
llama-bench (p512/n128)~180 tok/s out, 65 ms TTFT
Phases Reached
PhaseGateOutcome
0 ProvenanceNot recordedPASS
1 Admission (hard)PASSLoads; full GPU residency verified; clean inference/termination
2 PerformancePASS*Criterion interpreted (F-02)
3 Practical ViabilityPROTOCOL_BLOCKED12/12 screen but protocol defines no advancement threshold (F-01)
4–10, OMP-21BNOT RUNStopped per campaign rule
Practical Viability Evidence

12 tasks across instruction following, strict output, extraction, structured JSON, technical knowledge, false-premise rejection, uncertainty behavior, simple reasoning: 12/12, mechanical scoring. This is screening evidence, not a gate pass.

Reasoning Findings

Bounded observations only (dedicated module not reached): default mode emits <think> traces then answers; enable_thinking=false yields direct correct answers. Producer's reasoning-off benchmark framing is operationally reproducible on this runtime.

Role Classification

No deployment recommendation is supported by this evidence.

What the reached phases establish:

  • The model loaded cleanly and served stably across the tested phases
  • Performance was measured
  • The Phase-3 screening result was measured: 12/12
  • OMP connectivity was demonstrated
  • The reasoning-control mechanism was boundedly observed

Explicitly NOT TESTED: reliability, capability modules, context quality, variance, soak, and agent capability.

Important Limitations
  • Single-seed practical screen. Reliability, context quality, coding, tools, agent behavior, and soak were NOT TESTED.
  • Context wording: the server was configured with a 32K context window and inference ran within it. Useful 32K context behavior was not evaluated.
  • No LLMGauge Agent Harness result. LLMGauge was never invoked.
  • Protocol findings: this campaign identified 9 protocol ambiguities (F-01 through F-09) and 4 undocumented decisions required. These are documented in the canonical repo.
  • Results must not be generalized to a model review.
  • Canonical evaluation repo: https://github.com/WumboLabs/evaluations/tree/4f93731a9f042b9c1d86718a98a4e0e13d0fbd21/models/nemotron-3-nano-4b/profiles/nemotron-3-nano-4b-llamacpp-q4km
  • WELP protocol: https://github.com/WumboLabs/welp
  • Labs catalog: https://github.com/WumboLabs/evaluations/tree/4f93731a9f042b9c1d86718a98a4e0e13d0fbd21/models
  • LocalMaxxing: Speed run measured locally (~180 tok/s); submission blocked on missing credentials. No benchmark suite submissions.
Reproduction

Direct link to canonical reproduction material: Central evidence

This Lab Record is a summary; the canonical repo is the source of truth.

Identity
FieldValue
ModelNemotron 3 Nano 4B
ProducerNVIDIA
Evaluated artifactNVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf
Source revisionba223d14e45525f7fae81db77ea8cabeb2fc6c25
SHA-256be5d9a656a51922f24f1f09a759cebb694e1f5d9728bf0ef9f8c972c5a0b5ef2
Evaluation date2026-08-25
ProtocolWELP, first deliberate WELP validation campaign
Highest phase reachedPhase 3 (gate decision)
Campaign classificationPARTIAL_PROTOCOL_DEVELOPMENT

Limitations

  • Behavioral reliability has measurable hallucination/evidence-discipline defects under the adversarial corpus: clean-pass 0.30/0.25 and hallucination 0.20/0.15 across seeds 42/314159 on a 20-request stratified sample of the proven mechanical reliability corpus (transport success 40/40) Source events: 2026-09-11.
  • The model confidently fabricated a nonexistent CUDA API and capitulated to a false user assertion; independently verify technical/factual claims in adversarial or evidence-discipline-sensitive contexts Source events: 2026-09-11.
  • The 12/12 bounded quality screen is not evidence of universal reliability; the strict near-full context PASS is not proof of universal long-document reasoning Source events: 2026-09-11.
  • Q4_K_M quantization confound disclosed, not measured against BF16 Source events: 2026-09-11.

Tested configurations

Exact profile identities, sampling conditions and provenance are in each configuration's details below.

ConfigurationArtifact / precisionRuntimeHardwareContext / reasoningEvents
Q4_K_M weights; f16 KV cache; dense hybrid · llama.cpp · RTX 5070Q4_K_M weightsllama.cppRTX 507032,768 default / 131,072 guarded2 (2026-08-25, 2026-09-11)
Configuration details: Q4_K_M weights; f16 KV cache; dense hybrid · llama.cpp · RTX 5070
  • Full profile name: llama.cpp Q4_K_M.
  • Profile identity: nemotron-3-nano-4b-llamacpp-q4km.
  • Status: current canonical/recommended tested surface.
  • Artifact (as recorded): NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf (sole official quant; upstream rev ba223d14e45525f7fae81db77ea8cabeb2fc6c25)
  • Runtime (as recorded): llama.cpp build b10449, commit 0d9ceae1e38291035605613ab41a8f5e693d6fcd (CUDA SM120).
  • Hardware (as recorded): WumboJetsII (NVIDIA GeForce RTX 5070 12GB).
  • Profile Metadata

Events on this configuration:

Chronology

Compact index; full results are in Event results.

DateEventConfigurationStatus
2026-09-11Current-WELP recharacterizationQ4_K_M weights; f16 KV cache; dense hybrid · llama.cpp · RTX 5070READY_WITH_GUARDRAILS
2026-08-25Initial evaluation (protocol development, PROTOCOL_BLOCKED)Q4_K_M weights; f16 KV cache; dense hybrid · llama.cpp · RTX 5070PARTIAL_PROTOCOL_DEVELOPMENT

Technical provenance

Publication, acceptance and source-pin provenance for each event, collapsed; the scientific content above never depends on it.

Model evidence tree (GitHub)

Event provenance: 2026-09-11 WELP Recharacterization: llama.cpp Q4_K_M
  • Campaign: nemotron3-nano-4b-rtx5070-welp-recharacterization-2026-09-11
  • Export disposition: WEBSITE_READY; publication date 2026-09-12
  • Canonical citation (immutable commit/path pin): https://github.com/WumboLabs/evaluations/blob/479cb50c197f8ea0dd905a68ffd61f632f49c652/models/nemotron-3-nano-4b/events/welp-recharacterization-2026-09-11/REPORT.md
  • Export source pin: repository-cache WumboLabs/evaluations @ 479cb50c197f8ea0dd905a68ffd61f632f49c652 path models/nemotron-3-nano-4b/events/welp-recharacterization-2026-09-11/website-publication.json sha256 457ef6f472770425857963f5cb899dc4d7f6dd6ddb0bb12cbe33b54511ae07fa

Legacy provenance