WUMBOLABS / EVALUATIONS CANONICAL MODEL EVALUATION

/evaluations/minicpm5-2b

MiniCPM5-2B

Published evaluation evidence and testing history. Results remain attributed to their original configurations and methods.

latest evidence 2026-09-10

Overview

OpenBMB · 1 tested configuration · 1 evidence event · newest evidence 2026-09-10.

Contents: Findings · Event results · Limitations · Tested configurations · Chronology · Technical provenance

Findings

Each finding is attributed to the event and tested configuration that measured it; exact values, failures and limitations are in the event results. A historical event's recorded status describes that event's own sections at its date and is not a verdict on newer sections.

  • Model classification: READY_WITH_GUARDRAILS - attributed to the 2026-09-10 event, configuration BF16 weights and BF16 KV · vLLM · RTX 5070.
  • Recommended configuration: contained vLLM BF16 (current): event result.
  • Practical context: 32,768 default / 65,536 guarded tokens; native model-card maximum 131,072 (envelope complete: YES); source: event result
  • Context limit outcome: Exact 131,072 carries two completed dispositions: FIT_LIMIT on the BF16-KV surface (1,024 MiB reserve floor caps the pool below the maximum) and measured FAILED (strict useful-context gate) on the authorized fp8-KV alternate surface executed at 99.50% occupancy; the highest admitted BF16-KV rung was 98,304; source: event result
  • Latest evidence: 2026-09-10: event result
  • 2026-09-10 · BF16 weights and BF16 KV · vLLM · RTX 5070: classification READY_WITH_GUARDRAILS; 6/7 constrained checks; one deterministic temperature-0 over-refusal of a benign prime-list prompt failed the repeat-check content scorer; repeat consistency itself held; no repair attempted (frozen contract). Event result

Event results

Newest first. Each event is one independent testing/publication event with its full report linked once at the top of its section; scores, failures and limitations stay visible and raw section-provenance JSON stays collapsed.

2026-09-10: Initial evaluation (full characterization)

Initial Evaluation: contained vLLM BF16 · config BF16 weights and BF16 KV · vLLM · RTX 5070 · maturity: FULL_EVALUATION · status: READY_WITH_GUARDRAILS

Configuration: contained vLLM BF16.

Full report (REPORT.md)

WELP outcome

  • Outcome: PASS: MINICPM5_2B_RTX5070_CHARACTERIZED; MODEL-CARD CONTEXT ENVELOPE COMPLETE = YES
  • Classification: READY_WITH_GUARDRAILS
  • Artifact classification: Official BF16 full-precision candidate (characterized; not deployed); architecture-diversity control value HIGH

Publication state: published; canonical evidence linked above

Headline performance

SurfaceTTFTPrefillDecode
Short0.013037 sNot recorded118.622 tok/s
Moderate (2663-token input)0.203597 s13079.8 tok/s116.95 tok/s

Medians over five measured repetitions per surface at 8,192 baseline context; near-full ladder medians of two seeds per rung retained in campaign evidence

Quality and capabilities

  • Constrained result: 6/7 constrained checks; one deterministic temperature-0 over-refusal of a benign prime-list prompt failed the repeat-check content scorer; repeat consistency itself held; no repair attempted (frozen contract)
  • Thinking/reasoning TESTED_PASS; coding TESTED_PASS (4/4 frozen execution cases); native tool selection and tool-result use TESTED_PASS
  • English/Chinese text, instruction following, and structured output TESTED_PASS
  • Near-full performance inside practical gates through 98,304 (BF16-KV) with zero errors across 55 scientific requests
Guardrails and limitations
  • Strict long-context output compliance fails at all rungs (format + absent-field), even though content-level retrieval is strong
  • Deterministic temperature-0 over-refusal quirk observed on a benign prompt
  • fp8-KV scales uncalibrated (kv_scale 1.0); agent benchmarks deferred as out of scope; not autonomous-agent qualification

Reliability: 20/20 COMPLETE and passed on a fresh default-context server; zero HTTP/runtime/CUDA/OOM/Xid errors; bounded, not endurance certification

LocalMaxxing

  • Status: SUBMITTED
  • Canonical context: 32,768 tokens
  • tok/s out: 118.2
  • TTFT: 22.7 ms
  • Submission reference: cmtwcna5907f0ps01sfiwd60a
  • verifiedRun: NO

APPROVED, origin NEW (localmaxxing-backfill-2026-09-10); benchmarked on the canonical practical stack (BF16, vLLM contained runtime, 32K default, not the 131K fp8 boundary); actual prompt tokens 252 (endpoint usage); verifiedRun false reflects a client capture limitation

Tested condition: artifact BF16 weights and BF16 KV · vLLM · RTX 5070 · context 32,768 default / 65,536 guarded · native model-card maximum 131,072 tokens · envelope complete: YES.

Context limit outcome: Exact 131,072 carries two completed dispositions: FIT_LIMIT on the BF16-KV surface (1,024 MiB reserve floor caps the pool below the maximum) and measured FAILED (strict useful-context gate) on the authorized fp8-KV alternate surface executed at 99.50% occupancy; the highest admitted BF16-KV rung was 98,304

Context boundaries: The strict frozen useful-context gate FAILED at every rung (12/12 requests) with exact attribution: markdown-fenced JSON and an omitted absent field cascade the per-field gates. Supplementary content-level analysis (explicitly not a gate) shows 5/5 exact target retrieval at every depth through 65,536 on both seeds and at 131,072 on one of two seeds. A FAILED gate is a completed negative disposition, not VALIDATED; no rung is claimed VALIDATED.

Runtime conditions: Qualified contained CUDA 13.0 toolchain (compute_120f/sm_120f); full transformer GPU residency; fresh-cache containment proven; greedy temperature-0 primary surface

Hardware conditions: Single-user workstation; AMD Ryzen 7 9800X3D; Fedora Linux 44

Full tested identity, runtime and context details (exact hashes, versions and configuration)

Identity

FieldValue
Profile identityminicpm5-2b-vllm-bf16
ModelMiniCPM5-2B
ProducerOpenBMB
Official modelopenbmb/MiniCPM5-2B @ cd199ce3ee67549c42ef7372f809f2c63599a3e9
Tested artifactmodel-00000-of-00001.safetensors (official BF16; no quantization or conversion)
PrecisionBF16 weights and BF16 KV (primary surface); fp8(e4m3) KV only on the single authorized 131,072 alternate surface
Artifact SHA-25614fb8e7f0a18d53d1f239773758bf581cee7e456a4523a54622c3a245b64402c
Campaignminicpm5-2b-rtx5070-welp-characterization-2026-09-10
Record date2026-09-10

Runtime and hardware

FieldValue
EnginevLLM 0.27.1 (g6e448d0ea), Torch 2.13.0+cu130, Transformers 5.15.0, FlashInfer 0.6.16.post3
HardwareWumboJetsII (NVIDIA GeForce RTX 5070 12GB)
Runtime notesQualified contained CUDA 13.0 toolchain (compute_120f/sm_120f); full transformer GPU residency; fresh-cache containment proven; greedy temperature-0 primary surface
Hardware notesSingle-user workstation; AMD Ryzen 7 9800X3D; Fedora Linux 44

Context profile

FieldValue
Practical default32768 tokens
Guarded context65536 tokens
Native model-card maximum131072 tokens
Model-card envelope completeYES
Native maximum dispositionExact 131,072 carries two completed dispositions: FIT_LIMIT on the BF16-KV surface (1,024 MiB reserve floor caps the pool below the maximum) and measured FAILED (strict useful-context gate) on the authorized fp8-KV alternate surface executed at 99.50% occupancy; the highest admitted BF16-KV rung was 98,304

The strict frozen useful-context gate FAILED at every rung (12/12 requests) with exact attribution: markdown-fenced JSON and an omitted absent field cascade the per-field gates. Supplementary content-level analysis (explicitly not a gate) shows 5/5 exact target retrieval at every depth through 65,536 on both seeds and at 131,072 on one of two seeds. A FAILED gate is a completed negative disposition, not VALIDATED; no rung is claimed VALIDATED.

Limitations

  • Strict long-context output compliance fails at all rungs (format + absent-field), even though content-level retrieval is strong Source events: 2026-09-10.
  • Deterministic temperature-0 over-refusal quirk observed on a benign prompt Source events: 2026-09-10.
  • fp8-KV scales uncalibrated (kv_scale 1.0); agent benchmarks deferred as out of scope; not autonomous-agent qualification Source events: 2026-09-10.

Tested configurations

Exact profile identities, sampling conditions and provenance are in each configuration's details below.

ConfigurationArtifact / precisionRuntimeHardwareContext / reasoningEvents
BF16 weights and BF16 KV · vLLM · RTX 5070BF16 weights and BF16 KV (primary surface)vLLMRTX 507032,768 default / 65,536 guarded1 (2026-09-10)
Configuration details: BF16 weights and BF16 KV · vLLM · RTX 5070
  • Full profile name: contained vLLM BF16.
  • Profile identity: minicpm5-2b-vllm-bf16.
  • Status: current canonical/recommended tested surface.
  • Artifact (as recorded): model-00000-of-00001.safetensors (official BF16; no quantization or conversion)
  • Runtime (as recorded): vLLM 0.27.1 (g6e448d0ea), Torch 2.13.0+cu130, Transformers 5.15.0, FlashInfer 0.6.16.post3.
  • Hardware (as recorded): WumboJetsII (NVIDIA GeForce RTX 5070 12GB).
  • Profile Metadata

Events on this configuration:

Chronology

Compact index; full results are in Event results.

DateEventConfigurationStatus
2026-09-10Initial evaluation (full characterization)BF16 weights and BF16 KV · vLLM · RTX 5070READY_WITH_GUARDRAILS

Technical provenance

Publication, acceptance and source-pin provenance for each event, collapsed; the scientific content above never depends on it.

Model evidence tree (GitHub)

Event provenance: 2026-09-10 Initial Evaluation: contained vLLM BF16
  • Campaign: minicpm5-2b-rtx5070-welp-characterization-2026-09-10
  • Export disposition: WEBSITE_READY; publication date 2026-09-11
  • Canonical citation (immutable commit/path pin): https://github.com/WumboLabs/evaluations/blob/479cb50c197f8ea0dd905a68ffd61f632f49c652/models/minicpm5-2b/events/initial-evaluation-2026-09-10-minicpm5/REPORT.md
  • Export source pin: repository-cache WumboLabs/evaluations @ 479cb50c197f8ea0dd905a68ffd61f632f49c652 path models/minicpm5-2b/events/initial-evaluation-2026-09-10-minicpm5/website-publication.json sha256 20f817b5c706f0f68319bebe02e1766510c93a6c5249acb8104a57549eb7f7e1

Legacy provenance