WUMBOLABS / EVALUATIONS CANONICAL MODEL EVALUATION

/evaluations/qwen2.5-3b

Qwen2.5-3B Instruct

Published evaluation evidence and testing history. Results remain attributed to their original configurations and methods.

latest evidence 2026-07-15

Overview

Alibaba (Qwen team) · 2 tested configurations · 2 evidence events · newest evidence 2026-07-15.

Contents: Findings · Event results · Tested configurations · Chronology · Technical provenance

Findings

Each finding is attributed to the event and tested configuration that measured it; exact values, failures and limitations are in the event results. A historical event's recorded status describes that event's own sections at its date and is not a verdict on newer sections.

  • Recommended configuration: vLLM BF16 (cross-runtime lane) (current): event result.
  • Latest evidence: 2026-07-15: event result
  • 2026-07-15 · vLLM · RTX 5070: classification SPECIALIZED_TEST / RUNTIME_FINGERPRINT. Event result
  • 2026-07-15 · llama.cpp · RTX 5070: classification BENCHMARK_ONLY / CROSS_RUNTIME_METHODOLOGY_LANE. Event result

Event results

Newest first. Each event is one independent testing/publication event with its full report linked once at the top of its section; scores, failures and limitations stay visible and raw section-provenance JSON stays collapsed.

2026-07-15: vLLM fingerprint live smoke

Specialized Test: runtime fingerprint · config vLLM · RTX 5070 · maturity: SPECIALIZED_TEST · status: SPECIALIZED_TEST / RUNTIME_FINGERPRINT

Configuration: vLLM BF16 (cross-runtime lane).

Full report (REPORT.md)

vLLM runtime-fingerprint smoke using Qwen2.5-3B as payload.

This event is a bounded public-safe summary derived from retained WumboLabs evidence. It is not a WELP characterization, a universal model ranking, or a production-readiness proof. Values are attributed to the tested artifact, runtime, hardware, suite, and settings.

Identity and scope
  • Profile: qwen2.5-3b-vllm-bf16: vLLM BF16 (cross-runtime lane)
  • Evidence maturity: SPECIALIZED_TEST
  • Evidence scope: specialized
  • Hardware: WumboJetsII (RTX 5070 12GB)

2026-07-15: Cross-runtime comparison: llama.cpp F16 vs vLLM BF16

Benchmark Only: cross-runtime methodology lane · config llama.cpp · RTX 5070 · maturity: BENCHMARK_ONLY · status: BENCHMARK_ONLY / CROSS_RUNTIME_METHODOLOGY_LANE

Configuration: llama.cpp F16 (comparison reference).

Full report (REPORT.md)

LLMGauge cross-runtime methodology study with model-specific results: Qwen2.5-3B Instruct as F16 llama.cpp vs BF16 vLLM agent-backend comparison incl. failed-command-recovery probes. Primary subject is cross-runtime methodology; the Qwen2.5-3B results are retained as its benchmark-only evidence. Archaeology separately cites a full Qwen2.5-3B quant-lab record set (F16/Q8_0/Q6_K/Q5_K_M/Q4_K_M/IQ4_XS) from the Arch era.

This event is a bounded public-safe summary derived from retained WumboLabs evidence. It is not a WELP characterization, a universal model ranking, or a production-readiness proof. Values are attributed to the tested artifact, runtime, hardware, suite, and settings.

Identity and scope
  • Profile: qwen2.5-3b-llamacpp-f16: llama.cpp F16 (comparison reference)
  • Evidence maturity: BENCHMARK_ONLY
  • Evidence scope: performance, agent-backend
  • Hardware: WumboJetsII (RTX 5070 12GB)

Tested configurations

Exact profile identities, sampling conditions and provenance are in each configuration's details below.

ConfigurationArtifact / precisionRuntimeHardwareContext / reasoningEvents
llama.cpp · RTX 5070Qwen2.5-3B-Instruct F16 GGUF (6,178,317,312-byte fingerprint)llama.cppRTX 5070not recorded1 (2026-07-15)
vLLM · RTX 5070Qwen/Qwen2.5-3B-Instruct BF16vLLMRTX 5070not recorded1 (2026-07-15)
Configuration details: llama.cpp · RTX 5070
  • Full profile name: llama.cpp F16 (comparison reference).
  • Profile identity: qwen2.5-3b-llamacpp-f16.
  • Status: current tested surface; not the recommended profile.
  • Artifact (as recorded): Qwen2.5-3B-Instruct F16 GGUF (6,178,317,312-byte fingerprint)
  • Runtime (as recorded): llama.cpp.
  • Profile Metadata

Events on this configuration:

Configuration details: vLLM · RTX 5070
  • Full profile name: vLLM BF16 (cross-runtime lane).
  • Profile identity: qwen2.5-3b-vllm-bf16.
  • Status: current canonical/recommended tested surface.
  • Artifact (as recorded): Qwen/Qwen2.5-3B-Instruct BF16
  • Runtime (as recorded): vLLM.
  • Profile Metadata

Events on this configuration:

Chronology

Compact index; full results are in Event results.

DateEventConfigurationStatus
2026-07-15vLLM fingerprint live smokevLLM · RTX 5070SPECIALIZED_TEST / RUNTIME_FINGERPRINT
2026-07-15Cross-runtime comparison: llama.cpp F16 vs vLLM BF16llama.cpp · RTX 5070BENCHMARK_ONLY / CROSS_RUNTIME_METHODOLOGY_LANE

Technical provenance

Publication, acceptance and source-pin provenance for each event, collapsed; the scientific content above never depends on it.

Model evidence tree (GitHub)

Legacy provenance