WUMBOLABS / EVALUATIONS CANONICAL MODEL EVALUATION

/evaluations/lfm2.5-2.6b

LFM2.5 2.6B

Published evaluation evidence and testing history. Results remain attributed to their original configurations and methods.

latest evidence 2026-08-26

Overview

Liquid AI · 1 tested configuration · 1 evidence event · newest evidence 2026-08-26.

Contents: Findings · Event results · Tested configurations · Chronology · Technical provenance

Findings

Each finding is attributed to the event and tested configuration that measured it; exact values, failures and limitations are in the event results. A historical event's recorded status describes that event's own sections at its date and is not a verdict on newer sections.

  • Model classification: COMPLETE_SMALL_MODEL_EVALUATION - attributed to the 2026-08-26 event, configuration QAD Q4_0 · llama.cpp · RTX 5070.
  • Recommended configuration: llama.cpp QAD Q4_0 (current): event result.
  • Latest evidence: 2026-08-26: event result
  • 2026-08-26 · QAD Q4_0 · llama.cpp · RTX 5070: classification COMPLETE_SMALL_MODEL_EVALUATION. Event result

Event results

Newest first. Each event is one independent testing/publication event with its full report linked once at the top of its section; scores, failures and limitations stay visible and raw section-provenance JSON stays collapsed.

2026-08-26: Initial evaluation (small-model evaluation)

Initial Evaluation: llama.cpp QAD Q4_0 · config QAD Q4_0 · llama.cpp · RTX 5070 · maturity: FULL_EVALUATION · status: COMPLETE_SMALL_MODEL_EVALUATION

Configuration: llama.cpp QAD Q4_0.

Full report (REPORT.md)

Hardware
FieldValue
MachineWumboJetsII
GPUNVIDIA GeForce RTX 5070 12GB (SM120)
VRAM12,227 MiB physical
CPUAMD Ryzen 7 9800X3D
OSFedora Linux 44
Driver / CUDANVIDIA 610.57.04
Runtimellama.cpp b10449 (commit 0d9ceae1e38291035605613ab41a8f5e693d6fcd)
Headline Verdict

QAD Q4_0 is the rational ultra-fast deployment quant when 1.59 GB is the hard model-size target, but this campaign did not reproduce a material practical-quality win over ordinary PTQ Q4_0.

QAD exactly retained PTQ Q4_0's 4K GPU footprint and throughput, and it improved the fixed-corpus perplexity diagnostic. The small practical, tool, and coding lanes do not establish the producer's 96.6% aggregate-retention or Q4_K_M-equivalence claims.

Performance
QuantPeak GPU MiBPrompt tok/sDecode tok/sTTFT ms
BF1661001498.0105.120.7
PTQ Q4_024043235.4282.79.6
QAD Q4_024043221.8282.59.6
Q4_K_M24802777.0263.111.2
Q5_K_M27342649.9240.511.7

QAD is 0.07% slower than PTQ decode (within run variance) and has identical observed peak allocation. This reproduces the native-Q4_0 speed/memory claim.

Fidelity

llama-perplexity on fixed local repeated corpora found QAD lower than PTQ (natural 1.0408 vs 1.0882; code 1.1248 vs 1.1412). QAD also scored below BF16 on those corpora. This is evidence of a QAD-vs-PTQ log-probability improvement, not a BF16-quality percentage.

Practical Behavior

A compact mechanically scored exact-match set tested instruction following, JSON, absent-evidence responses, and document extraction. Results were noisy and small:

  • Structured output: no demonstrated QAD improvement over PTQ; QAD 1/2 vs PTQ 2/2.
  • Hallucination resistance: no demonstrated QAD improvement; QAD 1/2 vs PTQ 2/2.
  • Extraction/RAG: QAD 1/2 vs PTQ 0/2, but too small for a deployment claim.
Coding

Twenty isolated sandbox executions per primary quant. Each produced 4/20 execution successes. The suite exposed brittle single-code-block compliance and supports Liquid's warning that 2.6B is not for agentic coding. Coding is a poor fit regardless of the observed QAD quantization change.

Native Tools

All four compared quants passed the five deterministic local schema cases: 5/5. QAD did not improve this already-saturated simple suite. Nested-object, tool-result continuation, and recovery after an error remain unverified.

Context

All three finalists allocated and prefixed prompts through 127,777 actual tokens. QAD and PTQ retrieved the beginning needle at approximately 3.9K and 8.0K tokens, but neither produced a final answer above 16K within a 256-token generation budget. Useful long-context behavior is not established beyond 8K for QAD/PTQ.

Stability

The loopback QAD server completed 1,808.7 seconds (30m 8.7s), 175 varied requests, zero request errors. It shut down cleanly and GPU allocation returned to desktop baseline. No Xid/reset/GSP/channel-failure entries.

Role Classification

Supported roles:

  • Always-loaded router/extraction/tool helper with schema validation and evidence checks
  • Ultra-fast sidecar (when 1.59 GB is the hard size target)

Not supported:

  • Primary assistant
  • Long-context retriever
  • Coding agent
Claim Classifications
ClaimClassification
QAD preserves Q4_0 memoryREPRODUCED
QAD preserves Q4_0 decode throughputREPRODUCED
QAD materially beats ordinary PTQ Q4_0PARTIALLY_REPRODUCED: lower local PPL; no material practical win shown
QAD retains 96.6% BF16 aggregateINCONCLUSIVE
QAD recovers 48.4% of PTQ gapINCONCLUSIVE
QAD matches Q4_K_M qualityINCONCLUSIVE
QAD improves native toolsINCONCLUSIVE: all simple cases saturated
QAD improves structured outputNOT_REPRODUCED
QAD improves hallucination resistanceNOT_REPRODUCED
QAD improves long contextNOT_REPRODUCED
  • Canonical evaluation repo: https://github.com/WumboLabs/evaluations/tree/4f93731a9f042b9c1d86718a98a4e0e13d0fbd21/models/lfm2.5-2.6b/profiles/lfm2.5-2.6b-llamacpp-qad-q4-0
  • WELP protocol: https://github.com/WumboLabs/welp
  • Labs catalog: https://github.com/WumboLabs/evaluations/tree/4f93731a9f042b9c1d86718a98a4e0e13d0fbd21/models
Reproduction

Direct link to canonical reproduction material: Central evidence

This Lab Record is a summary; the canonical repo is the source of truth.

Identity
FieldValue
ModelLFM2.5 2.6B
ProducerLiquid AI
Evaluated artifactQAD Q4_0 from LiquidAI/LFM2.5-2.6B-GGUF @ f4a289c8a200a5ca71005ba7abc2dad33058a450
SHA-256a247afd6414918eac8e520a9e6137dc271235461ecbe1180462221d5b8d40b03
Evaluation date2026-08-26
ProtocolWELP
Campaign classificationCOMPLETE_SMALL_MODEL_EVALUATION

Tested configurations

Exact profile identities, sampling conditions and provenance are in each configuration's details below.

ConfigurationArtifact / precisionRuntimeHardwareContext / reasoningEvents
QAD Q4_0 · llama.cpp · RTX 5070QAD Q4_0llama.cppRTX 5070not recorded1 (2026-08-26)
Configuration details: QAD Q4_0 · llama.cpp · RTX 5070
  • Full profile name: llama.cpp QAD Q4_0.
  • Profile identity: lfm2.5-2.6b-llamacpp-qad-q4-0.
  • Status: current canonical/recommended tested surface.
  • Artifact (as recorded): QAD Q4_0 from LiquidAI/LFM2.5-2.6B-GGUF @ f4a289c8a200a5ca71005ba7abc2dad33058a450
  • Runtime (as recorded): llama.cpp.
  • Profile Metadata

Events on this configuration:

Chronology

Compact index; full results are in Event results.

DateEventConfigurationStatus
2026-08-26Initial evaluation (small-model evaluation)QAD Q4_0 · llama.cpp · RTX 5070COMPLETE_SMALL_MODEL_EVALUATION

Technical provenance

Publication, acceptance and source-pin provenance for each event, collapsed; the scientific content above never depends on it.

Model evidence tree (GitHub)

Legacy provenance