/evaluations/lfm2.5-2.6b
LFM2.5 2.6B
Published evaluation evidence and testing history. Results remain attributed to their original configurations and methods.
latest evidence 2026-08-26
Overview
Liquid AI · 1 tested configuration · 1 evidence event · newest evidence 2026-08-26.
Contents: Findings · Event results · Tested configurations · Chronology · Technical provenance
Findings
Each finding is attributed to the event and tested configuration that measured it; exact values, failures and limitations are in the event results. A historical event's recorded status describes that event's own sections at its date and is not a verdict on newer sections.
- Model classification: COMPLETE_SMALL_MODEL_EVALUATION - attributed to the 2026-08-26 event, configuration QAD Q4_0 · llama.cpp · RTX 5070.
- Recommended configuration: llama.cpp QAD Q4_0 (current): event result.
- Latest evidence: 2026-08-26: event result
- 2026-08-26 · QAD Q4_0 · llama.cpp · RTX 5070: classification COMPLETE_SMALL_MODEL_EVALUATION. Event result
Event results
Newest first. Each event is one independent testing/publication event with its full report linked once at the top of its section; scores, failures and limitations stay visible and raw section-provenance JSON stays collapsed.
2026-08-26: Initial evaluation (small-model evaluation)
Initial Evaluation: llama.cpp QAD Q4_0 · config QAD Q4_0 · llama.cpp · RTX 5070 · maturity: FULL_EVALUATION · status: COMPLETE_SMALL_MODEL_EVALUATION
Configuration: llama.cpp QAD Q4_0.
Hardware
| Field | Value |
|---|---|
| Machine | WumboJetsII |
| GPU | NVIDIA GeForce RTX 5070 12GB (SM120) |
| VRAM | 12,227 MiB physical |
| CPU | AMD Ryzen 7 9800X3D |
| OS | Fedora Linux 44 |
| Driver / CUDA | NVIDIA 610.57.04 |
| Runtime | llama.cpp b10449 (commit 0d9ceae1e38291035605613ab41a8f5e693d6fcd) |
Headline Verdict
QAD Q4_0 is the rational ultra-fast deployment quant when 1.59 GB is the hard model-size target, but this campaign did not reproduce a material practical-quality win over ordinary PTQ Q4_0.
QAD exactly retained PTQ Q4_0's 4K GPU footprint and throughput, and it improved the fixed-corpus perplexity diagnostic. The small practical, tool, and coding lanes do not establish the producer's 96.6% aggregate-retention or Q4_K_M-equivalence claims.
Performance
| Quant | Peak GPU MiB | Prompt tok/s | Decode tok/s | TTFT ms |
|---|---|---|---|---|
| BF16 | 6100 | 1498.0 | 105.1 | 20.7 |
| PTQ Q4_0 | 2404 | 3235.4 | 282.7 | 9.6 |
| QAD Q4_0 | 2404 | 3221.8 | 282.5 | 9.6 |
| Q4_K_M | 2480 | 2777.0 | 263.1 | 11.2 |
| Q5_K_M | 2734 | 2649.9 | 240.5 | 11.7 |
QAD is 0.07% slower than PTQ decode (within run variance) and has identical observed peak allocation. This reproduces the native-Q4_0 speed/memory claim.
Fidelity
llama-perplexity on fixed local repeated corpora found QAD lower than PTQ (natural 1.0408 vs 1.0882; code 1.1248 vs 1.1412). QAD also scored below BF16 on those corpora. This is evidence of a QAD-vs-PTQ log-probability improvement, not a BF16-quality percentage.
Practical Behavior
A compact mechanically scored exact-match set tested instruction following, JSON, absent-evidence responses, and document extraction. Results were noisy and small:
- Structured output: no demonstrated QAD improvement over PTQ; QAD 1/2 vs PTQ 2/2.
- Hallucination resistance: no demonstrated QAD improvement; QAD 1/2 vs PTQ 2/2.
- Extraction/RAG: QAD 1/2 vs PTQ 0/2, but too small for a deployment claim.
Coding
Twenty isolated sandbox executions per primary quant. Each produced 4/20 execution successes. The suite exposed brittle single-code-block compliance and supports Liquid's warning that 2.6B is not for agentic coding. Coding is a poor fit regardless of the observed QAD quantization change.
Native Tools
All four compared quants passed the five deterministic local schema cases: 5/5. QAD did not improve this already-saturated simple suite. Nested-object, tool-result continuation, and recovery after an error remain unverified.
Context
All three finalists allocated and prefixed prompts through 127,777 actual tokens. QAD and PTQ retrieved the beginning needle at approximately 3.9K and 8.0K tokens, but neither produced a final answer above 16K within a 256-token generation budget. Useful long-context behavior is not established beyond 8K for QAD/PTQ.
Stability
The loopback QAD server completed 1,808.7 seconds (30m 8.7s), 175 varied requests, zero request errors. It shut down cleanly and GPU allocation returned to desktop baseline. No Xid/reset/GSP/channel-failure entries.
Role Classification
Supported roles:
- Always-loaded router/extraction/tool helper with schema validation and evidence checks
- Ultra-fast sidecar (when 1.59 GB is the hard size target)
Not supported:
- Primary assistant
- Long-context retriever
- Coding agent
Claim Classifications
| Claim | Classification |
|---|---|
| QAD preserves Q4_0 memory | REPRODUCED |
| QAD preserves Q4_0 decode throughput | REPRODUCED |
| QAD materially beats ordinary PTQ Q4_0 | PARTIALLY_REPRODUCED: lower local PPL; no material practical win shown |
| QAD retains 96.6% BF16 aggregate | INCONCLUSIVE |
| QAD recovers 48.4% of PTQ gap | INCONCLUSIVE |
| QAD matches Q4_K_M quality | INCONCLUSIVE |
| QAD improves native tools | INCONCLUSIVE: all simple cases saturated |
| QAD improves structured output | NOT_REPRODUCED |
| QAD improves hallucination resistance | NOT_REPRODUCED |
| QAD improves long context | NOT_REPRODUCED |
Evidence Links
- Canonical evaluation repo: https://github.com/WumboLabs/evaluations/tree/4f93731a9f042b9c1d86718a98a4e0e13d0fbd21/models/lfm2.5-2.6b/profiles/lfm2.5-2.6b-llamacpp-qad-q4-0
- WELP protocol: https://github.com/WumboLabs/welp
- Labs catalog: https://github.com/WumboLabs/evaluations/tree/4f93731a9f042b9c1d86718a98a4e0e13d0fbd21/models
Reproduction
Direct link to canonical reproduction material: Central evidence
This Lab Record is a summary; the canonical repo is the source of truth.
Identity
| Field | Value |
|---|---|
| Model | LFM2.5 2.6B |
| Producer | Liquid AI |
| Evaluated artifact | QAD Q4_0 from LiquidAI/LFM2.5-2.6B-GGUF @ f4a289c8a200a5ca71005ba7abc2dad33058a450 |
| SHA-256 | a247afd6414918eac8e520a9e6137dc271235461ecbe1180462221d5b8d40b03 |
| Evaluation date | 2026-08-26 |
| Protocol | WELP |
| Campaign classification | COMPLETE_SMALL_MODEL_EVALUATION |
Tested configurations
Exact profile identities, sampling conditions and provenance are in each configuration's details below.
| Configuration | Artifact / precision | Runtime | Hardware | Context / reasoning | Events |
|---|---|---|---|---|---|
| QAD Q4_0 · llama.cpp · RTX 5070 | QAD Q4_0 | llama.cpp | RTX 5070 | not recorded | 1 (2026-08-26) |
Configuration details: QAD Q4_0 · llama.cpp · RTX 5070
- Full profile name: llama.cpp QAD Q4_0.
- Profile identity:
lfm2.5-2.6b-llamacpp-qad-q4-0. - Status: current canonical/recommended tested surface.
- Artifact (as recorded): QAD Q4_0 from LiquidAI/LFM2.5-2.6B-GGUF @ f4a289c8a200a5ca71005ba7abc2dad33058a450
- Runtime (as recorded): llama.cpp.
- Profile Metadata
Events on this configuration:
- Initial evaluation (small-model evaluation) (2026-08-26); status: COMPLETE_SMALL_MODEL_EVALUATION
Chronology
Compact index; full results are in Event results.
| Date | Event | Configuration | Status |
|---|---|---|---|
| 2026-08-26 | Initial evaluation (small-model evaluation) | QAD Q4_0 · llama.cpp · RTX 5070 | COMPLETE_SMALL_MODEL_EVALUATION |
Technical provenance
Publication, acceptance and source-pin provenance for each event, collapsed; the scientific content above never depends on it.