WUMBOLABS / EVALUATIONS CANONICAL MODEL EVALUATION

/evaluations/lfm2.5-1.2b

LFM2.5 1.2B

Published evaluation evidence and testing history. Results remain attributed to their original configurations and methods.

latest evidence 2026-08-26

Overview

Liquid AI · 1 tested configuration · 1 evidence event · newest evidence 2026-08-26.

Contents: Findings · Event results · Tested configurations · Chronology · Technical provenance

Findings

Each finding is attributed to the event and tested configuration that measured it; exact values, failures and limitations are in the event results. A historical event's recorded status describes that event's own sections at its date and is not a verdict on newer sections.

  • Model classification: COMPLETE_SMALL_MODEL_EVALUATION - attributed to the 2026-08-26 event, configuration QAD Q4_0 · llama.cpp · RTX 5070.
  • Recommended configuration: llama.cpp QAD Q4_0 (current): event result.
  • Latest evidence: 2026-08-26: event result
  • 2026-08-26 · QAD Q4_0 · llama.cpp · RTX 5070: classification COMPLETE_SMALL_MODEL_EVALUATION. Event result

Event results

Newest first. Each event is one independent testing/publication event with its full report linked once at the top of its section; scores, failures and limitations stay visible and raw section-provenance JSON stays collapsed.

2026-08-26: Initial evaluation (small-model evaluation)

Initial Evaluation: llama.cpp QAD Q4_0 · config QAD Q4_0 · llama.cpp · RTX 5070 · maturity: FULL_EVALUATION · status: COMPLETE_SMALL_MODEL_EVALUATION

Configuration: llama.cpp QAD Q4_0.

Full report (REPORT.md)

Hardware
FieldValue
MachineWumboJetsII
GPUNVIDIA GeForce RTX 5070 12GB (SM120)
VRAM12,227 MiB physical
CPUAMD Ryzen 7 9800X3D
OSFedora Linux 44, kernel 7.1.9-200.fc44
Driver / CUDANVIDIA 610.57.04
Runtimellama.cpp b10449 (commit 0d9ceae1e38291035605613ab41a8f5e693d6fcd)
Headline Verdict

QAD Q4_0 is the best deployment quant of the five-way matrix and the best always-resident sidecar candidate measured so far in this model family, but the campaign did not reproduce Liquid's hallucination or aggregate-retention story: at 1.2B every quant including BF16 fabricates confidently on invented subjects (0/12), and QAD's fixed-corpus perplexity advantage over PTQ inverted.

QAD kept exactly Q4_0 memory and within-variance Q4_0 speed, scored best of all quants on the 52-task practical suite (31/52), and is faster than both K-quant controls (~+6% decode). Useful context is <=8K for every candidate.

Performance
QuantPrompt tok/sDecode tok/sTTFT ms
BF163383232.69.7
PTQ Q4_06944592.34.8
QAD Q4_06600585.64.9
Q4_K_M6054554.45.5
UD-Q4_K_XL5947551.05.3
Q5_K_M5777510.76.0

Deltas: QAD -1.14% vs PTQ decode (within variance), +5.63% vs Liquid Q4_K_M, +6.28% vs UD-Q4_K_XL.

Fidelity

llama-perplexity on fixed local repeated corpora:

QuantNatural PPLCode PPL
BF161.52351.0514
PTQ Q4_01.24541.0378
QAD Q4_01.39411.0472
Q4_K_M1.31761.0457

The 2.6B finding inverted: at 1.2B PTQ beats QAD on natural text. The repeated-corpus pathology persists and worsens: BF16 is worse than most quants on natural text, so no BF16-relative percentage from this corpus is valid.

Practical Capability

52 unique mechanically-scored tasks, same set for all six models, seed 42:

ModelInstrExtrStructHallucLinuxUncertTotal
BF168/104/812/120/124/64/432/52
QAD Q4_08/104/811/120/124/64/431/52
PTQ Q4_08/104/810/120/124/64/430/52
Q5_K_M8/104/810/120/123/64/429/52
Q4_K_M8/103/810/120/123/63/427/52
UD-Q4_K_XL8/103/810/120/123/63/427/52
Hallucination Resistance

Twelve invented-subject probes (packages, flags, APIs, files, systemd units, sysctls, Kconfig options): every model, including BF16, fabricated confident detailed explanations: 0/12 across the entire matrix. This is a model-family property at 1.2B scale, not a quantization effect. QAD shows no improvement.

Structured Output

Mechanical JSON/schema scoring: QAD 11/12, PTQ/Q4_K_M/Q5_K_M/UD 10/12, BF16 12/12. A small QAD-over-PTQ edge appeared (+1).

Native Tools

v1 (exact five-case contract): saturated: 5/5 for BF16, PTQ, QAD, Q4_K_M, Q5_K_M; UD-Q4_K_XL 4/5.

v2 (ten harder cases): complete multi-turn tool sequences succeed only half the time even for BF16. QAD ties its controls.

Context

Three-needle retrieval at 4K/8K/16K/32K: at >=16K every model degenerates into repeating filler text. Useful context ceiling: 8K for all finalists.

Stability

The loopback-only QAD soak completed 1,805.9s (30m 6s), 180 varied requests, zero request errors. Peak power 33.6 W, max temperature 52 C. Zero Xid/reset/GSP/channel events.

Role Classification

GOOD FIT: always-loaded router, classifier, simple tool selector.

USABLE WITH GUARDRAILS: extraction helper, RAG helper (<=8K), structured formatter, conversational sidecar.

POOR FIT: coding helper, primary assistant, autonomous agent.

1.2B vs 2.6B Comparison
Dimension2.6B QAD1.2B QADDelta
GGUF bytes1,593,894,944695,755,488-898 MB
Peak VRAM2404 MiB1534 MiB-870 MiB
Decode tok/s282.5585.6x2.07
Useful context<=8K<=8Kequal
Hallucination resistanceno QAD effect shownabsent at model levelworse

The 1.2B creates a genuinely better always-resident micro-sidecar niche (twice as fast, two-thirds the VRAM) but is a strictly weaker knowledge/extraction assistant.

Claim Classifications
ClaimClassification
QAD keeps Q4_0 memoryREPRODUCED
QAD keeps Q4_0 speedREPRODUCED
97.4% BF16 aggregate retentionINCONCLUSIVE
Matches Liquid Q4_K_MPARTIALLY_REPRODUCED
Matches Unsloth UD-Q4_K_XLPARTIALLY_REPRODUCED
Meaningful recovery of PTQ lossPARTIALLY_REPRODUCED
Improves hallucination resistanceNOT_REPRODUCED
  • Canonical evaluation repo: https://github.com/WumboLabs/evaluations/tree/4f93731a9f042b9c1d86718a98a4e0e13d0fbd21/models/lfm2.5-1.2b/profiles/lfm2.5-1.2b-llamacpp-qad-q4-0
  • WELP protocol: https://github.com/WumboLabs/welp
  • Labs catalog: https://github.com/WumboLabs/evaluations/tree/4f93731a9f042b9c1d86718a98a4e0e13d0fbd21/models
Reproduction

Direct link to canonical reproduction material: Central evidence

This Lab Record is a summary; the canonical repo is the source of truth.

Identity
FieldValue
ModelLFM2.5 1.2B-Instruct
ProducerLiquid AI
Evaluated artifactQAD Q4_0 from LiquidAI/LFM2.5-1.2B-Instruct-GGUF @ 67672651…3615b
SHA-256bb741ebb106d543e9de114b843a3d3d73d51c74b5801e69da2abde821a0cb3e1
Evaluation date2026-08-26
ProtocolWELP
Campaign classificationCOMPLETE_SMALL_MODEL_EVALUATION

Tested configurations

Exact profile identities, sampling conditions and provenance are in each configuration's details below.

ConfigurationArtifact / precisionRuntimeHardwareContext / reasoningEvents
QAD Q4_0 · llama.cpp · RTX 5070QAD Q4_0llama.cppRTX 5070not recorded1 (2026-08-26)
Configuration details: QAD Q4_0 · llama.cpp · RTX 5070
  • Full profile name: llama.cpp QAD Q4_0.
  • Profile identity: lfm2.5-1.2b-llamacpp-qad-q4-0.
  • Status: current canonical/recommended tested surface.
  • Artifact (as recorded): QAD Q4_0 from LiquidAI/LFM2.5-1.2B-Instruct-GGUF @ 67672651…3615b
  • Runtime (as recorded): llama.cpp.
  • Profile Metadata

Events on this configuration:

Chronology

Compact index; full results are in Event results.

DateEventConfigurationStatus
2026-08-26Initial evaluation (small-model evaluation)QAD Q4_0 · llama.cpp · RTX 5070COMPLETE_SMALL_MODEL_EVALUATION

Technical provenance

Publication, acceptance and source-pin provenance for each event, collapsed; the scientific content above never depends on it.

Model evidence tree (GitHub)

Legacy provenance