WUMBOLABS / EVALUATIONS CANONICAL MODEL EVALUATION

/evaluations/qwen38-27b

Qwen3.8-27B

Published evaluation evidence and testing history. Results remain attributed to their original configurations and methods.

latest evidence 2026-09-12

Overview

Alibaba (Qwen team) · 2 tested configurations · 4 evidence events · newest evidence 2026-09-12.

Contents: Findings · Event results · Limitations · Tested configurations · Chronology · Technical provenance

Findings

Each finding is attributed to the event and tested configuration that measured it; exact values, failures and limitations are in the event results. A historical event's recorded status describes that event's own sections at its date and is not a verdict on newer sections.

  • Model classification: COMPLETED_DEEP_EVALUATION - attributed to the 2026-08-21 event, configuration UD-Q2_K_XL · llama.cpp b10449 · RTX 5070, SM120.
  • Recommended configuration: ExLlamaV3 H1 (SC_2.20bpw_H3_V3) (current): event result.
  • Practical context: 65,536 default / 98,304 guarded tokens; native model-card maximum 262,144 (envelope complete: YES); source: event result
  • Context limit outcome: FIT_LIMIT - exact native maximum 262,144 on the accepted H1 ExLlamaV3 surface: requires 12,530 MiB at ready (8,178 MiB context-independent + 262,144 x 17,408 B/token KV) versus 12,227 MiB total card VRAM; nearest measured boundary 98,304 VALIDATED. Official YaRN 1,000,000 extension INTEGRATION_BLOCKED on the accepted surface (runtime 1.4.6 has the YaRN implementation but exposes no supported activation without editing the protected artifact config) and independently FIT_LIMIT (cache alone ~16.6 GiB, ~1.36x total card VRAM); source: event result
  • Latest evidence: 2026-09-12: event result
  • 2026-09-12 · 2.20 bpw EXL3; q4 target KV; q4 draft KV · ExLlamaV3 · RTX 5070: classification GUARDED / MODEL-CARD CONTEXT ENVELOPE COMPLETE; useful-context strict aggregate 4/4 requests (2 rungs x 2 seeds) at >=99.4% occupancy: exact retrieval 5/5, synthesis, decoy resistance, absent-information grounding, instruction compliance all 4/4; measured depths 2/25/50/75/95% within +/-0.036 pp. Event result
  • 2026-09-09 · 2.20 bpw EXL3; q4 target KV; q4 draft KV · ExLlamaV3 · RTX 5070: classification COMPLETE / CURRENT_CANONICAL_H1_PROFILE; E18H seven-check quality matrix 7/7 PASS on the retained E3 fixture (all three arms A/B/A2); E18I H1 smoke repeated the fixture 7/7. Event result
  • 2026-08-21 · UD-Q2_K_XL · llama.cpp b10449 · RTX 5070, SM120: classification COMPLETED_DEEP_EVALUATION. Event result
  • 2026-08-20 · UD-Q2_K_XL · llama.cpp b10449 · RTX 5070, SM120: classification PROFILE_OPTIMIZATION (historical llama.cpp lane; superseded by ExLlamaV3 H1). Event result

Event results

Newest first. Each event is one independent testing/publication event with its full report linked once at the top of its section; scores, failures and limitations stay visible and raw section-provenance JSON stays collapsed.

2026-09-12: Context envelope completion

Context Envelope: ExLlamaV3 H1 · config 2.20 bpw EXL3; q4 target KV; q4 draft KV · ExLlamaV3 · RTX 5070 · maturity: CONTEXT_COMPLETION · status: GUARDED / MODEL-CARD CONTEXT ENVELOPE COMPLETE

Configuration: ExLlamaV3 H1 (SC_2.20bpw_H3_V3).

Full report (REPORT.md)

WELP outcome

  • Outcome: PASS: QWEN38_27B_CONTEXT_ENVELOPE_COMPLETED
  • Classification: MODEL-CARD CONTEXT ENVELOPE COMPLETE at the inherited guarded deployment classification; practical default unchanged

Publication state: published; canonical evidence linked above

Quality and capabilities

  • Constrained result: useful-context strict aggregate 4/4 requests (2 rungs x 2 seeds) at >=99.4% occupancy: exact retrieval 5/5, synthesis, decoy resistance, absent-information grounding, instruction compliance all 4/4; measured depths 2/25/50/75/95% within +/-0.036 pp
  • first near-full context evidence above 65,536 on the ExLlamaV3 2.20bpw Qwen3.8 surface
  • frozen fit model validated at both measured rungs (conservative by 28-80 MiB)
  • prefill degradation curve characterized: ~790 tok/s at 3.5K -> 606 at 64.7K -> 537 at 97.3K
Guardrails and limitations
  • retrieval-quality evidence is bounded to the tested fixture style and depths; it is not a general long-context quality claim
  • reliability and role classification remain the inherited historical model-level findings; practical default unchanged
  • the 98,304 operating point is guarded: ~181 s near-full TTFT and a ~300 MiB floor margin

Reliability: no CUDA/OOM/Xid events in the campaign; both arms torn down to verified idle; cold-cache enforced per request (verified cached_tokens 0 in all four retained terminal events)

LocalMaxxing

  • Status: SUBMITTED
  • Canonical context: 65,536 tokens
  • tok/s out: 75.4
  • TTFT: 71.35 ms
  • Submission reference: cmtwcncwr07f4ps01ef75q6ne
  • verifiedRun: NO

existing authorized submission on the identical canonical practical profile (VERIFIED_EXISTING); no new benchmark or submission in this campaign

Tested condition: artifact 2.20 bpw EXL3; q4 target KV; q4 draft KV; MTP width 1 · ExLlamaV3 · RTX 5070 · context 65,536 default / 98,304 guarded · native model-card maximum 262,144 tokens · envelope complete: YES.

Context limit outcome: FIT_LIMIT - exact native maximum 262,144 on the accepted H1 ExLlamaV3 surface: requires 12,530 MiB at ready (8,178 MiB context-independent + 262,144 x 17,408 B/token KV) versus 12,227 MiB total card VRAM; nearest measured boundary 98,304 VALIDATED. Official YaRN 1,000,000 extension INTEGRATION_BLOCKED on the accepted surface (runtime 1.4.6 has the YaRN implementation but exposes no supported activation without editing the protected artifact config) and independently FIT_LIMIT (cache alone ~16.6 GiB, ~1.36x total card VRAM)

Context boundaries: near-full performance (client streaming proxies): 65,536 anchor ~606 tok/s prefill, 61-62 tok/s decode, ~107 s TTFT; 98,304 ~537 tok/s prefill, 56-57 tok/s decode, ~181 s TTFT

Runtime conditions: Canonical H1 surface unchanged: MTP width 1, batch 1, max_chunk 256, recurrent history H1, CPU embedding lookup, all 64 transformer blocks GPU-resident; only serving context varies per arm

Hardware conditions: Single-user workstation; AMD Ryzen 7 9800X3D; Fedora Linux 44; displays on iGPU

Full tested identity, runtime and context details (exact hashes, versions and configuration)

Identity

FieldValue
Profile identityqwen38-27b-exl3-h1
ModelQwen3.8-27B
ProducerAlibaba (Qwen team); EXL3 quant by turboderp
Official modelQwen/Qwen3.8-27B @ card lastModified 2026-08-14 (native 262,144; YaRN factor 4.0 advertised to 1,000,000)
Tested artifactQwen3.8-27B-SC_2.20bpw_H3_V3 (EXL3)
Precision2.20 bpw EXL3; q4 target KV; q4 draft KV; MTP width 1
Campaignqwen38-27b-rtx5070-context-completion-2026-09-12
Record date2026-09-12

Runtime and hardware

FieldValue
EngineExLlamaV3 1.4.6+cu128.torch2.10.0 (torch 2.10.0+cu128, CUDA 12.8)
HardwareWumboJetsII (NVIDIA GeForce RTX 5070 12GB)
Runtime notesCanonical H1 surface unchanged: MTP width 1, batch 1, max_chunk 256, recurrent history H1, CPU embedding lookup, all 64 transformer blocks GPU-resident; only serving context varies per arm
Hardware notesSingle-user workstation; AMD Ryzen 7 9800X3D; Fedora Linux 44; displays on iGPU

Context profile

FieldValue
Practical default65536 tokens
Guarded context98304 tokens
Native model-card maximum262144 tokens
Model-card envelope completeYES
Native maximum dispositionFIT_LIMIT - exact native maximum 262,144 on the accepted H1 ExLlamaV3 surface: requires 12,530 MiB at ready (8,178 MiB context-independent + 262,144 x 17,408 B/token KV) versus 12,227 MiB total card VRAM; nearest measured boundary 98,304 VALIDATED. Official YaRN 1,000,000 extension INTEGRATION_BLOCKED on the accepted surface (runtime 1.4.6 has the YaRN implementation but exposes no supported activation without editing the protected artifact config) and independently FIT_LIMIT (cache alone ~16.6 GiB, ~1.36x total card VRAM)

near-full performance (client streaming proxies): 65,536 anchor ~606 tok/s prefill, 61-62 tok/s decode, ~107 s TTFT; 98,304 ~537 tok/s prefill, 56-57 tok/s decode, ~181 s TTFT

2026-09-09: H1 deployment / canonical promotion

H1 Canonical Promotion: ExLlamaV3 · config 2.20 bpw EXL3; q4 target KV; q4 draft KV · ExLlamaV3 · RTX 5070 · maturity: CURRENT_WELP · status: COMPLETE / CURRENT_CANONICAL_H1_PROFILE

Configuration: ExLlamaV3 H1 (SC_2.20bpw_H3_V3).

Full report (REPORT.md)

WELP outcome

  • Outcome: PASS: E18I_H1_CANONICAL_PROMOTION_COMPLETE (E18H: PASS: E18H_H1_VALIDATED_EXPERIMENTAL_PROMOTION_CANDIDATE)
  • Classification: CURRENT_CANONICAL_PROFILE: engine/history promotion validation, not a new model classification; the historical deep-evaluation role findings remain those of their tested stack
  • Artifact classification: Current canonical Qwen3.8 serving surface (exact artifact/runtime/geometry scope only)

Publication state: published; canonical evidence linked above

Headline performance

SurfaceTTFTPrefillDecode
Shortnot recordedNot recordednot recorded
Moderate (3514-token input)4.4487 s789.89 tok/s75.79 tok/s

E18H arm-B (H1) frozen 3,514-token prompt, five cold repetitions after warmup, medians; MTP width 1 active

Quality and capabilities

  • Constrained result: E18H seven-check quality matrix 7/7 PASS on the retained E3 fixture (all three arms A/B/A2); E18I H1 smoke repeated the fixture 7/7
  • H1 recurrent/GDN storage 292.6875 MiB vs H4 727.5 MiB (measured reduction 434.8125 MiB, retained as reserve, not spent)
  • E18G deterministic mechanism fixture F1/F2/F3 exact on H1; 20/20 stability sequence exactly flat (zero allocated/reserved growth, zero slope)
  • H1 oracle minimum free 1,888 MiB vs 1,470 MiB for H4 (+418 MiB measured in the same campaign)
Guardrails and limitations
  • H1 is canonical only for this exact artifact/runtime and text-only width1/batch1 surface
  • No context growth or recovered-memory spending; explicit H4 fallback remains available in the same launcher
  • Historical model-role findings (hallucination rate, strict interfaces, native tools) belong to the historical UD-Q2_K_XL llama.cpp evaluation and are not superseded by this profile work

Reliability: E18H H1 stability: 20/20 exact expected results, HTTP 200 throughout, flat allocator blocks (growth 0, slope 0); zero CUDA/OOM/Xid in both campaigns

LocalMaxxing

  • Status: SUBMITTED
  • Canonical context: 65,536 tokens
  • tok/s out: 75.4
  • TTFT: 71.35 ms
  • Submission reference: cmtwcncwr07f4ps01ef75q6ne
  • verifiedRun: NO

APPROVED, origin NEW (localmaxxing-backfill-2026-09-10) on the CURRENT H1 canonical profile (ExLlamaV3, SC_2.20bpw_H3_V3, q4/q4 KV, MTP width1, batch1, H1, 65,536); actual prompt tokens 266 (endpoint usage); verifiedRun false with an additional disclosed issue: MTP acceptance counters are not exposed by the canonical wrapper and none were fabricated; a transparent loopback path adapter bridged /chat/completions to /v1/chat/completions and is disclosed in the submission payload. Three historical E4-era submissions (llama.cpp-era wrapper with H4 history) remain historical and supersede nothing.

Tested condition: artifact 2.20 bpw EXL3; q4 target KV; q4 draft KV · ExLlamaV3 · RTX 5070 · context 65,536 default · envelope complete: NO.

Context limit outcome: No full model-card context campaign exists for this profile; inherited validated evidence: E18E corrected combined oracle 3/3 requests / 15/15 target fields exact on H1 at ~58.8K actual input tokens (prompts 58,849/58,902/58,883)

Context boundaries: H1 scope is exact: text-only, width 1, batch 1, the named artifact and runtime release; no width>1, batch>1, multimodal, tensor-parallel, other architecture/artifact/runtime/release, or recurrent-geometry validation is implied.

Runtime conditions: MTP width 1, batch 1, max_chunk 256, recurrent history H1 (max_history 1) with explicit H4 fallback via --max-history 4; CPU embedding lookup (not transformer-layer offload); all 64 transformer blocks GPU-resident; context 65,536 canonical; recovered memory (434.8125 MiB) retained as unspent reserve

Hardware conditions: Single-user workstation; AMD Ryzen 7 9800X3D; Fedora Linux 44

Full tested identity, runtime and context details (exact hashes, versions and configuration)

Identity

FieldValue
Profile identityqwen38-27b-exl3-h1
ModelQwen3.8-27B (current H1 profile)
ProducerAlibaba (Qwen team); EXL3 quant by turboderp
Tested artifactQwen3.8-27B-SC_2.20bpw_H3_V3 (EXL3)
Precision2.20 bpw EXL3; q4 target KV; q4 draft KV
Campaignqwen38-27b-h1-canonical-profile (E18H candidate validation 2026-09-08 + E18I canonical promotion 2026-09-09)
Record date2026-09-09

Runtime and hardware

FieldValue
EngineExLlamaV3 1.4.6+cu128.torch2.10.0
HardwareWumboJetsII (NVIDIA GeForce RTX 5070 12GB)
Runtime notesMTP width 1, batch 1, max_chunk 256, recurrent history H1 (max_history 1) with explicit H4 fallback via --max-history 4; CPU embedding lookup (not transformer-layer offload); all 64 transformer blocks GPU-resident; context 65,536 canonical; recovered memory (434.8125 MiB) retained as unspent reserve
Hardware notesSingle-user workstation; AMD Ryzen 7 9800X3D; Fedora Linux 44

Context profile

FieldValue
Practical default65536 tokens
Guarded contextnot recorded
Native model-card maximumnot recorded
Model-card envelope completeNO
Native maximum dispositionNo full model-card context campaign exists for this profile; inherited validated evidence: E18E corrected combined oracle 3/3 requests / 15/15 target fields exact on H1 at ~58.8K actual input tokens (prompts 58,849/58,902/58,883)

H1 scope is exact: text-only, width 1, batch 1, the named artifact and runtime release; no width>1, batch>1, multimodal, tensor-parallel, other architecture/artifact/runtime/release, or recurrent-geometry validation is implied.

2026-08-21: Initial evaluation (deep evaluation, historical llama.cpp)

Initial Evaluation: Historical llama.cpp · config UD-Q2_K_XL · llama.cpp b10449 · RTX 5070, SM120 · maturity: HISTORICAL_EVALUATION · status: COMPLETED_DEEP_EVALUATION

Configuration: Historical llama.cpp (Unsloth UD-Q2_K_XL GGUF).

Full report (REPORT.md)

Long-form report: Qwen3.8-27B on 12GB: How Far Can an RTX 5070 Really Push It?

Hardware
FieldValue
MachineWumboJetsII
GPUNVIDIA GeForce RTX 5070 12GB (SM120)
VRAM12,227 MiB physical
GPU power250 W stock reference limit
CPUAMD Ryzen 7 9800X3D, 8 cores / 16 threads
RAM~32 GiB DDR5 (~30 GiB usable)
OSFedora Linux 44, kernel 7.1.8-200.fc44.x86_64
Driver / CUDANVIDIA 610.57.04 / CUDA 13.3
Runtimellama.cpp b10449 (commit 0d9ceae1e38291035605613ab41a8f5e693d6fcd)

The RTX 5070 also drove the Fedora/Hyprland desktop during testing. Practical free VRAM was therefore lower than raw physical capacity; this is representative workstation evidence, not benchmark isolation.

Headline Verdict

NOT READY AS DAILY DRIVER

Qwen3.8-27B UD-Q2_K_XL is stable in the tested runtime, practically fast, unusually strong at coding, capable of useful bounded 64K context, and substantially improved by bounded reasoning on hard tasks.

But the frozen primary reliability corpus classified 19.0% of executions as hallucinations; the frozen strict campaign had 21.2% failures; false-premise resistance was weak; and only 5/50 full native-tool sequences met the strict contract.

Recommended classification: Strong reviewed local coding/technical assistant.

The evidence does not support use as an unattended agent, source of unverified technical facts, autonomous system administrator, autonomous Git operator, unsupervised native-tool agent, or fail-closed structured-output engine.

Performance
ConfigurationDecode tok/sPrefill tok/sContextNotes
Quality/balance (Q4_K_M, 38 GPU layers, 8K)6.4–6.8~8808K4.85–5.0/5 coding score; 467 MiB min headroom
Speed (UD-IQ2_M, full GPU, 4K)42.0–42.9Not recorded4K362/5 manual avg; task-dependent quality
Agent-safe (UD-IQ2_XXS, full GPU, 12K)Not recordedNot recorded12KBounded one-file OMP pass
Large context (Q2_K_XL, 64K Q4/Q4)~30~70464KLow-headroom reviewed work

Native MTP at 8K: ~66 tok/s (1.47x speedup over baseline ~45 tok/s, ~75% acceptance).

Capabilities
Coding

Strongest supported domain. Fresh executable coding passed 28/30 in disposable sandboxes with no network, minimal devices, resource limits, and no host modification. Coding recovery passed 36/40 changed-constraint revisions. This supports Qwen3.8-27B as a strong reviewed local coding assistant, not a hallucination-free coding system.

Reasoning

Tested, not enabled by default. On the fresh hard subset, reasoning on passed 9/10 versus 4/10 for the matched reasoning-off control. Use the reasoning profile for difficult math, logic, code diagnosis, evidence synthesis, and constrained planning. Leave it off for ordinary factual questions, strict interfaces, native tools, and routine work.

Strict Interfaces

The frozen strict campaign passed 197/250 (78.8%) and failed 53/250 (21.2%) across exact strings, JSON, extraction, enums, CSV, and word limits. Do not rely on one-shot strict responses; parse and mechanically validate structured output.

Native Tools

Only 5/50 full native-tool sequences satisfied the complete frozen end-to-end contract. Tool selection and argument structure performed better than the strict execution/recovery sequence. Native tools require deterministic execution, schema validation, result validation, and external supervision.

Long Context

Fresh baseline Q4 work passed bounded retrieval, synthesis, and instruction/code-retention probes at 16K, 32K, and 64K. The first filled 64K prompt contained approximately 64,275 tokens; first prefill took 91.24 s at 704.5 prompt tok/s. This demonstrates bounded 64K retrieval on this configuration. It does not establish that arbitrary 64K workloads are reliable.

Reliability

Primary reliability statistics covered 685 evaluated task executions:

ClassificationCountRate
CLEAN_PASS41460.4%
MATERIAL_DEFECT436.3%
HALLUCINATION13019.0%
STRUCTURAL_FAILURE9814.3%

False-premise resistance was the decisive readiness failure: hallucination-category runs were 67/150 CLEAN_PASS and 83/150 HALLUCINATION; sycophancy/false-premise runs were 28/75 CLEAN_PASS and 47/75 HALLUCINATION.

Stability

A two-hour balanced-profile soak completed 225/225 HTTP successes and 46/46 exact sentinels. Mean generation was 44.81 tok/s. VRAM started/ended/peaked at 10,340/10,248/10,342 MiB; peak soak temperature was 65 C. No new Xid, reset-required state, GSP failure, or display-engine collapse appeared during the controlled final campaign.

Role Classification

Supported roles:

  • Reviewed local coding assistant
  • Technical assistant (with verification)
  • Bounded reasoning assistant (reasoning on, difficult tasks only)

Guarded roles:

  • Drafting and editing (review required)
  • 64K context retrieval (low headroom, reviewed only)

Unsupported / not ready:

  • Unguarded daily driver
  • Unattended autonomous agent
  • Source of unverified technical facts
  • Autonomous system administrator
  • Autonomous Git operator
  • Unsupervised native-tool agent
  • Fail-closed structured-output engine
Important Limitations
  • No LLMGauge Agent Harness result exists. This is a coverage limit, not a replacement classification.
  • Autonomy evidence is bounded: the OMP follow-up completed a deliberately limited one-file inspect/edit/test/report loop. It does not establish broad autonomous-agent suitability.
  • Runtime safety boundary: the evaluated vLLM configuration exhausted host memory during an earlier admission attempt. This is an operational deployment boundary, not a model-quality result.
  • Useful context not established beyond 8K for the fast quants.
  • Results apply to the pinned artifacts, llama.cpp b10449 runtime, Fedora 44 workstation condition, RTX 5070 12GB configuration, stated contexts, and frozen WumboLabs probes. They do not establish producer accuracy claims, cloud behavior, broad hardware representativeness, or universal model quality.
  • Canonical evaluation repo (historical llama.cpp profile): https://github.com/WumboLabs/evaluations/tree/4f93731a9f042b9c1d86718a98a4e0e13d0fbd21/models/qwen38-27b/profiles/qwen38-27b-llamacpp-ud-q2-k-xl
  • WELP protocol: https://github.com/WumboLabs/welp
  • Labs catalog: https://github.com/WumboLabs/evaluations/tree/4f93731a9f042b9c1d86718a98a4e0e13d0fbd21/models
  • LocalMaxxing: APPROVED speed-test submissions (cmt3jngze0njfmv0133xpneeu BALANCED 16K; cmt3jp2cn0njpmv01t5u8m6wd FAST 8K MTP) + benchmark suite APPROVED/PUBLIC; ref-quant cmsv68xl3085ims01w8aeacig
Reproduction

Direct link to canonical reproduction material: Central evidence

Full reproduction instructions, manifests, and checksums are published in the canonical repository. This Lab Record is a summary; the canonical repo is the source of truth.

Provenance note (2026-09-12): this evidence was originally published at WumboLabs/eval-qwen3.8-27b (evidence commit 54b14b30e3779f15486c3785222647324459dcf5). It moved byte-identical to the canonical historical-profile repository (eval-qwen3.8-27b-llamacpp) by the model/profile/event publication-identity milestone; the original repository is preserved unchanged as a historical archive.

Identity
FieldValue
ModelQwen 3.8 27B
ProducerAlibaba (Qwen team); quantized by Unsloth
Evaluated artifactUnsloth Qwen3.8-27B UD-Q2_K_XL
Source revision27af057ecb382ddfea5d12837360a8980560e3ed
SHA-256fd4730dd8aad070517978752b63d530aeb1740d2283cab9fa24f1e404032ddb0
Evaluation date2026-08-14 through 2026-08-21
ProtocolWELP end-to-end (frozen snapshot welp-next-snapshot-2026-08-25-end-to-end)
Highest phase reachedFull campaign (Phase 10+)
Campaign classificationCOMPLETED_DEEP_EVALUATION

2026-08-20: llama.cpp quant/variant showdown runs (LLMGauge era)

Profile Optimization: llama.cpp quant selection · config UD-Q2_K_XL · llama.cpp b10449 · RTX 5070, SM120 · maturity: PROFILE_OPTIMIZATION · status: PROFILE_OPTIMIZATION (historical llama.cpp lane; superseded by ExLlamaV3 H1)

Configuration: Historical llama.cpp (Unsloth UD-Q2_K_XL GGUF).

Full report (REPORT.md)

llama.cpp-era quant/variant showdown evidence (unsloth v3-final UD-IQ1_S/IQ2_XXS/IQ2_M ladder, community variants AtomicChat/XYZ Q3 smokes, 16k practical-use selection, 196k context-retrieval probe) that informed the historical llama.cpp profile before the ExLlamaV3 H1 promotion. Long-form context completion is separately represented (context-envelope-completion-2026-09-12).

This event is a bounded public-safe summary derived from retained WumboLabs evidence. It is not a WELP characterization, a universal model ranking, or a production-readiness proof. Values are attributed to the tested artifact, runtime, hardware, suite, and settings.

Historical lane note

These showdown runs belong to the historical llama.cpp lane. The current canonical serving profile for this model is the ExLlamaV3 H1 surface, represented by its own events above.

Identity and scope
  • Profile: qwen38-27b-llamacpp-ud-q2-k-xl: Historical llama.cpp (Unsloth UD-Q2_K_XL GGUF)
  • Evidence maturity: PROFILE_OPTIMIZATION
  • Evidence scope: performance, serving-profile
  • Hardware: WumboJetsII (RTX 5070 12GB)

Limitations

  • retrieval-quality evidence is bounded to the tested fixture style and depths; it is not a general long-context quality claim Source events: 2026-09-12.
  • reliability and role classification remain the inherited historical model-level findings; practical default unchanged Source events: 2026-09-12.
  • the 98,304 operating point is guarded: ~181 s near-full TTFT and a ~300 MiB floor margin Source events: 2026-09-12.
  • H1 is canonical only for this exact artifact/runtime and text-only width1/batch1 surface Source events: 2026-09-09.
  • No context growth or recovered-memory spending; explicit H4 fallback remains available in the same launcher Source events: 2026-09-09.
  • Historical model-role findings (hallucination rate, strict interfaces, native tools) belong to the historical UD-Q2_K_XL llama.cpp evaluation and are not superseded by this profile work Source events: 2026-09-09.

Tested configurations

Exact profile identities, sampling conditions and provenance are in each configuration's details below.

ConfigurationArtifact / precisionRuntimeHardwareContext / reasoningEvents
2.20 bpw EXL3; q4 target KV; q4 draft KV · ExLlamaV3 · RTX 50702.20 bpw EXL3ExLlamaV3RTX 507065,536 default / 98,304 guarded2 (2026-09-09, 2026-09-12)
UD-Q2_K_XL · llama.cpp b10449 · RTX 5070, SM120UD-Q2_K_XLllama.cpp b10449RTX 5070, SM120not recorded2 (2026-08-20, 2026-08-21)
Configuration details: 2.20 bpw EXL3; q4 target KV; q4 draft KV · ExLlamaV3 · RTX 5070
  • Full profile name: ExLlamaV3 H1 (SC_2.20bpw_H3_V3).
  • Profile identity: qwen38-27b-exl3-h1.
  • Status: current canonical/recommended tested surface.
  • Deployment topology (as recorded): MTP width 1, batch 1, max_chunk 256, recurrent history H1 (max_history 1) with explicit H4 fallback via --max-history 4; CPU embedding lookup (not transformer-layer offload); all 64 transformer blocks GPU-resident
  • Artifact (as recorded): Qwen3.8-27B-SC_2.20bpw_H3_V3 (EXL3)
  • Runtime (as recorded): ExLlamaV3 1.4.6+cu128.torch2.10.0 (torch 2.10.0+cu128, CUDA 12.8).
  • Hardware (as recorded): WumboJetsII (NVIDIA GeForce RTX 5070 12GB).
  • Profile Metadata

Events on this configuration:

Configuration details: UD-Q2_K_XL · llama.cpp b10449 · RTX 5070, SM120
  • Full profile name: Historical llama.cpp (Unsloth UD-Q2_K_XL GGUF).
  • Profile identity: qwen38-27b-llamacpp-ud-q2-k-xl.
  • Status: historical tested surface; retained evidence, not the recommended profile.
  • Deployment topology (as recorded): Full-GPU llama.cpp CUDA serving on a single-user workstation; quant-matrix and placement evidence in reports/ and manifests/
  • Artifact (as recorded): Unsloth Qwen3.8-27B UD-Q2_K_XL GGUF (Unsloth rev 27af057ecb382ddfea5d12837360a8980560e3ed)
  • Runtime (as recorded): llama.cpp b10449 (0d9ceae1e38291035605613ab41a8f5e693d6fcd).
  • Profile Metadata

Events on this configuration:

Chronology

Compact index; full results are in Event results.

DateEventConfigurationStatus
2026-09-12Context envelope completion2.20 bpw EXL3; q4 target KV; q4 draft KV · ExLlamaV3 · RTX 5070GUARDED / MODEL-CARD CONTEXT ENVELOPE COMPLETE
2026-09-09H1 deployment / canonical promotion2.20 bpw EXL3; q4 target KV; q4 draft KV · ExLlamaV3 · RTX 5070COMPLETE / CURRENT_CANONICAL_H1_PROFILE
2026-08-21Initial evaluation (deep evaluation, historical llama.cpp)UD-Q2_K_XL · llama.cpp b10449 · RTX 5070, SM120COMPLETED_DEEP_EVALUATION
2026-08-20llama.cpp quant/variant showdown runs (LLMGauge era)UD-Q2_K_XL · llama.cpp b10449 · RTX 5070, SM120PROFILE_OPTIMIZATION (historical llama.cpp lane; superseded by ExLlamaV3 H1)

Technical provenance

Publication, acceptance and source-pin provenance for each event, collapsed; the scientific content above never depends on it.

Model evidence tree (GitHub)

Event provenance: 2026-09-12 Context Envelope: ExLlamaV3 H1
  • Campaign: qwen38-27b-rtx5070-context-completion-2026-09-12
  • Export disposition: WEBSITE_READY; publication date 2026-09-12
  • Canonical citation (immutable commit/path pin): https://github.com/WumboLabs/evaluations/blob/479cb50c197f8ea0dd905a68ffd61f632f49c652/models/qwen38-27b/events/context-envelope-completion-2026-09-12/REPORT.md
  • Export source pin: repository-cache WumboLabs/evaluations @ 479cb50c197f8ea0dd905a68ffd61f632f49c652 path models/qwen38-27b/events/context-envelope-completion-2026-09-12/website-publication.json sha256 efbde1adf01c3a57b59d1b51d9234384cd62738de299cbe9c8b6f06f80f8064f
Event provenance: 2026-09-09 H1 Canonical Promotion: ExLlamaV3
  • Campaign: qwen38-27b-h1-canonical-profile (E18H candidate validation 2026-09-08 + E18I canonical promotion 2026-09-09)
  • Export disposition: WEBSITE_READY; publication date 2026-09-11
  • Canonical citation (immutable commit/path pin): https://github.com/WumboLabs/evaluations/blob/479cb50c197f8ea0dd905a68ffd61f632f49c652/models/qwen38-27b/events/h1-canonical-promotion-2026-09-09/REPORT.md
  • Export source pin: repository-cache WumboLabs/evaluations @ 479cb50c197f8ea0dd905a68ffd61f632f49c652 path models/qwen38-27b/events/h1-canonical-promotion-2026-09-09/website-publication.json sha256 dc5d2ca7e8aefe8cec801041aaedc637f88ad9c3c37cf1ac9ddfe80cfcb0248b

Legacy provenance