Qwen3.6-27B-GGUF-4.256bpw

This is a 4.256 BPW quantized model for the GPU poors with 16 GiB of VRAM. It works in both ik_llama.cpp and mainline llama.cpp.

It was quantized using the simplest ever recipe – Q8_0 for the tiny ssm_alpha and ssm_beta tensors, IQ4_XS for the rest.

From local testing with llama-perplexity (wiki.test.raw, 580 chunks), it has the best quality and speed in the same size class:

• Compared to Q3_K_M from bartowski and UD-Q3_K_XL from unsloth, this IQ4_XS quant uses slightly less VRAM while having better quality.
• The IQ4_XS quant from mradermacher, bartowski, and unsloth have much better quality, but they use more VRAM and are harder to fit into 16 GiB of VRAM.

With 16 GiB of VRAM, we can fit a context size of 65536 with quantized KV cache:

# mainline llama.cpp
-c 65536 -ctk q8_0 -ctv q8_0 -np 1

For brave souls that seek the TurboQuant experience (see #21038), we can also fit a context size of 128000 with more heavily quantized KV cache:

# mainline llama.cpp
-c 128000 -ctk q4_0 -ctv q4_0 -np 1

Size

Size from llama-server output:

llm_load_print_meta: model size       = 13.327 GiB (4.256 BPW)
llm_load_print_meta: repeating layers = 12.069 GiB (4.257 BPW, 24.353 B parameters)
...
llm_load_tensors:  CUDA_Host buffer size =   644.14 MiB
llm_load_tensors:      CUDA0 buffer size = 13003.14 MiB

blk\..*\.attn_q\.weight=iq4_xs
blk\..*\.attn_k\.weight=iq4_xs
blk\..*\.attn_v\.weight=iq4_xs
blk\..*\.attn_output\.weight=iq4_xs
blk\..*\.attn_gate\.weight=iq4_xs
blk\..*\.attn_qkv\.weight=iq4_xs

blk\..*\.ssm_alpha\.weight=q8_0
blk\..*\.ssm_beta\.weight=q8_0
blk\..*\.ssm_out\.weight=iq4_xs

blk\..*\.ffn_down\.weight=iq4_xs
blk\..*\.ffn_(gate|up)\.weight=iq4_xs

token_embd\.weight=iq4_xs
output\.weight=iq4_xs

Speed

llama-sweep-bench result with a RTX 3090, with flags -ngl 99 -mqkv -muge -cuda graphs=1 -c 128000 -wgt 1 -wb:

Performance

This quant uses the imatrix from mradermacher. It performs well enough in long reasoning tasks and agentic tasks.