Qwen3.6-27B-GGUF-5.076bpw

This is a 5.076 BPW quantized model for the GPU poors with 24 GiB of VRAM. It uses the SOTA IQK quants, and thus works in ik_llama.cpp only.

From local testing with llama-perplexity (wiki.test.raw, 580 chunks), it has the best quality and speed in the same size class:

With 24 GiB of VRAM, we can fit a context size of 128000 with F16 KV cache:

-c 128000 -wgt 1

or a context size of 262144 with quantized KV cache:

-c 262144 -wgt 1 -ctk q8_0 -khad -ctv q6_0 -vhad

Size

Size from llama-server output:

llm_load_print_meta: model size       = 15.893 GiB (5.076 BPW)
llm_load_print_meta: repeating layers = 13.969 GiB (4.927 BPW, 24.353 B parameters)
...
llm_load_tensors:  CUDA_Host buffer size =   985.16 MiB
llm_load_tensors:      CUDA0 buffer size = 15288.91 MiB

This is sligtly bigger than the Qwen3.5-27B-4.915bpw quant, due to these changes:

• attention: mixture of IQ6_K and IQ5_K => Q6_0
• token_embd: IQ4_K => Q6_0
• output: IQ6_K => Q6_0

The recipe is almost identical with the IQ4_KS + Q6_0 recipe shared by IK in https://github.com/ikawrakow/ik_llama.cpp/discussions/1663, with ssm_alpha and ssm_beta getting a slight bump from Q6_0 to Q8_0.

IQ4_KS + Q6_0 form a good and fast combo, as noted by IK in https://github.com/ikawrakow/ik_llama.cpp/discussions/1663:

• IQ4_KS is better than IQ4_XS, has the same size, has the same performance on CUDA and CPU
• IQ5_K will wipe the floor with Q5_K in terms of quantization accuracy at the same bpw. One issue is that IQ5_K PP is lower on CUDA because of the block size of 16. It is about on par for TG on CUDA, about on par for TG on the CPU, and I think slightly faster PP on the CPU. If one does not want to take the CUDA performance penalty, one could replace Q5_K with IQ5_KS. This will be strictly faster, will use 0.25 bpw less than Q5_K, and will have about the same quantization accuracy.
• I think in may cases one can replace Q6_K with Q6_0, which is quite a bit faster on CUDA while giving about the same quantization accuracy as Q6_K. IQ6_K is better, but slower.
• The situation for Q4_K vs IQ4_KS and IQ4_K is similar to Q5_K vs IQ5_KS and IQ5_K

blk\..*\.attn_q\.weight=q6_0
blk\..*\.attn_k\.weight=q6_0
blk\..*\.attn_v\.weight=q6_0
blk\..*\.attn_output\.weight=q6_0
blk\..*\.attn_gate\.weight=q6_0
blk\..*\.attn_qkv\.weight=q6_0

blk\..*\.ssm_alpha\.weight=q8_0
blk\..*\.ssm_beta\.weight=q8_0
blk\..*\.ssm_out\.weight=q6_0

blk\..*\.ffn_down\.weight=iq4_ks
blk\..*\.ffn_(gate|up)\.weight=iq4_ks

token_embd\.weight=q6_0
output\.weight=q6_0

Speed

llama-sweep-bench result with a RTX 3090, with flags -ngl 99 -mqkv -muge -cuda graphs=1 -c 128000 -wgt 1 -wb:

Performance

This quant uses the imatrix from mradermacher. There are some long reasoning tasks that the full precision model served from https://dashscope-intl.aliyuncs.com/compatible-mode/v1 can solve at about 50:50 chance, and this quant can also solve at about 50:50 chance when using the imatrix from mradermacher. Without using any imatrix, this quant can't solve the tasks at all. This finding vindicates the importance of the importance matrix.

On agentic tasks tested with pi agent, the tasks that can be reliably solved by the full precision model can also be reliably solved by this quant.

For mainline users with 24 GiB of VRAM, I would recommend i1-Q4_K_M from mradermacher, which also performs quite well from limited testing.

For IK users that need even higher quality, I would recommend IQ5_KS from ubergarm, which is near lossless.

Verdict

We get Sonnet 4.5 at home with a used RTX 3090.