Notes on running Large Language models (LLMs) locally#

Running denser models on 16GB VRAM is a challenge, but it can be done:

For example, Qwen3.6 27B quantized to ~13G, leaving ~3GB of room
- Model weights do fit but context size is limited → needs KV-cache compression
KV-cache compression using Trellis-Coded Quantization (TCQ) reduces VRAM requirements
- 2-3x more context in the same VRAM
Multi Token Prediction (MTP) model variants need additional ~1GB VRAM
- nextn layer weights

You can play around with context size, KV compression levels, or MTP-ness of your model. For example, suppose you can run a IQ4_XS-pure quantized model with a 100,000 context using symmetric 3-bit TCQ (turbo3_tcq) without MTP. You are not likely to be able to run a MTP model variant without either reducing context size or compressing the KV cache further.

Furthermore, the MTP nextn layer itself is quantized, usually at a higher quantization level than the model e.g. you might find Q8nextn quants.

/etc/conf.d/llama.cpp
---
# NOTE: Configure as suggested: https://unsloth.ai/docs/models/qwen3.6#thinking-mode
#  Thinking general tasks:                       temp=1.0, top_p=0.95, top_k=20, min_p=0.0, pres_pen=1.5, rep_pen=1.0
#  Thinking precise coding tasks:                temp=0.6, top_p=0.95, top_k=20, min_p=0.0, pres_pen=0.0, rep_pen=1.0
#  Instruct (non-thinking) for general tasks:    temp=0.7, top_p=0.8,  top_k=20, min_p=0.0, pres_pen=1.5, rep_pen=1.0
#  Instruct (non-thinking) for reasoning tasks:  temp=1.0, top_p=0.95, top_k=20, min_p=0.0, pres_pen=1.5, rep_pen=1.0


# NOTE: Configure buun's fork per https://github.com/spiritbuun/buun-llama-cpp#recommended-configurations
#  turbo4 (4.25 bpv) -- lossless quality, great compression:    -ngl 99 -fa -ctk turbo4     -ctv turbo4
#  3-bit TCQ (3.25 bpv) -- best quality at 3-bit:               -ngl 99 -fa -ctk turbo3_tcq -ctv turbo3_tcq
#  2-bit TCQ (2.25 bpv) -- maximum compression:                 -ngl 99 -fa -ctk turbo2_tcq -ctv turbo2_tcq


# NOTE: To use MTP (Multi Token Prediction) you need GGUF with MTP layers, and additional ~1GiB VRAM so reduce context size
#  --spec-type draft-mtp --spec-draft-n-max 2


LLAMA_ARGS="\
    --host 0.0.0.0 -lv 2 \
    -hf Ununnilium/Qwen3.6-27B-IQ4_XS-pure-GGUF --alias qwen3.6-27b \
    --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 \
    --ctx-size 100000 -ngl 999 --fit-target 2 \
    -ctk turbo3_tcq -ctv turbo3_tcq -np 1 -fa on"


# or


LLAMA_ARGS="\
    --host 0.0.0.0 -lv 3 \
    --hf jpetrina/qwen3.6-27b-mtp-IQ4_XS-pure.gguf --alias qwen3.6-27b-mtp \
    --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 \
    --ctx-size 42000 -ngl 999 --fit-target 2 \
    -ctk turbo2_tcq -ctv turbo2_tcq -np 1 -fa on \
    --spec-type draft-mtp --spec-draft-n-max 2"

References: