Tencent Hy3 GGUF — 1-Bit and 4-Bit Quants for Single-GPU llama.cpp
Jul 14, 2026: Tencent Hunyuan released 1-bit and 4-bit Hy3 GGUF builds for llama.cpp with MTP — 295B MoE on a single 128GB-class GPU. explainx.ai maps VRAM tiers, copy-paste flags, DGX Spark benchmarks, and RTX 3060 reality checks.
"We've just released the 1-bit & 4-bit version of Hy3, a flagship-scale 295B model that can be served on a single GPU. Run Hy3 with llama.cpp, enable MTP, and experience powerful intelligence on dramatically lower hardware."
The thread (~14K views in its first hours) quoted the original July 6 launch — 295B MoE, Apache 2.0, 256K context, free OpenRouter window — and added the missing piece from that post's self-hosting section: you no longer need 8× H20 GPUs for every inference path.
Critical nuance upfront:single GPU here means 128 GB unified-memory class — DGX Spark, Mac Studio 128 GB, Strix Halo — not a 16 GB RTX 3060. One viral reply promised "can't wait to run this on my 3060 and 16GB RAM." That is not what Tencent's memory math supports.
+27–40% tok/s with MTP vs baseline on 128 GB rigs (community benchmarks on PR #25395 builds)
Still need 8 GPUs?
Only for full BF16/FP8 vLLM production — GGUF path is the consumer/prosumer lane
Agent tools?
Prompt-injected tools work; native OpenAI tool-call parsing on Hy3's bespoke format still rough in llama.cpp
India geo-block?
Separate thread reported Tencent 3D site blocked — unrelated to GGUF downloads via Hugging Face
Why July 14 matters — Hy3 goes from datacenter to desk
When Hy3 launched on July 6, explainx.ai's read was blunt: 299B BF16 parameters need 8-GPU tensor parallel via vLLM or SGLang. The 21B active MoE step is cheap per token; weight storage is not.
Hardware reality — what "single GPU" actually means
X thread questions answered
@adidshaft asked the right question: which card, context length, and tok/s for 1-bit?
Tier
Example hardware
Quant rung
Resident footprint
Reported decode
A — fits fully on GPU
DGX Spark GB10, M4 Max 128 GB
IQ3_XXS ~117 GB
~107 GiB (NextN layer skipped at inference)
~17 → 24 tok/s with MTP (+40%)
B — 1-bit ladder
128 GB Mac / Spark
IQ1_M ~62 GB
~60 GB + KV
Coherent; no_think mode recommended at IQ1
C — 4-bit quality
192 GB+ or offload
Q4_K_M + MTP ~183 GB
Split shards
10/10 needle scores to 524K (community cert)
D — not viable
RTX 3060 16 GB
any
CPU offload dominates
Thread aspiration ≠ shipping spec
@4xdotrip asked about Hy3 on a single DGX Spark — yes, that is the reference class. Community quant author vcruz305 benchmarked IQ2_M on Spark GB10: 18.0 → 22.8 tok/s with MTP (+27%, 90% draft acceptance).
@Macchilust's 3060 + 16 GB plan: treat as API-only via OpenRouter free route or smaller dense models — not this MoE.
llama.cpp stack — hy_v3 architecture + MTP
Hy3 uses a new hy_v3 architecture (295B MoE, 80 decoder layers + 1 MTP/NextN layer, 192 experts top-8). Mainline llama.cpp added support via PR #25395 — base graph, MoE router, and draft-mtp speculative head.
Why --spec-draft-p-min 0.75 is mandatory: Hy3's MTP head is single-depth trained. At default p-min, draft acceptance collapses (~39%) and speculation becomes slower than baseline. At 0.75, acceptance jumps to ~88–97% and decode wins appear.
Other gotchas from community testing:
--jinja required — chat template aborts without it
--parallel 1 required for draft-mtp on Hy3
First load of 100 GB+ quants: ~7–8 minutes before first token — do not kill early
Stop string: add <｜hy_eos:opensource｜> — occasional EOG leak without it
Tool calling: native OpenAI-style tool endpoints may 500; inject tools in prompt until parser lands
OpenAI-compatible smoke test
python
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="EMPTY")
r = client.chat.completions.create(
model="hy3",
messages=[{"role": "user", "content": "Write a FastAPI health check route."}],
temperature=0.9,
top_p=1.0,
)
print(r.choices[0].message.content)
Tencent's announcement names 1-bit and 4-bit; Hugging Face's quant tree (46+ variants as of July 14) fills the rungs:
Label
Approx size
Precision story
Best for
1-bit (IQ1_M)
~62 GB
Smallest certified ladder rung
128 GB Mac experiments, no_think tasks
2-bit (IQ2_M + MTP)
~100 GB
Recommended in satgeze ladder
DGX Spark daily driver
3-bit (IQ3_XXS-UD)
~117 GB
Asymmetric: hot tensors Q5–Q8, experts low-bit
Single-node 128 GB quality balance
4-bit (NVFP4 experts)
~174 GB
Routed experts 4-bit; router/attention BF16
Near-FP8 quality, still one Spark
4-bit (Q4_K_M + MTP)
~183 GB
General k-quant sweet spot
Long-context needle tests to 524K
MoE quantization insight: 192 routed experts are ~97% of parameters but only 8 activate per token — quant recipes put the bit budget on experts while keeping attention, shared expert, and output head at Q6–Q8. That is why aggressive IQ rungs remain usable.
Compare to GLM-5.2 Unsloth 2-bit at ~245 GB minimum — Hy3's smaller total parameter count wins the "fits on one desk" race even though GLM-5.2 leads some coding leaderboards.
Hy3 GGUF vs alternatives — July 2026 local stack
Model
MoE size
Local path
Min practical hardware
Hy3 GGUF
295B / 21B active
llama.cpp + MTP
128 GB unified
GLM-5.2
744B / ~32B active
Unsloth 2-bit GGUF
256 GB unified
GLM-5.2 Colibrì
744B
Disk-stream experts
25 GB RAM + 370 GB NVMe (slow)
Qwen 3.6 27B dense
27B dense
llama.cpp Q5
24–48 GB consumer
For MacBook vs dedicated GPU shoppers: Hy3 GGUF is the first frontier MoE that treats 128 GB Apple Silicon as a first-class target — not an afterthought.
For Fable 5 local hardware projections: closed-weight parity may lag years; Hy3 GGUF is proof Chinese open weights keep compressing the desk-scale window.
Agent and product context
Hy3's July 6 positioning was agent reliability — SWE-bench variance ≤4% across CodeBuddy, Cline, and KiloCode scaffolds. The GGUF release does not change model weights; it changes who can run those agents locally without routing prompts through Shenzhen-hosted APIs.
OpenClaw v2026.7.1 already ships Hy3 via TokenHub. Self-hosters can now mirror that stack on llama-server for air-gapped labs — same China AI playbook thesis: open weights + commodity inference.
Compliance reminder: Apache 2.0 clears license friction; it does not clear data-sovereignty review for regulated workloads — same framing as Asian AI alternatives.
Who should download today
Do it if:
You own 128 GB unified memory and want a 295B-class agent offline
You are A/B testing Hy3 vs GLM-5.2 on identical prompts
You need 256K context locally for repo-scale agents (watch KV cache math — 32K comfortable on 128 GB)
Skip if:
You have ≤24 GB VRAM — use hosted API or dense 27B models
You need production tool-call parsers today — stay on vLLM 8-GPU recipes until llama.cpp Hy3 tool support matures
You require maximum coding benchmark scores — GLM-5.2 may still edge Hy3 on your eval
Summary
On July 14, 2026, Tencent released 1-bit and 4-bit Hy3 GGUF quants aimed at single-GPU serving through llama.cpp + MTP — turning a model that launched on 8 GPUs into a 128 GB desk-scale option. The X thread's excitement is justified on DGX Spark and 128 GB Mac hardware; it is not a free pass for 16 GB consumer cards.
The load-bearing flags: --spec-type draft-mtp, --spec-draft-p-min 0.75, --parallel 1, --jinja. Get those right and community benchmarks show ~+27–40% tok/s over non-speculative decode. Get them wrong and MTP becomes a tax.
Quant sizes, tok/s benchmarks, and llama.cpp PR merge status reflect community reports as of July 14, 2026. Verify GGUF shard dates and arch metadata (hy_v3) before multi-hour downloads. Free OpenRouter promotions from the July 6 launch may expire — check current pricing.
