ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF
类型 text-generation · 框架 gguf · 下载 1,747,945 · 收藏 507
类型 text-generation · 框架 gguf · 下载 1,747,945 · 收藏 507
定位:这是 IST-DASLab 用 GSQ 与 RCO 方法为 DeepSeek-V4.1-Flash-1017 制作的 GGUF 非均匀量化版本,面向显存有限、想在本地跑 763B MoE 模型的部署场景。
核心能力
适用人群:熟悉 llama.cpp 部署、需要本地运行大 MoE 模型的开发者;需 llama.cpp b11870 或更新版本,并配合 Strata 在 12 GB 显存单卡上运行。
注意:需为每个构建手动建立指向 Engram 文件的软链接。
位置づけ:IST-DASLab が GSQ と RCO を用いて DeepSeek-V4.1-Flash-1017 を GGUF 化した非一様量子化版で、限られた VRAM で 763B の MoE をローカル実行する用途に向く。
主な機能
対象読者:llama.cpp での配備に慣れ、大規模 MoE をローカルで動かしたい開発者。llama.cpp b11870 以降が必要で、Strata と組み合わせれば 12 GB VRAM の単一 GPU で動作する。
注意:各構成ごとに Engram ファイルへのシンボリックリンクを手動で作成する必要がある。
정의: IST-DASLab이 GSQ와 RCO로 DeepSeek-V4.1-Flash-1017을 GGUF로 양자화한 비균일 버전이며, 제한된 VRAM에서 763B MoE를 로컬로 실행하려는 배포 환경을 겨냥한다.
핵심 기능
대상 사용자: llama.cpp 배포에 익숙하고 대형 MoE를 로컬에서 돌리려는 개발자. llama.cpp b11870 이상이 필요하며, Strata와 함께 쓰면 12 GB VRAM 단일 GPU에서 실행된다.
주의: 빌드마다 Engram 파일을 가리키는 심볼릭 링크를 직접 만들어야 한다.
What it is: GGUF non-uniform quantizations of DeepSeek-V4.1-Flash-1017 by IST-DASLab, built with GSQ and RCO for running a 763B MoE locally on limited VRAM.
Key capabilities
Who it is for: Developers comfortable with llama.cpp who want to run a large MoE locally; needs llama.cpp b11870 or newer and, with Strata, a single 12 GB GPU.
Caveats: Each build requires a manual symlink to the Engram file.
来源 · 查看原文
basemodel: deepseek-ai/DeepSeek-V4.1-Flash-1017 basemodelrelation: quantized pipelinetag: text-generation library_name: gguf license: mit tags: - gguf - gsq - rco - quantization - mixed-precision - ist-daslab - moe - deepseek ---
Non-uniform GGUF quantizations of a 763B-parameter, 384-expert MoE, produced with GSQ and RCO.
Base model vs GSQ-RCO quants on AIME25, GPQA-Diamond and LiveCodeBench v6
Update, October 27, 2026. The files now use llama.cpp's own DeepSeek-V4.1 layout and load in llama.cpp b11870 or newer, including images. - Downloaded the earlier files? You do not need to download the model again. Run convertds41upstream.py on your model folder: it rewrites the files locally into exactly the current ones and fetches only 80 KB. See Updating an existing download. - New download? The shared Engram file is shard 6 of every build, so link it once per build you use (<ID>isIQ1_M,IQ2_XXSorIQ2_XS; see llama.cpp): ``sh ln -s DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf DeepSeek-V4.1-Flash-1017-GSQ-RCO-<ID>-00006-of-00006.gguf``
This repository provides GGUF quantizations of DeepSeek-V4.1-Flash-1017 at three sizes: IQ1M (1.75 bits per weight), IQ2XXS (2.00) and IQ2_XS (2.25). In contrast to uniform quantization, which applies a single quantization type to all weight tensors, each model here assigns a separate quantization type to every tensor. The assignment is obtained by a gradient-based search that allocates precision according to per-tensor sensitivity, subject to a total size budget. The resulting files are standard GGUF.
DeepSeek-V4.1-Flash-1017 is a sparse mixture-of-experts model with 763.2B parameters: 384 routed experts per layer, of which 6 are active per token, and a 1M-token context window. The routed experts dominate the parameter count, so they also dominate the size budget, and the search spends most of its freedom there, assigning a quantization type per layer (not per expert, which GGUF cannot express).
With Strata, every build runs on a single GPU with 12 GB of VRAM. See Usage.
Method summary. GSQ provides accurate low-bit scalar quantization of each tensor at a given quantization type; RCO assigns the per-tensor quantization types under a size budget. Together they yield a non-uniform GGUF at the requested size.
| Method | Description |
|---|---|
| GSQ (Gumbel-Softmax Quantization, paper, code) | Post-training scalar quantization that jointly learns the per-coordinate grid assignments and the per-group scales via a Gumbel-Softmax relaxation. GSQ closes most of the gap between scalar and vector quantization at 2 to 3 bits while remaining deployable in standard scalar formats such as GGUF. |
| RCO (Riemannian Constrained Optimization, paper, code) | Assigns one of K quantization types to each of N tensors under a total size budget. The budget constraint is reformulated as a smooth Riemannian manifold in logit space, which permits gradient-based optimization directly on the task loss while enforcing the budget exactly, without constraint-specific hyperparameter tuning. |
Both methods were developed at the Deep Algorithms and Systems Lab (DASLab), Institute of Science and Technology Austria.
Each build is a backbone GGUF in five shards, named after its quantization identifier: DeepSeek-V4.1-Flash-1017-GSQ-RCO-<ID>-0000i-of-00006.gguf, i = 1 to 5. Shard 1 holds the metadata only, shards 2 to 5 the transformer weights, 551.9B parameters. The bpw is the average bit-width over these, which is what the search controls.
The Engram file, DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf, holds the model's two Engram lookup tables (196.6B parameters) at IQ4_NL, 4.50 bpw. It is identical for all builds, so it is stored once and needs to be downloaded only once, also when trying several builds. llama.cpp loads it as shard 6 of a build, so each build needs a link to it named ...-<ID>-00006-of-00006.gguf (one command, see llama.cpp). The tables are a lookup, not a matmul weight, so they are held at a fixed type in every build and excluded from the search. They are quantized in the evaluated models, not just the shipped ones.
| Build | bpw | Backbone | Engram | Total | Notes |
|---|---|---|---|---|---|
IQ1_M | 1.75 | 120.72 GB | 110.60 GB | 231.32 GB | Smallest |
IQ2_XXS | 2.00 | 138.24 GB | 110.60 GB | 248.84 GB | Recommended; closest to the base model |
IQ2_XS | 2.25 | 155.23 GB | 110.60 GB | 265.83 GB |
All three builds run on a single 12 GB GPU with Strata.
The Engram column is the same shared file in every row. The vision projector, mmproj-DeepSeek-V4.1-Flash-1017-BF16.gguf (0.97 GB), also serves all builds. GB is 109 bytes.
The IQ2XXS build is the stronger operating point: it matches the base model on all three benchmarks and so reaches 100.0% of its task average (93.54). The IQ1M build trades 1.2 points of task average for 17.5 GB less.
Only the backbone holds transformer weights. The Engram tables are read sparsely, one row per token, so they can stay memory-mapped on disk and do not need to occupy VRAM or RAM. With Strata, the backbone does not need to fit in VRAM either: every build runs on a 12 GB GPU, with the rest of the model held in system memory. Keep the Engram file on an SSD so the paging stays cheap.
All models are evaluated against the base model deepseek-ai/DeepSeek-V4.1-Flash-1017 as released (FP8 dense weights, FP4 routed experts). We report three reasoning and generation benchmarks: AIME25, GPQA-Diamond, and LiveCodeBench v6. Task average is the mean of the three; recovery is the task average relative to the base model.
All numbers in this card were measured at reasoning effort 50.
| Variant | bpw | AIME25↑ | GPQA-D↑ | LCB v6↑ | Task avg↑ | Recovery |
|---|---|---|---|---|---|---|
| Base (FP8 / FP4) | n/a | 100.00 | 90.91 | 89.71 | 93.54 | 100.0% |
| GSQ-RCO IQ1_M | 1.75 | 100.00 | 88.38 | 88.57 | 92.32 | 98.7% |
| GSQ-RCO IQ2_XXS | 2.00 | 100.00 | 90.91 | 89.71 | 93.54 | 100.0% |
| GSQ-RCO IQ2_XS | 2.25 | 100.00 | 90.40 | 89.71 | 93.37 | 99.8% |
IQ2XXS (2.00 bpw) keeps 100.0% of the base model's task average: it matches the base model on AIME25 (100.00), GPQA-Diamond (90.91) and LiveCodeBench v6 (89.71). IQ2XS (2.25 bpw) matches it on AIME25 and LiveCodeBench v6 and scores 90.40 on GPQA-Diamond, 99.8% overall. The smallest build, IQ1_M (1.75 bpw), also scores 100.00 on AIME25 and retains 98.7% overall.
Runtime support. llama.cpp supports DeepSeek-V4.1, including images, since release b11870 (October 25, 2026), and the files here use its layout since October 27, 2026. Downloads from before that date can be converted locally, see Updating an existing download. The GGUFs are also supported in Strata.
Strata runs these GGUFs on a single NVIDIA GPU with 12 GB of VRAM, with a one-click installer and an OpenAI-compatible endpoint on localhost. The builds were optimized together with Strata for local inference.
Requires llama.cpp b11870 or newer.
# download one build and the shared Engram file (requires: pip install -U "huggingface_hub[cli]")
hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF \
--include "*IQ2_XXS*" "*Engram*" --local-dir .
# the Engram file is shard 6 of every build
ln -s DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00006-of-00006.gguf
llama-cli -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf \
-p "Explain mixed-precision quantization." -ngl 99
-m takes the first shard; llama.cpp opens shards 2 to 6 from the same directory. To try another build, download only its shards (--include "*IQ1_M*") and add its link. On Windows, mklink /H <link> DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf makes the link; convertds41upstream.py, run on the folder, also creates missing links.
The Engram table stays memory-mapped on disk and is read one row per token (llama.cpp's default --lazy-mode auto does this for tensors over 4 GiB), so it needs neither VRAM nor RAM. See Memory requirements.
Download the vision projector and pass it with --mmproj:
hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF \
--include "mmproj-*" --local-dir .
llama-server -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf \
--mmproj mmproj-DeepSeek-V4.1-Flash-1017-BF16.gguf -ngl 99
With llama-mtmd-cli, also pass --jinja so that the chat template embedded in the GGUF is used.
The base repository ships no Jinja chat template; its prompt format is defined by DeepSeek's encoder, encoding/encoding.py. The GGUFs embed a template adapted from Unsloth's DeepSeek-V4.1 release, checked against that encoder for text chat, tool use, and image formatting. Reasoning effort accepts an integer from 1 to 100 (also a numeric string), or low, high, max (50, 75, 100). The default effort is DeepSeek's high (75), and effort applies only in thinking mode. Structured image blocks, including images in tool results, retain their image placeholders. Image inference needs the vision projector, see Images.
llama-server -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf -ngl 99 \
--reasoning on --reasoning-effort 50
Effort 50 is the setting all results in this card were measured with. Per request, set "reasoning_effort": "50" in the chat completion body (a string, also for numbers). --reasoning off disables thinking.
Files downloaded before October 27, 2026 (four shards per build and a separate DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf) were written for an earlier llama.cpp branch and do not load in llama.cpp. convertds41upstream.py turns them into the current files of this repository, byte for byte, without downloading the model again. It needs Python 3.8+ and no extra packages, and fetches only two small files of this repository (chat_template.jinja and exp_probs_b_vl.f32, 80 KB).
cd /path/to/downloaded/model-folder
hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF convert_ds41_upstream.py --local-dir .
python3 convert_ds41_upstream.py .
It converts every build in the folder and the mmproj, and writes the new backbone shards next to the old ones; every written tensor is read back and compared with its source. The Engram file is converted in place: only its header changes, its 110 GB of tables stay as they are, and the script adds the shard-6 links. The free space needed is the size of the new shards (121 to 155 GB per build). With --delete-old, the old shards of a build are deleted as soon as its new ones are verified, so space for one build suffices. The new first shards take the chat template from chat_template.jinja, so patch_chat_template.py is not needed.
The dense tensors (attention, shared experts, embeddings, output head) and the routed-expert matrices draw on separate quantization ladders, because the routed experts sit at a much lower bit-width than the dense path at any given budget: the dense tensors choose among Q5K, Q6K and Q80, the routed experts among types from IQ1S to IQ3_S. The search's teacher is the released checkpoint itself, with the routed experts in their native FP4, and its calibration data was generated by the model.
Reference implementations: GSQ at IST-DASLab/GSQ and RCO at IST-DASLab/RCO.
Each released build ships the files needed to audit how it was built:
| File | Contents |
|---|---|
tensor-allocation/DeepSeek-V4.1-Flash-1017-GSQ-RCO-<ID>.rco-allocation.txt | The quantization type assigned to every tensor of that build (all six shards, 1043 tensors), with a quant-type histogram and the target and measured bit-width. This is the RCO search result, so the allocation can be inspected without opening the model. |
convert_ds41_upstream.py, chat_template.jinja, exp_probs_b_vl.f32 | The converter for downloads from before October 27, 2026 (see Updating an existing download), the chat template the GGUFs embed, and the vision router bias the converter adds (40 x 384 F32, ffn.gate.bias_vl of the original checkpoint). |
If you use these models or methods, please cite both papers:
@article{gsq2026,
title = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurtic, Eldar and Kleinegger, Maximilian and Alistarh, Dan},
journal= {arXiv preprint arXiv:2604.18556},
year = {2026}
}
@article{rco2026,
title = {Model Compression with Exact Budget Constraints via Riemannian Manifolds},
author = {Helcig, Michael and Alistarh, Dan},
journal= {arXiv preprint arXiv:2605.00649},
year = {2026}
}
We thank Verda and Scientific Computing at the Institute of Science and Technology Austria for providing the compute resources used to produce these models.
These quantized weights inherit the license of the base model (DeepSeek-V4.1-Flash-1017, MIT). The GSQ-RCO tooling is released by the Deep Algorithms and Systems Lab under its repository license.