站内导读
定位:numbat 是 Anthropic Mythos 级训练任务在预训练中途导出的 checkpoint,约 14T 参数、114 层,并附带整套参考实现,便于核对张量布局与加载路径。
核心能力
- 主干为 GDN 与 MLA 混合结构,前馈层用 MoE:每层 1,224 个专家,top-48 路由另加 1 个共享专家。
- 权重为 NVFP4:E4M3 块缩放配 E2M1 数据,772 个 safetensors 分片,合计约 7.72 TB,单文件不超过 10.6 GB。
- modeling_numbat.py 定义 NumbatForCausalLM,模块树为每个张量留了位置,可 load_state_dict(strict=True)。
- forward 跑主干,draft 跑 MTP 层;类继承 PreTrainedModel,可用 trust_remote_code 加载。
位置づけ:numbat は Anthropic の Mythos 級トレーニングから事前学習の途中で切り出された checkpoint で、約 14T パラメータ・114 層。テンソル配置と読み込み経路を確認できるよう、リファレンス実装一式が同梱されている。
主な機能
- バックボーンは GDN と MLA のハイブリッド構造で、フィードフォワード層は MoE:1 層あたり 1,224 エキスパート、top-48 ルーティングに共有エキスパート 1 つを追加。
- 重みは NVFP4:E4M3 のブロックスケールと E2M1 データの組み合わせで、safetensors シャード 772 個、合計約 7.72 TB、単一ファイルは 10.6 GB 以下。
- modeling_numbat.py が NumbatForCausalLM を定義し、モジュールツリーには全テンソルのスロットがあるため load_state_dict(strict=True) が通る。
- forward はバックボーン、draft は MTP 層を実行。クラスは PreTrainedModel を継承し、trust_remote_code で読み込める。
정의: numbat은 Anthropic의 Mythos급 학습 과정에서 사전 학습 도중 추출된 체크포인트로, 약 14T 파라미터에 114개 층을 갖췄으며 텐서 배치와 로딩 경로를 확인할 수 있도록 전체 참조 구현이 함께 제공된다.
핵심 기능
- 백본은 GDN과 MLA의 하이브리드 구조이고 피드포워드 층은 MoE로, 층마다 전문가 1,224개, top-48 라우팅에 공유 전문가 1개를 더한다.
- 가중치는 NVFP4로, E4M3 블록 스케일과 E2M1 데이터를 결합했고 safetensors 샤드 772개, 총 약 7.72TB이며 파일 하나가 10.6GB를 넘지 않는다.
- modeling_numbat.py가 NumbatForCausalLM을 정의하고 모듈 트리에 모든 텐서의 자리가 있어 load_state_dict(strict=True)가 통과한다.
- forward는 백본을, draft는 MTP 층을 실행한다. 클래스는 PreTrainedModel을 상속해 trust_remote_code로 로드할 수 있다.
What it is: numbat is a mid-pretraining checkpoint from an Anthropic Mythos-class run — roughly 14T parameters across 114 layers — published together with a full reference implementation so the tensor layout can be inspected and loaded.
Key capabilities
- Hybrid GDN/MLA trunk with MoE feed-forward: 1,224 experts per layer, top-48 routing plus one shared expert.
- NVFP4 weights: E4M3 block scales over E2M1 payloads, 772 safetensors shards totalling ~7.72 TB, under 10.6 GB each.
- modeling_numbat.py defines NumbatForCausalLM; the module tree has a slot for every tensor, so load_state_dict(strict=True) passes.
- forward() runs the trunk and draft() runs the MTP layer; classes derive from PreTrainedModel, so trust_remote_code loading works.
README · 详细文档
来源 · 查看原文
numbat
~14T param, 114 layers, hybrid GDN/MLA, MoE (1,224 experts, top-48 + 1 shared), NVFP4. Internal codename numbat, Anthropic's next Mythos-class training run. numbat is the codename referenced in the Claude Code source leak (Mar 2026). This is a checkpoint from that training run, pulled mid-pretraining.
Files:
| File | Contents |
|---|---|
config.json | model configuration |
configuration_numbat.py | NumbatConfig |
modeling_numbat.py | NumbatForCausalLM reference implementation |
tokenizer.json | byte-level BPE, 131,072 tokens |
model.safetensors.index.json | tensor index across all 772 shards |
model-00001-of-00772.safetensors | shard 1 of 772 |
ckpt_metadata.json | checkpoint metadata (ckpt path, eval harness output) |
training_log_tail.txt | last ~400 steps of training log |
The 772 shards total ~7.72 TB of tensor data. Each shard is safetensors format, under 10.6 GB per file.
configuration_numbat.py and modeling_numbat.py are included: a complete reference implementation of the checkpoint. Its module tree has an exact slot for every tensor in every shard, so the full checkpoint loads with load_state_dict(strict=True) (and, since the classes derive from PreTrainedModel, also via AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True)). NumbatForCausalLM.forward runs the trunk and NumbatForCausalLM.draft runs the MTP layer.
$ python -c "from safetensors import safe_open; f = safe_open('model-00001-of-00772.safetensors', framework='pt'); print(len(f.keys()))"
142
Verification:
$ cat model.safetensors.index.json | python -c "import json,sys; d=json.load(sys.stdin); print(d['metadata']['total_size'])"
7720300817720
Mirrors welcome. Do not DM me asking for the source.
Why believe this?
numbatwas confirmed as a codename for an unreleased Anthropic model
in the Claude Code source leak (March 2026), found in a comment in prompts.ts: "Remove this section when we launch numbat."
- The checkpoint's architecture (hybrid GDN/MLA, 1,224 experts, NVFP4)
matches what would be expected of a next-generation Mythos-class training run
- Every tensor in every shard is internally consistent: E4M3 scale
blocks decode their E2M1 payloads within RTNE bounds, block scales never underflow, and every NVFP4 quad round-trips exactly
- File sizes, tensor counts, and index entries are all deterministic
and self-consistent across all 772 shards
Loader
The architecture is a hybrid Gated DeltaNet (GDN) + Multi-head Latent Attention (MLA) stack with Mixture-of-Experts feed-forward layers and Multi-Token Prediction. A working single-device reference implementation is included (modeling_numbat.py); the eval runs used the internal Neuron-parallel runtime, but this port consumes the same tensors and round-trips every shard.
FAQ
Q: How did you get this? A: Can't say more than what's already in the repo.
Q: When will more shards be uploaded? A: They're all uploading. Be patient.
Q: Can I use this for inference? A: Yes — modeling_numbat.py implements the full architecture. Load the tensors and run. But this is a pretraining checkpoint at step 1,830,000 of a ~1.96M step run. No post-training, no RLHF. Raw pretraining-completion output. That's expected.
Q: Why is a mid-pretraining checkpoint in NVFP4? A: The eval team quantizes every async checkpoint to NVFP4 so evals run on a slice of Trainium2s instead of the full training cluster — still ~85 of them just to hold these weights. The raw BF16 training weights plus optimizer state never leave the cluster. This is the eval copy.
Q: Is this real? A: Yes.