wpy/natural-memory-nm21
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
643e22ecb905392eedef8b52ce7c9d0c7f185ed1
natural-memory-nm21/router_scorecard_final.md
T
WpyQwq 643e22ecb9 Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据 
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线
- 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同）
- 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00）
- 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00%
- 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益）
- 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
2026-09-19 11:11:31 +08:00

17 KiB
Raw Blame History

metric V2-128 deployed(v3) V2-128 v6 final REPLAY-128 v7 final V2-512 v6 final XL-512 v6 final XL-128 v6 final
参数 2,037,774 2,037,774 2,037,774 4,741,902 7,898,127 6,714,639
router_dim 128 128 128 512 512 128
每条记录地址字节 512 512 512 2048 2048 512
Top-1 正确率 41.12% 94.37% 94.14% 94.18% 95.02% 95.19%
Recall@1 37.03% 88.82% 88.58% 88.65% 89.48% 89.64%
Recall@3 46.73% 96.40% 96.48% 96.66% 96.53% 97.45%
Recall@5 48.91% 97.40% 97.15% 97.64% 97.61% 98.20%
MRR 47.69% 95.97% 95.77% 95.87% 96.64% 96.70%
nDCG@3 44.22% 95.31% 95.37% 95.48% 95.60% 96.43%
多跳证据全中(Top-3) 43.43% 95.95% 96.22% 96.44% 95.35% 97.29%
hop 正确率 73.23% 100.00% 100.00% 100.00% 100.00% 100.00%
hop 欠预测率 4.68% 0.00% 0.00% 0.00% 0.00% 0.00%
need F1 89.58% 100.00% 100.00% 100.00% 100.00% 100.00%
need 召回 99.82% 100.00% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00%
已知问题被误拒率 0.18% 0.00% 0.00% 0.00% 0.00% 0.00%
未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00% 0.00%
仲裁准确率 81.12% 100.00% 100.00% 100.00% 100.00% 100.00%
已知问题 need 概率均值 97.63% 100.00% 100.00% 100.00% 100.00% 100.00%
未知问题 need 概率均值 99.89% 0.00% 0.00% 0.00% 0.00% 0.00%
平均分数余量 -9.2989 51.3153 40.1456 73.2482 215.7533 200.3346
单查询延迟 ms(GPU) 1.0995 0.9277 0.9663 0.9488 1.4501 1.5358
路由 QPS(GPU, 单查询) 909.4856 1077.8859 1034.8360 1053.9133 689.6254 651.1112
批量 QPS(GPU, batch=64) 66037.9306 72856.4274 69378.1979 71586.7654 46891.9434 33111.5549
批量 QPS(GPU, batch=256) 207287.4493 236293.1512 242759.8763 177958.2077 85734.5711 131545.1415
批量延迟 ms(GPU, batch=64) 0.0151 0.0137 0.0144 0.0140 0.0213 0.0302

family = benchmark_qa

metric V2-128 deployed(v3) V2-128 v6 final REPLAY-128 v7 final V2-512 v6 final XL-512 v6 final XL-128 v6 final
episodes 64 64 64 64 64 64
Top-1 正确率 - - - - - -
Recall@3 - - - - - -
MRR - - - - - -
hop 正确率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00%
已知问题被误拒率 - - - - - -
未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00% 0.00%
平均分数余量 - - - - - -

family = mega_validation

metric V2-128 deployed(v3) V2-128 v6 final REPLAY-128 v7 final V2-512 v6 final XL-512 v6 final XL-128 v6 final
episodes 20000 20000 20000 20000 20000 20000
Top-1 正确率 44.24% 98.38% 98.22% 98.38% 98.40% 98.44%
Recall@3 49.15% 99.51% 99.72% 99.75% 98.68% 99.82%
MRR 50.04% 99.19% 99.11% 99.18% 99.20% 99.21%
hop 正确率 74.95% 100.00% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00%
已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00% 0.00%
平均分数余量 -8.0919 57.1101 44.7334 81.8525 235.8911 217.4091

family = memory_policy

metric V2-128 deployed(v3) V2-128 v6 final REPLAY-128 v7 final V2-512 v6 final XL-512 v6 final XL-128 v6 final
episodes 1728 1728 1728 1728 1728 1728
Top-1 正确率 14.53% 59.40% 58.76% 57.70% 66.90% 67.55%
Recall@3 26.77% 68.42% 67.31% 68.60% 77.09% 75.92%
MRR 28.02% 67.60% 66.33% 66.64% 74.88% 74.81%
hop 正确率 57.41% 100.00% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00%
已知问题被误拒率 1.87% 0.00% 0.00% 0.00% 0.00% 0.00%
未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00% 0.00%
平均分数余量 -19.3599 0.2715 -0.2722 -2.6484 40.8526 52.4323

family = native_memory

metric V2-128 deployed(v3) V2-128 v6 final REPLAY-128 v7 final V2-512 v6 final XL-512 v6 final XL-128 v6 final
episodes 128 128 128 128 128 128
Top-1 正确率 0.00% 52.34% 47.66% 49.53% 38.32% 50.47%
Recall@3 2.80% 77.57% 77.57% 82.24% 85.98% 87.85%
MRR 10.23% 67.49% 65.73% 67.05% 61.74% 70.02%
hop 正确率 53.91% 100.00% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00%
已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00% 0.00%
平均分数余量 -29.2917 -0.8787 -1.0887 -2.5814 -5.2606 6.6553

读取/拒答门槛曲线

run threshold need F1 need 召回 未知拒答率 已知问题被误拒率 未知被误读率 tp/tn/fp/fn
V2-128 deployed(v3) 0.30 89.67% 100.00% 0.00% 0.00% 100.00% 17814/0/4106/0
V2-128 deployed(v3) 0.40 89.67% 100.00% 0.00% 0.00% 100.00% 17814/0/4106/0
V2-128 deployed(v3) 0.50 89.58% 99.82% 0.00% 0.18% 100.00% 17782/0/4106/32
V2-128 deployed(v3) 0.60 89.30% 99.26% 0.00% 0.74% 100.00% 17682/0/4106/132
V2-128 deployed(v3) 0.70 89.19% 99.05% 0.00% 0.95% 100.00% 17645/0/4106/169
V2-128 deployed(v3) 0.80 89.01% 98.68% 0.00% 1.32% 100.00% 17579/0/4106/235
V2-128 v6 final 0.30 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 final 0.40 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 final 0.50 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 final 0.60 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 final 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 final 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
REPLAY-128 v7 final 0.30 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
REPLAY-128 v7 final 0.40 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
REPLAY-128 v7 final 0.50 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
REPLAY-128 v7 final 0.60 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
REPLAY-128 v7 final 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
REPLAY-128 v7 final 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-512 v6 final 0.30 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-512 v6 final 0.40 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-512 v6 final 0.50 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-512 v6 final 0.60 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-512 v6 final 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-512 v6 final 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 final 0.30 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 final 0.40 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 final 0.50 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 final 0.60 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 final 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 final 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-128 v6 final 0.30 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-128 v6 final 0.40 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-128 v6 final 0.50 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-128 v6 final 0.60 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-128 v6 final 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-128 v6 final 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0

按 mega 类别拆解（每类 2,000 条）

类别 episodes metric V2-128 deployed(v3) V2-128 v6 final REPLAY-128 v7 final V2-512 v6 final XL-512 v6 final XL-128 v6 final
(未分类) - Top-1 正确率 - - - - - -
(未分类) - Recall@3 - - - - - -
(未分类) - MRR - - - - - -
(未分类) - 多跳证据全中 - - - - - -
(未分类) - hop 正确率 - - - - - -
(未分类) - 未知拒答率 - - - - - -
(未分类) - 已知问题被误拒率 - - - - - -
(未分类) - 未知问题被误读率 - - - - - -
conflict_update 2000 Top-1 正确率 79.60% 99.30% 99.10% 99.35% 99.25% 99.45%
conflict_update 2000 Recall@3 91.25% 100.00% 100.00% 100.00% 100.00% 100.00%
conflict_update 2000 MRR 85.83% 99.65% 99.55% 99.67% 99.62% 99.72%
conflict_update 2000 多跳证据全中 91.25% 100.00% 100.00% 100.00% 100.00% 100.00%
conflict_update 2000 hop 正确率 49.55% 100.00% 100.00% 100.00% 100.00% 100.00%
conflict_update 2000 未知拒答率 - - - - - -
conflict_update 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
conflict_update 2000 未知问题被误读率 - - - - - -
distractor_128 2000 Top-1 正确率 23.10% 97.50% 97.30% 98.00% 98.00% 97.85%
distractor_128 2000 Recall@3 25.35% 100.00% 100.00% 100.00% 100.00% 100.00%
distractor_128 2000 MRR 26.60% 98.75% 98.65% 99.00% 99.00% 98.92%
distractor_128 2000 多跳证据全中 25.35% 100.00% 100.00% 100.00% 100.00% 100.00%
distractor_128 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
distractor_128 2000 未知拒答率 - - - - - -
distractor_128 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
distractor_128 2000 未知问题被误读率 - - - - - -
distractor_32 2000 Top-1 正确率 22.55% 98.45% 98.30% 98.05% 98.20% 98.45%
distractor_32 2000 Recall@3 24.55% 100.00% 100.00% 100.00% 100.00% 100.00%
distractor_32 2000 MRR 25.96% 99.22% 99.15% 99.02% 99.09% 99.22%
distractor_32 2000 多跳证据全中 24.55% 100.00% 100.00% 100.00% 100.00% 100.00%
distractor_32 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
distractor_32 2000 未知拒答率 - - - - - -
distractor_32 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
distractor_32 2000 未知问题被误读率 - - - - - -
forget_correction 2000 Top-1 正确率 - - - - - -
forget_correction 2000 Recall@3 - - - - - -
forget_correction 2000 MRR - - - - - -
forget_correction 2000 多跳证据全中 - - - - - -
forget_correction 2000 hop 正确率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00%
forget_correction 2000 未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00%
forget_correction 2000 已知问题被误拒率 - - - - - -
forget_correction 2000 未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00% 0.00%
long_context 2000 Top-1 正确率 52.55% 97.95% 97.80% 97.70% 98.30% 98.15%
long_context 2000 Recall@3 62.45% 100.00% 100.00% 100.00% 100.00% 100.00%
long_context 2000 MRR 58.55% 98.97% 98.89% 98.84% 99.14% 99.07%
long_context 2000 多跳证据全中 62.45% 100.00% 100.00% 100.00% 100.00% 100.00%
long_context 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
long_context 2000 未知拒答率 - - - - - -
long_context 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
long_context 2000 未知问题被误读率 - - - - - -
multi_hop 2000 Top-1 正确率 73.00% 98.85% 99.05% 98.65% 98.65% 98.90%
multi_hop 2000 Recall@3 66.57% 96.05% 97.72% 98.02% 89.42% 98.52%
multi_hop 2000 MRR 84.53% 99.42% 99.52% 99.33% 99.33% 99.45%
multi_hop 2000 多跳证据全中 37.15% 92.10% 95.45% 96.05% 78.85% 97.05%
multi_hop 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
multi_hop 2000 未知拒答率 - - - - - -
multi_hop 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
multi_hop 2000 未知问题被误读率 - - - - - -
paraphrase 2000 Top-1 正确率 44.30% 99.50% 99.40% 99.55% 99.50% 99.70%
paraphrase 2000 Recall@3 55.25% 100.00% 100.00% 100.00% 100.00% 100.00%
paraphrase 2000 MRR 51.19% 99.75% 99.70% 99.78% 99.75% 99.85%
paraphrase 2000 多跳证据全中 55.25% 100.00% 100.00% 100.00% 100.00% 100.00%
paraphrase 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
paraphrase 2000 未知拒答率 - - - - - -
paraphrase 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
paraphrase 2000 未知问题被误读率 - - - - - -
random_position 2000 Top-1 正确率 24.90% 98.05% 97.80% 98.25% 98.05% 98.10%
random_position 2000 Recall@3 28.70% 100.00% 100.00% 100.00% 100.00% 100.00%
random_position 2000 MRR 29.20% 99.02% 98.89% 99.12% 99.02% 99.05%
random_position 2000 多跳证据全中 28.70% 100.00% 100.00% 100.00% 100.00% 100.00%
random_position 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
random_position 2000 未知拒答率 - - - - - -
random_position 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
random_position 2000 未知问题被误读率 - - - - - -
single_fact 2000 Top-1 正确率 33.90% 97.45% 97.05% 97.45% 97.25% 96.90%
single_fact 2000 Recall@3 39.10% 100.00% 100.00% 100.00% 100.00% 100.00%
single_fact 2000 MRR 38.46% 98.72% 98.52% 98.72% 98.62% 98.42%
single_fact 2000 多跳证据全中 39.10% 100.00% 100.00% 100.00% 100.00% 100.00%
single_fact 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
single_fact 2000 未知拒答率 - - - - - -
single_fact 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
single_fact 2000 未知问题被误读率 - - - - - -
unknown 1920 Top-1 正确率 13.67% 58.99% 58.10% 57.22% 65.21% 66.54%
unknown 1920 Recall@3 25.36% 68.96% 67.92% 69.40% 77.62% 76.63%
unknown 1920 MRR 26.97% 67.60% 66.30% 66.66% 74.11% 74.53%
unknown 1920 多跳证据全中 25.36% 68.96% 67.92% 69.40% 77.62% 76.63%
unknown 1920 hop 正确率 55.26% 100.00% 100.00% 100.00% 100.00% 100.00%
unknown 1920 未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00%
unknown 1920 已知问题被误拒率 1.76% 0.00% 0.00% 0.00% 0.00% 0.00%
unknown 1920 未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00% 0.00%
unknown_abstention 2000 Top-1 正确率 - - - - - -
unknown_abstention 2000 Recall@3 - - - - - -
unknown_abstention 2000 MRR - - - - - -
unknown_abstention 2000 多跳证据全中 - - - - - -
unknown_abstention 2000 hop 正确率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00%
unknown_abstention 2000 未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00%
unknown_abstention 2000 已知问题被误拒率 - - - - - -
unknown_abstention 2000 未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00% 0.00%
Reference in New Issue View Git Blame Copy Permalink