wpy/natural-memory-nm21
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
natural-memory-nm21/router_scorecard_v6.md
T
WpyQwq 643e22ecb9 Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据 
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线
- 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同）
- 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00）
- 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00%
- 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益）
- 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
2026-09-19 11:11:31 +08:00

19 KiB
Raw Permalink Blame History

metric V2-128 deployed(v3) V2-512 v3 best V2-512 v3 final V2-512 v6 best V2-512 v6 final XL-512 v6 best XL-512 v6 final
参数 2,037,774 4,741,902 4,741,902 4,741,902 4,741,902 7,898,127 7,898,127
router_dim 128 512 512 512 512 512 512
每条记录地址字节 512 2048 2048 2048 2048 2048 2048
Top-1 正确率 41.12% 80.75% 78.00% 95.05% 94.18% 95.78% 95.02%
Recall@1 37.03% 76.98% 73.56% 89.71% 88.65% 90.24% 89.48%
Recall@3 46.73% 95.02% 92.81% 98.84% 96.66% 97.59% 96.53%
Recall@5 48.91% 97.31% 95.30% 99.35% 97.64% 98.34% 97.61%
MRR 47.69% 88.46% 86.13% 97.12% 95.87% 97.24% 96.64%
nDCG@3 44.22% 89.53% 86.92% 97.24% 95.48% 96.64% 95.60%
多跳证据全中(Top-3) 43.43% 94.19% 91.98% 98.64% 96.44% 96.86% 95.35%
hop 正确率 73.23% 81.59% 81.75% 100.00% 100.00% 100.00% 100.00%
hop 欠预测率 4.68% 11.23% 11.23% 0.00% 0.00% 0.00% 0.00%
need F1 89.58% 88.68% 88.77% 99.98% 100.00% 100.00% 100.00%
need 召回 99.82% 88.77% 88.77% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 50.41% 51.29% 99.81% 100.00% 100.00% 100.00%
已知问题被误拒率 0.18% 11.23% 11.23% 0.00% 0.00% 0.00% 0.00%
未知问题被误读率 100.00% 49.59% 48.71% 0.19% 0.00% 0.00% 0.00%
仲裁准确率 81.12% 81.59% 81.75% 99.96% 100.00% 100.00% 100.00%
已知问题 need 概率均值 97.63% 88.77% 88.77% 99.88% 100.00% 100.00% 100.00%
未知问题 need 概率均值 99.89% 62.33% 45.28% 0.65% 0.00% 0.00% 0.00%
平均分数余量 -9.2989 9.6382 53.1605 12.3693 73.2482 45.5641 215.7533
单查询延迟 ms(GPU) 0.8439 0.8789 0.9095 0.8750 0.8725 1.2161 1.2505
路由 QPS(GPU, 单查询) 1184.9399 1137.7652 1099.5213 1142.8110 1146.1213 822.2900 799.6682
批量 QPS(GPU, batch=64) 72916.1919 69808.0279 67135.2145 73617.3737 61636.8434 57386.7509 53291.5883
批量 QPS(GPU, batch=256) 264812.9758 167235.0044 168449.8664 161941.2710 164167.8103 96746.8860 94345.8808
批量延迟 ms(GPU, batch=64) 0.0137 0.0143 0.0149 0.0136 0.0162 0.0174 0.0188

family = benchmark_qa

metric V2-128 deployed(v3) V2-512 v3 best V2-512 v3 final V2-512 v6 best V2-512 v6 final XL-512 v6 best XL-512 v6 final
episodes 64 64 64 64 64 64 64
Top-1 正确率 - - - - - - -
Recall@3 - - - - - - -
MRR - - - - - - -
hop 正确率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
已知问题被误拒率 - - - - - - -
未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
平均分数余量 - - - - - - -

family = mega_validation

metric V2-128 deployed(v3) V2-512 v3 best V2-512 v3 final V2-512 v6 best V2-512 v6 final XL-512 v6 best XL-512 v6 final
episodes 20000 20000 20000 20000 20000 20000 20000
Top-1 正确率 44.24% 84.44% 81.78% 97.00% 98.38% 98.38% 98.40%
Recall@3 49.15% 97.37% 95.43% 99.77% 99.75% 99.19% 98.68%
MRR 50.04% 91.19% 89.02% 98.49% 99.18% 99.18% 99.20%
hop 正确率 74.95% 80.00% 80.00% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 50.00% 50.00% 100.00% 100.00% 100.00% 100.00%
已知问题被误拒率 0.00% 12.50% 12.50% 0.00% 0.00% 0.00% 0.00%
未知问题被误读率 100.00% 50.00% 50.00% 0.00% 0.00% 0.00% 0.00%
平均分数余量 -8.0919 10.7052 59.2397 13.5425 81.8525 48.2819 235.8911

family = memory_policy

metric V2-128 deployed(v3) V2-512 v3 best V2-512 v3 final V2-512 v6 best V2-512 v6 final XL-512 v6 best XL-512 v6 final
episodes 1728 1728 1728 1728 1728 1728 1728
Top-1 正确率 14.53% 48.56% 44.64% 81.25% 57.70% 74.58% 66.90%
Recall@3 26.77% 73.76% 69.42% 92.44% 68.60% 83.48% 77.09%
MRR 28.02% 64.42% 60.50% 87.43% 66.64% 81.00% 74.88%
hop 正确率 57.41% 98.96% 100.00% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 14.29% 100.00% 80.95% 100.00% 100.00% 100.00%
已知问题被误拒率 1.87% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
未知问题被误读率 100.00% 85.71% 0.00% 19.05% 0.00% 0.00% 0.00%
平均分数余量 -19.3599 0.2432 -0.0168 2.2332 -2.6484 22.8851 40.8526

family = native_memory

metric V2-128 deployed(v3) V2-512 v3 best V2-512 v3 final V2-512 v6 best V2-512 v6 final XL-512 v6 best XL-512 v6 final
episodes 128 128 128 128 128 128 128
Top-1 正确率 0.00% 42.06% 44.86% 23.36% 49.53% 46.73% 38.32%
Recall@3 2.80% 82.24% 74.77% 61.68% 82.24% 83.18% 85.98%
MRR 10.23% 64.17% 62.77% 45.89% 67.05% 66.17% 61.74%
hop 正确率 53.91% 85.94% 100.00% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 14.29% 100.00% 80.95% 100.00% 100.00% 100.00%
已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
未知问题被误读率 100.00% 85.71% 0.00% 19.05% 0.00% 0.00% 0.00%
平均分数余量 -29.2917 -0.0322 -7.5310 -1.3701 -2.5814 0.9634 -5.2606

读取/拒答门槛曲线

run threshold need F1 need 召回 未知拒答率 已知问题被误拒率 未知被误读率 tp/tn/fp/fn
V2-128 deployed(v3) 0.30 89.67% 100.00% 0.00% 0.00% 100.00% 17814/0/4106/0
V2-128 deployed(v3) 0.40 89.67% 100.00% 0.00% 0.00% 100.00% 17814/0/4106/0
V2-128 deployed(v3) 0.50 89.58% 99.82% 0.00% 0.18% 100.00% 17782/0/4106/32
V2-128 deployed(v3) 0.60 89.30% 99.26% 0.00% 0.74% 100.00% 17682/0/4106/132
V2-128 deployed(v3) 0.70 89.19% 99.05% 0.00% 0.95% 100.00% 17645/0/4106/169
V2-128 deployed(v3) 0.80 89.01% 98.68% 0.00% 1.32% 100.00% 17579/0/4106/235
V2-512 v3 best 0.30 84.51% 89.00% 6.21% 11.00% 93.79% 15855/255/3851/1959
V2-512 v3 best 0.40 88.68% 88.78% 50.37% 11.22% 49.63% 15815/2068/2038/1999
V2-512 v3 best 0.50 88.68% 88.77% 50.41% 11.23% 49.59% 15814/2070/2036/2000
V2-512 v3 best 0.60 88.73% 88.77% 50.85% 11.23% 49.15% 15814/2088/2018/2000
V2-512 v3 best 0.70 88.77% 88.77% 51.24% 11.23% 48.76% 15814/2104/2002/2000
V2-512 v3 best 0.80 89.49% 88.77% 58.26% 11.23% 41.74% 15814/2392/1714/2000
V2-512 v3 final 0.30 88.77% 88.77% 51.29% 11.23% 48.71% 15814/2106/2000/2000
V2-512 v3 final 0.40 88.77% 88.77% 51.29% 11.23% 48.71% 15814/2106/2000/2000
V2-512 v3 final 0.50 88.77% 88.77% 51.29% 11.23% 48.71% 15814/2106/2000/2000
V2-512 v3 final 0.60 89.49% 88.77% 58.26% 11.23% 41.74% 15814/2392/1714/2000
V2-512 v3 final 0.70 89.49% 88.77% 58.26% 11.23% 41.74% 15814/2392/1714/2000
V2-512 v3 final 0.80 89.49% 88.77% 58.26% 11.23% 41.74% 15814/2392/1714/2000
V2-512 v6 best 0.30 99.97% 100.00% 99.71% 0.00% 0.29% 17814/4094/12/0
V2-512 v6 best 0.40 99.97% 100.00% 99.71% 0.00% 0.29% 17814/4094/12/0
V2-512 v6 best 0.50 99.98% 100.00% 99.81% 0.00% 0.19% 17814/4098/8/0
V2-512 v6 best 0.60 99.99% 100.00% 99.95% 0.00% 0.05% 17814/4104/2/0
V2-512 v6 best 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-512 v6 best 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-512 v6 final 0.30 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-512 v6 final 0.40 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-512 v6 final 0.50 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-512 v6 final 0.60 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-512 v6 final 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-512 v6 final 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 best 0.30 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 best 0.40 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 best 0.50 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 best 0.60 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 best 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 best 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 final 0.30 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 final 0.40 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 final 0.50 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 final 0.60 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 final 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 final 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0

按 mega 类别拆解（每类 2,000 条）

类别 episodes metric V2-128 deployed(v3) V2-512 v3 best V2-512 v3 final V2-512 v6 best V2-512 v6 final XL-512 v6 best XL-512 v6 final
(未分类) - Top-1 正确率 - - - - - - -
(未分类) - Recall@3 - - - - - - -
(未分类) - MRR - - - - - - -
(未分类) - 多跳证据全中 - - - - - - -
(未分类) - hop 正确率 - - - - - - -
(未分类) - 未知拒答率 - - - - - - -
(未分类) - 已知问题被误拒率 - - - - - - -
(未分类) - 未知问题被误读率 - - - - - - -
conflict_update 2000 Top-1 正确率 79.60% 86.05% 80.25% 99.05% 99.35% 99.20% 99.25%
conflict_update 2000 Recall@3 91.25% 99.95% 99.80% 100.00% 100.00% 100.00% 100.00%
conflict_update 2000 MRR 85.83% 92.82% 89.66% 99.52% 99.67% 99.60% 99.62%
conflict_update 2000 多跳证据全中 91.25% 99.95% 99.80% 100.00% 100.00% 100.00% 100.00%
conflict_update 2000 hop 正确率 49.55% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
conflict_update 2000 未知拒答率 - - - - - - -
conflict_update 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
conflict_update 2000 未知问题被误读率 - - - - - - -
distractor_128 2000 Top-1 正确率 23.10% 92.15% 89.80% 97.40% 98.00% 97.95% 98.00%
distractor_128 2000 Recall@3 25.35% 100.00% 99.95% 100.00% 100.00% 100.00% 100.00%
distractor_128 2000 MRR 26.60% 95.99% 94.74% 98.70% 99.00% 98.98% 99.00%
distractor_128 2000 多跳证据全中 25.35% 100.00% 99.95% 100.00% 100.00% 100.00% 100.00%
distractor_128 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
distractor_128 2000 未知拒答率 - - - - - - -
distractor_128 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
distractor_128 2000 未知问题被误读率 - - - - - - -
distractor_32 2000 Top-1 正确率 22.55% 92.15% 90.60% 96.90% 98.05% 98.05% 98.20%
distractor_32 2000 Recall@3 24.55% 100.00% 99.95% 100.00% 100.00% 100.00% 100.00%
distractor_32 2000 MRR 25.96% 95.97% 95.13% 98.44% 99.02% 99.02% 99.09%
distractor_32 2000 多跳证据全中 24.55% 100.00% 99.95% 100.00% 100.00% 100.00% 100.00%
distractor_32 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
distractor_32 2000 未知拒答率 - - - - - - -
distractor_32 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
distractor_32 2000 未知问题被误读率 - - - - - - -
forget_correction 2000 Top-1 正确率 - - - - - - -
forget_correction 2000 Recall@3 - - - - - - -
forget_correction 2000 MRR - - - - - - -
forget_correction 2000 多跳证据全中 - - - - - - -
forget_correction 2000 hop 正确率 0.00% 0.00% 0.00% 100.00% 100.00% 100.00% 100.00%
forget_correction 2000 未知拒答率 0.00% 0.00% 0.00% 100.00% 100.00% 100.00% 100.00%
forget_correction 2000 已知问题被误拒率 - - - - - - -
forget_correction 2000 未知问题被误读率 100.00% 100.00% 100.00% 0.00% 0.00% 0.00% 0.00%
long_context 2000 Top-1 正确率 52.55% 78.20% 69.65% 97.30% 97.70% 98.15% 98.30%
long_context 2000 Recall@3 62.45% 98.90% 84.65% 100.00% 100.00% 100.00% 100.00%
long_context 2000 MRR 58.55% 88.20% 77.64% 98.64% 98.84% 99.07% 99.14%
long_context 2000 多跳证据全中 62.45% 98.90% 84.65% 100.00% 100.00% 100.00% 100.00%
long_context 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
long_context 2000 未知拒答率 - - - - - - -
long_context 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
long_context 2000 未知问题被误读率 - - - - - - -
multi_hop 2000 Top-1 正确率 73.00% 67.10% 79.15% 95.10% 98.65% 98.75% 98.65%
multi_hop 2000 Recall@3 66.57% 91.20% 92.35% 98.17% 98.02% 93.53% 89.42%
multi_hop 2000 MRR 84.53% 82.14% 89.10% 97.53% 99.33% 99.37% 99.33%
multi_hop 2000 多跳证据全中 37.15% 83.85% 84.90% 96.35% 96.05% 87.05% 78.85%
multi_hop 2000 hop 正确率 100.00% 0.00% 0.00% 100.00% 100.00% 100.00% 100.00%
multi_hop 2000 未知拒答率 - - - - - - -
multi_hop 2000 已知问题被误拒率 0.00% 100.00% 100.00% 0.00% 0.00% 0.00% 0.00%
multi_hop 2000 未知问题被误读率 - - - - - - -
paraphrase 2000 Top-1 正确率 44.30% 77.25% 62.50% 97.55% 99.55% 99.65% 99.50%
paraphrase 2000 Recall@3 55.25% 88.95% 86.85% 100.00% 100.00% 100.00% 100.00%
paraphrase 2000 MRR 51.19% 83.29% 75.02% 98.78% 99.78% 99.83% 99.75%
paraphrase 2000 多跳证据全中 55.25% 88.95% 86.85% 100.00% 100.00% 100.00% 100.00%
paraphrase 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
paraphrase 2000 未知拒答率 - - - - - - -
paraphrase 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
paraphrase 2000 未知问题被误读率 - - - - - - -
random_position 2000 Top-1 正确率 24.90% 94.20% 92.00% 97.45% 98.25% 98.15% 98.05%
random_position 2000 Recall@3 28.70% 100.00% 99.90% 100.00% 100.00% 100.00% 100.00%
random_position 2000 MRR 29.20% 97.02% 95.84% 98.72% 99.12% 99.08% 99.02%
random_position 2000 多跳证据全中 28.70% 100.00% 99.90% 100.00% 100.00% 100.00% 100.00%
random_position 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
random_position 2000 未知拒答率 - - - - - - -
random_position 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
random_position 2000 未知问题被误读率 - - - - - - -
single_fact 2000 Top-1 正确率 33.90% 88.45% 90.30% 95.25% 97.45% 97.10% 97.25%
single_fact 2000 Recall@3 39.10% 99.95% 100.00% 100.00% 100.00% 100.00% 100.00%
single_fact 2000 MRR 38.46% 94.05% 95.00% 97.60% 98.72% 98.54% 98.62%
single_fact 2000 多跳证据全中 39.10% 99.95% 100.00% 100.00% 100.00% 100.00% 100.00%
single_fact 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
single_fact 2000 未知拒答率 - - - - - - -
single_fact 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
single_fact 2000 未知问题被误读率 - - - - - - -
unknown 1920 Top-1 正确率 13.67% 48.18% 44.65% 77.84% 57.22% 72.93% 65.21%
unknown 1920 Recall@3 25.36% 74.26% 69.74% 90.63% 69.40% 83.46% 77.62%
unknown 1920 MRR 26.97% 64.41% 60.64% 84.98% 66.66% 80.13% 74.11%
unknown 1920 多跳证据全中 25.36% 74.26% 69.74% 90.63% 69.40% 83.46% 77.62%
unknown 1920 hop 正确率 55.26% 98.12% 100.00% 100.00% 100.00% 100.00% 100.00%
unknown 1920 未知拒答率 0.00% 66.04% 100.00% 92.45% 100.00% 100.00% 100.00%
unknown 1920 已知问题被误拒率 1.76% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
unknown 1920 未知问题被误读率 100.00% 33.96% 0.00% 7.55% 0.00% 0.00% 0.00%
unknown_abstention 2000 Top-1 正确率 - - - - - - -
unknown_abstention 2000 Recall@3 - - - - - - -
unknown_abstention 2000 MRR - - - - - - -
unknown_abstention 2000 多跳证据全中 - - - - - - -
unknown_abstention 2000 hop 正确率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
unknown_abstention 2000 未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
unknown_abstention 2000 已知问题被误拒率 - - - - - - -
unknown_abstention 2000 未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
Reference in New Issue View Git Blame Copy Permalink