wpy/natural-memory-nm21
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
natural-memory-nm21/router_scorecard_v6_128.md
T
WpyQwq 643e22ecb9 Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据 
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线
- 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同）
- 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00）
- 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00%
- 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益）
- 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
2026-09-19 11:11:31 +08:00

19 KiB
Raw Permalink Blame History

metric V2-128 deployed(v3) V2-128 v6 best V2-128 v6 final XL-128 v6 best XL-128 v6 final XL-512 v6 best V2-512 v6 best
参数 2,037,774 2,037,774 2,037,774 6,714,639 6,714,639 7,898,127 4,741,902
router_dim 128 128 128 128 128 512 512
每条记录地址字节 512 512 512 512 512 2048 2048
Top-1 正确率 41.12% 94.62% 94.37% 95.66% 95.19% 95.78% 95.05%
Recall@1 37.03% 89.19% 88.82% 90.28% 89.64% 90.24% 89.71%
Recall@3 46.73% 97.81% 96.40% 98.75% 97.45% 97.59% 98.84%
Recall@5 48.91% 98.40% 97.40% 99.23% 98.20% 98.34% 99.35%
MRR 47.69% 96.49% 95.97% 97.37% 96.70% 97.24% 97.12%
nDCG@3 44.22% 96.49% 95.31% 97.51% 96.43% 96.64% 97.24%
多跳证据全中(Top-3) 43.43% 97.75% 95.95% 98.70% 97.29% 96.86% 98.64%
hop 正确率 73.23% 100.00% 100.00% 99.95% 100.00% 100.00% 100.00%
hop 欠预测率 4.68% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
need F1 89.58% 100.00% 100.00% 99.95% 100.00% 100.00% 99.98%
need 召回 99.82% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 100.00% 100.00% 99.59% 100.00% 100.00% 99.81%
已知问题被误拒率 0.18% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
未知问题被误读率 100.00% 0.00% 0.00% 0.41% 0.00% 0.00% 0.19%
仲裁准确率 81.12% 100.00% 100.00% 99.92% 100.00% 100.00% 99.96%
已知问题 need 概率均值 97.63% 100.00% 100.00% 99.01% 100.00% 100.00% 99.88%
未知问题 need 概率均值 99.89% 0.12% 0.00% 3.62% 0.00% 0.00% 0.65%
平均分数余量 -9.2989 14.9592 51.3153 16.7980 200.3346 45.5641 12.3693
单查询延迟 ms(GPU) 1.1558 1.1679 1.1653 1.5783 1.5959 1.5855 1.1623
路由 QPS(GPU, 单查询) 865.1642 856.2377 858.1113 633.5730 626.6057 630.7159 860.4001
批量 QPS(GPU, batch=64) 54038.5362 54149.1810 53471.4680 38630.5471 36843.4384 39909.7043 53757.1186
批量 QPS(GPU, batch=256) 199554.1212 196829.2045 198025.9290 141500.5693 134976.9590 94647.2541 154085.0598
批量延迟 ms(GPU, batch=64) 0.0185 0.0185 0.0187 0.0259 0.0271 0.0251 0.0186

family = benchmark_qa

metric V2-128 deployed(v3) V2-128 v6 best V2-128 v6 final XL-128 v6 best XL-128 v6 final XL-512 v6 best V2-512 v6 best
episodes 64 64 64 64 64 64 64
Top-1 正确率 - - - - - - -
Recall@3 - - - - - - -
MRR - - - - - - -
hop 正确率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 100.00% 100.00% 95.31% 100.00% 100.00% 100.00%
已知问题被误拒率 - - - - - - -
未知问题被误读率 100.00% 0.00% 0.00% 4.69% 0.00% 0.00% 0.00%
平均分数余量 - - - - - - -

family = mega_validation

metric V2-128 deployed(v3) V2-128 v6 best V2-128 v6 final XL-128 v6 best XL-128 v6 final XL-512 v6 best V2-512 v6 best
episodes 20000 20000 20000 20000 20000 20000 20000
Top-1 正确率 44.24% 97.69% 98.38% 97.66% 98.44% 98.38% 97.00%
Recall@3 49.15% 99.94% 99.51% 99.94% 99.82% 99.19% 99.77%
MRR 50.04% 98.84% 99.19% 98.82% 99.21% 99.18% 98.49%
hop 正确率 74.95% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
平均分数余量 -8.0919 16.4023 57.1101 17.9133 217.4091 48.2819 13.5425

family = memory_policy

metric V2-128 deployed(v3) V2-128 v6 best V2-128 v6 final XL-128 v6 best XL-128 v6 final XL-512 v6 best V2-512 v6 best
episodes 1728 1728 1728 1728 1728 1728 1728
Top-1 正确率 14.53% 68.54% 59.40% 80.61% 67.55% 74.58% 81.25%
Recall@3 26.77% 78.62% 68.42% 88.40% 75.92% 83.48% 92.44%
MRR 28.02% 76.18% 67.60% 86.01% 74.81% 81.00% 87.43%
hop 正确率 57.41% 100.00% 100.00% 99.71% 100.00% 100.00% 100.00%
未知拒答率 0.00% 100.00% 100.00% 66.67% 100.00% 100.00% 80.95%
已知问题被误拒率 1.87% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
未知问题被误读率 100.00% 0.00% 0.00% 33.33% 0.00% 0.00% 19.05%
平均分数余量 -19.3599 2.3780 0.2715 7.3970 52.4323 22.8851 2.2332

family = native_memory

metric V2-128 deployed(v3) V2-128 v6 best V2-128 v6 final XL-128 v6 best XL-128 v6 final XL-512 v6 best V2-512 v6 best
episodes 128 128 128 128 128 128 128
Top-1 正确率 0.00% 51.40% 52.34% 37.38% 50.47% 46.73% 23.36%
Recall@3 2.80% 85.05% 77.57% 85.05% 87.85% 83.18% 61.68%
MRR 10.23% 69.28% 67.49% 61.43% 70.02% 66.17% 45.89%
hop 正确率 53.91% 100.00% 100.00% 96.09% 100.00% 100.00% 100.00%
未知拒答率 0.00% 100.00% 100.00% 66.67% 100.00% 100.00% 80.95%
已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
未知问题被误读率 100.00% 0.00% 0.00% 33.33% 0.00% 0.00% 19.05%
平均分数余量 -29.2917 -0.1162 -0.8787 0.0072 6.6553 0.9634 -1.3701

读取/拒答门槛曲线

run threshold need F1 need 召回 未知拒答率 已知问题被误拒率 未知被误读率 tp/tn/fp/fn
V2-128 deployed(v3) 0.30 89.67% 100.00% 0.00% 0.00% 100.00% 17814/0/4106/0
V2-128 deployed(v3) 0.40 89.67% 100.00% 0.00% 0.00% 100.00% 17814/0/4106/0
V2-128 deployed(v3) 0.50 89.58% 99.82% 0.00% 0.18% 100.00% 17782/0/4106/32
V2-128 deployed(v3) 0.60 89.30% 99.26% 0.00% 0.74% 100.00% 17682/0/4106/132
V2-128 deployed(v3) 0.70 89.19% 99.05% 0.00% 0.95% 100.00% 17645/0/4106/169
V2-128 deployed(v3) 0.80 89.01% 98.68% 0.00% 1.32% 100.00% 17579/0/4106/235
V2-128 v6 best 0.30 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 best 0.40 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 best 0.50 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 best 0.60 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 best 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 best 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 final 0.30 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 final 0.40 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 final 0.50 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 final 0.60 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 final 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 final 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-128 v6 best 0.30 99.82% 100.00% 98.44% 0.00% 1.56% 17814/4042/64/0
XL-128 v6 best 0.40 99.91% 100.00% 99.20% 0.00% 0.80% 17814/4073/33/0
XL-128 v6 best 0.50 99.95% 100.00% 99.59% 0.00% 0.41% 17814/4089/17/0
XL-128 v6 best 0.60 99.97% 100.00% 99.71% 0.00% 0.29% 17814/4094/12/0
XL-128 v6 best 0.70 99.98% 100.00% 99.85% 0.00% 0.15% 17814/4100/6/0
XL-128 v6 best 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-128 v6 final 0.30 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-128 v6 final 0.40 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-128 v6 final 0.50 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-128 v6 final 0.60 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-128 v6 final 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-128 v6 final 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 best 0.30 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 best 0.40 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 best 0.50 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 best 0.60 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 best 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
XL-512 v6 best 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-512 v6 best 0.30 99.97% 100.00% 99.71% 0.00% 0.29% 17814/4094/12/0
V2-512 v6 best 0.40 99.97% 100.00% 99.71% 0.00% 0.29% 17814/4094/12/0
V2-512 v6 best 0.50 99.98% 100.00% 99.81% 0.00% 0.19% 17814/4098/8/0
V2-512 v6 best 0.60 99.99% 100.00% 99.95% 0.00% 0.05% 17814/4104/2/0
V2-512 v6 best 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-512 v6 best 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0

按 mega 类别拆解（每类 2,000 条）

类别 episodes metric V2-128 deployed(v3) V2-128 v6 best V2-128 v6 final XL-128 v6 best XL-128 v6 final XL-512 v6 best V2-512 v6 best
(未分类) - Top-1 正确率 - - - - - - -
(未分类) - Recall@3 - - - - - - -
(未分类) - MRR - - - - - - -
(未分类) - 多跳证据全中 - - - - - - -
(未分类) - hop 正确率 - - - - - - -
(未分类) - 未知拒答率 - - - - - - -
(未分类) - 已知问题被误拒率 - - - - - - -
(未分类) - 未知问题被误读率 - - - - - - -
conflict_update 2000 Top-1 正确率 79.60% 98.85% 99.30% 97.85% 99.45% 99.20% 99.05%
conflict_update 2000 Recall@3 91.25% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
conflict_update 2000 MRR 85.83% 99.42% 99.65% 98.92% 99.72% 99.60% 99.52%
conflict_update 2000 多跳证据全中 91.25% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
conflict_update 2000 hop 正确率 49.55% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
conflict_update 2000 未知拒答率 - - - - - - -
conflict_update 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
conflict_update 2000 未知问题被误读率 - - - - - - -
distractor_128 2000 Top-1 正确率 23.10% 97.55% 97.50% 97.80% 97.85% 97.95% 97.40%
distractor_128 2000 Recall@3 25.35% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
distractor_128 2000 MRR 26.60% 98.78% 98.75% 98.90% 98.92% 98.98% 98.70%
distractor_128 2000 多跳证据全中 25.35% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
distractor_128 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
distractor_128 2000 未知拒答率 - - - - - - -
distractor_128 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
distractor_128 2000 未知问题被误读率 - - - - - - -
distractor_32 2000 Top-1 正确率 22.55% 98.05% 98.45% 98.50% 98.45% 98.05% 96.90%
distractor_32 2000 Recall@3 24.55% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
distractor_32 2000 MRR 25.96% 99.02% 99.22% 99.24% 99.22% 99.02% 98.44%
distractor_32 2000 多跳证据全中 24.55% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
distractor_32 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
distractor_32 2000 未知拒答率 - - - - - - -
distractor_32 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
distractor_32 2000 未知问题被误读率 - - - - - - -
forget_correction 2000 Top-1 正确率 - - - - - - -
forget_correction 2000 Recall@3 - - - - - - -
forget_correction 2000 MRR - - - - - - -
forget_correction 2000 多跳证据全中 - - - - - - -
forget_correction 2000 hop 正确率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
forget_correction 2000 未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
forget_correction 2000 已知问题被误拒率 - - - - - - -
forget_correction 2000 未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
long_context 2000 Top-1 正确率 52.55% 98.00% 97.95% 97.65% 98.15% 98.15% 97.30%
long_context 2000 Recall@3 62.45% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
long_context 2000 MRR 58.55% 98.99% 98.97% 98.83% 99.07% 99.07% 98.64%
long_context 2000 多跳证据全中 62.45% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
long_context 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
long_context 2000 未知拒答率 - - - - - - -
long_context 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
long_context 2000 未知问题被误读率 - - - - - - -
multi_hop 2000 Top-1 正确率 73.00% 96.65% 98.85% 95.90% 98.90% 98.75% 95.10%
multi_hop 2000 Recall@3 66.57% 99.50% 96.05% 99.55% 98.52% 93.53% 98.17%
multi_hop 2000 MRR 84.53% 98.30% 99.42% 97.93% 99.45% 99.37% 97.53%
multi_hop 2000 多跳证据全中 37.15% 99.00% 92.10% 99.10% 97.05% 87.05% 96.35%
multi_hop 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
multi_hop 2000 未知拒答率 - - - - - - -
multi_hop 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
multi_hop 2000 未知问题被误读率 - - - - - - -
paraphrase 2000 Top-1 正确率 44.30% 98.55% 99.50% 98.70% 99.70% 99.65% 97.55%
paraphrase 2000 Recall@3 55.25% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
paraphrase 2000 MRR 51.19% 99.28% 99.75% 99.35% 99.85% 99.83% 98.78%
paraphrase 2000 多跳证据全中 55.25% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
paraphrase 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
paraphrase 2000 未知拒答率 - - - - - - -
paraphrase 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
paraphrase 2000 未知问题被误读率 - - - - - - -
random_position 2000 Top-1 正确率 24.90% 97.80% 98.05% 97.70% 98.10% 98.15% 97.45%
random_position 2000 Recall@3 28.70% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
random_position 2000 MRR 29.20% 98.90% 99.02% 98.85% 99.05% 99.08% 98.72%
random_position 2000 多跳证据全中 28.70% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
random_position 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
random_position 2000 未知拒答率 - - - - - - -
random_position 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
random_position 2000 未知问题被误读率 - - - - - - -
single_fact 2000 Top-1 正确率 33.90% 96.05% 97.45% 97.15% 96.90% 97.10% 95.25%
single_fact 2000 Recall@3 39.10% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
single_fact 2000 MRR 38.46% 98.00% 98.72% 98.56% 98.42% 98.54% 97.60%
single_fact 2000 多跳证据全中 39.10% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
single_fact 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
single_fact 2000 未知拒答率 - - - - - - -
single_fact 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
single_fact 2000 未知问题被误读率 - - - - - - -
unknown 1920 Top-1 正确率 13.67% 67.53% 58.99% 78.06% 66.54% 72.93% 77.84%
unknown 1920 Recall@3 25.36% 79.00% 68.96% 88.20% 76.63% 83.46% 90.63%
unknown 1920 MRR 26.97% 75.78% 67.60% 84.56% 74.53% 80.13% 84.98%
unknown 1920 多跳证据全中 25.36% 79.00% 68.96% 88.20% 76.63% 83.46% 90.63%
unknown 1920 hop 正确率 55.26% 100.00% 100.00% 99.48% 100.00% 100.00% 100.00%
unknown 1920 未知拒答率 0.00% 100.00% 100.00% 83.96% 100.00% 100.00% 92.45%
unknown 1920 已知问题被误拒率 1.76% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
unknown 1920 未知问题被误读率 100.00% 0.00% 0.00% 16.04% 0.00% 0.00% 7.55%
unknown_abstention 2000 Top-1 正确率 - - - - - - -
unknown_abstention 2000 Recall@3 - - - - - - -
unknown_abstention 2000 MRR - - - - - - -
unknown_abstention 2000 多跳证据全中 - - - - - - -
unknown_abstention 2000 hop 正确率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
unknown_abstention 2000 未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
unknown_abstention 2000 已知问题被误拒率 - - - - - - -
unknown_abstention 2000 未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
Reference in New Issue View Git Blame Copy Permalink