wpy/natural-memory-nm21
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
natural-memory-nm21/router_scorecard_prod.md
T
WpyQwq 643e22ecb9 Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据 
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线
- 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同）
- 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00）
- 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00%
- 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益）
- 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
2026-09-19 11:11:31 +08:00

15 KiB
Raw Permalink Blame History

metric V2-128 deployed(v3) REPLAY-128 v7 PROD-128 step20000 PROD-128 step10000 V2-128 v6 best
参数 2,037,774 2,037,774 2,037,774 2,037,774 2,037,774
router_dim 128 128 128 128 128
每条记录地址字节 512 512 512 512 512
Top-1 正确率 41.12% 94.14% 93.78% 93.81% 94.62%
Recall@1 37.03% 88.58% 88.34% 88.38% 89.19%
Recall@3 46.73% 96.48% 96.59% 96.78% 97.81%
Recall@5 48.91% 97.15% 97.33% 97.78% 98.40%
MRR 47.69% 95.77% 95.59% 95.70% 96.49%
nDCG@3 44.22% 95.37% 95.39% 95.53% 96.49%
多跳证据全中(Top-3) 43.43% 96.22% 96.50% 96.70% 97.75%
hop 正确率 73.23% 100.00% 100.00% 100.00% 100.00%
hop 欠预测率 4.68% 0.00% 0.00% 0.00% 0.00%
need F1 89.58% 100.00% 100.00% 100.00% 100.00%
need 召回 99.82% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00%
已知问题被误拒率 0.18% 0.00% 0.00% 0.00% 0.00%
未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00%
仲裁准确率 81.12% 100.00% 100.00% 100.00% 100.00%
已知问题 need 概率均值 97.63% 100.00% 100.00% 100.00% 100.00%
未知问题 need 概率均值 99.89% 0.00% 0.12% 0.12% 0.12%
平均分数余量 -9.2989 40.1456 21.8925 18.4219 14.9592
单查询延迟 ms(GPU) 0.9698 0.9359 0.9776 1.2315 1.2807
路由 QPS(GPU, 单查询) 1031.1135 1068.4453 1022.9307 812.0051 780.7949
批量 QPS(GPU, batch=64) 64355.2410 65927.7268 73335.6250 49115.9135 62429.2793
批量 QPS(GPU, batch=256) 185281.6861 262171.5170 273837.7940 141564.7326 201612.9032
批量延迟 ms(GPU, batch=64) 0.0155 0.0152 0.0136 0.0204 0.0160

family = benchmark_qa

metric V2-128 deployed(v3) REPLAY-128 v7 PROD-128 step20000 PROD-128 step10000 V2-128 v6 best
episodes 64 64 64 64 64
Top-1 正确率 - - - - -
Recall@3 - - - - -
MRR - - - - -
hop 正确率 0.00% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00%
已知问题被误拒率 - - - - -
未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00%
平均分数余量 - - - - -

family = mega_validation

metric V2-128 deployed(v3) REPLAY-128 v7 PROD-128 step20000 PROD-128 step10000 V2-128 v6 best
episodes 20000 20000 20000 20000 20000
Top-1 正确率 44.24% 98.22% 98.09% 97.98% 97.69%
Recall@3 49.15% 99.72% 99.90% 99.92% 99.94%
MRR 50.04% 99.11% 99.04% 98.99% 98.84%
hop 正确率 74.95% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00%
已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00%
未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00%
平均分数余量 -8.0919 44.7334 24.1982 20.2809 16.4023

family = memory_policy

metric V2-128 deployed(v3) REPLAY-128 v7 PROD-128 step20000 PROD-128 step10000 V2-128 v6 best
episodes 1728 1728 1728 1728 1728
Top-1 正确率 14.53% 58.76% 57.64% 59.23% 68.54%
Recall@3 26.77% 67.31% 67.96% 70.01% 78.62%
MRR 28.02% 66.33% 66.32% 68.25% 76.18%
hop 正确率 57.41% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00%
已知问题被误拒率 1.87% 0.00% 0.00% 0.00% 0.00%
未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00%
平均分数余量 -19.3599 -0.2722 1.8109 2.2951 2.3780

family = native_memory

metric V2-128 deployed(v3) REPLAY-128 v7 PROD-128 step20000 PROD-128 step10000 V2-128 v6 best
episodes 128 128 128 128 128
Top-1 正确率 0.00% 47.66% 25.23% 21.50% 51.40%
Recall@3 2.80% 77.57% 58.88% 53.27% 85.05%
MRR 10.23% 65.73% 46.57% 42.50% 69.28%
hop 正确率 53.91% 100.00% 100.00% 100.00% 100.00%
未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00%
已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00%
未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00%
平均分数余量 -29.2917 -1.0887 -2.5304 -2.2910 -0.1162

读取/拒答门槛曲线

run threshold need F1 need 召回 未知拒答率 已知问题被误拒率 未知被误读率 tp/tn/fp/fn
V2-128 deployed(v3) 0.30 89.67% 100.00% 0.00% 0.00% 100.00% 17814/0/4106/0
V2-128 deployed(v3) 0.40 89.67% 100.00% 0.00% 0.00% 100.00% 17814/0/4106/0
V2-128 deployed(v3) 0.50 89.58% 99.82% 0.00% 0.18% 100.00% 17782/0/4106/32
V2-128 deployed(v3) 0.60 89.30% 99.26% 0.00% 0.74% 100.00% 17682/0/4106/132
V2-128 deployed(v3) 0.70 89.19% 99.05% 0.00% 0.95% 100.00% 17645/0/4106/169
V2-128 deployed(v3) 0.80 89.01% 98.68% 0.00% 1.32% 100.00% 17579/0/4106/235
REPLAY-128 v7 0.30 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
REPLAY-128 v7 0.40 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
REPLAY-128 v7 0.50 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
REPLAY-128 v7 0.60 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
REPLAY-128 v7 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
REPLAY-128 v7 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
PROD-128 step20000 0.30 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
PROD-128 step20000 0.40 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
PROD-128 step20000 0.50 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
PROD-128 step20000 0.60 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
PROD-128 step20000 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
PROD-128 step20000 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
PROD-128 step10000 0.30 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
PROD-128 step10000 0.40 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
PROD-128 step10000 0.50 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
PROD-128 step10000 0.60 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
PROD-128 step10000 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
PROD-128 step10000 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 best 0.30 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 best 0.40 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 best 0.50 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 best 0.60 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 best 0.70 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0
V2-128 v6 best 0.80 100.00% 100.00% 100.00% 0.00% 0.00% 17814/4106/0/0

按 mega 类别拆解（每类 2,000 条）

类别 episodes metric V2-128 deployed(v3) REPLAY-128 v7 PROD-128 step20000 PROD-128 step10000 V2-128 v6 best
(未分类) - Top-1 正确率 - - - - -
(未分类) - Recall@3 - - - - -
(未分类) - MRR - - - - -
(未分类) - 多跳证据全中 - - - - -
(未分类) - hop 正确率 - - - - -
(未分类) - 未知拒答率 - - - - -
(未分类) - 已知问题被误拒率 - - - - -
(未分类) - 未知问题被误读率 - - - - -
conflict_update 2000 Top-1 正确率 79.60% 99.10% 99.30% 99.25% 98.85%
conflict_update 2000 Recall@3 91.25% 100.00% 100.00% 100.00% 100.00%
conflict_update 2000 MRR 85.83% 99.55% 99.65% 99.62% 99.42%
conflict_update 2000 多跳证据全中 91.25% 100.00% 100.00% 100.00% 100.00%
conflict_update 2000 hop 正确率 49.55% 100.00% 100.00% 100.00% 100.00%
conflict_update 2000 未知拒答率 - - - - -
conflict_update 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00%
conflict_update 2000 未知问题被误读率 - - - - -
distractor_128 2000 Top-1 正确率 23.10% 97.30% 97.55% 97.40% 97.55%
distractor_128 2000 Recall@3 25.35% 100.00% 100.00% 100.00% 100.00%
distractor_128 2000 MRR 26.60% 98.65% 98.78% 98.70% 98.78%
distractor_128 2000 多跳证据全中 25.35% 100.00% 100.00% 100.00% 100.00%
distractor_128 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00%
distractor_128 2000 未知拒答率 - - - - -
distractor_128 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00%
distractor_128 2000 未知问题被误读率 - - - - -
distractor_32 2000 Top-1 正确率 22.55% 98.30% 98.30% 98.15% 98.05%
distractor_32 2000 Recall@3 24.55% 100.00% 100.00% 100.00% 100.00%
distractor_32 2000 MRR 25.96% 99.15% 99.15% 99.08% 99.02%
distractor_32 2000 多跳证据全中 24.55% 100.00% 100.00% 100.00% 100.00%
distractor_32 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00%
distractor_32 2000 未知拒答率 - - - - -
distractor_32 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00%
distractor_32 2000 未知问题被误读率 - - - - -
forget_correction 2000 Top-1 正确率 - - - - -
forget_correction 2000 Recall@3 - - - - -
forget_correction 2000 MRR - - - - -
forget_correction 2000 多跳证据全中 - - - - -
forget_correction 2000 hop 正确率 0.00% 100.00% 100.00% 100.00% 100.00%
forget_correction 2000 未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00%
forget_correction 2000 已知问题被误拒率 - - - - -
forget_correction 2000 未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00%
long_context 2000 Top-1 正确率 52.55% 97.80% 98.45% 97.90% 98.00%
long_context 2000 Recall@3 62.45% 100.00% 100.00% 100.00% 100.00%
long_context 2000 MRR 58.55% 98.89% 99.22% 98.94% 98.99%
long_context 2000 多跳证据全中 62.45% 100.00% 100.00% 100.00% 100.00%
long_context 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00%
long_context 2000 未知拒答率 - - - - -
long_context 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00%
long_context 2000 未知问题被误读率 - - - - -
multi_hop 2000 Top-1 正确率 73.00% 99.05% 96.85% 96.75% 96.65%
multi_hop 2000 Recall@3 66.57% 97.72% 99.17% 99.38% 99.50%
multi_hop 2000 MRR 84.53% 99.52% 98.41% 98.36% 98.30%
multi_hop 2000 多跳证据全中 37.15% 95.45% 98.35% 98.75% 99.00%
multi_hop 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00%
multi_hop 2000 未知拒答率 - - - - -
multi_hop 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00%
multi_hop 2000 未知问题被误读率 - - - - -
paraphrase 2000 Top-1 正确率 44.30% 99.40% 99.55% 99.25% 98.55%
paraphrase 2000 Recall@3 55.25% 100.00% 100.00% 100.00% 100.00%
paraphrase 2000 MRR 51.19% 99.70% 99.78% 99.62% 99.28%
paraphrase 2000 多跳证据全中 55.25% 100.00% 100.00% 100.00% 100.00%
paraphrase 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00%
paraphrase 2000 未知拒答率 - - - - -
paraphrase 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00%
paraphrase 2000 未知问题被误读率 - - - - -
random_position 2000 Top-1 正确率 24.90% 97.80% 98.15% 97.90% 97.80%
random_position 2000 Recall@3 28.70% 100.00% 100.00% 100.00% 100.00%
random_position 2000 MRR 29.20% 98.89% 99.08% 98.95% 98.90%
random_position 2000 多跳证据全中 28.70% 100.00% 100.00% 100.00% 100.00%
random_position 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00%
random_position 2000 未知拒答率 - - - - -
random_position 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00%
random_position 2000 未知问题被误读率 - - - - -
single_fact 2000 Top-1 正确率 33.90% 97.05% 96.60% 97.25% 96.05%
single_fact 2000 Recall@3 39.10% 100.00% 100.00% 100.00% 100.00%
single_fact 2000 MRR 38.46% 98.52% 98.29% 98.62% 98.00%
single_fact 2000 多跳证据全中 39.10% 100.00% 100.00% 100.00% 100.00%
single_fact 2000 hop 正确率 100.00% 100.00% 100.00% 100.00% 100.00%
single_fact 2000 未知拒答率 - - - - -
single_fact 2000 已知问题被误拒率 0.00% 0.00% 0.00% 0.00% 0.00%
single_fact 2000 未知问题被误读率 - - - - -
unknown 1920 Top-1 正确率 13.67% 58.10% 55.73% 57.00% 67.53%
unknown 1920 Recall@3 25.36% 67.92% 67.42% 69.02% 79.00%
unknown 1920 MRR 26.97% 66.30% 65.15% 66.73% 75.78%
unknown 1920 多跳证据全中 25.36% 67.92% 67.42% 69.02% 79.00%
unknown 1920 hop 正确率 55.26% 100.00% 100.00% 100.00% 100.00%
unknown 1920 未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00%
unknown 1920 已知问题被误拒率 1.76% 0.00% 0.00% 0.00% 0.00%
unknown 1920 未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00%
unknown_abstention 2000 Top-1 正确率 - - - - -
unknown_abstention 2000 Recall@3 - - - - -
unknown_abstention 2000 MRR - - - - -
unknown_abstention 2000 多跳证据全中 - - - - -
unknown_abstention 2000 hop 正确率 0.00% 100.00% 100.00% 100.00% 100.00%
unknown_abstention 2000 未知拒答率 0.00% 100.00% 100.00% 100.00% 100.00%
unknown_abstention 2000 已知问题被误拒率 - - - - -
unknown_abstention 2000 未知问题被误读率 100.00% 0.00% 0.00% 0.00% 0.00%
Reference in New Issue View Git Blame Copy Permalink