wpy/natural-memory-nm21
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
natural-memory-nm21/zero_overlap_baseline.md
T
WpyQwq 643e22ecb9 Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据 
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线
- 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同）
- 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00）
- 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00%
- 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益）
- 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
2026-09-19 11:11:31 +08:00

4.8 KiB
Raw Permalink Blame History

metric V2-128 v6 best V2-128 v6 final V2-128 deployed(v3)
参数 2,037,774 2,037,774 2,037,774
router_dim 128 128 128
每条记录地址字节 512 512 512
Top-1 正确率 7.60% 18.40% 11.60%
Recall@1 7.60% 18.40% 11.60%
Recall@3 23.60% 36.00% 32.00%
Recall@5 37.20% 60.40% 52.00%
MRR 22.88% 35.07% 29.48%
nDCG@3 16.65% 28.40% 23.48%
多跳证据全中(Top-3) 23.60% 36.00% 32.00%
hop 正确率 83.33% 83.33% 62.33%
hop 欠预测率 0.00% 0.00% 22.80%
need F1 90.91% 90.91% 83.30%
need 召回 100.00% 100.00% 84.80%
未知拒答率 0.00% 0.00% 6.00%
已知问题被误拒率 0.00% 0.00% 15.20%
未知问题被误读率 100.00% 100.00% 94.00%
仲裁准确率 83.33% 83.33% 71.67%
已知问题 need 概率均值 99.73% 99.18% 75.24%
未知问题 need 概率均值 99.53% 98.22% 82.54%
平均分数余量 -7.8896 -18.6882 -22.8761
单查询延迟 ms(GPU) 0.8667 0.8890 0.9203
路由 QPS(GPU, 单查询) 1153.8630 1124.9066 1086.5613
批量 QPS(GPU, batch=64) 61288.5926 75940.9559 71565.9526
批量 QPS(GPU, batch=256) 273229.8761 296056.4359 257898.1302
批量延迟 ms(GPU, batch=64) 0.0163 0.0132 0.0140

family = zero_overlap_paraphrase

metric V2-128 v6 best V2-128 v6 final V2-128 deployed(v3)
episodes 300 300 300
Top-1 正确率 7.60% 18.40% 11.60%
Recall@3 23.60% 36.00% 32.00%
MRR 22.88% 35.07% 29.48%
hop 正确率 83.33% 83.33% 62.33%
未知拒答率 0.00% 0.00% 6.00%
已知问题被误拒率 0.00% 0.00% 15.20%
未知问题被误读率 100.00% 100.00% 94.00%
平均分数余量 -7.8896 -18.6882 -22.8761

读取/拒答门槛曲线

run threshold need F1 need 召回 未知拒答率 已知问题被误拒率 未知被误读率 tp/tn/fp/fn
V2-128 v6 best 0.30 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
V2-128 v6 best 0.40 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
V2-128 v6 best 0.50 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
V2-128 v6 best 0.60 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
V2-128 v6 best 0.70 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
V2-128 v6 best 0.80 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
V2-128 v6 final 0.30 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
V2-128 v6 final 0.40 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
V2-128 v6 final 0.50 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
V2-128 v6 final 0.60 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
V2-128 v6 final 0.70 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
V2-128 v6 final 0.80 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
V2-128 deployed(v3) 0.30 83.30% 84.80% 6.00% 15.20% 94.00% 212/3/47/38
V2-128 deployed(v3) 0.40 83.30% 84.80% 6.00% 15.20% 94.00% 212/3/47/38
V2-128 deployed(v3) 0.50 83.30% 84.80% 6.00% 15.20% 94.00% 212/3/47/38
V2-128 deployed(v3) 0.60 83.30% 84.80% 6.00% 15.20% 94.00% 212/3/47/38
V2-128 deployed(v3) 0.70 77.43% 74.80% 8.00% 25.20% 92.00% 187/4/46/63
V2-128 deployed(v3) 0.80 66.67% 56.40% 36.00% 43.60% 64.00% 141/18/32/109

按 mega 类别拆解（每类 2,000 条）

类别 episodes metric V2-128 v6 best V2-128 v6 final V2-128 deployed(v3)
(未分类) - Top-1 正确率 - - -
(未分类) - Recall@3 - - -
(未分类) - MRR - - -
(未分类) - 多跳证据全中 - - -
(未分类) - hop 正确率 - - -
(未分类) - 未知拒答率 - - -
(未分类) - 已知问题被误拒率 - - -
(未分类) - 未知问题被误读率 - - -
zero_overlap_paraphrase 300 Top-1 正确率 7.60% 18.40% 11.60%
zero_overlap_paraphrase 300 Recall@3 23.60% 36.00% 32.00%
zero_overlap_paraphrase 300 MRR 22.88% 35.07% 29.48%
zero_overlap_paraphrase 300 多跳证据全中 23.60% 36.00% 32.00%
zero_overlap_paraphrase 300 hop 正确率 83.33% 83.33% 62.33%
zero_overlap_paraphrase 300 未知拒答率 0.00% 0.00% 6.00%
zero_overlap_paraphrase 300 已知问题被误拒率 0.00% 0.00% 15.20%
zero_overlap_paraphrase 300 未知问题被误读率 100.00% 100.00% 94.00%
Reference in New Issue View Git Blame Copy Permalink