wpy/natural-memory-nm21
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
natural-memory-nm21/replay_check_zov.md
T
WpyQwq 643e22ecb9 Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据 
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线
- 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同）
- 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00）
- 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00%
- 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益）
- 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
2026-09-19 11:11:31 +08:00

4.8 KiB
Raw Permalink Blame History

metric V2-128 v6 final ZOV-128 final REPLAY-128 final
参数 2,037,774 2,037,774 2,037,774
router_dim 128 128 128
每条记录地址字节 512 512 512
Top-1 正确率 18.40% 66.80% 59.60%
Recall@1 18.40% 66.80% 59.60%
Recall@3 36.00% 84.40% 83.20%
Recall@5 60.40% 90.40% 90.80%
MRR 35.07% 76.95% 73.06%
nDCG@3 28.40% 76.96% 73.44%
多跳证据全中(Top-3) 36.00% 84.40% 83.20%
hop 正确率 83.33% 58.33% 83.33%
hop 欠预测率 0.00% 38.00% 0.00%
need F1 90.91% 71.26% 90.91%
need 召回 100.00% 62.00% 100.00%
未知拒答率 0.00% 40.00% 0.00%
已知问题被误拒率 0.00% 38.00% 0.00%
未知问题被误读率 100.00% 60.00% 100.00%
仲裁准确率 83.33% 58.33% 83.33%
已知问题 need 概率均值 99.18% 66.88% 99.18%
未知问题 need 概率均值 98.22% 64.00% 98.22%
平均分数余量 -18.6882 3.1264 1.5204
单查询延迟 ms(GPU) 0.9001 0.8862 0.8709
路由 QPS(GPU, 单查询) 1111.0124 1128.3880 1148.2401
批量 QPS(GPU, batch=64) 63459.3265 76974.8870 72823.2671
批量 QPS(GPU, batch=256) 250582.4084 291924.1911 234140.6307
批量延迟 ms(GPU, batch=64) 0.0158 0.0130 0.0137

family = zero_overlap_paraphrase

metric V2-128 v6 final ZOV-128 final REPLAY-128 final
episodes 300 300 300
Top-1 正确率 18.40% 66.80% 59.60%
Recall@3 36.00% 84.40% 83.20%
MRR 35.07% 76.95% 73.06%
hop 正确率 83.33% 58.33% 83.33%
未知拒答率 0.00% 40.00% 0.00%
已知问题被误拒率 0.00% 38.00% 0.00%
未知问题被误读率 100.00% 60.00% 100.00%
平均分数余量 -18.6882 3.1264 1.5204

读取/拒答门槛曲线

run threshold need F1 need 召回 未知拒答率 已知问题被误拒率 未知被误读率 tp/tn/fp/fn
V2-128 v6 final 0.30 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
V2-128 v6 final 0.40 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
V2-128 v6 final 0.50 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
V2-128 v6 final 0.60 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
V2-128 v6 final 0.70 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
V2-128 v6 final 0.80 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
ZOV-128 final 0.30 84.40% 84.40% 22.00% 15.60% 78.00% 211/11/39/39
ZOV-128 final 0.40 80.33% 76.80% 28.00% 23.20% 72.00% 192/14/36/58
ZOV-128 final 0.50 71.26% 62.00% 40.00% 38.00% 60.00% 155/20/30/95
ZOV-128 final 0.60 71.26% 62.00% 40.00% 38.00% 60.00% 155/20/30/95
ZOV-128 final 0.70 71.26% 62.00% 40.00% 38.00% 60.00% 155/20/30/95
ZOV-128 final 0.80 65.20% 53.20% 50.00% 46.80% 50.00% 133/25/25/117
REPLAY-128 final 0.30 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
REPLAY-128 final 0.40 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
REPLAY-128 final 0.50 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
REPLAY-128 final 0.60 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
REPLAY-128 final 0.70 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0
REPLAY-128 final 0.80 90.91% 100.00% 0.00% 0.00% 100.00% 250/0/50/0

按 mega 类别拆解（每类 2,000 条）

类别 episodes metric V2-128 v6 final ZOV-128 final REPLAY-128 final
(未分类) - Top-1 正确率 - - -
(未分类) - Recall@3 - - -
(未分类) - MRR - - -
(未分类) - 多跳证据全中 - - -
(未分类) - hop 正确率 - - -
(未分类) - 未知拒答率 - - -
(未分类) - 已知问题被误拒率 - - -
(未分类) - 未知问题被误读率 - - -
zero_overlap_paraphrase 300 Top-1 正确率 18.40% 66.80% 59.60%
zero_overlap_paraphrase 300 Recall@3 36.00% 84.40% 83.20%
zero_overlap_paraphrase 300 MRR 35.07% 76.95% 73.06%
zero_overlap_paraphrase 300 多跳证据全中 36.00% 84.40% 83.20%
zero_overlap_paraphrase 300 hop 正确率 83.33% 58.33% 83.33%
zero_overlap_paraphrase 300 未知拒答率 0.00% 40.00% 0.00%
zero_overlap_paraphrase 300 已知问题被误拒率 0.00% 38.00% 0.00%
zero_overlap_paraphrase 300 未知问题被误读率 100.00% 60.00% 100.00%
Reference in New Issue View Git Blame Copy Permalink