wpy/natural-memory-nm21
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
natural-memory-nm21/zero_overlap_baseline.md
T
WpyQwq 643e22ecb9 Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据 
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线
- 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同）
- 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00）
- 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00%
- 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益）
- 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
2026-09-19 11:11:31 +08:00

87 lines
4.8 KiB
Markdown
Raw Permalink Blame History

| metric | V2-128 v6 best | V2-128 v6 final | V2-128 deployed(v3) |
|---|---|---|---|
| 参数 | 2,037,774 | 2,037,774 | 2,037,774 |
| router_dim | 128 | 128 | 128 |
| 每条记录地址字节 | 512 | 512 | 512 |
| **Top-1 正确率** | 7.60% | 18.40% | 11.60% |
| Recall@1 | 7.60% | 18.40% | 11.60% |
| Recall@3 | 23.60% | 36.00% | 32.00% |
| Recall@5 | 37.20% | 60.40% | 52.00% |
| MRR | 22.88% | 35.07% | 29.48% |
| nDCG@3 | 16.65% | 28.40% | 23.48% |
| 多跳证据全中(Top-3) | 23.60% | 36.00% | 32.00% |
| hop 正确率 | 83.33% | 83.33% | 62.33% |
| hop 欠预测率 | 0.00% | 0.00% | 22.80% |
| need F1 | 90.91% | 90.91% | 83.30% |
| need 召回 | 100.00% | 100.00% | 84.80% |
| **未知拒答率** | 0.00% | 0.00% | 6.00% |
| **已知问题被误拒率** | 0.00% | 0.00% | 15.20% |
| 未知问题被误读率 | 100.00% | 100.00% | 94.00% |
| 仲裁准确率 | 83.33% | 83.33% | 71.67% |
| 已知问题 need 概率均值 | 99.73% | 99.18% | 75.24% |
| 未知问题 need 概率均值 | 99.53% | 98.22% | 82.54% |
| 平均分数余量 | -7.8896 | -18.6882 | -22.8761 |
| 单查询延迟 ms(GPU) | 0.8667 | 0.8890 | 0.9203 |
| 路由 QPS(GPU, 单查询) | 1153.8630 | 1124.9066 | 1086.5613 |
| 批量 QPS(GPU, batch=64) | 61288.5926 | 75940.9559 | 71565.9526 |
| 批量 QPS(GPU, batch=256) | 273229.8761 | 296056.4359 | 257898.1302 |
| 批量延迟 ms(GPU, batch=64) | 0.0163 | 0.0132 | 0.0140 |
### family = `zero_overlap_paraphrase`
| metric | V2-128 v6 best | V2-128 v6 final | V2-128 deployed(v3) |
|---|---|---|---|
| episodes | 300 | 300 | 300 |
| Top-1 正确率 | 7.60% | 18.40% | 11.60% |
| Recall@3 | 23.60% | 36.00% | 32.00% |
| MRR | 22.88% | 35.07% | 29.48% |
| hop 正确率 | 83.33% | 83.33% | 62.33% |
| 未知拒答率 | 0.00% | 0.00% | 6.00% |
| 已知问题被误拒率 | 0.00% | 0.00% | 15.20% |
| 未知问题被误读率 | 100.00% | 100.00% | 94.00% |
| 平均分数余量 | -7.8896 | -18.6882 | -22.8761 |
### 读取/拒答门槛曲线
| run | threshold | need F1 | need 召回 | 未知拒答率 | 已知问题被误拒率 | 未知被误读率 | tp/tn/fp/fn |
|---|---|---|---|---|---|---|---|
| V2-128 v6 best | 0.30 | 90.91% | 100.00% | 0.00% | 0.00% | 100.00% | 250/0/50/0 |
| V2-128 v6 best | 0.40 | 90.91% | 100.00% | 0.00% | 0.00% | 100.00% | 250/0/50/0 |
| V2-128 v6 best | 0.50 | 90.91% | 100.00% | 0.00% | 0.00% | 100.00% | 250/0/50/0 |
| V2-128 v6 best | 0.60 | 90.91% | 100.00% | 0.00% | 0.00% | 100.00% | 250/0/50/0 |
| V2-128 v6 best | 0.70 | 90.91% | 100.00% | 0.00% | 0.00% | 100.00% | 250/0/50/0 |
| V2-128 v6 best | 0.80 | 90.91% | 100.00% | 0.00% | 0.00% | 100.00% | 250/0/50/0 |
| V2-128 v6 final | 0.30 | 90.91% | 100.00% | 0.00% | 0.00% | 100.00% | 250/0/50/0 |
| V2-128 v6 final | 0.40 | 90.91% | 100.00% | 0.00% | 0.00% | 100.00% | 250/0/50/0 |
| V2-128 v6 final | 0.50 | 90.91% | 100.00% | 0.00% | 0.00% | 100.00% | 250/0/50/0 |
| V2-128 v6 final | 0.60 | 90.91% | 100.00% | 0.00% | 0.00% | 100.00% | 250/0/50/0 |
| V2-128 v6 final | 0.70 | 90.91% | 100.00% | 0.00% | 0.00% | 100.00% | 250/0/50/0 |
| V2-128 v6 final | 0.80 | 90.91% | 100.00% | 0.00% | 0.00% | 100.00% | 250/0/50/0 |
| V2-128 deployed(v3) | 0.30 | 83.30% | 84.80% | 6.00% | 15.20% | 94.00% | 212/3/47/38 |
| V2-128 deployed(v3) | 0.40 | 83.30% | 84.80% | 6.00% | 15.20% | 94.00% | 212/3/47/38 |
| V2-128 deployed(v3) | 0.50 | 83.30% | 84.80% | 6.00% | 15.20% | 94.00% | 212/3/47/38 |
| V2-128 deployed(v3) | 0.60 | 83.30% | 84.80% | 6.00% | 15.20% | 94.00% | 212/3/47/38 |
| V2-128 deployed(v3) | 0.70 | 77.43% | 74.80% | 8.00% | 25.20% | 92.00% | 187/4/46/63 |
| V2-128 deployed(v3) | 0.80 | 66.67% | 56.40% | 36.00% | 43.60% | 64.00% | 141/18/32/109 |
### 按 mega 类别拆解（每类 2,000 条）
| 类别 | episodes | metric | V2-128 v6 best | V2-128 v6 final | V2-128 deployed(v3) |
|---|---|---|---|---|---|
| (未分类) | - | Top-1 正确率 | - | - | - |
| (未分类) | - | Recall@3 | - | - | - |
| (未分类) | - | MRR | - | - | - |
| (未分类) | - | 多跳证据全中 | - | - | - |
| (未分类) | - | hop 正确率 | - | - | - |
| (未分类) | - | 未知拒答率 | - | - | - |
| (未分类) | - | 已知问题被误拒率 | - | - | - |
| (未分类) | - | 未知问题被误读率 | - | - | - |
| zero_overlap_paraphrase | 300 | Top-1 正确率 | 7.60% | 18.40% | 11.60% |
| zero_overlap_paraphrase | 300 | Recall@3 | 23.60% | 36.00% | 32.00% |
| zero_overlap_paraphrase | 300 | MRR | 22.88% | 35.07% | 29.48% |
| zero_overlap_paraphrase | 300 | 多跳证据全中 | 23.60% | 36.00% | 32.00% |
| zero_overlap_paraphrase | 300 | hop 正确率 | 83.33% | 83.33% | 62.33% |
| zero_overlap_paraphrase | 300 | 未知拒答率 | 0.00% | 0.00% | 6.00% |
| zero_overlap_paraphrase | 300 | 已知问题被误拒率 | 0.00% | 0.00% | 15.20% |
| zero_overlap_paraphrase | 300 | 未知问题被误读率 | 100.00% | 100.00% | 94.00% |
Reference in New Issue View Git Blame Copy Permalink