wpy/natural-memory-nm21
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据

Browse Source 
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线
- 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同）
- 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00）
- 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00%
- 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益）
- 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
This commit is contained in:
WpyQwq
2026-09-19 11:11:31 +08:00
commit 643e22ecb9
484 changed files with 306821 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
router_scorecard_v3eval.md
+106
View File
@@ -0,0 +1,106 @@
| metric | V2-512 best(step1000) | V2-512 final(step100k) | XL-512 best | XL-512 final(step100k) |
|---|---|---|---|---|
| 参数 | 4,741,902 | 4,741,902 | 7,898,127 | 7,898,127 |
| router_dim | 512 | 512 | 512 | 512 |
| 每条记录地址字节 | 2048 | 2048 | 2048 | 2048 |
| **Top-1 正确率** | 67.12% | 60.63% | 60.85% | 54.36% |
| Recall@1 | 67.12% | 60.63% | 60.85% | 54.36% |
| Recall@3 | 88.77% | 85.01% | 92.97% | 85.33% |
| Recall@5 | 94.93% | 90.13% | 97.33% | 91.93% |
| MRR | 79.08% | 73.26% | 75.98% | 69.50% |
| nDCG@3 | 80.00% | 74.57% | 79.28% | 71.71% |
| 多跳证据全中(Top-3) | 88.66% | 84.90% | 92.86% | 85.22% |
| hop 正确率 | 97.84% | 99.69% | 99.38% | 99.69% |
| hop 欠预测率 | 0.33% | 0.33% | 0.33% | 0.33% |
| need F1 | 98.92% | 99.89% | 99.62% | 99.89% |
| need 召回 | 99.78% | 99.78% | 99.78% | 99.78% |
| **未知拒答率** | 66.04% | 100.00% | 90.57% | 100.00% |
| **已知问题被误拒率** | 0.22% | 0.22% | 0.22% | 0.22% |
| 未知问题被误读率 | 33.96% | 0.00% | 9.43% | 0.00% |
| 仲裁准确率 | 97.94% | 99.79% | 99.28% | 99.79% |
| 已知问题 need 概率均值 | 98.47% | 99.78% | 99.62% | 99.78% |
| 未知问题 need 概率均值 | 25.36% | 1.52% | 13.50% | 1.24% |
| 平均分数余量 | 0.5880 | 3.8837 | 6.7097 | 25.3757 |
| 单查询延迟 ms(GPU) | 0.8951 | 0.9347 | 1.3102 | 1.2458 |
| 路由 QPS(GPU) | 1117.1437 | 1069.9032 | 763.2632 | 802.6816 |
### family = `benchmark_qa`
| metric | V2-512 best(step1000) | V2-512 final(step100k) | XL-512 best | XL-512 final(step100k) |
|---|---|---|---|---|
| episodes | 64 | 64 | 64 | 64 |
| Top-1 正确率 | 0.00% | 0.00% | 0.00% | 0.00% |
| Recall@3 | 0.00% | 0.00% | 0.00% | 0.00% |
| MRR | 0.00% | 0.00% | 0.00% | 0.00% |
| hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% |
| 未知拒答率 | 100.00% | 100.00% | 100.00% | 100.00% |
| 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% |
| 未知问题被误读率 | 0.00% | 0.00% | 0.00% | 0.00% |
| 平均分数余量 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
### family = `mega_validation`
| metric | V2-512 best(step1000) | V2-512 final(step100k) | XL-512 best | XL-512 final(step100k) |
|---|---|---|---|---|
| episodes | 20 | 20 | 20 | 20 |
| Top-1 正确率 | 40.00% | 35.00% | 40.00% | 50.00% |
| Recall@3 | 80.00% | 80.00% | 85.00% | 80.00% |
| MRR | 62.26% | 55.24% | 60.83% | 63.61% |
| hop 正确率 | 70.00% | 70.00% | 70.00% | 70.00% |
| 未知拒答率 | 0.00% | 0.00% | 0.00% | 0.00% |
| 已知问题被误拒率 | 20.00% | 20.00% | 20.00% | 20.00% |
| 未知问题被误读率 | 0.00% | 0.00% | 0.00% | 0.00% |
| 平均分数余量 | -0.2963 | -12.9083 | -0.6136 | 3.7291 |
### family = `memory_policy`
| metric | V2-512 best(step1000) | V2-512 final(step100k) | XL-512 best | XL-512 final(step100k) |
|---|---|---|---|---|
| episodes | 1728 | 1728 | 1728 | 1728 |
| Top-1 正确率 | 69.89% | 62.57% | 62.39% | 55.13% |
| Recall@3 | 90.04% | 85.94% | 94.20% | 85.76% |
| MRR | 81.00% | 74.59% | 77.16% | 70.03% |
| hop 正确率 | 98.96% | 100.00% | 99.83% | 100.00% |
| 未知拒答率 | 14.29% | 100.00% | 76.19% | 100.00% |
| 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% |
| 未知问题被误读率 | 85.71% | 0.00% | 23.81% | 0.00% |
| 平均分数余量 | 0.6516 | 5.3929 | 7.2586 | 27.0634 |
### family = `native_memory`
| metric | V2-512 best(step1000) | V2-512 final(step100k) | XL-512 best | XL-512 final(step100k) |
|---|---|---|---|---|
| episodes | 128 | 128 | 128 | 128 |
| Top-1 正确率 | 28.04% | 34.58% | 40.19% | 42.99% |
| Recall@3 | 70.09% | 71.03% | 74.77% | 79.44% |
| MRR | 51.73% | 55.38% | 59.96% | 62.29% |
| hop 正确率 | 85.94% | 100.00% | 97.66% | 100.00% |
| 未知拒答率 | 14.29% | 100.00% | 76.19% | 100.00% |
| 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% |
| 未知问题被误读率 | 85.71% | 0.00% | 23.81% | 0.00% |
| 平均分数余量 | -0.2618 | -17.0552 | -0.6793 | 2.4967 |
### 读取/拒答门槛曲线
| run | threshold | need F1 | need 召回 | 未知拒答率 | 已知问题被误拒率 | 未知被误读率 | tp/tn/fp/fn |
|---|---|---|---|---|---|---|---|
| V2-512 best(step1000) | 0.30 | 98.81% | 99.78% | 62.26% | 0.22% | 37.74% | 1830/66/40/4 |
| V2-512 best(step1000) | 0.40 | 98.87% | 99.78% | 64.15% | 0.22% | 35.85% | 1830/68/38/4 |
| V2-512 best(step1000) | 0.50 | 98.92% | 99.78% | 66.04% | 0.22% | 33.96% | 1830/70/36/4 |
| V2-512 best(step1000) | 0.60 | 99.40% | 99.78% | 83.02% | 0.22% | 16.98% | 1830/88/18/4 |
| V2-512 best(step1000) | 0.70 | 99.84% | 99.78% | 98.11% | 0.22% | 1.89% | 1830/104/2/4 |
| V2-512 final(step100k) | 0.30 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
| V2-512 final(step100k) | 0.40 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
| V2-512 final(step100k) | 0.50 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
| V2-512 final(step100k) | 0.60 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
| V2-512 final(step100k) | 0.70 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
| XL-512 best | 0.30 | 99.19% | 99.78% | 75.47% | 0.22% | 24.53% | 1830/80/26/4 |
| XL-512 best | 0.40 | 99.51% | 99.78% | 86.79% | 0.22% | 13.21% | 1830/92/14/4 |
| XL-512 best | 0.50 | 99.62% | 99.78% | 90.57% | 0.22% | 9.43% | 1830/96/10/4 |
| XL-512 best | 0.60 | 99.84% | 99.78% | 98.11% | 0.22% | 1.89% | 1830/104/2/4 |
| XL-512 best | 0.70 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
| XL-512 final(step100k) | 0.30 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
| XL-512 final(step100k) | 0.40 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
| XL-512 final(step100k) | 0.50 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
| XL-512 final(step100k) | 0.60 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
| XL-512 final(step100k) | 0.70 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |