WpyQwq
643e22ecb9
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线 - 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同） - 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00） - 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00% - 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益） - 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
102 lines
4.7 KiB
Markdown
102 lines
4.7 KiB
Markdown
| metric | V2-512 best(step1000) | V2-512 final(step100k) | XL-512 best |
|
|
|---|---|---|---|
|
|
| 参数 | 4,741,902 | 4,741,902 | 7,898,127 |
|
|
| router_dim | 512 | 512 | 512 |
|
|
| 每条记录地址字节 | 2048 | 2048 | 2048 |
|
|
| **Top-1 正确率** | 67.12% | 60.63% | 60.85% |
|
|
| Recall@1 | 67.12% | 60.63% | 60.85% |
|
|
| Recall@3 | 88.77% | 85.01% | 92.97% |
|
|
| Recall@5 | 94.93% | 90.13% | 97.33% |
|
|
| MRR | 79.08% | 73.26% | 75.98% |
|
|
| nDCG@3 | 80.00% | 74.57% | 79.28% |
|
|
| 多跳证据全中(Top-3) | 88.66% | 84.90% | 92.86% |
|
|
| hop 正确率 | 97.84% | 99.69% | 99.38% |
|
|
| hop 欠预测率 | 0.33% | 0.33% | 0.33% |
|
|
| need F1 | 98.92% | 99.89% | 99.62% |
|
|
| need 召回 | 99.78% | 99.78% | 99.78% |
|
|
| **未知拒答率** | 66.04% | 100.00% | 90.57% |
|
|
| **已知问题被误拒率** | 0.22% | 0.22% | 0.22% |
|
|
| 未知问题被误读率 | 33.96% | 0.00% | 9.43% |
|
|
| 仲裁准确率 | 97.94% | 99.79% | 99.28% |
|
|
| 已知问题 need 概率均值 | 98.47% | 99.78% | 99.62% |
|
|
| 未知问题 need 概率均值 | 25.36% | 1.52% | 13.50% |
|
|
| 平均分数余量 | 0.5880 | 3.8837 | 6.7097 |
|
|
| 单查询延迟 ms(GPU) | - | - | - |
|
|
| 路由 QPS(GPU) | - | - | - |
|
|
|
|
### family = `benchmark_qa`
|
|
|
|
| metric | V2-512 best(step1000) | V2-512 final(step100k) | XL-512 best |
|
|
|---|---|---|---|
|
|
| episodes | 64 | 64 | 64 |
|
|
| Top-1 正确率 | - | - | - |
|
|
| Recall@3 | - | - | - |
|
|
| MRR | - | - | - |
|
|
| hop 正确率 | 100.00% | 100.00% | 100.00% |
|
|
| 未知拒答率 | 100.00% | 100.00% | 100.00% |
|
|
| 已知问题被误拒率 | - | - | - |
|
|
| 未知问题被误读率 | 0.00% | 0.00% | 0.00% |
|
|
| 平均分数余量 | - | - | - |
|
|
|
|
### family = `mega_validation`
|
|
|
|
| metric | V2-512 best(step1000) | V2-512 final(step100k) | XL-512 best |
|
|
|---|---|---|---|
|
|
| episodes | 20 | 20 | 20 |
|
|
| Top-1 正确率 | 40.00% | 35.00% | 40.00% |
|
|
| Recall@3 | 80.00% | 80.00% | 85.00% |
|
|
| MRR | 62.26% | 55.24% | 60.83% |
|
|
| hop 正确率 | 70.00% | 70.00% | 70.00% |
|
|
| 未知拒答率 | - | - | - |
|
|
| 已知问题被误拒率 | 20.00% | 20.00% | 20.00% |
|
|
| 未知问题被误读率 | - | - | - |
|
|
| 平均分数余量 | -0.2963 | -12.9083 | -0.6136 |
|
|
|
|
### family = `memory_policy`
|
|
|
|
| metric | V2-512 best(step1000) | V2-512 final(step100k) | XL-512 best |
|
|
|---|---|---|---|
|
|
| episodes | 1728 | 1728 | 1728 |
|
|
| Top-1 正确率 | 69.89% | 62.57% | 62.39% |
|
|
| Recall@3 | 90.04% | 85.94% | 94.20% |
|
|
| MRR | 81.00% | 74.59% | 77.16% |
|
|
| hop 正确率 | 98.96% | 100.00% | 99.83% |
|
|
| 未知拒答率 | 14.29% | 100.00% | 76.19% |
|
|
| 已知问题被误拒率 | 0.00% | 0.00% | 0.00% |
|
|
| 未知问题被误读率 | 85.71% | 0.00% | 23.81% |
|
|
| 平均分数余量 | 0.6516 | 5.3929 | 7.2586 |
|
|
|
|
### family = `native_memory`
|
|
|
|
| metric | V2-512 best(step1000) | V2-512 final(step100k) | XL-512 best |
|
|
|---|---|---|---|
|
|
| episodes | 128 | 128 | 128 |
|
|
| Top-1 正确率 | 28.04% | 34.58% | 40.19% |
|
|
| Recall@3 | 70.09% | 71.03% | 74.77% |
|
|
| MRR | 51.73% | 55.38% | 59.96% |
|
|
| hop 正确率 | 85.94% | 100.00% | 97.66% |
|
|
| 未知拒答率 | 14.29% | 100.00% | 76.19% |
|
|
| 已知问题被误拒率 | 0.00% | 0.00% | 0.00% |
|
|
| 未知问题被误读率 | 85.71% | 0.00% | 23.81% |
|
|
| 平均分数余量 | -0.2618 | -17.0552 | -0.6793 |
|
|
|
|
### 读取/拒答门槛曲线
|
|
|
|
| run | threshold | need F1 | need 召回 | 未知拒答率 | 已知问题被误拒率 | 未知被误读率 | tp/tn/fp/fn |
|
|
|---|---|---|---|---|---|---|---|
|
|
| V2-512 best(step1000) | 0.30 | 98.81% | 99.78% | 62.26% | 0.22% | 37.74% | 1830/66/40/4 |
|
|
| V2-512 best(step1000) | 0.40 | 98.87% | 99.78% | 64.15% | 0.22% | 35.85% | 1830/68/38/4 |
|
|
| V2-512 best(step1000) | 0.50 | 98.92% | 99.78% | 66.04% | 0.22% | 33.96% | 1830/70/36/4 |
|
|
| V2-512 best(step1000) | 0.60 | 99.40% | 99.78% | 83.02% | 0.22% | 16.98% | 1830/88/18/4 |
|
|
| V2-512 best(step1000) | 0.70 | 99.84% | 99.78% | 98.11% | 0.22% | 1.89% | 1830/104/2/4 |
|
|
| V2-512 final(step100k) | 0.30 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
|
|
| V2-512 final(step100k) | 0.40 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
|
|
| V2-512 final(step100k) | 0.50 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
|
|
| V2-512 final(step100k) | 0.60 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
|
|
| V2-512 final(step100k) | 0.70 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
|
|
| XL-512 best | 0.30 | 99.19% | 99.78% | 75.47% | 0.22% | 24.53% | 1830/80/26/4 |
|
|
| XL-512 best | 0.40 | 99.51% | 99.78% | 86.79% | 0.22% | 13.21% | 1830/92/14/4 |
|
|
| XL-512 best | 0.50 | 99.62% | 99.78% | 90.57% | 0.22% | 9.43% | 1830/96/10/4 |
|
|
| XL-512 best | 0.60 | 99.84% | 99.78% | 98.11% | 0.22% | 1.89% | 1830/104/2/4 |
|
|
| XL-512 best | 0.70 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
|