WpyQwq
643e22ecb9
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线 - 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同） - 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00） - 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00% - 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益） - 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
203 lines
9.6 KiB
Markdown
203 lines
9.6 KiB
Markdown
| metric | V2-128 v6 final | ZOV-128 final |
|
|
|---|---|---|
|
|
| 参数 | 2,037,774 | 2,037,774 |
|
|
| router_dim | 128 | 128 |
|
|
| 每条记录地址字节 | 512 | 512 |
|
|
| **Top-1 正确率** | 94.37% | 90.72% |
|
|
| Recall@1 | 88.82% | 85.25% |
|
|
| Recall@3 | 96.40% | 93.69% |
|
|
| Recall@5 | 97.40% | 95.12% |
|
|
| MRR | 95.97% | 93.50% |
|
|
| nDCG@3 | 95.31% | 92.12% |
|
|
| 多跳证据全中(Top-3) | 95.95% | 91.96% |
|
|
| hop 正确率 | 100.00% | 91.59% |
|
|
| hop 欠预测率 | 0.00% | 10.35% |
|
|
| need F1 | 100.00% | 92.56% |
|
|
| need 召回 | 100.00% | 86.15% |
|
|
| **未知拒答率** | 100.00% | 100.00% |
|
|
| **已知问题被误拒率** | 0.00% | 13.85% |
|
|
| 未知问题被误读率 | 0.00% | 0.00% |
|
|
| 仲裁准确率 | 100.00% | 88.75% |
|
|
| 已知问题 need 概率均值 | 100.00% | 86.90% |
|
|
| 未知问题 need 概率均值 | 0.00% | 0.00% |
|
|
| 平均分数余量 | 51.3153 | 24.2131 |
|
|
| 单查询延迟 ms(GPU) | 0.9691 | 1.0321 |
|
|
| 路由 QPS(GPU, 单查询) | 1031.8867 | 968.8902 |
|
|
| 批量 QPS(GPU, batch=64) | 77868.3539 | 63339.9972 |
|
|
| 批量 QPS(GPU, batch=256) | 275832.3455 | 251226.6928 |
|
|
| 批量延迟 ms(GPU, batch=64) | 0.0128 | 0.0158 |
|
|
|
|
### family = `benchmark_qa`
|
|
|
|
| metric | V2-128 v6 final | ZOV-128 final |
|
|
|---|---|---|
|
|
| episodes | 64 | 64 |
|
|
| Top-1 正确率 | - | - |
|
|
| Recall@3 | - | - |
|
|
| MRR | - | - |
|
|
| hop 正确率 | 100.00% | 100.00% |
|
|
| 未知拒答率 | 100.00% | 100.00% |
|
|
| 已知问题被误拒率 | - | - |
|
|
| 未知问题被误读率 | 0.00% | 0.00% |
|
|
| 平均分数余量 | - | - |
|
|
|
|
### family = `mega_validation`
|
|
|
|
| metric | V2-128 v6 final | ZOV-128 final |
|
|
|---|---|---|
|
|
| episodes | 20000 | 20000 |
|
|
| Top-1 正确率 | 98.38% | 95.61% |
|
|
| Recall@3 | 99.51% | 97.52% |
|
|
| MRR | 99.19% | 97.57% |
|
|
| hop 正确率 | 100.00% | 92.09% |
|
|
| 未知拒答率 | 100.00% | 100.00% |
|
|
| 已知问题被误拒率 | 0.00% | 13.82% |
|
|
| 未知问题被误读率 | 0.00% | 0.00% |
|
|
| 平均分数余量 | 57.1101 | 27.2493 |
|
|
|
|
### family = `memory_policy`
|
|
|
|
| metric | V2-128 v6 final | ZOV-128 final |
|
|
|---|---|---|
|
|
| episodes | 1728 | 1728 |
|
|
| Top-1 正确率 | 59.40% | 49.68% |
|
|
| Recall@3 | 68.42% | 61.22% |
|
|
| MRR | 67.60% | 59.19% |
|
|
| hop 正确率 | 100.00% | 84.78% |
|
|
| 未知拒答率 | 100.00% | 100.00% |
|
|
| 已知问题被误拒率 | 0.00% | 15.00% |
|
|
| 未知问题被误读率 | 0.00% | 0.00% |
|
|
| 平均分数余量 | 0.2715 | -2.0129 |
|
|
|
|
### family = `native_memory`
|
|
|
|
| metric | V2-128 v6 final | ZOV-128 final |
|
|
|---|---|---|
|
|
| episodes | 128 | 128 |
|
|
| Top-1 正确率 | 52.34% | 13.08% |
|
|
| Recall@3 | 77.57% | 39.25% |
|
|
| MRR | 67.49% | 32.90% |
|
|
| hop 正确率 | 100.00% | 100.00% |
|
|
| 未知拒答率 | 100.00% | 100.00% |
|
|
| 已知问题被误拒率 | 0.00% | 0.00% |
|
|
| 未知问题被误读率 | 0.00% | 0.00% |
|
|
| 平均分数余量 | -0.8787 | -11.4113 |
|
|
|
|
### 读取/拒答门槛曲线
|
|
|
|
| run | threshold | need F1 | need 召回 | 未知拒答率 | 已知问题被误拒率 | 未知被误读率 | tp/tn/fp/fn |
|
|
|---|---|---|---|---|---|---|---|
|
|
| V2-128 v6 final | 0.30 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
|
| V2-128 v6 final | 0.40 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
|
| V2-128 v6 final | 0.50 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
|
| V2-128 v6 final | 0.60 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
|
| V2-128 v6 final | 0.70 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
|
| V2-128 v6 final | 0.80 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
|
| ZOV-128 final | 0.30 | 95.67% | 91.70% | 100.00% | 8.30% | 0.00% | 16335/4106/0/1479 |
|
|
| ZOV-128 final | 0.40 | 93.39% | 87.60% | 100.00% | 12.40% | 0.00% | 15605/4106/0/2209 |
|
|
| ZOV-128 final | 0.50 | 92.56% | 86.15% | 100.00% | 13.85% | 0.00% | 15347/4106/0/2467 |
|
|
| ZOV-128 final | 0.60 | 91.46% | 84.27% | 100.00% | 15.73% | 0.00% | 15012/4106/0/2802 |
|
|
| ZOV-128 final | 0.70 | 90.85% | 83.23% | 100.00% | 16.77% | 0.00% | 14826/4106/0/2988 |
|
|
| ZOV-128 final | 0.80 | 90.56% | 82.75% | 100.00% | 17.25% | 0.00% | 14741/4106/0/3073 |
|
|
|
|
### 按 mega 类别拆解（每类 2,000 条）
|
|
|
|
| 类别 | episodes | metric | V2-128 v6 final | ZOV-128 final |
|
|
|---|---|---|---|---|
|
|
| (未分类) | - | Top-1 正确率 | - | - |
|
|
| (未分类) | - | Recall@3 | - | - |
|
|
| (未分类) | - | MRR | - | - |
|
|
| (未分类) | - | 多跳证据全中 | - | - |
|
|
| (未分类) | - | hop 正确率 | - | - |
|
|
| (未分类) | - | 未知拒答率 | - | - |
|
|
| (未分类) | - | 已知问题被误拒率 | - | - |
|
|
| (未分类) | - | 未知问题被误读率 | - | - |
|
|
| conflict_update | 2000 | Top-1 正确率 | 99.30% | 96.80% |
|
|
| conflict_update | 2000 | Recall@3 | 100.00% | 100.00% |
|
|
| conflict_update | 2000 | MRR | 99.65% | 98.38% |
|
|
| conflict_update | 2000 | 多跳证据全中 | 100.00% | 100.00% |
|
|
| conflict_update | 2000 | hop 正确率 | 100.00% | 50.90% |
|
|
| conflict_update | 2000 | 未知拒答率 | - | - |
|
|
| conflict_update | 2000 | 已知问题被误拒率 | 0.00% | 76.10% |
|
|
| conflict_update | 2000 | 未知问题被误读率 | - | - |
|
|
| distractor_128 | 2000 | Top-1 正确率 | 97.50% | 94.80% |
|
|
| distractor_128 | 2000 | Recall@3 | 100.00% | 99.95% |
|
|
| distractor_128 | 2000 | MRR | 98.75% | 97.37% |
|
|
| distractor_128 | 2000 | 多跳证据全中 | 100.00% | 99.95% |
|
|
| distractor_128 | 2000 | hop 正确率 | 100.00% | 100.00% |
|
|
| distractor_128 | 2000 | 未知拒答率 | - | - |
|
|
| distractor_128 | 2000 | 已知问题被误拒率 | 0.00% | 0.00% |
|
|
| distractor_128 | 2000 | 未知问题被误读率 | - | - |
|
|
| distractor_32 | 2000 | Top-1 正确率 | 98.45% | 95.75% |
|
|
| distractor_32 | 2000 | Recall@3 | 100.00% | 100.00% |
|
|
| distractor_32 | 2000 | MRR | 99.22% | 97.86% |
|
|
| distractor_32 | 2000 | 多跳证据全中 | 100.00% | 100.00% |
|
|
| distractor_32 | 2000 | hop 正确率 | 100.00% | 100.00% |
|
|
| distractor_32 | 2000 | 未知拒答率 | - | - |
|
|
| distractor_32 | 2000 | 已知问题被误拒率 | 0.00% | 0.00% |
|
|
| distractor_32 | 2000 | 未知问题被误读率 | - | - |
|
|
| forget_correction | 2000 | Top-1 正确率 | - | - |
|
|
| forget_correction | 2000 | Recall@3 | - | - |
|
|
| forget_correction | 2000 | MRR | - | - |
|
|
| forget_correction | 2000 | 多跳证据全中 | - | - |
|
|
| forget_correction | 2000 | hop 正确率 | 100.00% | 100.00% |
|
|
| forget_correction | 2000 | 未知拒答率 | 100.00% | 100.00% |
|
|
| forget_correction | 2000 | 已知问题被误拒率 | - | - |
|
|
| forget_correction | 2000 | 未知问题被误读率 | 0.00% | 0.00% |
|
|
| long_context | 2000 | Top-1 正确率 | 97.95% | 95.40% |
|
|
| long_context | 2000 | Recall@3 | 100.00% | 100.00% |
|
|
| long_context | 2000 | MRR | 98.97% | 97.68% |
|
|
| long_context | 2000 | 多跳证据全中 | 100.00% | 100.00% |
|
|
| long_context | 2000 | hop 正确率 | 100.00% | 100.00% |
|
|
| long_context | 2000 | 未知拒答率 | - | - |
|
|
| long_context | 2000 | 已知问题被误拒率 | 0.00% | 0.00% |
|
|
| long_context | 2000 | 未知问题被误读率 | - | - |
|
|
| multi_hop | 2000 | Top-1 正确率 | 98.85% | 97.30% |
|
|
| multi_hop | 2000 | Recall@3 | 96.05% | 84.55% |
|
|
| multi_hop | 2000 | MRR | 99.42% | 98.64% |
|
|
| multi_hop | 2000 | 多跳证据全中 | 92.10% | 69.10% |
|
|
| multi_hop | 2000 | hop 正确率 | 100.00% | 100.00% |
|
|
| multi_hop | 2000 | 未知拒答率 | - | - |
|
|
| multi_hop | 2000 | 已知问题被误拒率 | 0.00% | 0.00% |
|
|
| multi_hop | 2000 | 未知问题被误读率 | - | - |
|
|
| paraphrase | 2000 | Top-1 正确率 | 99.50% | 98.20% |
|
|
| paraphrase | 2000 | Recall@3 | 100.00% | 100.00% |
|
|
| paraphrase | 2000 | MRR | 99.75% | 99.09% |
|
|
| paraphrase | 2000 | 多跳证据全中 | 100.00% | 100.00% |
|
|
| paraphrase | 2000 | hop 正确率 | 100.00% | 82.95% |
|
|
| paraphrase | 2000 | 未知拒答率 | - | - |
|
|
| paraphrase | 2000 | 已知问题被误拒率 | 0.00% | 21.55% |
|
|
| paraphrase | 2000 | 未知问题被误读率 | - | - |
|
|
| random_position | 2000 | Top-1 正确率 | 98.05% | 90.35% |
|
|
| random_position | 2000 | Recall@3 | 100.00% | 95.65% |
|
|
| random_position | 2000 | MRR | 99.02% | 93.37% |
|
|
| random_position | 2000 | 多跳证据全中 | 100.00% | 95.65% |
|
|
| random_position | 2000 | hop 正确率 | 100.00% | 87.10% |
|
|
| random_position | 2000 | 未知拒答率 | - | - |
|
|
| random_position | 2000 | 已知问题被误拒率 | 0.00% | 12.90% |
|
|
| random_position | 2000 | 未知问题被误读率 | - | - |
|
|
| single_fact | 2000 | Top-1 正确率 | 97.45% | 96.30% |
|
|
| single_fact | 2000 | Recall@3 | 100.00% | 100.00% |
|
|
| single_fact | 2000 | MRR | 98.72% | 98.12% |
|
|
| single_fact | 2000 | 多跳证据全中 | 100.00% | 100.00% |
|
|
| single_fact | 2000 | hop 正确率 | 100.00% | 100.00% |
|
|
| single_fact | 2000 | 未知拒答率 | - | - |
|
|
| single_fact | 2000 | 已知问题被误拒率 | 0.00% | 0.00% |
|
|
| single_fact | 2000 | 未知问题被误读率 | - | - |
|
|
| unknown | 1920 | Top-1 正确率 | 58.99% | 47.52% |
|
|
| unknown | 1920 | Recall@3 | 68.96% | 59.92% |
|
|
| unknown | 1920 | MRR | 67.60% | 57.64% |
|
|
| unknown | 1920 | 多跳证据全中 | 68.96% | 59.92% |
|
|
| unknown | 1920 | hop 正确率 | 100.00% | 86.30% |
|
|
| unknown | 1920 | 未知拒答率 | 100.00% | 100.00% |
|
|
| unknown | 1920 | 已知问题被误拒率 | 0.00% | 14.11% |
|
|
| unknown | 1920 | 未知问题被误读率 | 0.00% | 0.00% |
|
|
| unknown_abstention | 2000 | Top-1 正确率 | - | - |
|
|
| unknown_abstention | 2000 | Recall@3 | - | - |
|
|
| unknown_abstention | 2000 | MRR | - | - |
|
|
| unknown_abstention | 2000 | 多跳证据全中 | - | - |
|
|
| unknown_abstention | 2000 | hop 正确率 | 100.00% | 100.00% |
|
|
| unknown_abstention | 2000 | 未知拒答率 | 100.00% | 100.00% |
|
|
| unknown_abstention | 2000 | 已知问题被误拒率 | - | - |
|
|
| unknown_abstention | 2000 | 未知问题被误读率 | 0.00% | 0.00% |
|