Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线 - 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同） - 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00） - 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00% - 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益） - 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
This commit is contained in:
WpyQwq
@@ -0,0 +1,232 @@
|
||||
| metric | V2-128 deployed(v3) | V2-128 v6 best | V2-128 v6 final | XL-128 v6 best | XL-128 v6 final | XL-512 v6 best | V2-512 v6 best |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 参数 | 2,037,774 | 2,037,774 | 2,037,774 | 6,714,639 | 6,714,639 | 7,898,127 | 4,741,902 |
|
||||
| router_dim | 128 | 128 | 128 | 128 | 128 | 512 | 512 |
|
||||
| 每条记录地址字节 | 512 | 512 | 512 | 512 | 512 | 2048 | 2048 |
|
||||
| **Top-1 正确率** | 41.12% | 94.62% | 94.37% | 95.66% | 95.19% | 95.78% | 95.05% |
|
||||
| Recall@1 | 37.03% | 89.19% | 88.82% | 90.28% | 89.64% | 90.24% | 89.71% |
|
||||
| Recall@3 | 46.73% | 97.81% | 96.40% | 98.75% | 97.45% | 97.59% | 98.84% |
|
||||
| Recall@5 | 48.91% | 98.40% | 97.40% | 99.23% | 98.20% | 98.34% | 99.35% |
|
||||
| MRR | 47.69% | 96.49% | 95.97% | 97.37% | 96.70% | 97.24% | 97.12% |
|
||||
| nDCG@3 | 44.22% | 96.49% | 95.31% | 97.51% | 96.43% | 96.64% | 97.24% |
|
||||
| 多跳证据全中(Top-3) | 43.43% | 97.75% | 95.95% | 98.70% | 97.29% | 96.86% | 98.64% |
|
||||
| hop 正确率 | 73.23% | 100.00% | 100.00% | 99.95% | 100.00% | 100.00% | 100.00% |
|
||||
| hop 欠预测率 | 4.68% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
|
||||
| need F1 | 89.58% | 100.00% | 100.00% | 99.95% | 100.00% | 100.00% | 99.98% |
|
||||
| need 召回 | 99.82% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| **未知拒答率** | 0.00% | 100.00% | 100.00% | 99.59% | 100.00% | 100.00% | 99.81% |
|
||||
| **已知问题被误拒率** | 0.18% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
|
||||
| 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 0.41% | 0.00% | 0.00% | 0.19% |
|
||||
| 仲裁准确率 | 81.12% | 100.00% | 100.00% | 99.92% | 100.00% | 100.00% | 99.96% |
|
||||
| 已知问题 need 概率均值 | 97.63% | 100.00% | 100.00% | 99.01% | 100.00% | 100.00% | 99.88% |
|
||||
| 未知问题 need 概率均值 | 99.89% | 0.12% | 0.00% | 3.62% | 0.00% | 0.00% | 0.65% |
|
||||
| 平均分数余量 | -9.2989 | 14.9592 | 51.3153 | 16.7980 | 200.3346 | 45.5641 | 12.3693 |
|
||||
| 单查询延迟 ms(GPU) | 1.1558 | 1.1679 | 1.1653 | 1.5783 | 1.5959 | 1.5855 | 1.1623 |
|
||||
| 路由 QPS(GPU, 单查询) | 865.1642 | 856.2377 | 858.1113 | 633.5730 | 626.6057 | 630.7159 | 860.4001 |
|
||||
| 批量 QPS(GPU, batch=64) | 54038.5362 | 54149.1810 | 53471.4680 | 38630.5471 | 36843.4384 | 39909.7043 | 53757.1186 |
|
||||
| 批量 QPS(GPU, batch=256) | 199554.1212 | 196829.2045 | 198025.9290 | 141500.5693 | 134976.9590 | 94647.2541 | 154085.0598 |
|
||||
| 批量延迟 ms(GPU, batch=64) | 0.0185 | 0.0185 | 0.0187 | 0.0259 | 0.0271 | 0.0251 | 0.0186 |
|
||||
|
||||
### family = `benchmark_qa`
|
||||
|
||||
| metric | V2-128 deployed(v3) | V2-128 v6 best | V2-128 v6 final | XL-128 v6 best | XL-128 v6 final | XL-512 v6 best | V2-512 v6 best |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| episodes | 64 | 64 | 64 | 64 | 64 | 64 | 64 |
|
||||
| Top-1 正确率 | - | - | - | - | - | - | - |
|
||||
| Recall@3 | - | - | - | - | - | - | - |
|
||||
| MRR | - | - | - | - | - | - | - |
|
||||
| hop 正确率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| 未知拒答率 | 0.00% | 100.00% | 100.00% | 95.31% | 100.00% | 100.00% | 100.00% |
|
||||
| 已知问题被误拒率 | - | - | - | - | - | - | - |
|
||||
| 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 4.69% | 0.00% | 0.00% | 0.00% |
|
||||
| 平均分数余量 | - | - | - | - | - | - | - |
|
||||
|
||||
### family = `mega_validation`
|
||||
|
||||
| metric | V2-128 deployed(v3) | V2-128 v6 best | V2-128 v6 final | XL-128 v6 best | XL-128 v6 final | XL-512 v6 best | V2-512 v6 best |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| episodes | 20000 | 20000 | 20000 | 20000 | 20000 | 20000 | 20000 |
|
||||
| Top-1 正确率 | 44.24% | 97.69% | 98.38% | 97.66% | 98.44% | 98.38% | 97.00% |
|
||||
| Recall@3 | 49.15% | 99.94% | 99.51% | 99.94% | 99.82% | 99.19% | 99.77% |
|
||||
| MRR | 50.04% | 98.84% | 99.19% | 98.82% | 99.21% | 99.18% | 98.49% |
|
||||
| hop 正确率 | 74.95% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| 未知拒答率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
|
||||
| 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
|
||||
| 平均分数余量 | -8.0919 | 16.4023 | 57.1101 | 17.9133 | 217.4091 | 48.2819 | 13.5425 |
|
||||
|
||||
### family = `memory_policy`
|
||||
|
||||
| metric | V2-128 deployed(v3) | V2-128 v6 best | V2-128 v6 final | XL-128 v6 best | XL-128 v6 final | XL-512 v6 best | V2-512 v6 best |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| episodes | 1728 | 1728 | 1728 | 1728 | 1728 | 1728 | 1728 |
|
||||
| Top-1 正确率 | 14.53% | 68.54% | 59.40% | 80.61% | 67.55% | 74.58% | 81.25% |
|
||||
| Recall@3 | 26.77% | 78.62% | 68.42% | 88.40% | 75.92% | 83.48% | 92.44% |
|
||||
| MRR | 28.02% | 76.18% | 67.60% | 86.01% | 74.81% | 81.00% | 87.43% |
|
||||
| hop 正确率 | 57.41% | 100.00% | 100.00% | 99.71% | 100.00% | 100.00% | 100.00% |
|
||||
| 未知拒答率 | 0.00% | 100.00% | 100.00% | 66.67% | 100.00% | 100.00% | 80.95% |
|
||||
| 已知问题被误拒率 | 1.87% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
|
||||
| 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 33.33% | 0.00% | 0.00% | 19.05% |
|
||||
| 平均分数余量 | -19.3599 | 2.3780 | 0.2715 | 7.3970 | 52.4323 | 22.8851 | 2.2332 |
|
||||
|
||||
### family = `native_memory`
|
||||
|
||||
| metric | V2-128 deployed(v3) | V2-128 v6 best | V2-128 v6 final | XL-128 v6 best | XL-128 v6 final | XL-512 v6 best | V2-512 v6 best |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| episodes | 128 | 128 | 128 | 128 | 128 | 128 | 128 |
|
||||
| Top-1 正确率 | 0.00% | 51.40% | 52.34% | 37.38% | 50.47% | 46.73% | 23.36% |
|
||||
| Recall@3 | 2.80% | 85.05% | 77.57% | 85.05% | 87.85% | 83.18% | 61.68% |
|
||||
| MRR | 10.23% | 69.28% | 67.49% | 61.43% | 70.02% | 66.17% | 45.89% |
|
||||
| hop 正确率 | 53.91% | 100.00% | 100.00% | 96.09% | 100.00% | 100.00% | 100.00% |
|
||||
| 未知拒答率 | 0.00% | 100.00% | 100.00% | 66.67% | 100.00% | 100.00% | 80.95% |
|
||||
| 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
|
||||
| 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 33.33% | 0.00% | 0.00% | 19.05% |
|
||||
| 平均分数余量 | -29.2917 | -0.1162 | -0.8787 | 0.0072 | 6.6553 | 0.9634 | -1.3701 |
|
||||
|
||||
### 读取/拒答门槛曲线
|
||||
|
||||
| run | threshold | need F1 | need 召回 | 未知拒答率 | 已知问题被误拒率 | 未知被误读率 | tp/tn/fp/fn |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| V2-128 deployed(v3) | 0.30 | 89.67% | 100.00% | 0.00% | 0.00% | 100.00% | 17814/0/4106/0 |
|
||||
| V2-128 deployed(v3) | 0.40 | 89.67% | 100.00% | 0.00% | 0.00% | 100.00% | 17814/0/4106/0 |
|
||||
| V2-128 deployed(v3) | 0.50 | 89.58% | 99.82% | 0.00% | 0.18% | 100.00% | 17782/0/4106/32 |
|
||||
| V2-128 deployed(v3) | 0.60 | 89.30% | 99.26% | 0.00% | 0.74% | 100.00% | 17682/0/4106/132 |
|
||||
| V2-128 deployed(v3) | 0.70 | 89.19% | 99.05% | 0.00% | 0.95% | 100.00% | 17645/0/4106/169 |
|
||||
| V2-128 deployed(v3) | 0.80 | 89.01% | 98.68% | 0.00% | 1.32% | 100.00% | 17579/0/4106/235 |
|
||||
| V2-128 v6 best | 0.30 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| V2-128 v6 best | 0.40 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| V2-128 v6 best | 0.50 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| V2-128 v6 best | 0.60 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| V2-128 v6 best | 0.70 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| V2-128 v6 best | 0.80 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| V2-128 v6 final | 0.30 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| V2-128 v6 final | 0.40 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| V2-128 v6 final | 0.50 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| V2-128 v6 final | 0.60 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| V2-128 v6 final | 0.70 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| V2-128 v6 final | 0.80 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| XL-128 v6 best | 0.30 | 99.82% | 100.00% | 98.44% | 0.00% | 1.56% | 17814/4042/64/0 |
|
||||
| XL-128 v6 best | 0.40 | 99.91% | 100.00% | 99.20% | 0.00% | 0.80% | 17814/4073/33/0 |
|
||||
| XL-128 v6 best | 0.50 | 99.95% | 100.00% | 99.59% | 0.00% | 0.41% | 17814/4089/17/0 |
|
||||
| XL-128 v6 best | 0.60 | 99.97% | 100.00% | 99.71% | 0.00% | 0.29% | 17814/4094/12/0 |
|
||||
| XL-128 v6 best | 0.70 | 99.98% | 100.00% | 99.85% | 0.00% | 0.15% | 17814/4100/6/0 |
|
||||
| XL-128 v6 best | 0.80 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| XL-128 v6 final | 0.30 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| XL-128 v6 final | 0.40 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| XL-128 v6 final | 0.50 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| XL-128 v6 final | 0.60 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| XL-128 v6 final | 0.70 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| XL-128 v6 final | 0.80 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| XL-512 v6 best | 0.30 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| XL-512 v6 best | 0.40 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| XL-512 v6 best | 0.50 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| XL-512 v6 best | 0.60 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| XL-512 v6 best | 0.70 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| XL-512 v6 best | 0.80 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| V2-512 v6 best | 0.30 | 99.97% | 100.00% | 99.71% | 0.00% | 0.29% | 17814/4094/12/0 |
|
||||
| V2-512 v6 best | 0.40 | 99.97% | 100.00% | 99.71% | 0.00% | 0.29% | 17814/4094/12/0 |
|
||||
| V2-512 v6 best | 0.50 | 99.98% | 100.00% | 99.81% | 0.00% | 0.19% | 17814/4098/8/0 |
|
||||
| V2-512 v6 best | 0.60 | 99.99% | 100.00% | 99.95% | 0.00% | 0.05% | 17814/4104/2/0 |
|
||||
| V2-512 v6 best | 0.70 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
| V2-512 v6 best | 0.80 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
|
||||
|
||||
### 按 mega 类别拆解（每类 2,000 条）
|
||||
|
||||
| 类别 | episodes | metric | V2-128 deployed(v3) | V2-128 v6 best | V2-128 v6 final | XL-128 v6 best | XL-128 v6 final | XL-512 v6 best | V2-512 v6 best |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| (未分类) | - | Top-1 正确率 | - | - | - | - | - | - | - |
|
||||
| (未分类) | - | Recall@3 | - | - | - | - | - | - | - |
|
||||
| (未分类) | - | MRR | - | - | - | - | - | - | - |
|
||||
| (未分类) | - | 多跳证据全中 | - | - | - | - | - | - | - |
|
||||
| (未分类) | - | hop 正确率 | - | - | - | - | - | - | - |
|
||||
| (未分类) | - | 未知拒答率 | - | - | - | - | - | - | - |
|
||||
| (未分类) | - | 已知问题被误拒率 | - | - | - | - | - | - | - |
|
||||
| (未分类) | - | 未知问题被误读率 | - | - | - | - | - | - | - |
|
||||
| conflict_update | 2000 | Top-1 正确率 | 79.60% | 98.85% | 99.30% | 97.85% | 99.45% | 99.20% | 99.05% |
|
||||
| conflict_update | 2000 | Recall@3 | 91.25% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| conflict_update | 2000 | MRR | 85.83% | 99.42% | 99.65% | 98.92% | 99.72% | 99.60% | 99.52% |
|
||||
| conflict_update | 2000 | 多跳证据全中 | 91.25% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| conflict_update | 2000 | hop 正确率 | 49.55% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| conflict_update | 2000 | 未知拒答率 | - | - | - | - | - | - | - |
|
||||
| conflict_update | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
|
||||
| conflict_update | 2000 | 未知问题被误读率 | - | - | - | - | - | - | - |
|
||||
| distractor_128 | 2000 | Top-1 正确率 | 23.10% | 97.55% | 97.50% | 97.80% | 97.85% | 97.95% | 97.40% |
|
||||
| distractor_128 | 2000 | Recall@3 | 25.35% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| distractor_128 | 2000 | MRR | 26.60% | 98.78% | 98.75% | 98.90% | 98.92% | 98.98% | 98.70% |
|
||||
| distractor_128 | 2000 | 多跳证据全中 | 25.35% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| distractor_128 | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| distractor_128 | 2000 | 未知拒答率 | - | - | - | - | - | - | - |
|
||||
| distractor_128 | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
|
||||
| distractor_128 | 2000 | 未知问题被误读率 | - | - | - | - | - | - | - |
|
||||
| distractor_32 | 2000 | Top-1 正确率 | 22.55% | 98.05% | 98.45% | 98.50% | 98.45% | 98.05% | 96.90% |
|
||||
| distractor_32 | 2000 | Recall@3 | 24.55% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| distractor_32 | 2000 | MRR | 25.96% | 99.02% | 99.22% | 99.24% | 99.22% | 99.02% | 98.44% |
|
||||
| distractor_32 | 2000 | 多跳证据全中 | 24.55% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| distractor_32 | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| distractor_32 | 2000 | 未知拒答率 | - | - | - | - | - | - | - |
|
||||
| distractor_32 | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
|
||||
| distractor_32 | 2000 | 未知问题被误读率 | - | - | - | - | - | - | - |
|
||||
| forget_correction | 2000 | Top-1 正确率 | - | - | - | - | - | - | - |
|
||||
| forget_correction | 2000 | Recall@3 | - | - | - | - | - | - | - |
|
||||
| forget_correction | 2000 | MRR | - | - | - | - | - | - | - |
|
||||
| forget_correction | 2000 | 多跳证据全中 | - | - | - | - | - | - | - |
|
||||
| forget_correction | 2000 | hop 正确率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| forget_correction | 2000 | 未知拒答率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| forget_correction | 2000 | 已知问题被误拒率 | - | - | - | - | - | - | - |
|
||||
| forget_correction | 2000 | 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
|
||||
| long_context | 2000 | Top-1 正确率 | 52.55% | 98.00% | 97.95% | 97.65% | 98.15% | 98.15% | 97.30% |
|
||||
| long_context | 2000 | Recall@3 | 62.45% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| long_context | 2000 | MRR | 58.55% | 98.99% | 98.97% | 98.83% | 99.07% | 99.07% | 98.64% |
|
||||
| long_context | 2000 | 多跳证据全中 | 62.45% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| long_context | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| long_context | 2000 | 未知拒答率 | - | - | - | - | - | - | - |
|
||||
| long_context | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
|
||||
| long_context | 2000 | 未知问题被误读率 | - | - | - | - | - | - | - |
|
||||
| multi_hop | 2000 | Top-1 正确率 | 73.00% | 96.65% | 98.85% | 95.90% | 98.90% | 98.75% | 95.10% |
|
||||
| multi_hop | 2000 | Recall@3 | 66.57% | 99.50% | 96.05% | 99.55% | 98.52% | 93.53% | 98.17% |
|
||||
| multi_hop | 2000 | MRR | 84.53% | 98.30% | 99.42% | 97.93% | 99.45% | 99.37% | 97.53% |
|
||||
| multi_hop | 2000 | 多跳证据全中 | 37.15% | 99.00% | 92.10% | 99.10% | 97.05% | 87.05% | 96.35% |
|
||||
| multi_hop | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| multi_hop | 2000 | 未知拒答率 | - | - | - | - | - | - | - |
|
||||
| multi_hop | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
|
||||
| multi_hop | 2000 | 未知问题被误读率 | - | - | - | - | - | - | - |
|
||||
| paraphrase | 2000 | Top-1 正确率 | 44.30% | 98.55% | 99.50% | 98.70% | 99.70% | 99.65% | 97.55% |
|
||||
| paraphrase | 2000 | Recall@3 | 55.25% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| paraphrase | 2000 | MRR | 51.19% | 99.28% | 99.75% | 99.35% | 99.85% | 99.83% | 98.78% |
|
||||
| paraphrase | 2000 | 多跳证据全中 | 55.25% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| paraphrase | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| paraphrase | 2000 | 未知拒答率 | - | - | - | - | - | - | - |
|
||||
| paraphrase | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
|
||||
| paraphrase | 2000 | 未知问题被误读率 | - | - | - | - | - | - | - |
|
||||
| random_position | 2000 | Top-1 正确率 | 24.90% | 97.80% | 98.05% | 97.70% | 98.10% | 98.15% | 97.45% |
|
||||
| random_position | 2000 | Recall@3 | 28.70% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| random_position | 2000 | MRR | 29.20% | 98.90% | 99.02% | 98.85% | 99.05% | 99.08% | 98.72% |
|
||||
| random_position | 2000 | 多跳证据全中 | 28.70% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| random_position | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| random_position | 2000 | 未知拒答率 | - | - | - | - | - | - | - |
|
||||
| random_position | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
|
||||
| random_position | 2000 | 未知问题被误读率 | - | - | - | - | - | - | - |
|
||||
| single_fact | 2000 | Top-1 正确率 | 33.90% | 96.05% | 97.45% | 97.15% | 96.90% | 97.10% | 95.25% |
|
||||
| single_fact | 2000 | Recall@3 | 39.10% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| single_fact | 2000 | MRR | 38.46% | 98.00% | 98.72% | 98.56% | 98.42% | 98.54% | 97.60% |
|
||||
| single_fact | 2000 | 多跳证据全中 | 39.10% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| single_fact | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| single_fact | 2000 | 未知拒答率 | - | - | - | - | - | - | - |
|
||||
| single_fact | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
|
||||
| single_fact | 2000 | 未知问题被误读率 | - | - | - | - | - | - | - |
|
||||
| unknown | 1920 | Top-1 正确率 | 13.67% | 67.53% | 58.99% | 78.06% | 66.54% | 72.93% | 77.84% |
|
||||
| unknown | 1920 | Recall@3 | 25.36% | 79.00% | 68.96% | 88.20% | 76.63% | 83.46% | 90.63% |
|
||||
| unknown | 1920 | MRR | 26.97% | 75.78% | 67.60% | 84.56% | 74.53% | 80.13% | 84.98% |
|
||||
| unknown | 1920 | 多跳证据全中 | 25.36% | 79.00% | 68.96% | 88.20% | 76.63% | 83.46% | 90.63% |
|
||||
| unknown | 1920 | hop 正确率 | 55.26% | 100.00% | 100.00% | 99.48% | 100.00% | 100.00% | 100.00% |
|
||||
| unknown | 1920 | 未知拒答率 | 0.00% | 100.00% | 100.00% | 83.96% | 100.00% | 100.00% | 92.45% |
|
||||
| unknown | 1920 | 已知问题被误拒率 | 1.76% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
|
||||
| unknown | 1920 | 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 16.04% | 0.00% | 0.00% | 7.55% |
|
||||
| unknown_abstention | 2000 | Top-1 正确率 | - | - | - | - | - | - | - |
|
||||
| unknown_abstention | 2000 | Recall@3 | - | - | - | - | - | - | - |
|
||||
| unknown_abstention | 2000 | MRR | - | - | - | - | - | - | - |
|
||||
| unknown_abstention | 2000 | 多跳证据全中 | - | - | - | - | - | - | - |
|
||||
| unknown_abstention | 2000 | hop 正确率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| unknown_abstention | 2000 | 未知拒答率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
|
||||
| unknown_abstention | 2000 | 已知问题被误拒率 | - | - | - | - | - | - | - |
|
||||
| unknown_abstention | 2000 | 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
|
||||
Reference in New Issue
Block a user