wpy/natural-memory-nm21
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
natural-memory-nm21/router_scorecard_v6.md
T
WpyQwq 643e22ecb9 Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据 
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线
- 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同）
- 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00）
- 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00%
- 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益）
- 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
2026-09-19 11:11:31 +08:00

233 lines
19 KiB
Markdown
Raw Permalink Blame History

| metric | V2-128 deployed(v3) | V2-512 v3 best | V2-512 v3 final | V2-512 v6 best | V2-512 v6 final | XL-512 v6 best | XL-512 v6 final |
|---|---|---|---|---|---|---|---|
| 参数 | 2,037,774 | 4,741,902 | 4,741,902 | 4,741,902 | 4,741,902 | 7,898,127 | 7,898,127 |
| router_dim | 128 | 512 | 512 | 512 | 512 | 512 | 512 |
| 每条记录地址字节 | 512 | 2048 | 2048 | 2048 | 2048 | 2048 | 2048 |
| **Top-1 正确率** | 41.12% | 80.75% | 78.00% | 95.05% | 94.18% | 95.78% | 95.02% |
| Recall@1 | 37.03% | 76.98% | 73.56% | 89.71% | 88.65% | 90.24% | 89.48% |
| Recall@3 | 46.73% | 95.02% | 92.81% | 98.84% | 96.66% | 97.59% | 96.53% |
| Recall@5 | 48.91% | 97.31% | 95.30% | 99.35% | 97.64% | 98.34% | 97.61% |
| MRR | 47.69% | 88.46% | 86.13% | 97.12% | 95.87% | 97.24% | 96.64% |
| nDCG@3 | 44.22% | 89.53% | 86.92% | 97.24% | 95.48% | 96.64% | 95.60% |
| 多跳证据全中(Top-3) | 43.43% | 94.19% | 91.98% | 98.64% | 96.44% | 96.86% | 95.35% |
| hop 正确率 | 73.23% | 81.59% | 81.75% | 100.00% | 100.00% | 100.00% | 100.00% |
| hop 欠预测率 | 4.68% | 11.23% | 11.23% | 0.00% | 0.00% | 0.00% | 0.00% |
| need F1 | 89.58% | 88.68% | 88.77% | 99.98% | 100.00% | 100.00% | 100.00% |
| need 召回 | 99.82% | 88.77% | 88.77% | 100.00% | 100.00% | 100.00% | 100.00% |
| **未知拒答率** | 0.00% | 50.41% | 51.29% | 99.81% | 100.00% | 100.00% | 100.00% |
| **已知问题被误拒率** | 0.18% | 11.23% | 11.23% | 0.00% | 0.00% | 0.00% | 0.00% |
| 未知问题被误读率 | 100.00% | 49.59% | 48.71% | 0.19% | 0.00% | 0.00% | 0.00% |
| 仲裁准确率 | 81.12% | 81.59% | 81.75% | 99.96% | 100.00% | 100.00% | 100.00% |
| 已知问题 need 概率均值 | 97.63% | 88.77% | 88.77% | 99.88% | 100.00% | 100.00% | 100.00% |
| 未知问题 need 概率均值 | 99.89% | 62.33% | 45.28% | 0.65% | 0.00% | 0.00% | 0.00% |
| 平均分数余量 | -9.2989 | 9.6382 | 53.1605 | 12.3693 | 73.2482 | 45.5641 | 215.7533 |
| 单查询延迟 ms(GPU) | 0.8439 | 0.8789 | 0.9095 | 0.8750 | 0.8725 | 1.2161 | 1.2505 |
| 路由 QPS(GPU, 单查询) | 1184.9399 | 1137.7652 | 1099.5213 | 1142.8110 | 1146.1213 | 822.2900 | 799.6682 |
| 批量 QPS(GPU, batch=64) | 72916.1919 | 69808.0279 | 67135.2145 | 73617.3737 | 61636.8434 | 57386.7509 | 53291.5883 |
| 批量 QPS(GPU, batch=256) | 264812.9758 | 167235.0044 | 168449.8664 | 161941.2710 | 164167.8103 | 96746.8860 | 94345.8808 |
| 批量延迟 ms(GPU, batch=64) | 0.0137 | 0.0143 | 0.0149 | 0.0136 | 0.0162 | 0.0174 | 0.0188 |
### family = `benchmark_qa`
| metric | V2-128 deployed(v3) | V2-512 v3 best | V2-512 v3 final | V2-512 v6 best | V2-512 v6 final | XL-512 v6 best | XL-512 v6 final |
|---|---|---|---|---|---|---|---|
| episodes | 64 | 64 | 64 | 64 | 64 | 64 | 64 |
| Top-1 正确率 | - | - | - | - | - | - | - |
| Recall@3 | - | - | - | - | - | - | - |
| MRR | - | - | - | - | - | - | - |
| hop 正确率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| 未知拒答率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| 已知问题被误拒率 | - | - | - | - | - | - | - |
| 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| 平均分数余量 | - | - | - | - | - | - | - |
### family = `mega_validation`
| metric | V2-128 deployed(v3) | V2-512 v3 best | V2-512 v3 final | V2-512 v6 best | V2-512 v6 final | XL-512 v6 best | XL-512 v6 final |
|---|---|---|---|---|---|---|---|
| episodes | 20000 | 20000 | 20000 | 20000 | 20000 | 20000 | 20000 |
| Top-1 正确率 | 44.24% | 84.44% | 81.78% | 97.00% | 98.38% | 98.38% | 98.40% |
| Recall@3 | 49.15% | 97.37% | 95.43% | 99.77% | 99.75% | 99.19% | 98.68% |
| MRR | 50.04% | 91.19% | 89.02% | 98.49% | 99.18% | 99.18% | 99.20% |
| hop 正确率 | 74.95% | 80.00% | 80.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| 未知拒答率 | 0.00% | 50.00% | 50.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| 已知问题被误拒率 | 0.00% | 12.50% | 12.50% | 0.00% | 0.00% | 0.00% | 0.00% |
| 未知问题被误读率 | 100.00% | 50.00% | 50.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| 平均分数余量 | -8.0919 | 10.7052 | 59.2397 | 13.5425 | 81.8525 | 48.2819 | 235.8911 |
### family = `memory_policy`
| metric | V2-128 deployed(v3) | V2-512 v3 best | V2-512 v3 final | V2-512 v6 best | V2-512 v6 final | XL-512 v6 best | XL-512 v6 final |
|---|---|---|---|---|---|---|---|
| episodes | 1728 | 1728 | 1728 | 1728 | 1728 | 1728 | 1728 |
| Top-1 正确率 | 14.53% | 48.56% | 44.64% | 81.25% | 57.70% | 74.58% | 66.90% |
| Recall@3 | 26.77% | 73.76% | 69.42% | 92.44% | 68.60% | 83.48% | 77.09% |
| MRR | 28.02% | 64.42% | 60.50% | 87.43% | 66.64% | 81.00% | 74.88% |
| hop 正确率 | 57.41% | 98.96% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| 未知拒答率 | 0.00% | 14.29% | 100.00% | 80.95% | 100.00% | 100.00% | 100.00% |
| 已知问题被误拒率 | 1.87% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| 未知问题被误读率 | 100.00% | 85.71% | 0.00% | 19.05% | 0.00% | 0.00% | 0.00% |
| 平均分数余量 | -19.3599 | 0.2432 | -0.0168 | 2.2332 | -2.6484 | 22.8851 | 40.8526 |
### family = `native_memory`
| metric | V2-128 deployed(v3) | V2-512 v3 best | V2-512 v3 final | V2-512 v6 best | V2-512 v6 final | XL-512 v6 best | XL-512 v6 final |
|---|---|---|---|---|---|---|---|
| episodes | 128 | 128 | 128 | 128 | 128 | 128 | 128 |
| Top-1 正确率 | 0.00% | 42.06% | 44.86% | 23.36% | 49.53% | 46.73% | 38.32% |
| Recall@3 | 2.80% | 82.24% | 74.77% | 61.68% | 82.24% | 83.18% | 85.98% |
| MRR | 10.23% | 64.17% | 62.77% | 45.89% | 67.05% | 66.17% | 61.74% |
| hop 正确率 | 53.91% | 85.94% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| 未知拒答率 | 0.00% | 14.29% | 100.00% | 80.95% | 100.00% | 100.00% | 100.00% |
| 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| 未知问题被误读率 | 100.00% | 85.71% | 0.00% | 19.05% | 0.00% | 0.00% | 0.00% |
| 平均分数余量 | -29.2917 | -0.0322 | -7.5310 | -1.3701 | -2.5814 | 0.9634 | -5.2606 |
### 读取/拒答门槛曲线
| run | threshold | need F1 | need 召回 | 未知拒答率 | 已知问题被误拒率 | 未知被误读率 | tp/tn/fp/fn |
|---|---|---|---|---|---|---|---|
| V2-128 deployed(v3) | 0.30 | 89.67% | 100.00% | 0.00% | 0.00% | 100.00% | 17814/0/4106/0 |
| V2-128 deployed(v3) | 0.40 | 89.67% | 100.00% | 0.00% | 0.00% | 100.00% | 17814/0/4106/0 |
| V2-128 deployed(v3) | 0.50 | 89.58% | 99.82% | 0.00% | 0.18% | 100.00% | 17782/0/4106/32 |
| V2-128 deployed(v3) | 0.60 | 89.30% | 99.26% | 0.00% | 0.74% | 100.00% | 17682/0/4106/132 |
| V2-128 deployed(v3) | 0.70 | 89.19% | 99.05% | 0.00% | 0.95% | 100.00% | 17645/0/4106/169 |
| V2-128 deployed(v3) | 0.80 | 89.01% | 98.68% | 0.00% | 1.32% | 100.00% | 17579/0/4106/235 |
| V2-512 v3 best | 0.30 | 84.51% | 89.00% | 6.21% | 11.00% | 93.79% | 15855/255/3851/1959 |
| V2-512 v3 best | 0.40 | 88.68% | 88.78% | 50.37% | 11.22% | 49.63% | 15815/2068/2038/1999 |
| V2-512 v3 best | 0.50 | 88.68% | 88.77% | 50.41% | 11.23% | 49.59% | 15814/2070/2036/2000 |
| V2-512 v3 best | 0.60 | 88.73% | 88.77% | 50.85% | 11.23% | 49.15% | 15814/2088/2018/2000 |
| V2-512 v3 best | 0.70 | 88.77% | 88.77% | 51.24% | 11.23% | 48.76% | 15814/2104/2002/2000 |
| V2-512 v3 best | 0.80 | 89.49% | 88.77% | 58.26% | 11.23% | 41.74% | 15814/2392/1714/2000 |
| V2-512 v3 final | 0.30 | 88.77% | 88.77% | 51.29% | 11.23% | 48.71% | 15814/2106/2000/2000 |
| V2-512 v3 final | 0.40 | 88.77% | 88.77% | 51.29% | 11.23% | 48.71% | 15814/2106/2000/2000 |
| V2-512 v3 final | 0.50 | 88.77% | 88.77% | 51.29% | 11.23% | 48.71% | 15814/2106/2000/2000 |
| V2-512 v3 final | 0.60 | 89.49% | 88.77% | 58.26% | 11.23% | 41.74% | 15814/2392/1714/2000 |
| V2-512 v3 final | 0.70 | 89.49% | 88.77% | 58.26% | 11.23% | 41.74% | 15814/2392/1714/2000 |
| V2-512 v3 final | 0.80 | 89.49% | 88.77% | 58.26% | 11.23% | 41.74% | 15814/2392/1714/2000 |
| V2-512 v6 best | 0.30 | 99.97% | 100.00% | 99.71% | 0.00% | 0.29% | 17814/4094/12/0 |
| V2-512 v6 best | 0.40 | 99.97% | 100.00% | 99.71% | 0.00% | 0.29% | 17814/4094/12/0 |
| V2-512 v6 best | 0.50 | 99.98% | 100.00% | 99.81% | 0.00% | 0.19% | 17814/4098/8/0 |
| V2-512 v6 best | 0.60 | 99.99% | 100.00% | 99.95% | 0.00% | 0.05% | 17814/4104/2/0 |
| V2-512 v6 best | 0.70 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| V2-512 v6 best | 0.80 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| V2-512 v6 final | 0.30 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| V2-512 v6 final | 0.40 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| V2-512 v6 final | 0.50 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| V2-512 v6 final | 0.60 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| V2-512 v6 final | 0.70 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| V2-512 v6 final | 0.80 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| XL-512 v6 best | 0.30 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| XL-512 v6 best | 0.40 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| XL-512 v6 best | 0.50 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| XL-512 v6 best | 0.60 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| XL-512 v6 best | 0.70 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| XL-512 v6 best | 0.80 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| XL-512 v6 final | 0.30 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| XL-512 v6 final | 0.40 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| XL-512 v6 final | 0.50 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| XL-512 v6 final | 0.60 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| XL-512 v6 final | 0.70 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
| XL-512 v6 final | 0.80 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 |
### 按 mega 类别拆解（每类 2,000 条）
| 类别 | episodes | metric | V2-128 deployed(v3) | V2-512 v3 best | V2-512 v3 final | V2-512 v6 best | V2-512 v6 final | XL-512 v6 best | XL-512 v6 final |
|---|---|---|---|---|---|---|---|---|---|
| (未分类) | - | Top-1 正确率 | - | - | - | - | - | - | - |
| (未分类) | - | Recall@3 | - | - | - | - | - | - | - |
| (未分类) | - | MRR | - | - | - | - | - | - | - |
| (未分类) | - | 多跳证据全中 | - | - | - | - | - | - | - |
| (未分类) | - | hop 正确率 | - | - | - | - | - | - | - |
| (未分类) | - | 未知拒答率 | - | - | - | - | - | - | - |
| (未分类) | - | 已知问题被误拒率 | - | - | - | - | - | - | - |
| (未分类) | - | 未知问题被误读率 | - | - | - | - | - | - | - |
| conflict_update | 2000 | Top-1 正确率 | 79.60% | 86.05% | 80.25% | 99.05% | 99.35% | 99.20% | 99.25% |
| conflict_update | 2000 | Recall@3 | 91.25% | 99.95% | 99.80% | 100.00% | 100.00% | 100.00% | 100.00% |
| conflict_update | 2000 | MRR | 85.83% | 92.82% | 89.66% | 99.52% | 99.67% | 99.60% | 99.62% |
| conflict_update | 2000 | 多跳证据全中 | 91.25% | 99.95% | 99.80% | 100.00% | 100.00% | 100.00% | 100.00% |
| conflict_update | 2000 | hop 正确率 | 49.55% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| conflict_update | 2000 | 未知拒答率 | - | - | - | - | - | - | - |
| conflict_update | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| conflict_update | 2000 | 未知问题被误读率 | - | - | - | - | - | - | - |
| distractor_128 | 2000 | Top-1 正确率 | 23.10% | 92.15% | 89.80% | 97.40% | 98.00% | 97.95% | 98.00% |
| distractor_128 | 2000 | Recall@3 | 25.35% | 100.00% | 99.95% | 100.00% | 100.00% | 100.00% | 100.00% |
| distractor_128 | 2000 | MRR | 26.60% | 95.99% | 94.74% | 98.70% | 99.00% | 98.98% | 99.00% |
| distractor_128 | 2000 | 多跳证据全中 | 25.35% | 100.00% | 99.95% | 100.00% | 100.00% | 100.00% | 100.00% |
| distractor_128 | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| distractor_128 | 2000 | 未知拒答率 | - | - | - | - | - | - | - |
| distractor_128 | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| distractor_128 | 2000 | 未知问题被误读率 | - | - | - | - | - | - | - |
| distractor_32 | 2000 | Top-1 正确率 | 22.55% | 92.15% | 90.60% | 96.90% | 98.05% | 98.05% | 98.20% |
| distractor_32 | 2000 | Recall@3 | 24.55% | 100.00% | 99.95% | 100.00% | 100.00% | 100.00% | 100.00% |
| distractor_32 | 2000 | MRR | 25.96% | 95.97% | 95.13% | 98.44% | 99.02% | 99.02% | 99.09% |
| distractor_32 | 2000 | 多跳证据全中 | 24.55% | 100.00% | 99.95% | 100.00% | 100.00% | 100.00% | 100.00% |
| distractor_32 | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| distractor_32 | 2000 | 未知拒答率 | - | - | - | - | - | - | - |
| distractor_32 | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| distractor_32 | 2000 | 未知问题被误读率 | - | - | - | - | - | - | - |
| forget_correction | 2000 | Top-1 正确率 | - | - | - | - | - | - | - |
| forget_correction | 2000 | Recall@3 | - | - | - | - | - | - | - |
| forget_correction | 2000 | MRR | - | - | - | - | - | - | - |
| forget_correction | 2000 | 多跳证据全中 | - | - | - | - | - | - | - |
| forget_correction | 2000 | hop 正确率 | 0.00% | 0.00% | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| forget_correction | 2000 | 未知拒答率 | 0.00% | 0.00% | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| forget_correction | 2000 | 已知问题被误拒率 | - | - | - | - | - | - | - |
| forget_correction | 2000 | 未知问题被误读率 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| long_context | 2000 | Top-1 正确率 | 52.55% | 78.20% | 69.65% | 97.30% | 97.70% | 98.15% | 98.30% |
| long_context | 2000 | Recall@3 | 62.45% | 98.90% | 84.65% | 100.00% | 100.00% | 100.00% | 100.00% |
| long_context | 2000 | MRR | 58.55% | 88.20% | 77.64% | 98.64% | 98.84% | 99.07% | 99.14% |
| long_context | 2000 | 多跳证据全中 | 62.45% | 98.90% | 84.65% | 100.00% | 100.00% | 100.00% | 100.00% |
| long_context | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| long_context | 2000 | 未知拒答率 | - | - | - | - | - | - | - |
| long_context | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| long_context | 2000 | 未知问题被误读率 | - | - | - | - | - | - | - |
| multi_hop | 2000 | Top-1 正确率 | 73.00% | 67.10% | 79.15% | 95.10% | 98.65% | 98.75% | 98.65% |
| multi_hop | 2000 | Recall@3 | 66.57% | 91.20% | 92.35% | 98.17% | 98.02% | 93.53% | 89.42% |
| multi_hop | 2000 | MRR | 84.53% | 82.14% | 89.10% | 97.53% | 99.33% | 99.37% | 99.33% |
| multi_hop | 2000 | 多跳证据全中 | 37.15% | 83.85% | 84.90% | 96.35% | 96.05% | 87.05% | 78.85% |
| multi_hop | 2000 | hop 正确率 | 100.00% | 0.00% | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| multi_hop | 2000 | 未知拒答率 | - | - | - | - | - | - | - |
| multi_hop | 2000 | 已知问题被误拒率 | 0.00% | 100.00% | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| multi_hop | 2000 | 未知问题被误读率 | - | - | - | - | - | - | - |
| paraphrase | 2000 | Top-1 正确率 | 44.30% | 77.25% | 62.50% | 97.55% | 99.55% | 99.65% | 99.50% |
| paraphrase | 2000 | Recall@3 | 55.25% | 88.95% | 86.85% | 100.00% | 100.00% | 100.00% | 100.00% |
| paraphrase | 2000 | MRR | 51.19% | 83.29% | 75.02% | 98.78% | 99.78% | 99.83% | 99.75% |
| paraphrase | 2000 | 多跳证据全中 | 55.25% | 88.95% | 86.85% | 100.00% | 100.00% | 100.00% | 100.00% |
| paraphrase | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| paraphrase | 2000 | 未知拒答率 | - | - | - | - | - | - | - |
| paraphrase | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| paraphrase | 2000 | 未知问题被误读率 | - | - | - | - | - | - | - |
| random_position | 2000 | Top-1 正确率 | 24.90% | 94.20% | 92.00% | 97.45% | 98.25% | 98.15% | 98.05% |
| random_position | 2000 | Recall@3 | 28.70% | 100.00% | 99.90% | 100.00% | 100.00% | 100.00% | 100.00% |
| random_position | 2000 | MRR | 29.20% | 97.02% | 95.84% | 98.72% | 99.12% | 99.08% | 99.02% |
| random_position | 2000 | 多跳证据全中 | 28.70% | 100.00% | 99.90% | 100.00% | 100.00% | 100.00% | 100.00% |
| random_position | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| random_position | 2000 | 未知拒答率 | - | - | - | - | - | - | - |
| random_position | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| random_position | 2000 | 未知问题被误读率 | - | - | - | - | - | - | - |
| single_fact | 2000 | Top-1 正确率 | 33.90% | 88.45% | 90.30% | 95.25% | 97.45% | 97.10% | 97.25% |
| single_fact | 2000 | Recall@3 | 39.10% | 99.95% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| single_fact | 2000 | MRR | 38.46% | 94.05% | 95.00% | 97.60% | 98.72% | 98.54% | 98.62% |
| single_fact | 2000 | 多跳证据全中 | 39.10% | 99.95% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| single_fact | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| single_fact | 2000 | 未知拒答率 | - | - | - | - | - | - | - |
| single_fact | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| single_fact | 2000 | 未知问题被误读率 | - | - | - | - | - | - | - |
| unknown | 1920 | Top-1 正确率 | 13.67% | 48.18% | 44.65% | 77.84% | 57.22% | 72.93% | 65.21% |
| unknown | 1920 | Recall@3 | 25.36% | 74.26% | 69.74% | 90.63% | 69.40% | 83.46% | 77.62% |
| unknown | 1920 | MRR | 26.97% | 64.41% | 60.64% | 84.98% | 66.66% | 80.13% | 74.11% |
| unknown | 1920 | 多跳证据全中 | 25.36% | 74.26% | 69.74% | 90.63% | 69.40% | 83.46% | 77.62% |
| unknown | 1920 | hop 正确率 | 55.26% | 98.12% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| unknown | 1920 | 未知拒答率 | 0.00% | 66.04% | 100.00% | 92.45% | 100.00% | 100.00% | 100.00% |
| unknown | 1920 | 已知问题被误拒率 | 1.76% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| unknown | 1920 | 未知问题被误读率 | 100.00% | 33.96% | 0.00% | 7.55% | 0.00% | 0.00% | 0.00% |
| unknown_abstention | 2000 | Top-1 正确率 | - | - | - | - | - | - | - |
| unknown_abstention | 2000 | Recall@3 | - | - | - | - | - | - | - |
| unknown_abstention | 2000 | MRR | - | - | - | - | - | - | - |
| unknown_abstention | 2000 | 多跳证据全中 | - | - | - | - | - | - | - |
| unknown_abstention | 2000 | hop 正确率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| unknown_abstention | 2000 | 未知拒答率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| unknown_abstention | 2000 | 已知问题被误拒率 | - | - | - | - | - | - | - |
| unknown_abstention | 2000 | 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
Reference in New Issue View Git Blame Copy Permalink