| metric | V2-128 deployed(v3) | V2-128 v6 final | REPLAY-128 v7 final | V2-512 v6 final | XL-512 v6 final | XL-128 v6 final | |---|---|---|---|---|---|---| | 参数 | 2,037,774 | 2,037,774 | 2,037,774 | 4,741,902 | 7,898,127 | 6,714,639 | | router_dim | 128 | 128 | 128 | 512 | 512 | 128 | | 每条记录地址字节 | 512 | 512 | 512 | 2048 | 2048 | 512 | | **Top-1 正确率** | 41.12% | 94.37% | 94.14% | 94.18% | 95.02% | 95.19% | | Recall@1 | 37.03% | 88.82% | 88.58% | 88.65% | 89.48% | 89.64% | | Recall@3 | 46.73% | 96.40% | 96.48% | 96.66% | 96.53% | 97.45% | | Recall@5 | 48.91% | 97.40% | 97.15% | 97.64% | 97.61% | 98.20% | | MRR | 47.69% | 95.97% | 95.77% | 95.87% | 96.64% | 96.70% | | nDCG@3 | 44.22% | 95.31% | 95.37% | 95.48% | 95.60% | 96.43% | | 多跳证据全中(Top-3) | 43.43% | 95.95% | 96.22% | 96.44% | 95.35% | 97.29% | | hop 正确率 | 73.23% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | hop 欠预测率 | 4.68% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | need F1 | 89.58% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | need 召回 | 99.82% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | **未知拒答率** | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | **已知问题被误拒率** | 0.18% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | 仲裁准确率 | 81.12% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | 已知问题 need 概率均值 | 97.63% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | 未知问题 need 概率均值 | 99.89% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | 平均分数余量 | -9.2989 | 51.3153 | 40.1456 | 73.2482 | 215.7533 | 200.3346 | | 单查询延迟 ms(GPU) | 1.0995 | 0.9277 | 0.9663 | 0.9488 | 1.4501 | 1.5358 | | 路由 QPS(GPU, 单查询) | 909.4856 | 1077.8859 | 1034.8360 | 1053.9133 | 689.6254 | 651.1112 | | 批量 QPS(GPU, batch=64) | 66037.9306 | 72856.4274 | 69378.1979 | 71586.7654 | 46891.9434 | 33111.5549 | | 批量 QPS(GPU, batch=256) | 207287.4493 | 236293.1512 | 242759.8763 | 177958.2077 | 85734.5711 | 131545.1415 | | 批量延迟 ms(GPU, batch=64) | 0.0151 | 0.0137 | 0.0144 | 0.0140 | 0.0213 | 0.0302 | ### family = `benchmark_qa` | metric | V2-128 deployed(v3) | V2-128 v6 final | REPLAY-128 v7 final | V2-512 v6 final | XL-512 v6 final | XL-128 v6 final | |---|---|---|---|---|---|---| | episodes | 64 | 64 | 64 | 64 | 64 | 64 | | Top-1 正确率 | - | - | - | - | - | - | | Recall@3 | - | - | - | - | - | - | | MRR | - | - | - | - | - | - | | hop 正确率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | 未知拒答率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | 已知问题被误拒率 | - | - | - | - | - | - | | 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | 平均分数余量 | - | - | - | - | - | - | ### family = `mega_validation` | metric | V2-128 deployed(v3) | V2-128 v6 final | REPLAY-128 v7 final | V2-512 v6 final | XL-512 v6 final | XL-128 v6 final | |---|---|---|---|---|---|---| | episodes | 20000 | 20000 | 20000 | 20000 | 20000 | 20000 | | Top-1 正确率 | 44.24% | 98.38% | 98.22% | 98.38% | 98.40% | 98.44% | | Recall@3 | 49.15% | 99.51% | 99.72% | 99.75% | 98.68% | 99.82% | | MRR | 50.04% | 99.19% | 99.11% | 99.18% | 99.20% | 99.21% | | hop 正确率 | 74.95% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | 未知拒答率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | 平均分数余量 | -8.0919 | 57.1101 | 44.7334 | 81.8525 | 235.8911 | 217.4091 | ### family = `memory_policy` | metric | V2-128 deployed(v3) | V2-128 v6 final | REPLAY-128 v7 final | V2-512 v6 final | XL-512 v6 final | XL-128 v6 final | |---|---|---|---|---|---|---| | episodes | 1728 | 1728 | 1728 | 1728 | 1728 | 1728 | | Top-1 正确率 | 14.53% | 59.40% | 58.76% | 57.70% | 66.90% | 67.55% | | Recall@3 | 26.77% | 68.42% | 67.31% | 68.60% | 77.09% | 75.92% | | MRR | 28.02% | 67.60% | 66.33% | 66.64% | 74.88% | 74.81% | | hop 正确率 | 57.41% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | 未知拒答率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | 已知问题被误拒率 | 1.87% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | 平均分数余量 | -19.3599 | 0.2715 | -0.2722 | -2.6484 | 40.8526 | 52.4323 | ### family = `native_memory` | metric | V2-128 deployed(v3) | V2-128 v6 final | REPLAY-128 v7 final | V2-512 v6 final | XL-512 v6 final | XL-128 v6 final | |---|---|---|---|---|---|---| | episodes | 128 | 128 | 128 | 128 | 128 | 128 | | Top-1 正确率 | 0.00% | 52.34% | 47.66% | 49.53% | 38.32% | 50.47% | | Recall@3 | 2.80% | 77.57% | 77.57% | 82.24% | 85.98% | 87.85% | | MRR | 10.23% | 67.49% | 65.73% | 67.05% | 61.74% | 70.02% | | hop 正确率 | 53.91% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | 未知拒答率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | 平均分数余量 | -29.2917 | -0.8787 | -1.0887 | -2.5814 | -5.2606 | 6.6553 | ### 读取/拒答门槛曲线 | run | threshold | need F1 | need 召回 | 未知拒答率 | 已知问题被误拒率 | 未知被误读率 | tp/tn/fp/fn | |---|---|---|---|---|---|---|---| | V2-128 deployed(v3) | 0.30 | 89.67% | 100.00% | 0.00% | 0.00% | 100.00% | 17814/0/4106/0 | | V2-128 deployed(v3) | 0.40 | 89.67% | 100.00% | 0.00% | 0.00% | 100.00% | 17814/0/4106/0 | | V2-128 deployed(v3) | 0.50 | 89.58% | 99.82% | 0.00% | 0.18% | 100.00% | 17782/0/4106/32 | | V2-128 deployed(v3) | 0.60 | 89.30% | 99.26% | 0.00% | 0.74% | 100.00% | 17682/0/4106/132 | | V2-128 deployed(v3) | 0.70 | 89.19% | 99.05% | 0.00% | 0.95% | 100.00% | 17645/0/4106/169 | | V2-128 deployed(v3) | 0.80 | 89.01% | 98.68% | 0.00% | 1.32% | 100.00% | 17579/0/4106/235 | | V2-128 v6 final | 0.30 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | V2-128 v6 final | 0.40 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | V2-128 v6 final | 0.50 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | V2-128 v6 final | 0.60 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | V2-128 v6 final | 0.70 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | V2-128 v6 final | 0.80 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | REPLAY-128 v7 final | 0.30 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | REPLAY-128 v7 final | 0.40 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | REPLAY-128 v7 final | 0.50 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | REPLAY-128 v7 final | 0.60 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | REPLAY-128 v7 final | 0.70 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | REPLAY-128 v7 final | 0.80 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | V2-512 v6 final | 0.30 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | V2-512 v6 final | 0.40 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | V2-512 v6 final | 0.50 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | V2-512 v6 final | 0.60 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | V2-512 v6 final | 0.70 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | V2-512 v6 final | 0.80 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | XL-512 v6 final | 0.30 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | XL-512 v6 final | 0.40 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | XL-512 v6 final | 0.50 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | XL-512 v6 final | 0.60 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | XL-512 v6 final | 0.70 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | XL-512 v6 final | 0.80 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | XL-128 v6 final | 0.30 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | XL-128 v6 final | 0.40 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | XL-128 v6 final | 0.50 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | XL-128 v6 final | 0.60 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | XL-128 v6 final | 0.70 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | | XL-128 v6 final | 0.80 | 100.00% | 100.00% | 100.00% | 0.00% | 0.00% | 17814/4106/0/0 | ### 按 mega 类别拆解(每类 2,000 条) | 类别 | episodes | metric | V2-128 deployed(v3) | V2-128 v6 final | REPLAY-128 v7 final | V2-512 v6 final | XL-512 v6 final | XL-128 v6 final | |---|---|---|---|---|---|---|---|---| | (未分类) | - | Top-1 正确率 | - | - | - | - | - | - | | (未分类) | - | Recall@3 | - | - | - | - | - | - | | (未分类) | - | MRR | - | - | - | - | - | - | | (未分类) | - | 多跳证据全中 | - | - | - | - | - | - | | (未分类) | - | hop 正确率 | - | - | - | - | - | - | | (未分类) | - | 未知拒答率 | - | - | - | - | - | - | | (未分类) | - | 已知问题被误拒率 | - | - | - | - | - | - | | (未分类) | - | 未知问题被误读率 | - | - | - | - | - | - | | conflict_update | 2000 | Top-1 正确率 | 79.60% | 99.30% | 99.10% | 99.35% | 99.25% | 99.45% | | conflict_update | 2000 | Recall@3 | 91.25% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | conflict_update | 2000 | MRR | 85.83% | 99.65% | 99.55% | 99.67% | 99.62% | 99.72% | | conflict_update | 2000 | 多跳证据全中 | 91.25% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | conflict_update | 2000 | hop 正确率 | 49.55% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | conflict_update | 2000 | 未知拒答率 | - | - | - | - | - | - | | conflict_update | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | conflict_update | 2000 | 未知问题被误读率 | - | - | - | - | - | - | | distractor_128 | 2000 | Top-1 正确率 | 23.10% | 97.50% | 97.30% | 98.00% | 98.00% | 97.85% | | distractor_128 | 2000 | Recall@3 | 25.35% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | distractor_128 | 2000 | MRR | 26.60% | 98.75% | 98.65% | 99.00% | 99.00% | 98.92% | | distractor_128 | 2000 | 多跳证据全中 | 25.35% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | distractor_128 | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | distractor_128 | 2000 | 未知拒答率 | - | - | - | - | - | - | | distractor_128 | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | distractor_128 | 2000 | 未知问题被误读率 | - | - | - | - | - | - | | distractor_32 | 2000 | Top-1 正确率 | 22.55% | 98.45% | 98.30% | 98.05% | 98.20% | 98.45% | | distractor_32 | 2000 | Recall@3 | 24.55% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | distractor_32 | 2000 | MRR | 25.96% | 99.22% | 99.15% | 99.02% | 99.09% | 99.22% | | distractor_32 | 2000 | 多跳证据全中 | 24.55% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | distractor_32 | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | distractor_32 | 2000 | 未知拒答率 | - | - | - | - | - | - | | distractor_32 | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | distractor_32 | 2000 | 未知问题被误读率 | - | - | - | - | - | - | | forget_correction | 2000 | Top-1 正确率 | - | - | - | - | - | - | | forget_correction | 2000 | Recall@3 | - | - | - | - | - | - | | forget_correction | 2000 | MRR | - | - | - | - | - | - | | forget_correction | 2000 | 多跳证据全中 | - | - | - | - | - | - | | forget_correction | 2000 | hop 正确率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | forget_correction | 2000 | 未知拒答率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | forget_correction | 2000 | 已知问题被误拒率 | - | - | - | - | - | - | | forget_correction | 2000 | 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | long_context | 2000 | Top-1 正确率 | 52.55% | 97.95% | 97.80% | 97.70% | 98.30% | 98.15% | | long_context | 2000 | Recall@3 | 62.45% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | long_context | 2000 | MRR | 58.55% | 98.97% | 98.89% | 98.84% | 99.14% | 99.07% | | long_context | 2000 | 多跳证据全中 | 62.45% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | long_context | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | long_context | 2000 | 未知拒答率 | - | - | - | - | - | - | | long_context | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | long_context | 2000 | 未知问题被误读率 | - | - | - | - | - | - | | multi_hop | 2000 | Top-1 正确率 | 73.00% | 98.85% | 99.05% | 98.65% | 98.65% | 98.90% | | multi_hop | 2000 | Recall@3 | 66.57% | 96.05% | 97.72% | 98.02% | 89.42% | 98.52% | | multi_hop | 2000 | MRR | 84.53% | 99.42% | 99.52% | 99.33% | 99.33% | 99.45% | | multi_hop | 2000 | 多跳证据全中 | 37.15% | 92.10% | 95.45% | 96.05% | 78.85% | 97.05% | | multi_hop | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | multi_hop | 2000 | 未知拒答率 | - | - | - | - | - | - | | multi_hop | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | multi_hop | 2000 | 未知问题被误读率 | - | - | - | - | - | - | | paraphrase | 2000 | Top-1 正确率 | 44.30% | 99.50% | 99.40% | 99.55% | 99.50% | 99.70% | | paraphrase | 2000 | Recall@3 | 55.25% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | paraphrase | 2000 | MRR | 51.19% | 99.75% | 99.70% | 99.78% | 99.75% | 99.85% | | paraphrase | 2000 | 多跳证据全中 | 55.25% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | paraphrase | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | paraphrase | 2000 | 未知拒答率 | - | - | - | - | - | - | | paraphrase | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | paraphrase | 2000 | 未知问题被误读率 | - | - | - | - | - | - | | random_position | 2000 | Top-1 正确率 | 24.90% | 98.05% | 97.80% | 98.25% | 98.05% | 98.10% | | random_position | 2000 | Recall@3 | 28.70% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | random_position | 2000 | MRR | 29.20% | 99.02% | 98.89% | 99.12% | 99.02% | 99.05% | | random_position | 2000 | 多跳证据全中 | 28.70% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | random_position | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | random_position | 2000 | 未知拒答率 | - | - | - | - | - | - | | random_position | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | random_position | 2000 | 未知问题被误读率 | - | - | - | - | - | - | | single_fact | 2000 | Top-1 正确率 | 33.90% | 97.45% | 97.05% | 97.45% | 97.25% | 96.90% | | single_fact | 2000 | Recall@3 | 39.10% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | single_fact | 2000 | MRR | 38.46% | 98.72% | 98.52% | 98.72% | 98.62% | 98.42% | | single_fact | 2000 | 多跳证据全中 | 39.10% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | single_fact | 2000 | hop 正确率 | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | single_fact | 2000 | 未知拒答率 | - | - | - | - | - | - | | single_fact | 2000 | 已知问题被误拒率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | single_fact | 2000 | 未知问题被误读率 | - | - | - | - | - | - | | unknown | 1920 | Top-1 正确率 | 13.67% | 58.99% | 58.10% | 57.22% | 65.21% | 66.54% | | unknown | 1920 | Recall@3 | 25.36% | 68.96% | 67.92% | 69.40% | 77.62% | 76.63% | | unknown | 1920 | MRR | 26.97% | 67.60% | 66.30% | 66.66% | 74.11% | 74.53% | | unknown | 1920 | 多跳证据全中 | 25.36% | 68.96% | 67.92% | 69.40% | 77.62% | 76.63% | | unknown | 1920 | hop 正确率 | 55.26% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | unknown | 1920 | 未知拒答率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | unknown | 1920 | 已知问题被误拒率 | 1.76% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | unknown | 1920 | 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | | unknown_abstention | 2000 | Top-1 正确率 | - | - | - | - | - | - | | unknown_abstention | 2000 | Recall@3 | - | - | - | - | - | - | | unknown_abstention | 2000 | MRR | - | - | - | - | - | - | | unknown_abstention | 2000 | 多跳证据全中 | - | - | - | - | - | - | | unknown_abstention | 2000 | hop 正确率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | unknown_abstention | 2000 | 未知拒答率 | 0.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | | unknown_abstention | 2000 | 已知问题被误拒率 | - | - | - | - | - | - | | unknown_abstention | 2000 | 未知问题被误读率 | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |