wpy/natural-memory-nm21
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
natural-memory-nm21/router_verdict_v6.md
T
WpyQwq 643e22ecb9 Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据 
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线
- 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同）
- 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00）
- 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00%
- 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益）
- 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
2026-09-19 11:11:31 +08:00

94 lines
7.5 KiB
Markdown
Raw Permalink Blame History

| 指标 | 最强旧基线 | 基线值 | 候选值 | 差值 | 判定 |
|---|---|---:|---:|---:|---|
| Top-1 正确率 | V2-512 v3 best | 80.75% | 95.78% | +15.03pp | 通过 |
| Top-1 正确率 | V2-512 v3 best | 80.75% | 95.02% | +14.27pp | 通过 |
| Top-1 正确率 | V2-512 v3 best | 80.75% | 95.05% | +14.30pp | 通过 |
| Top-1 正确率 | V2-512 v3 best | 80.75% | 94.18% | +13.43pp | 通过 |
| Recall@1 | V2-512 v3 best | 76.98% | 90.24% | +13.26pp | 通过 |
| Recall@1 | V2-512 v3 best | 76.98% | 89.48% | +12.50pp | 通过 |
| Recall@1 | V2-512 v3 best | 76.98% | 89.71% | +12.73pp | 通过 |
| Recall@1 | V2-512 v3 best | 76.98% | 88.65% | +11.66pp | 通过 |
| Recall@3 | V2-512 v3 best | 95.02% | 97.59% | +2.57pp | 通过 |
| Recall@3 | V2-512 v3 best | 95.02% | 96.53% | +1.52pp | 通过 |
| Recall@3 | V2-512 v3 best | 95.02% | 98.84% | +3.83pp | 通过 |
| Recall@3 | V2-512 v3 best | 95.02% | 96.66% | +1.65pp | 通过 |
| Recall@5 | V2-512 v3 best | 97.31% | 98.34% | +1.04pp | 通过 |
| Recall@5 | V2-512 v3 best | 97.31% | 97.61% | +0.30pp | 通过 |
| Recall@5 | V2-512 v3 best | 97.31% | 99.35% | +2.04pp | 通过 |
| Recall@5 | V2-512 v3 best | 97.31% | 97.64% | +0.33pp | 通过 |
| MRR | V2-512 v3 best | 88.46% | 97.24% | +8.79pp | 通过 |
| MRR | V2-512 v3 best | 88.46% | 96.64% | +8.18pp | 通过 |
| MRR | V2-512 v3 best | 88.46% | 97.12% | +8.66pp | 通过 |
| MRR | V2-512 v3 best | 88.46% | 95.87% | +7.41pp | 通过 |
| nDCG@3 | V2-512 v3 best | 89.53% | 96.64% | +7.11pp | 通过 |
| nDCG@3 | V2-512 v3 best | 89.53% | 95.60% | +6.07pp | 通过 |
| nDCG@3 | V2-512 v3 best | 89.53% | 97.24% | +7.71pp | 通过 |
| nDCG@3 | V2-512 v3 best | 89.53% | 95.48% | +5.95pp | 通过 |
| 多跳证据全中(Top-3) | V2-512 v3 best | 94.19% | 96.86% | +2.67pp | 通过 |
| 多跳证据全中(Top-3) | V2-512 v3 best | 94.19% | 95.35% | +1.16pp | 通过 |
| 多跳证据全中(Top-3) | V2-512 v3 best | 94.19% | 98.64% | +4.45pp | 通过 |
| 多跳证据全中(Top-3) | V2-512 v3 best | 94.19% | 96.44% | +2.25pp | 通过 |
| 多跳证据全中(仅多正例) | V2-512 v3 final | 84.90% | 87.05% | +2.15pp | 通过 |
| 多跳证据全中(仅多正例) | V2-512 v3 final | 84.90% | 78.85% | -6.05pp | **未通过** |
| 多跳证据全中(仅多正例) | V2-512 v3 final | 84.90% | 96.35% | +11.45pp | 通过 |
| 多跳证据全中(仅多正例) | V2-512 v3 final | 84.90% | 96.05% | +11.15pp | 通过 |
| hop 正确率 | V2-512 v3 final | 81.75% | 100.00% | +18.25pp | 通过 |
| hop 正确率 | V2-512 v3 final | 81.75% | 100.00% | +18.25pp | 通过 |
| hop 正确率 | V2-512 v3 final | 81.75% | 100.00% | +18.25pp | 通过 |
| hop 正确率 | V2-512 v3 final | 81.75% | 100.00% | +18.25pp | 通过 |
| hop 欠预测率 | V2-128 deployed(v3) | 4.68% | 0.00% | -4.68pp | 通过 |
| hop 欠预测率 | V2-128 deployed(v3) | 4.68% | 0.00% | -4.68pp | 通过 |
| hop 欠预测率 | V2-128 deployed(v3) | 4.68% | 0.00% | -4.68pp | 通过 |
| hop 欠预测率 | V2-128 deployed(v3) | 4.68% | 0.00% | -4.68pp | 通过 |
| need F1 (门槛 0.50) | V2-128 deployed(v3) | 89.58% | 100.00% | +10.42pp | 通过 |
| need F1 (门槛 0.50) | V2-128 deployed(v3) | 89.58% | 100.00% | +10.42pp | 通过 |
| need F1 (门槛 0.50) | V2-128 deployed(v3) | 89.58% | 99.98% | +10.40pp | 通过 |
| need F1 (门槛 0.50) | V2-128 deployed(v3) | 89.58% | 100.00% | +10.42pp | 通过 |
| need 召回 (门槛 0.50) | V2-128 deployed(v3) | 99.82% | 100.00% | +0.18pp | 通过 |
| need 召回 (门槛 0.50) | V2-128 deployed(v3) | 99.82% | 100.00% | +0.18pp | 通过 |
| need 召回 (门槛 0.50) | V2-128 deployed(v3) | 99.82% | 100.00% | +0.18pp | 通过 |
| need 召回 (门槛 0.50) | V2-128 deployed(v3) | 99.82% | 100.00% | +0.18pp | 通过 |
| 未知拒答率 (门槛 0.50) | V2-512 v3 final | 51.29% | 100.00% | +48.71pp | 通过 |
| 未知拒答率 (门槛 0.50) | V2-512 v3 final | 51.29% | 100.00% | +48.71pp | 通过 |
| 未知拒答率 (门槛 0.50) | V2-512 v3 final | 51.29% | 99.81% | +48.51pp | 通过 |
| 未知拒答率 (门槛 0.50) | V2-512 v3 final | 51.29% | 100.00% | +48.71pp | 通过 |
| 已知问题被误拒率 | V2-128 deployed(v3) | 0.18% | 0.00% | -0.18pp | 通过 |
| 已知问题被误拒率 | V2-128 deployed(v3) | 0.18% | 0.00% | -0.18pp | 通过 |
| 已知问题被误拒率 | V2-128 deployed(v3) | 0.18% | 0.00% | -0.18pp | 通过 |
| 已知问题被误拒率 | V2-128 deployed(v3) | 0.18% | 0.00% | -0.18pp | 通过 |
| 未知问题被误读率 | V2-512 v3 final | 48.71% | 0.00% | -48.71pp | 通过 |
| 未知问题被误读率 | V2-512 v3 final | 48.71% | 0.00% | -48.71pp | 通过 |
| 未知问题被误读率 | V2-512 v3 final | 48.71% | 0.19% | -48.51pp | 通过 |
| 未知问题被误读率 | V2-512 v3 final | 48.71% | 0.00% | -48.71pp | 通过 |
| 仲裁准确率 | V2-512 v3 final | 81.75% | 100.00% | +18.25pp | 通过 |
| 仲裁准确率 | V2-512 v3 final | 81.75% | 100.00% | +18.25pp | 通过 |
| 仲裁准确率 | V2-512 v3 final | 81.75% | 99.96% | +18.21pp | 通过 |
| 仲裁准确率 | V2-512 v3 final | 81.75% | 100.00% | +18.25pp | 通过 |
| 平均分数余量 | V2-512 v3 final | 53.1605 | 45.5641 | -7.5964 | **未通过** |
| 平均分数余量 | V2-512 v3 final | 53.1605 | 215.7533 | +162.5928 | 通过 |
| 平均分数余量 | V2-512 v3 final | 53.1605 | 12.3693 | -40.7912 | **未通过** |
| 平均分数余量 | V2-512 v3 final | 53.1605 | 73.2482 | +20.0877 | 通过 |
| 单查询延迟 ms (GPU) | V2-128 deployed(v3) | 0.8439 | 1.2161 | +0.3722 | **未通过** |
| 单查询延迟 ms (GPU) | V2-128 deployed(v3) | 0.8439 | 1.2505 | +0.4066 | **未通过** |
| 单查询延迟 ms (GPU) | V2-128 deployed(v3) | 0.8439 | 0.8750 | +0.0311 | **未通过** |
| 单查询延迟 ms (GPU) | V2-128 deployed(v3) | 0.8439 | 0.8725 | +0.0286 | **未通过** |
| 路由 QPS (GPU, 单查询) | V2-128 deployed(v3) | 1184.9399 | 822.2900 | -362.6499 | **未通过** |
| 路由 QPS (GPU, 单查询) | V2-128 deployed(v3) | 1184.9399 | 799.6682 | -385.2717 | **未通过** |
| 路由 QPS (GPU, 单查询) | V2-128 deployed(v3) | 1184.9399 | 1142.8110 | -42.1289 | **未通过** |
| 路由 QPS (GPU, 单查询) | V2-128 deployed(v3) | 1184.9399 | 1146.1213 | -38.8186 | **未通过** |
| 批量 QPS (GPU, batch=64) | V2-128 deployed(v3) | 72916.1919 | 57386.7509 | -15529.4411 | **未通过** |
| 批量 QPS (GPU, batch=64) | V2-128 deployed(v3) | 72916.1919 | 53291.5883 | -19624.6036 | **未通过** |
| 批量 QPS (GPU, batch=64) | V2-128 deployed(v3) | 72916.1919 | 73617.3737 | +701.1818 | 通过 |
| 批量 QPS (GPU, batch=64) | V2-128 deployed(v3) | 72916.1919 | 61636.8434 | -11279.3485 | **未通过** |
| 批量 QPS (GPU, batch=256) | V2-128 deployed(v3) | 264812.9758 | 96746.8860 | -168066.0899 | **未通过** |
| 批量 QPS (GPU, batch=256) | V2-128 deployed(v3) | 264812.9758 | 94345.8808 | -170467.0950 | **未通过** |
| 批量 QPS (GPU, batch=256) | V2-128 deployed(v3) | 264812.9758 | 161941.2710 | -102871.7048 | **未通过** |
| 批量 QPS (GPU, batch=256) | V2-128 deployed(v3) | 264812.9758 | 164167.8103 | -100645.1656 | **未通过** |
| 地址字节/记录 | V2-128 deployed(v3) | 512.0000 | 2048.0000 | +1536.0000 | **未通过** |
| 地址字节/记录 | V2-128 deployed(v3) | 512.0000 | 2048.0000 | +1536.0000 | **未通过** |
| 地址字节/记录 | V2-128 deployed(v3) | 512.0000 | 2048.0000 | +1536.0000 | **未通过** |
| 地址字节/记录 | V2-128 deployed(v3) | 512.0000 | 2048.0000 | +1536.0000 | **未通过** |
**66 项通过 / 22 项未通过 / 0 项无数据 —— 全方位超越：否**
Reference in New Issue View Git Blame Copy Permalink