wpy/natural-memory-nm21
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
643e22ecb905392eedef8b52ce7c9d0c7f185ed1
natural-memory-nm21/router_verdict_prod.md
T
WpyQwq 643e22ecb9 Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据 
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线
- 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同）
- 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00）
- 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00%
- 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益）
- 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
2026-09-19 11:11:31 +08:00

103 lines
8.1 KiB
Markdown
Raw Blame History

| 指标 | 最强旧基线 | 基线值 | 候选值 | 差值 | 判定 |
|---|---|---:|---:|---:|---|
| Top-1 正确率 | V2-128 deployed(v3) | 41.12% | 94.14% | +53.01pp | 通过 |
| Top-1 正确率 | V2-128 deployed(v3) | 41.12% | 93.78% | +52.66pp | 通过 |
| Top-1 正确率 | V2-128 deployed(v3) | 41.12% | 93.81% | +52.68pp | 通过 |
| Top-1 正确率 | V2-128 deployed(v3) | 41.12% | 94.62% | +53.49pp | 通过 |
| Recall@1 | V2-128 deployed(v3) | 37.03% | 88.58% | +51.55pp | 通过 |
| Recall@1 | V2-128 deployed(v3) | 37.03% | 88.34% | +51.32pp | 通过 |
| Recall@1 | V2-128 deployed(v3) | 37.03% | 88.38% | +51.35pp | 通过 |
| Recall@1 | V2-128 deployed(v3) | 37.03% | 89.19% | +52.16pp | 通过 |
| Recall@3 | V2-128 deployed(v3) | 46.73% | 96.48% | +49.75pp | 通过 |
| Recall@3 | V2-128 deployed(v3) | 46.73% | 96.59% | +49.86pp | 通过 |
| Recall@3 | V2-128 deployed(v3) | 46.73% | 96.78% | +50.04pp | 通过 |
| Recall@3 | V2-128 deployed(v3) | 46.73% | 97.81% | +51.07pp | 通过 |
| Recall@5 | V2-128 deployed(v3) | 48.91% | 97.15% | +48.23pp | 通过 |
| Recall@5 | V2-128 deployed(v3) | 48.91% | 97.33% | +48.41pp | 通过 |
| Recall@5 | V2-128 deployed(v3) | 48.91% | 97.78% | +48.86pp | 通过 |
| Recall@5 | V2-128 deployed(v3) | 48.91% | 98.40% | +49.49pp | 通过 |
| MRR | V2-128 deployed(v3) | 47.69% | 95.77% | +48.08pp | 通过 |
| MRR | V2-128 deployed(v3) | 47.69% | 95.59% | +47.90pp | 通过 |
| MRR | V2-128 deployed(v3) | 47.69% | 95.70% | +48.01pp | 通过 |
| MRR | V2-128 deployed(v3) | 47.69% | 96.49% | +48.80pp | 通过 |
| nDCG@3 | V2-128 deployed(v3) | 44.22% | 95.37% | +51.15pp | 通过 |
| nDCG@3 | V2-128 deployed(v3) | 44.22% | 95.39% | +51.17pp | 通过 |
| nDCG@3 | V2-128 deployed(v3) | 44.22% | 95.53% | +51.31pp | 通过 |
| nDCG@3 | V2-128 deployed(v3) | 44.22% | 96.49% | +52.27pp | 通过 |
| 多跳证据全中(Top-3) | V2-128 deployed(v3) | 43.43% | 96.22% | +52.80pp | 通过 |
| 多跳证据全中(Top-3) | V2-128 deployed(v3) | 43.43% | 96.50% | +53.07pp | 通过 |
| 多跳证据全中(Top-3) | V2-128 deployed(v3) | 43.43% | 96.70% | +53.28pp | 通过 |
| 多跳证据全中(Top-3) | V2-128 deployed(v3) | 43.43% | 97.75% | +54.32pp | 通过 |
| 多跳证据全中(仅多正例) | V2-128 deployed(v3) | 37.15% | 95.45% | +58.30pp | 通过 |
| 多跳证据全中(仅多正例) | V2-128 deployed(v3) | 37.15% | 98.35% | +61.20pp | 通过 |
| 多跳证据全中(仅多正例) | V2-128 deployed(v3) | 37.15% | 98.75% | +61.60pp | 通过 |
| 多跳证据全中(仅多正例) | V2-128 deployed(v3) | 37.15% | 99.00% | +61.85pp | 通过 |
| hop 正确率 | V2-128 deployed(v3) | 73.23% | 100.00% | +26.77pp | 通过 |
| hop 正确率 | V2-128 deployed(v3) | 73.23% | 100.00% | +26.77pp | 通过 |
| hop 正确率 | V2-128 deployed(v3) | 73.23% | 100.00% | +26.77pp | 通过 |
| hop 正确率 | V2-128 deployed(v3) | 73.23% | 100.00% | +26.77pp | 通过 |
| hop 欠预测率 | V2-128 deployed(v3) | 4.68% | 0.00% | -4.68pp | 通过 |
| hop 欠预测率 | V2-128 deployed(v3) | 4.68% | 0.00% | -4.68pp | 通过 |
| hop 欠预测率 | V2-128 deployed(v3) | 4.68% | 0.00% | -4.68pp | 通过 |
| hop 欠预测率 | V2-128 deployed(v3) | 4.68% | 0.00% | -4.68pp | 通过 |
| need F1 (门槛 0.50) | V2-128 deployed(v3) | 89.58% | 100.00% | +10.42pp | 通过 |
| need F1 (门槛 0.50) | V2-128 deployed(v3) | 89.58% | 100.00% | +10.42pp | 通过 |
| need F1 (门槛 0.50) | V2-128 deployed(v3) | 89.58% | 100.00% | +10.42pp | 通过 |
| need F1 (门槛 0.50) | V2-128 deployed(v3) | 89.58% | 100.00% | +10.42pp | 通过 |
| need 召回 (门槛 0.50) | V2-128 deployed(v3) | 99.82% | 100.00% | +0.18pp | 通过 |
| need 召回 (门槛 0.50) | V2-128 deployed(v3) | 99.82% | 100.00% | +0.18pp | 通过 |
| need 召回 (门槛 0.50) | V2-128 deployed(v3) | 99.82% | 100.00% | +0.18pp | 通过 |
| need 召回 (门槛 0.50) | V2-128 deployed(v3) | 99.82% | 100.00% | +0.18pp | 通过 |
| 未知拒答率 (门槛 0.50) | V2-128 deployed(v3) | 0.00% | 100.00% | +100.00pp | 通过 |
| 未知拒答率 (门槛 0.50) | V2-128 deployed(v3) | 0.00% | 100.00% | +100.00pp | 通过 |
| 未知拒答率 (门槛 0.50) | V2-128 deployed(v3) | 0.00% | 100.00% | +100.00pp | 通过 |
| 未知拒答率 (门槛 0.50) | V2-128 deployed(v3) | 0.00% | 100.00% | +100.00pp | 通过 |
| 已知问题被误拒率 | V2-128 deployed(v3) | 0.18% | 0.00% | -0.18pp | 通过 |
| 已知问题被误拒率 | V2-128 deployed(v3) | 0.18% | 0.00% | -0.18pp | 通过 |
| 已知问题被误拒率 | V2-128 deployed(v3) | 0.18% | 0.00% | -0.18pp | 通过 |
| 已知问题被误拒率 | V2-128 deployed(v3) | 0.18% | 0.00% | -0.18pp | 通过 |
| 未知问题被误读率 | V2-128 deployed(v3) | 100.00% | 0.00% | -100.00pp | 通过 |
| 未知问题被误读率 | V2-128 deployed(v3) | 100.00% | 0.00% | -100.00pp | 通过 |
| 未知问题被误读率 | V2-128 deployed(v3) | 100.00% | 0.00% | -100.00pp | 通过 |
| 未知问题被误读率 | V2-128 deployed(v3) | 100.00% | 0.00% | -100.00pp | 通过 |
| 仲裁准确率 | V2-128 deployed(v3) | 81.12% | 100.00% | +18.88pp | 通过 |
| 仲裁准确率 | V2-128 deployed(v3) | 81.12% | 100.00% | +18.88pp | 通过 |
| 仲裁准确率 | V2-128 deployed(v3) | 81.12% | 100.00% | +18.88pp | 通过 |
| 仲裁准确率 | V2-128 deployed(v3) | 81.12% | 100.00% | +18.88pp | 通过 |
| 平均分数余量 | V2-128 deployed(v3) | -9.2989 | 40.1456 | +49.4445 | 通过 |
| 平均分数余量 | V2-128 deployed(v3) | -9.2989 | 21.8925 | +31.1914 | 通过 |
| 平均分数余量 | V2-128 deployed(v3) | -9.2989 | 18.4219 | +27.7208 | 通过 |
| 平均分数余量 | V2-128 deployed(v3) | -9.2989 | 14.9592 | +24.2581 | 通过 |
| 单查询延迟 ms (GPU) | V2-128 deployed(v3) | 0.9698 | 0.9359 | -0.0339 | 通过 |
| 单查询延迟 ms (GPU) | V2-128 deployed(v3) | 0.9698 | 0.9776 | +0.0078 | 通过 |
| 单查询延迟 ms (GPU) | V2-128 deployed(v3) | 0.9698 | 1.2315 | +0.2617 | **未通过** |
| 单查询延迟 ms (GPU) | V2-128 deployed(v3) | 0.9698 | 1.2807 | +0.3109 | **未通过** |
| 路由 QPS (GPU, 单查询) | V2-128 deployed(v3) | 1031.1135 | 1068.4453 | +37.3318 | 通过 |
| 路由 QPS (GPU, 单查询) | V2-128 deployed(v3) | 1031.1135 | 1022.9307 | -8.1828 | 通过 |
| 路由 QPS (GPU, 单查询) | V2-128 deployed(v3) | 1031.1135 | 812.0051 | -219.1084 | **未通过** |
| 路由 QPS (GPU, 单查询) | V2-128 deployed(v3) | 1031.1135 | 780.7949 | -250.3186 | **未通过** |
| 批量 QPS (GPU, batch=64) | V2-128 deployed(v3) | 64355.2410 | 65927.7268 | +1572.4858 | 通过 |
| 批量 QPS (GPU, batch=64) | V2-128 deployed(v3) | 64355.2410 | 73335.6250 | +8980.3841 | 通过 |
| 批量 QPS (GPU, batch=64) | V2-128 deployed(v3) | 64355.2410 | 49115.9135 | -15239.3274 | **未通过** |
| 批量 QPS (GPU, batch=64) | V2-128 deployed(v3) | 64355.2410 | 62429.2793 | -1925.9616 | 通过 |
| 批量 QPS (GPU, batch=256) | V2-128 deployed(v3) | 185281.6861 | 262171.5170 | +76889.8309 | 通过 |
| 批量 QPS (GPU, batch=256) | V2-128 deployed(v3) | 185281.6861 | 273837.7940 | +88556.1079 | 通过 |
| 批量 QPS (GPU, batch=256) | V2-128 deployed(v3) | 185281.6861 | 141564.7326 | -43716.9535 | **未通过** |
| 批量 QPS (GPU, batch=256) | V2-128 deployed(v3) | 185281.6861 | 201612.9032 | +16331.2171 | 通过 |
| 地址字节/记录 | V2-128 deployed(v3) | 512.0000 | 512.0000 | +0.0000 | 通过 |
| 地址字节/记录 | V2-128 deployed(v3) | 512.0000 | 512.0000 | +0.0000 | 通过 |
| 地址字节/记录 | V2-128 deployed(v3) | 512.0000 | 512.0000 | +0.0000 | 通过 |
| 地址字节/记录 | V2-128 deployed(v3) | 512.0000 | 512.0000 | +0.0000 | 通过 |
### 逐候选判定（全方位超越是对单个路由器说的）
| 路由器 | 通过 | 未通过 | 全方位超越 | 未通过的轴 |
|---|---:|---:|---|---|
| REPLAY-128 v7 | 22 | 0 | **是** | - |
| PROD-128 step20000 | 22 | 0 | **是** | - |
| PROD-128 step10000 | 18 | 4 | 否 | 单查询延迟 ms (GPU), 路由 QPS (GPU, 单查询), 批量 QPS (GPU, batch=64), 批量 QPS (GPU, batch=256) |
| V2-128 v6 best | 20 | 2 | 否 | 单查询延迟 ms (GPU), 路由 QPS (GPU, 单查询) |
**汇总 82 项通过 / 6 项未通过 / 0 项无数据**
Reference in New Issue View Git Blame Copy Permalink