Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线 - 修复 prepare_memory_router_dataset 候选池重建缺陷(mega 家族 3568x 加速,输出逐字节相同) - 修复 v5 被破坏的拒答与多跳标签(train 未知样本 319 -> 16319,multi_hop 平均正例 1.00 -> 2.00) - 同存储预算下 V2-128 v6 逐轴 22/22 通过:Top-1 41.12% -> 94.62%,未知拒答 0.00% -> 100.00% - 记录三条被实测推翻的显然优化(logits_to_keep=1 反而慢 55%、XL 容量未带来收益) - 记忆手术跨架构可移植性 14/14,读写关闭时与原生模型逐位相同
This commit is contained in:
@@ -0,0 +1,64 @@
|
||||
# 零重叠改写(zero-overlap paraphrase)阶段结果
|
||||
|
||||
所有速率均为百分比。数据来源为各阶段保存的 scorecard JSON,不引用滚动日志。
|
||||
|
||||
## 零重叠评测集(300 条,24 个**未见过的**改写问法,24 个同形候选,随机猜测 4.17%)
|
||||
|
||||
`基线` 含部署态路由器与 v6;`冻结策略重放` 为合并数据重放训练后的路由器。
|
||||
|
||||
| 指标 | V2-128 v6 best | V2-128 v6 final | V2-128 deployed(v3) | ZOV-128 final | REPLAY-128 final |
|
||||
|---|---|---|---|---|---|
|
||||
| Top-1 正确率 | 7.60% | 18.40% | 11.60% | 66.80% | 59.60% |
|
||||
| Recall@3 | 23.60% | 36.00% | 32.00% | 84.40% | 83.20% |
|
||||
| MRR | 22.88% | 35.07% | 29.48% | 76.95% | 73.06% |
|
||||
| 多跳证据全中(Top-3) | 23.60% | 36.00% | 32.00% | 84.40% | 83.20% |
|
||||
| hop 正确率 | 83.33% | 83.33% | 62.33% | 58.33% | 83.33% |
|
||||
| 未知拒答率 | 0.00% | 0.00% | 6.00% | 40.00% | 0.00% |
|
||||
| 已知问题被误拒率 | 0.00% | 0.00% | 15.20% | 38.00% | 0.00% |
|
||||
| 未知问题被误读率 | 100.00% | 100.00% | 94.00% | 60.00% | 100.00% |
|
||||
| 每条记录地址字节 | 512 | 512 | 512 | 512 | 512 |
|
||||
| 参数量 | 2,037,774 | 2,037,774 | 2,037,774 | 2,037,774 | 2,037,774 |
|
||||
| 单查询延迟中位数 ms (GPU) | 0.8446 | 0.8527 | 0.9079 | 0.8515 | 0.8460 |
|
||||
| 路由 QPS (GPU, 单查询) | 1153.8630 | 1124.9066 | 1086.5613 | 1128.3880 | 1148.2401 |
|
||||
| 批量 QPS (GPU, batch=64) | 61288.5926 | 75940.9559 | 71565.9526 | 76974.8870 | 72823.2671 |
|
||||
| 批量 QPS (GPU, batch=256) | 273229.8761 | 296056.4359 | 257898.1302 | 291924.1911 | 234140.6307 |
|
||||
|
||||
## v6 评测集(21,920 条,全部 22 轴)
|
||||
|
||||
`基线` 同时包含朴素微调(仅零重叠数据)以显示灾难性遗忘,以及 v6 原始路由器。
|
||||
|
||||
| 指标 | V2-128 v6 final | ZOV-128 final | REPLAY-128 final |
|
||||
|---|---|---|---|
|
||||
| Top-1 正确率 | 94.37% | 90.72% | 94.14% |
|
||||
| Recall@3 | 96.40% | 93.69% | 96.48% |
|
||||
| MRR | 95.97% | 93.50% | 95.77% |
|
||||
| 多跳证据全中(Top-3) | 95.95% | 91.96% | 96.22% |
|
||||
| hop 正确率 | 100.00% | 91.59% | 100.00% |
|
||||
| 未知拒答率 | 100.00% | 100.00% | 100.00% |
|
||||
| 已知问题被误拒率 | 0.00% | 13.85% | 0.00% |
|
||||
| 未知问题被误读率 | 0.00% | 0.00% | 0.00% |
|
||||
| 每条记录地址字节 | 512 | 512 | 512 |
|
||||
| 参数量 | 2,037,774 | 2,037,774 | 2,037,774 |
|
||||
| 单查询延迟中位数 ms (GPU) | 0.9020 | 0.9837 | 0.8828 |
|
||||
| 路由 QPS (GPU, 单查询) | 1031.8867 | 968.8902 | 1070.4178 |
|
||||
| 批量 QPS (GPU, batch=64) | 77868.3539 | 63339.9972 | 70303.4031 |
|
||||
| 批量 QPS (GPU, batch=256) | 275832.3455 | 251226.6928 | 244326.0991 |
|
||||
|
||||
### 基线 — 分类别 Top-1
|
||||
|
||||
| 类别 | V2-128 v6 final | ZOV-128 final | REPLAY-128 final |
|
||||
|---|---|---|---|
|
||||
| benchmark_qa | - | - | - |
|
||||
| mega_validation | 98.38% | 95.61% | - |
|
||||
| memory_policy | 59.40% | 49.68% | - |
|
||||
| native_memory | 52.34% | 13.08% | - |
|
||||
|
||||
### 冻结策略重放 — 分类别 Top-1
|
||||
|
||||
| 类别 | V2-128 v6 final | ZOV-128 final | REPLAY-128 final |
|
||||
|---|---|---|---|
|
||||
| benchmark_qa | - | - | - |
|
||||
| mega_validation | 98.38% | - | 98.22% |
|
||||
| memory_policy | 59.40% | - | 58.76% |
|
||||
| native_memory | 52.34% | - | 47.66% |
|
||||
|
||||
Reference in New Issue
Block a user