natural-memory-nm21/_regress_scorecard.md at main

Files

T

WpyQwq 643e22ecb9 Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据

- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线
- 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同）
- 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00）
- 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00%
- 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益）
- 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同

2026-09-19 11:11:31 +08:00

4.7 KiB

Raw Permalink Blame History

metric	V2-512 best(step1000)	V2-512 final(step100k)	XL-512 best
参数	4,741,902	4,741,902	7,898,127
router_dim	512	512	512
每条记录地址字节	2048	2048	2048
Top-1 正确率	67.12%	60.63%	60.85%
Recall@1	67.12%	60.63%	60.85%
Recall@3	88.77%	85.01%	92.97%
Recall@5	94.93%	90.13%	97.33%
MRR	79.08%	73.26%	75.98%
nDCG@3	80.00%	74.57%	79.28%
多跳证据全中(Top-3)	88.66%	84.90%	92.86%
hop 正确率	97.84%	99.69%	99.38%
hop 欠预测率	0.33%	0.33%	0.33%
need F1	98.92%	99.89%	99.62%
need 召回	99.78%	99.78%	99.78%
未知拒答率	66.04%	100.00%	90.57%
已知问题被误拒率	0.22%	0.22%	0.22%
未知问题被误读率	33.96%	0.00%	9.43%
仲裁准确率	97.94%	99.79%	99.28%
已知问题 need 概率均值	98.47%	99.78%	99.62%
未知问题 need 概率均值	25.36%	1.52%	13.50%
平均分数余量	0.5880	3.8837	6.7097
单查询延迟 ms(GPU)	-	-	-
路由 QPS(GPU)	-	-	-

family = `benchmark_qa`

metric	V2-512 best(step1000)	V2-512 final(step100k)	XL-512 best
episodes	64	64	64
Top-1 正确率	-	-	-
Recall@3	-	-	-
MRR	-	-	-
hop 正确率	100.00%	100.00%	100.00%
未知拒答率	100.00%	100.00%	100.00%
已知问题被误拒率	-	-	-
未知问题被误读率	0.00%	0.00%	0.00%
平均分数余量	-	-	-

family = `mega_validation`

metric	V2-512 best(step1000)	V2-512 final(step100k)	XL-512 best
episodes	20	20	20
Top-1 正确率	40.00%	35.00%	40.00%
Recall@3	80.00%	80.00%	85.00%
MRR	62.26%	55.24%	60.83%
hop 正确率	70.00%	70.00%	70.00%
未知拒答率	-	-	-
已知问题被误拒率	20.00%	20.00%	20.00%
未知问题被误读率	-	-	-
平均分数余量	-0.2963	-12.9083	-0.6136

family = `memory_policy`

metric	V2-512 best(step1000)	V2-512 final(step100k)	XL-512 best
episodes	1728	1728	1728
Top-1 正确率	69.89%	62.57%	62.39%
Recall@3	90.04%	85.94%	94.20%
MRR	81.00%	74.59%	77.16%
hop 正确率	98.96%	100.00%	99.83%
未知拒答率	14.29%	100.00%	76.19%
已知问题被误拒率	0.00%	0.00%	0.00%
未知问题被误读率	85.71%	0.00%	23.81%
平均分数余量	0.6516	5.3929	7.2586

family = `native_memory`

metric	V2-512 best(step1000)	V2-512 final(step100k)	XL-512 best
episodes	128	128	128
Top-1 正确率	28.04%	34.58%	40.19%
Recall@3	70.09%	71.03%	74.77%
MRR	51.73%	55.38%	59.96%
hop 正确率	85.94%	100.00%	97.66%
未知拒答率	14.29%	100.00%	76.19%
已知问题被误拒率	0.00%	0.00%	0.00%
未知问题被误读率	85.71%	0.00%	23.81%
平均分数余量	-0.2618	-17.0552	-0.6793

读取/拒答门槛曲线

run	threshold	need F1	need 召回	未知拒答率	已知问题被误拒率	未知被误读率	tp/tn/fp/fn
V2-512 best(step1000)	0.30	98.81%	99.78%	62.26%	0.22%	37.74%	1830/66/40/4
V2-512 best(step1000)	0.40	98.87%	99.78%	64.15%	0.22%	35.85%	1830/68/38/4
V2-512 best(step1000)	0.50	98.92%	99.78%	66.04%	0.22%	33.96%	1830/70/36/4
V2-512 best(step1000)	0.60	99.40%	99.78%	83.02%	0.22%	16.98%	1830/88/18/4
V2-512 best(step1000)	0.70	99.84%	99.78%	98.11%	0.22%	1.89%	1830/104/2/4
V2-512 final(step100k)	0.30	99.89%	99.78%	100.00%	0.22%	0.00%	1830/106/0/4
V2-512 final(step100k)	0.40	99.89%	99.78%	100.00%	0.22%	0.00%	1830/106/0/4
V2-512 final(step100k)	0.50	99.89%	99.78%	100.00%	0.22%	0.00%	1830/106/0/4
V2-512 final(step100k)	0.60	99.89%	99.78%	100.00%	0.22%	0.00%	1830/106/0/4
V2-512 final(step100k)	0.70	99.89%	99.78%	100.00%	0.22%	0.00%	1830/106/0/4
XL-512 best	0.30	99.19%	99.78%	75.47%	0.22%	24.53%	1830/80/26/4
XL-512 best	0.40	99.51%	99.78%	86.79%	0.22%	13.21%	1830/92/14/4
XL-512 best	0.50	99.62%	99.78%	90.57%	0.22%	9.43%	1830/96/10/4
XL-512 best	0.60	99.84%	99.78%	98.11%	0.22%	1.89%	1830/104/2/4
XL-512 best	0.70	99.89%	99.78%	100.00%	0.22%	0.00%	1830/106/0/4

4.7 KiB Raw Permalink Blame History

family = benchmark_qa

family = mega_validation

family = memory_policy

family = native_memory

读取/拒答门槛曲线

4.7 KiB

Raw Permalink Blame History

family = `benchmark_qa`

family = `mega_validation`

family = `memory_policy`

family = `native_memory`