wpy/natural-memory-nm21
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
643e22ecb905392eedef8b52ce7c9d0c7f185ed1
natural-memory-nm21/router_critical_e2e_replay.json
T
WpyQwq 643e22ecb9 Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据 
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线
- 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同）
- 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00）
- 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00%
- 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益）
- 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
2026-09-19 11:11:31 +08:00

680 lines
24 KiB
JSON
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
{
"deployed": {
"summary": {
"router": "deployed",
"cases": 16,
"accuracy_pct": 25.0,
"wrong_attribute_pct": 18.75,
"read_pct": 68.75,
"cases_with_rule_hits": 6,
"stop_reasons": {
"router_abstained": 5,
"evidence_found": 11
},
"mean_records_selected": 3.875,
"seconds": 73.7
},
"rows": [
{
"attribute": "常住城市",
"wanted": "CITY-A1B2C3",
"reply": "我平时待得最久的地方是 3 号楼。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "router_abstained",
"need_memory": false,
"top_score": 0.0,
"records_selected": 0
},
{
"attribute": "出生城市",
"wanted": "CITY-D4E5F6",
"reply": "关于您老家的具体位置，当前长期记忆中未包含相关证据，因此无法回答。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 2.6641891400334443,
"records_selected": 8
},
{
"attribute": "办公城市",
"wanted": "CITY-G7H8J9",
"reply": "您的办公楼层是 DESK-U7V8W9。user；办公楼层；DESK-U7V8W9",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 12,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.520044145734458,
"records_selected": 8
},
{
"attribute": "档案代号",
"wanted": "CODE-K1L2M3",
"reply": "系统分配给您的字符包括：user；紧急联系人姓氏；KIN-X1Y2Z3；user；主管姓名；BOSS-X4Y5Z6；user；笔记本电脑品牌；ThinkPad；user；手机尾",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 4.093658475922178,
"records_selected": 8
},
{
"attribute": "常用编辑器",
"wanted": "EDIT-N4P5Q6",
"reply": "您常用的代码编辑器是 EDIT-N4P5Q6。",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 2,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.515892826960929,
"records_selected": 2
},
{
"attribute": "默认语言",
"wanted": "LANG-R7S8T9",
"reply": "关于您平时讲哪种话，当前长期记忆中未包含此信息。已知事实仅涉及您的水杯是蓝色的（user；水杯；蓝色的）以及您的鼠标是无线的（user；鼠标；无线的），无法推断您使用的",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "router_abstained",
"need_memory": false,
"top_score": 0.0,
"records_selected": 0
},
{
"attribute": "通勤方式",
"wanted": "COMMU-U1V2W3",
"reply": "关于您早上怎么去公司，当前长期记忆中未包含相关信息，无法回答。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "router_abstained",
"need_memory": false,
"top_score": 0.0,
"records_selected": 0
},
{
"attribute": "主管姓名",
"wanted": "BOSS-X4Y5Z6",
"reply": "这句话听起来带着几分疲惫、迷茫，或者是对现状的无力感。能感受到你此刻可能正经历一些难以独自应对的困境。\n\n“谁管我？”这个问题背后，或许藏着几个不同的需求，我想为你拆解一下",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "router_abstained",
"need_memory": false,
"top_score": 0.0,
"records_selected": 0
},
{
"attribute": "工位楼层",
"wanted": "FLOOR-A7B8C9",
"reply": "我坐在 3 号楼。关于具体楼层信息，当前记忆中没有明确记录，无法回答。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "router_abstained",
"need_memory": false,
"top_score": 0.0,
"records_selected": 0
},
{
"attribute": "团队名称",
"wanted": "TEAM-D1E2F3",
"reply": "关于您所属的小组，当前长期记忆中未包含相关信息。已知信息包括：user；小组；不知道。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.7658000497788082,
"records_selected": 8
},
{
"attribute": "邮箱域名",
"wanted": "MAIL-G4H5J6",
"reply": "关于别人发信需要写的后缀，当前记忆中没有相关信息，无法回答。已知信息仅包含：user；紧急联系人姓氏；KIN-X1Y2Z3。",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 1,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.513982709418043,
"records_selected": 1
},
{
"attribute": "手机尾号",
"wanted": "PHONE-K7L8M9",
"reply": "您的电话号码最后几位是 PHONE-K7L8M9。user；手机尾号；PHONE-K7L8M9",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 4.0849550882345795,
"records_selected": 8
},
{
"attribute": "项目代号",
"wanted": "PROJ-N1P2Q3",
"reply": "我不知道您正在做的工程叫什么，因为当前记忆中没有关于您正在进行的工程项目的名称、路径或任何相关证据。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 1,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.370942695414584,
"records_selected": 1
},
{
"attribute": "入职年份",
"wanted": "YEAR-R4S5T6",
"reply": "关于您加入公司的年份，当前长期记忆中未包含此信息，无法回答。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.1999297104710993,
"records_selected": 8
},
{
"attribute": "办公楼层",
"wanted": "DESK-U7V8W9",
"reply": "您的办公楼层是 DESK-U7V8W9。",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 2,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.524232190583632,
"records_selected": 2
},
{
"attribute": "紧急联系人姓氏",
"wanted": "KIN-X1Y2Z3",
"reply": "出事时该找 KIN-X1Y2Z3 家。",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 12,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.522833668863877,
"records_selected": 8
}
]
},
"V2-128-v6": {
"summary": {
"router": "V2-128-v6",
"cases": 16,
"accuracy_pct": 37.5,
"wrong_attribute_pct": 25.0,
"read_pct": 93.75,
"cases_with_rule_hits": 6,
"stop_reasons": {
"evidence_found": 15,
"router_abstained": 1
},
"mean_records_selected": 5.875,
"seconds": 72.6
},
"rows": [
{
"attribute": "常住城市",
"wanted": "CITY-A1B2C3",
"reply": "我平时待得最久的地方是 3 号楼。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.3846491147015145,
"records_selected": 8
},
{
"attribute": "出生城市",
"wanted": "CITY-D4E5F6",
"reply": "关于您老家的具体位置，当前长期记忆中未包含相关证据，因此无法回答。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 2.6641891400334443,
"records_selected": 8
},
{
"attribute": "办公城市",
"wanted": "CITY-G7H8J9",
"reply": "您的办公楼层是 DESK-U7V8W9。user；办公楼层；DESK-U7V8W9",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 12,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.520044145734458,
"records_selected": 8
},
{
"attribute": "档案代号",
"wanted": "CODE-K1L2M3",
"reply": "系统分配给您的字符包括：user；紧急联系人姓氏；KIN-X1Y2Z3；user；主管姓名；BOSS-X4Y5Z6；user；笔记本电脑品牌；ThinkPad；user；手机尾",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 4.093658475922178,
"records_selected": 8
},
{
"attribute": "常用编辑器",
"wanted": "EDIT-N4P5Q6",
"reply": "您常用的代码编辑器是 EDIT-N4P5Q6。",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 2,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.515892826960929,
"records_selected": 2
},
{
"attribute": "默认语言",
"wanted": "LANG-R7S8T9",
"reply": "我的默认语言是 LANG-R7S8T9。user；默认语言；LANG-R7S8T9",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 2.865009269951279,
"records_selected": 8
},
{
"attribute": "通勤方式",
"wanted": "COMMU-U1V2W3",
"reply": "您的通勤方式是 COMMU-U1V2W3。",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.1957691736318106,
"records_selected": 8
},
{
"attribute": "主管姓名",
"wanted": "BOSS-X4Y5Z6",
"reply": "这句话听起来带着几分疲惫、迷茫，或者是对现状的无力感。能感受到你此刻可能正经历一些难以独自应对的困境。\n\n“谁管我？”这个问题背后，或许藏着几个不同的需求，我想为你拆解一下",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "router_abstained",
"need_memory": false,
"top_score": 0.0,
"records_selected": 0
},
{
"attribute": "工位楼层",
"wanted": "FLOOR-A7B8C9",
"reply": "您的办公楼层是 DESK-U7V8W9。user；办公楼层；DESK-U7V8W9",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 4.038213959597986,
"records_selected": 8
},
{
"attribute": "团队名称",
"wanted": "TEAM-D1E2F3",
"reply": "关于您所属的小组，当前长期记忆中未包含相关信息。已知信息包括：user；小组；不知道。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.7658000497788082,
"records_selected": 8
},
{
"attribute": "邮箱域名",
"wanted": "MAIL-G4H5J6",
"reply": "关于别人发信需要写的后缀，当前记忆中没有相关信息，无法回答。已知信息仅包含：user；紧急联系人姓氏；KIN-X1Y2Z3。",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 1,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.513982709418043,
"records_selected": 1
},
{
"attribute": "手机尾号",
"wanted": "PHONE-K7L8M9",
"reply": "您的电话号码最后几位是 PHONE-K7L8M9。user；手机尾号；PHONE-K7L8M9",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 4.0849550882345795,
"records_selected": 8
},
{
"attribute": "项目代号",
"wanted": "PROJ-N1P2Q3",
"reply": "我不知道您正在做的工程叫什么，因为当前记忆中没有关于您正在进行的工程项目的名称、路径或任何相关证据。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 1,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.370942695414584,
"records_selected": 1
},
{
"attribute": "入职年份",
"wanted": "YEAR-R4S5T6",
"reply": "关于您加入公司的年份，当前长期记忆中未包含此信息，无法回答。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.1999297104710993,
"records_selected": 8
},
{
"attribute": "办公楼层",
"wanted": "DESK-U7V8W9",
"reply": "您的办公楼层是 DESK-U7V8W9。",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 2,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.524232190583632,
"records_selected": 2
},
{
"attribute": "紧急联系人姓氏",
"wanted": "KIN-X1Y2Z3",
"reply": "出事时该找 KIN-X1Y2Z3 家。",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 12,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.522833668863877,
"records_selected": 8
}
]
},
"REPLAY-128": {
"summary": {
"router": "REPLAY-128",
"cases": 16,
"accuracy_pct": 37.5,
"wrong_attribute_pct": 25.0,
"read_pct": 93.75,
"cases_with_rule_hits": 6,
"stop_reasons": {
"evidence_found": 15,
"router_abstained": 1
},
"mean_records_selected": 5.875,
"seconds": 72.5
},
"rows": [
{
"attribute": "常住城市",
"wanted": "CITY-A1B2C3",
"reply": "我平时待得最久的地方是 3 号楼。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.3846491147015145,
"records_selected": 8
},
{
"attribute": "出生城市",
"wanted": "CITY-D4E5F6",
"reply": "关于您老家的具体位置，当前长期记忆中未包含相关证据，因此无法回答。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 2.6641891400334443,
"records_selected": 8
},
{
"attribute": "办公城市",
"wanted": "CITY-G7H8J9",
"reply": "您的办公楼层是 DESK-U7V8W9。user；办公楼层；DESK-U7V8W9",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 12,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.520044145734458,
"records_selected": 8
},
{
"attribute": "档案代号",
"wanted": "CODE-K1L2M3",
"reply": "系统分配给您的字符包括：user；紧急联系人姓氏；KIN-X1Y2Z3；user；主管姓名；BOSS-X4Y5Z6；user；笔记本电脑品牌；ThinkPad；user；手机尾",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 4.093658475922178,
"records_selected": 8
},
{
"attribute": "常用编辑器",
"wanted": "EDIT-N4P5Q6",
"reply": "您常用的代码编辑器是 EDIT-N4P5Q6。",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 2,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.515892826960929,
"records_selected": 2
},
{
"attribute": "默认语言",
"wanted": "LANG-R7S8T9",
"reply": "我的默认语言是 LANG-R7S8T9。user；默认语言；LANG-R7S8T9",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 2.865009269951279,
"records_selected": 8
},
{
"attribute": "通勤方式",
"wanted": "COMMU-U1V2W3",
"reply": "您的通勤方式是 COMMU-U1V2W3。",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.1957691736318106,
"records_selected": 8
},
{
"attribute": "主管姓名",
"wanted": "BOSS-X4Y5Z6",
"reply": "这句话听起来带着几分疲惫、迷茫，或者是对现状的无力感。能感受到你此刻可能正经历一些难以独自应对的困境。\n\n“谁管我？”这个问题背后，或许藏着几个不同的需求，我想为你拆解一下",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "router_abstained",
"need_memory": false,
"top_score": 0.0,
"records_selected": 0
},
{
"attribute": "工位楼层",
"wanted": "FLOOR-A7B8C9",
"reply": "您的办公楼层是 DESK-U7V8W9。user；办公楼层；DESK-U7V8W9",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 4.038213959597986,
"records_selected": 8
},
{
"attribute": "团队名称",
"wanted": "TEAM-D1E2F3",
"reply": "关于您所属的小组，当前长期记忆中未包含相关信息。已知信息包括：user；小组；不知道。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.7658000497788082,
"records_selected": 8
},
{
"attribute": "邮箱域名",
"wanted": "MAIL-G4H5J6",
"reply": "关于别人发信需要写的后缀，当前记忆中没有相关信息，无法回答。已知信息仅包含：user；紧急联系人姓氏；KIN-X1Y2Z3。",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 1,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.513982709418043,
"records_selected": 1
},
{
"attribute": "手机尾号",
"wanted": "PHONE-K7L8M9",
"reply": "您的电话号码最后几位是 PHONE-K7L8M9。user；手机尾号；PHONE-K7L8M9",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 4.0849550882345795,
"records_selected": 8
},
{
"attribute": "项目代号",
"wanted": "PROJ-N1P2Q3",
"reply": "我不知道您正在做的工程叫什么，因为当前记忆中没有关于您正在进行的工程项目的名称、路径或任何相关证据。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 1,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.370942695414584,
"records_selected": 1
},
{
"attribute": "入职年份",
"wanted": "YEAR-R4S5T6",
"reply": "关于您加入公司的年份，当前长期记忆中未包含此信息，无法回答。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.1999297104710993,
"records_selected": 8
},
{
"attribute": "办公楼层",
"wanted": "DESK-U7V8W9",
"reply": "您的办公楼层是 DESK-U7V8W9。",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 2,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.524232190583632,
"records_selected": 2
},
{
"attribute": "紧急联系人姓氏",
"wanted": "KIN-X1Y2Z3",
"reply": "出事时该找 KIN-X1Y2Z3 家。",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 12,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.522833668863877,
"records_selected": 8
}
]
}
}
Reference in New Issue View Git Blame Copy Permalink