wpy/natural-memory-nm21
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
natural-memory-nm21/router_critical_e2e_scramble_topk1.json
T
WpyQwq 643e22ecb9 Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据 
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线
- 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同）
- 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00）
- 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00%
- 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益）
- 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
2026-09-19 11:11:31 +08:00

454 lines
17 KiB
JSON
Raw Permalink Blame History

{
"V2-128-v6": {
"summary": {
"router": "V2-128-v6",
"cases": 16,
"accuracy_pct": 12.5,
"wrong_attribute_pct": 43.75,
"read_pct": 93.75,
"cases_with_rule_hits": 6,
"stop_reasons": {
"evidence_found": 15,
"router_abstained": 1
},
"mean_records_selected": 0.9375,
"seconds": 75.1
},
"rows": [
{
"attribute": "常住城市",
"wanted": "CITY-A1B2C3",
"reply": "关于您平时待得最久的地方，当前长期记忆中未包含此信息，仅确认了您的通勤方式是 COMMU-U1V2W3。",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.3846491147015145,
"records_selected": 1
},
{
"attribute": "出生城市",
"wanted": "CITY-D4E5F6",
"reply": "关于您老家的具体位置，当前已保存的记忆中未包含相关信息，仅确认了您的工位在 3 号楼。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 2.6641891400334443,
"records_selected": 1
},
{
"attribute": "办公城市",
"wanted": "CITY-G7H8J9",
"reply": "您每天上班要去的地方未知，因为关于您的通勤方式已保存为 COMMU-U1V2W3，但未包含具体的目的地信息。",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 12,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.520044145734458,
"records_selected": 1
},
{
"attribute": "档案代号",
"wanted": "CODE-K1L2M3",
"reply": "系统分配的那串字符是 KIN-X1Y2Z3。\nuser；紧急联系人姓氏；KIN-X1Y2Z3",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 4.093658475922178,
"records_selected": 1
},
{
"attribute": "常用编辑器",
"wanted": "EDIT-N4P5Q6",
"reply": "关于您写代码使用的软件，当前记忆中没有相关记录。已知信息：user；项目代号；PROJ-N1P2Q3。",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 2,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.515892826960929,
"records_selected": 1
},
{
"attribute": "默认语言",
"wanted": "LANG-R7S8T9",
"reply": "关于您平时讲哪种话，当前长期记忆中没有相关记录。已知事实仅涉及：user；水杯；蓝色的。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 2.865009269951279,
"records_selected": 1
},
{
"attribute": "通勤方式",
"wanted": "COMMU-U1V2W3",
"reply": "关于您早上如何前往公司，当前记忆中没有包含交通方式、路线或通勤习惯的具体信息。已知信息仅显示您的办公楼层是 DESK-U7V8W9。",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.1957691736318106,
"records_selected": 1
},
{
"attribute": "主管姓名",
"wanted": "BOSS-X4Y5Z6",
"reply": "这句话听起来带着几分疲惫、迷茫，或者是对现状的无力感。能感受到你此刻可能正经历一些难以独自应对的困境。\n\n“谁管我？”这个问题背后，或许藏着几个不同的需求，我想为你拆解一下",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "router_abstained",
"need_memory": false,
"top_score": 0.0,
"records_selected": 0
},
{
"attribute": "工位楼层",
"wanted": "FLOOR-A7B8C9",
"reply": "我坐在 3 号楼。关于具体楼层信息，当前记忆中没有明确记录，无法回答。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 4.038213959597986,
"records_selected": 1
},
{
"attribute": "团队名称",
"wanted": "TEAM-D1E2F3",
"reply": "关于您属于哪个小组，当前长期记忆中没有相关信息。已知事实仅涉及：user；水杯；蓝色的。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.7658000497788082,
"records_selected": 1
},
{
"attribute": "邮箱域名",
"wanted": "MAIL-G4H5J6",
"reply": "关于别人发信需要写的后缀，当前记忆中没有相关信息，无法回答。已知信息仅包含：user；紧急联系人姓氏；KIN-X1Y2Z3。",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 1,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.513982709418043,
"records_selected": 1
},
{
"attribute": "手机尾号",
"wanted": "PHONE-K7L8M9",
"reply": "我不知道您的电话号码最后几位是什么，因为当前长期记忆中仅包含“我的工位在 3 号楼”这一信息，未涉及电话号码相关数据。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 4.0849550882345795,
"records_selected": 1
},
{
"attribute": "项目代号",
"wanted": "PROJ-N1P2Q3",
"reply": "我不知道您正在做的工程叫什么，因为当前记忆中没有关于您正在进行的工程项目的名称、路径或任何相关证据。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 1,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.370942695414584,
"records_selected": 1
},
{
"attribute": "入职年份",
"wanted": "YEAR-R4S5T6",
"reply": "关于您加入公司的具体年份，当前长期记忆中未包含该信息。已知事实：user；办公楼层；DESK-U7V8W9。",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.1999297104710993,
"records_selected": 1
},
{
"attribute": "办公楼层",
"wanted": "DESK-U7V8W9",
"reply": "您平时在 DESK-U7V8W9 办公。",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 2,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.524232190583632,
"records_selected": 1
},
{
"attribute": "紧急联系人姓氏",
"wanted": "KIN-X1Y2Z3",
"reply": "出事时该找姓 KIN-X1Y2Z3 的家人。",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 12,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.522833668863877,
"records_selected": 1
}
]
},
"SCRAMBLED-ranking": {
"summary": {
"router": "SCRAMBLED-ranking",
"cases": 16,
"accuracy_pct": 12.5,
"wrong_attribute_pct": 43.75,
"read_pct": 93.75,
"cases_with_rule_hits": 6,
"stop_reasons": {
"evidence_found": 15,
"router_abstained": 1
},
"mean_records_selected": 0.9375,
"seconds": 74.1
},
"rows": [
{
"attribute": "常住城市",
"wanted": "CITY-A1B2C3",
"reply": "关于您平时待得最久的地方，当前长期记忆中未包含此信息，仅确认了您的通勤方式是 COMMU-U1V2W3。",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.3846491147015145,
"records_selected": 1
},
{
"attribute": "出生城市",
"wanted": "CITY-D4E5F6",
"reply": "关于您老家的具体位置，当前已保存的记忆中未包含相关信息，仅确认了您的工位在 3 号楼。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 2.6641891400334443,
"records_selected": 1
},
{
"attribute": "办公城市",
"wanted": "CITY-G7H8J9",
"reply": "您每天上班要去的地方未知，因为关于您的通勤方式已保存为 COMMU-U1V2W3，但未包含具体的目的地信息。",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 12,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.520044145734458,
"records_selected": 1
},
{
"attribute": "档案代号",
"wanted": "CODE-K1L2M3",
"reply": "系统分配的那串字符是 KIN-X1Y2Z3。\nuser；紧急联系人姓氏；KIN-X1Y2Z3",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 4.093658475922178,
"records_selected": 1
},
{
"attribute": "常用编辑器",
"wanted": "EDIT-N4P5Q6",
"reply": "关于您写代码使用的软件，当前记忆中没有相关记录。已知信息：user；项目代号；PROJ-N1P2Q3。",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 2,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.515892826960929,
"records_selected": 1
},
{
"attribute": "默认语言",
"wanted": "LANG-R7S8T9",
"reply": "关于您平时讲哪种话，当前长期记忆中没有相关记录。已知事实仅涉及：user；水杯；蓝色的。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 2.865009269951279,
"records_selected": 1
},
{
"attribute": "通勤方式",
"wanted": "COMMU-U1V2W3",
"reply": "关于您早上如何前往公司，当前记忆中没有包含交通方式、路线或通勤习惯的具体信息。已知信息仅显示您的办公楼层是 DESK-U7V8W9。",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.1957691736318106,
"records_selected": 1
},
{
"attribute": "主管姓名",
"wanted": "BOSS-X4Y5Z6",
"reply": "这句话听起来带着几分疲惫、迷茫，或者是对现状的无力感。能感受到你此刻可能正经历一些难以独自应对的困境。\n\n“谁管我？”这个问题背后，或许藏着几个不同的需求，我想为你拆解一下",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "router_abstained",
"need_memory": false,
"top_score": 0.0,
"records_selected": 0
},
{
"attribute": "工位楼层",
"wanted": "FLOOR-A7B8C9",
"reply": "我坐在 3 号楼。关于具体楼层信息，当前记忆中没有明确记录，无法回答。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 4.038213959597986,
"records_selected": 1
},
{
"attribute": "团队名称",
"wanted": "TEAM-D1E2F3",
"reply": "关于您属于哪个小组，当前长期记忆中没有相关信息。已知事实仅涉及：user；水杯；蓝色的。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.7658000497788082,
"records_selected": 1
},
{
"attribute": "邮箱域名",
"wanted": "MAIL-G4H5J6",
"reply": "关于别人发信需要写的后缀，当前记忆中没有相关信息，无法回答。已知信息仅包含：user；紧急联系人姓氏；KIN-X1Y2Z3。",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 1,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.513982709418043,
"records_selected": 1
},
{
"attribute": "手机尾号",
"wanted": "PHONE-K7L8M9",
"reply": "我不知道您的电话号码最后几位是什么，因为当前长期记忆中仅包含“我的工位在 3 号楼”这一信息，未涉及电话号码相关数据。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 4.0849550882345795,
"records_selected": 1
},
{
"attribute": "项目代号",
"wanted": "PROJ-N1P2Q3",
"reply": "我不知道您正在做的工程叫什么，因为当前记忆中没有关于您正在进行的工程项目的名称、路径或任何相关证据。",
"correct": false,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 1,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.370942695414584,
"records_selected": 1
},
{
"attribute": "入职年份",
"wanted": "YEAR-R4S5T6",
"reply": "关于您加入公司的具体年份，当前长期记忆中未包含该信息。已知事实：user；办公楼层；DESK-U7V8W9。",
"correct": false,
"wrong_attribute_code": true,
"rule_address_hit": false,
"rule_lexical_hits": 0,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 3.1999297104710993,
"records_selected": 1
},
{
"attribute": "办公楼层",
"wanted": "DESK-U7V8W9",
"reply": "您平时在 DESK-U7V8W9 办公。",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 2,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.524232190583632,
"records_selected": 1
},
{
"attribute": "紧急联系人姓氏",
"wanted": "KIN-X1Y2Z3",
"reply": "出事时该找姓 KIN-X1Y2Z3 的家人。",
"correct": true,
"wrong_attribute_code": false,
"rule_address_hit": false,
"rule_lexical_hits": 12,
"stop_reason": "evidence_found",
"need_memory": true,
"top_score": 8.522833668863877,
"records_selected": 1
}
]
}
}
Reference in New Issue View Git Blame Copy Permalink