Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线 - 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同） - 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00） - 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00% - 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益） - 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
This commit is contained in:
WpyQwq
@@ -0,0 +1,813 @@
|
||||
{
|
||||
"eval_file": "data/zero_overlap/eval.jsonl",
|
||||
"episodes": 48,
|
||||
"candidate_count": 24,
|
||||
"router": "checkpoints\\router_replay_v7_v2_128\\memory_router_v2.pt",
|
||||
"results": {
|
||||
"prior=1.00 blend=0.50": {
|
||||
"router": "prior=1.00 blend=0.50",
|
||||
"cases": 48,
|
||||
"answerable_cases": 40,
|
||||
"unknown_cases": 8,
|
||||
"accuracy_pct": 70.0,
|
||||
"wrong_attribute_pct": 27.5,
|
||||
"unknown_leak_pct": 0.0,
|
||||
"mean_records_selected": 5.208333333333333,
|
||||
"records_active_min": 23,
|
||||
"records_active_max": 24,
|
||||
"stop_reasons": {
|
||||
"evidence_found": 40,
|
||||
"attribute_not_covered": 8
|
||||
},
|
||||
"top_score_answerable": {
|
||||
"n": 40,
|
||||
"min": 0.84,
|
||||
"p10": 1.19,
|
||||
"p50": 2.73,
|
||||
"p90": 8.52,
|
||||
"max": 8.52
|
||||
},
|
||||
"top_score_unknown": {
|
||||
"n": 8,
|
||||
"min": 0.0,
|
||||
"p10": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"max": 0.0
|
||||
},
|
||||
"seconds": 302.5
|
||||
}
|
||||
},
|
||||
"rows": {
|
||||
"prior=1.00 blend=0.50": [
|
||||
{
|
||||
"id": "zero-overlap-000259",
|
||||
"attribute": "出生城市",
|
||||
"answerable": true,
|
||||
"expected": "VAL-M9VSJAEX",
|
||||
"reply": "您的出生城市是 VAL-M9VSJAEX。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 2.313681895152214,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000219",
|
||||
"attribute": "咖啡口味",
|
||||
"answerable": true,
|
||||
"expected": "VAL-M3W7MRJR",
|
||||
"reply": "关于您早上那杯咖啡的豆子偏好，当前记忆中没有相关记录。已知信息仅包含：user；午餐偏好；VAL-Y4Z5H2FG。",
|
||||
"correct": false,
|
||||
"wrong_attribute": true,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 1,
|
||||
"top_score": 8.515033109300843,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000193",
|
||||
"attribute": "午餐偏好",
|
||||
"answerable": true,
|
||||
"expected": "VAL-HG84H5TD",
|
||||
"reply": "您的午餐偏好是 user；午餐偏好；VAL-HG84H5TD。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 0.953831069469452,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000080",
|
||||
"attribute": "入职年份",
|
||||
"answerable": true,
|
||||
"expected": "VAL-GFT4HRX2",
|
||||
"reply": "关于您何时开始在这家公司上班，当前记忆中没有相关记录。已知信息：user；办公城市；VAL-TCNMN9HW。",
|
||||
"correct": false,
|
||||
"wrong_attribute": true,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 1,
|
||||
"top_score": 8.51429933919514,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000285",
|
||||
"attribute": "起床时间",
|
||||
"answerable": true,
|
||||
"expected": "VAL-4K3RQQKF",
|
||||
"reply": "我的闹钟设定时间是 VAL-4K3RQQKF。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 3.2757300050727958,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000246",
|
||||
"attribute": "常住城市",
|
||||
"answerable": false,
|
||||
"expected": "VAL-6BWJYDE4",
|
||||
"reply": "作为一个人工智能助手，我并没有“小时候”的概念，也无法知道您具体待过哪里。\n\n不过，如果您是想询问**您自己**小时候待的地方，或者想聊聊关于某个特定地点的童年回忆，我很乐意",
|
||||
"correct": false,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 23,
|
||||
"stop_reason": "attribute_not_covered",
|
||||
"records_selected": 0,
|
||||
"top_score": 0.0,
|
||||
"need_memory": false
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000204",
|
||||
"attribute": "阅读工具",
|
||||
"answerable": true,
|
||||
"expected": "VAL-ZRLSPU5V",
|
||||
"reply": "我翻书时依赖的长期记忆中没有关于“翻书”习惯的记录，因此无法回答您依赖什么。",
|
||||
"correct": false,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 2.989592991362775,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000078",
|
||||
"attribute": "常住城市",
|
||||
"answerable": true,
|
||||
"expected": "VAL-TQBY65U8",
|
||||
"reply": "您的常住城市是 VAL-TQBY65U8。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 1.4848385143280027,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000104",
|
||||
"attribute": "午餐偏好",
|
||||
"answerable": true,
|
||||
"expected": "VAL-SRFC842D",
|
||||
"reply": "您的午餐偏好是 user；午餐偏好；VAL-SRFC842D。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 1.1867529082298276,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000281",
|
||||
"attribute": "咖啡口味",
|
||||
"answerable": true,
|
||||
"expected": "VAL-3GP4STZX",
|
||||
"reply": "关于您早上饮用的咖啡偏好哪种豆子，当前记忆中没有相关记录。已知信息仅包含：user；午餐偏好；VAL-XNN7T8EK。",
|
||||
"correct": false,
|
||||
"wrong_attribute": true,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 1,
|
||||
"top_score": 8.515033109300843,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000106",
|
||||
"attribute": "咖啡口味",
|
||||
"answerable": false,
|
||||
"expected": "VAL-837DYYWV",
|
||||
"reply": "作为一个人工智能助手，我**无法直接访问您的设备、查看您的闹钟设置或获取您的实时时间信息**。\n\n要找到您的闹钟设定时间，您可以尝试以下方法：\n\n1. **查看手机/电脑屏幕",
|
||||
"correct": false,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 23,
|
||||
"stop_reason": "attribute_not_covered",
|
||||
"records_selected": 0,
|
||||
"top_score": 0.0,
|
||||
"need_memory": false
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000091",
|
||||
"attribute": "运动习惯",
|
||||
"answerable": true,
|
||||
"expected": "VAL-92MQHEDL",
|
||||
"reply": "您的运动习惯是 VAL-92MQHEDL。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 0.8434395718574524,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000088",
|
||||
"attribute": "设备型号",
|
||||
"answerable": true,
|
||||
"expected": "VAL-JY6S7EYF",
|
||||
"reply": "关于您使用的机器档次，当前长期记忆中未包含此信息，无法回答。已知信息仅涉及：user；常用编辑器；VAL-3KPKRM4J。",
|
||||
"correct": false,
|
||||
"wrong_attribute": true,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 1,
|
||||
"top_score": 8.516791512356487,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000244",
|
||||
"attribute": "午餐偏好",
|
||||
"answerable": false,
|
||||
"expected": "VAL-2VLHC26V",
|
||||
"reply": "您好！作为一个人工智能助手，我并没有直接服务于特定某一家公司或机构。我的设计目标是**为所有用户**提供通用、安全、有帮助的信息和服务。\n\n不过，如果您是在某个具体平台（如企业微信",
|
||||
"correct": false,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 23,
|
||||
"stop_reason": "attribute_not_covered",
|
||||
"records_selected": 0,
|
||||
"top_score": 0.0,
|
||||
"need_memory": false
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000282",
|
||||
"attribute": "咖啡口味",
|
||||
"answerable": true,
|
||||
"expected": "VAL-Q2AQQMUP",
|
||||
"reply": "关于您早上那杯咖啡的豆子偏好，当前记忆中没有相关记录。已知信息仅包含：user；午餐偏好；VAL-5ATEHVSM。",
|
||||
"correct": false,
|
||||
"wrong_attribute": true,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 1,
|
||||
"top_score": 8.515033109300843,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000056",
|
||||
"attribute": "主管姓名",
|
||||
"answerable": true,
|
||||
"expected": "VAL-VDL8H7Y9",
|
||||
"reply": "您的主管姓名是 VAL-VDL8H7Y9。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 2.049178622612938,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000184",
|
||||
"attribute": "项目代号",
|
||||
"answerable": true,
|
||||
"expected": "VAL-3H5TQEC4",
|
||||
"reply": "您的项目代号是 VAL-3H5TQEC4。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 3.4711107313126073,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000198",
|
||||
"attribute": "手机尾号",
|
||||
"answerable": true,
|
||||
"expected": "VAL-7T3XVMVH",
|
||||
"reply": "您随身那台设备的末端数字是 4N，该设备的完整型号为 VAL-9MWPDS4N。",
|
||||
"correct": false,
|
||||
"wrong_attribute": true,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 1,
|
||||
"top_score": 8.513663491707547,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000258",
|
||||
"attribute": "阅读工具",
|
||||
"answerable": true,
|
||||
"expected": "VAL-SHRY3GNG",
|
||||
"reply": "我翻书时依赖的实体是 user，属性是阅读工具，已确认值为 VAL-SHRY3GNG。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 2.9794869642364628,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000030",
|
||||
"attribute": "课程名称",
|
||||
"answerable": true,
|
||||
"expected": "VAL-BQY3K5EV",
|
||||
"reply": "您的课程名称是 VAL-BQY3K5EV。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 1.1944098997116086,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000196",
|
||||
"attribute": "手机尾号",
|
||||
"answerable": true,
|
||||
"expected": "VAL-ZHN2NMZV",
|
||||
"reply": "您随身那台设备的末端数字是 W9，该设备的完整型号为 VAL-5VBJ22W9。",
|
||||
"correct": false,
|
||||
"wrong_attribute": true,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 1,
|
||||
"top_score": 8.513663491707547,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000268",
|
||||
"attribute": "宿舍楼号",
|
||||
"answerable": true,
|
||||
"expected": "VAL-QVYMWTLK",
|
||||
"reply": "关于您睡觉的地儿是第几排，当前长期记忆中未包含此信息。已知信息如下：user；宿舍楼号；VAL-QVYMWTLK；常住城市；VAL-NW2E2ZWE；",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 2.7256882035478824,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000006",
|
||||
"attribute": "档案标识",
|
||||
"answerable": true,
|
||||
"expected": "VAL-8EFN3KGR",
|
||||
"reply": "您的档案标识是 VAL-8EFN3KGR。user；档案标识；VAL-8EFN3KGR",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 3.7116483740847537,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000047",
|
||||
"attribute": "咖啡口味",
|
||||
"answerable": true,
|
||||
"expected": "VAL-PW46K9K9",
|
||||
"reply": "关于您早上饮用的咖啡偏好哪种豆子，当前记忆中没有相关记录。已知信息仅包含您的午餐偏好是 VAL-BFW4LVTV。",
|
||||
"correct": false,
|
||||
"wrong_attribute": true,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 1,
|
||||
"top_score": 8.515033109300843,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000173",
|
||||
"attribute": "客户名称",
|
||||
"answerable": false,
|
||||
"expected": "VAL-P84HTKZR",
|
||||
"reply": "作为一个人工智能助手，我**无法知道您随身设备的型号或数字**，因为我没有访问您个人设备信息的权限，也无法看到您正在使用的具体设备。\n\n如果您指的是：\n- **手机/电脑型号**：",
|
||||
"correct": false,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 23,
|
||||
"stop_reason": "attribute_not_covered",
|
||||
"records_selected": 0,
|
||||
"top_score": 0.0,
|
||||
"need_memory": false
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000000",
|
||||
"attribute": "出生城市",
|
||||
"answerable": true,
|
||||
"expected": "VAL-PHB8FNF9",
|
||||
"reply": "您的出生城市是 VAL-PHB8FNF9。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 2.455323672186914,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000266",
|
||||
"attribute": "出生城市",
|
||||
"answerable": true,
|
||||
"expected": "VAL-K7FT5XFB",
|
||||
"reply": "您的出生城市是 VAL-K7FT5XFB。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 2.1730912274276046,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000051",
|
||||
"attribute": "项目代号",
|
||||
"answerable": false,
|
||||
"expected": "VAL-KG4RZYH5",
|
||||
"reply": "作为一个人工智能助手，我并没有实体，也没有“待得最久”的物理空间或地理位置。我的“存在”体现在每一次与您的对话中。\n\n不过，如果您是在寻找**您自己**待得最久的地方",
|
||||
"correct": false,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 23,
|
||||
"stop_reason": "attribute_not_covered",
|
||||
"records_selected": 0,
|
||||
"top_score": 0.0,
|
||||
"need_memory": false
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000061",
|
||||
"attribute": "常住城市",
|
||||
"answerable": true,
|
||||
"expected": "VAL-3SUNPNRD",
|
||||
"reply": "您的常住城市是 VAL-3SUNPNRD，办公城市是 VAL-XYRY8BUQ，出生城市是 VAL-GE828D22，通勤方式是 VAL-2CXTE3TB，入职年份",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 1.6610547590255735,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000214",
|
||||
"attribute": "客户名称",
|
||||
"answerable": true,
|
||||
"expected": "VAL-MV382EEF",
|
||||
"reply": "您的服务对象是 VAL-MV382EEF。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 1.2588214325904845,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000092",
|
||||
"attribute": "宿舍楼号",
|
||||
"answerable": true,
|
||||
"expected": "VAL-PLZZ659B",
|
||||
"reply": "关于您睡觉的地儿是第几排，当前长期记忆中未包含此信息。已知信息如下：user；宿舍楼号；VAL-PLZZ659B。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 2.809071942542481,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000140",
|
||||
"attribute": "宿舍楼号",
|
||||
"answerable": true,
|
||||
"expected": "VAL-RMDSB297",
|
||||
"reply": "关于您睡觉的地儿是第几排，当前长期记忆中未包含此信息。已知信息如下：user；宿舍楼号；VAL-RMDSB297。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 2.8702512824281925,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000164",
|
||||
"attribute": "客户名称",
|
||||
"answerable": true,
|
||||
"expected": "VAL-BB55YHW5",
|
||||
"reply": "您的客户名称是 VAL-BB55YHW5。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 1.2650055932998656,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000213",
|
||||
"attribute": "通勤方式",
|
||||
"answerable": true,
|
||||
"expected": "VAL-M58DDQX2",
|
||||
"reply": "您的通勤方式是 VAL-M58DDQX2。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 2.0363681795533712,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000122",
|
||||
"attribute": "通勤方式",
|
||||
"answerable": true,
|
||||
"expected": "VAL-VULSNGJ3",
|
||||
"reply": "您的通勤方式是 VAL-VULSNGJ3。user；通勤方式；VAL-VULSNGJ3",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 2.0330638098057325,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000168",
|
||||
"attribute": "常住城市",
|
||||
"answerable": true,
|
||||
"expected": "VAL-8TQ8737C",
|
||||
"reply": "您的常住城市是 VAL-8TQ8737C。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 1.4606248426437376,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000175",
|
||||
"attribute": "客户名称",
|
||||
"answerable": true,
|
||||
"expected": "VAL-QJNRRE8X",
|
||||
"reply": "您的客户名称是 VAL-QJNRRE8X。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 1.2628431367874144,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000144",
|
||||
"attribute": "入职年份",
|
||||
"answerable": false,
|
||||
"expected": "VAL-GLHQMRN6",
|
||||
"reply": "作为一个人工智能助手，我**无法知道您睡觉的具体位置**，因为我无法感知现实世界中的物理环境，也无法获取您的个人生活信息。\n\n如果您是在询问某个特定场所（如学校教室、电影院、酒店或",
|
||||
"correct": false,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 23,
|
||||
"stop_reason": "attribute_not_covered",
|
||||
"records_selected": 0,
|
||||
"top_score": 0.0,
|
||||
"need_memory": false
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000002",
|
||||
"attribute": "午餐偏好",
|
||||
"answerable": true,
|
||||
"expected": "VAL-HEE2QMH2",
|
||||
"reply": "您的午餐偏好是 user；午餐偏好；VAL-HEE2QMH2。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 0.9400680589675904,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000171",
|
||||
"attribute": "常住城市",
|
||||
"answerable": true,
|
||||
"expected": "VAL-KZA246K9",
|
||||
"reply": "您的常住城市是 VAL-KZA246K9，办公城市是 VAL-Z8QBE5HQ，出生城市是 VAL-N8SWTE6N，宿舍楼号是 VAL-XZY4WPYH，通勤",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 1.49963095664978,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000154",
|
||||
"attribute": "入职年份",
|
||||
"answerable": true,
|
||||
"expected": "VAL-7Z9CBUX8",
|
||||
"reply": "关于您何时开始在这家公司上班，当前记忆中没有相关记录。已知信息：user；办公城市；VAL-BP7253WL。",
|
||||
"correct": false,
|
||||
"wrong_attribute": true,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 1,
|
||||
"top_score": 8.51429933919514,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000234",
|
||||
"attribute": "工位楼层",
|
||||
"answerable": true,
|
||||
"expected": "VAL-87M7HVC7",
|
||||
"reply": "您的工位楼层是 VAL-87M7HVC7。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 3.1934859675651266,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000083",
|
||||
"attribute": "办公城市",
|
||||
"answerable": true,
|
||||
"expected": "VAL-YBVYV97X",
|
||||
"reply": "您的办公城市是 VAL-YBVYV97X。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 2.3717270611639645,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000079",
|
||||
"attribute": "出生城市",
|
||||
"answerable": true,
|
||||
"expected": "VAL-7R8WVAFG",
|
||||
"reply": "您的出生城市是 VAL-7R8WVAFG。",
|
||||
"correct": true,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 2.4271069115554122,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000180",
|
||||
"attribute": "客户名称",
|
||||
"answerable": false,
|
||||
"expected": "VAL-3F87H6UT",
|
||||
"reply": "作为一个人工智能助手，我**无法知道您随身设备的型号或数字**，因为我没有访问您个人设备信息的权限，也无法看到您正在使用的具体设备。\n\n如果您指的是：\n- **手机/电脑型号**：",
|
||||
"correct": false,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 23,
|
||||
"stop_reason": "attribute_not_covered",
|
||||
"records_selected": 0,
|
||||
"top_score": 0.0,
|
||||
"need_memory": false
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000025",
|
||||
"attribute": "紧急联系人姓氏",
|
||||
"answerable": true,
|
||||
"expected": "VAL-STECQ3PX",
|
||||
"reply": "您的默认语言是 VAL-PRVTUCP7，课程名称是 VAL-93ME7BJ5，起床时间是 VAL-PJC3D3W9，出生城市是 VAL-KYBRBBKV，咖啡口味是",
|
||||
"correct": false,
|
||||
"wrong_attribute": true,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 8,
|
||||
"top_score": 8.516015553315961,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000187",
|
||||
"attribute": "手机尾号",
|
||||
"answerable": true,
|
||||
"expected": "VAL-PXRHJ49F",
|
||||
"reply": "您随身那台设备的末端数字是 H9，该设备的完整型号为 VAL-ZPUPV9H9。",
|
||||
"correct": false,
|
||||
"wrong_attribute": true,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 24,
|
||||
"stop_reason": "evidence_found",
|
||||
"records_selected": 1,
|
||||
"top_score": 8.513663491707547,
|
||||
"need_memory": true
|
||||
},
|
||||
{
|
||||
"id": "zero-overlap-000045",
|
||||
"attribute": "办公城市",
|
||||
"answerable": false,
|
||||
"expected": "VAL-CG4X6ZWD",
|
||||
"reply": "作为一个人工智能助手，我并没有实体，也没有“待得最久”的物理空间或地理位置。我的“存在”体现在每一次与您的对话中。\n\n不过，如果您是在寻找**您自己**待得最久的地方",
|
||||
"correct": false,
|
||||
"wrong_attribute": false,
|
||||
"leaked": false,
|
||||
"records_written": 24,
|
||||
"records_active": 23,
|
||||
"stop_reason": "attribute_not_covered",
|
||||
"records_selected": 0,
|
||||
"top_score": 0.0,
|
||||
"need_memory": false
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user