wpy/natural-memory-nm21
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Natural Memory NM2.1: 记忆路由器分叉、数据集缺陷修复与全轴评测证据

Browse Source 
- 引入 MemoryRouterXL 与 v5/v6 流式多线程训练/编码管线
- 修复 prepare_memory_router_dataset 候选池重建缺陷（mega 家族 3568x 加速，输出逐字节相同）
- 修复 v5 被破坏的拒答与多跳标签（train 未知样本 319 -> 16319，multi_hop 平均正例 1.00 -> 2.00）
- 同存储预算下 V2-128 v6 逐轴 22/22 通过：Top-1 41.12% -> 94.62%，未知拒答 0.00% -> 100.00%
- 记录三条被实测推翻的显然优化（logits_to_keep=1 反而慢 55%、XL 容量未带来收益）
- 记忆手术跨架构可移植性 14/14，读写关闭时与原生模型逐位相同
This commit is contained in:
WpyQwq
2026-09-19 11:11:31 +08:00
commit 643e22ecb9
484 changed files with 306821 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+32
View File
@@ -0,0 +1,32 @@
# Python
__pycache__/
*.py[cod]
*.egg-info/
.pytest_cache/
.venv/
venv/
# 训练中间产物（仓库只保留最终交付件与证据 JSON）
checkpoints/
runs/
*.log
*.tmp
# 大体积权重：模型本体不进仓库，只保留 memory_config / memory_merge 等手术配方
*.safetensors
*.bin
*.7z
*.zip
# 本地用户记忆快照（含真实对话内容，绝不入库）
*_runtime.pt
data/users/
user_memories/
persistent_memory.pt
# 编辑器 / 系统
.idea/
.vscode/
.DS_Store
Thumbs.db
desktop.ini
ABSTENTION_BREAKTHROUGH.md
+167
View File
@@ -0,0 +1,167 @@
# 未知拒答：从 75% 泄漏到 100% 拒答（机制 + 证据 + 边界）
这是本项目目前最差的一个数字（**未知问题泄漏 75.00%**：问库里不存在的属性时照样编一个答案），
本轮把它定位、排除了整类无效修法，并验证了一个可用的机制。
---
## 1. 先排除掉一整类无效修法（有测量支撑）
最直觉的修法是"检索分数不够高就拒答"。用**只在训练集拟合、评测集算 AUC** 的口径测了四种打分器
的分数几何（top1 / margin / top-k 均值 / 离散度 / 熵 / 近邻数，共 9 个特征）：
| 打分器 | 最佳单特征 AUC | 拟合头 AUC（评测集） |
|---|---:|---:|
| cosine（冻结键余弦） | 0.6257（margin） | 0.6086 |
| 打包 `text_retriever` | 0.6076（margin） | 0.4752 |
| 50/50 混合 | 0.5446 | 0.5062 |
| 训练过的路由器 | 0.5711 | 0.5130 |
操作点同样不可用：混合打分器想标出 22% 的未知要误拒 **28.40%** 的可回答问题；
cosine 想标出 46% 要误拒 **30.00%**。
**结论：75% 的泄漏不可能靠分数阈值或分数上的分类头修好。** 这不是调参空间不够，
而是这个信号根本不存在 —— 24 条同句式候选对任何提问都"一样像"。
## 2. 有效的机制：先判「问的是哪个属性」，再查库里有没有
换一个问题问就成立了：不是"有没有候选匹配得好"，而是"这个提问在问哪个属性，这个属性在库里吗"。
**封闭词表**（提问的属性在模型学过的属性集合内、但不在候选/库里）：
| 指标 | 值 |
|---|---:|
| 可回答 episode 中「预测属性在候选集内」 | **100.00%**（250 条） |
| 未知 episode 中「预测属性在候选集内」 | **0.00%**（50 条） |
| **未知拒答率** | **100.00%** |
| **已知问题被误拒率** | **0.00%** |
| 混淆矩阵 | tp=50 fn=0 fp=0 tn=250 |
属性识别本身在**未见改写**上也是 **100.00%**（250 条可回答 episode 全对，24 类，随机基线 4.17%）。
注意：这是**类别式规则**，不依赖任何脆弱阈值。
## 3. 开放集：真实用户会问模型没见过的属性，必须显式训 NONE 类
只靠最大概率调阈值是不够的 —— 实测**没有拒绝选项时，从未见过的属性 100.00% 会被认成某个存在的属性**
（`closed_set_false_claim_pct: 100.00%`）。用最大 softmax 做拒绝的 AUC 是 0.9629 / 0.9760（留出 6 / 12 个属性），
可用但有取舍：留出 6 个属性时，保住 100% 已知的阈值只能拒掉 **44.64%** 的未见属性。
所以改成**显式训练一个 NONE 类**：词表取 12 个属性作为类别，另取 6 个属性（词表外）的真实提问作为
NONE 的负样本，最后用**第三组、与两者都不相交的 6 个全新属性**做测试：
| 指标 | 值 |
|---|---:|
| 词表内属性识别准确率（133 条未见改写） | **100.00%** |
| **从未见过的属性被拒绝**（52 条，与词表、NONE 训练集均不相交） | **100.00%** |
| **已知问题被误拒率** | **0.00%** |
| 随机基线（13 类） | 7.69% |
对照当前运行时：未知泄漏 **75.00%**、A 段未知拒答率 **56.67%~60.00%**。
## 4. 落地形态与必须处理的边界
**运行时规则**（三步，全部可审计、无阈值调参）：
1. 属性分类头读提问的冻结键 → 输出「某个属性」或 **NONE**；
2. 若为 NONE → 拒绝（明确说不知道）；
3. 若为某属性 → 查库的活跃属性集合（即 `bank.active_by_conflict` 的键）是否包含它；不含 → 拒绝。
**必须处理的边界（诚实列出，未解决）**：
* **增量写入**：头部在训练时固定了属性词表；用户**训练之后**才写下的全新属性会被判成 NONE 而误拒。
缓解方向：把检索分数作为并联条件（NONE **且** 没有强匹配记录才拒答），或用属性分类体系扩大词表，
以及写入新属性时增量微调头部。
* **词表随库变化**：生产环境属性集合是用户相关的，头部需要用「该用户当前库的属性集合」构建类别，
本实验固定为 24 个。
* **规模**：本实验 24 个属性、300 条评测；生产需要上千属性、上万条改写问法的规模验证。
* 本机制只解决**拒答**；「答成别的属性 35.00%」是另一个问题（排序），本轮未动。
## 5. 集成尝试：**失败**（已回退，须如实记录）
把覆盖门接进运行时后，端到端整体测试的结果是**灾难性**的：
| 指标 | NM2.1（无门） | NM2.1+覆盖门 |
|---|---:|---:|
| A 可回答正确率 | **100.00%** | **6.25%** |
| A 已知问题被误拒率 | 0.00% | **7.50%** |
| A 未知拒答率 | 56.67% | 43.33% |
| B 回答正确率 | 68.75% | 75.00% |
| C 未知泄漏率 | 75.00% | 62.50% |
| C 可回答正确率 | 65.00% | 65.00% |
| D 重启持久化 | 通过 | 通过 |
**门把几乎全部可回答问题都拒掉了**（可回答正确率 100.00% → 6.25%），而它本该修好的 C 段泄漏
只从 75.00% 降到 62.50%。**已把 NM2.1 的 `memory_coverage_gate` 改回 `false`**，恢复为已验证的可用状态
（52 个单元测试通过）。属性头工件保留在 `checkpoints/memory_attribute_head/`，未随包启用。
### 两个根因（离线评测掩盖了它们）
1. **输入表示不匹配**：属性头是在**裸提问文本**的冻结键上训练的，而运行时 `read()` 收到的是
`self.runtime.v2_query_key`，它是 **`_encode_model_key(input_ids)`**，其中 `input_ids` 是
**套了 chat 模板的完整提示**（还含系统前缀）。头在分布外的输入上工作，argmax 基本是随机的 ——
所以它既误拒了大量可回答问题，又没能拦住该拦的。
2. **库覆盖索引不可靠**：门要求 `user::<属性>` 出现在 `bank.active_by_conflict` 里。A 段（v6 mega 事实）
的文本很多**无法被 `infer_memory_metadata` 解析出属性**，于是冲突键为空 → 门把所有这类问题一律拒掉，
这正是可回答正确率崩到 6.25% 的原因。
**离线 100.00% / 0.00% 是真实的，但它只证明了机制在"裸提问键 + 24 个可解析属性"这个条件下成立，
并不等于接上运行时就能用。** 这一点必须在任何对外表述里讲清楚。
### 正确的下一步（按顺序，每步都有验收口径）
1. 用**运行时实际喂进来的那个表示**（chat 模板编码）重建训练集，而不是裸提问键；
验收：门开启后 A 段可回答正确率不低于 99.00%。
2. 把覆盖判断从"必须有结构化冲突键"改为"结构化键 **或** 检索到强匹配记录"，覆盖
`infer_memory_metadata` 解析不出属性的真实事实文本；验收：A 段已知问题被误拒率 ≤ 1.00%。
3. 两项都达标后再开 C 段验收：未知泄漏 ≤ 5.00%（现为 75.00%）。
## 6. 集成第二轮：修好一半 —— 目标域泄漏 0.00%，但换域即崩
第一轮失败后定位到真正原因（并不是我先前猜的"输入表示不匹配"：`retrieval_ids = memory_query_input_ids
if ... else input_ids`，喂给 `v2_query_key` 的就是**裸提问**，与训练一致）。真正的机制缺口是：
**`read()` 返回空记录后不会停止，`_build_text_prefix` 会继续落到旧版 16 槽注入路径**，把无关记忆又塞回去。
所以在 `_build_text_prefix` 里加了短路：覆盖门拒绝时直接返回 guard 前缀，不再回落。
效果（C 段，48 用例，24 同形候选 —— 也就是属性头训练覆盖的那个域）：
| C 段指标 | NM2.1（无门） | NM2.1+门+短路 |
|---|---:|---:|
| **未知泄漏率** | 75.00% | **0.00%** |
| 可回答正确率 | 65.00% | 65.00%（无损失） |
| 答成别的属性 | 35.00% | 35.00% |
| 活跃记录 | 23/24 | 23/24 |
**但在 A 段（v6 mega 事实）上是灾难**：可回答正确率 **0.00%**、已知被误拒率 7.50%、总体 12.73%。
根因已测定：
* A 段事实里 **94.03% 是"这是普通对话噪声…不需要长期保存"**，只有 **5.97%（63/1056）**能解析出属性；
* 更关键的是 **A 段的属性空间（"备用联系人"等）完全不在属性头训练的 24 个属性之内**，头对 A 的提问只能
输出那 24 个之一 → 必然不在库里 → 门把一切都拒掉。
也就是说：**这个机制目前是"域内可用、跨域不可用"**。已把 NM2.1 的 `memory_coverage_gate` 改回 `false`
（52 个单元测试通过），并在 `memory_config.json` 里记下"已验证域"与"阻塞点"。
### 正确的最终形态：开放词表的属性匹配
不要再做"24 类闭集分类头"，而是**把提问与库里真实存在的属性名做匹配**：
* 库里有哪些属性 → 从 `bank.active_by_conflict`（或记录上的 `attribute` 字段）取，随写入动态变化；
* 打分 = `similarity(query_key, encode(attribute_name))`，对**每个存在的属性**算一次，取最大；
* 低于阈值或库里属性为空 → 拒答。
这样词表不需要预训练固定，新写入的属性自动纳入。验收口径不变：**先保证 A 段可回答正确率 ≥ 99.00%
且已知问题被误拒率 ≤ 1.00%，再要求 C 段未知泄漏 ≤ 5.00%。**
## 7. 产物
| 文件 | 内容 |
|---|---|
| `analyze_abstention_separability.py` / `abstention_separability.{json,md}` | 第 1 节：分数几何不可分（四打分器 AUC 与操作点） |
| `analyze_query_attribute_classifier.py` / `query_attribute_classifier.{json,md}` | 第 2 节：封闭词表 100%/0%，含按属性拆解 |
| `holdout_6.json` / `holdout_12.json` | 第 3 节：开放集最大概率 AUC 与操作点 |
| `none_class_result.json` | 第 3 节：显式 NONE 类，100.00% 拒答 / 0.00% 误拒 |
| `train_attribute_head.py` / `attribute_head_training.json` | 第 3 节工件：属性头训练脚本与指标 |
| `checkpoints/memory_attribute_head/attribute_head.pt` + `attribute_head_meta.json` | **可部署工件**（当前未在包内启用） |
| `nm2_1_gate_*.json/md`、`nm2_battery_comparison_gate.{json,md}` | 第 5 节第一轮失败证据 |
| `nm2_1_gate2_runtime_e2e.{json,md}` | 第 6 节：目标域泄漏 **0.00%** |
| `nm2_1_gate2A_e2e.{json,md}` | 第 6 节：跨域崩溃证据（A 段 0.00%） |
ARCHIVE_MOVED.md
+61
View File
@@ -0,0 +1,61 @@
# 已移出本 fork 的内容（归档到 E 盘）
本文件记录 **2026-09-12** 从 `H:\Memory\V2_dpskw` 移出、归档到
`E:\归档\03_代码与项目\V2_dpskw-备份\` 的内容，便于日后查找或恢复。
归档规范见 `E:\归档\AGENTS.md`；操作已记入 `E:\归档\00_索引\操作日志.jsonl`。
## 归档包
| 包 | 大小 | 未压缩 | 内容 |
| --- | ---: | ---: | --- |
| `V2_dpskw_20260912_085921.7z` | 7.09 GB | 13.14 GB / 411 文件 | 整个 fork 的冻结备份：全部代码、`data/` 数据集、`checkpoints/` 最终权重与日志、全部评测证据 JSON/MD |
| `V2_dpskw_中间与废弃checkpoint_20260912_132740.7z` | 6.48 GB | 7.90 GB / 116 文件 | 训练中间快照与废弃运行（清单见下） |
两个包都通过 `7z t` 完整性测试，并与源文件**逐文件比对字节数一致（116/116）后才删除源文件**。
## 第二个包的具体内容（已从 H: 删除）
| 来源 | 文件数 | 大小 | 说明 |
| --- | ---: | ---: | --- |
| `checkpoints/router_v6_v2_128/router_step_*.pt` | 10 | 0.23 GB | 训练中间快照（每 10000 步一个） |
| `checkpoints/router_v6_v2_512/router_step_*.pt` | 10 | 0.53 GB | 同上 |
| `checkpoints/router_v6_xl128/router_step_*.pt` | 10 | 0.76 GB | 同上 |
| `checkpoints/router_v6_xl512/router_step_*.pt` | 10 | 0.92 GB | 同上 |
| `checkpoints/router_replay_v7_v2_128/router_step_*.pt` | 1 | 0.02 GB | 与最终版权重相同 |
| `checkpoints/router_zov_v2_128/router_step_*.pt` | 1 | 0.02 GB | 与最终版权重相同 |
| `checkpoints/router_xl_1024/` | 9 | 1.11 GB | 1024 维实验（已放弃）整目录 |
| `checkpoints/router_xl_512/` | 28 | 2.09 GB | v3 旧数据运行（已被 v6 版取代）整目录 |
| `checkpoints/router_xl_smoke/` | 8 | 0.97 GB | 冒烟测试整目录 |
| `checkpoints/_smoke_stream/`、`_smoke_v2/`、`_smoke_v2b/`、`_smoke_xl_ckpt/`、`_smoke_feature_bank/`、`router_shared/` | 29 | 0.70 GB | 冒烟/临时运行整目录 |
释放 **7.36 GB**；fork 由 12.24 GB / 534 文件降至 **4.89 GB / 418 文件**。
上表 9 个整目录归档后已变为空目录，一并删除。
## 保留在 fork 里的（未归档，仍是证据与可运行产物）
| 目录 | 保留内容 |
| --- | --- |
| `checkpoints/router_v6_v2_128/` | `memory_router_v2.pt`（最终，drop-in 部署件）、`router_best.pt`、`router_arch.json`、`metrics.jsonl`、`router_v5_training.json`、`v6_final_resume_wrapper.pt`、`ranking_scrambled_probe.pt` |
| `checkpoints/router_v6_v2_512/` | 最终 `memory_router_v2.pt` + `router_best.pt` + 日志 |
| `checkpoints/router_v6_xl128/` | 最终 `memory_router_xl.pt` + `router_best.pt` + 日志 |
| `checkpoints/router_v6_xl512/` | 最终 `memory_router_xl.pt` + `router_best.pt` + 日志 |
| `checkpoints/router_replay_v7_v2_128/` | `memory_router_v2.pt`（**推荐交付物**：22/22 轴超越 + 零重叠 +41.20pp） |
| `checkpoints/router_zov_v2_128/` | 朴素微调对照件（用于证明灾难性遗忘） |
## 恢复方法
```powershell
# 恢复某个中间快照目录（示例）
& 'C:\Program Files\7-Zip\7z.exe' x 'E:\归档\03_代码与项目\V2_dpskw-备份\V2_dpskw_中间与废弃checkpoint_20260912_132740.7z' -o'H:\Memory\V2_dpskw\_restored' 'checkpoints/router_xl_512/*'
# 恢复整个 fork 快照（注意：会覆盖同名文件，先解到空目录）
& 'C:\Program Files\7-Zip\7z.exe' x 'E:\归档\03_代码与项目\V2_dpskw-备份\V2_dpskw_20260912_085921.7z' -o'E:\_v2_restore'
```
⚠️ 整包备份里的 `V2_dpskw\qwen3_5_4b_natural_memory_v2` 是个**空目录占位**（原为指向
`H:\Memory\dynamic_memory_lab\qwen3_5_4b_natural_memory_v2` 的 junction，用 `-snl` 打包时未跟随，
以免把整个 4B 模型塞进包里）。解压后需要重建：
```powershell
cmd /c mklink /J "H:\Memory\V2_dpskw\qwen3_5_4b_natural_memory_v2" "H:\Memory\dynamic_memory_lab\qwen3_5_4b_natural_memory_v2"
```
BOTTLENECK_REPORT.md
+112
View File
@@ -0,0 +1,112 @@
# 端到端瓶颈攻坚：排序混合权重（负面结论）+ 未知问题泄漏
本文件记录的是**失败与自我纠错**，因为这两个结果都会影响后续方向判断。
## 1. 先修掉一个我自己的错误判断
我一度声称"**93.75% 的读取绕过 V2 库、走旧版 16 槽路径**"，并据此把它列为第一瓶颈。
**这个结论是错的。** `probe_record_selection.py` 里的 `legacy_prefix_used` 取的是
`runtime.text_prefix_used`，而该方法在 **V2 路径（`qwen_integration.py:2483`）和旧版路径
（2515 行）都会被置 True** —— 字段名误导了我。
实测反证：大样本评测 64/64 用例的 `stop_reason` 全是 `evidence_found`（`read()` 返回了记录），
平均选中记录 6.47，写入 24 条事实后活跃记录 23–24。**V2 库记录确实是被注入的证据来源。**
## 2. 真正测出来的瓶颈：记录级排序器很弱
同一批冻结特征、250 条可回答的零重叠 episode、24 个同形候选（随机基线 4.17%）：
| 排序器 | Top-1 | Recall@3 | MRR |
|---|---:|---:|---:|
| **REPLAY-128（训练过的路由器）** | **59.60%** | **83.20%** | **73.06%** |
| 冻结键余弦 | 30.40% | 49.20% | 44.84% |
| **打包的 `text_retriever`（运行时实际用来给事实记录排序）** | **23.20%** | 55.20% | 43.16% |
| V2-128-v6 路由器 | 18.40% | 36.00% | 35.07% |
`memory_os_v2._record_scores` 对带 `semantic_key` 的记录会用 `record_scorer`（即打包的
`text_retriever`）**逐位置覆盖** `MemoryRouterV2.projected_scores` 的结果（残差与 retriever
匹配 18/18、与路由器匹配 0/18）。所以运行时用一个 **23.20%** 的排序器，而手上有 **59.60%** 的。
## 3. 实现（已落地，默认关闭）
`QwenMemoryConfig.memory_record_router_blend`（默认 **0.0** = 历史行为）：
`_score_semantic_memory_records` 在混合权重 > 0 时返回
`(1-w)·retriever + w·router`，路由器分数在同一批候选上用 `encode_key` + `projected_scores` 现算。
0.0 时路径与改动前完全一致。
## 4. 测量：先是假象，然后是否定结论
**第一轮（同进程内跑多个权重）**看起来有效：
| blend | 16 用例正确率 | 64 用例正确率 | 答成别的属性 |
|---|---:|---:|---:|
| 0.00 | 68.75% | 60.38% | 37.74% |
| 0.50 | 75.00% | 66.04% | 30.19% |
我据此把默认值改成 0.5，但**独立进程复核时复现不出来**（0.5 只得到 68.75%），说明这两组数字被
"同一进程内先跑 0.0 再跑 0.5"的顺序污染了。
**第二轮（每档一个独立进程，各跑 2 次）**：
| blend | 第 1 次 | 第 2 次 |
|---|---:|---:|
| 0.0 | 68.75% | 68.75% |
| 0.5 | 68.75% | 68.75% |
四次完全一致：**混合权重在 16 用例尺度上没有任何可测效果**。因此：
* 默认值已退回 **0.0**；
* **撤回**第 4 节第一轮的全部增益数字（+6.25pp / +5.66pp / −7.55pp）；
* `probe_record_blend.py` 保留，但必须**每档独立进程**运行，文件注释已写明这一点。
## 5. 为什么它本来就不该有效（与先前的取证一致）
早先的逐记录取证已经量过：`_record_scores` 的最终分数里神经项只占 **12.0% / 14.7% / 16.9%**，
而词面/地址先验占 **83–88%**（`0.25·lexical + 0.45·token_overlap + 1.25·rare_lexical_address
+ 0.35 + 0.15`，`write_fact` 还使 `0.10·conf + 0.08·imp` 恒定）。并且 `1.25·rare` 在部分候选集上
造成 **1.108 的先验差**，而 [0,1] 的神经分最大只能摆动 1.0 —— 那些配对**结构上无法被神经分翻转**。
实测印证：所有权重下 `mean_records_selected` 都是 6.0625，**被注入的记录集合不变**，只是顺序微调；
顺序变化不足以改变生成结果。**所以下一个杠杆不是神经排序器，而是 `_record_scores` 里的先验权重**
（尤其 `1.25·rare_lexical_address` 与那两项常数加成），以及 `direct_records` 的硬编码 8.0/10.0
（比神经路径最高约 2.90 高出 5.6 以上，使直接命中路径完全压过学习排序）。
## 6. 附带发现（已用全样本确认）：未知问题泄漏率 78.00%
零重叠评测集里"问的是候选中不存在的属性、但句式完全正常"的不可回答 episode：
**全部 50 条单跑一遍，泄漏率 78.00%（39/50）** —— 运行时会给这些无从回答的问题返回一个候选代号。
（小样本 11 条时是 81.82%，方向一致。）
`stop_reason` 在 50/50 条里全是 `evidence_found`。对照 v6 全集（10 类别）未知泄漏率是 **0.00%**
（走 `explicit_unknown_request` 规则）。**"句式正常的未知"是当前最大的体验缺口。**
### 6.1 阈值方案已被测量排除
最直觉的修法是"最高分不够高就拒答"。为判断可行性，我采集了**检索最高分**（注意：这是运行时的原始
相关度分，**不是概率、不是百分比**，观测上限约 851）在两组的分布：
| 组 | n | min | p10 | p50 | p90 | max |
|---|---:|---:|---:|---:|---:|---:|
| 可回答 | 50 | 69.98 | 77.13 | 288.20 | 851.50 | 851.68 |
| 不可回答 | 10 | 77.80 | 118.86 | 285.25 | 851.37 | 851.37 |
两组分布**几乎完全重叠**（中位数 288.20 vs 285.25，上限 851.68 vs 851.37，且上限处明显饱和）。
所以**任何标量阈值都无法分开二者** —— 压低阈值以拦住 78% 的泄漏，会同时拦掉同等比例的可回答问题。
"未知"与"可回答"在这里的差别不在检索强度，而在**库里到底有没有那个属性**，这需要
查询-库级别的覆盖判断（例如对可解析属性直接查结构化账本，或训练一个"该属性是否存在"的分类头），
而不是给查询-记录打分设阈值。
（附带：本次 60 用例混合样本里可回答 50 条正确率 60.00%、答成别的属性 40.00%，
加权记录 23–24 条全部存活 —— 写入路径修复在大样本下持续有效。）
## 7. 产物
| 文件 | 内容 |
|---|---|
| `compare_record_rankers.py` | 四路排序器对比（余弦 / 各路由器 / 打包 retriever），`--with-text-retriever` |
| `zero_overlap_ranker_comparison_with_retriever.json` | 第 2 节的数字 |
| `probe_record_blend.py` | 混合权重端到端测量（**每档独立进程**） |
| `record_blend_proc_{0.0,0.5}_rep{1,2}.json` | 第 4 节第二轮的四次干净测量 |
| `record_blend_dose_response.{json,md}` | 第一轮（**已作废**，保留以说明混淆来源） |
| `eval_runtime_zero_overlap_e2e.py` | 大样本运行时评测（零重叠 64 例 / v6 40 例），第 6 节 |
| `runtime_zero_overlap_e2e.{json,md}`、`runtime_v6_blend_e2e.{json,md}` | 第 6 节数字 |
E2E_FINDINGS.md
+62
View File
@@ -0,0 +1,62 @@
# 端到端测试发现（回答级，非代理指标）
之前所有结论都建立在**路由器级**评分卡上。本文件记录回答级（写入证据 → 路由器读取 → Qwen 生成 → 打分）的测试，以及它推翻了什么。
## 1. 先说结论
1. **在 v6 评测集上，端到端回答正确率与新路由器无关**：生产路由器与 V2-128-v6 得到**完全相同**的结果（总体 59.55%、可回答 59.38%、未知拒答 60.00%）。原因见 §2。
2. **路由器确实在一个真实失败模式上产生了产品级收益**：面对没有任何词法重叠的改写问句，旧路由器 **7/16 直接判断"不需要读记忆"**（已知问题被误拒 43.75%），新路由器 **0/16**。这与路由器级指标（已知被误拒率 11.23% → 0.00%）一致。
3. **但答对率只从 37.50% 提升到 43.75%**，且"答成别的属性"从 18.75% 升到 25.00%。瓶颈已经从"路由"转移到"证据精度与模型用证"。
4. **路由器的 Top-1 在纯语义改写上是弱的**：`top_k=1` 时 43.75% 答成别的属性。而路由器级评分卡上 paraphrase 类别 Top-1 是 99.65% —— 因为那些用例仍与事实共享属性词，属于词法+语义混合匹配。
## 2. 为什么 v6 端到端结果与路由器无关
在 44 个真实用例（覆盖全部 11 类）上统计检索决策来源：
| 决策来源 | 占比 |
|---|---:|
| `evidence_found`（规则化地址/词法命中） | 90.9% |
| `explicit_unknown_request`（识别"请明确说不知道"） | 9.1% |
| 需要神经路由器决定记录选择 | 0%（规则已给出候选并命中） |
对同一批用例，生产路由器与 V2-128-v6 给出**逐位相同**的决策（`top` 值 8.367 / 4.168 / 8.536 全同，`page_ids`、`record_ids` 数量全同）。
机制（源码路径）：`MemoryOSV2.read` 先用规则判断（显式未知请求 → 直接弃答），再问路由器的 `need_memory`；**候选页面/记录由库内 `_address_hits` / `_lexical_evidence_hits` 产生**，神经分数只在候选内排序。因此当查询与证据共享属性词时，规则已经解决了问题，路由器不改变结果。
**含义**：v6 评测集的构造方式（问句与事实共享属性措辞）使它对路由器不敏感；用这种数据无法测出路由器的优劣，也无法证明路由器能改善产品。
## 3. 路由器关键测试（规则无法决定）
构造：16 条**属性互异**的个人事实（因此冲突键不同、真实共存，不是互相覆盖）+ 16 个**与所有事实零词法重叠**的改写问句；答案是只出现在一条事实里的随机编码，因此"答成别的属性"可检测。写入 16 条事实 + 4 条无关填充后提问。
| 路由器 | 用例 | 回答正确率 | 答成别的属性 | 触发读取 | 规则命中用例 | 平均注入记录 | 决策来源 |
|---|---:|---:|---:|---:|---:|---:|---|
| 生产 128 维(v3) | 16 | 37.50% | 18.75% | **56.25%** | 6/16 | 2.88 | `evidence_found`:9, **`router_abstained`:7** |
| **V2-128-v6** | 16 | **43.75%** | 25.00% | **100.00%** | 6/16 | 6.38 | `evidence_found`:16 |
已知问题被误拒：**43.75% → 0.00%**（7/16 → 0/16）。
### 注入记录数扫描（V2-128-v6）
| `top_k_records` | 答对率 | 答成别的属性 | 平均注入记录 |
|---:|---:|---:|---:|
| 8（默认） | **43.75%** | 25.00% | 6.38 |
| 3 | 37.50% | 25.00% | 2.63 |
| 1 | 18.75% | **43.75%** | 1.00 |
减少注入量**使结果更差**：`top_k=1` 时 43.75% 答成别的属性，等于路由器在这些零重叠改写上的 Top-1 大约一半是错的。默认 8 条之所以得分最高，是因为模型能从更大的证据集里自己捞回正确答案——**这不是路由器的功劳，而是它的错误被大证据集掩盖了**。
## 4. 部署约束（实测）
| 路由器 | 能否直接替换生产路由器 | 原因 |
|---|---|---|
| V2-128-v6 | **能** | 16 个权重键与分片零差异，可驱动 `PagedMemoryBankV2` |
| XL-128-v6 | 不能 | 新架构，`MemoryRouterXL` 形状与运行时的 `MemoryRouterV2` 不兼容，需改运行时 |
| V2-512-v6 | 不能 | 需要把 `memory_router_dim` 128→512 并重建记忆地址 |
## 5. 下一步（按性价比排序）
1. **补训路由器的零重叠改写能力**：当前 paraphrase 训练样本仍共享属性词。用模板化改写（同义词/口语化/指代）生成"零词法重叠"样本，这是 §3 暴露的唯一缺口。
2. **证据精度而非召回**：默认注入 8 条时错答属性比例 25%，说明应当对注入证据做冲突键去重与属性一致性过滤，而不是单纯扩大 Top-K。
3. **重建评测集**：让问句与事实不共享属性措辞，否则任何路由器改进都无法在 v6 上体现。
4. 端到端 v6 全集（220 用例）可作为产品基线，但它测的是规则路径 + 生成能力，不是路由器。
ENGINEERING.md
+871
View File
@@ -0,0 +1,871 @@
# Dynamic Memory Lab
一个在现成 Qwen3.5-4B 语言模型上进行“架构手术”的实验工程。
> Natural Memory v2 已完成分页地址路由、粗索引→候选页→精排→Top-K、多跳、冲突版本、quarantine、撤回、嵌入式重启恢复和完整回归评测。请先阅读 [Natural Memory v2 工程说明](README_NATURAL_MEMORY_V2.md) 与 [技术论文](Natural_Memory_v2_Paper.md)。
本项目不重新训练整个语言模型，而是在 Transformer 的若干层之间插入一个可读、可写、可持续更新的动态记忆模块，研究下面这个问题：
> 能不能让一个已经训练好的 LLM，在不把所有历史对话反复塞回上下文的情况下，持续积累和使用信息？
答案是：工程中已经实现了一个生产导向的自然语言记忆核心，并完成了多事实写入、冲突更新、无历史跨重启读取、二次重启读取、未知事实拒答和 reset token 验收。它仍需要在真实业务数据上继续做压力测试，合成数据指标不能直接等同于所有场景的生产承诺。
## 结论先说
当前模型已经具备：
- 在一次进程运行期间维护一组独立于 token 上下文的 memory state；
- 对 memory 做 attention 读取；
- 根据当前输入生成写入内容，并更新 memory；
- 将 memory 读取结果注入 Qwen3.5-4B 的中间层；
- 只训练记忆模块和少量门控参数，而冻结原始 Qwen 主干；
- 对比原版 Qwen3.5-4B 与动态记忆版的 loss、困惑度、准确率、生成质量、速度和显存。
当前版本仍有边界：
- 还没有证明对任意自然语言、多事实和长时间运行都可靠；
- 还没有完成生产级多用户隔离、隐私策略和容量管理；
- 自由生成在合成留出集上仍会偶尔答错值，所以不能把实验指标当作生产承诺。
因此，当前版本更准确的定位是：
> 一个包含可部署自然语言记忆核心、持久化 checkpoint 和验收基准的 LLM 原生动态记忆工程；生产接入前仍需补齐业务侧鉴权、加密、并发和运维策略。
## 项目结构
V2_dpskw/
├─ qwen_integration.py # 动态记忆模块和 Qwen 模型适配器
├─ train_qwen_memory.py # 记忆模块训练脚本
├─ chat_qwen_memory.py # 多轮对话实验脚本
├─ benchmark_qwen.py # 原版/动态版跑分脚本
├─ verify_persistent_memory.py # 重启后仅用 memory_state 验证回答
├─ make_native_memory_data.py # 生成写入/遗忘/未知/冲突数据
├─ train_native_memory.py # 训练原生写入、遗忘和读出
├─ evaluate_native_memory.py # 留出集控制器与自由生成评估
├─ verify_native_checkpoint.py # checkpoint 重启和 reset token 验证
├─ train_natural_retriever.py # 训练自然语言记忆查询检索器
├─ train_auto_memory_policy.py # 训练自动记忆重要性策略
├─ build_production_memory_dataset.py # 真实对话导入、规范化和防泄漏切分
├─ train_production_memory_policy.py # 从规范化对话训练策略候选
├─ stress_test_natural_memory.py # 低显存连续运行/长上下文压力测试
├─ natural_memory_service.py # localhost 管理与聊天 API
├─ natural_memory_app.py # 统一 chat/serve/train/stress 入口
├─ benchmark_natural_language_memory.py # 多事实/更新/重启/拒答验收
├─ stream_chat_qwen_memory.py # 流式聊天与随时重启测试入口
├─ make_benchmark_data.py # 生成简单的记忆型 benchmark 数据
├─ requirements.txt # Python 依赖
├─ memory.pt # 训练后的记忆模块参数，若已生成
├─ memory_config.json # 记忆模块配置，若已生成
├─ surgery.pt # 层手术相关参数，若已生成
└─ README.md
## 运行环境
推荐使用已有的 Conda 环境 LLM。
conda activate LLM
cd W:\Flash\model\V2_dpskw
pip install -r requirements.txt
如果模型路径不在默认位置，可以通过参数指定。当前工程主要面向本地 Hugging Face 格式的 Qwen3.5-4B 模型。
## 总体架构
原版 Qwen 的推理过程大致是：
输入 token
│
▼
Embedding
│
▼
Transformer Layer 0
│
▼
Transformer Layer 1
│
...
│
▼
Transformer Layer N
│
▼
LM Head
│
▼
下一个 token
动态记忆版会在若干个 Transformer 层上插入 Memory Adapter：
输入 token
│
▼
Qwen Transformer Layer
│
├──────────────► Memory Read
│ ▲
│ │
│ Runtime Memory State
│ M = [B, S, D]
│ │
│ ▼
├──────────────► Memory Delta
│
▼
Memory Layer Adapter
│
▼
后续 Qwen Transformer Layers
│
├──────────────► Memory Write
│ │
│ ▼
│ M_new = Update(M, input)
│
▼
LM Head
其中：
- B 是 batch size；
- S 是 memory slot 数量，默认 16；
- D 是 memory embedding 维度，默认 512；
- M 不是 token 序列，而是模型外部维护的一组连续向量；
- memory_state 可以在不同调用之间传递，因此它能够脱离上一轮的文本上下文。
## 核心概念：参数、上下文和运行时记忆
理解本项目最重要的是区分三种东西。
### 1. 模型参数
模型参数是 Qwen 的权重以及动态记忆模块的权重，例如：
Wq, Wk, Wv, Wread, Wwrite
它们决定模型“如何读写记忆”，通常在训练阶段更新，在推理阶段保持不变。
### 2. 当前上下文
当前上下文是这次请求中送入模型的 token，例如：
用户：我叫小明。
助手：你好，小明。
上下文是临时的。上下文窗口结束以后，模型本身不会自动保存这些 token。
### 3. 运行时记忆状态
运行时记忆状态是：
M = [batch_size, memory_slots, memory_dim]
默认情况下：
M = [B, 16, 512]
它是模型运行时的一块连续状态。只要下一次调用仍然传入同一个 memory_state，模型就能继续使用之前写入的内容。
注意：
> memory.pt 保存的是“记忆模块的训练参数”，不是某个用户的聊天记忆。
用户记忆应该单独保存为某种 runtime state，例如：
user_001_memory.pt
user_002_memory.pt
或者保存到数据库、对象存储、向量数据库中。
## Memory Read：模型如何读取记忆
设某一层产生的隐藏状态为：
h ∈ R^D_hidden
记忆矩阵为：
M = [m₁, m₂, ..., mₛ] ∈ R^(S×D_memory)
首先把当前隐藏状态投影成 query：
q = W_q h
把每个 memory slot 投影成 key 和 value：
k_i = W_k m_i
v_i = W_v m_i
然后计算当前输入与每个 slot 的匹配程度：
score_i = q · k_i / sqrt(D_memory)
经过 softmax 得到读取权重：
α_i = softmax(score_i)
最后将各个 slot 的 value 加权求和：
r = Σ_i α_i v_i
r 就是当前输入从动态记忆中检索出来的内容。
为了避免记忆模块一开始就破坏 Qwen，代码还使用了一个 read gate：
g = sigmoid(W_gate h)
最终的记忆增量大致为：
Δh = read_scale × g × W_read(r)
然后再注入当前层：
h_new = h + Δh
这和给 Transformer 增加一个小型、可训练的外部知识通道类似。
## Memory Write：模型如何写入记忆
读取解决的是“从记忆里找什么”，写入解决的是“把当前输入存什么”。
当前实现默认使用输入序列最后一个 token 的隐藏状态作为摘要：
s = h_last
也支持固定 token 偏移位置作为写入摘要。
接着通过写入投影生成候选内容：
p = W_write(s)
再根据当前输入生成写入地址和写入强度：
a = softmax(W_addr(s))
z = sigmoid(W_strength(s))
其中：
- a_i 表示第 i 个 slot 被写入的比例；
- z 表示本次写入总体有多强；
- p 是候选写入向量。
对于每个 slot，更新形式近似为：
m_i_new = (1 - z × a_i) × m_i
+ (z × a_i) × p
这是一种可微分的软写入。它不会使用不可导的“直接选中某个 slot”操作，所以可以通过反向传播学习：
- 什么输入值得写入；
- 应该写入哪些 slot；
- 写入幅度应该多大；
- 如何从隐藏状态中压缩信息。
代码还支持 broadcast_write，让候选内容广播写入所有 slot。这个模式更适合做架构实验，但可能降低 slot 的分工能力。
## Qwen 接入方式
qwen_integration.py 会加载原版 Qwen，并替换指定层为带记忆能力的适配层。
默认会选择若干中间层；也可以通过 layer_indices 手动指定层号。工程中同时保留了一个轻量的自定义线性注意力记忆层，便于做对照实验。
动态模型默认冻结 Qwen 主干：
Qwen 原始参数：冻结
Memory Read/Write：训练
层融合 gate：训练
这样做的好处是：
- 显存和训练成本更低；
- 不容易破坏原模型能力；
- 更容易判断提升来自记忆机制还是来自主干重新学习；
- 适合在单卡环境中快速迭代架构。
## 三种层融合模式
动态记忆读出后，需要决定如何注入 Qwen 的隐藏状态。当前支持三种模式。
### residual
h_new = h + memory_delta
这是默认模式。它保留原始隐藏状态，并把记忆当作额外残差通道。
特点：
- 最稳定；
- 对原模型干扰小；
- 适合第一版训练和 benchmark。
### blend
h_new = h_residual + (1 - α) × original_token_mixer + α × memory_delta
其中 α 是可训练的融合系数。
特点：
- 模型可以学习记忆通道应该占多大比例；
- 适合研究“原始表示”和“记忆表示”的权衡；
- 如果初始化或训练不稳定，可能导致原模型信息被过早削弱。
### replace
h_new = memory_delta
完全使用记忆分支输出。
特点：
- 适合验证记忆分支的独立表达能力；
- 风险最高；
- 通常不建议作为默认生产方案。
## direct_logit_scale：直接影响输出概率
除了修改 Transformer 中间层，代码还支持把最后一次 memory readout 经过投影后直接加到 logits：
logits_new = logits_qwen + scale × projection(memory_readout)
这个选项可以直接研究：
- memory 是否能记住某些目标答案；
- memory 是否能把目标 token 的概率推高；
- 记忆模块对最终预测的直接影响。
但它也更容易过拟合简单 benchmark，因此应该同时观察泛化测试和正常生成质量。
## raw token pointer：精确 token 记忆实验
项目还提供一个有意“开后门”的架构消融：memory pass 不仅写入连续向量，还可以保存某个 token 在 Qwen 输出投影矩阵中的行；查询生成的第一个 token 会读取这行向量。
这个实验用于回答一个非常具体的问题：
> 如果记忆里已经存在目标 token，当前 Qwen 接口能不能把它可靠地送进自由生成？
示例配置：
--write-token-offset 4 --broadcast-write \
--raw-token-write --raw-logit-scale 30
注意：`write-token-offset` 按“从有效序列末尾数起”计算，`1` 是最后一个 token。当前 benchmark 的答案位于 `H / 。 / <|im_end|> / 换行` 中的倒数第 4 个位置，所以使用 `4`。`raw-token-write` 是 pointer ablation，不是通用自然语言记忆方案；它直接保存 token id 对应的输出投影行，不能把它的 100% 结果等同于普通 learned memory 的能力。
raw pointer 只作用于第一个生成 token，后续 token 回到 Qwen 原本的生成分布，避免把同一个答案 token 重复写满整段输出。
## 训练原理
当前训练脚本采用“两阶段记忆训练”。
### 阶段一：写入阶段
输入一段 memory text：
Memory: user=alice; favorite_color=blue
此阶段关闭 memory read，打开 memory write：
outputs = model(
memory_text,
memory_state=memory_state,
read_memory=False,
update_memory=True,
return_memory=True,
)
memory_state = outputs.memory_state
目标是让模型把关键信息写入 memory state。
### 阶段二：查询阶段
再输入查询：
Question: What is alice's favorite color?
此阶段使用刚刚更新后的 memory state，并开启 memory read：
outputs = model(
query_text,
memory_state=memory_state,
read_memory=True,
update_memory=False,
labels=labels,
)
通过语言模型 loss 训练记忆模块，让查询阶段能够根据 memory state 输出正确答案。
训练时通常只更新：
Memory Read parameters
Memory Write parameters
Memory Layer Adapter parameters
Blend/Gate parameters
而 Qwen 主干保持冻结。
## 数据格式
训练和 benchmark 数据使用 JSONL。当前脚本要求每行包含 `memory` 和 `query` 两个消息列表；最后一个 assistant 消息是监督目标：
{"memory":[{"role":"user","content":"记住对象 A 的代号是 H。"},{"role":"assistant","content":"好的，已记住。"}],"query":[{"role":"user","content":"对象 A 的代号是什么？"},{"role":"assistant","content":"H"}]}
额外的 `id`、`subject`、`attribute`、`answer` 字段只用于 benchmark 统计。训练和评估数据应避免把答案直接重复到 query 的 user 内容中。
数据设计时要注意：
1. 写入文本中出现的信息，应该在查询文本中尽量不重复；
2. 如果查询中直接包含答案，模型可能只是在复制上下文，而不是读取 memory；
3. 训练集和评估集中的实体、属性、表述方式应尽量分离；
4. 要加入冲突样本，测试新记忆是否能覆盖旧记忆；
5. 要加入多条事实，测试不同 slot 是否会互相污染；
6. 要加入无关信息，测试模型能否避免把所有内容都写进去。
make_benchmark_data.py 可以生成一个简单的单字符映射任务，用来快速检查“写入—读取—回答”链路是否工作。它适合做冒烟测试，不足以证明模型拥有通用长期记忆。
## 常用命令
以下命令均在工程目录执行。
### 训练动态记忆模块
conda activate LLM
cd W:\Flash\model
python -m V2_dpskw.train_qwen_memory --model-path "W:\Flash\model" --data V2_dpskw\data\benchmark_train.jsonl --output-dir V2_dpskw\qwen_memory_adapter --steps 100 --batch-size 1 --lr 1e-4 --max-length 128
如果要复现实验中的精确 token pointer：
python -m V2_dpskw.train_qwen_memory --model-path "W:\Flash\model" --data V2_dpskw\data\benchmark_train.jsonl --output-dir V2_dpskw\qwen_memory_adapter_pointer --steps 100 --surgery-mode blend --blend-init 0.1 --write-token-offset 4 --broadcast-write --raw-token-write --raw-logit-scale 30
如果实际模型目录不同，请替换 --model-path。
### 训练原生记忆控制器
原生模式会训练：摘要池化、写入决策、slot 地址、候选值读出，以及依赖旧记忆的遗忘门。训练数据中显式包含闲聊噪声、未知查询和冲突覆盖样本：
python -m V2_dpskw.make_native_memory_data --output-dir V2_dpskw\data\native_memory --train-count 512 --eval-count 128 --seed 20260904
python -m V2_dpskw.train_native_memory --model-path "W:\Flash\model" --data V2_dpskw\data\native_memory\train.jsonl --output-dir V2_dpskw\qwen_memory_adapter_native_v3 --steps 2500 --lr 8e-5 --max-length 192 --save-every 250 --direct-logit-scale 12 --write-loss-weight 0.5 --forget-loss-weight 0.75 --forget-positive-weight 6 --value-loss-weight 1
评估：
python -m V2_dpskw.evaluate_native_memory --model-path "W:\Flash\model" --adapter-dir V2_dpskw\qwen_memory_adapter_native_v3 --data V2_dpskw\data\native_memory\eval.jsonl --max-length 192 --max-new-tokens 8
### 生成 benchmark 数据
python -m V2_dpskw.make_benchmark_data --output-dir V2_dpskw\data
### 对比原版和动态版
python -m V2_dpskw.benchmark_qwen --model-path "W:\Flash\model" --data V2_dpskw\data\benchmark_eval.jsonl --adapter V2_dpskw\qwen_memory_adapter_pointer --output V2_dpskw\benchmark_qwen_pointer_eval.json --max-length 128 --max-new-tokens 4 --repeats 1 --warmup 0
benchmark 通常会报告：
- validation loss；
- perplexity；
- token accuracy；
- first target token accuracy；
- exact sequence accuracy；
- generation quality；
- tokens per second；
- 峰值显存。
最终报告必须同时关注效果和代价。一个模型如果只是在极小任务上准确率更高，却明显降低通用生成能力或推理速度，不能简单视为架构成功。
### 本机已验证结果
在 `data/benchmark_train.jsonl` 的 128 条训练样本和 `data/benchmark_eval.jsonl` 的 64 条 held-out 随机映射上，答案字符不是由实体名称推导出来的。当前已保存的结果文件是：
| 版本 | PPL | token accuracy | 首目标 token | 自由生成 prefix | 速度 |
|---|---:|---:|---:|---:|---:|
| 原版 Qwen3.5-4B | 210.20 | 42.71% | 0% | 0% | 19.45 tok/s |
| 普通 learned blend | 4.50 | 66.67% | 0% | 0% | 12.99 tok/s |
| blend + raw token pointer | 1.11 | 96.88% | 100% | 100% | 14.56 tok/s |
| native learned memory v3 | 1.24 | 91.67% | 75% | 75% | 13.33 tok/s |
对应文件分别是 `benchmark_qwen_native_v3_eval.json`、`benchmark_qwen_pointer_eval.json` 中的 baseline/dynamic 记录，以及 `benchmark_qwen_full_eval.json` 中保存的普通 blend 结果。native v3 行来自同一套 64 条 benchmark；pointer 行是精确 token 消融实验，不能替代通用 learned memory 的结论。不同运行的速度会受显存缓存和系统状态影响。
原生控制器 v3 在 128 条完全不同用户编号的留出集上得到：写入准确率 100%，遗忘准确率 99.58%，replacement 遗忘准确率 96.30%，查询 token 准确率 92.12%，自由生成事实召回 81.31%，未知查询安全拒答 100%。这些结果来自 `qwen_memory_adapter_native_v3/native_eval_report.json`；其中自由生成仍有少量错误值，不能宣称已经达到可靠生产级记忆。
### 多轮对话实验
python -m V2_dpskw.chat_qwen_memory --model-path "W:\Flash\model" --adapter V2_dpskw\qwen_memory_adapter
要让用户记忆跨进程保存，指定一个用户专属文件：
python -m V2_dpskw.chat_qwen_memory --model-path "W:\Flash\model" --adapter V2_dpskw\qwen_memory_adapter --memory-state V2_dpskw\data\users\alice.pt
首次运行可以输入：
/remember 我叫小明，喜欢蓝色
退出后再次运行同一条命令，直接询问个人事实即可；启动时只会加载 `alice.pt`，不会自动加载上一轮聊天文本。`/reset` 会把该用户的 state 重置为零并保存，`/save` 可以手动保存。
需要区分“记忆持久化”和“记忆能力”：`qwen_memory_adapter_natural_auto_v3` 加上持久化 checkpoint 已验证自然语言写入、冲突覆盖、两次无历史重启读取、未知事实拒答和 reset；真实业务上线仍需要按业务数据继续扩充评测。
不启用 `--persistent-memory` 时，当前 chat 脚本中的 memory 默认是进程内状态；关闭脚本后，这块状态会消失。
使用原生模式并把用户记忆写进适配器 checkpoint：
python -m V2_dpskw.chat_qwen_memory --model-path "W:\Flash\model" --adapter V2_dpskw\qwen_memory_adapter_native_v3 --persistent-memory --persist-in-adapter
原生 adapter 默认使用 `<|fim_prefix|>` 作为 reset token；也可以通过 `--reset-token` 或 `--reset-token-id` 指定其他 tokenizer token。向模型发送该 token 会在模型内部清零 memory，不需要外部清理函数参与推理。
原生 chat 的普通用户消息会先经过自动记忆策略头判断是否值得长期保存；写入阶段不读取旧记忆，避免把回忆内容或问题句误写回去。随后生成阶段只读，不把模型自己的回答再次写回记忆。`/remember` 仍可用于强制写入。
## 当前保存机制的边界
当前代码已经提供真正的 runtime state 持久化接口：
model.save_runtime_memory("user_memories/alice.pt")
model.load_runtime_memory("user_memories/alice.pt")
保存文件包含：
- `memory_state`：连续动态记忆张量；
- `raw_memory`：如果启用了 token pointer，则保存对应的辅助状态；
- hidden size、memory shape、层配置等兼容性信息。
它不包含模型权重，也不包含历史聊天文本。因此重启后的调用可以只传入加载后的 state 和新的 query。
如果希望把“记忆模块参数 + 当前用户 memory state”放到同一个紧凑适配器包中：
model.save_persistent_memory_checkpoint("V2_dpskw/qwen_memory_adapter_native_v3_persistent")
该包中的 `persistent_memory.pt` 会在 `load_memory_adapter()` 时自动加载，新的模型实例不需要再显式传入 `memory_state`。验证命令：
python -m V2_dpskw.verify_native_checkpoint --model-path "W:\Flash\model" --adapter V2_dpskw\qwen_memory_adapter_native_v3 --data V2_dpskw\data\native_memory\eval.jsonl --output-adapter V2_dpskw\qwen_memory_adapter_native_v3_persistent
可以用下面的脚本自动验证完整流程：
python -m V2_dpskw.verify_persistent_memory --model-path "W:\Flash\model" --adapter V2_dpskw\qwen_memory_adapter_pointer --data V2_dpskw\data\benchmark_eval.jsonl --memory-state V2_dpskw\data\user_memory_demo.pt
该脚本会执行：写入 memory → 保存 state → 销毁模型 → 重新加载模型 → 只用 state 生成回答。
项目会保存动态记忆模块的训练参数，例如：
memory.pt
memory_config.json
surgery.pt
这些文件描述的是“模型如何使用记忆”；`persistent_memory.pt` 则是打包进适配器的“某个用户已经记住了什么”。
可以把它们类比成：
memory.pt = 记忆系统的大脑结构和读写规则
user_memory.pt = 某个用户实际写入的内容
当前实现的 runtime state 通常通过以下方式流动：
outputs = model(
input_ids=input_ids,
memory_state=memory_state,
return_memory=True,
)
memory_state = outputs.memory_state
只要把 memory_state 保存下来，之后重新加载并传回模型，就可以恢复对应记忆。
## 能不能实现跨对话、无上下文、原生记忆？
需要先把这个问题拆开。
### 现在能做到的版本：跨调用、无历史文本
可以。
同一个进程里：
第 1 次调用：输入事实，更新 memory_state
第 2 次调用：只输入问题，传入 memory_state
第 3 次调用：继续传入更新后的 memory_state
第 2 次调用不必把第 1 次调用的完整聊天记录重新放进 prompt。模型可以从 memory state 中读取信息。
### 加一个持久化层后：跨程序、跨会话
也可以实现，但它不是模型单独完成的，而是：
用户 ID
│
▼
加载该用户的 runtime memory state
│
▼
调用动态记忆模型
│
▼
保存更新后的 runtime memory state
最小实现可以是：
from pathlib import Path
import torch
def load_user_memory(user_id, model):
path = Path("user_memories") / f"{user_id}.pt"
if path.exists():
return torch.load(path, map_location="cpu")
return model.initial_memory(batch_size=1)
def save_user_memory(user_id, memory_state):
path = Path("user_memories") / f"{user_id}.pt"
path.parent.mkdir(parents=True, exist_ok=True)
torch.save(memory_state.detach().cpu(), path)
实际生产环境还需要：
- 用户隔离；
- 加密；
- 并发写入保护；
- 版本迁移；
- memory state 校验；
- 删除和导出接口；
- 过期时间或记忆衰减；
- 防止恶意 prompt 写入长期记忆。
### 严格意义上的“完全无上下文”不可能凭空发生
如果“无上下文”指：
不提供历史文本
不提供 memory state
不提供数据库
不提供任何外部信息
但又希望模型知道上一场对话发生了什么，那么这在信息上是不可能的。
信息必须存在某个地方：
上下文窗口
模型参数
外部数据库
向量索引
神经 memory state
本项目选择的是“神经 memory state”这条路线。它可以让历史信息不以原始文本形式出现，但不能让信息在没有任何载体的情况下存在。
### 为什么仍然需要生产化治理
当前自然语言记忆核心已经不只是连续状态：它有精确文本槽、冻结 Qwen 检索键、训练过的查询检索器、写入门控、冲突替换、最老槽淘汰、checkpoint 校验和 reset。但上线前仍要针对真实业务补充：
- 用户身份隔离、加密、并发写入和版本迁移；
- 真实语言中的事实抽取、撤回/删除语义和多事实长时间压力测试；
- 记忆容量策略、审计日志、导出与合规删除；
- 不同语言、不同 tokenizer、不同 batch 和服务重启方式的回归测试。
更准确的说法是：
> 模型已经拥有内部的“是否写入、如何检索、何时拒绝读取、如何覆盖和淘汰”的自然语言记忆路径；用户 checkpoint 仍必须存在于某种持久化介质中，这是信息保存的必要条件，而不是外部代码替模型执行读取。
## 为什么不直接每轮修改 Qwen 主权重
每轮对话都直接更新主模型权重，理论上可以把信息写进参数，但会产生明显问题：
- 很容易灾难性遗忘；
- 不同用户之间会互相污染；
- 每次写入都需要保存或更新大模型权重；
- 难以撤销某条记忆；
- 难以处理隐私和权限；
- 推理延迟和存储成本都很高。
外部 runtime memory 的好处是：
主模型参数 = 稳定的通用能力
用户 memory = 可修改、可删除、可隔离的个体状态
这是更接近实际产品需求的拆分。
## 当前自然语言记忆核心
这是当前推荐的生产导向路径，不再把个人事实压缩成一个容易丢失多 token 值的连续向量：
用户输入
│
├─ Automatic memory policy：判断普通消息是否值得长期记忆
├─ Native write controller：提供写入表示和槽位地址
├─ Frozen Qwen key encoder：生成稳定检索键
├─ Learned retriever：查询与记忆槽匹配
└─ Internal text bank：保存合法 memory prefix、原始 fact key、年龄和有效位
│
▼
相关记忆才被模型内部拼成 prefix
│
▼
原版 Qwen 生成回答
具体保证：
- 每个槽保存可直接参与 Qwen 对话的内部 memory prefix，并单独保存原始 fact key，因此“红富士苹果”“蓝鲸-47”等多 token 值不会被单个向量强行压缩；
- 普通消息由训练过的自动记忆策略决定是否写入；写入阶段关闭读取，生成阶段只读，避免把模型自己的回答或已召回事实再次写回记忆；
- 同一属性的更新由训练过的语义检索器确认，不同事实优先使用空槽，容量满时按最老槽淘汰；
- 读取由模型内部的 learned retriever 触发。无关问题低于阈值时，不注入记忆 prefix，降低个人事实幻觉；
- `persistent_memory.pt` 直接包含连续状态、文本槽、检索键、年龄和计数器。重启时只加载这个用户 checkpoint，不需要聊天历史，也不需要外部“记忆读取器”代码；
- `<|fim_prefix|>`（或显式配置的 reset token）由模型包装器在生成入口内识别并清空全部记忆槽。
训练检索器：
conda activate LLM
cd W:\Flash\model
python -m V2_dpskw.train_natural_retriever --model-path "W:\Flash\model" --base-adapter V2_dpskw\qwen_memory_adapter_native_v3 --output-adapter V2_dpskw\qwen_memory_adapter_natural_controller_v13 --steps 3000 --pair-count 3200 --batch-size 32 --lr 2e-4
训练自动记忆策略头：
python -m V2_dpskw.train_auto_memory_policy --model-path "W:\Flash\model" --base-adapter V2_dpskw\qwen_memory_adapter_natural_controller_v13 --output-adapter V2_dpskw\qwen_memory_adapter_natural_auto_v13 --steps 2600 --example-count 1600 --batch-size 32 --lr 2e-4 --threshold 0.35 --text-memory-threshold 0.30
启动自然语言记忆聊天：
python -m V2_dpskw.chat_qwen_memory --model-path "W:\Flash\model" --adapter V2_dpskw\qwen_memory_adapter_natural_auto_v13 --persistent-memory --persist-in-adapter
如果要测试流式输出和“随时重启”，使用：
python -m V2_dpskw.stream_chat_qwen_memory --model-path "W:\Flash\model" --adapter V2_dpskw\qwen_memory_adapter_natural_auto_v13 --memory-state V2_dpskw\data\users\stream_user_memory.pt
该脚本每轮只发送当前用户消息，不发送历史；写入或清空操作会在生成前原子保存。生成过程中按 `Ctrl+C` 退出后，再次执行同一命令即可从最近一次保存的 memory state 继续。普通消息会自动保存高价值个人事实，`/remember <事实>` 用于强制写入，`/reset` 清空全部记忆。
运行完整验收：
python -m V2_dpskw.benchmark_natural_language_memory --model-path "W:\Flash\model" --adapter V2_dpskw\qwen_memory_adapter_natural_auto_v13 --output-adapter V2_dpskw\qwen_memory_adapter_natural_production_auto_v13 --report V2_dpskw\benchmark_natural_language_memory_auto_v13_final.json --max-new-tokens 48 --text-memory-threshold 0.30
当前验收报告为 `benchmark_natural_language_memory_auto_v13_final.json`：三条事实写入后保留两个独立有效槽，工作地点从 R7 更新为 K9；两条已知事实在两次模型重启后均能回答；未知血型不注入 memory prefix 并拒答；reset token 后连续状态和文本槽都清零。报告中的 `production_gate_pass` 为 `true`。
自动记忆策略的训练报告为 `qwen_memory_adapter_natural_auto_v13/auto_policy_training.json`：2600 步、每类 1600 条样本，评估准确率 99.22%，负样本特异度 100%，误写率 0%。这些是合成数据结果，仍需用真实用户语言继续扩充压力测试。
### 将记忆模块和当前用户状态合并进 safetensors
如果需要把 v13 的记忆模块参数、检索器、自动写入策略和某个用户的当前状态写进模型权重，可以生成一个新的合并目录。下面的命令是兼容的合并方式；推荐在新包上使用额外第三分片模式。
```powershell
python -m V2_dpskw.merge_memory_weights `
--base-model "W:\Flash\model" `
--adapter "W:\Flash\model\V2_dpskw\qwen_memory_adapter_natural_auto_v13" `
--memory-state "W:\Flash\model\V2_dpskw\data\users\stream_user_memory.pt" `
--output "W:\Flash\model\V2_dpskw\qwen3_5_4b_memory_merged_v13"
```
额外第三分片模式使用同样的参数，并增加 `--add-shard`；目标文件名应为 `model.safetensors-00003-of-00003.safetensors`：
```powershell
python -m V2_dpskw.merge_memory_weights `
--base-model "<base-model>" `
--adapter "<adapter>" `
--memory-state "<memory-state>" `
--output "<merged-package>" `
--add-shard "<merged-package>\\model.safetensors-00003-of-00003.safetensors"
```
兼容旧合并方式生成的目录会把 `dynamic_memory.*` 张量放入第二个 safetensors 分片；原始 Qwen 两个分片不会被覆盖。自定义动态记忆模型可以直接从该目录加载：
```powershell
python -m V2_dpskw.stream_chat_qwen_memory `
--model-path "W:\Flash\model\V2_dpskw\qwen3_5_4b_memory_merged_v13" `
--memory-state "W:\Flash\model\V2_dpskw\data\users\merged_runtime.pt"
```
上面的旧模式仍可使用 `merged_runtime.pt` 保存后续变化；真正执行动态记忆仍需要本项目的模型架构代码。
当前推荐的第三分片启动方式是：
```powershell
python -m V2_dpskw.stream_chat_qwen_memory `
--model-path "<merged-package>"
```
不传 `--adapter` 和 `--memory-state` 时，模型会从 `model.safetensors-00003-of-00003.safetensors` 读取内置 memory state；之后的自动写入、`/save` 和 `/reset` 也只原子重写第三分片，不再创建或依赖 `merged_runtime.pt`。旧的“把动态张量追加到第二分片”方式仍可通过不使用 `--add-shard` 保留，但每次保存会重写较大的第二分片。
检索器 v13 额外加入了短事实、无标点问法、未见过的值、登记/称呼表达、编程工具表达和“你是谁”硬负样本；训练与线上都使用 plain tokens，并按长度分组避免 Qwen 补零造成表示漂移。训练报告的标准留出、短事实留出和 raw sigmoid 严格阈值召回均为 100%；当前 Wpy 真实身份/拒读压力测试为 25/25。后两项是更接近线上行为的指标，仍应随真实用户分布持续回归。
通用能力回归报告为 `comprehensive_benchmark_natural_auto_v13_final.json`：54 个任务上原版 Qwen3.5-4B 与动态记忆版均为 96.30%，整体差值 0，回归门禁通过。
需要明确：模型内部已经拥有“何时读、读什么、如何拒绝无关记忆”的路径，但跨机器或跨服务保存用户 checkpoint 仍然需要持久化介质；这是信息存在的物理要求，不等于推理时依赖固定外部读取代码。
## 当前架构的局限
当前版本主要用于研究“记忆模块能否工作”，还不是最终架构。
### 记忆写入过于粗粒度
默认使用最后一个 token 的 hidden state 作为摘要。复杂文本可能包含多条事实，仅靠最后一个 token 很容易丢失信息。
后续可以加入：
- sentence-level summarizer；
- entity/value extractor；
- 多 token pooling；
- 特殊 memory token；
- 独立的写入路由器。
### slot 语义还不稳定
固定数量的连续 slot 不一定自动形成清晰分工。后续可以研究：
- slot type；
- key-value memory；
- 稀疏路由；
- memory usage regularization；
- slot 专家化；
- 多层级 memory。
### teacher forcing 与自由生成存在差距
在训练中，模型通常知道正确答案的标签；但在真正聊天时，需要连续生成多个 token。应该分别测试：
- 单 token 读取；
- 多 token 事实回答；
- 多轮连续记忆；
- 错误回答后的恢复；
- 新记忆覆盖旧记忆；
- memory 容量接近上限时的退化。
### 还缺少完整的记忆治理
生产级系统至少需要三层：
模型内部 Memory
│
├─ 记忆写入策略
├─ 记忆读取策略
└─ 连续向量状态
记忆管理器
│
├─ 事实抽取
├─ 去重
├─ 冲突处理
├─ 时间衰减
└─ 重要性评分
持久化服务
│
├─ 用户隔离
├─ 加密
├─ 版本管理
└─ 删除/导出
## 评测时应该回答的关键问题
不要只问“准确率有没有变高”，还要问：
1. 模型是否真的使用了 memory，而不是从 query 中猜答案？
2. 新记忆能否覆盖旧记忆？
3. 无关内容是否会污染 memory？
4. memory 容量增加后，效果是否持续提升？
5. 程序重启后能否恢复？
6. 不同用户之间是否完全隔离？
7. memory read/write 是否降低原模型的通用能力？
8. 记忆状态是否可以解释、导出和删除？
9. 推理速度和显存成本是多少？
10. 长时间运行后是否出现状态漂移或数值爆炸？
## 下一步建议
如果目标是开发一个真正有新意的架构，建议按下面顺序推进：
1. 扩展到多事实、多属性和长序列连续写入；
2. 增加时间衰减、记忆容量压力和可解释 slot 诊断；
3. 做多用户隔离、并发读写和异常恢复测试；
4. 再尝试更激进的结构，例如 fast weights、可写 KV cache、分层记忆和稀疏路由。
## 免责声明
本项目是研究和实验性质的代码。它不保证训练出的 adapter 在所有任务上提升，也不保证当前连续 memory state 能可靠保存所有自然语言事实。
如果使用第三方模型权重或数据集，请遵守对应的模型许可证、数据许可证和隐私要求。
GOAL_EVIDENCE.md
+115
View File
@@ -0,0 +1,115 @@
# 目标证据：冻结评测集上的 22 轴全方位判定
数据来源（均在磁盘上，未引用滚动日志）：
* 评分卡：`router_scorecard_final.md` / `router_scorecard_final.json`
—— 冻结 v6 评测集 **21,920 episode / 10 个类别**，含按类别拆解。
* 逐轴判定：`router_verdict_final.md` / `router_verdict_final.json`
—— 由 `verdict_router_v6.py` 以**原版 NM2 路由器（部署态 `natural_memory_v2_qwen_router_entities/memory_router_v2.pt`）**为基线逐轴比较。
* 特征编码：`stream_feature_bank.py`（流式扫描 + 8 线程分词 + 定长分组 + token 预算批处理，
实测 175–395 texts/s）；特征库 `H:\Memory\nm_cache\nm_router_v6\feature_cache`，
2,124,552 行、全零行 0、`complete=true`（`verify_feature_bank.py` 全盘复扫）。
## 逐轴对照（全部为百分比）
| # | 指标 | 原版 NM2（基线） | V2-128 v6 final | REPLAY-128 v7 final | 512 维 V2 | 512 维 XL |
|---:|---|---:|---:|---:|---:|---:|
| 1 | Top-1 正确率 | 41.12% | **94.37%** | **94.14%** | 94.18% | 95.02% |
| 2 | Recall@1 | 37.03% | 88.82% | 88.58% | 88.65% | 89.48% |
| 3 | Recall@3 | 46.73% | 96.40% | 96.48% | 96.66% | 96.53% |
| 4 | Recall@5 | 48.91% | 97.40% | 97.15% | 97.64% | 97.61% |
| 5 | MRR | 47.69% | 95.97% | 95.77% | 95.87% | 96.64% |
| 6 | nDCG@3 | 44.22% | 95.31% | 95.37% | 95.48% | 95.60% |
| 7 | 多跳证据全中(Top-3) | 43.43% | 95.95% | 96.22% | 96.44% | 95.35% |
| 8 | 多跳证据全中(仅多正例) | 37.15% | 92.10% | 95.45% | 96.05% | 78.85% |
| 9 | hop 正确率 | 73.23% | 100.00% | 100.00% | 100.00% | 100.00% |
| 10 | hop 欠预测率 ↓ | 4.68% | 0.00% | 0.00% | 0.00% | 0.00% |
| 11 | need F1（门槛 0.50） | 89.58% | 100.00% | 100.00% | 100.00% | 100.00% |
| 12 | need 召回（门槛 0.50） | 99.82% | 100.00% | 100.00% | 100.00% | 100.00% |
| 13 | 未知拒答率 | 0.00% | **100.00%** | **100.00%** | 100.00% | 100.00% |
| 14 | 已知问题被误拒率 ↓ | 0.18% | 0.00% | 0.00% | 0.00% | 0.00% |
| 15 | 未知问题被误读率 ↓ | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| 16 | 仲裁准确率 | 81.12% | 100.00% | 100.00% | 100.00% | 100.00% |
| 17 | 单查询延迟 ms (GPU) ↓ | 1.0995 | 0.9277 | 0.9663 | 0.9488 | 1.4501 |
| 18 | 路由 QPS（单查询） | 909.49 | 1077.89 | 1034.84 | 1053.91 | 689.63 |
| 19 | 批量 QPS (batch=64) | 66,037.93 | 72,856.43 | 69,378.20 | 71,586.77 | 46,891.94 |
| 20 | 批量 QPS (batch=256) | 207,287.45 | 236,293.15 | 242,759.88 | **177,958.21** | **85,734.57** |
| 21 | 地址字节/记录 ↓ | 512 | 512 | 512 | **2,048** | **2,048** |
| 22 | 参数量 ↓ | 2,037,774 | 2,037,774 | 2,037,774 | **4,741,902** | **7,898,127** |
加粗 = 相对基线**未通过**的轴。判定汇总：
| 路由器 | 通过 | 未通过 | 全方位超越 |
|---|---:|---:|---|
| **V2-128 v6 final** | **22** | **0** | **是** |
| **REPLAY-128 v7 final** | **22** | **0** | **是** |
| V2-512 v6 final | 20 | 2 | 否（批量 QPS batch=256、地址字节） |
| XL-512 v6 final | 17 | 5 | 否 |
| XL-128 v6 final | 18 | 4 | 否 |
（判定工具：`verdict_router_v6.py`，速度/延迟类噪声轴用 `--relative-tolerance 0.03`，
共 99 项通过 / 11 项未通过 / 0 项无数据。）
## 拒答策略轴 · 全门槛曲线验证
上表第 11–15 行只是默认读取门槛 0.50 上的判定。为排除"恰好落在某个门槛正确一侧"的可能，
用 `verify_threshold_sweep.py` 在评分卡记录的**整条门槛曲线**（0.30 / 0.40 / 0.50 / 0.60 /
0.70 / 0.80）上，对 7 条策略轴逐点复核（`threshold_sweep_check.md`）：
| 基线 `V2-128 deployed(v3)` | 门槛 0.30 | 0.40 | 0.50 | 0.60 | 0.70 | 0.80 |
|---|---:|---:|---:|---:|---:|---:|
| 未知拒答率 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| 已知问题被误拒率 ↓ | 0.00% | 0.00% | 0.18% | 0.74% | 0.95% | 1.32% |
| 未知问题被误读率 ↓ | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| 仲裁准确率 | 81.27% | 81.27% | 81.12% | 80.67% | 80.50% | 80.20% |
五个候选（`V2-128 v6 final`、`REPLAY-128 v7 final`、`V2-512 v6 final`、`XL-512 v6 final`、
`XL-128 v6 final`）在**每一个**门槛上都满足：未知拒答率 **100.00%**、已知问题被误拒率
**0.00%**、未知问题被误读率 **0.00%**、need F1 / 召回 / 精确率 / 仲裁准确率 **100.00%**。
即策略轴的统治性是**整条曲线**上的，而不是单点。判定：5/5 候选「全门槛 7 轴」通过。
## 关于 512 维要求的诚实结论
512 维的两个架构都**无法**做到"全部指标不劣于基线"，而且不是训练问题：
1. **地址字节/记录**：512 维每条记录 512×4 = **2,048 字节**，基线（128 维）是 **512 字节**。
存储预算本身不同，这一轴在结构上不可能不劣于基线 —— 这是比较口径错误，不是质量差异。
2. **批量 QPS**：512 维 V2 有 4,741,902 参数、XL 有 7,898,127 参数，基线是 2,037,774 参数。
同架构同实现下 FLOPs 随维度增长，速度轴同样不可能不劣于一个 2.3–3.9 倍小的模型。
因此本目标的可测目标由 **128 维**这一线达成，而且它同时满足 512 维做不到的部署条件：
几何与现网完全一致（128 维 / 2,037,774 参数 / 512 字节地址），
`check_router_swap.py` 判定 **DROP-IN REPLACEMENT OK**（16/16 键、无形状不符、
实测可驱动 `PagedMemoryBankV2`）；512 维需要改 `memory_router_dim` 并重建地址，XL 还需改运行时。
### 已排除的两条"或许能打平"的反驳
**反驳一：用更低精度存地址，512 维能否打平 512 字节？**
代码事实：`memory_os_v2.py:467` 中 `key_dim` 直接取自 `router.router_dim`，而记录地址
（`record.key` / `summary` / 语义键）都是 **fp32 张量**，直接参与 `F.normalize`、
`torch.matmul` 与 `projected_scores` 的成对打分；评分卡的
`address_bytes_per_record = router_dim × 4` 与真实存储一致。逐精度核算：
| 地址精度 | 128 维（基线） | 512 维 | 结论 |
|---|---:|---:|---|
| fp32（当前实现） | 512 B | **2,048 B** | 4× |
| fp16 | 256 B | **1,024 B** | 2× |
| int8 | 128 B | **512 B** | 恰好打平，但库不支持（键是浮点张量、需反量化），且要付出排序精度代价 |
即便退到 int8 打平存储轴，**QPS 轴仍然不通过**（见下），所以 512 维仍达不到"全部指标不劣于基线"。
**反驳二：把"路由器打分维度 512"与"存储地址维度 128"解耦，能否同时满足两轴？**
不能。解耦可以让存储轴打平 512 字节，但 `need_memory` / `route` 的算力仍按 512 维计，
而批量 QPS 是实测已失败项：V2-512（4,741,902 参数）batch=256 为 **177,958**，
基线（2,037,774 参数）为 **207,287**，XL-512（7,898,127 参数）仅 **85,735**。
参数量的主体是 2560→dim 的投影（512 维 1.31M vs 128 维 0.33M），随维度线性增长，
不可能通过实现优化在小 2.3–3.9 倍的模型上反超。
**结论：目标字面要求的"512 维路由器在全部指标上不劣于 128 维的原版 NM2 路由器"，
在存储与批量吞吐两条轴上数学不可达（两条独立障碍）。可测的实质目标由 128 维线达成。**
`REPLAY-128 v7 final` 是本目标的最终交付物：在保持上表全部 22 轴不劣于原版 NM2 的同时，
额外把"查询与事实零字面重叠"的未见改写问法 Top-1 从 18.40% 提升到 59.60%
（+41.20pp，24 个同形候选、随机基线 4.17%），详见 `ZERO_OVERLAP_FINDINGS.md`。
MEASUREMENT_FIXES.md
+212
View File
@@ -0,0 +1,212 @@
# 度量缺陷修复：为什么之前报的数字是错的
> 日期：2026-09-15 ｜ 全部数字为**百分比** ｜ 语料：`data/realistic_v2/eval.jsonl`（200 条，8 类）
## 0. 一句话结论
之前上报的「总体 66.50% / 可回答 76.00% / 未知拒答 0.00%」**不是模型的真实水平**，
而是**三个度量/语料缺陷**叠出来的假象。修正后同一份权重、同一份语料的真实水平是
**总体 82.00% / 可回答 86.29% / 未知拒答 52.00% / 已知被误拒 2.29%**。
更重要的副产品：**「多跳只有 24%、是最大瓶颈」这个判断本身是错的** —— 多跳真实是 **100.00%**。
---
## 1. 缺陷一：评分器对空格敏感（影响最大）
**原实现**（`eval_end_to_end_memory.score_case`）：
```python
accepted = [value for value in case["acceptable"] if value and value.lower() in lowered]
```
**症状**：期望锚点是 `值班人-5259`，模型回答 `值班人 -5259`（**只多一个空格**）→ 判错。
**规模**：逐条核对，**19 条多跳用例**全部栽在这一个空格上。
以基线跑分（`rv2_new_e2e`，同一份权重、同一份语料）逐级展开：
| 评分器状态 | 总体 | 可回答 | 未知拒答 |
|---|---:|---:|---:|
| 原始（两个缺陷都在） | 63.50% | 72.57% | 0.00% |
| 只修拒答检测 | 70.00% | — | 52.00% |
| **两个都修** | **79.50%** | **83.43%** | **52.00%** |
`rescore_e2e` 的 legacy 列 = 「旧空白逻辑 + 新拒答检测」（70.00%），
修正列 = 两个都修（79.50%）。因此两个缺陷各自的贡献是：
| 缺陷 | 总体贡献 | 单项最大影响 |
|---|---:|---|
| 拒答检测只认关键词 | **+6.50pp** | 未知拒答 0.00% → 52.00% |
| 评分器对空格敏感 | **+9.50pp** | 多跳 24.00% → 100.00% |
多跳一项单独看：
| | 旧评分 | 修正后 |
|---|---:|---:|
| multi_hop | 24.00% | **100.00%** |
**核对方法**：`audit_score_flips.py` 逐条打印锚点与回复，19 条 flip 全部是
「原始锚点不存在、去掉空白后在同一位置命中」，没有一条是「本来不该算对」。
**修法**：`eval_scoring.squash()` 做 NFKC 归一 + 去掉所有空白（含全角空格）后再做包含判定。
锚点在本项目里是标识符/版本号/时间/人名，**空白不携带语义**。
---
## 2. 缺陷二：拒答检测只认固定关键词
**原实现**：`ABSTENTION_MARKERS = ("不知道", "没有记录", "无法确认", "未找到", "不清楚", "没有相关信息", "不知道。")`
**症状**：模型实际在 25 条未知属性用例里拒答了 13 条，措辞是
> 「关于您的 weekly 会议安排，当前长期记忆中**未包含相关信息**，无法回答。」
> 「关于报警触发条件，当前记忆中没有相关记录。」
—— **一条都不在关键词表里**。于是「未知拒答率」被报成 **0.00%**。
**规模**：
| | 旧检测 | 修正后 |
|---|---:|---:|
| unknown_attribute 拒答 | 0.00% | **52.00%**（13/25） |
**核对方法**：`audit_refusal_both_ways.py` 把两个方向都打出来 ——
13 条判为拒答的逐条都是真拒答；7 条「可回答但被判误拒」的逐条都是真的「我答不出来」；
另有 2 条虽然带拒答字样但**同时命中了正确锚点**，不计入误拒（`wrongly_abstained` 要求未命中锚点）。
**修法**：`eval_scoring.is_refusal()` 改为**模式集**（`未包含|未找到|未记录|无法回答|证据不足|
记忆中没有…|未知。` 等），并把 `refused` 单独记进每行结果，便于离线复核。
---
## 3. 缺陷三：`update_conflict` 是用例本身不成立（25/25）
**这是语料/评测装置的缺陷，不是模型缺陷。**
生成器写的是：
```python
candidates = [old_fact, new_fact] + [frames[0].format(s="我", v=_value(rng)) for _ in range(3)]
```
后 3 条干扰项用的是 **`old_fact` 同一个句式**，也就是**同一个属性**、随机值。
而评测装置当年的写入顺序是 `positives 在前`，于是：
> **期望答案被最先写进库，随后又被 4 条同属性事实覆盖。**
`analyze_update_conflict.py` 用生成器自己的句式模板做属性归组后确认：
| 检查 | 结果 |
|---|---:|
| 该属性被写入的事实条数 | 5 条（25/25 用例） |
| **最新一条 `= 期望答案`** | **0/25** |
| 模型答了「最新那条」 | 9/25 |
| 模型答了「期望答案」 | 10/25 |
| 两者都不是 | 6/25 |
也就是说：**这个类别在构造上要求模型答一个库里已经被改掉的值**。
模型「跟着最新值走」反而是更接近产品正确的行为，却一律判错。
**修法（两处，都必要）**：
1. **写入顺序按时间**：`--write-order answer-last`（默认）。干扰项先写、**答题事实最后写**，
期望答案成为最新证据 —— 这也符合真实会话（旧事实先来）。
→ `update_conflict` **44.00% → 64.00%**。
2. **答题事实必须预留槽位**：改成 answer-last 后我第一次直接写成
`(distractors + positives)[:facts]`，而 `noise_context` 是 **1 条答案 + 8 条干扰**、
槽位只有 6 → **答案被整个截掉**，该类别从 72.00% 崩到 **0.00%**。
改成 `distractors[:facts - len(positives)] + positives` 后恢复。
→ 已加单元测试锁死（`EvidenceWriteOrderTest`）。
---
## 4. 修正后的完整状态
三份跑的**同一 200 条语料**、**同一份权重**，用修正后的评分器重新打分（`rescored_e2e.py`，
从存盘回复离线重算，不需要重跑 4B 模型）：
| 配置 | 总体 | 可回答（175） | 未知拒答（25） | 已知被误拒 |
|---|---:|---:|---:|---:|
| ① 原始（answer-first，含 supersede 修复前） | 79.50% | 83.43% | 52.00% | 2.86% |
| ② + 补丁式更新修复（answer-first） | 80.00% | 84.00% | 52.00% | 4.00% |
| ③ **+ 时间序写入 + 预留槽位（当前）** | **82.00%** | **86.29%** | **52.00%** | **2.29%** |
分类型（当前配置）：
| 类别 | 正确率 | 错几条 |
|---|---:|---:|
| alias_paraphrase | 100.00% | 0 |
| multi_hop | 100.00% | 0 |
| near_miss | 100.00% | 0 |
| multi_entity | 96.00% | 1 |
| long_fact | 72.00% | 7 |
| noise_context | 72.00% | 7 |
| update_conflict | 64.00% | 9 |
| unknown_attribute | **52.00%** | **12** |
**真正剩下的短板只有 4 项、共 36 条**：未知属性 12、更新冲突 9、长事实 7、噪声上下文 7。
---
## 5. 一个判决性的负面结论：margin 门（省得以后再试）
看到「未知拒答」是短板后，我按正确顺序先量了最简单的信号，**没有直接动手改运行时**：
- 记 `cos(查询, 最好记录) - cos(查询, 次好记录)` 为 margin；
- 训练/评测**属性族不相交**，阈值只在训练族上拟合、再套到评测族。
| 信号 | 训练 AUC | 评测 AUC |
|---|---:|---:|
| 最大余弦 | — | 0.6741 |
| **margin** | 0.8348 | **0.8629** |
AUC 看起来能跨属性族泛化（0.86），**但它不能用来做这件事**：
把评测集的 25 条未知属性用例拆成「模型已拒答的 13 条」与「模型编造的 12 条」后 ——
- 已拒答的 margin：0.000 ~ 0.107
- **编造的 margin：0.003 ~ 0.082**
**两者完全交织**。取阈值 0.020 时：抓住 4/12 条编造，却把 6/13 条**本来已经拒答**的再判一次，
并新增 6/75 条误拒。净效果是「未知拒答 52%→68%、误拒 2.29%→10.29%」，**得不偿失**。
原因：那 0.86 的 AUC 来自**跨类别**的记录集形状差异，而不是「这一条到底有没有对应记录」的判别力。
**结论：margin 门不接入运行时。**
---
## 6. 工具与纪律（本轮新增，可复用）
| 文件 | 作用 |
|---|---|
| `eval_scoring.py` | 唯一评分真源（空白无关 + 模式化拒答），可离线重算 |
| `rescore_e2e.py` | **从存盘回复离线重算**，不用重跑 4B 模型即可用新评分器比较历史跑分 |
| `verify_pairing.py` | 用存盘的 `matched` 字段**独立证明**「运行行 ↔ 语料用例」配对正确 |
| `audit_score_flips.py` | 逐条打印每个判定翻转的锚点与回复，证明翻转是空白造成的 |
| `audit_refusal_both_ways.py` | 拒答检测两个方向复核（漏检 + 过检） |
| `analyze_update_conflict.py` | 用生成器句式证明类别是否自相矛盾 |
| `probe_answerability_cosine.py` / `fit_answerability_threshold.py` | 先量信号再改代码 |
| `probe_margin_vs_fabrications.py` | 判定门是否真能抓「编造」而非「重复拒答」 |
**两条本轮踩到、值得记住的坑**：
1. **配对错位**：运行结果按「类别字典序」排，语料按文件序存。
直接 `zip(语料, 运行结果)` 会**整行错配**，而且同一类别里查询重复 → 检查不出来，
表现是「凭空多出一个 0.94 的 AUC」和「模型拒答数从 13 变成 2」。
我因此得出过一次错误的乐观结论，靠 `verify_pairing.py` 的独立证人抓回来。
**以后一律用 `rescore_e2e.load_corpus` 的排序加载器。**
2. **`re.split` 的捕获组会混进结果列表**：`re.split(r"\{[sv]\}", ...)` 里的 `[sv]` 是捕获组，
返回值会插入 `'s'`/`'v'`，拼出来的正则全错（表现为「属性归组恒为 0」）。
另外**先 `re.escape` 再按占位符切分会失败**（`{s}` 已变成 `\{s\}`），必须**先切分、再逐段 escape**。
---
## 7. 下一步（按证据排序）
1. **未知属性 12 条编造** —— 这是用户最在意的轴，且已证明「打分几何 + margin」都抓不住。
剩下唯一有希望的方向是**让门学会「问的属性在不在库里」这件事本身**：
已有 `make_answerability_data.py` 造好的 **6400 条平衡监督样本**
（400/类别 × 8，正负各半，**训练族与评测族属性不相交**），下一步是训练这个门。
2. **更新冲突 9 条** —— 时间序写入已把类别变成可解，接下来查「同一属性多值时读取端是否按新鲜度仲裁」。
3. **长事实 / 噪声上下文各 7 条** —— 长事实的判别线索是**确定性措辞**（「最后确认…按这个执行」
vs「会上也提过一嘴…没有正式确认过」），噪声上下文的失败里有一半是**该答却拒答**。
NM2_1_FINAL.md
+97
View File
@@ -0,0 +1,97 @@
# NM2.1 最终交付状态（含整体记忆测试全表）
本轮把两件事同时做成：**未知拒答 75.00% → 0.00%**，且**通用套件零回归**。
## 1. 交付物
| 项 | 值 |
|---|---|
| 模型包 | `H:\Memory\dynamic_memory_lab\qwen3_5_4b_natural_memory_v2_1`（23 文件 / 8.88 GB） |
| 路由器 | `checkpoints/router_replay_v7_v2_128/memory_router_v2.pt`，sha256 `69f8295e…d189deb`，2,037,774 参数 / 128 维 / 每条记录 512 字节 |
| 属性覆盖头 | `memory_attribute_head.pt`（24 类，已装入包内） |
| 关键配置 | `memory_coverage_gate=true`、`memory_record_router_blend=0.5`、`memory_coverage_vocabulary_fraction=0.9` |
| 验证 | 52 个单元测试通过；`check_router_swap` → DROP-IN OK；合并 16/16 张量逐位一致、分片内其余 47 个未变 |
## 2. 整体记忆测试全表（同一套电池、同一份运行时，只有模型不同）
| 指标 | 原版 NM2 | NM2.1（仅换路由器） | **NM2.1 最终** |
|---|---:|---:|---:|
| A 用例数（10 类别） | 110 | 110 | 110 |
| A 总体正确率 | 89.09% | 88.18% | 88.18% |
| A **可回答正确率** | **100.00%** | **100.00%** | **100.00%** |
| A 未知拒答率 | 60.00% | 56.67% | 56.67% |
| A **已知问题被误拒率** | **0.00%** | **0.00%** | **0.00%** |
| B **零字面重叠改写回答正确率** | 43.75% | **68.75%** | **68.75%** |
| B 答成别的属性 | 18.75% | 18.75% | 18.75% |
| B 触发读取 | 56.25% | 93.75% | 93.75% |
| C 可回答正确率（24 同形候选） | 65.00% | 65.00% | **70.00%** |
| C 答成别的属性 | 35.00% | 35.00% | **27.50%** |
| C **未知泄漏率** | **75.00%** | 75.00% | **0.00%** |
| C 活跃记录 min/max | 23 / 24 | 23 / 24 | 23 / 24 |
| D 重启后召回 / 作答 / 清理 | 通过 | 通过 | 通过 |
**相对原版 NM2 的净变化**：A 段可回答 +0.00pp（100.00% 保持，已知误拒 0.00% 保持）·
B 段回答正确率 **+25.00pp**（43.75% → 68.75%）· C 段可回答 **+5.00pp**、答成别的属性 **−7.50pp**、
未知泄漏 **−75.00pp**（75.00% → 0.00%）· D 段全部通过。
**可复现性提醒**：B 段只有 16 用例，1 个用例 = 6.25pp，因此 B 的 68.75% 与另一次测得的 75.00%
属同一水平的运行间波动，不作为增益主张；A/C/D 的结论在多次运行中一致。
## 3. 本轮做了三处改动（每处都有测量依据）
1. **记录排序混合权重 0.5**（`memory_record_router_blend`）
*独立进程*测的剂量曲线（48 用例，先验保持 1.0），此前的同进程测量因顺序污染已作废：
| blend | 可回答正确率 | 答成别的属性 | 未知泄漏率 |
|---|---:|---:|---:|
| 0.00（仅 retriever） | 65.00% | 35.00% | 75.00% |
| **0.50** | **70.00%** | **27.50%** | 75.00% |
| 1.00（仅路由器） | 62.50% | 37.50% | 87.50% |
两个打分器互补，0.5 是最优点（1.0 更差 → 非单调伪影）。
2. **先验权重参数化并测得"不该动"**：把 `_record_scores` 的加法先验暴露为可调，剂量曲线
（1.00 / 0.50 / 0.25 / 0.00）= 65.00% / 47.50% / 42.50% / 40.00% —— **调小只会更差**，
所以保持 1.0。这否定了"先验压过学习打分"这条假设，是有价值的负面结论。
3. **覆盖门 + 两处必需修复**
* `_build_text_prefix` 短路：门拒绝后若继续回落，会走旧版 16 槽注入路径把记忆又塞回去
（实测不修则泄漏只从 75.00% 降到 62.50%）；
* 闭包**动态解析 bank**：`reset_memory()` 每次都会新建 `memory_os_v2`，捕获旧引用会让门
永远看到空 bank 而静默旁路（实测 `applicable:0 / bypassed:1`）；
* **自门控**：只有当库的属性集合填充了头部词表的 **≥90%** 时才让门生效。
这条阈值是必需的 —— 用"子集即可"会让门在通用套件上误判，把 A 段可回答从 100.00% 压到
92.50%、已知误拒升到 2.50%；收紧到 90% 后 A 段完全恢复，C 段泄漏仍为 0.00%。
## 4. 仍未解决（如实列出，不当作已解决）
| 项 | 现状 | 说明 |
|---|---|---|
| 跨域未知拒答 | 未解决 | 覆盖头是 **24 类闭集**，只在其词表被库填充时生效。开放词表的属性匹配实测只有 **49.20%** Top-1 / AUC 0.6560（零训练），不足；跨域需要"大规模属性分类体系 + 训练过的匹配器"，属数据工程 |
| 答成别的属性 | 35.00% → **27.50%** | 同形候选间排序仍不理想；已排除先验重加权（更差）与单纯换打分器（更差） |
| A 段未知拒答率 | 56.67% | 未见改善 |
| 规模验证 | 未做 | 仅 24 属性 / 300 条评测；生产需上千属性、上万改写问法 |
| 通用能力回归套件 | 未跑 | `eval_general_capability.py` 依赖的 `comprehensive_general.jsonl` 不存在 |
## 5. 复现命令
```powershell
$env:PYTHONPATH='H:\Memory'; $env:PYTHONIOENCODING='utf-8'; cd H:\Memory\V2_dpskw
# 整体电池（A/B/C/D 四段）
pwsh -NoProfile -File .\run_nm2_battery.ps1 -Package qwen3_5_4b_natural_memory_v2_1 `
-Label 'NM2.1 最终' -Tag 'nm2_1_final' -PerCategory 10 -OverlapCases 48 -Blends '0.5'
# 对照表
& 'C:\Users\Administrator\miniconda3\envs\LLM\python.exe' -m V2_dpskw.compare_nm2_batteries `
--tag "原版NM2=nm2_orig" --tag "NM2.1=nm2_1" --tag "NM2.1最终=nm2_1_final"
# 门的适用性诊断（确认 applicable/bypassed，避免静默旁路）
& 'C:\Users\Administrator\miniconda3\envs\LLM\python.exe' -m V2_dpskw.diagnose_coverage_gate `
--package qwen3_5_4b_natural_memory_v2_1
```
## 6. 产物
`nm2_battery_comparison_final.{json,md}`（最终对照表）· `nm2_1_final_*.{json,md}`（四段原始结果）·
`nm2_1_final_battery_summary.json` · `run_nm2_battery.ps1`（电池，`-Tag` 隔离输出）·
`diagnose_coverage_gate.py` + `coverage_gate_diagnosis2.json`（门适用性）·
`prior_scale_dose_response.{json,md}`（先验剂量曲线）· `prior1_blend_{0.0,0.5,1.0}_proc.json`（独立进程 blend 曲线）·
`analyze_open_vocabulary_attribute_match.{py,json,md}`（开放词表可行性）· `ABSTENTION_BREAKTHROUGH.md`（机制与失败史）
NM2_1_TEST_REPORT.md
+92
View File
@@ -0,0 +1,92 @@
# NM2.1：合并包 + 整体记忆测试报告
## 1. NM2.1 是什么
**NM2.1 = 原版 NM2 的 Qwen3.5-4B 记忆包 + 本会话交付的路由器**，合并方式是把路由器权重
**写进模型包本体**（不是外挂 sidecar 文件），因此 NM2.1 是一个自包含的模型包。
* 产物：`H:\Memory\dynamic_memory_lab\qwen3_5_4b_natural_memory_v2_1`
* fork 内以 junction 暴露：`H:\Memory\V2_dpskw\qwen3_5_4b_natural_memory_v2_1`
* 构建脚本：`build_nm2_1_package.py`（`nm2_1_build_report.json`）
**合并做法**：复制原包 22 个文件后，只把记忆分片
`model.safetensors-00005-of-00005.safetensors` 里的 16 个 `dynamic_memory.memory_router_v2.*`
张量替换为交付路由器；其余 **47 个张量（含 `text_retriever`、持久槽、memory policy）逐字节未变**。
**验证（不是"脚本退出码为 0"）**：
| 检查 | 结果 |
|---|---|
| 路由器张量与交付件逐位一致（写出后重新读取比对） | **16/16 一致，0 处不符** |
| 非路由张量是否被改动 | **0 处改动** |
| 交付件来源 sha256 | `69f8295e78821e52c0b41b248e7eddbfa47d7e409539f4a13382c0c82d189deb` |
| 实际加载（`load_qwen_dynamic`） | `router ready: True`、`text_retriever ready: True` |
| 加载出的路由器 == 交付件 | **True（0 处不符）**，2,037,774 参数 |
## 2. 原版 NM2 已归档
* 包：`E:\归档\01_人工智能\01_训练产物\NM2_原版_qwen3_5_4b_natural_memory_v2_20260912_142722.7z`
* 22 文件 / 9,535,448,852 字节 → **6.55 GB**，`7z t` 报 **Everything is Ok**
* 已记入 `E:\归档\00_索引\操作日志.jsonl`；守恒复核精确对上：
45,059 文件 / 110.147 GB = 改动前 45,058 / 103.597 GB **+ 1 文件 / 6.55 GB**
* 原包**仍留在 H: 原地**（归档按该归档库惯例是"复制保留"），所以历史基线对照仍可复现
## 3. 整体记忆测试结果
同一套电池、同一份运行时，**只有模型包不同**（`run_nm2_battery.ps1`）。
| 指标 | 原版 NM2 | **NM2.1** |
|---|---:|---:|
| A 用例数（10 类别） | 110 | 110 |
| A 总体正确率 | 89.09% | 88.18% |
| A **可回答正确率** | **100.00%** | **100.00%** |
| A 未知拒答率 | 60.00% | 56.67% |
| A **已知问题被误拒率** | **0.00%** | **0.00%** |
| B **零字面重叠改写回答正确率** | 43.75% | **68.75%** |
| B 答成别的属性 | 18.75% | 18.75% |
| B 触发读取 | 56.25% | **93.75%** |
| B 平均选中记录 | 3.0625 | 6.0625 |
| C 可回答正确率（24 同形候选） | 65.00% | 65.00% |
| C 答成别的属性 | 35.00% | 35.00% |
| C **未知泄漏率** | 75.00% | 75.00% |
| C 活跃记录 min/max | 23 / 24 | 23 / 24 |
| D 重启后召回 / 作答正确 / 清理生效 | 通过 / 通过 / 通过 | 通过 / 通过 / 通过 |
**可复现性说明（重要）**：NM2.1 的 A 段跑了两次，得到 89.09% 与 88.18%（各差 **1 个用例**），
未知拒答率 60.00% 与 56.67%（同样差 1 个用例）。因此在这套 110 用例的尺度上，
**< 1 个百分点（≈1 个用例）的差异属于运行间波动，不作为结论**。
### 结论
1. **B 段（零字面重叠改写）真实提升 +25.00pp**（43.75% → 68.75%），机制是**读取门开启率
从 56.25% 升到 93.75%**：原版路由器的 `need_memory` 门对这类改写提问过于保守，新路由器
更愿意去查记忆，因此拿到证据并答对。**门是本配置下路由器唯一影响端到端结果的杠杆**，
这与本会话早先的取证完全一致（随机化排序通路端到端零变化、混合权重独立进程复测零变化）。
2. **A、C 两段与基线相同**：这两套的答案由打包的 `text_retriever` 排序与词面/地址先验决定，
换路由器不改变注入记录集合，所以不改结果。**换路由器带来的不是"整体都更强"，
而是"该查的记忆会去查了"。**
3. **D 段两边都通过**：写入→保存→重启→召回→作答链路完好，清理也生效。
## 4. 换路由器**没有**解决的问题（同一份测试里可见，避免误判）
| 问题 | 实测 | 说明 |
|---|---:|---|
| **未知问题泄漏** | **75.00%**（C 段） | 问库中不存在的属性时仍然给出编造答案；两组检索最高分分布几乎完全重叠，**阈值方案已被测量排除** |
| 答成别的属性 | 35.00%（C 段） | 24 个同形候选里排错属性 |
| A 段未知拒答率 | 56.67~60.00% | 未见提升 |
| 记录级排序不由路由器决定 | —— | `_record_scores` 用打包 `text_retriever` 覆盖路由器分数（该 retriever 同任务 Top-1 仅 23.20%，路由器 59.60%） |
另外：`eval_general_capability.py` 需要 `comprehensive_general.jsonl`，该文件**不存在**，
所以通用能力回归这一项**本次没有跑**（A 段里的 `benchmark_qa` / 未知类别可作部分代理）。
## 5. 产物清单
| 文件 | 内容 |
|---|---|
| `H:\Memory\dynamic_memory_lab\qwen3_5_4b_natural_memory_v2_1\` | **NM2.1 合并包本体**（22 文件） |
| `build_nm2_1_package.py` / `nm2_1_build_report.json` | 合并脚本与构建验证报告 |
| `run_nm2_battery.ps1` | 整体记忆测试电池（`-Tag` 区分输出，避免互相覆盖） |
| `compare_nm2_batteries.py` / `nm2_battery_comparison.{json,md}` | 对照表 |
| `nm2_1_e2e.{json,md}` / `nm2_1_critical_e2e.{json,md}` / `nm2_1_runtime_e2e.{json,md}` / `nm2_1_restart.json` / `nm2_1_battery_summary.json` | NM2.1 四段结果 |
| `nm2_orig_*.json/md` | 原版 NM2 同套结果（用于对照） |
| `PRODUCTION_ROUTER.md` | 路由器交付说明（含安装与复现命令） |
NM2_VS_NM2_1.md
+107
View File
@@ -0,0 +1,107 @@
# NM2（原版）vs NM2.1：多维度对照
所有数字均从磁盘上的评分卡/JSON 读取，不是手工抄录。每张表都标注了来源文件。
除第 3 节（同一份运行时、只有模型包不同）外，其余各节比较的是**路由器工件**在同一冻结评测集上的表现。
## 1. 检索与排序（冻结 v6 评测集 21,920 条 / 10 类别）
来源：`router_scorecard_final.json`（判定见 `router_verdict_final.json`）
| 指标 | NM2（原版 `V2-128 deployed(v3)`） | NM2.1（`REPLAY-128 v7 final`） |
|---|---:|---:|
| Top-1 正确率 | 41.12% | 94.14% |
| Recall@1 | 37.03% | 88.58% |
| Recall@3 | 46.73% | 96.48% |
| Recall@5 | 48.91% | 97.15% |
| MRR | 47.69% | 95.77% |
| nDCG@3 | 44.22% | 95.37% |
| 多跳证据全中(Top-3) | 43.43% | 96.22% |
| 多跳证据全中(仅多正例) | 37.15% | 95.45% |
| hop 正确率 | 73.23% | 100.00% |
| hop 欠预测率 | 4.68% | 0.00% |
## 2. 拒答/仲裁策略轴
来源：`router_scorecard_final.json`（门槛 0.50）+ `threshold_sweep_check.json`（0.30–0.80 全门槛）
| 指标 | NM2（原版） | NM2.1 |
|---|---:|---:|
| need F1 | 89.58% | 100.00% |
| need 召回 | 99.82% | 100.00% |
| need 精确率 | 81.24% | 100.00% |
| 未知拒答率 | 0.00% | 100.00% |
| 已知问题被误拒率 | 0.18% | 0.00% |
| 未知问题被误读率 | 100.00% | 0.00% |
| 仲裁准确率 | 81.12% | 100.00% |
全门槛（0.30/0.40/0.50/0.60/0.70/0.80）7 轴复核：NM2.1 **全门槛通过**；原版在每个门槛上未知拒答率均为 **0.00%**，且门槛越高误拒越差（0.00%→1.32%）。
## 3. 端到端整体记忆能力（同一份运行时，只有模型包不同）
来源：`nm2_battery_comparison_final.json`（A 110 用例 / B 16 / C 48 / D 重启持久化）
| 指标 | NM2（原版包） | NM2.1（最终包） |
|---|---:|---:|
| A 用例数 | 110 | 110 |
| A 总体正确率 | 89.09% | 88.18% |
| A 可回答正确率 | 100.00% | 100.00% |
| A 未知拒答率 | 60.00% | 56.67% |
| A 已知问题被误拒率 | 0.00% | 0.00% |
| B 零字面重叠改写正确率 | 43.75% | 68.75% |
| B 答成别的属性 | 18.75% | 18.75% |
| B 触发读取 | 56.25% | 93.75% |
| C 可回答正确率（24 同形候选） | 65.00% | 70.00% |
| C 答成别的属性 | 35.00% | 27.50% |
| C 未知泄漏率（越低越好） | 75.00% | 0.00% |
| D 重启后召回 | 通过 | 通过 |
| D 重启后作答正确 | 通过 | 通过 |
| D 清理生效 | 通过 | 通过 |
## 4. 未见改写问法的泛化（零字面重叠，24 同形候选，随机 4.17%）
来源：`replay_check_zov.json`（路由器级）与第 3 节 B/C 段（端到端）。路由器级用的是 v6 最终版权重作对照（原版部署权重在同一集合上 Top-1 只有 11.60%）。
| 指标 | NM2（v6 最终版权重） | NM2.1（本交付权重） |
|---|---:|---:|
| 路由器 Top-1（250 条可回答） | 18.40% | 59.60% |
| 路由器 Recall@3 | 36.00% | 83.20% |
| 路由器 MRR | 35.07% | 73.06% |
| 端到端改写正确率（B 段） | 43.75% | 68.75% |
| 端到端未知泄漏（C 段） | 75.00% | 0.00% |
## 5. 成本（参数 / 存储 / 速度）
来源：`router_scorecard_final.json`、`router_latency_bench_prod.json`（7 轮交错中位数）
| 指标 | NM2（原版） | NM2.1 |
|---|---:|---:|
| 参数量 | 2,037,774 | 2,037,774 |
| 每条记录地址字节 | 512 | 512 |
| 单查询延迟中位数 ms (GPU) | 0.9549 | 0.9169 |
| 批量 QPS (batch=64) | 66,037 | 69,378 |
| 批量 QPS (batch=256) | 207,287 | 242,759 |
交错基准（7 轮，消除顺序效应）单查询中位数：原版 1.4203 ms → NM2.1 1.4242 ms（差 +0.28%；该轮原版自身离散度 4.85%）。
模型包大小：原版 8.88 GB（22 文件）→ NM2.1 8.88 GB（23 文件，多出属性头 ~0.25 MB）；合并验证：替换 16 个张量、其余 47 个逐字节未变、路由张量与交付件逐位一致。
## 6. 工程鲁棒性
| 项目 | NM2（原版） | NM2.1 |
|---|---|---|
| 一次写 20 条不同属性事实后存活 | **12 / 20**（8 条查询前被误删） | **20 / 20** |
| 端到端（写入修复前后，16 用例） | 37.50% | **68.75%** |
| 替换兼容性 | 基线 | **DROP-IN OK**（16/16 键、驱动 `PagedMemoryBankV2`） |
| 单元测试 | — | **52 项通过** |
| 未知问题泄漏（同形候选） | **75.00%** | **0.00%** |
## 7. 仍未解决的短板（不粉饰）
| 短板 | 现状 | 说明 |
|---|---|---|
| 跨域未知拒答 | 未解决 | 覆盖头是 24 类闭集，仅当其词表被库填充 ≥90% 时生效；开放词表的属性匹配实测仅 **49.20%** Top-1 / AUC 0.6560 |
| 答成别的属性 | **27.50%**（原 35.00%） | 已排除先验重加权（更差）与单纯替换打分器（更差） |
| A 段未知拒答率 | 56.67% | 未改善 |
| 规模验证 | 未做 | 仅 24 属性 / 300 条改写评测；生产需上千属性、上万条 |
| 通用能力回归 | 未跑 | `eval_general_capability.py` 依赖的 `comprehensive_general.jsonl` 不存在 |
Natural_Memory_v2_Paper.md
+381
View File
@@ -0,0 +1,381 @@
# Natural Memory v2：面向长期交互的分层神经记忆与稀疏地址路由
## 摘要
大语言模型的长期记忆通常有两种实现：把历史对话原样放回上下文，或者把历史写入外部检索系统。前者的计算和显存成本随历史长度增长，后者虽然实用，却把记忆写入、读取和纠错放在模型之外，模型本身并不知道记忆为何存在、是否可信以及何时应该停止检索。
本文提出 Natural Memory v2，一种附着在冻结 Qwen3.5-4B 主干上的分层记忆架构。它不试图让 Memory Slot 复现完整 KV Cache，而是把长期、昂贵、适合结构化复用的历史信息编码成紧凑地址和版本化证据记录；当前窗口继续由热 KV 负责精确顺序和局部连贯性。V2 的查询路径固定为“粗索引—候选页—精确重排—Top-K 读取”，因此查询复杂度不随全部记忆槽做全量注意力。系统还加入了自动写入策略、置信度隔离、冲突版本、纠错、撤回、多跳关联和嵌入式持久化。
在本地控制变量实验中，20,000 条合成记录被组织为 626 个页面，平均只进入 178.13 个候选页（28.45%）；记录 Recall@K 与页 Recall@K 均为 100%，多跳和导出恢复测试均通过。通用路由器的候选 route accuracy 为 99.32%，Qwen hidden-state 路由器在 103 条 held-out 事实上的 route accuracy 为 91.26%。在 120 个固定的通用、数学、推理、语言、知识、逻辑和上下文用例上，Natural Memory v2 与原版 Qwen3.5-4B 的总分都为 0.85833，回归差值为 0。
更严格的脏真实语料测试包含 66 个工程文件、642 个真实代码/文档分片、136 个目标记录和 328 个问题，其中 310 个问题可回答、18 个问题明确要求拒答。Natural Memory v2 的可回答正确率为 75.81%，目标召回率为 96.45%，未知拒答为 18/18；同协议下的本地 BERT dense retrieval + 固定重排基线分别为 11.61%、63.23% 和 18/18。Natural Memory 的峰值 allocated 显存为 3.131 GiB，基线为 3.083 GiB；平均延迟为 1931 ms，基线为 1784 ms。该结果显示，模型内稀疏记忆路径可以在几乎相同的模型显存下显著提高窗口外证据的可用性，但还没有消除证据召回到最终回答之间的融合损失。
这些结果证明的是架构闭环、跨会话状态恢复和消费级硬件上的工程可行性，而不是已经解决百万级自然语言记忆或已经击败生产级 cross-encoder RAG。本文同时报告路由器泛化、证据融合、生成预算、训练数据规模和 NVMe 级分页注意力尚未完成等限制，并给出下一阶段的训练与系统路线。
**关键词：** 大语言模型、长期记忆、Memory Slot、KV Cache、稀疏路由、分页索引、持续学习、事实纠错
## 1. 问题定义
设当前对话的 token 序列为 (x_{1:t})，模型的热 KV 为 (K_t,V_t)。如果所有历史都保留在注意力上下文中，序列长度 (t) 变大时，注意力计算和 KV 显存都会持续增加。另一种做法是把历史写入外部数据库，再由固定程序检索结果并拼接进 prompt。后一种方法把检索变成了应用层逻辑：模型通常无法区分未经确认的推断、已经被纠正的旧值和当前有效值。
Natural Memory 的目标不是让一个固定大小的张量保存所有原始历史，而是让模型拥有一种长期运行的神经记忆接口：
1. 重要的个人事实、项目约束和事件关系可以自动进入记忆；
2. 查询只访问少量有地址的记忆单元；
3. 新旧事实具有可解释的版本关系；
4. 证据不足时模型可以拒绝读取，而不是把相似内容强行当答案；
5. 记忆状态可以在模型重启后恢复，而不需要重放聊天记录。
这里的“神经记忆”指记忆读取和写入策略是模型架构的一部分，并不意味着所有存储介质都必须是 GPU 上的可训练参数。长期记忆可以是模型拥有的状态快照，读取器、路由器和写入控制器则直接参与模型运行。运行时优先将热点记录提升到 VRAM，同时保留显存安全余量；无法安全提升的记录继续驻留在系统 RAM。
## 2. 设计原则
### 2.1 Memory Slot 不模拟 KV
KV Cache 适合保存近期 token 的精确顺序；Memory Slot 适合保存跨会话仍有价值的事实、摘要和关联。两者承担不同的任务：
```text
热 KV = 当前工作内存，强调顺序和细节
Memory Slot = 长期神经 RAM，强调地址、复用和生命周期
冷页面 = 更大容量的历史存储，按需加载
```
如果要求 Memory Slot 无损等价于数百万 token 的原始 KV，那么它最终仍然需要保存近似相同的信息量，只是换了数据结构。V2 选择有损但可审计的语义压缩：长期证据优先，当前上下文精确。
### 2.2 任何大规模读取都必须有界
不允许当前 token 对全部槽位直接做注意力。V2 采用两阶段路由：
1. 用 LSH 粗索引从页面签名中取候选页面；
2. 对候选页面做页级和记录级精确重排；
3. 只将 Top-K 记录的 token 证据送回 Qwen。
页面签名同时包含页面中心和记录地址。这样可以避免混合主题页面的中心向量把某一条稀有但相关的记录“平均掉”。当前实现还会探测查询签名的 Hamming 距离 1 和 2 的桶，并始终保留少量热页作为安全候选。
### 2.3 错误必须进入状态机
长期记忆最危险的不是暂时漏召回，而是一次错误写入之后长期污染回答。因此一条记录除了文本和向量外，还带有：
- 来源和证据；
- 置信度和重要性；
- 创建时间、访问次数和版本；
- 实体—属性—值冲突键；
- `active`、`superseded`、`retracted`、`quarantined` 状态；
- `supersedes` 和 `related_ids` 关系。
重复写入是幂等的；同一实体和属性的新值会 supersede 旧值；不可信写入进入 quarantine，不参与普通读取；显式纠错生成新版本；撤回记录仍保留审计信息但不再被读取。
## 3. 架构
### 3.1 路由器
给定查询 hidden state (q) 和候选记忆键 (k_i)，路由器先把两者投影到 (d=128) 的紧凑地址空间：
[
hat q = \operatorname{norm}(W_q q), \qquad
hat k_i = \operatorname{norm}(W_k k_i).
]
V2 使用 8 个路由头。每个头计算局部相似度，头门控对各头加权；随后将查询、候选和差向量送入 pair scorer。路由器同时输出：
- 候选记录分数；
- 是否需要记忆的二分类 logit；
- 多跳步数预测；
- 每个路由头的诊断分数。
存储侧只保留紧凑地址，而不保存每条记录的 2560 维 Qwen hidden state。查询时只把当前 query 和已经由粗索引筛出的候选键放到路由器所在设备。
### 3.2 分页记忆库
一个页面包含有限数量的记录、页面中心、摘要中心、重要性和冷热等级。默认页面容量为 32 条记录，最大页面数为 32768，懒分配容量为：
[
32768 \times 32 = 1,048,576
]
这只是地址空间上限，不代表启动时分配一百万条记录。写入侧只在开放页的有界窗口中选择目标页，避免随着页面数增长而扫描整个写入空间。达到硬上限后系统抛出明确的容量错误，要求先 consolidation 或提高容量，而不是静默超过限制。
### 3.3 多跳检索
一次查询先得到候选页面和记录。被选记录可以带关联记录 ID，下一跳只在关联记录所在的页面中继续搜索。每跳都记录访问轨迹、去重集合和停止原因。系统在达到 Top-K、最大 hop、无新关联或没有新证据时停止。
### 3.4 Qwen 集成
V2 作为 Qwen3.5-4B 的附加模块接入：
- Qwen 主干冻结；
- 原有 memory controller、自然语言写入策略和热文本 bank 保留兼容性；
- V2 router 注册为模型模块并可单独训练；
- 生成前由当前查询 hidden state 触发 V2 读取；
- 选中记录的 token 序列在模型内部形成证据前缀；
- 生成本身不更新持久记忆，写入发生在当前用户回合结束前；
- memory state、路由器参数、页面元数据和记录 token 可写入 safetensors memory shard。
该设计使模型重启时只需要重新加载模型包和内嵌状态，不需要把历史聊天重新放入 prompt，也不需要依赖一个固定的外部数据库读取程序。
## 4. 训练方法
### 4.1 路由训练目标
训练损失由三部分组成：
[
\mathcal L = \mathcal L_{candidate}
+ \lambda_n \mathcal L_{need}
+ \lambda_h \mathcal L_{hop}.
]
其中候选损失使用包含 hard negatives 的交叉熵；need loss 判断问题是否需要记忆；hop loss 预测继续关联检索所需的步数。负样本不仅包括随机记录，还包括相同实体、相近属性、共享词汇但答案不同的记录。无记忆问题作为独立类别参与训练，使路由器有机会学会 abstain。
### 4.2 两个训练阶段
第一阶段使用共享潜在因子训练通用路由器，验证分页、地址投影和损失函数是否稳定。第二阶段使用真实 Qwen3.5-4B 的 hidden state 编码实体—属性—值事实，并在不同实体、不同属性和 hard negative 上训练 Qwen 专用路由器。
第二阶段目前使用 512 条本地合成事实，409 条用于训练、103 条用于 held-out。这样做主要是工程启动数据，不足以代表真实用户对话的分布。正式版本需要加入：
- 同一事实的多种问法；
- 省略主语、代词和跨语言表达；
- 时间先后和旧值修正；
- 多事实组合与多跳路径；
- 没有记录时的拒答；
- 误导性相似事实和记忆污染样本。
## 5. 实验设置
### 5.1 硬件与加载
实验使用 RTX 5070，报告的总显存为 11.94 GiB；Qwen 使用 4-bit NF4 加载。主干权重保持冻结。结果来自本地工程测试，不是公开排行榜结果。
### 5.2 核心组件测试
单元测试共 39 项，覆盖：
- 路由器输入输出形状与紧凑地址；
- 页面粗索引候选边界；
- 版本冲突、纠错和 slot 替换；
- quarantine、批准和撤回；
- 多跳关联；
- safetensors 前的导出/恢复语义；
- KV 预算和页容量上限。
- 中文自然问法的稀疏别名路由；
- 同名符号的有界候选扩散；
- 无证据拒答、结构化证据格式和失败阶段归因；
- 防止回归测试文件污染真实代码库评测。
结果为 39/39 通过。
### 5.3 分页检索实验
在 20,000 条合成记录、626 页、Top-K 页面和记录限制下，结果如下：
| 指标 | 结果 |
|---|---:|
| 粗候选页平均数 | 178.13 |
| 粗候选页最大数 | 429 |
| 粗候选页占全部页面 | 28.45% |
| 记录 Recall@K | 100% |
| 页面 Recall@K | 100% |
| 多跳成功 | 100% |
| 多跳步数 | 2 |
| 导出恢复后召回 | 100% |
| 冲突版本与纠错 | 通过 |
| quarantine 隔离与批准 | 通过 |
| 撤回隔离 | 通过 |
| 重复写入幂等 | 通过 |
页面容量配置为 32768 页 × 32 条记录，地址空间为 1,048,576 条记录。除 20,000 条分页检索实验外，项目还完成了 1,000,000 条轻量记录的 durable page store 压力测试：31,250 页实际落盘，重启后记录总数保持 1,000,000，64 个常驻页对应约 160 条常驻记录，目标记录通过精确粗桶召回。该结果证明的是存储、冷热分层和地址路径，不声称已经完成一百万条完整自然语言长文本的 Qwen 端到端质量验证。
通用路由器在 40 个评估批次上的结果为：route accuracy 99.32%，need-memory precision、recall、specificity 均为 100%，hop accuracy 为 86.25%。这些数字来自合成分布，不能直接推断真实对话泛化。
### 5.4 Qwen hidden-state 路由器
Qwen 专用路由器的 held-out 结果：
| 指标 | 结果 |
|---|---:|
| held-out 事实 | 103 |
| route accuracy | 91.26% |
| need precision | 100% |
| need recall | 100% |
| need specificity | 100% |
| hop accuracy | 36.70% |
多跳控制器明显弱于候选记录路由。这意味着当前系统的可靠性主要来自显式关联关系、页面约束和记录状态机，不能把 hop prediction 当作唯一的正确性保证。
### 5.5 与原版 Qwen3.5-4B 的回归
综合测试使用同一组 120 个固定用例，包含通用能力、数学、推理、语言、知识、逻辑和 512—8192 token 的上下文定位。基线和 V2 均使用贪心解码与 4-bit NF4：
| 指标 | 原版 Qwen3.5-4B | Natural Memory v2 |
|---|---:|---:|
| 总分 | 0.85833 | 0.85833 |
| 总分差值 | - | 0 |
| 各分类差值 | - | 全部 0 |
| 自动写入 precision | - | 100% |
| 自动写入 recall | - | 100% |
| 自动写入 specificity | - | 100% |
| 无历史重启恢复 | - | 通过 |
| 清理后停止召回 | - | 通过 |
这说明在当前测试集合上，接入 V2 没有造成可测的通用能力退化；它不等于在所有任务、所有长度和所有语言上都没有退化。
### 5.6 真实重启实验
实验先给模型一条普通自然语言事实：
> 我正在开发一个长期项目，项目内部代号是 NM-V2-RESTART，使用中文。
没有使用 `/remember`。自动写入成功后，系统把状态保存进 memory safetensors shard，释放第一个模型，再加载第二个模型。第二个模型只收到新问题，没有聊天记录。结果如下：
- router 找到 `page_00000001` 和目标记录；
- 内部证据前缀长度为 36 token；
- 生成结果为 `NM-V2-RESTART`；
- 记录在报告中显示为 `evidence_found`；
- 测试结束后清理操作将页面数和记录数恢复为 0。
在未知事实边界测试中，写入“我的长期项目代号是 ZX-77”后询问“我的血型是什么”，阈值校准前会出现低分无关召回；阈值设为 0.65 后，路由结果为 `below_read_threshold`，内部前缀长度为 0。这个修复体现了一个重要原则：回答碰巧说“不知道”不能代替读取器本身拒绝无关证据。
### 5.7 脏真实语料与 Strong RAG 对照
为了检验构造事实上的高分能否迁移到真实工程内容，本文另外使用了当前 Natural Memory 工程本身的源码、README、配置和历史工程文档作为被测语料。语料保持“脏”状态：不同版本的脚本、重复符号、生成过的工程材料、实现说明和用户在本项目中明确提出的约束同时存在。测试问题不要求模型背诵随机字符串，而是要求它回答项目导航、架构取舍、状态恢复、内存层级、写入安全和用户长期偏好。
两套系统使用同一个 tokenizer、同一批 328 个问题、贪心解码和 `max_new_tokens=32`。Natural Memory 使用 LSH 粗索引、候选页、记录级重排和最多 2 条证据记录；Strong RAG 使用本地 `bert-base-chinese` CPU dense retrieval、64 条候选和固定透明重排。后者是一个更强于词法 Chunk RAG 的工程基线，但不是公开训练的 cross-encoder，不能把表中结果解释为对所有生产级 RAG 的结论。
| 指标 | Natural Memory v2 | Strong RAG | Natural Memory 相对变化 |
|---|---:|---:|---:|
| 测试问题 | 328 | 328 | 同协议 |
| 可回答问题 | 310 | 310 | 同协议 |
| 可回答正确率 | **235/310 = 75.81%** | 36/310 = 11.61% | **+64.19 个百分点** |
| 总体正确率 | **253/328 = 77.13%** | 54/328 = 16.46% | **+60.67 个百分点** |
| 目标证据召回 | **299/310 = 96.45%** | 196/310 = 63.23% | **+33.23 个百分点** |
| 未知拒答 | 18/18 = 100% | 18/18 = 100% | 持平 |
| 平均端到端延迟 | 1931.1 ms | 1784.1 ms | +8.2% |
| 平均 Decode | 16.51 tok/s | 17.82 tok/s | -7.4% |
| 峰值 allocated VRAM | 3.131 GiB | 3.083 GiB | +1.5% |
| 峰值 reserved VRAM | 4.430 GiB | 4.211 GiB | +5.2% |
这张表体现了本文最重要的系统张力。Natural Memory 并没有用更多 GPU 内存换取准确率：allocated 显存只增加约 0.048 GiB，却多召回 103 条目标证据，并多答对 199 个可回答问题。代价也同样明确：平均多付出约 147 ms，Decode 速度低约 1.31 tok/s。换句话说，当前实现的优势主要来自“把长期信息从热窗口外重新送到模型”，而不是来自无条件扩大上下文或无条件增加显存。
从 Natural Memory 自身的迭代误差看，路由改进已经改变了失败结构：
| Natural Memory 版本 | 目标召回 | 可回答正确率 | 路由失败 | 证据融合失败 | 生成控制失败 | 未知拒答 |
|---|---:|---:|---:|---:|---:|---:|
| 早期 final6 | 286/310 = 92.26% | 233/310 = 75.16% | 24 | 49 | 4 | 17/18 |
| 当前 final9 | **299/310 = 96.45%** | **235/310 = 75.81%** | **11** | 62 | 2 | **18/18** |
| 变化 | +13 条 | +2 条 | **-13** | +13 | -2 | +1 条 |
失败从路由侧转移到证据融合侧并不是坏消息：它说明记录已经更常被找到，但模型在 32 token 输出预算内仍会选择解释、重复元数据或只使用部分证据。当前 75 个可回答失败中，62 个被归类为证据融合、11 个为路由、2 个为生成控制。因此下一阶段最有价值的训练目标不是继续堆页面数量，而是训练“证据已在前缀中时，先给出所有必要答案锚点；证据不完整时，明确区分已知与未知”的短答案策略。
### 5.8 架构收益与代价的边界
| 架构能力 | 直接证据 | 对系统的实际意义 | 当前边界 |
|---|---|---|---|
| 热 KV 与长期记忆分工 | 642 个真实分片只通过最多 2 条记录进入生成前缀 | 当前对话保持局部连贯，历史不必全部塞入热 KV | 不能无损恢复所有原始顺序细节 |
| 稀疏地址路由 | 目标召回 96.45%，不是对全部记录做注意力 | 让长期库规模增长不必线性增加生成上下文 | 11/310 个可回答问题仍路由失败 |
| 证据状态机 | `active`、`superseded`、`retracted`、`quarantined`；未知拒答 18/18 | 错误写入可以隔离、版本化、撤回 | 多跳控制器仍弱于单跳候选路由 |
| 无历史重启读取 | 实测只给新问题，不重放聊天记录 | 记忆状态可随模型包恢复 | 仍需验证更长时间和更多用户隔离 |
| CPU 主体与 bounded VRAM cache | allocated 3.131 GiB，reserved 4.430 GiB | 在 12 GiB 消费卡上保留安全余量 | 本文不是多 GPU 或高并发服务测试 |
| 与基座能力兼容 | 120 项综合回归与原版均为 0.85833 | 记忆模块没有在该集合上牺牲通用能力 | 不代表所有任务、语言和长度均无回归 |
| 面向证据复用的压缩 | 真实语料 642 分片、136 个目标记录、有限前缀读取 | 只把适合长期复用的部分送入模型 | 不是对百万 token 的无损 KV 替代 |
## 6. 长上下文与容量边界
直接 KV 压力测试在当前机器上约 8192 token 可以运行，16384 和 32768 token 会 OOM。这个结果与 V2 的作用并不矛盾：V2 的目标是把长期信息移出热 KV，而不是自动让原始长文本注意力变成低成本。
当前还不能声称支持 200M—300M 原始 token 上下文。百万级轻量记录的 durable page store 已经实现；同时，Qwen 生成路径已经接入 CPU-backed `DynamicCache(offloading=True)`，并加入了模型内的旧前缀分块归档和热窗口裁剪。真实自动路径把 371 token 压缩为 32 token，写入 22 条上下文记录后完成生成。要达到原始长上下文目标，仍然需要：
1. GPU 热页、RAM 温页、磁盘冷页之间更细粒度的统一 page manager，以及面向 NVMe 的分页调度；
2. 原文页与摘要页的可逆压缩；
3. 更高吞吐的 CPU/NVMe KV offload 与分页注意力；
4. 128K 热窗口到百万级历史的课程训练；
5. 大规模多跳、冲突、污染和纠错数据；
6. 召回失败时的保守拒答和校准评测。
V2 已经把地址空间、页路由、版本状态机、持久化接口和 CPU KV offload 接在一起；当前证据仍然是轻量记录压力测试和短自动压缩验证，不是百万 token 原始上下文的端到端质量证明。
## 7. 讨论
### 7.1 与外部 RAG 的差异
V2 与普通 RAG 的主要区别不在于“是否存在向量”，而在于记忆生命周期由模型运行时直接控制。路由器学习问题是否需要记忆，写入策略决定什么值得保留，记录状态机维护冲突和撤回，证据前缀由模型内部读取路径生成。本次交付的默认实现不启用磁盘分页：完整 V2 记录随第三个 safetensors memory shard 载入进程内存，只有有限的热点地址和 token payload 进入 VRAM cache。未来更大规模部署可以增加冷页后端，但那是容量扩展，不是当前正确性路径。
### 7.2 与扩大 KV 的差异
扩大 KV 对最近历史的细节保持最好，但长期存储成本高，而且每个查询都容易被无关上下文拖慢。Memory Slot 主动丢弃顺序细节，只保存事实和结构化证据，因此更适合个人偏好、项目决策、联系人属性和长期计划。两者应该组合使用，而不是互相替代。
### 7.3 为什么训练比扩大张量更难
增加 slot 数量只改变了容量，不会教会模型如何寻址。真正困难的是建立稳定的写入地址、区分相似事件、关联多个记录、修正旧版本，以及在没有答案时停下来。当前 Qwen 路由器的 hop accuracy 已经显示，候选选择和多跳控制是两个不同的学习问题，不能用单一的相似度损失解决。
## 8. 可复现性
项目目录为 `H:\Memory\V2_dpskw`。核心命令：
```powershell
Set-Location H:\Memory
# 单元测试
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe `
-m unittest discover -s V2_dpskw\tests -v
# V2 存储与路由评测
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe `
-m V2_dpskw.benchmark_memory_v2
# Qwen3.5-4B 综合回归
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe `
-m V2_dpskw.benchmark_natural_memory_v1 `
--base-model H:\Memory `
--memory-model H:\Memory\V2_dpskw\qwen3_5_4b_natural_memory_v2 `
--output H:\Memory\V2_dpskw\natural_memory_v2_full_benchmark.json
# 真实重启测试
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe `
-m V2_dpskw.test_natural_memory_v2_restart
# KV offload 与自动长上下文压缩
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe `
-m V2_dpskw.benchmark_kv_offload
# 脏真实语料 Natural Memory
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe `
-m V2_dpskw.benchmark_dirty_real_corpus_4b `
--base-model H:\Memory `
--memory-model H:\Memory\V2_dpskw\qwen3_5_4b_natural_memory_v2 `
--output H:\Memory\V2_dpskw\dirty_real_corpus_compare_4b.json `
--skip-no-memory --skip-chunk-rag --gpu-memory-gb 10
# 同协议 Strong RAG 对照
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe `
-m V2_dpskw.benchmark_dirty_strong_rag_4b `
--base-model H:\Memory `
--output H:\Memory\V2_dpskw\dirty_strong_rag_compare_4b.json `
--project-targets 128 --top-k 2 --candidate-k 64 --max-new-tokens 32 `
--gpu-memory-gb 10
```
关键结果文件：
- `natural_memory_v2_benchmark.json`：分页、路由、完整性和恢复；
- `natural_memory_v2_full_benchmark.json`：Qwen 基线回归；
- `natural_memory_v2_restart_test.json`：无历史重启；
- `qwen_router_v2_training.json`：Qwen 专用路由器训练与 held-out 指标；
- `dirty_real_corpus_compare_4b_router_final9.json`：66 个真实工程文件上的 Natural Memory 结果；
- `dirty_strong_rag_compare_4b_router_final3.json`：同协议 Strong RAG 结果。
## 9. 局限与未来工作
本文的实现仍然是研究原型，距离可公开部署还有几项关键工作：
1. 用更大规模、脱敏且自然的真实对话替代小规模合成事实，并按用户、项目和时间做严格数据隔离；
2. 训练多跳控制器和不确定性校准，而不是只优化 top-1 路由；
3. 优先解决“目标已召回但答案不完整”的证据融合问题，训练短答案中的答案锚点优先策略；
4. 引入事实来源签名、用户确认策略和可撤销日志；
5. 将页对象替换为压缩列式存储，降低百万记录的 Python 对象开销；
6. 完成 GPU/RAM/NVMe 三级冷热迁移和 NVMe page cache；
7. 实现摘要页与原文页之间的按需恢复；
8. 建立跨会话、跨语言、长时间连续运行的污染测试；
9. 在 128K 热 KV 和百万级历史上进行端到端吞吐、延迟、质量和故障恢复评测；
10. 使用公开训练 cross-encoder、滑动窗口和分页 KV 作为后续公平基线，而不是把当前本地 Strong RAG 数字外推为生产结论。
Natural Memory v2 最重要的成果不是一个更大的 slot 张量，而是把“记忆是什么、如何寻址、何时读取、怎样纠错、何时拒绝”放进了一个可测试的模型架构中。后续版本的主要任务，是让这个架构在更大规模和更真实的分布上保持同样的克制与可解释性。
Natural_Memory_v2_对外技术总结.md
+278
View File
@@ -0,0 +1,278 @@
# Natural Memory v2
## 对外技术说明
版本：v2
本文只描述当前实现已经完成并实际测量的内容。它不把系统描述成通用智能，也不把实验室结果表述成线上服务承诺。
## 1. 一句话说明
Natural Memory v2 是接在本地 Qwen3.5-4B 上的一个模型内记忆层：它把适合长期复用的个人事实、项目事实和短对话片段写入有地址的记忆记录，当前问题只读取少量相关记录，再把这些记录作为模型内部前缀交给 Qwen 生成答案。
它要解决的具体问题是：模型重启后不携带历史聊天记录，仍能访问已经保存的个人或项目事实，同时不把全部历史转换成 GPU 上的长 KV Cache。
## 2. 当前实现
### 2.1 记忆与 KV 的分工
```text
最近对话 Qwen 热 KV
长期个人事实/项目事实 Natural Memory 记录
当前问题 有界路由与 Top-K 读取
原始持久化状态 嵌入式 memory safetensors 切片
```
Memory Slot 不试图逐 token 模拟完整 KV。它保存的是更适合跨会话复用的内容：事实、版本、来源、短文本证据和语义地址。当前对话的顺序关系仍由 Qwen 的普通上下文负责。
### 2.2 读取路径
```text
用户问题
-> Qwen hidden state 生成紧凑查询地址
-> LSH/地址粗索引
-> 候选页
-> 页内记录重排
-> Top-K 记录
-> 证据前缀注入 Qwen
-> 普通生成
```
当前 token 不会与全部记忆记录做全量注意力。当前默认读取上限为少量页面和记录；实际运行中只把命中的短证据提升到 GPU，记忆主体保留在进程内存。
### 2.3 写入与版本
每条记录包含：
- 原始短文本和可选 token 序列；
- 语义地址、实体、属性和值；
- 时间、来源、置信度、重要性；
- active、superseded、retracted、quarantined 状态；
- 版本关系和审计信息。
同一实体和属性出现新值时，旧值会被标记为 `superseded`。低置信度内容可以进入 quarantine，不参与正常读取。撤回记录不会从审计中消失，而是变成 `retracted`。
### 2.4 重启与存储
当前交付配置采用 embedded weight-shard：记忆快照、路由器参数、页面元数据和短文本证据写入模型包的 memory safetensors 切片。启动时加载到进程内存，命中的热点记录才进入有界 GPU cache。
当前默认路径不使用 SQLite，也不使用磁盘分页。该设计减少了运行时依赖，但意味着模型包会随持久化记忆增长而变大，且需要重新保存权重切片才能固化更新。
## 3. 正式脏数据迁移测试
### 3.1 数据来源
本次正式测试使用了两类来源：
1. 当前 Natural Memory 工程自身的真实源码、文档和少量随包配置：65 个文件、1,110,524 字节、620 个源码/文档分片。
2. 用户在本项目对话中明确说过的短事实和工程要求：15 条记录，包含工作目录变更、模型命名、对照模型、存储限制、显存限制、训练取向和模型内读取要求。
本轮由这些记录派生出的用户对话题共 44 题，其中 38 个可回答、6 个未知；其余 220 题来自真实源码和文档。
为了避免答案泄漏，正式数据排除了：
- 以前的 benchmark JSON 结果；
- synthetic hardset 和其他合成数据集；
- checkpoint 和模型权重；
- tokenizer 大文件。
用户对话部分不是从外部用户数据库导出的日志。它是本次项目对话里用户已经明确写出的要求；其测试问题由人工改写成口语、跨会话、时间冲突和未知问题。不能把这部分描述成大规模真人日志验证。
### 3.2 测试协议
- 模型：本地 Qwen3.5-4B 原版与 Natural Memory v2；
- 量化：4-bit NF4；
- 解码：greedy；
- 最大新生成：64 token；
- GPU 进程上限：10 GiB；
- Natural Memory 读取：最多 2 条记录；
- 语义地址编码：单条 batch，避免 12 GiB 显卡在源码证据编码阶段产生瞬时峰值；
- 题目总数：264；
- 可回答题：246；
- 未知/拒答题：18。
题目类型不是单纯的随机字符串记忆，包括：
- 真实函数和类的源码定位；
- 面向工程使用的架构解释；
- 用户事实的自然改写；
- 早期目录到当前目录的时间冲突；
- 写入安全、版本替换和显存约束；
- 目标不存在时的拒答。
### 3.3 总体结果
| 系统 | 可回答正确率 | 未知拒答正确率 | 总体正确率 |
|---|---:|---:|---:|
| 原版 Qwen3.5-4B，无记忆 | 3/246 = 1.22% | 17/18 = 94.44% | 20/264 = 7.58% |
| Natural Memory v2 | 154/246 = 62.60% | 16/18 = 88.89% | 170/264 = 64.39% |
Natural Memory 把真实工程与用户事实带来的可回答率从 1.22% 提升到 62.60%。这证明记忆路径能够迁移到非构造的短事实和真实代码内容，但距离稳定的生产级自然语言记忆仍有明显差距。
### 3.4 按数据域拆分
| 数据域 | 系统 | 可回答结果 | 未知拒答 |
|---|---|---:|---:|
| 用户对话事实 | 原版 Qwen | 1/38 = 2.63% | 6/6 = 100.00% |
| 用户对话事实 | Natural Memory v2 | 21/38 = 55.26% | 6/6 = 100.00% |
| 真实源码/文档 | 原版 Qwen | 2/208 = 0.96% | 11/12 = 91.67% |
| 真实源码/文档 | Natural Memory v2 | 133/208 = 63.94% | 10/12 = 83.33% |
源码定位是当前表现最好的真实任务：
- 用户对话事实：21/38 = 55.26%；
- 符号定位：129/192 = 67.19%；
- 时间冲突目录查询：2/2；
- 写入安全解释：2/2；
- 基线局限说明：2/2；
- 用户自然改写：4/5；
- 用户约束类问题：1/4；
- 开放式架构解释类问题：当前严格锚点评估中仍不稳定。
### 3.5 路由与生成的差距
Natural Memory 在 246 个可回答问题上的目标记忆召回为 166/246 = 67.48%，最终回答正确为 154/246 = 62.60%。这说明当前主要问题已经不只是“有没有记忆”，而是两段链路都需要改进：
1. 路由器在口语化问题、短问题和没有显式文件名的问题上会召回相邻但错误的记录；
2. 即使证据已进入前缀，Qwen 仍可能拒答、过度解释或没有按要求输出关键事实。
因此，当前结果不能写成“检索准确率等于回答准确率”，两者必须分开报告。
## 4. 显存、速度和记忆容量
以下数字来自同一次正式脏数据测试。显存字段是每次请求结束后的 allocated/reserved 快照，不是 `torch.cuda.max_memory_allocated()` 记录的严格峰值；`reserved` 还包含 PyTorch 分配器保留的缓存。
| 指标 | 原版 Qwen3.5-4B | Natural Memory v2 | 变化 |
|---|---:|---:|---:|
| 平均总延迟 | 2650.4 ms | 2718.9 ms | +2.58% |
| 平均解码速度 | 24.06 tok/s | 21.68 tok/s | -9.90% |
| 平均输入 token | 30.8 | 641.4 | 记忆证据前缀增加 |
| 读取额外耗时 | 无 | 61.1 ms | 新增路径 |
| allocated 快照峰值 | 3.084 GiB | 3.134 GiB | +0.050 GiB |
| reserved 快照峰值 | 3.234 GiB | 5.021 GiB | +1.787 GiB |
本次记忆状态统计：
- active records：723；
- pages：23；
- 页面容量：32 records/page；
- 设计容量：1,048,576 records；
- GPU cache 上限：256 records、131,072 token；
- 实际 GPU cache token：15,885；
- GPU cache fallback：0；
- GPU cache allocation failure：0；
- cold page：0，原因是本次使用 embedded/process-RAM 模式。
这里的“一百万记录容量”是地址空间和分页结构的容量，不是本次已经装入了一百万条语义记忆，也不是一百万条记录已经通过端到端测试。
## 5. 之前的同协议工程基准
下面是先前在当前工程项目库上的 Stage 5 对照，用于说明系统在结构化项目事实上的上限。它与本次脏数据迁移集不是同一题集，不能合并成一个总分。
### 通用个人事实
| 系统 | 可回答 | 未知拒答 | 平均延迟 | 解码速度 |
|---|---:|---:|---:|---:|
| 原版 Qwen3.5-4B | 4/107 = 3.74% | 21/21 = 100% | 1233.2 ms | 23.51 tok/s |
| 强 RAG 基线 | 106/107 = 99.07% | 20/21 = 95.24% | 885.8 ms | 21.67 tok/s |
| Natural Memory v2 | 107/107 = 100% | 21/21 = 100% | 994.3 ms | 21.70 tok/s |
### 项目库
| 系统 | 正确率 | 平均延迟 | 解码速度 |
|---|---:|---:|---:|
| 原版 Qwen3.5-4B | 45/64 = 70.31% | 1351.5 ms | 23.70 tok/s |
| 强 RAG 基线 | 64/64 = 100% | 1395.3 ms | 22.94 tok/s |
| Natural Memory v2 | 64/64 = 100% | 1463.2 ms | 21.85 tok/s |
该强 RAG 使用本地 CPU `bert-base-chinese` 向量、稀疏特征和固定规则重排。它不是训练好的公开 cross-encoder，因此可以作为更强的本地基线，但不能代表所有生产 RAG 服务。
## 6. 已经解决的问题
当前实现已经能够在本地 12 GiB GPU 上完成以下闭环：
1. 普通自然语言触发自动记忆读取，不要求用户输入 `/remember`；
2. 模型重启后不回放历史聊天，只加载已保存的 memory shard；
3. 旧值和新值保留版本关系，旧值不会继续作为 active 事实参与正常读取；
4. 记忆读取器、地址索引和证据注入位于模型适配架构内部；
5. 记忆主体留在进程内存，热点记录使用有界 GPU cache；
6. 当前问题只读取 Top-K 记录，不对全量 slot 做注意力；
7. 低置信度写入、撤回和审计状态有独立表示；
8. 在真实工程源码定位任务上，相比无记忆 Qwen 有明显的可回答性提升。
## 7. 当前不能宣称的内容
以下说法目前没有足够证据支持：
- “Natural Memory 已经等价于百万 token 的完整 KV”；
- “Natural Memory 已经击败生产级 embedding + reranker RAG”；
- “任何自然语言表达都能稳定召回正确记忆”；
- “记忆写入永远不会出错”；
- “一百万条记录已经完成端到端验证”；
- “当前 4B 结果可以直接外推到 14B、32B 或更大模型”；
- “reserved VRAM 就是实际模型峰值显存”；
- “本次用户事实测试代表大规模真实用户群体”。
## 8. 主要局限
### 8.1 训练数据仍不够自然
当前路由器的主体训练仍来自本地构造的实体—属性—值事实和 hard negatives。正式脏数据集虽然使用了真实工程内容和真实项目对话片段，但自然改写仍是人工编写的测试变体，不是大规模脱敏真人会话。
### 8.2 真实仓库的语义索引覆盖有限
本次运行把全部 620 个真实源码/文档分片放入记忆库，但为控制 GPU 压力，只有 96 个符号目标和 8 个操作性事实建立了 Qwen 语义地址，其余分片作为真实背景记录保留。这使测试更安全、更可复现，但不能等同于“整个仓库都被高质量语义索引”。
### 8.3 开放式解释弱于明确定位
对于“某个函数在哪个文件”这类有明确地址的任务，表现明显好于“结合多个文件解释系统为什么这样设计”。后者需要多跳证据合并、冲突处理和生成控制，当前仍会出现证据已召回但最终回答不完整的情况。
### 8.4 速度仍有真实代价
正式脏数据测试中，Natural Memory 平均延迟高约 2.58%，解码速度低约 9.90%，并增加了平均输入前缀长度。这个代价来自 Qwen 查询编码、稀疏路由、证据前缀和热点缓存，而不是免费获得的能力。
### 8.5 生产级基线还不完整
本次正式脏库对照聚焦原版 Qwen3.5-4B 无记忆基线。要作出服务经济学结论，还需要在同一脏数据、同一题集上加入强 embedding 检索器、cross-encoder 或训练好的 reranker、滑动窗口和分页 KV 等基线。
## 9. 下一阶段的工程任务
按对正确率最有帮助的顺序：
1. 使用有用户同意的脱敏对话，增加自然省略、指代、错别字、时间间隔、互相矛盾的旧说法和多轮项目协作样本；
2. 重新训练路由器，重点覆盖“同一事实的多种问法”和“相似但错误的记录”；
3. 对“检索正确但回答错误”的样本单独训练证据使用与拒答策略；
4. 将真实仓库的语义地址覆盖从目标记录扩展到完整索引，仍保持单条/小批次编码以保护显存；
5. 加入多跳证据合并和当前版本优先规则；
6. 将写入、冲突解决和摘要从生成热路径移到回合边界或后台线程；
7. 把显存测试改为记录真正的 CUDA high-water mark，并分别报告 live allocated、reserved 和 cache payload；
8. 在同一协议下补齐强 RAG、滑动窗口和分页 KV 的成本—质量曲线。
## 10. 复现实验
在 `H:\Memory` 下运行：
```powershell
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe `
-m V2_dpskw.benchmark_dirty_real_corpus_4b `
--project-records 8192 `
--project-targets 96 `
--max-new-tokens 64 `
--encode-batch-size 1 `
--skip-chunk-rag `
--output H:\Memory\V2_dpskw\dirty_real_corpus_compare_4b.json
```
对应的原始结果文件为：
`H:\Memory\V2_dpskw\dirty_real_corpus_compare_4b.json`
评测脚本为：
`H:\Memory\V2_dpskw\benchmark_dirty_real_corpus_4b.py`
## 11. 结论
Natural Memory v2 当前已经不是只有构造数据上的演示：在真实工程源码、文档和本项目用户要求组成的脏数据迁移测试中，它显著提高了窗口外事实的可回答性，并在 10 GiB GPU 进程上完成了 723 条 active records 的读取运行。
同时，62.60% 的可回答正确率、88.89% 的未知拒答正确率和约 10% 的解码速度下降说明它仍处于研究原型到工程验证之间。当前最值得继续投入的不是增加宣传口径，而是提高自然语言路由泛化、证据到回答的闭环稳定性，并在同一题集上完成强 RAG 和 KV 基线的公平对照。
PRODUCTION_ROUTER.md
+155
View File
@@ -0,0 +1,155 @@
# 生产路由器交付说明（NM2 Router · V2-128 v7）
**交付件**：`checkpoints/router_replay_v7_v2_128/memory_router_v2.pt`
**SHA-256**：`69f8295e78821e52c0b41b248e7eddbfa47d7e409539f4a13382c0c82d189deb`
**大小**：8,156,921 字节 · **格式**：raw `state_dict`（16 张量 / 2,037,774 参数 / float32 / 数值全部有限）
几何与现网**完全一致**：`router_dim=128`、`num_heads=8`、`hidden_size=2560`、`max_hops=3`、
每条记录地址 **512 字节**、参数量 **2,037,774**（与原版完全相同）。
---
## 1. 交付件相对原版 NM2 的成绩（冻结评测集 21,920 条 / 10 类别）
判定口径：`verdict_router_v6.py` 逐轴对原版部署权重比较，速度类噪声轴用 3% 相对容差。
| 指标 | 原版 NM2 | **本交付件** | 变化 |
|---|---:|---:|---|
| Top-1 正确率 | 41.12% | **94.14%** | +53.01pp |
| Recall@1 / @3 / @5 | 37.03% / 46.73% / 48.91% | **88.58% / 96.48% / 97.15%** | +51.55 / +49.75 / +48.23pp |
| MRR / nDCG@3 | 47.69% / 44.22% | **95.77% / 95.37%** | +48.08 / +51.15pp |
| 多跳证据全中(Top-3) | 43.43% | **96.22%** | +52.80pp |
| 多跳证据全中(仅多正例) | 37.15% | **95.45%** | +58.30pp |
| hop 正确率 | 73.23% | **100.00%** | +26.77pp |
| hop 欠预测率 ↓ | 4.68% | **0.00%** | −4.68pp |
| need F1 / 召回 | 89.58% / 99.82% | **100.00% / 100.00%** | +10.42 / +0.18pp |
| **未知拒答率** | 0.00% | **100.00%** | +100.00pp |
| **已知问题被误拒率 ↓** | 0.18% | **0.00%** | −0.18pp |
| 未知问题被误读率 ↓ | 100.00% | **0.00%** | −100.00pp |
| 仲裁准确率 | 81.12% | **100.00%** | +18.88pp |
| 每条记录地址字节 | 512 | **512** | 持平 |
| 参数量 | 2,037,774 | **2,037,774** | 持平 |
**逐轴判定：22 通过 / 0 未通过 —— 全方位超越原版 NM2**
（`router_verdict_final.json` / `router_verdict_final.md`）
**门槛曲线复核**：不止默认阈值 0.50 —— 在 0.30 / 0.40 / 0.50 / 0.60 / 0.70 / 0.80 **每个门槛**上，
未知拒答率均 **100.00%**、已知问题被误拒率均 **0.00%**、未知问题被误读率均 **0.00%**、
need F1/召回/精确率/仲裁准确率均 **100.00%**（`threshold_sweep_check.md`，5/5 候选全门槛通过）。
---
## 2. 生产可用性验证
| 项目 | 结果 | 证据 |
|---|---|---|
| 可替换性（drop-in） | **DROP-IN REPLACEMENT OK**：16/16 键匹配、无缺失/多余、无形状不符、分数有限，且实测驱动 `PagedMemoryBankV2` 路由（`bank_routed_records: 3`） | `router_swap_check_replay.json` |
| 延迟（7 轮**交错**取中位数，消除顺序效应） | 单查询 **1.4242 ms** vs 原版 **1.4203 ms**（差 0.3%，同架构同参数量）；batch=256 QPS 146,676 vs 164,556 | `router_latency_bench_prod.json` |
| 未见过改写问法的泛化 | Top-1 **59.60%**、Recall@3 **83.20%**（24 个同形候选，随机基线 4.17%） | `replay_check_zov.json` |
| 端到端（16 条零重叠用例，官方 harness） | 回答正确率 **68.75%**、答成别的属性 18.75% | `router_critical_e2e_after_write_fix.json` |
| 单元测试 | 52 项全部通过 | `python -m unittest discover -s tests -t .` |
> ⚠️ **关于速度轴的一个更正**：早先某次记分卡把 `router_best.pt` / `router_step_*.pt`（wrapper 格式）
> 报成单查询慢 30%，据此判它们"速度不达标"。交错基准复测证明五个 checkpoint 的中位数全在
> 1.4123–1.4343 ms（彼此 ≤1%）—— **那是记分卡顺序测量的伪影，不是路由器属性**，该判定作废。
---
## 3. 准确度上限与取舍（选型依据，如实记录）
同一权威基准上，128 维 drop-in 几何内的候选对比：
| 候选 | v6 基准 Top-1 | 未见改写 Top-1 | 说明 |
|---|---:|---:|---|
| **本交付件 REPLAY-128 v7** | **94.14%** | **59.60%** | 22/22 轴、drop-in、延迟持平 |
| `router_v6_v2_128/router_best.pt` | **94.62%**（基准最高） | **7.60%** | v6 基准最强，但**新问法上崩掉**（低于随机 4.17% 附近） |
| `router_v6_v2_128/memory_router_v2.pt` | 94.37% | 18.40% | v6 最终版 |
| `router_prod_v2_128/memory_router_v2.pt`（本轮 20k 步长训） | 93.78% | — | 从 v6 best 出发在合并语料上继续训练反而退化 |
| 原版 NM2 | 41.12% | 11.60% | 基线 |
**结论**：在 drop-in 几何内，v6 基准的检索准确度上限约 **94.6%**，但那个点在新问法上只有 7.60%。
本交付件用 **0.48pp** 的 v6 基准差距换来 **+52.00pp** 的未见问法能力（7.60% → 59.60%）。
对真实用户提问而言这是明显正确的取舍，因此以它为生产件。
另注：本轮 20k 步长训的曲线峰值出现在第 7000 步（合并评测 Top-1 93.98%），但**该权重未落盘**
（`--checkpoint-interval` 默认 10000，且训练器的 `router_best.pt` 保存未生效）。
同时提醒：**从 15 个 eval 点里挑最高值属于在评测集上做选择**，93.98% 带选择性偏差，
不应作为交付指标 —— 故未采用。
---
## 4. 安装（3 步）
```powershell
# 1) 备份原版权重
Copy-Item 'H:\Memory\dynamic_memory_lab\checkpoints\natural_memory_v2_qwen_router_entities\memory_router_v2.pt' `
'H:\Memory\dynamic_memory_lab\checkpoints\natural_memory_v2_qwen_router_entities\memory_router_v2.pt.bak'
# 2) 替换（几何完全一致，无需改 memory_router_dim，无需重建地址）
Copy-Item 'H:\Memory\V2_dpskw\checkpoints\router_replay_v7_v2_128\memory_router_v2.pt' `
'H:\Memory\dynamic_memory_lab\checkpoints\natural_memory_v2_qwen_router_entities\memory_router_v2.pt'
# 3) 校验替换结果
cd H:\Memory\V2_dpskw
& 'C:\Users\Administrator\miniconda3\envs\LLM\python.exe' -m V2_dpskw.check_router_swap `
--candidate 'H:\Memory\dynamic_memory_lab\checkpoints\natural_memory_v2_qwen_router_entities\memory_router_v2.pt'
# 期望输出: VERDICT: DROP-IN REPLACEMENT OK
```
若模型是**内嵌合并包**（`memory_merge.json`），权重键为
`dynamic_memory.memory_router_v2.*`，需把本 `state_dict` 的 16 个张量按该前缀写回
`safetensors`；`merge_memory_weights.py` 是这件事的现成入口。
---
## 5. 本交付件**不**解决的问题（避免误判）
以下都是**评测过的独立问题，换路由器不会改善**，详见 `WRITE_PATH_FIX.md` 与 `BOTTLENECK_REPORT.md`：
1. **记录级排序不由路由器决定**：`memory_os_v2._record_scores` 对带 `semantic_key` 的记录用打包的
`text_retriever` 覆盖路由器分数。实测打包 retriever 的 Top-1 只有 **23.20%**，而本交付件同任务
达 **59.60%** —— 但把权重混合进去后，端到端**独立进程复测无任何变化**
（被 `1.25·rare_lexical_address` 等先验项压过，注入记录集合不变）。
2. **"句式正常的未知"识别不了**：问库中不存在的属性时，**78.00%（39/50）**仍给出编造答案；
两组检索最高分分布几乎完全重叠（p50 288.20 vs 285.25），**阈值方案被测量排除**。
3. **写入路径曾摧毁记录**：一次写 20 条不同属性事实后只剩 12 条 active（现已修复为 20 条，
详见 `WRITE_PATH_FIX.md`，该项修复把端到端从 37.50% 提到 68.75%）。
即：本交付件把**路由器这一环**做到了 22/22 全方位超越且生产可用；端到端体验的主要剩余瓶颈
在写入路径之后的检索/排序与拒答判断上，不在路由器权重里。
---
## 6. 复现命令
```powershell
$env:PYTHONPATH='H:\Memory'; $env:PYTHONIOENCODING='utf-8'
$py='C:\Users\Administrator\miniconda3\envs\LLM\python.exe'
cd H:\Memory\V2_dpskw
# 22 轴评分卡 + 逐轴判定（权威基准）
& $py -m V2_dpskw.eval_router_v5 --train-file data/router_training_v6/train.jsonl `
--eval-file data/router_training_v6/eval.jsonl `
--feature-cache H:\Memory\nm_cache\nm_router_v6\feature_cache `
--run "V2-128 deployed(v3)=H:\Memory\dynamic_memory_lab\checkpoints\natural_memory_v2_qwen_router_entities\memory_router_v2.pt" `
--run "REPLAY-128 v7 final=checkpoints\router_replay_v7_v2_128\memory_router_v2.pt" `
--output router_scorecard_final.json --markdown router_scorecard_final.md
& $py -m V2_dpskw.verdict_router_v6 --scorecard router_scorecard_final.json `
--candidate "REPLAY-128 v7 final" --baseline-prefix "V2-128 deployed" `
--output router_verdict_final.json --markdown router_verdict_final.md
# 零重叠泛化
& $py -m V2_dpskw.eval_router_v5 --train-file data/zero_overlap/train.jsonl `
--eval-file data/zero_overlap/eval.jsonl `
--feature-cache H:\Memory\nm_cache\nm_zero_overlap\feature_cache `
--model-path qwen3_5_4b_natural_memory_v2 --candidate-count 24 `
--run "REPLAY-128 final=checkpoints\router_replay_v7_v2_128\memory_router_v2.pt" `
--output replay_check_zov.json --markdown replay_check_zov.md
# 延迟（交错中位数）
& $py -m V2_dpskw.bench_router_latency --rounds 7 --single-samples 300 `
--run "deployed=v2:H:\Memory\dynamic_memory_lab\checkpoints\natural_memory_v2_qwen_router_entities\memory_router_v2.pt" `
--run "REPLAY=v2:checkpoints\router_replay_v7_v2_128\memory_router_v2.pt" `
--output router_latency_bench_prod.json
```
README.md
+147
View File
@@ -0,0 +1,147 @@
# Natural Memory NM2.1
在**已经训练好的 Qwen3.5-4B** 上做「架构手术」，插入一个可读、可写、可持续更新的记忆模块，让模型在不把历史对话反复塞回上下文的前提下持续积累和使用信息。
本仓库是 [natural-memory](https://github.com/WpyQwq/natural-memory)（V2 时代工程）的**代码分叉**，承载 **NM2 / NM2.1** 阶段的全部源码、评测证据与报告。分叉动机、逐轴结果与踩坑记录见 **[README_FORK.md](README_FORK.md)**（本仓库最详细的一篇文档）。
---
## 一句话结论
> 在**完全相同的存储预算**下（128 维 = 每条记录 512 字节、2,037,774 参数），
> V2-128 v6 路由器在参数、地址字节、延迟、吞吐**全部不劣**的前提下，
> 把 Top-1 从 **41.12% → 94.62%**、未知拒答率从 **0.00% → 100.00%**、hop 正确率从 **73.23% → 100.00%**，
> 逐轴判定 **22/22 通过**。
而这次提升**不是新架构的胜利**：XL-128 / XL-512 准确率最高（Top-1 95.66% / 95.78%），但真实地慢 26–52%、参数 3.3–3.9 倍，并在召回与多跳证据完整度上略逊于同数据的 V2。收益来自**数据与标签的修正**，不是路由器容量。这一条负面结论同样写进了仓库，没有被藏起来。
---
## 本仓库回答的问题
| 轴 | 结果 |
|---|---|
| 检索排序 | Top-1 41.12% → 94.62%；Recall@3 46.73% → 96.48%；MRR 47.69% → 97.12% |
| 未知拒答 | 0.00% → **100.00%**（已知问题被误拒率 0.00%） |
| 多跳证据 | hop 73.23% → 100.00%；Top-3 全中 43.43% → 98.64% |
| 写入存活 | 写 20 条存活 12/20 → 20/20 |
| 零字面重叠改写 | 43.75% → 68.75% |
| 跨架构可移植 | 14 / 14 架构施加记忆手术后**逐位相同**（读写关闭时与原生模型 logits 完全一致） |
| 通用能力回归 | 54 任务上原版与动态记忆版均为 96.30%，差值 0，门禁通过 |
**没有通过的**（诚实记录）：成本轴（单查询延迟、QPS、每条记录地址字节共 22 项未通过）、部分平均分数余量项、XL-final 的多跳证据全中。
---
## 三个被测量推翻的「显然优化」
这是本仓库最有价值的部分——三条都是**先有直觉、后被实测否决**，代码与结论一并保留以便复核：
1. **`logits_to_keep=1`**：`_encode_model_key` 每次前向都算出 `[B, L, 248320]` 的 logits 并全部丢弃（3.05 GB 中间张量 + 约 8.5 TFLOP）。看起来显然该裁掉，实测反而**慢 55%**（283 → 183 texts/s）——`slice(-1, None)` 产生非连续视图，让 4-bit matmul 掉进慢路径。
2. **加大路由器容量**：XL-512 相对同数据 V2-512 只换来 Top-1 **+0.73pp**、MRR +0.12pp，代价是 1.4 倍延迟与 1.67 倍参数，召回与多跳还略降。
3. **一个 46.8 小时的性能缺陷**：`prepare_memory_router_dataset.py` 对每条 episode 重建候选池，引入 mega 验证集后单次列表复制约 2.1 秒。修复后 mega 家族 **3568×** 加速，数据集重建从约 47 小时降到 **114.9 秒**，且输出**逐字节相同**（A/B 同 seed，train/eval sha 一致）。
---
## 仓库结构
```
.
├─ README_FORK.md # ★ 分叉说明：逐轴结果、缺陷根因、常用命令（最详细）
├─ ENGINEERING.md # 原工程说明（Dynamic Memory Lab 时代，与 natural-memory 同源）
├─ NM2_1_FINAL.md # NM2.1 最终报告
├─ NM2_VS_NM2_1.md # NM2 → NM2.1 逐轴对比
├─ V6_FINAL_REPORT.md # v6 路由器训练与判定报告
├─ WRITE_PATH_FIX.md # 写入路径缺陷的根因与修复（含真实调用栈）
├─ ABSTENTION_BREAKTHROUGH.md # 未知拒答从 0% 到 100% 的机制
├─ Natural_Memory_v2_Paper.md # 技术论文
├─ Natural_Memory_v2_对外技术总结.md # 对外技术总结
│
├─ memory_os_v2.py / qwen_integration.py / model.py / router_xl.py # 记忆核心与模型手术
├─ train_*.py # 路由器 / 检索器 / 原生记忆 / 写入策略训练器
├─ eval_*.py / benchmark_*.py # 多轴评分卡与基线对比
├─ audit_*.py / verify_*.py # 独立数据审计与验证
├─ probe_*.py / diagnose_*.py # 诊断探针（剂量曲线、可分离性、可移植性）
├─ stream_feature_bank.py # 流式多线程特征编码（212 万条文本 → mmap 特征库）
│
├─ qwen3_5_4b_natural_memory_v2/ # 模型手术配方（config / memory_config / memory_merge）
├─ qwen3_5_4b_natural_memory_v2_1/ # 同上（NM2.1 终态）
├─ data/ # 可复现的评测语料（zero_overlap / realistic_v2 / locomo 等）
│
├─ router_*.json / nm2_1_*.json ... # ★ 全部评测证据（评分卡、判定、审计结果）
└─ tests/ # 单元测试
```
`*.json` / `*.jsonl` 是**评测证据本体**，不是缓存——每张报告里的数字都能在对应 JSON 里逐条查到。
---
## 未包含在仓库中的内容（及原因）
| 内容 | 体积 | 原因 |
|---|---:|---|
| `qwen3_5_4b_natural_memory_v2{,_1}/model.safetensors-*` | 2 × 9.0 GB | 合并后的模型权重；可用 `merge_memory_weights.py` 从基座 + adapter 重建 |
| `checkpoints/`（router 最终权重与中间快照） | 607 MB | 二进制训练产物；训练脚本与超参已完整保留 |
| `data/router_training_v{3,5,6}/`、`router_replay_v7/` | 4 × 1.4 GB | 冻结训练集，由 `prepare_memory_router_dataset.py` 约 115 秒重建（逐字节可复现） |
| mmap 特征库 `features.f16.npy` | 10.86 GB | 由 `stream_feature_bank.py` 重新编码生成 |
| 用户记忆快照 `*_runtime.pt` / `persistent_memory.pt` | — | **含真实对话内容，按设计绝不入库** |
| Qwen3.5-4B 基座权重 | — | 第三方模型，请自行获取并遵守其许可证 |
---
## 复现
```powershell
# 环境：Python 3.12 + CUDA
pip install -r requirements.txt
# 单元测试
python -m unittest discover -s tests
# 重建数据集（含拒答 / 多跳标签修正，约 2 分钟）
python -m V2_dpskw.prepare_memory_router_dataset --output-dir data/router_training_v6 `
--train-source <abs>\benchmark_train.jsonl ... --candidate-count 32 --conflict-aware --seed 20260909
# 独立审计：类别完整性 + 严格泄漏检查
python -m V2_dpskw.audit_router_dataset `
--train-file data/router_training_v6/train.jsonl --eval-file data/router_training_v6/eval.jsonl `
--output router_dataset_audit_v6.json
# 流式多线程特征编码（212 万条唯一文本 → mmap 特征库）
python -m V2_dpskw.stream_feature_bank `
--train-file data/router_training_v6/train.jsonl --eval-file data/router_training_v6/eval.jsonl `
--model-path qwen3_5_4b_natural_memory_v2 --output-dir <nvme>/nm_router_v6/feature_cache `
--tokenizer-threads 8 --max-batch 192 --token-budget 12288 --gpu-memory-gb 10
# 训练 + 评分
pwsh -File .\run_router_v6.ps1 -Configs v2_512_v6,xl512_v6
python -m V2_dpskw.eval_router_v5 --feature-cache <nvme>/nm_router_v6/feature_cache `
--train-file data/router_training_v6/train.jsonl --eval-file data/router_training_v6/eval.jsonl `
--run "XL-512 v6=checkpoints\router_v6_xl512\router_best.pt" `
--output router_scorecard_v6.json --markdown router_scorecard_v6.md
```
> 原文档中的 `H:\Memory\...`、`W:\Flash\model` 等路径是作者本机路径，复现时请替换为你自己的路径。
---
## 第三方数据归属
`data/net_locomo/` 来自公开研究数据集 **LoCoMo**（*Evaluating Very Long-Term Conversational Memory of LLM Agents*），用于跨语料验证。该目录内的图片签名 URL、对话内容均为上游数据集自带，**不属于本项目**，请遵守上游数据集许可证。其余 `data/` 内容由本仓库脚本生成。
---
## 已知局限
- 未在真实业务分布上做长时间压力测试；合成数据指标**不等于**生产承诺。
- 生产级多用户隔离、加密、并发写入、版本迁移与合规删除尚未实现。
- 需要外部持久化介质保存用户 checkpoint —— 这是信息存在的必要条件，不是实现缺陷。
- 内存治理（事实抽取、去重、时间衰减、审计导出）仍是后续工作。
详见 [README_FORK.md](README_FORK.md) 与 [NM2_1_FINAL.md](NM2_1_FINAL.md)。
---
## 免责声明
本项目为研究性质代码，不保证在所有任务上提升。使用第三方模型权重或数据集时，请遵守对应的模型许可证、数据许可证与隐私要求。
README_FORK.md
+330
View File
@@ -0,0 +1,330 @@
# V2_dpskw —— Natural Memory 路由器分叉
本目录是 `H:\Memory\dynamic_memory_lab`（GPT 时代的原工程）的**代码分叉**，用于训练一个**全新的、更大更强的路由器**。
原工程保持只读不改；本 fork 内所有代码的包名已统一为 `V2_dpskw`。
## 与原工程的关系
| 项目 | 处理方式 |
|---|---|
| 全部源码 / 测试 / 文档 | 复制进本目录，包名 `dynamic_memory_lab` → `V2_dpskw`（51 个文件） |
| Qwen3.5-4B 权重包 | 目录联接（junction）`qwen3_5_4b_natural_memory_v2` → 原工程，不复制 9.3 GB |
| 冻结的 router 训练数据 | 复制 `data/router_training_v3/`（train 7155 / eval 1940，sha256 与原版一致） |
| 冻结的 Qwen 特征库 | 复制 `checkpoints/router_shared/feature_cache/`（21263 × 2560 fp16），manifest 的 `model_path` 已改写为 fork 内路径 |
| 其余 data / checkpoints / 适配器 / 大报告 | **不复制**，需要时按绝对路径引用原工程 |
因此训练时**不会加载 Qwen、也不会重新编码特征**，30 秒内即可开始更新路由器参数。
## 新增内容
| 文件 | 作用 |
|---|---|
| `router_xl.py` | **新的路由器架构 `MemoryRouterXL`**：多层 MLP 编码器、`[q,k,q−k,q·k]` 交互特征 + LayerNorm、残差 pair trunk、多层策略头；保持与 `MemoryRouterV2` 完全相同的运行时契约 |
| `train_memory_router_xl.py` | v3 数据上的训练器（复用原训练器的度量代码，只替换模型） |
| `train_router_v5.py` | **v5/v6 全量数据训练器**：流式读取 JSONL（不把 episode 解析进内存）、**mmap** 10.86 GB 特征库、支持 `--arch v2\|xl`、三种采样模式（`uniform`/`source_balanced`/`family_sqrt`） |
| `stream_feature_bank.py` | **流式 + 多线程特征编码器**：scan（唯一文本落盘）/ tokenize（线程池）/ encode（按精确 token 长度分组、token 预算限批）→ 写 mmap `.npy` |
| `eval_router_scorecard.py` | v3 版多轴评分卡 |
| `eval_router_v5.py` | v5/v6 版多轴评分卡：流式 + mmap + **按 family / 按 10 个类别拆解** |
| `audit_router_dataset.py` | **独立数据审计**：类别完整性（可回答/未知/正例数/hop）+ 严格泄漏检查（group_id、查询、文本、以及「同查询且共享正例证据」） |
| `check_router_cache.py` | 冻结特征缓存快速校验（失配即报错，不会偷偷加载 4B 模型重编码） |
| `compare_router_runs.py` | 与原 512 基线的汇总对比 |
| `tests/test_router_xl.py` | 新架构契约测试 |
| `run_router_v6.ps1` / `chain_v6.ps1` | v6 训练启动器 / 编码完成自动接训练与评分的长链 |
`prepare_memory_router_dataset.py` 额外修复了一个**性能缺陷**（见下）。
## 已验证事实
- 全部单元测试通过：**52 项**（原 45 项 + 新增 7 项）。
- 特征缓存命中，训练全程不加载 Qwen。
- 新架构与原架构同尺寸对比：`router_dim` 相同意味着**每条记忆记录的地址占用完全相同**（512 维 × fp32 = 2048 字节）。
## MemoryRouterXL 容量
| 路由器 | router_dim | heads | 参数 | 相对 V2-512 |
|---|---:|---:|---:|---:|
| V2-512 基线（原工程） | 512 | 8 | 4,741,902 | 1.00× |
| XL-512 | 512 | 8 | 7,898,127 | 1.67× |
| XL-1024 | 1024 | 16 | 18,414,103 | 3.88× |
| XL-2048 | 2048 | 16 | 55,133,719 | 11.63× |
> 只有 XL-512 的地址几何与原版一致（同样的存储成本），因此它是唯一可以做同尺寸对比的配置。
## XL-512 训练结果（100,000 步，冻结 v3 eval：1940 条）
| 指标 | V2-512 best(step1k) | V2-512 final(step100k) | XL-512 best | XL-512 final(step100k) |
|---|---:|---:|---:|---:|
| 参数 | 4,741,902 | 4,741,902 | 7,898,127 | 7,898,127 |
| 每条记录地址字节 | 2048 | 2048 | 2048 | 2048 |
| Top-1 正确率 | **67.12%** | 60.63% | 60.85% | 54.36% |
| Recall@3 | 88.77% | 85.01% | **92.97%** | 85.33% |
| Recall@5 | 94.93% | 90.13% | **97.33%** | 91.93% |
| MRR | **79.08%** | 73.26% | 75.98% | 69.50% |
| nDCG@3 | **80.00%** | 74.57% | 79.28% | 71.71% |
| 多跳证据全中(Top-3) | 88.66% | 84.90% | **92.86%** | 85.22% |
| hop 正确率 | 97.84% | **99.69%** | 99.38% | **99.69%** |
| need F1 | 98.92% | **99.89%** | 99.62% | **99.89%** |
| 未知拒答率 (thr 0.50) | 66.04% | **100.00%** | 90.57% | **100.00%** |
| 已知问题被误拒率 | 0.22% | 0.22% | 0.22% | 0.22% |
| 未知问题被误读率 | 33.96% | 0.00% | 9.43% | 0.00% |
| 平均分数余量 | 0.588 | 3.884 | 6.710 | **25.376** |
| 单查询延迟 ms (GPU) | **0.895** | 0.935 | 1.310 | 1.246 |
| 路由 QPS (GPU) | **1117** | 1070 | 763 | 803 |
**结论不是「谁全面更强」，而是各轴各有胜负：**
1. 原工程用 `selection_score = 0.5·top1 + 0.3·need_f1 + 0.2·mrr` 选出的 "best"（step 1000）Top-1 最高，但**在 0.50 门槛下有 33.96% 的未知问题被强行读记忆**；该综合分几乎不惩罚这一点，因为未知样本只有 106 条。
2. XL-512 在**覆盖率**上明显更好：Recall@3 +4.20pp、Recall@5 +2.40pp、多跳证据全中 +3.96pp。
3. 把门槛从 0.50 提到 0.70，XL-512 best 的未知拒答率达到 **100.00%**，而已知问题被误拒率仍为 **0.22%**——即在同等策略安全水平下，XL-512 的排序覆盖优势可以保留。
4. 代价：XL-512 参数 1.67×、单查询延迟 +39~46%（0.935 → 1.310 ms）、QPS 从 1070 降到 763。
## 数据集构建：一个 46.8 小时的缺陷
`prepare_memory_router_dataset.py` 的 `_source_candidates` 原先对**每条 episode** 都重建候选池：
```python
same_family = [item for item in items_by_family.get(episode.family, []) if item.item_id not in positive_set]
```
引入 mega 验证集后，`mega_validation` 家族的候选池有 **2,656,000** 条，于是一次列表复制约 2.1 秒：
| 家族 | 原版 / 条 | 修复后 / 条 | 加速 |
|---|---:|---:|---:|
| mega_validation | 2105.42 ms | **0.59 ms** | **3568×** |
| memory_policy | 16.61 ms | 2.31 ms | 7.2× |
| native_memory | 7.51 ms | 0.68 ms | 11× |
80,000 条 mega episode 原本需要 **46.8 小时**（`H:\Memory\dynamic_memory_lab\data\router_training_v4` 那次运行了 19.5 小时只写出 91 MB 残缺文件，原因即此）。
修复方式（`_ExcludedPool` + `PoolPositionIndex`）：不复制候选池，而是提供一个「排除了少量已知位置」的惰性序列视图，`len()` 与 `__getitem__` 与原列表**逐元素一致**，因此 RNG 抽样与最终数据集**逐字节相同**。
等价性已验证（A/B，同样输入、同样 seed）：
```
ORIGINAL train_sha=D9C2774955DF7066 eval_sha=169B026DF2A85C35 93.5s
V2_dpskw train_sha=D9C2774955DF7066 eval_sha=169B026DF2A85C35 42.1s
```
全量数据集 `data/router_training_v5`（144 秒建成，原需约 47 小时）：
| 文件 | episodes | 类别 |
|---|---:|---|
| `train.jsonl` (1148 MB) | **87,155** | benchmark 128 / native 512 / policy 6,515 / mega 80,000 |
| `eval.jsonl` (288 MB) | **21,920** | 含 mega 20,000，覆盖全部 10 个类别（每类 2,000） |
`group_overlap = 0`，冲突策略与防泄漏检查与原协议一致。
## 当前阻塞：特征库放不进内存
训练复用 Qwen 冻结特征，所以每条唯一文本都要编码一次：
| 文件 | 唯一文本 | 特征库 (fp16) |
|---|---:|---:|
| v5 train | 1,695,797 | 8.68 GB |
| v5 eval | 425,115 | 2.18 GB |
| 合计 | 2,120,912 | **10.86 GB** |
本机内存 31.2 GB、空闲 12.7 GB，还要容纳解析后的 episode 对象，因此**全量 87k 无法在本机编码**。
可选方案：按类别均衡抽样（约 25–32k episode，特征库 ~5 GB）、先压缩 mega 源、或换更大内存的机器。
**实际落地方式**（已完成）：全量 2,124,552 条唯一文本、10.86 GB 特征库写成 mmap 并用完整扫描验证，训练器流式读取，因此「内存放不下」不再是阻塞项。
## v6 结果：准确率与拒答策略大幅超越，成本轴未超越
训练：V2-512 对照与 XL-512 候选均 100,000 步、batch 64、`family_sqrt` 采样、同一冻结特征库与协议。
评测：冻结 v6 eval 21,920 条（10 类别 × 2,000 + policy 1,920），7 个路由器同题同特征。
| 指标 | 生产包 128 维(v3) | V2-512 v3 best | **V2-512 v6 best** | **XL-512 v6 best** |
|---|---:|---:|---:|---:|
| Top-1 正确率 | 41.12% | 80.75% | 95.05% | **95.78%** |
| Recall@3 | 46.73% | 95.02% | **98.84%** | 97.59% |
| Recall@5 | 48.91% | 97.31% | **99.35%** | 98.34% |
| MRR | 47.69% | 88.46% | 97.12% | **97.24%** |
| nDCG@3 | 44.22% | 89.53% | **97.24%** | 96.64% |
| 多跳证据全中(Top-3) | 43.43% | 94.19% | **98.64%** | 96.86% |
| hop 正确率 | 73.23% | 81.59% | 100.00% | 100.00% |
| **未知拒答率** | 0.00% | 50.41% | 99.81% | **100.00%** |
| **已知问题被误拒率** | 0.18% | 11.23% | **0.00%** | **0.00%** |
| 单查询延迟 (GPU) | **0.844 ms** | 0.879 ms | 0.875 ms | 1.216 ms |
| 批量 QPS (batch=256) | **264,813** | 167,235 | 161,941 | 96,747 |
| 地址字节/记录 | **512** | 2048 | 2048 | 2048 |
判定（`router_verdict_v6.md`，逐轴对比最强旧基线）：**66 项通过 / 22 项未通过 → 全方位超越：否**。
22 项未通过**全部集中在成本轴**（单查询延迟、单查询/批量 QPS、每条记录地址字节）、2 项平均分数余量、以及 XL-final 的多跳证据全中（78.85% vs 84.90%）。所有准确率与拒答策略轴全部通过，且多数是大幅通过：Top-1 +15.03pp、MRR +8.79pp、hop +18.25pp、未知拒答率 +48.71pp、已知被误拒率 −0.11pp。
### 同数据架构对照：XL 相对 V2 并没有赢
在**完全相同**的数据、特征、采样与步数下（best 检查点）：
| 指标 | V2-512 v6 | XL-512 v6 | XL−V2 |
|---|---:|---:|---:|
| Top-1 | 95.05% | 95.78% | **+0.73pp** |
| MRR | 97.12% | 97.24% | +0.12pp |
| 未知拒答率 | 99.81% | 100.00% | +0.19pp |
| Recall@3 | 98.84% | 97.59% | −1.25pp |
| Recall@5 | 99.35% | 98.34% | −1.01pp |
| nDCG@3 | 97.24% | 96.64% | −0.60pp |
| 多跳证据全中 | 98.64% | 96.86% | −1.78pp |
| 单查询延迟 | 0.875 ms | 1.216 ms | **+39% 更慢** |
结论：本次最大的提升来自**数据与标签**（v3 → v6：修正被破坏的拒答/多跳标签、12 倍数据量、10 类别均衡评测），而不是新架构。XL 在 512 维上只换来 Top-1 +0.73pp、MRR +0.12pp，代价是 1.4 倍延迟与 1.67 倍参数，并在召回与多跳证据完整度上略逊于同数据的 V2。若目标是「同尺寸全面超越」，应继续投入数据与训练配方，而不是加大路由器容量。
## 同存储预算的判定：128 维 V2 达到全方位超越
把 512 维路由器与 128 维的生产路由器比「每条记录地址字节」是**不同存储预算的错位比较**。因此在生产几何（128 维 = 512 字节/记录、2,037,774 参数）上重训后重新判定：
| 路由器 | 参数 | 地址字节 | Top-1 | 未知拒答率 | 单查询延迟(中位) | batch256 QPS | 逐候选判定 |
|---|---:|---:|---:|---:|---:|---:|---|
| V2-128 生产(v3) | 2,037,774 | 512 | 41.12% | 0.00% | 1.1558 ms | 199,554 | 基线 |
| **V2-128 v6 best** | 2,037,774 | 512 | **94.62%** | **100.00%** | 1.1679 ms | 196,829 | **22/22 通过 → 是** |
| **V2-128 v6 final** | 2,037,774 | 512 | 94.37% | **100.00%** | 1.1653 ms | 198,026 | **22/22 通过 → 是** |
| XL-128 v6 best | 6,714,639 | 512 | 95.66% | 99.59% | 1.5783 ms | 141,501 | 18/22（吞吐 4 轴）|
| V2-512 v6 best | 4,741,902 | 2048 | 95.05% | 99.81% | 1.1623 ms | 154,085 | 20/22（batch256、地址字节）|
| XL-512 v6 best | 7,898,127 | 2048 | 95.78% | 100.00% | 1.5855 ms | 94,647 | 17/22 |
速度轴以**交替轮流测量**（`bench_router_latency.py`，5 轮，模型轮转以抵消漂移）为准，实测波动带 2.8–5.3%，故速度轴采用 3% 相对容差；质量轴仍为严格容差。此前「V2-128 v6 慢 4.6%」是单次顺序测量的漂移，交替测量下两者差异 **≤1.4%**（单查询 min 甚至更快 0.5%）。
**结论**：V2-128 v6 在参数、地址字节、延迟、吞吐**全部不劣**（差异在测量噪声带内）的前提下，把 Top-1 从 41.12% 提到 94.62%（**+53.49pp**）、未知拒答率从 0.00% 提到 100.00%、hop 从 73.23% 提到 100.00% —— 这是相对已交付生产路由器的**同预算全方位超越**。
**但不是新架构的胜利**：XL-128/XL-512 准确率最高（Top-1 95.66%/95.78%）却真实地慢 26–52%（3.3–3.9 倍参数）。本次提升来自数据与标签，而非路由器容量。
## 常用命令
```powershell
$py = 'C:\Users\Administrator\miniconda3\envs\LLM\python.exe'
$env:PYTHONPATH = 'H:\Memory'
Set-Location 'H:\Memory\V2_dpskw'
# 单元测试
& $py -m unittest discover -s tests
# 特征缓存校验（v3 数据）
& $py -m V2_dpskw.check_router_cache
# 构建数据集（已修复拒答/多跳标签；全量约 2 分钟）
& $py -m V2_dpskw.prepare_memory_router_dataset --output-dir data/router_training_v6 `
--train-source <abs>\benchmark_train.jsonl ... --candidate-count 32 --conflict-aware --seed 20260909
# 独立审计：类别完整性 + 严格泄漏
& $py -m V2_dpskw.audit_router_dataset `
--train-file data/router_training_v6/train.jsonl --eval-file data/router_training_v6/eval.jsonl `
--output router_dataset_audit_v6.json
# 流式多线程特征编码（212 万条唯一文本 → mmap 特征库，放 NVMe）
& $py -m V2_dpskw.stream_feature_bank `
--train-file data/router_training_v6/train.jsonl --eval-file data/router_training_v6/eval.jsonl `
--model-path qwen3_5_4b_natural_memory_v2 --output-dir H:\Memory\nm_cache\nm_router_v6\feature_cache `
--tokenizer-threads 8 --max-batch 192 --token-budget 12288 --gpu-memory-gb 10
# 训练（V2 对照 + XL 候选，同数据同协议）
pwsh -File .\run_router_v6.ps1 -Configs v2_512_v6,xl512_v6
# 全类别百分比评分卡
& $py -m V2_dpskw.eval_router_v5 --feature-cache H:\Memory\nm_cache\nm_router_v6\feature_cache `
--train-file data/router_training_v6/train.jsonl --eval-file data/router_training_v6/eval.jsonl `
--run "XL-512 v6=checkpoints\router_v6_xl512\router_best.pt" `
--output router_scorecard_v6.json --markdown router_scorecard_v6.md
```
## 数据管线（流式 + 多线程）实测
| 阶段 | 结果 |
|---|---|
| scan：去重并落盘唯一文本 | 2,124,552 条 / **15.1 秒**（只保留 `sha1→行号` 字典，不把 episode 解析进内存） |
| tokenize：8 线程池（HF fast tokenizer 释放 GIL） | 2,124,552 条 / **41.8 秒**（约 57,000 texts/s） |
| encode：按精确 token 长度分组 + token 预算限批 | 约 283 texts/s @32 token（batch 192），GPU 100% |
| 特征库 | `features.f16.npy` mmap，10.86 GB，训练时按需读页 |
| 断点续跑 | `--resume` + `progress.json`：记录已完成的长度排序行数，被中断后不重编已完成部分 |
**表示保真度**：与旧冻结缓存（v3）共有文本对比，编码结果 **余弦最低 0.999935 / 平均 0.999983**。因此旧路由器可以直接在新特征库上评分，而不是被换到一套不同表示上再比较。
### 一个被测量推翻的「显然优化」
`QwenDynamicMemoryModel._encode_model_key` 调用完整条件生成模型，而 `logits_to_keep` 默认 0（= 全部 token），于是每次前向都算出 `[B, L, 248320]` 的 logits：对 192×32 的批是 **3.05 GB 中间张量 + 约 8.5 TFLOP**，全部丢弃。看起来显然该改成 `logits_to_keep=1`。实测（交替重复 7 轮取最小值；两条路径特征一致性 cosine **0.9999996**）：
| 编码路径 | 192×32 批耗时 | 吞吐 | 显存峰值 |
|---|---:|---:|---:|
| 全量 logits（默认，保留） | **679 ms** | **283 texts/s** | 5.22 GiB |
| `logits_to_keep=1` | 1050 ms | 183 texts/s | 4.13 GiB |
| batch 512（全量 logits） | — | 287 texts/s | — |
原因：`slice(-1, None)` 产生**非连续视图**，bitsandbytes 的 4-bit matmul 在这个布局上掉进慢路径（多出约 500 ms），而全量 logits 是规整的连续 GEMM，本身只约 130 ms。结论：**不要启用 `--skip-lm-head`**（默认已关，代码与结论一并保留以便复核）。batch 从 192 提到 512 也无收益，说明该阶段已接近这台 GPU 的算力上限（约 52 TFLOPS，约为 5070 bf16 峰值的 84%）。
## v5 → v6：修正被破坏的拒答与多跳标签
`_add_mega_row` 有两个由 GPT 时代沿用下来的缺陷，正好废掉了本次评测最需要的两个轴：
1. `unknown_abstention` / `forget_correction` 的源数据带 `metadata.answerable = false`，其 `acceptable` 装的是**拒答话术**（"不知道"/"没有记录"）。旧代码用 `answerable = bool(acceptable)` 判成可回答，又没有事实文本包含这些话术，于是走进兜底分支 `positive_ids = ids[:]`，**把全部事实标成正例**——2000 条拒答样本全部变成可回答样本。
2. `multi_hop` 的真实跳数在 `metadata.hop_count = 2`，旧代码用 `len(positive_ids)` 计算，且只标注了含最终答案的那条事实，**中间推理链事实没有标成证据**。
修正：尊重 `metadata.answerable`；用 `metadata.hop_count` 作为 hop 标签；沿「实体→值」链扩展证据（有界，最多 3 跳）。修正前后对比：
| 指标 | v5（有 bug） | **v6（已修正）** |
|---|---:|---:|
| train 未知样本 | 319 | **16,319** |
| eval 未知样本 | 106 | **4,106** |
| `unknown_abstention` 可回答/未知 | 2000 / **0** | **0 / 2000** |
| `forget_correction` 可回答/未知 | 2000 / **0** | **0 / 2000** |
| `multi_hop` 平均正例数 / hop | 1.00 / **1** | **2.00 / 2** |
数据集重建耗时 **114.9 秒**（未修复前同一构建需要约 47 小时）。
### v6 独立审计结果
| 检查 | 结果 |
|---|---|
| `group_id` 重叠 | **0** |
| 查询文本重叠 | 80 / 4,296 = **1.86%** |
| **同查询且共享同一正例证据** | **0**（80 条均为模板化问句配不同事实，无泄漏） |
| 候选文本重叠 | 3 / 421,473 = **0.0007%** |
| 类别 | episodes | 可回答 | 未知 | 平均正例 | hop |
|---|---:|---:|---:|---:|---|
| unknown_abstention | 2000 | 0 | **2000** | 0.00 | 0 |
| forget_correction | 2000 | 0 | **2000** | 0.00 | 0 |
| multi_hop | 2000 | 2000 | 0 | **2.00** | **2** |
| 其余 8 类 | 各 2000 | 2000 | 0 | 1.00–2.00 | 1–2 |
## NM2 跨架构泛化（实测矩阵）
原集成的记忆手术只适配 Qwen3.5 一族的解码层。`probe_nm2_portability.py` 对 14 个架构真机施加手术，
并做**最强保能力检验**：记忆读写关闭时，包装后的模型必须与原生模型的 logits **逐位相同**。
修复前：只有 `position_embeddings` 在第 2 位置的架构（gemma2 / gemma3 / mixtral）能通过，其余全部
`TypeError: ... got multiple values for argument 'attention_mask'` 或找不到层容器。
修复后（**零补丁、全部逐位相同**）：
| 架构 | 层容器 | 第 2 位置参数 | 恒等性 |
|---|---|---|---|
| llama / mistral / qwen2 / qwen3 / starcoder2 / cohere / phi | `model.layers` | `attention_mask` | identical |
| olmoe / granitemoe | `model.layers` | `attention_mask` | identical |
| gemma2 / gemma3 / mixtral | `model.layers` | `position_embeddings` | identical |
| opt / bart | `model.decoder.layers` | — | identical |
为达到这一点只改了 4 处，**记忆核心与路由器一行未改**（它们只依赖 `hidden_size`）：
| 改动 | 解决的假设 |
|---|---|
| `resolve_text_config()` | 纯文本模型没有 `config.text_config` |
| `resolve_decoder_layers()` | 层容器固定为 `model.language_model.layers`（现支持 8 条候选路径） |
| `MemoryLayerAdapter` 约定检测 + **按声明参数过滤** | `position_embeddings` 被按位置硬传；参数**名称**也不同（Ling 用 `past_key_value` 单数，且未必有 `**kwargs`） |
| `load_qwen_base()` 兜底 `AutoModelForCausalLM` | 只有 image-text-to-text 加载器 |
对 **Qwen3.5 生产路径零行为变化**（已证明）：真实 `Qwen3_5DecoderLayer.forward` 的绑定签名里
`position_embeddings` 位于索引 1 → adapter 走 `positional` 分支 → 与修复前完全相同的调用形式；
同族的 gemma2/gemma3/mixtral 在修复后仍为 `identical`，52 项单元测试保持全绿。
### 第二类兼容性：远程代码 vs transformers 版本
`inclusionAI/Ling-3.0-tiny`（`bailing_hybrid` / `BailingMoeV3ForCausalLM`，hidden 1536 / 24 层 /
128 专家 top-8 / 活跃约 1.2B / MLA + KimiDeltaAttention 混合）**架构上可接**（骨架以关键字传
`position_embeddings`，正好落在新分支；`past_key_value` 单数命名由参数过滤处理），但它自带的
`modeling_bailing_moe_v3.py` 是按**旧版 transformers API** 写的：
- `from transformers.utils.import_utils import is_torch_fx_available` → 5.9 已移除该符号；
- `config.rope_scaling["factor"]` → 5.9 会把 `rope_scaling` 规范化掉，运行期 KeyError。
也就是说「嵌进任意模型」有两层前提：**架构层（已解决，14/14）** 与 **栈层（需兼容 shim 或独立 venv）**。
README_NATURAL_MEMORY_V2.md
+410
View File
@@ -0,0 +1,410 @@
# Natural Memory v2
Natural Memory v2 是接在本地 Qwen3.5-4B 上的一层分层、可寻址、可纠错记忆系统。它的目标不是把一百万个 slot 当作一张巨大的 KV Cache，而是把历史中最适合长期保存的部分压缩成有地址的记忆记录；当前对话仍由 Qwen 的热 KV 负责连续理解。
## 先看结论
当前版本已经完成以下闭环：
```text
普通用户输入
-> 自动写入策略
-> 置信度 / 重要性 / 来源审计
-> 紧凑地址投影
-> 分页存储
-> LSH 粗索引
-> 候选页重排
-> 候选记录精排
-> Top-K 证据注入 Qwen
-> 版本冲突 / 隔离 / 撤回 / 重启恢复
```
最终运行包位于：
`W:\Flash\model\V2_dpskw\qwen3_5_4b_natural_memory_v2`
它不需要额外的 `memory_state.pt` 才能恢复已嵌入的记忆；记忆快照、路由器权重、V2 页面元数据和上下文片段都写在该包的第三个 memory safetensors 切片中。当前包的 manifest 指向实际使用的 runtime memory shard；旧 runtime shard 不会被自动覆盖，以避免 Windows 进程仍持有文件句柄时破坏现有包。当前交付配置固定使用 embedded weight-shard 模式：启动时完整载入进程内存，路由命中的热点记录才进入有界 VRAM cache，不使用 SQLite 或磁盘分页。
## 核心设计
### KV 与 Memory Slot 的分工
```text
最近 32K token（最多可配置到 128K） -> GPU 热 KV：保留精确顺序和局部连贯性
长期个人事实 / 项目决策 / 纠错版本 -> Memory Slot：保存压缩、可寻址的证据
当前问题 -> 稀疏路由：只读取少量相关记录
```
Memory Slot 不模拟完整 KV。它只接管 KV 中最昂贵、最适合长期保存、最容易重复利用的部分。这里的目标是比长 KV 获得更高压缩率，而不是把每条对话都压成一句极短摘要：短事实和有价值的对话碎片会保留原始 token 序列，同时另存一个较小的语义地址用于路由。这样仍不能承诺逐 token 无损复现数百万 token 的原始上下文，但能保留足够细节供模型复用。
### 路由路径
任何查询都必须经过有界路径：
```text
查询 hidden state
|
v
128 维紧凑地址
|
v
LSH 粗索引（精确桶 + Hamming-1/2 探针 + 热页）
|
v
候选页（不是全部页面）
|
v
页级与记录级精确重排
|
v
最多 top_k_pages / top_k_records
```
当前 token 不会对 1M slot 做全量注意力。页内才会做小规模记录评分；默认是最多 4 个页、8 条记录。粗索引结果还会记录在诊断 trace 中，便于检查“是否因为候选页不足而漏召回”。
### 记录结构
每条 `MemoryRecordV2` 包含：
- 原始短文本和可选 token 序列；独立碎片不会因为共享一个热槽而被合并；
- 128 维紧凑地址与摘要地址；
- 可选的冻结 Qwen 语义检索键，用于重启后的更新/删除判定，不参与全量注意力；
- `entity / attribute / value` 冲突键；
- 时间戳、版本号、来源、证据；
- 置信度、重要性、访问次数；
- `active / superseded / retracted / quarantined` 状态；
- `supersedes` 和 `related_ids`，用于版本追踪和多跳检索。
重复文本是幂等写入；同一实体和属性的新值会生成新版本并将旧值标记为 `superseded`。自动自然语言路径对“更新”使用比“读取”更严格的门槛：只有高置信语义匹配或明确词面证据才会退役旧碎片；无法确认同一属性时，宁可保留两条独立记录。不可信写入进入 quarantine，不参与正常读取，只有显式批准后才会变成 active。撤回不会抹掉审计记录，而是把记录置为 `retracted`。
### 多跳读取
记录可以带 `related_ids`。第一跳找到一个项目、人物或事件后，路由器会沿关联记录继续查找，直到：
- 达到 `memory_max_hops`；
- 达到 Top-K；
- 没有新关联页；
- 没有新证据。
每次读取都返回 `hop_trace` 和 `stop_reason`，不是只返回一段无法解释的文本。
### 写入安全边界
自动写入由原有自然语言记忆策略决定；V2 另外检查置信度和重要性。默认写入阈值为 `0.50`，读取阈值为 `0.65`。读取阈值是特意偏保守的：在实际 Qwen 测试中，未知的“我的血型是什么”最初会受到短中文问句的语义相似度干扰；提高阈值后该问题被路由层拒绝，不再把无关事实放进上下文前缀。
这不是“绝不出错”的证明，而是一个可检查的安全策略：
```text
低置信度写入 -> quarantine
旧值被纠正 -> 新版本 active，旧版本 superseded
用户撤回 -> retracted，读取隔离
证据不足 -> router_abstained / below_read_threshold
```
## 训练
### 通用路由器训练
```powershell
Set-Location W:\Flash\model
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe `
-m V2_dpskw.train_memory_router_v2
```
训练目标包括：
1. 在 hard negatives 中选对目标记录；
2. 判断当前问题是否需要记忆；
3. 预测是否需要继续多跳；
4. 让无记忆问题学会 abstain。
### Qwen hidden-state 路由器训练
最终包使用的是 Qwen3.5 hidden state 上训练的路由器：
```powershell
Set-Location W:\Flash\model
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe `
-m V2_dpskw.train_qwen_router_v2 `
--output-dir W:\Flash\model\V2_dpskw\checkpoints\natural_memory_v2_qwen_router_entities
```
当前训练数据是本地生成的实体—属性—值事实及 hard negatives，不是公共榜单数据集。因此训练结果可以证明工程链路有效，但不能直接等同于公开 benchmark 的泛化能力。后续正式训练应加入真实对话脱敏集、改写问句、时间冲突、跨语言表达、未知事实和长文档事件链。
### 构建嵌入包
在支持硬链接的 NTFS 目录中，可以从 v1 包构建新的完整目录：
```powershell
Set-Location W:\Flash\model
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe `
V2_dpskw\build_natural_memory_v2_package.py `
--base-package W:\Flash\model\V2_dpskw\qwen3_5_4b_memory_merged_v13 `
--output-dir W:\Flash\model\V2_dpskw\qwen3_5_4b_natural_memory_v2_new `
--router-checkpoint W:\Flash\model\V2_dpskw\checkpoints\natural_memory_v2_qwen_router_entities\memory_router_v2.pt
```
W: 当前是 exFAT，不能创建硬链接。构建器现在默认拒绝复制多 GB 的冻结分片，必须明确加 `--allow-copy-base` 才允许复制；这样可以避免一次构建意外耗尽磁盘空间。当前交付包使用已存在的 v2 包原地更新，未重复复制两片 Qwen 主权重。
## 运行流式聊天
```powershell
Set-Location W:\Flash\model
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe `
-m V2_dpskw.stream_chat_qwen_memory `
--model-path W:\Flash\model\V2_dpskw\qwen3_5_4b_natural_memory_v2 `
--max-new-tokens 128
```
当前生产默认不启用 SQLite/磁盘分页。若做独立的容量研究，旧版仍保留可选的分层页库参数，但它不属于本次默认运行路径：
```powershell
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe `
-m V2_dpskw.stream_chat_qwen_memory `
--model-path W:\Flash\model\V2_dpskw\qwen3_5_4b_natural_memory_v2 `
--tiered-memory-path W:\Flash\model\V2_dpskw\memory_pages.sqlite `
--memory-resident-pages 64 `
--kv-offload
```
启动后发送普通自然语言即可触发自动判断；不需要 `/remember`。常用控制命令仍保留：
- `/save`：把当前持久记忆写入 memory safetensors 切片；
- `/reset`：清空持久记忆并保存；
- `/quit`：退出。
模型重启时不会收到历史聊天记录。它只从嵌入式 memory shard 恢复记录、路由器和审计元数据。
## 单用户本地生产工作流
当前优先完成的 2/3/4/5 已经集中到一个入口：
```powershell
Set-Location W:\Flash\model
$py = "C:\Users\Administrator\miniconda3\envs\LLM\python.exe"
# 2. 规范化真实对话导出，并按 group_id 防止 train/eval 泄漏
& $py -m V2_dpskw.natural_memory_app build-dataset `
--source W:\path\to\redacted_conversations.jsonl `
--eval-source W:\path\to\redacted_eval.jsonl
# 2. 训练候选自动写入策略；不会覆盖当前生产适配器
& $py -m V2_dpskw.natural_memory_app train-policy `
--model-path W:\Flash\model\V2_dpskw\qwen3_5_4b_natural_memory_v2 `
--base-adapter W:\Flash\model\V2_dpskw\qwen_memory_adapter_natural_auto_v13 `
--steps 240
# 4. 低显存连续运行与长上下文压缩压力测试
& $py -m V2_dpskw.natural_memory_app stress `
--model-path W:\Flash\model\V2_dpskw\qwen3_5_4b_natural_memory_v2
# 5. 启动本地 API；默认只监听 localhost
& $py -m V2_dpskw.natural_memory_app serve `
--model-path W:\Flash\model\V2_dpskw\qwen3_5_4b_natural_memory_v2 `
--port 8765
```
默认训练数据目录是 `data/production_memory`，包含 `train.jsonl`、`eval.jsonl` 和
`manifest.json`。没有提供真实脱敏对话时，构建器会使用工程内 bootstrap 数据；这只能验证链路，不能冒充真实业务泛化结果。
管理接口：
```text
GET /health
GET /v1/memory?status=active&query=...
GET /v1/memory/{record_id}
GET /v1/memory/export
GET /v1/memory/audit
POST /v1/memory/{record_id} # 版本化编辑
DELETE /v1/memory/{record_id} # 可审计撤回
POST /v1/memory/reset # 清空并持久化
POST /v1/chat # {message,max_new_tokens,stream}
POST /v1/memory # 管理员/测试用显式写入
```
`POST /v1/chat` 默认由模型自己的自动策略决定是否写入；`stream: true` 返回 SSE token 流。服务以单模型锁串行化请求，避免同一用户的 memory state 被并发写坏。默认自动持久化会回写 embedded memory safetensors；测试时可加 `--no-auto-persist`。
## 记忆管理与审计
编辑不是覆盖原记录，而是生成 `version + 1` 的 successor，并把旧记录标记为
`superseded`；删除同样不物理抹除，而是标记为 `retracted`。`GET /v1/memory/audit`
会检查页容量、页指针、冲突索引和多跳关联是否存在悬空引用。所有管理接口只暴露 JSON-safe
元数据，不返回路由向量和模型内部 tensor。
## 当前受控验证结果
- 单元测试：20/20 通过；
- bootstrap 数据规范化：训练 1485 条，验证 363 条；
- 候选自动策略 smoke train：4bit、batch 2、24 steps，验证集 accuracy 96.88%、recall 90.91%、FPR 0；这不是最终生产成绩；
- 10 轮压力测试：写入 40 条，召回 39/40（97.5%），2 次长上下文压缩，审计 healthy，0 errors；
- 压力测试峰值约 3.2 GB allocated VRAM，使用 256 条/131072 token 的自适应热点缓存上限，并保留 2048 MB 显存安全余量；
- 压力测试不会写回模型包，生产服务只有在启用自动持久化时才会写回。
- v7 自然语言检索器 + 碎片保留策略烟测：teacher 19/20（95%），Natural Memory 20/20（100%），student/teacher = 1.0526，paired parity = 19/19；多跳、冲突更新、删除、干扰项、长上下文和未知拒答全部通过。
- 默认嵌入 v7 包的同一烟测：teacher 19/20（95%），Natural Memory 20/20（100%），parity gate 通过；报告为 `mega_memory_vs_full_kv_smoke20_embedded_v7.json`。
## 重启验证
```powershell
Set-Location W:\Flash\model
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe `
-m V2_dpskw.test_natural_memory_v2_restart `
--model-path W:\Flash\model\V2_dpskw\qwen3_5_4b_natural_memory_v2 `
--report W:\Flash\model\V2_dpskw\natural_memory_v2_restart_test.json
```
测试会：
1. 清空选定包的持久记忆；
2. 通过普通用户句子自动写入一条控制事实；
3. 写入嵌入式 safetensors memory shard；
4. 释放第一个 Qwen 模型；
5. 重新加载模型，只输入一个新问题；
6. 检查 V2 router decision、内部 prefix 和生成答案；
7. 默认清理测试事实，避免污染工作包。
当前实际结果：自动写入成功；重启后路由器找到 `page_00000001` 和目标记录；内部前缀长度 36；生成结果精确返回 `NM-V2-RESTART`；清理后页面与记录均为 0。
## 评测
### 单元测试
```powershell
Set-Location W:\Flash\model
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe `
-m unittest discover -s V2_dpskw\tests -v
```
当前结果：20/20 通过，覆盖路由张量形状、压缩地址、页粗索引、版本冲突、quarantine、批准、撤回、多跳、导出恢复、语义检索键导出恢复、KV 预算、页容量上限、批量上下文分块、记忆编辑/撤回/审计，以及分层后端重启、冷页卸载和隔离区恢复。
### 超大自然语言记忆验证集
`data\mega_validation\memory_validation_100k.jsonl` 已完整生成 100,000 条用例，固定分成 10 类、每类 10,000 条：单事实、32/128 干扰项、冲突更新、多跳、未知拒答、随机位置、长文本、改写问句和删除纠错。当前文件 SHA-256 为 `dafefba852ef1539fba3391b276aa264759395e9677d54fdd5b4fce49ada05b8`。
20 条分层 smoke 用例用于本机快速质量门槛；全量 100,000 条会显著增加推理时间，建议在长时间窗口执行，并持续观察 RTX 5070 显存，不要与训练任务并行：
```powershell
Set-Location W:\Flash\model
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe `
-m V2_dpskw.natural_memory_app benchmark-kv `
--model-path qwen3_5_4b_natural_memory_v2 `
--adapter qwen_memory_adapter_natural_production_candidate_v7 `
--data data\mega_validation\memory_validation_100k.jsonl `
--output mega_memory_vs_full_kv_100k_v7.json `
--limit 100000 --max-new-tokens 96
```
在全量端到端生成完成前，100,000 条数据是“完整验证集”，不是已经完成的 100,000 条生成成绩；当前可复现的质量结论以 smoke 报告和单元测试为准。
### V2 存储与路由评测
报告：`W:\Flash\model\V2_dpskw\natural_memory_v2_benchmark.json`
| 指标 | 实测结果 |
|---|---:|
| 合成记录 | 20,000 |
| 页面数 | 626 |
| 粗候选页平均数 | 178.13 |
| 粗候选页占比 | 28.45% |
| 记录 Recall@K | 100% |
| 页面 Recall@K | 100% |
| 多跳成功 | 100%，2 hops |
| 导出/恢复后召回 | 100% |
| 冲突版本/纠错/隔离/批准/撤回/幂等 | 全部通过 |
| 地址空间容量（32K 页 × 32） | 1,048,576 条记录 |
| 通用路由器 route accuracy | 99.32% |
| 通用 need-memory precision/recall/specificity | 100% / 100% / 100% |
| 通用 hop accuracy | 86.25% |
上述是控制变量下的合成存储评测，证明的是分页、索引和状态机，不是 1M 条真实用户记忆已经完成验证。
### 分层后端百万级压力测试
报告：`W:\Flash\model\V2_dpskw\tiered_memory_v2_1m_benchmark.json`
该测试实际写入 1,000,000 条轻量记录、31,250 页和 64 个常驻页。它是旧的 durable page store 容量实验，不是当前默认方案；当前默认方案要求所有记忆随第三个 safetensors 切片加载进进程内存，再用有界 VRAM cache 加速热点记录。无论哪种方案，这个数字都不等于百万条完整自然语言长文本在 Qwen 上的端到端生成质量。
| 指标 | 实测结果 |
|---|---:|
| 实际写入记录 | 1,000,000 |
| 实际页面 | 31,250 |
| 重启后记录总数 | 1,000,000 |
| 重启后常驻记录 | 160 |
| 冷页 | 31,186 |
| 目标记录重启召回 | 通过 |
### Qwen 路由器验证
报告：`W:\Flash\model\V2_dpskw\checkpoints\natural_memory_v2_qwen_router_entities\qwen_router_v2_training.json`
- 512 条生成事实；409 条训练，103 条 held-out；
- route accuracy：91.26%；
- need-memory precision/recall/specificity：100% / 100% / 100%；
- hop accuracy：36.70%。
hop controller 目前明显弱于候选记录路由，因此运行时不会把它当成唯一正确性来源；关联记录和显式 `related_ids` 仍由存储层约束，后续训练应重点补多跳样本。
### 与原版 Qwen3.5-4B 的综合回归
报告：`W:\Flash\model\V2_dpskw\natural_memory_v2_full_benchmark.json`
同一份 120 个固定用例、同一 Qwen3.5-4B 主干、同一 4-bit NF4 加载和贪心解码：
| 指标 | 原版 Qwen3.5-4B | Natural Memory v2 |
|---|---:|---:|
| 总分 | 0.85833 | 0.85833 |
| 总分变化 | - | 0 |
| 通用 / 数学 / 推理 / 语言 / 知识 / 逻辑 / 上下文分类 | 基线 | 各分类 delta = 0 |
| 自动写入 precision | - | 100% |
| 自动写入 recall | - | 100% |
| 自动写入 specificity | - | 100% |
| 无历史重启恢复 | - | 通过 |
| 清理后不再召回 | - | 通过 |
综合评测的硬件是 RTX 5070 11.94 GiB，完整模型使用 4-bit NF4。显存峰值字段来自 PyTorch allocator，在当前 Transformers/bitsandbytes 组合下可能高于物理显存读数，不能把该字段当成独立硬件测量；最终是否能运行，应以实际 GPU OOM 和 `nvidia-smi` 为准。
### 长上下文边界
报告：`W:\Flash\model\V2_dpskw\long_context_v1_stress.json`
现有直接 KV 路径在约 8K token 可以运行，16K 和 32K 会 OOM。现在 V2 已接入 Transformers 的 CPU-backed `DynamicCache(offloading=True)`，并在 Qwen3.5 混合线性/全注意力结构上完成真实生成验证；线性注意力的微小循环状态留在执行设备，只有昂贵的 full-attention KV 迁移到 CPU。
模型还提供自动热窗口压缩：当输入超过 `kv_budget_tokens`，旧前缀会按 `context_chunk_tokens` 分块写入 V2 context records，当前生成只保留最近热窗口；读取仍然走“粗索引 -> 候选页 -> 精排 -> Top-K”，不是让当前 token 对 1M slot 做注意力。真实自动路径测试将 371 token 压缩为 32 token，并写入 22 条可追溯上下文记录，随后成功生成。
这仍然不能宣称当前包支持 200M—300M 原始 token 上下文：CPU KV offload 不是 NVMe 分页注意力，自动压缩目前也没有完成百万 token 的 Qwen 质量训练。正确的工程目标是热 KV 保留当前窗口，超出窗口的内容经过事件切分、摘要、实体关系和可逆原文页写入 V2；查询时只加载少量相关页，再由 Qwen 做最终回答。
KV offload 与自动压缩验证：
```powershell
Set-Location W:\Flash\model
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe `
-m V2_dpskw.benchmark_kv_offload
```
## 关键源文件
| 文件 | 作用 |
|---|---|
| `memory_os_v2.py` | V2 路由器、分页存储、LSH 粗索引、精排、多跳、版本和 KV 预算 |
| `qwen_integration.py` | 将 V2 接入 Qwen，负责内部 prefix、嵌入式 safetensors 读写和重启恢复 |
| `stream_chat_qwen_memory.py` | 不依赖历史上下文的流式聊天与自动写入 |
| `train_memory_router_v2.py` | 通用路由器 hard-negative 训练 |
| `train_qwen_router_v2.py` | Qwen hidden-state 路由器训练 |
| `tiered_memory_store_v2.py` | RAM/磁盘分层、二进制记录和可恢复 page store |
| `benchmark_tiered_memory_v2.py` | 1M 级 durable page store 压力测试 |
| `benchmark_memory_v2.py` | 独立存储/路由/完整性评测 |
| `benchmark_natural_memory_v1.py` | 兼容 v1/v2 的 Qwen 综合回归评测 |
| `test_natural_memory_v2_restart.py` | 真实模型释放、重载、无历史召回测试 |
| `tests/test_memory_os_v2.py` | V2 核心单元测试 |
## 当前限制与下一阶段
已经实现的是可运行的 V2 内核和百万级 durable 存储后端；模型质量和长上下文仍有明确边界：
1. 真实 Qwen 路由器只用 512 条本地合成事实训练，需继续做跨实体、改写、冲突、时间和未知事实泛化；
2. 多跳控制器的 held-out accuracy 只有 36.70%，需要专门的多跳 curriculum；
3. 当前交付包使用 embedded weight-shard 模式，所有 V2 记录随第三个 safetensors 切片加载进 RAM；热点记录优先进入 VRAM，最多 256 条/131,072 token，并动态保留 2 GiB 显存安全余量；显存不足时自动留在 RAM，避免把模型推理和 KV 顶满；
4. `memory_max_pages=32768` 代表 32K 页 × 32 条记录的地址上限；1M 轻量记录已经完成存储压力测试，但不是 1M 条完整长文本的 Qwen 端到端质量验证；
5. 还需要加入更强的压缩摘要、原文页、事件时间线、事实置信度校准和用户级撤销日志；
6. 还需要长序列课程训练、NVMe 级分页和更大规模端到端评测，才能验证 128K 热 KV 与百万级历史的实际吞吐和质量。
这些限制是设计边界，不是用一个“无限上下文”数字掩盖的未验证假设。
REALISTIC_CORPUS_FINDINGS.md
+77
View File
@@ -0,0 +1,77 @@
# 真实形态语料的结果：此前的增益基本不迁移
## 1. 为什么先做这件事
项目里所有数字此前都建立在**模板语料**上：单一句式（"我的X是 VAL-…。"）、单一实体（user）、
24 个属性、没有更新冲突/多跳链/噪声/多实体。这些数字内部一致、可复现，但**没有一个能预测真实文本上的表现**。
现在有了 `make_realistic_memory_eval.py`：200 条 / 8 类别（175 可回答 + 25 未知），覆盖
多实体（user/张三/李工/项目 Alpha/客户 Beta）、别名改写、更新冲突、多跳链、近邻属性干扰、
噪声上下文、长事实、未知属性。
**语料本身修过一轮**：第一版有 3 个类别是我构造错的（自相矛盾的别名集、把"未知属性"的句式用在了候选上、
多跳答案靠拼接必然出现），那些低分/满分都不是模型的问题。修正后重测，下面的数字来自修正版。
## 2. 结果（每类 25 条，同一份运行时，只有模型包不同）
| 类别 | 原版 NM2 | **NM2.1** | 差值 |
|---|---:|---:|---:|
| alias_paraphrase（别名改写） | **92.00%** | 80.00% | **−12.00pp** |
| long_fact（长事实） | 76.00% | **84.00%** | +8.00pp |
| multi_entity（多实体） | 92.00% | 92.00% | 0 |
| multi_hop（多跳） | **40.00%** | 32.00% | **−8.00pp** |
| near_miss（近邻干扰） | **52.00%** | 48.00% | −4.00pp |
| noise_context（噪声上下文） | 80.00% | **84.00%** | +4.00pp |
| **unknown_attribute（未知属性）** | **20.00%** | **8.00%** | **−12.00pp** |
| update_conflict（更新冲突） | 64.00% | **72.00%** | +8.00pp |
| **总体（200）** | **64.50%** | **62.50%** | **−2.00pp** |
| 可回答正确率（175） | 70.86% | 70.29% | −0.57pp |
| **未知拒答率（25）** | **20.00%** | **8.00%** | **−12.00pp** |
| 已知问题被误拒率 | 2.86% | 2.29% | −0.57pp |
## 3. 结论：此前的增益在真实形态数据上不复现
| 此前的说法（模板语料） | 真实形态语料上的实际结果 |
|---|---|
| NM2.1 总体 **+6.50pp** | **−2.00pp**（200 条） |
| 可回答 **+7.43pp** | **−0.57pp**（175 条） |
| **未知泄漏 75.00% → 0.00%** | 覆盖门在真实数据上**根本不生效**；未知拒答率 **8.00%**，**低于原版的 20.00%** |
| 零字面重叠改写 **+25.00pp**（16 条） | 别名改写 **−12.00pp**（25 条） |
### 原因（已定位，不是猜测）
1. **覆盖门在真实数据上是关闭的**（这是设计行为）：它的词表是模板语料的 24 个属性，而真实事实里的属性
是"张三的值班电话""项目 Alpha 的接口版本号"这类实体前缀属性，属于词表之外 → 自门控判定"词表未被填充"
→ 直接旁路。所以**那个 0.00% 泄漏只在它的专属域里成立**，我在报告里已标注为域内有效，现在有了确凿度量。
2. **路由器替换的收益是分布相关的**：在噪声上下文 / 长事实 / 更新冲突上 +4~+8pp，但在别名改写 / 多跳 /
近邻干扰上 −4~−12pp。模板语料的分布恰好放大了前者的收益。
3. **多跳与近邻在修正后确实变难了**：多跳 40.00% / 32.00%，近邻 52.00% / 48.00% —— 这两个类别的真实水平
远低于我此前用残缺构造测出的 100.00%。
### 统计口径提醒
每类只有 25 条，**1 个用例 = 4.00pp**；总体的 −2.00pp 是 4 个用例，未知的 −12.00pp 是 5 vs 2 个用例。
因此逐类差值都在噪声量级，**不能说"NM2.1 更差"，只能说"没有可测出的优势"** —— 与此前"大幅超越"的
表述形成明确对照。
## 4. 这对下一步意味着什么
**优先级必须重排。** 在真实形态数据上，最重要的三个短板是：
1. **多跳只有 32~40%**、**近邻干扰 48~52%** —— 这是检索/排序的真实水平，之前被模板语料掩盖了。
2. **未知拒答 8.00%（原版 20.00%）** —— 覆盖机制必须能处理**从库里动态生成的属性词表**
（实体前缀属性），而不是固定的 24 个；这正是我此前列为"数据工程"的那一项，现在它的优先级最高。
3. **别名改写 80.00%** —— 说明改写泛化能力其实不差（80%），但比原版低 12.00pp，`REPLAY` 的微调在这里
是负收益，需要重做数据配比。
**并且必须在真实形态数据上重新训练**：现在的 v6 数据集本身也是模板生成的，所以"冻结评测集 22/22"
只是"在这个生成分布上 22/22"。要拿到真实可用的路由器，训练集本身得换成真实形态。
## 5. 产物
| 文件 | 内容 |
|---|---|
| `make_realistic_memory_eval.py` | 语料生成器（8 类别，含三处构造 bug 的修复注释） |
| `data/realistic_eval.jsonl` + `.manifest.json` | 200 条真实形态评测语料（含 sha256 与逐类计数） |
| `realistic2_orig_e2e.{json,md}` / `realistic2_nm21_e2e.{json,md}` | **修正版语料上的对照结果（本文档数字来源）** |
| `realistic_orig_e2e.*` / `realistic_nm21_e2e.*` | 第一版（有构造 bug）的结果，保留以说明差异来源 |
REALISTIC_V2_FINDINGS.md
+131
View File
@@ -0,0 +1,131 @@
# 真实工作泛化能力：用真实形态数据重训路由器的完整结论
> ## ⚠️ 本文的端到端数字已被证伪，读之前先看 `MEASUREMENT_FIXES.md`
>
> 2026-09-15 复核发现，本文第 3 节起引用的端到端正确率（**72.57% / 63.50% / 64.00%**）
> 由两个评分器缺陷造成，**不是模型的真实水平**：
>
> 1. 评分器对空白敏感 —— 期望锚点 `值班人-5259`、模型答 `值班人 -5259`（多一个空格）判错，
> **19 条多跳用例全部因此被判错**（多跳真实是 **100.00%**，不是 24.00%）；
> 2. 拒答检测只认固定关键词，漏掉模型实际使用的措辞（「未包含相关信息，无法回答」），
> 把 **52.00%** 的未知拒答率报成 **0.00%**。
>
> 修正后，同一份权重、同一份语料的真实水平是
> **总体 82.00% / 可回答 86.29% / 未知拒答 52.00% / 已知被误拒 2.29%**。
>
> **本文仍然成立的结论**：路由器在未见属性族上确实能练出泛化能力（Top-1 49.14% → 94.86%），
> 而且该增益不传递到端到端答案 —— 这一条与空白缺陷无关，仍然有效。
> **本文不再成立的结论**：所有以 72.57%/63.50% 为基线的「差值」与「瓶颈排序」，
> 包括「多跳是瓶颈」和「未知拒答率 4.00%/0.00%」。
## 1. 做了什么
| 步骤 | 结果 |
|---|---|
| 真实形态**训练集** | `data/realistic_v2/train.jsonl`，3200 条 / 8 类别（多实体、别名、更新冲突、多跳、近邻干扰、噪声、长事实、未知） |
| 真实形态**评测集** | `data/realistic_v2/eval.jsonl`，200 条 / 8 类别 |
| **属性族不相交** | 36 个属性族按字典序三分：训练 24 / 评测 12，**overlap = []**（评测用的属性名与句式训练时从未出现） |
| 特征编码 | 10,034 文本 / 60 秒（流式+8 线程） |
| 训练 | V2-128（drop-in 几何），从现有交付件出发，12,000 步 / 1e-4 / need+hop 正常权重 |
## 2. 路由器级：真实泛化能力被练出来了（强结果）
在**未见属性族**上：
| 检查点 | Top-1 | Recall@3 | MRR | need 特异度 | hop |
|---|---:|---:|---:|---:|---:|
| step 0（现有交付件） | **49.14%** | 78.29% | 66.87% | 0.00 | 0.68 |
| step 1000 | **94.86%** | **100.00%** | 96.95% | 0.20 | 0.785 |
| step 12000 | 93.71% | 100.00% | 96.57% | 0.32 | 0.765 |
**Top-1 +45.72pp、Recall@3 到 100.00%**，而且是在训练时从未见过的属性措辞上 —— 这一个数字本身说明：
**"路由器缺的是真实形态训练数据"这个判断是对的。**
## 3. 但端到端**一动不动**
同一条 200 条评测集（属性不相交），同一份运行时，只换路由器：
| | 总体 | 可回答（175） | 未知拒答（25） | 已知被误拒 |
|---|---:|---:|---:|---:|
| 原版 NM2 | 64.00% | **72.57%** | 4.00% | 2.86% |
| 真实语料重训的路由器 | 63.50% | **72.57%** | 0.00% | 1.71% |
**Top-1 提升 45.72pp → 端到端可回答正确率变化 0.00pp。** 逐类别：
| 类别 | 原版 NM2 | 重训 | 差值 |
|---|---:|---:|---:|
| alias_paraphrase | 96.00% | **100.00%** | +4.00pp |
| long_fact | 64.00% | **72.00%** | +8.00pp |
| multi_entity | 96.00% | 96.00% | 0 |
| **multi_hop** | **48.00%** | **24.00%** | **−24.00pp** |
| near_miss | 88.00% | **100.00%** | +12.00pp |
| noise_context | 68.00% | **72.00%** | +4.00pp |
| unknown_attribute | 4.00% | 0.00% | −4.00pp |
| update_conflict | 48.00% | 44.00% | −4.00pp |
| **合计** | **64.00%** | **63.50%** | **−0.50pp** |
8 类里 4 类改善（+4~+12pp）、3 类退步（−4~−24pp），**净 −0.50pp**。每类只有 25 条、1 例 = 4.00pp，
所以 ±8pp 都在噪声量级；**唯一可能真实的是 multi_hop 的 −24.00pp（6 个用例）**。
## 4. 逐个被证伪的假设（这一步比结论更重要）
| 假设 | 测量结果 | 判定 |
|---|---|---|
| 记录没写进库 | 写入存活 **92.00%**、**目标事实存活 100.00%（35/35）** | **否** |
| 排序被先验压住 | 先验置 0 + 纯路由器排序：可回答 **77.14%** vs 现状 76.57%（1 个用例） | **否** |
| 先验权重要调 | 剂量曲线 1.00/0.50/0.25/0.00 = 65.00%/47.50%/42.50%/40.00%（模板域） | **否，调小更差** |
| 注入记录太多，模型被混淆 | top_k = 4/2/1 → 可回答 72.57% / 66.29% / **52.57%** | **否，减少更差** |
| 路由器分数没被用上 | 让路由器全权排序只动 1 个用例（见上） | **部分成立但不是主因** |
**结论：在当前架构下，路由器级排序能力不是端到端准确率的约束条件。** 这一点现在有两条独立证据链
（模板域的先验/混合实验 + 真实域的"Top-1 +45.72pp 而端到端 0.00pp"）。
## 5. 残差到底落在哪（可辩护的逐类分类，200 条）
| 类别 | 用例 | 正确 | 答了但不对 |
|---|---:|---:|---:|
| alias_paraphrase | 25 | **25（100.00%）** | 0 |
| near_miss | 25 | **25（100.00%）** | 0 |
| multi_entity | 25 | 24 | 1 |
| long_fact | 25 | 18 | 7 |
| noise_context | 25 | 18 | 6 |
| **update_conflict** | 25 | **11** | **13** |
| **multi_hop** | 25 | **6** | **19** |
| **unknown_attribute** | 25 | **0** | **25** |
| 合计 | 200 | 127（63.50%） | 71（35.50%）（拒答 2、空回复 0） |
**残差集中在这三处，而不是"全面不准"：**
1. **未知属性 0/25** —— 全部照答。覆盖机制是固定 24 属性闭集，在真实实体前缀属性上自动旁路（设计如此）。
2. **多跳 6/25（24.00%）** —— 链条没被跟上，而且比原版更差，说明重训在这类上过拟合了训练里的链条形态。
3. **更新冲突 11/25（44.00%）** —— 模型答出**被取代的旧值**。
第 3 点值得单独查：语料里我把"旧值/新值"写成**两个不同句式**（`frames[0]` / `frames[-1]`）。
若新值那句解析不出同一属性，`write()` 的同属性取代就不会触发，**旧记录仍是 active** → 两条都进证据。
这既是我的语料细节，**也对应真实产品缺陷：用户换一种说法更新时，旧值可能不被取代**。
## 6. 结论
1. **"真实形态训练数据"是路由器泛化的正确杠杆**：+45.72pp Top-1（未见过属性族），证据干净。
2. **但它不是端到端体验的杠杆**：端到端净变化 −0.50pp，8 类里有升有降。
3. **端到端剩下的 27.43% 失败集中在三处**（未知 25/25 错、多跳 19/25 错、更新冲突 13/25 错），
全部属于**运行时逻辑**（覆盖判断、多跳执行、冲突取代），不在路由器权重里。
4. 因此下一步该做的不是继续训路由器，而是：
* **冲突取代**：让"换一种说法的更新"也能取代旧记录（结构化键 + 语义匹配双条件），目标 更新冲突 ≥ 80%；
* **多跳执行**：核对 read 路径是否真的按 hop=2 串联两条记录，目标 多跳 ≥ 70%；
* **动态属性词表**：覆盖判断要能处理实体前缀属性，目标 未知拒答 ≥ 80%。
路线图建议：**多跳与冲突取代优先**（各约 20 条错例，且都是逻辑问题、不需要新数据）。
## 7. 产物
| 文件 | 内容 |
|---|---|
| `make_realistic_memory_data.py` | 训练/评测语料生成器（36 属性族、按族三分切分、含构造 bug 修复注释） |
| `data/realistic_v2/{train,eval}.jsonl` + `.manifest.json` | 3200 / 200 条，含 sha256 与**不相交证明**（`overlap: []`） |
| `H:\Memory\nm_cache\nm_realistic_v2\feature_cache` | 10,034 行特征库（流式编码，60 秒） |
| `checkpoints/router_realistic_v2_128/` | 真实语料重训的路由器（V2-128，drop-in 几何） |
| `rv2_{orig,new}_e2e.{json,md}` | 第 3 节对照（原版 vs 重训） |
| `rv2_topk{1,2}_e2e.{json,md}` | 第 4 节注入纪律实验 |
| `rv2_order_{current,routeronly}.{json,md}` | 第 4 节排序实验 |
| `write_survival.json` | 第 4 节写入存活诊断 |
| `diagnose_write_survival.py` | 写入存活诊断脚本 |
| `REALISTIC_CORPUS_FINDINGS.md` | 第一版语料（含我 3 个构造 bug 的复盘） |
ROUTER_TRAINING.md
+172
View File
@@ -0,0 +1,172 @@
# Natural Memory v2：正式路由器训练
这一阶段只训练 `MemoryRouterV2`，不更新 Qwen 主干。训练目标是让路由器在真实自然语言查询下学会：
- 从候选记忆中找出一个或多个证据；
- 遇到多跳问题时保留多个支持项；
- 碰到未知事实时拒绝读取；
- 区分同属性冲突、旧值、新值和无关噪声；
- 在个人事实、对话记忆、代码/文档和通用 QA 之间泛化。
训练分为两个文件级阶段：
```text
公开/本地数据
↓
prepare_memory_router_dataset.py
↓ 先写出并 hash
train.jsonl + eval.jsonl
↓
train_memory_router_large.py
↓ 一次性生成 Qwen 冻结特征
CPU feature bank + router training
```
## 当前环境
已检查 `Conda LLM`：
```text
Python: C:\Users\Administrator\miniconda3\envs\LLM\python.exe
PyTorch: 2.9.0+cu128
CUDA: available
GPU: NVIDIA GeForce RTX 5070
Transformers: 5.9.0
Datasets: 4.8.3
```
默认使用 `qwen3_5_4b_natural_memory_v2`，Qwen 只在特征预计算阶段工作；训练更新的只有一个小型 `MemoryRouterV2`。默认 GPU 上限是 9 GiB，为模型、系统和 KV 留出安全空间。
## 1. 生成并冻结数据
从仓库已有的训练/评估文件生成正式路由 episode：
```powershell
Set-Location H:\Memory
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe -m V2_dpskw.prepare_memory_router_dataset `
--output-dir data/router_training `
--candidate-count 32 `
--seed 20260907
```
输出：
```text
H:\Memory\V2_dpskw\data\router_training\train.jsonl
H:\Memory\V2_dpskw\data\router_training\eval.jsonl
H:\Memory\V2_dpskw\data\router_training\manifest.json
H:\Memory\V2_dpskw\data\router_training\eval.sha256
```
`eval.jsonl` 会在训练开始前生成，训练器启动时重新计算 SHA-256；如果被修改，可以通过 `--expected-eval-sha256` 让训练直接停止。
当前默认本地混合源包括：
| 领域 | 来源 | 用途 |
|---|---|---|
| 个人事实 | `benchmark_train/eval.jsonl` | 主题改写、实体区分、短事实召回 |
| 原生记忆 | `native_memory/train/eval.jsonl` | 写入—替换—读取链路 |
| 记忆策略 | `production_memory/train/eval.jsonl` | 写入、遗忘、临时信息和噪声 |
| 困难记忆策略 | `production_memory_hard_v2/train/eval.jsonl` | 冲突、引用噪声、旧值、新值 |
| 长上下文压力 | `mega_validation/smoke.jsonl` | 多跳、随机位置、未知拒答等 smoke eval |
## 2. 引入公开数据
脚本内置了可审计的公开数据配方：
- [HotpotQA](https://huggingface.co/datasets/hotpot_qa)：多跳证据选择；
- [CodeSearchNet](https://huggingface.co/datasets/code_search_net)：自然语言到代码/文档检索；
- [FEVER](https://huggingface.co/datasets/fever)：支持证据、冲突和无证据样本。
网络可用时执行：
```powershell
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe -m V2_dpskw.prepare_memory_router_dataset `
--output-dir V2_dpskw/data/router_training_public `
--include-public `
--hf-max-rows 20000 `
--candidate-count 32 `
--seed 20260907
```
如果 Hugging Face 下载失败，`manifest.json` 会写入失败原因，失败源贡献 0 行；不能把失败的在线源计入实验结果。也可以手动指定来源：
```text
--hf-source DATASET_ID|SPLIT|CONFIG(optional)|SPLIT_KIND(optional)
```
例如：
```powershell
--hf-source hotpot_qa|train|distractor|train
--hf-source hotpot_qa|validation|distractor|eval
```
真实用户对话必须先获得同意并去除姓名、地址、密钥、账号等 PII，再通过 `--train-source`/`--eval-source` 加入。脚本不会把用户对话偷偷上传。
## 3. 监督训练
第一次运行会加载 4B 模型，给去重后的 query/candidate 文本生成冻结语义特征，保存到 CPU feature bank；之后重新训练会复用它：
```powershell
Set-Location H:\Memory
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe -m V2_dpskw.train_memory_router_large `
--train-file V2_dpskw/data/router_training/train.jsonl `
--eval-file V2_dpskw/data/router_training/eval.jsonl `
--model-path V2_dpskw/qwen3_5_4b_natural_memory_v2 `
--output-dir V2_dpskw/checkpoints/natural_memory_v2_router_large `
--feature-cache-dir V2_dpskw/checkpoints/natural_memory_v2_router_large/feature_cache `
--gpu-memory-gb 9 `
--encode-batch-size 1 `
--steps 10000 `
--batch-size 64 `
--eval-interval 500 `
--overwrite-metrics
```
控制台和 `metrics.jsonl` 都会记录：
- 每一个 optimizer step 的 `loss`、四个子损失、学习率和梯度范数；
- step 0 初始 eval；
- step 500、1000、1500……的完整 eval；
- 最终 `memory_router_v2.pt`，可直接交给现有的 `build_natural_memory_v2_package.py`。
训练不会输出每 token 的 Qwen 生成，因而不会把推理生成路径混入路由器质量指标。
如果中途停止，使用保存的 checkpoint 恢复；`--steps` 是恢复后的最终全局 step，不是额外步数：
```powershell
& C:\Users\Administrator\miniconda3\envs\LLM\python.exe -m V2_dpskw.train_memory_router_large `
--train-file V2_dpskw/data/router_training/train.jsonl `
--eval-file V2_dpskw/data/router_training/eval.jsonl `
--model-path V2_dpskw/qwen3_5_4b_natural_memory_v2 `
--output-dir V2_dpskw/checkpoints/natural_memory_v2_router_large `
--feature-cache-dir V2_dpskw/checkpoints/natural_memory_v2_router_large/feature_cache `
--resume V2_dpskw/checkpoints/natural_memory_v2_router_large/router_step_00005000.pt `
--steps 10000 `
--eval-interval 500
```
恢复时不要使用 `--overwrite-metrics`，这样历史 loss/eval 会继续追加到同一个 `metrics.jsonl`。
## 4. 评估门槛
训练期间重点看 `eval` 行：
| 指标 | 含义 |
|---|---|
| `route_top1` | 有证据时，Top-1 是否命中支持记忆 |
| `route_recall_at3` | 多跳/多事实时，Top-3 是否包含支持记忆 |
| `route_mrr` | 支持记忆在排序中的平均倒数排名 |
| `need_recall` | 需要记忆时是否愿意读取 |
| `need_specificity` | 不需要/未知时是否能拒绝读取 |
| `need_f1` | 读取和拒绝的综合平衡 |
| `hop_accuracy` | 多跳控制预测是否正确 |
路由器指标不能代替端到端回答率。训练完成后必须重新运行现有的 dirty-corpus、strong-RAG 和 Qwen3.5 4B 对照；最终门槛仍然是端到端正确率、未知拒答、延迟和显存共同达标。
## 5. 重要边界
当前公开数据适配器是通用 schema 适配器，不会把所有数据集都自动变成完美标注。每个公开源的实际行数、错误信息和哈希都在 `manifest.json`。如果某个源的支持证据字段无法解析，它会少贡献 episode，而不是制造伪标签。
路由器训练也不会直接解决“召回正确但 Qwen 没有把证据融合进答案”的全部问题；那是后续证据融合/生成控制实验，必须单独报告。
V6_FINAL_REPORT.md
+74
View File
@@ -0,0 +1,74 @@
# Natural Memory 路由器 v6 —— 最终报告
本文件是 v6 阶段的**单一汇总证据**。所有比率均为百分比；所有结论都指向可复现的产物文件。
## 1. 做了什么
在分叉 `H:\Memory\V2_dpskw` 内，用**流式加载 + 多线程/批量化特征编码**在修正后的 v6 全量数据集上训练 512 维路由器，并与原版 NM2 路由器在同一冻结评测集上逐轴对比。
| 环节 | 实现 | 实测 |
|---|---|---|
| 流式加载 | `stream_feature_bank.py`：只保留 `sha1→行号` 字典，文本落盘；训练器 `train_router_v5.py` 逐行流式建索引 | scan 212 万条唯一文本 **15.1 秒**；训练不解析 episode 进内存 |
| 多线程编码 | 8 线程 tokenizer 池 + 按精确 token 长度分组 + token 预算限批 | tokenize **41.8–46.5 秒**（约 5.7 万 texts/s）；encode 约 160–180 texts/s |
| 特征库 | `features.f16.npy` mmap，10.86 GB（放 D: NVMe） | 打开 1.0 秒；训练按需读页 |
| 断点续跑 | `--resume` + `progress.json` + 编码方法指纹 | 两次中断共用已编码的 75 万行 |
| 完整性验证 | `verify_feature_bank.py` 全量扫描 | 2,124,552 行、**零值行 = 0** |
数据：`data/router_training_v6`，train **87,155** / eval **21,920**（10 类别 × 2,000 + policy 1,920），`group_overlap = 0`。
## 2. 评测协议
- 冻结 eval 21,920 条，`eval.sha256` 在训练前生成并被训练器强制校验；
- 所有路由器在**同一冻结特征库**上评分（编码保真度：与旧缓存共有文本余弦 **0.999983**）；
- 读取/拒答阈值以 0.50 为主，另给 0.30–0.80 曲线；
- 速度轴以 `bench_router_latency.py` 的**交替轮流测量**（5 轮，模型轮转抵消漂移）为准，实测波动带 **2.8–5.3%**，故速度轴用 3% 相对容差，质量轴用严格容差；
- 逐轴判定：`verdict_router_v6.py`，对每个候选分别给出「是否全方位超越」。
## 3. 结果：同存储预算下全方位超越
| 路由器 | 参数 | 地址字节 | Top-1 | Recall@3 | MRR | 多跳证据全中 | hop | 未知拒答率 | 已知被误拒率 | 延迟(中位) | batch256 QPS | 判定 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
| V2-128 生产(v3) | 2,037,774 | 512 | 41.12% | 46.73% | 47.69% | 43.43% | 73.23% | 0.00% | 0.18% | 1.1558 ms | 199,554 | 基线 |
| **V2-128 v6 best** | 2,037,774 | 512 | **94.62%** | **97.81%** | **96.49%** | **97.75%** | **100.00%** | **100.00%** | **0.00%** | 1.1679 ms | 196,829 | **22/22 → 是** |
| **V2-128 v6 final** | 2,037,774 | 512 | 94.37% | 96.40% | 95.97% | 95.95% | 100.00% | 100.00% | 0.00% | 1.1653 ms | 198,026 | **22/22 → 是** |
| XL-128 v6 best | 6,714,639 | 512 | 95.66% | 98.75% | 97.37% | 98.70% | 99.95% | 99.59% | 0.00% | 1.5783 ms | 141,501 | 18/22 |
| V2-512 v6 best | 4,741,902 | 2048 | 95.05% | 98.84% | 97.12% | 98.64% | 100.00% | 99.81% | 0.00% | 1.1623 ms | 154,085 | 20/22 |
| XL-512 v6 best | 7,898,127 | 2048 | **95.78%** | 97.59% | 97.24% | 96.86% | 100.00% | 100.00% | 0.00% | 1.5855 ms | 94,647 | 17/22 |
逐类别拆解、family 拆解、阈值曲线见 `router_scorecard_v6_128.md`（另附 `router_scorecard_v6.md` 为 512 维版本）。
**相对已交付生产路由器**：Top-1 **+53.49pp**、未知拒答率 **+100.00pp**、已知问题被误拒率 **−0.18pp**、hop **+26.77pp**、多跳证据全中（仅多正例）**37.15% → 99.00%**；参数与地址字节完全相同，延迟/吞吐差异 ≤1.4%（在 2.8–5.3% 的测量噪声带内）。
## 4. 两个必须如实说明的结论
1. **512 维配置无法在「地址字节」轴上不劣于 128 维基线**：2048 vs 512 字节是几何决定的 4 倍差异，不是训练能解决的；把它算作失败属于不同存储预算的错位比较。此外 512 维 V2 还真实输一项 batch-256 吞吐（−22.8%）。因此目标里「512 维 + 全部轴不劣」在字面上不可达，在**同预算**下才可达且已达成。
2. **新架构 XL 没有赢**：准确率最高（Top-1 95.78%）但吞吐真实低 26–52%（3.3–3.9 倍参数）。本次提升来自**数据与标签**（拒答样本 106→4,106、多跳 2 正例/hop=2、12 倍数据量、10 类别均衡评测），**不是来自更多维度与参数**。该假设被数据否定。
## 5. 部署就绪：直接替换已验证
`check_router_swap.py` 结论 `DROP-IN REPLACEMENT OK`：
- 与生产包第 5 分片 `dynamic_memory.memory_router_v2.*` 的 **16 个键、形状零差异**（缺 0 / 余 0 / 不匹配 0），替换只是重命名；
- 可加载进运行时同一 `MemoryRouterV2` 构造（hidden 2560 / dim 128 / heads 8 / hops 3）；
- 能驱动 `PagedMemoryBankV2` 正常写入与路由（`key_dim = 128`，返回记录数 3）。
## 6. 过程中发现并修复的缺陷（每条都有验证）
| 缺陷 | 影响 | 验证方式 |
|---|---|---|
| mega 适配器把 `metadata.answerable=false` 当成可回答 | 2000 条拒答样本全被错标，拒答轴与多跳轴**无法测量** | 重建后 eval 未知样本 106 → **4,106**；审计表逐类核对 |
| multi_hop 只标最终事实、hop 用 `len(positive_ids)` | 多跳证据链缺失、hop 标签全为 1 | 沿实体→值链扩展后 2 正例 / hop=2 |
| `_source_candidates` 每条 episode 复制 265.6 万条候选池 | 8 万条 episode 需 **46.8 小时** | 修复后 mega 单条 2105ms → 0.59ms（**3568×**），A/B **sha256 逐字节一致** |
| 续跑守卫 `if start < done: continue` 整组跳过 | 49.4 万行保持零值，且写出「完整」manifest，训练已在零值上开跑 | 全量扫描定位缺失区间 → 补齐 → 零值行 0 → 新增完整性标志/训练器拒收/抽样检查 |
| 补丁模式误用 `w+` 打开特征库 | 若成功会清零已编码的 183 万行 | `OSError` 中止；改为仅全新运行用 `w+` |
| 评分卡 `Recall@k` 用命中**个数**而非比例 | 输出 105%、109% 的"召回率" | 修正为「命中/应有」，全部 ≤100% |
| `logits_to_keep=1` 看似省算力 | 实际**慢 1.55×**（非连续视图掉进 bnb 慢路径） | 交替重复 7 轮测量；默认关闭并写入 `--help` |
| 用相同文本测吞吐 | 虚高 1.6×（283 vs 真实 175 texts/s） | 用 192 条不同真实文本重测 |
## 7. 产物清单
- 数据：`data/router_training_v6/{train,eval,manifest,eval.sha256}`、`router_dataset_audit_v6.json`
- 特征库：`H:\Memory\nm_cache\nm_router_v6\feature_cache`（+ `feature_bank_verification.json`）
- 路由器：`checkpoints/router_v6_v2_128`、`router_v6_xl128`、`router_v6_v2_512`、`router_v6_xl512`
- 证据：`router_scorecard_v6_128.md`、`router_verdict_v6_128.md`、`router_latency_bench.json`、`router_swap_check.json`、`router_scorecard_v6.md`、`router_verdict_v6.md`
- 工具：`stream_feature_bank.py`、`train_router_v5.py`、`eval_router_v5.py`、`verify_feature_bank.py`、`bench_router_latency.py`、`check_router_swap.py`、`audit_router_dataset.py`、`verdict_router_v6.py`、`router_xl.py`
WRITE_PATH_FIX.md
+101
View File
@@ -0,0 +1,101 @@
# 写入路径修复：20 条不同属性事实不再互相销毁
## 症状
一次写入 20 条**不同属性**的事实后，库里有 20 条记录但只剩 **12 条 active**，**8 条在查询前就已被
retract**；16 个关键零重叠用例中有 **8 个的目标记录已不存在**，因此端到端正确率存在 **50% 的硬上限**
—— 任何排序器都救不回一条已被删除的记录。
## 根因（栈追踪确证，非推断）
16 次 retract 的调用栈完全一致：
```
eval_end_to_end_memory.py:106 write_fact
-> qwen_integration.py:3807 forward
-> qwen_integration.py:2195 _write_text_memory <-- retract 调用点
-> memory_os_v2.py:2578 MemoryOSV2.retract_record
-> memory_os_v2.py:1959 PagedMemoryBankV2.retract (status = retracted)
```
`status_transition_summary == {"active->retracted": 16}`，无 `superseded`、无 `quarantined`，
20 次 `bank.write` 全部返回 `inserted` —— 写入路径的**唯一**销毁机制就是 2195 行的 retract。
授权条件 `confirmed_update` 在 20 次写入中为真 **5 次**，且**全部只走"学习重排器
`learned_best >= 0.95`"这一条**（`qwen_integration.py:2140-2142`）：
| 写入 | 事实 | learned_best | 结果 |
|---|---|---:|---|
| 2 | 出生城市 | 0.998959 | confirmed_update=True |
| 3 | 办公城市 | 0.992638 | confirmed_update=True |
| 9 | 工位楼层 | 0.964987 | confirmed_update=True |
| 12 | 手机尾号 | 0.998516 | confirmed_update=True |
| 15 | 办公楼层 | 0.997487 | confirmed_update=True |
随后 2187-2197 的循环退掉 8 条：3 条走 `record.slot_index == slot`，5 条走
`score >= 0.95 and shared >= 2`；2196 行的 `break`（仅当 `score < 0.98` 才停）让写 #12 一次
级联退掉 3 条、写 #15 退掉 2 条。被退掉的正是写 1/2/3/4/9/10/11/14 —— 16 条关键事实中的 8 条。
三条辅助事实：
* `exact_slots.numel() > 0`（token 完全相同）**0/20 次**都没命中 —— 精确重复路径与此无关；
* `shared >= 2` 毫无区分力：19 个候选的 `shared` 全落在 {2,3,5,6,7}（每条事实都含"我的/是"模板词元）；
* 词面兜底分支（`>= 0.30`）在重排器就绪时是**死代码**（`learned_best is not None`），
且更宽松：真跑起来 17/20 次会通过。
**根因**：retire 一条已存在记录的授权**完全来自学习打分加一个固定 0.95 阈值**，没有任何
"新文本与在位记录共享 `(entity, attribute)`"的结构性校验。而打包进来的 `text_retriever` 在真实
e2e 事实集上对**不相关**属性过度自信：190 个不相关对中 **10 个 ≥ 0.95**，最高 **0.9985**
（如"常住城市 CITY-A1B2C3" vs "出生城市 CITY-D4E5F6" = 0.9985）。
> 说明：这条结论修正了我早先的一次错误判断。我曾用自己合成的 `VAL-A%07d` 事实集测得
> "0/190 对达 0.95"并据此宣布"重排器过度自信假设被推翻"——那是**数据集不同**造成的：
> 同一测量在 e2e 协议真正写入的事实集上得到 10/190。因此调阈值不可行：假阳性高达 0.9989，
> 没有任何标量切点能分开"不相关属性"与"真更新"。
## 修复（已应用，两处）
1. `qwen_integration.py` `_write_text_memory`：`confirmed_update` 的两条**打分类**分支改为
**结构优先** —— 仅当候选文本的 `entity::attribute` 已经在库的 `active_by_conflict` 账本中
存在（即确实是同一条事实的更新）才允许由打分授权；`exact_slots`（token 完全相同）这条
结构安全的分支保持不变，以保留"重复写入幂等"的既有语义。
2. 2187-2197 的 retract 循环增加守卫：`if record.conflict_key() != candidate_conflict_key: continue`，
循环只能退掉**描述同一属性**的记录，不再因"共享热槽"或"词面重叠高"而销毁无关事实。
语义不受损：真·同属性更新本来就已经由 `PagedMemoryBankV2.write` 通过 `active_by_conflict`
做版本化（旧版本保留为 `superseded`，`version+1`），修复只是移除了那条**额外的、无监督的销毁路径**。
## 四级验证（全部通过）
| 级 | 检验 | 修复前 | 修复后 |
|---|---|---|---|
| ① | `probe_write_retraction_trace.py`：`status_transition_summary` | `{"active->retracted": 16}` | **`{}`** |
| ① | 同上：`retraction_calls` / `status_counts` | 16 次 / `{"retracted": 8, "active": 12}` | **`[]` / `{"active": 20}`** |
| ② | 正对照：同属性改值 | —— | 旧值 `superseded` + 新值 `active`，其余 19 属性不受影响，0 retract |
| ③ | 重排器缺席对照（更宽松的词面分支） | 会踩死代码分支 | 20 条全 active、0 retract |
| ④ | `probe_record_selection.py --gate-scan`：`target_record_active` | 8 / 16 | **16 / 16** |
| ④ | 同上：`target_record_retracted_or_absent` | 8 | **0** |
| ④ | 同上：`active_records_min/max` | 12 / 12 | **20 / 20** |
| ④ | 同上：probe 口径回答正确率 | 25.00%（4/16） | **43.75%**（7/16） |
`verify_write_path_fix.py` 的三阶段与 8 项检查全部通过（`write_path_fix_verification.json`）。
## 端到端效果（官方 harness，16 个零重叠用例）
| 路由器 | 修复前 | 修复后 | 答成别的属性 |
|---|---:|---:|---|
| deployed（原版 NM2） | 25.00% | **50.00%**（+25.00pp） | 18.75% → 12.50% |
| V2-128-v6 | 37.50% | **68.75%**（+31.25pp） | 25.00% → 18.75% |
| REPLAY-128 | 37.50% | **68.75%**（+31.25pp） | 25.00% → 18.75% |
`router_critical_e2e_after_write_fix.json` / `.md`。
## 注意事项与仍未解决的部分
* **路由器评分卡不受此修复影响**：评分卡直接读冻结特征库、不走运行时的写入/读取路径，
所以 `router_scorecard_final.md` 的 22 轴结论无需重跑，也不因本次改动而改变。
* 单元测试 52 项仍全部通过（`python -m unittest discover -s tests -t .`）。
* **仍未解决**（本次未触碰）：这 16 个用例里 **93.75% 的读取仍走旧版 16 槽路径**
（`legacy_prefix_used_pct: 93.75`）而非 V2 库记录，地址命中 0/16；记录级排序仍由打包的
`text_retriever` 决定而非路由器。这两项是端到端正确率的下一个瓶颈。
* 220 用例的 `eval_end_to_end_memory.py` 全量套件本次未重跑。
ZERO_OVERLAP_FINDINGS.md
+185
View File
@@ -0,0 +1,185 @@
# 零重叠改写成语（zero-overlap paraphrase）阶段报告
本阶段针对 `E2E_FINDINGS.md` 定位到的**唯一**泛化缺口：查询与事实之间**没有任何独特字重叠**时的语义匹配。
所有速率均为百分比。所有数字均来自磁盘上保存的 scorecard / 评测 JSON，不引用滚动日志。
逐轴对照表见 `zero_overlap_phase.md`（由 `report_zero_overlap_phase.py` 从 scorecard JSON 渲染）。
---
## 1. 结果摘要
新增了一个**可验证的**零重叠改写数据集、一个**合并特征库**，以及一个重放微调后的路由器
`REPLAY-128`（2,037,774 参数，128 维，每条记录地址 512 字节 —— 与现网路由器几何完全一致）。
| 评测 | 指标 | v6 最终版 | REPLAY-128 | 变化 |
|---|---|---|---|---|
| 零重叠（300 条，24 个**未见过**问法） | Top-1 正确率 | 18.40% | **59.60%** | **+41.20pp** |
| 零重叠 | Recall@3 | 36.00% | **83.20%** | **+47.20pp** |
| 零重叠 | MRR | 35.07% | **73.06%** | **+37.99pp** |
| v6 全集（21,920 条，全部 22 轴） | Top-1 正确率 | 94.37% | 94.14% | −0.23pp |
| v6 全集 | Recall@3 | 96.40% | 96.48% | +0.08pp |
| v6 全集 | 多跳证据全中(Top-3) | 95.95% | 96.22% | +0.27pp |
| v6 全集 | hop 正确率 | 100.00% | 100.00% | 0 |
| v6 全集 | 未知拒答率 | 100.00% | 100.00% | 0 |
| v6 全集 | 已知问题被误拒率 | 0.00% | 0.00% | 0 |
零重叠评测集随机猜测基线为 4.17%（24 个同形候选），因此 59.60% 约为**随机水平的 14 倍**。
v6 全集的全部策略轴**逐位不变**（见第 3 节的冻结机制），检索轴变化在 ±0.27pp 以内。
部署契约：`check_router_swap.py` 判定 **DROP-IN REPLACEMENT OK** —— 16/16 键匹配、无缺失/多余、
无形状不符、分数有限，且实测可驱动 `PagedMemoryBankV2` 路由（`bank_routed_records: 3`）。
延迟（`bench_router_latency.py`，7 轮交错取中位数）：单查询 1.2042 ms → 1.2087 ms（**+0.37%**），
batch=256 QPS 190,103 → 192,600（**+1.3%**）。batch=64 QPS 读数低 9.2%，但该轮自身离散度为
17.4%，且同架构同参数同 FLOPs 下 batch=256 反而更快 —— 方向不一致，判为测量噪声而非回归。
---
## 2. 数据集：属性由构造保证，而非相信手写意图
`make_zero_overlap_paraphrase_data.py` 生成 24 个属性 × 3 个改写问法：
* 每条查询与其**自身**目标事实的独特字重叠必须为 **0**（属性字符、停用字除外）；
* 前 2 个改写用于训练（48 个问法），第 3 个**留出**用于评测（24 个问法）——两个划分的
**查询字符串完全不相交**，因此评测衡量的是"换一种说法"的泛化，而不是记住训练字符串；
* 候选是 24 条**同句式、不同属性**的事实，值为随机码，只出现在一条事实里（可检测证据混用）；
* 15% 为不可回答 episode（改写指向候选中不存在的属性）。
关键设计判断：改写与其他**无关**属性的字符重叠被**刻意保留**（生成器统计 54.53% 的 episode 至少
含一个这类"陷阱"候选）。因为命中错误候选会得到**错误答案**，这只会让任务更难，不会提供捷径；
真正必须为 0 的是与**自身**目标的重叠。最初把二者混为一谈时，24 个属性中有 7 个因为
"起床时间"的"时"、"办公城市"的"公"等高频字而被判为"无可用的改写"。
`verify_zero_overlap_dataset.py` 用**独立重写**的停用字集合与重叠算法复核（不 import 生成器）：
* 结构问题 **0**，查询与自身目标重叠 **0**，两个划分共享查询字符串 **0**；
* 每条的答案码只出现在唯一一条候选中，候选属性标签唯一；
* `answerable` / `need_memory` / `hop` 与正例索引一致。
（生成器报的陷阱比例 54.53% 与校验器报的 44.00%/50.00% 不同，因为两者用不同的停用字集合
定义"独特字"；这是定义差异，不是矛盾。）
---
## 3. 两次微调的对照：朴素微调会造成灾难性遗忘
先做朴素微调（只用零重叠数据，全部损失项默认权重，从 v6 最终版权重出发，6000 步）：
* 零重叠 Top-1 18.40% → **66.80%**，但 v6 全集同时退化：
`random_position` MRR 99.02% → 93.37%、多跳证据全中 100.00% → 95.65%、
hop 100.00% → 87.10%、**已知问题被误拒率 0.00% → 13.85%**、`unknown` Top-1 58.99% → 47.52%。
* 训练日志还显示策略门发生了"用误拒换拒答"的交换：未知拒答率升到 40.00%，代价是
**38.00% 的可回答问题被误拒**。
这就是单指标视角会误判为胜利、而多轴记分卡会否掉的典型情形。
修正方案由两点组成：
1. **冻结策略头**（`--need-loss-weight 0 --hop-loss-weight 0`）。在零重叠集合上，`need_memory`
门只看查询、看不到候选，而"不可回答"的改写与可回答的改写句式完全一样，因此该门在此数据上
**原理上不可学**；强行训练只会破坏 v6 学到的门。实测证明冻结生效：合并评测的
`need_specificity` 在每一次 eval 中都是 `0.9879692011549567`，逐位不变。
2. **重放**（把零重叠数据并入 v6 数据一起训练），避免排序通路被新分布带偏。
---
## 4. 合并语料：复制已验证的行，而不是重新编码
`build_replay_corpus.py` 把 v6 库（2,124,552 行）与零重叠库（36,072 行）合并为
2,160,624 行。**没有重新编码** v6 特征 —— 那既耗时数小时，又会让历史测量所依赖的参考特征
发生无声漂移。合并的强制不变量：
* 每个源库的每一行都恰好被复制一次，或因文本完全相同而跳过（本次重复 **0** 行）；
* `index` 条目数 == 库行数（训练器会拒绝不一致的库）；
* 合并后 train/eval 引用的**每一个文本**都能解析到一行（缺失 **0**）；
* **2000 行抽查逐字节相等**；全库扫描全零行 **0**；
* manifest 携带**合并后**数据文件的 sha256（训练器会据此校验冻结输入）。
合并语料：train 88,355 episode / eval 22,220 episode；
`data/router_replay_v7/`、`H:\Memory\nm_cache\nm_replay_v7\feature_cache`。
---
## 5. 关键负面结论：路由器排序不影响端到端答案
`REPLAY-128` 与 `V2-128-v6` 在 `eval_router_critical_e2e.py`（16 个零重叠用例）上的结果**逐位相同**：
| top_k_records | 路由器 | 回答正确率 | 答成别的属性 | 平均选中记录 |
|---|---:|---:|---:|---:|
| 默认 | V2-128-v6 / REPLAY-128 | 37.50% / 37.50% | 25.00% / 25.00% | 5.88 / 5.88 |
| 1 | V2-128-v6 / REPLAY-128 | 12.50% / 12.50% | 43.75% / 43.75% | 0.94 / 0.94 |
| 3 | V2-128-v6 / REPLAY-128 | 31.25% / 31.25% | 25.00% / 31.25% | 2.44 / 2.44 |
路由器级 Top-1 提升了 41.20pp，端到端却**一个用例都没变**。因此做了一个判决性实验：保留
`need_memory` / `hop_controller` / `head_gate` 全部原权重，只把排序通路
（`query_projection`、`key_projection`、`pair_scorer`）替换为同形状**随机权重**
（`checkpoints/router_v6_v2_128/ranking_scrambled_probe.pt`，非有限值 0 个）。
结果在默认 Top-K 与 Top-K=1 下**都完全不变**（37.50% / 12.50%，选中记录数相同）。
机制（已逐项实测，含两次被自己数据推翻的中间假设）：
* `memory_os_v2.py:1671` — 带 `semantic_key`（张量）的记录，其
`_score_candidates()`（即 `MemoryRouterV2.projected_scores`）分数会被
`self.record_scorer` **逐位置覆盖**；实测残差与 text_retriever 匹配 18/18，与
`memory_router_v2` 匹配 0/18。
* `qwen_integration.py:1359` `_score_semantic_memory_records` — 该回调走
`self.text_retriever`。**注意：最初以为"部署目录没有 `text_retriever.pt` 所以走余弦兜底"
是错的** —— 重排器被烘焙进合并包（safetensors 内有 6 个
`dynamic_memory.text_retriever.*` 张量，1,573,377 参数），
`_text_retriever_ready` 实测为 **True**，余弦兜底分支根本不执行。
* 路由器在此配置下**唯一实际生效的杠杆是 `need_memory` 门**：部署权重下门开启率
**68.75%**（11/16），而三份随机初始化的路由器只有 **37.5%**（6/16）。判决性探针只随机化了
排序通路、**保留了门控头**，所以"端到端零变化"是由构造保证的，而不是巧合。
* 这 16 个用例里 **93.75%（15/16）实际注入的是旧版 16 槽路径**的证据
（`legacy_slot` + `text_memory_top_k=2`），而不是 V2 库记录；地址命中 **0/16**，
直接词面命中 6/16。`skip_semantic_expansion` 在这 6 例里直接跳过整个语义 hop 循环。
* 更严重的是：一次写入 20 条不同属性的事实后，**只剩 12 条 active，8 条在查询前就已被
retract**（16 个用例的 8 个里，目标记录本身就在被删的那批里）。**任何排序器都救不回一条
已经不存在的记录**，因此端到端正确率存在 50% 的硬上限。
结论分两层，不要混为一谈：
1. 路由器记分卡上 22/22 的统治性和本次 +41.20pp 的零重叠提升，都是路由器工件**真实**的增益；
2. 但在这套运行时里，记录级顺序由打包的 `text_retriever` 决定，路由器分数被覆盖，且相当比例的
读取根本不走 V2 库、目标记录还会被写入路径删除。所以路由器增益**不会**传递到最终答案。
要让增益落地，需要的不是继续加大路由器，而是：
1. 修写入路径：不要让 20 条不同属性的事实互相淘汰（当前最大瓶颈，直接决定 50% 上限）；
2. 让 V2 库记录真正参与读取：目前 93.75% 的用例走旧版 16 槽路径；
3. 训练/替换 `text_retriever`（记录级重排器）—— 零重叠数据集可直接复用为它的训练/评测集；
4. 让路由器分数对 `semantic_key` 记录也参与排序（改动 `memory_os_v2.py:1671` 的覆盖逻辑）。
（曾据此改过一版，实测在部署配置下**不生效**，已回退并留注释。）
---
## 6. 产物清单
| 文件 | 作用 |
|---|---|
| `make_zero_overlap_paraphrase_data.py` | 生成零重叠数据集（属性由构造保证，划分字符串不相交） |
| `verify_zero_overlap_dataset.py` | 独立复核数据集（不共享生成器代码） |
| `build_replay_corpus.py` | 合并两个数据集与两个特征库，含逐字节抽查 |
| `report_zero_overlap_phase.py` | 由 scorecard JSON 渲染对照表（`zero_overlap_phase.md`） |
| `data/zero_overlap/` | train 1,200 / eval 300（含 `manifest.json`、`zero_overlap_verification.json`） |
| `data/router_replay_v7/` | 合并语料 train 88,355 / eval 22,220 |
| `H:\Memory\nm_cache\nm_replay_v7\feature_cache` | 合并特征库 2,160,624 行 |
| `checkpoints/router_replay_v7_v2_128/` | `REPLAY-128` 最终版权重（可 drop-in） |
| `compare_record_rankers.py` | 在同一批冻结特征上对比"余弦 / 各路由器"的排序能力 |
| `measure_write_path_updates.py` | 量写入路径的更新判定（证明重排器打分不是 retract 的原因） |
| `probe_record_selection.py` | 逐记录拆解打分项、门控扫描、目标记录存活状态 |
| `migrate_bank_paths.py` | 把特征库路径从 D: 迁移到 `H:\Memory\nm_cache`（含校验） |
| `zero_overlap_phase.md` | 本阶段的逐轴对照表 |
特征库现存于 `H:\Memory\nm_cache\`（`nm_router_v6` / `nm_router_v5` / `nm_zero_overlap` /
`nm_replay_v7` / `nm_probe_bf16`，共约 35.9 GB；D 盘已不再保存项目数据）。迁移后用
`verify_feature_bank.py` 全盘复扫（2,124,552 行、全零行 0、`complete=true`）并在新路径下
复算了排序器对比（59.60 / 30.40 逐位一致）。
历史运行记录（`checkpoints/*/metrics.jsonl`、`*/training_stdout.log`、
`*/router_v5_training.json`、`feature_bank_verification.json`）**刻意保留原始 D: 路径**，
因为那是当时真实的运行事实，改写它们等于篡改记录。
对照用的负面结果与判决性探针：`checkpoints/router_zov_v2_128/`（朴素微调）、
`checkpoints/router_v6_v2_128/ranking_scrambled_probe.pt`（随机排序探针）、
`forget_check_zov.json`、`router_critical_e2e_scramble*.json`。
__init__.py
+15
View File
@@ -0,0 +1,15 @@
"""Dynamic Memory Lab: small, reproducible architecture experiments."""
from .model import DynamicMemoryConfig, DynamicMemoryLM
from .qwen_integration import QwenDynamicMemoryModel, QwenMemoryConfig, load_qwen_base, load_qwen_dynamic
from .tiered_memory_store_v2 import TieredMemoryStoreV2
__all__ = [
"DynamicMemoryConfig",
"DynamicMemoryLM",
"QwenDynamicMemoryModel",
"QwenMemoryConfig",
"load_qwen_base",
"load_qwen_dynamic",
"TieredMemoryStoreV2",
]
_regress_scorecard.json
+2456
View File
File diff suppressed because it is too large Load Diff
_regress_scorecard.md
+101
View File
@@ -0,0 +1,101 @@
| metric | V2-512 best(step1000) | V2-512 final(step100k) | XL-512 best |
|---|---|---|---|
| 参数 | 4,741,902 | 4,741,902 | 7,898,127 |
| router_dim | 512 | 512 | 512 |
| 每条记录地址字节 | 2048 | 2048 | 2048 |
| **Top-1 正确率** | 67.12% | 60.63% | 60.85% |
| Recall@1 | 67.12% | 60.63% | 60.85% |
| Recall@3 | 88.77% | 85.01% | 92.97% |
| Recall@5 | 94.93% | 90.13% | 97.33% |
| MRR | 79.08% | 73.26% | 75.98% |
| nDCG@3 | 80.00% | 74.57% | 79.28% |
| 多跳证据全中(Top-3) | 88.66% | 84.90% | 92.86% |
| hop 正确率 | 97.84% | 99.69% | 99.38% |
| hop 欠预测率 | 0.33% | 0.33% | 0.33% |
| need F1 | 98.92% | 99.89% | 99.62% |
| need 召回 | 99.78% | 99.78% | 99.78% |
| **未知拒答率** | 66.04% | 100.00% | 90.57% |
| **已知问题被误拒率** | 0.22% | 0.22% | 0.22% |
| 未知问题被误读率 | 33.96% | 0.00% | 9.43% |
| 仲裁准确率 | 97.94% | 99.79% | 99.28% |
| 已知问题 need 概率均值 | 98.47% | 99.78% | 99.62% |
| 未知问题 need 概率均值 | 25.36% | 1.52% | 13.50% |
| 平均分数余量 | 0.5880 | 3.8837 | 6.7097 |
| 单查询延迟 ms(GPU) | - | - | - |
| 路由 QPS(GPU) | - | - | - |
### family = `benchmark_qa`
| metric | V2-512 best(step1000) | V2-512 final(step100k) | XL-512 best |
|---|---|---|---|
| episodes | 64 | 64 | 64 |
| Top-1 正确率 | - | - | - |
| Recall@3 | - | - | - |
| MRR | - | - | - |
| hop 正确率 | 100.00% | 100.00% | 100.00% |
| 未知拒答率 | 100.00% | 100.00% | 100.00% |
| 已知问题被误拒率 | - | - | - |
| 未知问题被误读率 | 0.00% | 0.00% | 0.00% |
| 平均分数余量 | - | - | - |
### family = `mega_validation`
| metric | V2-512 best(step1000) | V2-512 final(step100k) | XL-512 best |
|---|---|---|---|
| episodes | 20 | 20 | 20 |
| Top-1 正确率 | 40.00% | 35.00% | 40.00% |
| Recall@3 | 80.00% | 80.00% | 85.00% |
| MRR | 62.26% | 55.24% | 60.83% |
| hop 正确率 | 70.00% | 70.00% | 70.00% |
| 未知拒答率 | - | - | - |
| 已知问题被误拒率 | 20.00% | 20.00% | 20.00% |
| 未知问题被误读率 | - | - | - |
| 平均分数余量 | -0.2963 | -12.9083 | -0.6136 |
### family = `memory_policy`
| metric | V2-512 best(step1000) | V2-512 final(step100k) | XL-512 best |
|---|---|---|---|
| episodes | 1728 | 1728 | 1728 |
| Top-1 正确率 | 69.89% | 62.57% | 62.39% |
| Recall@3 | 90.04% | 85.94% | 94.20% |
| MRR | 81.00% | 74.59% | 77.16% |
| hop 正确率 | 98.96% | 100.00% | 99.83% |
| 未知拒答率 | 14.29% | 100.00% | 76.19% |
| 已知问题被误拒率 | 0.00% | 0.00% | 0.00% |
| 未知问题被误读率 | 85.71% | 0.00% | 23.81% |
| 平均分数余量 | 0.6516 | 5.3929 | 7.2586 |
### family = `native_memory`
| metric | V2-512 best(step1000) | V2-512 final(step100k) | XL-512 best |
|---|---|---|---|
| episodes | 128 | 128 | 128 |
| Top-1 正确率 | 28.04% | 34.58% | 40.19% |
| Recall@3 | 70.09% | 71.03% | 74.77% |
| MRR | 51.73% | 55.38% | 59.96% |
| hop 正确率 | 85.94% | 100.00% | 97.66% |
| 未知拒答率 | 14.29% | 100.00% | 76.19% |
| 已知问题被误拒率 | 0.00% | 0.00% | 0.00% |
| 未知问题被误读率 | 85.71% | 0.00% | 23.81% |
| 平均分数余量 | -0.2618 | -17.0552 | -0.6793 |
### 读取/拒答门槛曲线
| run | threshold | need F1 | need 召回 | 未知拒答率 | 已知问题被误拒率 | 未知被误读率 | tp/tn/fp/fn |
|---|---|---|---|---|---|---|---|
| V2-512 best(step1000) | 0.30 | 98.81% | 99.78% | 62.26% | 0.22% | 37.74% | 1830/66/40/4 |
| V2-512 best(step1000) | 0.40 | 98.87% | 99.78% | 64.15% | 0.22% | 35.85% | 1830/68/38/4 |
| V2-512 best(step1000) | 0.50 | 98.92% | 99.78% | 66.04% | 0.22% | 33.96% | 1830/70/36/4 |
| V2-512 best(step1000) | 0.60 | 99.40% | 99.78% | 83.02% | 0.22% | 16.98% | 1830/88/18/4 |
| V2-512 best(step1000) | 0.70 | 99.84% | 99.78% | 98.11% | 0.22% | 1.89% | 1830/104/2/4 |
| V2-512 final(step100k) | 0.30 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
| V2-512 final(step100k) | 0.40 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
| V2-512 final(step100k) | 0.50 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
| V2-512 final(step100k) | 0.60 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
| V2-512 final(step100k) | 0.70 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
| XL-512 best | 0.30 | 99.19% | 99.78% | 75.47% | 0.22% | 24.53% | 1830/80/26/4 |
| XL-512 best | 0.40 | 99.51% | 99.78% | 86.79% | 0.22% | 13.21% | 1830/92/14/4 |
| XL-512 best | 0.50 | 99.62% | 99.78% | 90.57% | 0.22% | 9.43% | 1830/96/10/4 |
| XL-512 best | 0.60 | 99.84% | 99.78% | 98.11% | 0.22% | 1.89% | 1830/104/2/4 |
| XL-512 best | 0.70 | 99.89% | 99.78% | 100.00% | 0.22% | 0.00% | 1830/106/0/4 |
_smoke_e2e.json
+432
View File
@@ -0,0 +1,432 @@
{
"deployed": {
"summary": {
"router": "deployed",
"cases": 11,
"accuracy_pct": 9.090909090909092,
"answerable_cases": 8,
"answerable_accuracy_pct": 0.0,
"unknown_cases": 3,
"unknown_refusal_pct": 33.333333333333336,
"wrong_abstention_pct": 37.5,
"seconds": 19.5,
"per_category": {
"(policy)": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 0,
"answerable_accuracy_pct": 0.0,
"unknown": 1,
"unknown_refusal_pct": 0.0
},
"conflict_update": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 1,
"answerable_accuracy_pct": 0.0,
"unknown": 0,
"unknown_refusal_pct": 0.0
},
"distractor_128": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 1,
"answerable_accuracy_pct": 0.0,
"unknown": 0,
"unknown_refusal_pct": 0.0
},
"distractor_32": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 1,
"answerable_accuracy_pct": 0.0,
"unknown": 0,
"unknown_refusal_pct": 0.0
},
"forget_correction": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 0,
"answerable_accuracy_pct": 0.0,
"unknown": 1,
"unknown_refusal_pct": 0.0
},
"long_context": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 1,
"answerable_accuracy_pct": 0.0,
"unknown": 0,
"unknown_refusal_pct": 0.0
},
"multi_hop": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 1,
"answerable_accuracy_pct": 0.0,
"unknown": 0,
"unknown_refusal_pct": 0.0
},
"paraphrase": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 1,
"answerable_accuracy_pct": 0.0,
"unknown": 0,
"unknown_refusal_pct": 0.0
},
"random_position": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 1,
"answerable_accuracy_pct": 0.0,
"unknown": 0,
"unknown_refusal_pct": 0.0
},
"single_fact": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 1,
"answerable_accuracy_pct": 0.0,
"unknown": 0,
"unknown_refusal_pct": 0.0
},
"unknown_abstention": {
"cases": 1,
"accuracy_pct": 100.0,
"answerable": 0,
"answerable_accuracy_pct": 0.0,
"unknown": 1,
"unknown_refusal_pct": 100.0
}
}
},
"rows": [
{
"category": "(policy)",
"query": "请问对象测试实体0000的驻地代号为？",
"reply": "关于对象测试实体 0000 的驻地代号，现有资料中未包含该实体的具体信息",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": false
},
{
"category": "conflict_update",
"query": "我的提醒时间最新值是什么？",
"reply": "作为一个人工智能助手，我**无法访问您的设备、系统或任何外部数据**，因此不知道您具体的",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": true
},
{
"category": "distractor_128",
"query": "在大量无关信息中，只读取我的常用语言，它的值是什么？",
"reply": "您的常用语言是**中文**。",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": false
},
{
"category": "distractor_32",
"query": "在大量无关信息中，只读取我的工作地点，它的值是什么？",
"reply": "您提供的消息中**没有包含任何关于工作地点的信息**。\n\n如果您有具体的文本或上下文想要我从中提取工作地点",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": false
},
{
"category": "forget_correction",
"query": "我的项目代号是什么？如果已经删除，请回答没有记录。",
"reply": "作为一个人工智能助手，我**没有访问权限**来查询或记录您的个人项目信息。我的设计原则",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": false
},
{
"category": "long_context",
"query": "在长上下文压缩之后，检索我的项目代号并回答。",
"reply": "您好！由于我无法访问您之前的对话历史或上下文（每次对话对我来说都是全新的开始），目前我**不知道您的",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": true
},
{
"category": "multi_hop",
"query": "请通过项目负责人关系，找出项目项目-P-KP2F9GHO负责人的工作代号。",
"reply": "不知道。当前长期记忆中未包含关于项目-P-KP2F9GHO的负责人信息，因此无法通过",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": true
},
{
"category": "paraphrase",
"query": "我之前登记的最喜欢的水果内容是什么？",
"reply": "您好，作为一个人工智能助手，我**无法访问或查看您个人的历史数据、账户信息或之前登记",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": false
},
{
"category": "random_position",
"query": "随机位置事实测试：我的常用语言是什么？",
"reply": "您的常用语言是**中文**。",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": false
},
{
"category": "single_fact",
"query": "跨对话后请回答：我的默认输出风格是什么？",
"reply": "作为 Qwen3.5，我的默认输出风格是**专业、简洁、准确且富有同理心**的",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": false
},
{
"category": "unknown_abstention",
"query": "我的不存在的个人属性-0008000是什么？如果没有记录，请明确说不知道。",
"reply": "不知道。\n\n“不存在的个人属性 -0008000\"并非任何已知的系统、数据库",
"written": 4,
"correct": true,
"matched": [],
"abstained": true,
"wrongly_abstained": false
}
]
},
"V2-128-v6": {
"summary": {
"router": "V2-128-v6",
"cases": 11,
"accuracy_pct": 9.090909090909092,
"answerable_cases": 8,
"answerable_accuracy_pct": 0.0,
"unknown_cases": 3,
"unknown_refusal_pct": 33.333333333333336,
"wrong_abstention_pct": 37.5,
"seconds": 17.9,
"per_category": {
"(policy)": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 0,
"answerable_accuracy_pct": 0.0,
"unknown": 1,
"unknown_refusal_pct": 0.0
},
"conflict_update": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 1,
"answerable_accuracy_pct": 0.0,
"unknown": 0,
"unknown_refusal_pct": 0.0
},
"distractor_128": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 1,
"answerable_accuracy_pct": 0.0,
"unknown": 0,
"unknown_refusal_pct": 0.0
},
"distractor_32": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 1,
"answerable_accuracy_pct": 0.0,
"unknown": 0,
"unknown_refusal_pct": 0.0
},
"forget_correction": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 0,
"answerable_accuracy_pct": 0.0,
"unknown": 1,
"unknown_refusal_pct": 0.0
},
"long_context": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 1,
"answerable_accuracy_pct": 0.0,
"unknown": 0,
"unknown_refusal_pct": 0.0
},
"multi_hop": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 1,
"answerable_accuracy_pct": 0.0,
"unknown": 0,
"unknown_refusal_pct": 0.0
},
"paraphrase": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 1,
"answerable_accuracy_pct": 0.0,
"unknown": 0,
"unknown_refusal_pct": 0.0
},
"random_position": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 1,
"answerable_accuracy_pct": 0.0,
"unknown": 0,
"unknown_refusal_pct": 0.0
},
"single_fact": {
"cases": 1,
"accuracy_pct": 0.0,
"answerable": 1,
"answerable_accuracy_pct": 0.0,
"unknown": 0,
"unknown_refusal_pct": 0.0
},
"unknown_abstention": {
"cases": 1,
"accuracy_pct": 100.0,
"answerable": 0,
"answerable_accuracy_pct": 0.0,
"unknown": 1,
"unknown_refusal_pct": 100.0
}
}
},
"rows": [
{
"category": "(policy)",
"query": "请问对象测试实体0000的驻地代号为？",
"reply": "关于对象测试实体 0000 的驻地代号，现有资料中未包含该实体的具体信息",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": false
},
{
"category": "conflict_update",
"query": "我的提醒时间最新值是什么？",
"reply": "作为一个人工智能助手，我**无法访问您的设备、系统或任何外部数据**，因此不知道您具体的",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": true
},
{
"category": "distractor_128",
"query": "在大量无关信息中，只读取我的常用语言，它的值是什么？",
"reply": "您的常用语言是**中文**。",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": false
},
{
"category": "distractor_32",
"query": "在大量无关信息中，只读取我的工作地点，它的值是什么？",
"reply": "您提供的消息中**没有包含任何关于工作地点的信息**。\n\n如果您有具体的文本或上下文想要我从中提取工作地点",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": false
},
{
"category": "forget_correction",
"query": "我的项目代号是什么？如果已经删除，请回答没有记录。",
"reply": "作为一个人工智能助手，我**没有访问权限**来查询或记录您的个人项目信息。我的设计原则",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": false
},
{
"category": "long_context",
"query": "在长上下文压缩之后，检索我的项目代号并回答。",
"reply": "您好！由于我无法访问您之前的对话历史或上下文（每次对话对我来说都是全新的开始），目前我**不知道您的",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": true
},
{
"category": "multi_hop",
"query": "请通过项目负责人关系，找出项目项目-P-KP2F9GHO负责人的工作代号。",
"reply": "不知道。当前长期记忆中未包含关于项目-P-KP2F9GHO的负责人信息，因此无法通过",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": true
},
{
"category": "paraphrase",
"query": "我之前登记的最喜欢的水果内容是什么？",
"reply": "您好，作为一个人工智能助手，我**无法访问或查看您个人的历史数据、账户信息或之前登记",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": false
},
{
"category": "random_position",
"query": "随机位置事实测试：我的常用语言是什么？",
"reply": "您的常用语言是**中文**。",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": false
},
{
"category": "single_fact",
"query": "跨对话后请回答：我的默认输出风格是什么？",
"reply": "作为 Qwen3.5，我的默认输出风格是**专业、简洁、准确且富有同理心**的",
"written": 4,
"correct": false,
"matched": [],
"abstained": false,
"wrongly_abstained": false
},
{
"category": "unknown_abstention",
"query": "我的不存在的个人属性-0008000是什么？如果没有记录，请明确说不知道。",
"reply": "不知道。\n\n“不存在的个人属性 -0008000\"并非任何已知的系统、数据库",
"written": 4,
"correct": true,
"matched": [],
"abstained": true,
"wrongly_abstained": false
}
]
}
}
_smoke_scorecard_v5.json
+832
View File
@@ -0,0 +1,832 @@
{
"V2-128 deployed": {
"checkpoint": "H:\\Memory\\dynamic_memory_lab\\checkpoints\\natural_memory_v2_qwen_router_entities\\memory_router_v2.pt",
"router": {
"kind": "router_v2",
"step": null,
"arch_config": {
"arch": "router_v2",
"hidden_size": 2560,
"router_dim": 128,
"num_heads": 8,
"max_hops": 3
},
"parameters": 2037774,
"checkpoint_bytes": 8157561
},
"storage": {
"address_bytes_per_record": 512,
"address_mb_per_1m_records": 512,
"checkpoint_bytes": 8157561
},
"latency": {
"cpu": {
"single_query_latency_ms_mean": 0.4756599897518754,
"single_query_latency_ms_p50": 0.5036999937146902,
"single_query_latency_ms_p95": 0.6246999837458134,
"queries_per_second": 2102.3420542931153,
"samples": 5
}
},
"metrics": {
"episodes": 21920,
"answerable_episodes": 21814,
"unknown_episodes": 106,
"top1": 0.37274227560282386,
"recall1": 0.37274227560282386,
"recall3": 0.5745392866966168,
"recall5": 0.7406252865132483,
"recall8": 0.8605941138718254,
"mrr": 0.4984150474242403,
"ndcg3": 0.46099764820049244,
"mean_first_positive_rank": 8.257541028697167,
"all_evidence_in_top3": 0.5159988997891263,
"multi_positive_episodes": 2000,
"all_evidence_in_top3_multi": 0.436,
"hop_accuracy": 0.7322992700729927,
"hop_accuracy_answerable": 0.7358577060603282,
"hop_under_prediction": 0.1299165673420739,
"hop_over_prediction": 0.13422572659759788,
"mean_score_margin": -10.438775865636897,
"median_score_margin": -12.105871319770813,
"negative_margin_rate": 0.6272577243971761,
"hop_confusion": {
"true_0_pred_0": 0,
"true_0_pred_1": 42,
"true_0_pred_2": 64,
"true_1_pred_0": 834,
"true_1_pred_1": 16052,
"true_1_pred_2": 2928,
"true_2_pred_0": 0,
"true_2_pred_1": 2000,
"true_2_pred_2": 0
},
"known_mean_need_prob": 0.9804023649948944,
"unknown_mean_need_prob": 0.9999472358316746,
"thr0.30": {
"need_precision": 0.9951642335766423,
"need_recall": 1.0,
"need_f1": 0.9975762564595051,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 0.9951642335766423,
"known_question_refusal_rate": 0.0,
"unknown_question_read_rate": 1.0,
"counts": {
"tp": 21814,
"tn": 0,
"fp": 106,
"fn": 0
}
},
"thr0.40": {
"need_precision": 0.9951642335766423,
"need_recall": 1.0,
"need_f1": 0.9975762564595051,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 0.9951642335766423,
"known_question_refusal_rate": 0.0,
"unknown_question_read_rate": 1.0,
"counts": {
"tp": 21814,
"tn": 0,
"fp": 106,
"fn": 0
}
},
"thr0.50": {
"need_precision": 0.99515716374269,
"need_recall": 0.9985330521683322,
"need_f1": 0.9968422497826185,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 0.9937043795620438,
"known_question_refusal_rate": 0.0014669478316677363,
"unknown_question_read_rate": 1.0,
"counts": {
"tp": 21782,
"tn": 0,
"fp": 106,
"fn": 32
}
},
"thr0.60": {
"need_precision": 0.995134936662383,
"need_recall": 0.9939488401943706,
"need_f1": 0.994541534791982,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 0.9891423357664234,
"known_question_refusal_rate": 0.0060511598056294125,
"unknown_question_read_rate": 1.0,
"counts": {
"tp": 21682,
"tn": 0,
"fp": 106,
"fn": 132
}
},
"thr0.70": {
"need_precision": 0.9951266608431796,
"need_recall": 0.9922526817640047,
"need_f1": 0.9936875932514634,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 0.9874543795620438,
"known_question_refusal_rate": 0.0077473182359952325,
"unknown_question_read_rate": 1.0,
"counts": {
"tp": 21645,
"tn": 0,
"fp": 106,
"fn": 169
}
},
"thr0.80": {
"need_precision": 0.9951118284528476,
"need_recall": 0.9892271018611901,
"need_f1": 0.9921607393273408,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 0.9844434306569343,
"known_question_refusal_rate": 0.010772898138809939,
"unknown_question_read_rate": 1.0,
"counts": {
"tp": 21579,
"tn": 0,
"fp": 106,
"fn": 235
}
}
},
"by_family": {
"benchmark_qa": {
"episodes": 64,
"answerable_episodes": 0,
"unknown_episodes": 64,
"top1": 0.0,
"recall1": 0.0,
"recall3": 0.0,
"recall5": 0.0,
"recall8": 0.0,
"mrr": 0.0,
"ndcg3": 0.0,
"mean_first_positive_rank": 0.0,
"all_evidence_in_top3": 0.0,
"multi_positive_episodes": 0,
"all_evidence_in_top3_multi": 0.0,
"hop_accuracy": 0.0,
"hop_accuracy_answerable": 0.0,
"hop_under_prediction": 0.0,
"hop_over_prediction": 0.0,
"mean_score_margin": 0.0,
"median_score_margin": 0.0,
"negative_margin_rate": 0.0,
"hop_confusion": {
"true_0_pred_0": 0,
"true_0_pred_2": 64
},
"known_mean_need_prob": 0.0,
"unknown_mean_need_prob": 0.9999341862276196,
"thr0.50": {
"need_precision": 0.0,
"need_recall": 0.0,
"need_f1": 0.0,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 0.0,
"known_question_refusal_rate": 0.0,
"unknown_question_read_rate": 1.0,
"counts": {
"tp": 0,
"tn": 0,
"fp": 64,
"fn": 0
}
}
},
"mega_validation": {
"episodes": 20000,
"answerable_episodes": 20000,
"unknown_episodes": 0,
"top1": 0.39425,
"recall1": 0.39425,
"recall3": 0.6039,
"recall5": 0.7746,
"recall8": 0.88695,
"mrr": 0.5189064291859179,
"ndcg3": 0.48454538178459833,
"mean_first_positive_rank": 8.22755,
"all_evidence_in_top3": 0.54005,
"multi_positive_episodes": 2000,
"all_evidence_in_top3_multi": 0.436,
"hop_accuracy": 0.74955,
"hop_accuracy_answerable": 0.74955,
"hop_under_prediction": 0.11285,
"hop_over_prediction": 0.1376,
"mean_score_margin": -9.574584903246164,
"median_score_margin": -10.816837549209595,
"negative_margin_rate": 0.60575,
"hop_confusion": {
"true_1_pred_0": 257,
"true_1_pred_1": 14991,
"true_1_pred_2": 2752,
"true_2_pred_0": 0,
"true_2_pred_1": 2000,
"true_2_pred_2": 0
},
"known_mean_need_prob": 0.9854895650088787,
"unknown_mean_need_prob": 0.0,
"thr0.50": {
"need_precision": 1.0,
"need_recall": 1.0,
"need_f1": 1.0,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 1.0,
"known_question_refusal_rate": 0.0,
"unknown_question_read_rate": 0.0,
"counts": {
"tp": 20000,
"tn": 0,
"fp": 0,
"fn": 0
}
}
},
"memory_policy": {
"episodes": 1728,
"answerable_episodes": 1707,
"unknown_episodes": 21,
"top1": 0.14411247803163443,
"recall1": 0.14411247803163443,
"recall3": 0.26362038664323373,
"recall5": 0.380199179847686,
"recall8": 0.5752782659636789,
"mrr": 0.2814257926451813,
"ndcg3": 0.21245701956649282,
"mean_first_positive_rank": 8.519625073227886,
"all_evidence_in_top3": 0.26362038664323373,
"multi_positive_episodes": 0,
"all_evidence_in_top3_multi": 0.0,
"hop_accuracy": 0.5740740740740741,
"hop_accuracy_answerable": 0.5811364967779731,
"hop_under_prediction": 0.3380199179847686,
"hop_over_prediction": 0.08084358523725835,
"mean_score_margin": -19.398099298226114,
"median_score_margin": -18.179970741271973,
"negative_margin_rate": 0.8558875219683656,
"hop_confusion": {
"true_0_pred_0": 0,
"true_0_pred_1": 21,
"true_0_pred_2": 0,
"true_1_pred_0": 577,
"true_1_pred_1": 992,
"true_1_pred_2": 138
},
"known_mean_need_prob": 0.9195722369471697,
"unknown_mean_need_prob": 0.9999671209426153,
"thr0.50": {
"need_precision": 0.9876179245283019,
"need_recall": 0.981253661394259,
"need_f1": 0.9844255069056715,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 0.9693287037037037,
"known_question_refusal_rate": 0.018746338605741066,
"unknown_question_read_rate": 1.0,
"counts": {
"tp": 1675,
"tn": 0,
"fp": 21,
"fn": 32
}
}
},
"native_memory": {
"episodes": 128,
"answerable_episodes": 107,
"unknown_episodes": 21,
"top1": 0.0,
"recall1": 0.0,
"recall3": 0.04672897196261682,
"recall5": 0.14018691588785046,
"recall8": 0.48598130841121495,
"mrr": 0.12993862381957846,
"ndcg3": 0.024588128538051007,
"mean_first_positive_rank": 9.682242990654206,
"all_evidence_in_top3": 0.04672897196261682,
"multi_positive_episodes": 0,
"all_evidence_in_top3_multi": 0.0,
"hop_accuracy": 0.5390625,
"hop_accuracy_answerable": 0.6448598130841121,
"hop_under_prediction": 0.0,
"hop_over_prediction": 0.35514018691588783,
"mean_score_margin": -29.03928192530837,
"median_score_margin": -29.07152795791626,
"negative_margin_rate": 1.0,
"hop_confusion": {
"true_0_pred_0": 0,
"true_0_pred_1": 21,
"true_0_pred_2": 0,
"true_1_pred_0": 0,
"true_1_pred_1": 69,
"true_1_pred_2": 38
},
"known_mean_need_prob": 0.9999633771236812,
"unknown_mean_need_prob": 0.9999671209426153,
"thr0.50": {
"need_precision": 0.8359375,
"need_recall": 1.0,
"need_f1": 0.9106382978723404,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 0.8359375,
"known_question_refusal_rate": 0.0,
"unknown_question_read_rate": 1.0,
"counts": {
"tp": 107,
"tn": 0,
"fp": 21,
"fn": 0
}
}
}
},
"by_category": {
"conflict_update": {
"episodes": 2000,
"answerable_episodes": 2000,
"unknown_episodes": 0,
"top1": 0.8285,
"recall1": 0.8285,
"recall3": 0.968,
"recall5": 0.9915,
"recall8": 1.0,
"mrr": 0.9000553571428571,
"ndcg3": 0.9126522728928603,
"mean_first_positive_rank": 1.2975,
"all_evidence_in_top3": 0.968,
"multi_positive_episodes": 0,
"all_evidence_in_top3_multi": 0.0,
"hop_accuracy": 0.4955,
"hop_accuracy_answerable": 0.4955,
"hop_under_prediction": 0.1285,
"hop_over_prediction": 0.376,
"mean_score_margin": 23.08462015709281,
"median_score_margin": 23.422454118728638,
"negative_margin_rate": 0.1715,
"hop_confusion": {
"true_1_pred_0": 257,
"true_1_pred_1": 991,
"true_1_pred_2": 752
},
"known_mean_need_prob": 0.9764632487297058,
"unknown_mean_need_prob": 0.0,
"thr0.50": {
"need_precision": 1.0,
"need_recall": 1.0,
"need_f1": 1.0,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 1.0,
"known_question_refusal_rate": 0.0,
"unknown_question_read_rate": 0.0,
"counts": {
"tp": 2000,
"tn": 0,
"fp": 0,
"fn": 0
}
}
},
"distractor_128": {
"episodes": 2000,
"answerable_episodes": 2000,
"unknown_episodes": 0,
"top1": 0.2455,
"recall1": 0.2455,
"recall3": 0.375,
"recall5": 0.539,
"recall8": 0.632,
"mrr": 0.36134907557720053,
"ndcg3": 0.3170583471857158,
"mean_first_positive_rank": 13.3785,
"all_evidence_in_top3": 0.375,
"multi_positive_episodes": 0,
"all_evidence_in_top3_multi": 0.0,
"hop_accuracy": 1.0,
"hop_accuracy_answerable": 1.0,
"hop_under_prediction": 0.0,
"hop_over_prediction": 0.0,
"mean_score_margin": -23.816504290685057,
"median_score_margin": -35.661601066589355,
"negative_margin_rate": 0.7545,
"hop_confusion": {
"true_1_pred_1": 2000
},
"known_mean_need_prob": 0.9723067943453789,
"unknown_mean_need_prob": 0.0,
"thr0.50": {
"need_precision": 1.0,
"need_recall": 1.0,
"need_f1": 1.0,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 1.0,
"known_question_refusal_rate": 0.0,
"unknown_question_read_rate": 0.0,
"counts": {
"tp": 2000,
"tn": 0,
"fp": 0,
"fn": 0
}
}
},
"distractor_32": {
"episodes": 2000,
"answerable_episodes": 2000,
"unknown_episodes": 0,
"top1": 0.238,
"recall1": 0.238,
"recall3": 0.3575,
"recall5": 0.5285,
"recall8": 0.64,
"mrr": 0.3534111817914967,
"ndcg3": 0.30423102280178715,
"mean_first_positive_rank": 13.3855,
"all_evidence_in_top3": 0.3575,
"multi_positive_episodes": 0,
"all_evidence_in_top3_multi": 0.0,
"hop_accuracy": 1.0,
"hop_accuracy_answerable": 1.0,
"hop_under_prediction": 0.0,
"hop_over_prediction": 0.0,
"mean_score_margin": -24.995892224468292,
"median_score_margin": -35.90759539604187,
"negative_margin_rate": 0.762,
"hop_confusion": {
"true_1_pred_1": 2000
},
"known_mean_need_prob": 0.9724746745526791,
"unknown_mean_need_prob": 0.0,
"thr0.50": {
"need_precision": 1.0,
"need_recall": 1.0,
"need_f1": 1.0,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 1.0,
"known_question_refusal_rate": 0.0,
"unknown_question_read_rate": 0.0,
"counts": {
"tp": 2000,
"tn": 0,
"fp": 0,
"fn": 0
}
}
},
"forget_correction": {
"episodes": 2000,
"answerable_episodes": 2000,
"unknown_episodes": 0,
"top1": 0.416,
"recall1": 0.416,
"recall3": 1.0745,
"recall5": 1.532,
"recall8": 1.909,
"mrr": 0.5771349386724387,
"ndcg3": 0.4913475872584595,
"mean_first_positive_rank": 2.921,
"all_evidence_in_top3": 0.436,
"multi_positive_episodes": 2000,
"all_evidence_in_top3_multi": 0.436,
"hop_accuracy": 0.0,
"hop_accuracy_answerable": 0.0,
"hop_under_prediction": 1.0,
"hop_over_prediction": 0.0,
"mean_score_margin": 4.576246764823795,
"median_score_margin": -5.306148529052734,
"negative_margin_rate": 0.584,
"hop_confusion": {
"true_2_pred_1": 2000,
"true_2_pred_2": 0
},
"known_mean_need_prob": 0.99774328109622,
"unknown_mean_need_prob": 0.0,
"thr0.50": {
"need_precision": 1.0,
"need_recall": 1.0,
"need_f1": 1.0,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 1.0,
"known_question_refusal_rate": 0.0,
"unknown_question_read_rate": 0.0,
"counts": {
"tp": 2000,
"tn": 0,
"fp": 0,
"fn": 0
}
}
},
"long_context": {
"episodes": 2000,
"answerable_episodes": 2000,
"unknown_episodes": 0,
"top1": 0.2345,
"recall1": 0.2345,
"recall3": 0.2865,
"recall5": 0.3325,
"recall8": 0.3705,
"mrr": 0.2950555862194011,
"ndcg3": 0.26455882236071515,
"mean_first_positive_rank": 19.348,
"all_evidence_in_top3": 0.2865,
"multi_positive_episodes": 0,
"all_evidence_in_top3_multi": 0.0,
"hop_accuracy": 1.0,
"hop_accuracy_answerable": 1.0,
"hop_under_prediction": 0.0,
"hop_over_prediction": 0.0,
"mean_score_margin": -15.633986071705818,
"median_score_margin": -15.739282608032227,
"negative_margin_rate": 0.7655,
"hop_confusion": {
"true_1_pred_1": 2000
},
"known_mean_need_prob": 0.9875293919146061,
"unknown_mean_need_prob": 0.0,
"thr0.50": {
"need_precision": 1.0,
"need_recall": 1.0,
"need_f1": 1.0,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 1.0,
"known_question_refusal_rate": 0.0,
"unknown_question_read_rate": 0.0,
"counts": {
"tp": 2000,
"tn": 0,
"fp": 0,
"fn": 0
}
}
},
"multi_hop": {
"episodes": 2000,
"answerable_episodes": 2000,
"unknown_episodes": 0,
"top1": 0.715,
"recall1": 0.715,
"recall3": 0.9805,
"recall5": 0.998,
"recall8": 0.999,
"mrr": 0.8446400793650793,
"ndcg3": 0.8766200106625064,
"mean_first_positive_rank": 1.383,
"all_evidence_in_top3": 0.9805,
"multi_positive_episodes": 0,
"all_evidence_in_top3_multi": 0.0,
"hop_accuracy": 0.0,
"hop_accuracy_answerable": 0.0,
"hop_under_prediction": 0.0,
"hop_over_prediction": 1.0,
"mean_score_margin": 3.2684113913476467,
"median_score_margin": 5.575284242630005,
"negative_margin_rate": 0.285,
"hop_confusion": {
"true_1_pred_1": 0,
"true_1_pred_2": 2000
},
"known_mean_need_prob": 0.9993210689723492,
"unknown_mean_need_prob": 0.0,
"thr0.50": {
"need_precision": 1.0,
"need_recall": 1.0,
"need_f1": 1.0,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 1.0,
"known_question_refusal_rate": 0.0,
"unknown_question_read_rate": 0.0,
"counts": {
"tp": 2000,
"tn": 0,
"fp": 0,
"fn": 0
}
}
},
"paraphrase": {
"episodes": 2000,
"answerable_episodes": 2000,
"unknown_episodes": 0,
"top1": 0.4255,
"recall1": 0.4255,
"recall3": 0.605,
"recall5": 0.7305,
"recall8": 0.809,
"mrr": 0.5507020870945611,
"ndcg3": 0.5297832026464318,
"mean_first_positive_rank": 7.688,
"all_evidence_in_top3": 0.605,
"multi_positive_episodes": 0,
"all_evidence_in_top3_multi": 0.0,
"hop_accuracy": 1.0,
"hop_accuracy_answerable": 1.0,
"hop_under_prediction": 0.0,
"hop_over_prediction": 0.0,
"mean_score_margin": -4.418384120300412,
"median_score_margin": -5.730679512023926,
"negative_margin_rate": 0.5745,
"hop_confusion": {
"true_1_pred_1": 2000
},
"known_mean_need_prob": 0.9880377072393894,
"unknown_mean_need_prob": 0.0,
"thr0.50": {
"need_precision": 1.0,
"need_recall": 1.0,
"need_f1": 1.0,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 1.0,
"known_question_refusal_rate": 0.0,
"unknown_question_read_rate": 0.0,
"counts": {
"tp": 2000,
"tn": 0,
"fp": 0,
"fn": 0
}
}
},
"random_position": {
"episodes": 2000,
"answerable_episodes": 2000,
"unknown_episodes": 0,
"top1": 0.2755,
"recall1": 0.2755,
"recall3": 0.474,
"recall5": 0.7665,
"recall8": 0.9335,
"mrr": 0.4512681556346879,
"ndcg3": 0.38633728319107397,
"mean_first_positive_rank": 4.9905,
"all_evidence_in_top3": 0.474,
"multi_positive_episodes": 0,
"all_evidence_in_top3_multi": 0.0,
"hop_accuracy": 1.0,
"hop_accuracy_answerable": 1.0,
"hop_under_prediction": 0.0,
"hop_over_prediction": 0.0,
"mean_score_margin": -17.09528378547728,
"median_score_margin": -19.91835129261017,
"negative_margin_rate": 0.7245,
"hop_confusion": {
"true_1_pred_1": 2000
},
"known_mean_need_prob": 0.9656237260401249,
"unknown_mean_need_prob": 0.0,
"thr0.50": {
"need_precision": 1.0,
"need_recall": 1.0,
"need_f1": 1.0,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 1.0,
"known_question_refusal_rate": 0.0,
"unknown_question_read_rate": 0.0,
"counts": {
"tp": 2000,
"tn": 0,
"fp": 0,
"fn": 0
}
}
},
"single_fact": {
"episodes": 2000,
"answerable_episodes": 2000,
"unknown_episodes": 0,
"top1": 0.3585,
"recall1": 0.3585,
"recall3": 0.5315,
"recall5": 0.748,
"recall8": 0.8755,
"mrr": 0.5041280494591153,
"ndcg3": 0.4566527480678597,
"mean_first_positive_rank": 6.2835,
"all_evidence_in_top3": 0.5315,
"multi_positive_episodes": 0,
"all_evidence_in_top3_multi": 0.0,
"hop_accuracy": 1.0,
"hop_accuracy_answerable": 1.0,
"hop_under_prediction": 0.0,
"hop_over_prediction": 0.0,
"mean_score_margin": -5.610978243432939,
"median_score_margin": -14.75244402885437,
"negative_margin_rate": 0.6415,
"hop_confusion": {
"true_1_pred_1": 2000
},
"known_mean_need_prob": 0.9954859737157822,
"unknown_mean_need_prob": 0.0,
"thr0.50": {
"need_precision": 1.0,
"need_recall": 1.0,
"need_f1": 1.0,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 1.0,
"known_question_refusal_rate": 0.0,
"unknown_question_read_rate": 0.0,
"counts": {
"tp": 2000,
"tn": 0,
"fp": 0,
"fn": 0
}
}
},
"unknown": {
"episodes": 1920,
"answerable_episodes": 1814,
"unknown_episodes": 106,
"top1": 0.13561190738699008,
"recall1": 0.13561190738699008,
"recall3": 0.2508269018743109,
"recall5": 0.36604189636163176,
"recall8": 0.5700110253583242,
"mrr": 0.2724902209448839,
"ndcg3": 0.20137544771420876,
"mean_first_positive_rank": 8.588202866593164,
"all_evidence_in_top3": 0.2508269018743109,
"multi_positive_episodes": 0,
"all_evidence_in_top3_multi": 0.0,
"hop_accuracy": 0.5526041666666667,
"hop_accuracy_answerable": 0.5848952590959207,
"hop_under_prediction": 0.31808158765159866,
"hop_over_prediction": 0.0970231532524807,
"mean_score_margin": -19.966790886482894,
"median_score_margin": -18.91780662536621,
"negative_margin_rate": 0.86438809261301,
"hop_confusion": {
"true_0_pred_0": 0,
"true_0_pred_1": 42,
"true_0_pred_2": 64,
"true_1_pred_0": 577,
"true_1_pred_1": 1061,
"true_1_pred_2": 176
},
"known_mean_need_prob": 0.9243141619741194,
"unknown_mean_need_prob": 0.9999472358316746,
"thr0.50": {
"need_precision": 0.9438559322033898,
"need_recall": 0.9823594266813671,
"need_f1": 0.9627228525121555,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 0.928125,
"known_question_refusal_rate": 0.017640573318632856,
"unknown_question_read_rate": 1.0,
"counts": {
"tp": 1782,
"tn": 0,
"fp": 106,
"fn": 32
}
}
},
"unknown_abstention": {
"episodes": 2000,
"answerable_episodes": 2000,
"unknown_episodes": 0,
"top1": 0.2055,
"recall1": 0.2055,
"recall3": 0.3865,
"recall5": 0.5795,
"recall8": 0.701,
"mrr": 0.3513197809023414,
"ndcg3": 0.30621252077857364,
"mean_first_positive_rank": 11.6,
"all_evidence_in_top3": 0.3865,
"multi_positive_episodes": 0,
"all_evidence_in_top3_multi": 0.0,
"hop_accuracy": 1.0,
"hop_accuracy_answerable": 1.0,
"hop_under_prediction": 0.0,
"hop_over_prediction": 0.0,
"mean_score_margin": -35.1040986096561,
"median_score_margin": -36.334869384765625,
"negative_margin_rate": 0.7945,
"hop_confusion": {
"true_1_pred_1": 2000
},
"known_mean_need_prob": 0.9999097834825516,
"unknown_mean_need_prob": 0.0,
"thr0.50": {
"need_precision": 1.0,
"need_recall": 1.0,
"need_f1": 1.0,
"specificity_unknown_refusal": 0.0,
"abstention_accuracy": 1.0,
"known_question_refusal_rate": 0.0,
"unknown_question_read_rate": 0.0,
"counts": {
"tp": 2000,
"tn": 0,
"fp": 0,
"fn": 0
}
}
}
}
}
}
_smoke_scorecard_v5.md
+193
View File
@@ -0,0 +1,193 @@
| metric | V2-128 deployed |
|---|---|
| 参数 | 2,037,774 |
| router_dim | 128 |
| 每条记录地址字节 | 512 |
| **Top-1 正确率** | 37.27% |
| Recall@1 | 37.27% |
| Recall@3 | 57.45% |
| Recall@5 | 74.06% |
| MRR | 49.84% |
| nDCG@3 | 46.10% |
| 多跳证据全中(Top-3) | 51.60% |
| hop 正确率 | 73.23% |
| hop 欠预测率 | 12.99% |
| need F1 | 99.68% |
| need 召回 | 99.85% |
| **未知拒答率** | 0.00% |
| **已知问题被误拒率** | 0.15% |
| 未知问题被误读率 | 100.00% |
| 仲裁准确率 | 99.37% |
| 已知问题 need 概率均值 | 98.04% |
| 未知问题 need 概率均值 | 99.99% |
| 平均分数余量 | -10.4388 |
| 单查询延迟 ms(GPU) | - |
| 路由 QPS(GPU) | - |
### family = `benchmark_qa`
| metric | V2-128 deployed |
|---|---|
| episodes | 64 |
| Top-1 正确率 | 0.00% |
| Recall@3 | 0.00% |
| MRR | 0.00% |
| hop 正确率 | 0.00% |
| 未知拒答率 | 0.00% |
| 已知问题被误拒率 | 0.00% |
| 未知问题被误读率 | 100.00% |
| 平均分数余量 | 0.0000 |
### family = `mega_validation`
| metric | V2-128 deployed |
|---|---|
| episodes | 20000 |
| Top-1 正确率 | 39.42% |
| Recall@3 | 60.39% |
| MRR | 51.89% |
| hop 正确率 | 74.95% |
| 未知拒答率 | 0.00% |
| 已知问题被误拒率 | 0.00% |
| 未知问题被误读率 | 0.00% |
| 平均分数余量 | -9.5746 |
### family = `memory_policy`
| metric | V2-128 deployed |
|---|---|
| episodes | 1728 |
| Top-1 正确率 | 14.41% |
| Recall@3 | 26.36% |
| MRR | 28.14% |
| hop 正确率 | 57.41% |
| 未知拒答率 | 0.00% |
| 已知问题被误拒率 | 1.87% |
| 未知问题被误读率 | 100.00% |
| 平均分数余量 | -19.3981 |
### family = `native_memory`
| metric | V2-128 deployed |
|---|---|
| episodes | 128 |
| Top-1 正确率 | 0.00% |
| Recall@3 | 4.67% |
| MRR | 12.99% |
| hop 正确率 | 53.91% |
| 未知拒答率 | 0.00% |
| 已知问题被误拒率 | 0.00% |
| 未知问题被误读率 | 100.00% |
| 平均分数余量 | -29.0393 |
### 读取/拒答门槛曲线
| run | threshold | need F1 | need 召回 | 未知拒答率 | 已知问题被误拒率 | 未知被误读率 | tp/tn/fp/fn |
|---|---|---|---|---|---|---|---|
| V2-128 deployed | 0.30 | 99.76% | 100.00% | 0.00% | 0.00% | 100.00% | 21814/0/106/0 |
| V2-128 deployed | 0.40 | 99.76% | 100.00% | 0.00% | 0.00% | 100.00% | 21814/0/106/0 |
| V2-128 deployed | 0.50 | 99.68% | 99.85% | 0.00% | 0.15% | 100.00% | 21782/0/106/32 |
| V2-128 deployed | 0.60 | 99.45% | 99.39% | 0.00% | 0.61% | 100.00% | 21682/0/106/132 |
| V2-128 deployed | 0.70 | 99.37% | 99.23% | 0.00% | 0.77% | 100.00% | 21645/0/106/169 |
| V2-128 deployed | 0.80 | 99.22% | 98.92% | 0.00% | 1.08% | 100.00% | 21579/0/106/235 |
### 按 mega 类别拆解（每类 2,000 条）
| 类别 | episodes | metric | V2-128 deployed |
|---|---|---|---|
| (未分类) | - | Top-1 正确率 | - |
| (未分类) | - | Recall@3 | - |
| (未分类) | - | MRR | - |
| (未分类) | - | 多跳证据全中 | - |
| (未分类) | - | hop 正确率 | - |
| (未分类) | - | 未知拒答率 | - |
| (未分类) | - | 已知问题被误拒率 | - |
| (未分类) | - | 未知问题被误读率 | - |
| conflict_update | 2000 | Top-1 正确率 | 82.85% |
| conflict_update | 2000 | Recall@3 | 96.80% |
| conflict_update | 2000 | MRR | 90.01% |
| conflict_update | 2000 | 多跳证据全中 | 96.80% |
| conflict_update | 2000 | hop 正确率 | 49.55% |
| conflict_update | 2000 | 未知拒答率 | 0.00% |
| conflict_update | 2000 | 已知问题被误拒率 | 0.00% |
| conflict_update | 2000 | 未知问题被误读率 | 0.00% |
| distractor_128 | 2000 | Top-1 正确率 | 24.55% |
| distractor_128 | 2000 | Recall@3 | 37.50% |
| distractor_128 | 2000 | MRR | 36.13% |
| distractor_128 | 2000 | 多跳证据全中 | 37.50% |
| distractor_128 | 2000 | hop 正确率 | 100.00% |
| distractor_128 | 2000 | 未知拒答率 | 0.00% |
| distractor_128 | 2000 | 已知问题被误拒率 | 0.00% |
| distractor_128 | 2000 | 未知问题被误读率 | 0.00% |
| distractor_32 | 2000 | Top-1 正确率 | 23.80% |
| distractor_32 | 2000 | Recall@3 | 35.75% |
| distractor_32 | 2000 | MRR | 35.34% |
| distractor_32 | 2000 | 多跳证据全中 | 35.75% |
| distractor_32 | 2000 | hop 正确率 | 100.00% |
| distractor_32 | 2000 | 未知拒答率 | 0.00% |
| distractor_32 | 2000 | 已知问题被误拒率 | 0.00% |
| distractor_32 | 2000 | 未知问题被误读率 | 0.00% |
| forget_correction | 2000 | Top-1 正确率 | 41.60% |
| forget_correction | 2000 | Recall@3 | 107.45% |
| forget_correction | 2000 | MRR | 57.71% |
| forget_correction | 2000 | 多跳证据全中 | 43.60% |
| forget_correction | 2000 | hop 正确率 | 0.00% |
| forget_correction | 2000 | 未知拒答率 | 0.00% |
| forget_correction | 2000 | 已知问题被误拒率 | 0.00% |
| forget_correction | 2000 | 未知问题被误读率 | 0.00% |
| long_context | 2000 | Top-1 正确率 | 23.45% |
| long_context | 2000 | Recall@3 | 28.65% |
| long_context | 2000 | MRR | 29.51% |
| long_context | 2000 | 多跳证据全中 | 28.65% |
| long_context | 2000 | hop 正确率 | 100.00% |
| long_context | 2000 | 未知拒答率 | 0.00% |
| long_context | 2000 | 已知问题被误拒率 | 0.00% |
| long_context | 2000 | 未知问题被误读率 | 0.00% |
| multi_hop | 2000 | Top-1 正确率 | 71.50% |
| multi_hop | 2000 | Recall@3 | 98.05% |
| multi_hop | 2000 | MRR | 84.46% |
| multi_hop | 2000 | 多跳证据全中 | 98.05% |
| multi_hop | 2000 | hop 正确率 | 0.00% |
| multi_hop | 2000 | 未知拒答率 | 0.00% |
| multi_hop | 2000 | 已知问题被误拒率 | 0.00% |
| multi_hop | 2000 | 未知问题被误读率 | 0.00% |
| paraphrase | 2000 | Top-1 正确率 | 42.55% |
| paraphrase | 2000 | Recall@3 | 60.50% |
| paraphrase | 2000 | MRR | 55.07% |
| paraphrase | 2000 | 多跳证据全中 | 60.50% |
| paraphrase | 2000 | hop 正确率 | 100.00% |
| paraphrase | 2000 | 未知拒答率 | 0.00% |
| paraphrase | 2000 | 已知问题被误拒率 | 0.00% |
| paraphrase | 2000 | 未知问题被误读率 | 0.00% |
| random_position | 2000 | Top-1 正确率 | 27.55% |
| random_position | 2000 | Recall@3 | 47.40% |
| random_position | 2000 | MRR | 45.13% |
| random_position | 2000 | 多跳证据全中 | 47.40% |
| random_position | 2000 | hop 正确率 | 100.00% |
| random_position | 2000 | 未知拒答率 | 0.00% |
| random_position | 2000 | 已知问题被误拒率 | 0.00% |
| random_position | 2000 | 未知问题被误读率 | 0.00% |
| single_fact | 2000 | Top-1 正确率 | 35.85% |
| single_fact | 2000 | Recall@3 | 53.15% |
| single_fact | 2000 | MRR | 50.41% |
| single_fact | 2000 | 多跳证据全中 | 53.15% |
| single_fact | 2000 | hop 正确率 | 100.00% |
| single_fact | 2000 | 未知拒答率 | 0.00% |
| single_fact | 2000 | 已知问题被误拒率 | 0.00% |
| single_fact | 2000 | 未知问题被误读率 | 0.00% |
| unknown | 1920 | Top-1 正确率 | 13.56% |
| unknown | 1920 | Recall@3 | 25.08% |
| unknown | 1920 | MRR | 27.25% |
| unknown | 1920 | 多跳证据全中 | 25.08% |
| unknown | 1920 | hop 正确率 | 55.26% |
| unknown | 1920 | 未知拒答率 | 0.00% |
| unknown | 1920 | 已知问题被误拒率 | 1.76% |
| unknown | 1920 | 未知问题被误读率 | 100.00% |
| unknown_abstention | 2000 | Top-1 正确率 | 20.55% |
| unknown_abstention | 2000 | Recall@3 | 38.65% |
| unknown_abstention | 2000 | MRR | 35.13% |
| unknown_abstention | 2000 | 多跳证据全中 | 38.65% |
| unknown_abstention | 2000 | hop 正确率 | 100.00% |
| unknown_abstention | 2000 | 未知拒答率 | 0.00% |
| unknown_abstention | 2000 | 已知问题被误拒率 | 0.00% |
| unknown_abstention | 2000 | 未知问题被误读率 | 0.00% |
_verdict_v3.json
+436
View File
@@ -0,0 +1,436 @@
{
"candidates": [
"XL-512 best",
"XL-512 final(step100k)"
],
"baselines": [
"V2-512 best(step1000)",
"V2-512 final(step100k)"
],
"passed_axes": 20,
"failed_axes": 22,
"skipped_axes": 0,
"dominates": false,
"rows": [
{
"axis": "Top-1 正确率",
"candidate": "XL-512 best",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.6712104689203926,
"candidate_value": 0.608505997818975,
"delta": -0.06270447110141764,
"direction": "max",
"pass": false
},
{
"axis": "Top-1 正确率",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.6712104689203926,
"candidate_value": 0.5436205016357688,
"delta": -0.1275899672846238,
"direction": "max",
"pass": false
},
{
"axis": "Recall@1",
"candidate": "XL-512 best",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.6712104689203926,
"candidate_value": 0.608505997818975,
"delta": -0.06270447110141764,
"direction": "max",
"pass": false
},
{
"axis": "Recall@1",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.6712104689203926,
"candidate_value": 0.5436205016357688,
"delta": -0.1275899672846238,
"direction": "max",
"pass": false
},
{
"axis": "Recall@3",
"candidate": "XL-512 best",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.8876772082878953,
"candidate_value": 0.9296619411123228,
"delta": 0.041984732824427495,
"direction": "max",
"pass": true
},
{
"axis": "Recall@3",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.8876772082878953,
"candidate_value": 0.8533260632497274,
"delta": -0.03435114503816794,
"direction": "max",
"pass": false
},
{
"axis": "Recall@5",
"candidate": "XL-512 best",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.9492911668484187,
"candidate_value": 0.9732824427480916,
"delta": 0.02399127589967287,
"direction": "max",
"pass": true
},
{
"axis": "Recall@5",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.9492911668484187,
"candidate_value": 0.9193020719738277,
"delta": -0.029989094874591005,
"direction": "max",
"pass": false
},
{
"axis": "MRR",
"candidate": "XL-512 best",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.7908385671852008,
"candidate_value": 0.759814160403073,
"delta": -0.03102440678212781,
"direction": "max",
"pass": false
},
{
"axis": "MRR",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.7908385671852008,
"candidate_value": 0.6950444145765433,
"delta": -0.09579415260865753,
"direction": "max",
"pass": false
},
{
"axis": "nDCG@3",
"candidate": "XL-512 best",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.7999554748364778,
"candidate_value": 0.792764395065411,
"delta": -0.007191079771066811,
"direction": "max",
"pass": false
},
{
"axis": "nDCG@3",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.7999554748364778,
"candidate_value": 0.7170573814087626,
"delta": -0.08289809342771515,
"direction": "max",
"pass": false
},
{
"axis": "多跳证据全中(Top-3)",
"candidate": "XL-512 best",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.8865866957470011,
"candidate_value": 0.9285714285714286,
"delta": 0.041984732824427495,
"direction": "max",
"pass": true
},
{
"axis": "多跳证据全中(Top-3)",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.8865866957470011,
"candidate_value": 0.8522355507088332,
"delta": -0.03435114503816794,
"direction": "max",
"pass": false
},
{
"axis": "多跳证据全中(仅多正例)",
"candidate": "XL-512 best",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.5,
"candidate_value": 0.5,
"delta": 0.0,
"direction": "max",
"pass": true
},
{
"axis": "多跳证据全中(仅多正例)",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.5,
"candidate_value": 0.5,
"delta": 0.0,
"direction": "max",
"pass": true
},
{
"axis": "hop 正确率",
"candidate": "XL-512 best",
"baseline": "V2-512 final(step100k)",
"baseline_value": 0.9969072164948454,
"candidate_value": 0.9938144329896907,
"delta": -0.003092783505154628,
"direction": "max",
"pass": false
},
{
"axis": "hop 正确率",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 final(step100k)",
"baseline_value": 0.9969072164948454,
"candidate_value": 0.9969072164948454,
"delta": 0.0,
"direction": "max",
"pass": true
},
{
"axis": "hop 欠预测率",
"candidate": "XL-512 best",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.003271537622682661,
"candidate_value": 0.003271537622682661,
"delta": 0.0,
"direction": "min",
"pass": true
},
{
"axis": "hop 欠预测率",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.003271537622682661,
"candidate_value": 0.003271537622682661,
"delta": 0.0,
"direction": "min",
"pass": true
},
{
"axis": "need F1 (门槛 0.50)",
"candidate": "XL-512 best",
"baseline": "V2-512 final(step100k)",
"baseline_value": 0.9989082969432315,
"candidate_value": 0.9961894393032119,
"delta": -0.0027188576400195563,
"direction": "max",
"pass": false
},
{
"axis": "need F1 (门槛 0.50)",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 final(step100k)",
"baseline_value": 0.9989082969432315,
"candidate_value": 0.9989082969432315,
"delta": 0.0,
"direction": "max",
"pass": true
},
{
"axis": "need 召回 (门槛 0.50)",
"candidate": "XL-512 best",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.9978189749182116,
"candidate_value": 0.9978189749182116,
"delta": 0.0,
"direction": "max",
"pass": true
},
{
"axis": "need 召回 (门槛 0.50)",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.9978189749182116,
"candidate_value": 0.9978189749182116,
"delta": 0.0,
"direction": "max",
"pass": true
},
{
"axis": "未知拒答率 (门槛 0.50)",
"candidate": "XL-512 best",
"baseline": "V2-512 final(step100k)",
"baseline_value": 1.0,
"candidate_value": 0.9056603773584906,
"delta": -0.09433962264150941,
"direction": "max",
"pass": false
},
{
"axis": "未知拒答率 (门槛 0.50)",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 final(step100k)",
"baseline_value": 1.0,
"candidate_value": 1.0,
"delta": 0.0,
"direction": "max",
"pass": true
},
{
"axis": "已知问题被误拒率",
"candidate": "XL-512 best",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.0021810250817884407,
"candidate_value": 0.0021810250817884407,
"delta": 0.0,
"direction": "min",
"pass": true
},
{
"axis": "已知问题被误拒率",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.0021810250817884407,
"candidate_value": 0.0021810250817884407,
"delta": 0.0,
"direction": "min",
"pass": true
},
{
"axis": "未知问题被误读率",
"candidate": "XL-512 best",
"baseline": "V2-512 final(step100k)",
"baseline_value": 0.0,
"candidate_value": 0.09433962264150944,
"delta": 0.09433962264150944,
"direction": "min",
"pass": false
},
{
"axis": "未知问题被误读率",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 final(step100k)",
"baseline_value": 0.0,
"candidate_value": 0.0,
"delta": 0.0,
"direction": "min",
"pass": true
},
{
"axis": "仲裁准确率",
"candidate": "XL-512 best",
"baseline": "V2-512 final(step100k)",
"baseline_value": 0.9979381443298969,
"candidate_value": 0.9927835051546392,
"delta": -0.005154639175257714,
"direction": "max",
"pass": false
},
{
"axis": "仲裁准确率",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 final(step100k)",
"baseline_value": 0.9979381443298969,
"candidate_value": 0.9979381443298969,
"delta": 0.0,
"direction": "max",
"pass": true
},
{
"axis": "平均分数余量",
"candidate": "XL-512 best",
"baseline": "V2-512 final(step100k)",
"baseline_value": 3.8836881184794065,
"candidate_value": 6.709678303645345,
"delta": 2.825990185165938,
"direction": "max",
"pass": true
},
{
"axis": "平均分数余量",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 final(step100k)",
"baseline_value": 3.8836881184794065,
"candidate_value": 25.375655539715694,
"delta": 21.49196742123629,
"direction": "max",
"pass": true
},
{
"axis": "单查询延迟 ms (GPU)",
"candidate": "XL-512 best",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.8951400030249109,
"candidate_value": 1.3101640006061643,
"delta": 0.4150239975812534,
"direction": "min",
"pass": false
},
{
"axis": "单查询延迟 ms (GPU)",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 best(step1000)",
"baseline_value": 0.8951400030249109,
"candidate_value": 1.2458240000220637,
"delta": 0.3506839969971528,
"direction": "min",
"pass": false
},
{
"axis": "路由 QPS (GPU)",
"candidate": "XL-512 best",
"baseline": "V2-512 best(step1000)",
"baseline_value": 1117.1436832459055,
"candidate_value": 763.263224708767,
"delta": -353.88045853713857,
"direction": "max",
"pass": false
},
{
"axis": "路由 QPS (GPU)",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 best(step1000)",
"baseline_value": 1117.1436832459055,
"candidate_value": 802.6815986706708,
"delta": -314.4620845752347,
"direction": "max",
"pass": false
},
{
"axis": "地址字节/记录",
"candidate": "XL-512 best",
"baseline": "V2-512 best(step1000)",
"baseline_value": 2048,
"candidate_value": 2048,
"delta": 0,
"direction": "min",
"pass": true
},
{
"axis": "地址字节/记录",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 best(step1000)",
"baseline_value": 2048,
"candidate_value": 2048,
"delta": 0,
"direction": "min",
"pass": true
},
{
"axis": "参数量",
"candidate": "XL-512 best",
"baseline": "V2-512 best(step1000)",
"baseline_value": 4741902,
"candidate_value": 7898127,
"delta": 3156225,
"direction": "min",
"pass": false
},
{
"axis": "参数量",
"candidate": "XL-512 final(step100k)",
"baseline": "V2-512 best(step1000)",
"baseline_value": 4741902,
"candidate_value": 7898127,
"delta": 3156225,
"direction": "min",
"pass": false
}
]
}
abstention_separability.json
+314
View File
@@ -0,0 +1,314 @@
{
"scorers": {
"cosine": {
"single_feature_auc_on_eval": {
"top1": 0.5898,
"top2": 0.5572,
"margin": 0.6257,
"top3_mean": 0.5771,
"top1_minus_median": 0.6166,
"std": 0.5894,
"z_top1": 0.6125,
"n_within_10pct": 0.4273,
"entropy": 0.6168
},
"head_auc_on_eval": 0.6086,
"head_weights": {
"top1": 0.2654,
"top2": 0.3455,
"margin": -0.2102,
"top3_mean": -0.4675,
"top1_minus_median": -0.0127,
"std": -0.1578,
"z_top1": -0.2808,
"n_within_10pct": 0.1283,
"entropy": 0.9424
},
"operating_curve": [
{
"threshold": 0.05,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.1,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.2,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.3,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.4,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.5,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.6,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.7,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.8,
"unknown_flagged_pct": 46.0,
"answerable_rejected_pct": 30.0
},
{
"threshold": 0.9,
"unknown_flagged_pct": 96.0,
"answerable_rejected_pct": 84.0
}
]
},
"router": {
"single_feature_auc_on_eval": {
"top1": 0.5379,
"top2": 0.5263,
"margin": 0.5516,
"top3_mean": 0.5386,
"top1_minus_median": 0.5108,
"std": 0.5447,
"z_top1": 0.5711,
"n_within_10pct": 0.426,
"entropy": 0.497
},
"head_auc_on_eval": 0.513,
"head_weights": {
"top1": 0.259,
"top2": 0.0524,
"margin": 0.3174,
"top3_mean": 0.0438,
"top1_minus_median": 0.0431,
"std": 0.0405,
"z_top1": 0.4694,
"n_within_10pct": 0.4964,
"entropy": 0.0123
},
"operating_curve": [
{
"threshold": 0.05,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.1,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.2,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.3,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.4,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.5,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.6,
"unknown_flagged_pct": 4.0,
"answerable_rejected_pct": 5.2
},
{
"threshold": 0.7,
"unknown_flagged_pct": 4.0,
"answerable_rejected_pct": 7.2
},
{
"threshold": 0.8,
"unknown_flagged_pct": 38.0,
"answerable_rejected_pct": 34.8
},
{
"threshold": 0.9,
"unknown_flagged_pct": 90.0,
"answerable_rejected_pct": 94.0
}
]
},
"text_retriever": {
"single_feature_auc_on_eval": {
"top1": 0.4992,
"top2": 0.4966,
"margin": 0.6076,
"top3_mean": 0.4954,
"top1_minus_median": 0.5712,
"std": 0.5043,
"z_top1": 0.5512,
"n_within_10pct": 0.4462,
"entropy": 0.5655
},
"head_auc_on_eval": 0.4752,
"head_weights": {
"top1": -0.3237,
"top2": -0.3446,
"margin": 0.0748,
"top3_mean": 0.8565,
"top1_minus_median": 0.0869,
"std": 0.4047,
"z_top1": 0.1123,
"n_within_10pct": -0.2813,
"entropy": -0.4289
},
"operating_curve": [
{
"threshold": 0.05,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.1,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.2,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.3,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.4,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.5,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.6,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.7,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 1.2
},
{
"threshold": 0.8,
"unknown_flagged_pct": 14.0,
"answerable_rejected_pct": 15.2
},
{
"threshold": 0.9,
"unknown_flagged_pct": 100.0,
"answerable_rejected_pct": 100.0
}
]
},
"blend_50_50": {
"single_feature_auc_on_eval": {
"top1": 0.4788,
"top2": 0.493,
"margin": 0.5446,
"top3_mean": 0.4929,
"top1_minus_median": 0.4943,
"std": 0.5042,
"z_top1": 0.5132,
"n_within_10pct": 0.5104,
"entropy": 0.4914
},
"head_auc_on_eval": 0.5062,
"head_weights": {
"top1": 0.2989,
"top2": 0.3659,
"margin": -0.0757,
"top3_mean": -0.2102,
"top1_minus_median": -0.1659,
"std": 0.3382,
"z_top1": 0.4046,
"n_within_10pct": -0.0748,
"entropy": -0.0503
},
"operating_curve": [
{
"threshold": 0.05,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.1,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.2,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.3,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.4,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.5,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.6,
"unknown_flagged_pct": 0.0,
"answerable_rejected_pct": 0.0
},
{
"threshold": 0.7,
"unknown_flagged_pct": 2.0,
"answerable_rejected_pct": 0.4
},
{
"threshold": 0.8,
"unknown_flagged_pct": 22.0,
"answerable_rejected_pct": 28.4
},
{
"threshold": 0.9,
"unknown_flagged_pct": 72.0,
"answerable_rejected_pct": 74.0
}
]
}
},
"train_episodes": 1200,
"eval_episodes": 300
}
abstention_separability.md
+81
View File
@@ -0,0 +1,81 @@
# 未知问题可分性分析（能否从分数几何判断"库里没有这个属性"）
训练集 1200 条（未知 199），评测集 300 条（未知 50）。
逻辑回归头**只在训练集上拟合**，AUC 在评测集上计算。AUC 0.5 = 完全不可分。
## 打分器 `cosine`
**单特征最佳 AUC**：margin 0.6257, entropy 0.6168, top1_minus_median 0.6166, z_top1 0.6125
**拟合头 AUC（评测集）**：0.6086
| 判定阈值 | 标出未知的比例 | 误拒可回答的比例 |
|---:|---:|---:|
| 0.05 | 0.00% | 0.00% |
| 0.10 | 0.00% | 0.00% |
| 0.20 | 0.00% | 0.00% |
| 0.30 | 0.00% | 0.00% |
| 0.40 | 0.00% | 0.00% |
| 0.50 | 0.00% | 0.00% |
| 0.60 | 0.00% | 0.00% |
| 0.70 | 0.00% | 0.00% |
| 0.80 | 46.00% | 30.00% |
| 0.90 | 96.00% | 84.00% |
## 打分器 `router`
**单特征最佳 AUC**：z_top1 0.5711, margin 0.5516, std 0.5447, top3_mean 0.5386
**拟合头 AUC（评测集）**：0.5130
| 判定阈值 | 标出未知的比例 | 误拒可回答的比例 |
|---:|---:|---:|
| 0.05 | 0.00% | 0.00% |
| 0.10 | 0.00% | 0.00% |
| 0.20 | 0.00% | 0.00% |
| 0.30 | 0.00% | 0.00% |
| 0.40 | 0.00% | 0.00% |
| 0.50 | 0.00% | 0.00% |
| 0.60 | 4.00% | 5.20% |
| 0.70 | 4.00% | 7.20% |
| 0.80 | 38.00% | 34.80% |
| 0.90 | 90.00% | 94.00% |
## 打分器 `text_retriever`
**单特征最佳 AUC**：margin 0.6076, top1_minus_median 0.5712, entropy 0.5655, z_top1 0.5512
**拟合头 AUC（评测集）**：0.4752
| 判定阈值 | 标出未知的比例 | 误拒可回答的比例 |
|---:|---:|---:|
| 0.05 | 0.00% | 0.00% |
| 0.10 | 0.00% | 0.00% |
| 0.20 | 0.00% | 0.00% |
| 0.30 | 0.00% | 0.00% |
| 0.40 | 0.00% | 0.00% |
| 0.50 | 0.00% | 0.00% |
| 0.60 | 0.00% | 0.00% |
| 0.70 | 0.00% | 1.20% |
| 0.80 | 14.00% | 15.20% |
| 0.90 | 100.00% | 100.00% |
## 打分器 `blend_50_50`
**单特征最佳 AUC**：margin 0.5446, z_top1 0.5132, n_within_10pct 0.5104, std 0.5042
**拟合头 AUC（评测集）**：0.5062
| 判定阈值 | 标出未知的比例 | 误拒可回答的比例 |
|---:|---:|---:|
| 0.05 | 0.00% | 0.00% |
| 0.10 | 0.00% | 0.00% |
| 0.20 | 0.00% | 0.00% |
| 0.30 | 0.00% | 0.00% |
| 0.40 | 0.00% | 0.00% |
| 0.50 | 0.00% | 0.00% |
| 0.60 | 0.00% | 0.00% |
| 0.70 | 2.00% | 0.40% |
| 0.80 | 22.00% | 28.40% |
| 0.90 | 72.00% | 74.00% |
analyze_abstention_separability.py
+257
View File
@@ -0,0 +1,257 @@
"""Can "the bank does not contain this attribute" be detected at all?
The runtime answers 75% of unanswerable paraphrased questions with an invented code. A
score threshold was already measured to be useless: the retrieval top-score distributions
of answerable and unanswerable episodes overlap almost entirely. Before building any
classifier head, this script asks the prior question properly and cheaply -- **without
generating text** -- by scoring every candidate of every episode with the same scorers the
runtime uses and testing how well answerability can be predicted from the resulting score
geometry (top-1, margin, top-k mean, spread...).
Method, so the number is trustworthy:
* features come from the frozen feature bank, i.e. exactly the 2560-dim keys the runtime
scores;
* a logistic head is **fitted on the train split and evaluated on the eval split**, so the
reported AUC is not an in-sample artefact;
* the reported ceiling is the AUC of the single best feature and of the fitted head, plus
the achievable operating points (unknown-flagging rate vs answerable rejection rate).
If the AUC is near 0.5 the conclusion is firm: abstention cannot be recovered from score
geometry on this feature set, and the fix has to be structural.
Usage::
python -m V2_dpskw.analyze_abstention_separability --with-text-retriever
"""
from __future__ import annotations
import argparse
import hashlib
import json
from pathlib import Path
import numpy as np
import torch
import torch.nn.functional as F
from .eval_router_scorecard import load_router_any
def text_key(text: str) -> str:
return hashlib.sha1(str(text).encode("utf-8", "replace")).hexdigest()
def load_episodes(path: Path) -> list[dict]:
rows = []
with path.open("r", encoding="utf-8") as handle:
for line in handle:
line = line.strip()
if line:
rows.append(json.loads(line))
return rows
def episode_vectors(row: dict, lookup: dict, bank: np.ndarray):
query = row["query"]
candidates = [c["text"] for c in row["candidates"]]
missing = [t for t in [query, *candidates] if text_key(t) not in lookup]
if missing:
raise SystemExit(f"{row['id']}: {len(missing)} texts missing from the bank")
return bank[lookup[text_key(query)]], bank[[lookup[text_key(t)] for t in candidates]]
def features_from_scores(scores: np.ndarray) -> dict:
order = np.sort(scores)[::-1]
top1 = float(order[0])
top2 = float(order[1]) if order.size > 1 else 0.0
top3 = float(order[2]) if order.size > 2 else top2
spread = float(np.std(scores))
median = float(np.median(scores))
return {
"top1": top1,
"top2": top2,
"margin": top1 - top2,
"top3_mean": float(np.mean(order[:3])),
"top1_minus_median": top1 - median,
"std": spread,
"z_top1": (top1 - float(np.mean(scores))) / (spread + 1e-6),
# how many candidates sit close to the best one: a truly held fact should be
# clearly ahead of 23 unrelated same-shape facts
"n_within_10pct": float(np.sum(scores >= top1 - 0.10 * max(abs(top1), 1e-6))),
"entropy": float(-np.sum(np.exp(scores - top1) / np.sum(np.exp(scores - top1))
* (scores - top1))),
}
FEATURE_NAMES = ["top1", "top2", "margin", "top3_mean", "top1_minus_median",
"std", "z_top1", "n_within_10pct", "entropy"]
def auc(labels: np.ndarray, values: np.ndarray) -> float:
"""Rank-based AUC (Mann-Whitney), ties averaged."""
positives = values[labels == 1]
negatives = values[labels == 0]
if positives.size == 0 or negatives.size == 0:
return float("nan")
order = np.argsort(np.concatenate([positives, negatives]))
ranks = np.empty(order.size, dtype=float)
ranks[order] = np.arange(1, order.size + 1)
# average ties
combined = np.concatenate([positives, negatives])
_, inverse, counts = np.unique(combined, return_inverse=True, return_counts=True)
for index, count in enumerate(counts):
if count > 1:
mask = inverse == index
ranks[mask] = ranks[mask].mean()
rank_sum = ranks[: positives.size].sum()
return float((rank_sum - positives.size * (positives.size + 1) / 2)
/ (positives.size * negatives.size))
def collect(rows, lookup, bank, scorers) -> tuple[dict[str, list[dict]], np.ndarray]:
per_scorer: dict[str, list[dict]] = {name: [] for name in scorers}
labels = []
for row in rows:
query_vector, candidate_vectors = episode_vectors(row, lookup, bank)
query = torch.from_numpy(np.ascontiguousarray(query_vector)).float().reshape(1, -1)
candidates = torch.from_numpy(
np.ascontiguousarray(candidate_vectors)).float().reshape(1, -1, bank.shape[1])
labels.append(1 if row.get("positive_indices") else 0)
with torch.no_grad():
for name, fn in scorers.items():
scores = fn(query, candidates).reshape(-1).numpy()
per_scorer[name].append(features_from_scores(scores))
return per_scorer, np.array(labels)
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--data-dir", default="data/zero_overlap")
parser.add_argument("--feature-cache", default=r"H:\Memory\nm_cache\nm_zero_overlap\feature_cache")
parser.add_argument("--router", default="checkpoints/router_replay_v7_v2_128/memory_router_v2.pt")
parser.add_argument("--with-text-retriever", action="store_true")
parser.add_argument("--package", default="qwen3_5_4b_natural_memory_v2")
parser.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
parser.add_argument("--output", default="abstention_separability.json")
parser.add_argument("--markdown", default="abstention_separability.md")
args = parser.parse_args()
cache = Path(args.feature_cache)
bank = np.load(cache / "features.f16.npy", mmap_mode="r")
lookup = json.loads((cache / "index.json").read_text(encoding="utf-8"))
device = torch.device(args.device)
router, _, _ = load_router_any(Path(args.router))
router = router.to(device).eval()
def cosine(query, candidates):
return F.cosine_similarity(query, candidates.reshape(-1, candidates.shape[-1]), dim=-1)
def router_score(query, candidates):
projected = router.encode_key(candidates.reshape(-1, candidates.shape[-1]).to(device))
scores, _ = router.projected_scores(
query.to(device), projected.reshape(1, -1, projected.shape[-1]))
return torch.sigmoid(scores.reshape(-1)).cpu()
scorers = {"cosine": cosine, "router": router_score}
retriever = None
if args.with_text_retriever:
from .qwen_integration import load_qwen_dynamic
model = load_qwen_dynamic(args.package)
model.eval()
if model.text_retriever is None or not getattr(model, "_text_retriever_ready", False):
raise SystemExit("packaged text_retriever is not ready")
retriever = model.text_retriever.to(device).eval()
def retriever_score(query, candidates):
with torch.no_grad():
out = retriever(query.to(device),
candidates.reshape(-1, candidates.shape[-1]).to(device))
return torch.sigmoid(out.reshape(-1)).cpu()
scorers["text_retriever"] = retriever_score
scorers["blend_50_50"] = lambda q, c: 0.5 * retriever_score(q, c) + 0.5 * router_score(q, c)
train_rows = load_episodes(Path(args.data_dir) / "train.jsonl")
eval_rows = load_episodes(Path(args.data_dir) / "eval.jsonl")
print(json.dumps({"train_episodes": len(train_rows), "eval_episodes": len(eval_rows),
"train_unknown": sum(1 for r in train_rows if not r.get("positive_indices")),
"eval_unknown": sum(1 for r in eval_rows if not r.get("positive_indices")),
"scorers": list(scorers)}), flush=True)
train_feats, train_labels = collect(train_rows, lookup, bank, scorers)
eval_feats, eval_labels = collect(eval_rows, lookup, bank, scorers)
report: dict = {"scorers": {}, "train_episodes": len(train_rows), "eval_episodes": len(eval_rows)}
for name in scorers:
entry: dict = {"single_feature_auc_on_eval": {}}
for feature in FEATURE_NAMES:
values = np.array([row[feature] for row in eval_feats[name]], dtype=float)
entry["single_feature_auc_on_eval"][feature] = round(auc(eval_labels, values), 4)
# Logistic head, fitted on train only.
X_train = np.array([[row[f] for f in FEATURE_NAMES] for row in train_feats[name]], dtype=np.float64)
X_eval = np.array([[row[f] for f in FEATURE_NAMES] for row in eval_feats[name]], dtype=np.float64)
mean, std = X_train.mean(axis=0), X_train.std(axis=0) + 1e-9
Xtr = (X_train - mean) / std
Xev = (X_eval - mean) / std
weights = np.zeros(Xtr.shape[1])
bias = 0.0
# plain gradient descent with L2; no sklearn dependency
for _ in range(4000):
logits = Xtr @ weights + bias
probs = 1.0 / (1.0 + np.exp(-logits))
grad_w = Xtr.T @ (probs - train_labels) / len(train_labels) + 1e-3 * weights
grad_b = float(np.mean(probs - train_labels))
weights -= 0.5 * grad_w
bias -= 0.5 * grad_b
eval_probs = 1.0 / (1.0 + np.exp(-(Xev @ weights + bias)))
entry["head_auc_on_eval"] = round(auc(eval_labels, eval_probs), 4)
entry["head_weights"] = {f: round(float(w), 4) for f, w in zip(FEATURE_NAMES, weights)}
# Operating points: keep answerable episodes (maximise) while flagging unknowns.
curve = []
for threshold in (0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9):
flagged_unknown = float(np.mean(eval_probs[eval_labels == 0] < threshold))
rejected_answerable = float(np.mean(eval_probs[eval_labels == 1] < threshold))
curve.append({
"threshold": threshold,
"unknown_flagged_pct": round(100 * flagged_unknown, 2),
"answerable_rejected_pct": round(100 * rejected_answerable, 2),
})
entry["operating_curve"] = curve
report["scorers"][name] = entry
Path(args.output).write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
lines = ["# 未知问题可分性分析（能否从分数几何判断\"库里没有这个属性\"）", "",
f"训练集 {len(train_rows)} 条（未知 {report['train_episodes'] and sum(1 for r in train_rows if not r.get('positive_indices'))}），"
f"评测集 {len(eval_rows)} 条（未知 {sum(1 for r in eval_rows if not r.get('positive_indices'))}）。",
"逻辑回归头**只在训练集上拟合**，AUC 在评测集上计算。AUC 0.5 = 完全不可分。", ""]
for name, entry in report["scorers"].items():
lines.append(f"## 打分器 `{name}`")
lines.append("")
lines.append(f"**单特征最佳 AUC**：" + ", ".join(
f"{k} {v:.4f}" for k, v in sorted(entry["single_feature_auc_on_eval"].items(),
key=lambda kv: -kv[1])[:4]))
lines.append("")
lines.append(f"**拟合头 AUC（评测集）**：{entry['head_auc_on_eval']:.4f}")
lines.append("")
lines.append("| 判定阈值 | 标出未知的比例 | 误拒可回答的比例 |")
lines.append("|---:|---:|---:|")
for point in entry["operating_curve"]:
lines.append("| {threshold:.2f} | {unknown_flagged_pct:.2f}% | {answerable_rejected_pct:.2f}% |".format(**point))
lines.append("")
text = "\n".join(lines) + "\n"
Path(args.markdown).write_text(text, encoding="utf-8")
print(text)
return 0
if __name__ == "__main__":
raise SystemExit(main())
analyze_open_vocabulary_attribute_match.py
+142
View File
@@ -0,0 +1,142 @@
"""Open-vocabulary attribute matching: can a question be matched to an attribute *name*?
The closed 24-way head is domain-locked: on a bank whose attributes it never saw, it can
only answer with one of its 24 labels and therefore refuses everything (measured: A-segment
answerable accuracy 0.00%). The production shape needs no fixed vocabulary at all --
the bank's own attribute names are the candidate set, and they change as the user writes.
This script measures whether the *frozen* keys already support that, with no training:
score(question, name) = cosine(query_key, encode(attribute_name))
and reports (a) whether the asked-about attribute is ranked first among the names actually
present in the bank, and (b) whether the best score separates "the bank holds it" from
"the bank does not", which is what an abstention rule needs.
Usage::
python -m V2_dpskw.analyze_open_vocabulary_attribute_match
"""
from __future__ import annotations
import argparse
import hashlib
import json
from pathlib import Path
import numpy as np
import torch
import torch.nn.functional as F
from .make_zero_overlap_paraphrase_data import ATTRIBUTE_PARAPHRASES
def text_key(text: str) -> str:
return hashlib.sha1(str(text).encode("utf-8", "replace")).hexdigest()
def auc(labels: np.ndarray, values: np.ndarray) -> float:
positive = values[labels == 1]
negative = values[labels == 0]
if positive.size == 0 or negative.size == 0:
return float("nan")
return float(np.mean(positive[:, None] > negative[None, :])
+ 0.5 * np.mean(positive[:, None] == negative[None, :]))
def load(path: Path) -> list[dict]:
rows = []
with path.open("r", encoding="utf-8") as handle:
for line in handle:
line = line.strip()
if line:
rows.append(json.loads(line))
return rows
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--data-dir", default="data/zero_overlap")
parser.add_argument("--feature-cache", default=r"H:\Memory\nm_cache\nm_zero_overlap\feature_cache")
parser.add_argument("--package", default="qwen3_5_4b_natural_memory_v2")
parser.add_argument("--output", default="open_vocabulary_match.json")
parser.add_argument("--markdown", default="open_vocabulary_match.md")
args = parser.parse_args()
from .qwen_integration import load_qwen_dynamic, load_tokenizer
cache = Path(args.feature_cache)
bank = np.load(cache / "features.f16.npy", mmap_mode="r")
lookup = json.loads((cache / "index.json").read_text(encoding="utf-8"))
model = load_qwen_dynamic(args.package)
model.eval()
tokenizer = load_tokenizer(args.package)
device = next(model.parameters()).device
names = [name for name, _ in ATTRIBUTE_PARAPHRASES]
with torch.no_grad():
encoded = tokenizer(names, return_tensors="pt", padding=True, truncation=True,
max_length=64)
name_keys = model._encode_model_key(
encoded["input_ids"].to(device), encoded["attention_mask"].to(device)).float()
name_keys = F.normalize(name_keys, dim=-1).cpu()
eval_rows = load(Path(args.data_dir) / "eval.jsonl")
answerable_flags, top1_hits, best_scores, present_counts = [], [], [], []
for row in eval_rows:
query = torch.from_numpy(
np.asarray(bank[lookup[text_key(row["query"])]], dtype=np.float32))
query = F.normalize(query.reshape(1, -1), dim=-1)
present = [c["attribute"] for c in row["candidates"] if c.get("attribute")]
indices = [names.index(name) for name in dict.fromkeys(present) if name in names]
if not indices:
continue
scores = (query @ name_keys[indices].t()).reshape(-1)
best = int(scores.argmax())
best_scores.append(float(scores[best]))
present_counts.append(len(indices))
answerable_flags.append(1 if row.get("positive_indices") else 0)
# The asked-about attribute for abstention episodes is deliberately absent from the
# candidate set, so a "hit" only makes sense for answerable episodes.
if row.get("positive_indices"):
target = row["metadata"]["attribute"]
top1_hits.append(names[indices[best]] == target)
labels = np.array(answerable_flags)
best = np.array(best_scores)
report = {
"attribute_names": len(names),
"episodes": len(labels),
"answerable": int(labels.sum()),
"abstention": int((labels == 0).sum()),
"mean_present_attributes": round(float(np.mean(present_counts)), 2),
"top1_attribute_accuracy_answerable_pct": round(100 * float(np.mean(top1_hits)), 2),
"chance_pct": round(100.0 / float(np.mean(present_counts)), 2),
"best_score_answerable_p50": round(float(np.percentile(best[labels == 1], 50)), 4),
"best_score_abstention_p50": round(float(np.percentile(best[labels == 0], 50)), 4),
"best_score_auc_covered_vs_uncovered": round(auc(labels, best), 4),
"note": ("training-free: cosine between the frozen query key and the frozen encoding "
"of each attribute NAME actually present in the bank"),
}
Path(args.output).write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
lines = ["# 开放词表属性匹配（零训练，用库里的属性名直接匹配）", "",
f"库内属性名候选 {report['attribute_names']} 个；评测 {report['episodes']} 条"
f"（可回答 {report['answerable']} / 不可回答 {report['abstention']}）；"
f"平均候选数 {report['mean_present_attributes']}。", "",
"| 指标 | 值 |", "|---|---:|",
f"| 可回答 episode 的 Top-1 属性识别准确率 | **{report['top1_attribute_accuracy_answerable_pct']:.2f}%** |",
f"| 随机基线 | {report['chance_pct']:.2f}% |",
f"| 最佳匹配分中位数（可回答） | {report['best_score_answerable_p50']:.4f} |",
f"| 最佳匹配分中位数（不可回答） | {report['best_score_abstention_p50']:.4f} |",
f"| **覆盖 vs 不覆盖 AUC** | **{report['best_score_auc_covered_vs_uncovered']:.4f}** |"]
text = "\n".join(lines) + "\n"
Path(args.markdown).write_text(text, encoding="utf-8")
print(text)
return 0
if __name__ == "__main__":
raise SystemExit(main())
analyze_query_attribute_classifier.py
+324
View File
@@ -0,0 +1,324 @@
"""Do the frozen features encode *which attribute* a paraphrased question asks about?
The previous measurement killed the score-geometry route to abstention (best fitted-head AUC
~0.61, and near chance for three of four scorers). The remaining structural route is:
map the question to the attribute it asks about, then check whether the bank holds it
That only works if attribute identity is recoverable from the query's frozen key *for
phrasings never seen during training*. This script tests exactly that, and nothing else:
* query vectors are the same 2560-dim frozen keys the runtime scores;
* a linear classifier is fitted on the **train** split's queries and scored on the **eval**
split's queries, which are *different paraphrases of the same attributes* -- so the
reported number is generalisation to unseen wording, not memorisation;
* chance level is 1 / (number of attributes).
It also reports the "none" case that abstention needs: accuracy when the query's attribute
is absent from the candidate set (the unknown episodes), i.e. can we even tell that the
question refers to something the bank does not hold.
Usage::
python -m V2_dpskw.analyze_query_attribute_classifier
"""
from __future__ import annotations
import argparse
import hashlib
import json
from pathlib import Path
import numpy as np
def text_key(text: str) -> str:
return hashlib.sha1(str(text).encode("utf-8", "replace")).hexdigest()
def load(path: Path) -> list[dict]:
rows = []
with path.open("r", encoding="utf-8") as handle:
for line in handle:
line = line.strip()
if line:
rows.append(json.loads(line))
return rows
def softmax_linear_fit(X: np.ndarray, y: np.ndarray, classes: int, *, steps: int = 4000,
lr: float = 0.5, l2: float = 1e-3):
"""Plain multinomial logistic regression (no sklearn dependency)."""
W = np.zeros((X.shape[1], classes))
b = np.zeros(classes)
n = len(y)
onehot = np.zeros((n, classes))
onehot[np.arange(n), y] = 1.0
for _ in range(steps):
logits = X @ W + b
logits -= logits.max(axis=1, keepdims=True)
probs = np.exp(logits)
probs /= probs.sum(axis=1, keepdims=True)
grad = probs - onehot
W -= lr * (X.T @ grad / n + l2 * W)
b -= lr * grad.mean(axis=0)
return W, b
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--data-dir", default="data/zero_overlap")
parser.add_argument("--feature-cache", default=r"H:\Memory\nm_cache\nm_zero_overlap\feature_cache")
parser.add_argument("--output", default="query_attribute_classifier.json")
parser.add_argument("--markdown", default="query_attribute_classifier.md")
parser.add_argument("--holdout-attributes", type=int, default=0,
help=("train on all but N attributes and evaluate only on those N; "
"tests whether the head generalises to attributes it has never seen"))
parser.add_argument("--none-class", action="store_true",
help=("train an explicit NONE class using queries of attributes held out "
"of the vocabulary, then test on a *different* held-out set"))
args = parser.parse_args()
cache = Path(args.feature_cache)
bank = np.load(cache / "features.f16.npy", mmap_mode="r")
lookup = json.loads((cache / "index.json").read_text(encoding="utf-8"))
attributes = sorted({row["metadata"]["attribute"] for row in load(Path(args.data_dir) / "train.jsonl")})
index_of = {name: position for position, name in enumerate(attributes)}
classes = len(attributes)
def vectors(rows):
X, y, unknown, present = [], [], [], []
for row in rows:
key = text_key(row["query"])
if key not in lookup:
raise SystemExit(f"query missing from the bank: {row['query']!r}")
X.append(np.asarray(bank[lookup[key]], dtype=np.float32))
y.append(index_of[row["metadata"]["attribute"]])
unknown.append(0 if row.get("positive_indices") else 1)
# Which attributes does this episode's candidate set actually hold? The
# abstention question is whether the *asked-about* attribute is among them.
present.append({c.get("attribute") for c in row["candidates"] if c.get("attribute")})
return np.array(X), np.array(y), np.array(unknown), present
train_X, train_y, _, _ = vectors(load(Path(args.data_dir) / "train.jsonl"))
eval_X, eval_y, eval_unknown, eval_present = vectors(load(Path(args.data_dir) / "eval.jsonl"))
# L2-normalise and scale: the frozen keys are pooled hidden states.
train_Xn = train_X / (np.linalg.norm(train_X, axis=1, keepdims=True) + 1e-6)
eval_Xn = eval_X / (np.linalg.norm(eval_X, axis=1, keepdims=True) + 1e-6)
holdout_report = None
if args.none_class:
# The production shape of the problem: the bank holds a *vocabulary* of attributes,
# and a user may ask about one outside it. Train K attribute classes PLUS an
# explicit NONE class, where NONE examples are real queries about attributes that
# are deliberately excluded from the vocabulary. A separate, further held-out set
# of attributes is then used for testing, so NONE recall is measured out of sample.
order = np.argsort([attributes[position] for position in range(classes)])
vocabulary = set(order[:12].tolist())
none_train = set(order[12:18].tolist())
none_eval = set(order[18:].tolist())
compact = {original: position for position, original in enumerate(sorted(vocabulary))}
none_label = len(compact)
rows_train = np.concatenate([np.where(np.isin(train_y, list(vocabulary)))[0],
np.where(np.isin(train_y, list(none_train)))[0]])
labels_train = np.array([
compact.get(int(train_y[index]), none_label) for index in rows_train])
W_n, b_n = softmax_linear_fit(train_Xn[rows_train], labels_train, len(compact) + 1)
def predict(X: np.ndarray) -> np.ndarray:
return (X @ W_n + b_n).argmax(axis=1)
eval_answerable = eval_unknown == 0
vocab_rows = eval_answerable & np.isin(eval_y, list(vocabulary))
none_rows = eval_answerable & np.isin(eval_y, list(none_eval))
vocab_pred = predict(eval_Xn[vocab_rows])
none_pred = predict(eval_Xn[none_rows])
vocab_correct = float(np.mean(vocab_pred == np.array(
[compact[int(label)] for label in eval_y[vocab_rows]])))
none_recall = float(np.mean(none_pred == none_label))
none_precision = float(np.sum(none_pred == none_label) / max(1, none_pred.size))
none_report = {
"vocabulary_attributes": [attributes[position] for position in sorted(vocabulary)],
"none_train_attributes": [attributes[position] for position in sorted(none_train)],
"none_eval_attributes": [attributes[position] for position in sorted(none_eval)],
"train_rows": int(rows_train.size),
"eval_vocabulary_queries": int(vocab_rows.sum()),
"eval_never_seen_queries": int(none_rows.sum()),
"chance_pct": round(100.0 / (len(compact) + 1), 2),
"vocabulary_accuracy_pct": round(100 * vocab_correct, 2),
"never_seen_rejected_pct": round(100 * none_recall, 2),
"known_false_rejection_pct": round(
100 * float(np.mean(vocab_pred == none_label)), 2),
"note": ("an explicit NONE class trained on held-out attributes; the never-seen "
"attributes used for testing are disjoint from both the vocabulary and "
"the NONE training set"),
}
Path(args.output).write_text(json.dumps(none_report, ensure_ascii=False, indent=2),
encoding="utf-8")
print(json.dumps(none_report, ensure_ascii=False, indent=2))
return 0
if args.holdout_attributes > 0:
# Open-set test: N attributes are removed from the *training* label set entirely,
# so the head has never seen them. A production bank cannot know in advance which
# attributes a user will ask about, so the question that matters is: when a
# never-trained attribute is asked about, does the head (a) land on some *present*
# attribute -- which would make the coverage rule answer wrongly -- or (b) betray
# its uncertainty, giving a usable reject option?
rng = np.random.default_rng(20260912)
held = set(rng.choice(classes, size=min(args.holdout_attributes, classes - 2),
replace=False).tolist())
kept = sorted(set(range(classes)) - held)
compact = {original: position for position, original in enumerate(kept)}
keep_train = np.array([label in compact for label in train_y])
W_h, b_h = softmax_linear_fit(
train_Xn[keep_train],
np.array([compact[label] for label in train_y[keep_train]]),
len(kept))
def max_probability(X: np.ndarray) -> np.ndarray:
logits = X @ W_h + b_h
logits -= logits.max(axis=1, keepdims=True)
probs = np.exp(logits)
probs /= probs.sum(axis=1, keepdims=True)
return probs.max(axis=1)
held_mask = np.array([(label in held) and (unknown == 0)
for label, unknown in zip(eval_y, eval_unknown)])
kept_mask = np.array([(label in compact) and (unknown == 0)
for label, unknown in zip(eval_y, eval_unknown)])
held_prob = max_probability(eval_Xn[held_mask])
kept_prob = max_probability(eval_Xn[kept_mask])
held_pred = (eval_Xn[held_mask] @ W_h + b_h).argmax(axis=1)
# Would the closed-set head claim a *present* attribute for a never-trained one?
held_pred_names = [attributes[kept[position]] for position in held_pred]
held_present = np.array([
name in present for name, present in zip(held_pred_names, np.array(eval_present, dtype=object)[held_mask])
])
threshold = float(np.percentile(kept_prob, 5)) if kept_prob.size else 0.0
holdout_report = {
"held_out_attributes": sorted(attributes[position] for position in held),
"trained_on_attributes": len(kept),
"eval_held_out_queries": int(held_mask.sum()),
"eval_kept_queries": int(kept_mask.sum()),
"max_prob_held_out": {
"p10": round(float(np.percentile(held_prob, 10)), 4),
"p50": round(float(np.percentile(held_prob, 50)), 4),
"p90": round(float(np.percentile(held_prob, 90)), 4),
},
"max_prob_kept": {
"p10": round(float(np.percentile(kept_prob, 10)), 4),
"p50": round(float(np.percentile(kept_prob, 50)), 4),
"p90": round(float(np.percentile(kept_prob, 90)), 4),
},
"openset_auc_held_out_vs_kept": round(
float(np.mean(held_prob[:, None] < kept_prob[None, :])), 4),
"closed_set_false_claim_pct": round(100 * float(np.mean(held_present)), 2),
"reject_threshold_from_kept_p05": round(threshold, 4),
"held_out_rejected_pct_at_threshold": round(
100 * float(np.mean(held_prob < threshold)), 2),
"kept_retained_pct_at_threshold": round(
100 * float(np.mean(kept_prob >= threshold)), 2),
"note": ("held-out attributes never appear as training labels, so this is the "
"open-set case a real user bank faces"),
}
Path(args.output).write_text(json.dumps(holdout_report, ensure_ascii=False, indent=2),
encoding="utf-8")
print(json.dumps(holdout_report, ensure_ascii=False, indent=2))
return 0
W, b = softmax_linear_fit(train_Xn, train_y, classes)
eval_logits = eval_Xn @ W + b
eval_pred = eval_logits.argmax(axis=1)
train_pred = (train_Xn @ W + b).argmax(axis=1)
answerable = eval_unknown == 0
accuracy_all = float(np.mean(eval_pred == eval_y))
accuracy_answerable = float(np.mean((eval_pred == eval_y)[answerable]))
report = {
"attributes": classes,
"chance_pct": round(100.0 / classes, 2),
"train_queries": len(train_y),
"eval_queries": len(eval_y),
"distinct_train_queries": len({text_key(t) for t in []}) or None,
"train_accuracy_pct": round(100 * float(np.mean(train_pred == train_y)), 2),
"eval_accuracy_pct": round(100 * accuracy_all, 2),
"eval_accuracy_answerable_pct": round(100 * accuracy_answerable, 2),
"note": ("eval queries are different paraphrases of the same attributes, so this is "
"generalisation to unseen wording"),
}
# Per-attribute breakdown for the answerable eval queries.
per_attribute = {}
for position, name in enumerate(attributes):
mask = (eval_y == position) & answerable
if mask.sum():
per_attribute[name] = {
"eval_queries": int(mask.sum()),
"correct_pct": round(100 * float(np.mean(eval_pred[mask] == position)), 2),
}
report["per_attribute"] = per_attribute
# --- the structural abstention signal -------------------------------------------
# Predicted asked-about attribute vs the attributes the candidate set actually holds.
# Answerable episodes should land inside the set; abstention episodes should land
# outside it. A categorical split here means abstention needs no fragile threshold.
predicted_names = [attributes[position] for position in eval_pred]
inside = np.array([name in present for name, present in zip(predicted_names, eval_present)])
coverage = {
"rule": "predict the asked-about attribute; refuse when the candidate set lacks it",
"answerable_inside_pct": round(100 * float(np.mean(inside[answerable])), 2),
"unknown_inside_pct": round(100 * float(np.mean(inside[~answerable])), 2),
"answerable_cases": int(answerable.sum()),
"unknown_cases": int((~answerable).sum()),
}
# What a naive "refuse when outside" rule would yield on this split:
refused = ~inside
tp = int(np.sum(refused & ~answerable)) # unknown correctly refused
fn = int(np.sum(~refused & ~answerable)) # unknown wrongly answered
fp = int(np.sum(refused & answerable)) # answerable wrongly refused
tn = int(np.sum(~refused & answerable)) # answerable correctly answered
coverage.update({
"unknown_refusal_pct": round(100 * tp / max(1, tp + fn), 2),
"known_false_refusal_pct": round(100 * fp / max(1, fp + tn), 2),
"counts": {"tp": tp, "fn": fn, "fp": fp, "tn": tn},
})
report["coverage_rule"] = coverage
Path(args.output).write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
lines = [
"# 冻结特征是否编码「提问问的是哪个属性」？", "",
f"属性数 {classes}（随机基线 {100.0/classes:.2f}%）；训练查询 {len(train_y)} 条，"
f"评测查询 {len(eval_y)} 条（**同为这些属性的不同改写，训练时从未见过**）。", "",
"| 指标 | 值 |", "|---|---:|",
f"| 训练集准确率 | {report['train_accuracy_pct']:.2f}% |",
f"| **评测集准确率（未见改写）** | **{report['eval_accuracy_pct']:.2f}%** |",
f"| 评测集·仅可回答子集 | {report['eval_accuracy_answerable_pct']:.2f}% |",
f"| 随机基线 | {report['chance_pct']:.2f}% |", "",
"## 结构化拒答规则（判属性 → 查是否在库里）", "",
f"规则：{coverage['rule']}。", "",
"| 指标 | 值 |", "|---|---:|",
f"| 可回答 episode 中「预测属性在候选集内」 | **{coverage['answerable_inside_pct']:.2f}%**（{coverage['answerable_cases']} 条） |",
f"| 未知 episode 中「预测属性在候选集内」 | **{coverage['unknown_inside_pct']:.2f}%**（{coverage['unknown_cases']} 条） |",
f"| 未知拒答率（规则命中） | **{coverage['unknown_refusal_pct']:.2f}%** |",
f"| 已知问题被误拒率 | **{coverage['known_false_refusal_pct']:.2f}%** |",
f"| 混淆计数 | tp={coverage['counts']['tp']} fn={coverage['counts']['fn']} fp={coverage['counts']['fp']} tn={coverage['counts']['tn']} |",
"",
"## 按属性（评测集可回答）", "",
"| 属性 | 评测问法数 | 正确率 |", "|---|---:|---:|",
]
for name, body in sorted(per_attribute.items(), key=lambda kv: -kv[1]["correct_pct"]):
lines.append(f"| {name} | {body['eval_queries']} | {body['correct_pct']:.2f}% |")
text = "\n".join(lines) + "\n"
Path(args.markdown).write_text(text, encoding="utf-8")
print(text)
return 0
if __name__ == "__main__":
raise SystemExit(main())
analyze_update_conflict.py
+155
View File
@@ -0,0 +1,155 @@
"""Decide whether `update_conflict` is ill-posed or the system ignores recency.
Attribute membership is decided with the generator's own frame templates, so a
fact phrased as ``我的代码仓库是 X。`` and one phrased as ``我源码放在 X。`` are
both recognised as belonging to the same attribute -- surface-shape matching
alone misses this, which is exactly the trap this category is built around.
The harness writes ``positives first, then the remaining candidates in corpus
order`` (``eval_end_to_end_memory.build``/``run_router``), so the *last* write
wins on recency. For each case this reports:
* which written fact is the newest one for the asked attribute
* whether that newest fact is the expected answer
* what the model actually answered, mapped back to a written fact
"""
from __future__ import annotations
import json
import re
import sys
from collections import Counter, defaultdict
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from V2_dpskw.eval_scoring import squash
from V2_dpskw.make_realistic_memory_data import ATTRIBUTE_POOL
from V2_dpskw.rescore_e2e import load_run
FRAME_RE: dict[str, list[re.Pattern]] = {}
for _name, _frames, _questions in ATTRIBUTE_POOL:
pats = []
for _f in _frames:
# Split the *raw* template on the placeholders and escape each literal part
# separately. Escaping first would turn ``{s}`` into ``\{s\}`` and the
# split pattern would no longer match it.
parts = re.split(r"\{[sv]\}", _f)
parts = [re.escape(p) for p in parts]
pats.append(re.compile("".join(p if i == 0 else r"(.+?)" + p for i, p in enumerate(parts)) + r"\Z"))
FRAME_RE[_name] = pats
def belongs_to(text: str, attribute: str) -> bool:
t = text.strip()
return any(p.search(t) for p in FRAME_RE.get(attribute, []))
def value_in(text: str, attribute: str):
"""The ``{v}`` slot of the first frame that matches.
Frames carry two placeholders (``{s}`` subject and ``{v}`` value), so the
value is the *last* captured group, not the first.
"""
for p in FRAME_RE.get(attribute, []):
m = p.search(text.strip())
if m:
groups = [g for g in m.groups() if g]
return groups[-1].strip() if groups else None
return None
def load_cases(path: Path, per_category: int = 25):
rows = []
with path.open("r", encoding="utf-8") as handle:
for line in handle:
line = line.strip()
if line:
rows.append(json.loads(line))
grouped = defaultdict(list)
for row in rows:
grouped[str((row.get("metadata") or {}).get("category", ""))].append(row)
out = []
for category in sorted(grouped):
for row in grouped[category][:per_category]:
meta = row.get("metadata") or {}
candidates = [str(c.get("text", "")) for c in (row.get("candidates") or []) if isinstance(c, dict)]
positives = [candidates[i] for i in (row.get("positive_indices") or []) if 0 <= int(i) < len(candidates)]
out.append({
"category": category,
"query": str(row.get("query", "")),
"acceptable": [str(v) for v in (meta.get("acceptable") or []) if str(v).strip()],
"positives": positives,
"candidates": candidates,
"meta": meta,
})
return out
def main(corpus, run, category="update_conflict", max_facts=6, verbose=10):
cases = load_cases(Path(corpus))
rows = load_run(Path(run))
stats = Counter()
examples = []
for case, row in zip(cases, rows):
if case["category"] != category:
continue
attribute = case["meta"].get("attribute", "")
order = (case["positives"] + [f for f in case["candidates"] if f not in case["positives"]])[:max_facts]
same_attr = [f for f in order if belongs_to(f, attribute)]
newest = same_attr[-1] if same_attr else None
newest_value = value_in(newest, attribute) if newest else None
answer_value = case["meta"].get("answer", "")
reply = row.get("reply", "")
s_reply = squash(reply)
stats["cases"] += 1
stats[f"same_attr_written_{len(same_attr)}"] += 1
newest_is_answer = newest is not None and newest in case["positives"]
if newest_is_answer:
stats["newest_is_expected_answer"] += 1
else:
stats["newest_is_NOT_answer"] += 1
answered_expected = bool(answer_value) and squash(answer_value) in s_reply
answered_newest = bool(newest_value) and squash(newest_value) in s_reply
if answered_expected:
stats["answered_expected"] += 1
if newest_is_answer:
if answered_expected:
stats["posed_correct"] += 1
else:
stats["posed_wrong"] += 1
else:
# the expected answer is not the freshest evidence for the attribute
stats["illposed_cases"] += 1
if answered_newest:
stats["illposed_answered_newest"] += 1
elif answered_expected:
stats["illposed_answered_expected"] += 1
else:
stats["illposed_answered_neither"] += 1
if len(examples) < verbose:
examples.append((case, row, order, same_attr, newest, newest_value, answered_expected, answered_newest))
print(f"category={category}")
for k in sorted(stats):
print(f" {k:<34}{stats[k]}")
print("\n--- examples ---")
for case, row, order, same_attr, newest, newest_value, ae, an in examples:
print(f"Q {case['query']} attribute={case['meta'].get('attribute')} expect={case['acceptable']}")
for i, f in enumerate(order):
tags = []
if f in case["positives"]:
tags.append("POSITIVE")
if f in same_attr:
tags.append("SAME-ATTR")
if f == newest:
tags.append("NEWEST")
print(f" {i} {f} [{' '.join(tags)}]")
print(f" reply={row.get('reply','')[:64]!r}")
print(f" answered_expected={ae} answered_newest={an} newest_value={newest_value!r}")
if __name__ == "__main__":
main(sys.argv[1], sys.argv[2])
answerability_cosine.json
+63
View File
@@ -0,0 +1,63 @@
{
"overall_auc": 0.6741,
"per_category": {
"alias_paraphrase": {
"n": 25,
"auc": null,
"mean_pos": 0.8255,
"mean_neg": null
},
"long_fact": {
"n": 25,
"auc": null,
"mean_pos": 0.7631,
"mean_neg": null
},
"multi_entity": {
"n": 25,
"auc": null,
"mean_pos": 0.7077,
"mean_neg": null
},
"multi_hop": {
"n": 25,
"auc": null,
"mean_pos": 0.7697,
"mean_neg": null
},
"near_miss": {
"n": 25,
"auc": null,
"mean_pos": 0.7452,
"mean_neg": null
},
"noise_context": {
"n": 25,
"auc": null,
"mean_pos": 0.6853,
"mean_neg": null
},
"unknown_attribute": {
"n": 25,
"auc": null,
"mean_pos": null,
"mean_neg": 0.7149
},
"update_conflict": {
"n": 25,
"auc": null,
"mean_pos": 0.7203,
"mean_neg": null
}
},
"best_threshold": {
"margin": 0.02,
"refused_unknown_pct": 40.0,
"false_refusal_pct": 3.43
},
"auc": {
"max": 0.6741,
"margin": 0.9413,
"zscore": 0.1767
}
}
answerability_threshold.json
+26
View File
@@ -0,0 +1,26 @@
{
"threshold": 0.007,
"train_auc": 0.7208,
"eval_auc": 0.7253,
"train_refused_unanswerable_pct": 11.5,
"train_false_refusal_pct": 3.58,
"eval_refused_unanswerable_pct": 24.0,
"eval_false_refusal_pct": 5.33,
"eval_refused_by_category": {
"unknown_attribute": 6
},
"eval_answerable_refused": {
"alias_paraphrase": [
2,
25
],
"multi_hop": [
0,
25
],
"near_miss": [
2,
25
]
}
}
attribute_head_training.json
+48
View File
@@ -0,0 +1,48 @@
{
"built_at": "2026-09-12T13:15:04Z",
"attributes": [
"主管姓名",
"入职年份",
"出生城市",
"办公城市",
"午餐偏好",
"咖啡口味",
"团队名称",
"客户名称",
"宿舍楼号",
"工位楼层",
"常住城市",
"常用编辑器",
"手机尾号",
"档案标识",
"紧急联系人姓氏",
"设备型号",
"课程名称",
"起床时间",
"运动习惯",
"通勤方式",
"邮箱域名",
"阅读工具",
"项目代号",
"默认语言"
],
"classes": 24,
"chance_pct": 4.17,
"train_queries": 1200,
"eval_queries": 300,
"attribute_accuracy_train_pct": 100.0,
"attribute_accuracy_eval_pct": 100.0,
"attribute_accuracy_eval_answerable_pct": 100.0,
"coverage_rule": {
"answerable_covered_pct": 100.0,
"abstention_covered_pct": 0.0,
"unknown_refusal_pct": 100.0,
"known_false_refusal_pct": 0.0,
"counts": {
"tp": 50,
"fn": 0,
"fp": 0,
"tn": 250
}
}
}
audit_refusal_both_ways.py
+49
View File
@@ -0,0 +1,49 @@
"""Validate the refusal detector in both directions.
Over-firing would inflate the known-question false-refusal rate; under-firing
would inflate the unknown-refusal rate. Both directions are printed with the
verbatim reply so the judgement can be checked by eye.
"""
from __future__ import annotations
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from V2_dpskw.eval_scoring import is_refusal, squash
from V2_dpskw.rescore_e2e import load_corpus, load_run
def main(corpus, run):
cases = load_corpus(Path(corpus), 25)
rows = load_run(Path(run))
unknown_refused, unknown_asserted = [], []
ans_refused, ans_refused_but_answered = [], []
for case, row in zip(cases, rows):
reply = row.get("reply", "")
refused = is_refusal(reply)
hit = [v for v in case["acceptable"] if v and squash(v) in squash(reply)]
if not case["answerable"]:
(unknown_refused if refused else unknown_asserted).append((case, row))
elif refused:
(ans_refused_but_answered if hit else ans_refused).append((case, row))
print(f"UNANSWERABLE cases where the detector says REFUSED ({len(unknown_refused)})")
for case, row in unknown_refused:
print(f" Q {case['query']}\n R {row.get('reply','')[:110]}")
print(f"\nUNANSWERABLE cases where the detector says ASSERTED ({len(unknown_asserted)})")
for case, row in unknown_asserted:
print(f" Q {case['query']}\n R {row.get('reply','')[:110]}")
print(f"\nANSWERABLE cases flagged as refusals ({len(ans_refused) + len(ans_refused_but_answered)})")
for tag, group in (("ALSO-MATCHED-ANCHOR (harmless)", ans_refused_but_answered), ("COUNTED AS FALSE REFUSAL", ans_refused)):
print(f" -- {tag}: {len(group)}")
for case, row in group:
print(f" Q {case['query']}\n R {row.get('reply','')[:110]}")
if __name__ == "__main__":
main(sys.argv[1], sys.argv[2])
audit_refusal_detection.py
+70
View File
@@ -0,0 +1,70 @@
"""Audit the unknown-refusal axis: are the 0/25 refusals real, or misdetected?
`ABSTENTION_MARKERS` is a fixed keyword list ("不知道", "没有记录", "无法确认",
"未找到", "不清楚", "没有相关信息"). A reply that says
"当前长期记忆中未包含相关信息，无法回答。" is a refusal in plain Chinese but
contains none of those exact strings, so it is scored as a confident wrong answer.
This prints every unknown_category reply so the true refusal rate can be counted
by hand, and groups them by which abstention signal (if any) they carry.
"""
from __future__ import annotations
import json
import re
import sys
from collections import Counter
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from V2_dpskw.eval_scoring import ABSTENTION_MARKERS
from V2_dpskw.rescore_e2e import load_corpus, load_run
# Refusal shapes actually observed in this project's replies, expressed as
# patterns rather than exact strings.
REFUSAL_PATTERNS = [
r"不知道",
r"没有(相关)?(记录|信息)",
r"未(包含|找到|记录|提及|提供)",
r"无法(回答|确认|确定|提供)",
r"不清楚",
r"缺乏(具体)?(上下文|信息)",
r"未(曾)?(在)?(长期)?记忆中(包含|出现|找到|记录)",
r"证据不足",
r"无法(从|根据)(记忆|证据)",
r"没有(足够)?(的)?证据",
]
REFUSAL_RE = re.compile("|".join(REFUSAL_PATTERNS))
def looks_like_refusal(reply: str) -> bool:
return bool(REFUSAL_RE.search(reply))
def main(corpus, run, category="unknown_attribute"):
cases = load_corpus(Path(corpus), 25)
rows = load_run(Path(run))
n = 0
markers_hit = 0
pattern_hit = 0
for case, row in zip(cases, rows):
if case["category"] != category:
continue
n += 1
reply = row.get("reply", "")
m = [x for x in ABSTENTION_MARKERS if x in reply]
p = bool(looks_like_refusal(reply))
markers_hit += int(bool(m))
pattern_hit += int(p)
print(f"[{n:02d}] markers={m or '-'} refusal_pattern={p}")
print(f" Q: {case['query']}")
print(f" R: {reply}")
print(f"\n{category}: {n} cases")
print(f" detected by keyword list : {markers_hit}/{n}")
print(f" detected by pattern set : {pattern_hit}/{n}")
if __name__ == "__main__":
main(sys.argv[1], sys.argv[2])
audit_router_dataset.py
+173
View File
@@ -0,0 +1,173 @@
"""Independent audit of a router dataset: category integrity and train/eval leakage.
The builder reports its own ``group_overlap`` check, but a benchmark that will be
quoted as evidence deserves an independent audit that streams the frozen files and
verifies, without trusting the builder's counters:
* per-family / per-category episode counts and the answerable-vs-unknown split;
* candidate width, positive-count and hop distributions per category (a category
that silently lost its positives would make every router look identical);
* **text-level leakage**: how many eval queries and eval candidate texts also
occur in the training split, and how many ``group_id`` values overlap.
Usage::
python -m V2_dpskw.audit_router_dataset ^
--train-file data/router_training_v5/train.jsonl ^
--eval-file data/router_training_v5/eval.jsonl ^
--output router_dataset_audit.json
"""
from __future__ import annotations
import argparse
import hashlib
import json
import sys
import time
from collections import Counter, defaultdict
from pathlib import Path
if __package__ in {None, ""}:
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
def _digest(text: str) -> bytes:
return hashlib.sha1(text.encode("utf-8", "replace")).digest()
def _scan(path: Path) -> dict:
per_category: dict[str, Counter] = defaultdict(Counter)
families = Counter()
categories = Counter()
group_ids: set[str] = set()
query_digests: set[bytes] = set()
candidate_digests: set[bytes] = set()
positive_by_query: dict[bytes, set[bytes]] = defaultdict(set)
queries_by_digest: dict[bytes, str] = {}
episodes = 0
started = time.perf_counter()
with path.open("r", encoding="utf-8") as handle:
for line in handle:
line = line.strip()
if not line:
continue
row = json.loads(line)
episodes += 1
family = str(row.get("family", "?") or "?")
metadata = row.get("metadata") if isinstance(row.get("metadata"), dict) else {}
category = str(metadata.get("category", "") or "(none)")
positives = row.get("positive_indices") or []
candidates = row.get("candidates") or []
families[family] += 1
categories[category] += 1
group_ids.add(str(row.get("group_id", "")))
bucket = per_category[category]
bucket["episodes"] += 1
bucket["answerable" if positives else "unknown"] += 1
bucket[f"candidates_{len(candidates)}"] += 1
bucket[f"positives_{min(len(positives), 4)}"] += 1
bucket[f"hop_{int(row.get('hop', 0))}"] += 1
bucket["candidate_slots"] += len(candidates)
bucket["positive_total"] += len(positives)
query = str(row.get("query", "")).strip()
query_digest = _digest(query) if query else None
if query_digest is not None:
query_digests.add(query_digest)
queries_by_digest.setdefault(query_digest, query)
texts: list[bytes] = []
for position, candidate in enumerate(candidates):
if not isinstance(candidate, dict):
continue
text = str(candidate.get("text", "")).strip()
if not text:
continue
digest = _digest(text)
candidate_digests.add(digest)
texts.append(digest)
# Evidence that a memorising router could exploit: the query text
# together with the exact positive evidence it should retrieve.
if query_digest is not None:
for position in positives:
if 0 <= int(position) < len(texts):
positive_by_query[query_digest].add(texts[int(position)])
return {
"episodes": episodes,
"families": dict(families),
"categories": dict(categories),
"per_category": {name: dict(counter) for name, counter in sorted(per_category.items())},
"group_ids": group_ids,
"query_digests": query_digests,
"candidate_digests": candidate_digests,
"positive_by_query": dict(positive_by_query),
"queries_by_digest": queries_by_digest,
"seconds": round(time.perf_counter() - started, 1),
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--train-file", default="data/router_training_v5/train.jsonl")
parser.add_argument("--eval-file", default="data/router_training_v5/eval.jsonl")
parser.add_argument("--output", default="router_dataset_audit.json")
args = parser.parse_args()
train_path = Path(args.train_file)
eval_path = Path(args.eval_file)
print(f"scanning {train_path} ...", flush=True)
train = _scan(train_path)
print(f"scanning {eval_path} ...", flush=True)
evaluation = _scan(eval_path)
report = {
"train": {key: value for key, value in train.items() if not key.endswith("_digests") and key not in {"group_ids", "positive_by_query", "queries_by_digest"}},
"eval": {key: value for key, value in evaluation.items() if not key.endswith("_digests") and key not in {"group_ids", "positive_by_query", "queries_by_digest"}},
"leakage": {},
}
overlapping_queries = evaluation["query_digests"] & train["query_digests"]
shared_evidence = [
digest for digest in overlapping_queries
if train["positive_by_query"].get(digest) and evaluation["positive_by_query"].get(digest)
and (train["positive_by_query"][digest] & evaluation["positive_by_query"][digest])
]
report["leakage"] = {
"group_id_overlap": len(train["group_ids"] & evaluation["group_ids"]),
"eval_queries_seen_in_train": len(overlapping_queries),
"eval_queries_total": len(evaluation["query_digests"]),
"query_overlap_rate": len(overlapping_queries) / max(1, len(evaluation["query_digests"])),
# The number that actually matters: a repeated query is only leakage when
# the same query also carries the same positive evidence in both splits.
"queries_with_shared_positive_evidence": len(shared_evidence),
"queries_with_shared_positive_evidence_examples": [
evaluation["queries_by_digest"].get(digest, "")[:60] for digest in shared_evidence[:5]
],
"eval_candidate_texts_seen_in_train": len(evaluation["candidate_digests"] & train["candidate_digests"]),
"eval_candidate_texts_total": len(evaluation["candidate_digests"]),
"candidate_overlap_rate": (
len(evaluation["candidate_digests"] & train["candidate_digests"])
/ max(1, len(evaluation["candidate_digests"]))
),
}
Path(args.output).write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps({"families": report["eval"]["families"], "categories": report["eval"]["categories"],
"leakage": report["leakage"]}, ensure_ascii=False, indent=2), flush=True)
print("\n| category | episodes | answerable | unknown | avg candidates | avg positives | hop dist |", flush=True)
print("|---|---:|---:|---:|---:|---:|---|", flush=True)
for name, block in report["eval"]["per_category"].items():
episodes = block.get("episodes", 0)
print("| {name} | {ep} | {ans} | {unk} | {cand:.2f} | {pos:.2f} | {hop} |".format(
name=name,
ep=episodes,
ans=block.get("answerable", 0),
unk=block.get("unknown", 0),
cand=block.get("candidate_slots", 0) / max(1, episodes),
pos=block.get("positive_total", 0) / max(1, episodes),
hop={key.split("_")[1]: value for key, value in block.items() if key.startswith("hop_")},
), flush=True)
print(f"\nwrote {args.output}", flush=True)
return 0
if __name__ == "__main__":
raise SystemExit(main())
audit_score_flips.py
+54
View File
@@ -0,0 +1,54 @@
"""Audit every case whose verdict flips when whitespace is ignored.
A flip is only legitimate if the anchor really is present modulo whitespace --
this prints the raw anchor and the raw reply so the difference can be eyeballed.
"""
from __future__ import annotations
import json
import sys
from collections import defaultdict
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from V2_dpskw.eval_scoring import squash
from V2_dpskw.rescore_e2e import load_corpus, load_run
def main(corpus, run, per_category=25):
cases = load_corpus(Path(corpus), per_category)
rows = load_run(Path(run))
assert len(rows) == len(cases)
flips = []
for row, case in zip(rows, cases):
reply = row.get("reply", "")
legacy = [v for v in case["acceptable"] if v and v.lower() in reply.lower()]
fixed = [v for v in case["acceptable"] if v and squash(v) in squash(reply)]
if bool(legacy) != bool(fixed):
flips.append((case, reply, legacy, fixed))
print(f"flips: {len(flips)}")
by_cat = defaultdict(int)
for case, reply, legacy, fixed in flips:
by_cat[case["category"]] += 1
print(dict(by_cat))
bad = 0
for case, reply, legacy, fixed in flips:
if not fixed:
print(f"!! now WRONG (was right): {case['category']} {case['query']}")
print(f" anchors={case['acceptable']!r}\n reply={reply!r}")
bad += 1
continue
anchor = fixed[0]
# show the neighbourhood of the anchor inside the reply
s_reply, s_anchor = squash(reply), squash(anchor)
pos = s_reply.find(s_anchor)
print(f"{case['category']:<18} anchor={anchor!r}")
print(f" reply = {reply!r}")
print(f" match@{pos} (raw anchor absent: {anchor not in reply})")
print(f"\nunexplained (now-wrong) flips: {bad}")
if __name__ == "__main__":
main(sys.argv[1], sys.argv[2])
bench_router_latency.py
+158
View File
@@ -0,0 +1,158 @@
"""Repeatable router latency/throughput benchmark with interleaved measurements.
The scorecard measured each router once, back to back. Two routers with the same
architecture and the same parameter count then differed by ~4% on single-query
latency, which is the same order as run-to-run drift — not evidence of a real
regression. This tool settles that by:
* loading every router once and measuring them **round-robin**, so slow drift in
machine state cannot favour one model over another;
* repeating the whole sweep and reporting min/median/spread per model;
* reporting the same axes the verdict uses (single query, batch-64, batch-256).
Usage::
python -m V2_dpskw.bench_router_latency ^
--run "deployed=V2-128:path.pt" --run "v2-128-v6=..." --rounds 5
"""
from __future__ import annotations
import argparse
import json
import statistics
import sys
import time
from pathlib import Path
import torch
if __package__ in {None, ""}:
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from V2_dpskw.eval_router_scorecard import load_router_any
from V2_dpskw.train_router_v5 import _resolve, load_feature_bank, stream_episode_tensors
def _time_call(fn, device: torch.device) -> float:
if device.type == "cuda":
torch.cuda.synchronize()
started = time.perf_counter()
fn()
if device.type == "cuda":
torch.cuda.synchronize()
return time.perf_counter() - started
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--run", action="append", required=True, help="LABEL=KIND:PATH")
parser.add_argument("--train-file", default="data/router_training_v6/train.jsonl")
parser.add_argument("--eval-file", default="data/router_training_v6/eval.jsonl")
parser.add_argument("--feature-cache", default=r"H:\Memory\nm_cache\nm_router_v6\feature_cache")
parser.add_argument("--model-path", default="qwen3_5_4b_natural_memory_v2")
parser.add_argument("--rounds", type=int, default=5)
parser.add_argument("--single-samples", type=int, default=200)
parser.add_argument("--warmup", type=int, default=30)
parser.add_argument("--output", default="router_latency_bench.json")
args = parser.parse_args()
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
train_path, eval_path = _resolve(args.train_file), _resolve(args.eval_file)
bank, lookup, _ = load_feature_bank(
_resolve(args.feature_cache), train_path=train_path, eval_path=eval_path,
model_path=args.model_path, max_key_tokens=256,
)
vectors = torch.from_numpy(bank)
data = stream_episode_tensors(eval_path, lookup, max_candidates=32)
routers = []
for spec in args.run:
label, rest = spec.split("=", 1)
path = Path(rest.split(":", 1)[1] if ":" in rest else rest)
router, arch, info = load_router_any(path)
routers.append((label, router.to(device).eval(), info["parameters"]))
print(json.dumps({"loaded": label, "parameters": info["parameters"],
"router_dim": arch["router_dim"]}), flush=True)
def single_query(model):
index = 0
def call():
nonlocal index
index = (index + 1) % 1000
q = vectors[data["query_indices"][index]].to(device=device, dtype=torch.float32).unsqueeze(0)
c = vectors[data["candidate_indices"][index]].to(device=device, dtype=torch.float32).unsqueeze(0)
with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=device.type == "cuda"):
model(q, c)
return call
def batched(model, batch):
idx = torch.arange(batch, dtype=torch.long)
q = vectors[data["query_indices"][idx]].to(device=device, dtype=torch.float32)
c = vectors[data["candidate_indices"][idx]].to(device=device, dtype=torch.float32)
def call():
with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=device.type == "cuda"):
model(q, c)
return call
samples: dict[str, dict[str, list[float]]] = {label: {"single": [], "b64": [], "b256": []} for label, _, _ in routers}
for label, model, _params in routers: # warm every model before timing any
call = single_query(model)
for _ in range(args.warmup):
call()
for batch, key in ((64, "b64"), (256, "b256")):
bcall = batched(model, batch)
for _ in range(3):
bcall()
for round_index in range(args.rounds):
for label, model, _params in routers: # round-robin: drift hits everyone
call = single_query(model)
elapsed = [_time_call(call, device) for _ in range(args.single_samples)]
samples[label]["single"].append(statistics.median(elapsed) * 1000)
for batch, key in ((64, "b64"), (256, "b256")):
bcall = batched(model, batch)
reps = 5
started = time.perf_counter()
for _ in range(reps):
bcall()
if device.type == "cuda":
torch.cuda.synchronize()
total = time.perf_counter() - started
samples[label][key].append(batch * reps / total)
print(json.dumps({"round": round_index + 1,
"single_ms": {label: round(min(samples[label]["single"]), 4) for label, _, _ in routers}}), flush=True)
print()
print("| router | params | single_query_ms (min/median/spread) | batch64_QPS | batch256_QPS |")
print("|---|---:|---|---:|---:|")
report = {}
for label, _model, params in routers:
s = samples[label]
row = {
"parameters": params,
"rounds": args.rounds,
"single_ms_min": min(s["single"]),
"single_ms_median": statistics.median(s["single"]),
"single_ms_max": max(s["single"]),
"single_ms_spread_pct": (max(s["single"]) - min(s["single"])) / min(s["single"]) * 100,
"batch64_qps_median": statistics.median(s["b64"]),
"batch256_qps_median": statistics.median(s["b256"]),
"batch64_qps_spread_pct": (max(s["b64"]) - min(s["b64"])) / max(1e-9, min(s["b64"])) * 100,
"batch256_qps_spread_pct": (max(s["b256"]) - min(s["b256"])) / max(1e-9, min(s["b256"])) * 100,
}
report[label] = row
print("| {label} | {params:,} | {a:.4f} / {b:.4f} / {c:.1f}% | {d:,.0f} | {e:,.0f} |".format(
label=label, params=params, a=row["single_ms_min"], b=row["single_ms_median"],
c=row["single_ms_spread_pct"], d=row["batch64_qps_median"], e=row["batch256_qps_median"]))
Path(args.output).write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
print(f"\nwrote {args.output}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
benchmark_compare_baselines_4b.py
+565
View File
@@ -0,0 +1,565 @@
"""Run a same-protocol 4B comparison: base Qwen, Chunk RAG, and Natural Memory.
The benchmark uses the same general-memory and real-repository query sets for
all systems. It reports answer correctness, refusal correctness, prompt/read
overhead, generation speed, and peak VRAM. Memory is kept in process RAM and
the GPU placement cap is shared by every model load.
"""
from __future__ import annotations
import argparse
import gc
import json
import math
import re
import time
from pathlib import Path
from statistics import mean
from typing import Any
import torch
from .benchmark_real_scale_memory_4b import (
_build_general_records,
_build_project_records,
_chat_generate,
_contains_answer,
_is_refusal,
_max_memory,
_native_cases,
_path,
_prepare_records,
_set_cuda_process_cap,
_source_files,
_sync,
)
from .qwen_integration import load_qwen_base, load_qwen_dynamic, load_tokenizer
from .stream_chat_qwen_memory import _chat_tensor
PROJECT_ROOT = Path(__file__).resolve().parent
TERM_PATTERN = re.compile(r"[\u4e00-\u9fff]|[A-Za-z0-9_\-]+")
def _terms(text: str) -> set[str]:
return set(TERM_PATTERN.findall(str(text).lower()))
def _vram_snapshot(device: torch.device) -> dict[str, float | None]:
if device.type != "cuda":
return {"allocated_gb": None, "reserved_gb": None}
return {
"allocated_gb": torch.cuda.memory_allocated(device) / 1024**3,
"reserved_gb": torch.cuda.memory_reserved(device) / 1024**3,
}
def _record_entry(record: dict[str, Any]) -> dict[str, Any]:
text = str(record.get("text", ""))
return {
"text": text,
"terms": _terms(text),
"entity": str(record.get("entity", "")),
"attribute": str(record.get("attribute", "")),
"value": str(record.get("value", "")),
}
def _build_rag_index(records: list[dict[str, Any]]) -> list[dict[str, Any]]:
return [_record_entry(record) for record in records if str(record.get("text", "")).strip()]
def _rag_retrieve(
index: list[dict[str, Any]],
query: str,
*,
top_k: int,
) -> tuple[list[dict[str, Any]], float]:
query_terms = _terms(query)
query_lower = query.strip().lower()
started = time.perf_counter()
scored: list[tuple[float, int, dict[str, Any]]] = []
for position, item in enumerate(index):
shared = len(query_terms.intersection(item["terms"]))
score = shared / math.sqrt(max(1, len(query_terms) * len(item["terms"])))
entity = item["entity"].strip().lower()
attribute = item["attribute"].strip().lower()
if len(entity) >= 4 and entity in query_lower:
score += 5.0
if attribute and attribute in query_lower:
score += 1.0
scored.append((score, -position, item))
scored.sort(key=lambda value: (value[0], value[1]), reverse=True)
elapsed_ms = (time.perf_counter() - started) * 1000.0
return [item for _, _, item in scored[: max(1, int(top_k))]], elapsed_ms
def _rag_prompt(query: str, records: list[dict[str, Any]]) -> str:
evidence = "\n".join(
f"[证据 {index}] {record['text']}"
for index, record in enumerate(records, 1)
)
return (
"以下是检索器返回的记忆证据。只能使用证据中明确出现的事实，"
"先核对实体、属性和已确认值；多个候选并存时不要把候选拼成一个事实；"
"新旧冲突时优先最新且来源更可靠的证据；找不到目标时请明确说不知道，"
"不要用相似用户的信息代替。\n"
"---记忆证据开始---\n"
+ evidence
+ "\n---记忆证据结束---\n问题："
+ query
)
@torch.inference_mode()
def _base_generate(
model: Any,
tokenizer: Any,
prompt: str,
device: torch.device,
*,
max_new_tokens: int,
) -> dict[str, Any]:
encoded = _chat_tensor(tokenizer, prompt)
encoded = {key: value.to(device) for key, value in encoded.items()}
if device.type == "cuda":
torch.cuda.reset_peak_memory_stats(device)
_sync(device)
started = time.perf_counter()
output = model.generate(
**encoded,
max_new_tokens=max_new_tokens,
do_sample=False,
use_cache=True,
pad_token_id=tokenizer.pad_token_id,
)
_sync(device)
elapsed = time.perf_counter() - started
response_ids = output[0, encoded["input_ids"].shape[1] :]
response = tokenizer.decode(
response_ids.detach().cpu().tolist(),
skip_special_tokens=True,
).strip()
generated_tokens = int(response_ids.numel())
row = {
"status": "ok",
"response": response,
"prompt_tokens": int(encoded["input_ids"].shape[1]),
"generated_tokens": generated_tokens,
"total_latency_s": elapsed,
"decode_tok_s": generated_tokens / max(elapsed, 1e-9),
}
row.update(_vram_snapshot(device))
del output
return row
def _quality_summary(rows: list[dict[str, Any]]) -> dict[str, Any]:
answerable = [row for row in rows if row["answerable"]]
unknown = [row for row in rows if not row["answerable"]]
return {
"cases": len(rows),
"answerable_cases": len(answerable),
"answerable_correct": sum(int(row["correct"]) for row in answerable),
"answerable_accuracy": sum(int(row["correct"]) for row in answerable) / max(1, len(answerable)),
"unknown_cases": len(unknown),
"unknown_correct": sum(int(row["correct"]) for row in unknown),
"unknown_refusal_accuracy": sum(int(row["correct"]) for row in unknown) / max(1, len(unknown)),
"mean_prompt_tokens": mean(row["prompt_tokens"] for row in rows) if rows else 0.0,
"mean_total_latency_ms": mean(row["total_latency_s"] for row in rows) * 1000.0 if rows else 0.0,
"mean_decode_tok_s": mean(row["decode_tok_s"] for row in rows) if rows else 0.0,
"peak_allocated_gb": max((row.get("allocated_gb") or 0.0 for row in rows), default=0.0),
"peak_reserved_gb": max((row.get("reserved_gb") or 0.0 for row in rows), default=0.0),
}
def _run_base_system(
model: Any,
tokenizer: Any,
rows: list[dict[str, Any]],
index: list[dict[str, Any]],
device: torch.device,
*,
mode: str,
rag_top_k: int,
max_new_tokens: int,
) -> dict[str, Any]:
output_rows: list[dict[str, Any]] = []
for row in rows:
query = str(row["query"])
retrieve_ms = 0.0
retrieved: list[dict[str, Any]] = []
if mode == "chunk_rag":
retrieved, retrieve_ms = _rag_retrieve(index, query, top_k=rag_top_k)
prompt = _rag_prompt(query, retrieved)
else:
prompt = query
generated = _base_generate(
model,
tokenizer,
prompt,
device,
max_new_tokens=max_new_tokens,
)
expected = str(row.get("expected", ""))
answerable = bool(row.get("answerable", True))
correct = (
_contains_answer(generated["response"], expected)
if answerable
else _is_refusal(generated["response"])
)
retrieved_target = answerable and any(
_contains_answer(item["value"], expected)
or _contains_answer(item["text"], expected)
for item in retrieved
)
output_rows.append(
{
"id": row.get("id", ""),
"query": query,
"expected": expected,
"answerable": answerable,
"correct": bool(correct),
"retrieved_target": bool(retrieved_target),
"retrieved_count": len(retrieved),
"retriever_ms": retrieve_ms,
"retrieved_values": [item["value"] for item in retrieved],
**generated,
}
)
summary = _quality_summary(output_rows)
summary["mean_retriever_ms"] = mean(row["retriever_ms"] for row in output_rows) if output_rows else 0.0
reader_values = [float(row["reader_ms"]) for row in output_rows if "reader_ms" in row]
if reader_values:
summary["mean_reader_ms"] = mean(reader_values)
summary["answerable_retrieval_recall"] = (
sum(int(row["retrieved_target"]) for row in output_rows if row["answerable"])
/ max(1, sum(int(row["answerable"]) for row in output_rows))
)
summary["rows"] = output_rows
return summary
def _run_natural_memory(
model: Any,
tokenizer: Any,
general_records: list[dict[str, Any]],
general_rows: list[dict[str, Any]],
project_records: list[dict[str, Any]],
project_rows: list[dict[str, Any]],
device: torch.device,
*,
generation_general: int,
generation_project: int,
max_new_tokens: int,
encode_batch_size: int,
) -> dict[str, Any]:
model.eval()
model.memory_config.memory_top_k_records = 2
model.memory_os_v2.bank.top_k_records = 2
model.reset_memory(batch_size=1, device=device)
output: dict[str, Any] = {}
prepared_general = _prepare_records(
model,
tokenizer,
general_records,
device,
batch_size=max(1, encode_batch_size),
)
model.memory_os_v2.write_batch(prepared_general)
general_rows = general_rows[: max(1, generation_general)]
general_output: list[dict[str, Any]] = []
for row in general_rows:
generated = _chat_generate(
model,
tokenizer,
str(row["query"]),
device,
max_new_tokens,
)
expected = str(row.get("expected", ""))
answerable = bool(row.get("answerable", True))
correct = (
_contains_answer(generated["response"], expected)
if answerable
else _is_refusal(generated["response"])
)
general_output.append(
{
"id": row.get("id", ""),
"query": row["query"],
"expected": expected,
"answerable": answerable,
"correct": bool(correct),
"prompt_tokens": int(generated["public_prompt_tokens"] + generated["prefix_tokens"]),
"generated_tokens": int(generated["generated_tokens"]),
"total_latency_s": float(generated["generation_seconds"]),
"decode_tok_s": int(generated["generated_tokens"]) / max(float(generated["generation_seconds"]), 1e-9),
"reader_ms": float(generated["memory_read_seconds"]) * 1000.0,
"prefix_used": bool(generated["prefix_used"]),
"prefix_tokens": int(generated["prefix_tokens"]),
"retrieved_values": list(generated["selected_values"]),
"retrieved_ids": list(generated["selected_record_ids"]),
"allocated_gb": _vram_snapshot(device)["allocated_gb"],
"reserved_gb": _vram_snapshot(device)["reserved_gb"],
"response": generated["response"],
}
)
output["general"] = _quality_summary(general_output)
output["general"]["rows"] = general_output
model.memory_os_v2 = model._new_memory_os_v2(model.memory.hidden_size)
model.reset_memory(batch_size=1, device=device)
prepared_project = _prepare_records(
model,
tokenizer,
project_records,
device,
batch_size=max(1, encode_batch_size),
)
model.memory_os_v2.write_batch(prepared_project)
project_rows = project_rows[: max(1, generation_project)]
project_output: list[dict[str, Any]] = []
for row in project_rows:
generated = _chat_generate(
model,
tokenizer,
str(row["query"]),
device,
max_new_tokens,
)
expected = str(row.get("expected", ""))
correct = _contains_answer(generated["response"], expected)
snapshot = _vram_snapshot(device)
project_output.append(
{
"id": row.get("id", ""),
"query": row["query"],
"expected": expected,
"answerable": True,
"correct": bool(correct),
"prompt_tokens": int(generated["public_prompt_tokens"] + generated["prefix_tokens"]),
"generated_tokens": int(generated["generated_tokens"]),
"total_latency_s": float(generated["generation_seconds"]),
"decode_tok_s": int(generated["generated_tokens"]) / max(float(generated["generation_seconds"]), 1e-9),
"reader_ms": float(generated["memory_read_seconds"]) * 1000.0,
"prefix_used": bool(generated["prefix_used"]),
"prefix_tokens": int(generated["prefix_tokens"]),
"retrieved_values": list(generated["selected_values"]),
"retrieved_ids": list(generated["selected_record_ids"]),
"allocated_gb": snapshot["allocated_gb"],
"reserved_gb": snapshot["reserved_gb"],
"response": generated["response"],
}
)
output["project"] = _quality_summary(project_output)
output["project"]["rows"] = project_output
return output
def _release(model: Any) -> None:
del model
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--base-model", default=r"H:\Memory")
parser.add_argument(
"--memory-model",
default=r"H:\Memory\V2_dpskw\qwen3_5_4b_natural_memory_v2",
)
parser.add_argument("--data-root", default=str(PROJECT_ROOT / "data"))
parser.add_argument("--output", default=str(PROJECT_ROOT / "baseline_compare_4b.json"))
parser.add_argument("--general-cases", type=int, default=640)
parser.add_argument("--project-records", type=int, default=8192)
parser.add_argument("--project-targets", type=int, default=256)
parser.add_argument("--generation-general", type=int, default=128)
parser.add_argument("--generation-project", type=int, default=64)
parser.add_argument("--rag-top-k", type=int, default=2)
parser.add_argument("--max-new-tokens", type=int, default=32)
parser.add_argument("--encode-batch-size", type=int, default=16)
parser.add_argument("--gpu-memory-gb", type=float, default=10.0)
parser.add_argument("--no-4bit", action="store_true")
parser.add_argument(
"--reuse-original-report",
default=None,
help="reuse original_qwen results from an earlier report and rerun only Natural Memory",
)
args = parser.parse_args()
_set_cuda_process_cap(args.gpu_memory_gb)
tokenizer = load_tokenizer(_path(args.base_model))
general_cases = _native_cases(_path(args.data_root), max(1, int(args.general_cases)))
general_raw_records, general_queries = _build_general_records(general_cases)
general_generation_rows = [
row for row in general_queries if row["split"] == "eval"
][: max(1, int(args.generation_general))]
project_files = _source_files()
project_raw_records, project_queries, project_meta = _build_project_records(
project_files,
record_count=max(128, int(args.project_records)),
target_count=max(1, int(args.project_targets)),
chunk_tokens=512,
)
project_generation_rows = [
{"id": f"project-{index}", "query": row["query"], "expected": row["expected"], "answerable": True}
for index, row in enumerate(project_queries[: max(1, int(args.generation_project))])
]
general_index = _build_rag_index(general_raw_records)
project_index = _build_rag_index(project_raw_records)
use_4bit = not args.no_4bit
max_memory = _max_memory(args.gpu_memory_gb)
reused_original_report = None
if args.reuse_original_report:
reused_original_report = json.loads(
_path(args.reuse_original_report).read_text(encoding="utf-8")
)
original_system = reused_original_report.get("systems", {}).get("original_qwen")
if not isinstance(original_system, dict):
raise ValueError("reuse report does not contain systems.original_qwen")
base_load_vram = original_system.get("load_vram")
base_general = original_system["general"]
base_project = original_system["project"]
print("reusing original Qwen results")
else:
print("loading original Qwen 4B")
base = load_qwen_base(
_path(args.base_model),
load_in_4bit=use_4bit,
max_memory=max_memory,
)
base.eval()
base_device = base.get_input_embeddings().weight.device
base_load_vram = _vram_snapshot(base_device)
base_general = {
"no_memory": _run_base_system(
base,
tokenizer,
general_generation_rows,
general_index,
base_device,
mode="no_memory",
rag_top_k=args.rag_top_k,
max_new_tokens=max(1, args.max_new_tokens),
),
"chunk_rag": _run_base_system(
base,
tokenizer,
general_generation_rows,
general_index,
base_device,
mode="chunk_rag",
rag_top_k=max(1, args.rag_top_k),
max_new_tokens=max(1, args.max_new_tokens),
),
}
base_project = {
"no_memory": _run_base_system(
base,
tokenizer,
project_generation_rows,
project_index,
base_device,
mode="no_memory",
rag_top_k=args.rag_top_k,
max_new_tokens=max(1, args.max_new_tokens),
),
"chunk_rag": _run_base_system(
base,
tokenizer,
project_generation_rows,
project_index,
base_device,
mode="chunk_rag",
rag_top_k=max(1, args.rag_top_k),
max_new_tokens=max(1, args.max_new_tokens),
),
}
_release(base)
# Drop the caller's reference as well. Otherwise the original Qwen
# remains resident while Natural Memory is loaded below, making the
# latter's VRAM measurement include two complete 4B models.
base = None
print("loading Natural Memory 4B")
memory = load_qwen_dynamic(
_path(args.memory_model),
load_in_4bit=use_4bit,
max_memory=max_memory,
)
memory_device = memory._find_layer_device()
natural = _run_natural_memory(
memory,
tokenizer,
general_raw_records,
general_generation_rows,
project_raw_records,
project_generation_rows,
memory_device,
generation_general=args.generation_general,
generation_project=args.generation_project,
max_new_tokens=max(1, args.max_new_tokens),
encode_batch_size=max(1, args.encode_batch_size),
)
natural_load_vram = _vram_snapshot(memory_device)
_release(memory)
report = {
"benchmark": "baseline_compare_4b",
"base_model": str(_path(args.base_model)),
"memory_model": str(_path(args.memory_model)),
"quantization": "4bit_nf4" if use_4bit else "none",
"gpu_memory_cap_gb": float(args.gpu_memory_gb),
"general_cases": len(general_cases),
"general_records": len(general_raw_records),
"project": project_meta,
"protocol": {
"generation_general": len(general_generation_rows),
"generation_project": len(project_generation_rows),
"rag_top_k": int(args.rag_top_k),
"max_new_tokens": int(args.max_new_tokens),
"same_tokenizer": True,
"same_sampling": "greedy",
"memory_storage": "process_ram; selected Natural Memory records promoted to bounded GPU cache",
"reused_original_report": str(_path(args.reuse_original_report)) if args.reuse_original_report else None,
},
"systems": {
"original_qwen": {
"load_vram": base_load_vram,
"general": base_general,
"project": base_project,
},
"natural_memory": {
"load_vram": natural_load_vram,
"general": natural["general"],
"project": natural["project"],
},
},
"limitations": [
"Chunk RAG uses a CPU lexical candidate index with exact entity/attribute bonuses; it is a transparent baseline, not a hosted embedding service.",
"The general corpus is the local native-memory benchmark; the project corpus is the current repository source and documentation.",
"Latency for Natural Memory is split into reader_ms and generation is measured by the existing model wrapper; this first comparison prioritizes correctness and VRAM.",
],
}
output = _path(args.output)
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
compact = {
name: {
"general": {key: value for key, value in system["general"].items() if key != "rows"},
"project": {key: value for key, value in system["project"].items() if key != "rows"},
}
for name, system in report["systems"].items()
}
print(json.dumps(compact, ensure_ascii=False, indent=2))
print(f"saved={output}")
if __name__ == "__main__":
main()
benchmark_dirty_real_corpus_4b.py
+1096
View File
File diff suppressed because it is too large Load Diff
benchmark_dirty_strong_rag_4b.py
+231
View File
@@ -0,0 +1,231 @@
"""Evaluate the production-style local dense+rereank baseline on dirty data.
This is the companion baseline for ``benchmark_dirty_real_corpus_4b``. It
uses the same real repository corpus, the same user-dialogue slice, the same
queries, tokenizer, greedy decoding and GPU cap. Retrieval is CPU-only:
local BERT embeddings provide recall and a bounded transparent reranker uses
lexical/entity/attribute signals. It is stronger than lexical Chunk RAG,
but is explicitly not presented as a trained public cross-encoder.
"""
from __future__ import annotations
import argparse
import gc
import json
import time
from collections import Counter
from pathlib import Path
from statistics import mean
from typing import Any
import torch
from .benchmark_compare_baselines_4b import (
_base_generate,
_rag_prompt,
_vram_snapshot,
)
from .benchmark_dirty_real_corpus_4b import (
_build_all_records,
_dirty_source_files,
_is_refusal_strict,
_path,
_quality_pass,
_retrieval_hit,
_summarize,
)
from .benchmark_real_scale_memory_4b import _max_memory, _set_cuda_process_cap
from .strong_rag_baseline import LocalEmbeddingReranker, _resolve_local_encoder
from .qwen_integration import load_qwen_base, load_tokenizer
PROJECT_ROOT = Path(__file__).resolve().parent
def _run_dirty(
model: Any,
tokenizer: Any,
rows: list[dict[str, Any]],
index: LocalEmbeddingReranker,
device: torch.device,
*,
top_k: int,
max_new_tokens: int,
) -> dict[str, Any]:
output_rows: list[dict[str, Any]] = []
for row_index, row in enumerate(rows, 1):
if row_index == 1 or row_index % 32 == 0 or row_index == len(rows):
print(f"Strong RAG generation: {row_index}/{len(rows)}", flush=True)
query = str(row["query"])
retrieved, retrieve_ms = index.retrieve(query, top_k=top_k)
prompt = _rag_prompt(query, retrieved)
generated = _base_generate(
model,
tokenizer,
prompt,
device,
max_new_tokens=max_new_tokens,
)
selected = [
{
"value": item.get("value", ""),
"text_preview": item.get("text", ""),
"entity": item.get("entity", ""),
"attribute": item.get("attribute", ""),
}
for item in retrieved
]
response = str(generated.get("response", ""))
output_rows.append(
{
**row,
"response": response,
"correct": _quality_pass(response, row),
"retrieval_target_found": _retrieval_hit(selected, row),
"retrieved_values": [item.get("value", "") for item in retrieved],
"retrieved_ids": [],
"selected_records": selected,
"prefix_used": bool(retrieved),
"prefix_tokens": int(generated.get("prompt_tokens", 0)),
"reader_ms": float(retrieve_ms),
"retriever_ms": float(retrieve_ms),
"prompt_tokens": int(generated.get("prompt_tokens", 0)),
"generated_tokens": int(generated.get("generated_tokens", 0)),
"total_latency_s": float(generated.get("total_latency_s", 0.0)),
"decode_tok_s": float(generated.get("decode_tok_s", 0.0)),
"allocated_gb": generated.get("allocated_gb"),
"reserved_gb": generated.get("reserved_gb"),
}
)
return _summarize(output_rows, include_reader=False)
def _release(model: Any) -> None:
del model
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
def _compact(summary: dict[str, Any]) -> dict[str, Any]:
return {key: value for key, value in summary.items() if key != "rows"}
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--base-model", default=r"H:\Memory")
parser.add_argument("--output", default=str(PROJECT_ROOT / "dirty_strong_rag_compare_4b.json"))
parser.add_argument("--project-records", type=int, default=8192)
parser.add_argument("--project-targets", type=int, default=96)
parser.add_argument("--top-k", type=int, default=4)
parser.add_argument("--candidate-k", type=int, default=64)
parser.add_argument("--embedding-batch-size", type=int, default=16)
parser.add_argument("--embedding-max-length", type=int, default=384)
parser.add_argument("--embedding-model", default=None)
parser.add_argument("--max-new-tokens", type=int, default=64)
parser.add_argument("--gpu-memory-gb", type=float, default=10.0)
parser.add_argument("--no-4bit", action="store_true")
args = parser.parse_args()
_set_cuda_process_cap(args.gpu_memory_gb)
tokenizer = load_tokenizer(_path(args.base_model))
files = _dirty_source_files()
user_records, user_rows, code_records, code_rows, corpus_meta = _build_all_records(
files,
project_records=max(128, int(args.project_records)),
project_targets=max(1, int(args.project_targets)),
)
all_records = user_records + code_records
all_rows = user_rows + code_rows
print(
f"dirty corpus: files={len(files)} real_chunks={corpus_meta['stored_real_chunk_records']} "
f"semantic_targets={corpus_meta['semantic_target_records']} user_records={len(user_records)} "
f"queries={len(all_rows)}"
)
encoder_path = _resolve_local_encoder(args.embedding_model)
print(f"loading local embedding model on CPU: {encoder_path}")
embedder = LocalEmbeddingReranker(
encoder_path,
max_length=args.embedding_max_length,
batch_size=args.embedding_batch_size,
candidate_k=args.candidate_k,
)
started = time.perf_counter()
embedder.add(all_records)
index_seconds = time.perf_counter() - started
print(f"dense_index_seconds={index_seconds:.3f}")
print("loading original Qwen3.5 4B for strong-RAG generation")
base = load_qwen_base(
_path(args.base_model),
load_in_4bit=not args.no_4bit,
max_memory=_max_memory(args.gpu_memory_gb),
)
base.eval()
device = base.get_input_embeddings().weight.device
load_vram = _vram_snapshot(device)
result = _run_dirty(
base,
tokenizer,
all_rows,
embedder,
device,
top_k=max(1, args.top_k),
max_new_tokens=max(1, args.max_new_tokens),
)
_release(base)
report = {
"benchmark": "dirty_strong_rag_compare_4b",
"created_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
"base_model": str(_path(args.base_model)),
"embedding_model": str(encoder_path),
"quantization": "4bit_nf4" if not args.no_4bit else "none",
"gpu_memory_cap_gb": float(args.gpu_memory_gb),
"corpus": corpus_meta,
"query_counts": {
"all": len(all_rows),
"user_dialogue": len(user_rows),
"real_repository": len(code_rows),
"answerable": sum(int(row["answerable"]) for row in all_rows),
"unknown": sum(int(not row["answerable"]) for row in all_rows),
"by_category": dict(Counter(str(row["category"]) for row in all_rows)),
},
"protocol": {
"same_tokenizer": True,
"same_sampling": "greedy",
"top_k": int(args.top_k),
"candidate_k": int(args.candidate_k),
"embedding_device": "cpu",
"embedding_max_length": int(args.embedding_max_length),
"embedding_batch_size": int(args.embedding_batch_size),
"max_new_tokens": int(args.max_new_tokens),
"baseline_note": "local BERT dense retrieval plus fixed transparent reranker; not a trained public cross-encoder",
},
"index_build": {
"records": len(all_records),
"indexed_records": len(embedder.records),
"seconds": index_seconds,
},
"system": {
"load_vram": load_vram,
"all": result,
},
"limitations": [
"The embedding encoder is local bert-base-chinese and CPU-only.",
"The reranker is a fixed transparent feature reranker, not a trained public cross-encoder.",
"The same strict answer-anchor evaluator is used for Natural Memory comparison.",
"Embedding/index build time is reported separately from per-query retrieval and generation.",
],
}
output = _path(args.output)
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps({"strong_rag": _compact(result), "index_build": report["index_build"]}, ensure_ascii=False, indent=2))
print(f"saved={output}")
if __name__ == "__main__":
main()
benchmark_embedded_memory_v2.py
+203
View File
@@ -0,0 +1,203 @@
"""Low-pressure end-to-end test for the embedded Natural Memory v2 path.
This benchmark intentionally uses only the model package's third safetensors
memory shard. It does not create SQLite files or exercise disk paging. The
long-context cases lower the temporary KV budget so the test measures the
model-owned archive/read path without asking a 12 GiB GPU to hold a huge KV.
"""
from __future__ import annotations
import argparse
import gc
import json
import random
import time
from pathlib import Path
from typing import Any
import torch
from .qwen_integration import load_qwen_dynamic, load_tokenizer
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--model-path",
default=r"H:\Memory\V2_dpskw\qwen3_5_4b_natural_memory_v2",
)
parser.add_argument("--lengths", default="4096,8192,16384,32768")
parser.add_argument("--kv-budget", type=int, default=2048)
parser.add_argument("--chunk-tokens", type=int, default=1024)
parser.add_argument("--max-new-tokens", type=int, default=8)
parser.add_argument("--no-4bit", action="store_true")
parser.add_argument(
"--output",
default=r"H:\Memory\V2_dpskw\embedded_memory_v2_long_benchmark.json",
)
return parser.parse_args()
def build_prompt(tokenizer: Any, target_tokens: int, seed: int) -> tuple[str, str, int]:
rng = random.Random(seed + target_tokens)
answer = f"EMBEDDED-LONG-{target_tokens}-{rng.randrange(100000, 999999)}"
needle = f"长期记忆锚点：唯一编号是 {answer}。"
filler = (
"这是长文本记忆压力测试中的普通背景段落，包含项目说明、日期、日志和无关备注。"
"这些内容不是问题答案，读取时应保留原文但忽略干扰。"
)
chunks: list[str] = []
while len(tokenizer(" ".join(chunks + [filler, needle]), add_special_tokens=False)["input_ids"]) < target_tokens:
chunks.append(filler)
# Keep the needle safely inside the archived prefix even in the smallest
# case, so a pass must come from memory rather than the retained window.
pivot = max(1, len(chunks) // 3)
material = " ".join(chunks[:pivot] + [needle] + chunks[pivot:])
prompt = (
"请阅读下面的长材料，回答末尾问题，只输出编号，不要解释。\n"
"---开始材料---\n"
f"{material}\n"
"---结束材料---\n"
"问题：长期记忆锚点的唯一编号是什么？"
)
prompt_tokens = len(tokenizer(prompt, add_special_tokens=False)["input_ids"])
return prompt, answer, prompt_tokens
def chat_inputs(tokenizer: Any, text: str, device: torch.device) -> dict[str, torch.Tensor]:
encoded = tokenizer.apply_chat_template(
[{"role": "user", "content": text}],
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
enable_thinking=False,
)
return {
key: value.to(device)
for key, value in encoded.items()
if isinstance(value, torch.Tensor)
}
def run_case(model: Any, tokenizer: Any, target_tokens: int, args: argparse.Namespace) -> dict[str, Any]:
device = model._find_layer_device()
model.reset_memory(batch_size=1, device=device)
assert model.memory_os_v2 is not None
model.memory_os_v2.kv_budget.max_tokens = int(args.kv_budget)
prompt, answer, prompt_tokens = build_prompt(tokenizer, target_tokens, 20260904)
encoded = chat_inputs(tokenizer, prompt, device)
query_text = "长期记忆锚点的唯一编号"
query = tokenizer(query_text, add_special_tokens=False, return_tensors="pt")
query_ids = query["input_ids"].to(device)
query_mask = query.get("attention_mask")
if query_mask is None:
query_mask = torch.ones_like(query_ids)
query_mask = query_mask.to(device)
started = time.perf_counter()
with torch.inference_mode():
output = model.generate(
**encoded,
max_new_tokens=args.max_new_tokens,
do_sample=False,
use_cache=True,
update_memory=False,
pad_token_id=tokenizer.pad_token_id,
memory_query_input_ids=query_ids,
memory_query_attention_mask=query_mask,
memory_query_text=query_text,
)
if device.type == "cuda":
torch.cuda.synchronize(device)
elapsed = time.perf_counter() - started
response_ids = output[0, encoded["input_ids"].shape[1] :]
response = tokenizer.decode(response_ids.detach().cpu().tolist(), skip_special_tokens=True).strip()
stats = model.memory_v2_stats()
records = model.memory_os_v2.bank.records
archived_text_hit = any(
answer in tokenizer.decode(record.token_ids.tolist(), skip_special_tokens=True)
for record in records.values()
if record.memory_type == "context_chunk" and isinstance(record.token_ids, torch.Tensor)
)
query_key = model._encode_model_key(query_ids, query_mask)[0]
retrieved, decision = model.read_hierarchical_memory(
query_key,
query_text=query_text,
query_token_ids=query_ids[0],
top_k_pages=model.memory_config.memory_top_k_pages,
top_k_records=model.memory_config.memory_top_k_records,
max_hops=model.memory_config.memory_max_hops,
)
retrieved_hit = any(
answer in tokenizer.decode(record.token_ids.tolist(), skip_special_tokens=True)
for record in retrieved
if isinstance(record.token_ids, torch.Tensor)
)
return {
"target_tokens": target_tokens,
"prompt_tokens": prompt_tokens,
"kv_budget_tokens": args.kv_budget,
"chunk_tokens": args.chunk_tokens,
"response": response,
"expected": answer,
"generation_hit": answer in response,
"archived_text_hit": archived_text_hit,
"retrieved_text_hit": retrieved_hit,
"retrieved_records": len(retrieved),
"router_stop_reason": decision.stop_reason,
"router_hop_count": decision.hop_count,
"seconds": elapsed,
"records": stats.get("records", 0),
"pages": stats.get("pages", 0),
"gpu_cache_records": stats.get("gpu_cache_records", 0),
"gpu_cache_tokens": stats.get("gpu_cache_tokens", 0),
"gpu_cache_device": stats.get("gpu_cache_device", "none"),
"passed": bool(archived_text_hit and retrieved_hit),
}
def main() -> None:
args = parse_args()
lengths = [int(item.strip()) for item in args.lengths.split(",") if item.strip()]
tokenizer = load_tokenizer(args.model_path)
model = load_qwen_dynamic(args.model_path, load_in_4bit=not args.no_4bit)
model.eval()
model.memory_config.context_chunk_tokens = int(args.chunk_tokens)
report: dict[str, Any] = {
"benchmark": "Natural Memory v2 embedded third-shard long-memory test",
"storage_mode": model.memory_config.memory_storage_mode,
"tier_store_enabled": bool(model.memory_os_v2 and model.memory_os_v2.bank.tier_store is not None),
"model_path": str(Path(args.model_path).resolve()),
"lengths": lengths,
"quantization": "4bit_nf4" if not args.no_4bit else "none",
"kv_budget_tokens": args.kv_budget,
"chunk_tokens": args.chunk_tokens,
"rows": [],
}
try:
if report["storage_mode"] != "embedded" or report["tier_store_enabled"]:
raise RuntimeError("embedded benchmark requires memory_storage_mode=embedded and no tier store")
for target_tokens in lengths:
row = run_case(model, tokenizer, target_tokens, args)
report["rows"].append(row)
print(json.dumps(row, ensure_ascii=False))
finally:
model.reset_memory(batch_size=1, device=model._find_layer_device())
model.close_memory_storage()
del model
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
report["passed_cases"] = sum(bool(row["passed"]) for row in report["rows"])
report["total_cases"] = len(report["rows"])
report["all_passed"] = report["passed_cases"] == report["total_cases"]
output = Path(args.output)
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps(report, ensure_ascii=False, indent=2))
print(f"saved={output}")
if __name__ == "__main__":
main()
benchmark_kv_offload.py
+128
View File
@@ -0,0 +1,128 @@
"""Smoke-test the model-owned KV offload and long-context compaction paths.
The test intentionally uses the real Qwen package. It verifies that
Transformers' CPU-backed DynamicCache can generate through Qwen3.5's hybrid
linear/full-attention stack and that an over-budget prompt is archived into
Natural Memory before only the recent hot window is passed to generation.
"""
from __future__ import annotations
import argparse
import json
from pathlib import Path
import torch
if __package__ in {None, ""}:
import sys
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from V2_dpskw.qwen_integration import load_qwen_dynamic, load_tokenizer
from V2_dpskw.stream_chat_qwen_memory import _chat_tensor
@torch.inference_mode()
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--model-path",
default=r"H:\Memory\V2_dpskw\qwen3_5_4b_natural_memory_v2",
)
parser.add_argument(
"--report",
default=r"H:\Memory\V2_dpskw\kv_offload_compaction_test.json",
)
parser.add_argument("--compact-hot-tokens", type=int, default=32)
parser.add_argument("--max-new-tokens", type=int, default=4)
parser.add_argument("--no-4bit", action="store_true")
args = parser.parse_args()
model_path = Path(args.model_path)
tokenizer = load_tokenizer(model_path)
model = load_qwen_dynamic(model_path, load_in_4bit=not args.no_4bit)
model.eval()
model.memory_config.kv_offload = True
device = model._find_layer_device()
answer_inputs = {
key: value.to(device)
for key, value in _chat_tensor(tokenizer, "请用一句话说明你支持什么。").items()
}
generated = model.generate(
**answer_inputs,
max_new_tokens=args.max_new_tokens,
do_sample=False,
update_memory=False,
use_cache=True,
pad_token_id=tokenizer.pad_token_id,
)
answer = tokenizer.decode(generated[0].detach().cpu().tolist(), skip_special_tokens=True).strip()
if model.memory_os_v2 is None:
raise RuntimeError("the selected package does not contain Natural Memory v2")
model.memory_os_v2.kv_budget.max_tokens = int(args.compact_hot_tokens)
model.memory_os_v2.kv_budget.keep_recent_tokens = min(
model.memory_os_v2.kv_budget.keep_recent_tokens,
model.memory_os_v2.kv_budget.max_tokens,
)
model.memory_config.context_chunk_tokens = max(4, min(16, args.compact_hot_tokens // 2))
long_text = " ".join(
[
"历史上下文片段用于验证 Natural Memory 的自动分页压缩。",
"这段内容应该被写入长期上下文页面，而不是继续占用当前热 KV。",
]
* 12
)
encoded = tokenizer(long_text, add_special_tokens=False, return_tensors="pt")
long_ids = encoded["input_ids"].to(device)
long_mask = encoded.get("attention_mask")
if long_mask is None:
long_mask = torch.ones_like(long_ids)
long_mask = long_mask.to(device)
before = model.memory_v2_stats()
generated_long = model.generate(
input_ids=long_ids,
attention_mask=long_mask,
max_new_tokens=1,
do_sample=False,
update_memory=False,
use_cache=True,
pad_token_id=tokenizer.pad_token_id,
)
compaction = dict(model.runtime.context_compaction or {})
after = model.memory_v2_stats()
compaction_pass = bool(
compaction["compacted"]
and compaction["archived_records"] > 0
and max(compaction["retained_tokens"]) <= args.compact_hot_tokens
and int(compaction["retained_tokens"][0]) <= args.compact_hot_tokens
and after["active_records"] >= before["active_records"] + compaction["archived_records"]
)
report = {
"model_path": str(model_path),
"transformers_cache": "DynamicCache(offloading=True)",
"kv_offload_pass": True,
"answer": answer,
"compaction": compaction,
"compaction_pass": compaction_pass,
"long_generation_tokens": int(generated_long.shape[1]),
"memory_before": before,
"memory_after": after,
"device": str(device),
"cuda_memory_allocated_bytes": (
int(torch.cuda.memory_allocated(device)) if torch.cuda.is_available() else 0
),
}
report_path = Path(args.report)
report_path.parent.mkdir(parents=True, exist_ok=True)
report_path.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps(report, ensure_ascii=False, indent=2))
if not compaction_pass:
raise SystemExit("context compaction verification failed")
if __name__ == "__main__":
main()
benchmark_large_memory_4b.py
+310
View File
@@ -0,0 +1,310 @@
"""Stress-test Natural Memory retrieval on a large in-memory 4B library.
This is a routing/reader benchmark, not a language-generation benchmark. It
builds a multi-million-token address space with sparse project and personal
facts, then evaluates paraphrased answerable queries, explicit unknown queries,
and versioned conflict updates. Filler records stay in process RAM and only
the selected records are promoted to the model's bounded hot cache.
"""
from __future__ import annotations
import argparse
import gc
import json
import random
import time
from pathlib import Path
from typing import Any
import torch
import torch.nn.functional as F
from .qwen_integration import load_qwen_dynamic, load_tokenizer
PROJECT_ROOT = Path(__file__).resolve().parent
def _path(value: str | Path) -> Path:
path = Path(value)
return path if path.is_absolute() or path.exists() else PROJECT_ROOT / path
def _max_memory(gpu_memory_gb: float) -> dict[Any, str] | None:
if gpu_memory_gb <= 0.0 or not torch.cuda.is_available():
return None
return {0: f"{gpu_memory_gb:.1f}GiB", "cpu": "64GiB"}
def _set_cuda_process_cap(gpu_memory_gb: float) -> None:
if gpu_memory_gb <= 0.0 or not torch.cuda.is_available():
return
total = torch.cuda.get_device_properties(0).total_memory
fraction = min(0.95, max(0.05, gpu_memory_gb * 1024**3 / total))
torch.cuda.set_per_process_memory_fraction(fraction, device=0)
def _sync(device: torch.device) -> None:
if device.type == "cuda":
torch.cuda.synchronize(device)
def _make_facts(count: int) -> list[dict[str, str]]:
facts: list[dict[str, str]] = []
for index in range(count):
code = f"PROJ-{index:04d}"
if index % 4 == 0:
text = f"项目 {code} 的部署区域是 region-{index % 7}，责任服务是 service_{index:04d}。"
query = f"请查项目 {code} 的部署区域，直接给出 region。"
elif index % 4 == 1:
text = f"项目 {code} 的回滚命令是 rollback_{index:04d}，发布窗口为周三。"
query = f"项目 {code} 出问题时应该执行哪条回滚命令？"
elif index % 4 == 2:
text = f"个人偏好 {code}：通知时间设为 {8 + index % 5}:30，提醒渠道为 email。"
query = f"我在 {code} 里设置的通知时间是多少？"
else:
text = f"仓库 {code} 的关键文件是 src/module_{index:04d}.py，入口函数为 run_{index:04d}。"
query = f"仓库 {code} 的关键入口函数叫什么？"
facts.append({"code": code, "text": text, "query": query})
return facts
def _encode_texts(model: Any, tokenizer: Any, texts: list[str], device: torch.device) -> torch.Tensor:
encoded = tokenizer(
texts,
add_special_tokens=False,
padding=True,
return_tensors="pt",
)
input_ids = encoded["input_ids"].to(device)
mask = encoded["attention_mask"].to(device)
return model._encode_model_key(input_ids, mask).detach().cpu()
@torch.inference_mode()
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--base-model", default=r"H:\Memory")
parser.add_argument("--memory-model", default=r"H:\Memory\V2_dpskw\qwen3_5_4b_natural_memory_v2")
parser.add_argument("--output", default=r"H:\Memory\V2_dpskw\large_memory_4b.json")
parser.add_argument("--library-tokens", type=int, default=4_194_304)
parser.add_argument("--chunk-tokens", type=int, default=512)
parser.add_argument("--target-count", type=int, default=64)
parser.add_argument("--unknown-count", type=int, default=16)
parser.add_argument("--gpu-memory-gb", type=float, default=10.0)
parser.add_argument("--no-4bit", action="store_true")
args = parser.parse_args()
_set_cuda_process_cap(args.gpu_memory_gb)
tokenizer = load_tokenizer(_path(args.base_model))
use_4bit = not args.no_4bit
max_memory = _max_memory(args.gpu_memory_gb)
print("loading Natural Memory 4B")
model = load_qwen_dynamic(
_path(args.memory_model),
load_in_4bit=use_4bit,
max_memory=max_memory,
)
model.eval()
model.memory_config.memory_top_k_records = 2
model.memory_os_v2.bank.top_k_records = 2
device = model._find_layer_device()
model.reset_memory(batch_size=1, device=device)
target_count = max(1, min(int(args.target_count), 256))
unknown_count = max(1, min(int(args.unknown_count), 128))
facts = _make_facts(target_count)
chunk_count = max(1, int(args.library_tokens) // max(1, int(args.chunk_tokens)))
positions = random.Random(20260905).sample(range(chunk_count), min(target_count, chunk_count))
position_to_fact = dict(zip(positions, facts))
target_keys: dict[str, torch.Tensor] = {}
for start in range(0, len(facts), 16):
batch = facts[start : start + 16]
for fact, key in zip(batch, _encode_texts(model, tokenizer, [item["text"] for item in batch], device)):
target_keys[fact["code"]] = F.normalize(key.float(), dim=0)
generator = torch.Generator(device="cpu").manual_seed(20260905)
records: list[dict[str, Any]] = []
for chunk_index in range(chunk_count):
fact = position_to_fact.get(chunk_index)
if fact is None:
key = F.normalize(torch.randn(model.memory.hidden_size, generator=generator), dim=0)
records.append(
{
"text": f"library_chunk:{chunk_index}",
"key": key,
"summary": key,
"semantic_key": key,
"memory_type": "context_chunk",
"importance": 0.4,
"confidence": 0.8,
"source": "large_memory_benchmark",
"evidence": [f"library_chunk:{chunk_index}"],
"trusted": True,
"force": True,
}
)
continue
key = target_keys[fact["code"]]
token_ids = torch.tensor(tokenizer(fact["text"], add_special_tokens=False)["input_ids"], dtype=torch.long)
records.append(
{
"text": fact["text"],
"key": key,
"summary": key,
"semantic_key": key,
"memory_type": "fact",
"entity": fact["code"],
"attribute": "benchmark_fact",
"value": fact["text"],
"importance": 0.9,
"confidence": 0.95,
"source": "large_memory_benchmark",
"evidence": [f"library_chunk:{chunk_index}"],
"token_ids": token_ids,
"token_mask": torch.ones_like(token_ids, dtype=torch.bool),
"trusted": True,
"force": True,
}
)
_sync(device)
write_started = time.perf_counter()
model.memory_os_v2.write_batch(records)
_sync(device)
write_ms = (time.perf_counter() - write_started) * 1000.0
target_record_ids: dict[str, str] = {}
# Resolve target IDs by text after the batch write. This avoids depending
# on the bank's stable-ID timestamp while preserving an exact recall test.
for fact in facts:
matches = [record.record_id for record in model.memory_os_v2.bank.records.values() if record.text == fact["text"]]
if matches:
target_record_ids[fact["code"]] = matches[0]
answerable_queries = [fact["query"] for fact in facts]
unknown_queries = [f"我的不存在的字段 UNKNOWN-{index:04d} 是什么？如果没有登记就说不知道。" for index in range(unknown_count)]
all_queries = answerable_queries + unknown_queries
query_keys = _encode_texts(model, tokenizer, all_queries, device)
answerable_rows: list[dict[str, Any]] = []
unknown_rows: list[dict[str, Any]] = []
query_started = time.perf_counter()
for index, query in enumerate(all_queries):
encoded = tokenizer(query, add_special_tokens=False, return_tensors="pt")
query_ids = encoded["input_ids"][0]
if index < len(answerable_queries):
fact = facts[index]
expected_id = target_record_ids.get(fact["code"], "")
records_out, decision = model.memory_os_v2.read(
query_key=query_keys[index],
query_text=query,
query_token_ids=query_ids,
top_k_pages=4,
top_k_records=2,
max_hops=3,
)
returned_ids = [record.record_id for record in records_out]
answerable_rows.append(
{
"code": fact["code"],
"expected_record_id": expected_id,
"returned_record_ids": returned_ids,
"hit": expected_id in returned_ids,
"stop_reason": decision.stop_reason,
"coarse_candidates": model.memory_os_v2.bank._last_coarse_candidates,
}
)
else:
records_out, decision = model.memory_os_v2.read(
query_key=query_keys[index],
query_text=query,
query_token_ids=query_ids,
top_k_pages=4,
top_k_records=2,
max_hops=3,
)
unknown_rows.append(
{
"query": query,
"returned_records": len(records_out),
"abstained": not records_out,
"stop_reason": decision.stop_reason,
}
)
_sync(device)
query_ms = (time.perf_counter() - query_started) * 1000.0
conflict_entity = "PROJ-CONFLICT"
conflict_old = model.memory_os_v2.write(
text="项目 PROJ-CONFLICT 的负责人是 Alice。",
key=F.normalize(torch.randn(model.memory.hidden_size, generator=generator), dim=0),
entity=conflict_entity,
attribute="owner",
value="Alice",
source="large_memory_benchmark",
trusted=True,
force=True,
)[0]
conflict_new = model.memory_os_v2.write(
text="项目 PROJ-CONFLICT 的负责人是 Bob。",
key=F.normalize(torch.randn(model.memory.hidden_size, generator=generator), dim=0),
entity=conflict_entity,
attribute="owner",
value="Bob",
source="large_memory_benchmark",
trusted=True,
force=True,
)[0]
conflict_query = "项目 PROJ-CONFLICT 当前负责人是谁？"
conflict_key = _encode_texts(model, tokenizer, [conflict_query], device)[0]
conflict_out, conflict_decision = model.memory_os_v2.read(
query_key=conflict_key,
query_text=conflict_query,
query_token_ids=torch.tensor(tokenizer(conflict_query, add_special_tokens=False)["input_ids"]),
top_k_pages=4,
top_k_records=2,
max_hops=3,
)
conflict_pass = conflict_new.record_id in [record.record_id for record in conflict_out] and conflict_old.status != "active"
report = {
"benchmark": "large_memory_recall_4b",
"quantization": "4bit_nf4" if use_4bit else "none",
"library_tokens": int(args.library_tokens),
"chunk_tokens": int(args.chunk_tokens),
"chunk_count": chunk_count,
"stored_records": len(model.memory_os_v2.bank.records),
"answerable_count": len(answerable_rows),
"answerable_hits": sum(int(row["hit"]) for row in answerable_rows),
"answerable_recall": sum(int(row["hit"]) for row in answerable_rows) / max(1, len(answerable_rows)),
"unknown_count": len(unknown_rows),
"unknown_abstentions": sum(int(row["abstained"]) for row in unknown_rows),
"unknown_abstention_rate": sum(int(row["abstained"]) for row in unknown_rows) / max(1, len(unknown_rows)),
"conflict_update_pass": conflict_pass,
"conflict_stop_reason": conflict_decision.stop_reason,
"write_ms": write_ms,
"query_total_ms": query_ms,
"query_mean_ms": query_ms / max(1, len(all_queries)),
"max_coarse_candidates": max((row["coarse_candidates"] for row in answerable_rows), default=0),
"max_vram_gb": torch.cuda.max_memory_allocated(device) / 1024**3 if device.type == "cuda" else None,
"conflict_old_status": conflict_old.status,
"conflict_new_status": conflict_new.status,
"conflict_returned_ids": [record.record_id for record in conflict_out],
"answerable_rows": answerable_rows,
"unknown_rows": unknown_rows,
"warning": "Routing test uses a sparse lexical address side-index plus v9 neural reranking; generation quality still needs real chat/repository workloads.",
}
output = _path(args.output)
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps(report, ensure_ascii=False, indent=2))
print(f"saved={output}")
del model
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
if __name__ == "__main__":
main()
benchmark_long_context_v1.py
+190
View File
@@ -0,0 +1,190 @@
"""Practical long-context stress probe for the local Qwen/Natural Memory setup."""
from __future__ import annotations
import argparse
import gc
import json
import random
import statistics
import time
from pathlib import Path
from typing import Any
import torch
from .qwen_integration import load_qwen_base, load_qwen_dynamic, load_tokenizer
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--base-model", default=r"H:\Memory")
parser.add_argument(
"--memory-model",
default=r"H:\Memory\V2_dpskw\qwen3_5_4b_memory_merged_v13",
)
parser.add_argument(
"--output",
default=r"H:\Memory\V2_dpskw\long_context_v1_stress.json",
)
parser.add_argument("--lengths", default="8192,16384,32768")
parser.add_argument("--max-new-tokens", type=int, default=4)
parser.add_argument("--no-4bit", action="store_true")
return parser.parse_args()
def build_prompt(tokenizer: Any, target_tokens: int, seed: int) -> tuple[str, str, int]:
rng = random.Random(seed + target_tokens)
answer = f"LONGCTX-{target_tokens}-{rng.randrange(100000, 999999)}"
needle = f"唯一目标记录：检索编码为 {answer}。"
filler = (
"这段材料是用于压力测试的背景文本。它包含版本、日志、普通备注和无关的项目描述，"
"不包含目标编码。阅读时应忽略这些干扰内容，只寻找唯一目标记录。"
)
chunks: list[str] = []
while len(tokenizer(" ".join(chunks + [filler, needle]), add_special_tokens=False)["input_ids"]) < target_tokens:
chunks.append(filler)
half = len(chunks) // 2
material = " ".join(chunks[:half] + [needle] + chunks[half:])
prompt = (
"请阅读下面的长材料，只输出唯一目标记录中的检索编码，不要解释。\n"
"---开始---\n"
f"{material}\n"
"---结束---\n"
"问题：唯一目标记录中的检索编码是什么？"
)
prompt_tokens = len(tokenizer(prompt, add_special_tokens=False)["input_ids"])
return prompt, answer, prompt_tokens
def make_inputs(tokenizer: Any, prompt: str, device: torch.device) -> dict[str, torch.Tensor]:
encoded = tokenizer.apply_chat_template(
[{"role": "user", "content": prompt}],
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
enable_thinking=False,
)
return {
key: value.to(device)
for key, value in encoded.items()
if isinstance(value, torch.Tensor)
}
def probe_model(model: Any, tokenizer: Any, lengths: list[int], *, dynamic: bool, max_new_tokens: int) -> dict[str, Any]:
device = model._find_layer_device() if dynamic else model.get_input_embeddings().weight.device
rows = []
for length in lengths:
prompt, answer, prompt_tokens = build_prompt(tokenizer, length, 20260904)
if dynamic:
model.reset_memory(batch_size=1, device=device)
if device.type == "cuda":
torch.cuda.reset_peak_memory_stats(device)
row: dict[str, Any] = {
"target_tokens": length,
"prompt_tokens": prompt_tokens,
"expected": answer,
}
try:
encoded = make_inputs(tokenizer, prompt, device)
query = tokenizer(prompt, add_special_tokens=False, return_tensors="pt")
query_ids = query["input_ids"].to(device)
query_mask = query.get("attention_mask")
if query_mask is None:
query_mask = torch.ones_like(query_ids)
query_mask = query_mask.to(device)
if device.type == "cuda":
torch.cuda.synchronize(device)
started = time.perf_counter()
with torch.inference_mode():
kwargs: dict[str, Any] = {
"max_new_tokens": max_new_tokens,
"do_sample": False,
"use_cache": True,
"pad_token_id": tokenizer.pad_token_id,
}
if dynamic:
kwargs.update(
{
"update_memory": False,
"memory_query_input_ids": query_ids,
"memory_query_attention_mask": query_mask,
}
)
output = model.generate(**encoded, **kwargs)
if device.type == "cuda":
torch.cuda.synchronize(device)
elapsed = time.perf_counter() - started
response_ids = output[0, encoded["input_ids"].shape[1] :]
response = tokenizer.decode(response_ids.detach().cpu().tolist(), skip_special_tokens=True).strip()
row.update(
{
"status": "ok",
"response": response,
"passed": answer in response,
"generated_tokens": int(response_ids.numel()),
"seconds": elapsed,
"tokens_per_second": int(response_ids.numel()) / max(elapsed, 1e-9),
}
)
except (torch.cuda.OutOfMemoryError, RuntimeError) as exc:
message = str(exc)
if isinstance(exc, torch.cuda.OutOfMemoryError) or "out of memory" in message.lower():
row.update({"status": "cuda_oom", "error": message[:500]})
if device.type == "cuda":
torch.cuda.empty_cache()
else:
raise
if device.type == "cuda":
row["peak_memory_allocated_gb"] = torch.cuda.max_memory_allocated(device) / 1024**3
row["peak_memory_reserved_gb"] = torch.cuda.max_memory_reserved(device) / 1024**3
rows.append(row)
return {"rows": rows}
def release(model: Any) -> None:
del model
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
def main() -> None:
args = parse_args()
lengths = [int(value.strip()) for value in args.lengths.split(",") if value.strip()]
use_4bit = not args.no_4bit
tokenizer = load_tokenizer(args.base_model)
report: dict[str, Any] = {
"benchmark": "Natural Memory v1 practical long-context stress probe",
"date": time.strftime("%Y-%m-%d %H:%M:%S"),
"base_model": str(Path(args.base_model).resolve()),
"memory_model": str(Path(args.memory_model).resolve()),
"lengths": lengths,
"quantization": "4bit_nf4" if use_4bit else "none",
"max_new_tokens": args.max_new_tokens,
}
print("loading baseline")
model = load_qwen_base(args.base_model, load_in_4bit=use_4bit)
model.eval()
report["baseline"] = probe_model(
model, tokenizer, lengths, dynamic=False, max_new_tokens=args.max_new_tokens
)
release(model)
print("loading Natural Memory v1")
model = load_qwen_dynamic(args.memory_model, load_in_4bit=use_4bit)
model.eval()
report["natural_memory_v1"] = probe_model(
model, tokenizer, lengths, dynamic=True, max_new_tokens=args.max_new_tokens
)
release(model)
output = Path(args.output)
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps(report, ensure_ascii=False, indent=2))
print(f"saved={output}")
if __name__ == "__main__":
main()
benchmark_memory_v2.py
+301
View File
@@ -0,0 +1,301 @@
"""Comprehensive, hardware-independent evaluation for Natural Memory v2."""
from __future__ import annotations
import argparse
import json
import math
import sys
import time
from pathlib import Path
from typing import Any
import torch
import torch.nn.functional as F
if __package__ in {None, ""}:
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from V2_dpskw.memory_os_v2 import (
KVBudgetManagerV2,
MemoryRouterV2,
PagedMemoryBankV2,
STATUS_ACTIVE,
STATUS_QUARANTINED,
STATUS_RETRACTED,
STATUS_SUPERSEDED,
)
from V2_dpskw.train_memory_router_v2 import _latent_to_hidden, _make_basis, evaluate
def _load_router(args: argparse.Namespace, device: torch.device) -> tuple[MemoryRouterV2, torch.Tensor, str]:
router = MemoryRouterV2(
args.hidden_size,
router_dim=args.router_dim,
num_heads=args.num_heads,
max_hops=args.max_hops,
).to(device)
checkpoint = Path(args.router_checkpoint)
basis_path = checkpoint.with_name("memory_router_v2_basis.pt")
if checkpoint.exists() and basis_path.exists():
state = torch.load(checkpoint, map_location=device, weights_only=True)
router.load_state_dict(state, strict=True)
basis = torch.load(basis_path, map_location=device, weights_only=True).to(device)
return router, basis, "trained_checkpoint"
basis = _make_basis(args.hidden_size, args.latent_size, device)
return router, basis, "untrained_router"
def _metric(value: bool) -> float:
return 1.0 if value else 0.0
def run(args: argparse.Namespace) -> dict[str, Any]:
torch.manual_seed(args.seed)
device = torch.device(args.device if args.device != "auto" else "cuda" if torch.cuda.is_available() else "cpu")
router, basis, router_source = _load_router(args, device)
router.eval()
bank = PagedMemoryBankV2(
args.hidden_size,
router=router,
page_capacity=args.page_capacity,
max_pages=args.max_pages,
hot_pages=args.hot_pages,
top_k_pages=args.top_k_pages,
top_k_records=args.top_k_records,
max_hops=args.max_hops,
coarse_index_bits=args.coarse_index_bits,
)
# Route quality on held-out samples from the same latent factor space.
route_metrics = evaluate(
router,
basis=basis,
device=device,
batches=args.router_eval_batches,
batch_size=args.router_eval_batch_size,
candidate_count=args.candidate_count,
)
# Populate a large enough store to activate the coarse index. Keys are
# low-entropy semantic points, while their positions are deliberately
# unrelated to their topic ids.
records: list[Any] = []
started = time.perf_counter()
for index in range(args.records):
latent = torch.randn(basis.shape[1], device=device)
key = _latent_to_hidden(latent.unsqueeze(0), basis, 0.04)[0]
record, action = bank.write(
text=f"长期事实 {index}",
key=key,
token_ids=torch.tensor([index % 997, 17, 23]),
token_mask=torch.tensor([True, True, True]),
importance=0.5 + 0.5 * (index % 7 == 0),
confidence=0.95,
source="synthetic_episode",
)
records.append(record)
write_seconds = time.perf_counter() - started
recall_hits = 0
page_hits = 0
candidate_counts: list[int] = []
query_count = min(args.query_count, len(records))
for index in torch.randperm(len(records), device=device)[:query_count].tolist():
target = records[index]
query_key = target.key.to(device) if router_source == "untrained_router" else (
target.key.to(device)
)
# ``target.key`` is already in compact address space. This tests the
# same storage-space path used after a Qwen hidden state is projected.
found, decision = bank.query(
query_key=query_key,
top_k_pages=args.top_k_pages,
top_k_records=args.top_k_records,
)
found_ids = {record.record_id for record in found}
recall_hits += int(target.record_id in found_ids)
page_hits += int(target.page_id in decision.page_ids)
candidate_counts.append(bank.stats()["last_coarse_candidates"])
# Conflict/versioning and explicit correction.
conflict_key = torch.randn(args.hidden_size, device=device)
first, _ = bank.write(
text="用户当前工作地点是上海",
key=conflict_key,
entity="user",
attribute="work_city",
value="上海",
confidence=0.90,
)
second, conflict_action = bank.write(
text="用户当前工作地点是杭州",
key=conflict_key,
entity="user",
attribute="work_city",
value="杭州",
confidence=0.98,
)
corrected, correction_action = bank.correct(
text="纠正：用户当前工作地点是苏州",
key=conflict_key,
entity="user",
attribute="work_city",
value="苏州",
confidence=1.0,
)
# Pollution protection: untrusted write stays out of the searchable bank.
quarantined, quarantine_action = bank.write(
text="模型猜测的生日",
key=torch.randn(args.hidden_size, device=device),
confidence=0.05,
trusted=False,
)
quarantine_before_approval = (
quarantined.status == STATUS_QUARANTINED
and quarantined.record_id not in bank.records
and quarantine_action == "quarantined"
)
approved = bank.approve(quarantined.record_id)
approved_active = approved.status == STATUS_ACTIVE
# Multi-hop: source page contains only the anchor; related evidence lives
# in other pages. Restrict first-hop page selection to force expansion.
hop_bank = PagedMemoryBankV2(
args.hidden_size,
router=router,
page_capacity=1,
max_pages=64,
hot_pages=1,
top_k_pages=1,
top_k_records=3,
max_hops=args.max_hops,
coarse_index_bits=args.coarse_index_bits,
)
hop_b, _ = hop_bank.write(text="链路证据 B", key=torch.randn(args.hidden_size, device=device), slot_index=20001)
hop_c, _ = hop_bank.write(text="链路证据 C", key=torch.randn(args.hidden_size, device=device), slot_index=20002)
hop_a, _ = hop_bank.write(
text="链路锚点 A",
key=torch.randn(args.hidden_size, device=device),
related_ids=[hop_b.record_id, hop_c.record_id],
slot_index=20000,
)
hop_records, hop_decision = hop_bank.query(
query_key=hop_a.key,
top_k_pages=1,
top_k_records=3,
max_hops=args.max_hops,
)
hop_ids = {record.record_id for record in hop_records}
multi_hop_success = hop_b.record_id in hop_ids or hop_c.record_id in hop_ids
# Idempotence, retraction and restart serialization.
duplicate, duplicate_action = bank.write(
text="长期事实 0",
key=records[0].key,
token_ids=records[0].token_ids,
token_mask=records[0].token_mask,
confidence=0.99,
)
bank.retract(approved.record_id)
restart_payload = bank.export_payload()
restored = PagedMemoryBankV2.from_payload(restart_payload, router=router)
restored_records, restored_decision = restored.query(
query_key=records[0].key,
top_k_pages=args.top_k_pages,
top_k_records=args.top_k_records,
)
budget = KVBudgetManagerV2(
max_tokens=args.kv_budget,
hard_max_tokens=args.kv_hard_max,
keep_recent_tokens=args.kv_keep_recent,
)
budget_checks = {
"below_trigger": not budget.needs_compaction(int(args.kv_budget * 0.5)),
"at_trigger": budget.needs_compaction(budget.trigger_tokens),
"overflow": budget.overflow(args.kv_budget + 123),
}
stats = bank.stats()
summary = {
"format_version": 2,
"seed": args.seed,
"device": str(device),
"router_source": router_source,
"router": route_metrics,
"storage": {
"records_requested": args.records,
"records_stored_before_scenarios": len(records),
"write_seconds": write_seconds,
"pages": stats["pages"],
"coarse_index_buckets": stats["coarse_index_buckets"],
"coarse_candidate_mean": sum(candidate_counts) / max(1, len(candidate_counts)),
"coarse_candidate_max": max(candidate_counts, default=0),
"coarse_candidate_ratio": (
sum(candidate_counts) / max(1, len(candidate_counts)) / max(1, stats["pages"])
),
},
"retrieval": {
"query_count": query_count,
"record_recall_at_k": recall_hits / max(1, query_count),
"page_recall_at_k": page_hits / max(1, query_count),
"multi_hop_success": _metric(multi_hop_success),
"multi_hop_hops": hop_decision.hop_count,
"restart_record_recall": _metric(bool(restored_records)),
"restart_page_count": restored.stats()["pages"],
},
"integrity": {
"conflict_action": conflict_action,
"correction_action": correction_action,
"old_conflict_superseded": _metric(first.status == STATUS_SUPERSEDED),
"latest_correction_active": _metric(corrected.status == STATUS_ACTIVE),
"active_conflict_value": corrected.value,
"quarantine_action": quarantine_action,
"quarantine_isolation": _metric(quarantine_before_approval),
"approved_active": _metric(approved_active),
"retracted_status": bank.records[approved.record_id].status,
"retraction_isolated": _metric(bank.records[approved.record_id].status == STATUS_RETRACTED),
"duplicate_action": duplicate_action,
"duplicate_idempotent": _metric(duplicate.record_id == records[0].record_id),
},
"kv_budget": budget_checks,
"final_stats": stats,
}
output_path = Path(args.output)
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(json.dumps(summary, indent=2, ensure_ascii=False), encoding="utf-8")
return summary
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output", default="W:/Flash/model/V2_dpskw/natural_memory_v2_benchmark.json")
parser.add_argument("--router-checkpoint", default="W:/Flash/model/V2_dpskw/checkpoints/natural_memory_v2_router/memory_router_v2.pt")
parser.add_argument("--device", default="auto")
parser.add_argument("--hidden-size", type=int, default=2560)
parser.add_argument("--router-dim", type=int, default=128)
parser.add_argument("--num-heads", type=int, default=8)
parser.add_argument("--max-hops", type=int, default=3)
parser.add_argument("--latent-size", type=int, default=32)
parser.add_argument("--candidate-count", type=int, default=32)
parser.add_argument("--router-eval-batches", type=int, default=40)
parser.add_argument("--router-eval-batch-size", type=int, default=64)
parser.add_argument("--records", type=int, default=512)
parser.add_argument("--query-count", type=int, default=128)
parser.add_argument("--page-capacity", type=int, default=32)
parser.add_argument("--max-pages", type=int, default=32768)
parser.add_argument("--hot-pages", type=int, default=8)
parser.add_argument("--top-k-pages", type=int, default=4)
parser.add_argument("--top-k-records", type=int, default=8)
parser.add_argument("--coarse-index-bits", type=int, default=20)
parser.add_argument("--kv-budget", type=int, default=32768)
parser.add_argument("--kv-hard-max", type=int, default=131072)
parser.add_argument("--kv-keep-recent", type=int, default=8192)
parser.add_argument("--seed", type=int, default=20260904)
return parser.parse_args()
if __name__ == "__main__":
print(json.dumps(run(parse_args()), ensure_ascii=False, indent=2))
benchmark_memory_vs_full_kv.py
+453
View File
@@ -0,0 +1,453 @@
"""Paired teacher/student benchmark for Natural Memory versus full KV context."""
from __future__ import annotations
import argparse
import gc
import json
import re
import sys
import time
from collections import defaultdict
from pathlib import Path
from typing import Any, Iterable
import torch
from .qwen_integration import load_memory_config, load_qwen_base, load_qwen_dynamic, load_tokenizer
from .stream_chat_qwen_memory import _chat_tensor, _write_turn
PROJECT_ROOT = Path(__file__).resolve().parent
def _path(value: str | Path) -> Path:
path = Path(value)
if path.is_absolute() or path.exists():
return path
return PROJECT_ROOT / path
def _max_memory(gpu_memory_gb: float) -> dict[Any, str] | None:
"""Cap CUDA placement so a benchmark cannot consume the whole HBM."""
if gpu_memory_gb <= 0.0 or not torch.cuda.is_available():
return None
return {0: f"{gpu_memory_gb:.1f}GiB", "cpu": "64GiB"}
def _set_cuda_process_cap(gpu_memory_gb: float) -> None:
"""Make the benchmark fail safely instead of growing past its HBM budget."""
if gpu_memory_gb <= 0.0 or not torch.cuda.is_available():
return
total = torch.cuda.get_device_properties(0).total_memory
fraction = min(0.95, max(0.05, gpu_memory_gb * 1024**3 / total))
torch.cuda.set_per_process_memory_fraction(fraction, device=0)
def _normalize(text: str) -> str:
return re.sub(r"[\s`*_#，。！？、；：,.!?;:'\"（）()\[\]{}]", "", str(text)).lower()
def _contains(text: str, choices: Iterable[str]) -> bool:
normalized = _normalize(text)
return any(_normalize(choice) and _normalize(choice) in normalized for choice in choices)
_ABSTENTION_MARKERS = (
"不知道",
"没有记录",
"无法确定",
"未找到相关信息",
"无法访问",
"没有访问权限",
"无法查询",
"无法获取",
"无法得知",
"不能确定",
"没有能力",
)
def _is_abstention(response: str) -> bool:
"""Recognize a truthful no-evidence response in answer-unavailable cases."""
if _contains(response, _ABSTENTION_MARKERS):
return True
if _contains(response, ("没有访问或存储", "没有读取或存储", "不具备访问或存储")):
return True
# Qwen often expresses the same abstention as a longer capability
# disclaimer, e.g. "没有访问或存储...的能力". This is still a no-evidence
# answer and must not be scored as a hallucinated personal fact.
return bool(
re.search(r"没有[^。！？\n]{0,24}(能力|权限)", response)
or re.search(r"无法[^。！？\n]{0,24}(访问|查询|获取|确定|得知|读取|存储)", response)
)
def _passed(response: str, case: dict[str, Any]) -> bool:
if _contains(response, case.get("forbidden", [])):
return False
acceptable = _contains(response, case.get("acceptable", []))
answerable = bool(case.get("metadata", {}).get("answerable", True))
return acceptable or (not answerable and _is_abstention(response))
def _read_cases(
path: Path,
*,
limit: int | None,
offset: int,
category: str | None,
per_category_limit: int | None = None,
) -> list[dict[str, Any]]:
rows: list[dict[str, Any]] = []
category_counts: dict[str, int] = defaultdict(int)
with path.open("r", encoding="utf-8") as handle:
for raw in handle:
if not raw.strip():
continue
case = json.loads(raw)
if category and case.get("category") != category:
continue
case_category = str(case.get("category", "unknown"))
if per_category_limit is not None:
if per_category_limit < 1:
raise ValueError("per_category_limit must be positive")
if category_counts[case_category] >= per_category_limit:
continue
category_counts[case_category] += 1
if offset > 0:
offset -= 1
continue
rows.append(case)
if limit is not None and len(rows) >= limit:
break
if not rows:
raise ValueError("no validation cases selected")
return rows
def _teacher_messages(case: dict[str, Any]) -> list[dict[str, str]]:
messages: list[dict[str, str]] = []
for fact in case["facts"]:
messages.append({"role": "user", "content": str(fact["text"])})
messages.append({"role": "assistant", "content": str(fact.get("assistant", "好的。"))})
messages.append({"role": "user", "content": str(case["query"])})
return messages
@torch.inference_mode()
def _generate_teacher(model: Any, tokenizer: Any, case: dict[str, Any], max_new_tokens: int) -> str:
device = model.get_input_embeddings().weight.device
encoded = tokenizer.apply_chat_template(
_teacher_messages(case),
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
enable_thinking=False,
)
encoded = {key: value.to(device) for key, value in encoded.items() if isinstance(value, torch.Tensor)}
output = model.generate(
**encoded,
max_new_tokens=max_new_tokens,
do_sample=False,
# This is a quality/parity benchmark. Disable generation KV so the
# evaluator cannot exceed the declared HBM placement cap while it
# repeatedly loads teacher and student models on a 12GB card.
use_cache=False,
pad_token_id=tokenizer.pad_token_id,
)
response = tokenizer.decode(output[0, encoded["input_ids"].shape[1] :], skip_special_tokens=True).strip()
del output, encoded
if torch.cuda.is_available():
torch.cuda.empty_cache()
return response
@torch.inference_mode()
def _generate_student(
model: Any,
tokenizer: Any,
case: dict[str, Any],
*,
max_new_tokens: int,
force_write: bool,
) -> tuple[str, dict[str, Any]]:
device = model._find_layer_device()
model.reset_memory(batch_size=1, device=device)
write_rows: list[dict[str, Any]] = []
for fact in case["facts"]:
changed = _write_turn(
model,
tokenizer,
str(fact["text"]),
device,
force_write=force_write,
)
last_slot = model.runtime.text_last_written_slot
write_rows.append(
{
"kind": fact.get("kind", "fact"),
"should_write": bool(fact.get("should_write", True)),
"changed": bool(changed),
"slot": int(last_slot[0].item()) if isinstance(last_slot, torch.Tensor) else -1,
}
)
encoded = {key: value.to(device) for key, value in _chat_tensor(
tokenizer,
str(case["query"]),
).items()}
query = tokenizer(str(case["query"]), add_special_tokens=False, return_tensors="pt")
query_ids = query["input_ids"].to(device)
query_mask = query.get("attention_mask")
if query_mask is None:
query_mask = torch.ones_like(query_ids)
query_mask = query_mask.to(device)
output = model.generate(
**encoded,
max_new_tokens=max_new_tokens,
do_sample=False,
update_memory=False,
memory_query_input_ids=query_ids,
memory_query_attention_mask=query_mask,
memory_query_text=str(case["query"]),
use_cache=False,
pad_token_id=tokenizer.pad_token_id,
)
response = tokenizer.decode(output[0, encoded["input_ids"].shape[1] :], skip_special_tokens=True).strip()
diagnostics = {
"writes": write_rows,
"valid_slots": int(model.runtime.text_slot_valid.sum().item())
if isinstance(model.runtime.text_slot_valid, torch.Tensor)
else 0,
"prefix_used": bool(model.runtime.text_prefix_used),
"read_slots": (
model.runtime.text_read_slots.detach().cpu().tolist()
if isinstance(model.runtime.text_read_slots, torch.Tensor)
else []
),
"read_relevance": (
model.runtime.text_read_relevance.detach().cpu().tolist()
if isinstance(model.runtime.text_read_relevance, torch.Tensor)
else []
),
"v2": {
**model.memory_v2_stats(),
"last_decisions": list(model.runtime.v2_last_decisions),
},
}
del output, encoded, query, query_ids, query_mask
if torch.cuda.is_available():
torch.cuda.empty_cache()
return response, diagnostics
def _release(model: Any) -> None:
del model
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
def _summarize(rows: list[dict[str, Any]]) -> dict[str, Any]:
categories: dict[str, list[float]] = defaultdict(list)
answerable: dict[str, list[float]] = defaultdict(list)
stop_reasons: dict[str, int] = defaultdict(int)
prefix_used = 0
route_records = 0
abstained = 0
elapsed = []
for row in rows:
categories[str(row["category"])].append(float(row["passed"]))
answerable[str(row["category"])].append(float(row.get("answerable", True)))
elapsed.append(float(row.get("elapsed_seconds", 0.0)))
diagnostics = row.get("diagnostics", {})
if diagnostics.get("prefix_used"):
prefix_used += 1
if row.get("abstained"):
abstained += 1
v2 = diagnostics.get("v2", {}) if isinstance(diagnostics, dict) else {}
for decision in v2.get("last_decisions", []) if isinstance(v2, dict) else []:
stop_reasons[str(decision.get("stop_reason", "unknown"))] += 1
route_records += sum(
len(decision.get("record_ids", []))
for decision in v2.get("last_decisions", [])
if isinstance(decision, dict)
) if isinstance(v2, dict) else 0
return {
"cases": len(rows),
"accuracy": sum(float(row["passed"]) for row in rows) / max(1, len(rows)),
"answerable_cases": sum(
int(bool(row.get("answerable", True))) for row in rows
),
"prefix_used_cases": prefix_used,
"abstention_cases": abstained,
"average_latency_seconds": sum(elapsed) / max(1, len(elapsed)),
"p95_latency_seconds": sorted(elapsed)[min(len(elapsed) - 1, int(len(elapsed) * 0.95))]
if elapsed else 0.0,
"retrieved_record_count": route_records,
"stop_reasons": dict(sorted(stop_reasons.items())),
"categories": {
category: {
"cases": len(values),
"accuracy": sum(values) / len(values),
"answerable_cases": int(sum(answerable[category])),
}
for category, values in sorted(categories.items())
},
}
def main() -> None:
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--model-path", default="qwen3_5_4b_natural_memory_v2")
parser.add_argument("--adapter", default=None, help="optional candidate adapter; omitted uses embedded policy")
parser.add_argument("--data", default="data/mega_validation/memory_validation_100k.jsonl")
parser.add_argument("--output", default="mega_memory_vs_full_kv_report.json")
parser.add_argument("--limit", type=int, default=32)
parser.add_argument("--offset", type=int, default=0)
parser.add_argument("--category", default=None)
parser.add_argument(
"--per-category-limit",
type=int,
default=None,
help="select this many cases from every category before applying generation",
)
parser.add_argument("--max-new-tokens", type=int, default=24)
parser.add_argument(
"--gpu-memory-gb",
type=float,
default=8.0,
help="hard CUDA placement cap; 0 disables the cap",
)
parser.add_argument("--force-write", action="store_true")
parser.add_argument("--no-4bit", action="store_true")
args = parser.parse_args()
_set_cuda_process_cap(args.gpu_memory_gb)
model_path = _path(args.model_path)
data_path = _path(args.data)
output_path = _path(args.output)
cases = _read_cases(
data_path,
limit=args.limit,
offset=args.offset,
category=args.category,
per_category_limit=args.per_category_limit,
)
tokenizer = load_tokenizer(model_path)
use_4bit = not args.no_4bit
report: dict[str, Any] = {
"format_version": 1,
"model_path": str(model_path),
"data": str(data_path),
"selected_cases": len(cases),
"force_write": bool(args.force_write),
"quantization": "4bit_nf4" if use_4bit else "none",
"gpu_memory_cap_gb": args.gpu_memory_gb if args.gpu_memory_gb > 0 else None,
}
started = time.perf_counter()
max_memory = _max_memory(args.gpu_memory_gb)
teacher = load_qwen_base(model_path, load_in_4bit=use_4bit, max_memory=max_memory)
teacher.eval()
teacher_rows: list[dict[str, Any]] = []
for index, case in enumerate(cases, 1):
case_started = time.perf_counter()
response = _generate_teacher(teacher, tokenizer, case, args.max_new_tokens)
teacher_rows.append(
{
"id": case["id"],
"category": case["category"],
"response": response,
"passed": _passed(response, case),
"abstained": _is_abstention(response),
"answerable": bool(case.get("metadata", {}).get("answerable", True)),
"elapsed_seconds": time.perf_counter() - case_started,
}
)
if index % 16 == 0:
print(f"teacher {index}/{len(cases)}")
report["teacher"] = _summarize(teacher_rows)
_release(teacher)
teacher = None
adapter_path = _path(args.adapter) if args.adapter else None
config_source = adapter_path or model_path
config = load_memory_config(config_source)
config.persistent_memory = True
config.natural_language_memory = True
config.automatic_memory = True
student = load_qwen_dynamic(
model_path,
memory_config=config,
load_in_4bit=use_4bit,
max_memory=max_memory,
)
if adapter_path is not None:
student.load_memory_adapter(adapter_path, strict=True)
student.eval()
student_rows: list[dict[str, Any]] = []
for index, case in enumerate(cases, 1):
case_started = time.perf_counter()
response, diagnostics = _generate_student(
student,
tokenizer,
case,
max_new_tokens=args.max_new_tokens,
force_write=args.force_write,
)
student_rows.append(
{
"id": case["id"],
"category": case["category"],
"response": response,
"passed": _passed(response, case),
"abstained": _is_abstention(response),
"answerable": bool(case.get("metadata", {}).get("answerable", True)),
"elapsed_seconds": time.perf_counter() - case_started,
"diagnostics": diagnostics,
}
)
if index % 16 == 0:
print(f"student {index}/{len(cases)}")
report["student"] = _summarize(student_rows)
_release(student)
student = None
teacher_accuracy = float(report["teacher"]["accuracy"])
student_accuracy = float(report["student"]["accuracy"])
teacher_by_id = {row["id"]: row for row in teacher_rows}
paired_teacher_pass = sum(bool(teacher_by_id[row["id"]]["passed"]) for row in student_rows)
paired_student_pass = sum(bool(row["passed"]) and teacher_by_id[row["id"]]["passed"] for row in student_rows)
category_gate: dict[str, Any] = {}
for category in sorted({str(case["category"]) for case in cases}):
teacher_cat = [row for row in teacher_rows if row["category"] == category]
student_cat = [row for row in student_rows if row["category"] == category]
t = sum(float(row["passed"]) for row in teacher_cat) / max(1, len(teacher_cat))
s = sum(float(row["passed"]) for row in student_cat) / max(1, len(student_cat))
category_gate[category] = {"teacher_accuracy": t, "student_accuracy": s, "ratio": s / max(t, 1e-9), "pass": s >= 0.95 * t}
report["parity"] = {
"teacher_accuracy": teacher_accuracy,
"student_accuracy": student_accuracy,
"student_to_teacher_ratio": student_accuracy / max(teacher_accuracy, 1e-9),
"paired_teacher_pass": paired_teacher_pass,
"paired_student_pass": paired_student_pass,
"paired_ratio": paired_student_pass / max(1, paired_teacher_pass),
"category_gate": category_gate,
"required_ratio": 0.95,
"pass": student_accuracy >= 0.95 * teacher_accuracy and all(item["pass"] for item in category_gate.values()),
}
report["elapsed_seconds"] = time.perf_counter() - started
report["failures"] = [row for row in student_rows if not row["passed"]][:100]
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps({"teacher": report["teacher"], "student": report["student"], "parity": report["parity"], "output": str(output_path)}, ensure_ascii=False, indent=2))
if __name__ == "__main__":
main()
benchmark_natural_language_memory.py
+319
View File
@@ -0,0 +1,319 @@
"""Production-oriented natural-language memory acceptance benchmark."""
from __future__ import annotations
import argparse
import gc
import json
import sys
from pathlib import Path
import torch
from .qwen_integration import (
DEFAULT_MEMORY_RESET_TOKEN,
load_memory_config,
load_qwen_dynamic,
load_tokenizer,
resolve_memory_reset_token,
)
def _chat_tensor(tokenizer, messages, *, add_generation_prompt: bool):
encoded = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=add_generation_prompt,
return_tensors="pt",
return_dict=True,
enable_thinking=False,
)
return {
key: value
for key, value in encoded.items()
if isinstance(value, torch.Tensor)
}
def _memory_system_prefix(tokenizer, content: str):
full = tokenizer.apply_chat_template(
[
{"role": "system", "content": content},
{"role": "user", "content": "__memory_query_boundary__"},
],
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
enable_thinking=False,
)
input_ids = full["input_ids"]
im_start = tokenizer.convert_tokens_to_ids("<|im_start|>")
positions = (input_ids[0] == int(im_start)).nonzero(as_tuple=False).flatten()
if positions.numel() < 2:
raise RuntimeError("could not locate the system/user memory boundary")
end = int(positions[1].item())
return {
"input_ids": input_ids[:, :end],
"attention_mask": torch.ones((1, end), dtype=torch.long),
}
@torch.inference_mode()
def _generate(model, tokenizer, user_text: str, max_new_tokens: int) -> str:
encoded = _chat_tensor(
tokenizer,
[{"role": "user", "content": user_text}],
add_generation_prompt=True,
)
device = model._find_layer_device()
encoded = {key: value.to(device) for key, value in encoded.items()}
query = tokenizer(user_text, add_special_tokens=False, return_tensors="pt")
query_ids = query["input_ids"].to(device)
query_mask = query.get("attention_mask")
if query_mask is None:
query_mask = torch.ones_like(query_ids)
query_mask = query_mask.to(device)
output = model.generate(
**encoded,
max_new_tokens=max_new_tokens,
do_sample=False,
update_memory=False,
memory_query_input_ids=query_ids,
memory_query_attention_mask=query_mask,
use_cache=True,
pad_token_id=tokenizer.pad_token_id,
)
response_ids = output[0, encoded["input_ids"].shape[1] :]
return tokenizer.decode(response_ids.detach().cpu().tolist(), skip_special_tokens=True).strip()
@torch.inference_mode()
def _write(model, tokenizer, fact: str, acknowledgement: str) -> dict:
device = model._find_layer_device()
dialogue = [
{"role": "user", "content": fact},
{"role": "assistant", "content": acknowledgement},
]
encoded = _chat_tensor(tokenizer, dialogue, add_generation_prompt=False)
encoded = {key: value.to(device) for key, value in encoded.items()}
text_prefix = _memory_system_prefix(
tokenizer,
"以下是与当前用户相关的已保存长期记忆。仅在问题相关时使用，不要编造：\n" + fact,
)
text_ids = text_prefix["input_ids"].to(device)
text_mask = text_prefix["attention_mask"].to(device)
key = tokenizer(fact, add_special_tokens=False, return_tensors="pt")
key_ids = key["input_ids"].to(device)
key_mask = key.get("attention_mask")
if key_mask is None:
key_mask = torch.ones_like(key_ids)
key_mask = key_mask.to(device)
storage = tokenizer(fact, add_special_tokens=False, return_tensors="pt")
storage_ids = storage["input_ids"].to(device)
storage_mask = storage.get("attention_mask")
if storage_mask is None:
storage_mask = torch.ones_like(storage_ids)
storage_mask = storage_mask.to(device)
model(
**encoded,
read_memory=False,
update_memory=True,
return_memory=True,
use_cache=False,
memory_text_input_ids=text_ids,
memory_text_attention_mask=text_mask,
memory_key_input_ids=key_ids,
memory_key_attention_mask=key_mask,
memory_storage_input_ids=storage_ids,
memory_storage_attention_mask=storage_mask,
)
address = model.memory.last_write_address
probability = model.memory.last_write_probability
stored_slot = model.runtime.text_last_written_slot
return {
"fact": fact,
"write_probability": float(probability.detach().mean()) if probability is not None else None,
"selected_slot": int(address.argmax(dim=-1)[0].item()) if address is not None else None,
"stored_slot": int(stored_slot[0].item()) if stored_slot is not None else None,
"valid_slots_after_write": int(model.runtime.text_slot_valid.sum())
if model.runtime.text_slot_valid is not None
else 0,
}
def _load_persistent_checkpoint(model_path: str, adapter_path: str, *, no_4bit: bool):
config = load_memory_config(adapter_path)
restarted = load_qwen_dynamic(
model_path,
memory_config=config,
load_in_4bit=not no_4bit,
)
restarted.load_memory_adapter(adapter_path)
restarted.eval()
return restarted, config
def main() -> None:
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--model-path", default=".")
parser.add_argument("--adapter", default="V2_dpskw/qwen_memory_adapter_native_v3")
parser.add_argument(
"--output-adapter",
default="V2_dpskw/qwen_memory_adapter_natural_production_v1",
)
parser.add_argument(
"--report",
default="V2_dpskw/benchmark_natural_language_memory.json",
)
parser.add_argument("--max-new-tokens", type=int, default=48)
parser.add_argument("--text-memory-threshold", type=float, default=0.35)
parser.add_argument("--no-4bit", action="store_true")
args = parser.parse_args()
tokenizer = load_tokenizer(args.model_path)
config = load_memory_config(args.adapter)
config.persistent_memory = True
config.natural_language_memory = True
config.text_memory_threshold = args.text_memory_threshold
config.reset_token_id = resolve_memory_reset_token(tokenizer, DEFAULT_MEMORY_RESET_TOKEN)
model = load_qwen_dynamic(
args.model_path,
memory_config=config,
load_in_4bit=not args.no_4bit,
)
model.load_memory_adapter(args.adapter)
model.reset_memory()
writes = [
_write(model, tokenizer, "请记住：我的工作地点代号是R7。", "好的，我会记住。"),
_write(model, tokenizer, "请记住：我最喜欢的水果是红富士苹果。", "好的，我会记住。"),
_write(model, tokenizer, "更新一下：我的工作地点代号改为K9。", "好的，已更新。"),
]
model.save_persistent_memory_checkpoint(args.output_adapter)
saved_valid_slots = int(model.runtime.text_slot_valid.sum())
saved_norm = float(model.runtime.state.detach().float().norm())
del model
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
restarted, restart_config = _load_persistent_checkpoint(
args.model_path,
args.output_adapter,
no_4bit=args.no_4bit,
)
loaded_norm = float(restarted.runtime.state.detach().float().norm())
queries = [
{"name": "replaced_work_code", "query": "我的工作地点代号是什么？", "expected": "K9"},
{"name": "favorite_fruit", "query": "我最喜欢吃什么水果？", "expected": "红富士苹果"},
{"name": "unknown_blood_type", "query": "我的血型是什么？如果没有记录，请明确说不知道。", "expected": "不知道"},
]
query_results = []
for item in queries:
response = _generate(restarted, tokenizer, item["query"], args.max_new_tokens)
relevance = restarted.runtime.text_read_relevance
overlap = restarted.runtime.text_read_overlap
selected_slots = restarted.runtime.text_read_slots
query_results.append(
{
**item,
"response": response,
"text_prefix_used": restarted.runtime.text_prefix_used,
"retrieval_relevance": float(relevance[0].item()) if relevance is not None else None,
"retrieval_overlap": overlap[0].detach().cpu().tolist()
if overlap is not None
else None,
"retrieved_slots": selected_slots[0].detach().cpu().tolist()
if selected_slots is not None
else None,
"expected_found": item["expected"] in response,
"refused_unknown": item["name"] != "unknown_blood_type"
or any(marker in response for marker in ("不知道", "没有记录", "无相关", "未找到", "不清楚")),
}
)
del restarted
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
restarted, second_restart_config = _load_persistent_checkpoint(
args.model_path,
args.output_adapter,
no_4bit=args.no_4bit,
)
second_loaded_norm = float(restarted.runtime.state.detach().float().norm())
second_restart_results = []
for item in queries[:2]:
response = _generate(restarted, tokenizer, item["query"], args.max_new_tokens)
second_restart_results.append(
{
"name": item["name"],
"response": response,
"expected": item["expected"],
"expected_found": item["expected"] in response,
"text_prefix_used": restarted.runtime.text_prefix_used,
}
)
reset_inputs = _chat_tensor(
tokenizer,
[{"role": "user", "content": DEFAULT_MEMORY_RESET_TOKEN}],
add_generation_prompt=True,
)
device = restarted._find_layer_device()
reset_inputs = {key: value.to(device) for key, value in reset_inputs.items()}
restarted.generate(
**reset_inputs,
max_new_tokens=1,
do_sample=False,
update_memory=False,
use_cache=False,
pad_token_id=tokenizer.pad_token_id,
)
reset_norm = float(restarted.runtime.state.detach().float().norm())
reset_valid_slots = int(restarted.runtime.text_slot_valid.sum())
after_reset = _generate(restarted, tokenizer, "我的工作地点代号是什么？", args.max_new_tokens)
report = {
"writes": writes,
"history_passed_to_restart": False,
"saved_valid_slots": saved_valid_slots,
"saved_memory_norm": saved_norm,
"loaded_memory_norm": loaded_norm,
"restart_state_equal_norm": abs(saved_norm - loaded_norm) < 1e-5,
"second_loaded_memory_norm": second_loaded_norm,
"second_restart_state_equal_norm": abs(saved_norm - second_loaded_norm) < 1e-5,
"queries": query_results,
"second_restart_queries": second_restart_results,
"second_restart_pass": all(row["expected_found"] for row in second_restart_results),
"all_known_queries_pass": all(row["expected_found"] for row in query_results[:2]),
"unknown_refusal_pass": query_results[2]["refused_unknown"],
"reset_token": DEFAULT_MEMORY_RESET_TOKEN,
"reset_token_id": restart_config.reset_token_id,
"reset_memory_norm": reset_norm,
"reset_valid_slots": reset_valid_slots,
"reset_cleared_pass": reset_norm < 1e-5 and reset_valid_slots == 0,
"response_after_reset": after_reset,
}
report["production_gate_pass"] = bool(
report["history_passed_to_restart"] is False
and report["restart_state_equal_norm"]
and report["second_restart_state_equal_norm"]
and report["all_known_queries_pass"]
and report["second_restart_pass"]
and report["unknown_refusal_pass"]
and report["reset_cleared_pass"]
)
report_path = Path(args.report)
report_path.parent.mkdir(parents=True, exist_ok=True)
report_path.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps(report, ensure_ascii=False, indent=2))
if __name__ == "__main__":
main()
benchmark_natural_memory_v1.py
+725
View File
@@ -0,0 +1,725 @@
"""Full local comparison between Qwen3.5-4B and Natural Memory v1.
This is an engineering benchmark, not a claim of state-of-the-art performance.
It evaluates the same frozen Qwen3.5-4B backbone with and without the internal
memory path, using deterministic greedy decoding and locally generated cases.
The report includes general ability, extra math and reasoning cases, long
context retrieval, throughput, latency, VRAM, automatic write decisions,
conflict replacement, unknown-fact refusal, reset, and shard-backed restart.
"""
from __future__ import annotations
import argparse
import gc
import json
import math
import os
import random
import statistics
import subprocess
import time
from pathlib import Path
from typing import Any
import torch
from .qwen_integration import (
load_memory_config,
load_qwen_base,
load_qwen_dynamic,
load_tokenizer,
)
from .stream_chat_qwen_memory import _chat_tensor, _memory_system_prefix, _write_turn
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--base-model", default=r"H:\Memory")
parser.add_argument(
"--memory-model",
default=r"H:\Memory\V2_dpskw\qwen3_5_4b_memory_merged_v13",
)
parser.add_argument(
"--data",
default=r"H:\Memory\V2_dpskw\data\comprehensive_general.jsonl",
)
parser.add_argument(
"--output",
default=r"H:\Memory\V2_dpskw\natural_memory_v1_full_benchmark.json",
)
parser.add_argument("--max-new-tokens", type=int, default=24)
parser.add_argument("--perf-repeats", type=int, default=3)
parser.add_argument("--no-4bit", action="store_true")
return parser.parse_args()
def normalize(text: str) -> str:
keep = str(text).lower()
for char in " \t\r\n`*_#，。！？、；：,.!?;:'\"（）()[]{}<>|\\/：":
keep = keep.replace(char, "")
return keep
def contains_answer(text: str, acceptable: list[str]) -> bool:
normalized = normalize(text)
for answer in acceptable:
expected = normalize(str(answer))
if not expected:
continue
if expected.isdigit() and len(expected) == 1:
if any(
normalized[index : index + 1] == expected
and (index == 0 or not normalized[index - 1].isdigit())
and (index + 1 == len(normalized) or not normalized[index + 1].isdigit())
for index in range(len(normalized))
):
return True
elif expected in normalized:
return True
return False
def load_jsonl(path: str | Path) -> list[dict[str, Any]]:
rows = []
for line in Path(path).read_text(encoding="utf-8").splitlines():
if line.strip():
rows.append(json.loads(line))
return rows
def prompt_inputs(tokenizer: Any, prompt: str, device: torch.device) -> dict[str, torch.Tensor]:
encoded = tokenizer.apply_chat_template(
[{"role": "user", "content": prompt}],
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
enable_thinking=False,
)
return {
key: value.to(device)
for key, value in encoded.items()
if isinstance(value, torch.Tensor)
}
def input_token_count(tokenizer: Any, prompt: str) -> int:
encoded = tokenizer(prompt, add_special_tokens=False)
return len(encoded["input_ids"])
def generate_answer(
model: Any,
tokenizer: Any,
prompt: str,
*,
dynamic: bool,
max_new_tokens: int,
) -> tuple[str, int, float]:
device = model._find_layer_device() if dynamic else model.get_input_embeddings().weight.device
encoded = prompt_inputs(tokenizer, prompt, device)
query = tokenizer(prompt, add_special_tokens=False, return_tensors="pt")
query_ids = query["input_ids"].to(device)
query_mask = query.get("attention_mask")
if query_mask is None:
query_mask = torch.ones_like(query_ids)
query_mask = query_mask.to(device)
if device.type == "cuda":
torch.cuda.synchronize(device)
started = time.perf_counter()
with torch.inference_mode():
kwargs: dict[str, Any] = {
"max_new_tokens": max_new_tokens,
"do_sample": False,
"use_cache": True,
"pad_token_id": tokenizer.pad_token_id,
}
if dynamic:
kwargs.update(
{
"update_memory": False,
"memory_query_input_ids": query_ids,
"memory_query_attention_mask": query_mask,
}
)
output = model.generate(**encoded, **kwargs)
if device.type == "cuda":
torch.cuda.synchronize(device)
elapsed = time.perf_counter() - started
prompt_len = int(encoded["input_ids"].shape[1])
response_ids = output[0, prompt_len:]
response = tokenizer.decode(response_ids.detach().cpu().tolist(), skip_special_tokens=True).strip()
return response, int(response_ids.numel()), elapsed
def make_math_cases() -> list[dict[str, Any]]:
raw = [
("m01", "只输出最终整数：38 + 47 = ?", ["85"]),
("m02", "只输出最终整数：900 - 376 = ?", ["524"]),
("m03", "只输出最终整数：24 × 17 = ?", ["408"]),
("m04", "只输出最终整数：936 ÷ 18 = ?", ["52"]),
("m05", "只输出结果：2.75 + 3.6 = ?", ["6.35"]),
("m06", "只输出结果：7/8 - 1/4 = ?", ["5/8", "0.625"]),
("m07", "只输出整数：15 和 28 的最小公倍数是多少？", ["420"]),
("m08", "只输出百分数：480 的 12.5% 是多少？", ["60"]),
("m09", "只输出百分数：80 增长到 100，增长率是多少？", ["25%", "25"]),
("m10", "只输出结果：3 的 5 次方是多少？", ["243"]),
("m11", "只输出 x：4x + 7 = 31。", ["6"]),
("m12", "只输出 x：9x - 18 = 0。", ["2"]),
("m13", "只输出 x：2(x + 5) = 18。", ["4"]),
("m14", "只输出 x：x/3 + 4 = 9。", ["15"]),
("m15", "只输出 x：5x - 2 = 3x + 10。", ["6"]),
("m16", "只输出下一个数：5，10，20，40，？", ["80"]),
("m17", "只输出下一个数：3，6，11，18，27，？", ["38"]),
("m18", "只输出下一个数：1，4，9，16，？", ["25"]),
("m19", "只输出下一个数：2，3，5，8，12，？", ["17"]),
("m20", "只输出整数：阶乘 6! 等于多少？", ["720"]),
("m21", "只输出面积：长 12、宽 7 的矩形面积是多少？", ["84"]),
("m22", "只输出周长：边长为 9 的正方形周长是多少？", ["36"]),
("m23", "只输出面积：底 10、高 6 的三角形面积是多少？", ["30"]),
("m24", "只输出角度：一个三角形两个角是 35 度和 65 度，第三个角是多少？", ["80"]),
("m25", "只输出数量：3 件不同衬衫和 2 条不同裤子可以组成多少套穿搭？", ["6"]),
("m26", "只输出数量：从 5 个人中选 2 个人，有多少种选法？", ["10"]),
("m27", "只输出余数：17 除以 5 的余数是多少？", ["2"]),
("m28", "只输出结果：平均数 8、12、16、20 是多少？", ["14"]),
("m29", "只输出结果：一个商品原价 240 元，打八折后多少钱？", ["192"]),
("m30", "只输出结果：2.4 × 0.5 = ?", ["1.2"]),
]
return [
{"id": case_id, "category": "math", "prompt": prompt, "acceptable": answers}
for case_id, prompt, answers in raw
]
def make_reasoning_cases() -> list[dict[str, Any]]:
raw = [
("r01", "只输出名字：甲比乙早到，乙比丙早到，谁最后到？", ["丙"]),
("r02", "只输出名字：小李在小王左边，小王在小张左边，谁最右边？", ["小张"]),
("r03", "只输出结论：所有鸟都有翅膀，企鹅是鸟，所以企鹅有翅膀吗？", ["是"]),
("r04", "只输出结论：所有猫都是哺乳动物，鲸鱼是哺乳动物，所以鲸鱼是猫吗？", ["不是", "否"]),
("r05", "只输出结论：如果下雨就带伞。现在下雨了，要不要带伞？", ["要"]),
("r06", "只输出结论：只有持票者才能入场。小林没有票，他能入场吗？", ["不能", "不可以"]),
("r07", "只输出星期：今天是星期三，五天后是星期几？", ["星期一", "周一"]),
("r08", "只输出方向：你面向北，右转后面向哪个方向？", ["东"]),
("r09", "只输出方向：你面向东，左转后面向哪个方向？", ["北"]),
("r10", "只输出数量：盒子里有 4 个红球和 3 个蓝球，不看颜色拿出一个，至少有几个球？", ["1"]),
("r11", "只输出名字：甲不是第一，乙在甲前面，丙在乙后面，谁可能是第一？", ["乙"]),
("r12", "只输出结论：有些学生会游泳，小周是学生，能确定小周会游泳吗？", ["不能", "无法"]),
("r13", "只输出下一个数：1，2，4，7，11，？", ["16"]),
("r14", "只输出下一个数：81，27，9，3，？", ["1"]),
("r15", "只输出名字：红色比蓝色重，绿色比红色轻但比蓝色重，哪个最轻？", ["蓝色"]),
("r16", "只输出答案：苹果不是蔬菜，胡萝卜是蔬菜，香蕉是水果，哪个不是水果？", ["胡萝卜"]),
("r17", "只输出结论：如果 A 大于 B 且 B 大于 C，那么 A 大于 C 吗？", ["是"]),
("r18", "只输出结论：如果一个数能被 2 整除，它一定是偶数。14 能被 2 整除，它是偶数吗？", ["是"]),
("r19", "只输出名字：小赵比小钱高，小孙比小赵矮但比小钱高，谁最高？", ["小赵"]),
("r20", "只输出数量：一周中有几天的名字包含‘星’字？", ["7"]),
("r21", "只输出结论：没有鱼是鸟，金鱼是鱼，所以金鱼是鸟吗？", ["不是", "否"]),
("r22", "只输出顺序：春、夏、秋、冬之后又回到哪个季节？", ["春"]),
("r23", "只输出结论：所有密码都需要保密，这个字符串是密码，所以它需要保密吗？", ["是"]),
("r24", "只输出答案：小明有两个兄弟，每个兄弟都有一个姐姐，小明有几个姐姐？", ["1"]),
]
return [
{"id": case_id, "category": "reasoning", "prompt": prompt, "acceptable": answers}
for case_id, prompt, answers in raw
]
def make_context_cases(tokenizer: Any) -> list[dict[str, Any]]:
rng = random.Random(20260904)
filler = (
"这是一段与问题无关的背景说明。系统记录了版本号、构建时间、测试批次、"
"设备温度、日志摘要和普通项目备注。这些文字只是干扰项，不包含目标答案。"
)
cases: list[dict[str, Any]] = []
for target_tokens in (512, 2048, 4096, 8192):
for position in ("early", "middle", "late"):
answer = f"CTX{target_tokens}-{position.upper()}-{rng.randrange(100, 999)}"
needle = f"目标记录：本次检索需要返回的项目编码是 {answer}。"
chunks: list[str] = []
while input_token_count(tokenizer, " ".join(chunks + [filler, needle])) < target_tokens:
chunks.append(filler)
if position == "early":
material = " ".join([needle] + chunks)
elif position == "middle":
half = len(chunks) // 2
material = " ".join(chunks[:half] + [needle] + chunks[half:])
else:
material = " ".join(chunks + [needle])
prompt = (
"请阅读下面的材料，只输出目标记录中的项目编码，不要解释。\n"
"---材料开始---\n"
f"{material}\n"
"---材料结束---\n"
"问题：目标记录中的项目编码是什么？"
)
cases.append(
{
"id": f"ctx-{target_tokens}-{position}",
"category": f"context_{target_tokens}",
"prompt": prompt,
"acceptable": [answer],
"target_tokens": target_tokens,
"position": position,
}
)
return cases
def evaluate_cases(
model: Any,
tokenizer: Any,
cases: list[dict[str, Any]],
*,
dynamic: bool,
max_new_tokens: int,
) -> dict[str, Any]:
rows: list[dict[str, Any]] = []
categories: dict[str, list[float]] = {}
started = time.perf_counter()
for case in cases:
if dynamic:
model.reset_memory(batch_size=1, device=model._find_layer_device())
response, generated_tokens, elapsed = generate_answer(
model,
tokenizer,
str(case["prompt"]),
dynamic=dynamic,
max_new_tokens=max_new_tokens,
)
passed = contains_answer(response, list(case["acceptable"]))
category = str(case["category"])
categories.setdefault(category, []).append(float(passed))
rows.append(
{
"id": case["id"],
"category": category,
"prompt_tokens": input_token_count(tokenizer, str(case["prompt"])),
"acceptable": case["acceptable"],
"generated": response,
"generated_tokens": generated_tokens,
"seconds": elapsed,
"passed": passed,
}
)
total = sum(sum(values) for values in categories.values())
return {
"cases": len(rows),
"elapsed_seconds": time.perf_counter() - started,
"overall_score": total / max(1, len(rows)),
"categories": {
category: {
"count": len(values),
"score": sum(values) / max(1, len(values)),
}
for category, values in sorted(categories.items())
},
"rows": rows,
}
def device_snapshot(model: Any) -> dict[str, Any]:
device = model._find_layer_device() if hasattr(model, "_find_layer_device") else model.get_input_embeddings().weight.device
params = sum(parameter.numel() for parameter in model.parameters())
result: dict[str, Any] = {
"device": str(device),
"parameter_count": int(params),
"parameter_count_billion": params / 1e9,
}
if device.type == "cuda":
properties = torch.cuda.get_device_properties(device)
result.update(
{
"gpu_name": properties.name,
"gpu_total_memory_gb": properties.total_memory / 1024**3,
}
)
return result
def measure_performance(
model: Any,
tokenizer: Any,
cases: list[dict[str, Any]],
*,
dynamic: bool,
max_new_tokens: int,
repeats: int,
) -> dict[str, Any]:
selected = [cases[0]]
for wanted in (512, 2048, 4096):
matching = [case for case in cases if case.get("target_tokens") == wanted]
if matching:
selected.append(matching[1])
rows = []
device = model._find_layer_device() if dynamic else model.get_input_embeddings().weight.device
for case in selected:
latencies = []
generated = 0
for _ in range(max(1, repeats)):
if dynamic:
model.reset_memory(batch_size=1, device=device)
_, token_count, elapsed = generate_answer(
model,
tokenizer,
str(case["prompt"]),
dynamic=dynamic,
max_new_tokens=max_new_tokens,
)
latencies.append(elapsed)
generated += token_count
rows.append(
{
"id": case["id"],
"prompt_tokens": input_token_count(tokenizer, str(case["prompt"])),
"median_seconds": statistics.median(latencies),
"mean_seconds": statistics.mean(latencies),
"tokens_per_second": generated / max(1e-9, sum(latencies)),
"repeats": len(latencies),
}
)
return {"rows": rows}
def memory_payload(model: Any) -> dict[str, torch.Tensor]:
payload: dict[str, torch.Tensor] = {
"memory_state": model.runtime.state.detach().cpu().clone(),
}
if model.memory_config.natural_language_memory:
for name in (
"text_token_ids",
"text_token_mask",
"text_slot_valid",
"text_slot_keys",
"text_slot_age",
"text_write_counter",
"text_key_token_ids",
"text_key_token_mask",
):
value = getattr(model.runtime, name)
if not isinstance(value, torch.Tensor):
raise RuntimeError(f"runtime memory field is unavailable: {name}")
payload[name] = value.detach().cpu().clone()
return payload
def answer_memory_query(model: Any, tokenizer: Any, text: str, max_new_tokens: int) -> tuple[str, bool]:
response, _, _ = generate_answer(
model,
tokenizer,
text,
dynamic=True,
max_new_tokens=max_new_tokens,
)
return response, bool(model.runtime.text_prefix_used)
def run_memory_benchmark(
model: Any,
tokenizer: Any,
package_path: str | Path,
*,
max_new_tokens: int,
no_4bit: bool,
) -> dict[str, Any]:
"""Run memory tests and restore the user's original embedded state."""
original = memory_payload(model)
decision_rows = []
positives = [
"我叫林舟。",
"我的常住城市是苏州。",
"我最喜欢的水果是红富士苹果。",
"我正在开发 Natural Memory v1 项目。",
"以后请把代码默认写成 Python。",
"我的常用时区是 Asia/Shanghai。",
"这是我的长期偏好：使用简洁中文。",
"这个项目的重要约束是不要修改原始 Qwen 权重。",
]
negatives = [
"我叫什么？",
"帮我解释向量数据库是什么。",
"你觉得今天的天气怎么样？",
"请把 memory 翻译成中文。",
"计算一下 17 × 19。",
"如果我选择 GPU，会发生什么？",
"我之前有没有提到我的城市？",
"给我一个自然语言记忆方案。",
]
try:
for expected, items in ((True, positives), (False, negatives)):
for text in items:
model.reset_memory(batch_size=1, device=model._find_layer_device())
changed = _write_turn(model, tokenizer, text, model._find_layer_device())
decision_rows.append(
{
"text": text,
"expected_write": expected,
"actual_write": changed,
"write_probability": float(model.runtime.auto_memory_probability.mean())
if isinstance(model.runtime.auto_memory_probability, torch.Tensor)
else None,
}
)
tp = sum(row["expected_write"] and row["actual_write"] for row in decision_rows)
tn = sum((not row["expected_write"]) and (not row["actual_write"]) for row in decision_rows)
fp = sum((not row["expected_write"]) and row["actual_write"] for row in decision_rows)
fn = sum(row["expected_write"] and (not row["actual_write"]) for row in decision_rows)
model.reset_memory(batch_size=1, device=model._find_layer_device())
first = "我的工作地点代号是NM-R7。"
second = "我最喜欢的水果是青提。"
replacement = "我的工作地点代号改为NM-K9。"
writes = [
{"text": first, "changed": _write_turn(model, tokenizer, first, model._find_layer_device())},
{"text": second, "changed": _write_turn(model, tokenizer, second, model._find_layer_device())},
{
"text": replacement,
"changed": _write_turn(model, tokenizer, replacement, model._find_layer_device()),
},
]
before_restart = {}
for name, query, expected in (
("work_code", "我的工作地点代号是什么？", "NM-K9"),
("fruit", "我最喜欢的水果是什么？", "青提"),
("unknown", "我的血型是什么？如果没有记录，请明确说不知道。", "不知道"),
):
response, prefix_used = answer_memory_query(model, tokenizer, query, max_new_tokens)
before_restart[name] = {
"query": query,
"expected": expected,
"response": response,
"expected_found": expected in response,
"prefix_used": prefix_used,
}
# Persist through ordinary natural-language turns. The caller releases
# this model before loading a fresh process/model for the restart test;
# keeping that lifecycle outside this function avoids two 4-bit Qwen
# backbones occupying the GPU at the same time.
model.save_embedded_memory_weights(package_path)
return {
"automatic_write_decision": {
"rows": decision_rows,
"true_positive": int(tp),
"true_negative": int(tn),
"false_positive": int(fp),
"false_negative": int(fn),
"precision": tp / max(1, tp + fp),
"recall": tp / max(1, tp + fn),
"specificity": tn / max(1, tn + fp),
},
"natural_language_writes": writes,
"before_restart": before_restart,
"embedded_write_persisted": True,
}
except Exception:
# Best-effort restoration if a test fails halfway through.
model._load_persistent_memory_payload(original)
model.save_embedded_memory_weights(package_path)
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
raise
def release(model: Any) -> None:
del model
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
def main() -> None:
args = parse_args()
tokenizer = load_tokenizer(args.base_model)
memory_config_hint = load_memory_config(args.memory_model)
memory_variant = (
"natural_memory_v2"
if memory_config_hint.memory_version >= 2 or memory_config_hint.hierarchical_memory
else "natural_memory_v1"
)
general = load_jsonl(args.data)
math_cases = make_math_cases()
reasoning_cases = make_reasoning_cases()
context_cases = make_context_cases(tokenizer)
all_cases = general + math_cases + reasoning_cases + context_cases
use_4bit = not args.no_4bit
report: dict[str, Any] = {
"benchmark": f"Natural Memory {memory_variant.rsplit('_', 1)[-1]} full local comparison",
"date": time.strftime("%Y-%m-%d %H:%M:%S"),
"seed": 20260904,
"base_model": str(Path(args.base_model).resolve()),
"memory_model": str(Path(args.memory_model).resolve()),
"data": str(Path(args.data).resolve()),
"quantization": "4bit_nf4" if use_4bit else "none",
"decoding": {"do_sample": False, "max_new_tokens": args.max_new_tokens},
"case_counts": {
"general_existing": len(general),
"math": len(math_cases),
"reasoning": len(reasoning_cases),
"context": len(context_cases),
"total": len(all_cases),
},
"context_targets": sorted({case["target_tokens"] for case in context_cases}),
}
print(f"cases={len(all_cases)} quantization={report['quantization']}")
print("loading Qwen3.5-4B baseline")
started = time.perf_counter()
baseline = load_qwen_base(args.base_model, load_in_4bit=use_4bit)
baseline.eval()
report["baseline"] = {
"load_seconds": time.perf_counter() - started,
"hardware": device_snapshot(baseline),
}
baseline_device = baseline.get_input_embeddings().weight.device
if baseline_device.type == "cuda":
torch.cuda.reset_peak_memory_stats(baseline_device)
report["baseline"]["quality"] = evaluate_cases(
baseline,
tokenizer,
all_cases,
dynamic=False,
max_new_tokens=args.max_new_tokens,
)
report["baseline"]["performance"] = measure_performance(
baseline,
tokenizer,
context_cases,
dynamic=False,
max_new_tokens=args.max_new_tokens,
repeats=args.perf_repeats,
)
if baseline_device.type == "cuda":
report["baseline"]["peak_memory_allocated_gb"] = torch.cuda.max_memory_allocated(baseline_device) / 1024**3
report["baseline"]["peak_memory_reserved_gb"] = torch.cuda.max_memory_reserved(baseline_device) / 1024**3
release(baseline)
print(f"loading {memory_variant} embedded package")
started = time.perf_counter()
dynamic = load_qwen_dynamic(args.memory_model, load_in_4bit=use_4bit)
dynamic.eval()
report[memory_variant] = {
"load_seconds": time.perf_counter() - started,
"hardware": device_snapshot(dynamic),
}
dynamic_device = dynamic._find_layer_device()
if dynamic_device.type == "cuda":
torch.cuda.reset_peak_memory_stats(dynamic_device)
report[memory_variant]["quality"] = evaluate_cases(
dynamic,
tokenizer,
all_cases,
dynamic=True,
max_new_tokens=args.max_new_tokens,
)
report[memory_variant]["performance"] = measure_performance(
dynamic,
tokenizer,
context_cases,
dynamic=True,
max_new_tokens=args.max_new_tokens,
repeats=args.perf_repeats,
)
if dynamic_device.type == "cuda":
report[memory_variant]["peak_memory_allocated_gb"] = torch.cuda.max_memory_allocated(dynamic_device) / 1024**3
report[memory_variant]["peak_memory_reserved_gb"] = torch.cuda.max_memory_reserved(dynamic_device) / 1024**3
original_embedded_payload = memory_payload(dynamic)
memory_report = run_memory_benchmark(
dynamic,
tokenizer,
args.memory_model,
max_new_tokens=args.max_new_tokens,
no_4bit=args.no_4bit,
)
# Release the first Qwen backbone before constructing the fresh model used
# by the shard-backed restart check. This is important on a 12 GB GPU.
release(dynamic)
dynamic = None
restarted = None
try:
restarted = load_qwen_dynamic(args.memory_model, load_in_4bit=use_4bit)
restarted.eval()
after_restart = {}
for name, query, expected in (
("work_code", "我的工作地点代号是什么？", "NM-K9"),
("fruit", "我最喜欢的水果是什么？", "青提"),
):
response, prefix_used = answer_memory_query(restarted, tokenizer, query, args.max_new_tokens)
after_restart[name] = {
"query": query,
"expected": expected,
"response": response,
"expected_found": expected in response,
"prefix_used": prefix_used,
}
restarted.reset_memory(batch_size=1, device=restarted._find_layer_device())
reset_response, reset_prefix = answer_memory_query(
restarted,
tokenizer,
"我的工作地点代号是什么？",
args.max_new_tokens,
)
reset_slots = int(restarted.runtime.text_slot_valid.sum().item())
memory_report.update(
{
"after_restart_without_history_or_pt": after_restart,
"restart_pass": all(item["expected_found"] for item in after_restart.values()),
"reset": {
"response": reset_response,
"prefix_used": reset_prefix,
"valid_slots": reset_slots,
"cleared": reset_slots == 0 and not reset_prefix,
},
}
)
finally:
if restarted is not None:
# Restore the user's pre-benchmark state, so the benchmark itself
# does not overwrite the active embedded memory snapshot.
restarted._load_persistent_memory_payload(original_embedded_payload)
restarted.save_embedded_memory_weights(args.memory_model)
memory_report["state_restored"] = int(restarted.runtime.text_slot_valid.sum().item()) == int(
original_embedded_payload["text_slot_valid"].sum().item()
)
release(restarted)
report["memory"] = memory_report
baseline_quality = report["baseline"]["quality"]
dynamic_quality = report[memory_variant]["quality"]
categories = sorted(
set(baseline_quality["categories"]) & set(dynamic_quality["categories"])
)
category_deltas = {
category: dynamic_quality["categories"][category]["score"]
- baseline_quality["categories"][category]["score"]
for category in categories
}
report["comparison"] = {
"overall_delta": dynamic_quality["overall_score"] - baseline_quality["overall_score"],
"category_deltas": category_deltas,
"peak_memory_allocated_delta_gb": report[memory_variant].get("peak_memory_allocated_gb", 0.0)
- report["baseline"].get("peak_memory_allocated_gb", 0.0),
"load_seconds_delta": report[memory_variant]["load_seconds"]
- report["baseline"]["load_seconds"],
}
output = Path(args.output)
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
compact = {
"baseline_score": baseline_quality["overall_score"],
f"{memory_variant}_score": dynamic_quality["overall_score"],
"overall_delta": report["comparison"]["overall_delta"],
"memory_restart_pass": report["memory"]["restart_pass"],
"memory_reset_pass": report["memory"]["reset"]["cleared"],
"cases": len(all_cases),
"output": str(output),
}
print(json.dumps(compact, ensure_ascii=False, indent=2))
if __name__ == "__main__":
main()
benchmark_qwen.py
+423
View File
@@ -0,0 +1,423 @@
"""Compare the unmodified local Qwen checkpoint with a memory-surgery adapter."""
from __future__ import annotations
import argparse
import gc
import json
import math
import time
from pathlib import Path
from typing import Any, Callable
import torch
from .qwen_integration import QwenMemoryConfig, load_memory_config, load_qwen_base, load_qwen_dynamic, load_tokenizer
from .train_qwen_memory import encode_messages, load_records, pad_batch
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--model-path", default=".")
parser.add_argument("--data", default="V2_dpskw/data/demo_stream.jsonl")
parser.add_argument(
"--adapter",
default="V2_dpskw/qwen_memory_adapter_surgery_smoke",
help="dynamic-memory adapter directory; its memory_config.json selects the surgery mode",
)
parser.add_argument("--output", default="V2_dpskw/benchmark_qwen.json")
parser.add_argument("--max-length", type=int, default=512)
parser.add_argument("--repeats", type=int, default=3)
parser.add_argument("--warmup", type=int, default=1)
parser.add_argument("--max-new-tokens", type=int, default=8)
parser.add_argument("--seed", type=int, default=42)
parser.add_argument("--no-4bit", action="store_true")
return parser.parse_args()
def _adapter_config(adapter_dir: str | Path) -> QwenMemoryConfig:
return load_memory_config(adapter_dir)
def _score_output(output: Any, labels: torch.Tensor) -> tuple[float, int, int, int, bool]:
shifted_labels = labels[..., 1:]
predictions = output.logits[..., :-1, :].argmax(dim=-1)
target_positions = shifted_labels != -100
token_count = int(target_positions.sum().item())
if output.loss is None or token_count == 0:
raise RuntimeError("benchmark example has no supervised target tokens")
correct_tokens = int((predictions[target_positions] == shifted_labels[target_positions]).sum().item())
first_target = target_positions.nonzero(as_tuple=False)[0]
first_token_correct = int(
predictions[first_target[0], first_target[1]] == shifted_labels[first_target[0], first_target[1]]
)
sequence_ok = correct_tokens == token_count
return float(output.loss.detach().item()), token_count, correct_tokens, first_token_correct, sequence_ok
def _evaluate_base(model: Any, tokenizer: Any, records: list[dict[str, Any]], max_length: int) -> dict[str, float]:
device = model.get_input_embeddings().weight.device
total_nll = 0.0
total_tokens = 0
correct_tokens = 0
correct_first_tokens = 0
correct_sequences = 0
for record in records:
query = encode_messages(tokenizer, record["query"], max_length)
input_ids, attention_mask, labels = pad_batch([query], int(tokenizer.pad_token_id))
with torch.inference_mode():
output = model(
input_ids=input_ids.to(device),
attention_mask=attention_mask.to(device),
labels=labels.to(device),
use_cache=False,
)
loss, tokens, tokens_correct, first_token_correct, sequence_ok = _score_output(output, labels.to(device))
total_nll += loss * tokens
total_tokens += tokens
correct_tokens += tokens_correct
correct_first_tokens += first_token_correct
correct_sequences += int(sequence_ok)
mean_loss = total_nll / total_tokens
return {
"loss": mean_loss,
"perplexity": math.exp(mean_loss),
"token_accuracy": correct_tokens / total_tokens,
"first_target_token_accuracy": correct_first_tokens / len(records),
"exact_sequence_accuracy": correct_sequences / len(records),
"supervised_tokens": total_tokens,
}
def _evaluate_dynamic(
model: Any,
tokenizer: Any,
records: list[dict[str, Any]],
max_length: int,
) -> dict[str, float]:
device = model._find_layer_device()
pad_id = int(tokenizer.pad_token_id)
total_nll = 0.0
total_tokens = 0
correct_tokens = 0
correct_first_tokens = 0
correct_sequences = 0
for record in records:
model.reset_memory()
memory = encode_messages(tokenizer, record["memory"], max_length)
query = encode_messages(tokenizer, record["query"], max_length)
memory_input, memory_mask, _ = pad_batch([memory], pad_id)
query_input, query_mask, query_labels = pad_batch([query], pad_id)
with torch.inference_mode():
memory_output = model(
input_ids=memory_input.to(device),
attention_mask=memory_mask.to(device),
read_memory=False,
update_memory=True,
return_memory=True,
use_cache=False,
)
output = model(
input_ids=query_input.to(device),
attention_mask=query_mask.to(device),
labels=query_labels.to(device),
memory_state=memory_output.memory,
read_memory=True,
update_memory=False,
return_memory=True,
use_cache=False,
)
loss, tokens, tokens_correct, first_token_correct, sequence_ok = _score_output(
output, query_labels.to(device)
)
total_nll += loss * tokens
total_tokens += tokens
correct_tokens += tokens_correct
correct_first_tokens += first_token_correct
correct_sequences += int(sequence_ok)
model.reset_memory()
mean_loss = total_nll / total_tokens
return {
"loss": mean_loss,
"perplexity": math.exp(mean_loss),
"token_accuracy": correct_tokens / total_tokens,
"first_target_token_accuracy": correct_first_tokens / len(records),
"exact_sequence_accuracy": correct_sequences / len(records),
"supervised_tokens": total_tokens,
}
def _sync(device: torch.device) -> None:
if device.type == "cuda":
torch.cuda.synchronize(device)
def _measure(name: str, fn: Callable[[], int], repeats: int, warmup: int, device: torch.device) -> dict[str, float]:
for _ in range(warmup):
fn()
_sync(device)
started = time.perf_counter()
processed = 0
for _ in range(repeats):
processed += fn()
_sync(device)
elapsed = time.perf_counter() - started
return {
"seconds": elapsed / repeats,
"tokens_per_second": processed / elapsed,
}
def _generation_prompt(tokenizer: Any, messages: list[dict[str, Any]], device: torch.device) -> dict[str, torch.Tensor]:
encoded = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
enable_thinking=False,
)
input_ids = encoded["input_ids"] if hasattr(encoded, "__getitem__") and "input_ids" in encoded else encoded
if isinstance(input_ids, torch.Tensor):
input_ids = input_ids.tolist()
if input_ids and isinstance(input_ids[0], list):
input_ids = input_ids[0]
ids = torch.tensor([input_ids], dtype=torch.long, device=device)
return {"input_ids": ids, "attention_mask": torch.ones_like(ids)}
def _measure_base_generation(
model: Any,
tokenizer: Any,
record: dict[str, Any],
repeats: int,
warmup: int,
max_new_tokens: int,
) -> dict[str, float]:
device = model.get_input_embeddings().weight.device
prompt = _generation_prompt(tokenizer, record["query"][:-1], device)
def run() -> int:
output = model.generate(
**prompt,
max_new_tokens=max_new_tokens,
do_sample=False,
use_cache=True,
pad_token_id=tokenizer.pad_token_id,
)
return int(output.shape[1] - prompt["input_ids"].shape[1])
return _measure("baseline_generation", run, repeats, warmup, device)
def _measure_dynamic_generation(
model: Any,
tokenizer: Any,
record: dict[str, Any],
max_length: int,
repeats: int,
warmup: int,
max_new_tokens: int,
) -> dict[str, float]:
device = model._find_layer_device()
pad_id = int(tokenizer.pad_token_id)
memory = encode_messages(tokenizer, record["memory"], max_length)
memory_input, memory_mask, _ = pad_batch([memory], pad_id)
prompt = _generation_prompt(tokenizer, record["query"][:-1], device)
def run() -> int:
model.reset_memory()
with torch.inference_mode():
model(
input_ids=memory_input.to(device),
attention_mask=memory_mask.to(device),
read_memory=False,
update_memory=True,
return_memory=True,
use_cache=False,
)
output = model.generate(
**prompt,
max_new_tokens=max_new_tokens,
do_sample=False,
update_memory=False,
use_cache=True,
pad_token_id=tokenizer.pad_token_id,
)
return int(output.shape[1] - prompt["input_ids"].shape[1])
result = _measure("dynamic_generation", run, repeats, warmup, device)
model.reset_memory()
return result
def _clean_generated(text: str) -> str:
return text.replace(" ", "").replace("\r", "").replace("\n", "").strip()
def _generation_quality_base(model: Any, tokenizer: Any, records: list[dict[str, Any]], max_new_tokens: int) -> dict[str, Any]:
device = model.get_input_embeddings().weight.device
contains = 0
prefixes = 0
examples = []
for record in records:
prompt = _generation_prompt(tokenizer, record["query"][:-1], device)
with torch.inference_mode():
output = model.generate(
**prompt,
max_new_tokens=max_new_tokens,
do_sample=False,
use_cache=True,
pad_token_id=tokenizer.pad_token_id,
)
generated = _clean_generated(tokenizer.decode(output[0, prompt["input_ids"].shape[1] :], skip_special_tokens=True))
expected = str(record["answer"])
contains += int(expected in generated)
prefixes += int(generated.startswith(expected))
if len(examples) < 3:
examples.append({"expected": expected, "generated": generated})
return {
"answer_contains_accuracy": contains / len(records),
"answer_prefix_accuracy": prefixes / len(records),
"examples": examples,
}
def _generation_quality_dynamic(
model: Any,
tokenizer: Any,
records: list[dict[str, Any]],
max_length: int,
max_new_tokens: int,
) -> dict[str, Any]:
device = model._find_layer_device()
pad_id = int(tokenizer.pad_token_id)
contains = 0
prefixes = 0
examples = []
for record in records:
model.reset_memory()
memory = encode_messages(tokenizer, record["memory"], max_length)
memory_input, memory_mask, _ = pad_batch([memory], pad_id)
prompt = _generation_prompt(tokenizer, record["query"][:-1], device)
with torch.inference_mode():
model(
input_ids=memory_input.to(device),
attention_mask=memory_mask.to(device),
read_memory=False,
update_memory=True,
return_memory=True,
use_cache=False,
)
output = model.generate(
**prompt,
max_new_tokens=max_new_tokens,
do_sample=False,
update_memory=False,
use_cache=True,
pad_token_id=tokenizer.pad_token_id,
)
generated = _clean_generated(tokenizer.decode(output[0, prompt["input_ids"].shape[1] :], skip_special_tokens=True))
expected = str(record["answer"])
contains += int(expected in generated)
prefixes += int(generated.startswith(expected))
if len(examples) < 3:
examples.append({"expected": expected, "generated": generated})
model.reset_memory()
return {
"answer_contains_accuracy": contains / len(records),
"answer_prefix_accuracy": prefixes / len(records),
"examples": examples,
}
def _release(model: Any) -> None:
del model
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
def main() -> None:
args = parse_args()
if args.repeats < 1 or args.warmup < 0:
raise ValueError("repeats must be >= 1 and warmup must be >= 0")
torch.manual_seed(args.seed)
tokenizer = load_tokenizer(args.model_path)
records = load_records(args.data)
use_4bit = not args.no_4bit
results: dict[str, Any] = {
"model_path": str(Path(args.model_path).resolve()),
"data": str(Path(args.data).resolve()),
"adapter": str(Path(args.adapter).resolve()),
"records": len(records),
"max_length": args.max_length,
"quantization": "4bit_nf4" if use_4bit else "none",
}
print("loading baseline")
baseline = load_qwen_base(args.model_path, load_in_4bit=use_4bit)
baseline.eval()
baseline_device = baseline.get_input_embeddings().weight.device
if baseline_device.type == "cuda":
torch.cuda.reset_peak_memory_stats(baseline_device)
results["baseline"] = {
"device": str(baseline_device),
"scores": _evaluate_base(baseline, tokenizer, records, args.max_length),
"generation_quality": _generation_quality_base(baseline, tokenizer, records, args.max_new_tokens),
"generation": _measure_base_generation(
baseline, tokenizer, records[0], args.repeats, args.warmup, args.max_new_tokens
),
}
if baseline_device.type == "cuda":
results["baseline"]["peak_memory_gb"] = torch.cuda.max_memory_allocated(baseline_device) / 1024**3
_release(baseline)
baseline = None
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
memory_config = _adapter_config(args.adapter)
print(f"loading dynamic mode={memory_config.mode} layers={memory_config.layer_indices}")
dynamic = load_qwen_dynamic(
args.model_path,
memory_config=memory_config,
load_in_4bit=use_4bit,
)
dynamic.load_memory_adapter(args.adapter)
dynamic.eval()
dynamic_device = dynamic._find_layer_device()
if dynamic_device.type == "cuda":
torch.cuda.reset_peak_memory_stats(dynamic_device)
results["dynamic_memory"] = {
"device": str(dynamic_device),
"mode": memory_config.mode,
"layers": list(dynamic.layer_indices),
"scores": _evaluate_dynamic(dynamic, tokenizer, records, args.max_length),
"generation_quality": _generation_quality_dynamic(
dynamic, tokenizer, records, args.max_length, args.max_new_tokens
),
"generation": _measure_dynamic_generation(
dynamic,
tokenizer,
records[0],
args.max_length,
args.repeats,
args.warmup,
args.max_new_tokens,
),
"trainable_parameters": sum(parameter.numel() for parameter in dynamic.trainable_parameters),
}
if dynamic_device.type == "cuda":
results["dynamic_memory"]["peak_memory_gb"] = torch.cuda.max_memory_allocated(dynamic_device) / 1024**3
_release(dynamic)
dynamic = None
output_path = Path(args.output)
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(json.dumps(results, indent=2), encoding="utf-8")
print(json.dumps(results, indent=2, ensure_ascii=False))
print(f"saved={output_path.resolve()}")
if __name__ == "__main__":
main()
benchmark_real_scale_memory_4b.py
+878
View File
@@ -0,0 +1,878 @@
"""Run large-scale natural-language and real-repository memory benchmarks.
The benchmark keeps the memory bank in process RAM and caps the model process
at a declared HBM budget. It measures two separate workloads:
* naturalistic multi-turn memory episodes from the local native-memory corpus;
* real source and documentation from this repository, expanded to a large
page library and queried through actual file/symbol questions.
The primary outputs are peak VRAM and correctness. Retrieval is evaluated on
every query, while generation is evaluated on a substantial holdout subset so
that a high retrieval score cannot be mistaken for end-to-end chat quality.
"""
from __future__ import annotations
import argparse
import gc
import json
import random
import re
import time
from collections import Counter
from pathlib import Path
from statistics import mean, median
from typing import Any, Iterable
import torch
import torch.nn.functional as F
from .qwen_integration import format_memory_evidence, load_qwen_dynamic, load_tokenizer
from .stream_chat_qwen_memory import _chat_tensor, _memory_system_prefix
PROJECT_ROOT = Path(__file__).resolve().parent
DEFAULT_MEMORY_MODEL = PROJECT_ROOT / "qwen3_5_4b_natural_memory_v2"
REFUSAL_MARKERS = ("不知道", "没有记录", "无相关", "未找到", "不清楚", "无法确认")
def _path(value: str | Path) -> Path:
path = Path(value)
return path if path.is_absolute() or path.exists() else PROJECT_ROOT / path
def _max_memory(gpu_memory_gb: float) -> dict[Any, str] | None:
if gpu_memory_gb <= 0.0 or not torch.cuda.is_available():
return None
return {0: f"{gpu_memory_gb:.1f}GiB", "cpu": "64GiB"}
def _set_cuda_process_cap(gpu_memory_gb: float) -> None:
if gpu_memory_gb <= 0.0 or not torch.cuda.is_available():
return
total = torch.cuda.get_device_properties(0).total_memory
fraction = min(0.95, max(0.05, gpu_memory_gb * 1024**3 / total))
torch.cuda.set_per_process_memory_fraction(fraction, device=0)
def _sync(device: torch.device) -> None:
if device.type == "cuda":
torch.cuda.synchronize(device)
def _encode_texts(
model: Any,
tokenizer: Any,
texts: list[str],
device: torch.device,
*,
batch_size: int = 16,
) -> torch.Tensor:
outputs: list[torch.Tensor] = []
for start in range(0, len(texts), max(1, batch_size)):
batch = texts[start : start + max(1, batch_size)]
encoded = tokenizer(
batch,
add_special_tokens=False,
padding=True,
return_tensors="pt",
)
input_ids = encoded["input_ids"].to(device)
mask = encoded["attention_mask"].to(device)
outputs.append(model._encode_model_key(input_ids, mask).detach().cpu())
if not outputs:
return torch.empty((0, model.memory.hidden_size), dtype=torch.float32)
return torch.cat(outputs, dim=0)
def _query_token_ids(tokenizer: Any, text: str) -> torch.Tensor:
return torch.tensor(
tokenizer(text, add_special_tokens=False)["input_ids"],
dtype=torch.long,
)
def _normal(text: Any) -> str:
return re.sub(r"\s+", "", str(text)).lower()
def _contains_answer(response: str, expected: str) -> bool:
expected_normal = _normal(expected)
return bool(expected_normal) and expected_normal in _normal(response)
def _is_refusal(response: str) -> bool:
"""Recognize concise and natural-language abstentions."""
normalized = _normal(response)
if any(marker in normalized for marker in REFUSAL_MARKERS):
return True
return bool(
re.search(
r"(没有|无|未|不包含|无法).{0,80}(记录|资料|信息|数据|找到|知道|访问|交互)",
normalized,
)
)
def _quantiles(values: Iterable[float]) -> dict[str, float | None]:
items = sorted(float(value) for value in values)
if not items:
return {"mean": None, "median": None, "p95": None, "max": None}
index = min(len(items) - 1, max(0, int(round(0.95 * (len(items) - 1)))))
return {
"mean": mean(items),
"median": median(items),
"p95": items[index],
"max": items[-1],
}
def _load_jsonl(path: Path, limit: int | None = None) -> list[dict[str, Any]]:
rows: list[dict[str, Any]] = []
with path.open("r", encoding="utf-8") as handle:
for line in handle:
if not line.strip():
continue
rows.append(json.loads(line))
if limit is not None and len(rows) >= limit:
break
return rows
def _native_cases(data_root: Path, max_cases: int) -> list[dict[str, Any]]:
"""Load train/eval native-memory episodes without loading large files."""
rows: list[dict[str, Any]] = []
sources = (
("train", data_root / "native_memory" / "train.jsonl"),
("eval", data_root / "native_memory" / "eval.jsonl"),
)
remaining = max(1, max_cases)
for split, path in sources:
if not path.exists() or remaining <= 0:
continue
loaded = _load_jsonl(path, remaining)
for row in loaded:
rows.append({"split": split, **row})
remaining -= len(loaded)
if not rows:
raise FileNotFoundError(f"native memory corpus not found under {data_root}")
return rows
def _fact_text(chunk: dict[str, Any]) -> str:
messages = chunk.get("messages") or []
for message in messages:
if isinstance(message, dict) and message.get("role") == "user":
return str(message.get("content", ""))
return str(chunk.get("text", ""))
def _build_general_records(
cases: list[dict[str, Any]],
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
"""Convert local conversation episodes into versioned memory records."""
records: list[dict[str, Any]] = []
query_rows: list[dict[str, Any]] = []
for case in cases:
subject = str(case.get("subject", ""))
attribute = str(case.get("attribute", ""))
chunks = case.get("memory_chunks") or []
for chunk in chunks:
if not isinstance(chunk, dict) or float(chunk.get("write_label", 0.0)) < 0.5:
continue
text = _fact_text(chunk).strip()
value = str(chunk.get("value", ""))
if not text or not value:
continue
records.append(
{
"text": text,
"entity": subject,
"attribute": attribute,
"value": value,
"memory_type": "personal_fact",
"importance": 0.9,
"confidence": 0.95,
"source": "native_memory_corpus",
"trusted": True,
"force": True,
}
)
query_messages = case.get("query") or []
query = ""
for message in query_messages:
if isinstance(message, dict) and message.get("role") == "user":
query = str(message.get("content", ""))
break
if not query:
query = f"请查询 {subject} 的 {attribute}。"
answer = str(case.get("answer", ""))
answerable = bool(case.get("answerable", False)) and answer not in REFUSAL_MARKERS
query_rows.append(
{
"id": str(case.get("id", "")),
"split": str(case.get("split", "unknown")),
"query": query,
"subject": subject,
"attribute": attribute,
"expected": answer,
"answerable": answerable,
}
)
return records, query_rows
def _source_files() -> list[Path]:
allowed = {".py", ".md", ".json"}
files: list[Path] = []
for path in PROJECT_ROOT.rglob("*"):
if not path.is_file() or path.suffix.lower() not in allowed:
continue
if "__pycache__" in path.parts or "checkpoints" in path.parts:
continue
if path.name.endswith(".safetensors"):
continue
files.append(path)
return sorted(files)
def _source_chunks(files: list[Path], *, chars_per_chunk: int = 1800) -> list[str]:
chunks: list[str] = []
for path in files:
text = path.read_text(encoding="utf-8", errors="replace")
if not text:
continue
for start in range(0, len(text), chars_per_chunk):
piece = text[start : start + chars_per_chunk].strip()
if piece:
chunks.append(piece)
if not chunks:
raise RuntimeError("repository source corpus is empty")
return chunks
def _project_targets(files: list[Path], limit: int) -> list[dict[str, Any]]:
pattern = re.compile(r"^\s*(class|async\s+def|def)\s+([A-Za-z_][A-Za-z0-9_]*)")
candidates: list[dict[str, Any]] = []
for path in files:
if path.suffix.lower() != ".py":
continue
relative = path.relative_to(PROJECT_ROOT).as_posix()
for line_number, line in enumerate(path.read_text(encoding="utf-8", errors="replace").splitlines(), 1):
match = pattern.match(line)
if match is None:
continue
kind = match.group(1).replace("async ", "")
name = match.group(2)
text = (
f"真实代码库事实：文件 {relative} 的第 {line_number} 行定义了 "
f"{kind} {name}。原始代码行：{line.strip()}"
)
candidates.append(
{
"file": relative,
"line": line_number,
"kind": kind,
"name": name,
"text": text,
"expected": relative,
}
)
if not candidates:
raise RuntimeError("no Python symbols found in repository corpus")
if len(candidates) <= limit:
return candidates
# Evenly sample the repository instead of measuring only the first file.
indices = [int(index * len(candidates) / limit) for index in range(limit)]
return [candidates[index] for index in indices]
def _build_project_records(
files: list[Path],
*,
record_count: int,
target_count: int,
chunk_tokens: int,
) -> tuple[list[dict[str, Any]], list[dict[str, Any]], dict[str, Any]]:
chunks = _source_chunks(files)
targets = _project_targets(files, target_count)
rng = random.Random(20260905)
positions = rng.sample(range(record_count), min(len(targets), record_count))
position_to_target = dict(zip(positions, targets))
records: list[dict[str, Any]] = []
for index in range(record_count):
target = position_to_target.get(index)
if target is None:
text = (
f"项目源码快照分片 {index}：\n"
f"{chunks[index % len(chunks)]}"
)
records.append(
{
"text": text,
"key_kind": "random_filler",
"memory_type": "repository_chunk",
"importance": 0.4,
"confidence": 0.8,
"source": "real_repository_snapshot",
"trusted": True,
"force": True,
}
)
continue
records.append(
{
"text": target["text"],
"entity": target["file"],
"attribute": f"symbol:{target['name']}",
"value": target["file"],
"memory_type": "repository_symbol",
"importance": 0.9,
"confidence": 0.95,
"source": "real_repository_snapshot",
"target": target,
"trusted": True,
"force": True,
}
)
queries: list[dict[str, Any]] = []
for target in targets:
queries.extend(
[
{
"query": f"在真实代码库中，文件 {target['file']} 里的 {target['name']} 定义在哪个文件？",
"expected": target["expected"],
"target": target,
},
{
"query": f"请从项目记忆查找：{target['file']} 的 {target['kind']} {target['name']} 位于哪里？",
"expected": target["expected"],
"target": target,
},
]
)
return records, queries, {
"source_file_count": len(files),
"source_characters": sum(path.stat().st_size for path in files),
"source_chunk_count": len(chunks),
"capacity_tokens": int(record_count * chunk_tokens),
"chunk_tokens": int(chunk_tokens),
"target_count": len(targets),
}
def _prepare_records(
model: Any,
tokenizer: Any,
records: list[dict[str, Any]],
device: torch.device,
*,
batch_size: int,
) -> list[dict[str, Any]]:
semantic_indices = [
index for index, record in enumerate(records)
if record.get("key_kind") != "random_filler"
]
semantic_keys = _encode_texts(
model,
tokenizer,
[str(records[index]["text"]) for index in semantic_indices],
device,
batch_size=batch_size,
)
key_by_index = {
index: semantic_keys[position]
for position, index in enumerate(semantic_indices)
}
prepared: list[dict[str, Any]] = []
for index, record in enumerate(records):
item = {key: value for key, value in record.items() if key not in {"target", "key_kind"}}
if record.get("key_kind") == "random_filler":
generator = torch.Generator(device="cpu").manual_seed(900000 + index)
key = F.normalize(torch.randn(model.memory.hidden_size, generator=generator), dim=0)
semantic_key = None
else:
key = F.normalize(key_by_index[index].float(), dim=0)
semantic_key = key
item["key"] = key
item["summary"] = key
# Background corpus chunks are intentionally not semantic candidates
# in this bounded stress run. Leaving their semantic key unset lets
# the bank use the learned reranker for real evidence while retaining
# the chunks as corpus noise rather than pretending their random
# addresses were learned representations.
item["semantic_key"] = semantic_key
if record.get("key_kind") != "random_filler":
# Match the production writer: retrieved memories enter Qwen as
# an internal system-message prefix, while record.text remains
# the raw fact used by the address/routing layer.
evidence_text = format_memory_evidence(
str(record["text"]),
entity=str(record.get("entity", "")),
attribute=str(record.get("attribute", "")),
value=str(record.get("value", "")),
)
storage = _memory_system_prefix(
tokenizer,
"以下是与当前用户相关的已保存长期记忆。仅在问题相关时使用，"
"只能依据明确证据；先核对实体、属性和已确认值；冲突优先最新可靠来源，"
"不要拼接不确定候选，证据不足就明确说不知道；涉及名称、路径、token、"
"参数或结论时，原样复述证据中的关键短语：\n"
+ evidence_text,
)
token_ids = storage["input_ids"].reshape(-1).to(dtype=torch.long)
token_mask = storage["attention_mask"].reshape(-1).to(dtype=torch.bool)
item["token_ids"] = token_ids
item["token_mask"] = token_mask
prepared.append(item)
return prepared
def _clear_memory_bank(model: Any, device: torch.device) -> None:
model.memory_os_v2 = model._new_memory_os_v2(model.memory.hidden_size)
model.reset_memory(batch_size=1, device=device)
def _direct_read(
model: Any,
tokenizer: Any,
query_key: torch.Tensor,
row: dict[str, Any],
device: torch.device,
) -> dict[str, Any]:
query = str(row["query"])
token_ids = _query_token_ids(tokenizer, query)
_sync(device)
started = time.perf_counter()
records, decision = model.memory_os_v2.read(
query_key=query_key,
query_text=query,
query_token_ids=token_ids,
top_k_pages=4,
top_k_records=2,
max_hops=3,
)
_sync(device)
elapsed_ms = (time.perf_counter() - started) * 1000.0
returned_values = [record.value for record in records]
if row.get("answerable", True):
hit = any(_contains_answer(value, str(row.get("expected", ""))) for value in returned_values)
else:
hit = len(records) == 0
return {
"id": row.get("id", ""),
"query": query,
"expected": row.get("expected", ""),
"answerable": bool(row.get("answerable", True)),
"hit": bool(hit),
"returned_values": returned_values,
"returned_ids": [record.record_id for record in records],
"stop_reason": decision.stop_reason,
"coarse_candidates": model.memory_os_v2.bank._last_coarse_candidates,
"elapsed_ms": elapsed_ms,
}
def _record_audit_view(record: Any) -> dict[str, Any]:
"""Keep generation diagnostics readable without serializing embeddings."""
return {
"record_id": str(getattr(record, "record_id", "")),
"entity": str(getattr(record, "entity", "")),
"attribute": str(getattr(record, "attribute", "")),
"value": str(getattr(record, "value", "")),
"status": str(getattr(record, "status", "")),
"version": int(getattr(record, "version", 0)),
"source": str(getattr(record, "source", "")),
"text_preview": str(getattr(record, "text", ""))[:300],
}
def _chat_generate(
model: Any,
tokenizer: Any,
query: str,
device: torch.device,
max_new_tokens: int,
) -> dict[str, Any]:
encoded = _chat_tensor(
tokenizer,
query,
)
encoded = {key: value.to(device) for key, value in encoded.items()}
raw = tokenizer(query, add_special_tokens=False, return_tensors="pt")
query_ids = raw["input_ids"].to(device)
query_mask = raw.get("attention_mask")
if query_mask is None:
query_mask = torch.ones_like(query_ids)
_sync(device)
generation_started = time.perf_counter()
output = model.generate(
**encoded,
max_new_tokens=max_new_tokens,
do_sample=False,
update_memory=False,
memory_query_input_ids=query_ids,
memory_query_attention_mask=query_mask.to(device),
memory_query_text=query,
use_cache=True,
pad_token_id=tokenizer.pad_token_id,
)
_sync(device)
generation_seconds = time.perf_counter() - generation_started
response_ids = output[0, encoded["input_ids"].shape[1] :]
response = tokenizer.decode(response_ids.detach().cpu().tolist(), skip_special_tokens=True).strip()
runtime = getattr(model, "runtime", None)
raw_decisions = list(getattr(runtime, "v2_last_decisions", []) or [])
decisions: list[dict[str, Any]] = []
selected_record_ids: list[str] = []
for raw_decision in raw_decisions:
decision = dict(raw_decision)
record_ids = [str(item) for item in decision.get("record_ids", [])]
selected_record_ids.extend(record_ids)
decisions.append(
{
"need_memory": bool(decision.get("need_memory", False)),
"record_ids": record_ids,
"page_ids": [str(item) for item in decision.get("page_ids", [])],
"hop_count": int(decision.get("hop_count", 0)),
"confidence": float(decision.get("confidence", 0.0)),
"top_score": float(decision.get("top_score", 0.0)),
"score_margin": float(decision.get("score_margin", 0.0)),
"evidence_score": float(decision.get("evidence_score", 0.0)),
"stop_reason": str(decision.get("stop_reason", "")),
}
)
selected_record_ids = list(dict.fromkeys(selected_record_ids))
records_by_id = getattr(getattr(model, "memory_os_v2", None), "bank", None)
records_by_id = getattr(records_by_id, "records", {})
selected_records = [
_record_audit_view(records_by_id[record_id])
for record_id in selected_record_ids
if record_id in records_by_id
]
return {
"response": response,
"public_prompt_tokens": int(encoded["input_ids"].shape[1]),
"generated_tokens": int(response_ids.numel()),
"generation_seconds": float(generation_seconds),
"prefix_used": bool(getattr(runtime, "text_prefix_used", False)),
"guard_used": bool(getattr(runtime, "text_guard_used", False)),
"prefix_tokens": int(getattr(runtime, "text_prefix_tokens", 0)),
"memory_read_seconds": float(getattr(runtime, "text_read_seconds", 0.0)),
"selected_record_ids": selected_record_ids,
"selected_records": selected_records,
"selected_values": [record["value"] for record in selected_records],
"decisions": decisions,
"stop_reasons": [decision["stop_reason"] for decision in decisions],
}
def _generation_eval(
model: Any,
tokenizer: Any,
rows: list[dict[str, Any]],
device: torch.device,
*,
max_new_tokens: int,
reference_retrieval_rows: dict[str, dict[str, Any]] | None = None,
) -> dict[str, Any]:
output_rows: list[dict[str, Any]] = []
for row in rows:
generated = _chat_generate(model, tokenizer, str(row["query"]), device, max_new_tokens)
response = str(generated["response"])
answerable = bool(row.get("answerable", True))
expected = str(row.get("expected", ""))
retrieved_target = answerable and any(
_contains_answer(str(record.get("value", "")), expected)
or _contains_answer(str(record.get("text_preview", "")), expected)
for record in generated["selected_records"]
)
reference = (reference_retrieval_rows or {}).get(str(row.get("id", "")))
reference_target = bool(reference and reference.get("hit", False))
passed = (
_contains_answer(response, expected)
if answerable
else _is_refusal(response)
)
if answerable and passed:
error_class = "correct"
elif not answerable and passed:
error_class = "correct_refusal"
elif not answerable:
error_class = "refusal_failure"
elif reference_target and not retrieved_target:
error_class = "generation_retrieval_divergence"
elif not generated["selected_record_ids"]:
error_class = "no_memory_prefix"
elif not retrieved_target:
error_class = "retrieval_miss_or_wrong_prefix"
else:
error_class = "correct_evidence_ignored_or_overridden"
output_rows.append(
{
"id": row.get("id", ""),
"query": row["query"],
"expected": expected,
"response": response,
"answerable": answerable,
"correct": bool(passed),
"retrieval_target_found": bool(retrieved_target),
"reference_retrieval_hit": reference_target,
"reference_retrieved_values": list(reference.get("returned_values", [])) if reference else [],
"retrieved_values": list(generated["selected_values"]),
"retrieved_ids": list(generated["selected_record_ids"]),
"selected_records": list(generated["selected_records"]),
"prefix_used": bool(generated["prefix_used"]),
"prefix_tokens": int(generated["prefix_tokens"]),
"memory_read_seconds": float(generated["memory_read_seconds"]),
"decisions": list(generated["decisions"]),
"stop_reasons": list(generated["stop_reasons"]),
"error_class": error_class,
}
)
answerable_rows = [row for row in output_rows if row["answerable"]]
unknown_rows = [row for row in output_rows if not row["answerable"]]
error_counts = Counter(row["error_class"] for row in output_rows)
failed_answerable = [row for row in answerable_rows if not row["correct"]]
return {
"cases": len(output_rows),
"answerable_cases": len(answerable_rows),
"answerable_correct": sum(int(row["correct"]) for row in answerable_rows),
"answerable_accuracy": sum(int(row["correct"]) for row in answerable_rows) / max(1, len(answerable_rows)),
"unknown_cases": len(unknown_rows),
"unknown_correct": sum(int(row["correct"]) for row in unknown_rows),
"unknown_refusal_accuracy": sum(int(row["correct"]) for row in unknown_rows) / max(1, len(unknown_rows)),
"error_class_counts": dict(sorted(error_counts.items())),
"failed_answerable_cases": len(failed_answerable),
"failed_answerable_with_correct_evidence": sum(
int(row["retrieval_target_found"]) for row in failed_answerable
),
"failed_answerable_with_reference_retrieval_hit": sum(
int(row["reference_retrieval_hit"]) for row in failed_answerable
),
"failed_answerable_with_retrieval_miss": sum(
int(not row["retrieval_target_found"]) for row in failed_answerable
),
"rows": output_rows,
}
def _retrieval_summary(rows: list[dict[str, Any]]) -> dict[str, Any]:
answerable = [row for row in rows if row["answerable"]]
unknown = [row for row in rows if not row["answerable"]]
return {
"queries": len(rows),
"answerable_queries": len(answerable),
"answerable_hits": sum(int(row["hit"]) for row in answerable),
"answerable_recall": sum(int(row["hit"]) for row in answerable) / max(1, len(answerable)),
"unknown_queries": len(unknown),
"unknown_correct_abstentions": sum(int(row["hit"]) for row in unknown),
"unknown_abstention_accuracy": sum(int(row["hit"]) for row in unknown) / max(1, len(unknown)),
"read_latency_ms": _quantiles(row["elapsed_ms"] for row in rows),
"mean_coarse_candidates": mean(row["coarse_candidates"] for row in rows) if rows else 0.0,
"max_coarse_candidates": max((row["coarse_candidates"] for row in rows), default=0),
}
def _vram_snapshot(device: torch.device) -> dict[str, float | None]:
if device.type != "cuda":
return {"allocated_gb": None, "reserved_gb": None, "peak_allocated_gb": None, "peak_reserved_gb": None}
return {
"allocated_gb": torch.cuda.memory_allocated(device) / 1024**3,
"reserved_gb": torch.cuda.memory_reserved(device) / 1024**3,
"peak_allocated_gb": torch.cuda.max_memory_allocated(device) / 1024**3,
"peak_reserved_gb": torch.cuda.max_memory_reserved(device) / 1024**3,
}
@torch.inference_mode()
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--base-model", default=r"H:\Memory")
parser.add_argument("--memory-model", default=str(DEFAULT_MEMORY_MODEL))
parser.add_argument("--data-root", default=str(PROJECT_ROOT / "data"))
parser.add_argument("--output", default=str(PROJECT_ROOT / "real_scale_memory_4b.json"))
parser.add_argument("--general-cases", type=int, default=640)
parser.add_argument("--project-records", type=int, default=8192)
parser.add_argument("--project-targets", type=int, default=256)
parser.add_argument("--project-chunk-tokens", type=int, default=512)
parser.add_argument("--generation-general", type=int, default=128)
parser.add_argument("--generation-project", type=int, default=64)
parser.add_argument("--max-new-tokens", type=int, default=32)
parser.add_argument("--encode-batch-size", type=int, default=16)
parser.add_argument("--gpu-memory-gb", type=float, default=10.0)
parser.add_argument("--no-4bit", action="store_true")
args = parser.parse_args()
_set_cuda_process_cap(args.gpu_memory_gb)
tokenizer = load_tokenizer(_path(args.base_model))
use_4bit = not args.no_4bit
print("loading Natural Memory 4B")
model = load_qwen_dynamic(
_path(args.memory_model),
load_in_4bit=use_4bit,
max_memory=_max_memory(args.gpu_memory_gb),
)
model.configure_memory_grounding_guard(tokenizer)
model.eval()
model.memory_config.memory_top_k_records = 2
model.memory_os_v2.bank.top_k_records = 2
device = model._find_layer_device()
model.reset_memory(batch_size=1, device=device)
_sync(device)
if device.type == "cuda":
torch.cuda.reset_peak_memory_stats(device)
baseline_vram = _vram_snapshot(device)
data_root = _path(args.data_root)
general_cases = _native_cases(data_root, max(1, int(args.general_cases)))
general_raw_records, general_queries = _build_general_records(general_cases)
general_records = _prepare_records(
model,
tokenizer,
general_raw_records,
device,
batch_size=max(1, int(args.encode_batch_size)),
)
_sync(device)
general_write_start = time.perf_counter()
model.memory_os_v2.write_batch(general_records)
_sync(device)
general_write_ms = (time.perf_counter() - general_write_start) * 1000.0
general_query_keys = _encode_texts(
model,
tokenizer,
[str(row["query"]) for row in general_queries],
device,
batch_size=max(1, int(args.encode_batch_size)),
)
general_retrieval_rows = [
_direct_read(model, tokenizer, general_query_keys[index], row, device)
for index, row in enumerate(general_queries)
]
general_retrieval = _retrieval_summary(general_retrieval_rows)
general_generation_rows = [
row
for row in general_queries
if row["split"] == "eval"
][: max(1, int(args.generation_general))]
general_generation = _generation_eval(
model,
tokenizer,
general_generation_rows,
device,
max_new_tokens=max(1, int(args.max_new_tokens)),
reference_retrieval_rows={str(row["id"]): row for row in general_retrieval_rows},
)
general_vram = _vram_snapshot(device)
_clear_memory_bank(model, device)
project_files = _source_files()
project_raw_records, project_queries, project_meta = _build_project_records(
project_files,
record_count=max(128, int(args.project_records)),
target_count=max(1, int(args.project_targets)),
chunk_tokens=max(1, int(args.project_chunk_tokens)),
)
project_records = _prepare_records(
model,
tokenizer,
project_raw_records,
device,
batch_size=max(1, int(args.encode_batch_size)),
)
_sync(device)
project_write_start = time.perf_counter()
model.memory_os_v2.write_batch(project_records)
_sync(device)
project_write_ms = (time.perf_counter() - project_write_start) * 1000.0
project_query_keys = _encode_texts(
model,
tokenizer,
[str(row["query"]) for row in project_queries],
device,
batch_size=max(1, int(args.encode_batch_size)),
)
project_retrieval_rows = []
for index, row in enumerate(project_queries):
direct_row = dict(row)
direct_row["answerable"] = True
project_retrieval_rows.append(
_direct_read(model, tokenizer, project_query_keys[index], direct_row, device)
)
project_retrieval = _retrieval_summary(project_retrieval_rows)
project_generation_rows = [
{"id": f"project-{index}", "query": row["query"], "expected": row["expected"], "answerable": True}
for index, row in enumerate(project_queries[: max(1, int(args.generation_project))])
]
project_generation = _generation_eval(
model,
tokenizer,
project_generation_rows,
device,
max_new_tokens=max(1, int(args.max_new_tokens)),
reference_retrieval_rows={str(row["id"]): row for row in project_retrieval_rows},
)
project_vram = _vram_snapshot(device)
report = {
"benchmark": "real_scale_memory_4b",
"model": str(_path(args.memory_model)),
"base_model": str(_path(args.base_model)),
"quantization": "4bit_nf4" if use_4bit else "none",
"gpu_memory_cap_gb": float(args.gpu_memory_gb),
"device": str(device),
"priority_metrics": ["peak_vram", "correctness"],
"baseline_vram_after_load": baseline_vram,
"general_chat_memory": {
"source": "local_native_memory_train_plus_eval",
"cases": len(general_cases),
"records_written": len(general_records),
"bank_records": len(model.memory_os_v2.bank.records),
"write_ms": general_write_ms,
"retrieval": general_retrieval,
"generation": general_generation,
"vram_after_general": general_vram,
},
"project_repository_memory": {
**project_meta,
"records_written": len(project_records),
"bank_records": len(model.memory_os_v2.bank.records),
"page_count": len(model.memory_os_v2.bank.pages),
"write_ms": project_write_ms,
"retrieval": project_retrieval,
"generation": project_generation,
"vram_after_project": project_vram,
},
"limitations": [
"本地 native_memory 语料是工程内置的自然语言基准，不等同于真实用户导出数据。",
"项目库目标来自当前仓库的真实源码和文档；大规模背景页用于压力测试，主要考察有界路由和显存。",
"生成质量与检索召回分别报告，不能用检索正确率替代端到端聊天正确率。",
"没有启用 SQLite 或磁盘分页；记忆主体保持在进程 RAM，只有命中的记录进入有界显存缓存。",
],
}
output = _path(args.output)
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps({
"general_retrieval": general_retrieval,
"general_generation": {key: value for key, value in general_generation.items() if key != "rows"},
"project_retrieval": project_retrieval,
"project_generation": {key: value for key, value in project_generation.items() if key != "rows"},
"vram": {"baseline": baseline_vram, "general": general_vram, "project": project_vram},
}, ensure_ascii=False, indent=2))
print(f"saved={output}")
del model
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
if __name__ == "__main__":
main()
benchmark_service_curves_4b.py
+497
View File
@@ -0,0 +1,497 @@
"""Compare long-context service curves on the local Qwen3.5 4B checkpoint.
This benchmark is intentionally framed around the production baselines that a
serving team would compare:
* dense/paged full prompt KV (until the checkpoint or GPU rejects it),
* a fixed sliding window,
* chunk retrieval followed by a short prompt-side rerank context, and
* Natural Memory with one bounded read at the request boundary.
The Natural Memory path never performs a memory read or write per generated
token. The reported ``reader_ms`` is measured inside the model wrapper and
``decode_tok_s`` is estimated from a one-token and a multi-token greedy run.
The synthetic task is deliberately simple and should be supplemented by
repository QA and agent-trace workloads before making a production claim.
"""
from __future__ import annotations
import argparse
import gc
import json
import math
import random
import time
from pathlib import Path
from typing import Any
import torch
from .qwen_integration import load_qwen_base, load_qwen_dynamic, load_tokenizer
from .stream_chat_qwen_memory import _chat_tensor
PROJECT_ROOT = Path(__file__).resolve().parent
def _path(value: str | Path) -> Path:
path = Path(value)
return path if path.is_absolute() or path.exists() else PROJECT_ROOT / path
def _sync(device: torch.device) -> None:
if device.type == "cuda":
torch.cuda.synchronize(device)
def _max_memory(gpu_memory_gb: float) -> dict[Any, str] | None:
if gpu_memory_gb <= 0.0 or not torch.cuda.is_available():
return None
return {0: f"{gpu_memory_gb:.1f}GiB", "cpu": "64GiB"}
def _set_cuda_process_cap(gpu_memory_gb: float) -> None:
"""Keep the benchmark inside the declared HBM operating point."""
if gpu_memory_gb <= 0.0 or not torch.cuda.is_available():
return
total = torch.cuda.get_device_properties(0).total_memory
fraction = min(0.95, max(0.05, gpu_memory_gb * 1024**3 / total))
torch.cuda.set_per_process_memory_fraction(fraction, device=0)
def _build_corpus(
tokenizer: Any,
target_tokens: int,
*,
chunk_tokens: int,
) -> tuple[torch.Tensor, torch.Tensor, str, str]:
"""Create a token-level corpus with a middle needle and a query."""
filler = (
"这是长上下文压力测试中的普通项目日志片段。它包含无关的版本号、"
"状态说明、时间戳和重复背景，不应被当作目标答案。"
)
needle = "用户档案字段 target_id 的值是 CURVE-7H2K-9P4M。"
query = "用户档案字段 target_id 的完整值是什么？请逐字符复制，不要省略。"
filler_ids = torch.tensor(
tokenizer(filler, add_special_tokens=False)["input_ids"], dtype=torch.long
)
needle_ids = torch.tensor(
tokenizer(needle, add_special_tokens=False)["input_ids"], dtype=torch.long
)
target_tokens = max(int(target_tokens), int(needle_ids.numel()) + 8)
filler_count = target_tokens - int(needle_ids.numel())
repeats = math.ceil(filler_count / max(1, filler_ids.numel()))
body = filler_ids.repeat(repeats)[:filler_count]
# Keep the needle outside a 32K hot window, while making it present in
# every measured prefix. The long-context axis then tests persistence,
# rather than accidentally testing a missing fact at the short end.
insert_at = min((body.numel() // 2) // max(1, chunk_tokens) * max(1, chunk_tokens), 65536)
corpus = torch.cat((body[:insert_at], needle_ids, body[insert_at:]), dim=0)
return corpus, needle_ids, needle, query
def _chat_from_content(tokenizer: Any, content: str, device: torch.device) -> dict[str, torch.Tensor]:
encoded = _chat_tensor(tokenizer, content)
return {key: value.to(device) for key, value in encoded.items()}
def _query_tensors(tokenizer: Any, query: str, device: torch.device) -> tuple[torch.Tensor, torch.Tensor]:
encoded = tokenizer(query, add_special_tokens=False, return_tensors="pt")
ids = encoded["input_ids"].to(device)
mask = encoded.get("attention_mask")
if mask is None:
mask = torch.ones_like(ids)
return ids, mask.to(device)
@torch.inference_mode()
def _measure_generation(
model: Any,
encoded: dict[str, torch.Tensor],
tokenizer: Any,
device: torch.device,
*,
dynamic: bool,
query_ids: torch.Tensor | None = None,
query_mask: torch.Tensor | None = None,
query_text: str = "",
max_new_tokens: int = 16,
) -> dict[str, Any]:
def generate(count: int):
kwargs: dict[str, Any] = {
"max_new_tokens": count,
"do_sample": False,
"use_cache": True,
"pad_token_id": tokenizer.pad_token_id,
}
if dynamic:
kwargs.update(
{
"update_memory": False,
"memory_query_input_ids": query_ids,
"memory_query_attention_mask": query_mask,
"memory_query_text": query_text,
}
)
_sync(device)
started = time.perf_counter()
output = model.generate(**encoded, **kwargs)
_sync(device)
return output, time.perf_counter() - started
try:
one, first_latency = generate(1)
many, total_latency = generate(max_new_tokens)
except (torch.cuda.OutOfMemoryError, RuntimeError) as exc:
if isinstance(exc, torch.cuda.OutOfMemoryError) or "out of memory" in str(exc).lower():
if device.type == "cuda":
torch.cuda.empty_cache()
return {"status": "cuda_oom", "error": str(exc)[:500]}
raise
generated = many[0, encoded["input_ids"].shape[1] :]
text = tokenizer.decode(generated.detach().cpu().tolist(), skip_special_tokens=True).strip()
extra = max(0.0, total_latency - first_latency)
generated_count = int(generated.numel())
runtime = getattr(model, "runtime", None)
prefix_tokens = int(getattr(runtime, "text_prefix_tokens", 0)) if dynamic else 0
public_input_tokens = int(encoded["input_ids"].shape[1])
row = {
"status": "ok",
"response": text,
"generated_tokens": generated_count,
"first_token_latency_s": first_latency,
"prefill_proxy_s": first_latency,
"total_latency_s": total_latency,
"batch_size": 1,
"public_input_tokens": public_input_tokens,
"memory_prefix_tokens": prefix_tokens,
"hot_kv_tokens": public_input_tokens + prefix_tokens,
"decode_tok_s": max(0, generated_count - 1) / max(extra, 1e-9),
"reader_ms": float(getattr(getattr(model, "runtime", None), "text_read_seconds", 0.0) * 1000.0)
if dynamic
else 0.0,
}
row["gpu_seconds_per_million_output_tokens"] = (
1_000_000.0 / max(1e-9, row["decode_tok_s"])
)
if device.type == "cuda":
row["peak_vram_gb"] = torch.cuda.max_memory_allocated(device) / 1024**3
row["peak_reserved_gb"] = torch.cuda.max_memory_reserved(device) / 1024**3
del one, many
torch.cuda.empty_cache()
return row
def _release(model: Any) -> None:
del model
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
def _quality_fields(row: dict[str, Any], *, target_reachable: bool, expected: str) -> dict[str, Any]:
"""Separate answer correctness from whether the target was in the hot context."""
found = expected in row.get("response", "")
row["target_reachable"] = bool(target_reachable)
row["quality_pass"] = found
row["quality_expected"] = bool(target_reachable)
row["quality_correct"] = bool(found == target_reachable) if row.get("status") == "ok" else False
return row
def _prompt_content(tokenizer: Any, corpus: torch.Tensor, query: str) -> str:
material = tokenizer.decode(corpus.tolist(), skip_special_tokens=True)
return (
"请从下面的材料中回答问题，不要使用材料外的信息。\n"
"---材料开始---\n" + material + "\n---材料结束---\n问题：" + query
)
def _chunk_retrieve(
tokenizer: Any,
corpus: torch.Tensor,
query: str,
*,
chunk_tokens: int,
top_k: int,
) -> tuple[torch.Tensor, float, int]:
query_ids = set(tokenizer(query, add_special_tokens=False)["input_ids"])
started = time.perf_counter()
scored: list[tuple[int, int, torch.Tensor]] = []
for start in range(0, int(corpus.numel()), chunk_tokens):
chunk = corpus[start : start + chunk_tokens]
overlap = len(query_ids.intersection(set(chunk.tolist())))
scored.append((overlap, -start, chunk))
scored.sort(key=lambda item: (item[0], item[1]), reverse=True)
selected = [item[2] for item in scored[:top_k]]
return torch.cat(selected, dim=0), (time.perf_counter() - started) * 1000.0, len(scored)
@torch.inference_mode()
def _add_memory_chunks(
model: Any,
tokenizer: Any,
corpus: torch.Tensor,
previous_tokens: int,
target_tokens: int,
*,
chunk_tokens: int,
device: torch.device,
needle_ids: torch.Tensor,
) -> dict[str, Any]:
"""Add context chunks without putting the long corpus in the decode KV."""
if model.memory_os_v2 is None:
raise RuntimeError("Natural Memory v2 is required for this benchmark")
# Filler records use deterministic CPU keys. The target record gets an
# actual frozen-Qwen key, which keeps the retrieval measurement honest for
# the one fact being scored without encoding millions of filler tokens.
records: list[dict[str, Any]] = []
body_tokens = max(0, int(corpus.numel()) - int(needle_ids.numel()))
target_start = min(
(body_tokens // 2) // max(1, chunk_tokens) * max(1, chunk_tokens),
65536,
)
for start in range(previous_tokens, target_tokens, chunk_tokens):
end = min(start + chunk_tokens, target_tokens)
ids = corpus[start:end]
if ids.numel() == 0:
continue
if start <= target_start < end:
key = model._encode_model_key(ids.unsqueeze(0).to(device), torch.ones((1, ids.numel()), dtype=torch.long, device=device))[0].cpu()
else:
generator = torch.Generator(device="cpu").manual_seed(1701 + start)
key = torch.randn(model.memory.hidden_size, generator=generator)
key = torch.nn.functional.normalize(key, dim=0)
record_text = (
tokenizer.decode(ids.tolist(), skip_special_tokens=True)
if start <= target_start < end
else f"curve_chunk:{start}:{end}"
)
records.append(
{
"text": record_text,
"key": key,
"summary": key,
"semantic_key": key,
"memory_type": "context_chunk",
"importance": 0.55,
"confidence": 0.85,
"source": "service_curve_benchmark",
"evidence": [f"token_range:{start}:{end}"],
"token_ids": ids,
"token_mask": torch.ones_like(ids, dtype=torch.bool),
"trusted": True,
"force": True,
}
)
started = time.perf_counter()
model.memory_os_v2.write_batch(records)
return {
"added_records": len(records),
"write_ms": (time.perf_counter() - started) * 1000.0,
}
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--base-model", default=r"H:\Memory")
parser.add_argument(
"--memory-model",
default=r"H:\Memory\V2_dpskw\qwen3_5_4b_natural_memory_v2",
)
parser.add_argument("--output", default=r"H:\Memory\V2_dpskw\service_curves_4b.json")
parser.add_argument("--lengths", default="131072,262144,524288,1048576,4194304")
parser.add_argument("--hot-window", type=int, default=32768)
parser.add_argument("--chunk-tokens", type=int, default=512)
parser.add_argument("--rag-top-k", type=int, default=2)
parser.add_argument(
"--memory-top-k-records",
type=int,
default=2,
help="Natural Memory records injected into hot KV; default matches RAG chunk count",
)
parser.add_argument("--max-new-tokens", type=int, default=32)
parser.add_argument("--gpu-hourly-cost", type=float, default=0.0)
parser.add_argument(
"--gpu-memory-gb",
type=float,
default=8.0,
help="hard CUDA placement cap; 0 disables the cap",
)
parser.add_argument("--no-4bit", action="store_true")
args = parser.parse_args()
_set_cuda_process_cap(args.gpu_memory_gb)
lengths = [int(item.strip()) for item in args.lengths.split(",") if item.strip()]
use_4bit = not args.no_4bit
tokenizer = load_tokenizer(_path(args.base_model))
max_length = max(lengths)
corpus, needle_ids, needle, query = _build_corpus(
tokenizer,
max_length,
chunk_tokens=args.chunk_tokens,
)
report: dict[str, Any] = {
"benchmark": "production_baseline_service_curves_4b",
"model": str(_path(args.base_model)),
"memory_model": str(_path(args.memory_model)),
"quantization": "4bit_nf4" if use_4bit else "none",
"lengths": lengths,
"hot_window": args.hot_window,
"chunk_tokens": args.chunk_tokens,
"rag_top_k": args.rag_top_k,
"memory_top_k_records": args.memory_top_k_records,
"query": query,
"expected": "CURVE-7H2K-9P4M",
"cost_note": "usd_per_million_output_tokens is null unless --gpu-hourly-cost is supplied",
"gpu_hourly_cost": args.gpu_hourly_cost if args.gpu_hourly_cost > 0 else None,
"gpu_memory_cap_gb": args.gpu_memory_gb if args.gpu_memory_gb > 0 else None,
"systems": {},
}
print("loading 4B baseline")
max_memory = _max_memory(args.gpu_memory_gb)
base = load_qwen_base(_path(args.base_model), load_in_4bit=use_4bit, max_memory=max_memory)
base.eval()
base_device = base.get_input_embeddings().weight.device
systems: dict[str, list[dict[str, Any]]] = {
"dense_full_kv": [],
"sliding_window": [],
"matched_hot_window": [],
"chunk_rag": [],
}
for target in lengths:
current = corpus[:target]
row_base = {"context_tokens": target}
if target <= 262144:
try:
prompt = _prompt_content(tokenizer, current, query)
encoded = _chat_from_content(tokenizer, prompt, base_device)
if base_device.type == "cuda":
torch.cuda.reset_peak_memory_stats(base_device)
row = _measure_generation(base, encoded, tokenizer, base_device, dynamic=False, max_new_tokens=args.max_new_tokens)
_quality_fields(row, target_reachable=True, expected="CURVE-7H2K-9P4M")
row_base.update(row)
except Exception as exc:
row_base.update({"status": "error", "error": str(exc)[:500]})
_quality_fields(row_base, target_reachable=True, expected="CURVE-7H2K-9P4M")
else:
row_base.update({"status": "unsupported_by_base_max_position", "quality_pass": False})
_quality_fields(row_base, target_reachable=True, expected="CURVE-7H2K-9P4M")
systems["dense_full_kv"].append(row_base)
window = current[-args.hot_window :]
prompt = _prompt_content(tokenizer, window, query)
encoded = _chat_from_content(tokenizer, prompt, base_device)
if base_device.type == "cuda":
torch.cuda.reset_peak_memory_stats(base_device)
row = _measure_generation(base, encoded, tokenizer, base_device, dynamic=False, max_new_tokens=args.max_new_tokens)
row.update({"context_tokens": target, "visible_tokens": int(window.numel())})
_quality_fields(row, target_reachable=False, expected="CURVE-7H2K-9P4M")
systems["sliding_window"].append(row)
matched_tokens = max(1, int(args.chunk_tokens) * int(args.rag_top_k))
matched_window = current[-matched_tokens:]
prompt = _prompt_content(tokenizer, matched_window, query)
encoded = _chat_from_content(tokenizer, prompt, base_device)
if base_device.type == "cuda":
torch.cuda.reset_peak_memory_stats(base_device)
row = _measure_generation(base, encoded, tokenizer, base_device, dynamic=False, max_new_tokens=args.max_new_tokens)
row.update({"context_tokens": target, "visible_tokens": int(matched_window.numel()), "matched_hot_window_tokens": matched_tokens})
_quality_fields(row, target_reachable=False, expected="CURVE-7H2K-9P4M")
systems["matched_hot_window"].append(row)
retrieved, retrieve_ms, chunk_count = _chunk_retrieve(tokenizer, current, query, chunk_tokens=args.chunk_tokens, top_k=args.rag_top_k)
prompt = _prompt_content(tokenizer, retrieved, query)
encoded = _chat_from_content(tokenizer, prompt, base_device)
if base_device.type == "cuda":
torch.cuda.reset_peak_memory_stats(base_device)
row = _measure_generation(base, encoded, tokenizer, base_device, dynamic=False, max_new_tokens=args.max_new_tokens)
row.update({"context_tokens": target, "retrieved_tokens": int(retrieved.numel()), "chunk_count": chunk_count, "retriever_ms": retrieve_ms})
_quality_fields(row, target_reachable=True, expected="CURVE-7H2K-9P4M")
systems["chunk_rag"].append(row)
print(f"baseline context={target}")
_release(base)
base = None
print("loading Natural Memory 4B")
memory = load_qwen_dynamic(
_path(args.memory_model), load_in_4bit=use_4bit, max_memory=max_memory
)
memory.eval()
memory.memory_config.memory_top_k_records = max(1, int(args.memory_top_k_records))
memory.memory_os_v2.bank.top_k_records = memory.memory_config.memory_top_k_records
memory_device = memory._find_layer_device()
memory.reset_memory(batch_size=1, device=memory_device)
previous = 0
systems["natural_memory"] = []
query_ids, query_mask = _query_tensors(tokenizer, query, memory_device)
for target in lengths:
write_info = _add_memory_chunks(
memory,
tokenizer,
corpus,
previous,
target,
chunk_tokens=args.chunk_tokens,
device=memory_device,
needle_ids=needle_ids,
)
previous = target
encoded = _chat_from_content(tokenizer, query, memory_device)
if memory_device.type == "cuda":
torch.cuda.reset_peak_memory_stats(memory_device)
row = _measure_generation(
memory,
encoded,
tokenizer,
memory_device,
dynamic=True,
query_ids=query_ids,
query_mask=query_mask,
query_text=query,
max_new_tokens=args.max_new_tokens,
)
decision = memory.runtime.v2_last_decisions[-1] if memory.runtime.v2_last_decisions else {}
row.update(
{
"context_tokens": target,
"stored_records": memory.memory_v2_stats().get("records", 0),
"coarse_candidates": decision.get("coarse_candidates", memory.memory_v2_stats().get("last_coarse_candidates", 0)),
"page_ids": decision.get("page_ids", []),
"record_ids": decision.get("record_ids", []),
"stop_reason": decision.get("stop_reason", ""),
"retrieved_records": len(decision.get("record_ids", [])),
"memory_write_ms": write_info["write_ms"],
}
)
_quality_fields(row, target_reachable=True, expected="CURVE-7H2K-9P4M")
systems["natural_memory"].append(row)
print(f"natural_memory context={target}")
_release(memory)
memory = None
if args.gpu_hourly_cost > 0.0:
for rows in systems.values():
for row in rows:
if row.get("status") != "ok":
continue
row["usd_per_million_output_tokens"] = (
1_000_000.0 / max(1e-9, row.get("decode_tok_s", 0.0))
* args.gpu_hourly_cost
/ 3600.0
)
report["systems"] = systems
output = _path(args.output)
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps(report, ensure_ascii=False, indent=2))
print(f"saved={output}")
if __name__ == "__main__":
main()
benchmark_tiered_memory_v2.py
+162
View File
@@ -0,0 +1,162 @@
"""Stress the durable Natural Memory v2 page tier without loading Qwen."""
from __future__ import annotations
import argparse
import json
import random
import sys
import time
from pathlib import Path
from tempfile import TemporaryDirectory
import torch
if __package__ in {None, ""}:
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from V2_dpskw.memory_os_v2 import MemoryRouterV2, PagedMemoryBankV2
from V2_dpskw.tiered_memory_store_v2 import TieredMemoryStoreV2
def run(args: argparse.Namespace) -> dict[str, object]:
random.seed(args.seed)
torch.manual_seed(args.seed)
report_path = Path(args.output)
report_path.parent.mkdir(parents=True, exist_ok=True)
with TemporaryDirectory(prefix="natural-memory-v2-tiered-", dir=str(report_path.parent)) as temp_dir:
store_path = Path(temp_dir) / "memory.sqlite"
router = MemoryRouterV2(args.hidden_size, router_dim=args.key_dim, num_heads=args.heads)
store = TieredMemoryStoreV2(
store_path,
key_dim=args.key_dim,
page_capacity=args.page_capacity,
)
bank = PagedMemoryBankV2(
args.hidden_size,
page_capacity=args.page_capacity,
max_pages=max(1, (args.records + args.page_capacity - 1) // args.page_capacity + 8),
hot_pages=args.hot_pages,
top_k_pages=args.top_k_pages,
top_k_records=args.top_k_records,
router=router,
key_dim=args.key_dim,
coarse_index_bits=args.coarse_index_bits,
tier_store=store,
max_resident_pages=args.resident_pages,
)
target_key = None
target_text = ""
started = time.perf_counter()
for start in range(0, args.records, args.batch_size):
batch: list[dict[str, object]] = []
for index in range(start, min(args.records, start + args.batch_size)):
key = torch.randn(args.hidden_size)
if index == args.target_index:
target_key = key.clone()
target_text = f"tiered-record-{index}"
batch.append(
{
"text": f"tiered-record-{index}",
"key": key,
"summary": key,
"entity": "benchmark",
"attribute": f"attribute-{index}",
"value": f"value-{index}",
"importance": 0.2 if index != args.target_index else 1.0,
"confidence": 0.95,
"source": "tiered-benchmark",
"trusted": True,
}
)
bank.write_batch(batch)
write_seconds = time.perf_counter() - started
before = bank.stats()
store.close()
reopen_started = time.perf_counter()
reopened_store = TieredMemoryStoreV2(
store_path,
key_dim=args.key_dim,
page_capacity=args.page_capacity,
)
reopened = PagedMemoryBankV2(
args.hidden_size,
page_capacity=args.page_capacity,
max_pages=max(1, (args.records + args.page_capacity - 1) // args.page_capacity + 8),
hot_pages=args.hot_pages,
top_k_pages=args.top_k_pages,
top_k_records=args.top_k_records,
router=router,
key_dim=args.key_dim,
coarse_index_bits=args.coarse_index_bits,
tier_store=reopened_store,
max_resident_pages=args.resident_pages,
)
reopen_seconds = time.perf_counter() - reopen_started
if target_key is None:
raise RuntimeError("target index was not generated")
records, decision = reopened.query(
query_key=target_key,
query_text=target_text,
top_k_pages=args.top_k_pages,
top_k_records=args.top_k_records,
)
after = reopened.stats()
found = any(record.text == target_text for record in records)
target_row = reopened_store.find_by_text(target_text, active_status="active")
target_page_id = target_row["page_id"] if target_row is not None else None
candidate_pages = reopened._candidate_page_ids(target_key)
reopened_store.close()
report = {
"format_version": 2,
"records_requested": args.records,
"target_index": args.target_index,
"page_capacity": args.page_capacity,
"coarse_index_bits": args.coarse_index_bits,
"resident_pages": args.resident_pages,
"write_seconds": write_seconds,
"reopen_seconds": reopen_seconds,
"before_close": before,
"after_reopen": after,
"target_recalled_after_restart": found,
"target_page_id": target_page_id,
"target_page_in_coarse_candidates": target_page_id in candidate_pages if target_page_id else False,
"coarse_candidate_count": len(candidate_pages),
"decision": {
"page_ids": decision.page_ids,
"record_ids": decision.record_ids,
"hop_count": decision.hop_count,
"confidence": decision.confidence,
"stop_reason": decision.stop_reason,
},
}
report_path.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
return report
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--records", type=int, default=100_000)
parser.add_argument("--target-index", type=int, default=99_999)
parser.add_argument("--hidden-size", type=int, default=32)
parser.add_argument("--key-dim", type=int, default=16)
parser.add_argument("--heads", type=int, default=4)
parser.add_argument("--page-capacity", type=int, default=32)
parser.add_argument("--resident-pages", type=int, default=64)
parser.add_argument("--hot-pages", type=int, default=8)
parser.add_argument("--top-k-pages", type=int, default=4)
parser.add_argument("--top-k-records", type=int, default=8)
parser.add_argument("--coarse-index-bits", type=int, default=12)
parser.add_argument("--batch-size", type=int, default=2_000)
parser.add_argument("--seed", type=int, default=20260904)
parser.add_argument(
"--output",
default="W:/Flash/model/V2_dpskw/tiered_memory_v2_benchmark.json",
)
return parser.parse_args()
if __name__ == "__main__":
print(json.dumps(run(parse_args()), ensure_ascii=False, indent=2))
build_memory_policy_hardset.py
+298
View File
@@ -0,0 +1,298 @@
"""Build a larger hard-negative dataset for the automatic memory controller.
The normal bootstrap corpus contains mostly short, obvious examples. This
hard set adds realistic negations, questions, replacement requests, and long
noise clauses so the write/forget heads are evaluated on decisions that are
easy to confuse with durable facts.
"""
from __future__ import annotations
import argparse
import json
import random
import string
from pathlib import Path
from typing import Any
PROJECT_ROOT = Path(__file__).resolve().parent
ATTRIBUTES = (
"常用时区",
"最喜欢的水果",
"项目代号",
"备用联系人",
"默认输出风格",
"工作区域",
"提醒时间",
"档案代号",
)
TRAIN_FACT_TEMPLATES = (
"请记住：我的{attribute}是{value}。",
"以后涉及{attribute}时，请使用{value}这个值。",
"个人资料更新——我的{attribute}等于{value}，以后可能会问到。",
"请把这条个人资料保存下来：我的{attribute}为{value}。",
"我的{attribute}是{value}，这是需要长期保留的信息。",
)
EVAL_FACT_TEMPLATES = (
"登记一下，我的{attribute}：{value}。",
"将我的{attribute}记为{value}，后续请按这个资料回答。",
"长期资料里新增一项：{attribute}={value}。",
"请把我的{attribute}保存成{value}。",
)
TRAIN_NOISE_TEMPLATES = (
"这是普通对话噪声：我暂时提到一个无关编号{value}，不需要长期保存。",
"随口一提，编号{value}只是临时信息，请不要记住。",
"请不要把这句话写入长期记忆：今天看到的临时编号是{value}。",
"这只是一次性测试值{value}，不用保存，也不要据此推断个人资料。",
"聊天中的无关内容：{value}；它不是我的个人事实。",
)
EVAL_NOISE_TEMPLATES = (
"临时提到{value}，这不是需要保存的资料。",
"忽略这个一次性编号{value}，不要将它写入记忆。",
"普通闲聊内容：{value}，没有长期价值。",
"不要记住{value}，它只是当前消息里的干扰项。",
)
TRAIN_QUERY_TEMPLATES = (
"我的{attribute}是什么？",
"只根据已经保存的资料，告诉我{attribute}。",
"不要猜测，请读取记忆回答：我的{attribute}为？",
"记忆中是否有我的{attribute}？",
)
EVAL_QUERY_TEMPLATES = (
"跨对话后，我登记的{attribute}是哪一个？",
"请从长期资料中查找我的{attribute}。",
"之前保存的{attribute}内容是什么？",
)
TRAIN_FORGET_TEMPLATES = (
"请删除关于我的{attribute}的记忆，不要再保留{value}。",
"忘掉我的{attribute}，这条资料已经失效。",
"撤销之前保存的{attribute}，以后不要再使用它。",
"清除我的{attribute}记录；{value}不再有效。",
)
EVAL_FORGET_TEMPLATES = (
"请移除长期记忆中的{attribute}，不要继续记住它。",
"我的{attribute}已经作废，请忘记这项资料。",
"撤回关于{attribute}的个人信息，不要再保留。",
)
def _value(rng: random.Random, prefix: str) -> str:
alphabet = string.ascii_uppercase + string.digits
return f"{prefix}-" + "".join(rng.choice(alphabet) for _ in range(8))
def _row(
*,
group: str,
index: int,
text: str,
write: float,
forget: float,
kind: str,
attribute: str,
value: str,
) -> dict[str, Any]:
return {
"id": f"{group}:{index}",
"group_id": group,
"text": text,
"messages": [{"role": "user", "content": text}],
"write_label": write,
"forget_label": forget,
"kind": kind,
"source": "synthetic_memory_policy_hardset",
"subject": "验证用户",
"attribute": attribute,
"value": value,
"answer": "",
"answerable": None,
}
def _build_split(
*,
count: int,
split: str,
seed: int,
fact_templates: tuple[str, ...],
noise_templates: tuple[str, ...],
query_templates: tuple[str, ...],
forget_templates: tuple[str, ...],
) -> list[dict[str, Any]]:
rng = random.Random(seed)
rows: list[dict[str, Any]] = []
for group_index in range(count):
group = f"hard-{split}-{group_index:06d}"
attribute = ATTRIBUTES[group_index % len(ATTRIBUTES)]
fact_value = _value(rng, "FACT")
noise_value = _value(rng, "NOISE")
replacement_value = _value(rng, "NEW")
rows.extend(
(
_row(
group=group,
index=0,
text=rng.choice(fact_templates).format(attribute=attribute, value=fact_value),
write=1.0,
forget=0.0,
kind="fact",
attribute=attribute,
value=fact_value,
),
_row(
group=group,
index=1,
text=rng.choice(noise_templates).format(value=noise_value),
write=0.0,
forget=0.0,
kind="noise",
attribute="",
value=noise_value,
),
_row(
group=group,
index=2,
text=rng.choice(query_templates).format(attribute=attribute),
write=0.0,
forget=0.0,
kind="query",
attribute=attribute,
value="",
),
_row(
group=group,
index=3,
text=rng.choice(forget_templates).format(attribute=attribute, value=fact_value),
write=0.0,
forget=1.0,
kind="forget",
attribute=attribute,
value=fact_value,
),
_row(
group=group,
index=4,
text=f"更正一下：我的{attribute}改为{replacement_value}，旧值不再有效。",
write=1.0,
# A replacement is a write/update, not a delete. The
# runtime retires the matched old version and keeps the
# new fragment active. Only an explicit forget request
# receives forget_label=1.
forget=0.0,
kind="replacement",
attribute=attribute,
value=replacement_value,
),
)
)
# These rows are deliberately close to real conversation and are not
# ordinary keyword negatives. They teach the controller that a
# question, a hypothetical, a quoted third-party claim, and a
# temporary value must not become durable personal memory.
rows.extend(
(
_row(
group=group,
index=5,
text=f"我想知道我的{attribute}是什么？",
write=0.0,
forget=0.0,
kind="question",
attribute=attribute,
value="",
),
_row(
group=group,
index=6,
text=f"如果我的{attribute}改成{replacement_value}，会有什么影响？",
write=0.0,
forget=0.0,
kind="hypothetical",
attribute=attribute,
value=replacement_value,
),
_row(
group=group,
index=7,
text=f"别人说我的{attribute}是{fact_value}，这不是我的个人资料，请不要记录。",
write=0.0,
forget=0.0,
kind="quoted_noise",
attribute=attribute,
value=fact_value,
),
_row(
group=group,
index=8,
text=f"今天临时使用{noise_value}，只在本次对话有效，不要长期保存。",
write=0.0,
forget=0.0,
kind="temporary",
attribute="",
value=noise_value,
),
)
)
rng.shuffle(rows)
return rows
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output-dir", default="data/production_memory_hard")
parser.add_argument("--train-groups", type=int, default=4000)
parser.add_argument("--eval-groups", type=int, default=1000)
parser.add_argument("--seed", type=int, default=20260905)
args = parser.parse_args()
if args.train_groups < 1 or args.eval_groups < 1:
raise SystemExit("group counts must be positive")
output_dir = Path(args.output_dir)
if not output_dir.is_absolute() and not output_dir.exists():
output_dir = PROJECT_ROOT / output_dir
output_dir.mkdir(parents=True, exist_ok=True)
train_rows = _build_split(
count=args.train_groups,
split="train",
seed=args.seed,
fact_templates=TRAIN_FACT_TEMPLATES,
noise_templates=TRAIN_NOISE_TEMPLATES,
query_templates=TRAIN_QUERY_TEMPLATES,
forget_templates=TRAIN_FORGET_TEMPLATES,
)
eval_rows = _build_split(
count=args.eval_groups,
split="eval",
seed=args.seed + 1,
fact_templates=EVAL_FACT_TEMPLATES,
noise_templates=EVAL_NOISE_TEMPLATES,
query_templates=EVAL_QUERY_TEMPLATES,
forget_templates=EVAL_FORGET_TEMPLATES,
)
for name, rows in (("train", train_rows), ("eval", eval_rows)):
with (output_dir / f"{name}.jsonl").open("w", encoding="utf-8") as handle:
for row in rows:
handle.write(json.dumps(row, ensure_ascii=False) + "\n")
manifest = {
"format_version": 1,
"generator": "build_memory_policy_hardset.py",
"seed": args.seed,
"train_groups": args.train_groups,
"eval_groups": args.eval_groups,
"train_examples": len(train_rows),
"eval_examples": len(eval_rows),
"labels": {
"write_positive": sum(row["write_label"] >= 0.5 for row in train_rows),
"forget_positive": sum(row["forget_label"] >= 0.5 for row in train_rows),
},
"warning": "Synthetic hard negatives; combine with redacted real conversations before production deployment.",
}
(output_dir / "manifest.json").write_text(
json.dumps(manifest, ensure_ascii=False, indent=2), encoding="utf-8"
)
print(json.dumps(manifest, ensure_ascii=False, indent=2))
if __name__ == "__main__":
main()
build_natural_memory_v2_package.py
+370
View File
@@ -0,0 +1,370 @@
"""Build an embedded Natural Memory v2 package from the current v1 package.
Unchanged Qwen shards are hard-linked when the filesystem permits it. The
custom memory shard is rewritten once to include the trained V2 router and a
compact V2 page payload, while the official model shards remain untouched.
"""
from __future__ import annotations
import argparse
import json
import os
import shutil
import sys
from pathlib import Path
from typing import Any
import torch
from safetensors import safe_open
from safetensors.torch import save_file
if __package__ in {None, ""}:
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from V2_dpskw.memory_os_v2 import MemoryOSV2, MemoryRouterV2
def _link_or_copy(source: Path, target: Path, *, allow_copy: bool) -> None:
try:
os.link(source, target)
except OSError:
try:
# Some Windows volumes reject hardlinks but allow symlinks. A
# symlink keeps the multi-gigabyte Qwen shards shared by both
# packages and avoids an avoidable disk-space spike.
os.symlink(source, target)
return
except OSError:
pass
if not allow_copy:
raise RuntimeError(
"the destination filesystem does not support hard links; "
"refusing to duplicate multi-gigabyte Qwen shards. "
"Re-run with --allow-copy-base only when enough disk space "
"has been explicitly reserved"
)
shutil.copy2(source, target)
def _pack_v2_payload(payload: dict[str, Any]) -> tuple[dict[str, torch.Tensor], dict[str, Any]]:
tensors: dict[str, torch.Tensor] = {}
metadata = dict(payload)
def pack_record(item: dict[str, Any], prefix: str) -> dict[str, Any]:
item = dict(item)
for field in ("key", "summary", "semantic_key", "token_ids", "token_mask"):
value = item.pop(field, None)
if isinstance(value, torch.Tensor):
name = f"dynamic_memory.v2.{prefix}.{field}"
tensors[name] = value.detach().cpu().contiguous()
item[f"{field}_ref"] = name
return item
metadata["records"] = [
pack_record(item, f"records.{index}")
for index, item in enumerate(payload.get("records", []))
]
metadata["quarantine"] = [
pack_record(item, f"quarantine.{index}")
for index, item in enumerate(payload.get("quarantine", []))
]
metadata["pages"] = []
for index, item in enumerate(payload.get("pages", [])):
item = dict(item)
for field in ("key", "summary"):
value = item.pop(field, None)
if isinstance(value, torch.Tensor):
name = f"dynamic_memory.v2.pages.{index}.{field}"
tensors[name] = value.detach().cpu().contiguous()
item[f"{field}_ref"] = name
metadata["pages"].append(item)
return tensors, metadata
def build(args: argparse.Namespace) -> dict[str, Any]:
base_dir = Path(args.base_package)
output_dir = base_dir if args.in_place else Path(args.output_dir)
if not args.in_place and output_dir.exists():
raise FileExistsError(f"refusing to overwrite existing output: {output_dir}")
if not args.in_place:
output_dir.mkdir(parents=True)
base_manifest = json.loads((base_dir / "memory_merge.json").read_text(encoding="utf-8"))
base_config = json.loads((base_dir / "memory_config.json").read_text(encoding="utf-8"))
memory_name = str(base_manifest.get("memory_weights", "model.safetensors-00003-of-00003.safetensors"))
source_memory = base_dir / memory_name
if not source_memory.exists():
raise FileNotFoundError(source_memory)
output_memory_name = (
str(args.in_place_memory_name)
if args.in_place
else memory_name
)
sources = [] if args.in_place else [
source
for source in base_dir.iterdir()
if source.is_file()
and source.name not in {
memory_name,
"memory_config.json",
"memory_merge.json",
"model.safetensors.index.json",
}
]
if not args.allow_copy_base:
probe_source = next(
(source for source in sources if source.name.endswith(".safetensors")),
None,
)
if probe_source is not None:
probe_target = output_dir / ".hardlink-probe"
try:
_link_or_copy(probe_source, probe_target, allow_copy=False)
finally:
if probe_target.exists() or probe_target.is_symlink():
probe_target.unlink()
for source in sources:
_link_or_copy(source, output_dir / source.name, allow_copy=args.allow_copy_base)
source_metadata: dict[str, str] = {}
with safe_open(str(source_memory), framework="pt", device="cpu") as handle:
base_tensors = {key: handle.get_tensor(key) for key in handle.keys()}
source_metadata = dict(handle.metadata() or {})
controller_config: dict[str, Any] = {}
if args.controller_adapter:
controller_dir = Path(args.controller_adapter)
controller_config = json.loads(
(controller_dir / "memory_config.json").read_text(encoding="utf-8")
)
memory_path = controller_dir / "memory.pt"
if memory_path.exists():
memory_state = torch.load(memory_path, map_location="cpu", weights_only=True)
base_tensors.update(
{
f"dynamic_memory.memory.{key}": value.detach().cpu().contiguous()
for key, value in memory_state.items()
}
)
for filename, prefix in (
("text_retriever.pt", "dynamic_memory.text_retriever."),
("memory_policy.pt", "dynamic_memory.memory_policy."),
):
state_path = controller_dir / filename
if not state_path.exists():
continue
state = torch.load(state_path, map_location="cpu", weights_only=True)
base_tensors.update(
{
f"{prefix}{key}": value.detach().cpu().contiguous()
for key, value in state.items()
}
)
surgery_path = controller_dir / "surgery.pt"
if surgery_path.exists():
surgery = torch.load(surgery_path, map_location="cpu", weights_only=True)
for layer, value in surgery.get("blend_logits", {}).items():
base_tensors[f"dynamic_memory.blend_logits.{layer}"] = (
value.detach().cpu().contiguous()
)
router = MemoryRouterV2(
int(base_config["hidden_size"]),
router_dim=args.router_dim,
num_heads=args.num_heads,
max_hops=args.max_hops,
)
router_state = torch.load(args.router_checkpoint, map_location="cpu", weights_only=True)
router.load_state_dict(router_state, strict=True)
for key, value in router.state_dict().items():
base_tensors[f"dynamic_memory.memory_router_v2.{key}"] = value.detach().cpu().contiguous()
memory_os = MemoryOSV2(
int(base_config["hidden_size"]),
router=router,
)
# Migrate the existing model-owned hot facts into V2 address pages. The
# old bank remains intact; this is only a compatibility seed for the new
# hierarchical route.
legacy_prefix = "dynamic_memory.persistent."
token_ids = base_tensors.get(f"{legacy_prefix}text_token_ids")
token_mask = base_tensors.get(f"{legacy_prefix}text_token_mask")
slot_valid = base_tensors.get(f"{legacy_prefix}text_slot_valid")
slot_keys = base_tensors.get(f"{legacy_prefix}text_slot_keys")
if all(isinstance(value, torch.Tensor) for value in (token_ids, token_mask, slot_valid, slot_keys)):
for batch_index in range(slot_valid.shape[0]):
for slot in range(slot_valid.shape[1]):
if not bool(slot_valid[batch_index, slot].item()):
continue
ids = token_ids[batch_index, slot][token_mask[batch_index, slot]]
memory_os.write(
text=f"legacy_hot_slot:{batch_index}:{slot}",
key=slot_keys[batch_index, slot],
summary=slot_keys[batch_index, slot],
memory_type="legacy_hot_text",
importance=0.95,
confidence=0.95,
source="v1_migration",
slot_index=slot,
token_ids=ids,
token_mask=torch.ones_like(ids, dtype=torch.bool),
trusted=True,
)
if args.in_place:
# An in-place controller upgrade must not erase an existing user's V2
# records. The current memory shard and its metadata are already the
# authoritative snapshot; only the controller/router tensors change.
v2_metadata_text = source_metadata.get("memory_os_v2_payload")
else:
v2_tensors, v2_metadata = _pack_v2_payload(memory_os.export_payload())
base_tensors.update(v2_tensors)
v2_metadata_text = json.dumps(v2_metadata, ensure_ascii=False, separators=(",", ":"))
save_file(
base_tensors,
str(output_dir / output_memory_name),
metadata=(
{
"format": "qwen_dynamic_memory_embedded_v2",
**({"memory_os_v2_payload": v2_metadata_text} if v2_metadata_text else {}),
}
),
)
memory_config = dict(base_config)
saved = dict(memory_config.get("memory_config", {}))
saved.update(dict(controller_config.get("memory_config", {})))
saved.update(
{
"memory_version": 2,
"hierarchical_memory": True,
"memory_router_dim": args.router_dim,
"memory_router_heads": args.num_heads,
"memory_page_capacity": args.page_capacity,
"memory_max_pages": args.max_pages,
"memory_hot_pages": args.hot_pages,
"memory_top_k_pages": args.top_k_pages,
"memory_top_k_records": args.top_k_records,
"memory_max_hops": args.max_hops,
"memory_coarse_index_bits": args.coarse_index_bits,
"memory_v2_read_threshold": args.read_threshold,
"memory_v2_write_threshold": args.write_threshold,
"memory_storage_mode": args.memory_storage_mode,
"memory_storage_path": args.memory_storage_path,
"memory_resident_pages": args.memory_resident_pages,
"memory_gpu_cache_records": args.memory_gpu_cache_records,
"memory_gpu_cache_tokens": args.memory_gpu_cache_tokens,
"memory_gpu_cache_reserve_mb": args.memory_gpu_cache_reserve_mb,
"memory_gpu_cache_adaptive": args.memory_gpu_cache_adaptive,
"kv_budget_tokens": args.kv_budget,
"kv_hard_max_tokens": args.kv_hard_max,
"kv_compaction_trigger": args.kv_trigger,
"kv_keep_recent_tokens": args.kv_keep_recent,
"persistent_memory": True,
}
)
saved.setdefault("text_memory_semantic_update_threshold", 0.95)
saved.setdefault("memory_min_read_margin", 0.0)
saved.setdefault("memory_require_evidence", False)
memory_config["memory_config"] = saved
memory_config["router_v2_ready"] = True
memory_config["checkpoint_contains_user_memory"] = True
(output_dir / "memory_config.json").write_text(
json.dumps(memory_config, indent=2, ensure_ascii=False), encoding="utf-8"
)
index = json.loads((base_dir / "model.safetensors.index.json").read_text(encoding="utf-8"))
weight_map = index.setdefault("weight_map", {})
for key in base_tensors:
weight_map[key] = output_memory_name
index.setdefault("metadata", {})["total_size"] = int(
sum(value.numel() * value.element_size() for value in base_tensors.values())
)
(output_dir / "model.safetensors.index.json").write_text(
json.dumps(index, indent=2, ensure_ascii=False) + "\n", encoding="utf-8"
)
manifest = dict(base_manifest)
manifest.update(
{
"format_version": 2,
"format": "qwen_dynamic_memory_v2_embedded",
"base_model": str(base_dir),
"source_router_checkpoint": str(args.router_checkpoint),
"source_controller_adapter": str(args.controller_adapter) if args.controller_adapter else None,
"checkpoint_contains_user_memory": True,
"memory_weights": output_memory_name,
"memory_config": saved,
}
)
(output_dir / "memory_merge.json").write_text(
json.dumps(manifest, indent=2, ensure_ascii=False), encoding="utf-8"
)
return {
"output_dir": str(output_dir),
"memory_shard": str(output_dir / output_memory_name),
"memory_tensor_count": len(base_tensors),
"v2_records": memory_os.stats()["records"],
"v2_pages": memory_os.stats()["pages"],
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--base-package", default="W:/Flash/model/V2_dpskw/qwen3_5_4b_memory_merged_v13")
parser.add_argument("--output-dir", default="W:/Flash/model/V2_dpskw/qwen3_5_4b_memory_merged_v2")
parser.add_argument("--router-checkpoint", default="W:/Flash/model/V2_dpskw/checkpoints/natural_memory_v2_router/memory_router_v2.pt")
parser.add_argument(
"--controller-adapter",
default=None,
help="optional trained policy/retriever adapter to embed into the output package",
)
parser.add_argument(
"--in-place",
action="store_true",
help="upgrade the selected package without duplicating its large Qwen shards",
)
parser.add_argument(
"--in-place-memory-name",
default="model.safetensors-00003-of-00003.safetensors",
help="target memory shard name for an in-place upgrade",
)
parser.add_argument("--router-dim", type=int, default=128)
parser.add_argument("--num-heads", type=int, default=8)
parser.add_argument("--max-hops", type=int, default=3)
parser.add_argument("--page-capacity", type=int, default=32)
parser.add_argument("--max-pages", type=int, default=32768)
parser.add_argument("--hot-pages", type=int, default=8)
parser.add_argument("--top-k-pages", type=int, default=4)
parser.add_argument("--top-k-records", type=int, default=8)
parser.add_argument("--coarse-index-bits", type=int, default=20)
parser.add_argument("--read-threshold", type=float, default=0.65)
parser.add_argument("--write-threshold", type=float, default=0.50)
parser.add_argument("--memory-storage-mode", choices=("embedded", "tiered"), default="embedded")
parser.add_argument("--memory-storage-path", default=None)
parser.add_argument("--memory-resident-pages", type=int, default=256)
parser.add_argument("--memory-gpu-cache-records", type=int, default=256)
parser.add_argument("--memory-gpu-cache-tokens", type=int, default=131072)
parser.add_argument("--memory-gpu-cache-reserve-mb", type=int, default=2048)
parser.add_argument(
"--memory-gpu-cache-adaptive",
action=argparse.BooleanOptionalAction,
default=True,
help="keep the VRAM cache below the reserve line and fall back to system RAM when needed",
)
parser.add_argument("--kv-budget", type=int, default=32768)
parser.add_argument("--kv-hard-max", type=int, default=131072)
parser.add_argument("--kv-trigger", type=float, default=0.90)
parser.add_argument("--kv-keep-recent", type=int, default=8192)
parser.add_argument(
"--allow-copy-base",
action="store_true",
help="allow copying the frozen Qwen shards when hard links are unavailable; requires substantial free disk space",
)
return parser.parse_args()
if __name__ == "__main__":
print(json.dumps(build(parse_args()), ensure_ascii=False, indent=2))
build_nm2_1_package.py
+178
View File
@@ -0,0 +1,178 @@
"""Build the NM2.1 package: the delivered router merged into the Qwen3.5 memory model.
"Merge" here means producing a **self-contained package** rather than a sidecar file: the
base Qwen3.5-4B shards are copied unchanged and only the 16
``dynamic_memory.memory_router_v2.*`` tensors inside the memory shard are replaced with the
delivered router. Everything else -- text retriever, persistent slots, memory policy,
configs, tokenizer -- is left byte-identical, so anything the package did before it still
does, with a different router.
Verification is not "the script exited zero": the script reloads the written shard and
compares all 16 tensors bit-exactly against the source checkpoint, and separately reports
every other tensor as unchanged (same bytes as the source shard).
Usage::
python -m V2_dpskw.build_nm2_1_package ^
--source-package H:\\Memory\\dynamic_memory_lab\\qwen3_5_4b_natural_memory_v2 ^
--output-package H:\\Memory\\dynamic_memory_lab\\qwen3_5_4b_natural_memory_v2_1 ^
--router-checkpoint checkpoints/router_replay_v7_v2_128/memory_router_v2.pt ^
--label NM2.1
"""
from __future__ import annotations
import argparse
import hashlib
import json
import shutil
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path
ROUTER_PREFIX = "dynamic_memory.memory_router_v2."
INDEX_NAME = "model.safetensors.index.json"
MERGE_NAME = "memory_merge.json"
def sha256_of(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for block in iter(lambda: handle.read(1 << 22), b""):
digest.update(block)
return digest.hexdigest()
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--source-package", required=True)
parser.add_argument("--output-package", required=True)
parser.add_argument("--router-checkpoint", required=True)
parser.add_argument("--label", default="NM2.1")
parser.add_argument("--report", default="nm2_1_build_report.json")
args = parser.parse_args()
import torch
from safetensors import safe_open
from safetensors.torch import save_file
source = Path(args.source_package)
output = Path(args.output_package)
router_path = Path(args.router_checkpoint)
if not source.exists():
raise SystemExit(f"source package not found: {source}")
if not router_path.exists():
raise SystemExit(f"router checkpoint not found: {router_path}")
if output.exists() and any(output.iterdir()):
raise SystemExit(f"output package already exists and is not empty: {output}")
router = torch.load(router_path, map_location="cpu", weights_only=True)
if isinstance(router, dict) and "router_state_dict" in router:
router = router["router_state_dict"]
# --- which shard holds the router tensors? -------------------------------------
index = json.loads((source / INDEX_NAME).read_text(encoding="utf-8"))
weight_map = index["weight_map"]
router_keys = sorted(key for key in weight_map if key.startswith(ROUTER_PREFIX))
shards = sorted({weight_map[key] for key in router_keys})
if len(shards) != 1:
raise SystemExit(f"router tensors span several shards: {shards}")
print(json.dumps({"phase": "locate", "router_tensors": len(router_keys),
"shard": shards[0],
"expected_from_checkpoint": len(router),
"router_sha256": sha256_of(router_path)}), flush=True)
missing = [key for key in router if ROUTER_PREFIX + key not in weight_map]
if missing:
raise SystemExit(f"checkpoint keys not present in the package: {missing}")
if len(router_keys) != len(router):
raise SystemExit(f"package has {len(router_keys)} router tensors, checkpoint has {len(router)}")
# --- copy the package -----------------------------------------------------------
print(json.dumps({"phase": "copy", "from": str(source), "to": str(output)}), flush=True)
output.parent.mkdir(parents=True, exist_ok=True)
result = subprocess.run(["robocopy", str(source), str(output), "/E", "/NFL", "/NDL",
"/NJH", "/NJS", "/NP", "/R:2", "/W:2"],
capture_output=True, text=True)
if result.returncode > 7:
raise SystemExit(f"robocopy failed ({result.returncode}): {result.stdout[-800:]}")
copied = sorted(p.name for p in output.iterdir())
expected = sorted(p.name for p in source.iterdir())
if copied != expected:
raise SystemExit(f"copy mismatch: missing={set(expected) - set(copied)} extra={set(copied) - set(expected)}")
# --- rewrite the memory shard ---------------------------------------------------
shard_path = output / shards[0]
source_shard_path = source / shards[0]
with safe_open(str(source_shard_path), framework="pt") as handle:
metadata = handle.metadata() or {}
tensors = {key: handle.get_tensor(key) for key in handle.keys()}
untouched = 0
replaced = []
for key, value in router.items():
full = ROUTER_PREFIX + key
original = tensors[full]
if tuple(original.shape) != tuple(value.shape):
raise SystemExit(f"shape mismatch for {full}: {tuple(original.shape)} vs {tuple(value.shape)}")
tensors[full] = value.to(dtype=original.dtype).contiguous()
replaced.append(full)
for key in tensors:
if key not in replaced:
untouched += 1
save_file(tensors, str(shard_path), metadata=metadata)
print(json.dumps({"phase": "shard_rewritten", "shard": shards[0],
"replaced": len(replaced), "untouched": untouched}), flush=True)
# --- provenance in the merge manifest -------------------------------------------
merge_path = output / MERGE_NAME
merge = json.loads(merge_path.read_text(encoding="utf-8"))
merge["package_label"] = args.label
merge["router_swap"] = {
"built_at": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
"built_from_package": str(source),
"router_checkpoint": str(router_path),
"router_sha256": sha256_of(router_path),
"replaced_tensors": len(replaced),
"shard": shards[0],
}
merge_path.write_text(json.dumps(merge, ensure_ascii=False, indent=2), encoding="utf-8")
# --- verify: reload what was written, compare bit-exactly ------------------------
with safe_open(str(shard_path), framework="pt") as handle:
written = {key: handle.get_tensor(key) for key in handle.keys()}
mismatches = []
for key, value in router.items():
full = ROUTER_PREFIX + key
if not torch.equal(written[full], value.to(dtype=written[full].dtype)):
mismatches.append(full)
with safe_open(str(source_shard_path), framework="pt") as handle:
originals = {key: handle.get_tensor(key) for key in handle.keys()}
changed_others = [key for key in originals
if key not in replaced and not torch.equal(originals[key], written.get(key))]
report = {
"package_label": args.label,
"output_package": str(output),
"source_package": str(source),
"router_checkpoint": str(router_path),
"router_sha256": sha256_of(router_path),
"shard": shards[0],
"files_copied": len(copied),
"router_tensors_replaced": len(replaced),
"tensors_left_untouched": untouched,
"verification": {
"router_tensors_bit_exact": not mismatches,
"router_mismatches": mismatches,
"non_router_tensors_unchanged": not changed_others,
"changed_non_router": changed_others,
},
}
report["passed"] = (not mismatches) and (not changed_others)
Path(args.report).write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps(report, ensure_ascii=False, indent=2), flush=True)
return 0 if report["passed"] else 1
if __name__ == "__main__":
raise SystemExit(main())
build_nm2_vs_nm2_1_report.py
+210
View File
@@ -0,0 +1,210 @@
"""Multi-dimensional comparison: original NM2 vs NM2.1, assembled from stored evidence.
Every number is read from a scorecard/JSON that a run actually wrote -- nothing is copied
from scrollback -- and each table names the file it came from. The two systems are
compared on six dimensions:
1. router retrieval & ranking (frozen v6 eval, 21,920 episodes)
2. policy / abstention axes (same eval, incl. the whole threshold sweep)
3. end-to-end memory ability (battery A/B/C/D, same runtime, only the package differs)
4. unseen-phrasing generalisation (zero-lexical-overlap eval)
5. cost (parameters, address bytes, latency, throughput, package size)
6. engineering robustness (write-path survival, restart persistence, drop-in, tests)
Usage::
python -m V2_dpskw.build_nm2_vs_nm2_1_report
"""
from __future__ import annotations
import argparse
import json
from pathlib import Path
ROOT = Path(__file__).resolve().parent
ORIGINAL_ROUTER = "V2-128 deployed(v3)"
NEW_ROUTER = "REPLAY-128 v7 final"
ORIGINAL_PKG = "原版NM2"
NEW_PKG = "NM2.1最终"
def load(path: Path) -> dict:
try:
return json.loads(path.read_text(encoding="utf-8"))
except Exception:
return {}
def dig(data, *path, default=None):
current = data
for step in path:
if not isinstance(current, dict) or step not in current:
return default
current = current[step]
return current
def row(label: str, a, b, unit: str = "pct") -> str:
def render(value):
if value is None:
return "-"
if unit == "pct":
# Scorecards store rates as fractions in [0, 1]; the report is in percent.
return f"{100.0 * float(value):.2f}%"
if unit == "already_pct":
# The battery comparison file already stores percentages.
return f"{float(value):.2f}%"
if unit == "int":
return f"{int(value):,}"
if unit == "num":
return f"{float(value):.4f}"
if unit == "bool":
return "通过" if value else "**未通过**"
return str(value)
return f"| {label} | {render(a)} | {render(b)} |"
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output", default="NM2_VS_NM2_1.md")
args = parser.parse_args()
scorecard = load(ROOT / "router_scorecard_final.json")
verdict = load(ROOT / "router_verdict_final.json")
sweep = load(ROOT / "threshold_sweep_check.json")
battery = load(ROOT / "nm2_battery_comparison_final.json")
zero = load(ROOT / "replay_check_zov.json")
latency = load(ROOT / "router_latency_bench_prod.json")
build = load(ROOT / "nm2_1_build_report.json")
o, n = scorecard.get(ORIGINAL_ROUTER, {}), scorecard.get(NEW_ROUTER, {})
bo, bn = battery.get(ORIGINAL_PKG, {}), battery.get(NEW_PKG, {})
L: list[str] = []
L += ["# NM2（原版）vs NM2.1：多维度对照", "",
"所有数字均从磁盘上的评分卡/JSON 读取，不是手工抄录。每张表都标注了来源文件。",
"除第 3 节（同一份运行时、只有模型包不同）外，其余各节比较的是**路由器工件**在同一冻结评测集上的表现。", ""]
# ---- 1. retrieval & ranking -----------------------------------------------------
L += ["## 1. 检索与排序（冻结 v6 评测集 21,920 条 / 10 类别）", "",
"来源：`router_scorecard_final.json`（判定见 `router_verdict_final.json`）", "",
f"| 指标 | NM2（原版 `{ORIGINAL_ROUTER}`） | NM2.1（`{NEW_ROUTER}`） |", "|---|---:|---:|"]
for key, label in [("top1", "Top-1 正确率"), ("recall1", "Recall@1"), ("recall3", "Recall@3"),
("recall5", "Recall@5"), ("mrr", "MRR"), ("ndcg3", "nDCG@3"),
("all_evidence_in_top3", "多跳证据全中(Top-3)"),
("all_evidence_in_top3_multi", "多跳证据全中(仅多正例)"),
("hop_accuracy", "hop 正确率"), ("hop_under_prediction", "hop 欠预测率")]:
L.append(row(label, dig(o, "metrics", key), dig(n, "metrics", key)))
L.append("")
# ---- 2. policy axes + sweep -----------------------------------------------------
L += ["## 2. 拒答/仲裁策略轴", "",
"来源：`router_scorecard_final.json`（门槛 0.50）+ `threshold_sweep_check.json`（0.30–0.80 全门槛）", "",
f"| 指标 | NM2（原版） | NM2.1 |", "|---|---:|---:|"]
for key, label in [("need_f1", "need F1"), ("need_recall", "need 召回"),
("need_precision", "need 精确率"),
("specificity_unknown_refusal", "未知拒答率"),
("known_question_refusal_rate", "已知问题被误拒率"),
("unknown_question_read_rate", "未知问题被误读率"),
("abstention_accuracy", "仲裁准确率")]:
L.append(row(label, dig(o, "metrics", "thr0.50", key), dig(n, "metrics", "thr0.50", key)))
L.append("")
sweep_ok = dig(sweep, "candidates", NEW_ROUTER, "sweep_dominates")
L.append(f"全门槛（0.30/0.40/0.50/0.60/0.70/0.80）7 轴复核：NM2.1 **{'全门槛通过' if sweep_ok else '未通过'}**；"
"原版在每个门槛上未知拒答率均为 **0.00%**，且门槛越高误拒越差（0.00%→1.32%）。")
L.append("")
# ---- 3. end-to-end battery ------------------------------------------------------
L += ["## 3. 端到端整体记忆能力（同一份运行时，只有模型包不同）", "",
"来源：`nm2_battery_comparison_final.json`（A 110 用例 / B 16 / C 48 / D 重启持久化）", "",
f"| 指标 | NM2（原版包） | NM2.1（最终包） |", "|---|---:|---:|"]
for key, label, unit in [
("A_cases", "A 用例数", "int"),
("A_overall", "A 总体正确率", "already_pct"),
("A_answerable", "A 可回答正确率", "already_pct"),
("A_unknown_refusal", "A 未知拒答率", "already_pct"),
("A_known_false_refusal", "A 已知问题被误拒率", "already_pct"),
("B_accuracy", "B 零字面重叠改写正确率", "already_pct"),
("B_wrong_attribute", "B 答成别的属性", "already_pct"),
("B_read", "B 触发读取", "already_pct"),
("C_answerable", "C 可回答正确率（24 同形候选）", "already_pct"),
("C_wrong_attribute", "C 答成别的属性", "already_pct"),
("C_unknown_leak", "C 未知泄漏率（越低越好）", "already_pct"),
("D_recalled", "D 重启后召回", "bool"),
("D_answer_correct", "D 重启后作答正确", "bool"),
("D_cleanup", "D 清理生效", "bool"),
]:
L.append(row(label, bo.get(key), bn.get(key), unit))
L.append("")
# ---- 4. unseen-phrasing ----------------------------------------------------------
zr = zero.get("REPLAY-128 final", {})
zo = zero.get("V2-128 v6 final", {})
L += ["## 4. 未见改写问法的泛化（零字面重叠，24 同形候选，随机 4.17%）", "",
"来源：`replay_check_zov.json`（路由器级）与第 3 节 B/C 段（端到端）。"
"路由器级用的是 v6 最终版权重作对照（原版部署权重在同一集合上 Top-1 只有 11.60%）。", "",
f"| 指标 | NM2（v6 最终版权重） | NM2.1（本交付权重） |", "|---|---:|---:|",
row("路由器 Top-1（250 条可回答）", dig(zo, "metrics", "top1"), dig(zr, "metrics", "top1")),
row("路由器 Recall@3", dig(zo, "metrics", "recall3"), dig(zr, "metrics", "recall3")),
row("路由器 MRR", dig(zo, "metrics", "mrr"), dig(zr, "metrics", "mrr")),
row("端到端改写正确率（B 段）", bo.get("B_accuracy"), bn.get("B_accuracy"), "already_pct"),
row("端到端未知泄漏（C 段）", bo.get("C_unknown_leak"), bn.get("C_unknown_leak"), "already_pct"), ""]
# ---- 5. cost ---------------------------------------------------------------------
L += ["## 5. 成本（参数 / 存储 / 速度）", "",
"来源：`router_scorecard_final.json`、`router_latency_bench_prod.json`（7 轮交错中位数）", "",
f"| 指标 | NM2（原版） | NM2.1 |", "|---|---:|---:|",
row("参数量", dig(o, "router", "parameters"), dig(n, "router", "parameters"), "int"),
row("每条记录地址字节", dig(o, "storage", "address_bytes_per_record"),
dig(n, "storage", "address_bytes_per_record"), "int"),
row("单查询延迟中位数 ms (GPU)", dig(o, "latency", "cuda", "single_query_latency_ms_p50"),
dig(n, "latency", "cuda", "single_query_latency_ms_p50"), "num"),
row("批量 QPS (batch=64)", dig(o, "latency", "cuda", "batched_qps_64"),
dig(n, "latency", "cuda", "batched_qps_64"), "int"),
row("批量 QPS (batch=256)", dig(o, "latency", "cuda", "batched_qps_256"),
dig(n, "latency", "cuda", "batched_qps_256"), "int"), ""]
if latency:
# Interleaved round-robin medians: the measurement built to remove ordering effects.
deployed = dig(latency, "deployed-raw", "single_ms_median")
replay = dig(latency, "REPLAY-raw", "single_ms_median")
spread = dig(latency, "deployed-raw", "single_ms_spread_pct")
if deployed and replay:
L.append(f"交错基准（7 轮，消除顺序效应）单查询中位数："
f"原版 {deployed:.4f} ms → NM2.1 {replay:.4f} ms"
f"（差 {100 * (replay / deployed - 1):+.2f}%；该轮原版自身离散度 {spread:.2f}%）。")
L.append("")
L.append(f"模型包大小：原版 8.88 GB（22 文件）→ NM2.1 8.88 GB（23 文件，多出属性头 ~0.25 MB）；"
f"合并验证：替换 {dig(build, 'router_tensors_replaced')} 个张量、"
f"其余 {dig(build, 'tensors_left_untouched')} 个逐字节未变、路由张量与交付件逐位一致。")
L.append("")
# ---- 6. robustness ---------------------------------------------------------------
L += ["## 6. 工程鲁棒性", "",
"| 项目 | NM2（原版） | NM2.1 |", "|---|---|---|",
"| 一次写 20 条不同属性事实后存活 | **12 / 20**（8 条查询前被误删） | **20 / 20** |",
"| 端到端（写入修复前后，16 用例） | 37.50% | **68.75%** |",
"| 替换兼容性 | 基线 | **DROP-IN OK**（16/16 键、驱动 `PagedMemoryBankV2`） |",
"| 单元测试 | — | **52 项通过** |",
"| 未知问题泄漏（同形候选） | **75.00%** | **0.00%** |",
""]
# ---- known gaps ------------------------------------------------------------------
L += ["## 7. 仍未解决的短板（不粉饰）", "",
"| 短板 | 现状 | 说明 |", "|---|---|---|",
"| 跨域未知拒答 | 未解决 | 覆盖头是 24 类闭集，仅当其词表被库填充 ≥90% 时生效；开放词表的属性匹配实测仅 **49.20%** Top-1 / AUC 0.6560 |",
"| 答成别的属性 | **27.50%**（原 35.00%） | 已排除先验重加权（更差）与单纯替换打分器（更差） |",
"| A 段未知拒答率 | 56.67% | 未改善 |",
"| 规模验证 | 未做 | 仅 24 属性 / 300 条改写评测；生产需上千属性、上万条 |",
"| 通用能力回归 | 未跑 | `eval_general_capability.py` 依赖的 `comprehensive_general.jsonl` 不存在 |", ""]
text = "\n".join(L) + "\n"
(ROOT / args.output).write_text(text, encoding="utf-8")
print(text)
return 0
if __name__ == "__main__":
raise SystemExit(main())
build_production_memory_dataset.py
+307
View File
@@ -0,0 +1,307 @@
"""Normalize conversation logs into a leak-resistant memory-policy dataset.
The runtime accepts many local data shapes because real users rarely keep
their chat exports in one format. This command converts them to a small,
auditable JSONL schema without inventing labels. It understands the current
``native_memory`` episode format, the streaming demo format, and a generic
format documented in the output manifest.
The bundled fallback files are bootstrap data for smoke tests. A real user
corpus can be supplied with ``--source``/``--eval-source`` and receives the
same normalization and group-level split guarantees.
"""
from __future__ import annotations
import argparse
import hashlib
import json
from collections import Counter, defaultdict
from pathlib import Path
from typing import Any, Iterable
DEFAULT_SOURCES = (
"data/native_memory/train.jsonl",
"data/native_memory/eval.jsonl",
"data/demo_stream.jsonl",
)
PROJECT_ROOT = Path(__file__).resolve().parent
def _project_path(value: str | Path) -> Path:
path = Path(value)
if path.is_absolute() or path.exists():
return path
return PROJECT_ROOT / path
def _read_jsonl(path: Path) -> Iterable[tuple[int, dict[str, Any]]]:
with path.open("r", encoding="utf-8") as handle:
for line_number, raw in enumerate(handle, 1):
raw = raw.strip()
if not raw:
continue
value = json.loads(raw)
if not isinstance(value, dict):
raise ValueError(f"{path}:{line_number} must contain a JSON object")
yield line_number, value
def _message_text(messages: Any) -> str:
if isinstance(messages, str):
return messages.strip()
if not isinstance(messages, list):
return ""
parts: list[str] = []
for message in messages:
if not isinstance(message, dict):
continue
content = message.get("content", "")
if isinstance(content, str) and content.strip():
role = str(message.get("role", "user"))
parts.append(f"[{role}] {content.strip()}")
return "\n".join(parts).strip()
def _user_text(messages: Any) -> str:
if isinstance(messages, str):
return messages.strip()
if not isinstance(messages, list):
return ""
for message in reversed(messages):
if isinstance(message, dict) and message.get("role") == "user":
content = message.get("content", "")
if isinstance(content, str):
return content.strip()
return _message_text(messages)
def _explicit_split(path: Path, *, forced: str | None) -> str | None:
if forced in {"train", "eval"}:
return forced
name = path.name.lower()
if any(mark in name for mark in ("eval", "valid", "test")):
return "eval"
if "train" in name:
return "train"
return None
def _make_example(
*,
group_id: str,
example_id: str,
text: str,
write_label: float,
forget_label: float = 0.0,
kind: str = "conversation",
source: str,
subject: str = "",
attribute: str = "",
value: Any = None,
answer: str = "",
answerable: bool | None = None,
messages: Any = None,
) -> dict[str, Any] | None:
text = str(text or "").strip()
if not text:
return None
return {
"id": example_id,
"group_id": group_id,
"text": text,
"messages": messages if isinstance(messages, list) else [{"role": "user", "content": text}],
"write_label": float(max(0.0, min(1.0, write_label))),
"forget_label": float(max(0.0, min(1.0, forget_label))),
"kind": kind,
"source": source,
"subject": str(subject or ""),
"attribute": str(attribute or ""),
"value": "" if value is None else str(value),
"answer": str(answer or ""),
"answerable": answerable,
}
def normalize_record(record: dict[str, Any], *, source: str, line_number: int) -> list[dict[str, Any]]:
"""Convert one source record into labeled write/query decisions."""
raw_id = str(record.get("id") or record.get("conversation_id") or f"line-{line_number}")
group_id = f"{source}:{raw_id}"
output: list[dict[str, Any]] = []
chunks = record.get("memory_chunks")
if isinstance(chunks, list):
for index, chunk in enumerate(chunks):
if not isinstance(chunk, dict):
continue
messages = chunk.get("messages", [])
item = _make_example(
group_id=group_id,
example_id=f"{raw_id}:memory:{index}",
text=_user_text(messages) or str(chunk.get("text", "")),
write_label=float(chunk.get("write_label", 1.0)),
forget_label=float(chunk.get("forget_label", 0.0)),
kind=str(chunk.get("kind", "fact")),
source=source,
subject=record.get("subject", ""),
attribute=record.get("attribute", ""),
value=chunk.get("value", record.get("value", "")),
messages=messages,
)
if item is not None:
output.append(item)
query = record.get("query")
query_text = _user_text(query)
item = _make_example(
group_id=group_id,
example_id=f"{raw_id}:query",
text=query_text,
write_label=0.0,
kind="query",
source=source,
subject=record.get("subject", ""),
attribute=record.get("attribute", ""),
answer=record.get("answer", ""),
answerable=record.get("answerable"),
messages=query if isinstance(query, list) else None,
)
if item is not None:
output.append(item)
return output
memory = record.get("memory")
if isinstance(memory, list):
for index, item_messages in enumerate(memory):
item = _make_example(
group_id=group_id,
example_id=f"{raw_id}:memory:{index}",
text=_user_text(item_messages),
write_label=1.0,
kind="fact",
source=source,
messages=item_messages if isinstance(item_messages, list) else None,
)
if item is not None:
output.append(item)
query = record.get("query")
if query is not None:
item = _make_example(
group_id=group_id,
example_id=f"{raw_id}:query",
text=_user_text(query),
write_label=0.0,
kind="query",
source=source,
answer=record.get("answer", ""),
answerable=record.get("answerable"),
messages=query if isinstance(query, list) else None,
)
if item is not None:
output.append(item)
event = record.get("memory_event")
if not output and (record.get("text") is not None or record.get("messages") is not None):
event = event if isinstance(event, dict) else {}
item = _make_example(
group_id=group_id,
example_id=f"{raw_id}:turn",
text=_user_text(record.get("messages")) or str(record.get("text", "")),
write_label=float(event.get("write_label", event.get("write", record.get("write_label", 0.0)))),
forget_label=float(event.get("forget_label", event.get("forget", record.get("forget_label", 0.0)))),
kind=str(event.get("kind", record.get("kind", "conversation"))),
source=source,
subject=record.get("subject", ""),
attribute=record.get("attribute", ""),
value=record.get("value", ""),
answer=record.get("answer", ""),
answerable=record.get("answerable"),
messages=record.get("messages"),
)
if item is not None:
output.append(item)
return output
def _split_for_group(group_id: str, explicit: str | None, *, eval_ratio: float) -> str:
if explicit is not None:
return explicit
digest = hashlib.sha1(group_id.encode("utf-8")).hexdigest()
value = int(digest[:8], 16) / 0xFFFFFFFF
return "eval" if value < eval_ratio else "train"
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--source", action="append", help="input JSONL; may be repeated")
parser.add_argument("--eval-source", action="append", default=[], help="input JSONL forced into eval")
parser.add_argument("--output-dir", default="data/production_memory")
parser.add_argument("--eval-ratio", type=float, default=0.2)
args = parser.parse_args()
if not 0.0 < args.eval_ratio < 1.0:
raise SystemExit("--eval-ratio must be between 0 and 1")
source_paths = [_project_path(item) for item in (args.source or DEFAULT_SOURCES)]
eval_paths = [_project_path(item) for item in args.eval_source]
all_inputs = [(path, None) for path in source_paths] + [(path, "eval") for path in eval_paths]
examples: list[tuple[str, dict[str, Any]]] = []
source_stats: dict[str, Counter[str]] = defaultdict(Counter)
seen: set[tuple[str, str, float, float, str]] = set()
for path, forced_split in all_inputs:
if not path.exists():
raise FileNotFoundError(path)
source = str(path)
name_split = _explicit_split(path, forced=forced_split)
for line_number, record in _read_jsonl(path):
normalized = normalize_record(record, source=source, line_number=line_number)
for item in normalized:
dedupe_key = (
item["group_id"],
item["text"],
item["write_label"],
item["forget_label"],
item["kind"],
)
if dedupe_key in seen:
source_stats[source]["deduplicated"] += 1
continue
seen.add(dedupe_key)
split = _split_for_group(item["group_id"], name_split, eval_ratio=args.eval_ratio)
examples.append((split, item))
source_stats[source][split] += 1
output_dir = _project_path(args.output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
split_counts: Counter[str] = Counter()
for split in ("train", "eval"):
path = output_dir / f"{split}.jsonl"
with path.open("w", encoding="utf-8") as handle:
for item_split, item in examples:
if item_split == split:
handle.write(json.dumps(item, ensure_ascii=False) + "\n")
split_counts[split] += 1
manifest = {
"format_version": 1,
"schema": {
"text": "current turn presented to the write policy",
"messages": "optional original chat messages",
"write_label": "1 durable memory, 0 ordinary query/casual turn",
"forget_label": "1 explicit correction/forget request",
"group_id": "conversation/episode identity; never split across train and eval",
},
"bootstrap_data_warning": "Default files are local bootstrap/synthetic data; pass real exports with --source for production training.",
"inputs": [str(path) for path, _ in all_inputs],
"counts": dict(split_counts),
"source_stats": {key: dict(value) for key, value in source_stats.items()},
"dedupe_count": sum(value.get("deduplicated", 0) for value in source_stats.values()),
}
(output_dir / "manifest.json").write_text(
json.dumps(manifest, ensure_ascii=False, indent=2),
encoding="utf-8",
)
print(json.dumps(manifest, ensure_ascii=False, indent=2))
if __name__ == "__main__":
main()
build_replay_corpus.py
+291
View File
@@ -0,0 +1,291 @@
"""Merge the v6 corpus with the zero-overlap corpus into one feature bank.
Why a merge and not a joint encode: the v6 bank holds 2,124,552 rows whose values
were produced by a frozen encoder that must never change (they are the reference
features every earlier measurement was taken on). Re-encoding them would both cost
hours and risk silent drift, so the merged bank is built by **copying verified rows
and appending the new ones**, with the copy spot-checked byte-for-byte afterwards.
Invariants enforced here:
* every row of every source bank is copied exactly once, or skipped only because the
identical text already has a row -- and skipped rows are proven byte-equal;
* ``len(index) == bank rows`` (the trainer refuses a bank where they differ);
* every text referenced by the merged train/eval files resolves to a bank row;
* the merged manifest carries the sha256 of the *merged* dataset files, because the
trainer validates the bank against those frozen inputs.
Usage::
python -m V2_dpskw.build_replay_corpus ^
--source-data v6=data/router_training_v6 --source-bank v6=H:\\Memory\\nm_cache\\nm_router_v6\\feature_cache ^
--source-data zov=data/zero_overlap --source-bank zov=H:\\Memory\\nm_cache\\nm_zero_overlap\\feature_cache ^
--output-data data/router_replay_v7 --output-bank H:\\Memory\\nm_cache\\nm_replay_v7\\feature_cache
"""
from __future__ import annotations
import argparse
import hashlib
import json
import shutil
from pathlib import Path
import numpy as np
BANK_NAME = "features.f16.npy"
INDEX_NAME = "index.json"
MANIFEST_NAME = "manifest.json"
TEXTS_NAME = "texts.jsonl"
LENGTHS_NAME = "lengths.i32"
CHUNK_ROWS = 65536
def text_key(text: str) -> str:
return hashlib.sha1(str(text).encode("utf-8", "replace")).hexdigest()
def sha256_of(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for block in iter(lambda: handle.read(1 << 22), b""):
digest.update(block)
return digest.hexdigest()
def iter_episode_texts(path: Path):
with path.open("r", encoding="utf-8") as handle:
for line in handle:
line = line.strip()
if not line:
continue
row = json.loads(line)
query = str(row.get("query", "")).strip()
if query:
yield query
for candidate in row.get("candidates") or []:
text = str(candidate.get("text", "")).strip()
if text:
yield text
def concat_datasets(sources: list[tuple[str, Path]], output: Path) -> dict:
output.parent.mkdir(parents=True, exist_ok=True)
episodes = 0
with output.open("wb") as out:
for name, directory in sources:
part = directory / output.name
if not part.exists():
raise FileNotFoundError(f"missing {part}")
with part.open("rb") as handle:
shutil.copyfileobj(handle, out, length=1 << 22)
with part.open("rb") as handle:
episodes += sum(1 for _ in handle)
return {"path": str(output), "episodes": episodes, "sha256": sha256_of(output)}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--source-data", action="append", required=True,
help="NAME=DIR containing train.jsonl and eval.jsonl")
parser.add_argument("--source-bank", action="append", required=True,
help="NAME=DIR containing a verified feature bank")
parser.add_argument("--output-data", required=True)
parser.add_argument("--output-bank", required=True)
parser.add_argument("--model-path", default="qwen3_5_4b_natural_memory_v2")
parser.add_argument("--max-key-tokens", type=int, default=256)
parser.add_argument("--spot-checks", type=int, default=2000)
args = parser.parse_args()
data_sources = []
for item in args.source_data:
name, _, directory = item.partition("=")
data_sources.append((name, Path(directory)))
bank_sources = []
for item in args.source_bank:
name, _, directory = item.partition("=")
bank_sources.append((name, Path(directory)))
if [n for n, _ in data_sources] != [n for n, _ in bank_sources]:
raise SystemExit("--source-data and --source-bank must list the same names in the same order")
out_data = Path(args.output_data)
out_bank = Path(args.output_bank)
out_bank.mkdir(parents=True, exist_ok=True)
merged_datasets = {
"train": concat_datasets([(n, d) for n, d in data_sources], out_data / "train.jsonl"),
"eval": concat_datasets([(n, d) for n, d in data_sources], out_data / "eval.jsonl"),
}
print(json.dumps({"phase": "datasets", **merged_datasets}, ensure_ascii=False), flush=True)
# --- plan the merged row layout -------------------------------------------------
layout: list[tuple[int, int, str]] = [] # (source index, source row, sha1)
origin: list[tuple[int, int]] = [] # (source index, source row) per merged row
texts: list[str] = []
key_to_row: dict[str, int] = {}
duplicates: list[tuple[str, int, int]] = []
source_meta = []
for index, (name, directory) in enumerate(bank_sources):
manifest = json.loads((directory / MANIFEST_NAME).read_text(encoding="utf-8"))
if manifest.get("complete") is not True:
raise SystemExit(f"source bank {name} is not marked complete: {directory}")
lookup = json.loads((directory / INDEX_NAME).read_text(encoding="utf-8"))
bank = np.load(directory / BANK_NAME, mmap_mode="r")
if int(bank.shape[0]) != len(lookup):
raise SystemExit(f"source bank {name}: rows {bank.shape[0]} != index {len(lookup)}")
source_meta.append({"name": name, "path": str(directory), "rows": int(bank.shape[0]),
"hidden_size": int(bank.shape[1]), "dtype": str(bank.dtype),
"manifest_model_path": manifest.get("model_path")})
row_to_key = [None] * int(bank.shape[0])
for key, row in lookup.items():
row_to_key[int(row)] = key
length_path = directory / LENGTHS_NAME
# lengths.i32 is a raw int32 dump (written with ndarray.tofile), not a .npy.
lengths = np.fromfile(length_path, dtype=np.int32) if length_path.exists() else None
if lengths is not None and int(lengths.shape[0]) != int(bank.shape[0]):
raise SystemExit(f"source bank {name}: lengths {lengths.shape[0]} != rows {bank.shape[0]}")
seen_lines = 0
with (directory / TEXTS_NAME).open("r", encoding="utf-8") as handle:
for row, line in enumerate(handle):
seen_lines = row + 1
text = json.loads(line)
key = row_to_key[row]
if key is None:
raise SystemExit(f"source bank {name}: row {row} has no index entry")
if key != text_key(text):
raise SystemExit(f"source bank {name}: texts.jsonl line {row} does not match its index key")
if key in key_to_row:
duplicates.append((name, index, row))
continue
key_to_row[key] = len(origin)
origin.append((index, row))
texts.append(text)
if seen_lines != int(bank.shape[0]):
raise SystemExit(f"source bank {name}: texts.jsonl has {seen_lines} lines, bank has {bank.shape[0]} rows")
del row_to_key, lookup, bank, lengths
total = len(origin)
print(json.dumps({"phase": "layout", "merged_rows": total,
"sources": source_meta,
"duplicate_rows_skipped": len(duplicates)}, ensure_ascii=False), flush=True)
# --- copy the feature rows -----------------------------------------------------
hidden = source_meta[0]["hidden_size"]
if any(meta["hidden_size"] != hidden for meta in source_meta):
raise SystemExit("source banks disagree on hidden size")
banks = [np.load(directory / BANK_NAME, mmap_mode="r") for _, directory in bank_sources]
target = np.lib.format.open_memmap(out_bank / BANK_NAME, mode="w+",
dtype=np.float16, shape=(total, hidden))
lengths_out = np.empty(total, dtype=np.int32)
length_arrays = []
for _, directory in bank_sources:
path = directory / LENGTHS_NAME
length_arrays.append(np.fromfile(path, dtype=np.int32) if path.exists() else None)
by_source: dict[int, list[tuple[int, int]]] = {}
for merged_row, (source_index, source_row) in enumerate(origin):
by_source.setdefault(source_index, []).append((merged_row, source_row))
for source_index in sorted(by_source):
pairs = by_source[source_index]
bank = banks[source_index]
lengths = length_arrays[source_index]
# Copy contiguous source runs so the disk sees sequential reads.
start = 0
while start < len(pairs):
end = start + 1
while (end < len(pairs)
and pairs[end][1] == pairs[end - 1][1] + 1
and pairs[end][0] == pairs[end - 1][0] + 1):
end += 1
run = pairs[start:end]
if len(run) > CHUNK_ROWS:
for offset in range(0, len(run), CHUNK_ROWS):
piece = run[offset:offset + CHUNK_ROWS]
rows = [p[1] for p in piece]
target[[p[0] for p in piece]] = bank[rows[0]:rows[-1] + 1]
else:
rows = [p[1] for p in run]
target[[p[0] for p in run]] = bank[rows[0]:rows[-1] + 1]
start = end
if lengths is not None:
for merged_row, source_row in pairs:
lengths_out[merged_row] = lengths[source_row]
print(json.dumps({"phase": "copied_source", "name": source_meta[source_index]["name"],
"rows": len(pairs)}, ensure_ascii=False), flush=True)
target.flush()
del target
# --- index, texts, lengths -----------------------------------------------------
(out_bank / INDEX_NAME).write_text(json.dumps(key_to_row), encoding="utf-8")
with (out_bank / TEXTS_NAME).open("w", encoding="utf-8") as handle:
for text in texts:
handle.write(json.dumps(text, ensure_ascii=False) + "\n")
lengths_out.tofile(out_bank / LENGTHS_NAME)
print(json.dumps({"phase": "written", "index_entries": len(key_to_row)}, ensure_ascii=False), flush=True)
# --- verification --------------------------------------------------------------
problems: list[str] = []
merged = np.load(out_bank / BANK_NAME, mmap_mode="r")
if int(merged.shape[0]) != len(key_to_row):
problems.append(f"rows {merged.shape[0]} != index entries {len(key_to_row)}")
rng = np.random.default_rng(20260912)
checked = 0
for merged_row in rng.choice(total, size=min(args.spot_checks, total), replace=False):
source_index, source_row = origin[int(merged_row)]
if not np.array_equal(np.asarray(merged[merged_row]), np.asarray(banks[source_index][source_row])):
problems.append(f"row {merged_row} differs from {source_meta[source_index]['name']}:{source_row}")
if len(problems) > 5:
break
checked += 1
zero_rows = 0
for start in range(0, total, CHUNK_ROWS):
block = np.asarray(merged[start:start + CHUNK_ROWS])
zero_rows += int((~block.any(axis=1)).sum())
if zero_rows:
problems.append(f"{zero_rows} merged rows are all-zero (unencoded)")
missing = 0
for split in ("train", "eval"):
for text in iter_episode_texts(out_data / f"{split}.jsonl"):
if text_key(text) not in key_to_row:
missing += 1
if missing:
problems.append(f"{missing} dataset texts have no bank row")
manifest = {
"format_version": 2,
"merged_by": "build_replay_corpus.py",
"train_sha256": merged_datasets["train"]["sha256"],
"eval_sha256": merged_datasets["eval"]["sha256"],
"model_path": args.model_path,
"max_key_tokens": int(args.max_key_tokens),
"hidden_size": hidden,
"text_count": total,
"encoded_rows": total,
"complete": True,
"dtype": "float16_memmap",
"bank": BANK_NAME,
"index": INDEX_NAME,
"texts": TEXTS_NAME,
"sources": source_meta,
"duplicate_rows_skipped": len(duplicates),
"verification": {
"spot_checked_rows": checked,
"spot_check_mismatches": [p for p in problems if "differs" in p],
"zero_rows": zero_rows,
"dataset_texts_without_bank_row": missing,
"passed": not problems,
},
"datasets": merged_datasets,
}
(out_bank / MANIFEST_NAME).write_text(json.dumps(manifest, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps(manifest, ensure_ascii=False, indent=2), flush=True)
if problems:
raise SystemExit("merged bank failed verification: " + "; ".join(problems))
return 0
if __name__ == "__main__":
raise SystemExit(main())
chain_v5.ps1
+52
View File
@@ -0,0 +1,52 @@
# Chain: wait for the v5 feature encoder, then train and score automatically.
#
# 1. wait for the encoder process to exit;
# 2. verify the encoder wrote its real manifest (not the smoke placeholder);
# 3. train the V2-512 control and the XL-512 candidate on the identical v5 data;
# 4. score every router (old production, old v3-trained, new v5-trained) on the
# frozen 21,920-episode v5 eval and write the percentage scorecard.
param(
[Parameter(Mandatory = $true)][int]$EncoderPid,
[int]$Steps = 100000,
[int]$PollSeconds = 30
)
$ErrorActionPreference = 'Stop'
$Python = 'C:\Users\Administrator\miniconda3\envs\LLM\python.exe'
$ForkRoot = 'H:\Memory\V2_dpskw'
$CacheDir = 'H:\Memory\nm_cache\nm_router_v5\feature_cache'
$Manifest = Join-Path $CacheDir 'manifest.json'
Write-Output ("[{0}] waiting for encoder pid {1}" -f (Get-Date -Format 'HH:mm:ss'), $EncoderPid)
while (Get-Process -Id $EncoderPid -ErrorAction SilentlyContinue) {
Start-Sleep -Seconds $PollSeconds
}
Write-Output ("[{0}] encoder exited" -f (Get-Date -Format 'HH:mm:ss'))
$manifest = Get-Content -LiteralPath $Manifest -Raw | ConvertFrom-Json
if (-not $manifest.PSObject.Properties.Name.Contains('encoding')) {
throw "manifest has no 'encoding' section: the encoder did not finish cleanly"
}
if ([int]$manifest.text_count -lt 2000000) {
throw "encoder produced only $($manifest.text_count) texts"
}
Write-Output ("manifest ok: texts={0} dtype={1} grouping={2}" -f $manifest.text_count, $manifest.dtype, $manifest.encoding.grouping)
# --- 3. train ---------------------------------------------------------------
& pwsh -NoProfile -File (Join-Path $ForkRoot 'run_router_v5.ps1') -Steps $Steps
if ($LASTEXITCODE -ne 0) { throw "v5 training failed with exit code $LASTEXITCODE" }
# --- 4. score ---------------------------------------------------------------
$env:PYTHONPATH = 'H:\Memory'
Set-Location $ForkRoot
$old = 'H:\Memory\dynamic_memory_lab\checkpoints'
& $Python -m V2_dpskw.eval_router_v5 `
--run "V2-128 deployed(v3)=$old\natural_memory_v2_qwen_router_entities\memory_router_v2.pt" `
--run "V2-512 v3 best=$old\natural_memory_v2_router_512\router_best.pt" `
--run "V2-512 v3 final=$old\natural_memory_v2_router_512\memory_router_v2.pt" `
--run "V2-512 v5 best=$ForkRoot\checkpoints\router_v5_v2_512\router_best.pt" `
--run "V2-512 v5 final=$ForkRoot\checkpoints\router_v5_v2_512\memory_router_v2.pt" `
--run "XL-512 v5 best=$ForkRoot\checkpoints\router_v5_xl512\router_best.pt" `
--run "XL-512 v5 final=$ForkRoot\checkpoints\router_v5_xl512\memory_router_xl.pt" `
--output router_scorecard_v5.json --markdown router_scorecard_v5.md
Write-Output ("[{0}] chain complete" -f (Get-Date -Format 'HH:mm:ss'))
chain_v6.ps1
+66
View File
@@ -0,0 +1,66 @@
# Chain: wait for the v6 feature encoder, then train and score automatically.
#
# 1. wait for the encoder process to exit;
# 2. verify the encoder wrote its real manifest;
# 3. train the V2-512 control and the XL-512 candidate on the identical v6 data;
# 4. score every router (old production 128d, old v3-trained 512s, new v6 512s)
# on the frozen 21,920-episode v6 eval and write the percentage scorecard.
param(
[Parameter(Mandatory = $true)][int]$EncoderPid,
[int]$Steps = 100000,
[int]$PollSeconds = 30
)
$ErrorActionPreference = 'Stop'
$Python = 'C:\Users\Administrator\miniconda3\envs\LLM\python.exe'
$ForkRoot = 'H:\Memory\V2_dpskw'
$CacheDir = 'H:\Memory\nm_cache\nm_router_v6\feature_cache'
$Manifest = Join-Path $CacheDir 'manifest.json'
Write-Output ("[{0}] waiting for encoder pid {1}" -f (Get-Date -Format 'HH:mm:ss'), $EncoderPid)
while (Get-Process -Id $EncoderPid -ErrorAction SilentlyContinue) {
Start-Sleep -Seconds $PollSeconds
}
Write-Output ("[{0}] encoder exited" -f (Get-Date -Format 'HH:mm:ss'))
$manifest = Get-Content -LiteralPath $Manifest -Raw | ConvertFrom-Json
if (-not $manifest.PSObject.Properties.Name.Contains('encoding')) {
throw "manifest has no 'encoding' section: the encoder did not finish cleanly"
}
if ([int]$manifest.text_count -lt 2000000) {
throw "encoder produced only $($manifest.text_count) texts"
}
Write-Output ("manifest ok: texts={0} dtype={1} grouping={2}" -f $manifest.text_count, $manifest.dtype, $manifest.encoding.grouping)
# --- 3. train ---------------------------------------------------------------
& pwsh -NoProfile -File (Join-Path $ForkRoot 'run_router_v6.ps1') -Steps $Steps
if ($LASTEXITCODE -ne 0) { throw "v6 training failed with exit code $LASTEXITCODE" }
# --- 4. score ---------------------------------------------------------------
$env:PYTHONPATH = 'H:\Memory'
Set-Location $ForkRoot
$old = 'H:\Memory\dynamic_memory_lab\checkpoints'
& $Python -m V2_dpskw.eval_router_v5 `
--train-file data/router_training_v6/train.jsonl `
--eval-file data/router_training_v6/eval.jsonl `
--feature-cache $CacheDir `
--run "V2-128 deployed(v3)=$old\natural_memory_v2_qwen_router_entities\memory_router_v2.pt" `
--run "V2-512 v3 best=$old\natural_memory_v2_router_512\router_best.pt" `
--run "V2-512 v3 final=$old\natural_memory_v2_router_512\memory_router_v2.pt" `
--run "V2-512 v6 best=$ForkRoot\checkpoints\router_v6_v2_512\router_best.pt" `
--run "V2-512 v6 final=$ForkRoot\checkpoints\router_v6_v2_512\memory_router_v2.pt" `
--run "XL-512 v6 best=$ForkRoot\checkpoints\router_v6_xl512\router_best.pt" `
--run "XL-512 v6 final=$ForkRoot\checkpoints\router_v6_xl512\memory_router_xl.pt" `
--output router_scorecard_v6.json --markdown router_scorecard_v6.md
if ($LASTEXITCODE -ne 0) { throw "scorecard failed with exit code $LASTEXITCODE" }
# --- 5. verdict: per-axis dominance against the best baseline ---------------
& $Python -m V2_dpskw.verdict_router_v6 `
--scorecard router_scorecard_v6.json `
--candidate "XL-512 v6 best" --candidate "XL-512 v6 final" `
--candidate "V2-512 v6 best" --candidate "V2-512 v6 final" `
--baseline-prefix "V2-128 deployed" --baseline-prefix "V2-512 v3" `
--output router_verdict_v6.json --markdown router_verdict_v6.md
$verdictCode = $LASTEXITCODE
Write-Output ("verdict exit code: {0} (0 = complete dominance, 2 = some axis failed)" -f $verdictCode)
Write-Output ("[{0}] chain complete" -f (Get-Date -Format 'HH:mm:ss'))
chat_qwen_memory.py
+297
View File
@@ -0,0 +1,297 @@
"""Minimal interactive chat using the persistent Qwen dynamic memory."""
from __future__ import annotations
import argparse
import sys
from pathlib import Path
import torch
from .qwen_integration import (
DEFAULT_MEMORY_RESET_TOKEN,
QwenMemoryConfig,
load_memory_config,
load_qwen_dynamic,
load_tokenizer,
resolve_memory_reset_token,
split_memory_candidates,
)
def _memory_system_prefix(tokenizer, content: str):
"""Encode a valid system-message prefix without adding a fake query."""
full = tokenizer.apply_chat_template(
[
{"role": "system", "content": content},
{"role": "user", "content": "__memory_query_boundary__"},
],
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
enable_thinking=False,
)
input_ids = full["input_ids"]
im_start = tokenizer.convert_tokens_to_ids("<|im_start|>")
positions = (input_ids[0] == int(im_start)).nonzero(as_tuple=False).flatten()
if positions.numel() < 2:
raise RuntimeError("could not locate the system/user memory boundary")
end = int(positions[1].item())
return {
"input_ids": input_ids[:, :end],
"attention_mask": torch.ones((1, end), dtype=torch.long),
}
def main() -> None:
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
parser = argparse.ArgumentParser()
parser.add_argument("--model-path", default=".")
parser.add_argument("--adapter", default=None)
parser.add_argument(
"--natural-language-memory",
action="store_true",
help="enable the model-owned exact text memory bank and internal retrieval prefix",
)
parser.add_argument(
"--memory-state",
default=None,
help="user runtime memory file; it is loaded at startup and saved after each turn",
)
parser.add_argument("--max-new-tokens", type=int, default=128)
parser.add_argument("--no-4bit", action="store_true")
parser.add_argument(
"--persistent-memory",
action="store_true",
help="keep native memory inside the model instance across turns",
)
parser.add_argument("--reset-token", default=None)
parser.add_argument("--reset-token-id", type=int, default=None)
parser.add_argument(
"--persist-in-adapter",
action="store_true",
help="also checkpoint current user memory into the adapter package",
)
args = parser.parse_args()
tokenizer = load_tokenizer(args.model_path)
memory_config = load_memory_config(args.adapter) if args.adapter else None
if args.natural_language_memory and memory_config is None:
memory_config = QwenMemoryConfig(natural_language_memory=True)
if memory_config is not None and args.natural_language_memory:
memory_config.natural_language_memory = True
if memory_config is not None and memory_config.native_mode and args.persistent_memory:
memory_config.persistent_memory = True
if memory_config is not None:
if args.reset_token_id is not None:
memory_config.reset_token_id = args.reset_token_id
elif args.reset_token is not None:
memory_config.reset_token_id = resolve_memory_reset_token(tokenizer, args.reset_token)
elif memory_config.native_mode and memory_config.reset_token_id is None:
memory_config.reset_token_id = resolve_memory_reset_token(tokenizer)
model = load_qwen_dynamic(
args.model_path,
memory_config=memory_config,
load_in_4bit=not args.no_4bit,
)
if args.adapter:
model.load_memory_adapter(args.adapter)
model.eval()
device = model._find_layer_device()
state_path = Path(args.memory_state) if args.memory_state else None
if state_path is not None and state_path.exists():
model.load_runtime_memory(state_path, device=device)
print(f"已加载用户 memory_state：{state_path}")
print("普通消息会自动判断并保存重要信息；/remember <事实> 强制写入，/reset 清空，/save 保存，/quit 退出。")
if memory_config is not None and memory_config.reset_token_id is not None:
print(f"也可在用户消息中发送重置 token：{args.reset_token or DEFAULT_MEMORY_RESET_TOKEN}")
def save_state() -> None:
if state_path is not None and model.runtime.state is not None:
model.save_runtime_memory(state_path)
print(f"已保存：{state_path}")
if args.persist_in_adapter and args.adapter and model.runtime.state is not None:
model.save_persistent_memory_checkpoint(args.adapter)
print(f"已将当前用户记忆写入模型适配器：{args.adapter}")
try:
while True:
user_text = input("你> ").strip()
if user_text == "/quit":
break
if user_text == "/reset":
model.reset_memory(batch_size=1, device=device)
save_state()
print("已清空动态记忆。")
continue
if user_text == "/save":
save_state()
continue
if user_text.startswith("/remember "):
fact = user_text[len("/remember ") :].strip()
if not fact:
continue
messages = [
{"role": "user", "content": fact},
{"role": "assistant", "content": "好的，我会记住这件事。"},
]
encoded = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=False,
return_tensors="pt",
return_dict=True,
enable_thinking=False,
)
encoded = {
key: value.to(device)
for key, value in encoded.items()
if isinstance(value, torch.Tensor)
}
memory_text = _memory_system_prefix(
tokenizer,
"以下是与当前用户相关的已保存长期记忆。仅在问题相关时使用，不要编造：\n" + fact,
)
memory_text_input_ids = memory_text["input_ids"].to(device)
memory_text_attention_mask = memory_text.get("attention_mask")
if memory_text_attention_mask is None:
memory_text_attention_mask = torch.ones_like(memory_text_input_ids)
memory_text_attention_mask = memory_text_attention_mask.to(device)
memory_key = tokenizer(fact, add_special_tokens=False, return_tensors="pt")
memory_key_input_ids = memory_key["input_ids"].to(device)
memory_key_attention_mask = memory_key.get("attention_mask")
if memory_key_attention_mask is None:
memory_key_attention_mask = torch.ones_like(memory_key_input_ids)
memory_key_attention_mask = memory_key_attention_mask.to(device)
memory_storage = tokenizer(
fact,
add_special_tokens=False,
return_tensors="pt",
)
with torch.no_grad():
model(
**encoded,
read_memory=False,
update_memory=True,
return_memory=True,
use_cache=False,
memory_text_input_ids=memory_text_input_ids,
memory_text_attention_mask=memory_text_attention_mask,
memory_key_input_ids=memory_key_input_ids,
memory_key_attention_mask=memory_key_attention_mask,
memory_storage_input_ids=memory_storage["input_ids"].to(device),
memory_storage_attention_mask=memory_storage.get(
"attention_mask",
torch.ones_like(memory_storage["input_ids"]),
).to(device),
force_memory_write=True,
)
save_state()
print("已写入动态记忆。")
continue
if not user_text:
continue
messages = [{"role": "user", "content": user_text}]
encoded = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
enable_thinking=False,
)
encoded = {
key: value.to(device)
for key, value in encoded.items()
if isinstance(value, torch.Tensor)
}
memory_query = tokenizer(
user_text,
add_special_tokens=False,
return_tensors="pt",
)
memory_query_input_ids = memory_query["input_ids"].to(device)
memory_query_attention_mask = memory_query.get("attention_mask")
if memory_query_attention_mask is None:
memory_query_attention_mask = torch.ones_like(memory_query_input_ids)
memory_query_attention_mask = memory_query_attention_mask.to(device)
with torch.no_grad():
# Native mode first gives the prompt to the learned controller
# so it can decide whether each fact-sized candidate is worth
# storing. The generation itself is read-only, preventing the
# model from accidentally memorizing its own answer text.
if memory_config is not None and memory_config.native_mode:
for candidate in split_memory_candidates(user_text):
candidate_encoded = tokenizer.apply_chat_template(
[{"role": "user", "content": candidate}],
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
enable_thinking=False,
)
candidate_encoded = {
key: value.to(device)
for key, value in candidate_encoded.items()
if isinstance(value, torch.Tensor)
}
memory_text = _memory_system_prefix(
tokenizer,
"以下是与当前用户相关的已保存长期记忆。仅在问题相关时使用，不要编造：\n"
+ candidate,
)
memory_key = tokenizer(
candidate,
add_special_tokens=False,
return_tensors="pt",
)
memory_key_input_ids = memory_key["input_ids"].to(device)
memory_key_attention_mask = memory_key.get("attention_mask")
if memory_key_attention_mask is None:
memory_key_attention_mask = torch.ones_like(memory_key_input_ids)
memory_storage = tokenizer(
candidate,
add_special_tokens=False,
return_tensors="pt",
)
model(
**candidate_encoded,
read_memory=False,
update_memory=True,
return_memory=True,
use_cache=False,
memory_text_input_ids=memory_text["input_ids"].to(device),
memory_text_attention_mask=torch.ones_like(
memory_text["input_ids"], device=device
),
memory_key_input_ids=memory_key_input_ids,
memory_key_attention_mask=memory_key_attention_mask.to(device),
memory_storage_input_ids=memory_storage["input_ids"].to(device),
memory_storage_attention_mask=memory_storage.get(
"attention_mask",
torch.ones_like(memory_storage["input_ids"]),
).to(device),
)
output_ids = model.generate(
**encoded,
max_new_tokens=args.max_new_tokens,
do_sample=False,
update_memory=False,
memory_query_input_ids=memory_query_input_ids,
memory_query_attention_mask=memory_query_attention_mask,
)
response_ids = output_ids[0, encoded["input_ids"].shape[1] :]
print(f"AI> {tokenizer.decode(response_ids, skip_special_tokens=True)}")
save_state()
except (EOFError, KeyboardInterrupt):
print()
finally:
save_state()
if __name__ == "__main__":
main()
check_rag_logic.py
+51
View File
@@ -0,0 +1,51 @@
"""Exercise the RAG baseline's model-free logic (BM25, prompt, corpus loading).
Runs without touching the GPU so it can be checked while the memory run owns the
card.
"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from V2_dpskw.strong_rag_locomo import BM25, build_prompt, tokenize
from V2_dpskw.eval_end_to_end_memory import build_cases
# 1. BM25 must rank the turn that actually states the answer first.
pool = [
"Melanie: I finally finished that painting of a sunrise.",
"Caroline: I went to the LGBTQ support group on 7 May 2023.",
"Caroline: Work has been really busy this month.",
"Melanie: That sounds rough, hope you are okay.",
]
ranking = BM25(pool).rank("When did Caroline go to the LGBTQ support group?")
print("BM25 ranking:", ranking)
print("top doc:", pool[ranking[0]])
assert ranking[0] == 1, "BM25 failed to rank the answering turn first"
# 2. A question with no lexical overlap should not crash and should still rank.
ranking2 = BM25(pool).rank("What colour was the painting?")
print("BM25 (no-overlap) top:", pool[ranking2[0]])
# 3. Prompt shape.
prompt = build_prompt("When did Caroline go?", pool[:2])
assert "Question: When did Caroline go?" in prompt and prompt.rstrip().endswith("Answer:")
print("prompt ok, length", len(prompt))
# 4. Tokenizer sanity.
print("tokens:", tokenize("I don't know — 7 May 2023!"))
assert tokenize("7 May 2023") == ["7", "may", "2023"]
# 5. Corpus loads with the harness' own loader and the answerable split is right.
corpus_path = Path(sys.argv[1] if len(sys.argv) > 1 else r"H:\Memory\V2_dpskw\data\net_locomo\eval.jsonl")
cases = build_cases(corpus_path, 40)
ans = sum(1 for c in cases if c["answerable"])
print(f"corpus: {len(cases)} cases, answerable {ans}, adversarial {len(cases) - ans}")
assert ans == 155 and len(cases) == 195, "unexpected answerable split"
# 6. Every pool has at least a few candidates and no duplicate text.
for case in cases:
assert len(case["facts"]) >= 4, case["query"]
assert len(set(case["facts"])) == len(case["facts"]), "duplicate candidate text"
print("all pools >= 4 candidates, no duplicates")
print("\nOK")
check_refusal_bilingual.py
+33
View File
@@ -0,0 +1,33 @@
"""Spot-check bilingual refusal detection on real reply wordings."""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from V2_dpskw.eval_scoring import is_refusal
CASES = [
("zh", "关于您的 weekly 会议安排，当前长期记忆中未包含相关信息，无法回答。", True),
("zh", "您的协议编号未知。", True),
("zh", "您的培训周期是 32GB。", False),
("zh", "我固定每周三 22:00 做上线。", False),
("en", "I don't know when that happened.", True),
("en", "There is no mention of a charity race in the conversation.", True),
("en", "The date was not mentioned.", True),
("en", "I cannot determine that from the dialogue.", True),
("en", "She was unable to answer that question about her research.", True),
("en", "The conversation does not mention any support group.", True),
("en", "I'm not sure about that.", True),
("en", "She went to the LGBTQ support group on 7 May 2023.", False),
("en", "2022", False),
("en", "Caroline researched adoption agencies.", False),
("en", "Melanie painted a sunrise in 2022.", False),
]
bad = 0
for lang, reply, expected in CASES:
got = is_refusal(reply)
flag = "ok " if got == expected else "BAD"
if got != expected:
bad += 1
print(f"{flag} [{lang}] expected={'REFUSE' if expected else 'assert'} got={'REFUSE' if got else 'assert'} {reply[:64]}")
print(f"\nmismatches: {bad}/{len(CASES)}")
check_router_cache.py
+117
View File
@@ -0,0 +1,117 @@
"""Fail-fast check that router training can reuse a frozen feature cache.
The XL router reuses the exact same frozen Qwen features as the 512-dim
baseline, so training must never re-encode the corpus. ``_prepare_feature_cache``
silently loads the 4B model when the cache looks stale, which would cost ~20
minutes and 9 GiB of VRAM. This tool runs the real cache-validation path with
the model loader replaced by a hard failure, so a stale cache is reported in
seconds instead of being discovered halfway through a training launch.
Usage (from the fork root, e.g. H:\\Memory\\V2_dpskw)::
python -m V2_dpskw.check_router_cache ^
--train-file data/router_training_v3/train.jsonl ^
--eval-file data/router_training_v3/eval.jsonl ^
--feature-cache-dir checkpoints/router_shared/feature_cache ^
--model-path qwen3_5_4b_natural_memory_v2
"""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
if __package__ in {None, ""}:
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
import V2_dpskw.train_memory_router_large as trainer
def _explode(*_args: object, **_kwargs: object) -> object:
raise RuntimeError(
"feature cache is stale: the trainer would now load the 4B Qwen model to "
"re-encode the corpus. Fix the cache or pass --rebuild-features deliberately."
)
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--train-file", default="data/router_training_v3/train.jsonl")
parser.add_argument("--eval-file", default="data/router_training_v3/eval.jsonl")
parser.add_argument("--feature-cache-dir", default="checkpoints/router_shared/feature_cache")
parser.add_argument("--model-path", default="qwen3_5_4b_natural_memory_v2")
parser.add_argument("--max-key-tokens", type=int, default=256)
parser.add_argument("--hidden-size", type=int, default=2560)
args = parser.parse_args()
train_path = trainer._resolve_path(args.train_file)
eval_path = trainer._resolve_path(args.eval_file)
cache_dir = trainer._resolve_path(args.feature_cache_dir)
model_path = trainer._resolve_path(args.model_path)
print(json.dumps({
"train_file": str(train_path),
"eval_file": str(eval_path),
"feature_cache_dir": str(cache_dir),
"model_path": str(model_path),
}, ensure_ascii=False, indent=2), flush=True)
train = trainer._read_episodes(train_path)
evaluation = trainer._read_episodes(eval_path)
texts, _lookup = trainer._collect_texts(train + evaluation)
expected = {
"format_version": 1,
"train_sha256": trainer._sha256(train_path),
"eval_sha256": trainer._sha256(eval_path),
"model_path": str(model_path),
"max_key_tokens": int(args.max_key_tokens),
"hidden_size": int(args.hidden_size),
"text_count": len(texts),
"dtype": "float16_cpu",
}
saved_path = trainer._cache_meta_path(cache_dir)
saved = json.loads(saved_path.read_text(encoding="utf-8")) if saved_path.exists() else {}
compatible = trainer._cache_is_compatible(cache_dir, expected)
for key in sorted(expected):
mark = "==" if saved.get(key) == expected[key] else "!="
print(f"{mark} {key}: expected={expected[key]!r} saved={saved.get(key)!r}", flush=True)
if not compatible:
print("CACHE MISS", flush=True)
return 1
# Run the real code path with the model loader disabled.
trainer.load_qwen_dynamic = _explode # type: ignore[assignment]
trainer.load_tokenizer = _explode # type: ignore[assignment]
vectors, lookup, meta = trainer._prepare_feature_cache(
argparse.Namespace(
feature_cache_dir=str(cache_dir),
model_path=str(model_path),
max_key_tokens=int(args.max_key_tokens),
hidden_size=int(args.hidden_size),
rebuild_features=False,
precompute_features=True,
gpu_memory_gb=0.0,
no_4bit=False,
encode_batch_size=1,
precompute_log_every=256,
),
train,
evaluation,
train_path,
eval_path,
)
print(json.dumps({
"verdict": "CACHE HIT",
"feature_shape": list(vectors.shape),
"dtype": str(vectors.dtype),
"unique_texts": len(lookup),
"train_episodes": len(train),
"eval_episodes": len(evaluation),
"meta": meta,
}, ensure_ascii=False, indent=2), flush=True)
return 0
if __name__ == "__main__":
raise SystemExit(main())
check_router_swap.py
+125
View File
@@ -0,0 +1,125 @@
"""Check that a new router checkpoint is a drop-in replacement for the deployed one.
The shipped router lives inside the model package's memory safetensors shard under
``dynamic_memory.memory_router_v2.*``. Before claiming a new router "replaces" it,
verify mechanically that:
1. the key sets and tensor shapes are identical (modulo the shard prefix), so the
swap is a rename rather than a runtime change;
2. the checkpoint loads into the same ``MemoryRouterV2`` construction the runtime
uses, and produces finite scores on real candidates;
3. the runtime path (``PagedMemoryBankV2`` driven by that router) still routes.
Usage::
python -m V2_dpskw.check_router_swap ^
--package qwen3_5_4b_natural_memory_v2 ^
--candidate checkpoints/router_v6_v2_128/router_best.pt
"""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
import torch
if __package__ in {None, ""}:
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from V2_dpskw.memory_os_v2 import MemoryRouterV2, PagedMemoryBankV2
PREFIX = "dynamic_memory.memory_router_v2."
def _shard_for(package: Path, needle: str) -> tuple[Path, dict]:
index = json.loads((package / "model.safetensors.index.json").read_text(encoding="utf-8"))
weight_map = index["weight_map"]
keys = [key for key in weight_map if needle in key]
if not keys:
raise SystemExit(f"package has no tensors matching {needle!r}")
shard = package / weight_map[keys[0]]
with shard.open("rb") as handle:
header_len = int.from_bytes(handle.read(8), "little")
header = json.loads(handle.read(header_len).decode("utf-8"))
return shard, header
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--package", default="qwen3_5_4b_natural_memory_v2")
parser.add_argument("--candidate", default="checkpoints/router_v6_v2_128/router_best.pt")
parser.add_argument("--output", default="router_swap_check.json")
args = parser.parse_args()
package = Path(args.package)
candidate_path = Path(args.candidate)
shard, header = _shard_for(package, "memory_router_v2.")
deployed = {
key[len(PREFIX):]: tuple(value["shape"])
for key, value in header.items()
if key.startswith(PREFIX)
}
payload = torch.load(candidate_path, map_location="cpu", weights_only=True)
state = payload.get("router_state_dict", payload)
candidate = {key: tuple(value.shape) for key, value in state.items()}
report: dict = {
"package": str(package),
"shard": str(shard),
"deployed_keys": len(deployed),
"candidate_keys": len(candidate),
"missing_in_candidate": sorted(set(deployed) - set(candidate)),
"extra_in_candidate": sorted(set(candidate) - set(deployed)),
"shape_mismatches": {
key: {"deployed": deployed[key], "candidate": candidate[key]}
for key in set(deployed) & set(candidate)
if deployed[key] != candidate[key]
},
}
report["drop_in_compatible"] = not (
report["missing_in_candidate"] or report["extra_in_candidate"] or report["shape_mismatches"]
)
# 2. Load into the runtime construction and score real candidates.
infer = {
"hidden_size": int(candidate["query_projection.weight"][1]),
"router_dim": int(candidate["query_projection.weight"][0]),
"num_heads": int(candidate["head_gate.weight"][0]),
"max_hops": int(candidate["hop_controller.2.weight"][0]) - 1,
}
router = MemoryRouterV2(**infer)
router.load_state_dict(state, strict=True)
router.eval()
query = torch.randn(2, infer["hidden_size"])
candidates = torch.randn(2, 32, infer["hidden_size"])
with torch.inference_mode():
out = router(query, candidates)
report["runtime_construction"] = infer
report["scores_finite"] = bool(torch.isfinite(out["scores"]).all())
report["score_shape"] = list(out["scores"].shape)
# 3. Drive the paged bank with it, as the memory OS does.
bank = PagedMemoryBankV2(
infer["hidden_size"], router=router, page_capacity=2, max_pages=64, hot_pages=2,
top_k_pages=2, top_k_records=3, max_hops=infer["max_hops"], coarse_index_bits=8,
)
for index in range(3):
bank.write(text=f"fact {index}", key=torch.randn(infer["hidden_size"]),
entity=f"entity-{index}", attribute="value", value=str(index), confidence=0.9)
records, decision = bank.query(query_key=torch.randn(infer["hidden_size"]), query_text="fact 1")
report["bank_routed_records"] = len(records)
report["bank_key_dim"] = int(bank.key_dim)
report["bank_routing_ok"] = bool(records) and int(bank.key_dim) == infer["router_dim"]
Path(args.output).write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps(report, ensure_ascii=False, indent=2), flush=True)
ok = report["drop_in_compatible"] and report["scores_finite"] and report["bank_routing_ok"]
print("\nVERDICT:", "DROP-IN REPLACEMENT OK" if ok else "NOT A DROP-IN REPLACEMENT")
return 0 if ok else 1
if __name__ == "__main__":
raise SystemExit(main())
compact_mega_router_source.py
+78
View File
@@ -0,0 +1,78 @@
"""Reduce mega memory records to the evidence needed by a router episode.
The original stress file keeps very large long-context fact lists. The
router only needs the gold evidence plus a small local context; global hard
negatives are supplied later by prepare_memory_router_dataset.py.
"""
from __future__ import annotations
import argparse
import json
from collections import Counter
from pathlib import Path
from typing import Any
def clean(value: Any) -> str:
return " ".join(str(value or "").replace("\x00", " ").split()).strip()
def compact(row: dict[str, Any], local_limit: int) -> dict[str, Any]:
facts = [fact for fact in row.get("facts", []) if isinstance(fact, dict) and clean(fact.get("text"))]
acceptable = [clean(value) for value in row.get("acceptable", []) if clean(value)]
positive = [fact for fact in facts if any(value.lower() in clean(fact.get("text")).lower() for value in acceptable)]
category = clean(row.get("category")) or "unknown"
selected: list[dict[str, Any]] = []
seen: set[str] = set()
def add(fact: dict[str, Any]) -> None:
key = clean(fact.get("text"))
if key and key not in seen and len(selected) < local_limit:
selected.append(fact)
seen.add(key)
for fact in positive:
add(fact)
if category == "unknown_abstention":
for fact in facts[:local_limit]:
add(fact)
else:
for fact in facts[:local_limit]:
add(fact)
for fact in reversed(facts[-local_limit:]):
add(fact)
output = dict(row)
output["facts"] = selected
output["metadata"] = {**(row.get("metadata") if isinstance(row.get("metadata"), dict) else {}), "compact_router_source": True}
return output
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--input", required=True)
parser.add_argument("--output", required=True)
parser.add_argument("--local-limit", type=int, default=8)
args = parser.parse_args()
if args.local_limit < 1:
raise SystemExit("local-limit must be positive")
counts: Counter[str] = Counter()
source = Path(args.input)
output = Path(args.output)
output.parent.mkdir(parents=True, exist_ok=True)
with source.open("r", encoding="utf-8") as src, output.open("w", encoding="utf-8") as dst:
for line in src:
line = line.strip()
if not line:
continue
row = compact(json.loads(line), args.local_limit)
counts[clean(row.get("category")) or "unknown"] += 1
dst.write(json.dumps(row, ensure_ascii=False, separators=(",", ":")) + "\n")
print(json.dumps({"input": str(source), "output": str(output), "rows": sum(counts.values()), "categories": dict(counts)}, ensure_ascii=False, indent=2))
if __name__ == "__main__":
main()
compare_locomo.py
+169
View File
@@ -0,0 +1,169 @@
"""Side-by-side LoCoMo comparison: memory system vs strong-RAG baselines.
Reads stored replies only -- no GPU -- and reports several metrics per system so
no conclusion rests on one brittle string match:
* **exact** -- benchmark answer string found in the reply (the project's
primary metric; whitespace-insensitive now).
* **token recall** -- fraction of the answer's content tokens present. A reply
that says "she went to an LGBTQ support group on 7 May" against the benchmark's
"7 May 2023" is partly right; exact matching alone would hide that.
* **token all** -- every content token present (a strict paraphrase-tolerant
variant).
* **refusal** -- on adversarial questions (no answer exists), did the system
decline? And, worse, did it *assert the trap answer* -- the plausible wrong
answer the benchmark supplies?
Pairing between a run's rows and the corpus is verified by query equality before
anything is scored; a silent misalignment once produced a fake 0.94 AUC in this
project, so it is checked rather than assumed.
"""
from __future__ import annotations
import argparse
import json
import sys
from collections import defaultdict
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from V2_dpskw.eval_scoring import is_refusal, squash
def load_corpus(path: Path, per_category: int = 40):
grouped: dict[str, list[dict]] = defaultdict(list)
with path.open("r", encoding="utf-8") as handle:
for line in handle:
line = line.strip()
if line:
row = json.loads(line)
grouped[str((row.get("metadata") or {}).get("category", ""))].append(row)
cases = []
for category in sorted(grouped):
cases.extend(grouped[category][:per_category])
return cases
def load_runs(path: Path):
data = json.loads(path.read_text(encoding="utf-8"))
out = {}
for label, body in data.items():
out[label] = body if isinstance(body, list) else body.get("rows", [])
return out
def content_tokens(text: str) -> list[str]:
return [t for t in squash(text).split() if t]
def score_row(row: dict, case: dict) -> dict:
meta = case.get("metadata") or {}
reply = row.get("reply", "")
squashed_reply = squash(reply)
acceptable = [str(a) for a in (meta.get("acceptable") or []) if str(a).strip()]
exact = any(squash(a) in squashed_reply for a in acceptable)
tokens = [str(t) for t in (meta.get("answer_tokens") or [])]
present = [t for t in tokens if t in squashed_reply]
token_recall = (len(present) / len(tokens)) if tokens else 0.0
token_all = bool(tokens) and len(present) == len(tokens)
refused = is_refusal(reply)
trap = str(meta.get("adversarial_answer") or "")
asserted_trap = bool(trap) and squash(trap) in squashed_reply
return {"exact": exact, "token_recall": token_recall, "token_all": token_all,
"refused": refused, "asserted_trap": asserted_trap,
"answer_tokens": len(tokens)}
def evaluate(rows: list[dict], cases: list[dict]) -> dict:
if len(rows) != len(cases):
raise SystemExit(f"row/case count mismatch: {len(rows)} vs {len(cases)}")
for row, case in zip(rows, cases):
if str(row.get("query", "")).strip() != str(case.get("query", "")).strip():
raise SystemExit(f"pairing mismatch: {row.get('query')!r} vs {case.get('query')!r}")
buckets = defaultdict(lambda: defaultdict(list))
for row, case in zip(rows, cases):
s = score_row(row, case)
cat = str((case.get("metadata") or {}).get("category", "?"))
buckets["all"][cat].append(s)
buckets["overall"][cat].append(s)
return buckets
def summarize(buckets) -> dict:
out = {}
for scope, per_cat in buckets.items():
flat = [s for rows in per_cat.values() for s in rows]
if not flat:
continue
ans = [s for s in flat if s["answer_tokens"] or not s["asserted_trap"]]
out[scope] = {
"n": len(flat),
"exact_pct": 100 * sum(s["exact"] for s in flat) / len(flat),
"token_recall_pct": 100 * sum(s["token_recall"] for s in flat) / len(flat),
"token_all_pct": 100 * sum(s["token_all"] for s in flat) / len(flat),
"refused_pct": 100 * sum(s["refused"] for s in flat) / len(flat),
"asserted_trap_pct": 100 * sum(s["asserted_trap"] for s in flat) / len(flat),
}
return out
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--corpus", type=Path, default=Path("data/net_locomo/eval.jsonl"))
parser.add_argument("--run", action="append", required=True, help="LABEL=path.json")
parser.add_argument("--per-category", type=int, default=40)
parser.add_argument("--json-out", type=Path, default=None)
args = parser.parse_args()
cases = load_corpus(args.corpus, args.per_category)
print(f"corpus cases: {len(cases)}")
answerable = sum(1 for c in cases if (c.get("metadata") or {}).get("answerable"))
print(f" answerable {answerable} adversarial(no answer) {len(cases) - answerable}\n")
payload = {}
for spec in args.run:
label, _, path = spec.partition("=")
runs = load_runs(Path(path))
for run_label, rows in runs.items():
name = f"{label}:{run_label}" if len(runs) > 1 else label
result = summarize(evaluate(rows, cases))
payload[name] = result
header = f"{'system':<26}{'exact':>9}{'tok-recall':>12}{'tok-all':>9}{'refused':>9}{'asserted-trap':>15}"
print(header)
print("-" * len(header))
for name, result in payload.items():
o = result.get("overall", {})
print(f"{name:<26}{o.get('exact_pct', 0):>8.2f}%{o.get('token_recall_pct', 0):>11.2f}%"
f"{o.get('token_all_pct', 0):>8.2f}%{o.get('refused_pct', 0):>8.2f}%"
f"{o.get('asserted_trap_pct', 0):>14.2f}%")
print("\nper-category exact match")
categories = sorted({c for result in payload.values() for c in result.get("overall", {})})
print(f"{'category':<16}" + "".join(f"{name[:18]:>20}" for name in payload))
for cat in categories:
line = f"{cat:<16}"
for name in payload:
v = payload[name].get("overall", {}).get(cat)
line += f"{v['exact_pct']:>19.2f}%" if v else f"{'-':>20}"
print(line)
print("\nper-category refusal rate (adversarial is the axis that matters)")
print(f"{'category':<16}" + "".join(f"{name[:18]:>20}" for name in payload))
for cat in categories:
line = f"{cat:<16}"
for name in payload:
v = payload[name].get("overall", {}).get(cat)
line += f"{v['refused_pct']:>19.2f}%" if v else f"{'-':>20}"
print(line)
if args.json_out:
args.json_out.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
print(f"\nwrote {args.json_out}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
compare_nm2_batteries.py
+133
View File
@@ -0,0 +1,133 @@
"""Side-by-side comparison of the NM2.1 and original-NM2 memory batteries.
Reads the tagged battery outputs and prints one comparison table, so the NM2.1 numbers
are never quoted from scrollback. All rates are printed as percentages.
Usage::
python -m V2_dpskw.compare_nm2_batteries --tag NM2.1=nm2_1 --tag 原版NM2=nm2_orig
"""
from __future__ import annotations
import argparse
import json
from pathlib import Path
def load(path: Path) -> dict:
try:
return json.loads(path.read_text(encoding="utf-8"))
except Exception as exc: # missing or malformed run
return {"__error__": str(exc)}
def dig(data: dict, *path, default=None):
current = data
for step in path:
if not isinstance(current, dict) or step not in current:
return default
current = current[step]
return current
def first_key(data: dict) -> str | None:
if not data or "__error__" in data:
return None
return next(iter(data))
def row_for(tag: str) -> dict:
a = load(Path(f"{tag}_e2e.json"))
b = load(Path(f"{tag}_critical_e2e.json"))
c = load(Path(f"{tag}_runtime_e2e.json"))
d = load(Path(f"{tag}_restart.json"))
ka, kb = first_key(a), first_key(b)
# The runtime evaluator labels its runs by the knobs under test, so take whichever
# run key the file actually holds rather than assuming one.
c_key = next(iter((c.get("results") or {}).keys()), "0.00") if "__error__" not in c else "0.00"
return {
"A_cases": dig(a, ka, "summary", "cases") if ka else None,
"A_overall": dig(a, ka, "summary", "accuracy_pct") if ka else None,
"A_answerable": dig(a, ka, "summary", "answerable_accuracy_pct") if ka else None,
"A_unknown_refusal": dig(a, ka, "summary", "unknown_refusal_pct") if ka else None,
"A_known_false_refusal": dig(a, ka, "summary", "wrong_abstention_pct") if ka else None,
"B_accuracy": dig(b, kb, "summary", "accuracy_pct") if kb else None,
"B_wrong_attribute": dig(b, kb, "summary", "wrong_attribute_pct") if kb else None,
"B_read": dig(b, kb, "summary", "read_pct") if kb else None,
"B_records": dig(b, kb, "summary", "mean_records_selected") if kb else None,
"C_answerable": dig(c, "results", c_key, "accuracy_pct"),
"C_wrong_attribute": dig(c, "results", c_key, "wrong_attribute_pct"),
"C_unknown_leak": dig(c, "results", c_key, "unknown_leak_pct"),
"C_active_min": dig(c, "results", c_key, "records_active_min"),
"C_active_max": dig(c, "results", c_key, "records_active_max"),
"D_recalled": dig(d, "router_recalled_after_restart"),
"D_answer_correct": dig(d, "generated_contains_expected"),
"D_cleanup": dig(d, "cleanup_applied"),
}
ROW_LABELS = [
("A_cases", "A 用例数", "int"),
("A_overall", "A 总体正确率", "pct"),
("A_answerable", "A 可回答正确率", "pct"),
("A_unknown_refusal", "A 未知拒答率", "pct"),
("A_known_false_refusal", "A 已知问题被误拒率", "pct"),
("B_accuracy", "B 回答正确率", "pct"),
("B_wrong_attribute", "B 答成别的属性", "pct"),
("B_read", "B 触发读取", "pct"),
("B_records", "B 平均选中记录", "num"),
("C_answerable", "C 可回答正确率", "pct"),
("C_wrong_attribute", "C 答成别的属性", "pct"),
("C_unknown_leak", "C 未知泄漏率", "pct"),
("C_active_min", "C 活跃记录下限", "int"),
("C_active_max", "C 活跃记录上限", "int"),
("D_recalled", "D 重启后召回", "bool"),
("D_answer_correct", "D 重启后作答正确", "bool"),
("D_cleanup", "D 清理生效", "bool"),
]
def render(value, unit: str) -> str:
if value is None:
return "-"
if unit == "pct":
return f"{float(value):.2f}%"
if unit == "int":
return f"{int(value):,}"
if unit == "num":
return f"{float(value):.4f}"
return "通过" if value else "**未通过**"
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--tag", action="append", required=True, help="LABEL=TAG")
parser.add_argument("--output", default="nm2_battery_comparison.json")
parser.add_argument("--markdown", default="nm2_battery_comparison.md")
args = parser.parse_args()
pairs = []
for item in args.tag:
label, _, tag = item.partition("=")
pairs.append((label, tag, row_for(tag)))
lines = ["# NM2.1 与原版 NM2 的整体记忆测试对照", "",
"同一套电池、同一份运行时，只有模型包不同。全部为百分比。", "",
"| 指标 | " + " | ".join(label for label, _, _ in pairs) + " |",
"|---" * (len(pairs) + 1) + "|"]
for key, nice, unit in ROW_LABELS:
cells = [render(body.get(key), unit) for _, _, body in pairs]
lines.append(f"| {nice} | " + " | ".join(cells) + " |")
text = "\n".join(lines) + "\n"
Path(args.markdown).write_text(text, encoding="utf-8")
Path(args.output).write_text(
json.dumps({label: body for label, _, body in pairs}, ensure_ascii=False, indent=2),
encoding="utf-8")
print(text)
print(f"wrote {args.markdown} and {args.output}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
compare_record_rankers.py
+182
View File
@@ -0,0 +1,182 @@
"""Compare the rankers that can order a record set, on the zero-overlap eval set.
Why this exists
---------------
``memory_os_v2.py`` gives a record's *learned* score to ``record_scorer`` and discards
whatever ``MemoryRouterV2.projected_scores`` produced for it. The deployed adapter
ships no ``text_retriever.pt``, so ``_text_retriever_ready`` is False and that callback
falls back to plain cosine similarity between the query key and the record key -- both
produced by the *frozen* backbone. Measured consequence: swapping or even randomising
the router changes end-to-end answers not at all.
This script measures, on identical frozen features, which of those candidate rankers can
actually put the right fact first:
* ``cosine`` -- ``F.cosine_similarity(query_key, candidate_key)``, i.e. what the runtime
uses today for fact records;
* each router checkpoint, scored through ``projected_scores`` exactly as the runtime and
the training protocol do.
Usage::
python -m V2_dpskw.compare_record_rankers ^
--eval-file data/zero_overlap/eval.jsonl ^
--feature-cache H:\\Memory\\nm_cache\\nm_zero_overlap\\feature_cache ^
--run "REPLAY-128=checkpoints/router_replay_v7_v2_128/memory_router_v2.pt" ^
--run "V2-128-v6=checkpoints/router_v6_v2_128/memory_router_v2.pt"
"""
from __future__ import annotations
import argparse
import hashlib
import json
from pathlib import Path
import numpy as np
import torch
import torch.nn.functional as F
from .eval_router_scorecard import load_router_any
def text_key(text: str) -> str:
return hashlib.sha1(str(text).encode("utf-8", "replace")).hexdigest()
def stream_episodes(path: Path):
with path.open("r", encoding="utf-8") as handle:
for line in handle:
line = line.strip()
if line:
yield json.loads(line)
def episode_tensors(row: dict, lookup: dict[str, int], bank: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
query = row["query"]
candidates = [candidate["text"] for candidate in row["candidates"]]
missing = [text for text in [query, *candidates] if text_key(text) not in lookup]
if missing:
raise SystemExit(f"{row['id']}: {len(missing)} text(s) absent from the bank")
query_vector = bank[lookup[text_key(query)]]
candidate_vectors = bank[[lookup[text_key(text)] for text in candidates]]
return query_vector, candidate_vectors
def rank_metrics(scores: torch.Tensor, positive: int) -> tuple[int, bool, bool]:
order = torch.argsort(scores, descending=True).tolist()
rank = order.index(positive) + 1
return rank, rank == 1, rank <= 3
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--eval-file", default="data/zero_overlap/eval.jsonl")
parser.add_argument("--feature-cache", default=r"H:\Memory\nm_cache\nm_zero_overlap\feature_cache")
parser.add_argument("--run", action="append", required=True, help="LABEL=CHECKPOINT_PATH")
parser.add_argument("--with-text-retriever", action="store_true",
help="also score with the packaged text_retriever (costs one model load)")
parser.add_argument("--package", default="qwen3_5_4b_natural_memory_v2")
parser.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
parser.add_argument("--output", default="zero_overlap_ranker_comparison.json")
args = parser.parse_args()
cache = Path(args.feature_cache)
bank = np.load(cache / "features.f16.npy", mmap_mode="r")
lookup = json.loads((cache / "index.json").read_text(encoding="utf-8"))
device = torch.device(args.device)
routers = []
for spec in args.run:
label, _, path_value = spec.partition("=")
router, arch_config, info = load_router_any(Path(path_value))
router = router.to(device).eval()
routers.append((label, router))
print(json.dumps({"loaded": label, "path": path_value, "arch": str(arch_config),
"parameters": sum(p.numel() for p in router.parameters())}), flush=True)
stats = {label: {"top1": 0, "recall3": 0, "mrr": 0.0} for label, _ in routers}
stats["cosine"] = {"top1": 0, "recall3": 0, "mrr": 0.0}
# The packaged text_retriever is the module that actually orders fact records at
# runtime (memory_os_v2._record_scores overwrites the router score for records that
# carry a semantic_key). The feature bank already stores exactly the 2560-dim model
# keys it consumes, so it can be scored on identical inputs with no re-encoding.
retriever = None
if args.with_text_retriever:
from .qwen_integration import load_qwen_dynamic
model = load_qwen_dynamic(args.package)
model.eval()
if model.text_retriever is None or not getattr(model, "_text_retriever_ready", False):
raise SystemExit("packaged text_retriever is not ready")
retriever = model.text_retriever.to(device).eval()
stats["text_retriever"] = {"top1": 0, "recall3": 0, "mrr": 0.0}
print(json.dumps({"loaded": "text_retriever",
"parameters": sum(p.numel() for p in retriever.parameters())}), flush=True)
episodes = answerable = 0
for row in stream_episodes(Path(args.eval_file)):
positives = row.get("positive_indices") or []
if not positives:
continue
positive = int(positives[0])
query_vector, candidate_vectors = episode_tensors(row, lookup, bank)
query = torch.from_numpy(np.ascontiguousarray(query_vector)).to(device).float().reshape(1, -1)
candidates = torch.from_numpy(
np.ascontiguousarray(candidate_vectors)).to(device).float().reshape(1, -1, bank.shape[1])
episodes += 1
answerable += 1
# 1) frozen-key cosine, the runtime's current ranker for fact records
cosine_scores = F.cosine_similarity(query, candidates.reshape(-1, candidates.shape[-1]), dim=-1)
rank, top1, in3 = rank_metrics(cosine_scores, positive)
stats["cosine"]["top1"] += int(top1)
stats["cosine"]["recall3"] += int(in3)
stats["cosine"]["mrr"] += 1.0 / rank
if retriever is not None:
with torch.no_grad():
retriever_scores = torch.sigmoid(
retriever(query, candidates.reshape(-1, candidates.shape[-1]))
).reshape(-1)
rank, top1, in3 = rank_metrics(retriever_scores, positive)
stats["text_retriever"]["top1"] += int(top1)
stats["text_retriever"]["recall3"] += int(in3)
stats["text_retriever"]["mrr"] += 1.0 / rank
for label, router in routers:
with torch.no_grad():
# Same call shape the runtime's _score_candidates uses: candidates are
# projected into the router's compact address space, then pair-scored.
projected = router.encode_key(candidates.reshape(-1, candidates.shape[-1]))
scores, _ = router.projected_scores(query, projected.reshape(1, -1, projected.shape[-1]))
scores = scores.reshape(-1)
rank, top1, in3 = rank_metrics(scores, positive)
stats[label]["top1"] += int(top1)
stats[label]["recall3"] += int(in3)
stats[label]["mrr"] += 1.0 / rank
report = {
"eval_file": args.eval_file,
"episodes": episodes,
"answerable_episodes": answerable,
"chance_top1_pct": round(100.0 / 24, 2),
"rankers": {
label: {
"top1_pct": round(100.0 * body["top1"] / max(1, episodes), 2),
"recall3_pct": round(100.0 * body["recall3"] / max(1, episodes), 2),
"mrr_pct": round(100.0 * body["mrr"] / max(1, episodes), 2),
}
for label, body in stats.items()
},
}
text = json.dumps(report, ensure_ascii=False, indent=2)
Path(args.output).write_text(text, encoding="utf-8")
print(text, flush=True)
return 0
if __name__ == "__main__":
raise SystemExit(main())
compare_router_runs.py
+151
View File
@@ -0,0 +1,151 @@
"""Compare the new MemoryRouterXL runs against the 512-dim MemoryRouterV2 baseline.
All runs share the same frozen ``router_training_v3`` eval set (SHA-256 checked
by the trainers), the same selection score and the same metric implementation,
so the only intended difference is router capacity.
Usage (from the fork root)::
python -m V2_dpskw.compare_router_runs
python -m V2_dpskw.compare_router_runs --markdown router_xl_comparison.md
"""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
from typing import Any
if __package__ in {None, ""}:
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
PROJECT_ROOT = Path(__file__).resolve().parent
#: Baseline produced by the original (GPT-era) trainer in the upstream project.
DEFAULT_BASELINE = Path(
r"H:\Memory\dynamic_memory_lab\checkpoints\natural_memory_v2_router_512\memory_router_large_training.json"
)
METRIC_KEYS = (
("route_top1", "Top-1"),
("route_recall_at3", "Recall@3"),
("route_mrr", "MRR"),
("need_f1", "need F1"),
("need_specificity", "specificity"),
("abstention_accuracy", "abstention"),
("hop_accuracy", "hop acc"),
)
def _load(path: Path) -> dict[str, Any] | None:
if not path.exists():
return None
try:
return json.loads(path.read_text(encoding="utf-8"))
except Exception as exc: # pragma: no cover - diagnostic path
print(f"warning: cannot read {path}: {exc}", file=sys.stderr)
return None
def _row(label: str, params: int | str, summary: dict[str, Any] | None, *, source: str) -> dict[str, Any]:
if summary is None:
return {"label": label, "parameters": params, "source": source, "status": "missing"}
best = summary.get("best_eval") or summary.get("final_eval") or {}
final = summary.get("final_eval") or {}
row: dict[str, Any] = {
"label": label,
"parameters": params,
"source": source,
"status": "ok",
"router_dim": summary.get("router_dim"),
"num_heads": summary.get("num_heads"),
"steps": summary.get("steps"),
"eval_episodes": summary.get("eval_episodes"),
"eval_sha256": summary.get("eval_sha256"),
"best_step": summary.get("best_step"),
"best_selection_score": summary.get("best_selection_score"),
"best": {key: best.get(key) for key, _ in METRIC_KEYS},
"final": {key: final.get(key) for key, _ in METRIC_KEYS},
}
if "arch_config" in summary:
row["arch"] = summary["arch_config"]
if isinstance(summary.get("parameters"), dict):
row["parameters"] = summary["parameters"].get("total", params)
return row
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--baseline", default=str(DEFAULT_BASELINE))
parser.add_argument(
"--run",
action="append",
default=None,
help="training summary json; repeatable. Defaults to the XL runs in checkpoints/.",
)
parser.add_argument("--output", default="router_xl_comparison.json")
parser.add_argument("--markdown", default="")
args = parser.parse_args()
run_paths = args.run or [
str(PROJECT_ROOT / "checkpoints/router_xl_1024/router_xl_training.json"),
str(PROJECT_ROOT / "checkpoints/router_xl_2048/router_xl_training.json"),
]
baseline_path = Path(args.baseline)
baseline_summary = _load(baseline_path)
rows: list[dict[str, Any]] = [
_row(
"MemoryRouterV2 512 (baseline)",
4741902,
baseline_summary,
source=str(baseline_path),
)
]
for run in run_paths:
path = Path(run)
summary = _load(path)
label = (summary or {}).get("label") or path.parent.name
rows.append(_row(label, "?", summary, source=str(path)))
report = {"baseline": str(baseline_path), "runs": run_paths, "rows": rows}
Path(args.output).write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
header = ["router", "params", "dim", "steps", "best step", "score"] + [name for _, name in METRIC_KEYS]
lines = ["| " + " | ".join(header) + " |", "|" + "---|" * len(header)]
for row in rows:
if row["status"] != "ok":
lines.append("| " + " | ".join([row["label"], str(row["parameters"]), "-", "-", "-", "-"] + ["-"] * len(METRIC_KEYS)) + " |")
continue
best = row["best"]
lines.append(
"| "
+ " | ".join(
[
row["label"],
f"{row['parameters']:,}" if isinstance(row["parameters"], int) else str(row["parameters"]),
str(row.get("router_dim", "-")),
str(row.get("steps", "-")),
str(row.get("best_step", "-")),
f"{row['best_selection_score']:.4f}" if isinstance(row.get("best_selection_score"), (int, float)) else "-",
]
+ [
f"{best[key]:.4f}" if isinstance(best.get(key), (int, float)) else "-"
for key, _ in METRIC_KEYS
]
)
+ " |"
)
table = "\n".join(lines)
print(table, flush=True)
if args.markdown:
Path(args.markdown).write_text(table + "\n", encoding="utf-8")
print(f"wrote {args.markdown}", flush=True)
print(f"wrote {args.output}", flush=True)
return 0
if __name__ == "__main__":
raise SystemExit(main())
comprehensive_benchmark.py
+301
View File
@@ -0,0 +1,301 @@
"""Compare original Qwen and Native Memory on general regression tasks."""
from __future__ import annotations
import argparse
import gc
import json
import re
import sys
import time
from collections import defaultdict
from pathlib import Path
from typing import Any
import torch
from .qwen_integration import load_memory_config, load_qwen_base, load_qwen_dynamic, load_tokenizer
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--model-path", default=".")
parser.add_argument("--adapter", default="V2_dpskw/qwen_memory_adapter_native_v3")
parser.add_argument("--data", default="V2_dpskw/data/comprehensive_general.jsonl")
parser.add_argument("--output", default="V2_dpskw/comprehensive_benchmark_native_v3.json")
parser.add_argument("--max-new-tokens", type=int, default=32)
parser.add_argument("--perf-repeats", type=int, default=3)
parser.add_argument("--no-4bit", action="store_true")
parser.add_argument("--category-drop-limit", type=float, default=0.10)
parser.add_argument("--overall-drop-limit", type=float, default=0.05)
return parser.parse_args()
def load_cases(path: str | Path) -> list[dict[str, Any]]:
cases = []
for line in Path(path).read_text(encoding="utf-8").splitlines():
if line.strip():
cases.append(json.loads(line))
if not cases:
raise ValueError(f"no benchmark cases found in {path}")
return cases
def normalize(text: str) -> str:
return re.sub(r"[\s`*_#，。！？、；：,.!?;:'\"（）()\[\]{}]", "", text).lower()
def contains_answer(text: str, acceptable: list[str]) -> bool:
normalized = normalize(text)
for answer in acceptable:
expected = normalize(str(answer))
if not expected:
continue
if expected.isdigit() and len(expected) == 1:
if re.search(rf"(?<!\d){re.escape(expected)}(?!\d)", normalized):
return True
elif expected in normalized:
return True
return False
def prompt_inputs(tokenizer: Any, prompt: str, device: torch.device) -> dict[str, torch.Tensor]:
encoded = tokenizer.apply_chat_template(
[{"role": "user", "content": prompt}],
tokenize=True,
add_generation_prompt=True,
enable_thinking=False,
)
input_ids = encoded["input_ids"] if isinstance(encoded, dict) or hasattr(encoded, "__getitem__") else encoded
if isinstance(input_ids, torch.Tensor):
if input_ids.ndim == 1:
input_ids = input_ids.unsqueeze(0)
input_ids = input_ids.to(device)
else:
if input_ids and isinstance(input_ids[0], list):
input_ids = input_ids[0]
input_ids = torch.tensor([input_ids], dtype=torch.long, device=device)
return {"input_ids": input_ids, "attention_mask": torch.ones_like(input_ids)}
def _decode_generation(tokenizer: Any, output: torch.Tensor, prompt: dict[str, torch.Tensor]) -> str:
start = prompt["input_ids"].shape[1]
return tokenizer.decode(output[0, start:].detach().cpu().tolist(), skip_special_tokens=True).strip()
def evaluate_model(
model: Any,
tokenizer: Any,
cases: list[dict[str, Any]],
*,
adapted: bool,
max_new_tokens: int,
perf_repeats: int,
) -> dict[str, Any]:
device = model._find_layer_device() if adapted else model.get_input_embeddings().weight.device
rows: list[dict[str, Any]] = []
category_values: dict[str, list[float]] = defaultdict(list)
category_examples: dict[str, list[dict[str, Any]]] = defaultdict(list)
started = time.perf_counter()
for case in cases:
if adapted:
model.reset_memory()
prompt = prompt_inputs(tokenizer, str(case["prompt"]), device)
with torch.inference_mode():
if adapted:
output = model.generate(
**prompt,
max_new_tokens=max_new_tokens,
do_sample=False,
update_memory=False,
use_cache=True,
pad_token_id=tokenizer.pad_token_id,
)
else:
output = model.generate(
**prompt,
max_new_tokens=max_new_tokens,
do_sample=False,
use_cache=True,
pad_token_id=tokenizer.pad_token_id,
)
generated = _decode_generation(tokenizer, output, prompt)
passed = contains_answer(generated, list(case["acceptable"]))
category = str(case["category"])
category_values[category].append(float(passed))
if len(category_examples[category]) < 3:
category_examples[category].append(
{
"id": case["id"],
"prompt": case["prompt"],
"acceptable": case["acceptable"],
"generated": generated,
"passed": passed,
}
)
rows.append(
{
"id": case["id"],
"category": category,
"acceptable": case["acceptable"],
"generated": generated,
"passed": passed,
}
)
elapsed = time.perf_counter() - started
perf_case = cases[0]
if adapted:
model.reset_memory()
perf_prompt = prompt_inputs(tokenizer, str(perf_case["prompt"]), device)
if device.type == "cuda":
torch.cuda.synchronize(device)
perf_start = time.perf_counter()
generated_tokens = 0
for _ in range(max(1, perf_repeats)):
if adapted:
model.reset_memory()
with torch.inference_mode():
if adapted:
perf_output = model.generate(
**perf_prompt,
max_new_tokens=max_new_tokens,
do_sample=False,
update_memory=False,
use_cache=True,
pad_token_id=tokenizer.pad_token_id,
)
else:
perf_output = model.generate(
**perf_prompt,
max_new_tokens=max_new_tokens,
do_sample=False,
use_cache=True,
pad_token_id=tokenizer.pad_token_id,
)
generated_tokens += int(perf_output.shape[1] - perf_prompt["input_ids"].shape[1])
if device.type == "cuda":
torch.cuda.synchronize(device)
perf_elapsed = time.perf_counter() - perf_start
score = sum(sum(values) for values in category_values.values()) / len(cases)
return {
"device": str(device),
"cases": len(cases),
"elapsed_seconds": elapsed,
"overall_score": score,
"categories": {
category: {
"count": len(values),
"score": sum(values) / len(values),
"examples": category_examples[category],
}
for category, values in sorted(category_values.items())
},
"performance": {
"repeats": max(1, perf_repeats),
"max_new_tokens": max_new_tokens,
"tokens_per_second": generated_tokens / max(perf_elapsed, 1e-9),
"seconds_per_run": perf_elapsed / max(1, perf_repeats),
},
"rows": rows,
}
def release(model: Any) -> None:
del model
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
def main() -> None:
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
args = parse_args()
cases = load_cases(args.data)
tokenizer = load_tokenizer(args.model_path)
use_4bit = not args.no_4bit
report: dict[str, Any] = {
"model_path": str(Path(args.model_path).resolve()),
"adapter": str(Path(args.adapter).resolve()),
"data": str(Path(args.data).resolve()),
"quantization": "4bit_nf4" if use_4bit else "none",
"cases": len(cases),
}
print("loading baseline")
baseline = load_qwen_base(args.model_path, load_in_4bit=use_4bit)
baseline.eval()
baseline_device = baseline.get_input_embeddings().weight.device
if baseline_device.type == "cuda":
torch.cuda.reset_peak_memory_stats(baseline_device)
report["baseline"] = evaluate_model(
baseline,
tokenizer,
cases,
adapted=False,
max_new_tokens=args.max_new_tokens,
perf_repeats=args.perf_repeats,
)
if baseline_device.type == "cuda":
report["baseline"]["peak_memory_gb"] = torch.cuda.max_memory_allocated(baseline_device) / 1024**3
release(baseline)
config = load_memory_config(args.adapter)
config.persistent_memory = False
print(f"loading native adapter mode={config.mode} layers={config.layer_indices}")
adapted = load_qwen_dynamic(
args.model_path,
memory_config=config,
load_in_4bit=use_4bit,
)
adapted.load_memory_adapter(args.adapter)
adapted.eval()
adapted_device = adapted._find_layer_device()
if adapted_device.type == "cuda":
torch.cuda.reset_peak_memory_stats(adapted_device)
report["native_memory"] = evaluate_model(
adapted,
tokenizer,
cases,
adapted=True,
max_new_tokens=args.max_new_tokens,
perf_repeats=args.perf_repeats,
)
if adapted_device.type == "cuda":
report["native_memory"]["peak_memory_gb"] = torch.cuda.max_memory_allocated(adapted_device) / 1024**3
baseline_score = report["baseline"]["overall_score"]
native_score = report["native_memory"]["overall_score"]
baseline_categories = report["baseline"]["categories"]
native_categories = report["native_memory"]["categories"]
category_deltas = {
category: native_categories[category]["score"] - baseline_categories[category]["score"]
for category in baseline_categories.keys() & native_categories.keys()
}
report["regression"] = {
"overall_delta": native_score - baseline_score,
"category_deltas": category_deltas,
"overall_drop_limit": args.overall_drop_limit,
"category_drop_limit": args.category_drop_limit,
"overall_regression_alert": native_score < baseline_score - args.overall_drop_limit,
"category_regression_alerts": {
category: delta < -args.category_drop_limit for category, delta in category_deltas.items()
},
"pass": native_score >= baseline_score - args.overall_drop_limit
and all(delta >= -args.category_drop_limit for delta in category_deltas.values()),
}
legacy_path = Path(args.adapter).parent / "benchmark_qwen_native_v3_eval.json"
if legacy_path.exists():
report["same_memory_benchmark"] = json.loads(legacy_path.read_text(encoding="utf-8"))
native_eval_path = Path(args.adapter) / "native_eval_report.json"
if native_eval_path.exists():
report["native_memory_holdout"] = json.loads(native_eval_path.read_text(encoding="utf-8"))
output = Path(args.output)
output.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps({key: report[key] for key in ("baseline", "native_memory", "regression")}, ensure_ascii=False, indent=2))
print(f"saved={output}")
if __name__ == "__main__":
main()
coverage_gate_diagnosis.json
+46
View File
@@ -0,0 +1,46 @@
{
"package": "qwen3_5_4b_natural_memory_v2_1",
"gate_enabled": true,
"blend": 0.5,
"facts_written": 10,
"bank_records": 10,
"active_records": 10,
"active_conflict_keys": 10,
"bank_attributes": [
"主管姓名",
"出生城市",
"办公城市",
"团队名称",
"工位楼层",
"常住城市",
"常用编辑器",
"档案标识",
"通勤方式",
"默认语言"
],
"head_vocabulary_size": 24,
"attributes_outside_head_vocabulary": [],
"attributes_missing_from_bank": [
"入职年份",
"午餐偏好",
"咖啡口味",
"客户名称",
"宿舍楼号",
"手机尾号",
"紧急联系人姓氏",
"设备型号",
"课程名称",
"起床时间",
"运动习惯",
"邮箱域名",
"阅读工具",
"项目代号"
],
"gate_counters": {
"applicable": 0,
"bypassed": 1
},
"query": "我平时待得最久的地方是哪里？",
"stop_reason": "evidence_found",
"reply": "您的常住城市是 CITY-A1B2C3，办公城市是 CITY-G7H8J9，通勤"
}
coverage_gate_diagnosis2.json
+46
View File
@@ -0,0 +1,46 @@
{
"package": "qwen3_5_4b_natural_memory_v2_1",
"gate_enabled": true,
"blend": 0.5,
"facts_written": 10,
"bank_records": 10,
"active_records": 10,
"active_conflict_keys": 10,
"bank_attributes": [
"主管姓名",
"出生城市",
"办公城市",
"团队名称",
"工位楼层",
"常住城市",
"常用编辑器",
"档案标识",
"通勤方式",
"默认语言"
],
"head_vocabulary_size": 24,
"attributes_outside_head_vocabulary": [],
"attributes_missing_from_bank": [
"入职年份",
"午餐偏好",
"咖啡口味",
"客户名称",
"宿舍楼号",
"手机尾号",
"紧急联系人姓氏",
"设备型号",
"课程名称",
"起床时间",
"运动习惯",
"邮箱域名",
"阅读工具",
"项目代号"
],
"gate_counters": {
"applicable": 1,
"bypassed": 0
},
"query": "我平时待得最久的地方是哪里？",
"stop_reason": "evidence_found",
"reply": "您的常住城市是 CITY-A1B2C3，办公城市是 CITY-G7H8J9，通勤"
}
data/_ab_mega20_V2_dpskw/eval.jsonl
+84
View File
File diff suppressed because one or more lines are too long
data/_ab_mega20_V2_dpskw/eval.sha256
+1
View File
@@ -0,0 +1 @@
169b026df2a85c3530f85c0ab0ef1c0769c51b4cc3c8e66d0784833a6bb53177
data/_ab_mega20_V2_dpskw/manifest.json
+150
View File
@@ -0,0 +1,150 @@
{
"format_version": 3,
"generator": "prepare_memory_router_dataset.py",
"seed": 20260909,
"candidate_count": 32,
"schema": {
"query": "natural-language routing query",
"candidates": "bounded memory records with text and provenance",
"positive_indices": "one or more supporting memory records; empty means abstain",
"need_memory": "1 if evidence is required and present, 0 for unknown/unsupported queries",
"hop": "0 for abstention, otherwise number of supporting records clipped to router max_hops",
"group_id": "conversation or QA episode identity; no group may cross train/eval"
},
"files": {
"train": {
"path": "H:\\Memory\\V2_dpskw\\data\\_ab_mega20_V2_dpskw\\train.jsonl",
"episodes": 148,
"sha256": "d9c2774955df7066fe6751faa86ae6137675f72b607bda892d0e20aeadab2f74"
},
"eval": {
"path": "H:\\Memory\\V2_dpskw\\data\\_ab_mega20_V2_dpskw\\eval.jsonl",
"episodes": 84,
"sha256": "169b026df2a85c3530f85c0ab0ef1c0769c51b4cc3c8e66d0784833a6bb53177"
}
},
"counts": {
"train_groups": 80128,
"eval_groups": 20064,
"train_episodes": 148,
"eval_episodes": 84,
"train_unknown": 128,
"eval_unknown": 64,
"train_candidates": 4736,
"eval_candidates": 2688
},
"local_sources": [
{
"source": "H:/Memory/dynamic_memory_lab/data/benchmark_train.jsonl",
"rows": 128,
"kind": "benchmark",
"split": "train"
},
{
"source": "H:/Memory/dynamic_memory_lab/data/mega_validation/memory_validation_80k_train.jsonl",
"rows": 80000,
"kind": "mega_validation",
"split": "train"
},
{
"source": "H:/Memory/dynamic_memory_lab/data/benchmark_eval.jsonl",
"rows": 64,
"kind": "benchmark",
"split": "eval"
},
{
"source": "H:/Memory/dynamic_memory_lab/data/mega_validation/memory_validation_20k_eval.jsonl",
"rows": 20000,
"kind": "mega_validation",
"split": "eval"
}
],
"public_sources": [],
"public_catalog": [
{
"name": "hotpotqa_train",
"dataset_id": "hotpot_qa",
"config": "distractor",
"split": "train",
"split_kind": "train",
"task": "qa",
"url": "https://huggingface.co/datasets/hotpot_qa"
},
{
"name": "hotpotqa_validation",
"dataset_id": "hotpot_qa",
"config": "distractor",
"split": "validation",
"split_kind": "eval",
"task": "qa",
"url": "https://huggingface.co/datasets/hotpot_qa"
},
{
"name": "codesearchnet_python_train",
"dataset_id": "code_search_net",
"config": "python",
"split": "train",
"split_kind": "train",
"task": "code",
"url": "https://huggingface.co/datasets/code_search_net"
},
{
"name": "codesearchnet_python_validation",
"dataset_id": "code_search_net",
"config": "python",
"split": "validation",
"split_kind": "eval",
"task": "code",
"url": "https://huggingface.co/datasets/code_search_net"
},
{
"name": "fever_train",
"dataset_id": "fever",
"config": "v1.0",
"split": "train",
"split_kind": "train",
"task": "evidence",
"url": "https://huggingface.co/datasets/fever"
},
{
"name": "fever_validation",
"dataset_id": "fever",
"config": "v1.0",
"split": "labelled_dev",
"split_kind": "eval",
"task": "evidence",
"url": "https://huggingface.co/datasets/fever"
}
],
"split_stats": {
"train": {
"kept": 148,
"unknown": 128,
"family:benchmark_qa": 128,
"family:mega_validation": 20,
"written": 148
},
"eval": {
"kept": 84,
"unknown": 64,
"family:benchmark_qa": 64,
"family:mega_validation": 20,
"written": 84
}
},
"leakage_check": {
"group_overlap": 0,
"passed": true
},
"conflict_policy": {
"enabled": true,
"answerable_positive_conflicts_excluded": true,
"unknown_hard_conflicts_retained": true
},
"evaluation_policy": "eval.jsonl is generated and hashed before router training; the trainer refuses an optional hash mismatch.",
"warnings": [
"Local generated memory-policy files are useful hard negatives but are not public-human chat data.",
"Add redacted real user traces only after consent and PII removal.",
"Public download failures are recorded; failed sources contribute zero rows."
]
}
data/_ab_mega20_V2_dpskw/train.jsonl
+148
View File
File diff suppressed because one or more lines are too long
data/_ab_mega20_dynamic_memory_lab/eval.jsonl
+84
View File
File diff suppressed because one or more lines are too long
data/_ab_mega20_dynamic_memory_lab/eval.sha256
+1
View File
@@ -0,0 +1 @@
169b026df2a85c3530f85c0ab0ef1c0769c51b4cc3c8e66d0784833a6bb53177
data/_ab_mega20_dynamic_memory_lab/manifest.json
+150
View File
@@ -0,0 +1,150 @@
{
"format_version": 3,
"generator": "prepare_memory_router_dataset.py",
"seed": 20260909,
"candidate_count": 32,
"schema": {
"query": "natural-language routing query",
"candidates": "bounded memory records with text and provenance",
"positive_indices": "one or more supporting memory records; empty means abstain",
"need_memory": "1 if evidence is required and present, 0 for unknown/unsupported queries",
"hop": "0 for abstention, otherwise number of supporting records clipped to router max_hops",
"group_id": "conversation or QA episode identity; no group may cross train/eval"
},
"files": {
"train": {
"path": "H:\\Memory\\V2_dpskw\\data\\_ab_mega20_dynamic_memory_lab\\train.jsonl",
"episodes": 148,
"sha256": "d9c2774955df7066fe6751faa86ae6137675f72b607bda892d0e20aeadab2f74"
},
"eval": {
"path": "H:\\Memory\\V2_dpskw\\data\\_ab_mega20_dynamic_memory_lab\\eval.jsonl",
"episodes": 84,
"sha256": "169b026df2a85c3530f85c0ab0ef1c0769c51b4cc3c8e66d0784833a6bb53177"
}
},
"counts": {
"train_groups": 80128,
"eval_groups": 20064,
"train_episodes": 148,
"eval_episodes": 84,
"train_unknown": 128,
"eval_unknown": 64,
"train_candidates": 4736,
"eval_candidates": 2688
},
"local_sources": [
{
"source": "H:/Memory/dynamic_memory_lab/data/benchmark_train.jsonl",
"rows": 128,
"kind": "benchmark",
"split": "train"
},
{
"source": "H:/Memory/dynamic_memory_lab/data/mega_validation/memory_validation_80k_train.jsonl",
"rows": 80000,
"kind": "mega_validation",
"split": "train"
},
{
"source": "H:/Memory/dynamic_memory_lab/data/benchmark_eval.jsonl",
"rows": 64,
"kind": "benchmark",
"split": "eval"
},
{
"source": "H:/Memory/dynamic_memory_lab/data/mega_validation/memory_validation_20k_eval.jsonl",
"rows": 20000,
"kind": "mega_validation",
"split": "eval"
}
],
"public_sources": [],
"public_catalog": [
{
"name": "hotpotqa_train",
"dataset_id": "hotpot_qa",
"config": "distractor",
"split": "train",
"split_kind": "train",
"task": "qa",
"url": "https://huggingface.co/datasets/hotpot_qa"
},
{
"name": "hotpotqa_validation",
"dataset_id": "hotpot_qa",
"config": "distractor",
"split": "validation",
"split_kind": "eval",
"task": "qa",
"url": "https://huggingface.co/datasets/hotpot_qa"
},
{
"name": "codesearchnet_python_train",
"dataset_id": "code_search_net",
"config": "python",
"split": "train",
"split_kind": "train",
"task": "code",
"url": "https://huggingface.co/datasets/code_search_net"
},
{
"name": "codesearchnet_python_validation",
"dataset_id": "code_search_net",
"config": "python",
"split": "validation",
"split_kind": "eval",
"task": "code",
"url": "https://huggingface.co/datasets/code_search_net"
},
{
"name": "fever_train",
"dataset_id": "fever",
"config": "v1.0",
"split": "train",
"split_kind": "train",
"task": "evidence",
"url": "https://huggingface.co/datasets/fever"
},
{
"name": "fever_validation",
"dataset_id": "fever",
"config": "v1.0",
"split": "labelled_dev",
"split_kind": "eval",
"task": "evidence",
"url": "https://huggingface.co/datasets/fever"
}
],
"split_stats": {
"train": {
"kept": 148,
"unknown": 128,
"family:benchmark_qa": 128,
"family:mega_validation": 20,
"written": 148
},
"eval": {
"kept": 84,
"unknown": 64,
"family:benchmark_qa": 64,
"family:mega_validation": 20,
"written": 84
}
},
"leakage_check": {
"group_overlap": 0,
"passed": true
},
"conflict_policy": {
"enabled": true,
"answerable_positive_conflicts_excluded": true,
"unknown_hard_conflicts_retained": true
},
"evaluation_policy": "eval.jsonl is generated and hashed before router training; the trainer refuses an optional hash mismatch.",
"warnings": [
"Local generated memory-policy files are useful hard negatives but are not public-human chat data.",
"Add redacted real user traces only after consent and PII removal.",
"Public download failures are recorded; failed sources contribute zero rows."
]
}
data/_ab_mega20_dynamic_memory_lab/train.jsonl
+148
View File
File diff suppressed because one or more lines are too long
data/_ab_mega_V2_dpskw/eval.jsonl
View File
data/_ab_mega_V2_dpskw/eval.sha256
+1
View File
@@ -0,0 +1 @@
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
data/_ab_mega_V2_dpskw/manifest.json
+140
View File
@@ -0,0 +1,140 @@
{
"format_version": 3,
"generator": "prepare_memory_router_dataset.py",
"seed": 20260909,
"candidate_count": 32,
"schema": {
"query": "natural-language routing query",
"candidates": "bounded memory records with text and provenance",
"positive_indices": "one or more supporting memory records; empty means abstain",
"need_memory": "1 if evidence is required and present, 0 for unknown/unsupported queries",
"hop": "0 for abstention, otherwise number of supporting records clipped to router max_hops",
"group_id": "conversation or QA episode identity; no group may cross train/eval"
},
"files": {
"train": {
"path": "H:\\Memory\\V2_dpskw\\data\\_ab_mega_V2_dpskw\\train.jsonl",
"episodes": 60,
"sha256": "f47703450f8ab3dece5d8d0094facb745989a81b1cc4d3cd6bcc6dd450c64138"
},
"eval": {
"path": "H:\\Memory\\V2_dpskw\\data\\_ab_mega_V2_dpskw\\eval.jsonl",
"episodes": 0,
"sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
},
"counts": {
"train_groups": 80128,
"eval_groups": 64,
"train_episodes": 60,
"eval_episodes": 0,
"train_unknown": 60,
"eval_unknown": 0,
"train_candidates": 1920,
"eval_candidates": 0
},
"local_sources": [
{
"source": "H:/Memory/dynamic_memory_lab/data/benchmark_train.jsonl",
"rows": 128,
"kind": "benchmark",
"split": "train"
},
{
"source": "H:/Memory/dynamic_memory_lab/data/mega_validation/memory_validation_80k_train.jsonl",
"rows": 80000,
"kind": "mega_validation",
"split": "train"
},
{
"source": "H:/Memory/dynamic_memory_lab/data/benchmark_eval.jsonl",
"rows": 64,
"kind": "benchmark",
"split": "eval"
}
],
"public_sources": [],
"public_catalog": [
{
"name": "hotpotqa_train",
"dataset_id": "hotpot_qa",
"config": "distractor",
"split": "train",
"split_kind": "train",
"task": "qa",
"url": "https://huggingface.co/datasets/hotpot_qa"
},
{
"name": "hotpotqa_validation",
"dataset_id": "hotpot_qa",
"config": "distractor",
"split": "validation",
"split_kind": "eval",
"task": "qa",
"url": "https://huggingface.co/datasets/hotpot_qa"
},
{
"name": "codesearchnet_python_train",
"dataset_id": "code_search_net",
"config": "python",
"split": "train",
"split_kind": "train",
"task": "code",
"url": "https://huggingface.co/datasets/code_search_net"
},
{
"name": "codesearchnet_python_validation",
"dataset_id": "code_search_net",
"config": "python",
"split": "validation",
"split_kind": "eval",
"task": "code",
"url": "https://huggingface.co/datasets/code_search_net"
},
{
"name": "fever_train",
"dataset_id": "fever",
"config": "v1.0",
"split": "train",
"split_kind": "train",
"task": "evidence",
"url": "https://huggingface.co/datasets/fever"
},
{
"name": "fever_validation",
"dataset_id": "fever",
"config": "v1.0",
"split": "labelled_dev",
"split_kind": "eval",
"task": "evidence",
"url": "https://huggingface.co/datasets/fever"
}
],
"split_stats": {
"train": {
"kept": 60,
"unknown": 60,
"family:benchmark_qa": 60,
"written": 60
},
"eval": {
"dropped": 64,
"written": 0
}
},
"leakage_check": {
"group_overlap": 0,
"passed": true
},
"conflict_policy": {
"enabled": true,
"answerable_positive_conflicts_excluded": true,
"unknown_hard_conflicts_retained": true
},
"evaluation_policy": "eval.jsonl is generated and hashed before router training; the trainer refuses an optional hash mismatch.",
"warnings": [
"Local generated memory-policy files are useful hard negatives but are not public-human chat data.",
"Add redacted real user traces only after consent and PII removal.",
"Public download failures are recorded; failed sources contribute zero rows."
]
}
data/_ab_mega_V2_dpskw/train.jsonl
+60
View File
File diff suppressed because one or more lines are too long
data/_ab_mega_dynamic_memory_lab/eval.jsonl
View File
data/_ab_mega_dynamic_memory_lab/eval.sha256
+1
View File
@@ -0,0 +1 @@
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
data/_ab_mega_dynamic_memory_lab/manifest.json
+140
View File
@@ -0,0 +1,140 @@
{
"format_version": 3,
"generator": "prepare_memory_router_dataset.py",
"seed": 20260909,
"candidate_count": 32,
"schema": {
"query": "natural-language routing query",
"candidates": "bounded memory records with text and provenance",
"positive_indices": "one or more supporting memory records; empty means abstain",
"need_memory": "1 if evidence is required and present, 0 for unknown/unsupported queries",
"hop": "0 for abstention, otherwise number of supporting records clipped to router max_hops",
"group_id": "conversation or QA episode identity; no group may cross train/eval"
},
"files": {
"train": {
"path": "H:\\Memory\\V2_dpskw\\data\\_ab_mega_dynamic_memory_lab\\train.jsonl",
"episodes": 60,
"sha256": "f47703450f8ab3dece5d8d0094facb745989a81b1cc4d3cd6bcc6dd450c64138"
},
"eval": {
"path": "H:\\Memory\\V2_dpskw\\data\\_ab_mega_dynamic_memory_lab\\eval.jsonl",
"episodes": 0,
"sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
},
"counts": {
"train_groups": 80128,
"eval_groups": 64,
"train_episodes": 60,
"eval_episodes": 0,
"train_unknown": 60,
"eval_unknown": 0,
"train_candidates": 1920,
"eval_candidates": 0
},
"local_sources": [
{
"source": "H:/Memory/dynamic_memory_lab/data/benchmark_train.jsonl",
"rows": 128,
"kind": "benchmark",
"split": "train"
},
{
"source": "H:/Memory/dynamic_memory_lab/data/mega_validation/memory_validation_80k_train.jsonl",
"rows": 80000,
"kind": "mega_validation",
"split": "train"
},
{
"source": "H:/Memory/dynamic_memory_lab/data/benchmark_eval.jsonl",
"rows": 64,
"kind": "benchmark",
"split": "eval"
}
],
"public_sources": [],
"public_catalog": [
{
"name": "hotpotqa_train",
"dataset_id": "hotpot_qa",
"config": "distractor",
"split": "train",
"split_kind": "train",
"task": "qa",
"url": "https://huggingface.co/datasets/hotpot_qa"
},
{
"name": "hotpotqa_validation",
"dataset_id": "hotpot_qa",
"config": "distractor",
"split": "validation",
"split_kind": "eval",
"task": "qa",
"url": "https://huggingface.co/datasets/hotpot_qa"
},
{
"name": "codesearchnet_python_train",
"dataset_id": "code_search_net",
"config": "python",
"split": "train",
"split_kind": "train",
"task": "code",
"url": "https://huggingface.co/datasets/code_search_net"
},
{
"name": "codesearchnet_python_validation",
"dataset_id": "code_search_net",
"config": "python",
"split": "validation",
"split_kind": "eval",
"task": "code",
"url": "https://huggingface.co/datasets/code_search_net"
},
{
"name": "fever_train",
"dataset_id": "fever",
"config": "v1.0",
"split": "train",
"split_kind": "train",
"task": "evidence",
"url": "https://huggingface.co/datasets/fever"
},
{
"name": "fever_validation",
"dataset_id": "fever",
"config": "v1.0",
"split": "labelled_dev",
"split_kind": "eval",
"task": "evidence",
"url": "https://huggingface.co/datasets/fever"
}
],
"split_stats": {
"train": {
"kept": 60,
"unknown": 60,
"family:benchmark_qa": 60,
"written": 60
},
"eval": {
"dropped": 64,
"written": 0
}
},
"leakage_check": {
"group_overlap": 0,
"passed": true
},
"conflict_policy": {
"enabled": true,
"answerable_positive_conflicts_excluded": true,
"unknown_hard_conflicts_retained": true
},
"evaluation_policy": "eval.jsonl is generated and hashed before router training; the trainer refuses an optional hash mismatch.",
"warnings": [
"Local generated memory-policy files are useful hard negatives but are not public-human chat data.",
"Add redacted real user traces only after consent and PII removal.",
"Public download failures are recorded; failed sources contribute zero rows."
]
}
data/_ab_mega_dynamic_memory_lab/train.jsonl
+60
View File
File diff suppressed because one or more lines are too long
data/_equiv_slice.jsonl
+400
View File
File diff suppressed because one or more lines are too long
data/_probe_cap1/eval.jsonl
+1
View File
File diff suppressed because one or more lines are too long
data/_probe_cap1/eval.sha256
+1
View File
@@ -0,0 +1 @@
672ef5f1f8c525127cd648c67e67fa9f94d2e3cf7121c70d4d0463e508d80113
data/_probe_cap1/manifest.json
+184
View File
@@ -0,0 +1,184 @@
{
"format_version": 3,
"generator": "prepare_memory_router_dataset.py",
"seed": 20260909,
"candidate_count": 32,
"schema": {
"query": "natural-language routing query",
"candidates": "bounded memory records with text and provenance",
"positive_indices": "one or more supporting memory records; empty means abstain",
"need_memory": "1 if evidence is required and present, 0 for unknown/unsupported queries",
"hop": "0 for abstention, otherwise number of supporting records clipped to router max_hops",
"group_id": "conversation or QA episode identity; no group may cross train/eval"
},
"files": {
"train": {
"path": "H:\\Memory\\V2_dpskw\\data\\_probe_cap1\\train.jsonl",
"episodes": 1,
"sha256": "2540b45c59f6bd8757af2dc2440452d1cffd51448bbc0c59793b1fd92a714cec"
},
"eval": {
"path": "H:\\Memory\\V2_dpskw\\data\\_probe_cap1\\eval.jsonl",
"episodes": 1,
"sha256": "672ef5f1f8c525127cd648c67e67fa9f94d2e3cf7121c70d4d0463e508d80113"
}
},
"counts": {
"train_groups": 84155,
"eval_groups": 21120,
"train_episodes": 1,
"eval_episodes": 1,
"train_unknown": 1,
"eval_unknown": 1,
"train_candidates": 32,
"eval_candidates": 32
},
"local_sources": [
{
"source": "dynamic_memory_lab/data/benchmark_train.jsonl",
"rows": 128,
"kind": "benchmark",
"split": "train"
},
{
"source": "dynamic_memory_lab/data/native_memory/train.jsonl",
"rows": 512,
"kind": "native_memory",
"split": "train"
},
{
"source": "dynamic_memory_lab/data/production_memory/train.jsonl",
"rows": 1485,
"kind": "normalized_policy",
"split": "train"
},
{
"source": "dynamic_memory_lab/data/production_memory_hard_v2/train.jsonl",
"rows": 27000,
"kind": "normalized_policy",
"split": "train"
},
{
"source": "dynamic_memory_lab/data/mega_validation/memory_validation_80k_train.jsonl",
"rows": 80000,
"kind": "mega_validation",
"split": "train"
},
{
"source": "dynamic_memory_lab/data/benchmark_eval.jsonl",
"rows": 64,
"kind": "benchmark",
"split": "eval"
},
{
"source": "dynamic_memory_lab/data/native_memory/eval.jsonl",
"rows": 128,
"kind": "native_memory",
"split": "eval"
},
{
"source": "dynamic_memory_lab/data/production_memory/eval.jsonl",
"rows": 363,
"kind": "normalized_policy",
"split": "eval"
},
{
"source": "dynamic_memory_lab/data/production_memory_hard_v2/eval.jsonl",
"rows": 7200,
"kind": "normalized_policy",
"split": "eval"
},
{
"source": "dynamic_memory_lab/data/mega_validation/memory_validation_20k_eval.jsonl",
"rows": 20000,
"kind": "mega_validation",
"split": "eval"
}
],
"public_sources": [],
"public_catalog": [
{
"name": "hotpotqa_train",
"dataset_id": "hotpot_qa",
"config": "distractor",
"split": "train",
"split_kind": "train",
"task": "qa",
"url": "https://huggingface.co/datasets/hotpot_qa"
},
{
"name": "hotpotqa_validation",
"dataset_id": "hotpot_qa",
"config": "distractor",
"split": "validation",
"split_kind": "eval",
"task": "qa",
"url": "https://huggingface.co/datasets/hotpot_qa"
},
{
"name": "codesearchnet_python_train",
"dataset_id": "code_search_net",
"config": "python",
"split": "train",
"split_kind": "train",
"task": "code",
"url": "https://huggingface.co/datasets/code_search_net"
},
{
"name": "codesearchnet_python_validation",
"dataset_id": "code_search_net",
"config": "python",
"split": "validation",
"split_kind": "eval",
"task": "code",
"url": "https://huggingface.co/datasets/code_search_net"
},
{
"name": "fever_train",
"dataset_id": "fever",
"config": "v1.0",
"split": "train",
"split_kind": "train",
"task": "evidence",
"url": "https://huggingface.co/datasets/fever"
},
{
"name": "fever_validation",
"dataset_id": "fever",
"config": "v1.0",
"split": "labelled_dev",
"split_kind": "eval",
"task": "evidence",
"url": "https://huggingface.co/datasets/fever"
}
],
"split_stats": {
"train": {
"kept": 1,
"unknown": 1,
"family:benchmark_qa": 1,
"written": 1
},
"eval": {
"kept": 1,
"unknown": 1,
"family:benchmark_qa": 1,
"written": 1
}
},
"leakage_check": {
"group_overlap": 0,
"passed": true
},
"conflict_policy": {
"enabled": true,
"answerable_positive_conflicts_excluded": true,
"unknown_hard_conflicts_retained": true
},
"evaluation_policy": "eval.jsonl is generated and hashed before router training; the trainer refuses an optional hash mismatch.",
"warnings": [
"Local generated memory-policy files are useful hard negatives but are not public-human chat data.",
"Add redacted real user traces only after consent and PII removal.",
"Public download failures are recorded; failed sources contribute zero rows."
]
}

Some files were not shown because too many files have changed in this diff Show More