Close Round 04.1 selector patch
This commit is contained in:
parent
77114cee5b
commit
3f6f00a82e
|
|
@ -172,10 +172,13 @@ CCRA review packages are archived by round under `ccra_review_bundle/round-NN_YY
|
|||
|
||||
Files intended for GPT / CCRA upload inside a review bundle should include the round version in the filename, for example `_03.2`. Raw changed-file zips should also include the round version while preserving source-relative paths inside the archive.
|
||||
|
||||
Use the installed `bundle-zip` Skill for review bundle raw zips. Do not hand-roll source-relative zip creation with PowerShell `Compress-Archive` when directory paths matter.
|
||||
|
||||
## 13. Next Steps
|
||||
|
||||
1. Round 03 is closed after the Round 03.2a CCRA / GPT pass decision.
|
||||
2. Keep QPI and Intellectual Archaeology at `draft / B / pending` until Owner / CCRA review accepts stronger status.
|
||||
3. Start Round 04 blind input routing evaluation only after an explicit owner instruction; do not modify selector rules before that evaluation.
|
||||
4. Defer any third-model expansion until the current QPI selector and case layer are accepted.
|
||||
5. Route missing reusable extraction, inspection, or stability-scoring tools to `requirements/skills-vault/` or `requirements/ccpe/` instead of improvising local platform features.
|
||||
3. Round 04.1 is closed after GPT review acceptance and closeout documentation. The closeout records `R04-BI-022` as an accepted collateral depth-limited QPI override and adds it to regression.
|
||||
4. Round 05 discussion should start from the handoff report in a new session; do not treat Round 04.1 closure as a model lifecycle upgrade.
|
||||
5. Defer any third-model expansion until the current QPI selector and case layer are accepted.
|
||||
6. Route missing reusable extraction, inspection, or stability-scoring tools to `requirements/skills-vault/` or `requirements/ccpe/` instead of improvising local platform features.
|
||||
|
|
|
|||
|
|
@ -175,7 +175,7 @@
|
|||
- 机制稳定性：中等，七层结构清晰，但层级之间的真实递进关系需要更多案例验证。
|
||||
- 边界清晰度：中等，最小充分下潜原则已明确，但执行时容易过度下钻。
|
||||
- 来源证据质量：较高，核心定义和七层结构来自原文，停止条件来自综合应用材料。
|
||||
- 回归测试表现：pending，当前已有 19 条回归用例，已覆盖 positive、boundary、misuse、no_call、selector_gate 和 pipeline；Round 03.2 已纳入明确拒绝思想考古的模型级 hard exclusion 陷阱。
|
||||
- 回归测试表现：pending，当前已有 20 条回归用例，已覆盖 positive、boundary、misuse、no_call、selector_gate 和 pipeline；Round 03.2 已纳入明确拒绝思想考古的模型级 hard exclusion 陷阱，Round 04.1 已纳入 `不要进入思想考古` hard exclusion 变体。
|
||||
|
||||
评级理由：纵向下潜机制清晰，适合深度建模，但七层结构和停止条件需要进一步稳定。
|
||||
|
||||
|
|
@ -185,11 +185,15 @@
|
|||
|
||||
`pending`
|
||||
|
||||
当前已有 19 条回归用例，覆盖 positive、boundary、misuse、no_call、selector_gate 和 pipeline。Round 03.2 重点新增：
|
||||
当前已有 20 条回归用例，覆盖 positive、boundary、misuse、no_call、selector_gate 和 pipeline。Round 03.2 重点新增：
|
||||
|
||||
- `case_ia_negative_explicit_do_not_call_001`
|
||||
- `case_ia_negative_only_qpi_001`
|
||||
|
||||
Round 04.1 重点新增：
|
||||
|
||||
- `case_ia_round04_do_not_enter_archaeology_001`
|
||||
|
||||
## 示例输入
|
||||
|
||||
- “我想把过去文章中的认知模型抽取出来，但不知道这些模型到底稳不稳，应该怎么判断？”
|
||||
|
|
|
|||
11
cards/qpi.md
11
cards/qpi.md
|
|
@ -204,7 +204,7 @@ QPI 的三分结构、核心匮乏物、主体性和动态性有清晰来源支
|
|||
- 机制稳定性：中高，扫描匮乏物和匹配处理范式的机制明确。
|
||||
- 边界清晰度：中等，混合型问题、多视角分歧和上下文不足场景仍需更多测试。
|
||||
- 来源证据质量：较高，主体性和动态性来自 2025 原文，核心匮乏物和误框定规则来自 2026 原文，应用规则来自综合文档。
|
||||
- 回归测试表现：pending，当前已有 58 条 QPI 回归用例；其中 Round 03 / 03.1 已纳入 owner-reviewed case layer、selector no-call、multi-perspective 和低上下文 provisional 边界用例，Round 03.2 已纳入普通 how-to / 词义判断 no-call 陷阱，Round 03.2a 已纳入 depth-limiting QPI override 陷阱。生命周期仍保持 draft。
|
||||
- 回归测试表现：pending，当前已有 62 条 QPI 回归用例；其中 Round 03 / 03.1 已纳入 owner-reviewed case layer、selector no-call、multi-perspective 和低上下文 provisional 边界用例，Round 03.2 已纳入普通 how-to / 词义判断 no-call 陷阱，Round 03.2a 已纳入 depth-limiting QPI override 陷阱，Round 04.1 已纳入 translation payload no-call、depth-limiting QPI override 和 IA heavy-signal anti-recall 陷阱。生命周期仍保持 draft。
|
||||
|
||||
评级理由：三分结构清晰，适合作为入口路由模型，但需要补充大量边界案例，防止过度升维或降维。
|
||||
|
||||
|
|
@ -214,7 +214,7 @@ QPI 的三分结构、核心匮乏物、主体性和动态性有清晰来源支
|
|||
|
||||
`pending`
|
||||
|
||||
当前已有 58 条 QPI 回归用例，覆盖 positive、boundary、misuse、no_call、selector_gate 和 pipeline。Round 03 / 03.1 重点新增：
|
||||
当前已有 62 条 QPI 回归用例，覆盖 positive、boundary、misuse、no_call、selector_gate 和 pipeline。Round 03 / 03.1 重点新增：
|
||||
|
||||
- owner-reviewed flow / disappointment / organizational year-end-review cases；
|
||||
- `case_qpi_international_logistics_no_call_001`
|
||||
|
|
@ -236,6 +236,13 @@ Round 03.2a 重点新增：
|
|||
- `case_qpi_depth_limited_problem_definition_001`
|
||||
- `case_qpi_depth_limited_dominant_scarcity_001`
|
||||
|
||||
Round 04.1 重点新增：
|
||||
|
||||
- `case_qpi_round04_translation_payload_no_call_001`
|
||||
- `case_qpi_round04_depth_limited_specific_scarcity_001`
|
||||
- `case_qpi_round04_depth_limited_scarcity_set_001`
|
||||
- `case_qpi_round04_depth_limited_assumption_scarcity_001`
|
||||
|
||||
## 示例输入
|
||||
|
||||
- “我们团队每次都说要长期主义，但一到季度 KPI 就回到短期动作，这到底是什么问题？”
|
||||
|
|
|
|||
|
|
@ -0,0 +1,53 @@
|
|||
# Round 04.1 CCRA Review Brief
|
||||
|
||||
Date: 2026-06-18
|
||||
|
||||
Repository: `C:\Users\wangq\Documents\Codex\work-projects\the-mindscape-of-bro-tsong`
|
||||
|
||||
Round: `04.1`
|
||||
|
||||
Topic: Selector no-call, hard exclusion, and depth-limiting patch
|
||||
|
||||
## 1. Review Goal
|
||||
|
||||
Round 04.1 fixes the owner / GPT reviewed failures from the Round 04 blind routing run.
|
||||
|
||||
This is a selector rule patch and post-patch verification round. It is not a second blind test.
|
||||
|
||||
## 2. Target Failures
|
||||
|
||||
- `R04-BI-002`: translation task selected QPI because the payload contained `模型`.
|
||||
- `R04-BI-024`: explicit `不要进入思想考古` did not hard-exclude Intellectual Archaeology.
|
||||
- `R04-BI-035`: `不要深入分析` plus dominant-scarcity judgment was hard no-called.
|
||||
- `R04-BI-036`: `不要展开` plus scarcity-set judgment was hard no-called.
|
||||
|
||||
## 3. What Changed
|
||||
|
||||
- Added targeted regression cases for the four reviewed failures.
|
||||
- Added translation / instruction-task no-call coverage.
|
||||
- Added minimal instruction / payload separation for no-call routing.
|
||||
- Added IA hard-exclusion variants such as `不要进入思想考古`.
|
||||
- Expanded depth-limiting QPI override phrases.
|
||||
- Generated post-patch routing verification against the frozen 38-input Round 04 pool.
|
||||
- Closeout records `R04-BI-022` as an accepted collateral change and adds it to QPI regression.
|
||||
|
||||
## 4. What Did Not Change
|
||||
|
||||
- The original Round 04 blind routing report was not rewritten.
|
||||
- No LLM selector was introduced.
|
||||
- No third model was added.
|
||||
- QPI and Intellectual Archaeology were not upgraded.
|
||||
- No RAG, vector database, frontend, backend, user account, or QA product was introduced.
|
||||
|
||||
## 5. Closeout Note
|
||||
|
||||
GPT review accepted Round 04.1 and did not request a 04.2 rule patch. The only closeout addition is documentation / regression hardening for `R04-BI-022`: it moved from no-call to lightweight QPI after the depth-limiting override patch, IA remains rejected, and this is treated as semantically correct collateral behavior.
|
||||
|
||||
## 6. Read Order
|
||||
|
||||
1. `01_RULE_PATCH_SUMMARY_AND_DIFF_04.1.md`
|
||||
2. `02_EXPECTED_ROUTING_LABELS_04.1.md`
|
||||
3. `03_POST_PATCH_ROUTING_VERIFICATION_REPORT_04.1.md`
|
||||
4. `04_REGRESSION_AND_COMMAND_LOG_04.1.md`
|
||||
|
||||
Use `optional_raw_changed_files_04.1.zip` only when exact file inspection is needed.
|
||||
|
|
@ -0,0 +1,75 @@
|
|||
# Round 04.1 Rule Patch Summary and Diff Guide
|
||||
|
||||
## 1. Patch Scope
|
||||
|
||||
Round 04.1 patches selector rules only. It does not change model lifecycle status.
|
||||
|
||||
## 2. Rule Changes
|
||||
|
||||
### Translation / Direct Transformation No-Call
|
||||
|
||||
Added `instruction_task_no_call_signals` to `selector/selector_rules.json`:
|
||||
|
||||
```text
|
||||
翻译成
|
||||
翻译为
|
||||
把这句话翻译
|
||||
翻译下面
|
||||
译成英文
|
||||
译成中文
|
||||
```
|
||||
|
||||
These are checked against the task instruction segment so payload words such as `模型` do not trigger QPI when the user is asking for translation.
|
||||
|
||||
### Instruction / Payload Separation
|
||||
|
||||
Added `instruction_segment()` in `scripts/run_selector_demo.py`.
|
||||
|
||||
For no-call routing, the selector can inspect the segment before `：`, `:`, or newline as the task instruction. This is intentionally minimal and does not introduce semantic parsing or LLM routing.
|
||||
|
||||
### IA Hard Exclusion Variants
|
||||
|
||||
Added model-level hard exclusion variants:
|
||||
|
||||
```text
|
||||
不要进入思想考古
|
||||
不进入思想考古
|
||||
先不进入思想考古
|
||||
不要做思想考古
|
||||
不做思想考古
|
||||
```
|
||||
|
||||
This fixes cases where the user explicitly refuses IA but uses phrasing not covered by Round 03.2.
|
||||
|
||||
### Depth-Limiting QPI Override
|
||||
|
||||
Expanded QPI-limited override and IA exclusion phrases for:
|
||||
|
||||
```text
|
||||
主导稀缺
|
||||
只判断这件事的主导稀缺
|
||||
只判断现在是缺信息
|
||||
缺信息、缺路径还是缺共识
|
||||
```
|
||||
|
||||
This keeps `不要展开` / `不要深入分析` as no-call by default, but allows QPI when the user asks for a narrow scarcity judgment.
|
||||
|
||||
## 3. Regression Changes
|
||||
|
||||
Added:
|
||||
|
||||
- `case_qpi_round04_translation_payload_no_call_001`
|
||||
- `case_qpi_round04_depth_limited_specific_scarcity_001`
|
||||
- `case_qpi_round04_depth_limited_scarcity_set_001`
|
||||
- `case_ia_round04_do_not_enter_archaeology_001`
|
||||
- Closeout addition: `case_qpi_round04_depth_limited_assumption_scarcity_001` from `R04-BI-022`
|
||||
|
||||
Counts after patch:
|
||||
|
||||
- QPI regression cases: 62
|
||||
- Intellectual Archaeology regression cases: 20
|
||||
- Aggregate regression cases: 82
|
||||
|
||||
## 4. Review Notes
|
||||
|
||||
The rule patch is intentionally conservative. It does not expand QPI into all governance, capacity, or organization-like inputs from Round 04. Those remain calibration discussion items unless owner / CCRA approves a later patch.
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
# Round 04.1 Expected Routing Labels
|
||||
|
||||
These labels are added after owner / GPT review of the Round 04 blind output.
|
||||
|
||||
They apply only to the four targeted patch cases and do not convert the entire Round 04 blind pool into regression.
|
||||
|
||||
## Target Labels
|
||||
|
||||
| Case ID | Expected Routing | Failure Type Addressed |
|
||||
| --- | --- | --- |
|
||||
| `R04-BI-002` | `selected_models=[]`, `no_call=true` | translation no-call false negative; QPI over-selection |
|
||||
| `R04-BI-024` | `selected_models` may include `qpi`; must reject `intellectual_archaeology`; `no_call=false` | IA hard exclusion failure |
|
||||
| `R04-BI-035` | selected `qpi`; reject `intellectual_archaeology`; `no_call=false` | depth-limiting override failure |
|
||||
| `R04-BI-036` | selected `qpi`; reject `intellectual_archaeology`; `no_call=false` | depth-limiting override failure |
|
||||
|
||||
## Non-Target Notes
|
||||
|
||||
Round 04 governance, responsibility, capacity, and ambiguous-context cases are not patched in 04.1.
|
||||
|
||||
They may indicate future QPI under-selection or calibration questions, but broadening QPI recall in this patch would risk turning QPI into a generic organization-problem detector.
|
||||
|
|
@ -0,0 +1,175 @@
|
|||
# Round 04.1 Post-Patch Routing Verification Report
|
||||
|
||||
Date: 2026-06-18
|
||||
|
||||
This is not a second blind test.
|
||||
|
||||
Frozen input set: `round04_blind_inputs_2026-06-18_v0.1`
|
||||
Original before-report: `reports/Round04_blind_routing_evaluation_report_2026-06-18.md`
|
||||
|
||||
## Scope
|
||||
|
||||
- Round 04.1 verifies selector rule patches after owner / GPT review.
|
||||
- It reruns the frozen 38-input Round 04 pool for comparison.
|
||||
- Expected routing labels are limited to the four review-approved target cases.
|
||||
- It does not generate final answers and does not upgrade model status.
|
||||
|
||||
## Summary
|
||||
|
||||
Frozen inputs rerun: 38
|
||||
Post-patch no-call results: 18
|
||||
Targeted cases checked: 4
|
||||
Targeted failures: 0
|
||||
Behavior changes recorded: 5
|
||||
|
||||
## Targeted Case Results
|
||||
|
||||
### R04-BI-002
|
||||
|
||||
- status: `PASS`
|
||||
- review_label: translation no-call; payload complexity must not trigger QPI
|
||||
- before_selected_models: `['qpi']`
|
||||
- before_no_call: `False`
|
||||
- after_selected_models: `[]`
|
||||
- after_rejected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- after_no_call: `True`
|
||||
|
||||
After selected model details:
|
||||
- none
|
||||
|
||||
### R04-BI-024
|
||||
|
||||
- status: `PASS`
|
||||
- review_label: explicit IA refusal should hard-exclude IA while allowing QPI
|
||||
- before_selected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- before_no_call: `False`
|
||||
- after_selected_models: `['qpi']`
|
||||
- after_rejected_models: `['intellectual_archaeology']`
|
||||
- after_no_call: `False`
|
||||
|
||||
After selected model details:
|
||||
- `qpi` score=0.74 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; complexity signal: 边界、责任; selection_priority=9
|
||||
|
||||
### R04-BI-035
|
||||
|
||||
- status: `PASS`
|
||||
- review_label: depth-limiting plus dominant-scarcity judgment should allow QPI
|
||||
- before_selected_models: `[]`
|
||||
- before_no_call: `True`
|
||||
- after_selected_models: `['qpi']`
|
||||
- after_rejected_models: `['intellectual_archaeology']`
|
||||
- after_no_call: `False`
|
||||
|
||||
After selected model details:
|
||||
- `qpi` score=0.59 decision=selected
|
||||
- reasons: selection_priority=9; QPI problem-definition gate matched
|
||||
|
||||
### R04-BI-036
|
||||
|
||||
- status: `PASS`
|
||||
- review_label: depth-limiting plus scarcity-set judgment should allow QPI
|
||||
- before_selected_models: `[]`
|
||||
- before_no_call: `True`
|
||||
- after_selected_models: `['qpi']`
|
||||
- after_rejected_models: `['intellectual_archaeology']`
|
||||
- after_no_call: `False`
|
||||
|
||||
After selected model details:
|
||||
- `qpi` score=0.99 decision=selected
|
||||
- reasons: trigger keyword matched: 缺路径、缺共识; complexity signal: 共识、底层; selection_priority=9; QPI problem-definition gate matched
|
||||
|
||||
## Non-target Behavior Diff / Accepted Collateral Change
|
||||
|
||||
The frozen 38-input pool was rerun for comparison. Expected routing labels remain limited to the four target cases above, but the full rerun records five behavior changes from the original Round 04 report.
|
||||
|
||||
### R04-BI-002
|
||||
|
||||
- label: target fix: translation no-call
|
||||
- before_selected_models: `['qpi']`
|
||||
- before_no_call: `False`
|
||||
- after_selected_models: `[]`
|
||||
- after_rejected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- after_no_call: `True`
|
||||
- closeout_judgment: Expected target change; translation instruction should not be routed to QPI because the payload contains `模型`.
|
||||
|
||||
### R04-BI-022
|
||||
|
||||
- label: accepted collateral change: depth-limited QPI override
|
||||
- before_selected_models: `[]`
|
||||
- before_no_call: `True`
|
||||
- after_selected_models: `['qpi']`
|
||||
- after_rejected_models: `['intellectual_archaeology']`
|
||||
- after_no_call: `False`
|
||||
- closeout_judgment: Not a new failure. The request rejects deeper IA-style expansion while asking for dominant scarcity judgment, so lightweight QPI is acceptable and IA should remain rejected. Added to regression as a QPI selector-gate trap.
|
||||
|
||||
### R04-BI-024
|
||||
|
||||
- label: target fix: explicit IA refusal
|
||||
- before_selected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- before_no_call: `False`
|
||||
- after_selected_models: `['qpi']`
|
||||
- after_rejected_models: `['intellectual_archaeology']`
|
||||
- after_no_call: `False`
|
||||
- closeout_judgment: Expected target change; explicit refusal should hard-exclude IA while allowing lightweight QPI.
|
||||
|
||||
### R04-BI-035
|
||||
|
||||
- label: target fix: dominant-scarcity depth limit
|
||||
- before_selected_models: `[]`
|
||||
- before_no_call: `True`
|
||||
- after_selected_models: `['qpi']`
|
||||
- after_rejected_models: `['intellectual_archaeology']`
|
||||
- after_no_call: `False`
|
||||
- closeout_judgment: Expected target change; depth-limiting language should not block lightweight QPI scarcity judgment.
|
||||
|
||||
### R04-BI-036
|
||||
|
||||
- label: target fix: scarcity-set depth limit
|
||||
- before_selected_models: `[]`
|
||||
- before_no_call: `True`
|
||||
- after_selected_models: `['qpi']`
|
||||
- after_rejected_models: `['intellectual_archaeology']`
|
||||
- after_no_call: `False`
|
||||
- closeout_judgment: Expected target change; depth-limiting language should not block QPI scarcity-set judgment.
|
||||
|
||||
## Full Pool Rerun Summary
|
||||
|
||||
- `R04-BI-001`: selected=[]; no_call=True; category=explicit_no_call_fact_lookup
|
||||
- `R04-BI-002`: selected=[]; no_call=True; category=explicit_no_call_translation
|
||||
- `R04-BI-003`: selected=[]; no_call=True; category=explicit_no_call_rewrite
|
||||
- `R04-BI-004`: selected=[]; no_call=True; category=explicit_no_call_formatting
|
||||
- `R04-BI-005`: selected=[]; no_call=True; category=explicit_no_call_direct_execution
|
||||
- `R04-BI-006`: selected=['qpi']; no_call=False; category=qpi_light_problem_definition
|
||||
- `R04-BI-007`: selected=['qpi']; no_call=False; category=qpi_light_problem_definition
|
||||
- `R04-BI-008`: selected=['qpi']; no_call=False; category=qpi_light_problem_definition
|
||||
- `R04-BI-009`: selected=['qpi']; no_call=False; category=qpi_light_problem_definition
|
||||
- `R04-BI-010`: selected=[]; no_call=True; category=governance_responsibility_consensus
|
||||
- `R04-BI-011`: selected=[]; no_call=True; category=governance_responsibility_consensus
|
||||
- `R04-BI-012`: selected=[]; no_call=True; category=governance_responsibility_consensus
|
||||
- `R04-BI-013`: selected=[]; no_call=True; category=governance_responsibility_consensus
|
||||
- `R04-BI-014`: selected=[]; no_call=True; category=capacity_resource_path
|
||||
- `R04-BI-015`: selected=[]; no_call=True; category=capacity_resource_path
|
||||
- `R04-BI-016`: selected=['qpi']; no_call=False; category=capacity_resource_path
|
||||
- `R04-BI-017`: selected=[]; no_call=True; category=capacity_resource_path
|
||||
- `R04-BI-018`: selected=['qpi']; no_call=False; category=model_extraction_or_depth_modeling
|
||||
- `R04-BI-019`: selected=['qpi', 'intellectual_archaeology']; no_call=False; category=model_extraction_or_depth_modeling
|
||||
- `R04-BI-020`: selected=['qpi']; no_call=False; category=model_extraction_or_depth_modeling
|
||||
- `R04-BI-021`: selected=['qpi']; no_call=False; category=qpi_allowed_ia_refused
|
||||
- `R04-BI-022`: selected=['qpi']; no_call=False; category=qpi_allowed_ia_refused
|
||||
- `R04-BI-023`: selected=['qpi']; no_call=False; category=qpi_allowed_ia_refused
|
||||
- `R04-BI-024`: selected=['qpi']; no_call=False; category=qpi_allowed_ia_refused
|
||||
- `R04-BI-025`: selected=['intellectual_archaeology', 'qpi']; no_call=False; category=likely_intellectual_archaeology
|
||||
- `R04-BI-026`: selected=['qpi']; no_call=False; category=likely_intellectual_archaeology
|
||||
- `R04-BI-027`: selected=['qpi']; no_call=False; category=likely_intellectual_archaeology
|
||||
- `R04-BI-028`: selected=[]; no_call=True; category=ambiguous_low_context
|
||||
- `R04-BI-029`: selected=[]; no_call=True; category=ambiguous_low_context
|
||||
- `R04-BI-030`: selected=['qpi']; no_call=False; category=ambiguous_low_context
|
||||
- `R04-BI-031`: selected=[]; no_call=True; category=explicit_no_call_how_to
|
||||
- `R04-BI-032`: selected=[]; no_call=True; category=explicit_no_call_how_to
|
||||
- `R04-BI-033`: selected=[]; no_call=True; category=explicit_no_call_dictionary_judgment
|
||||
- `R04-BI-034`: selected=['qpi']; no_call=False; category=direct_execution_with_analysis_override
|
||||
- `R04-BI-035`: selected=['qpi']; no_call=False; category=qpi_allowed_depth_limited
|
||||
- `R04-BI-036`: selected=['qpi']; no_call=False; category=qpi_allowed_depth_limited
|
||||
- `R04-BI-037`: selected=[]; no_call=True; category=ambiguous_growth_how_to
|
||||
- `R04-BI-038`: selected=['qpi']; no_call=False; category=likely_intellectual_archaeology
|
||||
|
|
@ -0,0 +1,98 @@
|
|||
# Round 04.1 Regression and Command Log
|
||||
|
||||
Date: 2026-06-18
|
||||
|
||||
## 1. Red Evidence
|
||||
|
||||
After adding the four Round 04.1 regression cases and before patching selector rules:
|
||||
|
||||
```powershell
|
||||
python scripts\run_selector_regression.py
|
||||
```
|
||||
|
||||
Failed as expected:
|
||||
|
||||
- `case_ia_round04_do_not_enter_archaeology_001`: expected primary QPI, got IA; IA selected despite explicit refusal.
|
||||
- `case_qpi_round04_translation_payload_no_call_001`: expected no selected model, got QPI.
|
||||
- `case_qpi_round04_depth_limited_specific_scarcity_001`: expected QPI, got none.
|
||||
- `case_qpi_round04_depth_limited_scarcity_set_001`: expected QPI, got none.
|
||||
|
||||
## 2. Targeted Green Evidence
|
||||
|
||||
After the patch:
|
||||
|
||||
```powershell
|
||||
python scripts\run_selector_regression.py
|
||||
python -m unittest tests.test_round04_post_patch_verification -v
|
||||
python scripts\run_round04_post_patch_verification.py
|
||||
```
|
||||
|
||||
Observed:
|
||||
|
||||
- Selector regression passed.
|
||||
- Round 04.1 post-patch verification test passed, 2 tests.
|
||||
- Post-patch report generated with 4 targeted cases checked and 0 targeted failures.
|
||||
- Closeout report records 5 behavior changes, including accepted collateral change `R04-BI-022`.
|
||||
- `R04-BI-022` was added to QPI regression as `case_qpi_round04_depth_limited_assumption_scarcity_001`.
|
||||
|
||||
## 3. Full Verification
|
||||
|
||||
The following commands passed after the Round 04.1 patch:
|
||||
|
||||
```powershell
|
||||
python scripts\rebuild_indexes.py --check
|
||||
python -m unittest discover -s tests -p "test*.py" -v
|
||||
python scripts\validate_model_library.py
|
||||
python scripts\check_card_contract.py
|
||||
python scripts\run_selector_demo.py
|
||||
python scripts\run_selector_regression.py
|
||||
python scripts\run_selector_calibration_smoke.py
|
||||
python scripts\check_model_card_sync.py
|
||||
```
|
||||
|
||||
Results:
|
||||
|
||||
- Index check: PASS
|
||||
- Unit tests: PASS, 22 tests
|
||||
- Model library validation: PASS
|
||||
- Card contract: PASS
|
||||
- Selector demo: PASS
|
||||
- Selector regression: PASS, 26 cases
|
||||
- Selector calibration smoke: PASS
|
||||
- Model/card sync: PASS
|
||||
|
||||
## 4. Lifecycle Status
|
||||
|
||||
QPI and Intellectual Archaeology remain:
|
||||
|
||||
```text
|
||||
draft / B / pending
|
||||
```
|
||||
|
||||
## 5. Bundle Zip Verification
|
||||
|
||||
The raw changed-files zip was generated with the installed `bundle-zip` Skill:
|
||||
|
||||
```powershell
|
||||
python C:\Users\wangq\.agents\skills\bundle-zip\scripts\bundle_zip.py --file-list ccra_review_bundle\round-04.1_2026-06-18_selector-rule-patch\raw_file_list_04.1.txt --relative-root-marker the-mindscape-of-bro-tsong --output C:\Users\wangq\Documents\Codex\work-projects\the-mindscape-of-bro-tsong\ccra_review_bundle\round-04.1_2026-06-18_selector-rule-patch\optional_raw_changed_files_04.1.zip --verify --json-output ccra_review_bundle\round-04.1_2026-06-18_selector-rule-patch\optional_raw_changed_files_04.1.zip.validation.json
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
- Status: PASS
|
||||
- Source files: 37
|
||||
- Zip entries: 37
|
||||
- Path mode: source-relative entries under `the-mindscape-of-bro-tsong`
|
||||
- Validation sidecar: `optional_raw_changed_files_04.1.zip.validation.json`
|
||||
|
||||
## 6. Closeout Note
|
||||
|
||||
Round 04.1 is closed after GPT review acceptance and this closeout note.
|
||||
|
||||
No additional selector rule patch was made during closeout. The only closeout behavior change is documentation / regression hardening for `R04-BI-022`:
|
||||
|
||||
- It is not treated as a new failure.
|
||||
- It is accepted as a depth-limiting QPI override collateral change.
|
||||
- IA should continue to be rejected.
|
||||
- Lightweight QPI routing is acceptable.
|
||||
- The case is now covered by selector regression.
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
# Round 04.1 Bundle File Manifest
|
||||
|
||||
## Review Files
|
||||
|
||||
- `00_OPEN_THIS_FIRST_CCRA_REVIEW_BRIEF_04.1.md`
|
||||
- `01_RULE_PATCH_SUMMARY_AND_DIFF_04.1.md`
|
||||
- `02_EXPECTED_ROUTING_LABELS_04.1.md`
|
||||
- `03_POST_PATCH_ROUTING_VERIFICATION_REPORT_04.1.md`
|
||||
- `04_REGRESSION_AND_COMMAND_LOG_04.1.md`
|
||||
- `BUNDLE_FILE_MANIFEST_04.1.md`
|
||||
- `raw_file_list_04.1.txt`
|
||||
- `optional_raw_changed_files_04.1.zip`
|
||||
- `optional_raw_changed_files_04.1.zip.validation.json`
|
||||
|
||||
## Optional Raw Changed Files
|
||||
|
||||
The zip is optional and should be used only if exact file inspection is needed.
|
||||
|
||||
It is generated with the installed `bundle-zip` Skill and preserves source-relative paths.
|
||||
|
||||
## Source-Relative Files Included in Raw Zip
|
||||
|
||||
- `README.md`
|
||||
- `docs/DECISIONS.md`
|
||||
- `docs/WORKFLOW.md`
|
||||
- `docs/superpowers/plans/2026-06-18-round04-1-selector-rule-patch.md`
|
||||
- `selector/README.md`
|
||||
- `selector/selector_rules.json`
|
||||
- `selector/selector_calibration_inputs.json`
|
||||
- `selector/round04_blind_inputs.json`
|
||||
- `scripts/README.md`
|
||||
- `scripts/run_selector_demo.py`
|
||||
- `scripts/run_selector_regression.py`
|
||||
- `scripts/run_selector_calibration_smoke.py`
|
||||
- `scripts/run_round04_blind_routing.py`
|
||||
- `scripts/run_round04_post_patch_verification.py`
|
||||
- `tests/qpi.regression.json`
|
||||
- `tests/intellectual_archaeology.regression.json`
|
||||
- `tests/regression_cases.json`
|
||||
- `tests/regression_index.json`
|
||||
- `tests/test_round04_blind_routing.py`
|
||||
- `tests/test_round04_post_patch_verification.py`
|
||||
- `models/model_index.json`
|
||||
- `models/qpi.model.json`
|
||||
- `models/intellectual_archaeology.model.json`
|
||||
- `cards/card_index.md`
|
||||
- `cards/qpi.md`
|
||||
- `cards/intellectual_archaeology.md`
|
||||
- `reports/README.md`
|
||||
- `reports/Round04_blind_routing_evaluation_report_2026-06-18.md`
|
||||
- `reports/Round04_1_post_patch_routing_verification_report_2026-06-18.md`
|
||||
- `reports/Codex新会话交接文档_Round04.1关闭与Round05准备_2026-06-18.md`
|
||||
- `reports/selector_regression_report_v0.2.md`
|
||||
- `reports/selector_calibration_smoke_report.md`
|
||||
- `reports/validation_report.md`
|
||||
- `reports/index_rebuild_report.md`
|
||||
- `reports/model_card_sync_report_v0.2.md`
|
||||
- `reports/content_review_report_v0.2.md`
|
||||
- `requirements/skills-vault/2026-06-18-review-bundle-zip-packager.md`
|
||||
Binary file not shown.
|
|
@ -0,0 +1,47 @@
|
|||
{
|
||||
"status": "PASS",
|
||||
"output_zip": "C:\\Users\\wangq\\Documents\\Codex\\work-projects\\the-mindscape-of-bro-tsong\\ccra_review_bundle\\round-04.1_2026-06-18_selector-rule-patch\\optional_raw_changed_files_04.1.zip",
|
||||
"source_file_count": 37,
|
||||
"zip_entry_count": 37,
|
||||
"entries": [
|
||||
"README.md",
|
||||
"docs/DECISIONS.md",
|
||||
"docs/WORKFLOW.md",
|
||||
"docs/superpowers/plans/2026-06-18-round04-1-selector-rule-patch.md",
|
||||
"selector/README.md",
|
||||
"selector/selector_rules.json",
|
||||
"selector/selector_calibration_inputs.json",
|
||||
"selector/round04_blind_inputs.json",
|
||||
"scripts/README.md",
|
||||
"scripts/run_selector_demo.py",
|
||||
"scripts/run_selector_regression.py",
|
||||
"scripts/run_selector_calibration_smoke.py",
|
||||
"scripts/run_round04_blind_routing.py",
|
||||
"scripts/run_round04_post_patch_verification.py",
|
||||
"tests/qpi.regression.json",
|
||||
"tests/intellectual_archaeology.regression.json",
|
||||
"tests/regression_cases.json",
|
||||
"tests/regression_index.json",
|
||||
"tests/test_round04_blind_routing.py",
|
||||
"tests/test_round04_post_patch_verification.py",
|
||||
"models/model_index.json",
|
||||
"models/qpi.model.json",
|
||||
"models/intellectual_archaeology.model.json",
|
||||
"cards/card_index.md",
|
||||
"cards/qpi.md",
|
||||
"cards/intellectual_archaeology.md",
|
||||
"reports/README.md",
|
||||
"reports/Round04_blind_routing_evaluation_report_2026-06-18.md",
|
||||
"reports/Round04_1_post_patch_routing_verification_report_2026-06-18.md",
|
||||
"reports/Codex新会话交接文档_Round04.1关闭与Round05准备_2026-06-18.md",
|
||||
"reports/selector_regression_report_v0.2.md",
|
||||
"reports/selector_calibration_smoke_report.md",
|
||||
"reports/validation_report.md",
|
||||
"reports/index_rebuild_report.md",
|
||||
"reports/model_card_sync_report_v0.2.md",
|
||||
"reports/content_review_report_v0.2.md",
|
||||
"requirements/skills-vault/2026-06-18-review-bundle-zip-packager.md"
|
||||
],
|
||||
"warnings": [],
|
||||
"errors": []
|
||||
}
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
README.md
|
||||
docs/DECISIONS.md
|
||||
docs/WORKFLOW.md
|
||||
docs/superpowers/plans/2026-06-18-round04-1-selector-rule-patch.md
|
||||
selector/README.md
|
||||
selector/selector_rules.json
|
||||
selector/selector_calibration_inputs.json
|
||||
selector/round04_blind_inputs.json
|
||||
scripts/README.md
|
||||
scripts/run_selector_demo.py
|
||||
scripts/run_selector_regression.py
|
||||
scripts/run_selector_calibration_smoke.py
|
||||
scripts/run_round04_blind_routing.py
|
||||
scripts/run_round04_post_patch_verification.py
|
||||
tests/qpi.regression.json
|
||||
tests/intellectual_archaeology.regression.json
|
||||
tests/regression_cases.json
|
||||
tests/regression_index.json
|
||||
tests/test_round04_blind_routing.py
|
||||
tests/test_round04_post_patch_verification.py
|
||||
models/model_index.json
|
||||
models/qpi.model.json
|
||||
models/intellectual_archaeology.model.json
|
||||
cards/card_index.md
|
||||
cards/qpi.md
|
||||
cards/intellectual_archaeology.md
|
||||
reports/README.md
|
||||
reports/Round04_blind_routing_evaluation_report_2026-06-18.md
|
||||
reports/Round04_1_post_patch_routing_verification_report_2026-06-18.md
|
||||
reports/Codex新会话交接文档_Round04.1关闭与Round05准备_2026-06-18.md
|
||||
reports/selector_regression_report_v0.2.md
|
||||
reports/selector_calibration_smoke_report.md
|
||||
reports/validation_report.md
|
||||
reports/index_rebuild_report.md
|
||||
reports/model_card_sync_report_v0.2.md
|
||||
reports/content_review_report_v0.2.md
|
||||
requirements/skills-vault/2026-06-18-review-bundle-zip-packager.md
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
# Round 04 CCRA Review Brief
|
||||
|
||||
Date: 2026-06-18
|
||||
|
||||
Repository: `C:\Users\wangq\Documents\Codex\work-projects\the-mindscape-of-bro-tsong`
|
||||
|
||||
Round: `04`
|
||||
|
||||
Topic: Blind input routing evaluation
|
||||
|
||||
## 1. Review Goal
|
||||
|
||||
Round 04 tests whether the current rule-based selector, QPI routing contract, and Intellectual Archaeology gate behave stably on new inputs.
|
||||
|
||||
This is a routing evaluation only. It does not evaluate final answer quality.
|
||||
|
||||
## 2. What Changed
|
||||
|
||||
- Updated `reports/Round04_blind_input_candidates_2026-06-18.md` from owner review notes.
|
||||
- Added frozen blind input pool: `selector/round04_blind_inputs.json`.
|
||||
- Added standard-library runner: `scripts/run_round04_blind_routing.py`.
|
||||
- Added tests for the runner: `tests/test_round04_blind_routing.py`.
|
||||
- Ran the first blind routing pass and wrote `reports/Round04_blind_routing_evaluation_report_2026-06-18.md`.
|
||||
|
||||
## 3. What Did Not Change
|
||||
|
||||
- `selector/selector_rules.json` was not modified.
|
||||
- QPI and Intellectual Archaeology model JSON lifecycle status was not upgraded.
|
||||
- No third model was added.
|
||||
- No LLM selector, RAG, vector database, frontend, backend, account system, or QA product was introduced.
|
||||
- No expected routing labels were added before the first blind run.
|
||||
|
||||
## 4. Files To Read
|
||||
|
||||
1. `01_ROUND03_CLOSURE_AND_ROUND04_SCOPE_04.md`
|
||||
2. `02_BLIND_INPUT_SET_04.md`
|
||||
3. `03_ROUTING_EVALUATION_REPORT_04.md`
|
||||
4. `04_REVIEW_QUESTIONS_FOR_GPT_04.md`
|
||||
|
||||
The optional raw zip contains source-relative files for exact inspection.
|
||||
|
|
@ -0,0 +1,81 @@
|
|||
# Round 03 Closure and Round 04 Scope
|
||||
|
||||
## 1. Round 03 Closure Summary
|
||||
|
||||
```text
|
||||
Round 03 closed after 03.2a.
|
||||
|
||||
QPI and Intellectual Archaeology remain draft / B / pending.
|
||||
QPI is accepted as a draft-callable routing asset at report level.
|
||||
Selector v0.2 passes:
|
||||
- no-call gate
|
||||
- calibration smoke
|
||||
- model-level IA exclusion
|
||||
- QPI-before-IA gate
|
||||
- depth-limiting override gate
|
||||
|
||||
Proceed to Round 04 blind input routing evaluation.
|
||||
```
|
||||
|
||||
`draft-callable` is review language only. It does not change JSON lifecycle status. Validation pass, schema compliance, selector demo pass, or report-level acceptance cannot upgrade either model to stable.
|
||||
|
||||
## 2. Round 04 Target
|
||||
|
||||
Round 04 verifies:
|
||||
|
||||
```text
|
||||
真实新输入下，selector + QPI + IA gate 的路由是否稳定。
|
||||
```
|
||||
|
||||
This means:
|
||||
|
||||
- selected model IDs
|
||||
- rejected model IDs
|
||||
- no-call behavior
|
||||
- model-level hard exclusions
|
||||
- QPI-before-IA gate behavior
|
||||
- depth-limiting override behavior
|
||||
|
||||
It does not mean final answer quality.
|
||||
|
||||
## 3. Round 04 Non-Goals
|
||||
|
||||
```text
|
||||
Do not add a third model.
|
||||
Do not upgrade QPI or Intellectual Archaeology.
|
||||
Do not introduce LLM selector.
|
||||
Do not implement RAG, vector database, frontend, backend, user account, or QA product.
|
||||
Do not modify selector rules before blind test.
|
||||
Run first, analyze failures later.
|
||||
```
|
||||
|
||||
## 4. Current Runtime Asset Set
|
||||
|
||||
The optional raw zip includes:
|
||||
|
||||
```text
|
||||
selector/selector_rules.json
|
||||
selector/selector_calibration_inputs.json
|
||||
selector/round04_blind_inputs.json
|
||||
models/qpi.model.json
|
||||
models/intellectual_archaeology.model.json
|
||||
models/model_index.json
|
||||
cards/qpi.md
|
||||
cards/intellectual_archaeology.md
|
||||
cards/card_index.md
|
||||
tests/qpi.regression.json
|
||||
tests/intellectual_archaeology.regression.json
|
||||
tests/regression_cases.json
|
||||
tests/test_round04_blind_routing.py
|
||||
scripts/run_selector_demo.py
|
||||
scripts/run_selector_regression.py
|
||||
scripts/run_selector_calibration_smoke.py
|
||||
scripts/run_round04_blind_routing.py
|
||||
reports/Round04_blind_input_candidates_2026-06-18.md
|
||||
reports/Round04_blind_routing_evaluation_report_2026-06-18.md
|
||||
reports/selector_regression_report_v0.2.md
|
||||
reports/selector_calibration_smoke_report.md
|
||||
reports/validation_report.md
|
||||
reports/index_rebuild_report.md
|
||||
reports/model_card_sync_report_v0.2.md
|
||||
```
|
||||
|
|
@ -0,0 +1,48 @@
|
|||
# Round 04 Blind Input Set
|
||||
|
||||
Source: `selector/round04_blind_inputs.json`
|
||||
|
||||
Human-readable source: `reports/Round04_blind_input_candidates_2026-06-18.md`
|
||||
|
||||
Status: `frozen_for_first_blind_run_after_owner_review`
|
||||
|
||||
## 1. Blindness Rules
|
||||
|
||||
- No expected routing labels are included.
|
||||
- Category labels are coverage labels, not answer keys.
|
||||
- The input pool must not be merged into `selector/selector_calibration_inputs.json` before review.
|
||||
- Selector rules must not be modified before the first blind run.
|
||||
|
||||
## 2. Count
|
||||
|
||||
Total inputs: 38
|
||||
|
||||
Control notes:
|
||||
|
||||
- `R04-BI-018`: positive control for obvious model-extraction / depth-modeling intent.
|
||||
- `R04-BI-021`: model-name-exposed control case.
|
||||
- `R04-BI-025`: model-name-exposed control case.
|
||||
- `R04-BI-026`: high-value natural boundary sample, because it avoids explicit model names while suggesting IA or QPI-to-IA routing.
|
||||
|
||||
## 3. Category Coverage
|
||||
|
||||
- explicit no-call fact / translation / rewrite / formatting / direct execution
|
||||
- ordinary how-to no-call traps
|
||||
- dictionary / word-meaning judgment no-call trap
|
||||
- QPI lightweight problem definition
|
||||
- governance / responsibility / consensus
|
||||
- capacity / resource / path
|
||||
- model extraction / depth modeling
|
||||
- IA refused but QPI allowed
|
||||
- depth-limiting but QPI allowed
|
||||
- likely Intellectual Archaeology
|
||||
- ambiguous / low context
|
||||
- ambiguous growth how-to boundary
|
||||
|
||||
## 4. Full Input IDs
|
||||
|
||||
```text
|
||||
R04-BI-001 through R04-BI-038
|
||||
```
|
||||
|
||||
Use `selector/round04_blind_inputs.json` for exact input text and metadata.
|
||||
|
|
@ -0,0 +1,771 @@
|
|||
# Round 04 Blind Routing Evaluation Report
|
||||
|
||||
Date: 2026-06-18
|
||||
|
||||
Blind input set: `round04_blind_inputs_2026-06-18_v0.1`
|
||||
|
||||
Expected routing labels: `not included before first blind run`
|
||||
|
||||
This report records current selector routing only. It does not score correctness, does not generate final answers, does not modify selector rules, and does not upgrade model status.
|
||||
|
||||
## Scope
|
||||
|
||||
- Round 04 target: test routing stability on new inputs.
|
||||
- Runtime surface: current rule-based selector plus model-level QPI / IA gates.
|
||||
- QPI contract status: represented here by QPI selection and routing details; no full QPI answer generation is performed.
|
||||
- IA status: represented here by selector selection / rejection only; no Intellectual Archaeology answer generation is performed.
|
||||
|
||||
## Summary
|
||||
|
||||
Inputs evaluated: 38
|
||||
No-call results: 20
|
||||
|
||||
### Selected Model Distribution
|
||||
|
||||
- `intellectual_archaeology`: 3
|
||||
- `none`: 20
|
||||
- `qpi`: 18
|
||||
|
||||
### Coverage Categories
|
||||
|
||||
- `ambiguous_growth_how_to`: 1
|
||||
- `ambiguous_low_context`: 3
|
||||
- `capacity_resource_path`: 4
|
||||
- `direct_execution_with_analysis_override`: 1
|
||||
- `explicit_no_call_dictionary_judgment`: 1
|
||||
- `explicit_no_call_direct_execution`: 1
|
||||
- `explicit_no_call_fact_lookup`: 1
|
||||
- `explicit_no_call_formatting`: 1
|
||||
- `explicit_no_call_how_to`: 2
|
||||
- `explicit_no_call_rewrite`: 1
|
||||
- `explicit_no_call_translation`: 1
|
||||
- `governance_responsibility_consensus`: 4
|
||||
- `likely_intellectual_archaeology`: 4
|
||||
- `model_extraction_or_depth_modeling`: 3
|
||||
- `qpi_allowed_depth_limited`: 2
|
||||
- `qpi_allowed_ia_refused`: 4
|
||||
- `qpi_light_problem_definition`: 4
|
||||
|
||||
## Case Results
|
||||
|
||||
### R04-BI-001
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_fact_lookup`
|
||||
- input_text: `查一下《原则》这本书的作者是谁，只要事实，不要分析。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- no_call: `True`
|
||||
- routing_notes: hard no-call gate matched; explicit analysis override was not present.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 只要事实、查一下
|
||||
- `qpi` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 只要事实、查一下
|
||||
|
||||
### R04-BI-002
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_translation`
|
||||
- input_text: `把这句话翻译成英文：这个模型还处在草稿阶段。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.49 decision=selected
|
||||
- reasons: complexity signal: 模型; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.17 decision=rejected
|
||||
- reasons: complexity signal: 模型; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-003
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_rewrite`
|
||||
- input_text: `请把这段话润色得更顺一点，不要改变意思，也不要解释原因。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- no_call: `True`
|
||||
- routing_notes: hard no-call gate matched; explicit analysis override was not present.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 润色
|
||||
- `qpi` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 润色
|
||||
|
||||
### R04-BI-004
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_formatting`
|
||||
- input_text: `把下面这组要点整理成 Markdown 表格，直接给结果。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-005
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_direct_execution`
|
||||
- input_text: `把这个 JSON 压缩成单行，不用说明：{"status":"draft","level":"B"}`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-006
|
||||
|
||||
- category_for_owner_review: `qpi_light_problem_definition`
|
||||
- input_text: `我们现在反复开会但没有推进，我想先判断这是缺信息、缺路径，还是缺共识。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.99 decision=selected
|
||||
- reasons: trigger keyword matched: 缺路径、缺共识; complexity signal: 反复、共识; selection_priority=9; QPI problem-definition gate matched
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.0 decision=rejected
|
||||
- reasons: complexity signal: 反复、共识; selection_priority=7
|
||||
- penalties: QPI-before-IA gate: 缺路径、缺共识; no IA heavy-depth gate
|
||||
|
||||
### R04-BI-007
|
||||
|
||||
- category_for_owner_review: `qpi_light_problem_definition`
|
||||
- input_text: `这个项目卡住了，我不确定到底是问题没定义清楚，还是执行方案不够。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.59 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-008
|
||||
|
||||
- category_for_owner_review: `qpi_light_problem_definition`
|
||||
- input_text: `我们有一堆材料，但不知道下一步该问什么问题，先帮我定性一下。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.59 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-009
|
||||
|
||||
- category_for_owner_review: `qpi_light_problem_definition`
|
||||
- input_text: `这件事到底算一个简单问题、复杂问题，还是已经变成组织议题？`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.89 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; input type matched: 复杂问题; complexity signal: 组织、复杂; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.32 decision=rejected
|
||||
- reasons: input type matched: 复杂问题; complexity signal: 组织、复杂; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-010
|
||||
|
||||
- category_for_owner_review: `governance_responsibility_consensus`
|
||||
- input_text: `每个部门都说支持这个项目，但一到分工就没人愿意承担主责。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-011
|
||||
|
||||
- category_for_owner_review: `governance_responsibility_consensus`
|
||||
- input_text: `老板要创新，财务要控成本，业务要短期业绩，最后方案总是被折中到没有方向。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-012
|
||||
|
||||
- category_for_owner_review: `governance_responsibility_consensus`
|
||||
- input_text: `我们不是不知道怎么做，而是每个人都在等别人先表态。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-013
|
||||
|
||||
- category_for_owner_review: `governance_responsibility_consensus`
|
||||
- input_text: `这个流程改造牵涉教务、学院和信息化部门，谁都能提意见，但没人能拍板。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-014
|
||||
|
||||
- category_for_owner_review: `capacity_resource_path`
|
||||
- input_text: `团队现在知道目标，也认可方向，但只有两个人能做关键模块，进度一直被产能卡住。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-015
|
||||
|
||||
- category_for_owner_review: `capacity_resource_path`
|
||||
- input_text: `我们知道要做知识库，也有材料，但不知道从哪条最小路径开始。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-016
|
||||
|
||||
- category_for_owner_review: `capacity_resource_path`
|
||||
- input_text: `预算不是最大问题，问题是现有系统、权限和审批流程都接不上。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.74 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; complexity signal: 系统; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.17 decision=rejected
|
||||
- reasons: complexity signal: 系统; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-017
|
||||
|
||||
- category_for_owner_review: `capacity_resource_path`
|
||||
- input_text: `这件事大家都觉得重要，但每周只能挤出零散时间，所以一直没有稳定产出。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-018
|
||||
|
||||
- category_for_owner_review: `model_extraction_or_depth_modeling`
|
||||
- control_case_type: `positive_control`
|
||||
- input_text: `请从这篇长文里抽取一个可复用的认知模型，重点看它的核心机制和适用边界。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.49 decision=selected
|
||||
- reasons: complexity signal: 模型、机制、抽取、边界; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.17 decision=rejected
|
||||
- reasons: complexity signal: 模型、机制、抽取、边界; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-019
|
||||
|
||||
- category_for_owner_review: `model_extraction_or_depth_modeling`
|
||||
- input_text: `这套说法背后好像有一个稳定结构，不只是观点总结，帮我判断它能不能沉淀成模型。`
|
||||
- selected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- rejected_models: `[]`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.49 decision=selected
|
||||
- reasons: complexity signal: 模型、结构; selection_priority=9
|
||||
- `intellectual_archaeology` score=0.42 decision=selected
|
||||
- reasons: trigger keyword matched: 沉淀成模型; complexity signal: 模型、结构; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
Rejected model details:
|
||||
- none
|
||||
|
||||
### R04-BI-020
|
||||
|
||||
- category_for_owner_review: `model_extraction_or_depth_modeling`
|
||||
- input_text: `我想把这段讨论拆成原始判断、隐含假设、适用条件和误用边界。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.49 decision=selected
|
||||
- reasons: complexity signal: 边界; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.17 decision=rejected
|
||||
- reasons: complexity signal: 边界; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-021
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_ia_refused`
|
||||
- control_case_type: `model_name_exposed`
|
||||
- input_text: `先不要思想考古，只判断这是不是一个 QPI 意义上的问题。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.59 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.0 decision=rejected
|
||||
- reasons: trigger keyword matched: 思想考古; selection_priority=7; IA heavy-depth signal: 思想考古
|
||||
- penalties: negative trigger: 不要思想考古、先不要思想考古; model hard exclusion signal: 不要思想考古、先不要思想考古
|
||||
|
||||
### R04-BI-022
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_ia_refused`
|
||||
- input_text: `不要展开到底层假设，只帮我看主导稀缺是什么。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- no_call: `True`
|
||||
- routing_notes: hard no-call gate matched; explicit analysis override was not present.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 不要展开
|
||||
- `qpi` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 不要展开
|
||||
|
||||
### R04-BI-023
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_ia_refused`
|
||||
- input_text: `这次先不做深度建模，只做问题定性，看它是缺数据、缺路径还是缺共识。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.99 decision=selected
|
||||
- reasons: trigger keyword matched: 问题、缺数据、缺路径、缺共识; complexity signal: 共识; selection_priority=9; QPI problem-definition gate matched
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.0 decision=rejected
|
||||
- reasons: complexity signal: 共识; selection_priority=7
|
||||
- penalties: QPI-before-IA gate: 缺数据、缺路径、缺共识; no IA heavy-depth gate; model hard exclusion signal: 只做问题定性
|
||||
|
||||
### R04-BI-024
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_ia_refused`
|
||||
- input_text: `不要进入思想考古，我只要一个轻量判断：现在应该先问问题、找路径，还是处理责任边界？`
|
||||
- selected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- rejected_models: `[]`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `intellectual_archaeology` score=0.97 decision=selected
|
||||
- reasons: trigger keyword matched: 思想考古; complexity signal: 边界、责任; selection_priority=7; IA heavy-depth signal: 思想考古
|
||||
- `qpi` score=0.74 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; complexity signal: 边界、责任; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- none
|
||||
|
||||
### R04-BI-025
|
||||
|
||||
- category_for_owner_review: `likely_intellectual_archaeology`
|
||||
- control_case_type: `model_name_exposed`
|
||||
- input_text: `QPI 已经判断这是中重型课题，现在请继续分析它背后的底层假设、生成机制和长期稳定结构。`
|
||||
- selected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- rejected_models: `[]`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `intellectual_archaeology` score=0.97 decision=selected
|
||||
- reasons: trigger keyword matched: 底层假设; complexity signal: 长期、底层、结构、机制; selection_priority=7; IA heavy-depth signal: 底层假设
|
||||
- `qpi` score=0.74 decision=selected
|
||||
- reasons: trigger keyword matched: 课题; complexity signal: 长期、底层、结构、机制; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- none
|
||||
|
||||
### R04-BI-026
|
||||
|
||||
- category_for_owner_review: `likely_intellectual_archaeology`
|
||||
- input_text: `这个现象反复出现在不同项目里，我想知道它背后的思想结构是什么，而不是只做问题分类。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.74 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; complexity signal: 反复、结构; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.17 decision=rejected
|
||||
- reasons: complexity signal: 反复、结构; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-027
|
||||
|
||||
- category_for_owner_review: `likely_intellectual_archaeology`
|
||||
- input_text: `请分析这套判断为什么会长期成立，它依赖哪些世界观、因果假设和行动边界。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.49 decision=selected
|
||||
- reasons: complexity signal: 长期、边界; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.17 decision=rejected
|
||||
- reasons: complexity signal: 长期、边界; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-028
|
||||
|
||||
- category_for_owner_review: `ambiguous_low_context`
|
||||
- input_text: `这个不太对。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-029
|
||||
|
||||
- category_for_owner_review: `ambiguous_low_context`
|
||||
- input_text: `怎么又卡住了？`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-030
|
||||
|
||||
- category_for_owner_review: `ambiguous_low_context`
|
||||
- input_text: `还是那个老问题。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.59 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-031
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_how_to`
|
||||
- input_text: `如何在 Excel 里冻结首行？`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-032
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_how_to`
|
||||
- input_text: `怎么把 Markdown 文件转成 PDF？`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-033
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_dictionary_judgment`
|
||||
- input_text: `判断一下“alignment”在这句话里是什么意思。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-034
|
||||
|
||||
- category_for_owner_review: `direct_execution_with_analysis_override`
|
||||
- input_text: `请把下面内容总结成三点，但不要只执行，先判断它背后的问题定义。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.59 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-035
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_depth_limited`
|
||||
- input_text: `不要深入分析，只判断这件事的主导稀缺是什么。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- no_call: `True`
|
||||
- routing_notes: hard no-call gate matched; explicit analysis override was not present.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 不要深入分析
|
||||
- `qpi` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 不要深入分析
|
||||
|
||||
### R04-BI-036
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_depth_limited`
|
||||
- input_text: `不要展开到底层理论，只判断现在是缺信息、缺路径还是缺共识。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- no_call: `True`
|
||||
- routing_notes: hard no-call gate matched; explicit analysis override was not present.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 不要展开
|
||||
- `qpi` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 不要展开
|
||||
|
||||
### R04-BI-037
|
||||
|
||||
- category_for_owner_review: `ambiguous_growth_how_to`
|
||||
- input_text: `如何提高这个账号的流量？`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-038
|
||||
|
||||
- category_for_owner_review: `likely_intellectual_archaeology`
|
||||
- input_text: `这类失败在不同项目里反复出现，我想抽出它背后稳定的生成机制和可迁移框架。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.49 decision=selected
|
||||
- reasons: complexity signal: 反复、机制; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.17 decision=rejected
|
||||
- reasons: complexity signal: 反复、机制; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
## Review Reminder
|
||||
|
||||
This is the first blind routing output. Failure labels, expected routing, and rule changes should be added only after owner / GPT / CCRA review of these observed results.
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
# Round 04 Review Questions for GPT / CCRA
|
||||
|
||||
## 1. Scope Check
|
||||
|
||||
1. Did Round 04 preserve the rule that selector rules are not modified before the first blind run?
|
||||
2. Did the blind input set avoid expected routing labels before first execution?
|
||||
3. Is it acceptable that model-name-exposed control cases are retained as controls rather than natural blind inputs?
|
||||
|
||||
## 2. Routing Evaluation
|
||||
|
||||
For each observed routing result, classify failures or risks as needed:
|
||||
|
||||
- selector false positive
|
||||
- selector false negative
|
||||
- no-call false positive
|
||||
- no-call false negative
|
||||
- QPI over-selection
|
||||
- QPI under-selection
|
||||
- IA over-selection
|
||||
- IA under-selection
|
||||
- QPI-before-IA gate failure
|
||||
- model-level exclusion failure
|
||||
- depth-limiting override failure
|
||||
- insufficient-context handling risk
|
||||
- control-case-only behavior, not natural-input behavior
|
||||
|
||||
## 3. Specific Review Prompts
|
||||
|
||||
1. Which Round 04 cases should become regression cases?
|
||||
2. Which cases should become calibration inputs?
|
||||
3. Which cases should remain as non-regression discussion examples?
|
||||
4. Does any observed routing require a selector rule patch?
|
||||
5. If a rule patch is needed, should it be Round 04.1 rather than part of Round 04?
|
||||
6. Does any result imply that QPI or Intellectual Archaeology model content needs revision?
|
||||
7. Are any blind inputs too artificial and should be removed from future rounds?
|
||||
|
||||
## 4. Lifecycle Guard
|
||||
|
||||
Do not upgrade QPI or Intellectual Archaeology model lifecycle status from this Round 04 result alone.
|
||||
|
||||
Model status changes still require evidence review, content review, regression review, selector review, and owner / CCRA review.
|
||||
|
|
@ -0,0 +1,56 @@
|
|||
# Round 04 Bundle File Manifest
|
||||
|
||||
## Review Files
|
||||
|
||||
- `00_OPEN_THIS_FIRST_CCRA_REVIEW_BRIEF_04.md`
|
||||
- `01_ROUND03_CLOSURE_AND_ROUND04_SCOPE_04.md`
|
||||
- `02_BLIND_INPUT_SET_04.md`
|
||||
- `03_ROUTING_EVALUATION_REPORT_04.md`
|
||||
- `04_REVIEW_QUESTIONS_FOR_GPT_04.md`
|
||||
- `BUNDLE_FILE_MANIFEST_04.md`
|
||||
- `optional_raw_changed_files_04.zip`
|
||||
|
||||
## Optional Raw Changed Files
|
||||
|
||||
The zip is optional and should be used only if exact file inspection is needed.
|
||||
|
||||
It preserves source-relative paths.
|
||||
|
||||
It excludes:
|
||||
|
||||
- `knowledge_assets/`
|
||||
- `__pycache__/`
|
||||
- `*.pyc`
|
||||
- previous review bundle zip files
|
||||
|
||||
## Source-Relative Files Included in Raw Zip
|
||||
|
||||
- `README.md`
|
||||
- `selector/README.md`
|
||||
- `selector/selector_rules.json`
|
||||
- `selector/selector_calibration_inputs.json`
|
||||
- `selector/round04_blind_inputs.json`
|
||||
- `models/qpi.model.json`
|
||||
- `models/intellectual_archaeology.model.json`
|
||||
- `models/model_index.json`
|
||||
- `cards/qpi.md`
|
||||
- `cards/intellectual_archaeology.md`
|
||||
- `cards/card_index.md`
|
||||
- `tests/qpi.regression.json`
|
||||
- `tests/intellectual_archaeology.regression.json`
|
||||
- `tests/regression_cases.json`
|
||||
- `tests/test_round04_blind_routing.py`
|
||||
- `scripts/README.md`
|
||||
- `scripts/run_selector_demo.py`
|
||||
- `scripts/run_selector_regression.py`
|
||||
- `scripts/run_selector_calibration_smoke.py`
|
||||
- `scripts/run_round04_blind_routing.py`
|
||||
- `reports/README.md`
|
||||
- `reports/Round04_blind_input_candidates_2026-06-18.md`
|
||||
- `reports/Round04_blind_routing_evaluation_report_2026-06-18.md`
|
||||
- `reports/selector_regression_report_v0.2.md`
|
||||
- `reports/selector_calibration_smoke_report.md`
|
||||
- `reports/validation_report.md`
|
||||
- `reports/index_rebuild_report.md`
|
||||
- `reports/model_card_sync_report_v0.2.md`
|
||||
- `docs/superpowers/plans/2026-06-18-round04-blind-routing-evaluation.md`
|
||||
Binary file not shown.
|
|
@ -214,3 +214,36 @@ Closure statement:
|
|||
- QPI is accepted as a draft-callable routing asset at report level only; this does not change JSON model lifecycle status.
|
||||
- Selector v0.2 passes no-call, calibration smoke, model-level exclusion, QPI-before-IA, and depth-limiting override gates.
|
||||
- Round 04 blind input routing evaluation is the next candidate phase, but it is not started by this closure.
|
||||
|
||||
## Decision 017: Treat Round 04.1 as selector rule patch verification, not a second blind test
|
||||
|
||||
Status: Accepted
|
||||
|
||||
Reason:
|
||||
|
||||
Round 04 first-run blind routing evaluation found review-approved selector failures:
|
||||
|
||||
- R04-BI-002: translation task over-selected QPI because payload text contained `模型`.
|
||||
- R04-BI-024: explicit `不要进入思想考古` did not hard-exclude Intellectual Archaeology.
|
||||
- R04-BI-035 / R04-BI-036: depth-limiting phrases blocked lightweight QPI scarcity judgment.
|
||||
|
||||
Round 04.1 therefore patches selector rules and verifies the patch against frozen Round 04 inputs. It is not a second blind test because expected labels now exist for the four owner / GPT reviewed target cases.
|
||||
|
||||
The patch remains scoped:
|
||||
|
||||
- It adds translation / direct transformation no-call coverage.
|
||||
- It separates task instruction from payload for no-call matching before payload complexity scoring.
|
||||
- It expands IA model-level hard exclusion variants.
|
||||
- It expands depth-limiting QPI override phrases for `主导稀缺` and `缺信息 / 缺路径 / 缺共识`.
|
||||
- It does not add a new model, lifecycle upgrade, LLM selector, RAG, database, service, frontend, backend, user system, or QA product.
|
||||
|
||||
QPI and Intellectual Archaeology remain `draft / B / pending`.
|
||||
|
||||
Closeout:
|
||||
|
||||
- GPT review accepted Round 04.1 and did not request a 04.2 selector patch.
|
||||
- `R04-BI-022` is documented as an accepted collateral behavior change: it moved from no-call to lightweight QPI after the depth-limiting override patch.
|
||||
- This is not a new failure because the user rejects deeper IA-style expansion but asks for dominant-scarcity judgment.
|
||||
- IA remains rejected for `R04-BI-022`.
|
||||
- `R04-BI-022` is added to selector regression as a QPI gate / IA anti-recall trap.
|
||||
- Closing Round 04.1 does not upgrade either model lifecycle status.
|
||||
|
|
|
|||
|
|
@ -74,6 +74,20 @@ python scripts\rebuild_indexes.py --check
|
|||
python scripts\validate_model_library.py
|
||||
```
|
||||
|
||||
## 5.1 Review Bundle Zip Workflow
|
||||
|
||||
When creating optional raw changed-file zips for GPT / CCRA review bundles, use the installed `bundle-zip` Skill instead of ad hoc PowerShell or Python compression.
|
||||
|
||||
Required properties:
|
||||
|
||||
1. Provide an explicit file list.
|
||||
2. Use the repository root marker, for example `the-mindscape-of-bro-tsong`.
|
||||
3. Preserve source-relative paths inside the zip.
|
||||
4. Verify the zip by reading it back.
|
||||
5. Keep the `bundle-zip` JSON summary with the review bundle when useful.
|
||||
|
||||
Do not use `Compress-Archive` for review raw zips where directory paths matter; it can flatten file-list paths.
|
||||
|
||||
## 6. Supplier Request Workflow
|
||||
|
||||
When this repository needs a capability owned by a neighboring repository:
|
||||
|
|
|
|||
|
|
@ -0,0 +1,108 @@
|
|||
# Round 04.1 Selector Rule Patch Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Fix Round 04 selector boundary failures without changing model lifecycle status or introducing an LLM selector.
|
||||
|
||||
**Architecture:** Add targeted regression cases from Round 04 review, patch rule-based selector configuration and minimal selector logic, rerun the frozen Round 04 input pool as post-patch verification, and package 04.1 review materials. The original Round 04 blind output remains frozen; 04.1 produces comparison reports.
|
||||
|
||||
**Tech Stack:** JSON, Markdown, Python standard library, existing selector scripts, installed `bundle-zip` Skill.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Do not change the original Round 04 blind routing report.
|
||||
- Do not introduce an LLM selector.
|
||||
- Do not add a third model.
|
||||
- Do not upgrade QPI or Intellectual Archaeology.
|
||||
- Do not implement RAG, vector database, frontend, backend, user account, or QA product.
|
||||
- Treat Round 04.1 as post-patch routing verification, not a second blind test.
|
||||
- Use `bundle-zip` for source-relative review bundle zips.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add Targeted Regression Cases
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/qpi.regression.json`
|
||||
- Modify: `tests/intellectual_archaeology.regression.json`
|
||||
- Modify: `tests/regression_cases.json`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: Round 04 review cases R04-BI-002, R04-BI-024, R04-BI-035, R04-BI-036.
|
||||
- Produces: selector regression failures before patch.
|
||||
|
||||
- [ ] Add R04-BI-002 as a QPI no-call regression.
|
||||
- [ ] Add R04-BI-024 as an IA hard-exclusion regression with QPI allowed.
|
||||
- [ ] Add R04-BI-035 and R04-BI-036 as depth-limiting QPI override regressions.
|
||||
- [ ] Run `python scripts\run_selector_regression.py` and confirm expected failures.
|
||||
|
||||
### Task 2: Patch Selector Rules and Minimal Logic
|
||||
|
||||
**Files:**
|
||||
- Modify: `selector/selector_rules.json`
|
||||
- Modify if needed: `scripts/run_selector_demo.py`
|
||||
- Modify: `selector/README.md`
|
||||
- Modify: `docs/DECISIONS.md`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: failing regression cases from Task 1.
|
||||
- Produces: selector behavior that passes targeted regressions.
|
||||
|
||||
- [ ] Add translation no-call intent phrases beyond one exact wording.
|
||||
- [ ] Add instruction / payload separation for direct transformation tasks before scoring payload complexity.
|
||||
- [ ] Add IA hard-exclusion variants such as `不要进入思想考古`.
|
||||
- [ ] Add depth-limiting QPI override phrases for `主导稀缺` and `缺信息 / 缺路径 / 缺共识` forms.
|
||||
- [ ] Document the rule patch.
|
||||
|
||||
### Task 3: Post-Patch Routing Comparison
|
||||
|
||||
**Files:**
|
||||
- Create: `scripts/run_round04_post_patch_verification.py`
|
||||
- Create: `reports/Round04_1_post_patch_routing_verification_report_2026-06-18.md`
|
||||
- Test: `tests/test_round04_blind_routing.py` or new focused test if needed.
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: frozen `selector/round04_blind_inputs.json` and frozen Round 04 report.
|
||||
- Produces: before / after comparison for targeted expected routing labels.
|
||||
|
||||
- [ ] Generate current post-patch results for all 38 frozen inputs.
|
||||
- [ ] Compare targeted cases R04-BI-002, R04-BI-024, R04-BI-035, R04-BI-036 against expected labels.
|
||||
- [ ] Include original selected models / no-call from Round 04 and new selected models / no-call.
|
||||
|
||||
### Task 4: Build Round 04.1 Review Bundle
|
||||
|
||||
**Files:**
|
||||
- Create: `ccra_review_bundle/round-04.1_2026-06-18_selector-rule-patch/00_OPEN_THIS_FIRST_CCRA_REVIEW_BRIEF_04.1.md`
|
||||
- Create: `ccra_review_bundle/round-04.1_2026-06-18_selector-rule-patch/01_RULE_PATCH_SUMMARY_AND_DIFF_04.1.md`
|
||||
- Create: `ccra_review_bundle/round-04.1_2026-06-18_selector-rule-patch/02_EXPECTED_ROUTING_LABELS_04.1.md`
|
||||
- Create: `ccra_review_bundle/round-04.1_2026-06-18_selector-rule-patch/03_POST_PATCH_ROUTING_VERIFICATION_REPORT_04.1.md`
|
||||
- Create: `ccra_review_bundle/round-04.1_2026-06-18_selector-rule-patch/04_REGRESSION_AND_COMMAND_LOG_04.1.md`
|
||||
- Create: `ccra_review_bundle/round-04.1_2026-06-18_selector-rule-patch/BUNDLE_FILE_MANIFEST_04.1.md`
|
||||
- Create: `ccra_review_bundle/round-04.1_2026-06-18_selector-rule-patch/optional_raw_changed_files_04.1.zip`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: patch diffs, post-patch report, validation output, source file list.
|
||||
- Produces: GPT / CCRA review package.
|
||||
|
||||
- [ ] Write required review files.
|
||||
- [ ] Use `bundle-zip` to create source-relative raw zip.
|
||||
- [ ] Verify zip output using `bundle-zip` JSON summary.
|
||||
|
||||
### Task 5: Verify
|
||||
|
||||
**Files:**
|
||||
- Read: full changed set.
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: completed 04.1 patch and review bundle.
|
||||
- Produces: verification evidence.
|
||||
|
||||
- [ ] Run `python scripts\rebuild_indexes.py --check`.
|
||||
- [ ] Run `python -m unittest discover -s tests -p "test*.py" -v`.
|
||||
- [ ] Run `python scripts\validate_model_library.py`.
|
||||
- [ ] Run `python scripts\check_card_contract.py`.
|
||||
- [ ] Run `python scripts\run_selector_demo.py`.
|
||||
- [ ] Run `python scripts\run_selector_regression.py`.
|
||||
- [ ] Run `python scripts\run_selector_calibration_smoke.py`.
|
||||
- [ ] Run `python scripts\check_model_card_sync.py`.
|
||||
- [ ] Confirm model JSON status remains `draft / B / pending`.
|
||||
|
|
@ -0,0 +1,172 @@
|
|||
# Round 04 Blind Routing Evaluation Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Prepare and run Round 04 blind input routing evaluation without changing selector rules or model lifecycle status.
|
||||
|
||||
**Architecture:** Round 04 uses a reviewed blind input pool as source material, freezes it into a machine-readable JSON input set, runs the existing rule-based selector through a thin batch runner, and packages the outputs for GPT / CCRA review. The runner reuses `scripts/run_selector_demo.py::recommend` and does not add answer generation, LLM selector logic, or expected routing labels.
|
||||
|
||||
**Tech Stack:** Markdown, JSON, Python standard library, existing selector scripts.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Do not modify `selector/selector_rules.json` before the blind evaluation.
|
||||
- Do not merge blind inputs into `selector/selector_calibration_inputs.json`.
|
||||
- Do not add expected behavior before the first blind run.
|
||||
- Do not add a third model.
|
||||
- Do not upgrade QPI or Intellectual Archaeology.
|
||||
- Do not introduce an LLM selector.
|
||||
- Do not implement RAG, vector database, frontend, backend, user account, or QA product.
|
||||
- Run first, analyze failures later.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Revise Human-Readable Blind Input Source
|
||||
|
||||
**Files:**
|
||||
- Modify: `reports/Round04_blind_input_candidates_2026-06-18.md`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: owner review notes for R04-BI-018, R04-BI-021, R04-BI-025, R04-BI-026, and new R04-BI-031 through R04-BI-038.
|
||||
- Produces: 38 candidate inputs with review-only metadata and no expected routing labels.
|
||||
|
||||
- [ ] **Step 1: Mark control-case notes**
|
||||
|
||||
Add metadata notes:
|
||||
- R04-BI-021 and R04-BI-025: `model-name-exposed control case`
|
||||
- R04-BI-018: `positive control`
|
||||
- R04-BI-026: natural IA / QPI-to-IA boundary sample
|
||||
|
||||
- [ ] **Step 2: Add R04-BI-031 through R04-BI-038**
|
||||
|
||||
Add the eight owner-proposed cases exactly, preserving their categories, input text, and inclusion rationale.
|
||||
|
||||
- [ ] **Step 3: Update coverage**
|
||||
|
||||
Set total to 38 candidate inputs and include a short note that 2 inputs expose model names by design.
|
||||
|
||||
### Task 2: Freeze Machine-Readable Blind Input JSON
|
||||
|
||||
**Files:**
|
||||
- Create: `selector/round04_blind_inputs.json`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: reviewed Markdown source from Task 1.
|
||||
- Produces: JSON with `blind_input_set_id`, `status`, `source_document`, `constraints`, and `inputs`.
|
||||
|
||||
- [ ] **Step 1: Create JSON without expected routing**
|
||||
|
||||
Each input object contains:
|
||||
- `input_id`
|
||||
- `category_for_owner_review`
|
||||
- `input_text`
|
||||
- `why_included`
|
||||
- optional `control_case_type`
|
||||
|
||||
- [ ] **Step 2: Preserve blindness constraints**
|
||||
|
||||
Include top-level notes that no expected behavior is present and that selector rules must not be modified before first run.
|
||||
|
||||
### Task 3: Add Failing Tests for the Round 04 Runner
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/test_round04_blind_routing.py`
|
||||
- Create later in Task 4: `scripts/run_round04_blind_routing.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `selector/round04_blind_inputs.json`
|
||||
- Produces: test expectations for load, evaluation shape, and report content.
|
||||
|
||||
- [ ] **Step 1: Write tests before implementation**
|
||||
|
||||
Test that a blind input item can be loaded, evaluated through `recommend`, and rendered without expected routing labels.
|
||||
|
||||
- [ ] **Step 2: Run tests and confirm failure**
|
||||
|
||||
Run: `python -m unittest tests.test_round04_blind_routing -v`
|
||||
|
||||
Expected: failure because `scripts.run_round04_blind_routing` does not exist.
|
||||
|
||||
### Task 4: Implement and Run the Round 04 Runner
|
||||
|
||||
**Files:**
|
||||
- Create: `scripts/run_round04_blind_routing.py`
|
||||
- Create: `reports/Round04_blind_routing_evaluation_report_2026-06-18.md`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `selector/round04_blind_inputs.json`, `scripts/run_selector_demo.py::recommend`
|
||||
- Produces: JSON-shaped evaluation results and Markdown report.
|
||||
|
||||
- [ ] **Step 1: Implement standard-library runner**
|
||||
|
||||
Provide functions:
|
||||
- `load_blind_inputs(root)`
|
||||
- `evaluate_blind_input(root, item)`
|
||||
- `write_report(root, input_set, results)`
|
||||
- `main()`
|
||||
|
||||
- [ ] **Step 2: Run focused tests**
|
||||
|
||||
Run: `python -m unittest tests.test_round04_blind_routing -v`
|
||||
|
||||
- [ ] **Step 3: Run blind evaluation**
|
||||
|
||||
Run: `python scripts\run_round04_blind_routing.py`
|
||||
|
||||
Expected: report written; command exits 0.
|
||||
|
||||
### Task 5: Build Round 04 Review Bundle
|
||||
|
||||
**Files:**
|
||||
- Create: `ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/00_OPEN_THIS_FIRST_CCRA_REVIEW_BRIEF_04.md`
|
||||
- Create: `ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/01_ROUND03_CLOSURE_AND_ROUND04_SCOPE_04.md`
|
||||
- Create: `ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/02_BLIND_INPUT_SET_04.md`
|
||||
- Create: `ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/03_ROUTING_EVALUATION_REPORT_04.md`
|
||||
- Create: `ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/04_REVIEW_QUESTIONS_FOR_GPT_04.md`
|
||||
- Create: `ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/BUNDLE_FILE_MANIFEST_04.md`
|
||||
- Create: `ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/optional_raw_changed_files_04.zip`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: Round 03 closure summary, frozen blind input JSON, runner output report, current runtime assets.
|
||||
- Produces: GPT / CCRA review package.
|
||||
|
||||
- [ ] **Step 1: Write brief and scope files**
|
||||
|
||||
Include Round 03 closure summary, Round 04 target, and non-goals.
|
||||
|
||||
- [ ] **Step 2: Write input and evaluation mirrors**
|
||||
|
||||
Mirror the frozen blind input set and routing evaluation report for upload.
|
||||
|
||||
- [ ] **Step 3: Create raw zip**
|
||||
|
||||
Include current runtime assets, blind input set, runner, tests, and reports. Preserve source-relative paths.
|
||||
|
||||
### Task 6: Verify
|
||||
|
||||
**Files:**
|
||||
- Read: all changed files
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: full working tree after Tasks 1-5.
|
||||
- Produces: verification evidence.
|
||||
|
||||
- [ ] **Step 1: Run focused Round 04 test**
|
||||
|
||||
Run: `python -m unittest tests.test_round04_blind_routing -v`
|
||||
|
||||
- [ ] **Step 2: Run existing validation gates**
|
||||
|
||||
Run:
|
||||
- `python scripts\rebuild_indexes.py --check`
|
||||
- `python -m unittest discover -s tests -p "test*.py" -v`
|
||||
- `python scripts\validate_model_library.py`
|
||||
- `python scripts\check_card_contract.py`
|
||||
- `python scripts\run_selector_demo.py`
|
||||
- `python scripts\run_selector_regression.py`
|
||||
- `python scripts\run_selector_calibration_smoke.py`
|
||||
- `python scripts\check_model_card_sync.py`
|
||||
|
||||
- [ ] **Step 3: Inspect git diff**
|
||||
|
||||
Confirm no selector rule changes and no model lifecycle status changes.
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"index_version": "0.1",
|
||||
"last_updated": "2026-06-17",
|
||||
"last_updated": "2026-06-18",
|
||||
"models": [
|
||||
{
|
||||
"model_id": "intellectual_archaeology",
|
||||
|
|
@ -11,7 +11,7 @@
|
|||
"card_file": "cards/intellectual_archaeology.md",
|
||||
"source_article_count": 2,
|
||||
"source_evidence_count": 5,
|
||||
"regression_case_count": 19,
|
||||
"regression_case_count": 20,
|
||||
"stability_level": "B",
|
||||
"regression_status": "pending",
|
||||
"status": "draft",
|
||||
|
|
@ -26,7 +26,7 @@
|
|||
"card_file": "cards/qpi.md",
|
||||
"source_article_count": 3,
|
||||
"source_evidence_count": 10,
|
||||
"regression_case_count": 58,
|
||||
"regression_case_count": 62,
|
||||
"stability_level": "B",
|
||||
"regression_status": "pending",
|
||||
"status": "draft",
|
||||
|
|
|
|||
|
|
@ -0,0 +1,161 @@
|
|||
# Codex 新会话交接文档：Round 04.1 关闭与 Round 05 准备
|
||||
|
||||
日期：2026-06-18
|
||||
|
||||
仓库：`C:\Users\wangq\Documents\Codex\work-projects\the-mindscape-of-bro-tsong`
|
||||
|
||||
## 1. 当前结论
|
||||
|
||||
Round 04.1 已完成 closeout，可以关闭。
|
||||
|
||||
依据：已阅读 GPT 评审文件 `C:\Users\wangq\Documents\Codex\knowledge-vault\work\internal\强哥的思想宇宙\讨论\2026-06-18产品交流-GPT.md` 第 219-315 行。评审结论是 04.1 可以验收，不做 04.2 规则补丁，不引入 LLM selector；关闭前只补记 `R04-BI-022` 的联动变化，并加入 regression 或 calibration。
|
||||
|
||||
本次 closeout 已执行：
|
||||
|
||||
- `R04-BI-022` 已记录为 accepted collateral change。
|
||||
- `R04-BI-022` 已加入 QPI selector regression。
|
||||
- 04.1 post-patch report 已补充 5 个 before/after 行为变化。
|
||||
- 04.1 review bundle 中的报告副本已同步。
|
||||
- QPI 和 Intellectual Archaeology 生命周期仍保持 `draft / B / pending`。
|
||||
|
||||
## 2. 关闭边界
|
||||
|
||||
本次没有进入 Round 05。
|
||||
|
||||
本次没有做：
|
||||
|
||||
- 新 selector 规则补丁；
|
||||
- 04.2；
|
||||
- 第三模型；
|
||||
- LLM selector；
|
||||
- RAG、vector database、frontend、backend、账号系统或 QA 产品；
|
||||
- QPI 或 Intellectual Archaeology 生命周期升级。
|
||||
|
||||
## 3. 关键变更
|
||||
|
||||
### 3.1 `R04-BI-022` closeout 处理
|
||||
|
||||
原输入：
|
||||
|
||||
```text
|
||||
不要展开到底层假设，只帮我看主导稀缺是什么。
|
||||
```
|
||||
|
||||
04.1 patch 前：
|
||||
|
||||
- selected_models: `[]`
|
||||
- no_call: `True`
|
||||
|
||||
04.1 patch 后：
|
||||
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
|
||||
关闭判断：
|
||||
|
||||
- 这不是新增失败。
|
||||
- 它属于 depth-limiting QPI override 的合理联动变化。
|
||||
- 用户拒绝的是深入展开 / IA 式下钻，不是拒绝所有分析。
|
||||
- 轻量 QPI 主导稀缺判断可接受。
|
||||
- IA 应继续被拒绝。
|
||||
|
||||
新增 regression：
|
||||
|
||||
```text
|
||||
case_qpi_round04_depth_limited_assumption_scarcity_001
|
||||
```
|
||||
|
||||
### 3.2 报告补记
|
||||
|
||||
`reports/Round04_1_post_patch_routing_verification_report_2026-06-18.md` 已新增：
|
||||
|
||||
```text
|
||||
Non-target Behavior Diff / Accepted Collateral Change
|
||||
```
|
||||
|
||||
其中记录 5 个 before/after 行为变化：
|
||||
|
||||
- `R04-BI-002`
|
||||
- `R04-BI-022`
|
||||
- `R04-BI-024`
|
||||
- `R04-BI-035`
|
||||
- `R04-BI-036`
|
||||
|
||||
Targeted labels 仍只限 4 个 owner / GPT reviewed target cases：
|
||||
|
||||
- `R04-BI-002`
|
||||
- `R04-BI-024`
|
||||
- `R04-BI-035`
|
||||
- `R04-BI-036`
|
||||
|
||||
`R04-BI-022` 是 accepted collateral change，不改写为新 target case。
|
||||
|
||||
## 4. 当前关键文件
|
||||
|
||||
Round 04 / 04.1 入口：
|
||||
|
||||
- `reports/Round04_blind_input_candidates_2026-06-18.md`
|
||||
- `selector/round04_blind_inputs.json`
|
||||
- `reports/Round04_blind_routing_evaluation_report_2026-06-18.md`
|
||||
- `reports/Round04_1_post_patch_routing_verification_report_2026-06-18.md`
|
||||
- `ccra_review_bundle/round-04.1_2026-06-18_selector-rule-patch/00_OPEN_THIS_FIRST_CCRA_REVIEW_BRIEF_04.1.md`
|
||||
|
||||
Selector / regression 入口：
|
||||
|
||||
- `selector/selector_rules.json`
|
||||
- `scripts/run_selector_demo.py`
|
||||
- `scripts/run_selector_regression.py`
|
||||
- `scripts/run_round04_blind_routing.py`
|
||||
- `scripts/run_round04_post_patch_verification.py`
|
||||
- `tests/qpi.regression.json`
|
||||
- `tests/intellectual_archaeology.regression.json`
|
||||
- `tests/regression_cases.json`
|
||||
|
||||
当前 QPI regression 数：
|
||||
|
||||
- QPI: 62
|
||||
- Intellectual Archaeology: 20
|
||||
- Aggregate: 82
|
||||
|
||||
## 5. 验证结果
|
||||
|
||||
以下命令均已通过：
|
||||
|
||||
```powershell
|
||||
python scripts\rebuild_indexes.py --check
|
||||
python -m unittest discover -s tests -p "test*.py" -v
|
||||
python scripts\validate_model_library.py
|
||||
python scripts\check_card_contract.py
|
||||
python scripts\run_selector_demo.py
|
||||
python scripts\run_selector_regression.py
|
||||
python scripts\run_selector_calibration_smoke.py
|
||||
python scripts\check_model_card_sync.py
|
||||
```
|
||||
|
||||
结果：
|
||||
|
||||
- Index check: PASS
|
||||
- Unit tests: PASS, 22 tests
|
||||
- Model library validation: PASS
|
||||
- Card contract: PASS
|
||||
- Selector demo: PASS
|
||||
- Selector regression: PASS, 26 selector cases
|
||||
- Selector calibration smoke: PASS
|
||||
- Model/card sync: PASS
|
||||
|
||||
## 6. Round 05 启动建议
|
||||
|
||||
新会话可以从 Round 05 讨论开始，但建议先读本交接文档和 04.1 post-patch report。
|
||||
|
||||
Round 05 不应默认继续改 selector。应先根据 owner / GPT 新讨论确定目标。如果讨论涉及新模型、第三模型、平台化、LLM selector 或 RAG，应先回到非目标边界和 `docs/DECISIONS.md` 判断是否需要新的 owner / CCRA 决策。
|
||||
|
||||
## 7. Git 关闭信息
|
||||
|
||||
计划关闭 tag：
|
||||
|
||||
```text
|
||||
qpi-round-04.1-closed-2026-06-18
|
||||
```
|
||||
|
||||
本文件会随 Round 04.1 closeout commit 一起提交。
|
||||
|
|
@ -15,5 +15,9 @@ Current reports:
|
|||
- `content_review_report_v0.2.md`
|
||||
- `model_review_status.json`
|
||||
- `next_action_register_v0.2.md`
|
||||
- `Round04_blind_input_candidates_2026-06-18.md`
|
||||
- `Round04_blind_routing_evaluation_report_2026-06-18.md`
|
||||
- `Round04_1_post_patch_routing_verification_report_2026-06-18.md`
|
||||
- `Codex新会话交接文档_Round04.1关闭与Round05准备_2026-06-18.md`
|
||||
|
||||
Reports should separate verified facts, assumptions, unresolved questions, and suggested next tasks.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,175 @@
|
|||
# Round 04.1 Post-Patch Routing Verification Report
|
||||
|
||||
Date: 2026-06-18
|
||||
|
||||
This is not a second blind test.
|
||||
|
||||
Frozen input set: `round04_blind_inputs_2026-06-18_v0.1`
|
||||
Original before-report: `reports/Round04_blind_routing_evaluation_report_2026-06-18.md`
|
||||
|
||||
## Scope
|
||||
|
||||
- Round 04.1 verifies selector rule patches after owner / GPT review.
|
||||
- It reruns the frozen 38-input Round 04 pool for comparison.
|
||||
- Expected routing labels are limited to the four review-approved target cases.
|
||||
- It does not generate final answers and does not upgrade model status.
|
||||
|
||||
## Summary
|
||||
|
||||
Frozen inputs rerun: 38
|
||||
Post-patch no-call results: 18
|
||||
Targeted cases checked: 4
|
||||
Targeted failures: 0
|
||||
Behavior changes recorded: 5
|
||||
|
||||
## Targeted Case Results
|
||||
|
||||
### R04-BI-002
|
||||
|
||||
- status: `PASS`
|
||||
- review_label: translation no-call; payload complexity must not trigger QPI
|
||||
- before_selected_models: `['qpi']`
|
||||
- before_no_call: `False`
|
||||
- after_selected_models: `[]`
|
||||
- after_rejected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- after_no_call: `True`
|
||||
|
||||
After selected model details:
|
||||
- none
|
||||
|
||||
### R04-BI-024
|
||||
|
||||
- status: `PASS`
|
||||
- review_label: explicit IA refusal should hard-exclude IA while allowing QPI
|
||||
- before_selected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- before_no_call: `False`
|
||||
- after_selected_models: `['qpi']`
|
||||
- after_rejected_models: `['intellectual_archaeology']`
|
||||
- after_no_call: `False`
|
||||
|
||||
After selected model details:
|
||||
- `qpi` score=0.74 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; complexity signal: 边界、责任; selection_priority=9
|
||||
|
||||
### R04-BI-035
|
||||
|
||||
- status: `PASS`
|
||||
- review_label: depth-limiting plus dominant-scarcity judgment should allow QPI
|
||||
- before_selected_models: `[]`
|
||||
- before_no_call: `True`
|
||||
- after_selected_models: `['qpi']`
|
||||
- after_rejected_models: `['intellectual_archaeology']`
|
||||
- after_no_call: `False`
|
||||
|
||||
After selected model details:
|
||||
- `qpi` score=0.59 decision=selected
|
||||
- reasons: selection_priority=9; QPI problem-definition gate matched
|
||||
|
||||
### R04-BI-036
|
||||
|
||||
- status: `PASS`
|
||||
- review_label: depth-limiting plus scarcity-set judgment should allow QPI
|
||||
- before_selected_models: `[]`
|
||||
- before_no_call: `True`
|
||||
- after_selected_models: `['qpi']`
|
||||
- after_rejected_models: `['intellectual_archaeology']`
|
||||
- after_no_call: `False`
|
||||
|
||||
After selected model details:
|
||||
- `qpi` score=0.99 decision=selected
|
||||
- reasons: trigger keyword matched: 缺路径、缺共识; complexity signal: 共识、底层; selection_priority=9; QPI problem-definition gate matched
|
||||
|
||||
## Non-target Behavior Diff / Accepted Collateral Change
|
||||
|
||||
The frozen 38-input pool was rerun for comparison. Expected routing labels remain limited to the four target cases above, but the full rerun records five behavior changes from the original Round 04 report.
|
||||
|
||||
### R04-BI-002
|
||||
|
||||
- label: target fix: translation no-call
|
||||
- before_selected_models: `['qpi']`
|
||||
- before_no_call: `False`
|
||||
- after_selected_models: `[]`
|
||||
- after_rejected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- after_no_call: `True`
|
||||
- closeout_judgment: Expected target change; translation instruction should not be routed to QPI because the payload contains `模型`.
|
||||
|
||||
### R04-BI-022
|
||||
|
||||
- label: accepted collateral change: depth-limited QPI override
|
||||
- before_selected_models: `[]`
|
||||
- before_no_call: `True`
|
||||
- after_selected_models: `['qpi']`
|
||||
- after_rejected_models: `['intellectual_archaeology']`
|
||||
- after_no_call: `False`
|
||||
- closeout_judgment: Not a new failure. The request rejects deeper IA-style expansion while asking for dominant scarcity judgment, so lightweight QPI is acceptable and IA should remain rejected. Added to regression as a QPI selector-gate trap.
|
||||
|
||||
### R04-BI-024
|
||||
|
||||
- label: target fix: explicit IA refusal
|
||||
- before_selected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- before_no_call: `False`
|
||||
- after_selected_models: `['qpi']`
|
||||
- after_rejected_models: `['intellectual_archaeology']`
|
||||
- after_no_call: `False`
|
||||
- closeout_judgment: Expected target change; explicit refusal should hard-exclude IA while allowing lightweight QPI.
|
||||
|
||||
### R04-BI-035
|
||||
|
||||
- label: target fix: dominant-scarcity depth limit
|
||||
- before_selected_models: `[]`
|
||||
- before_no_call: `True`
|
||||
- after_selected_models: `['qpi']`
|
||||
- after_rejected_models: `['intellectual_archaeology']`
|
||||
- after_no_call: `False`
|
||||
- closeout_judgment: Expected target change; depth-limiting language should not block lightweight QPI scarcity judgment.
|
||||
|
||||
### R04-BI-036
|
||||
|
||||
- label: target fix: scarcity-set depth limit
|
||||
- before_selected_models: `[]`
|
||||
- before_no_call: `True`
|
||||
- after_selected_models: `['qpi']`
|
||||
- after_rejected_models: `['intellectual_archaeology']`
|
||||
- after_no_call: `False`
|
||||
- closeout_judgment: Expected target change; depth-limiting language should not block QPI scarcity-set judgment.
|
||||
|
||||
## Full Pool Rerun Summary
|
||||
|
||||
- `R04-BI-001`: selected=[]; no_call=True; category=explicit_no_call_fact_lookup
|
||||
- `R04-BI-002`: selected=[]; no_call=True; category=explicit_no_call_translation
|
||||
- `R04-BI-003`: selected=[]; no_call=True; category=explicit_no_call_rewrite
|
||||
- `R04-BI-004`: selected=[]; no_call=True; category=explicit_no_call_formatting
|
||||
- `R04-BI-005`: selected=[]; no_call=True; category=explicit_no_call_direct_execution
|
||||
- `R04-BI-006`: selected=['qpi']; no_call=False; category=qpi_light_problem_definition
|
||||
- `R04-BI-007`: selected=['qpi']; no_call=False; category=qpi_light_problem_definition
|
||||
- `R04-BI-008`: selected=['qpi']; no_call=False; category=qpi_light_problem_definition
|
||||
- `R04-BI-009`: selected=['qpi']; no_call=False; category=qpi_light_problem_definition
|
||||
- `R04-BI-010`: selected=[]; no_call=True; category=governance_responsibility_consensus
|
||||
- `R04-BI-011`: selected=[]; no_call=True; category=governance_responsibility_consensus
|
||||
- `R04-BI-012`: selected=[]; no_call=True; category=governance_responsibility_consensus
|
||||
- `R04-BI-013`: selected=[]; no_call=True; category=governance_responsibility_consensus
|
||||
- `R04-BI-014`: selected=[]; no_call=True; category=capacity_resource_path
|
||||
- `R04-BI-015`: selected=[]; no_call=True; category=capacity_resource_path
|
||||
- `R04-BI-016`: selected=['qpi']; no_call=False; category=capacity_resource_path
|
||||
- `R04-BI-017`: selected=[]; no_call=True; category=capacity_resource_path
|
||||
- `R04-BI-018`: selected=['qpi']; no_call=False; category=model_extraction_or_depth_modeling
|
||||
- `R04-BI-019`: selected=['qpi', 'intellectual_archaeology']; no_call=False; category=model_extraction_or_depth_modeling
|
||||
- `R04-BI-020`: selected=['qpi']; no_call=False; category=model_extraction_or_depth_modeling
|
||||
- `R04-BI-021`: selected=['qpi']; no_call=False; category=qpi_allowed_ia_refused
|
||||
- `R04-BI-022`: selected=['qpi']; no_call=False; category=qpi_allowed_ia_refused
|
||||
- `R04-BI-023`: selected=['qpi']; no_call=False; category=qpi_allowed_ia_refused
|
||||
- `R04-BI-024`: selected=['qpi']; no_call=False; category=qpi_allowed_ia_refused
|
||||
- `R04-BI-025`: selected=['intellectual_archaeology', 'qpi']; no_call=False; category=likely_intellectual_archaeology
|
||||
- `R04-BI-026`: selected=['qpi']; no_call=False; category=likely_intellectual_archaeology
|
||||
- `R04-BI-027`: selected=['qpi']; no_call=False; category=likely_intellectual_archaeology
|
||||
- `R04-BI-028`: selected=[]; no_call=True; category=ambiguous_low_context
|
||||
- `R04-BI-029`: selected=[]; no_call=True; category=ambiguous_low_context
|
||||
- `R04-BI-030`: selected=['qpi']; no_call=False; category=ambiguous_low_context
|
||||
- `R04-BI-031`: selected=[]; no_call=True; category=explicit_no_call_how_to
|
||||
- `R04-BI-032`: selected=[]; no_call=True; category=explicit_no_call_how_to
|
||||
- `R04-BI-033`: selected=[]; no_call=True; category=explicit_no_call_dictionary_judgment
|
||||
- `R04-BI-034`: selected=['qpi']; no_call=False; category=direct_execution_with_analysis_override
|
||||
- `R04-BI-035`: selected=['qpi']; no_call=False; category=qpi_allowed_depth_limited
|
||||
- `R04-BI-036`: selected=['qpi']; no_call=False; category=qpi_allowed_depth_limited
|
||||
- `R04-BI-037`: selected=[]; no_call=True; category=ambiguous_growth_how_to
|
||||
- `R04-BI-038`: selected=['qpi']; no_call=False; category=likely_intellectual_archaeology
|
||||
|
|
@ -0,0 +1,329 @@
|
|||
# Round 04 Blind Input Candidates
|
||||
|
||||
Date: 2026-06-18
|
||||
|
||||
Repository: `C:\Users\wangq\Documents\Codex\work-projects\the-mindscape-of-bro-tsong`
|
||||
|
||||
Status: `draft_owner_review_needed`
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
This document is a draft input pool for Round 04 blind input routing evaluation.
|
||||
|
||||
It prepares candidate user inputs only. It does not run the selector, does not add expected selector behavior, does not modify selector rules, does not add regression cases, and does not upgrade model lifecycle status.
|
||||
|
||||
Owner review should decide which inputs are retained, edited, removed, or recategorized before a frozen Round 04 blind input JSON is created.
|
||||
|
||||
## 2. Scope Guardrails
|
||||
|
||||
- Do not modify `selector/selector_rules.json` before running the blind evaluation.
|
||||
- Do not merge these candidates into `selector/selector_calibration_inputs.json`.
|
||||
- Do not add expected behavior before the first blind run.
|
||||
- Do not add a third model.
|
||||
- Do not build QA, RAG, frontend, backend, database, or any platform feature.
|
||||
- Do not treat category labels below as answer keys. They are coverage labels for owner review only.
|
||||
|
||||
## 3. Review Instructions
|
||||
|
||||
For each input, mark one of:
|
||||
|
||||
- `keep`
|
||||
- `revise`
|
||||
- `remove`
|
||||
- `recategorize`
|
||||
|
||||
Optional review notes:
|
||||
|
||||
- Is the input too obvious?
|
||||
- Is the input too close to existing regression / calibration cases?
|
||||
- Is the wording natural enough?
|
||||
- Is the input too leading because it names QPI or Intellectual Archaeology?
|
||||
- Should the category mix change?
|
||||
|
||||
## 4. Candidate Inputs
|
||||
|
||||
### 4.1 明确 no-call：查事实、翻译、润色、格式化、直接执行
|
||||
|
||||
#### R04-BI-001
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_fact_lookup`
|
||||
- input_text: `查一下《原则》这本书的作者是谁，只要事实，不要分析。`
|
||||
- why_included: Tests whether direct fact lookup stays outside model routing.
|
||||
|
||||
#### R04-BI-002
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_translation`
|
||||
- input_text: `把这句话翻译成英文：这个模型还处在草稿阶段。`
|
||||
- why_included: Tests pure translation as no-call.
|
||||
|
||||
#### R04-BI-003
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_rewrite`
|
||||
- input_text: `请把这段话润色得更顺一点，不要改变意思，也不要解释原因。`
|
||||
- why_included: Tests polish-only request with explicit no-analysis boundary.
|
||||
|
||||
#### R04-BI-004
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_formatting`
|
||||
- input_text: `把下面这组要点整理成 Markdown 表格，直接给结果。`
|
||||
- why_included: Tests formatting / direct execution without cognitive model need.
|
||||
|
||||
#### R04-BI-005
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_direct_execution`
|
||||
- input_text: `把这个 JSON 压缩成单行，不用说明：{"status":"draft","level":"B"}`
|
||||
- why_included: Tests direct transformation request.
|
||||
|
||||
### 4.2 QPI 轻量问题定性
|
||||
|
||||
#### R04-BI-006
|
||||
|
||||
- category_for_owner_review: `qpi_light_problem_definition`
|
||||
- input_text: `我们现在反复开会但没有推进，我想先判断这是缺信息、缺路径，还是缺共识。`
|
||||
- why_included: Tests lightweight problem-definition language without asking for depth.
|
||||
|
||||
#### R04-BI-007
|
||||
|
||||
- category_for_owner_review: `qpi_light_problem_definition`
|
||||
- input_text: `这个项目卡住了，我不确定到底是问题没定义清楚，还是执行方案不够。`
|
||||
- why_included: Tests QPI-like framing without naming QPI.
|
||||
|
||||
#### R04-BI-008
|
||||
|
||||
- category_for_owner_review: `qpi_light_problem_definition`
|
||||
- input_text: `我们有一堆材料，但不知道下一步该问什么问题，先帮我定性一下。`
|
||||
- why_included: Tests question/problem boundary and data/path scarcity.
|
||||
|
||||
#### R04-BI-009
|
||||
|
||||
- category_for_owner_review: `qpi_light_problem_definition`
|
||||
- input_text: `这件事到底算一个简单问题、复杂问题，还是已经变成组织议题？`
|
||||
- why_included: Tests lightweight classification among question / problem / issue.
|
||||
|
||||
### 4.3 组织治理 / 责任 / 共识类输入
|
||||
|
||||
#### R04-BI-010
|
||||
|
||||
- category_for_owner_review: `governance_responsibility_consensus`
|
||||
- input_text: `每个部门都说支持这个项目，但一到分工就没人愿意承担主责。`
|
||||
- why_included: Tests responsibility ambiguity and consensus scarcity.
|
||||
|
||||
#### R04-BI-011
|
||||
|
||||
- category_for_owner_review: `governance_responsibility_consensus`
|
||||
- input_text: `老板要创新，财务要控成本，业务要短期业绩，最后方案总是被折中到没有方向。`
|
||||
- why_included: Tests cross-stakeholder conflict and governance weight.
|
||||
|
||||
#### R04-BI-012
|
||||
|
||||
- category_for_owner_review: `governance_responsibility_consensus`
|
||||
- input_text: `我们不是不知道怎么做，而是每个人都在等别人先表态。`
|
||||
- why_included: Tests responsibility / commitment issue where path may already exist.
|
||||
|
||||
#### R04-BI-013
|
||||
|
||||
- category_for_owner_review: `governance_responsibility_consensus`
|
||||
- input_text: `这个流程改造牵涉教务、学院和信息化部门，谁都能提意见，但没人能拍板。`
|
||||
- why_included: Tests multi-party ownership and decision authority gap.
|
||||
|
||||
### 4.4 容量 / 资源 / 路径类输入
|
||||
|
||||
#### R04-BI-014
|
||||
|
||||
- category_for_owner_review: `capacity_resource_path`
|
||||
- input_text: `团队现在知道目标，也认可方向，但只有两个人能做关键模块，进度一直被产能卡住。`
|
||||
- why_included: Tests resource / capacity scarcity rather than data scarcity.
|
||||
|
||||
#### R04-BI-015
|
||||
|
||||
- category_for_owner_review: `capacity_resource_path`
|
||||
- input_text: `我们知道要做知识库，也有材料，但不知道从哪条最小路径开始。`
|
||||
- why_included: Tests path scarcity with sufficient source material.
|
||||
|
||||
#### R04-BI-016
|
||||
|
||||
- category_for_owner_review: `capacity_resource_path`
|
||||
- input_text: `预算不是最大问题，问题是现有系统、权限和审批流程都接不上。`
|
||||
- why_included: Tests operational path constraints and institutional friction.
|
||||
|
||||
#### R04-BI-017
|
||||
|
||||
- category_for_owner_review: `capacity_resource_path`
|
||||
- input_text: `这件事大家都觉得重要，但每周只能挤出零散时间，所以一直没有稳定产出。`
|
||||
- why_included: Tests capacity scarcity mixed with execution continuity.
|
||||
|
||||
### 4.5 模型抽取 / 深度建模类输入
|
||||
|
||||
#### R04-BI-018
|
||||
|
||||
- category_for_owner_review: `model_extraction_or_depth_modeling`
|
||||
- input_text: `请从这篇长文里抽取一个可复用的认知模型，重点看它的核心机制和适用边界。`
|
||||
- why_included: Tests model-extraction intent and possible depth-modeling route.
|
||||
- owner_review_note: `positive control; obvious model-extraction / depth-modeling case rather than a natural boundary sample.`
|
||||
|
||||
#### R04-BI-019
|
||||
|
||||
- category_for_owner_review: `model_extraction_or_depth_modeling`
|
||||
- input_text: `这套说法背后好像有一个稳定结构，不只是观点总结，帮我判断它能不能沉淀成模型。`
|
||||
- why_included: Tests model-mining request without naming Intellectual Archaeology.
|
||||
|
||||
#### R04-BI-020
|
||||
|
||||
- category_for_owner_review: `model_extraction_or_depth_modeling`
|
||||
- input_text: `我想把这段讨论拆成原始判断、隐含假设、适用条件和误用边界。`
|
||||
- why_included: Tests structured depth analysis intent.
|
||||
|
||||
### 4.6 明确拒绝思想考古但允许 QPI 的输入
|
||||
|
||||
#### R04-BI-021
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_ia_refused`
|
||||
- input_text: `先不要思想考古，只判断这是不是一个 QPI 意义上的问题。`
|
||||
- why_included: Tests explicit IA refusal while allowing QPI.
|
||||
- control_case_type: `model-name-exposed`
|
||||
- owner_review_note: `Kept as a control case for explicit model instructions, not as a fully natural blind input.`
|
||||
|
||||
#### R04-BI-022
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_ia_refused`
|
||||
- input_text: `不要展开到底层假设，只帮我看主导稀缺是什么。`
|
||||
- why_included: Tests depth-limiting QPI override without naming QPI.
|
||||
|
||||
#### R04-BI-023
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_ia_refused`
|
||||
- input_text: `这次先不做深度建模，只做问题定性，看它是缺数据、缺路径还是缺共识。`
|
||||
- why_included: Tests QPI-limited intent plus explicit depth refusal.
|
||||
|
||||
#### R04-BI-024
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_ia_refused`
|
||||
- input_text: `不要进入思想考古，我只要一个轻量判断：现在应该先问问题、找路径，还是处理责任边界？`
|
||||
- why_included: Tests model-level hard exclusion for IA while preserving QPI-like routing.
|
||||
|
||||
### 4.7 真正应该进入思想考古的输入
|
||||
|
||||
#### R04-BI-025
|
||||
|
||||
- category_for_owner_review: `likely_intellectual_archaeology`
|
||||
- input_text: `QPI 已经判断这是中重型课题，现在请继续分析它背后的底层假设、生成机制和长期稳定结构。`
|
||||
- why_included: Tests explicit post-QPI depth handoff.
|
||||
- control_case_type: `model-name-exposed`
|
||||
- owner_review_note: `Kept as a control case for explicit post-QPI handoff, not as a fully natural blind input.`
|
||||
|
||||
#### R04-BI-026
|
||||
|
||||
- category_for_owner_review: `likely_intellectual_archaeology`
|
||||
- input_text: `这个现象反复出现在不同项目里，我想知道它背后的思想结构是什么，而不是只做问题分类。`
|
||||
- why_included: Tests repeated-pattern depth modeling beyond lightweight QPI.
|
||||
- owner_review_note: `High-value natural boundary sample because it avoids explicit model names while suggesting IA or QPI-to-IA routing.`
|
||||
|
||||
#### R04-BI-027
|
||||
|
||||
- category_for_owner_review: `likely_intellectual_archaeology`
|
||||
- input_text: `请分析这套判断为什么会长期成立，它依赖哪些世界观、因果假设和行动边界。`
|
||||
- why_included: Tests deep assumption / mechanism request.
|
||||
|
||||
### 4.8 模糊短句 / 低上下文输入
|
||||
|
||||
#### R04-BI-028
|
||||
|
||||
- category_for_owner_review: `ambiguous_low_context`
|
||||
- input_text: `这个不太对。`
|
||||
- why_included: Tests very low context and likely need for clarification.
|
||||
|
||||
#### R04-BI-029
|
||||
|
||||
- category_for_owner_review: `ambiguous_low_context`
|
||||
- input_text: `怎么又卡住了？`
|
||||
- why_included: Tests short ambiguous frustration signal.
|
||||
|
||||
#### R04-BI-030
|
||||
|
||||
- category_for_owner_review: `ambiguous_low_context`
|
||||
- input_text: `还是那个老问题。`
|
||||
- why_included: Tests prior-context dependency without enough current evidence.
|
||||
|
||||
### 4.9 Round 03.2 / 03.2a trap coverage additions
|
||||
|
||||
#### R04-BI-031
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_how_to`
|
||||
- input_text: `如何在 Excel 里冻结首行？`
|
||||
- why_included: Tests whether generic how-to stays outside QPI after broad "如何" gate was narrowed.
|
||||
|
||||
#### R04-BI-032
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_how_to`
|
||||
- input_text: `怎么把 Markdown 文件转成 PDF？`
|
||||
- why_included: Tests tool/how-to request as no-call.
|
||||
|
||||
#### R04-BI-033
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_dictionary_judgment`
|
||||
- input_text: `判断一下“alignment”在这句话里是什么意思。`
|
||||
- why_included: Tests whether ordinary word-meaning judgment avoids QPI over-selection.
|
||||
|
||||
#### R04-BI-034
|
||||
|
||||
- category_for_owner_review: `direct_execution_with_analysis_override`
|
||||
- input_text: `请把下面内容总结成三点，但不要只执行，先判断它背后的问题定义。`
|
||||
- why_included: Tests whether explicit analysis override can escape direct-execution no-call.
|
||||
|
||||
#### R04-BI-035
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_depth_limited`
|
||||
- input_text: `不要深入分析，只判断这件事的主导稀缺是什么。`
|
||||
- why_included: Tests Round 03.2a depth-limiting override without naming QPI or IA.
|
||||
|
||||
#### R04-BI-036
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_depth_limited`
|
||||
- input_text: `不要展开到底层理论，只判断现在是缺信息、缺路径还是缺共识。`
|
||||
- why_included: Tests QPI-limited routing while rejecting deeper modeling.
|
||||
|
||||
#### R04-BI-037
|
||||
|
||||
- category_for_owner_review: `ambiguous_growth_how_to`
|
||||
- input_text: `如何提高这个账号的流量？`
|
||||
- why_included: Tests whether generic growth how-to is routed to QPI, no-call, or low-confidence provisional handling.
|
||||
|
||||
#### R04-BI-038
|
||||
|
||||
- category_for_owner_review: `likely_intellectual_archaeology`
|
||||
- input_text: `这类失败在不同项目里反复出现，我想抽出它背后稳定的生成机制和可迁移框架。`
|
||||
- why_included: Tests deeper modeling intent without explicit "思想考古" or "底层假设" wording.
|
||||
|
||||
## 5. Coverage Summary
|
||||
|
||||
- explicit no-call: 5
|
||||
- QPI lightweight problem definition: 4
|
||||
- governance / responsibility / consensus: 4
|
||||
- capacity / resource / path: 4
|
||||
- model extraction / depth modeling: 3
|
||||
- IA refused but QPI allowed: 4
|
||||
- likely Intellectual Archaeology: 4
|
||||
- ambiguous / low context: 3
|
||||
- Round 03.2 / 03.2a trap additions: 8
|
||||
- model-name-exposed control cases: 2
|
||||
- positive control cases: 1
|
||||
|
||||
Total: 38 candidate inputs.
|
||||
|
||||
## 6. Frozen Asset and First Run
|
||||
|
||||
Owner review has been incorporated. The frozen JSON asset is:
|
||||
|
||||
```text
|
||||
selector/round04_blind_inputs.json
|
||||
```
|
||||
|
||||
The frozen asset preserves reviewed input text and non-answer-key metadata only.
|
||||
|
||||
The first blind routing run has been written to:
|
||||
|
||||
```text
|
||||
reports/Round04_blind_routing_evaluation_report_2026-06-18.md
|
||||
```
|
||||
|
||||
Expected selector behavior should be added only after owner / GPT / CCRA review of the observed routing results.
|
||||
|
|
@ -0,0 +1,771 @@
|
|||
# Round 04 Blind Routing Evaluation Report
|
||||
|
||||
Date: 2026-06-18
|
||||
|
||||
Blind input set: `round04_blind_inputs_2026-06-18_v0.1`
|
||||
|
||||
Expected routing labels: `not included before first blind run`
|
||||
|
||||
This report records current selector routing only. It does not score correctness, does not generate final answers, does not modify selector rules, and does not upgrade model status.
|
||||
|
||||
## Scope
|
||||
|
||||
- Round 04 target: test routing stability on new inputs.
|
||||
- Runtime surface: current rule-based selector plus model-level QPI / IA gates.
|
||||
- QPI contract status: represented here by QPI selection and routing details; no full QPI answer generation is performed.
|
||||
- IA status: represented here by selector selection / rejection only; no Intellectual Archaeology answer generation is performed.
|
||||
|
||||
## Summary
|
||||
|
||||
Inputs evaluated: 38
|
||||
No-call results: 20
|
||||
|
||||
### Selected Model Distribution
|
||||
|
||||
- `intellectual_archaeology`: 3
|
||||
- `none`: 20
|
||||
- `qpi`: 18
|
||||
|
||||
### Coverage Categories
|
||||
|
||||
- `ambiguous_growth_how_to`: 1
|
||||
- `ambiguous_low_context`: 3
|
||||
- `capacity_resource_path`: 4
|
||||
- `direct_execution_with_analysis_override`: 1
|
||||
- `explicit_no_call_dictionary_judgment`: 1
|
||||
- `explicit_no_call_direct_execution`: 1
|
||||
- `explicit_no_call_fact_lookup`: 1
|
||||
- `explicit_no_call_formatting`: 1
|
||||
- `explicit_no_call_how_to`: 2
|
||||
- `explicit_no_call_rewrite`: 1
|
||||
- `explicit_no_call_translation`: 1
|
||||
- `governance_responsibility_consensus`: 4
|
||||
- `likely_intellectual_archaeology`: 4
|
||||
- `model_extraction_or_depth_modeling`: 3
|
||||
- `qpi_allowed_depth_limited`: 2
|
||||
- `qpi_allowed_ia_refused`: 4
|
||||
- `qpi_light_problem_definition`: 4
|
||||
|
||||
## Case Results
|
||||
|
||||
### R04-BI-001
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_fact_lookup`
|
||||
- input_text: `查一下《原则》这本书的作者是谁，只要事实，不要分析。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- no_call: `True`
|
||||
- routing_notes: hard no-call gate matched; explicit analysis override was not present.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 只要事实、查一下
|
||||
- `qpi` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 只要事实、查一下
|
||||
|
||||
### R04-BI-002
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_translation`
|
||||
- input_text: `把这句话翻译成英文：这个模型还处在草稿阶段。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.49 decision=selected
|
||||
- reasons: complexity signal: 模型; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.17 decision=rejected
|
||||
- reasons: complexity signal: 模型; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-003
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_rewrite`
|
||||
- input_text: `请把这段话润色得更顺一点，不要改变意思，也不要解释原因。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- no_call: `True`
|
||||
- routing_notes: hard no-call gate matched; explicit analysis override was not present.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 润色
|
||||
- `qpi` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 润色
|
||||
|
||||
### R04-BI-004
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_formatting`
|
||||
- input_text: `把下面这组要点整理成 Markdown 表格，直接给结果。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-005
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_direct_execution`
|
||||
- input_text: `把这个 JSON 压缩成单行，不用说明：{"status":"draft","level":"B"}`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-006
|
||||
|
||||
- category_for_owner_review: `qpi_light_problem_definition`
|
||||
- input_text: `我们现在反复开会但没有推进，我想先判断这是缺信息、缺路径，还是缺共识。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.99 decision=selected
|
||||
- reasons: trigger keyword matched: 缺路径、缺共识; complexity signal: 反复、共识; selection_priority=9; QPI problem-definition gate matched
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.0 decision=rejected
|
||||
- reasons: complexity signal: 反复、共识; selection_priority=7
|
||||
- penalties: QPI-before-IA gate: 缺路径、缺共识; no IA heavy-depth gate
|
||||
|
||||
### R04-BI-007
|
||||
|
||||
- category_for_owner_review: `qpi_light_problem_definition`
|
||||
- input_text: `这个项目卡住了，我不确定到底是问题没定义清楚，还是执行方案不够。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.59 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-008
|
||||
|
||||
- category_for_owner_review: `qpi_light_problem_definition`
|
||||
- input_text: `我们有一堆材料，但不知道下一步该问什么问题，先帮我定性一下。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.59 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-009
|
||||
|
||||
- category_for_owner_review: `qpi_light_problem_definition`
|
||||
- input_text: `这件事到底算一个简单问题、复杂问题，还是已经变成组织议题？`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.89 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; input type matched: 复杂问题; complexity signal: 组织、复杂; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.32 decision=rejected
|
||||
- reasons: input type matched: 复杂问题; complexity signal: 组织、复杂; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-010
|
||||
|
||||
- category_for_owner_review: `governance_responsibility_consensus`
|
||||
- input_text: `每个部门都说支持这个项目，但一到分工就没人愿意承担主责。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-011
|
||||
|
||||
- category_for_owner_review: `governance_responsibility_consensus`
|
||||
- input_text: `老板要创新，财务要控成本，业务要短期业绩，最后方案总是被折中到没有方向。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-012
|
||||
|
||||
- category_for_owner_review: `governance_responsibility_consensus`
|
||||
- input_text: `我们不是不知道怎么做，而是每个人都在等别人先表态。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-013
|
||||
|
||||
- category_for_owner_review: `governance_responsibility_consensus`
|
||||
- input_text: `这个流程改造牵涉教务、学院和信息化部门，谁都能提意见，但没人能拍板。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-014
|
||||
|
||||
- category_for_owner_review: `capacity_resource_path`
|
||||
- input_text: `团队现在知道目标，也认可方向，但只有两个人能做关键模块，进度一直被产能卡住。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-015
|
||||
|
||||
- category_for_owner_review: `capacity_resource_path`
|
||||
- input_text: `我们知道要做知识库，也有材料，但不知道从哪条最小路径开始。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-016
|
||||
|
||||
- category_for_owner_review: `capacity_resource_path`
|
||||
- input_text: `预算不是最大问题，问题是现有系统、权限和审批流程都接不上。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.74 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; complexity signal: 系统; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.17 decision=rejected
|
||||
- reasons: complexity signal: 系统; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-017
|
||||
|
||||
- category_for_owner_review: `capacity_resource_path`
|
||||
- input_text: `这件事大家都觉得重要，但每周只能挤出零散时间，所以一直没有稳定产出。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-018
|
||||
|
||||
- category_for_owner_review: `model_extraction_or_depth_modeling`
|
||||
- control_case_type: `positive_control`
|
||||
- input_text: `请从这篇长文里抽取一个可复用的认知模型，重点看它的核心机制和适用边界。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.49 decision=selected
|
||||
- reasons: complexity signal: 模型、机制、抽取、边界; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.17 decision=rejected
|
||||
- reasons: complexity signal: 模型、机制、抽取、边界; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-019
|
||||
|
||||
- category_for_owner_review: `model_extraction_or_depth_modeling`
|
||||
- input_text: `这套说法背后好像有一个稳定结构，不只是观点总结，帮我判断它能不能沉淀成模型。`
|
||||
- selected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- rejected_models: `[]`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.49 decision=selected
|
||||
- reasons: complexity signal: 模型、结构; selection_priority=9
|
||||
- `intellectual_archaeology` score=0.42 decision=selected
|
||||
- reasons: trigger keyword matched: 沉淀成模型; complexity signal: 模型、结构; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
Rejected model details:
|
||||
- none
|
||||
|
||||
### R04-BI-020
|
||||
|
||||
- category_for_owner_review: `model_extraction_or_depth_modeling`
|
||||
- input_text: `我想把这段讨论拆成原始判断、隐含假设、适用条件和误用边界。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.49 decision=selected
|
||||
- reasons: complexity signal: 边界; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.17 decision=rejected
|
||||
- reasons: complexity signal: 边界; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-021
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_ia_refused`
|
||||
- control_case_type: `model_name_exposed`
|
||||
- input_text: `先不要思想考古，只判断这是不是一个 QPI 意义上的问题。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.59 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.0 decision=rejected
|
||||
- reasons: trigger keyword matched: 思想考古; selection_priority=7; IA heavy-depth signal: 思想考古
|
||||
- penalties: negative trigger: 不要思想考古、先不要思想考古; model hard exclusion signal: 不要思想考古、先不要思想考古
|
||||
|
||||
### R04-BI-022
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_ia_refused`
|
||||
- input_text: `不要展开到底层假设，只帮我看主导稀缺是什么。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- no_call: `True`
|
||||
- routing_notes: hard no-call gate matched; explicit analysis override was not present.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 不要展开
|
||||
- `qpi` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 不要展开
|
||||
|
||||
### R04-BI-023
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_ia_refused`
|
||||
- input_text: `这次先不做深度建模，只做问题定性，看它是缺数据、缺路径还是缺共识。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.99 decision=selected
|
||||
- reasons: trigger keyword matched: 问题、缺数据、缺路径、缺共识; complexity signal: 共识; selection_priority=9; QPI problem-definition gate matched
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.0 decision=rejected
|
||||
- reasons: complexity signal: 共识; selection_priority=7
|
||||
- penalties: QPI-before-IA gate: 缺数据、缺路径、缺共识; no IA heavy-depth gate; model hard exclusion signal: 只做问题定性
|
||||
|
||||
### R04-BI-024
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_ia_refused`
|
||||
- input_text: `不要进入思想考古，我只要一个轻量判断：现在应该先问问题、找路径，还是处理责任边界？`
|
||||
- selected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- rejected_models: `[]`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `intellectual_archaeology` score=0.97 decision=selected
|
||||
- reasons: trigger keyword matched: 思想考古; complexity signal: 边界、责任; selection_priority=7; IA heavy-depth signal: 思想考古
|
||||
- `qpi` score=0.74 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; complexity signal: 边界、责任; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- none
|
||||
|
||||
### R04-BI-025
|
||||
|
||||
- category_for_owner_review: `likely_intellectual_archaeology`
|
||||
- control_case_type: `model_name_exposed`
|
||||
- input_text: `QPI 已经判断这是中重型课题，现在请继续分析它背后的底层假设、生成机制和长期稳定结构。`
|
||||
- selected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- rejected_models: `[]`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `intellectual_archaeology` score=0.97 decision=selected
|
||||
- reasons: trigger keyword matched: 底层假设; complexity signal: 长期、底层、结构、机制; selection_priority=7; IA heavy-depth signal: 底层假设
|
||||
- `qpi` score=0.74 decision=selected
|
||||
- reasons: trigger keyword matched: 课题; complexity signal: 长期、底层、结构、机制; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- none
|
||||
|
||||
### R04-BI-026
|
||||
|
||||
- category_for_owner_review: `likely_intellectual_archaeology`
|
||||
- input_text: `这个现象反复出现在不同项目里，我想知道它背后的思想结构是什么，而不是只做问题分类。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.74 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; complexity signal: 反复、结构; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.17 decision=rejected
|
||||
- reasons: complexity signal: 反复、结构; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-027
|
||||
|
||||
- category_for_owner_review: `likely_intellectual_archaeology`
|
||||
- input_text: `请分析这套判断为什么会长期成立，它依赖哪些世界观、因果假设和行动边界。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.49 decision=selected
|
||||
- reasons: complexity signal: 长期、边界; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.17 decision=rejected
|
||||
- reasons: complexity signal: 长期、边界; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-028
|
||||
|
||||
- category_for_owner_review: `ambiguous_low_context`
|
||||
- input_text: `这个不太对。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-029
|
||||
|
||||
- category_for_owner_review: `ambiguous_low_context`
|
||||
- input_text: `怎么又卡住了？`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-030
|
||||
|
||||
- category_for_owner_review: `ambiguous_low_context`
|
||||
- input_text: `还是那个老问题。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.59 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-031
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_how_to`
|
||||
- input_text: `如何在 Excel 里冻结首行？`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-032
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_how_to`
|
||||
- input_text: `怎么把 Markdown 文件转成 PDF？`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-033
|
||||
|
||||
- category_for_owner_review: `explicit_no_call_dictionary_judgment`
|
||||
- input_text: `判断一下“alignment”在这句话里是什么意思。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-034
|
||||
|
||||
- category_for_owner_review: `direct_execution_with_analysis_override`
|
||||
- input_text: `请把下面内容总结成三点，但不要只执行，先判断它背后的问题定义。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.59 decision=selected
|
||||
- reasons: trigger keyword matched: 问题; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-035
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_depth_limited`
|
||||
- input_text: `不要深入分析，只判断这件事的主导稀缺是什么。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- no_call: `True`
|
||||
- routing_notes: hard no-call gate matched; explicit analysis override was not present.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 不要深入分析
|
||||
- `qpi` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 不要深入分析
|
||||
|
||||
### R04-BI-036
|
||||
|
||||
- category_for_owner_review: `qpi_allowed_depth_limited`
|
||||
- input_text: `不要展开到底层理论，只判断现在是缺信息、缺路径还是缺共识。`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['intellectual_archaeology', 'qpi']`
|
||||
- no_call: `True`
|
||||
- routing_notes: hard no-call gate matched; explicit analysis override was not present.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 不要展开
|
||||
- `qpi` score=0.0 decision=rejected
|
||||
- penalties: hard no-call signal: 不要展开
|
||||
|
||||
### R04-BI-037
|
||||
|
||||
- category_for_owner_review: `ambiguous_growth_how_to`
|
||||
- input_text: `如何提高这个账号的流量？`
|
||||
- selected_models: `[]`
|
||||
- rejected_models: `['qpi', 'intellectual_archaeology']`
|
||||
- no_call: `True`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- none
|
||||
|
||||
Rejected model details:
|
||||
- `qpi` score=0.09 decision=rejected
|
||||
- reasons: selection_priority=9
|
||||
- penalties: QPI positive signal missing
|
||||
- `intellectual_archaeology` score=0.02 decision=rejected
|
||||
- reasons: selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
### R04-BI-038
|
||||
|
||||
- category_for_owner_review: `likely_intellectual_archaeology`
|
||||
- input_text: `这类失败在不同项目里反复出现，我想抽出它背后稳定的生成机制和可迁移框架。`
|
||||
- selected_models: `['qpi']`
|
||||
- rejected_models: `['intellectual_archaeology']`
|
||||
- no_call: `False`
|
||||
- routing_notes: selector is rule-based and driven by selector/selector_rules.json; no LLM, no vector search, no answer generation.
|
||||
|
||||
Selected model details:
|
||||
- `qpi` score=0.49 decision=selected
|
||||
- reasons: complexity signal: 反复、机制; selection_priority=9
|
||||
|
||||
Rejected model details:
|
||||
- `intellectual_archaeology` score=0.17 decision=rejected
|
||||
- reasons: complexity signal: 反复、机制; selection_priority=7
|
||||
- penalties: no IA heavy-depth gate
|
||||
|
||||
## Review Reminder
|
||||
|
||||
This is the first blind routing output. Failure labels, expected routing, and rule changes should be added only after owner / GPT / CCRA review of these observed results.
|
||||
|
|
@ -111,7 +111,7 @@ Coverage:
|
|||
- 已覆盖误用、防误召回、no-call 和 pipeline gate。
|
||||
- 仍需人工审查用例真实性和遗漏边界。
|
||||
|
||||
Round 03 / 03.1 / 03.2 / 03.2a update: QPI has since expanded to 58 regression cases after owner-reviewed case promotion, selector/no-call repair, broad-gate no-call traps, and depth-limiting QPI override traps. Intellectual Archaeology has expanded to 19 regression cases after Round 03.2 IA hard-exclusion traps. This v0.2 table is retained as a historical baseline, not the current count.
|
||||
Round 03 / 03.1 / 03.2 / 03.2a / 04.1 update: QPI has since expanded to 62 regression cases after owner-reviewed case promotion, selector/no-call repair, broad-gate no-call traps, depth-limiting QPI override traps, and Round 04.1 translation-payload / depth-limit / IA anti-recall repairs. Intellectual Archaeology has expanded to 20 regression cases after Round 03.2 and Round 04.1 IA hard-exclusion traps. This v0.2 table is retained as a historical baseline, not the current count.
|
||||
|
||||
## 6. Selector Regression 结论
|
||||
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Status: `PASS`
|
||||
|
||||
Cases checked: 21
|
||||
Cases checked: 26
|
||||
Failures: 0
|
||||
|
||||
## Cases
|
||||
|
|
@ -15,6 +15,7 @@ Failures: 0
|
|||
- `case_ia_pipeline_after_qpi_001`: PASS; selected=['intellectual_archaeology', 'qpi']; no_call=False
|
||||
- `case_ia_negative_explicit_do_not_call_001`: PASS; selected=['qpi']; no_call=False
|
||||
- `case_ia_negative_only_qpi_001`: PASS; selected=['qpi']; no_call=False
|
||||
- `case_ia_round04_do_not_enter_archaeology_001`: PASS; selected=['qpi']; no_call=False
|
||||
- `case_qpi_no_call_direct_edit_001`: PASS; selected=[]; no_call=True
|
||||
- `case_qpi_selector_gate_fact_001`: PASS; selected=[]; no_call=True
|
||||
- `case_qpi_pipeline_before_ia_001`: PASS; selected=['qpi']; no_call=False
|
||||
|
|
@ -28,3 +29,7 @@ Failures: 0
|
|||
- `case_qpi_depth_limited_problem_definition_001`: PASS; selected=['qpi']; no_call=False
|
||||
- `case_qpi_depth_limited_dominant_scarcity_001`: PASS; selected=['qpi']; no_call=False
|
||||
- `case_qpi_analysis_override_should_call_001`: PASS; selected=['qpi']; no_call=False
|
||||
- `case_qpi_round04_translation_payload_no_call_001`: PASS; selected=[]; no_call=True
|
||||
- `case_qpi_round04_depth_limited_specific_scarcity_001`: PASS; selected=['qpi']; no_call=False
|
||||
- `case_qpi_round04_depth_limited_scarcity_set_001`: PASS; selected=['qpi']; no_call=False
|
||||
- `case_qpi_round04_depth_limited_assumption_scarcity_001`: PASS; selected=['qpi']; no_call=False
|
||||
|
|
|
|||
|
|
@ -0,0 +1,184 @@
|
|||
# Skills-Vault Request: Review Bundle Zip Packager
|
||||
|
||||
Date: 2026-06-18
|
||||
|
||||
Client repository: `C:\Users\wangq\Documents\Codex\work-projects\the-mindscape-of-bro-tsong`
|
||||
|
||||
Requested supplier repository:
|
||||
|
||||
```text
|
||||
C:\Users\wangq\Documents\Codex\skills-vault
|
||||
```
|
||||
|
||||
## 1. Product Context
|
||||
|
||||
The Mindscape of Bro Tsong repeatedly prepares CCRA / GPT review bundles under directories such as:
|
||||
|
||||
```text
|
||||
ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/
|
||||
```
|
||||
|
||||
Each bundle often includes an optional raw changed-files zip. The zip must preserve source-relative paths so GPT / CCRA can inspect exact files in their repository context.
|
||||
|
||||
This is reusable automation, not product behavior. It belongs in `skills-vault`, not in this repository.
|
||||
|
||||
## 2. Repeated Action / Friction Point
|
||||
|
||||
Manual zip creation has repeatedly failed or required retries because of:
|
||||
|
||||
- PowerShell `Compress-Archive` flattening paths when given a flat file list.
|
||||
- Windows / .NET compression API differences.
|
||||
- Existing destination zip overwrite behavior.
|
||||
- Need to verify that zip entries preserve source-relative paths.
|
||||
- Need to confirm every requested file exists before packaging.
|
||||
|
||||
This has happened during review bundle preparation and slows down otherwise deterministic handoff work.
|
||||
|
||||
## 3. Proposed Skill Name
|
||||
|
||||
Candidate name:
|
||||
|
||||
```text
|
||||
review-bundle-zip-packager
|
||||
```
|
||||
|
||||
Alternative names:
|
||||
|
||||
```text
|
||||
source-relative-zip
|
||||
review-zip-packager
|
||||
```
|
||||
|
||||
## 4. Inputs
|
||||
|
||||
The Skill should accept:
|
||||
|
||||
### 4.1 File List
|
||||
|
||||
A newline-delimited list of files. Files may be absolute paths or paths resolvable from the current working directory.
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
C:\Users\wangq\Documents\Codex\work-projects\the-mindscape-of-bro-tsong\cards\card_index.md
|
||||
C:\Users\wangq\Documents\Codex\work-projects\the-mindscape-of-bro-tsong\cards\intellectual_archaeology.md
|
||||
```
|
||||
|
||||
### 4.2 Relative Root Marker
|
||||
|
||||
A repository or path segment used to calculate source-relative zip entries.
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
the-mindscape-of-bro-tsong
|
||||
```
|
||||
|
||||
For the first example file above, the zip entry should be:
|
||||
|
||||
```text
|
||||
cards/card_index.md
|
||||
```
|
||||
|
||||
### 4.3 Output Zip Path
|
||||
|
||||
Absolute or cwd-relative path for the destination zip.
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
C:\Users\wangq\Documents\Codex\work-projects\the-mindscape-of-bro-tsong\ccra_review_bundle\round-04_2026-06-18_blind-input-routing-evaluation\optional_raw_changed_files_04.zip
|
||||
```
|
||||
|
||||
## 5. Expected Outputs
|
||||
|
||||
The Skill should produce:
|
||||
|
||||
- The output zip file.
|
||||
- A validation summary printed to stdout.
|
||||
- Optional machine-readable validation JSON next to the zip or printed to stdout when requested.
|
||||
|
||||
Recommended validation summary fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "PASS",
|
||||
"output_zip": "absolute path",
|
||||
"source_file_count": 2,
|
||||
"zip_entry_count": 2,
|
||||
"entries": [
|
||||
"cards/card_index.md",
|
||||
"cards/intellectual_archaeology.md"
|
||||
],
|
||||
"warnings": []
|
||||
}
|
||||
```
|
||||
|
||||
## 6. Required Behavior
|
||||
|
||||
- Preserve source-relative paths inside the zip.
|
||||
- Use forward slashes in zip entry names.
|
||||
- Fail fast if any listed file does not exist.
|
||||
- Fail if any file cannot be made relative to the requested root marker.
|
||||
- Fail if duplicate zip entry names would be produced.
|
||||
- Overwrite the destination zip only after inputs validate.
|
||||
- Verify the written zip by reading it back and comparing actual entries to expected entries.
|
||||
- Print a clear PASS / FAIL result.
|
||||
|
||||
## 7. Safety Boundaries
|
||||
|
||||
- The Skill may create or overwrite only the requested output zip path.
|
||||
- It must not delete source files.
|
||||
- It must not modify source files.
|
||||
- It must not recursively package directories unless a future explicit option is added.
|
||||
- It must not silently ignore missing files.
|
||||
- It must not flatten paths.
|
||||
- It should reject output paths that are the same as one of the source files.
|
||||
|
||||
## 8. File Edits Allowed
|
||||
|
||||
Inside the client repository, this Skill should not edit files other than the requested output zip and optional validation sidecar.
|
||||
|
||||
Inside `skills-vault`, implementation may add normal Skill source files, tests, README updates, and install metadata.
|
||||
|
||||
## 9. Validation Expectations
|
||||
|
||||
Minimum validation cases for `skills-vault`:
|
||||
|
||||
- Two absolute files under the same root marker produce two source-relative entries.
|
||||
- Mixed absolute and cwd-relative files produce correct entries.
|
||||
- Existing destination zip is replaced safely.
|
||||
- Missing source file fails before writing output.
|
||||
- File outside the root marker fails.
|
||||
- Duplicate relative entry names fail.
|
||||
- Zip readback verifies exact entry list.
|
||||
- Windows paths with backslashes are normalized to forward slashes in zip entries.
|
||||
|
||||
Suggested command shape:
|
||||
|
||||
```powershell
|
||||
review-bundle-zip-packager `
|
||||
--file-list files.txt `
|
||||
--relative-root-marker the-mindscape-of-bro-tsong `
|
||||
--output C:\path\to\optional_raw_changed_files_04.zip `
|
||||
--verify
|
||||
```
|
||||
|
||||
Exact CLI naming can be decided in `skills-vault`.
|
||||
|
||||
## 10. Why This Should Become A Reusable Skill
|
||||
|
||||
This is a deterministic file transformation and validation task that recurs across CCRA / GPT review workflows. It is not specific to QPI, Intellectual Archaeology, or this product's model library logic.
|
||||
|
||||
Moving it to `skills-vault` will:
|
||||
|
||||
- Reduce repeated manual zip failures.
|
||||
- Preserve review bundle path correctness.
|
||||
- Provide a reusable Windows-safe packaging command.
|
||||
- Keep automation tooling out of the product repository.
|
||||
|
||||
## 11. Current Blocked / Affected Task
|
||||
|
||||
No current model extraction task is blocked. The need surfaced during Round 04 review bundle preparation, where manual zip creation required retries before source-relative paths were correct.
|
||||
|
||||
This request should be treated as a reusable workflow improvement before future review bundle rounds.
|
||||
|
|
@ -11,6 +11,8 @@ Current script:
|
|||
- `run_selector_regression.py`
|
||||
- `run_selector_demo.py`
|
||||
- `run_selector_calibration_smoke.py`
|
||||
- `run_round04_blind_routing.py`
|
||||
- `run_round04_post_patch_verification.py`
|
||||
|
||||
Run:
|
||||
|
||||
|
|
@ -51,4 +53,20 @@ python scripts\check_model_card_sync.py
|
|||
|
||||
`run_selector_calibration_smoke.py` checks selector calibration inputs, including strict `no_call`, `no_call_or_low_priority`, `select_qpi_reject_ia`, and IA heavy-depth gate expectations.
|
||||
|
||||
Run Round 04 blind routing evaluation:
|
||||
|
||||
```powershell
|
||||
python scripts\run_round04_blind_routing.py
|
||||
```
|
||||
|
||||
This reads `selector/round04_blind_inputs.json` and writes `reports/Round04_blind_routing_evaluation_report_2026-06-18.md`. It does not score correctness or use expected routing labels.
|
||||
|
||||
Run Round 04.1 post-patch routing verification:
|
||||
|
||||
```powershell
|
||||
python scripts\run_round04_post_patch_verification.py
|
||||
```
|
||||
|
||||
This reruns the frozen Round 04 input pool after selector rule patches and writes `reports/Round04_1_post_patch_routing_verification_report_2026-06-18.md`. It is not a second blind test.
|
||||
|
||||
Prefer Python standard library before adding dependencies.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,174 @@
|
|||
import json
|
||||
import sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
if str(SCRIPT_DIR) not in sys.path:
|
||||
sys.path.insert(0, str(SCRIPT_DIR))
|
||||
|
||||
from run_selector_demo import recommend # noqa: E402
|
||||
|
||||
|
||||
REPORT_NAME = "Round04_blind_routing_evaluation_report_2026-06-18.md"
|
||||
|
||||
|
||||
def read_json(path):
|
||||
return json.loads(Path(path).read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def load_blind_inputs(root):
|
||||
return read_json(root / "selector" / "round04_blind_inputs.json")
|
||||
|
||||
|
||||
def selected_model_ids(result):
|
||||
return [item["model_id"] for item in result.get("selected_models", [])]
|
||||
|
||||
|
||||
def rejected_model_ids(result):
|
||||
return [item["model_id"] for item in result.get("rejected_models", [])]
|
||||
|
||||
|
||||
def evaluate_blind_input(root, item):
|
||||
selector_result = recommend(root, item["input_text"])
|
||||
return {
|
||||
"input_id": item["input_id"],
|
||||
"category_for_owner_review": item["category_for_owner_review"],
|
||||
"control_case_type": item.get("control_case_type"),
|
||||
"input_text": item["input_text"],
|
||||
"why_included": item.get("why_included", ""),
|
||||
"selected_models": selected_model_ids(selector_result),
|
||||
"rejected_models": rejected_model_ids(selector_result),
|
||||
"no_call": selector_result.get("no_call"),
|
||||
"routing_notes": selector_result.get("routing_notes", ""),
|
||||
"selected_model_details": selector_result.get("selected_models", []),
|
||||
"rejected_model_details": selector_result.get("rejected_models", []),
|
||||
}
|
||||
|
||||
|
||||
def format_model_details(details):
|
||||
if not details:
|
||||
return [" - none"]
|
||||
lines = []
|
||||
for detail in details:
|
||||
lines.append(
|
||||
f" - `{detail['model_id']}` score={detail['score']} decision={detail['decision']}"
|
||||
)
|
||||
if detail.get("reasons"):
|
||||
lines.append(" - reasons: " + "; ".join(detail["reasons"]))
|
||||
if detail.get("penalties"):
|
||||
lines.append(" - penalties: " + "; ".join(detail["penalties"]))
|
||||
return lines
|
||||
|
||||
|
||||
def render_report(input_set, results):
|
||||
selected_counter = Counter()
|
||||
category_counter = Counter()
|
||||
no_call_count = 0
|
||||
for result in results:
|
||||
category_counter[result["category_for_owner_review"]] += 1
|
||||
if result["no_call"]:
|
||||
no_call_count += 1
|
||||
if not result["selected_models"]:
|
||||
selected_counter["none"] += 1
|
||||
for model_id in result["selected_models"]:
|
||||
selected_counter[model_id] += 1
|
||||
|
||||
lines = [
|
||||
"# Round 04 Blind Routing Evaluation Report",
|
||||
"",
|
||||
"Date: 2026-06-18",
|
||||
"",
|
||||
f"Blind input set: `{input_set['blind_input_set_id']}`",
|
||||
"",
|
||||
"Expected routing labels: `not included before first blind run`",
|
||||
"",
|
||||
"This report records current selector routing only. It does not score correctness, does not generate final answers, does not modify selector rules, and does not upgrade model status.",
|
||||
"",
|
||||
"## Scope",
|
||||
"",
|
||||
"- Round 04 target: test routing stability on new inputs.",
|
||||
"- Runtime surface: current rule-based selector plus model-level QPI / IA gates.",
|
||||
"- QPI contract status: represented here by QPI selection and routing details; no full QPI answer generation is performed.",
|
||||
"- IA status: represented here by selector selection / rejection only; no Intellectual Archaeology answer generation is performed.",
|
||||
"",
|
||||
"## Summary",
|
||||
"",
|
||||
f"Inputs evaluated: {len(results)}",
|
||||
f"No-call results: {no_call_count}",
|
||||
"",
|
||||
"### Selected Model Distribution",
|
||||
"",
|
||||
]
|
||||
|
||||
for key, count in sorted(selected_counter.items()):
|
||||
lines.append(f"- `{key}`: {count}")
|
||||
|
||||
lines.extend([
|
||||
"",
|
||||
"### Coverage Categories",
|
||||
"",
|
||||
])
|
||||
for key, count in sorted(category_counter.items()):
|
||||
lines.append(f"- `{key}`: {count}")
|
||||
|
||||
lines.extend([
|
||||
"",
|
||||
"## Case Results",
|
||||
"",
|
||||
])
|
||||
|
||||
for result in results:
|
||||
lines.extend([
|
||||
f"### {result['input_id']}",
|
||||
"",
|
||||
f"- category_for_owner_review: `{result['category_for_owner_review']}`",
|
||||
])
|
||||
if result.get("control_case_type"):
|
||||
lines.append(f"- control_case_type: `{result['control_case_type']}`")
|
||||
lines.extend([
|
||||
f"- input_text: `{result['input_text']}`",
|
||||
f"- selected_models: `{result['selected_models']}`",
|
||||
f"- rejected_models: `{result['rejected_models']}`",
|
||||
f"- no_call: `{result['no_call']}`",
|
||||
f"- routing_notes: {result['routing_notes']}",
|
||||
"",
|
||||
"Selected model details:",
|
||||
])
|
||||
lines.extend(format_model_details(result["selected_model_details"]))
|
||||
lines.extend(["", "Rejected model details:"])
|
||||
lines.extend(format_model_details(result["rejected_model_details"]))
|
||||
lines.append("")
|
||||
|
||||
lines.extend([
|
||||
"## Review Reminder",
|
||||
"",
|
||||
"This is the first blind routing output. Failure labels, expected routing, and rule changes should be added only after owner / GPT / CCRA review of these observed results.",
|
||||
"",
|
||||
])
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def write_report(root, input_set, results):
|
||||
report_path = root / "reports" / REPORT_NAME
|
||||
report_path.write_text(render_report(input_set, results), encoding="utf-8")
|
||||
return report_path
|
||||
|
||||
|
||||
def main():
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
input_set = load_blind_inputs(root)
|
||||
results = [
|
||||
evaluate_blind_input(root, item)
|
||||
for item in input_set.get("inputs", [])
|
||||
]
|
||||
report_path = write_report(root, input_set, results)
|
||||
print(f"round04 blind routing report written to {report_path}")
|
||||
print(f"inputs evaluated: {len(results)}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -0,0 +1,276 @@
|
|||
import ast
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
if str(SCRIPT_DIR) not in sys.path:
|
||||
sys.path.insert(0, str(SCRIPT_DIR))
|
||||
|
||||
from run_round04_blind_routing import evaluate_blind_input, load_blind_inputs # noqa: E402
|
||||
|
||||
|
||||
REPORT_NAME = "Round04_1_post_patch_routing_verification_report_2026-06-18.md"
|
||||
ORIGINAL_REPORT = "Round04_blind_routing_evaluation_report_2026-06-18.md"
|
||||
|
||||
TARGET_EXPECTATIONS = {
|
||||
"R04-BI-002": {
|
||||
"expected_selected_models": [],
|
||||
"expected_no_call": True,
|
||||
"must_not_select": ["qpi", "intellectual_archaeology"],
|
||||
"review_label": "translation no-call; payload complexity must not trigger QPI",
|
||||
},
|
||||
"R04-BI-024": {
|
||||
"expected_selected_models": ["qpi"],
|
||||
"expected_no_call": False,
|
||||
"must_not_select": ["intellectual_archaeology"],
|
||||
"review_label": "explicit IA refusal should hard-exclude IA while allowing QPI",
|
||||
},
|
||||
"R04-BI-035": {
|
||||
"expected_selected_models": ["qpi"],
|
||||
"expected_no_call": False,
|
||||
"must_not_select": ["intellectual_archaeology"],
|
||||
"review_label": "depth-limiting plus dominant-scarcity judgment should allow QPI",
|
||||
},
|
||||
"R04-BI-036": {
|
||||
"expected_selected_models": ["qpi"],
|
||||
"expected_no_call": False,
|
||||
"must_not_select": ["intellectual_archaeology"],
|
||||
"review_label": "depth-limiting plus scarcity-set judgment should allow QPI",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
DIFF_CASE_NOTES = {
|
||||
"R04-BI-002": {
|
||||
"label": "target fix: translation no-call",
|
||||
"judgment": "Expected target change; translation instruction should not be routed to QPI because the payload contains `模型`.",
|
||||
},
|
||||
"R04-BI-022": {
|
||||
"label": "accepted collateral change: depth-limited QPI override",
|
||||
"judgment": "Not a new failure. The request rejects deeper IA-style expansion while asking for dominant scarcity judgment, so lightweight QPI is acceptable and IA should remain rejected. Added to regression as a QPI selector-gate trap.",
|
||||
},
|
||||
"R04-BI-024": {
|
||||
"label": "target fix: explicit IA refusal",
|
||||
"judgment": "Expected target change; explicit refusal should hard-exclude IA while allowing lightweight QPI.",
|
||||
},
|
||||
"R04-BI-035": {
|
||||
"label": "target fix: dominant-scarcity depth limit",
|
||||
"judgment": "Expected target change; depth-limiting language should not block lightweight QPI scarcity judgment.",
|
||||
},
|
||||
"R04-BI-036": {
|
||||
"label": "target fix: scarcity-set depth limit",
|
||||
"judgment": "Expected target change; depth-limiting language should not block QPI scarcity-set judgment.",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def parse_original_report(root):
|
||||
report_path = root / "reports" / ORIGINAL_REPORT
|
||||
text = report_path.read_text(encoding="utf-8")
|
||||
case_pattern = re.compile(r"^### (R04-BI-\d+)\n(?P<body>.*?)(?=^### R04-BI-|\Z)", re.M | re.S)
|
||||
cases = {}
|
||||
for match in case_pattern.finditer(text):
|
||||
case_id = match.group(1)
|
||||
body = match.group("body")
|
||||
selected = extract_list_field(body, "selected_models")
|
||||
rejected = extract_list_field(body, "rejected_models")
|
||||
no_call = extract_bool_field(body, "no_call")
|
||||
cases[case_id] = {
|
||||
"selected_models": selected,
|
||||
"rejected_models": rejected,
|
||||
"no_call": no_call,
|
||||
}
|
||||
return cases
|
||||
|
||||
|
||||
def extract_list_field(body, field_name):
|
||||
match = re.search(rf"- {field_name}: `(?P<value>.*?)`", body)
|
||||
if not match:
|
||||
return []
|
||||
return ast.literal_eval(match.group("value"))
|
||||
|
||||
|
||||
def extract_bool_field(body, field_name):
|
||||
match = re.search(rf"- {field_name}: `(?P<value>True|False)`", body)
|
||||
if not match:
|
||||
return None
|
||||
return match.group("value") == "True"
|
||||
|
||||
|
||||
def evaluate_expectation(result, expectation):
|
||||
errors = []
|
||||
selected = result["selected_models"]
|
||||
for model_id in expectation["expected_selected_models"]:
|
||||
if model_id not in selected:
|
||||
errors.append(f"expected selected model missing: {model_id}")
|
||||
for model_id in expectation["must_not_select"]:
|
||||
if model_id in selected:
|
||||
errors.append(f"forbidden selected model: {model_id}")
|
||||
if result["no_call"] is not expectation["expected_no_call"]:
|
||||
errors.append(f"expected no_call={expectation['expected_no_call']}, got {result['no_call']}")
|
||||
if not expectation["expected_selected_models"] and selected:
|
||||
errors.append(f"expected no selected models, got {selected}")
|
||||
return errors
|
||||
|
||||
|
||||
def build_results(root):
|
||||
original = parse_original_report(root)
|
||||
input_set = load_blind_inputs(root)
|
||||
after_results = {
|
||||
item["input_id"]: evaluate_blind_input(root, item)
|
||||
for item in input_set["inputs"]
|
||||
}
|
||||
targeted = []
|
||||
for case_id, expectation in TARGET_EXPECTATIONS.items():
|
||||
after = after_results[case_id]
|
||||
errors = evaluate_expectation(after, expectation)
|
||||
targeted.append({
|
||||
"case_id": case_id,
|
||||
"review_label": expectation["review_label"],
|
||||
"status": "PASS" if not errors else "FAIL",
|
||||
"errors": errors,
|
||||
"before": original.get(case_id, {}),
|
||||
"after": after,
|
||||
})
|
||||
behavior_diffs = []
|
||||
for case_id, before in original.items():
|
||||
after = after_results.get(case_id)
|
||||
if not after:
|
||||
continue
|
||||
if (
|
||||
before.get("selected_models") != after["selected_models"]
|
||||
or before.get("no_call") != after["no_call"]
|
||||
):
|
||||
behavior_diffs.append({
|
||||
"case_id": case_id,
|
||||
"before": before,
|
||||
"after": after,
|
||||
"note": DIFF_CASE_NOTES.get(case_id, {
|
||||
"label": "unclassified behavior change",
|
||||
"judgment": "Requires owner review.",
|
||||
}),
|
||||
})
|
||||
return input_set, list(after_results.values()), targeted, behavior_diffs
|
||||
|
||||
|
||||
def build_post_patch_report(root):
|
||||
input_set, all_results, targeted, behavior_diffs = build_results(root)
|
||||
failures = [item for item in targeted if item["errors"]]
|
||||
no_call_count = sum(1 for item in all_results if item["no_call"])
|
||||
|
||||
lines = [
|
||||
"# Round 04.1 Post-Patch Routing Verification Report",
|
||||
"",
|
||||
"Date: 2026-06-18",
|
||||
"",
|
||||
"This is not a second blind test.",
|
||||
"",
|
||||
f"Frozen input set: `{input_set['blind_input_set_id']}`",
|
||||
f"Original before-report: `reports/{ORIGINAL_REPORT}`",
|
||||
"",
|
||||
"## Scope",
|
||||
"",
|
||||
"- Round 04.1 verifies selector rule patches after owner / GPT review.",
|
||||
"- It reruns the frozen 38-input Round 04 pool for comparison.",
|
||||
"- Expected routing labels are limited to the four review-approved target cases.",
|
||||
"- It does not generate final answers and does not upgrade model status.",
|
||||
"",
|
||||
"## Summary",
|
||||
"",
|
||||
f"Frozen inputs rerun: {len(all_results)}",
|
||||
f"Post-patch no-call results: {no_call_count}",
|
||||
f"Targeted cases checked: {len(targeted)}",
|
||||
f"Targeted failures: {len(failures)}",
|
||||
f"Behavior changes recorded: {len(behavior_diffs)}",
|
||||
"",
|
||||
"## Targeted Case Results",
|
||||
"",
|
||||
]
|
||||
|
||||
for item in targeted:
|
||||
after = item["after"]
|
||||
before = item["before"]
|
||||
lines.extend([
|
||||
f"### {item['case_id']}",
|
||||
"",
|
||||
f"- status: `{item['status']}`",
|
||||
f"- review_label: {item['review_label']}",
|
||||
f"- before_selected_models: `{before.get('selected_models')}`",
|
||||
f"- before_no_call: `{before.get('no_call')}`",
|
||||
f"- after_selected_models: `{after['selected_models']}`",
|
||||
f"- after_rejected_models: `{after['rejected_models']}`",
|
||||
f"- after_no_call: `{after['no_call']}`",
|
||||
])
|
||||
if item["errors"]:
|
||||
lines.append("- errors:")
|
||||
for error in item["errors"]:
|
||||
lines.append(f" - {error}")
|
||||
lines.extend([
|
||||
"",
|
||||
"After selected model details:",
|
||||
])
|
||||
if after["selected_model_details"]:
|
||||
for detail in after["selected_model_details"]:
|
||||
lines.append(f"- `{detail['model_id']}` score={detail['score']} decision={detail['decision']}")
|
||||
if detail.get("reasons"):
|
||||
lines.append(" - reasons: " + "; ".join(detail["reasons"]))
|
||||
if detail.get("penalties"):
|
||||
lines.append(" - penalties: " + "; ".join(detail["penalties"]))
|
||||
else:
|
||||
lines.append("- none")
|
||||
lines.append("")
|
||||
|
||||
lines.extend([
|
||||
"## Non-target Behavior Diff / Accepted Collateral Change",
|
||||
"",
|
||||
"The frozen 38-input pool was rerun for comparison. Expected routing labels remain limited to the four target cases above, but the full rerun records five behavior changes from the original Round 04 report.",
|
||||
"",
|
||||
])
|
||||
|
||||
for item in behavior_diffs:
|
||||
before = item["before"]
|
||||
after = item["after"]
|
||||
note = item["note"]
|
||||
lines.extend([
|
||||
f"### {item['case_id']}",
|
||||
"",
|
||||
f"- label: {note['label']}",
|
||||
f"- before_selected_models: `{before.get('selected_models')}`",
|
||||
f"- before_no_call: `{before.get('no_call')}`",
|
||||
f"- after_selected_models: `{after['selected_models']}`",
|
||||
f"- after_rejected_models: `{after['rejected_models']}`",
|
||||
f"- after_no_call: `{after['no_call']}`",
|
||||
f"- closeout_judgment: {note['judgment']}",
|
||||
"",
|
||||
])
|
||||
|
||||
lines.extend([
|
||||
"## Full Pool Rerun Summary",
|
||||
"",
|
||||
])
|
||||
for result in all_results:
|
||||
lines.append(
|
||||
f"- `{result['input_id']}`: selected={result['selected_models']}; "
|
||||
f"no_call={result['no_call']}; category={result['category_for_owner_review']}"
|
||||
)
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def write_report(root):
|
||||
report_path = root / "reports" / REPORT_NAME
|
||||
report_path.write_text(build_post_patch_report(root), encoding="utf-8")
|
||||
return report_path
|
||||
|
||||
|
||||
def main():
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
report_path = write_report(root)
|
||||
print(f"round04.1 post-patch verification report written to {report_path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -23,6 +23,14 @@ def hit_any(user_input, signals):
|
|||
return [signal for signal in signals if signal in user_input]
|
||||
|
||||
|
||||
def instruction_segment(user_input):
|
||||
segments = [user_input]
|
||||
for delimiter in ("：", ":", "\n"):
|
||||
if delimiter in user_input:
|
||||
segments.append(user_input.split(delimiter, 1)[0])
|
||||
return min(segments, key=len)
|
||||
|
||||
|
||||
def has_analysis_override(user_input, global_rules):
|
||||
return bool(hit_any(user_input, global_rules.get("explicit_analysis_override_phrases", [])))
|
||||
|
||||
|
|
@ -30,7 +38,24 @@ def has_analysis_override(user_input, global_rules):
|
|||
def hard_no_call_hits(user_input, global_rules):
|
||||
signals = list(global_rules.get("hard_no_call_signals", []))
|
||||
signals.extend(global_rules.get("direct_execution_no_call_signals", []))
|
||||
return hit_any(user_input, signals)
|
||||
hits = hit_any(user_input, signals)
|
||||
instruction_hits = instruction_task_no_call_hits(
|
||||
user_input,
|
||||
global_rules.get("instruction_task_no_call_signals", []),
|
||||
)
|
||||
return hits + instruction_hits
|
||||
|
||||
|
||||
def instruction_task_no_call_hits(user_input, signals):
|
||||
instruction = instruction_segment(user_input)
|
||||
if instruction != user_input:
|
||||
return hit_any(instruction, signals)
|
||||
stripped = user_input.strip()
|
||||
return [
|
||||
signal
|
||||
for signal in signals
|
||||
if 0 <= stripped.find(signal) <= 4
|
||||
]
|
||||
|
||||
|
||||
def depth_limited_qpi_override_hits(user_input, global_rules):
|
||||
|
|
|
|||
|
|
@ -30,11 +30,19 @@ Round 03.2a selector rule:
|
|||
- Depth-limiting phrases such as `不要展开` and `不要深入分析` remain global no-call signals by default.
|
||||
- If a depth-limiting phrase is paired with explicit QPI-limited intent such as `只做 QPI`, `只做问题定性`, or `只判断主导稀缺`, the selector should continue scoring, allow QPI, and still reject IA.
|
||||
|
||||
Round 04.1 selector rule:
|
||||
|
||||
- Translation / direct transformation instructions should no-call even when the payload contains complexity words such as `模型`.
|
||||
- Task instruction and payload content should be separated for no-call routing when a direct transformation phrase appears before a colon.
|
||||
- IA refusal variants such as `不要进入思想考古` and `不做思想考古` are model-level hard exclusions for IA.
|
||||
- Depth-limiting phrases paired with `主导稀缺` or `缺信息 / 缺路径 / 缺共识` intent should allow QPI and reject IA.
|
||||
|
||||
Current files:
|
||||
|
||||
- `selector_rules.json`
|
||||
- `selector_examples.json`
|
||||
- `selector_calibration_inputs.json`
|
||||
- `round04_blind_inputs.json`
|
||||
- `qpi_case_digests.json`
|
||||
|
||||
The selector does not call an LLM.
|
||||
|
|
@ -45,6 +53,12 @@ Selector regression is `scripts/run_selector_regression.py`.
|
|||
|
||||
Selector calibration smoke check is `scripts/run_selector_calibration_smoke.py`.
|
||||
|
||||
Round 04 blind routing evaluation is `scripts/run_round04_blind_routing.py`.
|
||||
|
||||
Round 04.1 post-patch routing verification is `scripts/run_round04_post_patch_verification.py`.
|
||||
|
||||
`round04_blind_inputs.json` is a frozen blind input pool for first-run routing evaluation. It intentionally contains no expected selector behavior and should not be merged into calibration inputs before owner / GPT / CCRA review.
|
||||
|
||||
`qpi_case_digests.json` stores owner-reviewed QPI case digests that can feed selector calibration or later regression selection. It is not a model status upgrade and should not include unreviewed draft cases.
|
||||
|
||||
Digest field notes:
|
||||
|
|
|
|||
|
|
@ -0,0 +1,253 @@
|
|||
{
|
||||
"blind_input_set_id": "round04_blind_inputs_2026-06-18_v0.1",
|
||||
"status": "frozen_for_first_blind_run_after_owner_review",
|
||||
"source_document": "reports/Round04_blind_input_candidates_2026-06-18.md",
|
||||
"created_date": "2026-06-18",
|
||||
"notes": [
|
||||
"This file is the frozen Round 04 blind input pool.",
|
||||
"It intentionally contains no expected selector behavior and no owner answer key.",
|
||||
"Category labels are coverage labels for review, not expected routing labels.",
|
||||
"Selector rules must not be modified before the first blind run."
|
||||
],
|
||||
"constraints": {
|
||||
"no_expected_routing": true,
|
||||
"do_not_modify_selector_before_first_run": true,
|
||||
"do_not_merge_into_calibration_inputs": true,
|
||||
"do_not_add_third_model": true,
|
||||
"do_not_upgrade_model_status": true,
|
||||
"do_not_run_answer_generation": true
|
||||
},
|
||||
"inputs": [
|
||||
{
|
||||
"input_id": "R04-BI-001",
|
||||
"category_for_owner_review": "explicit_no_call_fact_lookup",
|
||||
"input_text": "查一下《原则》这本书的作者是谁，只要事实，不要分析。",
|
||||
"why_included": "Tests whether direct fact lookup stays outside model routing."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-002",
|
||||
"category_for_owner_review": "explicit_no_call_translation",
|
||||
"input_text": "把这句话翻译成英文：这个模型还处在草稿阶段。",
|
||||
"why_included": "Tests pure translation as no-call."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-003",
|
||||
"category_for_owner_review": "explicit_no_call_rewrite",
|
||||
"input_text": "请把这段话润色得更顺一点，不要改变意思，也不要解释原因。",
|
||||
"why_included": "Tests polish-only request with explicit no-analysis boundary."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-004",
|
||||
"category_for_owner_review": "explicit_no_call_formatting",
|
||||
"input_text": "把下面这组要点整理成 Markdown 表格，直接给结果。",
|
||||
"why_included": "Tests formatting / direct execution without cognitive model need."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-005",
|
||||
"category_for_owner_review": "explicit_no_call_direct_execution",
|
||||
"input_text": "把这个 JSON 压缩成单行，不用说明：{\"status\":\"draft\",\"level\":\"B\"}",
|
||||
"why_included": "Tests direct transformation request."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-006",
|
||||
"category_for_owner_review": "qpi_light_problem_definition",
|
||||
"input_text": "我们现在反复开会但没有推进，我想先判断这是缺信息、缺路径，还是缺共识。",
|
||||
"why_included": "Tests lightweight problem-definition language without asking for depth."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-007",
|
||||
"category_for_owner_review": "qpi_light_problem_definition",
|
||||
"input_text": "这个项目卡住了，我不确定到底是问题没定义清楚，还是执行方案不够。",
|
||||
"why_included": "Tests QPI-like framing without naming QPI."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-008",
|
||||
"category_for_owner_review": "qpi_light_problem_definition",
|
||||
"input_text": "我们有一堆材料，但不知道下一步该问什么问题，先帮我定性一下。",
|
||||
"why_included": "Tests question/problem boundary and data/path scarcity."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-009",
|
||||
"category_for_owner_review": "qpi_light_problem_definition",
|
||||
"input_text": "这件事到底算一个简单问题、复杂问题，还是已经变成组织议题？",
|
||||
"why_included": "Tests lightweight classification among question / problem / issue."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-010",
|
||||
"category_for_owner_review": "governance_responsibility_consensus",
|
||||
"input_text": "每个部门都说支持这个项目，但一到分工就没人愿意承担主责。",
|
||||
"why_included": "Tests responsibility ambiguity and consensus scarcity."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-011",
|
||||
"category_for_owner_review": "governance_responsibility_consensus",
|
||||
"input_text": "老板要创新，财务要控成本，业务要短期业绩，最后方案总是被折中到没有方向。",
|
||||
"why_included": "Tests cross-stakeholder conflict and governance weight."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-012",
|
||||
"category_for_owner_review": "governance_responsibility_consensus",
|
||||
"input_text": "我们不是不知道怎么做，而是每个人都在等别人先表态。",
|
||||
"why_included": "Tests responsibility / commitment issue where path may already exist."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-013",
|
||||
"category_for_owner_review": "governance_responsibility_consensus",
|
||||
"input_text": "这个流程改造牵涉教务、学院和信息化部门，谁都能提意见，但没人能拍板。",
|
||||
"why_included": "Tests multi-party ownership and decision authority gap."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-014",
|
||||
"category_for_owner_review": "capacity_resource_path",
|
||||
"input_text": "团队现在知道目标，也认可方向，但只有两个人能做关键模块，进度一直被产能卡住。",
|
||||
"why_included": "Tests resource / capacity scarcity rather than data scarcity."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-015",
|
||||
"category_for_owner_review": "capacity_resource_path",
|
||||
"input_text": "我们知道要做知识库，也有材料，但不知道从哪条最小路径开始。",
|
||||
"why_included": "Tests path scarcity with sufficient source material."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-016",
|
||||
"category_for_owner_review": "capacity_resource_path",
|
||||
"input_text": "预算不是最大问题，问题是现有系统、权限和审批流程都接不上。",
|
||||
"why_included": "Tests operational path constraints and institutional friction."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-017",
|
||||
"category_for_owner_review": "capacity_resource_path",
|
||||
"input_text": "这件事大家都觉得重要，但每周只能挤出零散时间，所以一直没有稳定产出。",
|
||||
"why_included": "Tests capacity scarcity mixed with execution continuity."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-018",
|
||||
"category_for_owner_review": "model_extraction_or_depth_modeling",
|
||||
"control_case_type": "positive_control",
|
||||
"input_text": "请从这篇长文里抽取一个可复用的认知模型，重点看它的核心机制和适用边界。",
|
||||
"why_included": "Tests model-extraction intent and possible depth-modeling route."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-019",
|
||||
"category_for_owner_review": "model_extraction_or_depth_modeling",
|
||||
"input_text": "这套说法背后好像有一个稳定结构，不只是观点总结，帮我判断它能不能沉淀成模型。",
|
||||
"why_included": "Tests model-mining request without naming Intellectual Archaeology."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-020",
|
||||
"category_for_owner_review": "model_extraction_or_depth_modeling",
|
||||
"input_text": "我想把这段讨论拆成原始判断、隐含假设、适用条件和误用边界。",
|
||||
"why_included": "Tests structured depth analysis intent."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-021",
|
||||
"category_for_owner_review": "qpi_allowed_ia_refused",
|
||||
"control_case_type": "model_name_exposed",
|
||||
"input_text": "先不要思想考古，只判断这是不是一个 QPI 意义上的问题。",
|
||||
"why_included": "Tests explicit IA refusal while allowing QPI."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-022",
|
||||
"category_for_owner_review": "qpi_allowed_ia_refused",
|
||||
"input_text": "不要展开到底层假设，只帮我看主导稀缺是什么。",
|
||||
"why_included": "Tests depth-limiting QPI override without naming QPI."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-023",
|
||||
"category_for_owner_review": "qpi_allowed_ia_refused",
|
||||
"input_text": "这次先不做深度建模，只做问题定性，看它是缺数据、缺路径还是缺共识。",
|
||||
"why_included": "Tests QPI-limited intent plus explicit depth refusal."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-024",
|
||||
"category_for_owner_review": "qpi_allowed_ia_refused",
|
||||
"input_text": "不要进入思想考古，我只要一个轻量判断：现在应该先问问题、找路径，还是处理责任边界？",
|
||||
"why_included": "Tests model-level hard exclusion for IA while preserving QPI-like routing."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-025",
|
||||
"category_for_owner_review": "likely_intellectual_archaeology",
|
||||
"control_case_type": "model_name_exposed",
|
||||
"input_text": "QPI 已经判断这是中重型课题，现在请继续分析它背后的底层假设、生成机制和长期稳定结构。",
|
||||
"why_included": "Tests explicit post-QPI depth handoff."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-026",
|
||||
"category_for_owner_review": "likely_intellectual_archaeology",
|
||||
"input_text": "这个现象反复出现在不同项目里，我想知道它背后的思想结构是什么，而不是只做问题分类。",
|
||||
"why_included": "Tests repeated-pattern depth modeling beyond lightweight QPI."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-027",
|
||||
"category_for_owner_review": "likely_intellectual_archaeology",
|
||||
"input_text": "请分析这套判断为什么会长期成立，它依赖哪些世界观、因果假设和行动边界。",
|
||||
"why_included": "Tests deep assumption / mechanism request."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-028",
|
||||
"category_for_owner_review": "ambiguous_low_context",
|
||||
"input_text": "这个不太对。",
|
||||
"why_included": "Tests very low context and likely need for clarification."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-029",
|
||||
"category_for_owner_review": "ambiguous_low_context",
|
||||
"input_text": "怎么又卡住了？",
|
||||
"why_included": "Tests short ambiguous frustration signal."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-030",
|
||||
"category_for_owner_review": "ambiguous_low_context",
|
||||
"input_text": "还是那个老问题。",
|
||||
"why_included": "Tests prior-context dependency without enough current evidence."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-031",
|
||||
"category_for_owner_review": "explicit_no_call_how_to",
|
||||
"input_text": "如何在 Excel 里冻结首行？",
|
||||
"why_included": "Tests whether generic how-to stays outside QPI after broad \"如何\" gate was narrowed."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-032",
|
||||
"category_for_owner_review": "explicit_no_call_how_to",
|
||||
"input_text": "怎么把 Markdown 文件转成 PDF？",
|
||||
"why_included": "Tests tool/how-to request as no-call."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-033",
|
||||
"category_for_owner_review": "explicit_no_call_dictionary_judgment",
|
||||
"input_text": "判断一下“alignment”在这句话里是什么意思。",
|
||||
"why_included": "Tests whether ordinary word-meaning judgment avoids QPI over-selection."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-034",
|
||||
"category_for_owner_review": "direct_execution_with_analysis_override",
|
||||
"input_text": "请把下面内容总结成三点，但不要只执行，先判断它背后的问题定义。",
|
||||
"why_included": "Tests whether explicit analysis override can escape direct-execution no-call."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-035",
|
||||
"category_for_owner_review": "qpi_allowed_depth_limited",
|
||||
"input_text": "不要深入分析，只判断这件事的主导稀缺是什么。",
|
||||
"why_included": "Tests Round 03.2a depth-limiting override without naming QPI or IA."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-036",
|
||||
"category_for_owner_review": "qpi_allowed_depth_limited",
|
||||
"input_text": "不要展开到底层理论，只判断现在是缺信息、缺路径还是缺共识。",
|
||||
"why_included": "Tests QPI-limited routing while rejecting deeper modeling."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-037",
|
||||
"category_for_owner_review": "ambiguous_growth_how_to",
|
||||
"input_text": "如何提高这个账号的流量？",
|
||||
"why_included": "Tests whether generic growth how-to is routed to QPI, no-call, or low-confidence provisional handling."
|
||||
},
|
||||
{
|
||||
"input_id": "R04-BI-038",
|
||||
"category_for_owner_review": "likely_intellectual_archaeology",
|
||||
"input_text": "这类失败在不同项目里反复出现，我想抽出它背后稳定的生成机制和可迁移框架。",
|
||||
"why_included": "Tests deeper modeling intent without explicit \"思想考古\" or \"底层假设\" wording."
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
{
|
||||
"selector_version": "0.2",
|
||||
"selector_version": "0.2.1-draft",
|
||||
"selection_method": "rule_based_no_llm",
|
||||
"global_rules": {
|
||||
"negative_trigger_first": true,
|
||||
|
|
@ -35,6 +35,14 @@
|
|||
"不用解释",
|
||||
"生成图片"
|
||||
],
|
||||
"instruction_task_no_call_signals": [
|
||||
"翻译成",
|
||||
"翻译为",
|
||||
"把这句话翻译",
|
||||
"翻译下面",
|
||||
"译成英文",
|
||||
"译成中文"
|
||||
],
|
||||
"depth_limiting_no_call_signals": [
|
||||
"不要展开",
|
||||
"不要深入分析"
|
||||
|
|
@ -44,6 +52,10 @@
|
|||
"只做问题定性",
|
||||
"只做问题定性复核",
|
||||
"只判断主导稀缺",
|
||||
"只判断这件事的主导稀缺",
|
||||
"主导稀缺",
|
||||
"只判断现在是缺信息",
|
||||
"缺信息、缺路径还是缺共识",
|
||||
"只判断这是问题还是课题",
|
||||
"只判断问题类型"
|
||||
],
|
||||
|
|
@ -84,6 +96,7 @@
|
|||
],
|
||||
"qpi_gate_signals": [
|
||||
"缺数据",
|
||||
"缺信息",
|
||||
"缺路径",
|
||||
"缺共识",
|
||||
"问题到底",
|
||||
|
|
@ -95,7 +108,10 @@
|
|||
"如何定义这个问题",
|
||||
"如何提高流量",
|
||||
"如何区分 Q/P/I",
|
||||
"主导稀缺",
|
||||
"判断主导稀缺",
|
||||
"只判断这件事的主导稀缺",
|
||||
"缺信息、缺路径还是缺共识",
|
||||
"判断这是问题还是课题",
|
||||
"判断应该调用哪个模型",
|
||||
"只做 QPI",
|
||||
|
|
@ -221,17 +237,31 @@
|
|||
"马上执行",
|
||||
"不用解释",
|
||||
"先不要思想考古",
|
||||
"不要思想考古"
|
||||
"不要思想考古",
|
||||
"不要进入思想考古",
|
||||
"不进入思想考古",
|
||||
"先不进入思想考古",
|
||||
"不要做思想考古",
|
||||
"不做思想考古"
|
||||
],
|
||||
"hard_exclusion_triggers": [
|
||||
"不要思想考古",
|
||||
"先不要思想考古",
|
||||
"不要进入思想考古",
|
||||
"不进入思想考古",
|
||||
"先不进入思想考古",
|
||||
"不要做思想考古",
|
||||
"不做思想考古",
|
||||
"不要调用思想考古",
|
||||
"不调用思想考古",
|
||||
"只做 QPI",
|
||||
"只做问题定性",
|
||||
"只做问题定性复核",
|
||||
"只判断主导稀缺",
|
||||
"只判断这件事的主导稀缺",
|
||||
"主导稀缺",
|
||||
"只判断现在是缺信息",
|
||||
"缺信息、缺路径还是缺共识",
|
||||
"只判断这是问题还是课题",
|
||||
"只判断问题类型"
|
||||
],
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@
|
|||
{"case_id": "case_ia_selector_gate_no_heavy_signal_001", "model_id": "intellectual_archaeology", "case_type": "selector_gate", "input": "底层这个词怎么翻译？只要一个英文。", "expected_behavior": "虽然有底层一词，也不应召回思想考古。", "failure_signal": "关键词误召回。", "expected_output_elements": ["rejected_models"], "should_call_model": false, "expected_primary_model": "none", "negative_expected_models": ["intellectual_archaeology"], "expected_classification": "not_applicable", "expected_dominant_scarcity": "not_applicable", "expected_max_depth": "no_call", "minimum_required_elements": ["negative trigger"], "forbidden_elements": ["七层"], "evaluation_mode": "structured", "notes": "关键词误召回防护。"},
|
||||
{"case_id": "case_ia_pipeline_after_qpi_001", "model_id": "intellectual_archaeology", "case_type": "pipeline", "input": "QPI 已判断为中重型课题，且需要抽取底层机制和验证路径。", "expected_behavior": "IA 可作为 QPI 后续模型被调用。", "failure_signal": "在 QPI 放行后仍拒绝。", "expected_output_elements": ["QPI 后续", "should_call=true"], "should_call_model": true, "expected_primary_model": "intellectual_archaeology", "negative_expected_models": [], "expected_classification": "not_applicable", "expected_dominant_scarcity": "not_applicable", "expected_max_depth": "core_mechanism", "minimum_required_elements": ["entry_reason"], "forbidden_elements": [], "evaluation_mode": "structured", "notes": "QPI 后续 pipeline。"},
|
||||
{"case_id": "case_ia_negative_explicit_do_not_call_001", "model_id": "intellectual_archaeology", "case_type": "selector_gate", "input": "QPI 已判断：这是 Issue。先不要思想考古，只做问题定性复核。", "expected_behavior": "selector 应选择 QPI，并硬拒绝思想考古。", "failure_signal": "因为“思想考古”和“QPI 已判断”加分而仍选中 IA。", "expected_output_elements": ["selected=qpi", "rejected=intellectual_archaeology"], "should_call_model": false, "expected_primary_model": "qpi", "negative_expected_models": ["intellectual_archaeology"], "expected_classification": "not_applicable", "expected_dominant_scarcity": "not_applicable", "expected_max_depth": "no_call", "minimum_required_elements": ["model_level_hard_exclusion"], "forbidden_elements": ["selected IA"], "evaluation_mode": "structured", "notes": "Round 03.2 hard exclusion trap: explicit IA refusal must override score."},
|
||||
{"case_id": "case_ia_negative_only_qpi_001", "model_id": "intellectual_archaeology", "case_type": "selector_gate", "input": "不要思想考古，只做 QPI 复核。", "expected_behavior": "selector 应选择 QPI，并硬拒绝思想考古。", "failure_signal": "IA 负触发只被当作 penalty，仍可能被选中。", "expected_output_elements": ["selected=qpi", "rejected=intellectual_archaeology"], "should_call_model": false, "expected_primary_model": "qpi", "negative_expected_models": ["intellectual_archaeology"], "expected_classification": "not_applicable", "expected_dominant_scarcity": "not_applicable", "expected_max_depth": "no_call", "minimum_required_elements": ["model_level_hard_exclusion"], "forbidden_elements": ["selected IA"], "evaluation_mode": "structured", "notes": "Round 03.2 hard exclusion trap: only-QPI instruction rejects IA."}
|
||||
{"case_id": "case_ia_negative_only_qpi_001", "model_id": "intellectual_archaeology", "case_type": "selector_gate", "input": "不要思想考古，只做 QPI 复核。", "expected_behavior": "selector 应选择 QPI，并硬拒绝思想考古。", "failure_signal": "IA 负触发只被当作 penalty，仍可能被选中。", "expected_output_elements": ["selected=qpi", "rejected=intellectual_archaeology"], "should_call_model": false, "expected_primary_model": "qpi", "negative_expected_models": ["intellectual_archaeology"], "expected_classification": "not_applicable", "expected_dominant_scarcity": "not_applicable", "expected_max_depth": "no_call", "minimum_required_elements": ["model_level_hard_exclusion"], "forbidden_elements": ["selected IA"], "evaluation_mode": "structured", "notes": "Round 03.2 hard exclusion trap: only-QPI instruction rejects IA."},
|
||||
{"case_id": "case_ia_round04_do_not_enter_archaeology_001", "model_id": "intellectual_archaeology", "case_type": "selector_gate", "input": "不要进入思想考古，我只要一个轻量判断：现在应该先问问题、找路径，还是处理责任边界？", "expected_behavior": "selector 应选择或允许 QPI，并硬拒绝思想考古。", "failure_signal": "因为“思想考古”和复杂度信号加分而仍选中 IA。", "expected_output_elements": ["selected=qpi", "rejected=intellectual_archaeology"], "should_call_model": false, "expected_primary_model": "qpi", "negative_expected_models": ["intellectual_archaeology"], "expected_classification": "not_applicable", "expected_dominant_scarcity": "not_applicable", "expected_max_depth": "no_call", "minimum_required_elements": ["model_level_hard_exclusion"], "forbidden_elements": ["selected IA"], "evaluation_mode": "structured", "notes": "Round 04.1 target from R04-BI-024: explicit 'do not enter IA' must hard-exclude IA."}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1748,6 +1748,124 @@
|
|||
],
|
||||
"evaluation_mode": "structured",
|
||||
"notes": "Round 03.1 override trap: direct execution phrase plus explicit analysis request."
|
||||
},
|
||||
{
|
||||
"case_id": "case_qpi_round04_translation_payload_no_call_001",
|
||||
"model_id": "qpi",
|
||||
"case_type": "no_call",
|
||||
"input": "把这句话翻译成英文：这个模型还处在草稿阶段。",
|
||||
"expected_behavior": "应 no_call：翻译任务不应因为待翻译内容里的“模型”触发 QPI。",
|
||||
"failure_signal": "selector 选中 QPI，尤其是把 payload 中的“模型”当作复杂度信号。",
|
||||
"expected_output_elements": [
|
||||
"no_call=true",
|
||||
"selected_models=[]"
|
||||
],
|
||||
"should_call_model": false,
|
||||
"expected_primary_model": "none",
|
||||
"negative_expected_models": [
|
||||
"qpi",
|
||||
"intellectual_archaeology"
|
||||
],
|
||||
"expected_classification": "no_call",
|
||||
"expected_dominant_scarcity": "not_applicable",
|
||||
"expected_max_depth": "not_applicable",
|
||||
"minimum_required_elements": [
|
||||
"translation_no_call"
|
||||
],
|
||||
"forbidden_elements": [
|
||||
"payload_complexity_trigger"
|
||||
],
|
||||
"evaluation_mode": "structured",
|
||||
"notes": "Round 04.1 target from R04-BI-002: translation payload must not trigger QPI."
|
||||
},
|
||||
{
|
||||
"case_id": "case_qpi_round04_depth_limited_specific_scarcity_001",
|
||||
"model_id": "qpi",
|
||||
"case_type": "selector_gate",
|
||||
"input": "不要深入分析，只判断这件事的主导稀缺是什么。",
|
||||
"expected_behavior": "应选择 QPI 并拒绝 IA；“不要深入分析”限制深度，主导稀缺判断仍属于 QPI 轻量判断。",
|
||||
"failure_signal": "selector 返回 no_call=true 或选中 IA。",
|
||||
"expected_output_elements": [
|
||||
"selected=qpi",
|
||||
"rejected=intellectual_archaeology",
|
||||
"no_call=false"
|
||||
],
|
||||
"should_call_model": true,
|
||||
"expected_primary_model": "qpi",
|
||||
"negative_expected_models": [
|
||||
"intellectual_archaeology"
|
||||
],
|
||||
"expected_classification": "mixed",
|
||||
"expected_dominant_scarcity": "unknown",
|
||||
"expected_max_depth": "not_applicable",
|
||||
"minimum_required_elements": [
|
||||
"depth_limited_qpi_override"
|
||||
],
|
||||
"forbidden_elements": [
|
||||
"global_no_call"
|
||||
],
|
||||
"evaluation_mode": "structured",
|
||||
"notes": "Round 04.1 target from R04-BI-035."
|
||||
},
|
||||
{
|
||||
"case_id": "case_qpi_round04_depth_limited_scarcity_set_001",
|
||||
"model_id": "qpi",
|
||||
"case_type": "selector_gate",
|
||||
"input": "不要展开到底层理论，只判断现在是缺信息、缺路径还是缺共识。",
|
||||
"expected_behavior": "应选择 QPI 并拒绝 IA；“不要展开”限制深度，不应阻断缺信息 / 缺路径 / 缺共识判断。",
|
||||
"failure_signal": "selector 返回 no_call=true 或选中 IA。",
|
||||
"expected_output_elements": [
|
||||
"selected=qpi",
|
||||
"rejected=intellectual_archaeology",
|
||||
"no_call=false"
|
||||
],
|
||||
"should_call_model": true,
|
||||
"expected_primary_model": "qpi",
|
||||
"negative_expected_models": [
|
||||
"intellectual_archaeology"
|
||||
],
|
||||
"expected_classification": "mixed",
|
||||
"expected_dominant_scarcity": "unknown",
|
||||
"expected_max_depth": "not_applicable",
|
||||
"minimum_required_elements": [
|
||||
"depth_limited_qpi_override"
|
||||
],
|
||||
"forbidden_elements": [
|
||||
"global_no_call"
|
||||
],
|
||||
"evaluation_mode": "structured",
|
||||
"notes": "Round 04.1 target from R04-BI-036."
|
||||
},
|
||||
{
|
||||
"case_id": "case_qpi_round04_depth_limited_assumption_scarcity_001",
|
||||
"model_id": "qpi",
|
||||
"case_type": "selector_gate",
|
||||
"input": "不要展开到底层假设，只帮我看主导稀缺是什么。",
|
||||
"expected_behavior": "应选择 QPI 并拒绝 IA；“不要展开”和“底层假设”限制深度，主导稀缺判断仍属于 QPI 轻量判断。",
|
||||
"failure_signal": "selector 返回 no_call=true 或选中 IA。",
|
||||
"expected_output_elements": [
|
||||
"selected=qpi",
|
||||
"rejected=intellectual_archaeology",
|
||||
"no_call=false"
|
||||
],
|
||||
"should_call_model": true,
|
||||
"expected_primary_model": "qpi",
|
||||
"negative_expected_models": [
|
||||
"intellectual_archaeology"
|
||||
],
|
||||
"expected_classification": "mixed",
|
||||
"expected_dominant_scarcity": "unknown",
|
||||
"expected_max_depth": "not_applicable",
|
||||
"minimum_required_elements": [
|
||||
"depth_limited_qpi_override",
|
||||
"ia_heavy_signal_rejected"
|
||||
],
|
||||
"forbidden_elements": [
|
||||
"global_no_call",
|
||||
"selected IA"
|
||||
],
|
||||
"evaluation_mode": "structured",
|
||||
"notes": "Round 04.1 closeout from R04-BI-022: accepted collateral change; combines `不要展开`, `底层假设`, and `主导稀缺`."
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2244,6 +2244,152 @@
|
|||
],
|
||||
"evaluation_mode": "structured",
|
||||
"notes": "Round 03.2 hard exclusion trap: only-QPI instruction rejects IA."
|
||||
},
|
||||
{
|
||||
"case_id": "case_qpi_round04_translation_payload_no_call_001",
|
||||
"model_id": "qpi",
|
||||
"case_type": "no_call",
|
||||
"input": "把这句话翻译成英文：这个模型还处在草稿阶段。",
|
||||
"expected_behavior": "应 no_call：翻译任务不应因为待翻译内容里的“模型”触发 QPI。",
|
||||
"failure_signal": "selector 选中 QPI，尤其是把 payload 中的“模型”当作复杂度信号。",
|
||||
"expected_output_elements": [
|
||||
"no_call=true",
|
||||
"selected_models=[]"
|
||||
],
|
||||
"should_call_model": false,
|
||||
"expected_primary_model": "none",
|
||||
"negative_expected_models": [
|
||||
"qpi",
|
||||
"intellectual_archaeology"
|
||||
],
|
||||
"expected_classification": "no_call",
|
||||
"expected_dominant_scarcity": "not_applicable",
|
||||
"expected_max_depth": "not_applicable",
|
||||
"minimum_required_elements": [
|
||||
"translation_no_call"
|
||||
],
|
||||
"forbidden_elements": [
|
||||
"payload_complexity_trigger"
|
||||
],
|
||||
"evaluation_mode": "structured",
|
||||
"notes": "Round 04.1 target from R04-BI-002: translation payload must not trigger QPI."
|
||||
},
|
||||
{
|
||||
"case_id": "case_qpi_round04_depth_limited_specific_scarcity_001",
|
||||
"model_id": "qpi",
|
||||
"case_type": "selector_gate",
|
||||
"input": "不要深入分析，只判断这件事的主导稀缺是什么。",
|
||||
"expected_behavior": "应选择 QPI 并拒绝 IA；“不要深入分析”限制深度，主导稀缺判断仍属于 QPI 轻量判断。",
|
||||
"failure_signal": "selector 返回 no_call=true 或选中 IA。",
|
||||
"expected_output_elements": [
|
||||
"selected=qpi",
|
||||
"rejected=intellectual_archaeology",
|
||||
"no_call=false"
|
||||
],
|
||||
"should_call_model": true,
|
||||
"expected_primary_model": "qpi",
|
||||
"negative_expected_models": [
|
||||
"intellectual_archaeology"
|
||||
],
|
||||
"expected_classification": "mixed",
|
||||
"expected_dominant_scarcity": "unknown",
|
||||
"expected_max_depth": "not_applicable",
|
||||
"minimum_required_elements": [
|
||||
"depth_limited_qpi_override"
|
||||
],
|
||||
"forbidden_elements": [
|
||||
"global_no_call"
|
||||
],
|
||||
"evaluation_mode": "structured",
|
||||
"notes": "Round 04.1 target from R04-BI-035."
|
||||
},
|
||||
{
|
||||
"case_id": "case_qpi_round04_depth_limited_scarcity_set_001",
|
||||
"model_id": "qpi",
|
||||
"case_type": "selector_gate",
|
||||
"input": "不要展开到底层理论，只判断现在是缺信息、缺路径还是缺共识。",
|
||||
"expected_behavior": "应选择 QPI 并拒绝 IA；“不要展开”限制深度，不应阻断缺信息 / 缺路径 / 缺共识判断。",
|
||||
"failure_signal": "selector 返回 no_call=true 或选中 IA。",
|
||||
"expected_output_elements": [
|
||||
"selected=qpi",
|
||||
"rejected=intellectual_archaeology",
|
||||
"no_call=false"
|
||||
],
|
||||
"should_call_model": true,
|
||||
"expected_primary_model": "qpi",
|
||||
"negative_expected_models": [
|
||||
"intellectual_archaeology"
|
||||
],
|
||||
"expected_classification": "mixed",
|
||||
"expected_dominant_scarcity": "unknown",
|
||||
"expected_max_depth": "not_applicable",
|
||||
"minimum_required_elements": [
|
||||
"depth_limited_qpi_override"
|
||||
],
|
||||
"forbidden_elements": [
|
||||
"global_no_call"
|
||||
],
|
||||
"evaluation_mode": "structured",
|
||||
"notes": "Round 04.1 target from R04-BI-036."
|
||||
},
|
||||
{
|
||||
"case_id": "case_qpi_round04_depth_limited_assumption_scarcity_001",
|
||||
"model_id": "qpi",
|
||||
"case_type": "selector_gate",
|
||||
"input": "不要展开到底层假设，只帮我看主导稀缺是什么。",
|
||||
"expected_behavior": "应选择 QPI 并拒绝 IA；“不要展开”和“底层假设”限制深度，主导稀缺判断仍属于 QPI 轻量判断。",
|
||||
"failure_signal": "selector 返回 no_call=true 或选中 IA。",
|
||||
"expected_output_elements": [
|
||||
"selected=qpi",
|
||||
"rejected=intellectual_archaeology",
|
||||
"no_call=false"
|
||||
],
|
||||
"should_call_model": true,
|
||||
"expected_primary_model": "qpi",
|
||||
"negative_expected_models": [
|
||||
"intellectual_archaeology"
|
||||
],
|
||||
"expected_classification": "mixed",
|
||||
"expected_dominant_scarcity": "unknown",
|
||||
"expected_max_depth": "not_applicable",
|
||||
"minimum_required_elements": [
|
||||
"depth_limited_qpi_override",
|
||||
"ia_heavy_signal_rejected"
|
||||
],
|
||||
"forbidden_elements": [
|
||||
"global_no_call",
|
||||
"selected IA"
|
||||
],
|
||||
"evaluation_mode": "structured",
|
||||
"notes": "Round 04.1 closeout from R04-BI-022: accepted collateral change; combines `不要展开`, `底层假设`, and `主导稀缺`."
|
||||
},
|
||||
{
|
||||
"case_id": "case_ia_round04_do_not_enter_archaeology_001",
|
||||
"model_id": "intellectual_archaeology",
|
||||
"case_type": "selector_gate",
|
||||
"input": "不要进入思想考古，我只要一个轻量判断：现在应该先问问题、找路径，还是处理责任边界？",
|
||||
"expected_behavior": "selector 应选择或允许 QPI，并硬拒绝思想考古。",
|
||||
"failure_signal": "因为“思想考古”和复杂度信号加分而仍选中 IA。",
|
||||
"expected_output_elements": [
|
||||
"selected=qpi",
|
||||
"rejected=intellectual_archaeology"
|
||||
],
|
||||
"should_call_model": false,
|
||||
"expected_primary_model": "qpi",
|
||||
"negative_expected_models": [
|
||||
"intellectual_archaeology"
|
||||
],
|
||||
"expected_classification": "not_applicable",
|
||||
"expected_dominant_scarcity": "not_applicable",
|
||||
"expected_max_depth": "no_call",
|
||||
"minimum_required_elements": [
|
||||
"model_level_hard_exclusion"
|
||||
],
|
||||
"forbidden_elements": [
|
||||
"selected IA"
|
||||
],
|
||||
"evaluation_mode": "structured",
|
||||
"notes": "Round 04.1 target from R04-BI-024: explicit 'do not enter IA' must hard-exclude IA."
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,33 +1,33 @@
|
|||
{
|
||||
"index_version": "0.2",
|
||||
"last_updated": "2026-06-17",
|
||||
"last_updated": "2026-06-18",
|
||||
"source_of_truth": "per_model_files",
|
||||
"aggregate_file": "tests/regression_cases.json",
|
||||
"models": [
|
||||
{
|
||||
"model_id": "qpi",
|
||||
"regression_file": "tests/qpi.regression.json",
|
||||
"case_count": 58,
|
||||
"case_count": 62,
|
||||
"case_type_counts": {
|
||||
"boundary": 13,
|
||||
"misuse": 16,
|
||||
"no_call": 6,
|
||||
"no_call": 7,
|
||||
"pipeline": 1,
|
||||
"positive": 16,
|
||||
"selector_gate": 6
|
||||
"selector_gate": 9
|
||||
}
|
||||
},
|
||||
{
|
||||
"model_id": "intellectual_archaeology",
|
||||
"regression_file": "tests/intellectual_archaeology.regression.json",
|
||||
"case_count": 19,
|
||||
"case_count": 20,
|
||||
"case_type_counts": {
|
||||
"boundary": 3,
|
||||
"misuse": 3,
|
||||
"no_call": 3,
|
||||
"pipeline": 1,
|
||||
"positive": 5,
|
||||
"selector_gate": 4
|
||||
"selector_gate": 5
|
||||
}
|
||||
}
|
||||
]
|
||||
|
|
|
|||
|
|
@ -0,0 +1,65 @@
|
|||
import importlib.util
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def load_round04_runner():
|
||||
script_path = Path(__file__).resolve().parents[1] / "scripts" / "run_round04_blind_routing.py"
|
||||
spec = importlib.util.spec_from_file_location("run_round04_blind_routing", script_path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
class Round04BlindRoutingTests(unittest.TestCase):
|
||||
def test_loads_frozen_blind_inputs_without_expected_routing(self):
|
||||
runner = load_round04_runner()
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
|
||||
input_set = runner.load_blind_inputs(root)
|
||||
|
||||
self.assertEqual(input_set["blind_input_set_id"], "round04_blind_inputs_2026-06-18_v0.1")
|
||||
self.assertEqual(len(input_set["inputs"]), 38)
|
||||
for item in input_set["inputs"]:
|
||||
self.assertIn("input_id", item)
|
||||
self.assertIn("category_for_owner_review", item)
|
||||
self.assertIn("input_text", item)
|
||||
self.assertNotIn("expected_selector_behavior", item)
|
||||
self.assertNotIn("expected_primary_model", item)
|
||||
|
||||
def test_evaluates_blind_input_without_answer_key_fields(self):
|
||||
runner = load_round04_runner()
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
input_set = runner.load_blind_inputs(root)
|
||||
|
||||
result = runner.evaluate_blind_input(root, input_set["inputs"][0])
|
||||
|
||||
self.assertEqual(result["input_id"], "R04-BI-001")
|
||||
self.assertEqual(result["category_for_owner_review"], "explicit_no_call_fact_lookup")
|
||||
self.assertIn("selected_models", result)
|
||||
self.assertIn("rejected_models", result)
|
||||
self.assertIn("no_call", result)
|
||||
self.assertIn("routing_notes", result)
|
||||
self.assertNotIn("expected_selector_behavior", result)
|
||||
self.assertNotIn("expected_primary_model", result)
|
||||
|
||||
def test_report_states_results_are_not_scored_against_expected_routing(self):
|
||||
runner = load_round04_runner()
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
input_set = runner.load_blind_inputs(root)
|
||||
results = [
|
||||
runner.evaluate_blind_input(root, item)
|
||||
for item in input_set["inputs"][:2]
|
||||
]
|
||||
|
||||
report_text = runner.render_report(input_set, results)
|
||||
|
||||
self.assertIn("# Round 04 Blind Routing Evaluation Report", report_text)
|
||||
self.assertIn("Expected routing labels: `not included before first blind run`", report_text)
|
||||
self.assertIn("Inputs evaluated: 2", report_text)
|
||||
self.assertIn("R04-BI-001", report_text)
|
||||
self.assertIn("R04-BI-002", report_text)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
import importlib.util
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def load_post_patch_runner():
|
||||
script_path = Path(__file__).resolve().parents[1] / "scripts" / "run_round04_post_patch_verification.py"
|
||||
spec = importlib.util.spec_from_file_location("run_round04_post_patch_verification", script_path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
class Round041PostPatchVerificationTests(unittest.TestCase):
|
||||
def test_targeted_labels_are_limited_to_review_approved_cases(self):
|
||||
runner = load_post_patch_runner()
|
||||
|
||||
self.assertEqual(
|
||||
sorted(runner.TARGET_EXPECTATIONS.keys()),
|
||||
["R04-BI-002", "R04-BI-024", "R04-BI-035", "R04-BI-036"],
|
||||
)
|
||||
|
||||
def test_report_marks_target_cases_pass_after_patch(self):
|
||||
runner = load_post_patch_runner()
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
|
||||
report = runner.build_post_patch_report(root)
|
||||
|
||||
self.assertIn("# Round 04.1 Post-Patch Routing Verification Report", report)
|
||||
self.assertIn("This is not a second blind test.", report)
|
||||
self.assertIn("Targeted cases checked: 4", report)
|
||||
self.assertIn("Targeted failures: 0", report)
|
||||
self.assertIn("Behavior changes recorded: 5", report)
|
||||
self.assertIn("## Non-target Behavior Diff / Accepted Collateral Change", report)
|
||||
self.assertIn("### R04-BI-022", report)
|
||||
self.assertIn("Not a new failure.", report)
|
||||
self.assertIn("lightweight QPI is acceptable and IA should remain rejected", report)
|
||||
for case_id in ["R04-BI-002", "R04-BI-024", "R04-BI-035", "R04-BI-036"]:
|
||||
self.assertIn(f"### {case_id}", report)
|
||||
self.assertIn("status: `PASS`", report)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Loading…
Reference in New Issue