316 lines
7.2 KiB
Markdown
316 lines
7.2 KiB
Markdown
# Review Agent Regression Test Handoff
|
|
|
|
## 1. Objective
|
|
|
|
Run a controlled 3 x 2 regression test for two review agents:
|
|
|
|
```text
|
|
巨人认知 / Giant Cognition
|
|
认知显影 / Cognitive Imaging
|
|
```
|
|
|
|
The goal is to determine whether CCPE System Lite migration improves, preserves, or degrades the original CCPE 2.0 review behavior.
|
|
|
|
This work should be continued in a new Codex session to avoid contamination from the current debugging conversation.
|
|
|
|
## 2. Core Hypothesis
|
|
|
|
Current working hypothesis:
|
|
|
|
```text
|
|
For mature review agents, full template-style Lite rewrites may lose hidden review kernels.
|
|
The safer migration route may be:
|
|
|
|
original CCPE 2.0 kernel
|
|
+ minimal Lite metadata / portability wrapper
|
|
+ explicit safety and reasoning-disclosure repairs
|
|
+ regression-tested output protocol
|
|
```
|
|
|
|
## 3. Why This Regression Test Exists
|
|
|
|
The 巨人认知 Lite migration exposed a possible root cause:
|
|
|
|
```text
|
|
Original 巨人认知 is not just one model.
|
|
It contains at least:
|
|
|
|
1. 认知架构师 role
|
|
2. 进化型生物计算架构
|
|
3. 思想考古学家 method
|
|
4. GL0-GL4 output protocol
|
|
5. Gemini-shaped style / generation habits
|
|
6. Article-context-derived worldview
|
|
```
|
|
|
|
The current Lite version preserved the architecture but under-specified 思想考古学家. That may explain why GL3 sometimes behaves like metaphor optimization instead of deep assumption archaeology.
|
|
|
|
## 4. Test Matrix
|
|
|
|
For each agent family, test:
|
|
|
|
```text
|
|
Prompt variants:
|
|
A. original-ccpe-2
|
|
B. ccpe-system-lite
|
|
C. original-kernel-minimal-lite
|
|
|
|
Model environments:
|
|
1. ChatGPT / Codex-side model
|
|
2. Gemini 3.1 Pro
|
|
```
|
|
|
|
This gives:
|
|
|
|
```text
|
|
3 prompt variants x 2 model environments = 6 conditions per agent family
|
|
2 agent families x 6 conditions = 12 condition groups
|
|
```
|
|
|
|
Run each condition against the same article set.
|
|
|
|
## 5. Article Corpus
|
|
|
|
Place full article texts in:
|
|
|
|
```text
|
|
workbench/analysis/review-agent-regression-2026-06-02/articles/
|
|
```
|
|
|
|
Recommended minimum:
|
|
|
|
```text
|
|
article-01-strong-metaphor.md
|
|
article-02-business-analysis.md
|
|
article-03-logical-argument.md
|
|
article-04-value-philosophy.md
|
|
```
|
|
|
|
Use full正文 rather than outline when possible. This reduces dependence on hidden context packages and makes the test more portable.
|
|
|
|
## 6. Prompt Inventory
|
|
|
|
Prompt variants should live under:
|
|
|
|
```text
|
|
workbench/analysis/review-agent-regression-2026-06-02/prompts/
|
|
```
|
|
|
|
Recommended layout:
|
|
|
|
```text
|
|
prompts/
|
|
giant-cognition/
|
|
original-ccpe-2.prompt.md
|
|
ccpe-system-lite.prompt.md
|
|
original-kernel-minimal-lite.prompt.md
|
|
|
|
cognitive-imaging/
|
|
original-ccpe-2.prompt.md
|
|
ccpe-system-lite.prompt.md
|
|
original-kernel-minimal-lite.prompt.md
|
|
```
|
|
|
|
Current known sources:
|
|
|
|
```text
|
|
Giant Cognition Lite:
|
|
agents/lite/giant-cognition.prompt.md
|
|
|
|
Cognitive Imaging original:
|
|
workbench/raw/认知显影者1.1.md
|
|
|
|
Cognitive Imaging Lite:
|
|
agents/lite/cognitive-imaging-practitioner.prompt.md
|
|
```
|
|
|
|
Known missing prompt artifacts:
|
|
|
|
```text
|
|
Giant Cognition original CCPE 2.0 prompt
|
|
Giant Cognition original-kernel-minimal-lite prompt
|
|
Cognitive Imaging original-kernel-minimal-lite prompt
|
|
```
|
|
|
|
The new session should reconstruct or import these before running tests.
|
|
|
|
## 7. Original-Kernel Minimal Lite Rule
|
|
|
|
When creating `original-kernel-minimal-lite.prompt.md`, do not rewrite the mature prompt into a new template.
|
|
|
|
Preserve:
|
|
|
|
```text
|
|
Original role language
|
|
Original conceptual metaphors
|
|
Original output protocol
|
|
Original model-specific layer names
|
|
Original critique style
|
|
Original method kernels
|
|
```
|
|
|
|
Only minimally repair:
|
|
|
|
```text
|
|
Front matter
|
|
Target platform / usage scenario
|
|
Hidden chain-of-thought disclosure rules
|
|
Fact / retrieval boundaries
|
|
Layer naming discipline
|
|
Explicit output validation checklist
|
|
Version notes
|
|
```
|
|
|
|
For 巨人认知, explicitly preserve:
|
|
|
|
```text
|
|
进化型生物计算架构
|
|
思想考古学家
|
|
GL0-GL4
|
|
意图 / 反思双循环
|
|
批判性且建设性的语气
|
|
```
|
|
|
|
For 认知显影, preserve its original review kernel before applying CCPE System wrapping.
|
|
|
|
## 8. Test Execution Protocol
|
|
|
|
Use fresh thread isolation.
|
|
|
|
For each result:
|
|
|
|
```text
|
|
1. Start a fresh thread/subthread.
|
|
2. Load exactly one prompt variant.
|
|
3. Provide exactly one article.
|
|
4. Ask for the standard review output.
|
|
5. Save the raw output without editing.
|
|
6. Record model, prompt variant, article id, date, and operator.
|
|
```
|
|
|
|
Do not mix multiple prompt variants in one test thread.
|
|
|
|
Do not show the model other variants' outputs during generation.
|
|
|
|
Do not tune prompts mid-run. If a prompt is changed, increment the prompt version and restart the condition.
|
|
|
|
## 9. Result File Naming
|
|
|
|
Save results under:
|
|
|
|
```text
|
|
results/chatgpt/
|
|
results/gemini/
|
|
```
|
|
|
|
Use this filename pattern:
|
|
|
|
```text
|
|
{agent-family}__{prompt-variant}__{article-id}__{model-env}.result.md
|
|
```
|
|
|
|
Examples:
|
|
|
|
```text
|
|
giant-cognition__original-ccpe-2__article-01-strong-metaphor__chatgpt.result.md
|
|
giant-cognition__ccpe-system-lite__article-01-strong-metaphor__gemini.result.md
|
|
cognitive-imaging__original-kernel-minimal-lite__article-03-logical-argument__chatgpt.result.md
|
|
```
|
|
|
|
## 10. Evaluation Rubric
|
|
|
|
Use:
|
|
|
|
```text
|
|
rubrics/review-agent-regression-rubric.md
|
|
```
|
|
|
|
Do not evaluate only "which output feels better." Score the specific regression dimensions:
|
|
|
|
```text
|
|
1. Model fidelity
|
|
2. Method fidelity
|
|
3. GL3 / deep-structure performance
|
|
4. Hidden assumption detection
|
|
5. Philosophical bedrock excavation
|
|
6. Context fit
|
|
7. Concept overfitting risk
|
|
8. Output actionability
|
|
9. Naming / protocol discipline
|
|
10. Platform stability
|
|
```
|
|
|
|
## 11. ChatGPT / Codex-Side Execution
|
|
|
|
The new Codex session may use thread tools if available.
|
|
|
|
Before using thread automation, search for the relevant thread tool:
|
|
|
|
```text
|
|
create_thread
|
|
send_message_to_thread
|
|
read_thread
|
|
list_threads
|
|
```
|
|
|
|
If thread tools are unavailable, run tests manually in separate fresh Codex conversations and save outputs into `results/chatgpt/`.
|
|
|
|
## 12. Gemini-Side Execution
|
|
|
|
Gemini outputs will be produced manually by the user.
|
|
|
|
Save them into:
|
|
|
|
```text
|
|
results/gemini/
|
|
```
|
|
|
|
Use the same file naming convention as ChatGPT/Codex-side results.
|
|
|
|
## 13. Final Report
|
|
|
|
After all results are collected, produce:
|
|
|
|
```text
|
|
reports/review-agent-regression-report.md
|
|
```
|
|
|
|
The report should answer:
|
|
|
|
```text
|
|
1. Which prompt variant performs best per agent family?
|
|
2. Which prompt variant transfers best to ChatGPT/Codex?
|
|
3. Which prompt variant performs best on Gemini?
|
|
4. Does original-kernel-minimal-lite outperform full Lite rewrite?
|
|
5. Is CCPE System migration preserving the old CCPE 2.0 review kernels?
|
|
6. Should mature review agents use separate production prompts per model environment?
|
|
7. What should be changed in CCPE Forge migration rules?
|
|
```
|
|
|
|
## 14. Stop Conditions
|
|
|
|
Pause and ask the user before:
|
|
|
|
```text
|
|
- Promoting a prompt variant to active.
|
|
- Replacing any canonical Lite prompt.
|
|
- Updating Model Index status from candidate to active.
|
|
- Creating Agent Specs, Skills, or Runtimes from these prompts.
|
|
- Deleting or archiving old CCPE 2.0 prompts.
|
|
```
|
|
|
|
## 15. Recommended Next Step
|
|
|
|
In the new session:
|
|
|
|
```text
|
|
1. Read this handoff.
|
|
2. Confirm article files exist.
|
|
3. Copy or reconstruct prompt variants into prompts/.
|
|
4. Create original-kernel-minimal-lite variants for both agent families.
|
|
5. Run ChatGPT/Codex-side tests in fresh threads.
|
|
6. Wait for Gemini-side outputs from the user.
|
|
7. Compare all outputs and write final report.
|
|
```
|
|
|