173 lines
6.4 KiB
Markdown
173 lines
6.4 KiB
Markdown
# Round 04 Blind Routing Evaluation Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Prepare and run Round 04 blind input routing evaluation without changing selector rules or model lifecycle status.
|
|
|
|
**Architecture:** Round 04 uses a reviewed blind input pool as source material, freezes it into a machine-readable JSON input set, runs the existing rule-based selector through a thin batch runner, and packages the outputs for GPT / CCRA review. The runner reuses `scripts/run_selector_demo.py::recommend` and does not add answer generation, LLM selector logic, or expected routing labels.
|
|
|
|
**Tech Stack:** Markdown, JSON, Python standard library, existing selector scripts.
|
|
|
|
## Global Constraints
|
|
|
|
- Do not modify `selector/selector_rules.json` before the blind evaluation.
|
|
- Do not merge blind inputs into `selector/selector_calibration_inputs.json`.
|
|
- Do not add expected behavior before the first blind run.
|
|
- Do not add a third model.
|
|
- Do not upgrade QPI or Intellectual Archaeology.
|
|
- Do not introduce an LLM selector.
|
|
- Do not implement RAG, vector database, frontend, backend, user account, or QA product.
|
|
- Run first, analyze failures later.
|
|
|
|
---
|
|
|
|
### Task 1: Revise Human-Readable Blind Input Source
|
|
|
|
**Files:**
|
|
- Modify: `reports/Round04_blind_input_candidates_2026-06-18.md`
|
|
|
|
**Interfaces:**
|
|
- Consumes: owner review notes for R04-BI-018, R04-BI-021, R04-BI-025, R04-BI-026, and new R04-BI-031 through R04-BI-038.
|
|
- Produces: 38 candidate inputs with review-only metadata and no expected routing labels.
|
|
|
|
- [ ] **Step 1: Mark control-case notes**
|
|
|
|
Add metadata notes:
|
|
- R04-BI-021 and R04-BI-025: `model-name-exposed control case`
|
|
- R04-BI-018: `positive control`
|
|
- R04-BI-026: natural IA / QPI-to-IA boundary sample
|
|
|
|
- [ ] **Step 2: Add R04-BI-031 through R04-BI-038**
|
|
|
|
Add the eight owner-proposed cases exactly, preserving their categories, input text, and inclusion rationale.
|
|
|
|
- [ ] **Step 3: Update coverage**
|
|
|
|
Set total to 38 candidate inputs and include a short note that 2 inputs expose model names by design.
|
|
|
|
### Task 2: Freeze Machine-Readable Blind Input JSON
|
|
|
|
**Files:**
|
|
- Create: `selector/round04_blind_inputs.json`
|
|
|
|
**Interfaces:**
|
|
- Consumes: reviewed Markdown source from Task 1.
|
|
- Produces: JSON with `blind_input_set_id`, `status`, `source_document`, `constraints`, and `inputs`.
|
|
|
|
- [ ] **Step 1: Create JSON without expected routing**
|
|
|
|
Each input object contains:
|
|
- `input_id`
|
|
- `category_for_owner_review`
|
|
- `input_text`
|
|
- `why_included`
|
|
- optional `control_case_type`
|
|
|
|
- [ ] **Step 2: Preserve blindness constraints**
|
|
|
|
Include top-level notes that no expected behavior is present and that selector rules must not be modified before first run.
|
|
|
|
### Task 3: Add Failing Tests for the Round 04 Runner
|
|
|
|
**Files:**
|
|
- Create: `tests/test_round04_blind_routing.py`
|
|
- Create later in Task 4: `scripts/run_round04_blind_routing.py`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `selector/round04_blind_inputs.json`
|
|
- Produces: test expectations for load, evaluation shape, and report content.
|
|
|
|
- [ ] **Step 1: Write tests before implementation**
|
|
|
|
Test that a blind input item can be loaded, evaluated through `recommend`, and rendered without expected routing labels.
|
|
|
|
- [ ] **Step 2: Run tests and confirm failure**
|
|
|
|
Run: `python -m unittest tests.test_round04_blind_routing -v`
|
|
|
|
Expected: failure because `scripts.run_round04_blind_routing` does not exist.
|
|
|
|
### Task 4: Implement and Run the Round 04 Runner
|
|
|
|
**Files:**
|
|
- Create: `scripts/run_round04_blind_routing.py`
|
|
- Create: `reports/Round04_blind_routing_evaluation_report_2026-06-18.md`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `selector/round04_blind_inputs.json`, `scripts/run_selector_demo.py::recommend`
|
|
- Produces: JSON-shaped evaluation results and Markdown report.
|
|
|
|
- [ ] **Step 1: Implement standard-library runner**
|
|
|
|
Provide functions:
|
|
- `load_blind_inputs(root)`
|
|
- `evaluate_blind_input(root, item)`
|
|
- `write_report(root, input_set, results)`
|
|
- `main()`
|
|
|
|
- [ ] **Step 2: Run focused tests**
|
|
|
|
Run: `python -m unittest tests.test_round04_blind_routing -v`
|
|
|
|
- [ ] **Step 3: Run blind evaluation**
|
|
|
|
Run: `python scripts\run_round04_blind_routing.py`
|
|
|
|
Expected: report written; command exits 0.
|
|
|
|
### Task 5: Build Round 04 Review Bundle
|
|
|
|
**Files:**
|
|
- Create: `ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/00_OPEN_THIS_FIRST_CCRA_REVIEW_BRIEF_04.md`
|
|
- Create: `ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/01_ROUND03_CLOSURE_AND_ROUND04_SCOPE_04.md`
|
|
- Create: `ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/02_BLIND_INPUT_SET_04.md`
|
|
- Create: `ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/03_ROUTING_EVALUATION_REPORT_04.md`
|
|
- Create: `ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/04_REVIEW_QUESTIONS_FOR_GPT_04.md`
|
|
- Create: `ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/BUNDLE_FILE_MANIFEST_04.md`
|
|
- Create: `ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/optional_raw_changed_files_04.zip`
|
|
|
|
**Interfaces:**
|
|
- Consumes: Round 03 closure summary, frozen blind input JSON, runner output report, current runtime assets.
|
|
- Produces: GPT / CCRA review package.
|
|
|
|
- [ ] **Step 1: Write brief and scope files**
|
|
|
|
Include Round 03 closure summary, Round 04 target, and non-goals.
|
|
|
|
- [ ] **Step 2: Write input and evaluation mirrors**
|
|
|
|
Mirror the frozen blind input set and routing evaluation report for upload.
|
|
|
|
- [ ] **Step 3: Create raw zip**
|
|
|
|
Include current runtime assets, blind input set, runner, tests, and reports. Preserve source-relative paths.
|
|
|
|
### Task 6: Verify
|
|
|
|
**Files:**
|
|
- Read: all changed files
|
|
|
|
**Interfaces:**
|
|
- Consumes: full working tree after Tasks 1-5.
|
|
- Produces: verification evidence.
|
|
|
|
- [ ] **Step 1: Run focused Round 04 test**
|
|
|
|
Run: `python -m unittest tests.test_round04_blind_routing -v`
|
|
|
|
- [ ] **Step 2: Run existing validation gates**
|
|
|
|
Run:
|
|
- `python scripts\rebuild_indexes.py --check`
|
|
- `python -m unittest discover -s tests -p "test*.py" -v`
|
|
- `python scripts\validate_model_library.py`
|
|
- `python scripts\check_card_contract.py`
|
|
- `python scripts\run_selector_demo.py`
|
|
- `python scripts\run_selector_regression.py`
|
|
- `python scripts\run_selector_calibration_smoke.py`
|
|
- `python scripts\check_model_card_sync.py`
|
|
|
|
- [ ] **Step 3: Inspect git diff**
|
|
|
|
Confirm no selector rule changes and no model lifecycle status changes.
|