the-mindscape-of-bro-tsong/docs/superpowers/plans/2026-06-18-round04-blind-routing-evaluation.md

Round 04 Blind Routing Evaluation Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Prepare and run Round 04 blind input routing evaluation without changing selector rules or model lifecycle status.

Architecture: Round 04 uses a reviewed blind input pool as source material, freezes it into a machine-readable JSON input set, runs the existing rule-based selector through a thin batch runner, and packages the outputs for GPT / CCRA review. The runner reuses scripts/run_selector_demo.py::recommend and does not add answer generation, LLM selector logic, or expected routing labels.

Tech Stack: Markdown, JSON, Python standard library, existing selector scripts.

Global Constraints

• Do not modify selector/selector_rules.json before the blind evaluation.
• Do not merge blind inputs into selector/selector_calibration_inputs.json.
• Do not add expected behavior before the first blind run.
• Do not add a third model.
• Do not upgrade QPI or Intellectual Archaeology.
• Do not introduce an LLM selector.
• Do not implement RAG, vector database, frontend, backend, user account, or QA product.
• Run first, analyze failures later.

---

Task 1: Revise Human-Readable Blind Input Source

Files:

• Modify: reports/Round04_blind_input_candidates_2026-06-18.md

Interfaces:

• Consumes: owner review notes for R04-BI-018, R04-BI-021, R04-BI-025, R04-BI-026, and new R04-BI-031 through R04-BI-038.

• Produces: 38 candidate inputs with review-only metadata and no expected routing labels.

• Step 1: Mark control-case notes

Add metadata notes:

• R04-BI-021 and R04-BI-025: model-name-exposed control case

• R04-BI-018: positive control

• R04-BI-026: natural IA / QPI-to-IA boundary sample

• Step 2: Add R04-BI-031 through R04-BI-038

Add the eight owner-proposed cases exactly, preserving their categories, input text, and inclusion rationale.

• Step 3: Update coverage

Set total to 38 candidate inputs and include a short note that 2 inputs expose model names by design.

Task 2: Freeze Machine-Readable Blind Input JSON

Files:

• Create: selector/round04_blind_inputs.json

Interfaces:

• Consumes: reviewed Markdown source from Task 1.

• Produces: JSON with blind_input_set_id, status, source_document, constraints, and inputs.

• Step 1: Create JSON without expected routing

Each input object contains:

• input_id

• category_for_owner_review

• input_text

• why_included

• optional control_case_type

• Step 2: Preserve blindness constraints

Include top-level notes that no expected behavior is present and that selector rules must not be modified before first run.

Task 3: Add Failing Tests for the Round 04 Runner

Files:

• Create: tests/test_round04_blind_routing.py
• Create later in Task 4: scripts/run_round04_blind_routing.py

Interfaces:

• Consumes: selector/round04_blind_inputs.json

• Produces: test expectations for load, evaluation shape, and report content.

• Step 1: Write tests before implementation

Test that a blind input item can be loaded, evaluated through recommend, and rendered without expected routing labels.

• Step 2: Run tests and confirm failure

Run: python -m unittest tests.test_round04_blind_routing -v

Expected: failure because scripts.run_round04_blind_routing does not exist.

Task 4: Implement and Run the Round 04 Runner

Files:

• Create: scripts/run_round04_blind_routing.py
• Create: reports/Round04_blind_routing_evaluation_report_2026-06-18.md

Interfaces:

• Consumes: selector/round04_blind_inputs.json, scripts/run_selector_demo.py::recommend

• Produces: JSON-shaped evaluation results and Markdown report.

• Step 1: Implement standard-library runner

Provide functions:

• load_blind_inputs(root)

• evaluate_blind_input(root, item)

• write_report(root, input_set, results)

• main()

• Step 2: Run focused tests

Run: python -m unittest tests.test_round04_blind_routing -v

• Step 3: Run blind evaluation

Run: python scripts\run_round04_blind_routing.py

Expected: report written; command exits 0.

Task 5: Build Round 04 Review Bundle

Files:

• Create: ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/00_OPEN_THIS_FIRST_CCRA_REVIEW_BRIEF_04.md
• Create: ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/01_ROUND03_CLOSURE_AND_ROUND04_SCOPE_04.md
• Create: ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/02_BLIND_INPUT_SET_04.md
• Create: ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/03_ROUTING_EVALUATION_REPORT_04.md
• Create: ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/04_REVIEW_QUESTIONS_FOR_GPT_04.md
• Create: ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/BUNDLE_FILE_MANIFEST_04.md
• Create: ccra_review_bundle/round-04_2026-06-18_blind-input-routing-evaluation/optional_raw_changed_files_04.zip

Interfaces:

• Consumes: Round 03 closure summary, frozen blind input JSON, runner output report, current runtime assets.

• Produces: GPT / CCRA review package.

• Step 1: Write brief and scope files

Include Round 03 closure summary, Round 04 target, and non-goals.

• Step 2: Write input and evaluation mirrors

Mirror the frozen blind input set and routing evaluation report for upload.

• Step 3: Create raw zip

Include current runtime assets, blind input set, runner, tests, and reports. Preserve source-relative paths.

Task 6: Verify

Files:

• Read: all changed files

Interfaces:

• Consumes: full working tree after Tasks 1-5.

• Produces: verification evidence.

• Step 1: Run focused Round 04 test

Run: python -m unittest tests.test_round04_blind_routing -v

• Step 2: Run existing validation gates

Run:

• python scripts\rebuild_indexes.py --check

• python -m unittest discover -s tests -p "test*.py" -v

• python scripts\validate_model_library.py

• python scripts\check_card_contract.py

• python scripts\run_selector_demo.py

• python scripts\run_selector_regression.py

• python scripts\run_selector_calibration_smoke.py

• python scripts\check_model_card_sync.py

• Step 3: Inspect git diff

Confirm no selector rule changes and no model lifecycle status changes.