301 lines
16 KiB
Markdown
301 lines
16 KiB
Markdown
# Local CCRA Next Review Requirements
|
|
|
|
## 1. Return Metadata
|
|
|
|
- project_id: `the-mindscape-of-bro-tsong`
|
|
- review_round: `round-04`
|
|
- local_pass: `pilot-02`
|
|
- turn_type: `planning_turn`
|
|
- runtime_id: `ccra-local-review`
|
|
- runtime_version: `0.1.1`
|
|
- agent_id: `ccra-local-reviewer`
|
|
- agent_version: `0.1.1`
|
|
- invocation_mode: `prompt_path_reference`
|
|
- carrier_type: `codex_thread`
|
|
- child_session_id: `019edcde-24be-7593-99b2-5cf014467340`
|
|
- same_child_session_as_review_turn: `yes`
|
|
- first_turn_report_path: `local_ccra_reviews/round-04/pilot-02/04_LOCAL_CCRA_REVIEW_REPORT.md`
|
|
- owner_decision_path: `local_ccra_reviews/round-04/pilot-02/owner-decision.md`
|
|
- planning_output_path: `local_ccra_reviews/round-04/pilot-02/next-review-requirements.md`
|
|
- generated_at: `2026-06-19`
|
|
|
|
## 2. Planning Scope
|
|
|
|
- next_review_goal: define the submitted materials and verification evidence required for a Round 04.1 post-patch routing verification / regression pass.
|
|
- target_round_or_pass: `round-04.1_selector-rule-patch_verification`
|
|
- planning_basis:
|
|
- `local_ccra_reviews/round-04/pilot-02/04_LOCAL_CCRA_REVIEW_REPORT.md`
|
|
- `local_ccra_reviews/round-04/pilot-02/owner-decision.md`
|
|
- `local_ccra_reviews/round-04/pilot-02/agent-invocation-packet.md`
|
|
- explicit_non_goals:
|
|
- Do not re-review the full initial Round 04 bundle in this planning turn.
|
|
- Do not use this planning output as Owner approval.
|
|
- Do not treat Round 04.1 verification as Round 05 acceptance.
|
|
- Do not promote QPI or Intellectual Archaeology lifecycle status.
|
|
- Do not modify selector rules, tests, knowledge assets, model files, or review bundles in this turn.
|
|
|
|
Owner decision status is still `pending`. This plan therefore defines required review materials and recommended defaults; it does not convert the first-turn findings into accepted work.
|
|
|
|
## 3. Required Submitted Materials
|
|
|
|
```text
|
|
- material_id: M-01-review-brief
|
|
required_path_or_bundle_slot: ccra_review_bundle/<round-04.1>/00_OPEN_THIS_FIRST_*.md
|
|
purpose: state the review goal, target failures, non-goals, and authority boundary.
|
|
required: yes
|
|
notes: Must say this is a post-patch verification pass, not a second blind test and not lifecycle promotion.
|
|
|
|
- material_id: M-02-rule-patch-summary-and-diff
|
|
required_path_or_bundle_slot: ccra_review_bundle/<round-04.1>/01_RULE_PATCH_SUMMARY_AND_DIFF_*.md
|
|
purpose: show exactly what selector rule surfaces changed and why.
|
|
required: yes
|
|
notes: Must map changes to R04-BI-002, R04-BI-024, R04-BI-035, R04-BI-036, and any accepted same-class case such as R04-BI-022.
|
|
|
|
- material_id: M-03-expected-routing-labels-or-owner-decisions
|
|
required_path_or_bundle_slot: ccra_review_bundle/<round-04.1>/02_EXPECTED_ROUTING_LABELS_*.md or owner-decision-backed equivalent
|
|
purpose: prevent Web CCRA from inferring target labels from prose.
|
|
required: yes
|
|
notes: Must separate must-fix, same-class boundary, calibration-only, and explicitly-not-patched cases.
|
|
|
|
- material_id: M-04-post-patch-routing-verification-report
|
|
required_path_or_bundle_slot: reports/Round04_1_post_patch_routing_verification_report_*.md and bundle copy
|
|
purpose: provide full frozen-pool rerun after the patch.
|
|
required: yes
|
|
notes: Must include all 38 Round 04 inputs, not only target cases.
|
|
|
|
- material_id: M-05-before-after-routing-diff
|
|
required_path_or_bundle_slot: helper-outputs/routing-behavior-diff.md or a report section with equivalent table
|
|
purpose: show target changes, non-target changes, accepted collateral behavior, and regressions.
|
|
required: yes
|
|
notes: If the helper cannot run, include a manual before/after table and explain the missing helper risk.
|
|
|
|
- material_id: M-06-regression-and-command-log
|
|
required_path_or_bundle_slot: ccra_review_bundle/<round-04.1>/04_REGRESSION_AND_COMMAND_LOG_*.md
|
|
purpose: prove executed tests and validation gates, not dry-run declarations.
|
|
required: yes
|
|
notes: Include exact commands, exit codes, timestamps or run context, and changed-file impact.
|
|
|
|
- material_id: M-07-existing-selector-regression
|
|
required_path_or_bundle_slot: reports/selector_regression_report*.md or command log section
|
|
purpose: prove existing no-call, model-exclusion, QPI-before-IA, and depth-limiting protections still pass.
|
|
required: yes
|
|
notes: Must not rely only on targeted target-case checks.
|
|
|
|
- material_id: M-08-calibration-smoke
|
|
required_path_or_bundle_slot: reports/selector_calibration_smoke_report*.md or command log section
|
|
purpose: prove direct-execution, QPI, and IA boundaries remain sane after patch.
|
|
required: yes
|
|
notes: Especially watch translation/direct transformation and IA trigger over-expansion.
|
|
|
|
- material_id: M-09-bundle-manifest
|
|
required_path_or_bundle_slot: ccra_review_bundle/<round-04.1>/BUNDLE_FILE_MANIFEST_*.md
|
|
purpose: let reviewers inspect package completeness without relying on upload order.
|
|
required: yes
|
|
notes: Use versioned filenames to avoid Web ChatGPT file confusion.
|
|
|
|
- material_id: M-10-optional-raw-changed-files
|
|
required_path_or_bundle_slot: ccra_review_bundle/<round-04.1>/optional_raw_changed_files_<round>.zip
|
|
purpose: allow exact inspection of selector, tests, scripts, reports, and relevant docs.
|
|
required: yes
|
|
notes: Preserve source-relative paths. Include zip validation/readback summary.
|
|
|
|
- material_id: M-11-local-ccra-first-turn-report
|
|
required_path_or_bundle_slot: local_ccra_reviews/round-04/pilot-02/04_LOCAL_CCRA_REVIEW_REPORT.md
|
|
purpose: optional context for Web CCRA on why these verification requirements exist.
|
|
required: no
|
|
notes: Include in Web upload only if Owner explicitly asks Web CCRA to review Local CCRA findings.
|
|
|
|
- material_id: M-12-owner-decision-record
|
|
required_path_or_bundle_slot: local_ccra_reviews/round-04/pilot-02/owner-decision.md or round-specific decision note
|
|
purpose: show which findings and same-class boundaries Owner accepted, rejected, or deferred.
|
|
required: yes before final acceptance claims
|
|
notes: Current status is pending; absence blocks acceptance language, not evidence collection.
|
|
```
|
|
|
|
## 4. Required Verification Evidence
|
|
|
|
```text
|
|
targeted_regression:
|
|
required: yes
|
|
cases: R04-BI-002, R04-BI-024, R04-BI-035, R04-BI-036
|
|
conditional_cases: R04-BI-022 if Owner accepts it as same-class QPI-limited intent
|
|
expected: target failures change in the intended direction only.
|
|
|
|
full_frozen_pool_rerun_or_before_after_comparison:
|
|
required: yes
|
|
scope: all 38 Round 04 inputs
|
|
expected: selected_models, rejected_models, no_call, score, reasons, penalties, and routing_notes before/after.
|
|
|
|
existing_regression_or_unit_tests:
|
|
required: yes
|
|
minimum: round04 runner tests, selector regression, any post-patch verification tests, and touched selector helper tests.
|
|
expected: executed, not dry-run.
|
|
|
|
calibration_smoke:
|
|
required: yes
|
|
expected: no-call, direct execution, QPI problem-definition, IA refusal, QPI-before-IA, and depth-limiting smoke coverage.
|
|
|
|
validation_report:
|
|
required: yes
|
|
minimum: index check and model library validation if package claims model-library consistency.
|
|
expected: no lifecycle upgrade from validation pass.
|
|
|
|
command_log:
|
|
required: yes
|
|
minimum: command, working directory, exit code, output file path, and whether files changed.
|
|
expected: dry-run must be labeled dry-run and cannot support pass claims.
|
|
|
|
dry_run_acceptable: no for final verification; yes only for planning/preflight manifest readability.
|
|
```
|
|
|
|
## 5. Expected Before / After Comparisons
|
|
|
|
```text
|
|
- comparison_id: C-01-target-fixes
|
|
before_source: reports/Round04_blind_routing_evaluation_report_2026-06-18.md
|
|
after_source: reports/Round04_1_post_patch_routing_verification_report_<date>.md
|
|
expected_fields: input_id, input_text, selected_models, rejected_models, no_call, score, reasons, penalties, routing_notes
|
|
target_cases: R04-BI-002, R04-BI-024, R04-BI-035, R04-BI-036
|
|
non_target_cases_to_watch: R04-BI-001, R04-BI-003, R04-BI-004, R04-BI-005, R04-BI-021, R04-BI-025, R04-BI-031, R04-BI-032, R04-BI-033, R04-BI-034
|
|
|
|
- comparison_id: C-02-same-class-boundary
|
|
before_source: reports/Round04_blind_routing_evaluation_report_2026-06-18.md
|
|
after_source: post-patch frozen-pool rerun
|
|
expected_fields: no_call, QPI selected/rejected, IA selected/rejected, exclusion reason
|
|
target_cases: R04-BI-022 if accepted into patch scope
|
|
non_target_cases_to_watch: all depth-limiting and IA-refusal cases, especially R04-BI-021 and R04-BI-023
|
|
|
|
- comparison_id: C-03-direct-execution-collateral
|
|
before_source: initial Round 04 report plus existing selector regression
|
|
after_source: post-patch report plus regression output
|
|
expected_fields: no_call status and reasons for fact lookup, translation, rewrite, formatting, direct execution, how-to, dictionary judgment
|
|
target_cases: R04-BI-002
|
|
non_target_cases_to_watch: R04-BI-001, R04-BI-003, R04-BI-004, R04-BI-005, R04-BI-031, R04-BI-032, R04-BI-033, R04-BI-034
|
|
|
|
- comparison_id: C-04-calibration-not-patched
|
|
before_source: first-turn report section 12
|
|
after_source: post-patch report calibration section
|
|
expected_fields: unchanged/deferred status, reason, Owner/Web question
|
|
target_cases: none
|
|
non_target_cases_to_watch: R04-BI-010 through R04-BI-017, R04-BI-018, R04-BI-026, R04-BI-027, R04-BI-030, R04-BI-037, R04-BI-038
|
|
```
|
|
|
|
## 6. Open Owner Decisions
|
|
|
|
```text
|
|
- decision_id: OD-R04-001
|
|
decision_needed: accept the four must-fix initial Round 04 target failures as the patch target.
|
|
options: accept_all | accept_subset | request_web_baseline_comparison_first
|
|
recommended_default: accept_all
|
|
blocks_next_review: no for evidence collection; yes for final acceptance language.
|
|
|
|
- decision_id: OD-R04-002
|
|
decision_needed: classify R04-BI-022 as must-fix same-class, calibration-only, or acceptable conservative no-call.
|
|
options: include_in_patch | calibration_only | leave_no_call
|
|
recommended_default: include_in_patch
|
|
blocks_next_review: yes if the next package claims R04-BI-022 as accepted collateral or target behavior.
|
|
|
|
- decision_id: OD-R04-003
|
|
decision_needed: define policy for descriptive governance/capacity statements without explicit requests.
|
|
options: provisional_qpi | ask_clarifying_question | no_call_until_request
|
|
recommended_default: ask_clarifying_question or provisional_qpi with low confidence.
|
|
blocks_next_review: no for narrow patch verification; yes for broader selector maturity claims.
|
|
|
|
- decision_id: OD-R04-004
|
|
decision_needed: define direct IA versus QPI-first route for natural model-extraction/deep-mechanism inputs.
|
|
options: direct_ia_when_depth_intent_clear | qpi_first_always | qpi_plus_ia_when_both_signals_present
|
|
recommended_default: qpi_plus_ia_when both clear depth intent and model-extraction intent appear.
|
|
blocks_next_review: no for narrow patch verification; yes for IA route maturity claims.
|
|
|
|
- decision_id: OD-R04-005
|
|
decision_needed: decide whether helper profile tuning is required before formal Round 05 Local CCRA use.
|
|
options: tune_now | document_limitations | defer
|
|
recommended_default: document_limitations for this pilot, tune before repeated formal use.
|
|
blocks_next_review: no, if limitations are explicit.
|
|
```
|
|
|
|
## 7. Explicit Non-Goals
|
|
|
|
```text
|
|
- non_goal: no lifecycle status upgrade
|
|
reason: QPI and Intellectual Archaeology remain draft / B / pending until Owner / CCRA lifecycle authority changes that.
|
|
|
|
- non_goal: no Round 05 acceptance claim
|
|
reason: Round 04.1 verification only proves a scoped selector patch if evidence passes.
|
|
|
|
- non_goal: no second blind test claim
|
|
reason: after expected labels and target cases exist, this is a post-patch verification pass, not blind evaluation.
|
|
|
|
- non_goal: no broad governance/capacity keyword patch
|
|
reason: those cases are calibration-only until Owner/Web policy is explicit.
|
|
|
|
- non_goal: no IA trigger broadening without calibration
|
|
reason: IA is a deep model and should not become default because of words like structure, mechanism, long-term, or model.
|
|
|
|
- non_goal: no LLM selector, RAG, database, frontend, backend, user system, or third model
|
|
reason: outside model_library_mvp selector patch scope.
|
|
|
|
- non_goal: no helper exit-code absolutism
|
|
reason: helper output is evidence only; profile mismatch, dry-run, and broad scans must be judged by the reviewer.
|
|
```
|
|
|
|
## 8. Recommended Web CCRA Review Questions
|
|
|
|
1. Do the Round 04.1 submitted materials prove the four target cases were fixed without creating non-target regressions across the full frozen 38-case pool?
|
|
2. Should `R04-BI-022` be treated as an accepted same-class QPI-limited override, calibration-only, or conservative no-call?
|
|
3. Are governance/capacity descriptive cases (`R04-BI-010` through `R04-BI-017`) better handled as provisional QPI, clarification, or no-call until the user asks for analysis?
|
|
4. Should natural model-extraction/deep-mechanism cases (`R04-BI-018`, `026`, `027`, `038`) directly select IA, select QPI first, or select both when depth intent is explicit?
|
|
5. Is the verification package sufficient to avoid a further small patch round, or should remaining boundary issues be deferred into a broader calibration round?
|
|
6. Does any wording in the package imply lifecycle promotion, Round 05 acceptance, or Owner/Web approval beyond the evidence?
|
|
|
|
## 9. Material Package Checklist
|
|
|
|
```text
|
|
review_brief:
|
|
required: yes
|
|
pass_condition: states target scope, non-goals, and post-patch verification identity.
|
|
|
|
rule_or_model_diff:
|
|
required: yes
|
|
pass_condition: selector rule changes mapped to target cases; no model content/lifecycle changes unless separately justified.
|
|
|
|
expected_labels_or_decisions:
|
|
required: yes
|
|
pass_condition: target, same-class, calibration-only, and explicitly-not-patched cases are separated.
|
|
|
|
post_patch_verification_report:
|
|
required: yes
|
|
pass_condition: all 38 frozen inputs rerun with score/reason details.
|
|
|
|
regression_and_command_log:
|
|
required: yes
|
|
pass_condition: commands executed with exit codes; no dry-run used as pass evidence.
|
|
|
|
bundle_manifest:
|
|
required: yes
|
|
pass_condition: versioned manifest names recognized by human reviewer even if generic helper profile is noisy.
|
|
|
|
optional_raw_changed_files:
|
|
required: yes
|
|
pass_condition: source-relative zip entries and readback/validation summary included.
|
|
|
|
local_ccra_report:
|
|
required: optional
|
|
pass_condition: included only with Owner approval; otherwise referenced locally, not uploaded as formal Web evidence.
|
|
|
|
owner_decision_record:
|
|
required: yes before acceptance claims
|
|
pass_condition: records accepted, rejected, deferred, or escalated findings; current `pending` is not acceptance.
|
|
```
|
|
|
|
## 10. Limits And Carry-Forward Risks
|
|
|
|
- Owner decision remains pending. The next package may gather evidence, but it should not claim accepted fixes without a decision record.
|
|
- The first-turn report identified clear `fix_now` targets but did not inspect any post-patch evidence. The next review must not cite this plan as proof that the patch worked.
|
|
- `routing-behavior-diff-audit` was skipped in review_turn for good reason, but before/after routing comparison is required for the post-patch pass.
|
|
- `regression-validation-gate-runner` was dry-run only. The next package needs executed gates if it claims verification.
|
|
- `review-bundle-audit` used a generic profile that did not recognize versioned bundle manifests. Human-readable bundle manifest evidence is still required, and helper profile limitations should be documented.
|
|
- `lifecycle-status-guard-scan` was broad and noisy. The next review should include a lifecycle/scope guard section focused on the submitted package, not a whole-repository false-positive dump.
|
|
- Calibration-only cases must stay explicitly out of the narrow patch acceptance claim. They can be listed as next-round policy questions, not evidence that the patch is incomplete.
|
|
- Current project docs may contain later Round 04.1 language. The next review should clearly distinguish source evidence, patch evidence, closeout language, and lifecycle claims.
|