93 lines
3.1 KiB
Markdown
93 lines
3.1 KiB
Markdown
# Scripts
|
|
|
|
This folder contains simple local scripts for validation and selector demos.
|
|
|
|
Current script:
|
|
|
|
- `validate_model_library.py`
|
|
- `check_card_contract.py`
|
|
- `rebuild_indexes.py`
|
|
- `check_model_card_sync.py`
|
|
- `run_selector_regression.py`
|
|
- `run_selector_demo.py`
|
|
- `run_selector_calibration_smoke.py`
|
|
- `run_round04_blind_routing.py`
|
|
- `run_round04_post_patch_verification.py`
|
|
- `init_round05_context.py`
|
|
- `run_round05_1_selector_patch_audit.py`
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
python scripts\validate_model_library.py
|
|
```
|
|
|
|
The script checks JSON parsing, required model fields, unique IDs, source references, excerpt references, regression case model references, and model/card index drift. It writes `reports/validation_report.md`.
|
|
|
|
Rebuild or check model/card indexes:
|
|
|
|
```powershell
|
|
python scripts\rebuild_indexes.py --write
|
|
python scripts\rebuild_indexes.py --check
|
|
```
|
|
|
|
The index script regenerates `models/model_index.json` and `cards/card_index.md` from model JSON, card files, and regression cases. It writes `reports/index_rebuild_report.md`.
|
|
|
|
Run the card contract checker:
|
|
|
|
```powershell
|
|
python scripts\check_card_contract.py
|
|
```
|
|
|
|
Run the selector demo:
|
|
|
|
```powershell
|
|
python scripts\run_selector_demo.py
|
|
```
|
|
|
|
Run selector regression and model/card sync checks:
|
|
|
|
```powershell
|
|
python scripts\run_selector_regression.py
|
|
python scripts\run_selector_calibration_smoke.py
|
|
python scripts\check_model_card_sync.py
|
|
```
|
|
|
|
`run_selector_calibration_smoke.py` checks selector calibration inputs, including strict `no_call`, `no_call_or_low_priority`, `select_qpi_reject_ia`, and IA heavy-depth gate expectations.
|
|
|
|
Run Round 04 blind routing evaluation:
|
|
|
|
```powershell
|
|
python scripts\run_round04_blind_routing.py
|
|
```
|
|
|
|
This reads `selector/round04_blind_inputs.json` and writes `reports/Round04_blind_routing_evaluation_report_2026-06-18.md`. It does not score correctness or use expected routing labels.
|
|
|
|
Run Round 04.1 post-patch routing verification:
|
|
|
|
```powershell
|
|
python scripts\run_round04_post_patch_verification.py
|
|
```
|
|
|
|
This reruns the frozen Round 04 input pool after selector rule patches and writes `reports/Round04_1_post_patch_routing_verification_report_2026-06-18.md`. It is not a second blind test.
|
|
|
|
Initialize Round 05 new-session context:
|
|
|
|
```powershell
|
|
python scripts\init_round05_context.py
|
|
```
|
|
|
|
This reads Round 04 / 04.1 reports, selector source files, regression files, and workflow decisions, then writes `ccra_review_bundle/round-05_init_selector-calibration-policy/00_ROUND05_NEW_SESSION_CONTEXT.md`. It does not modify selector rules, model cards, model specs, or regression cases.
|
|
|
|
Run Round 05.1 selector patch audit:
|
|
|
|
```powershell
|
|
python scripts\run_round05_1_selector_patch_audit.py baseline
|
|
python scripts\run_round05_1_selector_patch_audit.py after
|
|
python scripts\run_round05_1_selector_patch_audit.py diff
|
|
```
|
|
|
|
This captures before / after selector behavior for the Round 04 blind pool, selector calibration inputs, and aggregate regression cases. The diff report includes selected models, rejected models, scores, no-call status, reasons, penalties, target classification, changed fields, and non-target disposition placeholders.
|
|
|
|
Prefer Python standard library before adding dependencies.
|