93 lines
4.2 KiB
Markdown
93 lines
4.2 KiB
Markdown
---
|
|
name: routing-behavior-diff-audit
|
|
description: Use when Codex needs to compare before/after selector, routing, QPI, classifier, rules-engine, triage, assignment, or model-governance outputs to identify targeted route changes, collateral non-target changes, no-call changes, missing cases, and new or resolved expected-route mismatches before review, release, handoff, CCRA/local review, or regression closeout.
|
|
---
|
|
|
|
# Routing Behavior Diff Audit
|
|
|
|
## Purpose
|
|
|
|
Run a deterministic before/after behavior diff for rules-based routing outputs. Use the bundled script to compare case IDs, routes, optional expected routes, missing cases, no-call transitions, and targeted versus non-target changes.
|
|
|
|
This Skill reports observed differences only. It must not edit routing rules, change regression cases, update expected labels, infer product meaning, or decide whether a patch should close.
|
|
|
|
## Command
|
|
|
|
```powershell
|
|
conda run -n skills-vault python .\scripts\routing_behavior_diff_audit.py `
|
|
--before-results C:\path\before-results `
|
|
--after-results C:\path\after-results `
|
|
--output-dir C:\path\review-run `
|
|
--case-id-field case_id `
|
|
--route-field route `
|
|
--expected-route-field expected_route `
|
|
--targeted-case CASE-001
|
|
```
|
|
|
|
When installed under `C:\Users\wangq\.agents\skills\routing-behavior-diff-audit`, run from that Skill directory or use the installed script path.
|
|
|
|
## Inputs
|
|
|
|
- `--before-results`: JSON, JSONL, or Markdown file, or a directory containing those files.
|
|
- `--after-results`: JSON, JSONL, or Markdown file, or a directory containing those files.
|
|
- `--output-dir`: directory where reports are written.
|
|
- `--case-id-field`: field containing the stable case ID. Default: `case_id`.
|
|
- `--route-field`: field containing the routing decision. Default: `route`.
|
|
- `--expected-route-field`: optional field containing the expected route for failure comparison.
|
|
- `--targeted-case`: repeatable case ID that the patch intentionally targets.
|
|
- `--targeted-cases-file`: optional JSON/YAML list, or object with `targeted_cases`, `case_ids`, or `cases`.
|
|
|
|
JSON is the stable input target. Directory inputs are scanned recursively for `.json`, `.jsonl`, `.md`, and `.markdown`; Markdown support expects a fenced JSON or YAML block.
|
|
|
|
## Outputs
|
|
|
|
The script writes under `output_dir`:
|
|
|
|
```text
|
|
routing-behavior-diff.md
|
|
routing-behavior-diff.json
|
|
```
|
|
|
|
The JSON report includes:
|
|
|
|
- `total_cases_compared`
|
|
- `unchanged_cases`
|
|
- `changed_cases`
|
|
- `targeted_changes`
|
|
- `non_target_changes`
|
|
- `new_failures`
|
|
- `resolved_failures`
|
|
- `missing_before_cases`
|
|
- `missing_after_cases`
|
|
- `diff_table`
|
|
- `machine_readable_summary`
|
|
|
|
`machine_readable_summary.status` is `PASS`, `ATTENTION`, or `FAIL` for automation routing only. It is not a patch approval, owner approval, CCRA decision, or release decision.
|
|
|
|
## Comparison Rules
|
|
|
|
- Compare only case IDs present in both before and after for `total_cases_compared`.
|
|
- Classify after-only IDs as `missing_before_cases`.
|
|
- Classify before-only IDs as `missing_after_cases`.
|
|
- Treat a common case as changed when the route, expected route, no-call status, or expected-route failure status changes.
|
|
- Classify changed cases listed by `--targeted-case` or `--targeted-cases-file` as `targeted_changes`.
|
|
- Classify all other changed cases as `non_target_changes`.
|
|
- Compute `new_failures` and `resolved_failures` only when `--expected-route-field` is supplied and present in the records.
|
|
- Treat empty, null-like, and common no-call spellings such as `no_call`, `no-call`, and `no call` as no-call values.
|
|
|
|
## Safety
|
|
|
|
- Read only configured before/after result paths and optional targeted-cases file.
|
|
- Write only `routing-behavior-diff.md` and `routing-behavior-diff.json` under `output_dir`.
|
|
- Do not edit selectors, routing rules, regression cases, expected labels, review bundles, model cards, or governance artifacts.
|
|
- Do not decide acceptance, lifecycle status, patch closure, release readiness, or owner approval.
|
|
|
|
## Validation
|
|
|
|
After changing diff behavior, run:
|
|
|
|
```powershell
|
|
conda run -n skills-vault python -B -m unittest discover -s skills/routing-behavior-diff-audit/tests -v
|
|
conda run -n skills-vault python -B C:\Users\wangq\.codex\skills\.system\skill-creator\scripts\quick_validate.py skills/routing-behavior-diff-audit
|
|
```
|