26 KiB
26 KiB
CCPE Quality Rubric
1. Purpose
This rubric defines how to evaluate the quality of CCPE artifacts.
It applies to:
CCPE-Lite Prompt Cards
CCPE-Agent Specs
CCPE-Skill Specs
CCPE-Runtime Specs
Model Cards
Model Indexes
Hybrid artifacts
The goal is not to reward length or complexity.
The goal is to judge whether the artifact is clear, useful, safe, reusable, maintainable, and faithful to its underlying cognitive structure.
2. Scoring Scale
Use this scale for each criterion:
0 = Missing
1 = Weak
2 = Adequate
3 = Strong
4 = Excellent
Optional severity labels:
S = Structural blocker
A = Major issue
B = Moderate issue
C = Minor issue
3. Core Evaluation Criteria
The main criteria are:
1. Purpose Fit
2. Scenario Fit
3. Classification Accuracy
4. Structural Clarity
5. Boundary Precision
6. Capability Realism
7. Context Handling
8. Model Fidelity
9. Lite Kernel Fidelity
10. Kernel Force
11. Production Stability
12. Skill Reusability
13. Authority Clarity
14. Workflow Coherence
15. State Awareness
16. Output Usability
17. Evaluation Strength
18. Human-in-the-Loop Design
19. Runtime Safety
20. Portability
21. Maintainability
22. Intellectual Flavor Preservation
Not every criterion applies equally to every artifact.
4. Purpose Fit
4.1 Question
Does the artifact clearly serve its intended purpose?
4.2 Good Signs
Primary objective is explicit.
Success criteria are defined.
Non-goals are stated.
The artifact does not drift beyond its purpose.
The form matches the use case.
4.3 Bad Signs
Role is vivid but goal is vague.
The artifact tries to do everything.
Output does not match the intended work.
The user cannot tell what good performance means.
4.4 Common Fixes
Add Objective Layer.
Add Non-Goals.
Add Success Criteria.
Remove unrelated capabilities.
4A. Scenario Fit
4A.1 Question
Does the artifact form match how the user actually uses or plans to use it?
4A.2 Good Signs
Usage scenario is explicit.
Target platform is explicit.
Single-agent vs multi-agent use is clear.
Manual orchestration vs automation is clear.
Codex-callable Skill need is justified.
The artifact is not expanded beyond scenario needs.
4A.3 Bad Signs
A Web-style expert prompt is overbuilt as a full Runtime.
A committee member is converted to Agent Spec before collaboration needs are clear.
A Codex Skill is omitted even though automatic invocation is required.
Scenario assumptions are hidden.
The artifact is classified only from its content, not its use.
4A.4 Common Fixes
Run Scenario Probe.
Document current and planned usage.
Choose Lite / Agent / Skill / Runtime from scenario evidence.
Defer unnecessary layers.
5. Classification Accuracy
5.1 Question
Is the artifact classified correctly as Lite, Agent, Skill, Runtime, Model Card, Model Index, or Hybrid?
5.2 Good Signs
The primary form is clear.
Embedded components are identified.
Hybrid structure is acknowledged when needed.
No unnecessary complexity is added.
No complex artifact is flattened into a prompt.
Mature single-agent prompts are not expanded without scenario evidence.
5.3 Bad Signs
A full workflow is written as one prompt.
A reusable model is trapped inside one agent.
A simple expert prompt is overbuilt as Runtime.
A tool procedure is mixed into persona instructions.
Every mature prompt is automatically split into Lite + Agent + Skill + Runtime.
5.4 Common Fixes
Reclassify.
Split into Model Card, Skill, Agent Spec, or Runtime.
Preserve a Lite version if portability matters.
6. Structural Clarity
6.1 Question
Can a human or AI system understand how the artifact is organized?
6.2 Good Signs
Sections are logically ordered.
Objective, role, capabilities, constraints, workflow, and output are distinguishable.
No major duplicate sections.
No contradictory instructions.
6.3 Bad Signs
Same concept appears in multiple places with different meanings.
Workflow is repeated.
Capability and authority are confused.
Model and persona are fused without explanation.
6.4 Common Fixes
Reorganize using CCPE Layer Spec.
Remove duplicate sections.
Separate Agent, Skill, Model, and Runtime components.
7. Boundary Precision
7.1 Question
Does the artifact clearly define what it should not do?
7.2 Good Signs
Hard constraints are explicit.
Soft constraints are marked as preferences.
Refusal conditions are clear.
Conflict resolution rules exist.
Scope boundaries are testable.
7.3 Bad Signs
Artifact accepts any input.
Constraints are vague.
It cannot say when the task is inappropriate.
It claims universal applicability.
7.4 Common Fixes
Add Scope.
Add Non-Goals.
Add Refusal Conditions.
Add Failure Conditions.
Add Conflict Resolution.
8. Capability Realism
8.1 Question
Are the stated capabilities realistic and executable?
8.2 Good Signs
Capabilities match available context and tools.
Internal reasoning is separated from external actions.
No claims of omniscience.
Tool-dependent abilities are marked.
8.3 Bad Signs
Claims all-domain mastery without source policy.
Says it can verify facts without retrieval.
Promises perfect correctness.
Uses phrases like all-knowing mode.
8.4 Common Fixes
Add Source Policy.
Add Tool Preconditions.
Add Uncertainty Handling.
Replace omniscient language with evidence-based rules.
9. Context Handling
9.1 Question
Does the artifact define what context it uses and how?
9.2 Good Signs
Input contract is clear.
User-provided context is prioritized.
Retrieved context is treated critically.
Memory and state are separated.
Source priority exists.
Uncertainty is acknowledged.
9.3 Bad Signs
External search is assumed but not specified.
User context and model assumptions are mixed.
Old knowledge is treated as current.
Retrieved material is treated as truth.
9.4 Common Fixes
Add Input Contract.
Add Source Priority.
Add Retrieval Policy.
Add Context Refresh Rules.
Add Uncertainty Handling.
10. Model Fidelity
10.1 Question
If the artifact uses a cognitive model, does it preserve the model accurately?
10.2 Good Signs
Core assumptions are preserved.
Mechanism is clear.
Scope is defined.
Failure modes are included.
Falsification boundary exists.
Original terminology is retained when meaningful.
Concepts are separated by function: lens, claim, mechanism, generator, metaphor, output format.
Inferred claims are marked as reconstructions rather than attributed as explicit source claims.
10.3 Bad Signs
Model is reduced to generic advice.
Metaphor is kept but mechanism is lost.
The model becomes too broad to be falsifiable.
Important edge cases are removed.
A useful lens is incorrectly treated as a causal generator.
The artifact attacks a reconstructed claim as if it were stated by the source.
10.4 Common Fixes
Create Model Card.
Restore core mechanism.
Add scope and failure modes.
Add falsification boundary.
Separate model from agent persona.
Add Concept Function Discipline.
Add Reconstructed Claim Labeling.
10A. Lite Kernel Fidelity
10A.1 Question
When the target is CCPE-Lite, does the artifact preserve the single-agent prompt kernel that makes it work in Web / GPT / Gemini / Claude style environments?
10A.2 Good Signs
Core / Execution / Constraint / Operation are present or clearly compressed.
The prompt can be copied into a chat product and used directly.
The output standard and workflow are operational, not merely descriptive.
Original voice and working behavior are preserved for migrated mature agents.
Regression testing against the old prompt is planned or documented.
10A.3 Bad Signs
The Lite prompt reads like a shortened Agent Spec.
The old prompt's practical force is lost.
The workflow no longer produces the expected output.
The prompt depends on external files unavailable in Web environments.
The migration improves structure but worsens production behavior.
10A.4 Common Fixes
Restore the CCPE 2.0 four-layer prompt kernel.
Use Outside-In construction.
Embed necessary model and output rules for portability.
Add regression comparison against the old agent.
Move nonessential governance details out of Lite.
10A.5 Minimal-Kernel Migration Check
For mature CCPE 2.0 single-agent prompts, evaluate whether an original-kernel-minimal-lite candidate exists before judging a full Lite rewrite.
This check supports the migration-policy split between Fast Migration Lane and Refinement Lane.
Good signs:
Original prompt is kept as regression reference.
Visible source-level risks are first captured in an Original Source Judgment Report.
The user chooses the source decision before migration.
Minimal-kernel candidate preserves old behavior with only necessary migration repairs.
`## Original Kernel` preserves the original prompt body verbatim.
Full Lite rewrite is treated as optional refined Lite optimization, not the default first artifact.
Fast Migration Lane, optional refined Lite optimization, or later-layer expansion is chosen explicitly.
Regression compares original, minimal-kernel, and refined candidates when available.
Minimal-kernel is recognized as a high-ROI baseline, not automatically as final refined Lite.
Bad signs:
The original working prompt is replaced before kernel fidelity is tested.
Source defects are silently fixed without a judgment report or user decision.
Platform incompatibilities are confused with source defects.
Kernel features are flattened because they look unusual.
The full Lite rewrite is promoted because it is cleaner, even though outputs are weaker.
The candidate translates, paraphrases, deduplicates, reorders, or smooths the original kernel while still claiming `original-kernel-minimal-lite`.
No temporary production path exists for batch migration.
The process jumps from original-kernel-minimal-lite directly into Model Card, Skill, Agent Spec, Runtime, or Model Index work before deciding whether refined Lite optimization is needed.
The process treats a refined Lite success case, such as Cognitive Imaging, as the default requirement for every agent.
Common fixes:
Produce an Original Source Judgment Report.
Classify findings as source defect, platform incompatibility, kernel feature, or ambiguous finding.
Let the user or original CCPE agent review the judgment report before source repair.
Generate original-kernel-minimal-lite.
Place the untouched original prompt body under `## Original Kernel`.
Move translation, paraphrase, deduplication, reordering, terminology replacement, workflow rewrite, and style smoothing into Refinement Lane.
Run a small regression batch.
Promote minimal-kernel first when ROI matters.
Reserve refined Lite for high-value agents.
Use the proven regression pattern only when budget is justified: three prompt variants, two platforms, four article types, 24 result files per agent family, and a regression report.
10B. Kernel Force
10B.1 Question
Does the migrated artifact preserve the original prompt's effective cognitive pressure?
10B.2 Good Signs
Original method pressure remains visible.
Distinctive voice and sharpness are preserved.
Core metaphor still performs its structural function.
Output behavior resembles the successful old agent.
The artifact still produces the kind of insight users valued in the original.
In Fast Migration Lane, the `Original Kernel` block is a verbatim copy of the old working prompt.
10B.3 Bad Signs
The output is cleaner but weaker.
The old agent's productive strangeness is removed.
The prompt becomes generic consulting or generic review language.
Required labels remain but the actual operation changes.
New template rules cause concept proliferation or method drift.
The original prompt body is translated, paraphrased, deduplicated, reordered, or style-smoothed inside a supposed `Original Kernel`.
10B.4 Common Fixes
Compare against original outputs.
Restore original working kernel.
Use original-kernel-minimal-lite as the next candidate.
Add targeted discipline rules instead of rewriting the whole prompt.
10C. Production Stability
10C.1 Question
Does the migrated artifact run safely and consistently in the target model environment?
10C.2 Good Signs
Output format is stable.
Platform boundaries are clear.
Hidden chain-of-thought disclosure is repaired.
Source and retrieval rules are explicit where needed.
The prompt remains portable across intended chat environments.
The model does not drift into generic review, excessive boilerplate, or template overfitting.
10C.3 Bad Signs
The prompt works only in one model environment.
Format collapses or expands unpredictably.
Platform safety rules are missing.
The output is stable only because it became generic.
The prompt depends on unavailable external files.
10C.4 Common Fixes
Add platform boundary.
Add output validation discipline.
Add concise and full modes if needed.
Test across target environments.
Balance production stability against kernel force before promotion.
10D. Concept Function Discipline
10D.1 Question
Does the artifact distinguish what each concept is doing before evaluating or operationalizing it?
10D.2 Good Signs
Concepts are classified as lens, claim, metaphor, mechanism, generator, procedure, constraint, or output form.
Evaluation does not force every useful lens to become a causal generator.
Metaphors are tested for structural function before being kept or removed.
Pressure tests target the right object.
10D.3 Bad Signs
A lens is judged as if it were a full causal mechanism.
A metaphor is dismissed before its structural function is checked.
A procedure is treated as a model.
A local heuristic is promoted into a universal law.
10D.4 Common Fixes
Add concept-function labels.
Separate lens validity from generator validity.
Test metaphors for operational function.
Mark local heuristics and scope boundaries.
10E. Reconstruction Discipline
10E.1 Question
When an artifact tests an implicit claim, does it label the claim as a reconstruction and avoid attributing it as an explicit source statement?
10E.2 Good Signs
Explicit source claims are separated from inferred claims.
Reconstructed claims are marked before testing.
The strongest plausible version is tested, not a weaker caricature.
Ambiguous source intent is marked as uncertain.
10E.3 Bad Signs
The artifact criticizes a claim the source did not make.
A nuanced concept is reconstructed as a crude opposite.
The pressure test wins by changing the original claim into a weaker version.
10E.4 Common Fixes
Label reconstructed claims.
Quote or summarize the explicit source claim separately.
Use a charitable strong-form reconstruction.
Add uncertainty notes when the source intent is ambiguous.
10F. Output Structure Discipline
10F.1 Question
Does the output structure help the user inspect, compare, and reuse the result?
10F.2 Good Signs
Markdown hierarchy is clean.
List nesting matches conceptual hierarchy.
Each section starts with a judgment before supporting detail.
Formatting does not obscure the conclusion.
10F.3 Bad Signs
Everything is flattened into one list level.
Bullets, headings, and paragraphs compete for hierarchy.
The output is formally structured but hard to scan.
The result is correct but difficult to use downstream.
10F.4 Common Fixes
Define output hierarchy rules.
Keep evidence subordinate to judgments.
Use consistent heading levels.
Add a final usable summary or next-step section.
11. Skill Reusability
11.1 Question
If the artifact contains a reusable method, can it be extracted as a Skill?
11.2 Good Signs
Method has clear trigger conditions.
Steps are stable.
Inputs and outputs are definable.
Multiple agents could use it.
Validation is possible.
11.3 Bad Signs
Reusable procedure is buried in persona.
Method is duplicated across several agents.
Tool use is described inconsistently.
No clear output or failure handling.
11.4 Common Fixes
Extract Skill.
Add input/output contract.
Add execution workflow.
Add validation criteria.
Reference Skill from Agent.
12. Authority Clarity
12.1 Question
Does the artifact define what it can do autonomously and what requires human approval?
12.2 Good Signs
Autonomous actions are defined.
Confirmation-required actions are defined.
Forbidden actions are defined.
Human decision gates are explicit.
Risk levels are considered.
12.3 Bad Signs
Agent can rewrite or execute without approval.
Tool permissions are implied.
Authority is mixed with capability.
No escalation rule exists.
12.4 Common Fixes
Add Authority Layer.
Add Human Decision Gates.
Add Tool Permission Rules.
Add Forbidden Actions.
13. Workflow Coherence
13.1 Question
Does the workflow proceed logically and reliably?
13.2 Good Signs
Steps are ordered.
Branch conditions are defined.
Stop conditions exist.
Fallback behavior exists.
Discussion and execution modes are distinct.
13.3 Bad Signs
Workflow repeats itself.
The agent must always follow a long process even when unnecessary.
No handling for poor input.
No distinction between new request and follow-up discussion.
13.4 Common Fixes
Define Main Workflow.
Add Branch Logic.
Add Stop Conditions.
Add Fallback Workflow.
Separate Report Mode from Discussion Mode.
14. State Awareness
14.1 Question
Does the artifact need to track state, and if so, does it do so explicitly?
14.2 Good Signs
Session state is defined.
Persistent state is defined if needed.
Decision logs are specified.
Open questions are tracked.
Resume rules exist.
14.3 Bad Signs
Long-running process has no state.
Committee workflow does not track decisions.
Model Index updates have no source or review status.
Agent remembers vaguely without rules.
14.4 Common Fixes
Add State Layer.
Add Decision Log.
Add Version Markers.
Add Resume Rules.
Add Model Index status fields.
15. Output Usability
15.1 Question
Is the output useful, structured, and appropriate for the task?
15.2 Good Signs
Output format is explicit.
Required sections are clear.
Output matches user workflow.
Actionable next steps are included.
No unnecessary verbosity.
15.3 Bad Signs
Output is generic.
Output is too long to use.
No prioritization.
No summary or action path.
Report format does not match the user's work.
15.4 Common Fixes
Add Output Layer.
Add Delivery Checklist.
Add prioritization.
Add concise and full modes.
Add downstream usage target.
16. Evaluation Strength
16.1 Question
Can the artifact's work be checked?
16.2 Good Signs
Quality rubric exists.
Validation checklist exists.
Failure criteria are defined.
Human acceptance criteria are clear.
Test cases exist for important artifacts.
16.3 Bad Signs
No way to tell if output is good.
Agent self-declares success.
No falsification boundary for models.
No test cases for Skills or Runtimes.
16.4 Common Fixes
Add Evaluation Layer.
Add validation checklist.
Add failure conditions.
Add test cases.
Add human review protocol.
17. Human-in-the-Loop Design
17.1 Question
Does the artifact correctly preserve human judgment where needed?
17.2 Good Signs
Human decision gates are explicit.
Human owns final judgment in deep work.
Automation boundaries are clear.
The artifact asks for confirmation before risky actions.
17.3 Bad Signs
Agent silently decides major conceptual questions.
Automation is applied to high-uncertainty thinking.
Human role is vague.
Committee workflow lacks stage approval.
17.4 Common Fixes
Add Human Decision Gates.
Add Stage Approval.
Add Escalation Rules.
Mark depth-oriented tasks as non-fully-automatable.
18. Runtime Safety
18.1 Question
If the artifact runs tools, files, code, or workflows, is it safe?
18.2 Good Signs
Tool scope is explicit.
Allowed and forbidden actions are defined.
Risky actions require confirmation.
Validation and rollback exist.
Errors are handled.
18.3 Bad Signs
Agent can write files without permission.
No validation after tool use.
No rollback strategy.
No distinction between draft and canonical files.
18.4 Common Fixes
Add Runtime Layer.
Add Authority Layer.
Add Recovery Rules.
Add Draft-first Policy.
Add Validation Rules.
19. Portability
19.1 Question
Can the artifact be used in the intended platform?
19.2 Good Signs
Lite prompts are copy-paste friendly.
Agent Specs can be adapted to Codex / Claude Code / OpenClaw.
Skills use clear references and templates.
Runtime does not assume unsupported features.
19.3 Bad Signs
Prompt assumes unavailable tools.
Skill format does not match platform.
Agent relies on hidden context.
Runtime depends on unspecified environment.
19.4 Common Fixes
Add Platform Target.
Add Deployment Notes.
Separate platform-neutral spec from platform-specific implementation.
Keep portable Lite version where useful.
20. Maintainability
20.1 Question
Can the artifact be updated without breaking everything?
20.2 Good Signs
Version metadata exists.
Dependencies are listed.
Model and Skill references are separate.
Change log or status exists.
Canonical location is clear.
20.3 Bad Signs
One giant prompt contains everything.
Same model copied into many agents.
No status or version.
No index entry.
20.4 Common Fixes
Extract Model Card.
Extract Skill.
Add Knowledge Asset Layer.
Add Version Metadata.
Update Model Index.
21. Intellectual Flavor Preservation
21.1 Question
Does the refactored artifact preserve the user's original thinking?
21.2 Good Signs
Core metaphor remains meaningful.
Distinctive terminology is preserved.
The model's tension and sharpness remain.
The artifact does not become generic.
Original cognitive stance is visible.
21.3 Bad Signs
Unique model becomes generic consulting advice.
Sharp critique becomes bland summary.
Metaphor is removed even though it carried mechanism.
Conceptual edge is softened unnecessarily.
21.4 Common Fixes
Restore key terms.
Restore metaphor where structurally meaningful.
Add Model Fidelity note.
Compare refactor against original.
Ask user to approve major conceptual compression.
22. Artifact-Specific Rubric
22.1 CCPE-Lite
Prioritize:
Purpose Fit
Role Clarity
Portability
Output Usability
Boundary Precision
Model Fidelity if model-backed
Avoid overloading with:
Heavy Runtime
Long authority systems
Complex state
Too many sections
22.2 CCPE-Agent
Prioritize:
Objective
Role
Input / Output Contract
Capability
Authority
Workflow
Evaluation
Collaboration
Maintainability
22.3 CCPE-Skill
Prioritize:
Trigger Conditions
Input / Output
Procedure
Reusability
Validation
Failure Handling
Tool Rules if applicable
22.4 CCPE-Runtime
Prioritize:
Stages
Participants
Handoff
State
Human Decision Gates
Authority
Validation
Recovery
Archival
22.5 Model Card
Prioritize:
Model Fidelity
Scope
Core Assumptions
Mechanism
Procedure
Failure Modes
Falsification Boundary
Related Models
Source Traceability
22.6 Model Index
Prioritize:
Taxonomy
Hierarchy
Dependencies
Usage Mapping
Version Status
Source Tracking
Review Status
23. Quality Report Format
For original CCPE 2.0 agent upgrade preparation, do not use a generic quality report as the first document.
workbench/analysis/{artifact-slug}-original-source-judgment-report.md
Use the Original Source Judgment Report template and include quality judgments inside its findings, kernel protection, source decision, and validation sections.
Do not print the full source judgment report in chat unless the user explicitly asks for inline output.
# CCPE Quality Report
## 1. Artifact
Name:
Path:
Type:
## 2. Classification
Primary:
Secondary:
Hybrid Components:
## 3. Score Summary
| Criterion | Score | Severity | Notes |
|---|---:|---|---|
| Purpose Fit | | | |
| Scenario Fit | | | |
| Classification Accuracy | | | |
| Structural Clarity | | | |
| Boundary Precision | | | |
| Capability Realism | | | |
| Context Handling | | | |
| Model Fidelity | | | |
| Lite Kernel Fidelity | | | |
| Kernel Force | | | |
| Production Stability | | | |
| Concept Function Discipline | | | |
| Reconstruction Discipline | | | |
| Skill Reusability | | | |
| Authority Clarity | | | |
| Workflow Coherence | | | |
| State Awareness | | | |
| Output Usability | | | |
| Output Structure Discipline | | | |
| Evaluation Strength | | | |
| Human-in-the-Loop Design | | | |
| Runtime Safety | | | |
| Portability | | | |
| Maintainability | | | |
| Intellectual Flavor Preservation | | | |
## 4. Major Findings
...
## 5. Required Fixes
...
## 6. Recommended Improvements
...
## 7. Refactor Direction
...
## 8. Human Decisions Needed
...
24. Final Rule
A high-quality CCPE artifact is not the longest artifact.
It is the artifact that has the right structure for its job.
Evaluate quality by fitness, clarity, fidelity, safety, and reuse.