4.5 KiB
Plan2Code Metrics
Part of AGENTS.md — project guidance for AI coding agents.
A recursive self-improvement toolchain for plan2code contributors. Collects run metrics, aggregates by prompt generation, diagnoses weak steps via AI, and proposes surgical prompt edits.
Metrics Data Flow
Collect → Aggregate → Analyze → Improve → Apply
- Collector reads project artifacts (
specs/<feature>/) → writesRunMetricsJSON per run - Aggregator groups runs by prompt SHA fingerprint (cohorts) →
aggregated.json - Analyzer invokes AI with aggregated metrics + prompt contents → diagnosis markdown
- Improver invokes AI with diagnosis → validated
PromptEdit[]proposals (char limit + verbatim checks) - Applier shows interactive diffs → patches
src/plan2code-*.mdfiles
Metrics Commands
Run plan2code-metrics (fully interactive, no flags). Build and test commands: see Development Commands.
Metrics CLI Menu
| Option | Action |
|---|---|
| Collect | Read spec artifacts → run JSON |
| Import | Copy run JSON from another project |
| View | Display cohort metrics with health indicators |
| Analyze | AI diagnosis of weak metrics |
| Propose | AI improvement proposals with validation |
| Apply | Interactive diff review + file patching |
| Fetch community submissions | List/parse/import open community-feedback GitHub issues from jparkerweb/plan2code, close on success |
| Delete metrics data | Delete selected runs, all runs + aggregated data, or everything including proposals |
Community submissions arrive as GitHub issues labeled community-feedback on jparkerweb/plan2code, created by the finalize prompt's post-Step-6 submission flow; the "Fetch community submissions" option requires an authenticated gh CLI to list/close them.
Key Source Files
| File | Purpose |
|---|---|
types.ts |
All interfaces (RunMetrics, UserFeedback, CohortMetrics, etc.) + METRIC_TARGETS |
collector.ts |
Reads project artifacts → run JSON (parses plan drafts, overview.md, loop logs) |
aggregator.ts |
Merges runs by prompt generation (SHA cohort) → aggregated.json |
community.ts |
Lists/parses/closes community-feedback-labeled GitHub issues via gh CLI |
analyzer.ts |
AI diagnosis via prompts/analyze.md template |
improver.ts |
AI proposals via prompts/improve.md + validation (char count, old_text match) |
applier.ts |
Interactive diff review + file patching |
cli.ts |
Menu-driven interactive CLI (100% prompts, no flags) |
invoke-llm.ts |
Unified LLM interface (Claude Code, GitHub Copilot CLI, or Devin CLI) |
*.test.ts |
vitest suites for aggregator, community, improver (npm test in plan2code-metrics/) |
User Feedback
The collector parses an optional ## User Feedback table from overview.md:
## User Feedback
| Field | Value |
|-------|-------|
| Rating | 8 |
| Reason | Smooth workflow |
| Went Well | Planning was thorough |
| Went Poorly | Some tasks unclear |
Feedback is collected during finalize (Step 5) or retroactively via the CLI. Pipe characters in values are escaped as \|. The aggregator computes avg_user_rating and feedback_count per cohort.
Supported AI Agents
- Claude Code (recommended):
claude --print --dangerously-skip-permissions, prompt written to a temp file and piped to stdin (execainputFile) - GitHub Copilot CLI:
copilot --allow-all-tools -s, prompt on stdin (default modelclaude-sonnet-4) - Devin CLI:
devinCLI with--print --prompt-file <file> --permission-mode dangerous
Metric Targets
| Metric | Target | Direction |
|---|---|---|
avg_confidence |
≥ 90 | higher is better |
avg_clarification_rounds |
≤ 2.0 | lower is better |
avg_verification_gaps_found |
≤ 2.0 | lower is better |
avg_parallel_groups |
≥ 0.5 | higher is better |
avg_verification_items_added |
≤ 1.5 | lower is better |
avg_task_completion_rate |
≥ 0.95 | higher is better |
avg_blocker_count |
≤ 1.5 | lower is better |
avg_verification_failures_found |
≤ 1.0 | lower is better |
archival_success_rate |
≥ 0.99 | higher is better |
avg_user_rating |
≥ 7.0 | higher is better |
On start the CLI asks for the plan2code repo path (default: saved plan2codeRepoPath in ~/.plan2code-metrics.json, else auto-detected up to 5 parent dirs) and the project path (default: cwd). Data is stored in <project>/.plan2code-metrics/ (runs/, aggregated.json, proposals/).