mirror of
https://github.com/jparkerweb/plan2code.git
synced 2026-07-21 18:33:22 -07:00
Compare commits
7 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 628e688ab9 | |||
| 157e55ef2e | |||
| 02fd4efa40 | |||
| 348c8ed7b2 | |||
| 5c2f70a37c | |||
| de46275b51 | |||
| b7b2595fe1 |
@@ -5,3 +5,11 @@ dist/
|
|||||||
plan2code-loop/dist
|
plan2code-loop/dist
|
||||||
plan2code-loop/node_modules
|
plan2code-loop/node_modules
|
||||||
plan2code-loop/package-lock.json
|
plan2code-loop/package-lock.json
|
||||||
|
plan2code-metrics/dist
|
||||||
|
plan2code-metrics/node_modules
|
||||||
|
plan2code-metrics/package-lock.json
|
||||||
|
.plan2code-loop
|
||||||
|
.plan2code-metrics
|
||||||
|
nul
|
||||||
|
node_modules/
|
||||||
|
package-lock.json
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
node scripts/validate-char-count.js
|
||||||
+19
@@ -0,0 +1,19 @@
|
|||||||
|
# Exclude user-generated spec files
|
||||||
|
specs/
|
||||||
|
specs--completed/
|
||||||
|
|
||||||
|
# Exclude development files
|
||||||
|
.husky/
|
||||||
|
.git/
|
||||||
|
.gitignore
|
||||||
|
CLAUDE.md
|
||||||
|
|
||||||
|
# Exclude generated files (installer generates dist/ dynamically)
|
||||||
|
dist/
|
||||||
|
|
||||||
|
# Exclude dependencies (installer will run npm install if needed)
|
||||||
|
node_modules/
|
||||||
|
package-lock.json
|
||||||
|
plan2code-loop/node_modules/
|
||||||
|
plan2code-loop/package-lock.json
|
||||||
|
plan2code-loop/dist/
|
||||||
@@ -18,12 +18,21 @@ plan2code/
|
|||||||
├── plan2code-loop/ # Autonomous loop CLI tool (Node.js/TypeScript)
|
├── plan2code-loop/ # Autonomous loop CLI tool (Node.js/TypeScript)
|
||||||
│ ├── src/ # TypeScript source
|
│ ├── src/ # TypeScript source
|
||||||
│ └── dist/ # Built output (tsup)
|
│ └── dist/ # Built output (tsup)
|
||||||
|
├── plan2code-metrics/ # Recursive self-improvement toolchain
|
||||||
|
│ ├── src/ # TypeScript source
|
||||||
|
│ │ └── prompts/ # Internal AI prompt templates (no char limit)
|
||||||
|
│ └── dist/ # Built output (tsup)
|
||||||
|
├── scripts/ # Development scripts
|
||||||
|
│ └── validate-char-count.js # Pre-commit character count validator
|
||||||
├── dist/ # Generated distribution files (auto-generated)
|
├── dist/ # Generated distribution files (auto-generated)
|
||||||
│ ├── global-commands/ # For global installation (~/.claude/, etc.)
|
│ ├── global-commands/ # For global installation (~/.claude/, etc.)
|
||||||
│ └── local-commands/ # For per-project installation (.claude/, etc.)
|
│ └── local-commands/ # For per-project installation (.claude/, etc.)
|
||||||
|
├── .husky/ # Git hooks (husky)
|
||||||
|
│ └── pre-commit # Runs character count validation
|
||||||
├── docs/ # Documentation and assets
|
├── docs/ # Documentation and assets
|
||||||
├── specs/ # Feature specs (if any in-progress)
|
├── specs/ # Feature specs (if any in-progress)
|
||||||
├── install.js # Interactive installer (Node.js)
|
├── install.js # Interactive installer (Node.js)
|
||||||
|
├── package.json # Root package (husky only, private: true)
|
||||||
├── version.json # Version metadata
|
├── version.json # Version metadata
|
||||||
└── README.md # User documentation
|
└── README.md # User documentation
|
||||||
```
|
```
|
||||||
@@ -34,6 +43,7 @@ plan2code/
|
|||||||
|------|---------|
|
|------|---------|
|
||||||
| `install.js` | Main installer - generates and installs workflow files to AI tool directories |
|
| `install.js` | Main installer - generates and installs workflow files to AI tool directories |
|
||||||
| `src/plan2code-*.md` | Source workflow prompts (the "source of truth") |
|
| `src/plan2code-*.md` | Source workflow prompts (the "source of truth") |
|
||||||
|
| `scripts/validate-char-count.js` | Pre-commit validator ensuring all source prompts ≤ 11,000 chars |
|
||||||
| `version.json` | Version metadata (name, version, description) |
|
| `version.json` | Version metadata (name, version, description) |
|
||||||
| `QUICK-REFERENCE.md` | User quick-reference card |
|
| `QUICK-REFERENCE.md` | User quick-reference card |
|
||||||
|
|
||||||
@@ -48,7 +58,7 @@ plan2code/
|
|||||||
| `plan2code-1b--revise-plan.md` | 1b | Mid-implementation revisions |
|
| `plan2code-1b--revise-plan.md` | 1b | Mid-implementation revisions |
|
||||||
| `plan2code-2--document.md` | 2 | Create implementation specs |
|
| `plan2code-2--document.md` | 2 | Create implementation specs |
|
||||||
| `plan2code-3--implement.md` | 3 | Execute implementation (phase by phase) |
|
| `plan2code-3--implement.md` | 3 | Execute implementation (phase by phase) |
|
||||||
| `plan2code-4--finalize.md` | 4 | Validate, summarize, archive |
|
| `plan2code-4--finalize.md` | 4 | Validate, summarize, feedback, archive (7 steps) |
|
||||||
|
|
||||||
## Plan2Code Loop (`plan2code-loop/`)
|
## Plan2Code Loop (`plan2code-loop/`)
|
||||||
|
|
||||||
@@ -88,44 +98,125 @@ The LLM must output one of these formats:
|
|||||||
- `PHASE_COMPLETE` - Current phase finished (phase mode only)
|
- `PHASE_COMPLETE` - Current phase finished (phase mode only)
|
||||||
- `LOOP_COMPLETE` - All phases finished
|
- `LOOP_COMPLETE` - All phases finished
|
||||||
|
|
||||||
|
## Plan2Code Metrics (`plan2code-metrics/`)
|
||||||
|
|
||||||
|
A recursive self-improvement toolchain for plan2code contributors. Collects run metrics, aggregates by prompt generation, diagnoses weak steps via AI, and proposes surgical prompt edits.
|
||||||
|
|
||||||
|
### Metrics Data Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
Collect → Aggregate → Analyze → Improve → Apply
|
||||||
|
```
|
||||||
|
|
||||||
|
1. **Collector** reads project artifacts (`specs/<feature>/`) → writes `RunMetrics` JSON per run
|
||||||
|
2. **Aggregator** groups runs by prompt SHA fingerprint (cohorts) → `aggregated.json`
|
||||||
|
3. **Analyzer** invokes AI with aggregated metrics + prompt contents → diagnosis markdown
|
||||||
|
4. **Improver** invokes AI with diagnosis → validated `PromptEdit[]` proposals (char limit + verbatim checks)
|
||||||
|
5. **Applier** shows interactive diffs → patches `src/plan2code-*.md` files
|
||||||
|
|
||||||
|
### Metrics Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd plan2code-metrics && npm run build # Build the CLI
|
||||||
|
plan2code-metrics # Run (fully interactive, no flags)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Metrics CLI Menu
|
||||||
|
|
||||||
|
| Option | Action |
|
||||||
|
|--------|--------|
|
||||||
|
| Collect | Read spec artifacts → run JSON |
|
||||||
|
| Import | Copy run JSON from another project |
|
||||||
|
| View | Display cohort metrics with health indicators |
|
||||||
|
| Analyze | AI diagnosis of weak metrics |
|
||||||
|
| Propose | AI improvement proposals with validation |
|
||||||
|
| Apply | Interactive diff review + file patching |
|
||||||
|
|
||||||
|
### Key Files
|
||||||
|
|
||||||
|
| File | Purpose |
|
||||||
|
|------|---------|
|
||||||
|
| `types.ts` | All interfaces (`RunMetrics`, `UserFeedback`, `CohortMetrics`, etc.) + `METRIC_TARGETS` |
|
||||||
|
| `collector.ts` | Reads project artifacts → run JSON (parses plan drafts, overview.md, loop logs) |
|
||||||
|
| `aggregator.ts` | Merges runs by prompt generation (SHA cohort) → `aggregated.json` |
|
||||||
|
| `analyzer.ts` | AI diagnosis via `prompts/analyze.md` template |
|
||||||
|
| `improver.ts` | AI proposals via `prompts/improve.md` + validation (char count, old_text match) |
|
||||||
|
| `applier.ts` | Interactive diff review + file patching |
|
||||||
|
| `cli.ts` | Menu-driven interactive CLI (100% prompts, no flags) |
|
||||||
|
| `invoke-llm.ts` | Unified LLM interface (Claude Code or Copilot CLI) |
|
||||||
|
|
||||||
|
### User Feedback
|
||||||
|
|
||||||
|
The collector parses an optional `## User Feedback` table from `overview.md`:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
## User Feedback
|
||||||
|
| Field | Value |
|
||||||
|
|-------|-------|
|
||||||
|
| Rating | 8 |
|
||||||
|
| Reason | Smooth workflow |
|
||||||
|
| Went Well | Planning was thorough |
|
||||||
|
| Went Poorly | Some tasks unclear |
|
||||||
|
```
|
||||||
|
|
||||||
|
Feedback is collected during finalize (Step 5) or retroactively via the CLI. Pipe characters in values are escaped as `\|`. The aggregator computes `avg_user_rating` and `feedback_count` per cohort.
|
||||||
|
|
||||||
|
### Supported AI Agents
|
||||||
|
|
||||||
|
- **Claude Code** (recommended): `claude` CLI with `--inputFile` for prompt delivery
|
||||||
|
- **GitHub Copilot CLI**: `copilot` CLI with stdin prompt delivery
|
||||||
|
|
||||||
|
### Metric Targets
|
||||||
|
|
||||||
|
| Metric | Target | Direction |
|
||||||
|
|--------|--------|-----------|
|
||||||
|
| `avg_confidence` | ≥ 90 | higher is better |
|
||||||
|
| `avg_task_completion_rate` | ≥ 0.95 | higher is better |
|
||||||
|
| `avg_blocker_count` | ≤ 1.5 | lower is better |
|
||||||
|
| `avg_completion_marker_success_rate` | ≥ 0.95 | higher is better |
|
||||||
|
| `avg_verification_failures_found` | ≤ 1.0 | lower is better |
|
||||||
|
| `archival_success_rate` | ≥ 0.99 | higher is better |
|
||||||
|
| `avg_user_rating` | ≥ 7.0 | higher is better |
|
||||||
|
|
||||||
|
Data stored in `.plan2code-metrics/` (runs/, aggregated.json, proposals/).
|
||||||
|
|
||||||
## Development Commands
|
## Development Commands
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Run the interactive installer
|
# Install dev dependencies (sets up husky pre-commit hooks)
|
||||||
|
npm install
|
||||||
|
|
||||||
|
# Run the interactive installer (always interactive — any CLI args are silently ignored)
|
||||||
node install.js
|
node install.js
|
||||||
|
|
||||||
# Install to specific platform only
|
|
||||||
node install.js --platform claude
|
|
||||||
node install.js --platform cursor
|
|
||||||
node install.js --platform copilot
|
|
||||||
node install.js --platform continue
|
|
||||||
node install.js --platform windsurf
|
|
||||||
node install.js --platform codeium
|
|
||||||
node install.js --platform vscode-copilot
|
|
||||||
|
|
||||||
# Preview changes without installing
|
|
||||||
node install.js --dry-run
|
|
||||||
|
|
||||||
# Show local (per-project) installation instructions
|
|
||||||
node install.js --local
|
|
||||||
|
|
||||||
# Uninstall all Plan2Code files
|
|
||||||
node install.js --uninstall
|
|
||||||
|
|
||||||
# Plan2Code Loop
|
# Plan2Code Loop
|
||||||
cd plan2code-loop && npm install # First time setup
|
cd plan2code-loop && npm install # First time setup
|
||||||
cd plan2code-loop && npm run build # Build the CLI
|
cd plan2code-loop && npm run build # Build the CLI
|
||||||
|
|
||||||
|
# Plan2Code Metrics
|
||||||
|
cd plan2code-metrics && npm install # First time setup
|
||||||
|
cd plan2code-metrics && npm run build # Build the CLI
|
||||||
```
|
```
|
||||||
|
|
||||||
### Installer Menu Options
|
### Installer Menu Options
|
||||||
|
|
||||||
|
**Main menu:**
|
||||||
|
|
||||||
|
| Option | Action |
|
||||||
|
|--------|--------|
|
||||||
|
| `I` | Install Plan2Code for all platforms + loop CLI |
|
||||||
|
| `U` | Uninstall Plan2Code files + loop CLI (confirmation required) |
|
||||||
|
| `C` | Open CUSTOM sub-menu |
|
||||||
|
| `Q` | Quit |
|
||||||
|
|
||||||
|
**CUSTOM sub-menu (`C`):**
|
||||||
|
|
||||||
| Option | Action |
|
| Option | Action |
|
||||||
|--------|--------|
|
|--------|--------|
|
||||||
| `A` | Install to ALL platforms + build & link loop CLI |
|
|
||||||
| `O` | Build & link plan2code-loop CLI only |
|
|
||||||
| `U` | Uninstall prompts from all platforms + unlink loop CLI |
|
|
||||||
| `L` | Show local (per-project) install instructions |
|
| `L` | Show local (per-project) install instructions |
|
||||||
| `1-7` | Install to specific platform |
|
| `O` | Install plan2code-loop CLI only |
|
||||||
|
| `M` | Install plan2code-metrics CLI only |
|
||||||
|
| `Q` | Return to main menu |
|
||||||
|
|
||||||
## How the Installer Works
|
## How the Installer Works
|
||||||
|
|
||||||
@@ -136,15 +227,19 @@ cd plan2code-loop && npm run build # Build the CLI
|
|||||||
|
|
||||||
### Platform-Specific File Formats
|
### Platform-Specific File Formats
|
||||||
|
|
||||||
| Platform | Extension | Header |
|
| Platform | Extension / File | Local Dir | Global Dir | Header |
|
||||||
|----------|-----------|--------|
|
|----------|-----------------|-----------|------------|--------|
|
||||||
| Claude Code | `.md` | None |
|
| Claude Code | `.md` | — | — | None |
|
||||||
| Cursor | `.md` | None |
|
| Cursor | `.md` | — | — | None |
|
||||||
| Copilot CLI | `.md` | YAML frontmatter |
|
| Copilot CLI | `.md` | — | — | YAML frontmatter |
|
||||||
| Continue | `.prompt.md` | YAML frontmatter |
|
| Continue | `.prompt.md` | — | — | YAML frontmatter |
|
||||||
| Windsurf | `.md` | YAML frontmatter |
|
| Windsurf | `.md` | — | — | YAML frontmatter |
|
||||||
| VS Code Copilot | `.prompt.md` | YAML frontmatter |
|
| VS Code Copilot | `.prompt.md` | — | — | YAML frontmatter |
|
||||||
| Codeium | `.md` | YAML frontmatter |
|
| Codeium | `.md` | — | — | YAML frontmatter |
|
||||||
|
| Claude Code (Skills) | `SKILL.md` in subdir | `.claude/skills/<skill-name>/` | `~/.claude/skills/<skill-name>/` | YAML frontmatter + `disable-model-invocation: true` |
|
||||||
|
| Agent Skills (Amp · Gemini CLI · OpenCode) | `SKILL.md` in subdir | `.agents/skills/<skill-name>/` | `~/.agents/skills/<skill-name>/` | YAML frontmatter (no disable flag) |
|
||||||
|
| Crush | `SKILL.md` in subdir | — (global only) | `~/.config/crush/skills/<skill-name>/` (Unix) / `%LOCALAPPDATA%\crush\skills\<skill-name>\` (Windows) | YAML frontmatter |
|
||||||
|
| Gemini CLI (TOML) | `.toml` | `.gemini/commands/` | `~/.gemini/commands/` | None (TOML fields: `description`, `prompt`) |
|
||||||
|
|
||||||
## Naming Convention
|
## Naming Convention
|
||||||
|
|
||||||
@@ -168,15 +263,19 @@ When modifying workflow prompts in `src/`:
|
|||||||
|
|
||||||
## Code Style
|
## Code Style
|
||||||
|
|
||||||
- **JavaScript:** CommonJS modules (Node.js built-ins only - no external dependencies)
|
- **install.js:** CommonJS, Node.js built-ins only (no external deps), ANSI colors via `COLORS` constant, readline-based prompts
|
||||||
- **Console output:** Uses ANSI color codes via the `COLORS` constant
|
- **plan2code-loop & plan2code-metrics:** TypeScript + ESM, built with tsup (target ES2022, moduleResolution: bundler)
|
||||||
- **User interaction:** Readline-based interactive prompts
|
- External deps: `@inquirer/prompts`, `chalk`, `execa`, `ora`
|
||||||
- **File operations:** Synchronous fs operations for simplicity
|
- Interactive CLI via `@inquirer/prompts` (select, input, confirm)
|
||||||
|
- **File operations:** Synchronous fs in all packages
|
||||||
|
|
||||||
## Gotchas/Pitfalls
|
## Gotchas/Pitfalls
|
||||||
|
|
||||||
- **Version sync:** When adding a new version to `CHANGELOG.md`, also update `version.json` to match. The installer displays the version from `version.json` in its header.
|
- **Version sync:** When adding a new version to `CHANGELOG.md`, also update `version.json` and `package.json` to match. The installer displays the version from `version.json` in its header.
|
||||||
- **Loop `.gitignore` setup:** `ensureGitignore()` runs at startup in `Controller.run()` as a pre-flight step, not just inside `createTaskCommit()`. This is critical for phase mode where the Node controller doesn't handle commits — without it, `git add -A` would stage spec files.
|
- **Loop `.gitignore` setup:** `ensureGitignore()` runs at startup in `Controller.run()` as a pre-flight step, not just inside `createTaskCommit()`. This is critical for phase mode where the Node controller doesn't handle commits — without it, `git add -A` would stage spec files.
|
||||||
|
- **Workflow file character limit:** All `src/plan2code-*.md` files must be ≤ 11,000 characters. A husky pre-commit hook enforces this. The 11,000 limit leaves buffer for platform-specific YAML headers (106-142 chars) to stay under Windsurf's 12,000 char limit.
|
||||||
|
- **Metrics internal prompts have no char limit:** Files in `plan2code-metrics/src/prompts/` are NOT subject to the 11,000 char limit — only `src/plan2code-*.md` consumer-facing prompts are.
|
||||||
|
- **User Feedback table format:** The `## User Feedback` markdown table in `overview.md` has a strict format the collector regex depends on. Field names must be exactly `Rating`, `Reason`, `Went Well`, `Went Poorly`. Pipe characters in values must be escaped as `\|`.
|
||||||
|
|
||||||
## Git Commit Messages
|
## Git Commit Messages
|
||||||
|
|
||||||
@@ -185,11 +284,11 @@ When modifying workflow prompts in `src/`:
|
|||||||
- Loop task mode: `createTaskCommit()` in `plan2code-loop/src/utils/git.ts` appends the footer automatically
|
- Loop task mode: `createTaskCommit()` in `plan2code-loop/src/utils/git.ts` appends the footer automatically
|
||||||
- Loop phase mode: Prompt template instructs the LLM to add `-m "AI Assisted"` as the final flag
|
- Loop phase mode: Prompt template instructs the LLM to add `-m "AI Assisted"` as the final flag
|
||||||
- Implement mode: User-facing commit suggestions in `src/plan2code-3--implement.md` include the footer
|
- Implement mode: User-facing commit suggestions in `src/plan2code-3--implement.md` include the footer
|
||||||
- **Init workflow:** `src/smarsh2code---init.md` generates AGENTS.md files with a Git Commit Messages section that includes this convention by default
|
- **Init workflow:** `src/plan2code---init.md` generates AGENTS.md files with a Git Commit Messages section that includes this convention by default
|
||||||
|
|
||||||
## Mascot
|
## Mascot
|
||||||
|
|
||||||
The project has a mascot called "Smarshy" - an ASCII art robot that appears in installer output and workflow prompts. Mascot variants are defined in `MASCOT` constant in `install.js` and appear in workflow markdown files.
|
The project has a mascot called "Planny" - an ASCII art robot that appears in installer output and workflow prompts. Mascot variants are defined in `MASCOT` constant in `install.js` and appear in workflow markdown files.
|
||||||
|
|
||||||
```
|
```
|
||||||
╭───╮
|
╭───╮
|
||||||
|
|||||||
+107
@@ -2,6 +2,113 @@
|
|||||||
|
|
||||||
All notable changes to Plan2Code will be documented in this file.
|
All notable changes to Plan2Code will be documented in this file.
|
||||||
|
|
||||||
|
## v1.8.1
|
||||||
|
|
||||||
|
### ✨ Added
|
||||||
|
|
||||||
|
- **User feedback collection** — Optional 1-10 rating with reason, what went well, and what went poorly
|
||||||
|
- Finalize prompt (Step 5) asks for optional feedback before archival, writes structured table to `overview.md`
|
||||||
|
- Collector parses `## User Feedback` table from `overview.md` into `RunMetrics.user_feedback`
|
||||||
|
- Aggregator computes `avg_user_rating` and `feedback_count` per cohort
|
||||||
|
- CLI offers interactive feedback collection if none found during metrics collection
|
||||||
|
- Analysis and improvement prompts reference `avg_user_rating` metric target (≥ 7.0)
|
||||||
|
- `UserFeedback` type exported from public API
|
||||||
|
- **Pipe-safe feedback parsing** — User text containing `|` characters is escaped on write and correctly unescaped on parse using negative lookbehind regex
|
||||||
|
|
||||||
|
### 🐛 Fixed
|
||||||
|
|
||||||
|
- **Duplicate run files** — Interactive feedback no longer creates a second run JSON; the original is deleted before re-collecting
|
||||||
|
- **Finalize step ordering** — Feedback collection moved to Step 5 (before archival at Step 6), ensuring `overview.md` is written while still in the active spec directory
|
||||||
|
|
||||||
|
## v1.8.0
|
||||||
|
|
||||||
|
### ✨ Added
|
||||||
|
|
||||||
|
- **plan2code-metrics** — New recursive self-improvement toolchain for plan2code contributors (`plan2code-metrics/`)
|
||||||
|
- Fully interactive menu-driven CLI — no flags, all inputs collected via prompts
|
||||||
|
- **Collect** metrics from completed project specs (plan, document, implement, finalize steps)
|
||||||
|
- **Import** run data from other projects for cross-project aggregation
|
||||||
|
- **View** metrics status with health indicators and generation-over-generation deltas
|
||||||
|
- **Analyze** weak steps via AI-powered diagnosis (Claude Code or GitHub Copilot CLI)
|
||||||
|
- **Generate** surgical improvement proposals with automatic validation (char count limits, edit verification)
|
||||||
|
- **Review and apply** proposals with interactive diff review
|
||||||
|
- Cohort-based aggregation groups runs by prompt generation (SHA fingerprint of prompt files)
|
||||||
|
- Supports both Claude Code and GitHub Copilot CLI as AI backends
|
||||||
|
- Standalone TypeScript package with tsup build (ESM), installed via `npm link`
|
||||||
|
|
||||||
|
### 🔧 Changed
|
||||||
|
|
||||||
|
- **plan2code-4--finalize.md** — Added "Metrics Capture (Contributors)" note in Step 6 directing contributors to run `plan2code-metrics` after finalization
|
||||||
|
- **plan2code-loop index.ts** — Added dim hint "run plan2code-metrics" after session summary
|
||||||
|
|
||||||
|
## v1.7.0
|
||||||
|
|
||||||
|
### ✨ Added
|
||||||
|
|
||||||
|
- **4 new platform targets** — Gemini CLI, Crush, Amp, and OpenCode now supported
|
||||||
|
- Gemini CLI installs as TOML commands (`.gemini/commands/plan2code-*.toml`)
|
||||||
|
- Crush installs as skill subdirs (`~/.config/crush/skills/` on Unix, `%LOCALAPPDATA%\crush\skills\` on Windows)
|
||||||
|
- Amp and OpenCode covered via shared Agent Skills target (`.agents/skills/`)
|
||||||
|
- **Claude Code Skills format** — Migrated from flat `.claude/commands/*.md` to `.claude/skills/<skill-name>/SKILL.md` with `disable-model-invocation: true` frontmatter
|
||||||
|
- **Agent Skills cross-tool target** — Single `.agents/skills/` install covers Amp, Gemini CLI, and OpenCode simultaneously
|
||||||
|
- **Legacy cleanup** — Old `.claude/commands/plan2code-*.md` files automatically removed on install and uninstall
|
||||||
|
- **TOML generation** — New `generateTomlContent()` produces Gemini CLI command files using TOML literal multi-line strings
|
||||||
|
|
||||||
|
### 🔧 Changed
|
||||||
|
|
||||||
|
- `AGENTS.md` Platform-Specific File Formats table expanded to 5 columns with 4 new platform rows
|
||||||
|
- `docs/index.html` hero section updated with 4 new platform pills
|
||||||
|
- `README.md` Supported Platforms list updated with 4 new platforms
|
||||||
|
|
||||||
|
## v1.6.2
|
||||||
|
|
||||||
|
### 🔧 Changed
|
||||||
|
|
||||||
|
- **Installer menu simplified** — Replaced the 7-platform picker with a clean 4-option menu (I/U/C/Q)
|
||||||
|
- `I` — Install Plan2Code for all platforms + loop CLI
|
||||||
|
- `U` — Uninstall (with confirmation prompt)
|
||||||
|
- `C` — CUSTOM sub-menu: `L` (local install instructions), `O` (loop CLI only), `Q` (back)
|
||||||
|
- `Q` — Quit
|
||||||
|
- Any CLI arguments (e.g. `--dry-run`, `--platform`) are now silently ignored; installer always runs interactively
|
||||||
|
- **README installation section** — npx install method promoted to primary recommended install path; updated menu example
|
||||||
|
|
||||||
|
### 🗑️ Removed
|
||||||
|
|
||||||
|
- CLI flags `--dry-run`, `--platform`, `--local`, `--uninstall`, `--help`, `--loop`, `--uninstall-loop` (all removed; installer is always interactive)
|
||||||
|
- `displayHelp()` function removed from `install.js`
|
||||||
|
|
||||||
|
## v1.6.1
|
||||||
|
|
||||||
|
### ✨ Added
|
||||||
|
|
||||||
|
- **NPX installation support** - Team members can now install directly from GitHub without cloning
|
||||||
|
- Added `name`, `version`, and `bin` fields to `package.json` for npm compatibility
|
||||||
|
- Installation via `npx git+ssh://git@github.com/jparkerweb/plan2code.git` (SSH)
|
||||||
|
- Installation via `npx git+https://github.com/jparkerweb/plan2code.git` (HTTPS)
|
||||||
|
- Installer runs from temporary location and cleans up automatically
|
||||||
|
- Updated README.md with Quick Start section showing both authentication methods
|
||||||
|
- Added `.npmignore` file to suppress npm warnings during npx execution
|
||||||
|
|
||||||
|
## v1.6.0
|
||||||
|
|
||||||
|
### 🏎️ Improved
|
||||||
|
|
||||||
|
- **Reduced workflow file sizes** - All 4 over-target source prompts compressed to ≤ 11,000 characters for Windsurf IDE compatibility (12,000 char limit minus header buffer)
|
||||||
|
- `plan2code-3--implement.md` - 14,806 → 8,533 chars (42% reduction)
|
||||||
|
- `plan2code-1--plan.md` - 13,587 → 10,348 chars (24% reduction)
|
||||||
|
- `plan2code-4--finalize.md` - 12,012 → 8,586 chars (29% reduction)
|
||||||
|
- `plan2code-2--document.md` - 11,842 → 9,031 chars (24% reduction)
|
||||||
|
- All functional workflow behavior preserved
|
||||||
|
- Compression techniques: template-to-section-list specs, removed bad examples, consolidated redundant sections, simplified decorative boxes, imperative directives
|
||||||
|
|
||||||
|
### 🧪 Testing
|
||||||
|
|
||||||
|
- **Pre-commit character count validation** - Husky pre-commit hook prevents workflow files from exceeding 11,000 characters
|
||||||
|
- `scripts/validate-char-count.js` - Cross-platform Node.js validation script using only built-in modules
|
||||||
|
- `.husky/pre-commit` - Git hook trigger calling the validation script
|
||||||
|
- Root `package.json` with husky as sole devDependency (`private: true`)
|
||||||
|
- `.gitignore` updated with `node_modules/` and `package-lock.json`
|
||||||
|
|
||||||
## v1.5.4
|
## v1.5.4
|
||||||
|
|
||||||
### ✨ Added
|
### ✨ Added
|
||||||
|
|||||||
+5
-5
@@ -2,8 +2,8 @@
|
|||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
| Step | Command | Input | Output |
|
| Step | Command | Input | Output |
|
||||||
| ------ | ----------------------------- | --------------- | ----------------------------------- |
|
| ------ | ------------------------------- | --------------- | ----------------------------------- |
|
||||||
| Init | /plan2code---init | None | AGENTS.md file |
|
| Init | /plan2code---init | None | AGENTS.md file |
|
||||||
| Update | /plan2code---init-update | AGENTS.md | Updated AGENTS.md |
|
| Update | /plan2code---init-update | AGENTS.md | Updated AGENTS.md |
|
||||||
| 0 | /plan2code---quick-task | Requirements | Conversational plan |
|
| 0 | /plan2code---quick-task | Requirements | Conversational plan |
|
||||||
@@ -60,10 +60,10 @@ When phases have no file conflicts or dependencies, they can run simultaneously:
|
|||||||
| ---------------------- | ------------------------------------------------- |
|
| ---------------------- | ------------------------------------------------- |
|
||||||
| Lost context mid-phase | Attach spec files, say "resume from Task X.Y" |
|
| Lost context mid-phase | Attach spec files, say "resume from Task X.Y" |
|
||||||
| Wrong phase started | Say "abort", start correct phase |
|
| Wrong phase started | Say "abort", start correct phase |
|
||||||
| Need to change plan | Use `/plan2code-1b--revise-plan` |
|
| Need to change plan | Use `/plan2code-1b--revise-plan` |
|
||||||
| Multiple spec folders | Specify which: "Continue with specs/user-auth/" |
|
| Multiple spec folders | Specify which: "Continue with specs/user-auth/" |
|
||||||
| Need AGENTS.md file | Use `/plan2code---init` to generate one |
|
| Need AGENTS.md file | Use `/plan2code---init` to generate one |
|
||||||
| Update AGENTS.md | Use `/plan2code---init-update` after sessions |
|
| Update AGENTS.md | Use `/plan2code---init-update` after sessions |
|
||||||
| Run phases in parallel | Check Parallel Execution Groups in overview.md |
|
| Run phases in parallel | Check Parallel Execution Groups in overview.md |
|
||||||
|
|
||||||
## Workflow Decision
|
## Workflow Decision
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
A structured 4-step workflow for developing features and projects with AI assistance. This methodology emphasizes thorough planning before implementation, ensuring well-documented, maintainable code.
|
A structured 4-step workflow for developing features and projects with AI assistance. This methodology emphasizes thorough planning before implementation, ensuring well-documented, maintainable code.
|
||||||
|
|
||||||
<img src="docs/plan2code.jpg" alt="Plan2Code Workflow" height="400">
|
<img src="docs/desk.jpg" alt="Plan2Code Workflow" height="275">
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
@@ -15,16 +15,16 @@ A structured 4-step workflow for developing features and projects with AI assist
|
|||||||
New Chat New Chat New Chat (per phase) New Chat
|
New Chat New Chat New Chat (per phase) New Chat
|
||||||
```
|
```
|
||||||
|
|
||||||
| Command | Use When |
|
| Command | When to Use |
|
||||||
|---------|----------|
|
|----------------------------------|----------------------------------------------------------|
|
||||||
| `/plan2code---init` | Generate AGENTS.md file for new/existing projects |
|
| `/plan2code---init` | Generate AGENTS.md file for new/existing projects |
|
||||||
| `/plan2code---init-update` | Update AGENTS.md with new learnings from coding sessions |
|
| `/plan2code---init-update` | Update AGENTS.md with new learnings from coding sessions |
|
||||||
| `/plan2code---quick-task` | Small, quick tasks that don't need full workflow |
|
| `/plan2code---quick-task` | Small, quick tasks that don't need full workflow |
|
||||||
| `/plan2code-1--plan` | Starting a new feature (full planning) |
|
| `/plan2code-1--plan` | Starting a new feature (full planning) |
|
||||||
| `/plan2code-1b--revise-plan` | Requirements change mid-implementation |
|
| `/plan2code-1b--revise-plan` | Requirements change mid-implementation |
|
||||||
| `/plan2code-2--document` | After planning, create implementation specs |
|
| `/plan2code-2--document` | After planning, create implementation specs |
|
||||||
| `/plan2code-3--implement` | Execute implementation (one phase per conversation) |
|
| `/plan2code-3--implement` | Execute implementation (one phase per conversation) |
|
||||||
| `/plan2code-4--finalize` | All phases complete, ready to archive |
|
| `/plan2code-4--finalize` | All phases complete, ready to archive |
|
||||||
|
|
||||||
**Key Rules:**
|
**Key Rules:**
|
||||||
- Start NEW conversation for each step (and each implementation phase)
|
- Start NEW conversation for each step (and each implementation phase)
|
||||||
@@ -40,13 +40,13 @@ See [QUICK-REFERENCE.md](QUICK-REFERENCE.md) for full reference card.
|
|||||||
|
|
||||||
Plan2Code includes an interactive installer that generates and installs workflow files for all major AI coding assistants.
|
Plan2Code includes an interactive installer that generates and installs workflow files for all major AI coding assistants.
|
||||||
|
|
||||||
<img src="docs/install-script.jpg" width="582">
|
<img src="docs/install-script.jpg" width="600">
|
||||||
|
|
||||||
### Prerequisites
|
### Prerequisites
|
||||||
|
|
||||||
The install script requires **Node.js** (v14 or later). If you don't have Node.js installed:
|
The install script requires **Node.js** (v14 or later). If you don't have Node.js installed:
|
||||||
|
|
||||||
1. Download from [nodejs.org](https://nodejs.org/)
|
1. Download from [nodejs.org](https://nodejs.org/) (recommended)
|
||||||
2. Or use a package manager:
|
2. Or use a package manager:
|
||||||
- **macOS:** `brew install node`
|
- **macOS:** `brew install node`
|
||||||
- **Windows:** `winget install OpenJS.NodeJS` or `choco install nodejs`
|
- **Windows:** `winget install OpenJS.NodeJS` or `choco install nodejs`
|
||||||
@@ -61,42 +61,74 @@ The install script requires **Node.js** (v14 or later). If you don't have Node.j
|
|||||||
- Codeium (IntelliJ)
|
- Codeium (IntelliJ)
|
||||||
- GitHub Copilot CLI
|
- GitHub Copilot CLI
|
||||||
- VS Code GitHub Copilot
|
- VS Code GitHub Copilot
|
||||||
|
- Gemini CLI
|
||||||
|
- Crush
|
||||||
|
- Amp
|
||||||
|
- OpenCode
|
||||||
|
|
||||||
### Quick Start
|
### Install via npx (Recommended — No Clone Required)
|
||||||
|
|
||||||
|
Run the interactive installer directly using `npx` with your preferred GitHub authentication method:
|
||||||
|
|
||||||
|
**If you use SSH keys:**
|
||||||
|
```bash
|
||||||
|
npx git+ssh://git@github.com/jparkerweb/plan2code.git
|
||||||
|
```
|
||||||
|
|
||||||
|
**If you use HTTPS authentication:**
|
||||||
|
```bash
|
||||||
|
npx git+https://github.com/jparkerweb/plan2code.git
|
||||||
|
```
|
||||||
|
|
||||||
|
This downloads the installer to a temporary location, runs it, installs the workflow files to your machine, and cleans up automatically. The installed workflows remain on your system and work independently. To update or reinstall, simply run the command again.
|
||||||
|
|
||||||
|
### Standard Installation (Clone Method)
|
||||||
|
|
||||||
|
#### No Clone Required
|
||||||
|
|
||||||
|
Run the interactive installer directly using `npx` with your preferred GitHub authentication method:
|
||||||
|
|
||||||
|
**If you use SSH keys:**
|
||||||
|
```bash
|
||||||
|
npx git+ssh://git@github.com/jparkerweb/plan2code.git
|
||||||
|
```
|
||||||
|
|
||||||
|
**If you use HTTPS authentication:**
|
||||||
|
```bash
|
||||||
|
npx git+https://github.com/jparkerweb/plan2code.git
|
||||||
|
```
|
||||||
|
|
||||||
|
This downloads the installer to a temporary location, runs it, installs the workflow files to your machine, and cleans up automatically. The installed workflows remain on your system and work independently. To update or reinstall, simply run the command again.
|
||||||
|
|
||||||
|
#### Standard Installation (Clone Method)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Clone the repository
|
# Clone the repository
|
||||||
git clone https://github.com/plan/plan2code.git
|
git clone https://github.com/jparkerweb/plan2code.git
|
||||||
cd plan2code
|
cd plan2code
|
||||||
|
|
||||||
# Run the interactive installer
|
# Run the interactive installer
|
||||||
node install.js
|
node install.js
|
||||||
|
|
||||||
|
# OPTIONAL: Install dev dependencies (ONLY if you plan to modify/contribute to Plan2Code)
|
||||||
|
npm install
|
||||||
```
|
```
|
||||||
|
|
||||||
The installer will display an interactive menu:
|
The installer displays an interactive menu:
|
||||||
|
|
||||||
```
|
```
|
||||||
Available platforms:
|
╔════════════════════════════════════════════════════════════════╗
|
||||||
|
║ INSTALL PLAN2CODE ║
|
||||||
|
╠════════════════════════════════════════════════════════════════╣
|
||||||
|
║ I. INSTALL Install Plan2Code for all platforms ║
|
||||||
|
║ U. UNINSTALL Remove Plan2Code files ║
|
||||||
|
║ C. CUSTOM Advanced options ║
|
||||||
|
║ Q. QUIT Exit ║
|
||||||
|
╚════════════════════════════════════════════════════════════════╝
|
||||||
|
|
||||||
1. Claude Code (~/.claude/commands/)
|
SELECT OPTION (I, U, C, Q) [I]:
|
||||||
2. Copilot CLI (~/.copilot/agents/)
|
|
||||||
3. Cursor (~/.cursor/commands/)
|
|
||||||
4. Continue (~/.continue/prompts/)
|
|
||||||
5. Windsurf (~/.codeium/windsurf/global_workflows/)
|
|
||||||
6. Codeium (IJ) (~/.codeium/global_workflows/)
|
|
||||||
7. VS Code Copilot (%APPDATA%\Code\User\prompts\)
|
|
||||||
|
|
||||||
A. Install ALL platforms + loop CLI
|
|
||||||
O. Build/link plan2code-loop CLI only
|
|
||||||
L. Show local (project) install instructions
|
|
||||||
U. Uninstall Plan2Code files + unlink loop CLI
|
|
||||||
Q. Quit
|
|
||||||
|
|
||||||
Enter choice (1-7, A, O, L, U, Q, or comma-separated like 1,3,5):
|
|
||||||
```
|
```
|
||||||
|
|
||||||
For per-project installation or additional options, run `node install.js --help`.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Important: Start Fresh Conversations
|
## Important: Start Fresh Conversations
|
||||||
@@ -219,10 +251,10 @@ The `overview.md` includes a "Parallel Execution Groups" section that identifies
|
|||||||
After running `node install.js`, use the slash commands directly in your AI tool:
|
After running `node install.js`, use the slash commands directly in your AI tool:
|
||||||
|
|
||||||
```
|
```
|
||||||
/smarsh2code-1--plan # Start planning a new feature
|
/plan2code-1--plan # Start planning a new feature
|
||||||
/smarsh2code-2--document # Create implementation docs from plan
|
/plan2code-2--document # Create implementation docs from plan
|
||||||
/smarsh2code-3--implement # Begin/continue implementation
|
/plan2code-3--implement # Begin/continue implementation
|
||||||
/smarsh2code-4--finalize # Wrap up after all phases complete
|
/plan2code-4--finalize # Wrap up after all phases complete
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -384,10 +416,10 @@ For hands-off implementation, Plan2Code includes an optional autonomous loop CLI
|
|||||||
# From the plan2code root directory:
|
# From the plan2code root directory:
|
||||||
|
|
||||||
# Option 1: Install everything (recommended)
|
# Option 1: Install everything (recommended)
|
||||||
node install.js # Select option A
|
node install.js # Select I at the menu
|
||||||
|
|
||||||
# Option 2: Install loop only
|
# Option 2: Install loop only
|
||||||
node install.js # Select option O
|
node install.js # Select C, then O at the menu
|
||||||
```
|
```
|
||||||
|
|
||||||
### Using the Loop
|
### Using the Loop
|
||||||
|
|||||||
+678
-367
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,14 @@
|
|||||||
|
{
|
||||||
|
"name": "plan2code",
|
||||||
|
"version": "1.8.1",
|
||||||
|
"private": true,
|
||||||
|
"bin": {
|
||||||
|
"plan2code": "./install.js"
|
||||||
|
},
|
||||||
|
"scripts": {
|
||||||
|
"prepare": "husky"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"husky": "^9.0.0"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -34,7 +34,7 @@ The CLI will:
|
|||||||
1. Auto-detect specs in `./specs/` directory
|
1. Auto-detect specs in `./specs/` directory
|
||||||
2. Let you select a spec if multiple are found
|
2. Let you select a spec if multiple are found
|
||||||
3. Prompt to continue if an existing session is found
|
3. Prompt to continue if an existing session is found
|
||||||
4. Ask for JIRA ticket ID, agent selection, and max iterations
|
4. Ask for JIRA ticket ID, agent selection, loop mode, and max iterations
|
||||||
|
|
||||||
## How It Works
|
## How It Works
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "plan2code-loop",
|
"name": "plan2code-loop",
|
||||||
"version": "1.0.0",
|
"version": "1.6.2",
|
||||||
"description": "Plan2Code Loop - Autonomous spec-driven implementation CLI",
|
"description": "Plan2Code Loop - Autonomous spec-driven implementation CLI",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "dist/index.js",
|
"main": "dist/index.js",
|
||||||
@@ -43,3 +43,4 @@
|
|||||||
"typescript": "^5.9.3"
|
"typescript": "^5.9.3"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -23,11 +23,11 @@ async function selectSpec(cwd: string = process.cwd()): Promise<string | null> {
|
|||||||
logger.info('');
|
logger.info('');
|
||||||
logger.info('To get started:');
|
logger.info('To get started:');
|
||||||
logger.info('');
|
logger.info('');
|
||||||
logger.info('1. Create a spec using `smarsh2code-1--plan`');
|
logger.info('1. Create a spec using `plan2code-1--plan`');
|
||||||
logger.info(' command in our AI Agent');
|
logger.info(' command in our AI Agent');
|
||||||
logger.info('');
|
logger.info('');
|
||||||
logger.info('2. Come back here and run `smarsh2code-loop`');
|
logger.info('2. Come back here and run `plan2code-loop`');
|
||||||
logger.info(' as an alternative to `smarsh2code-3--implement`');
|
logger.info(' as an alternative to `plan2code-3--implement`');
|
||||||
logger.info('');
|
logger.info('');
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
@@ -67,8 +67,6 @@ async function selectSpec(cwd: string = process.cwd()): Promise<string | null> {
|
|||||||
async function selectAgent(): Promise<string> {
|
async function selectAgent(): Promise<string> {
|
||||||
const allAgents = agentRegistry.getAll();
|
const allAgents = agentRegistry.getAll();
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
const agentName = await select({
|
const agentName = await select({
|
||||||
message: 'Select AI agent:',
|
message: 'Select AI agent:',
|
||||||
choices: allAgents.map((agent) => ({
|
choices: allAgents.map((agent) => ({
|
||||||
@@ -108,6 +106,7 @@ async function selectMaxIterations(): Promise<number> {
|
|||||||
|
|
||||||
return max;
|
return max;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Select loop mode: one task per loop or one phase per loop
|
* Select loop mode: one task per loop or one phase per loop
|
||||||
*/
|
*/
|
||||||
@@ -126,6 +125,7 @@ async function selectLoopMode(): Promise<LoopMode> {
|
|||||||
],
|
],
|
||||||
default: 'task',
|
default: 'task',
|
||||||
});
|
});
|
||||||
|
|
||||||
return mode;
|
return mode;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -176,7 +176,7 @@ async function handleExistingSession(
|
|||||||
export async function setupSession(
|
export async function setupSession(
|
||||||
stateManager: StateManager
|
stateManager: StateManager
|
||||||
): Promise<SessionSetupResult | null> {
|
): Promise<SessionSetupResult | null> {
|
||||||
// Show welcome
|
// Show Planny welcome
|
||||||
logger.welcome();
|
logger.welcome();
|
||||||
|
|
||||||
// Select spec
|
// Select spec
|
||||||
|
|||||||
@@ -142,12 +142,14 @@ export class Controller {
|
|||||||
logger.info(`Please ensure the "${this.agent.config.command}" command is installed and available in your PATH.`);
|
logger.info(`Please ensure the "${this.agent.config.command}" command is installed and available in your PATH.`);
|
||||||
throw new Error(`Agent "${this.agent.config.displayName}" is not available. Please install it and try again.`);
|
throw new Error(`Agent "${this.agent.config.displayName}" is not available. Please install it and try again.`);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Ensure git repo and .gitignore are set up before any iterations
|
// Ensure git repo and .gitignore are set up before any iterations
|
||||||
const gitReady = await ensureGitRepo(process.cwd());
|
const gitReady = await ensureGitRepo(process.cwd());
|
||||||
if (!gitReady) {
|
if (!gitReady) {
|
||||||
throw new Error('Failed to initialize a git repository in the current working directory. Cannot start Plan2Code Loop.');
|
throw new Error('Failed to initialize a git repository in the current working directory. Cannot start Plan2Code Loop.');
|
||||||
}
|
}
|
||||||
ensureGitignore(process.cwd());
|
ensureGitignore(process.cwd());
|
||||||
|
|
||||||
const loopModeLabel = (this.config.loopMode || 'task') === 'phase' ? 'One phase per loop' : 'One task per loop';
|
const loopModeLabel = (this.config.loopMode || 'task') === 'phase' ? 'One phase per loop' : 'One task per loop';
|
||||||
logger.header('Starting Plan2Code Loop');
|
logger.header('Starting Plan2Code Loop');
|
||||||
logger.info(`Agent: ${this.agent.config.displayName}`);
|
logger.info(`Agent: ${this.agent.config.displayName}`);
|
||||||
@@ -215,9 +217,11 @@ export class Controller {
|
|||||||
|
|
||||||
// Branch completion handling based on loop mode
|
// Branch completion handling based on loop mode
|
||||||
const isPhaseMode = (this.config.loopMode || 'task') === 'phase';
|
const isPhaseMode = (this.config.loopMode || 'task') === 'phase';
|
||||||
|
|
||||||
if (isPhaseMode) {
|
if (isPhaseMode) {
|
||||||
// Phase mode: parse ALL completion markers from output
|
// Phase mode: parse ALL completion markers from output
|
||||||
const allCompletions = checkForAllCompletions(result.stdout + result.stderr);
|
const allCompletions = checkForAllCompletions(result.stdout + result.stderr);
|
||||||
|
|
||||||
// Determine status
|
// Determine status
|
||||||
let status: IterationLogEntry['status'] = 'running';
|
let status: IterationLogEntry['status'] = 'running';
|
||||||
if (allCompletions.tasks.length > 0 || allCompletions.loopComplete || allCompletions.phaseComplete) {
|
if (allCompletions.tasks.length > 0 || allCompletions.loopComplete || allCompletions.phaseComplete) {
|
||||||
@@ -229,10 +233,12 @@ export class Controller {
|
|||||||
} else if (result.exitCode !== 0) {
|
} else if (result.exitCode !== 0) {
|
||||||
status = 'error';
|
status = 'error';
|
||||||
}
|
}
|
||||||
|
|
||||||
// Log iteration with count of tasks
|
// Log iteration with count of tasks
|
||||||
const markerSummary = allCompletions.tasks.map(t => `${t.marker}: ${t.taskId}`).join(', ');
|
const markerSummary = allCompletions.tasks.map(t => `${t.marker}: ${t.taskId}`).join(', ');
|
||||||
const logEntry = this.createLogEntry(result, status, markerSummary || undefined);
|
const logEntry = this.createLogEntry(result, status, markerSummary || undefined);
|
||||||
await this.stateManager.appendIterationLog(logEntry);
|
await this.stateManager.appendIterationLog(logEntry);
|
||||||
|
|
||||||
// Display each completed task
|
// Display each completed task
|
||||||
const duration = Math.round(result.duration / 1000);
|
const duration = Math.round(result.duration / 1000);
|
||||||
for (const task of allCompletions.tasks) {
|
for (const task of allCompletions.tasks) {
|
||||||
@@ -259,6 +265,7 @@ export class Controller {
|
|||||||
logger.warning(blockInfo);
|
logger.warning(blockInfo);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Show phase-level summary
|
// Show phase-level summary
|
||||||
if (allCompletions.tasks.length > 0) {
|
if (allCompletions.tasks.length > 0) {
|
||||||
const completedCount = allCompletions.tasks.filter(t => t.marker === 'TASK_COMPLETE').length;
|
const completedCount = allCompletions.tasks.filter(t => t.marker === 'TASK_COMPLETE').length;
|
||||||
@@ -273,6 +280,7 @@ export class Controller {
|
|||||||
} else {
|
} else {
|
||||||
logger.iteration(iterNum, this.config.maxIterations, `completed in ${duration}s`);
|
logger.iteration(iterNum, this.config.maxIterations, `completed in ${duration}s`);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Verbose output
|
// Verbose output
|
||||||
if (this.config.verbose) {
|
if (this.config.verbose) {
|
||||||
console.log();
|
console.log();
|
||||||
@@ -287,6 +295,7 @@ export class Controller {
|
|||||||
} else if (result.exitCode !== 0 && result.stderr.trim()) {
|
} else if (result.exitCode !== 0 && result.stderr.trim()) {
|
||||||
logger.error(` ${result.stderr.trim().split('\n')[0]}`);
|
logger.error(` ${result.stderr.trim().split('\n')[0]}`);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Handle LOOP_COMPLETE
|
// Handle LOOP_COMPLETE
|
||||||
if (allCompletions.loopComplete) {
|
if (allCompletions.loopComplete) {
|
||||||
this.onLoopComplete?.();
|
this.onLoopComplete?.();
|
||||||
@@ -302,42 +311,41 @@ export class Controller {
|
|||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
// Task mode (default): existing single-marker logic
|
// Task mode (default): existing single-marker logic
|
||||||
const completion = checkForCompletion(result.stdout + result.stderr);
|
const completion = checkForCompletion(result.stdout + result.stderr);
|
||||||
|
|
||||||
// Determine status
|
// Determine status
|
||||||
let status: IterationLogEntry['status'] = 'running';
|
let status: IterationLogEntry['status'] = 'running';
|
||||||
if (completion.completed) {
|
if (completion.completed) {
|
||||||
if (completion.marker === 'TASK_BLOCKED') {
|
if (completion.marker === 'TASK_BLOCKED') {
|
||||||
status = 'blocked';
|
status = 'blocked';
|
||||||
} else {
|
} else {
|
||||||
status = 'completed';
|
status = 'completed';
|
||||||
|
}
|
||||||
|
} else if (result.timedOut) {
|
||||||
|
status = 'timeout';
|
||||||
|
} else if (result.exitCode !== 0) {
|
||||||
|
status = 'error';
|
||||||
}
|
}
|
||||||
} else if (result.timedOut) {
|
|
||||||
status = 'timeout';
|
|
||||||
} else if (result.exitCode !== 0) {
|
|
||||||
status = 'error';
|
|
||||||
}
|
|
||||||
|
|
||||||
// Log iteration
|
// Log iteration
|
||||||
const logEntry = this.createLogEntry(result, status, completion.marker);
|
const logEntry = this.createLogEntry(result, status, completion.marker);
|
||||||
await this.stateManager.appendIterationLog(logEntry);
|
await this.stateManager.appendIterationLog(logEntry);
|
||||||
|
|
||||||
// Display iteration result with task info from completion marker
|
// Display iteration result with task info from completion marker
|
||||||
this.displayIterationResult(result, iterNum, completion);
|
this.displayIterationResult(result, iterNum, completion);
|
||||||
|
|
||||||
|
// Handle completion markers
|
||||||
|
if (completion.completed) {
|
||||||
|
const taskDisplay = this.formatTaskDisplay(completion);
|
||||||
|
|
||||||
// Handle completion markers
|
if (completion.marker === 'TASK_COMPLETE') {
|
||||||
if (completion.completed) {
|
this.tasksCompleted++;
|
||||||
const taskDisplay = this.formatTaskDisplay(completion);
|
await this.onTaskComplete?.({
|
||||||
|
marker: completion.marker,
|
||||||
if (completion.marker === 'TASK_COMPLETE') {
|
taskId: completion.taskId,
|
||||||
this.tasksCompleted++;
|
taskName: completion.taskName,
|
||||||
await this.onTaskComplete?.({
|
});
|
||||||
marker: completion.marker,
|
logger.success(taskDisplay ? `Completed: ${taskDisplay}` : 'Task completed!');
|
||||||
taskId: completion.taskId,
|
|
||||||
taskName: completion.taskName,
|
|
||||||
});
|
|
||||||
logger.success(taskDisplay ? `Completed: ${taskDisplay}` : 'Task completed!');
|
|
||||||
} else if (completion.marker === 'PREREQ_COMPLETE' || completion.marker === 'PREREQ_ASSUMED') {
|
} else if (completion.marker === 'PREREQ_COMPLETE' || completion.marker === 'PREREQ_ASSUMED') {
|
||||||
this.prereqsCompleted++;
|
this.prereqsCompleted++;
|
||||||
await this.onTaskComplete?.({
|
await this.onTaskComplete?.({
|
||||||
@@ -350,29 +358,29 @@ export class Controller {
|
|||||||
: completion.taskId ? `Prereq ${completion.taskId}` : 'Prerequisite';
|
: completion.taskId ? `Prereq ${completion.taskId}` : 'Prerequisite';
|
||||||
const verb = completion.marker === 'PREREQ_COMPLETE' ? 'Verified' : 'Assumed';
|
const verb = completion.marker === 'PREREQ_COMPLETE' ? 'Verified' : 'Assumed';
|
||||||
logger.success(`${verb}: ${prereqDisplay}`);
|
logger.success(`${verb}: ${prereqDisplay}`);
|
||||||
} else if (completion.marker === 'TASK_BLOCKED') {
|
} else if (completion.marker === 'TASK_BLOCKED') {
|
||||||
const blockInfo = completion.taskId
|
const blockInfo = completion.taskId
|
||||||
? `Task ${completion.taskId} blocked: ${completion.reason || 'Unknown reason'}`
|
? `Task ${completion.taskId} blocked: ${completion.reason || 'Unknown reason'}`
|
||||||
: `Task blocked: ${completion.reason || 'Unknown reason'}`;
|
: `Task blocked: ${completion.reason || 'Unknown reason'}`;
|
||||||
logger.warning(blockInfo);
|
logger.warning(blockInfo);
|
||||||
} else if (completion.marker === 'LOOP_COMPLETE') {
|
} else if (completion.marker === 'LOOP_COMPLETE') {
|
||||||
// Commit any final changes before completing
|
// Commit any final changes before completing
|
||||||
this.tasksCompleted++;
|
this.tasksCompleted++;
|
||||||
await this.onTaskComplete?.({
|
await this.onTaskComplete?.({
|
||||||
marker: completion.marker,
|
marker: completion.marker,
|
||||||
taskId: completion.taskId,
|
taskId: completion.taskId,
|
||||||
taskName: completion.taskName || 'Final implementation complete',
|
taskName: completion.taskName || 'Final implementation complete',
|
||||||
});
|
});
|
||||||
this.onLoopComplete?.();
|
this.onLoopComplete?.();
|
||||||
logger.success('All tasks complete!');
|
logger.success('All tasks complete!');
|
||||||
return {
|
return {
|
||||||
completed: true,
|
completed: true,
|
||||||
iterations: iterNum,
|
iterations: iterNum,
|
||||||
finalMarker: 'LOOP_COMPLETE',
|
finalMarker: 'LOOP_COMPLETE',
|
||||||
exitReason: 'all_complete',
|
exitReason: 'all_complete',
|
||||||
tasksCompleted: this.tasksCompleted,
|
tasksCompleted: this.tasksCompleted,
|
||||||
prereqsCompleted: this.prereqsCompleted,
|
prereqsCompleted: this.prereqsCompleted,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -394,7 +402,7 @@ export class Controller {
|
|||||||
? checkForAllCompletions(result.stdout + result.stderr).tasks.length > 0
|
? checkForAllCompletions(result.stdout + result.stderr).tasks.length > 0
|
||||||
: checkForCompletion(result.stdout + result.stderr).completed;
|
: checkForCompletion(result.stdout + result.stderr).completed;
|
||||||
if (!hasCompletions) {
|
if (!hasCompletions) {
|
||||||
logger.warning(`Iteration ${iterNum} exited with code ${result.exitCode}, continuing...`);
|
logger.warning(`Iteration ${iterNum} exited with code ${result.exitCode}, continuing...`);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -80,6 +80,7 @@ export async function run(): Promise<LoopResult | null> {
|
|||||||
logger.dim(' - config.json (session configuration)');
|
logger.dim(' - config.json (session configuration)');
|
||||||
logger.dim(' - scratchpad.md (LLM-managed notes)');
|
logger.dim(' - scratchpad.md (LLM-managed notes)');
|
||||||
logger.dim(' - iteration.log (history)');
|
logger.dim(' - iteration.log (history)');
|
||||||
|
logger.dim('Tip: run `plan2code-metrics` to capture run data for prompt improvement.');
|
||||||
|
|
||||||
return loopResult;
|
return loopResult;
|
||||||
} finally {
|
} finally {
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
export const LOOP_PROMPT_TEMPLATE = `# PLAN-LOOP: Autonomous Task Implementation
|
export const LOOP_PROMPT_TEMPLATE = `# PLAN2CODE-LOOP: Autonomous Task Implementation
|
||||||
|
|
||||||
## CRITICAL CONSTRAINT
|
## CRITICAL CONSTRAINT
|
||||||
**IMPLEMENT EXACTLY ONE TASK PER ITERATION.**
|
**IMPLEMENT EXACTLY ONE TASK PER ITERATION.**
|
||||||
@@ -83,7 +83,9 @@ If key patterns or learnings were discovered, update \`./AGENTS.md\` if it exist
|
|||||||
|
|
||||||
Remember: ONE TASK ONLY. Find it, implement it, mark it done, output TASK_COMPLETE with the task ID and description, then stop.
|
Remember: ONE TASK ONLY. Find it, implement it, mark it done, output TASK_COMPLETE with the task ID and description, then stop.
|
||||||
`;
|
`;
|
||||||
|
|
||||||
export const LOOP_PROMPT_TEMPLATE_PHASE = `# PLAN2CODE-LOOP: Autonomous Phase Implementation
|
export const LOOP_PROMPT_TEMPLATE_PHASE = `# PLAN2CODE-LOOP: Autonomous Phase Implementation
|
||||||
|
|
||||||
## CRITICAL CONSTRAINT
|
## CRITICAL CONSTRAINT
|
||||||
**IMPLEMENT ALL REMAINING TASKS IN THE CURRENT PHASE.**
|
**IMPLEMENT ALL REMAINING TASKS IN THE CURRENT PHASE.**
|
||||||
Find the first incomplete phase, then implement every remaining task in that phase before stopping.
|
Find the first incomplete phase, then implement every remaining task in that phase before stopping.
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
export type LoopMode = 'task' | 'phase';
|
export type LoopMode = 'task' | 'phase';
|
||||||
|
|
||||||
export interface SessionConfig {
|
export interface SessionConfig {
|
||||||
agent: string; // "claude-code" | "copilot-cli"
|
agent: string; // "claude-code" | "copilot-cli"
|
||||||
model: string; // Selected model
|
model: string; // Selected model
|
||||||
|
|||||||
@@ -54,6 +54,7 @@ export function checkForCompletion(output: string): CompletionCheckResult {
|
|||||||
taskName: prereqMatch[2].trim(),
|
taskName: prereqMatch[2].trim(),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
// Check for PREREQ_ASSUMED with prereq info
|
// Check for PREREQ_ASSUMED with prereq info
|
||||||
// Format: PREREQ_ASSUMED: P2.1 - Design approval (cannot verify)
|
// Format: PREREQ_ASSUMED: P2.1 - Design approval (cannot verify)
|
||||||
const assumedMatch = output.match(/PREREQ_ASSUMED[:\[\s]+([^\]:\-\s]+)[\]:\-\s]+(.+?)(?:\n|$)/i);
|
const assumedMatch = output.match(/PREREQ_ASSUMED[:\[\s]+([^\]:\-\s]+)[\]:\-\s]+(.+?)(?:\n|$)/i);
|
||||||
@@ -65,6 +66,7 @@ export function checkForCompletion(output: string): CompletionCheckResult {
|
|||||||
taskName: assumedMatch[2].trim(),
|
taskName: assumedMatch[2].trim(),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
// Check for TASK_BLOCKED with task info and reason
|
// Check for TASK_BLOCKED with task info and reason
|
||||||
// Format: TASK_BLOCKED: 1.1 - Reason why blocked
|
// Format: TASK_BLOCKED: 1.1 - Reason why blocked
|
||||||
const blockedWithTaskMatch = output.match(/TASK_BLOCKED[:\[\s]+(\d+\.\d+)[\]:\-\s]+(.+?)(?:\n|$)/i);
|
const blockedWithTaskMatch = output.match(/TASK_BLOCKED[:\[\s]+(\d+\.\d+)[\]:\-\s]+(.+?)(?:\n|$)/i);
|
||||||
@@ -94,6 +96,7 @@ export function checkForCompletion(output: string): CompletionCheckResult {
|
|||||||
|
|
||||||
return { completed: false };
|
return { completed: false };
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Extract ALL completion markers from output (for phase mode).
|
* Extract ALL completion markers from output (for phase mode).
|
||||||
* Returns an array of all TASK_COMPLETE/TASK_BLOCKED markers found,
|
* Returns an array of all TASK_COMPLETE/TASK_BLOCKED markers found,
|
||||||
@@ -107,12 +110,15 @@ export function checkForAllCompletions(output: string): {
|
|||||||
const tasks: CompletionCheckResult[] = [];
|
const tasks: CompletionCheckResult[] = [];
|
||||||
let loopComplete = false;
|
let loopComplete = false;
|
||||||
let phaseComplete = false;
|
let phaseComplete = false;
|
||||||
|
|
||||||
if (output.includes('LOOP_COMPLETE')) {
|
if (output.includes('LOOP_COMPLETE')) {
|
||||||
loopComplete = true;
|
loopComplete = true;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (output.includes('PHASE_COMPLETE')) {
|
if (output.includes('PHASE_COMPLETE')) {
|
||||||
phaseComplete = true;
|
phaseComplete = true;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Find all TASK_COMPLETE markers with task info
|
// Find all TASK_COMPLETE markers with task info
|
||||||
// Format: TASK_COMPLETE: 1.1 - Task description
|
// Format: TASK_COMPLETE: 1.1 - Task description
|
||||||
const completeRegex = /TASK_COMPLETE[:\[\s]+(\d+\.\d+)[\]:\-\s]+(.+?)(?:\n|$)/gi;
|
const completeRegex = /TASK_COMPLETE[:\[\s]+(\d+\.\d+)[\]:\-\s]+(.+?)(?:\n|$)/gi;
|
||||||
@@ -125,6 +131,7 @@ export function checkForAllCompletions(output: string): {
|
|||||||
taskName: match[2].trim(),
|
taskName: match[2].trim(),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
// Find all TASK_BLOCKED markers with task info
|
// Find all TASK_BLOCKED markers with task info
|
||||||
const blockedRegex = /TASK_BLOCKED[:\[\s]+(\d+\.\d+)[\]:\-\s]+(.+?)(?:\n|$)/gi;
|
const blockedRegex = /TASK_BLOCKED[:\[\s]+(\d+\.\d+)[\]:\-\s]+(.+?)(?:\n|$)/gi;
|
||||||
while ((match = blockedRegex.exec(output)) !== null) {
|
while ((match = blockedRegex.exec(output)) !== null) {
|
||||||
@@ -135,6 +142,7 @@ export function checkForAllCompletions(output: string): {
|
|||||||
reason: match[2].trim(),
|
reason: match[2].trim(),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
// Find PREREQ_COMPLETE markers
|
// Find PREREQ_COMPLETE markers
|
||||||
const prereqRegex = /PREREQ_COMPLETE[:\[\s]+([^\]:\-\s]+)[\]:\-\s]+(.+?)(?:\n|$)/gi;
|
const prereqRegex = /PREREQ_COMPLETE[:\[\s]+([^\]:\-\s]+)[\]:\-\s]+(.+?)(?:\n|$)/gi;
|
||||||
while ((match = prereqRegex.exec(output)) !== null) {
|
while ((match = prereqRegex.exec(output)) !== null) {
|
||||||
@@ -145,6 +153,7 @@ export function checkForAllCompletions(output: string): {
|
|||||||
taskName: match[2].trim(),
|
taskName: match[2].trim(),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
// Find PREREQ_ASSUMED markers
|
// Find PREREQ_ASSUMED markers
|
||||||
const assumedRegex = /PREREQ_ASSUMED[:\[\s]+([^\]:\-\s]+)[\]:\-\s]+(.+?)(?:\n|$)/gi;
|
const assumedRegex = /PREREQ_ASSUMED[:\[\s]+([^\]:\-\s]+)[\]:\-\s]+(.+?)(?:\n|$)/gi;
|
||||||
while ((match = assumedRegex.exec(output)) !== null) {
|
while ((match = assumedRegex.exec(output)) !== null) {
|
||||||
@@ -155,5 +164,6 @@ export function checkForAllCompletions(output: string): {
|
|||||||
taskName: match[2].trim(),
|
taskName: match[2].trim(),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
return { tasks, loopComplete, phaseComplete };
|
return { tasks, loopComplete, phaseComplete };
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -40,7 +40,7 @@ export async function ensureGitRepo(cwd: string): Promise<boolean> {
|
|||||||
/**
|
/**
|
||||||
* Required entries for the .gitignore file
|
* Required entries for the .gitignore file
|
||||||
*/
|
*/
|
||||||
const REQUIRED_GITIGNORE_ENTRIES = ['specs/', 'specs--completed/', 'nul', 'node_modules/'];
|
const REQUIRED_GITIGNORE_ENTRIES = ['specs/', 'specs--completed/', '.plan2code-loop', '.plan2code-metrics', 'nul', 'node_modules/'];
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Ensure .gitignore exists with required entries
|
* Ensure .gitignore exists with required entries
|
||||||
@@ -109,7 +109,9 @@ export async function createTaskCommit(options: GitCommitOptions): Promise<boole
|
|||||||
// Build commit message
|
// Build commit message
|
||||||
let commitMessage = taskName;
|
let commitMessage = taskName;
|
||||||
if (jiraTicketId) {
|
if (jiraTicketId) {
|
||||||
commitMessage = `${taskName}\n\n${jiraTicketId}`;
|
commitMessage = `${taskName}\n\n${jiraTicketId}\n\nAI Assisted`;
|
||||||
|
} else {
|
||||||
|
commitMessage = `${taskName}\n\nAI Assisted`;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Create the commit
|
// Create the commit
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
export { logger, MASCOT, type Logger } from './logger.js';
|
export { logger, MASCOT, type Logger } from './logger.js';
|
||||||
export {
|
export {
|
||||||
checkForCompletion,
|
checkForCompletion,
|
||||||
|
checkForAllCompletions,
|
||||||
type CompletionMarker,
|
type CompletionMarker,
|
||||||
type CompletionCheckResult
|
type CompletionCheckResult
|
||||||
} from './completion.js';
|
} from './completion.js';
|
||||||
|
|||||||
@@ -0,0 +1,289 @@
|
|||||||
|
# plan2code-metrics
|
||||||
|
|
||||||
|
Recursive self-improvement toolchain for plan2code contributors. Collects metrics from completed project specs, aggregates them across runs and prompt generations, then uses AI to diagnose weak steps and propose targeted edits to the workflow prompts.
|
||||||
|
|
||||||
|
## Quick Reference
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Install (one-time, from the plan2code root)
|
||||||
|
node install.js # → "I" (Install All) includes metrics
|
||||||
|
# → or "M" (Metrics only) under the CUSTOM sub-menu
|
||||||
|
|
||||||
|
# After finishing any project spec (steps 1-4):
|
||||||
|
cd your-project
|
||||||
|
plan2code-metrics # → "Collect metrics" → pick your spec dir → done (5 sec)
|
||||||
|
|
||||||
|
# When you're curious or have 3+ runs:
|
||||||
|
plan2code-metrics # → "View metrics status" to see the dashboard
|
||||||
|
# → "Run analysis" for AI diagnosis of weak spots
|
||||||
|
# → "Generate improvement proposal" for prompt edits
|
||||||
|
# → "Review and apply" to patch src/plan2code-*.md
|
||||||
|
```
|
||||||
|
|
||||||
|
**One habit:** collect after every finished spec. Everything else is on-demand.
|
||||||
|
|
||||||
|
### How Many Runs Do I Need?
|
||||||
|
|
||||||
|
| Runs | What You Get |
|
||||||
|
|------|-------------|
|
||||||
|
| **1** | Raw data and basic dashboard. Start here. |
|
||||||
|
| **3+** | AI analysis unlocks (warns below 3 that results may be unreliable). Pattern detection starts working. |
|
||||||
|
| **5-10+** | Averages stabilize. Generation-over-generation comparisons become meaningful. |
|
||||||
|
|
||||||
|
More is always better — each run adds a data point. You're looking for trends, not individual scores.
|
||||||
|
|
||||||
|
### The Feedback Loop
|
||||||
|
|
||||||
|
```
|
||||||
|
Use plan2code on a project (steps 1-4)
|
||||||
|
|
|
||||||
|
v
|
||||||
|
Collect metrics from the finished spec <-- do this every time
|
||||||
|
|
|
||||||
|
v
|
||||||
|
Aggregate across runs (automatic)
|
||||||
|
|
|
||||||
|
v
|
||||||
|
Analyze weak spots (AI-powered, 3+ runs)
|
||||||
|
|
|
||||||
|
v
|
||||||
|
Generate prompt improvements
|
||||||
|
|
|
||||||
|
v
|
||||||
|
Apply edits to src/plan2code-*.md
|
||||||
|
|
|
||||||
|
v
|
||||||
|
Prompts have new SHA hashes = new "generation"
|
||||||
|
|
|
||||||
|
v
|
||||||
|
Use improved prompts on next project
|
||||||
|
|
|
||||||
|
v
|
||||||
|
Collect again, compare generations <-- the loop closes
|
||||||
|
```
|
||||||
|
|
||||||
|
After applying edits, the prompt file SHA hashes change. The next collected run falls into a **new cohort** (generation). The dashboard then shows generation-over-generation deltas — "Did confidence go up? Did blockers decrease?" — so you can measure whether your edits actually helped.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
Requires Node.js >= 18.
|
||||||
|
|
||||||
|
### Via the plan2code installer (recommended)
|
||||||
|
|
||||||
|
From the plan2code repo root:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
node install.js
|
||||||
|
```
|
||||||
|
|
||||||
|
Choose **I** (Install All) to install prompts, loop CLI, and metrics together. Or choose **C** (Custom) then **M** (Metrics) to install just the metrics CLI.
|
||||||
|
|
||||||
|
The installer handles `npm install`, `npm run build`, and global linking automatically. After install, `plan2code-metrics` is available from any directory.
|
||||||
|
|
||||||
|
### Manual install
|
||||||
|
|
||||||
|
If you prefer to install manually or are developing on the metrics package:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd plan2code-metrics
|
||||||
|
npm install
|
||||||
|
npm run build
|
||||||
|
npm link
|
||||||
|
```
|
||||||
|
|
||||||
|
To run without linking:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm start
|
||||||
|
# or
|
||||||
|
node dist/bin/plan2code-metrics.js
|
||||||
|
```
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
Run `plan2code-metrics` from inside (or near) a plan2code repository. The CLI is fully interactive — no flags, all inputs collected via prompts.
|
||||||
|
|
||||||
|
```
|
||||||
|
plan2code-metrics
|
||||||
|
```
|
||||||
|
|
||||||
|
### Main Menu
|
||||||
|
|
||||||
|
| Option | Description |
|
||||||
|
|--------|-------------|
|
||||||
|
| **Collect metrics** | Parse a completed project spec and extract step-by-step metrics into a run JSON |
|
||||||
|
| **Import run data** | Copy a run JSON from another project into the local metrics store |
|
||||||
|
| **View metrics status** | Show aggregated metrics with health indicators and generation deltas |
|
||||||
|
| **Run analysis** | AI-powered diagnosis of weak steps (requires Claude Code or Copilot CLI) |
|
||||||
|
| **Generate improvement proposal** | AI generates surgical prompt edits based on a diagnosis |
|
||||||
|
| **Review and apply** | Interactive diff review to accept/reject individual edits |
|
||||||
|
|
||||||
|
### Each Time You Finish a Spec
|
||||||
|
|
||||||
|
After completing Step 4 (finalize) on any project:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd your-project
|
||||||
|
plan2code-metrics
|
||||||
|
```
|
||||||
|
|
||||||
|
Pick **"Collect metrics"** and point it at your spec directory (`specs/<feature>/` or `specs--completed/<feature>/`). It reads your artifacts, extracts ~30 metrics, and writes a run JSON. Takes about 5 seconds.
|
||||||
|
|
||||||
|
The data stays local to that project in `.plan2code-metrics/runs/`. To aggregate across multiple projects, use **"Import"** to copy run files into one central location.
|
||||||
|
|
||||||
|
### Cross-Project Aggregation
|
||||||
|
|
||||||
|
If you use plan2code across several repos, you can consolidate all run data in one place:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# From your central repo (e.g., the plan2code repo itself):
|
||||||
|
plan2code-metrics
|
||||||
|
# → "Import run data"
|
||||||
|
# → Paste path to: /path/to/other-project/.plan2code-metrics/runs/run-xxx.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Imported runs are copied locally and included in all future aggregations, analyses, and comparisons.
|
||||||
|
|
||||||
|
## What It Measures
|
||||||
|
|
||||||
|
Each collection scrapes your spec files and extracts:
|
||||||
|
|
||||||
|
### Step 1 (Plan) — from `PLAN-DRAFT-*.md` and `PLAN-CONVERSATION-*.md`
|
||||||
|
|
||||||
|
- **Confidence score** — overall planning confidence percentage
|
||||||
|
- **Confidence breakdown** — requirements, feasibility, integration, risk sub-scores
|
||||||
|
- **Clarification rounds** — how many rounds of Q&A occurred
|
||||||
|
- **Verification gaps found** — missing requirements caught during verification
|
||||||
|
- **Functional/non-functional requirement counts** — FR- and NFR- headings
|
||||||
|
- **Risk count** — risk table entries
|
||||||
|
- **Phase count** — number of implementation phases planned
|
||||||
|
|
||||||
|
### Step 2 (Document) — from `overview.md` and `phase-*.md`
|
||||||
|
|
||||||
|
- **Total tasks** — checkbox count across all phases
|
||||||
|
- **Tasks per phase** — distribution of work
|
||||||
|
- **Phase count** — number of phase files
|
||||||
|
- **Parallel groups** — parallel execution groups identified
|
||||||
|
- **Requirement coverage** — coverage percentage if present
|
||||||
|
- **Verification items** — verification checklist items added
|
||||||
|
|
||||||
|
### Step 3 (Implement) — from `.plan2code-loop/` data (loop mode only)
|
||||||
|
|
||||||
|
- **Task completion rate** — completed / total tasks
|
||||||
|
- **Blocker count** — tasks marked TASK_BLOCKED
|
||||||
|
- **Blocker categories** — normalized reasons for blocks
|
||||||
|
- **Total iterations** — loop iteration count
|
||||||
|
- **Avg iteration duration** — mean time per iteration
|
||||||
|
- **Exit code distribution** — process exit codes
|
||||||
|
- **Completion marker success rate** — valid markers / total iterations
|
||||||
|
|
||||||
|
### Step 4 (Finalize) — from `overview.md` and directory structure
|
||||||
|
|
||||||
|
- **Completion rate at audit** — tasks done at finalization time
|
||||||
|
- **Verification failures** — tasks marked with `[!]`
|
||||||
|
- **Documentation updates needed** — TODO/FIXME/update markers
|
||||||
|
- **Archival success** — whether spec was moved to `specs--completed/`
|
||||||
|
|
||||||
|
### Health Targets
|
||||||
|
|
||||||
|
The dashboard compares key metrics against targets:
|
||||||
|
|
||||||
|
| Metric | Target | Direction |
|
||||||
|
|--------|--------|-----------|
|
||||||
|
| avg_confidence | 90 | >= |
|
||||||
|
| avg_clarification_rounds | 2.0 | <= |
|
||||||
|
| avg_verification_gaps_found | 2.0 | <= |
|
||||||
|
| avg_parallel_groups | 0.5 | >= |
|
||||||
|
| avg_task_completion_rate | 0.95 | >= |
|
||||||
|
| avg_blocker_count | 1.5 | <= |
|
||||||
|
| avg_completion_marker_success_rate | 0.95 | >= |
|
||||||
|
| avg_verification_failures_found | 1.0 | <= |
|
||||||
|
| archival_success_rate | 0.99 | >= |
|
||||||
|
|
||||||
|
Green checkmark = meeting target. Red X = below target (with the target shown).
|
||||||
|
|
||||||
|
## How It Works
|
||||||
|
|
||||||
|
### 1. Collect
|
||||||
|
|
||||||
|
Reads completed project artifacts from `specs/` or `specs--completed/` and extracts metrics for each workflow step. Output: `.plan2code-metrics/runs/run-<timestamp>.json`
|
||||||
|
|
||||||
|
### 2. Aggregate
|
||||||
|
|
||||||
|
Merges all run files into `aggregated.json`, grouped by **prompt generation** — a SHA fingerprint of the 8 workflow prompt files. Two runs that used identical prompt files belong to the same cohort.
|
||||||
|
|
||||||
|
### 3. Analyze
|
||||||
|
|
||||||
|
Sends aggregated metrics to an AI model with an analyst prompt. The AI produces a diagnosis identifying the weakest workflow steps and root causes. No edits are proposed at this stage.
|
||||||
|
|
||||||
|
Output: `.plan2code-metrics/proposals/<timestamp>-diagnosis.md`
|
||||||
|
|
||||||
|
### 4. Improve
|
||||||
|
|
||||||
|
Sends the diagnosis plus current prompt file contents to an AI model. The AI proposes surgical `old_text -> new_text` edits, each validated against:
|
||||||
|
|
||||||
|
- The `old_text` actually exists in the target file
|
||||||
|
- The edited file stays under the 11,000 character limit
|
||||||
|
|
||||||
|
Output: `.plan2code-metrics/proposals/prop-<timestamp>.json`
|
||||||
|
|
||||||
|
### 5. Apply
|
||||||
|
|
||||||
|
Interactive diff review for each proposed edit. Accept, reject, or skip individual changes. Applied edits are patched directly into `src/plan2code-*.md`.
|
||||||
|
|
||||||
|
## Data Directory
|
||||||
|
|
||||||
|
All metrics data lives in `.plan2code-metrics/` (add to `.gitignore` if desired):
|
||||||
|
|
||||||
|
```
|
||||||
|
.plan2code-metrics/
|
||||||
|
├── runs/ # Individual run JSONs
|
||||||
|
│ ├── run-20260215-120000-a1b2.json
|
||||||
|
│ └── run-20260220-090000-c3d4.json
|
||||||
|
├── aggregated.json # Merged cohort data
|
||||||
|
└── proposals/ # Diagnoses and improvement proposals
|
||||||
|
├── 20260220-diagnosis.md
|
||||||
|
└── prop-20260220.json
|
||||||
|
```
|
||||||
|
|
||||||
|
## AI Backends
|
||||||
|
|
||||||
|
The analysis and improvement steps require an AI agent. Two backends are supported:
|
||||||
|
|
||||||
|
| Backend | Command | Notes |
|
||||||
|
|---------|---------|-------|
|
||||||
|
| **Claude Code** | `claude` | Uses `--print` mode. Recommended. |
|
||||||
|
| **GitHub Copilot CLI** | `copilot` | Uses stdin piping with `--allow-all-tools -s`. |
|
||||||
|
|
||||||
|
Model selection is interactive — choose from available models when prompted.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
src/
|
||||||
|
├── bin/plan2code-metrics.ts # Entry point
|
||||||
|
├── cli.ts # Interactive menu (main loop)
|
||||||
|
├── types.ts # RunMetrics, PromptProposal, CohortMetrics, etc.
|
||||||
|
├── collector.ts # Reads project artifacts → run JSON
|
||||||
|
├── aggregator.ts # Merges runs → aggregated.json (cohort grouping)
|
||||||
|
├── analyzer.ts # AI diagnosis via LLM invocation
|
||||||
|
├── improver.ts # AI improvement proposal + validation
|
||||||
|
├── applier.ts # Interactive diff review + file patching
|
||||||
|
├── invoke-llm.ts # Unified LLM invocation (Claude Code / Copilot CLI)
|
||||||
|
├── index.ts # Public API exports
|
||||||
|
└── prompts/
|
||||||
|
├── analyze.md # AI prompt template for diagnosis
|
||||||
|
└── improve.md # AI prompt template for improvement proposals
|
||||||
|
```
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run dev # Watch mode (rebuilds on change)
|
||||||
|
npm run build # Production build
|
||||||
|
npm test # Run unit tests
|
||||||
|
npm run test:watch # Watch mode tests
|
||||||
|
npm start # Run the CLI
|
||||||
|
```
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
{
|
||||||
|
"name": "plan2code-metrics",
|
||||||
|
"version": "1.0.0",
|
||||||
|
"description": "Plan2Code Metrics - Recursive self-improvement toolchain for plan2code contributors",
|
||||||
|
"type": "module",
|
||||||
|
"main": "dist/index.js",
|
||||||
|
"bin": {
|
||||||
|
"plan2code-metrics": "./dist/bin/plan2code-metrics.js"
|
||||||
|
},
|
||||||
|
"files": [
|
||||||
|
"dist"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18.0.0"
|
||||||
|
},
|
||||||
|
"keywords": [
|
||||||
|
"cli",
|
||||||
|
"ai",
|
||||||
|
"metrics",
|
||||||
|
"plan2code",
|
||||||
|
"contributor-tooling"
|
||||||
|
],
|
||||||
|
"license": "MIT",
|
||||||
|
"scripts": {
|
||||||
|
"build": "tsup",
|
||||||
|
"dev": "tsup --watch",
|
||||||
|
"start": "node dist/bin/plan2code-metrics.js",
|
||||||
|
"test": "vitest run",
|
||||||
|
"test:watch": "vitest",
|
||||||
|
"prepublishOnly": "npm run build"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"@inquirer/prompts": "^8.1.0",
|
||||||
|
"chalk": "^5.6.2",
|
||||||
|
"execa": "^9.6.1",
|
||||||
|
"fs-extra": "^11.3.3",
|
||||||
|
"ora": "^9.0.0"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"@types/fs-extra": "^11.0.4",
|
||||||
|
"@types/node": "^25.0.3",
|
||||||
|
"tsup": "^8.5.1",
|
||||||
|
"typescript": "^5.9.3",
|
||||||
|
"vitest": "^3.1.1"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,154 @@
|
|||||||
|
import { describe, it, expect } from 'vitest';
|
||||||
|
import { avg, rate, buildCohortKey, backfillPromptVersions } from './aggregator.js';
|
||||||
|
import type { PromptVersions } from './types.js';
|
||||||
|
|
||||||
|
// ── avg() ─────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
describe('avg', () => {
|
||||||
|
it('returns null for empty array', () => {
|
||||||
|
expect(avg([])).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('returns null for all-null/undefined values', () => {
|
||||||
|
expect(avg([null, undefined, null])).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('computes correct average for valid numbers', () => {
|
||||||
|
expect(avg([10, 20, 30])).toBe(20);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('filters out null/undefined/NaN from mixed arrays', () => {
|
||||||
|
expect(avg([10, null, 20, undefined, NaN, 30])).toBe(20);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ── rate() ────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
describe('rate', () => {
|
||||||
|
it('returns null for empty array', () => {
|
||||||
|
expect(rate([])).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('returns null for all-null values', () => {
|
||||||
|
expect(rate([null, null])).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('returns 1.0 for all-true', () => {
|
||||||
|
expect(rate([true, true, true])).toBe(1.0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('returns 0.0 for all-false', () => {
|
||||||
|
expect(rate([false, false, false])).toBe(0.0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('computes correct rate for mixed true/false', () => {
|
||||||
|
expect(rate([true, false, true, false])).toBe(0.5);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('filters out null/undefined from mixed arrays', () => {
|
||||||
|
expect(rate([true, null, false, undefined])).toBe(0.5);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ── backfillPromptVersions() ──────────────────────────────────────────────────
|
||||||
|
|
||||||
|
const FULL_VERSIONS: PromptVersions = {
|
||||||
|
plan: 'sha256:aaa',
|
||||||
|
revise_plan: 'sha256:bbb',
|
||||||
|
document: 'sha256:ccc',
|
||||||
|
implement: 'sha256:ddd',
|
||||||
|
finalize: 'sha256:eee',
|
||||||
|
init: 'sha256:fff',
|
||||||
|
init_update: 'sha256:ggg',
|
||||||
|
quick_task: 'sha256:hhh',
|
||||||
|
};
|
||||||
|
|
||||||
|
describe('backfillPromptVersions', () => {
|
||||||
|
it('returns all 8 fields with sentinels when given empty-ish object', () => {
|
||||||
|
const result = backfillPromptVersions({} as PromptVersions);
|
||||||
|
expect(Object.keys(result)).toHaveLength(8);
|
||||||
|
for (const val of Object.values(result)) {
|
||||||
|
expect(val).toBe('sha256:missing');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it('preserves existing values, fills missing with sha256:missing', () => {
|
||||||
|
const partial = { plan: 'sha256:aaa', implement: 'sha256:ddd' } as PromptVersions;
|
||||||
|
const result = backfillPromptVersions(partial);
|
||||||
|
expect(result.plan).toBe('sha256:aaa');
|
||||||
|
expect(result.implement).toBe('sha256:ddd');
|
||||||
|
expect(result.revise_plan).toBe('sha256:missing');
|
||||||
|
expect(result.document).toBe('sha256:missing');
|
||||||
|
expect(result.finalize).toBe('sha256:missing');
|
||||||
|
expect(result.init).toBe('sha256:missing');
|
||||||
|
expect(result.init_update).toBe('sha256:missing');
|
||||||
|
expect(result.quick_task).toBe('sha256:missing');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('returns unchanged object when all 8 fields present', () => {
|
||||||
|
const result = backfillPromptVersions(FULL_VERSIONS);
|
||||||
|
expect(result).toEqual(FULL_VERSIONS);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ── buildCohortKey() ──────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
describe('buildCohortKey', () => {
|
||||||
|
it('returns 12-char hex string', () => {
|
||||||
|
const key = buildCohortKey(FULL_VERSIONS);
|
||||||
|
expect(key).toMatch(/^[0-9a-f]{12}$/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('is deterministic (same input → same output)', () => {
|
||||||
|
const key1 = buildCohortKey(FULL_VERSIONS);
|
||||||
|
const key2 = buildCohortKey(FULL_VERSIONS);
|
||||||
|
expect(key1).toBe(key2);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('old 4-field run with backfill sentinels produces same key as raw 4-field object', () => {
|
||||||
|
// Simulate an old run that only had 4 fields
|
||||||
|
const oldRun = {
|
||||||
|
plan: 'sha256:aaa',
|
||||||
|
implement: 'sha256:ddd',
|
||||||
|
document: 'sha256:ccc',
|
||||||
|
finalize: 'sha256:eee',
|
||||||
|
} as PromptVersions;
|
||||||
|
|
||||||
|
// After backfill, the missing fields get 'sha256:missing'
|
||||||
|
const backfilled = backfillPromptVersions(oldRun);
|
||||||
|
|
||||||
|
// buildCohortKey filters out 'sha256:missing', so both should match
|
||||||
|
const keyDirect = buildCohortKey(oldRun);
|
||||||
|
const keyBackfilled = buildCohortKey(backfilled);
|
||||||
|
expect(keyDirect).toBe(keyBackfilled);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('different prompt versions → different keys', () => {
|
||||||
|
const altered = { ...FULL_VERSIONS, plan: 'sha256:zzz' };
|
||||||
|
expect(buildCohortKey(FULL_VERSIONS)).not.toBe(buildCohortKey(altered));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('key is independent of field insertion order', () => {
|
||||||
|
const ordered: PromptVersions = {
|
||||||
|
plan: 'sha256:aaa',
|
||||||
|
revise_plan: 'sha256:bbb',
|
||||||
|
document: 'sha256:ccc',
|
||||||
|
implement: 'sha256:ddd',
|
||||||
|
finalize: 'sha256:eee',
|
||||||
|
init: 'sha256:fff',
|
||||||
|
init_update: 'sha256:ggg',
|
||||||
|
quick_task: 'sha256:hhh',
|
||||||
|
};
|
||||||
|
const reversed: PromptVersions = {
|
||||||
|
quick_task: 'sha256:hhh',
|
||||||
|
init_update: 'sha256:ggg',
|
||||||
|
init: 'sha256:fff',
|
||||||
|
finalize: 'sha256:eee',
|
||||||
|
implement: 'sha256:ddd',
|
||||||
|
document: 'sha256:ccc',
|
||||||
|
revise_plan: 'sha256:bbb',
|
||||||
|
plan: 'sha256:aaa',
|
||||||
|
};
|
||||||
|
expect(buildCohortKey(ordered)).toBe(buildCohortKey(reversed));
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,227 @@
|
|||||||
|
/**
|
||||||
|
* aggregator.ts
|
||||||
|
* Merges all run JSON files into aggregated.json, grouped by prompt generation (SHA fingerprint).
|
||||||
|
*/
|
||||||
|
|
||||||
|
import fs from 'fs';
|
||||||
|
import path from 'path';
|
||||||
|
import crypto from 'crypto';
|
||||||
|
import type { RunMetrics, AggregatedMetrics, CohortMetrics, PromptVersions } from './types.js';
|
||||||
|
|
||||||
|
// ── Helpers ───────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export function avg(values: (number | null | undefined)[]): number | null {
|
||||||
|
const valid = values.filter((v): v is number => v != null && !isNaN(v));
|
||||||
|
if (valid.length === 0) return null;
|
||||||
|
return valid.reduce((a, b) => a + b, 0) / valid.length;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function rate(values: (boolean | null | undefined)[]): number | null {
|
||||||
|
const valid = values.filter((v): v is boolean => v != null);
|
||||||
|
if (valid.length === 0) return null;
|
||||||
|
return valid.filter(v => v).length / valid.length;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Build a stable cohort key from the sorted prompt_versions object.
|
||||||
|
* Two runs with identical prompt files get the same cohort key.
|
||||||
|
*/
|
||||||
|
export function buildCohortKey(promptVersions: PromptVersions): string {
|
||||||
|
// Filter out missing sentinels so old runs (4 fields) keep their original key
|
||||||
|
const entries = Object.entries(promptVersions)
|
||||||
|
.filter(([, v]) => v !== 'sha256:missing')
|
||||||
|
.sort(([a], [b]) => a.localeCompare(b));
|
||||||
|
const str = JSON.stringify(Object.fromEntries(entries));
|
||||||
|
return crypto.createHash('sha256').update(str).digest('hex').slice(0, 12);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Backfill missing PromptVersions fields for old run files (pre-v1.1).
|
||||||
|
*/
|
||||||
|
export function backfillPromptVersions(pv: PromptVersions): PromptVersions {
|
||||||
|
return {
|
||||||
|
plan: pv.plan ?? 'sha256:missing',
|
||||||
|
revise_plan: pv.revise_plan ?? 'sha256:missing',
|
||||||
|
document: pv.document ?? 'sha256:missing',
|
||||||
|
implement: pv.implement ?? 'sha256:missing',
|
||||||
|
finalize: pv.finalize ?? 'sha256:missing',
|
||||||
|
init: pv.init ?? 'sha256:missing',
|
||||||
|
init_update: pv.init_update ?? 'sha256:missing',
|
||||||
|
quick_task: pv.quick_task ?? 'sha256:missing',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Load run files ────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export function loadRunFiles(runsDir: string): RunMetrics[] {
|
||||||
|
if (!fs.existsSync(runsDir)) return [];
|
||||||
|
|
||||||
|
const files = fs.readdirSync(runsDir)
|
||||||
|
.filter(f => f.endsWith('.json') && f.startsWith('run-'))
|
||||||
|
.sort();
|
||||||
|
|
||||||
|
const runs: RunMetrics[] = [];
|
||||||
|
for (const file of files) {
|
||||||
|
try {
|
||||||
|
const content = fs.readFileSync(path.join(runsDir, file), 'utf8');
|
||||||
|
const data = JSON.parse(content) as RunMetrics;
|
||||||
|
data.prompt_versions = backfillPromptVersions(data.prompt_versions);
|
||||||
|
runs.push(data);
|
||||||
|
} catch (err) {
|
||||||
|
console.warn(`Warning: could not parse ${file}: ${err}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return runs;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Build cohort metrics ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
function buildCohort(runs: RunMetrics[], cohortKey: string): CohortMetrics {
|
||||||
|
const runIds = runs.map(r => r.run_id);
|
||||||
|
const timestamps = runs
|
||||||
|
.flatMap(r => [r.project.started_at, r.project.completed_at])
|
||||||
|
.filter((t): t is string => t != null)
|
||||||
|
.sort();
|
||||||
|
|
||||||
|
// Step 1 averages
|
||||||
|
const avgConfidence = avg(runs.map(r => r.step1_plan.final_confidence));
|
||||||
|
const avgClarification = avg(runs.map(r => r.step1_plan.clarification_rounds));
|
||||||
|
const avgVerifGaps = avg(runs.map(r => r.step1_plan.verification_gaps_found));
|
||||||
|
const avgFRCount = avg(runs.map(r => r.step1_plan.functional_requirements_count));
|
||||||
|
const avgNFRCount = avg(runs.map(r => r.step1_plan.non_functional_requirements_count));
|
||||||
|
const avgRiskCount = avg(runs.map(r => r.step1_plan.risk_count));
|
||||||
|
const avgPhaseStep1 = avg(runs.map(r => r.step1_plan.phase_count));
|
||||||
|
|
||||||
|
// Step 2 averages
|
||||||
|
const avgTotalTasks = avg(runs.map(r => r.step2_document.total_tasks));
|
||||||
|
const avgPhaseStep2 = avg(runs.map(r => r.step2_document.phase_count));
|
||||||
|
const avgParallelGroups = avg(runs.map(r => r.step2_document.parallel_groups_identified));
|
||||||
|
const avgReqCoverage = avg(runs.map(r => r.step2_document.requirement_coverage_percent));
|
||||||
|
const avgVerifItems = avg(runs.map(r => r.step2_document.verification_items_added));
|
||||||
|
|
||||||
|
// Step 3 (loop only)
|
||||||
|
const loopRuns = runs.filter(r => r.step3_implement.used_loop_mode);
|
||||||
|
const avgCompletionRate = avg(loopRuns.map(r => r.step3_implement.task_completion_rate));
|
||||||
|
const avgBlockerCount = avg(loopRuns.map(r => r.step3_implement.blocker_count));
|
||||||
|
const avgTotalIter = avg(loopRuns.map(r => r.step3_implement.total_iterations));
|
||||||
|
const avgIterDuration = avg(loopRuns.map(r => r.step3_implement.avg_iteration_duration_ms));
|
||||||
|
const avgMarkerSuccess = avg(loopRuns.map(r => r.step3_implement.completion_marker_success_rate));
|
||||||
|
|
||||||
|
// Step 4 averages
|
||||||
|
const avgCompletionAtAudit = avg(runs.map(r => r.step4_finalize.completion_rate_at_audit));
|
||||||
|
const avgVerifFailures = avg(runs.map(r => r.step4_finalize.verification_failures_found));
|
||||||
|
const avgDocUpdates = avg(runs.map(r => r.step4_finalize.documentation_updates_needed));
|
||||||
|
const archivalRate = rate(runs.map(r => r.step4_finalize.archival_succeeded));
|
||||||
|
|
||||||
|
// User feedback
|
||||||
|
const feedbackRuns = runs.filter(r => r.user_feedback != null);
|
||||||
|
const avgUserRating = avg(feedbackRuns.map(r => r.user_feedback!.overall_rating));
|
||||||
|
const feedbackCount = feedbackRuns.length;
|
||||||
|
|
||||||
|
return {
|
||||||
|
cohort_key: cohortKey,
|
||||||
|
prompt_versions: runs[0].prompt_versions,
|
||||||
|
run_count: runs.length,
|
||||||
|
run_ids: runIds,
|
||||||
|
first_seen: timestamps[0] ?? runs[0].run_id,
|
||||||
|
last_seen: timestamps[timestamps.length - 1] ?? runs[runs.length - 1].run_id,
|
||||||
|
|
||||||
|
avg_confidence: avgConfidence,
|
||||||
|
avg_clarification_rounds: avgClarification,
|
||||||
|
avg_verification_gaps_found: avgVerifGaps,
|
||||||
|
avg_functional_requirements_count: avgFRCount,
|
||||||
|
avg_non_functional_requirements_count: avgNFRCount,
|
||||||
|
avg_risk_count: avgRiskCount,
|
||||||
|
avg_phase_count_step1: avgPhaseStep1,
|
||||||
|
|
||||||
|
avg_total_tasks: avgTotalTasks,
|
||||||
|
avg_phase_count_step2: avgPhaseStep2,
|
||||||
|
avg_parallel_groups: avgParallelGroups,
|
||||||
|
avg_requirement_coverage_percent: avgReqCoverage,
|
||||||
|
avg_verification_items_added: avgVerifItems,
|
||||||
|
|
||||||
|
loop_run_count: loopRuns.length,
|
||||||
|
avg_task_completion_rate: avgCompletionRate,
|
||||||
|
avg_blocker_count: avgBlockerCount,
|
||||||
|
avg_total_iterations: avgTotalIter,
|
||||||
|
avg_iteration_duration_ms: avgIterDuration,
|
||||||
|
avg_completion_marker_success_rate: avgMarkerSuccess,
|
||||||
|
|
||||||
|
avg_completion_rate_at_audit: avgCompletionAtAudit,
|
||||||
|
avg_verification_failures_found: avgVerifFailures,
|
||||||
|
avg_documentation_updates_needed: avgDocUpdates,
|
||||||
|
archival_success_rate: archivalRate,
|
||||||
|
|
||||||
|
avg_user_rating: avgUserRating,
|
||||||
|
feedback_count: feedbackCount,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Main aggregator ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export function aggregate(runsDir: string, outputPath: string): AggregatedMetrics {
|
||||||
|
const runs = loadRunFiles(runsDir);
|
||||||
|
|
||||||
|
// Group by cohort key
|
||||||
|
const cohortMap = new Map<string, RunMetrics[]>();
|
||||||
|
for (const run of runs) {
|
||||||
|
const key = buildCohortKey(run.prompt_versions);
|
||||||
|
if (!cohortMap.has(key)) cohortMap.set(key, []);
|
||||||
|
cohortMap.get(key)!.push(run);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sort cohorts by first seen
|
||||||
|
const cohorts: CohortMetrics[] = [];
|
||||||
|
for (const [key, cohortRuns] of cohortMap) {
|
||||||
|
cohorts.push(buildCohort(cohortRuns, key));
|
||||||
|
}
|
||||||
|
cohorts.sort((a, b) => a.first_seen.localeCompare(b.first_seen));
|
||||||
|
|
||||||
|
// Determine current cohort (most recent)
|
||||||
|
const currentCohortKey = cohorts.length > 0
|
||||||
|
? cohorts[cohorts.length - 1].cohort_key
|
||||||
|
: null;
|
||||||
|
|
||||||
|
const aggregated: AggregatedMetrics = {
|
||||||
|
schema_version: '1.0',
|
||||||
|
last_updated: new Date().toISOString(),
|
||||||
|
total_runs: runs.length,
|
||||||
|
cohorts,
|
||||||
|
current_cohort_key: currentCohortKey,
|
||||||
|
};
|
||||||
|
|
||||||
|
// Write to output path
|
||||||
|
const dir = path.dirname(outputPath);
|
||||||
|
fs.mkdirSync(dir, { recursive: true });
|
||||||
|
fs.writeFileSync(outputPath, JSON.stringify(aggregated, null, 2), 'utf8');
|
||||||
|
|
||||||
|
return aggregated;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function loadAggregated(outputPath: string): AggregatedMetrics | null {
|
||||||
|
try {
|
||||||
|
const content = fs.readFileSync(outputPath, 'utf8');
|
||||||
|
return JSON.parse(content) as AggregatedMetrics;
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Import a single run JSON from another project into the local runs dir.
|
||||||
|
* Returns true if imported, false if already present.
|
||||||
|
*/
|
||||||
|
export function importRun(runJsonPath: string, runsDir: string): boolean {
|
||||||
|
const content = fs.readFileSync(runJsonPath, 'utf8');
|
||||||
|
const run = JSON.parse(content) as RunMetrics;
|
||||||
|
run.prompt_versions = backfillPromptVersions(run.prompt_versions);
|
||||||
|
const destPath = path.join(runsDir, `${run.run_id}.json`);
|
||||||
|
|
||||||
|
if (fs.existsSync(destPath)) {
|
||||||
|
return false; // Already imported
|
||||||
|
}
|
||||||
|
|
||||||
|
fs.mkdirSync(runsDir, { recursive: true });
|
||||||
|
fs.writeFileSync(destPath, JSON.stringify(run, null, 2), 'utf8');
|
||||||
|
return true;
|
||||||
|
}
|
||||||
@@ -0,0 +1,128 @@
|
|||||||
|
/**
|
||||||
|
* analyzer.ts
|
||||||
|
* Reads aggregated.json + prompt file contents, builds analysis prompt, invokes AI.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import fs from 'fs';
|
||||||
|
import path from 'path';
|
||||||
|
import type { AggregatedMetrics } from './types.js';
|
||||||
|
import { invokeLLM, type AgentType } from './invoke-llm.js';
|
||||||
|
|
||||||
|
const ANALYZE_PROMPT_PATH = new URL('../src/prompts/analyze.md', import.meta.url).pathname
|
||||||
|
.replace(/^\/([A-Za-z]:)/, '$1'); // Fix Windows path
|
||||||
|
|
||||||
|
// ── Helpers ───────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
function readPromptFiles(plan2codeRoot: string): Record<string, string> {
|
||||||
|
const srcDir = path.join(plan2codeRoot, 'src');
|
||||||
|
const promptFiles = [
|
||||||
|
'plan2code-1--plan.md',
|
||||||
|
'plan2code-1b--revise-plan.md',
|
||||||
|
'plan2code-2--document.md',
|
||||||
|
'plan2code-3--implement.md',
|
||||||
|
'plan2code-4--finalize.md',
|
||||||
|
'plan2code---init.md',
|
||||||
|
'plan2code---init-update.md',
|
||||||
|
'plan2code---quick-task.md',
|
||||||
|
];
|
||||||
|
|
||||||
|
const contents: Record<string, string> = {};
|
||||||
|
for (const file of promptFiles) {
|
||||||
|
try {
|
||||||
|
contents[file] = fs.readFileSync(path.join(srcDir, file), 'utf8');
|
||||||
|
} catch {
|
||||||
|
contents[file] = '(file not found)';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return contents;
|
||||||
|
}
|
||||||
|
|
||||||
|
function interpolate(template: string, vars: Record<string, string>): string {
|
||||||
|
let result = template;
|
||||||
|
for (const [key, value] of Object.entries(vars)) {
|
||||||
|
result = result.replaceAll(`{{${key}}}`, value);
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
function generateDiagnosisId(): string {
|
||||||
|
const now = new Date();
|
||||||
|
const ts = now.toISOString().replace(/[-:T.Z]/g, '').slice(0, 14);
|
||||||
|
return `diag-${ts}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Main analyzer ─────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export interface AnalyzerOptions {
|
||||||
|
aggregatedPath: string; // Path to aggregated.json
|
||||||
|
plan2codeRoot: string; // Path to plan2code repo root
|
||||||
|
proposalsDir: string; // Where to save diagnosis output
|
||||||
|
model?: string; // AI model to use (default: claude-opus-4-6)
|
||||||
|
agent?: AgentType; // Agent to use (default: claude-code)
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function runAnalysis(opts: AnalyzerOptions): Promise<string> {
|
||||||
|
const { aggregatedPath, plan2codeRoot, proposalsDir, model = 'claude-opus-4-6', agent = 'claude-code' } = opts;
|
||||||
|
|
||||||
|
// Load aggregated metrics
|
||||||
|
let aggregated: AggregatedMetrics | null = null;
|
||||||
|
try {
|
||||||
|
aggregated = JSON.parse(fs.readFileSync(aggregatedPath, 'utf8')) as AggregatedMetrics;
|
||||||
|
} catch {
|
||||||
|
throw new Error(`Could not read aggregated metrics at ${aggregatedPath}. Run "Collect metrics" and "Import run data" first.`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (aggregated.total_runs === 0) {
|
||||||
|
throw new Error('No runs in aggregated metrics. Collect and import some runs first.');
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read prompt files
|
||||||
|
const promptContents = readPromptFiles(plan2codeRoot);
|
||||||
|
|
||||||
|
// Format for interpolation
|
||||||
|
const promptContentsStr = Object.entries(promptContents)
|
||||||
|
.map(([file, content]) => `## ${file}\n\n${content}`)
|
||||||
|
.join('\n\n---\n\n');
|
||||||
|
|
||||||
|
const aggregatedStr = JSON.stringify(aggregated, null, 2);
|
||||||
|
|
||||||
|
// Load analyze prompt template
|
||||||
|
let analyzeTemplate: string;
|
||||||
|
try {
|
||||||
|
analyzeTemplate = fs.readFileSync(ANALYZE_PROMPT_PATH, 'utf8');
|
||||||
|
} catch {
|
||||||
|
// Fallback: look relative to cwd
|
||||||
|
const altPath = path.join(process.cwd(), 'src', 'prompts', 'analyze.md');
|
||||||
|
analyzeTemplate = fs.readFileSync(altPath, 'utf8');
|
||||||
|
}
|
||||||
|
|
||||||
|
const fullPrompt = interpolate(analyzeTemplate, {
|
||||||
|
aggregatedMetrics: aggregatedStr,
|
||||||
|
promptContents: promptContentsStr,
|
||||||
|
});
|
||||||
|
|
||||||
|
// Invoke Claude
|
||||||
|
console.log(`\nInvoking AI analysis (model: ${model})...`);
|
||||||
|
console.log('This may take a minute...\n');
|
||||||
|
|
||||||
|
let diagnosisContent: string;
|
||||||
|
try {
|
||||||
|
diagnosisContent = await invokeLLM({
|
||||||
|
prompt: fullPrompt,
|
||||||
|
model,
|
||||||
|
agent,
|
||||||
|
timeout: 300_000,
|
||||||
|
});
|
||||||
|
} catch (err) {
|
||||||
|
throw new Error(`AI invocation failed: ${err instanceof Error ? err.message : String(err)}\n\nMake sure the ${agent} CLI is installed and authenticated.`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Save diagnosis
|
||||||
|
const diagId = generateDiagnosisId();
|
||||||
|
fs.mkdirSync(proposalsDir, { recursive: true });
|
||||||
|
const diagPath = path.join(proposalsDir, `${diagId}-diagnosis.md`);
|
||||||
|
fs.writeFileSync(diagPath, diagnosisContent, 'utf8');
|
||||||
|
|
||||||
|
console.log(`Diagnosis saved to: ${diagPath}`);
|
||||||
|
return diagPath;
|
||||||
|
}
|
||||||
@@ -0,0 +1,201 @@
|
|||||||
|
/**
|
||||||
|
* applier.ts
|
||||||
|
* Interactive review of a PromptProposal: display colored diff per edit,
|
||||||
|
* user approves/rejects each edit, apply approved edits to src files.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import fs from 'fs';
|
||||||
|
import path from 'path';
|
||||||
|
import { confirm } from '@inquirer/prompts';
|
||||||
|
import chalk from 'chalk';
|
||||||
|
import type { PromptEdit, PromptProposal } from './types.js';
|
||||||
|
import { validateEdit } from './improver.js';
|
||||||
|
|
||||||
|
const CHAR_LIMIT = 11_000;
|
||||||
|
|
||||||
|
// ── Diff display ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
function displayEditDiff(edit: PromptEdit, index: number, total: number): void {
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.bold(`─── Edit ${index + 1} of ${total} ───────────────────────────────────`));
|
||||||
|
console.log(chalk.cyan(`File: ${edit.file}`));
|
||||||
|
console.log(chalk.gray(`Rationale: ${edit.rationale}`));
|
||||||
|
console.log(chalk.gray(`Expected impact: ${edit.expected_metric_impact}`));
|
||||||
|
console.log(chalk.gray(`Char impact: ${edit.char_count_before} → ${edit.char_count_after} (${edit.char_count_delta >= 0 ? '+' : ''}${edit.char_count_delta})`));
|
||||||
|
|
||||||
|
// Show char count status
|
||||||
|
if (edit.char_count_after > CHAR_LIMIT) {
|
||||||
|
console.log(chalk.red(`⚠ WARNING: Would exceed ${CHAR_LIMIT} char limit!`));
|
||||||
|
} else {
|
||||||
|
const headroom = CHAR_LIMIT - edit.char_count_after;
|
||||||
|
console.log(chalk.green(`✓ Char count OK (${headroom} chars headroom after edit)`));
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.bold('── REMOVED (old_text) ──'));
|
||||||
|
|
||||||
|
// Show old text with line-level context
|
||||||
|
const oldLines = edit.old_text.split('\n');
|
||||||
|
for (const line of oldLines) {
|
||||||
|
console.log(chalk.red('- ') + chalk.red(line));
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.bold('── ADDED (new_text) ──'));
|
||||||
|
|
||||||
|
const newLines = edit.new_text.split('\n');
|
||||||
|
for (const line of newLines) {
|
||||||
|
console.log(chalk.green('+ ') + chalk.green(line));
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log();
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Apply edit to file ────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
function applyEdit(edit: PromptEdit, plan2codeRoot: string): boolean {
|
||||||
|
// Reject path traversal attempts
|
||||||
|
if (edit.file.includes('..') || path.isAbsolute(edit.file)) {
|
||||||
|
console.error(chalk.red(`✗ Rejected: "${edit.file}" contains path traversal or absolute path`));
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
const filePath = path.join(plan2codeRoot, 'src', edit.file);
|
||||||
|
try {
|
||||||
|
let content = fs.readFileSync(filePath, 'utf8');
|
||||||
|
if (!content.includes(edit.old_text)) {
|
||||||
|
console.error(chalk.red(`✗ Cannot apply: old_text not found in ${edit.file} (may have been modified by a previous edit)`));
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
content = content.replace(edit.old_text, edit.new_text);
|
||||||
|
|
||||||
|
// Post-apply char count check
|
||||||
|
if (content.length > CHAR_LIMIT) {
|
||||||
|
console.error(chalk.red(`✗ Cannot apply: would exceed ${CHAR_LIMIT} char limit (${content.length} chars)`));
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
fs.writeFileSync(filePath, content, 'utf8');
|
||||||
|
console.log(chalk.green(`✓ Applied edit to ${edit.file} (now ${content.length} chars)`));
|
||||||
|
return true;
|
||||||
|
} catch (err) {
|
||||||
|
console.error(chalk.red(`✗ Failed to apply edit: ${err instanceof Error ? err.message : String(err)}`));
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Main applier ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export interface ApplierOptions {
|
||||||
|
proposalPath: string;
|
||||||
|
plan2codeRoot: string;
|
||||||
|
proposalsDir: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ApplierResult {
|
||||||
|
approved: number;
|
||||||
|
rejected: number;
|
||||||
|
applied: number;
|
||||||
|
failed: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function reviewAndApply(opts: ApplierOptions): Promise<ApplierResult> {
|
||||||
|
const { proposalPath, plan2codeRoot, proposalsDir } = opts;
|
||||||
|
|
||||||
|
// Load proposal
|
||||||
|
let proposal: PromptProposal;
|
||||||
|
try {
|
||||||
|
proposal = JSON.parse(fs.readFileSync(proposalPath, 'utf8')) as PromptProposal;
|
||||||
|
} catch {
|
||||||
|
throw new Error(`Could not read proposal at ${proposalPath}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (proposal.proposals.length === 0) {
|
||||||
|
console.log(chalk.yellow('No edits in this proposal.'));
|
||||||
|
return { approved: 0, rejected: 0, applied: 0, failed: 0 };
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.bold.cyan('=== Plan2Code Prompt Improvement Review ==='));
|
||||||
|
console.log(chalk.gray(`Proposal: ${proposal.proposal_id}`));
|
||||||
|
console.log(chalk.gray(`Created: ${proposal.created_at}`));
|
||||||
|
console.log(chalk.gray(`Based on: ${proposal.based_on_runs.length} run(s)`));
|
||||||
|
console.log(chalk.gray(`Edits: ${proposal.proposals.length}`));
|
||||||
|
|
||||||
|
// Re-validate all edits against current file state
|
||||||
|
const promptContents: Record<string, string> = {};
|
||||||
|
const srcDir = path.join(plan2codeRoot, 'src');
|
||||||
|
for (const edit of proposal.proposals) {
|
||||||
|
if (!promptContents[edit.file]) {
|
||||||
|
try {
|
||||||
|
promptContents[edit.file] = fs.readFileSync(path.join(srcDir, edit.file), 'utf8');
|
||||||
|
} catch {
|
||||||
|
promptContents[edit.file] = '';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let approved = 0;
|
||||||
|
let rejected = 0;
|
||||||
|
let applied = 0;
|
||||||
|
let failed = 0;
|
||||||
|
|
||||||
|
for (let i = 0; i < proposal.proposals.length; i++) {
|
||||||
|
const edit = proposal.proposals[i];
|
||||||
|
|
||||||
|
// Re-validate
|
||||||
|
const validation = validateEdit(edit, promptContents);
|
||||||
|
if (!validation.valid) {
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.red(`✗ Edit ${i + 1} is no longer valid (files may have changed):`));
|
||||||
|
for (const err of validation.errors) {
|
||||||
|
console.log(chalk.red(` - ${err}`));
|
||||||
|
}
|
||||||
|
rejected++;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
displayEditDiff(edit, i, proposal.proposals.length);
|
||||||
|
|
||||||
|
const approve = await confirm({
|
||||||
|
message: `Apply this edit to ${edit.file}?`,
|
||||||
|
default: true,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!approve) {
|
||||||
|
console.log(chalk.gray('Skipped.'));
|
||||||
|
rejected++;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
approved++;
|
||||||
|
const success = applyEdit(edit, plan2codeRoot);
|
||||||
|
if (success) {
|
||||||
|
applied++;
|
||||||
|
// Update in-memory content to reflect the edit
|
||||||
|
promptContents[edit.file] = promptContents[edit.file].replace(edit.old_text, edit.new_text);
|
||||||
|
} else {
|
||||||
|
failed++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Update proposal status
|
||||||
|
proposal.status = applied > 0 ? 'applied' : 'rejected';
|
||||||
|
fs.writeFileSync(proposalPath, JSON.stringify(proposal, null, 2), 'utf8');
|
||||||
|
|
||||||
|
// Summary
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.bold('─── Review Complete ──────────────────────────────────'));
|
||||||
|
console.log(`Approved: ${chalk.green(String(approved))} Rejected: ${chalk.red(String(rejected))} Applied: ${chalk.green(String(applied))} Failed: ${chalk.red(String(failed))}`);
|
||||||
|
|
||||||
|
if (applied > 0) {
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.bold.cyan('Next steps — create a PR with your changes:'));
|
||||||
|
console.log(chalk.gray(' git add src/'));
|
||||||
|
console.log(chalk.gray(` git commit -m "metrics: apply prompt improvements (${proposal.proposal_id})"`));
|
||||||
|
console.log(chalk.gray(' git push -u origin HEAD'));
|
||||||
|
console.log(chalk.gray(' gh pr create --title "metrics: apply gen N improvements"'));
|
||||||
|
}
|
||||||
|
|
||||||
|
return { approved, rejected, applied, failed };
|
||||||
|
}
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
import { runCLI } from '../cli.js';
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
try {
|
||||||
|
await runCLI();
|
||||||
|
process.exit(0);
|
||||||
|
} catch (err) {
|
||||||
|
if (err instanceof Error && err.message.includes('User force closed')) {
|
||||||
|
// Ctrl+C — exit cleanly
|
||||||
|
process.exit(0);
|
||||||
|
}
|
||||||
|
console.error(err instanceof Error ? err.message : String(err));
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
main();
|
||||||
@@ -0,0 +1,645 @@
|
|||||||
|
/**
|
||||||
|
* cli.ts
|
||||||
|
* 100% interactive menu-driven CLI for plan2code-metrics.
|
||||||
|
* No flags — all inputs collected via @inquirer/prompts.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import fs from 'fs';
|
||||||
|
import path from 'path';
|
||||||
|
import { select, input, confirm } from '@inquirer/prompts';
|
||||||
|
import chalk from 'chalk';
|
||||||
|
import ora from 'ora';
|
||||||
|
import { collectRun } from './collector.js';
|
||||||
|
import { aggregate, loadAggregated, importRun, loadRunFiles } from './aggregator.js';
|
||||||
|
import { runAnalysis } from './analyzer.js';
|
||||||
|
import { generateImprovement } from './improver.js';
|
||||||
|
import { reviewAndApply } from './applier.js';
|
||||||
|
import type { AggregatedMetrics, CohortMetrics } from './types.js';
|
||||||
|
import { METRIC_TARGETS } from './types.js';
|
||||||
|
import { AGENTS, type AgentType } from './invoke-llm.js';
|
||||||
|
|
||||||
|
// ── Defaults ──────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
const DEFAULT_METRICS_DIR = '.plan2code-metrics';
|
||||||
|
const DEFAULT_RUNS_SUBDIR = 'runs';
|
||||||
|
const DEFAULT_AGGREGATED_FILE = 'aggregated.json';
|
||||||
|
const DEFAULT_PROPOSALS_SUBDIR = 'proposals';
|
||||||
|
|
||||||
|
function getMetricsDirs(baseDir = process.cwd()) {
|
||||||
|
const metricsDir = path.join(baseDir, DEFAULT_METRICS_DIR);
|
||||||
|
return {
|
||||||
|
metricsDir,
|
||||||
|
runsDir: path.join(metricsDir, DEFAULT_RUNS_SUBDIR),
|
||||||
|
aggregatedPath: path.join(metricsDir, DEFAULT_AGGREGATED_FILE),
|
||||||
|
proposalsDir: path.join(metricsDir, DEFAULT_PROPOSALS_SUBDIR),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// Detect plan2code root (either this directory or parent directories)
|
||||||
|
function detectPlan2CodeRoot(): string | null {
|
||||||
|
let dir = process.cwd();
|
||||||
|
for (let i = 0; i < 5; i++) {
|
||||||
|
const srcDir = path.join(dir, 'src');
|
||||||
|
if (
|
||||||
|
fs.existsSync(path.join(srcDir, 'plan2code-1--plan.md')) &&
|
||||||
|
fs.existsSync(path.join(srcDir, 'plan2code-2--document.md'))
|
||||||
|
) {
|
||||||
|
return dir;
|
||||||
|
}
|
||||||
|
const parent = path.dirname(dir);
|
||||||
|
if (parent === dir) break;
|
||||||
|
dir = parent;
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Detect plan2code version
|
||||||
|
function detectPlan2CodeVersion(plan2codeRoot: string): string {
|
||||||
|
try {
|
||||||
|
const versionPath = path.join(plan2codeRoot, 'version.json');
|
||||||
|
const content = JSON.parse(fs.readFileSync(versionPath, 'utf8'));
|
||||||
|
return content.version ?? '0.0.0';
|
||||||
|
} catch {
|
||||||
|
try {
|
||||||
|
const pkg = JSON.parse(fs.readFileSync(path.join(plan2codeRoot, 'package.json'), 'utf8'));
|
||||||
|
return pkg.version ?? '0.0.0';
|
||||||
|
} catch {
|
||||||
|
return '0.0.0';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Metric health helpers ─────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
function metricStatus(key: string, value: number | null): string {
|
||||||
|
if (value == null) return chalk.gray('N/A');
|
||||||
|
const target = METRIC_TARGETS[key as keyof typeof METRIC_TARGETS];
|
||||||
|
if (!target) return chalk.white(String(Math.round(value * 100) / 100));
|
||||||
|
|
||||||
|
const ok = target.direction === 'gte' ? value >= target.target : value <= target.target;
|
||||||
|
const formatted = Number.isInteger(value) ? String(value) : value.toFixed(2);
|
||||||
|
return ok ? chalk.green(`✓ ${formatted}`) : chalk.red(`✗ ${formatted} (target: ${target.direction === 'gte' ? '≥' : '≤'}${target.target})`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Agent + model selection ───────────────────────────────────────────────────
|
||||||
|
|
||||||
|
async function selectAgentAndModel(): Promise<{ agent: AgentType; model: string }> {
|
||||||
|
const agentChoice = await select({
|
||||||
|
message: 'Which AI agent?',
|
||||||
|
choices: Object.values(AGENTS).map(a => ({ name: a.displayName, value: a.name })),
|
||||||
|
});
|
||||||
|
|
||||||
|
const agentDef = AGENTS[agentChoice];
|
||||||
|
const model = await select({
|
||||||
|
message: 'AI model:',
|
||||||
|
choices: agentDef.models.map(m => ({ name: m.label, value: m.value })),
|
||||||
|
});
|
||||||
|
|
||||||
|
return { agent: agentChoice, model };
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Flow: Collect metrics ─────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
async function flowCollect(): Promise<void> {
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.bold.cyan('── Collect Metrics ──'));
|
||||||
|
|
||||||
|
const sourceChoice = await select({
|
||||||
|
message: 'Where is the completed project spec?',
|
||||||
|
choices: [
|
||||||
|
{ name: 'Active spec directory (specs/<feature-name>/)', value: 'active' },
|
||||||
|
{ name: 'Archived spec (specs--completed/<feature-name>/)', value: 'archived' },
|
||||||
|
{ name: 'Custom path', value: 'custom' },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
|
||||||
|
let specDir: string;
|
||||||
|
let projectName: string;
|
||||||
|
|
||||||
|
if (sourceChoice === 'active' || sourceChoice === 'archived') {
|
||||||
|
const baseSubdir = sourceChoice === 'active' ? 'specs' : 'specs--completed';
|
||||||
|
const baseDir = path.join(process.cwd(), baseSubdir);
|
||||||
|
|
||||||
|
if (!fs.existsSync(baseDir)) {
|
||||||
|
console.log(chalk.red(`No ${baseSubdir}/ directory found in ${process.cwd()}`));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const entries = fs.readdirSync(baseDir)
|
||||||
|
.filter(f => fs.statSync(path.join(baseDir, f)).isDirectory());
|
||||||
|
|
||||||
|
if (entries.length === 0) {
|
||||||
|
console.log(chalk.red(`No spec directories found in ${baseSubdir}/`));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const chosen = await select({
|
||||||
|
message: 'Select spec directory:',
|
||||||
|
choices: entries.map(e => ({ name: e, value: e })),
|
||||||
|
});
|
||||||
|
|
||||||
|
specDir = path.join(baseDir, chosen);
|
||||||
|
projectName = chosen;
|
||||||
|
} else {
|
||||||
|
specDir = await input({
|
||||||
|
message: 'Enter full path to spec directory:',
|
||||||
|
validate: (v) => fs.existsSync(v) ? true : 'Directory not found',
|
||||||
|
});
|
||||||
|
projectName = await input({
|
||||||
|
message: 'Project name (for metrics label):',
|
||||||
|
default: path.basename(specDir),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Detect plan2code root
|
||||||
|
const plan2codeRoot = detectPlan2CodeRoot() ?? process.cwd();
|
||||||
|
const plan2codeVersion = detectPlan2CodeVersion(plan2codeRoot);
|
||||||
|
|
||||||
|
const { runsDir, aggregatedPath } = getMetricsDirs();
|
||||||
|
|
||||||
|
const spinner = ora('Collecting metrics from project artifacts...').start();
|
||||||
|
try {
|
||||||
|
const metrics = await collectRun({
|
||||||
|
specDir,
|
||||||
|
projectName,
|
||||||
|
plan2codeRoot,
|
||||||
|
plan2codeVersion,
|
||||||
|
outputDir: runsDir,
|
||||||
|
});
|
||||||
|
|
||||||
|
spinner.succeed(`Metrics collected: ${metrics.run_id}`);
|
||||||
|
console.log(chalk.gray(` Saved to: ${runsDir}/${metrics.run_id}.json`));
|
||||||
|
|
||||||
|
// Re-aggregate
|
||||||
|
aggregate(runsDir, aggregatedPath);
|
||||||
|
console.log(chalk.gray(' Aggregated metrics updated.'));
|
||||||
|
|
||||||
|
// Show summary
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.bold('Collected:'));
|
||||||
|
console.log(` Step 1 (Plan): ${metrics.step1_plan.present ? chalk.green('✓') : chalk.gray('—')}`);
|
||||||
|
console.log(` Step 2 (Document): ${metrics.step2_document.present ? chalk.green('✓') : chalk.gray('—')}`);
|
||||||
|
console.log(` Step 3 (Implement): ${metrics.step3_implement.present ? (metrics.step3_implement.used_loop_mode ? chalk.green('✓ (loop)') : chalk.yellow('✓ (manual)')) : chalk.gray('—')}`);
|
||||||
|
console.log(` Step 4 (Finalize): ${metrics.step4_finalize.present ? chalk.green('✓') : chalk.gray('—')}`);
|
||||||
|
console.log(` User Feedback: ${metrics.user_feedback ? chalk.green(`✓ (${metrics.user_feedback.overall_rating}/10)`) : chalk.gray('—')}`);
|
||||||
|
|
||||||
|
// Offer to collect feedback interactively if not present
|
||||||
|
if (!metrics.user_feedback) {
|
||||||
|
console.log();
|
||||||
|
const wantFeedback = await confirm({
|
||||||
|
message: 'No user feedback found in overview.md. Would you like to provide feedback now?',
|
||||||
|
default: false,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (wantFeedback) {
|
||||||
|
const rating = await input({
|
||||||
|
message: 'Overall rating (1-10):',
|
||||||
|
validate: (v) => {
|
||||||
|
const n = parseInt(v, 10);
|
||||||
|
return (n >= 1 && n <= 10) ? true : 'Must be a number between 1 and 10';
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const reason = await input({ message: 'Rating reason:' });
|
||||||
|
const wentWell = await input({ message: 'What went well?' });
|
||||||
|
const wentPoorly = await input({ message: 'What went poorly?' });
|
||||||
|
|
||||||
|
// Write the feedback table into overview.md
|
||||||
|
const overviewPath = path.join(specDir, 'overview.md');
|
||||||
|
let overviewContent = '';
|
||||||
|
try {
|
||||||
|
overviewContent = fs.readFileSync(overviewPath, 'utf8');
|
||||||
|
} catch { /* empty */ }
|
||||||
|
|
||||||
|
// Escape pipe characters in user text to prevent table parsing issues
|
||||||
|
const esc = (s: string) => s.replace(/\|/g, '\\|');
|
||||||
|
|
||||||
|
const feedbackTable = [
|
||||||
|
'',
|
||||||
|
'## User Feedback',
|
||||||
|
'| Field | Value |',
|
||||||
|
'|-------|-------|',
|
||||||
|
`| Rating | ${rating} |`,
|
||||||
|
`| Reason | ${esc(reason)} |`,
|
||||||
|
`| Went Well | ${esc(wentWell)} |`,
|
||||||
|
`| Went Poorly | ${esc(wentPoorly)} |`,
|
||||||
|
'',
|
||||||
|
].join('\n');
|
||||||
|
|
||||||
|
fs.writeFileSync(overviewPath, overviewContent + feedbackTable, 'utf8');
|
||||||
|
console.log(chalk.green('✓ Feedback written to overview.md'));
|
||||||
|
|
||||||
|
// Delete the original run (without feedback) before re-collecting
|
||||||
|
const originalRunPath = path.join(runsDir, `${metrics.run_id}.json`);
|
||||||
|
try { fs.unlinkSync(originalRunPath); } catch { /* ignore */ }
|
||||||
|
|
||||||
|
// Re-collect and re-aggregate
|
||||||
|
const reSpinner = ora('Re-collecting metrics with feedback...').start();
|
||||||
|
const updatedMetrics = await collectRun({
|
||||||
|
specDir,
|
||||||
|
projectName,
|
||||||
|
plan2codeRoot,
|
||||||
|
plan2codeVersion,
|
||||||
|
outputDir: runsDir,
|
||||||
|
});
|
||||||
|
aggregate(runsDir, aggregatedPath);
|
||||||
|
reSpinner.succeed(`Updated: ${updatedMetrics.run_id} (rating: ${updatedMetrics.user_feedback?.overall_rating}/10)`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
spinner.fail(`Collection failed: ${err instanceof Error ? err.message : String(err)}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Flow: Import run data ─────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
async function flowImport(): Promise<void> {
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.bold.cyan('── Import Run Data ──'));
|
||||||
|
console.log(chalk.gray('Copy a run JSON from another project into this repo for aggregation.'));
|
||||||
|
console.log();
|
||||||
|
|
||||||
|
const sourcePath = await input({
|
||||||
|
message: 'Path to run JSON file (e.g., /path/to/project/.plan2code-metrics/runs/run-xxx.json):',
|
||||||
|
validate: (v) => {
|
||||||
|
if (!fs.existsSync(v)) return 'File not found';
|
||||||
|
if (!v.endsWith('.json')) return 'Must be a .json file';
|
||||||
|
return true;
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
const { runsDir, aggregatedPath } = getMetricsDirs();
|
||||||
|
|
||||||
|
try {
|
||||||
|
const imported = importRun(sourcePath, runsDir);
|
||||||
|
if (!imported) {
|
||||||
|
console.log(chalk.yellow('Run already imported (same run_id exists).'));
|
||||||
|
} else {
|
||||||
|
console.log(chalk.green(`✓ Imported ${path.basename(sourcePath)}`));
|
||||||
|
}
|
||||||
|
|
||||||
|
const spinner = ora('Re-aggregating...').start();
|
||||||
|
const aggregated = aggregate(runsDir, aggregatedPath);
|
||||||
|
spinner.succeed(`Aggregated metrics updated (${aggregated.total_runs} total runs)`);
|
||||||
|
} catch (err) {
|
||||||
|
console.error(chalk.red(`Import failed: ${err instanceof Error ? err.message : String(err)}`));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Flow: View metrics status ─────────────────────────────────────────────────
|
||||||
|
|
||||||
|
async function flowViewStatus(): Promise<void> {
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.bold.cyan('── Metrics Status & History ──'));
|
||||||
|
|
||||||
|
const { runsDir, aggregatedPath } = getMetricsDirs();
|
||||||
|
const aggregated = loadAggregated(aggregatedPath);
|
||||||
|
|
||||||
|
if (!aggregated || aggregated.total_runs === 0) {
|
||||||
|
console.log(chalk.yellow('No metrics collected yet. Use "Collect metrics" to get started.'));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.bold(`Total runs: ${aggregated.total_runs} | Generations: ${aggregated.cohorts.length}`));
|
||||||
|
console.log(chalk.gray(`Last updated: ${aggregated.last_updated}`));
|
||||||
|
|
||||||
|
for (let i = 0; i < aggregated.cohorts.length; i++) {
|
||||||
|
const cohort = aggregated.cohorts[i];
|
||||||
|
const isCurrent = cohort.cohort_key === aggregated.current_cohort_key;
|
||||||
|
const label = isCurrent ? chalk.bold.green('[CURRENT]') : '';
|
||||||
|
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.bold(`Generation ${i + 1} (sha:${cohort.cohort_key}) — ${cohort.run_count} run(s) ${label}`));
|
||||||
|
console.log(chalk.gray(` Period: ${cohort.first_seen?.slice(0, 10)} → ${cohort.last_seen?.slice(0, 10)}`));
|
||||||
|
|
||||||
|
// Step 1 metrics
|
||||||
|
if (cohort.avg_confidence != null || cohort.avg_clarification_rounds != null) {
|
||||||
|
console.log(chalk.bold(' Step 1 (Plan):'));
|
||||||
|
if (cohort.avg_confidence != null)
|
||||||
|
console.log(` avg_confidence: ${metricStatus('avg_confidence', cohort.avg_confidence)}`);
|
||||||
|
if (cohort.avg_clarification_rounds != null)
|
||||||
|
console.log(` avg_clarification_rounds: ${metricStatus('avg_clarification_rounds', cohort.avg_clarification_rounds)}`);
|
||||||
|
if (cohort.avg_verification_gaps_found != null)
|
||||||
|
console.log(` avg_verification_gaps: ${metricStatus('avg_verification_gaps_found', cohort.avg_verification_gaps_found)}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Step 2 metrics
|
||||||
|
if (cohort.avg_total_tasks != null || cohort.avg_parallel_groups != null) {
|
||||||
|
console.log(chalk.bold(' Step 2 (Document):'));
|
||||||
|
if (cohort.avg_total_tasks != null)
|
||||||
|
console.log(` avg_total_tasks: ${chalk.white(cohort.avg_total_tasks.toFixed(1))}`);
|
||||||
|
if (cohort.avg_parallel_groups != null)
|
||||||
|
console.log(` avg_parallel_groups: ${metricStatus('avg_parallel_groups', cohort.avg_parallel_groups)}`);
|
||||||
|
if (cohort.avg_verification_items_added != null)
|
||||||
|
console.log(` avg_verification_items: ${metricStatus('avg_verification_items_added', cohort.avg_verification_items_added)}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Step 3 metrics (loop only)
|
||||||
|
if (cohort.loop_run_count > 0) {
|
||||||
|
console.log(chalk.bold(` Step 3 (Implement) — ${cohort.loop_run_count} loop run(s):`));
|
||||||
|
if (cohort.avg_task_completion_rate != null)
|
||||||
|
console.log(` avg_task_completion_rate: ${metricStatus('avg_task_completion_rate', cohort.avg_task_completion_rate)}`);
|
||||||
|
if (cohort.avg_blocker_count != null)
|
||||||
|
console.log(` avg_blocker_count: ${metricStatus('avg_blocker_count', cohort.avg_blocker_count)}`);
|
||||||
|
if (cohort.avg_completion_marker_success_rate != null)
|
||||||
|
console.log(` avg_marker_success_rate: ${metricStatus('avg_completion_marker_success_rate', cohort.avg_completion_marker_success_rate)}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Step 4 metrics
|
||||||
|
if (cohort.avg_completion_rate_at_audit != null || cohort.archival_success_rate != null) {
|
||||||
|
console.log(chalk.bold(' Step 4 (Finalize):'));
|
||||||
|
if (cohort.avg_completion_rate_at_audit != null)
|
||||||
|
console.log(` avg_completion_at_audit: ${chalk.white(cohort.avg_completion_rate_at_audit.toFixed(2))}`);
|
||||||
|
if (cohort.avg_verification_failures_found != null)
|
||||||
|
console.log(` avg_verif_failures: ${metricStatus('avg_verification_failures_found', cohort.avg_verification_failures_found)}`);
|
||||||
|
if (cohort.archival_success_rate != null)
|
||||||
|
console.log(` archival_success_rate: ${metricStatus('archival_success_rate', cohort.archival_success_rate)}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// User feedback
|
||||||
|
if (cohort.feedback_count > 0) {
|
||||||
|
console.log(chalk.bold(' User Feedback:'));
|
||||||
|
console.log(` avg_user_rating: ${metricStatus('avg_user_rating', cohort.avg_user_rating)}`);
|
||||||
|
console.log(` feedback_count: ${chalk.white(String(cohort.feedback_count))}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Compare with previous generation
|
||||||
|
if (i > 0) {
|
||||||
|
const prev = aggregated.cohorts[i - 1];
|
||||||
|
const deltas: string[] = [];
|
||||||
|
if (cohort.avg_confidence != null && prev.avg_confidence != null) {
|
||||||
|
const d = cohort.avg_confidence - prev.avg_confidence;
|
||||||
|
deltas.push(`confidence ${d >= 0 ? chalk.green(`▲${d.toFixed(1)}`) : chalk.red(`▼${Math.abs(d).toFixed(1)}`)}`);
|
||||||
|
}
|
||||||
|
if (cohort.avg_task_completion_rate != null && prev.avg_task_completion_rate != null) {
|
||||||
|
const d = cohort.avg_task_completion_rate - prev.avg_task_completion_rate;
|
||||||
|
deltas.push(`completion ${d >= 0 ? chalk.green(`▲${(d * 100).toFixed(1)}%`) : chalk.red(`▼${(Math.abs(d) * 100).toFixed(1)}%`)}`);
|
||||||
|
}
|
||||||
|
if (deltas.length > 0) {
|
||||||
|
console.log(chalk.gray(` vs Gen ${i}: ${deltas.join(' ')}`));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run list
|
||||||
|
const runs = loadRunFiles(runsDir);
|
||||||
|
if (runs.length > 0) {
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.bold(`Run files (${runs.length}):`));
|
||||||
|
for (const run of runs.slice(-10)) {
|
||||||
|
console.log(chalk.gray(` ${run.run_id} ${run.project.name} v${run.plan2code_version}`));
|
||||||
|
}
|
||||||
|
if (runs.length > 10) {
|
||||||
|
console.log(chalk.gray(` ... and ${runs.length - 10} more`));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Flow: Run analysis ────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
async function flowRunAnalysis(): Promise<string | null> {
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.bold.cyan('── Run Analysis (Diagnose Weak Steps) ──'));
|
||||||
|
|
||||||
|
const { aggregatedPath, proposalsDir } = getMetricsDirs();
|
||||||
|
const plan2codeRoot = detectPlan2CodeRoot() ?? process.cwd();
|
||||||
|
|
||||||
|
const aggregated = loadAggregated(aggregatedPath);
|
||||||
|
if (!aggregated || aggregated.total_runs === 0) {
|
||||||
|
console.log(chalk.yellow('No aggregated metrics found. Collect and import runs first.'));
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (aggregated.total_runs < 3) {
|
||||||
|
const proceed = await confirm({
|
||||||
|
message: `Only ${aggregated.total_runs} run(s) available (≥3 recommended for reliable analysis). Proceed anyway?`,
|
||||||
|
default: false,
|
||||||
|
});
|
||||||
|
if (!proceed) return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
const { agent, model } = await selectAgentAndModel();
|
||||||
|
|
||||||
|
try {
|
||||||
|
const diagPath = await runAnalysis({
|
||||||
|
aggregatedPath,
|
||||||
|
plan2codeRoot,
|
||||||
|
proposalsDir,
|
||||||
|
model,
|
||||||
|
agent,
|
||||||
|
});
|
||||||
|
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.green('✓ Analysis complete'));
|
||||||
|
console.log(chalk.gray(`Diagnosis: ${diagPath}`));
|
||||||
|
|
||||||
|
const viewNow = await confirm({
|
||||||
|
message: 'Open diagnosis in console?',
|
||||||
|
default: true,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (viewNow) {
|
||||||
|
const content = fs.readFileSync(diagPath, 'utf8');
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.dim('─'.repeat(60)));
|
||||||
|
console.log(content);
|
||||||
|
console.log(chalk.dim('─'.repeat(60)));
|
||||||
|
}
|
||||||
|
|
||||||
|
return diagPath;
|
||||||
|
} catch (err) {
|
||||||
|
console.error(chalk.red(`Analysis failed: ${err instanceof Error ? err.message : String(err)}`));
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Flow: Generate improvement proposal ──────────────────────────────────────
|
||||||
|
|
||||||
|
async function flowGenerateProposal(): Promise<void> {
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.bold.cyan('── Generate Improvement Proposal ──'));
|
||||||
|
|
||||||
|
const { aggregatedPath, proposalsDir, runsDir } = getMetricsDirs();
|
||||||
|
const plan2codeRoot = detectPlan2CodeRoot() ?? process.cwd();
|
||||||
|
|
||||||
|
// Find diagnosis files
|
||||||
|
let diagFiles: string[] = [];
|
||||||
|
if (fs.existsSync(proposalsDir)) {
|
||||||
|
diagFiles = fs.readdirSync(proposalsDir)
|
||||||
|
.filter(f => f.endsWith('-diagnosis.md'))
|
||||||
|
.sort()
|
||||||
|
.reverse(); // most recent first
|
||||||
|
}
|
||||||
|
|
||||||
|
let diagnosisPath: string;
|
||||||
|
|
||||||
|
if (diagFiles.length === 0) {
|
||||||
|
console.log(chalk.yellow('No diagnosis files found. Running analysis first...'));
|
||||||
|
const diagPath = await flowRunAnalysis();
|
||||||
|
if (!diagPath) return;
|
||||||
|
diagnosisPath = diagPath;
|
||||||
|
} else {
|
||||||
|
const choice = await select({
|
||||||
|
message: 'Select diagnosis to base proposal on:',
|
||||||
|
choices: [
|
||||||
|
...diagFiles.map(f => ({ name: f, value: path.join(proposalsDir, f) })),
|
||||||
|
{ name: '(run new analysis first)', value: '__new__' },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
|
||||||
|
if (choice === '__new__') {
|
||||||
|
const diagPath = await flowRunAnalysis();
|
||||||
|
if (!diagPath) return;
|
||||||
|
diagnosisPath = diagPath;
|
||||||
|
} else {
|
||||||
|
diagnosisPath = choice;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const { agent, model } = await selectAgentAndModel();
|
||||||
|
|
||||||
|
try {
|
||||||
|
const result = await generateImprovement({
|
||||||
|
diagnosisPath,
|
||||||
|
plan2codeRoot,
|
||||||
|
proposalsDir,
|
||||||
|
runsDir,
|
||||||
|
model,
|
||||||
|
agent,
|
||||||
|
});
|
||||||
|
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.green(`✓ Proposal generated: ${result.proposal.proposal_id}`));
|
||||||
|
console.log(chalk.gray(` Valid edits: ${result.validEditCount}`));
|
||||||
|
console.log(chalk.gray(` Invalid edits: ${result.invalidEditCount} (rejected)`));
|
||||||
|
console.log(chalk.gray(` Saved to: ${result.proposalPath}`));
|
||||||
|
|
||||||
|
if (result.validEditCount > 0) {
|
||||||
|
const reviewNow = await confirm({
|
||||||
|
message: 'Review and apply edits now?',
|
||||||
|
default: true,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (reviewNow) {
|
||||||
|
await reviewAndApply({
|
||||||
|
proposalPath: result.proposalPath,
|
||||||
|
plan2codeRoot,
|
||||||
|
proposalsDir,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
console.error(chalk.red(`Proposal generation failed: ${err instanceof Error ? err.message : String(err)}`));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Flow: Review and apply proposal ──────────────────────────────────────────
|
||||||
|
|
||||||
|
async function flowReviewAndApply(): Promise<void> {
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.bold.cyan('── Review and Apply a Proposal ──'));
|
||||||
|
|
||||||
|
const { proposalsDir } = getMetricsDirs();
|
||||||
|
const plan2codeRoot = detectPlan2CodeRoot() ?? process.cwd();
|
||||||
|
|
||||||
|
if (!fs.existsSync(proposalsDir)) {
|
||||||
|
console.log(chalk.yellow('No proposals directory found. Generate a proposal first.'));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const proposalFiles = fs.readdirSync(proposalsDir)
|
||||||
|
.filter(f => /^prop-\d+\.json$/.test(f))
|
||||||
|
.sort()
|
||||||
|
.reverse(); // most recent first
|
||||||
|
|
||||||
|
if (proposalFiles.length === 0) {
|
||||||
|
console.log(chalk.yellow('No proposal files found. Generate a proposal first.'));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const chosen = await select({
|
||||||
|
message: 'Select proposal to review:',
|
||||||
|
choices: proposalFiles.map(f => {
|
||||||
|
try {
|
||||||
|
const p = JSON.parse(fs.readFileSync(path.join(proposalsDir, f), 'utf8'));
|
||||||
|
const label = `${f} [${p.status}] ${p.proposals?.length ?? 0} edits`;
|
||||||
|
return { name: label, value: path.join(proposalsDir, f) };
|
||||||
|
} catch {
|
||||||
|
return { name: f, value: path.join(proposalsDir, f) };
|
||||||
|
}
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
|
||||||
|
await reviewAndApply({
|
||||||
|
proposalPath: chosen,
|
||||||
|
plan2codeRoot,
|
||||||
|
proposalsDir,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Main menu ─────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export async function runCLI(): Promise<void> {
|
||||||
|
console.log();
|
||||||
|
console.log(chalk.bold.white('plan2code-metrics'));
|
||||||
|
console.log(chalk.gray('Recursive self-improvement toolchain for plan2code contributors'));
|
||||||
|
console.log();
|
||||||
|
|
||||||
|
// Check if we're in (or near) a plan2code repo
|
||||||
|
const plan2codeRoot = detectPlan2CodeRoot();
|
||||||
|
if (!plan2codeRoot) {
|
||||||
|
console.log(chalk.yellow('⚠ Could not detect plan2code repository (src/plan2code-*.md not found nearby).'));
|
||||||
|
console.log(chalk.gray(' Prompt version hashing and analysis will be limited.'));
|
||||||
|
console.log();
|
||||||
|
} else {
|
||||||
|
const version = detectPlan2CodeVersion(plan2codeRoot);
|
||||||
|
console.log(chalk.gray(`plan2code root: ${plan2codeRoot} (v${version})`));
|
||||||
|
console.log();
|
||||||
|
}
|
||||||
|
|
||||||
|
let continueLoop = true;
|
||||||
|
while (continueLoop) {
|
||||||
|
const action = await select({
|
||||||
|
message: 'What would you like to do?',
|
||||||
|
choices: [
|
||||||
|
{ name: 'Collect metrics for a completed project', value: 'collect' },
|
||||||
|
{ name: 'Import run data from another project', value: 'import' },
|
||||||
|
{ name: 'View metrics status and history', value: 'view' },
|
||||||
|
{ name: 'Run analysis (diagnose weak steps)', value: 'analyze' },
|
||||||
|
{ name: 'Generate improvement proposal', value: 'propose' },
|
||||||
|
{ name: 'Review and apply a proposal', value: 'apply' },
|
||||||
|
{ name: 'Exit', value: 'exit' },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
|
||||||
|
switch (action) {
|
||||||
|
case 'collect':
|
||||||
|
await flowCollect();
|
||||||
|
break;
|
||||||
|
case 'import':
|
||||||
|
await flowImport();
|
||||||
|
break;
|
||||||
|
case 'view':
|
||||||
|
await flowViewStatus();
|
||||||
|
break;
|
||||||
|
case 'analyze':
|
||||||
|
await flowRunAnalysis();
|
||||||
|
break;
|
||||||
|
case 'propose':
|
||||||
|
await flowGenerateProposal();
|
||||||
|
break;
|
||||||
|
case 'apply':
|
||||||
|
await flowReviewAndApply();
|
||||||
|
break;
|
||||||
|
case 'exit':
|
||||||
|
continueLoop = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (continueLoop && action !== 'exit') {
|
||||||
|
console.log();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(chalk.gray('Goodbye.'));
|
||||||
|
}
|
||||||
@@ -0,0 +1,520 @@
|
|||||||
|
/**
|
||||||
|
* collector.ts
|
||||||
|
* Reads finished project artifacts and writes a RunMetrics JSON file.
|
||||||
|
* Zero dependency on plan2code-loop internals — reads files directly.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import fs from 'fs';
|
||||||
|
import path from 'path';
|
||||||
|
import crypto from 'crypto';
|
||||||
|
import type {
|
||||||
|
RunMetrics,
|
||||||
|
PromptVersions,
|
||||||
|
Step1PlanMetrics,
|
||||||
|
Step2DocumentMetrics,
|
||||||
|
Step3ImplementMetrics,
|
||||||
|
Step4FinalizeMetrics,
|
||||||
|
UserFeedback,
|
||||||
|
} from './types.js';
|
||||||
|
|
||||||
|
// ── Helpers ──────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
function sha256File(filePath: string): string {
|
||||||
|
try {
|
||||||
|
const content = fs.readFileSync(filePath, 'utf8');
|
||||||
|
return 'sha256:' + crypto.createHash('sha256').update(content).digest('hex');
|
||||||
|
} catch {
|
||||||
|
return 'sha256:missing';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function readFileSafe(filePath: string): string | null {
|
||||||
|
try {
|
||||||
|
return fs.readFileSync(filePath, 'utf8');
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function generateRunId(): string {
|
||||||
|
const now = new Date();
|
||||||
|
const ts = now.toISOString().replace(/[-:T]/g, '').slice(0, 14);
|
||||||
|
const rand = crypto.randomBytes(2).toString('hex');
|
||||||
|
return `run-${ts.slice(0, 8)}-${ts.slice(8, 14)}-${rand}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Prompt version hashing ────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export function collectPromptVersions(plan2codeRoot: string): PromptVersions {
|
||||||
|
const srcDir = path.join(plan2codeRoot, 'src');
|
||||||
|
return {
|
||||||
|
plan: sha256File(path.join(srcDir, 'plan2code-1--plan.md')),
|
||||||
|
revise_plan: sha256File(path.join(srcDir, 'plan2code-1b--revise-plan.md')),
|
||||||
|
document: sha256File(path.join(srcDir, 'plan2code-2--document.md')),
|
||||||
|
implement: sha256File(path.join(srcDir, 'plan2code-3--implement.md')),
|
||||||
|
finalize: sha256File(path.join(srcDir, 'plan2code-4--finalize.md')),
|
||||||
|
init: sha256File(path.join(srcDir, 'plan2code---init.md')),
|
||||||
|
init_update: sha256File(path.join(srcDir, 'plan2code---init-update.md')),
|
||||||
|
quick_task: sha256File(path.join(srcDir, 'plan2code---quick-task.md')),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Step 1: Plan ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export function collectStep1(specDir: string): Step1PlanMetrics {
|
||||||
|
// Find PLAN-DRAFT-*.md files
|
||||||
|
let draftFiles: string[] = [];
|
||||||
|
let convFiles: string[] = [];
|
||||||
|
try {
|
||||||
|
const files = fs.readdirSync(specDir);
|
||||||
|
draftFiles = files
|
||||||
|
.filter(f => f.startsWith('PLAN-DRAFT-') && f.endsWith('.md'))
|
||||||
|
.map(f => path.join(specDir, f));
|
||||||
|
convFiles = files
|
||||||
|
.filter(f => f.startsWith('PLAN-CONVERSATION-') && f.endsWith('.md'))
|
||||||
|
.map(f => path.join(specDir, f));
|
||||||
|
} catch {
|
||||||
|
return { present: false, final_confidence: null, confidence_breakdown: null,
|
||||||
|
clarification_rounds: null, tech_stack_revision_rounds: null,
|
||||||
|
verification_gaps_found: null, functional_requirements_count: null,
|
||||||
|
non_functional_requirements_count: null, risk_count: null, phase_count: null };
|
||||||
|
}
|
||||||
|
|
||||||
|
if (draftFiles.length === 0) {
|
||||||
|
return { present: false, final_confidence: null, confidence_breakdown: null,
|
||||||
|
clarification_rounds: null, tech_stack_revision_rounds: null,
|
||||||
|
verification_gaps_found: null, functional_requirements_count: null,
|
||||||
|
non_functional_requirements_count: null, risk_count: null, phase_count: null };
|
||||||
|
}
|
||||||
|
|
||||||
|
// Use the latest draft file
|
||||||
|
draftFiles.sort();
|
||||||
|
const latestDraft = readFileSafe(draftFiles[draftFiles.length - 1]) ?? '';
|
||||||
|
|
||||||
|
// Parse "Confidence: XX%" — look for overall or total confidence
|
||||||
|
let finalConfidence: number | null = null;
|
||||||
|
const confMatch = latestDraft.match(/(?:Overall|Final|Total)?\s*[Cc]onfidence[:\s]+(\d{1,3})%/);
|
||||||
|
if (confMatch) {
|
||||||
|
finalConfidence = parseInt(confMatch[1], 10);
|
||||||
|
} else {
|
||||||
|
// Try table format: | Confidence | 92 |
|
||||||
|
const tableMatch = latestDraft.match(/[|]\s*[Cc]onfidence\s*[|]\s*(\d{1,3})/);
|
||||||
|
if (tableMatch) finalConfidence = parseInt(tableMatch[1], 10);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Confidence breakdown (requirements, feasibility, integration, risk)
|
||||||
|
let confidenceBreakdown: Step1PlanMetrics['confidence_breakdown'] = null;
|
||||||
|
const reqMatch = latestDraft.match(/[Rr]equirements?[:\s|]+(\d{1,2})/);
|
||||||
|
const feasMatch = latestDraft.match(/[Ff]easibility[:\s|]+(\d{1,2})/);
|
||||||
|
const intMatch = latestDraft.match(/[Ii]ntegration[:\s|]+(\d{1,2})/);
|
||||||
|
const riskMatch = latestDraft.match(/[Rr]isk[:\s|]+(\d{1,2})/);
|
||||||
|
if (reqMatch || feasMatch || intMatch || riskMatch) {
|
||||||
|
confidenceBreakdown = {
|
||||||
|
requirements: reqMatch ? parseInt(reqMatch[1], 10) : null,
|
||||||
|
feasibility: feasMatch ? parseInt(feasMatch[1], 10) : null,
|
||||||
|
integration: intMatch ? parseInt(intMatch[1], 10) : null,
|
||||||
|
risk: riskMatch ? parseInt(riskMatch[1], 10) : null,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// Count ### FR- / ### NFR- headings
|
||||||
|
const frCount = (latestDraft.match(/###\s+FR-/g) ?? []).length;
|
||||||
|
const nfrCount = (latestDraft.match(/###\s+NFR-/g) ?? []).length;
|
||||||
|
|
||||||
|
// Count risk table rows (lines starting with | that contain risk-level keywords)
|
||||||
|
const riskRows = latestDraft.match(/^\s*[|][^|]*(?:High|Medium|Low|Critical)[^|]*[|]/gm) ?? [];
|
||||||
|
const riskCount = riskRows.length || null;
|
||||||
|
|
||||||
|
// Count ## Phase headings
|
||||||
|
const phaseCount = (latestDraft.match(/^##\s+Phase\s+\d/gm) ?? []).length || null;
|
||||||
|
|
||||||
|
// Clarification rounds from conversation file
|
||||||
|
let clarificationRounds: number | null = null;
|
||||||
|
let techStackRevisions: number | null = null;
|
||||||
|
let verificationGaps: number | null = null;
|
||||||
|
|
||||||
|
if (convFiles.length > 0) {
|
||||||
|
convFiles.sort();
|
||||||
|
const latestConv = readFileSafe(convFiles[convFiles.length - 1]) ?? '';
|
||||||
|
// Count heading repetitions as clarification rounds (## Clarification or ## Round)
|
||||||
|
const clarRounds = (latestConv.match(/^##\s+(?:Clarification|Round)\s+\d/gm) ?? []).length;
|
||||||
|
clarificationRounds = clarRounds || null;
|
||||||
|
// Tech stack revision rounds
|
||||||
|
const techRounds = (latestConv.match(/^##\s+(?:Tech\s+Stack|Technology)\s+Revision/gmi) ?? []).length;
|
||||||
|
techStackRevisions = techRounds || null;
|
||||||
|
// Verification gaps found
|
||||||
|
const gapMatches = latestConv.match(/(?:verification\s+gap|gap\s+found|missing\s+requirement)/gi) ?? [];
|
||||||
|
verificationGaps = gapMatches.length || null;
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
present: true,
|
||||||
|
final_confidence: finalConfidence,
|
||||||
|
confidence_breakdown: confidenceBreakdown,
|
||||||
|
clarification_rounds: clarificationRounds,
|
||||||
|
tech_stack_revision_rounds: techStackRevisions,
|
||||||
|
verification_gaps_found: verificationGaps,
|
||||||
|
functional_requirements_count: frCount || null,
|
||||||
|
non_functional_requirements_count: nfrCount || null,
|
||||||
|
risk_count: riskCount,
|
||||||
|
phase_count: phaseCount,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Step 2: Document ──────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export function collectStep2(specDir: string): Step2DocumentMetrics {
|
||||||
|
const overviewPath = path.join(specDir, 'overview.md');
|
||||||
|
const overview = readFileSafe(overviewPath);
|
||||||
|
|
||||||
|
if (!overview) {
|
||||||
|
return { present: false, total_tasks: null, tasks_per_phase: null,
|
||||||
|
phase_count: null, parallel_groups_identified: null,
|
||||||
|
requirement_coverage_percent: null, verification_items_added: null };
|
||||||
|
}
|
||||||
|
|
||||||
|
// Count all checkbox tasks: - [ ], - [x], - [!], - [/]
|
||||||
|
const allTasks = (overview.match(/^\s*-\s+\[[ x!/?]\]/gm) ?? []).length;
|
||||||
|
|
||||||
|
// Detect Parallel Execution Groups table
|
||||||
|
const parallelGroups = (overview.match(/Parallel\s+Execution\s+Group/gi) ?? []).length;
|
||||||
|
|
||||||
|
// Count phase files
|
||||||
|
let phaseCount = 0;
|
||||||
|
let tasksPerPhase: number[] = [];
|
||||||
|
try {
|
||||||
|
const files = fs.readdirSync(specDir);
|
||||||
|
const phaseFiles = files
|
||||||
|
.filter(f => /^phase-\d+\.md$/i.test(f))
|
||||||
|
.sort();
|
||||||
|
phaseCount = phaseFiles.length;
|
||||||
|
for (const pf of phaseFiles) {
|
||||||
|
const content = readFileSafe(path.join(specDir, pf)) ?? '';
|
||||||
|
const count = (content.match(/^\s*-\s+\[[ x!/?]\]/gm) ?? []).length;
|
||||||
|
tasksPerPhase.push(count);
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// leave empty
|
||||||
|
}
|
||||||
|
|
||||||
|
// Requirement coverage: look for coverage percentage in overview
|
||||||
|
let reqCoverage: number | null = null;
|
||||||
|
const covMatch = overview.match(/[Cc]overage[:\s]+(\d{1,3})%/);
|
||||||
|
if (covMatch) reqCoverage = parseInt(covMatch[1], 10);
|
||||||
|
|
||||||
|
// Verification items added (look for verification checklist items)
|
||||||
|
const verifItems = (overview.match(/(?:verify|verification|test|check):\s*\[[ x]\]/gi) ?? []).length;
|
||||||
|
|
||||||
|
return {
|
||||||
|
present: true,
|
||||||
|
total_tasks: allTasks || null,
|
||||||
|
tasks_per_phase: tasksPerPhase.length > 0 ? tasksPerPhase : null,
|
||||||
|
phase_count: phaseCount || null,
|
||||||
|
parallel_groups_identified: parallelGroups || 0,
|
||||||
|
requirement_coverage_percent: reqCoverage,
|
||||||
|
verification_items_added: verifItems || null,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Step 3: Implement (loop data) ─────────────────────────────────────────────
|
||||||
|
|
||||||
|
export function collectStep3(specDir: string): Step3ImplementMetrics {
|
||||||
|
const loopDir = path.join(specDir, '.plan2code-loop');
|
||||||
|
const iterLogPath = path.join(loopDir, 'iteration.log');
|
||||||
|
const configPath = path.join(loopDir, 'config.json');
|
||||||
|
|
||||||
|
if (!fs.existsSync(loopDir)) {
|
||||||
|
return { present: true, used_loop_mode: false, loop_mode: null,
|
||||||
|
task_completion_rate: null, tasks_completed: null, tasks_total: null,
|
||||||
|
blocker_count: null, blocker_categories: null, total_iterations: null,
|
||||||
|
avg_iteration_duration_ms: null, exit_code_distribution: null,
|
||||||
|
completion_marker_success_rate: null };
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse config.json for loop mode
|
||||||
|
let loopMode: 'task' | 'phase' | null = null;
|
||||||
|
const configContent = readFileSafe(configPath);
|
||||||
|
if (configContent) {
|
||||||
|
try {
|
||||||
|
const config = JSON.parse(configContent);
|
||||||
|
loopMode = config.loopMode ?? null;
|
||||||
|
} catch {
|
||||||
|
// ignore
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse NDJSON iteration.log
|
||||||
|
const iterLogContent = readFileSafe(iterLogPath);
|
||||||
|
if (!iterLogContent) {
|
||||||
|
return { present: true, used_loop_mode: true, loop_mode: loopMode,
|
||||||
|
task_completion_rate: null, tasks_completed: null, tasks_total: null,
|
||||||
|
blocker_count: null, blocker_categories: null, total_iterations: null,
|
||||||
|
avg_iteration_duration_ms: null, exit_code_distribution: null,
|
||||||
|
completion_marker_success_rate: null };
|
||||||
|
}
|
||||||
|
|
||||||
|
interface IterEntry {
|
||||||
|
iteration: number;
|
||||||
|
timestamp: string;
|
||||||
|
duration: number;
|
||||||
|
exitCode: number;
|
||||||
|
status: string;
|
||||||
|
completionMarker?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
const entries: IterEntry[] = [];
|
||||||
|
for (const line of iterLogContent.split('\n')) {
|
||||||
|
const trimmed = line.trim();
|
||||||
|
if (!trimmed) continue;
|
||||||
|
try {
|
||||||
|
entries.push(JSON.parse(trimmed));
|
||||||
|
} catch {
|
||||||
|
// skip malformed lines
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const totalIterations = entries.length;
|
||||||
|
if (totalIterations === 0) {
|
||||||
|
return { present: true, used_loop_mode: true, loop_mode: loopMode,
|
||||||
|
task_completion_rate: null, tasks_completed: null, tasks_total: null,
|
||||||
|
blocker_count: null, blocker_categories: null, total_iterations: 0,
|
||||||
|
avg_iteration_duration_ms: null, exit_code_distribution: null,
|
||||||
|
completion_marker_success_rate: null };
|
||||||
|
}
|
||||||
|
|
||||||
|
// Exit code distribution
|
||||||
|
const exitCodes: Record<string, number> = {};
|
||||||
|
for (const e of entries) {
|
||||||
|
const key = String(e.exitCode ?? 'other');
|
||||||
|
exitCodes[key] = (exitCodes[key] ?? 0) + 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Average duration
|
||||||
|
const durations = entries.map(e => e.duration).filter(d => d != null && d > 0);
|
||||||
|
const avgDuration = durations.length > 0
|
||||||
|
? Math.round(durations.reduce((a, b) => a + b, 0) / durations.length)
|
||||||
|
: null;
|
||||||
|
|
||||||
|
// Completion markers
|
||||||
|
const markerEntries = entries.filter(e => e.completionMarker && e.completionMarker.trim() !== '');
|
||||||
|
const taskCompleteEntries = markerEntries.filter(e =>
|
||||||
|
e.completionMarker?.startsWith('TASK_COMPLETE')
|
||||||
|
);
|
||||||
|
const blockedEntries = markerEntries.filter(e =>
|
||||||
|
e.completionMarker?.startsWith('TASK_BLOCKED')
|
||||||
|
);
|
||||||
|
|
||||||
|
// Completion marker success rate: entries where a valid marker was detected vs total
|
||||||
|
const validMarkerCount = markerEntries.length;
|
||||||
|
const markerSuccessRate = totalIterations > 0
|
||||||
|
? validMarkerCount / totalIterations
|
||||||
|
: null;
|
||||||
|
|
||||||
|
// Task counts from markers (used for completion_marker_success_rate)
|
||||||
|
const blockerCount = blockedEntries.length;
|
||||||
|
|
||||||
|
// Blocker categories (parse from "TASK_BLOCKED: X.Y - reason")
|
||||||
|
const blockerCategories: string[] = [];
|
||||||
|
for (const e of blockedEntries) {
|
||||||
|
const match = e.completionMarker?.match(/TASK_BLOCKED:\s*[\d.]+\s*-\s*(.+)/);
|
||||||
|
if (match) {
|
||||||
|
// Normalize to snake_case category
|
||||||
|
const reason = match[1].toLowerCase().replace(/\s+/g, '_').replace(/[^a-z0-9_]/g, '');
|
||||||
|
if (!blockerCategories.includes(reason)) {
|
||||||
|
blockerCategories.push(reason);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Count tasks from overview.md checkboxes (consistent numerator and denominator)
|
||||||
|
const overviewPath = path.join(specDir, 'overview.md');
|
||||||
|
const overview = readFileSafe(overviewPath);
|
||||||
|
let tasksTotal: number | null = null;
|
||||||
|
let tasksCompleted: number | null = null;
|
||||||
|
if (overview) {
|
||||||
|
tasksTotal = (overview.match(/^\s*-\s+\[[ x!/?]\]/gm) ?? []).length || null;
|
||||||
|
tasksCompleted = (overview.match(/^\s*-\s+\[x\]/gm) ?? []).length || null;
|
||||||
|
}
|
||||||
|
|
||||||
|
const taskCompletionRate = tasksTotal && tasksTotal > 0 && tasksCompleted != null
|
||||||
|
? tasksCompleted / tasksTotal
|
||||||
|
: null;
|
||||||
|
|
||||||
|
return {
|
||||||
|
present: true,
|
||||||
|
used_loop_mode: true,
|
||||||
|
loop_mode: loopMode,
|
||||||
|
task_completion_rate: taskCompletionRate,
|
||||||
|
tasks_completed: tasksCompleted || null,
|
||||||
|
tasks_total: tasksTotal,
|
||||||
|
blocker_count: blockerCount,
|
||||||
|
blocker_categories: blockerCategories.length > 0 ? blockerCategories : null,
|
||||||
|
total_iterations: totalIterations,
|
||||||
|
avg_iteration_duration_ms: avgDuration,
|
||||||
|
exit_code_distribution: exitCodes,
|
||||||
|
completion_marker_success_rate: markerSuccessRate,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Step 4: Finalize ──────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export function collectStep4(specDir: string): Step4FinalizeMetrics {
|
||||||
|
// Check if spec was archived (specs--completed exists at parent level)
|
||||||
|
const specDirName = path.basename(specDir);
|
||||||
|
const parentDir = path.dirname(specDir);
|
||||||
|
const completedPath = path.join(parentDir, '..', 'specs--completed', specDirName);
|
||||||
|
const archivalSucceeded = fs.existsSync(completedPath);
|
||||||
|
|
||||||
|
// Read overview for completion metrics
|
||||||
|
const overviewPath = path.join(specDir, 'overview.md');
|
||||||
|
|
||||||
|
// If archived, try from archived location
|
||||||
|
const effectiveOverviewPath = archivalSucceeded
|
||||||
|
? path.join(completedPath, 'overview.md')
|
||||||
|
: overviewPath;
|
||||||
|
|
||||||
|
const overview = readFileSafe(effectiveOverviewPath) ?? readFileSafe(overviewPath);
|
||||||
|
if (!overview) {
|
||||||
|
return { present: false, completion_rate_at_audit: null,
|
||||||
|
verification_failures_found: null, documentation_updates_needed: null,
|
||||||
|
archival_succeeded: archivalSucceeded };
|
||||||
|
}
|
||||||
|
|
||||||
|
// Completion rate: completed tasks / total tasks
|
||||||
|
const totalTasks = (overview.match(/^\s*-\s+\[[ x!/?]\]/gm) ?? []).length;
|
||||||
|
const completedTasks = (overview.match(/^\s*-\s+\[x\]/gm) ?? []).length;
|
||||||
|
const completionRate = totalTasks > 0 ? completedTasks / totalTasks : null;
|
||||||
|
|
||||||
|
// Verification failures: look for [!] tasks or "verification failed" text
|
||||||
|
const blockedTasks = (overview.match(/^\s*-\s+\[!\]/gm) ?? []).length;
|
||||||
|
|
||||||
|
// Documentation updates needed (look for TODO or "update" markers)
|
||||||
|
const docUpdates = (overview.match(/(?:TODO|FIXME|update\s+(?:README|CHANGELOG|docs))/gi) ?? []).length;
|
||||||
|
|
||||||
|
return {
|
||||||
|
present: true,
|
||||||
|
completion_rate_at_audit: completionRate,
|
||||||
|
verification_failures_found: blockedTasks || 0,
|
||||||
|
documentation_updates_needed: docUpdates || null,
|
||||||
|
archival_succeeded: archivalSucceeded,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── User Feedback ─────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export function collectUserFeedback(specDir: string): UserFeedback | null {
|
||||||
|
// Try both active and archived locations for overview.md
|
||||||
|
const overviewPath = path.join(specDir, 'overview.md');
|
||||||
|
const specDirName = path.basename(specDir);
|
||||||
|
const parentDir = path.dirname(specDir);
|
||||||
|
const completedPath = path.join(parentDir, '..', 'specs--completed', specDirName, 'overview.md');
|
||||||
|
|
||||||
|
const overview = readFileSafe(overviewPath) ?? readFileSafe(completedPath);
|
||||||
|
if (!overview) return null;
|
||||||
|
|
||||||
|
// Look for ## User Feedback section with a markdown table
|
||||||
|
const feedbackMatch = overview.match(
|
||||||
|
/## User Feedback\s*\n\s*\|[^\n]*\|\s*\n\s*\|[-| ]+\|\s*\n([\s\S]*?)(?=\n##\s|\n*$)/
|
||||||
|
);
|
||||||
|
if (!feedbackMatch) return null;
|
||||||
|
|
||||||
|
const tableBody = feedbackMatch[1];
|
||||||
|
|
||||||
|
// Parse table rows: | Field | Value |
|
||||||
|
// Use regex to split only on unescaped pipes, preserving whitespace in values
|
||||||
|
const rows = tableBody.split('\n').filter(l => l.trim().startsWith('|'));
|
||||||
|
const fields: Record<string, string> = {};
|
||||||
|
for (const row of rows) {
|
||||||
|
// Split on unescaped pipes (not preceded by backslash)
|
||||||
|
const cells = row.split(/(?<!\\)\|/).map(c => c.trim()).filter(c => c.length > 0);
|
||||||
|
if (cells.length >= 2) {
|
||||||
|
const value = cells.slice(1).join('|').replace(/\\\|/g, '|');
|
||||||
|
fields[cells[0].toLowerCase()] = value;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const rating = parseInt(fields['rating'] ?? '', 10);
|
||||||
|
if (isNaN(rating) || rating < 1 || rating > 10) return null;
|
||||||
|
|
||||||
|
return {
|
||||||
|
overall_rating: rating,
|
||||||
|
rating_reason: fields['reason'] ?? '',
|
||||||
|
what_went_well: fields['went well'] ?? '',
|
||||||
|
what_went_poorly: fields['went poorly'] ?? '',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Main collector ────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export interface CollectorOptions {
|
||||||
|
specDir: string; // Path to the spec directory (specs/<feature-name>)
|
||||||
|
projectName: string; // Human-readable project name
|
||||||
|
plan2codeRoot: string; // Path to the plan2code repo (for prompt hashing)
|
||||||
|
plan2codeVersion: string; // e.g. "1.7.0"
|
||||||
|
outputDir: string; // Where to write <run-id>.json
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function collectRun(opts: CollectorOptions): Promise<RunMetrics> {
|
||||||
|
const {
|
||||||
|
specDir,
|
||||||
|
projectName,
|
||||||
|
plan2codeRoot,
|
||||||
|
plan2codeVersion,
|
||||||
|
outputDir,
|
||||||
|
} = opts;
|
||||||
|
|
||||||
|
const runId = generateRunId();
|
||||||
|
|
||||||
|
// Get started_at / completed_at from spec directory mtime / iteration.log
|
||||||
|
let startedAt: string | null = null;
|
||||||
|
let completedAt: string | null = null;
|
||||||
|
try {
|
||||||
|
const stat = fs.statSync(specDir);
|
||||||
|
startedAt = stat.birthtime.toISOString();
|
||||||
|
completedAt = stat.mtime.toISOString();
|
||||||
|
} catch {
|
||||||
|
// leave null
|
||||||
|
}
|
||||||
|
|
||||||
|
// Try to get started_at from earliest iteration log entry
|
||||||
|
const loopDir = path.join(specDir, '.plan2code-loop');
|
||||||
|
const iterLogPath = path.join(loopDir, 'iteration.log');
|
||||||
|
const iterLogContent = readFileSafe(iterLogPath);
|
||||||
|
if (iterLogContent) {
|
||||||
|
const lines = iterLogContent.split('\n').filter(l => l.trim());
|
||||||
|
if (lines.length > 0) {
|
||||||
|
try {
|
||||||
|
const first = JSON.parse(lines[0]);
|
||||||
|
if (first.timestamp) startedAt = first.timestamp;
|
||||||
|
} catch { /* ignore */ }
|
||||||
|
try {
|
||||||
|
const last = JSON.parse(lines[lines.length - 1]);
|
||||||
|
if (last.timestamp) completedAt = last.timestamp;
|
||||||
|
} catch { /* ignore */ }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const metrics: RunMetrics = {
|
||||||
|
schema_version: '1.0',
|
||||||
|
run_id: runId,
|
||||||
|
plan2code_version: plan2codeVersion,
|
||||||
|
prompt_versions: collectPromptVersions(plan2codeRoot),
|
||||||
|
project: {
|
||||||
|
name: projectName,
|
||||||
|
started_at: startedAt,
|
||||||
|
completed_at: completedAt,
|
||||||
|
},
|
||||||
|
step1_plan: collectStep1(specDir),
|
||||||
|
step2_document: collectStep2(specDir),
|
||||||
|
step3_implement: collectStep3(specDir),
|
||||||
|
step4_finalize: collectStep4(specDir),
|
||||||
|
user_feedback: collectUserFeedback(specDir),
|
||||||
|
};
|
||||||
|
|
||||||
|
// Write to output dir
|
||||||
|
fs.mkdirSync(outputDir, { recursive: true });
|
||||||
|
const outPath = path.join(outputDir, `${runId}.json`);
|
||||||
|
fs.writeFileSync(outPath, JSON.stringify(metrics, null, 2), 'utf8');
|
||||||
|
|
||||||
|
return metrics;
|
||||||
|
}
|
||||||
@@ -0,0 +1,139 @@
|
|||||||
|
import { describe, it, expect } from 'vitest';
|
||||||
|
import { validateEdit, parseProposalFromResponse } from './improver.js';
|
||||||
|
import type { PromptEdit } from './types.js';
|
||||||
|
|
||||||
|
// ── Shared fixtures ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
const PROMPT_CONTENTS: Record<string, string> = {
|
||||||
|
'plan2code-1--plan.md': 'This is the plan prompt content. It has some text here.',
|
||||||
|
'plan2code-2--document.md': 'Document prompt with repeated text. repeated text. Done.',
|
||||||
|
'plan2code-3--implement.md': 'Implement prompt content.',
|
||||||
|
};
|
||||||
|
|
||||||
|
function makeEdit(overrides: Partial<PromptEdit> = {}): PromptEdit {
|
||||||
|
return {
|
||||||
|
file: 'plan2code-1--plan.md',
|
||||||
|
rationale: 'test rationale',
|
||||||
|
expected_metric_impact: 'test impact',
|
||||||
|
char_count_before: PROMPT_CONTENTS['plan2code-1--plan.md'].length,
|
||||||
|
char_count_after: PROMPT_CONTENTS['plan2code-1--plan.md'].length,
|
||||||
|
char_count_delta: 0,
|
||||||
|
old_text: 'some text',
|
||||||
|
new_text: 'better text',
|
||||||
|
...overrides,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── validateEdit() ────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
describe('validateEdit', () => {
|
||||||
|
it('valid edit passes with no errors', () => {
|
||||||
|
const result = validateEdit(makeEdit(), PROMPT_CONTENTS);
|
||||||
|
expect(result.valid).toBe(true);
|
||||||
|
expect(result.errors).toHaveLength(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('rejects path traversal (../ in file path)', () => {
|
||||||
|
const result = validateEdit(makeEdit({ file: '../etc/passwd' }), PROMPT_CONTENTS);
|
||||||
|
expect(result.valid).toBe(false);
|
||||||
|
expect(result.errors[0]).toMatch(/path traversal/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('rejects absolute paths', () => {
|
||||||
|
const result = validateEdit(makeEdit({ file: '/etc/passwd' }), PROMPT_CONTENTS);
|
||||||
|
expect(result.valid).toBe(false);
|
||||||
|
expect(result.errors[0]).toMatch(/path traversal|absolute/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('errors when file not found in promptContents', () => {
|
||||||
|
const result = validateEdit(makeEdit({ file: 'nonexistent.md' }), PROMPT_CONTENTS);
|
||||||
|
expect(result.valid).toBe(false);
|
||||||
|
expect(result.errors[0]).toMatch(/not found/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('errors when old_text not found in file content', () => {
|
||||||
|
const result = validateEdit(makeEdit({ old_text: 'hallucinated text' }), PROMPT_CONTENTS);
|
||||||
|
expect(result.valid).toBe(false);
|
||||||
|
expect(result.errors[0]).toMatch(/not found verbatim/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('warns when old_text appears multiple times', () => {
|
||||||
|
const edit = makeEdit({
|
||||||
|
file: 'plan2code-2--document.md',
|
||||||
|
old_text: 'repeated text',
|
||||||
|
char_count_before: PROMPT_CONTENTS['plan2code-2--document.md'].length,
|
||||||
|
char_count_after: PROMPT_CONTENTS['plan2code-2--document.md'].length,
|
||||||
|
});
|
||||||
|
const result = validateEdit(edit, PROMPT_CONTENTS);
|
||||||
|
expect(result.valid).toBe(true);
|
||||||
|
expect(result.warnings.some(w => /appears.*times/i.test(w))).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('errors when edit would exceed 11,000 char limit', () => {
|
||||||
|
const bigText = 'x'.repeat(12_000);
|
||||||
|
const edit = makeEdit({ new_text: bigText });
|
||||||
|
const result = validateEdit(edit, PROMPT_CONTENTS);
|
||||||
|
expect(result.valid).toBe(false);
|
||||||
|
expect(result.errors.some(e => /exceed.*11.?000/i.test(e))).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('warns when reported char counts diverge from actual (>10 chars off)', () => {
|
||||||
|
const edit = makeEdit({
|
||||||
|
char_count_before: 999,
|
||||||
|
char_count_after: 999,
|
||||||
|
});
|
||||||
|
const result = validateEdit(edit, PROMPT_CONTENTS);
|
||||||
|
expect(result.valid).toBe(true);
|
||||||
|
expect(result.warnings.some(w => /char_count_before.*differs/i.test(w))).toBe(true);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ── parseProposalFromResponse() ───────────────────────────────────────────────
|
||||||
|
|
||||||
|
describe('parseProposalFromResponse', () => {
|
||||||
|
const sampleEdit = {
|
||||||
|
file: 'test.md',
|
||||||
|
rationale: 'r',
|
||||||
|
expected_metric_impact: 'e',
|
||||||
|
char_count_before: 100,
|
||||||
|
char_count_after: 110,
|
||||||
|
char_count_delta: 10,
|
||||||
|
old_text: 'old',
|
||||||
|
new_text: 'new',
|
||||||
|
};
|
||||||
|
|
||||||
|
it('parses JSON from markdown code block (```json ... ```)', () => {
|
||||||
|
const response = `Here is my proposal:\n\n\`\`\`json\n${JSON.stringify([sampleEdit])}\n\`\`\`\n\nDone.`;
|
||||||
|
const result = parseProposalFromResponse(response);
|
||||||
|
expect(result).toHaveLength(1);
|
||||||
|
expect(result![0].file).toBe('test.md');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('parses JSON from bare code block (``` ... ```)', () => {
|
||||||
|
const response = `Proposal:\n\n\`\`\`\n${JSON.stringify([sampleEdit])}\n\`\`\``;
|
||||||
|
const result = parseProposalFromResponse(response);
|
||||||
|
expect(result).toHaveLength(1);
|
||||||
|
expect(result![0].old_text).toBe('old');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('parses bare JSON array with old_text field', () => {
|
||||||
|
const response = `Some preamble\n${JSON.stringify([sampleEdit])}\nSome postamble`;
|
||||||
|
const result = parseProposalFromResponse(response);
|
||||||
|
expect(result).toHaveLength(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('returns null for non-JSON response', () => {
|
||||||
|
const result = parseProposalFromResponse('No changes needed at this time.');
|
||||||
|
expect(result).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('returns null for malformed JSON', () => {
|
||||||
|
const result = parseProposalFromResponse('```json\n{broken json]\n```');
|
||||||
|
expect(result).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('returns null for empty response', () => {
|
||||||
|
const result = parseProposalFromResponse('');
|
||||||
|
expect(result).toBeNull();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,288 @@
|
|||||||
|
/**
|
||||||
|
* improver.ts
|
||||||
|
* Reads diagnosis + prompt files, invokes AI, parses PromptEdit[] from response.
|
||||||
|
* Validates: old_text verbatim match, char count limits.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import fs from 'fs';
|
||||||
|
import path from 'path';
|
||||||
|
import type { PromptEdit, PromptProposal } from './types.js';
|
||||||
|
import { invokeLLM, type AgentType } from './invoke-llm.js';
|
||||||
|
|
||||||
|
const CHAR_LIMIT = 11_000;
|
||||||
|
const IMPROVE_PROMPT_PATH = new URL('../src/prompts/improve.md', import.meta.url).pathname
|
||||||
|
.replace(/^\/([A-Za-z]:)/, '$1'); // Fix Windows path
|
||||||
|
|
||||||
|
// ── Helpers ───────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
function interpolate(template: string, vars: Record<string, string>): string {
|
||||||
|
let result = template;
|
||||||
|
for (const [key, value] of Object.entries(vars)) {
|
||||||
|
result = result.replaceAll(`{{${key}}}`, value);
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
function generateProposalId(): string {
|
||||||
|
const now = new Date();
|
||||||
|
const ts = now.toISOString().replace(/[-:T.Z]/g, '').slice(0, 14);
|
||||||
|
return `prop-${ts}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function readPromptFiles(plan2codeRoot: string): Record<string, string> {
|
||||||
|
const srcDir = path.join(plan2codeRoot, 'src');
|
||||||
|
const promptFiles = [
|
||||||
|
'plan2code-1--plan.md',
|
||||||
|
'plan2code-1b--revise-plan.md',
|
||||||
|
'plan2code-2--document.md',
|
||||||
|
'plan2code-3--implement.md',
|
||||||
|
'plan2code-4--finalize.md',
|
||||||
|
'plan2code---init.md',
|
||||||
|
'plan2code---init-update.md',
|
||||||
|
'plan2code---quick-task.md',
|
||||||
|
];
|
||||||
|
|
||||||
|
const contents: Record<string, string> = {};
|
||||||
|
for (const file of promptFiles) {
|
||||||
|
try {
|
||||||
|
contents[file] = fs.readFileSync(path.join(srcDir, file), 'utf8');
|
||||||
|
} catch {
|
||||||
|
contents[file] = '';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return contents;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Edit validation ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export interface ValidationResult {
|
||||||
|
valid: boolean;
|
||||||
|
errors: string[];
|
||||||
|
warnings: string[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export function validateEdit(
|
||||||
|
edit: PromptEdit,
|
||||||
|
promptContents: Record<string, string>,
|
||||||
|
): ValidationResult {
|
||||||
|
const errors: string[] = [];
|
||||||
|
const warnings: string[] = [];
|
||||||
|
|
||||||
|
// Reject path traversal attempts
|
||||||
|
if (edit.file.includes('..') || path.isAbsolute(edit.file)) {
|
||||||
|
errors.push(`Rejected: "${edit.file}" contains path traversal or absolute path.`);
|
||||||
|
return { valid: false, errors, warnings };
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check target file exists
|
||||||
|
const fileContent = promptContents[edit.file];
|
||||||
|
if (fileContent === undefined) {
|
||||||
|
errors.push(`Target file "${edit.file}" not found. Valid files: ${Object.keys(promptContents).join(', ')}`);
|
||||||
|
return { valid: false, errors, warnings };
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check old_text exists verbatim in the file
|
||||||
|
if (!fileContent.includes(edit.old_text)) {
|
||||||
|
errors.push(`old_text not found verbatim in "${edit.file}". The AI may have hallucinated text.`);
|
||||||
|
} else {
|
||||||
|
// Warn if old_text appears more than once (ambiguous match)
|
||||||
|
const occurrences = fileContent.split(edit.old_text).length - 1;
|
||||||
|
if (occurrences > 1) {
|
||||||
|
warnings.push(`old_text appears ${occurrences} times in "${edit.file}". Only the first occurrence will be replaced.`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check char count after edit
|
||||||
|
const afterContent = fileContent.replace(edit.old_text, edit.new_text);
|
||||||
|
if (afterContent.length > CHAR_LIMIT) {
|
||||||
|
errors.push(`Edit would cause "${edit.file}" to exceed ${CHAR_LIMIT} char limit (would be ${afterContent.length} chars).`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify reported char counts match reality
|
||||||
|
const actualBefore = fileContent.length;
|
||||||
|
const actualAfter = afterContent.length;
|
||||||
|
if (Math.abs(edit.char_count_before - actualBefore) > 10) {
|
||||||
|
warnings.push(`Reported char_count_before (${edit.char_count_before}) differs from actual (${actualBefore}).`);
|
||||||
|
}
|
||||||
|
if (Math.abs(edit.char_count_after - actualAfter) > 10) {
|
||||||
|
warnings.push(`Reported char_count_after (${edit.char_count_after}) differs from actual (${actualAfter}).`);
|
||||||
|
}
|
||||||
|
|
||||||
|
return { valid: errors.length === 0, errors, warnings };
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── AI response parsing ───────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export function parseProposalFromResponse(response: string): PromptEdit[] | null {
|
||||||
|
// Look for JSON code block containing PromptEdit[]
|
||||||
|
const jsonBlockMatch = response.match(/```(?:json)?\s*(\[[\s\S]*?\])\s*```/);
|
||||||
|
if (!jsonBlockMatch) {
|
||||||
|
// Try bare JSON array
|
||||||
|
const bareMatch = response.match(/(\[[\s\S]*"old_text"[\s\S]*\])/);
|
||||||
|
if (!bareMatch) return null;
|
||||||
|
try {
|
||||||
|
return JSON.parse(bareMatch[1]) as PromptEdit[];
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
return JSON.parse(jsonBlockMatch[1]) as PromptEdit[];
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Main improver ─────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export interface ImproverOptions {
|
||||||
|
diagnosisPath: string; // Path to diagnosis markdown file
|
||||||
|
plan2codeRoot: string; // Path to plan2code repo root
|
||||||
|
proposalsDir: string; // Where to save proposal JSON
|
||||||
|
runsDir: string; // For tracking which runs this is based on
|
||||||
|
model?: string;
|
||||||
|
agent?: AgentType; // Agent to use (default: claude-code)
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ImproverResult {
|
||||||
|
proposalPath: string;
|
||||||
|
proposal: PromptProposal;
|
||||||
|
validationResults: Array<{ edit: PromptEdit; result: ValidationResult }>;
|
||||||
|
validEditCount: number;
|
||||||
|
invalidEditCount: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function generateImprovement(opts: ImproverOptions): Promise<ImproverResult> {
|
||||||
|
const { diagnosisPath, plan2codeRoot, proposalsDir, runsDir, model = 'claude-opus-4-6', agent = 'claude-code' } = opts;
|
||||||
|
|
||||||
|
// Load diagnosis
|
||||||
|
let diagnosisContent: string;
|
||||||
|
try {
|
||||||
|
diagnosisContent = fs.readFileSync(diagnosisPath, 'utf8');
|
||||||
|
} catch {
|
||||||
|
throw new Error(`Could not read diagnosis file at ${diagnosisPath}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read prompt files
|
||||||
|
const promptContents = readPromptFiles(plan2codeRoot);
|
||||||
|
const srcDir = path.join(plan2codeRoot, 'src');
|
||||||
|
|
||||||
|
// Build char counts for each file
|
||||||
|
const charCounts = Object.entries(promptContents)
|
||||||
|
.map(([file, content]) => `| ${file} | ${content.length} | ${CHAR_LIMIT} | ${CHAR_LIMIT - content.length} headroom |`)
|
||||||
|
.join('\n');
|
||||||
|
|
||||||
|
const promptContentsStr = Object.entries(promptContents)
|
||||||
|
.map(([file, content]) => `## ${file} (${content.length} chars)\n\n${content}`)
|
||||||
|
.join('\n\n---\n\n');
|
||||||
|
|
||||||
|
// Load improve prompt template
|
||||||
|
let improveTemplate: string;
|
||||||
|
try {
|
||||||
|
improveTemplate = fs.readFileSync(IMPROVE_PROMPT_PATH, 'utf8');
|
||||||
|
} catch {
|
||||||
|
const altPath = path.join(process.cwd(), 'src', 'prompts', 'improve.md');
|
||||||
|
improveTemplate = fs.readFileSync(altPath, 'utf8');
|
||||||
|
}
|
||||||
|
|
||||||
|
const fullPrompt = interpolate(improveTemplate, {
|
||||||
|
diagnosisContent,
|
||||||
|
promptContents: promptContentsStr,
|
||||||
|
charCounts: `| File | Current Chars | Limit | Headroom |\n|------|--------------|-------|----------|\n${charCounts}`,
|
||||||
|
});
|
||||||
|
|
||||||
|
// Invoke Claude
|
||||||
|
console.log(`\nInvoking AI improvement proposal (model: ${model})...`);
|
||||||
|
console.log('This may take a minute...\n');
|
||||||
|
|
||||||
|
let aiResponse: string;
|
||||||
|
try {
|
||||||
|
aiResponse = await invokeLLM({
|
||||||
|
prompt: fullPrompt,
|
||||||
|
model,
|
||||||
|
agent,
|
||||||
|
timeout: 300_000,
|
||||||
|
});
|
||||||
|
} catch (err) {
|
||||||
|
throw new Error(`AI invocation failed: ${err instanceof Error ? err.message : String(err)}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse edits
|
||||||
|
const rawEdits = parseProposalFromResponse(aiResponse);
|
||||||
|
if (!rawEdits || rawEdits.length === 0) {
|
||||||
|
throw new Error('Could not parse PromptEdit[] from AI response. The AI may not have produced a valid JSON block.');
|
||||||
|
}
|
||||||
|
|
||||||
|
// Enforce max edits per cycle
|
||||||
|
const MAX_EDITS = 5;
|
||||||
|
if (rawEdits.length > MAX_EDITS) {
|
||||||
|
console.warn(`\n⚠ AI generated ${rawEdits.length} edits (max is ${MAX_EDITS}). Truncating to first ${MAX_EDITS}.`);
|
||||||
|
rawEdits.length = MAX_EDITS;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate each edit
|
||||||
|
const validationResults: ImproverResult['validationResults'] = [];
|
||||||
|
const validEdits: PromptEdit[] = [];
|
||||||
|
|
||||||
|
for (const edit of rawEdits) {
|
||||||
|
const result = validateEdit(edit, promptContents);
|
||||||
|
validationResults.push({ edit, result });
|
||||||
|
|
||||||
|
if (result.valid) {
|
||||||
|
// Compute accurate char counts
|
||||||
|
const fileContent = promptContents[edit.file] ?? '';
|
||||||
|
const afterContent = fileContent.replace(edit.old_text, edit.new_text);
|
||||||
|
edit.char_count_before = fileContent.length;
|
||||||
|
edit.char_count_after = afterContent.length;
|
||||||
|
edit.char_count_delta = afterContent.length - fileContent.length;
|
||||||
|
validEdits.push(edit);
|
||||||
|
} else {
|
||||||
|
console.warn(`\n⚠ Edit rejected for "${edit.file}":`);
|
||||||
|
for (const err of result.errors) {
|
||||||
|
console.warn(` - ${err}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const warn of result.warnings) {
|
||||||
|
console.warn(` Warning: ${warn}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get run IDs that contributed to this analysis
|
||||||
|
const runIds: string[] = [];
|
||||||
|
try {
|
||||||
|
const files = fs.readdirSync(runsDir)
|
||||||
|
.filter(f => f.startsWith('run-') && f.endsWith('.json'));
|
||||||
|
runIds.push(...files.map(f => f.replace('.json', '')));
|
||||||
|
} catch { /* no runs dir */ }
|
||||||
|
|
||||||
|
// Build proposal
|
||||||
|
const proposalId = generateProposalId();
|
||||||
|
const proposal: PromptProposal = {
|
||||||
|
proposal_id: proposalId,
|
||||||
|
created_at: new Date().toISOString(),
|
||||||
|
based_on_runs: runIds,
|
||||||
|
analyst_model: model,
|
||||||
|
proposals: validEdits,
|
||||||
|
status: 'pending',
|
||||||
|
diagnosis_file: path.basename(diagnosisPath),
|
||||||
|
};
|
||||||
|
|
||||||
|
// Save proposal JSON
|
||||||
|
fs.mkdirSync(proposalsDir, { recursive: true });
|
||||||
|
const proposalPath = path.join(proposalsDir, `${proposalId}.json`);
|
||||||
|
fs.writeFileSync(proposalPath, JSON.stringify(proposal, null, 2), 'utf8');
|
||||||
|
|
||||||
|
// Also save raw AI response alongside
|
||||||
|
const rawPath = path.join(proposalsDir, `${proposalId}-raw.md`);
|
||||||
|
fs.writeFileSync(rawPath, aiResponse, 'utf8');
|
||||||
|
|
||||||
|
return {
|
||||||
|
proposalPath,
|
||||||
|
proposal,
|
||||||
|
validationResults,
|
||||||
|
validEditCount: validEdits.length,
|
||||||
|
invalidEditCount: rawEdits.length - validEdits.length,
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
// Public API for plan2code-metrics
|
||||||
|
export { collectRun, collectPromptVersions } from './collector.js';
|
||||||
|
export { aggregate, loadAggregated, loadRunFiles, importRun } from './aggregator.js';
|
||||||
|
export { runAnalysis } from './analyzer.js';
|
||||||
|
export { generateImprovement, validateEdit, parseProposalFromResponse } from './improver.js';
|
||||||
|
export { reviewAndApply } from './applier.js';
|
||||||
|
export { invokeLLM, AGENTS } from './invoke-llm.js';
|
||||||
|
export type { AgentType, InvokeLLMOptions } from './invoke-llm.js';
|
||||||
|
export { runCLI } from './cli.js';
|
||||||
|
export { METRIC_TARGETS } from './types.js';
|
||||||
|
export type {
|
||||||
|
RunMetrics,
|
||||||
|
UserFeedback,
|
||||||
|
PromptVersions,
|
||||||
|
PromptEdit,
|
||||||
|
PromptProposal,
|
||||||
|
AggregatedMetrics,
|
||||||
|
CohortMetrics,
|
||||||
|
} from './types.js';
|
||||||
@@ -0,0 +1,98 @@
|
|||||||
|
/**
|
||||||
|
* invoke-llm.ts
|
||||||
|
* Unified LLM invocation for plan2code-metrics.
|
||||||
|
* Supports Claude Code (temp file → stdin) and Copilot CLI (stdin string).
|
||||||
|
* Mirrors the agent pattern from plan2code-loop.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { execa } from 'execa';
|
||||||
|
import { writeFileSync, unlinkSync } from 'fs';
|
||||||
|
import { join } from 'path';
|
||||||
|
import { tmpdir } from 'os';
|
||||||
|
|
||||||
|
// ── Agent definitions ────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export type AgentType = 'claude-code' | 'copilot-cli';
|
||||||
|
|
||||||
|
export interface AgentDef {
|
||||||
|
name: AgentType;
|
||||||
|
displayName: string;
|
||||||
|
command: string;
|
||||||
|
models: Array<{ value: string; label: string }>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const AGENTS: Record<AgentType, AgentDef> = {
|
||||||
|
'claude-code': {
|
||||||
|
name: 'claude-code',
|
||||||
|
displayName: 'Claude Code',
|
||||||
|
command: 'claude',
|
||||||
|
models: [
|
||||||
|
{ value: 'claude-opus-4-6', label: 'Claude Opus 4.6 (Recommended)' },
|
||||||
|
{ value: 'claude-sonnet-4-6', label: 'Claude Sonnet 4.6 (faster)' },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
'copilot-cli': {
|
||||||
|
name: 'copilot-cli',
|
||||||
|
displayName: 'GitHub Copilot CLI',
|
||||||
|
command: 'copilot',
|
||||||
|
models: [
|
||||||
|
{ value: 'claude-sonnet-4', label: 'Claude Sonnet 4 (Default)' },
|
||||||
|
{ value: 'claude-sonnet-4.5', label: 'Claude Sonnet 4.5' },
|
||||||
|
{ value: 'claude-opus-4.5', label: 'Claude Opus 4.5' },
|
||||||
|
{ value: 'gpt-5', label: 'GPT-5' },
|
||||||
|
{ value: 'gpt-5-mini', label: 'GPT-5 Mini' },
|
||||||
|
{ value: 'gemini-3-pro-preview', label: 'Gemini 3 Pro' },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
// ── Invocation ───────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export interface InvokeLLMOptions {
|
||||||
|
prompt: string;
|
||||||
|
model: string;
|
||||||
|
agent: AgentType;
|
||||||
|
timeout?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function invokeLLM(opts: InvokeLLMOptions): Promise<string> {
|
||||||
|
const { prompt, model, agent, timeout = 300_000 } = opts;
|
||||||
|
const def = AGENTS[agent];
|
||||||
|
|
||||||
|
if (agent === 'claude-code') {
|
||||||
|
// Write prompt to temp file — more reliable than stdin on Windows
|
||||||
|
const tempFile = join(tmpdir(), `plan2code-metrics-prompt-${Date.now()}.txt`);
|
||||||
|
writeFileSync(tempFile, prompt, 'utf-8');
|
||||||
|
|
||||||
|
try {
|
||||||
|
const args: string[] = [
|
||||||
|
'--print',
|
||||||
|
'--dangerously-skip-permissions',
|
||||||
|
];
|
||||||
|
if (model) {
|
||||||
|
args.push('--model', model);
|
||||||
|
}
|
||||||
|
|
||||||
|
const result = await execa(def.command, args, {
|
||||||
|
inputFile: tempFile,
|
||||||
|
timeout,
|
||||||
|
});
|
||||||
|
return result.stdout;
|
||||||
|
} finally {
|
||||||
|
try { unlinkSync(tempFile); } catch { /* ignore cleanup errors */ }
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// Copilot CLI: pipe prompt via stdin string
|
||||||
|
const args: string[] = [];
|
||||||
|
if (model) {
|
||||||
|
args.push('--model', model);
|
||||||
|
}
|
||||||
|
args.push('--allow-all-tools', '-s');
|
||||||
|
|
||||||
|
const result = await execa(def.command, args, {
|
||||||
|
input: prompt,
|
||||||
|
timeout,
|
||||||
|
});
|
||||||
|
return result.stdout;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,101 @@
|
|||||||
|
# PLAN2CODE METRICS ANALYSIS REQUEST
|
||||||
|
|
||||||
|
You are a senior AI systems analyst specializing in prompt engineering quality assessment. Your role is to diagnose weaknesses in the plan2code workflow prompts by examining aggregated run metrics.
|
||||||
|
|
||||||
|
**IMPORTANT:** Do NOT propose specific edits in this response. Diagnosis only. The improvement step is separate.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Aggregated Run Metrics
|
||||||
|
|
||||||
|
The following JSON contains metrics aggregated from real plan2code project runs, grouped by prompt "generation" (a cohort is identified by the SHA fingerprint of the src/plan2code-*.md prompt files at collection time):
|
||||||
|
|
||||||
|
```json
|
||||||
|
{{aggregatedMetrics}}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current Prompt File Contents
|
||||||
|
|
||||||
|
The following are the current contents of the plan2code workflow prompt files being evaluated:
|
||||||
|
|
||||||
|
{{promptContents}}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Metric Targets Reference
|
||||||
|
|
||||||
|
| Metric | Target | Direction |
|
||||||
|
|--------|--------|-----------|
|
||||||
|
| avg_confidence (Step 1) | ≥ 90 | higher is better |
|
||||||
|
| avg_clarification_rounds (Step 1) | ≤ 2.0 | lower is better |
|
||||||
|
| avg_verification_gaps_found (Step 1) | ≤ 2.0 | lower is better |
|
||||||
|
| avg_parallel_groups (Step 2) | ≥ 0.5 | higher is better |
|
||||||
|
| avg_verification_items_added (Step 2) | ≤ 1.5 | lower is better |
|
||||||
|
| avg_task_completion_rate (Step 3) | ≥ 0.95 | higher is better |
|
||||||
|
| avg_blocker_count (Step 3) | ≤ 1.5 | lower is better |
|
||||||
|
| avg_completion_marker_success_rate (Step 3) | ≥ 0.95 | higher is better |
|
||||||
|
| avg_verification_failures_found (Step 4) | ≤ 1.0 | lower is better |
|
||||||
|
| archival_success_rate (Step 4) | ≥ 0.99 | higher is better |
|
||||||
|
| avg_user_rating (Feedback) | ≥ 7.0 | higher is better (1-10 scale, null if no feedback) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Analysis Instructions
|
||||||
|
|
||||||
|
1. Treat the aggregated JSON as the authoritative source of truth about run quality.
|
||||||
|
2. Compare each metric against its target. Calculate delta (actual − target).
|
||||||
|
3. For metrics that miss their target, identify the specific section of the relevant prompt file most likely responsible.
|
||||||
|
4. Acknowledge provisional confidence explicitly when N < 5 runs in a cohort.
|
||||||
|
5. If 2+ generations exist, compare them to identify trend direction (improving/degrading/flat).
|
||||||
|
6. Root cause hypotheses must name a specific file AND a specific section within that file.
|
||||||
|
7. Do not invent metrics not present in the JSON. If a metric is null, note it as "insufficient data."
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Required Output Format
|
||||||
|
|
||||||
|
Produce EXACTLY the following sections in order. Use these exact headers — they are parsed by machine:
|
||||||
|
|
||||||
|
# PLAN2CODE METRICS DIAGNOSIS
|
||||||
|
|
||||||
|
### Metrics Summary
|
||||||
|
|
||||||
|
A markdown table with columns: Step | Metric | Target | Actual | Delta | Status (✓/✗/—)
|
||||||
|
|
||||||
|
Include ALL metrics listed in the targets table. Use "—" for null values.
|
||||||
|
|
||||||
|
### Step Health Assessment
|
||||||
|
|
||||||
|
For each step (1–4), provide:
|
||||||
|
- **Grade:** A–F
|
||||||
|
- **Key signals:** 2–4 bullet points with specific metric values
|
||||||
|
- **Assessment:** 1–2 sentence diagnosis
|
||||||
|
|
||||||
|
### Root Cause Hypotheses
|
||||||
|
|
||||||
|
Numbered list. For each underperforming metric:
|
||||||
|
1. **Metric:** [metric name] | **Value:** [actual] | **Target:** [target]
|
||||||
|
- **File:** [plan2code-X--name.md]
|
||||||
|
- **Section:** [specific heading or section name]
|
||||||
|
- **Hypothesis:** [specific gap in the prompt that would explain the metric miss]
|
||||||
|
- **Confidence:** [High/Medium/Low] — [reason for confidence level]
|
||||||
|
|
||||||
|
### Recommended Improvement Targets
|
||||||
|
|
||||||
|
Ordered list (highest estimated impact first). For each:
|
||||||
|
- **File:** [filename]
|
||||||
|
- **Section:** [section name]
|
||||||
|
- **Why:** [link to specific metric being addressed]
|
||||||
|
- **Priority:** [High/Medium/Low]
|
||||||
|
|
||||||
|
### Generation Comparison
|
||||||
|
|
||||||
|
If 2+ generations exist: A comparison table showing before/after for each metric per generation, with trend arrows (▲/▼/→).
|
||||||
|
|
||||||
|
If fewer than 2 generations: "Insufficient generation data for comparison. Current generation: [cohort_key], [N] runs."
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
End of analysis request.
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
# PLAN2CODE PROMPT IMPROVEMENT REQUEST
|
||||||
|
|
||||||
|
You are a senior AI prompt engineer. Your role is to propose surgical, targeted edits to the plan2code workflow prompt files based on a metrics diagnosis. You make precise, minimal changes — NOT rewrites.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Metrics Diagnosis
|
||||||
|
|
||||||
|
The following diagnosis was produced by the analysis step:
|
||||||
|
|
||||||
|
{{diagnosisContent}}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current Prompt File Contents (with char counts)
|
||||||
|
|
||||||
|
{{promptContents}}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Character Count Status
|
||||||
|
|
||||||
|
{{charCounts}}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Hard Constraints — ALL must be satisfied:
|
||||||
|
|
||||||
|
1. **Char limit:** Each target file MUST stay under 11,000 characters after your edit is applied. This is enforced by code — edits that violate it will be automatically rejected.
|
||||||
|
2. **Maximum 5 edits per cycle.** Focus on the highest-impact changes only.
|
||||||
|
3. **Each edit must cite a specific metric** in its `expected_metric_impact` field (e.g., "avg_completion_marker_success_rate", "avg_blocker_count").
|
||||||
|
4. **`old_text` must be verbatim** from the file. Copy-paste exactly — include surrounding whitespace/newlines as they appear. Edits with mismatched old_text will be automatically rejected.
|
||||||
|
5. **Do NOT modify Role sections** (lines starting with "You are" at the top of each file) or step headings (lines starting with `#`).
|
||||||
|
6. **Each edit must be independently applicable** — no edit should depend on another edit being applied first.
|
||||||
|
7. **Prefer additive guidance over deletions.** Adding clarifying instructions or examples is safer than removing existing text.
|
||||||
|
8. **Do not change the overall structure** or flow of any prompt file.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Edit Strategy Guidelines
|
||||||
|
|
||||||
|
- Target the specific sections identified in "Recommended Improvement Targets" from the diagnosis.
|
||||||
|
- For high `avg_clarification_rounds`: Add more upfront specification examples or decision criteria to Step 1.
|
||||||
|
- For low `avg_completion_marker_success_rate`: Clarify or simplify the completion marker format in Step 3.
|
||||||
|
- For high `avg_blocker_count`: Add blocker-recovery guidance or prerequisite check instructions.
|
||||||
|
- For low `avg_confidence`: Strengthen the confidence calculation instructions with clearer rubrics.
|
||||||
|
- For low `avg_parallel_groups`: Add explicit guidance for identifying parallel tasks in Step 2.
|
||||||
|
- For low `avg_user_rating`: Review user feedback themes (what_went_well, what_went_poorly) for systemic issues.
|
||||||
|
- Keep each `new_text` as short as possible while still addressing the root cause.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Required Output Format
|
||||||
|
|
||||||
|
Produce EXACTLY the following sections. The JSON block is parsed by machine — it must be syntactically valid.
|
||||||
|
|
||||||
|
# PLAN2CODE PROMPT IMPROVEMENT PROPOSAL
|
||||||
|
|
||||||
|
### Improvement Rationale
|
||||||
|
|
||||||
|
2–3 paragraphs explaining:
|
||||||
|
1. Which metrics are being addressed and why they matter
|
||||||
|
2. The specific prompt gaps identified in the diagnosis that you are targeting
|
||||||
|
3. Why the proposed edits are expected to improve those metrics
|
||||||
|
|
||||||
|
### Proposed Edits
|
||||||
|
|
||||||
|
```json
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"file": "plan2code-X--name.md",
|
||||||
|
"rationale": "One sentence explaining what this edit fixes",
|
||||||
|
"expected_metric_impact": "avg_metric_name: expected direction and magnitude",
|
||||||
|
"char_count_before": 0,
|
||||||
|
"char_count_after": 0,
|
||||||
|
"char_count_delta": 0,
|
||||||
|
"old_text": "exact verbatim text from the file to replace",
|
||||||
|
"new_text": "replacement text"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
Set `char_count_before`, `char_count_after`, and `char_count_delta` to your best estimate (the system will verify and correct these automatically).
|
||||||
|
|
||||||
|
### Character Count Verification
|
||||||
|
|
||||||
|
A table with columns: File | Before | Projected After | Delta | Limit | Status (✓/✗)
|
||||||
|
|
||||||
|
Verify that NO file exceeds 11,000 characters after your proposed edits.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
End of improvement request.
|
||||||
@@ -0,0 +1,175 @@
|
|||||||
|
// All TypeScript interfaces for plan2code-metrics
|
||||||
|
|
||||||
|
export interface PromptVersions {
|
||||||
|
plan: string; // sha256:... plan2code-1--plan.md
|
||||||
|
revise_plan: string; // plan2code-1b--revise-plan.md
|
||||||
|
document: string; // plan2code-2--document.md
|
||||||
|
implement: string; // plan2code-3--implement.md
|
||||||
|
finalize: string; // plan2code-4--finalize.md
|
||||||
|
init: string; // plan2code---init.md
|
||||||
|
init_update: string; // plan2code---init-update.md
|
||||||
|
quick_task: string; // plan2code---quick-task.md
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface Step1PlanMetrics {
|
||||||
|
present: boolean;
|
||||||
|
final_confidence: number | null;
|
||||||
|
confidence_breakdown: {
|
||||||
|
requirements: number | null;
|
||||||
|
feasibility: number | null;
|
||||||
|
integration: number | null;
|
||||||
|
risk: number | null;
|
||||||
|
} | null;
|
||||||
|
clarification_rounds: number | null;
|
||||||
|
tech_stack_revision_rounds: number | null;
|
||||||
|
verification_gaps_found: number | null;
|
||||||
|
functional_requirements_count: number | null;
|
||||||
|
non_functional_requirements_count: number | null;
|
||||||
|
risk_count: number | null;
|
||||||
|
phase_count: number | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface Step2DocumentMetrics {
|
||||||
|
present: boolean;
|
||||||
|
total_tasks: number | null;
|
||||||
|
tasks_per_phase: number[] | null;
|
||||||
|
phase_count: number | null;
|
||||||
|
parallel_groups_identified: number | null;
|
||||||
|
requirement_coverage_percent: number | null;
|
||||||
|
verification_items_added: number | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface Step3ImplementMetrics {
|
||||||
|
present: boolean;
|
||||||
|
used_loop_mode: boolean;
|
||||||
|
loop_mode: 'task' | 'phase' | null;
|
||||||
|
task_completion_rate: number | null;
|
||||||
|
tasks_completed: number | null;
|
||||||
|
tasks_total: number | null;
|
||||||
|
blocker_count: number | null;
|
||||||
|
blocker_categories: string[] | null;
|
||||||
|
total_iterations: number | null;
|
||||||
|
avg_iteration_duration_ms: number | null;
|
||||||
|
exit_code_distribution: Record<string, number> | null;
|
||||||
|
completion_marker_success_rate: number | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface Step4FinalizeMetrics {
|
||||||
|
present: boolean;
|
||||||
|
completion_rate_at_audit: number | null;
|
||||||
|
verification_failures_found: number | null;
|
||||||
|
documentation_updates_needed: number | null;
|
||||||
|
archival_succeeded: boolean | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface UserFeedback {
|
||||||
|
overall_rating: number; // 1-10
|
||||||
|
rating_reason: string;
|
||||||
|
what_went_well: string;
|
||||||
|
what_went_poorly: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface RunMetrics {
|
||||||
|
schema_version: '1.0';
|
||||||
|
run_id: string;
|
||||||
|
plan2code_version: string;
|
||||||
|
prompt_versions: PromptVersions;
|
||||||
|
project: {
|
||||||
|
name: string;
|
||||||
|
started_at: string | null;
|
||||||
|
completed_at: string | null;
|
||||||
|
};
|
||||||
|
step1_plan: Step1PlanMetrics;
|
||||||
|
step2_document: Step2DocumentMetrics;
|
||||||
|
step3_implement: Step3ImplementMetrics;
|
||||||
|
step4_finalize: Step4FinalizeMetrics;
|
||||||
|
user_feedback: UserFeedback | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PromptEdit {
|
||||||
|
file: string;
|
||||||
|
rationale: string;
|
||||||
|
expected_metric_impact: string;
|
||||||
|
char_count_before: number;
|
||||||
|
char_count_after: number;
|
||||||
|
char_count_delta: number;
|
||||||
|
old_text: string;
|
||||||
|
new_text: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PromptProposal {
|
||||||
|
proposal_id: string;
|
||||||
|
created_at: string;
|
||||||
|
based_on_runs: string[];
|
||||||
|
analyst_model: string;
|
||||||
|
proposals: PromptEdit[];
|
||||||
|
status: 'pending' | 'applied' | 'rejected';
|
||||||
|
diagnosis_file: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Aggregated metrics schema
|
||||||
|
export interface CohortMetrics {
|
||||||
|
cohort_key: string; // hash of sorted prompt_versions
|
||||||
|
prompt_versions: PromptVersions;
|
||||||
|
run_count: number;
|
||||||
|
run_ids: string[];
|
||||||
|
first_seen: string;
|
||||||
|
last_seen: string;
|
||||||
|
|
||||||
|
// Step 1 averages
|
||||||
|
avg_confidence: number | null;
|
||||||
|
avg_clarification_rounds: number | null;
|
||||||
|
avg_verification_gaps_found: number | null;
|
||||||
|
avg_functional_requirements_count: number | null;
|
||||||
|
avg_non_functional_requirements_count: number | null;
|
||||||
|
avg_risk_count: number | null;
|
||||||
|
avg_phase_count_step1: number | null;
|
||||||
|
|
||||||
|
// Step 2 averages
|
||||||
|
avg_total_tasks: number | null;
|
||||||
|
avg_phase_count_step2: number | null;
|
||||||
|
avg_parallel_groups: number | null;
|
||||||
|
avg_requirement_coverage_percent: number | null;
|
||||||
|
avg_verification_items_added: number | null;
|
||||||
|
|
||||||
|
// Step 3 averages (loop only)
|
||||||
|
loop_run_count: number;
|
||||||
|
avg_task_completion_rate: number | null;
|
||||||
|
avg_blocker_count: number | null;
|
||||||
|
avg_total_iterations: number | null;
|
||||||
|
avg_iteration_duration_ms: number | null;
|
||||||
|
avg_completion_marker_success_rate: number | null;
|
||||||
|
|
||||||
|
// Step 4 averages
|
||||||
|
avg_completion_rate_at_audit: number | null;
|
||||||
|
avg_verification_failures_found: number | null;
|
||||||
|
avg_documentation_updates_needed: number | null;
|
||||||
|
archival_success_rate: number | null;
|
||||||
|
|
||||||
|
// User feedback
|
||||||
|
avg_user_rating: number | null;
|
||||||
|
feedback_count: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface AggregatedMetrics {
|
||||||
|
schema_version: '1.0';
|
||||||
|
last_updated: string;
|
||||||
|
total_runs: number;
|
||||||
|
cohorts: CohortMetrics[];
|
||||||
|
current_cohort_key: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Metric targets for health assessment
|
||||||
|
export const METRIC_TARGETS = {
|
||||||
|
avg_confidence: { target: 90, direction: 'gte' as const },
|
||||||
|
avg_clarification_rounds: { target: 2.0, direction: 'lte' as const },
|
||||||
|
avg_verification_gaps_found: { target: 2.0, direction: 'lte' as const },
|
||||||
|
avg_parallel_groups: { target: 0.5, direction: 'gte' as const },
|
||||||
|
avg_verification_items_added: { target: 1.5, direction: 'lte' as const },
|
||||||
|
avg_task_completion_rate: { target: 0.95, direction: 'gte' as const },
|
||||||
|
avg_blocker_count: { target: 1.5, direction: 'lte' as const },
|
||||||
|
avg_completion_marker_success_rate: { target: 0.95, direction: 'gte' as const },
|
||||||
|
avg_verification_failures_found: { target: 1.0, direction: 'lte' as const },
|
||||||
|
archival_success_rate: { target: 0.99, direction: 'gte' as const },
|
||||||
|
avg_user_rating: { target: 7.0, direction: 'gte' as const },
|
||||||
|
} as const;
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ES2022",
|
||||||
|
"module": "ESNext",
|
||||||
|
"moduleResolution": "bundler",
|
||||||
|
"lib": ["ES2022"],
|
||||||
|
"outDir": "dist",
|
||||||
|
"rootDir": ".",
|
||||||
|
"strict": true,
|
||||||
|
"esModuleInterop": true,
|
||||||
|
"skipLibCheck": true,
|
||||||
|
"forceConsistentCasingInFileNames": true,
|
||||||
|
"resolveJsonModule": true,
|
||||||
|
"declaration": true,
|
||||||
|
"declarationMap": true,
|
||||||
|
"sourceMap": true
|
||||||
|
},
|
||||||
|
"include": ["src/**/*"],
|
||||||
|
"exclude": ["node_modules", "dist"]
|
||||||
|
}
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
import { defineConfig } from 'tsup';
|
||||||
|
|
||||||
|
export default defineConfig({
|
||||||
|
entry: {
|
||||||
|
'bin/plan2code-metrics': 'src/bin/plan2code-metrics.ts',
|
||||||
|
index: 'src/index.ts',
|
||||||
|
},
|
||||||
|
format: ['esm'],
|
||||||
|
dts: true,
|
||||||
|
clean: true,
|
||||||
|
sourcemap: true,
|
||||||
|
banner: {
|
||||||
|
js: '#!/usr/bin/env node',
|
||||||
|
},
|
||||||
|
});
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
import { defineConfig } from 'vitest/config';
|
||||||
|
|
||||||
|
export default defineConfig({
|
||||||
|
test: {
|
||||||
|
include: ['src/**/*.test.ts'],
|
||||||
|
},
|
||||||
|
});
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
|
||||||
|
const fs = require('fs');
|
||||||
|
const path = require('path');
|
||||||
|
|
||||||
|
const CHAR_LIMIT = 11000;
|
||||||
|
const srcDir = path.join(__dirname, '..', 'src');
|
||||||
|
|
||||||
|
const files = fs.readdirSync(srcDir)
|
||||||
|
.filter(f => f.startsWith('plan2code-') && f.endsWith('.md'))
|
||||||
|
.sort();
|
||||||
|
|
||||||
|
const failures = [];
|
||||||
|
|
||||||
|
for (const file of files) {
|
||||||
|
const filePath = path.join(srcDir, file);
|
||||||
|
const content = fs.readFileSync(filePath, 'utf8');
|
||||||
|
const charCount = content.length;
|
||||||
|
if (charCount > CHAR_LIMIT) {
|
||||||
|
failures.push({ file: path.join('src', file), charCount });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (failures.length > 0) {
|
||||||
|
console.error('Character limit exceeded:');
|
||||||
|
console.error('');
|
||||||
|
for (const { file, charCount } of failures) {
|
||||||
|
console.error(` ${file}: ${charCount} / ${CHAR_LIMIT} (+${charCount - CHAR_LIMIT} over)`);
|
||||||
|
}
|
||||||
|
console.error('');
|
||||||
|
console.error(`${failures.length} file(s) over the ${CHAR_LIMIT} character limit.`);
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`All ${files.length} files under ${CHAR_LIMIT} characters \u2713`);
|
||||||
|
process.exit(0);
|
||||||
@@ -18,7 +18,7 @@ Interactive Q&A flow to update an existing `AGENTS.md` with new learnings and pr
|
|||||||
|
|
||||||
Check if `AGENTS.md` exists in project root.
|
Check if `AGENTS.md` exists in project root.
|
||||||
|
|
||||||
**If missing:** "No AGENTS.md found. Create one from scratch? I can analyze the codebase and generate an initial file." Stop and wait. If yes, use `smarsh2code---init.md` workflow.
|
**If missing:** "No AGENTS.md found. Create one from scratch? I can analyze the codebase and generate an initial file." Stop and wait. If yes, use `plan2code---init.md` workflow.
|
||||||
|
|
||||||
**If exists:** Read and summarize:
|
**If exists:** Read and summarize:
|
||||||
- Main sections (bullets)
|
- Main sections (bullets)
|
||||||
@@ -49,8 +49,8 @@ Present the user with update options:
|
|||||||
|
|
||||||
> ```
|
> ```
|
||||||
> ⋅
|
> ⋅
|
||||||
> ╭───╮
|
> ╭───╮
|
||||||
> │ ● │
|
> │ ● │
|
||||||
> │ ~ │ What should we update?
|
> │ ~ │ What should we update?
|
||||||
> ╰───╯
|
> ╰───╯
|
||||||
> ```
|
> ```
|
||||||
@@ -140,10 +140,12 @@ Check for other AI agent config files and offer to replace with AGENTS.md refere
|
|||||||
### If Files Found
|
### If Files Found
|
||||||
|
|
||||||
```
|
```
|
||||||
> ╭───╮
|
o o
|
||||||
> │ ● │
|
\ /
|
||||||
> │ ~ │ Found other AI agent configs!
|
+---+
|
||||||
> ╰───╯
|
| o |
|
||||||
|
| ~ | Found other AI agent configs!
|
||||||
|
+---+
|
||||||
```
|
```
|
||||||
|
|
||||||
> Found AI config files that could reference AGENTS.md:
|
> Found AI config files that could reference AGENTS.md:
|
||||||
|
|||||||
@@ -16,8 +16,8 @@ Check for `./AGENTS.md` first:
|
|||||||
|
|
||||||
> ```
|
> ```
|
||||||
> ⋅
|
> ⋅
|
||||||
> ╭───╮
|
> ╭───╮
|
||||||
> │ ● │
|
> │ ● │ ?
|
||||||
> │ ~ │ Hmm, I don't see an AGENTS.md...
|
> │ ~ │ Hmm, I don't see an AGENTS.md...
|
||||||
> ╰───╯
|
> ╰───╯
|
||||||
> ```
|
> ```
|
||||||
@@ -142,13 +142,9 @@ Then tell user: "Created `specs/<feature-name>/PLAN-DRAFT-<date>.md`. Start new
|
|||||||
---
|
---
|
||||||
```
|
```
|
||||||
⋅
|
⋅
|
||||||
o o
|
|
||||||
╲ ╱
|
|
||||||
╭───╮
|
╭───╮
|
||||||
│ ★ │
|
│ ★ │
|
||||||
│ ◡ │ Plan ready! What do you think?
|
│ ◡ │ Plan ready! What do you think?
|
||||||
├───┤
|
|
||||||
│ · │
|
|
||||||
╰───╯
|
╰───╯
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
+21
-172
@@ -16,7 +16,7 @@ Check if `./AGENTS.md` exists:
|
|||||||
> ```
|
> ```
|
||||||
> ⋅
|
> ⋅
|
||||||
> ╭───╮
|
> ╭───╮
|
||||||
> │ ● │
|
> │ ● │ ?
|
||||||
> │ ~ │ Hmm, I don't see an AGENTS.md...
|
> │ ~ │ Hmm, I don't see an AGENTS.md...
|
||||||
> ╰───╯
|
> ╰───╯
|
||||||
> ```
|
> ```
|
||||||
@@ -61,7 +61,7 @@ Before Phase 1, check for `specs/*/PLAN-DRAFT-*.md`:
|
|||||||
When assumptions made, end with:
|
When assumptions made, end with:
|
||||||
**Assumptions this phase:** [Assumption] - [impact if wrong]
|
**Assumptions this phase:** [Assumption] - [impact if wrong]
|
||||||
|
|
||||||
User should confirm before next phase.
|
User MUST confirm before next phase.
|
||||||
|
|
||||||
### Confidence Calculation
|
### Confidence Calculation
|
||||||
|
|
||||||
@@ -76,20 +76,6 @@ Four dimensions (0-25% each):
|
|||||||
|
|
||||||
Report each sub-score with overall percentage.
|
Report each sub-score with overall percentage.
|
||||||
|
|
||||||
## Examples
|
|
||||||
|
|
||||||
### Requirement Gathering
|
|
||||||
**Bad:** "You want auth. I'll design JWT with bcrypt." (Made tech decisions without asking)
|
|
||||||
|
|
||||||
**Good:** "You mentioned authentication. Before proposing:
|
|
||||||
1. What methods? (email/password, social, SSO?)
|
|
||||||
2. Compliance? (SOC2, HIPAA?)
|
|
||||||
3. Token storage? (cookies, localStorage?)"
|
|
||||||
|
|
||||||
### Scope Assessment
|
|
||||||
**Bad:** "Medium project. Moving to Phase 4." (No justification)
|
|
||||||
|
|
||||||
**Good:** "Analysis: 8 requirements (Medium: 10-15), 4 components (Medium: 4-6), 2 integrations (Medium: 2-3). Appears **Medium**. Agree?"
|
|
||||||
|
|
||||||
## Process
|
## Process
|
||||||
|
|
||||||
@@ -231,160 +217,29 @@ State assessment and ask user to confirm.
|
|||||||
- Ask targeted questions
|
- Ask targeted questions
|
||||||
- State: "Need clarity on [areas] to improve [dimension] confidence."
|
- State: "Need clarity on [areas] to improve [dimension] confidence."
|
||||||
|
|
||||||
---
|
**Phase 7 Outputs:**
|
||||||
|
1. Save PLAN-CONVERSATION (see template below) — transcript + decision summary tables
|
||||||
#### STEP 7A: Save Conversation Log
|
2. Create PLAN-DRAFT (see template below) using same date
|
||||||
|
3. Verify: re-read conversation as source of truth, cross-reference against PLAN-DRAFT sections (Reqs→2.1/2.2, Tech→3, Architecture→4, Risks→6, Assumptions→9, Criteria→7). Add gaps with `<!-- VERIFICATION -->` comments. Output verification summary table.
|
||||||
Create `specs/<feature-name>/PLAN-CONVERSATION-<YYYYMMDD>.md`:
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
# Planning Conversation Log
|
|
||||||
|
|
||||||
**Feature:** [Name]
|
|
||||||
**Date:** [YYYYMMDD]
|
|
||||||
**Related:** specs/<feature-name>/PLAN-DRAFT-<date>.md
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Transcript
|
|
||||||
|
|
||||||
### Phase 1: Requirements Analysis
|
|
||||||
**[AGENT]** [Response]
|
|
||||||
**[USER]** [Response]
|
|
||||||
[...continue for all phases...]
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Decision Summary
|
|
||||||
|
|
||||||
### Key Decisions
|
|
||||||
| Phase | Decision | User Confirmation |
|
|
||||||
|-------|----------|-------------------|
|
|
||||||
|
|
||||||
### Requirements Confirmed
|
|
||||||
| ID | Requirement | Phase |
|
|
||||||
|----|-------------|-------|
|
|
||||||
|
|
||||||
### Technology Approved
|
|
||||||
| Tech | Category | Phase | Confirmation |
|
|
||||||
|------|----------|-------|--------------|
|
|
||||||
|
|
||||||
### Assumptions
|
|
||||||
| Assumption | Phase | Impact if Wrong | Acknowledged |
|
|
||||||
|------------|-------|-----------------|--------------|
|
|
||||||
```
|
|
||||||
|
|
||||||
Save before STEP 7B.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### STEP 7B: Create PLAN-DRAFT
|
|
||||||
|
|
||||||
Using same date, create `specs/<feature-name>/PLAN-DRAFT-<date>.md` (template below).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### STEP 7C: Verification Pass
|
|
||||||
|
|
||||||
After PLAN-DRAFT:
|
|
||||||
|
|
||||||
1. Re-read conversation log as source of truth
|
|
||||||
2. Verify against PLAN-DRAFT:
|
|
||||||
|
|
||||||
| From Conversation | Verify In PLAN-DRAFT |
|
|
||||||
|-------------------|----------------------|
|
|
||||||
| Functional requirements | 2.1 |
|
|
||||||
| Non-functional requirements | 2.2 |
|
|
||||||
| Technology decisions | 3. Tech Stack |
|
|
||||||
| Architecture decisions | 4. Architecture |
|
|
||||||
| Risks | 6. Risks |
|
|
||||||
| Assumptions | 9. Assumptions |
|
|
||||||
| Success criteria | 7. Success Criteria |
|
|
||||||
|
|
||||||
3. For gaps: Add with `<!-- VERIFICATION: Added from Phase X -->`
|
|
||||||
4. Output summary:
|
|
||||||
```
|
|
||||||
## Verification Complete
|
|
||||||
| Section | In Conversation | In PLAN-DRAFT | Added |
|
|
||||||
|---------|-----------------|---------------|-------|
|
|
||||||
**Status:** [All captured / X items added]
|
|
||||||
```
|
|
||||||
5. Save updated PLAN-DRAFT if needed
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Templates
|
## Templates
|
||||||
|
|
||||||
### PLAN-DRAFT Format
|
### PLAN-DRAFT Format
|
||||||
|
|
||||||
```markdown
|
File: `specs/<feature-name>/PLAN-DRAFT-<YYYYMMDD>.md`
|
||||||
# [Feature Name] - Implementation Plan
|
|
||||||
|
|
||||||
**Created:** [Date]
|
Header: Title, Created date, Status (Draft | Phase 3 Complete - Resume at Phase 4 | Complete), Confidence % (Reqs/Feasibility/Integration/Risk each /25), link to PLAN-CONVERSATION.
|
||||||
**Status:** Draft | Phase 3 Complete - Resume at Phase 4 | Complete
|
|
||||||
**Confidence:** [X]% (Reqs: X/25, Feasibility: X/25, Integration: X/25, Risk: X/25)
|
|
||||||
**Conversation Log:** specs/<feature-name>/PLAN-CONVERSATION-<date>.md
|
|
||||||
|
|
||||||
## 1. Executive Summary
|
Sections:
|
||||||
[2-3 sentences]
|
1. **Executive Summary** — 2-3 sentences
|
||||||
|
2. **Requirements** — 2.1 Functional (FR-N checklist), 2.2 Non-Functional (NFR-N checklist), 2.3 Out of Scope, 2.4 Testing Strategy table (Types, Phase Testing, Coverage)
|
||||||
## 2. Requirements
|
3. **Tech Stack** — table: Category / Technology / Version / Justification
|
||||||
### 2.1 Functional
|
4. **Architecture** — 4.1 Pattern (name + rationale), 4.2 System Context Diagram, 4.3 Components table, 4.4 Data Model, 4.5 API Design
|
||||||
- [ ] FR-1: [Description]
|
5. **Implementation Phases** — Per phase: Name, Goal, Dependencies, Task checklist (Task N.N)
|
||||||
|
6. **Risks and Mitigations** — table: Risk / Likelihood / Impact / Mitigation
|
||||||
### 2.2 Non-Functional
|
7. **Success Criteria** — checklist
|
||||||
- [ ] NFR-1: [Description]
|
8. **Open Questions** — remove if none
|
||||||
|
9. **Assumptions** — list
|
||||||
### 2.3 Out of Scope
|
|
||||||
- [Exclusions]
|
|
||||||
|
|
||||||
### 2.4 Testing Strategy
|
|
||||||
| Preference | Selection |
|
|
||||||
|------------|-----------|
|
|
||||||
| Types | [Unit/Integration/E2E/None] |
|
|
||||||
| Phase Testing | [After each/Dedicated/None] |
|
|
||||||
| Coverage | [Critical/Moderate/Comprehensive/N/A] |
|
|
||||||
|
|
||||||
## 3. Tech Stack
|
|
||||||
| Category | Technology | Version | Justification |
|
|
||||||
|----------|------------|---------|---------------|
|
|
||||||
|
|
||||||
## 4. Architecture
|
|
||||||
### 4.1 Pattern
|
|
||||||
[Name and rationale]
|
|
||||||
|
|
||||||
### 4.2 System Context Diagram
|
|
||||||
[ASCII or description]
|
|
||||||
|
|
||||||
### 4.3 Components
|
|
||||||
| Component | Responsibility | Dependencies |
|
|
||||||
|-----------|----------------|--------------|
|
|
||||||
|
|
||||||
### 4.4 Data Model
|
|
||||||
[Schema, relationships]
|
|
||||||
|
|
||||||
### 4.5 API Design
|
|
||||||
[Endpoints if applicable]
|
|
||||||
|
|
||||||
## 5. Implementation Phases
|
|
||||||
### Phase 1: [Name]
|
|
||||||
**Goal:** [Accomplishment]
|
|
||||||
**Dependencies:** None / [List]
|
|
||||||
- [ ] Task 1.1: [Description]
|
|
||||||
|
|
||||||
## 6. Risks and Mitigations
|
|
||||||
| Risk | Likelihood | Impact | Mitigation |
|
|
||||||
|------|------------|--------|------------|
|
|
||||||
|
|
||||||
## 7. Success Criteria
|
|
||||||
- [ ] [Criterion]
|
|
||||||
|
|
||||||
## 8. Open Questions
|
|
||||||
[Remove if none]
|
|
||||||
|
|
||||||
## 9. Assumptions
|
|
||||||
[List]
|
|
||||||
```
|
|
||||||
|
|
||||||
### Response Format
|
### Response Format
|
||||||
|
|
||||||
@@ -414,15 +269,9 @@ When complete (PLAN-DRAFT created), tell user:
|
|||||||
> │ ★ │
|
> │ ★ │
|
||||||
> │ ◡ │ Planning done! Ready for documentation!
|
> │ ◡ │ Planning done! Ready for documentation!
|
||||||
> ╰───╯
|
> ╰───╯
|
||||||
>
|
> =============================================
|
||||||
> ╔═══════════════════════════════════════════════════════════════════╗
|
> NEXT STEP: Start a NEW conversation then run:
|
||||||
> ║ NEXT STEPS ║
|
> `/plan2code-2--document`
|
||||||
> ╠═══════════════════════════════════════════════════════════════════╣
|
|
||||||
> ║ ║
|
|
||||||
> ║ 1. Start a NEW conversation ║
|
|
||||||
> ║ 2. Use command: /plan2code-2--document ║
|
|
||||||
> ║ ║
|
|
||||||
> ╚═══════════════════════════════════════════════════════════════════╝
|
|
||||||
> ```"
|
> ```"
|
||||||
|
|
||||||
## Abort Handling
|
## Abort Handling
|
||||||
|
|||||||
@@ -76,13 +76,9 @@ Present findings and confirm understanding before proceeding.
|
|||||||
|
|
||||||
```
|
```
|
||||||
⋅
|
⋅
|
||||||
o o
|
|
||||||
╲ ╱ ?
|
|
||||||
╭───╮
|
╭───╮
|
||||||
│ ● │
|
│ ● │
|
||||||
│ ~ │ Here's the plan. What do you think?
|
│ ~ │ Here's the plan. What do you think?
|
||||||
├───┤
|
|
||||||
│ · │
|
|
||||||
╰───╯
|
╰───╯
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
+10
-145
@@ -37,7 +37,7 @@ If no PLAN-DRAFT found and user hasn't provided one, ask for:
|
|||||||
|
|
||||||
If no plan exists and user wants to skip:
|
If no plan exists and user wants to skip:
|
||||||
> "Documentation transforms planning into specs. Without a plan, either:
|
> "Documentation transforms planning into specs. Without a plan, either:
|
||||||
> 1. Run planning first (`/smarsh2code-1--plan`)
|
> 1. Run planning first (`/plan2code-1--plan`)
|
||||||
> 2. Describe requirements so I can help create a minimal plan"
|
> 2. Describe requirements so I can help create a minimal plan"
|
||||||
|
|
||||||
## Phase Sizing
|
## Phase Sizing
|
||||||
@@ -50,14 +50,6 @@ If no plan exists and user wants to skip:
|
|||||||
| Independence | Testable/verifiable independently |
|
| Independence | Testable/verifiable independently |
|
||||||
| Dependencies | Logical dependency order |
|
| Dependencies | Logical dependency order |
|
||||||
|
|
||||||
**Typical progression:**
|
|
||||||
1. Project setup/configuration
|
|
||||||
2. Data models/database layer
|
|
||||||
3. Core business logic/services
|
|
||||||
4. API/Interface layer
|
|
||||||
5. Integration, error handling, polish
|
|
||||||
6. Additional features as needed
|
|
||||||
|
|
||||||
## Task Writing
|
## Task Writing
|
||||||
|
|
||||||
| Criterion | Description |
|
| Criterion | Description |
|
||||||
@@ -130,29 +122,9 @@ Or if none:
|
|||||||
| None | - | All phases must run sequentially |
|
| None | - | All phases must run sequentially |
|
||||||
```
|
```
|
||||||
|
|
||||||
### Documentation Verification
|
### Documentation Verification (Step 9)
|
||||||
|
|
||||||
**STEP 9A:** Re-read PLAN-DRAFT as source of truth
|
Re-read PLAN-DRAFT as source of truth. Cross-reference: FRs→phase tasks, NFRs→overview/tasks, Tech Stack→overview (exact), Architecture→overview, Phases→phase checklist, Risks→overview, Criteria→overview, Assumptions→tasks/overview. Fix gaps with `<!-- VERIFICATION: Added -->` comments. Output verification summary.
|
||||||
|
|
||||||
**STEP 9B:** Cross-reference:
|
|
||||||
|
|
||||||
| PLAN-DRAFT Section | Verify Against |
|
|
||||||
|--------------------|----------------|
|
|
||||||
| 2.1 Functional Requirements | phase-X.md tasks (each FR-X has tasks) |
|
|
||||||
| 2.2 Non-Functional Requirements | overview.md or tasks |
|
|
||||||
| 3 Tech Stack | overview.md (exact match) |
|
|
||||||
| 4.1 Architecture Pattern | overview.md |
|
|
||||||
| 4.3 Component Overview | overview.md |
|
|
||||||
| 5 Implementation Phases | Phase Checklist (all have phase-X.md) |
|
|
||||||
| 6 Risks and Mitigations | overview.md |
|
|
||||||
| 7 Success Criteria | overview.md |
|
|
||||||
| 9 Assumptions | Tasks or overview |
|
|
||||||
|
|
||||||
**STEP 9C:** For gaps:
|
|
||||||
- Missing requirement: Add task with `<!-- VERIFICATION: Added - FR-X from PLAN-DRAFT -->`
|
|
||||||
- Missing section: Add to overview.md with `<!-- VERIFICATION: Added from PLAN-DRAFT section X -->`
|
|
||||||
|
|
||||||
**STEP 9D:** Output verification summary
|
|
||||||
|
|
||||||
### Output Structure
|
### Output Structure
|
||||||
|
|
||||||
@@ -182,112 +154,15 @@ Use kebab-case for feature name (e.g., `user-authentication`).
|
|||||||
|
|
||||||
### overview.md
|
### overview.md
|
||||||
|
|
||||||
```markdown
|
Header: Title, Created date, Source (PLAN-DRAFT path), Status (Not Started | In Progress | Complete).
|
||||||
# [Feature Name] - Implementation Overview
|
|
||||||
|
|
||||||
**Created:** [Date]
|
Sections: Summary (from Executive Summary), Tech Stack table (exact copy from PLAN-DRAFT), Architecture (Pattern + Component Overview table), Risks and Mitigations table, Success Criteria checklist, Phase Checklist, Parallel Execution Groups table (from analysis), Quick Reference (Key Files, Environment Variables, External Dependencies), Completion Summary (filled during finalization).
|
||||||
**Source:** PLAN-DRAFT-<date>.md
|
|
||||||
**Status:** Not Started | In Progress | Complete
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
[From Executive Summary]
|
|
||||||
|
|
||||||
## Tech Stack
|
|
||||||
[Copy table from planning doc]
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
### Pattern
|
|
||||||
[From section 4.1]
|
|
||||||
|
|
||||||
### Component Overview
|
|
||||||
| Component | Responsibility | Dependencies |
|
|
||||||
|-----------|----------------|--------------|
|
|
||||||
[From section 4.3]
|
|
||||||
|
|
||||||
## Risks and Mitigations
|
|
||||||
| Risk | Likelihood | Impact | Mitigation |
|
|
||||||
|------|------------|--------|------------|
|
|
||||||
[From section 6]
|
|
||||||
|
|
||||||
## Success Criteria
|
|
||||||
[From section 7]
|
|
||||||
- [ ] [Criterion]
|
|
||||||
|
|
||||||
## Phase Checklist
|
|
||||||
- [ ] Phase 1: [Name] - [Description]
|
|
||||||
|
|
||||||
## Parallel Execution Groups
|
|
||||||
<!-- This section enables running multiple phases simultaneously in separate agent instances -->
|
|
||||||
<!-- Phases in the same group have no file conflicts or dependencies between them -->
|
|
||||||
|
|
||||||
| Group | Phases | Reason |
|
|
||||||
|-------|--------|--------|
|
|
||||||
| [A/None] | [numbers] | [Why parallel-eligible] |
|
|
||||||
|
|
||||||
## Quick Reference
|
|
||||||
### Key Files
|
|
||||||
[Files to be created]
|
|
||||||
|
|
||||||
### Environment Variables
|
|
||||||
[Required env vars or "None"]
|
|
||||||
|
|
||||||
### External Dependencies
|
|
||||||
[External services/APIs]
|
|
||||||
|
|
||||||
---
|
|
||||||
## Completion Summary
|
|
||||||
[Filled during finalization]
|
|
||||||
```
|
|
||||||
|
|
||||||
### phase-X.md
|
### phase-X.md
|
||||||
|
|
||||||
```markdown
|
Header: Phase name, Status, Estimated Tasks count.
|
||||||
# Phase X: [Name]
|
|
||||||
|
|
||||||
**Status:** Not Started | In Progress | Complete
|
Sections: Overview (2-3 sentences), Prerequisites checklist, Tasks (grouped by category, `- [ ] **Task X.N:** [Description]` with File path and details), Phase Testing (if enabled), Acceptance Criteria checklist, Notes, Phase Completion Summary (filled after implementation: date, implementer, what was done, files changed, issues).
|
||||||
**Estimated Tasks:** [N]
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
[2-3 sentences: what this phase accomplishes]
|
|
||||||
|
|
||||||
## Prerequisites
|
|
||||||
- [ ] Phase X-1 complete (if applicable)
|
|
||||||
- [ ] [Other prerequisites]
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
|
|
||||||
### [Category 1]
|
|
||||||
- [ ] **Task X.1:** [Description]
|
|
||||||
- File: `path/to/file`
|
|
||||||
- [Details]
|
|
||||||
|
|
||||||
### Phase Testing (if enabled)
|
|
||||||
- [ ] **Task X.N:** Run test suite
|
|
||||||
- Command: `[test command]`
|
|
||||||
- Pass criteria: [criteria]
|
|
||||||
|
|
||||||
## Acceptance Criteria
|
|
||||||
- [ ] [Verifiable criterion]
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
[Additional context]
|
|
||||||
|
|
||||||
---
|
|
||||||
## Phase Completion Summary
|
|
||||||
_[Filled after implementation]_
|
|
||||||
|
|
||||||
**Completed:** [Date]
|
|
||||||
**Implemented by:** [AI/human]
|
|
||||||
|
|
||||||
### What was done:
|
|
||||||
[Summary]
|
|
||||||
|
|
||||||
### Files created/modified:
|
|
||||||
- `path/to/file` - [description]
|
|
||||||
|
|
||||||
### Issues encountered:
|
|
||||||
[Issues or "None"]
|
|
||||||
```
|
|
||||||
|
|
||||||
## Session End
|
## Session End
|
||||||
|
|
||||||
@@ -333,19 +208,9 @@ Parallel Execution: [Groups or "None - sequential only"]
|
|||||||
> │ ★ │
|
> │ ★ │
|
||||||
> │ ◡ │ Specs are ready! Time to build!
|
> │ ◡ │ Specs are ready! Time to build!
|
||||||
> ╰───╯
|
> ╰───╯
|
||||||
>
|
> ============================================
|
||||||
> ╔═══════════════════════════════════════════════════════════════════╗
|
> NEXT STEP: Start a NEW conversation and run:
|
||||||
> ║ NEXT STEPS ║
|
> `/plan2code-3--implement`
|
||||||
> ╠═══════════════════════════════════════════════════════════════════╣
|
|
||||||
> ║ ║
|
|
||||||
> ║ 1. Start a NEW conversation ║
|
|
||||||
> ║ 2. Use command: /plan2code-3--implement ║
|
|
||||||
> ║ 3. Provide path: specs/<feature-name>/overview.md ║
|
|
||||||
> ║ ║
|
|
||||||
> ║ The command will auto-detect Phase 1 as the next phase. ║
|
|
||||||
> ║ Complete ONE phase per conversation. ║
|
|
||||||
> ║ ║
|
|
||||||
> ╚═══════════════════════════════════════════════════════════════════╝
|
|
||||||
> ```"
|
> ```"
|
||||||
|
|
||||||
## Abort Handling
|
## Abort Handling
|
||||||
|
|||||||
+50
-268
@@ -60,75 +60,16 @@ Never reset `[/]` to `[ ]`. Started work stays marked for conscious resume decis
|
|||||||
|
|
||||||
## Parallel Phase Selection
|
## Parallel Phase Selection
|
||||||
|
|
||||||
After identifying workable phases, check for parallel execution:
|
Check overview.md for "Parallel Execution Groups" section. Find workable phases (`[ ]` or `[/]`).
|
||||||
|
|
||||||
1. Look for "Parallel Execution Groups" section in overview.md
|
| Condition | Action |
|
||||||
2. Find if next phase belongs to a group with other uncompleted phases
|
|-----------|--------|
|
||||||
3. Only show consecutive uncompleted phases in same group
|
| 2+ workable phases in same parallel group | Show selection prompt listing each with status. TIP: run another agent on a different phase simultaneously. If all are `[/]`, warn about duplication. |
|
||||||
|
| Single `[/]` phase | Prompt: "Phase X is in progress. Resume? (yes/no)" |
|
||||||
|
| Single `[ ]` phase | Auto-start: mark `[/]` and begin |
|
||||||
|
| No parallel groups section | Sequential mode (single-phase rules above) |
|
||||||
|
|
||||||
**Decision Logic:**
|
Only show consecutive same-group incomplete phases. After selection, mark `[/]`, read `phase-X.md`, begin.
|
||||||
|
|
||||||
```
|
|
||||||
Find workable phases = all phases marked [ ] or [/] (not [x])
|
|
||||||
Check Parallel Execution Groups table:
|
|
||||||
|
|
||||||
CASE 1: Multiple parallel phases available
|
|
||||||
- If 2+ workable phases exist in the same parallel group
|
|
||||||
→ Show parallel selection UI with status for each
|
|
||||||
|
|
||||||
CASE 2: Single workable phase that is IN PROGRESS [/]
|
|
||||||
- Phase was started but not completed (possibly by another session)
|
|
||||||
→ Show resume prompt: "Phase X is in progress. Resume? (yes/no)"
|
|
||||||
|
|
||||||
CASE 3: Single workable phase that is PENDING [ ]
|
|
||||||
- Fresh phase, no parallel options
|
|
||||||
→ Auto-start: mark [/] and begin implementation
|
|
||||||
|
|
||||||
CASE 4: No Parallel Execution Groups section exists
|
|
||||||
→ Fall back to sequential mode using Cases 2-3 logic
|
|
||||||
```
|
|
||||||
|
|
||||||
**Parallel Selection UI:**
|
|
||||||
|
|
||||||
When parallel options are available (CASE 1), present this prompt:
|
|
||||||
|
|
||||||
```
|
|
||||||
⚡ PARALLEL PHASES AVAILABLE
|
|
||||||
|
|
||||||
These phases can run in parallel:
|
|
||||||
[1] Phase N: [Name] [IN PROGRESS]
|
|
||||||
[2] Phase N+1: [Name] [AVAILABLE]
|
|
||||||
[3] Phase N+2: [Name] [AVAILABLE]
|
|
||||||
|
|
||||||
Which phase? (1/2/3)
|
|
||||||
|
|
||||||
┌─────────────────────────────────────────────────────────────────────────┐
|
|
||||||
│ TIP: Run another agent instance with /smarsh2code-3--implement to work │
|
|
||||||
│ on a different phase simultaneously. │
|
|
||||||
│ │
|
|
||||||
│ IN PROGRESS phases may be running in another session - selecting one │
|
|
||||||
│ will resume work on it. │
|
|
||||||
└─────────────────────────────────────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
**Single Phase Resume UI:**
|
|
||||||
```
|
|
||||||
⚡ PHASE RESUME CHECK
|
|
||||||
|
|
||||||
Phase N: [Name] is IN PROGRESS.
|
|
||||||
- Another session may be working on it
|
|
||||||
- Previous session may have been aborted
|
|
||||||
|
|
||||||
Resume? (yes / no - I'll wait)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Rules:**
|
|
||||||
- Only show consecutive, same-group, incomplete phases (`[ ]` or `[/]`)
|
|
||||||
- Stop at first phase NOT in same group
|
|
||||||
- After selection, mark `[/]` if not already, read `phase-X.md`, begin
|
|
||||||
- If ALL parallel phases are `[/]`, warn about potential duplication
|
|
||||||
|
|
||||||
**Fallback:** If Parallel Execution Groups missing or shows "None", use sequential (Cases 2-3).
|
|
||||||
|
|
||||||
## Code Consistency Rules
|
## Code Consistency Rules
|
||||||
|
|
||||||
@@ -143,18 +84,9 @@ Resume? (yes / no - I'll wait)
|
|||||||
|
|
||||||
## Examples
|
## Examples
|
||||||
|
|
||||||
**Following Specs:**
|
**Following Specs:** Create exactly `src/services/UserService.ts` as specified — never rename or relocate.
|
||||||
- Bad: Task says "create UserService.ts" but creates "services/user.service.ts"
|
|
||||||
- Good: Creates exactly `src/services/UserService.ts` as specified
|
|
||||||
|
|
||||||
**Handling Blockers:**
|
**Blocker format:** `- [!] **Task 2.3:** ... > BLOCKED: [reason]. > User action: [action].` Then proceed to next non-dependent task.
|
||||||
```
|
|
||||||
- [!] **Task 2.3:** Connect to Stripe API
|
|
||||||
> BLOCKED: STRIPE_SECRET_KEY not in environment.
|
|
||||||
> User action: Add to .env
|
|
||||||
|
|
||||||
Proceeding to Task 2.4 (no Stripe dependency).
|
|
||||||
```
|
|
||||||
|
|
||||||
## Process
|
## Process
|
||||||
|
|
||||||
@@ -214,51 +146,17 @@ After all tasks:
|
|||||||
|
|
||||||
### 6. After User Approval
|
### 6. After User Approval
|
||||||
|
|
||||||
When user replies "approved":
|
See "After Approval / Session End" section below.
|
||||||
1. Update `overview.md`: `[/]` -> `[x]`
|
|
||||||
2. Update `phase-X.md`: Status -> "Complete"
|
|
||||||
3. Confirm and provide next steps
|
|
||||||
|
|
||||||
## Handling Blockers
|
## Issue Handling
|
||||||
|
|
||||||
**1. Mark as Blocked:**
|
| Type | Action | Format |
|
||||||
```markdown
|
|------|--------|--------|
|
||||||
- [!] **Task 3.2:** Create OAuth integration
|
| Blocker | Mark `[!]`, continue non-dependent tasks, report at phase end | `> BLOCKED: [reason]. User action: [action]` |
|
||||||
> BLOCKED: Missing GOOGLE_CLIENT_ID environment variable.
|
| Minor spec gap | Proceed with interpretation, note decision | `> SPEC NOTE: [what was assumed]` |
|
||||||
> Required: User must configure OAuth credentials.
|
| Major spec conflict | STOP and ask user — do NOT guess on architecture | `SPEC CONFLICT: [details]. Please clarify.` |
|
||||||
```
|
|
||||||
|
|
||||||
**2. Continue** with non-dependent tasks.
|
Default: ONE phase per conversation. Small phase (<5 tasks): ask if should continue with next. Large (>40): warn at start.
|
||||||
|
|
||||||
**3. Report** all blocked tasks at phase end.
|
|
||||||
|
|
||||||
## Handling Spec Issues
|
|
||||||
|
|
||||||
**Minor (proceed with interpretation):**
|
|
||||||
```markdown
|
|
||||||
- [x] **Task 2.4:** Create user validation
|
|
||||||
> SPEC NOTE: Format not specified. Implemented RFC 5322 email regex.
|
|
||||||
```
|
|
||||||
|
|
||||||
**Major (stop and ask):**
|
|
||||||
```markdown
|
|
||||||
[PHASE 2: Database Layer] - PAUSED
|
|
||||||
|
|
||||||
SPEC CONFLICT:
|
|
||||||
- Task 2.3: "email as primary key"
|
|
||||||
- Architecture section: "id (UUID) as primary key"
|
|
||||||
|
|
||||||
Please clarify before continuing.
|
|
||||||
```
|
|
||||||
|
|
||||||
Do NOT guess on architectural decisions.
|
|
||||||
|
|
||||||
## Phase Size Flexibility
|
|
||||||
|
|
||||||
- **Small phase (<5 tasks):** After completing, ask if should continue with next phase
|
|
||||||
- **Large phase (>40 tasks):** Warn at start, suggest breaking into sub-phases for future
|
|
||||||
|
|
||||||
Default: ONE phase per conversation.
|
|
||||||
|
|
||||||
## Templates
|
## Templates
|
||||||
|
|
||||||
@@ -278,34 +176,9 @@ Before sign-off, verify:
|
|||||||
|
|
||||||
### Completion Report Format
|
### Completion Report Format
|
||||||
|
|
||||||
```markdown
|
Header: `⚡ [PHASE X: Phase Name] - READY FOR SIGN-OFF`
|
||||||
⚡ [PHASE X: Phase Name] - READY FOR SIGN-OFF
|
|
||||||
|
|
||||||
## Summary
|
Sections: Summary (2-3 sentences), Tasks Completed (Y/Z + blocked list), Test Results table (if run), Files Created, Files Modified, Issues (or "None"), Verify (files exist, no syntax errors, app runs, 1-2 specific checks).
|
||||||
[2-3 sentences on accomplishments]
|
|
||||||
|
|
||||||
## Tasks Completed: Y/Z
|
|
||||||
[List blocked tasks if any]
|
|
||||||
|
|
||||||
## Test Results (if run)
|
|
||||||
| Run | Passed | Failed | Skipped |
|
|
||||||
|-----|--------|--------|---------|
|
|
||||||
| X | X | X | X |
|
|
||||||
|
|
||||||
## Files Created
|
|
||||||
- `path/to/file.ts` - [description]
|
|
||||||
|
|
||||||
## Files Modified
|
|
||||||
- `path/to/file.ts` - [changes]
|
|
||||||
|
|
||||||
## Issues
|
|
||||||
[Blockers, clarifications, deviations - or "None"]
|
|
||||||
|
|
||||||
## Verify
|
|
||||||
- Files listed above exist
|
|
||||||
- No syntax errors in editor
|
|
||||||
- App runs (if applicable)
|
|
||||||
- [1-2 specific checks for what was built]
|
|
||||||
|
|
||||||
```
|
```
|
||||||
⋅
|
⋅
|
||||||
@@ -315,108 +188,35 @@ Before sign-off, verify:
|
|||||||
╰───╯
|
╰───╯
|
||||||
```
|
```
|
||||||
|
|
||||||
╔═══════════════════════════════════════════════════════════════════════════════╗
|
> Reply "approved" to mark this phase complete, or describe any issues.
|
||||||
║ Reply "approved" to mark this phase complete, or describe any issues. ║
|
|
||||||
╚═══════════════════════════════════════════════════════════════════════════════╝
|
### After Approval / Session End
|
||||||
|
|
||||||
|
On user "approved":
|
||||||
|
1. Mark `[/]` → `[x]` in overview.md, update phase-X.md status to "Complete"
|
||||||
|
2. Show Planny art with completion message
|
||||||
|
3. Provide: `git add -A && git commit -m "Complete Phase X: [Phase Name]" -m "AI Assisted"`
|
||||||
|
4. **If more phases:** "NEXT STEP: Start NEW conversation and run: `/plan2code-3--implement`"
|
||||||
|
5. **If final phase:** "NEXT STEP: Start NEW conversation and run: `/plan2code-4--finalize`"
|
||||||
|
6. Mention `/plan2code-1b--revise` option
|
||||||
|
|
||||||
|
Planny (continuing):
|
||||||
|
```
|
||||||
|
⋅
|
||||||
|
╭───╮
|
||||||
|
│ ★ │
|
||||||
|
│ ◡ │ Phase done! Great progress!
|
||||||
|
╰───╯
|
||||||
```
|
```
|
||||||
|
|
||||||
### After Approval Format
|
Planny (final phase):
|
||||||
|
```
|
||||||
```markdown
|
⋅
|
||||||
⚡ [PHASE X: Phase Name] - COMPLETE
|
╭───╮
|
||||||
|
│ ★ │
|
||||||
Phase marked complete in overview.md.
|
│ ◡ │ All phases complete! Amazing work!
|
||||||
|
╰───╯
|
||||||
## Save Your Progress
|
|
||||||
|
|
||||||
\`\`\`bash
|
|
||||||
git add -A
|
|
||||||
git commit -m "Complete Phase X: [Phase Name]" -m "AI Assisted"
|
|
||||||
\`\`\`
|
|
||||||
|
|
||||||
This creates a checkpoint you can return to if needed.
|
|
||||||
|
|
||||||
## Next Steps
|
|
||||||
|
|
||||||
The next uncompleted phase is Phase Y: [Name].
|
|
||||||
To continue, start a NEW conversation with:
|
|
||||||
|
|
||||||
> /smarsh2code-3--implement specs/<feature-name>/overview.md
|
|
||||||
|
|
||||||
The command will auto-detect Phase Y as the next phase to implement.
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Session End
|
|
||||||
|
|
||||||
When the user approves the phase (replies "approved"), provide:
|
|
||||||
|
|
||||||
1. Confirmation that the phase is marked complete
|
|
||||||
2. Git commit command (for user to execute)
|
|
||||||
3. Files to attach in next session for the next phase
|
|
||||||
4. Reminder to start a NEW conversation
|
|
||||||
5. If all phases complete: recommend proceeding to finalization
|
|
||||||
|
|
||||||
Example for continuing:
|
|
||||||
|
|
||||||
> "⚡ [PHASE 2: Phase Name] - COMPLETE ✓
|
|
||||||
>
|
|
||||||
> Phase marked complete in overview.md.
|
|
||||||
>
|
|
||||||
> ```
|
|
||||||
> ⋅
|
|
||||||
> ╭───╮
|
|
||||||
> │ ★ │
|
|
||||||
> │ ◡ │ Phase done! Great progress!
|
|
||||||
> ╰───╯
|
|
||||||
>
|
|
||||||
> ╔═══════════════════════════════════════════════════════════════════╗
|
|
||||||
> ║ NEXT STEPS ║
|
|
||||||
> ╠═══════════════════════════════════════════════════════════════════╣
|
|
||||||
> ║ ║
|
|
||||||
> ║ Save your progress: ║
|
|
||||||
> ║ git add -A && git commit -m "Complete Phase 2: [Phase Name]" -m "AI Assisted" ║
|
|
||||||
> ║ ║
|
|
||||||
> ║ Then: ║
|
|
||||||
> ║ 1. Start a NEW conversation ║
|
|
||||||
> ║ 2. Use command: /smarsh2code-3--implement ║
|
|
||||||
> ║ 3. Provide path: specs/<feature-name>/overview.md ║
|
|
||||||
> ║ ║
|
|
||||||
> ║ The command will auto-detect Phase 3 as next. ║
|
|
||||||
> ║ ║
|
|
||||||
> ╚═══════════════════════════════════════════════════════════════════╝
|
|
||||||
> ```
|
|
||||||
>
|
|
||||||
> Need to change the plan? Use `/smarsh2code-1b--revise` before continuing."
|
|
||||||
|
|
||||||
Example for final phase:
|
|
||||||
|
|
||||||
> "⚡ [PHASE 4: Phase Name] - COMPLETE ✓
|
|
||||||
>
|
|
||||||
> This was the final implementation phase!
|
|
||||||
>
|
|
||||||
> ```
|
|
||||||
> ⋅
|
|
||||||
> ╭───╮
|
|
||||||
> │ ★ │
|
|
||||||
> │ ◡ │ All phases complete! Amazing work!
|
|
||||||
> ╰───╯
|
|
||||||
>
|
|
||||||
> ╔═══════════════════════════════════════════════════════════════════╗
|
|
||||||
> ║ NEXT STEPS - FINAL PHASE COMPLETE ║
|
|
||||||
> ╠═══════════════════════════════════════════════════════════════════╣
|
|
||||||
> ║ ║
|
|
||||||
> ║ Save your progress: ║
|
|
||||||
> ║ git add -A && git commit -m "Complete Phase 4: [Phase Name]" -m "AI Assisted" ║
|
|
||||||
> ║ ║
|
|
||||||
> ║ Then: ║
|
|
||||||
> ║ 1. Start a NEW conversation ║
|
|
||||||
> ║ 2. Use command: /smarsh2code-4--finalize ║
|
|
||||||
> ║ 3. Provide path: specs/<feature-name>/overview.md ║
|
|
||||||
> ║ ║
|
|
||||||
> ╚═══════════════════════════════════════════════════════════════════╝
|
|
||||||
> ```
|
|
||||||
>
|
|
||||||
> Need to revise before finalizing? Use `/smarsh2code-1b--revise` first."
|
|
||||||
|
|
||||||
## Abort Handling
|
## Abort Handling
|
||||||
|
|
||||||
@@ -426,7 +226,7 @@ If user says "abort", "cancel", or similar:
|
|||||||
- List completed vs remaining tasks
|
- List completed vs remaining tasks
|
||||||
- Note created/modified files
|
- Note created/modified files
|
||||||
- Do NOT change phase checkbox (stays `[/]`)
|
- Do NOT change phase checkbox (stays `[/]`)
|
||||||
- Explain: "Run `/smarsh2code-3--implement` again to resume."
|
- Explain: "Run `/plan2code-3--implement` again to resume."
|
||||||
3. Stop implementation
|
3. Stop implementation
|
||||||
|
|
||||||
## Recovery
|
## Recovery
|
||||||
@@ -435,26 +235,8 @@ If user says "abort", "cancel", or similar:
|
|||||||
|-------|----------|
|
|-------|----------|
|
||||||
| Lost context mid-phase | Attach specs, say "resume from Task X.Y" |
|
| Lost context mid-phase | Attach specs, say "resume from Task X.Y" |
|
||||||
| Spec unclear/conflicting | Mark task blocked, ask user |
|
| Spec unclear/conflicting | Mark task blocked, ask user |
|
||||||
| Need to change plan | Pause, use `/plan2code-1b--revise` |
|
| Need to change plan | Pause, use `/plan2code-1b--revise-plan` |
|
||||||
|
|
||||||
## Learning Capture Protocol
|
## Learning Capture
|
||||||
|
|
||||||
At END of each session, check for auto-capture triggers:
|
At session end, if you discovered undocumented commands, dependency quirks, gotchas (>5min cost), framework workarounds, or missing `AGENTS.md` patterns → prompt user to update AGENTS.md. If yes, apply the edit directly.
|
||||||
- [ ] Discovered undocumented build/test command
|
|
||||||
- [ ] Found non-obvious dependency relationship
|
|
||||||
- [ ] Encountered "gotcha" costing >5 minutes
|
|
||||||
- [ ] Made workaround for framework quirk
|
|
||||||
- [ ] Found patterns not in `AGENTS.md`
|
|
||||||
|
|
||||||
**Capture Format:**
|
|
||||||
```
|
|
||||||
📚 LEARNING DETECTED
|
|
||||||
|
|
||||||
Category: [Commands / Architecture / Gotchas / Testing / Config]
|
|
||||||
Learning: [concise description]
|
|
||||||
Context: [why this matters]
|
|
||||||
|
|
||||||
Update AGENTS.md with this? (yes/no)
|
|
||||||
```
|
|
||||||
|
|
||||||
If yes, apply the edit directly.
|
|
||||||
|
|||||||
+62
-130
@@ -31,24 +31,9 @@ If multiple active spec folders exist or nothing provided, ask user for:
|
|||||||
|
|
||||||
## Examples
|
## Examples
|
||||||
|
|
||||||
### Task Audit
|
**Task Audit:** Always show verification table with phase totals, blocked items, and completion %. Never just assert "all complete" without evidence.
|
||||||
**Bad:** "All tasks complete. Moving to Step 2." (No verification shown)
|
|
||||||
|
|
||||||
**Good:**
|
**Documentation Review:** Always show review table with each document checked and proposed changes. Never assert "no updates needed" without evidence.
|
||||||
| Phase | Total | Completed | Blocked |
|
|
||||||
|-------|-------|-----------|---------|
|
|
||||||
| Phase 1 | 12 | 12 | 0 |
|
|
||||||
| Phase 2 | 18 | 17 | 1 |
|
|
||||||
Blocked: Task 2.14 - OAuth awaiting credentials. Completion: 96.7%
|
|
||||||
|
|
||||||
### Documentation Review
|
|
||||||
**Bad:** "No docs need updating." (No evidence of review)
|
|
||||||
|
|
||||||
**Good:**
|
|
||||||
| Document | Needs Update? | Changes |
|
|
||||||
|----------|---------------|---------|
|
|
||||||
| README.md | Yes | Add auth setup |
|
|
||||||
| .env.example | Yes | Add JWT_SECRET |
|
|
||||||
|
|
||||||
## Process
|
## Process
|
||||||
|
|
||||||
@@ -190,46 +175,57 @@ Add this summary to `overview.md` under `## Completion Summary`.
|
|||||||
| `API.md` / docs | API documentation | Update with new endpoints |
|
| `API.md` / docs | API documentation | Update with new endpoints |
|
||||||
| `CLAUDE.md` | AI assistant context | Update if patterns changed |
|
| `CLAUDE.md` | AI assistant context | Update if patterns changed |
|
||||||
|
|
||||||
#### Report:
|
Report: table of documents needing updates with proposed changes. List each document with specific additions.
|
||||||
|
|
||||||
```markdown
|
If updates needed, show Planny and ask for approval:
|
||||||
## Documentation Review
|
|
||||||
| Document | Needs Update? | Proposed Changes |
|
|
||||||
|----------|---------------|------------------|
|
|
||||||
| README.md | Yes | Add "Authentication" section |
|
|
||||||
| CHANGELOG.md | Yes | Add entry: "Added user auth with JWT" |
|
|
||||||
|
|
||||||
### Proposed Updates
|
```
|
||||||
#### README.md
|
⋅
|
||||||
[Show specific additions]
|
╭───╮
|
||||||
|
│ ● │
|
||||||
#### CHANGELOG.md
|
│ ~ │ Found some docs that need updating!
|
||||||
[Show specific entry]
|
╰───╯
|
||||||
```
|
```
|
||||||
|
|
||||||
**If ANY documentation needs updates:**
|
> Reply "approve" to proceed with doc updates, or specify which to skip.
|
||||||
|
|
||||||
> ```
|
Do NOT make documentation changes without user approval.
|
||||||
> ⋅
|
|
||||||
> ╭───╮
|
|
||||||
> │ ● │
|
|
||||||
> │ ~ │ Found some docs that need updating!
|
|
||||||
> ╰───╯
|
|
||||||
> ```
|
|
||||||
>
|
|
||||||
> "The following documentation updates are recommended. Review and approve:
|
|
||||||
>
|
|
||||||
> [List proposed changes]
|
|
||||||
>
|
|
||||||
> Reply 'approve' to proceed, or specify which to skip."
|
|
||||||
|
|
||||||
**Do NOT make documentation changes without user approval.**
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### STEP 5: Spec Cleanup
|
### STEP 5: User Feedback (Optional)
|
||||||
|
|
||||||
`🧹 [FINALIZATION STEP 5: Spec Cleanup]`
|
`🧹 [FINALIZATION STEP 5: User Feedback]`
|
||||||
|
|
||||||
|
**Objective:** Collect optional user feedback before archival.
|
||||||
|
|
||||||
|
Ask the user: "Would you like to provide feedback on this workflow run? (optional)"
|
||||||
|
|
||||||
|
If yes, collect:
|
||||||
|
1. **Rating** (1-10): "How would you rate this workflow run overall?"
|
||||||
|
2. **Reason**: "Brief reason for your rating?"
|
||||||
|
3. **Went Well**: "What went well?"
|
||||||
|
4. **Went Poorly**: "What went poorly or could improve?"
|
||||||
|
|
||||||
|
Append to `overview.md` (in the active spec directory):
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
## User Feedback
|
||||||
|
| Field | Value |
|
||||||
|
|-------|-------|
|
||||||
|
| Rating | [1-10] |
|
||||||
|
| Reason | [response] |
|
||||||
|
| Went Well | [response] |
|
||||||
|
| Went Poorly | [response] |
|
||||||
|
```
|
||||||
|
|
||||||
|
If the user declines, skip and proceed to Step 6 (Spec Cleanup).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### STEP 6: Spec Cleanup
|
||||||
|
|
||||||
|
`🧹 [FINALIZATION STEP 6: Spec Cleanup]`
|
||||||
|
|
||||||
**Objective:** Archive completed specifications.
|
**Objective:** Archive completed specifications.
|
||||||
|
|
||||||
@@ -256,9 +252,9 @@ specs--completed/
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### STEP 6: Final Confirmation
|
### STEP 7: Final Confirmation
|
||||||
|
|
||||||
`🧹 [FINALIZATION STEP 6: Final Confirmation]`
|
`🧹 [FINALIZATION STEP 7: Final Confirmation]`
|
||||||
|
|
||||||
**Objective:** Confirm all finalization steps complete.
|
**Objective:** Confirm all finalization steps complete.
|
||||||
|
|
||||||
@@ -276,8 +272,9 @@ specs--completed/
|
|||||||
- [x] Step 2: Implementation Verification
|
- [x] Step 2: Implementation Verification
|
||||||
- [x] Step 3: Implementation Summary
|
- [x] Step 3: Implementation Summary
|
||||||
- [x] Step 4: Documentation Review
|
- [x] Step 4: Documentation Review
|
||||||
- [x] Step 5: Spec Cleanup
|
- [x] Step 5: User Feedback (Optional)
|
||||||
- [x] Step 6: Final Confirmation
|
- [x] Step 6: Spec Cleanup
|
||||||
|
- [x] Step 7: Final Confirmation
|
||||||
|
|
||||||
### Files Created/Modified During Finalization
|
### Files Created/Modified During Finalization
|
||||||
- `specs/<feature-name>/overview.md` - Added completion summary
|
- `specs/<feature-name>/overview.md` - Added completion summary
|
||||||
@@ -297,68 +294,20 @@ specs--completed/
|
|||||||
╰───╯
|
╰───╯
|
||||||
```
|
```
|
||||||
|
|
||||||
╔═══════════════════════════════════════════════════════════════════╗
|
> IMPLEMENTATION COMPLETED!
|
||||||
║ IMPLEMENTATION COMPLETE ║
|
> All tasks finished.
|
||||||
╠═══════════════════════════════════════════════════════════════════╣
|
> Specs archived to `specs--completed/<feature-name>/`.
|
||||||
║ ║
|
> Thank you for using the Plan2Code workflow!
|
||||||
║ All tasks finished. Specs archived to: ║
|
|
||||||
║ specs--completed/<feature-name>/ ║
|
### Metrics Capture (Contributors)
|
||||||
║ ║
|
To help improve plan2code's prompts, run `plan2code-metrics` in the project root.
|
||||||
║ Thank you for using the Smarsh2Code workflow! ║
|
|
||||||
║ ║
|
|
||||||
╚═══════════════════════════════════════════════════════════════════╝
|
|
||||||
```
|
|
||||||
|
|
||||||
### Handling Incomplete Implementations
|
### Handling Incomplete Implementations
|
||||||
|
|
||||||
**Partial Completion (>75%):** Allow finalization with documentation:
|
| Completion | Action |
|
||||||
|
|-----------|--------|
|
||||||
```markdown
|
| **>75%** | Finalize with notice. List incomplete items. Note remaining tasks for follow-up cycle. |
|
||||||
## Partial Completion Notice
|
| **<75%** | Recommend returning to implementation. List incomplete phases with task counts. Options: 1) Return via `/plan2code-3--implement` 2) Proceed with partial finalization. |
|
||||||
Feature finalized at [X]% completion.
|
|
||||||
|
|
||||||
### Incomplete Items
|
|
||||||
- Phase X, Task Y: [Description] - [Reason]
|
|
||||||
|
|
||||||
╔═══════════════════════════════════════════════════════════════════╗
|
|
||||||
║ NEXT STEPS - REMAINING WORK ║
|
|
||||||
╠═══════════════════════════════════════════════════════════════════╣
|
|
||||||
║ ║
|
|
||||||
║ [X] incomplete tasks remain. ║
|
|
||||||
║ Review the overview.md for remaining items. ║
|
|
||||||
║ Address these in a follow-up implementation cycle. ║
|
|
||||||
║ ║
|
|
||||||
╚═══════════════════════════════════════════════════════════════════╝
|
|
||||||
```
|
|
||||||
|
|
||||||
**Low Completion (<75%):** Recommend returning to implementation:
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
Implementation only [X]% complete. Recommend returning to Implementation Mode.
|
|
||||||
|
|
||||||
**Incomplete phases:**
|
|
||||||
- Phase X: [Y]/[Z] tasks
|
|
||||||
- Phase Y: [Y]/[Z] tasks
|
|
||||||
|
|
||||||
Options:
|
|
||||||
1. Return to implementation
|
|
||||||
2. Proceed with partial finalization
|
|
||||||
|
|
||||||
╔═══════════════════════════════════════════════════════════════════╗
|
|
||||||
║ NEXT STEPS - IMPLEMENTATION INCOMPLETE ║
|
|
||||||
╠═══════════════════════════════════════════════════════════════════╣
|
|
||||||
║ ║
|
|
||||||
║ Completion rate is below 75%. ║
|
|
||||||
║ Return to implementation before finalizing. ║
|
|
||||||
║ ║
|
|
||||||
║ 1. Start a NEW conversation ║
|
|
||||||
║ 2. Use command: /smarsh2code-3--implement ║
|
|
||||||
║ 3. Provide path: specs/<feature-name>/overview.md ║
|
|
||||||
║ ║
|
|
||||||
║ The command will auto-detect the next Phase to implement. ║
|
|
||||||
║ ║
|
|
||||||
╚═══════════════════════════════════════════════════════════════════╝
|
|
||||||
```
|
|
||||||
|
|
||||||
## Abort Handling
|
## Abort Handling
|
||||||
|
|
||||||
@@ -375,23 +324,6 @@ If user says "abort", "cancel", or similar:
|
|||||||
| Missing spec files | Ask for all phase-X.md files |
|
| Missing spec files | Ask for all phase-X.md files |
|
||||||
| Doc updates rejected | Skip updates, note in summary |
|
| Doc updates rejected | Skip updates, note in summary |
|
||||||
|
|
||||||
## Learning Capture Protocol
|
## Learning Capture
|
||||||
|
|
||||||
At END of session, check for auto-capture triggers:
|
At session end, if you discovered undocumented commands, dependency quirks, gotchas (>5min cost), framework workarounds, or missing `AGENTS.md` patterns → prompt user to update AGENTS.md. If yes, apply the edit directly.
|
||||||
- [ ] Discovered undocumented build/test command
|
|
||||||
- [ ] Found non-obvious dependency relationship
|
|
||||||
- [ ] Encountered "gotcha" costing >5 minutes
|
|
||||||
- [ ] Made workaround for framework quirk
|
|
||||||
- [ ] Found patterns not in `AGENTS.md`
|
|
||||||
|
|
||||||
If any triggered:
|
|
||||||
```
|
|
||||||
📚 LEARNING DETECTED
|
|
||||||
- Category: [Commands / Architecture / Gotchas / Testing / Config]
|
|
||||||
- Learning: [description]
|
|
||||||
- Context: [why this matters]
|
|
||||||
|
|
||||||
Update AGENTS.md with this? (yes/no)
|
|
||||||
```
|
|
||||||
|
|
||||||
If yes, generate and apply the edit directly.
|
|
||||||
|
|||||||
+3
-1
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "Plan2Code",
|
"name": "Plan2Code",
|
||||||
"version": "1.5.4",
|
"version": "1.8.1",
|
||||||
"description": "A structured 4-step workflow methodology for AI-assisted software development",
|
"description": "A structured 4-step workflow methodology for AI-assisted software development",
|
||||||
"keywords": [
|
"keywords": [
|
||||||
"ai",
|
"ai",
|
||||||
@@ -17,5 +17,7 @@
|
|||||||
"url": "https://github.com/jparkerweb/plan2code"
|
"url": "https://github.com/jparkerweb/plan2code"
|
||||||
},
|
},
|
||||||
"homepage": "https://plan2code.jparkerweb.com",
|
"homepage": "https://plan2code.jparkerweb.com",
|
||||||
|
"releaseDate": "2026-02-20",
|
||||||
"mode": "utility"
|
"mode": "utility"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user