Compare commits
50 Commits
v1.6.1
..
06195cf692
| Author | SHA1 | Date | |
|---|---|---|---|
| 06195cf692 | |||
| 2b247c9093 | |||
| 1085039110 | |||
| e57dad052c | |||
| 32d1487dcf | |||
| 4cbf426df2 | |||
| 8a8cc14e0d | |||
| 1907281b40 | |||
| cf9fe6d80f | |||
| 0c17ef1454 | |||
| 0767c6b4c7 | |||
| b78b3cd6a3 | |||
| 474c76e564 | |||
| 161e424632 | |||
| 09aa565559 | |||
| 0501260098 | |||
| 48a7cf68bd | |||
| 30860a7654 | |||
| fda0146969 | |||
| 75605e5e41 | |||
| 3b18b42e30 | |||
| 8e457fa6e4 | |||
| 6740e26261 | |||
| aedfe944b2 | |||
| 9b3c06b498 | |||
| 8fe1bfa805 | |||
| 68542fd778 | |||
| 4656a6ebb7 | |||
| 7e9de755f1 | |||
| 4b20f93bb9 | |||
| 5b3b84f0a9 | |||
| 1015b99a6a | |||
| 20a9e2f040 | |||
| 28bda05cbf | |||
| 5274bdbd51 | |||
| 25d874d87d | |||
| 5e48e98293 | |||
| 8e0b1a2ab2 | |||
| c2579a7b6a | |||
| 9d1104d105 | |||
| 72fed09aab | |||
| 37311a7e68 | |||
| c92865c395 | |||
| 63656d339d | |||
| 628e688ab9 | |||
| 157e55ef2e | |||
| 02fd4efa40 | |||
| 348c8ed7b2 | |||
| 5c2f70a37c | |||
| de46275b51 |
@@ -0,0 +1,133 @@
|
||||
# Architecture
|
||||
> Part of [AGENTS.md](../AGENTS.md) — project guidance for AI coding agents.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
plan2code/
|
||||
├── src/ # Source workflow prompts (11 markdown files)
|
||||
│ ├── plan2code-0-pathfinder-references/ # Reference files for pathfinder skill
|
||||
│ │ ├── chart.md # MODE A: destination + frontier grills, templates
|
||||
│ │ ├── grilling.md # Folded-in grilling + domain-modeling
|
||||
│ │ ├── questions.md # On-disk question-file format + markers
|
||||
│ │ ├── resolve.md # Per-type resolution + graduating the fog
|
||||
│ │ ├── handoff.md # Clearing gate + PLAN-DRAFT handoff
|
||||
│ │ └── trail.md # Every-response map visual + pathed resume command
|
||||
│ ├── plan2code-review-references/ # Reference files for review skill
|
||||
│ │ ├── verification-protocol.md # Deep verification + confidence calibration
|
||||
│ │ ├── dimensions.md # 11 dimensions with detailed checklists
|
||||
│ │ ├── false-positives.md # Known false-positive patterns
|
||||
│ │ └── session-end.md # Next-step routing at review session end
|
||||
│ ├── plan2code-init-update-references/ # Reference files for init-update skill
|
||||
│ │ └── ai-agent-file-sync.md # Step 7: replace AI configs with AGENTS.md refs
|
||||
│ └── plan2code-4-finalize-references/ # Reference files for finalize skill
|
||||
│ └── community-feedback-submission.md # STEP 6.5 payload schema + submission tiers
|
||||
├── plan2code-loop/ # Autonomous loop CLI tool (Node.js/TypeScript)
|
||||
│ ├── src/ # TypeScript source
|
||||
│ └── dist/ # Built output (tsup)
|
||||
├── plan2code-metrics/ # Recursive self-improvement toolchain
|
||||
│ ├── src/ # TypeScript source
|
||||
│ │ └── prompts/ # Internal AI prompt templates (no char limit)
|
||||
│ └── dist/ # Built output (tsup)
|
||||
├── src/statusline-claude/ # Claude CLI status line (Node.js, zero deps, single file)
|
||||
│ ├── statusline.js # Self-contained: config, git, formatters, render
|
||||
│ └── statusline-config.json # Default config template
|
||||
├── scripts/ # Development scripts
|
||||
│ └── validate-char-count.js # Pre-commit character count validator
|
||||
├── skills/ # Committed build artifact — one Agent Skill per src/ prompt
|
||||
│ └── plan2code-<name>/ # SKILL.md plus references/ where present
|
||||
│ # Generated by npm run build:skills; consumed by skills add
|
||||
├── .husky/ # Git hooks (husky)
|
||||
│ └── pre-commit # Runs character count validation
|
||||
├── .claude/ # Repo-local Claude Code config (NOT installed by install.js)
|
||||
│ └── skills/ # Maintainer-only dev skills, e.g. plan2code-publish/
|
||||
├── docs/ # Documentation and assets
|
||||
├── specs/ # Feature specs (if any in-progress)
|
||||
├── install.js # Interactive installer (Node.js)
|
||||
├── package.json # Root package (husky only, private: true)
|
||||
├── version.json # Version metadata
|
||||
└── README.md # User documentation
|
||||
```
|
||||
|
||||
## Key Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `install.js` | Interactive installer — builds `skills/` from `src/`, then delegates installation to the skills CLI |
|
||||
| `src/plan2code-*.md` | Source workflow prompts (the "source of truth") |
|
||||
| `skills/` | Generated Agent Skills committed for installation and drift verification |
|
||||
| `scripts/validate-char-count.js` | Pre-commit validator ensuring all source prompts ≤ 11,000 chars |
|
||||
| `version.json` | Version metadata (name, version, description) |
|
||||
| `QUICK-REFERENCE.md` | User quick-reference card |
|
||||
| `src/statusline-claude/` | Claude CLI status bar (included in `A` Install All + dev tools; also via Custom → S) |
|
||||
|
||||
## Workflow Prompts (in `src/`)
|
||||
|
||||
| File | Step | Purpose |
|
||||
|------|------|---------|
|
||||
| `plan2code-init.md` | Init | Generate AGENTS.md as index + `.agents-docs/` section files (progressive discovery) |
|
||||
| `plan2code-init-update.md` | Update | Update AGENTS.md with learnings; detects and routes edits to `.agents-docs/` files |
|
||||
| `plan2code-0-pathfinder.md` | 0 | Chart a foggy idea as a local map of decision questions under `specs/<idea>/pathfinder/`, resolve one per session, hand a seeded PLAN-DRAFT to Step 1 |
|
||||
| `plan2code-quick-task.md` | quick | Lightweight planning for small tasks (standalone — not a pipeline step) |
|
||||
| `plan2code-1-plan.md` | 1 | Requirements analysis & architecture |
|
||||
| `plan2code-1b-revise-plan.md` | 1b | Mid-implementation revisions |
|
||||
| `plan2code-2-document.md` | 2 | Create implementation specs |
|
||||
| `plan2code-3-implement.md` | 3 | Execute implementation (phase by phase) |
|
||||
| `plan2code-review.md` | review | Post-implementation comprehensive review |
|
||||
| `plan2code-4-finalize.md` | 4 | Validate, summarize, feedback, archive (steps 1–7, +optional 6.5) |
|
||||
| `plan2code-handoff.md` | handoff | Compact the conversation into a self-contained handoff document for a fresh session |
|
||||
|
||||
## Naming Convention
|
||||
|
||||
Workflow files follow a strict naming pattern:
|
||||
- **Utilities:** `plan2code-<name>.md` (single dash)
|
||||
- **Numbered steps:** `plan2code-<N>-<name>.md` (single dash, number, single dash)
|
||||
|
||||
Examples:
|
||||
- `plan2code-init.md` (utility)
|
||||
- `plan2code-1-plan.md` (step 1)
|
||||
- `plan2code-1b-revise-plan.md` (step 1b)
|
||||
|
||||
## Reference Files
|
||||
|
||||
Some workflows use companion reference files for depth that exceeds the 11k char limit. The orchestrator (main workflow file) loads them via `Read` directives during execution.
|
||||
|
||||
**Pattern:** `src/<source-filename-without-extension>-references/` (e.g., `plan2code-review-references/`)
|
||||
|
||||
**How the installer handles them:** the directory is copied verbatim to `skills/<skill-name>/references/`, so `Read references/<file>.md` resolves consistently for every agent. There is one output format, with no flat-file sibling directory or path rewrite.
|
||||
|
||||
Reference files are NOT subject to the 11,000 character limit. The review workflow pioneered this pattern (`verification-protocol`, `dimensions`, `false-positives`, `session-end`); the init-update workflow also uses it (`ai-agent-file-sync` for its Step 7), `plan2code-4-finalize.md` uses it for STEP 6.5 (`community-feedback-submission`), and `plan2code-0-pathfinder.md` leans on it hardest (`chart`, `grilling`, `questions`, `resolve`, `handoff`, `trail`, `github-issues` — the orchestrator is a dispatcher, the depth lives in the references). Other workflows can adopt it when a source file's detail exceeds the 11k limit.
|
||||
|
||||
## Repo-Local Skills (`.claude/skills/`)
|
||||
|
||||
Maintainer-only skills committed to the repo but deliberately excluded from the generated `skills/` product artifact:
|
||||
|
||||
- `plan2code-publish/` — publishes a GitHub Release after version files agree.
|
||||
- `plan2code-changelog/` — validates release classification and keeps version files synchronized.
|
||||
- `sync-repo/` — decrypts the protected upstream-sync workflow in memory.
|
||||
|
||||
Do not run `skills add` against the repository root: recursive discovery also finds these maintainer skills. `install.js` targets `skills/` and passes explicit workflow names instead. Global install/uninstall cleanup removes `plan2code-*` workflow skills from user skill directories; it never operates on this repository's `.claude/skills/` directory, and `sync-repo` intentionally has no `plan2code-` prefix.
|
||||
|
||||
## Status Line
|
||||
|
||||
Optional Claude Code status bar living in `src/statusline-claude/`. Three-line bar (icon + content per line) showing model, project, branch, uncommitted diff stats, session duration + cost, context window usage, and plan/quota usage.
|
||||
|
||||
**Design constraints:**
|
||||
- **Zero runtime dependencies** — `statusline.js` is self-contained (config loader, git helpers, formatters, render). Copied verbatim to `~/.claude/plan2code-statusline.js` on install; no bundler step.
|
||||
- **Stdin-driven** — all data comes from Claude Code's stdin JSON (`model`, `workspace`, `context_window`, `rate_limits`, `cost`). No API calls, no auth, no background processes.
|
||||
- **Silent failure** — outer `try/catch` around `main()` plus `process.exit(0)` on missing stdin guarantees the script never crashes the CLI. All git ops are timeout-bounded (1.5s) and non-git workspaces short-circuit via `fs.existsSync('.git')`.
|
||||
- **Atomic settings writes** — installer writes `~/.claude/settings.json` via temp file + rename so a crash never leaves the file truncated.
|
||||
- **Custom-config respect** — installer detects non-plan2code `statusLine` entries, prompts before replacing, and backs up to `statusline-previous.json`. Uninstall only removes `settings.statusLine` if it points to the plan2code bundle.
|
||||
|
||||
**Layout:**
|
||||
|
||||
```
|
||||
src/statusline-claude/
|
||||
├── statusline.js # Self-contained: config, git, formatters, render
|
||||
├── statusline-config.json # Default config template
|
||||
└── README.md # User docs: install, config, debugging
|
||||
```
|
||||
|
||||
**Adaptive plan-usage display:** the formatter auto-selects between `5h/7d` rate-limit percentages (Pro/Max/Teams — when `rate_limits` present in stdin) and `Nk in · Nk out` session-token counts (Bedrock/Vertex/PAYG — when `rate_limits` absent). Segment is hidden when neither shape is available.
|
||||
|
||||
**Installer integration** lives in `install.js` under the `STATUS LINE INSTALLATION` section (`installStatusLine`, `uninstallStatusLine`). Included in `A` (Install All + dev tools); also available individually via Custom → `S`.
|
||||
@@ -0,0 +1,26 @@
|
||||
# Code Style & Gotchas
|
||||
> Part of [AGENTS.md](../AGENTS.md) — project guidance for AI coding agents.
|
||||
|
||||
## Code Style
|
||||
|
||||
- **install.js:** CommonJS, Node.js built-ins only (no external deps), ANSI colors via `COLORS` constant, readline-based prompts
|
||||
- **plan2code-loop & plan2code-metrics:** TypeScript + ESM, built with tsup (target ES2022, moduleResolution: bundler)
|
||||
- External deps: `@inquirer/prompts`, `chalk`, `execa`, `ora`
|
||||
- Interactive CLI via `@inquirer/prompts` (select, input, confirm)
|
||||
- **File operations:** Synchronous fs in all packages
|
||||
|
||||
## Gotchas / Pitfalls
|
||||
|
||||
- **Version sync:** When adding a new version to `CHANGELOG.md`, also update `version.json` and `package.json` (root) to match. Check `README.md` for any version badges or references that need updating. The installer displays the version from `version.json` in its header. All three files (`CHANGELOG.md`, `version.json`, `package.json`) must always show the same version number.
|
||||
- **CHANGELOG ordering:** Entries in `CHANGELOG.md` must be in reverse-chronological order — newest version at the top, oldest at the bottom. New entries are always inserted immediately after the file header.
|
||||
- **CHANGELOG house format is not Keep a Changelog:** version headings are `## vX.Y.Z` — with a `v` prefix and **no date**. The release date is recorded only in `version.json`'s `releaseDate`. Category headings carry emoji: `### ✨ Added`, `### 🔧 Changed`, `### 🐛 Fixed`, `### 💥 Breaking`, `### 🗑️ Removed`, `### 📚 Documentation`. Older entries contain one-off variants (`🎁 Added`, `📦 Updated`, `📝 Documentation`, `🏎️ Improved`, `🧪 Testing`) — do not introduce new ones. The `.claude/skills/plan2code-changelog/` skill automates version selection and formatting; use it rather than hand-rolling an entry.
|
||||
- **PowerShell mangles the CHANGELOG emoji:** `Get-Content` / `Select-String` render the `###` heading emoji as `?` under the default Windows console encoding, so a heading audit done that way reports garbage. Read `CHANGELOG.md` with a file-read or grep tool instead.
|
||||
- **`.claude/skills/` is tracked, not ignored:** repo-local skills (`plan2code-changelog`, `plan2code-publish`, `sync-repo`) live there and are committed. Nothing in `.gitignore` touches `.claude/`, so a new skill only needs `git add`. Per the Failure Log convention in `AGENTS.md`, a correction that is a *workflow* rather than a rule belongs here as a skill, linked from AGENTS.md — not as a Failure log line.
|
||||
- **Loop `.gitignore` setup:** `ensureGitignore()` runs at startup in `Controller.run()` as a pre-flight step, not just inside `createTaskCommit()`. This is critical for phase mode where the Node controller doesn't handle commits — without it, `git add -A` would stage spec files.
|
||||
- **Workflow file character limit:** All `src/plan2code-*.md` files must be ≤ 11,000 characters. A husky pre-commit hook enforces this. The limit predates v2.2.0's single-format skill build and is retained as prompt-size discipline; generated `SKILL.md` files add a small YAML header.
|
||||
- **`skills/` is a committed build artifact:** edit `src/`, run `npm run build:skills`, and commit the regenerated skills in the same change. `npm test` runs `install.js --verify-skills` and fails on missing, unexpected, or stale files. Never edit `skills/` by hand.
|
||||
- **Metrics internal prompts have no char limit:** Files in `plan2code-metrics/src/prompts/` are NOT subject to the 11,000 char limit — only `src/plan2code-*.md` consumer-facing prompts are.
|
||||
- **User Feedback table format:** The `## User Feedback` markdown table in `overview.md` has a strict format the collector regex depends on. Field names must be exactly `Rating`, `Reason`, `Went Well`, `Went Poorly`. Pipe characters in values must be escaped as `\|`.
|
||||
- **PLAN-DRAFT confidence numbers are scraped by regex:** when a `specs/<feature>/PLAN-DRAFT-*.md` contains no `<!-- METRICS_JSON ... -->` comment, `collector.ts` falls back to prose scraping (`collector.ts:186-241`). The overall-confidence pattern requires a literal `%`, but the four *breakdown* patterns (`collector.ts:201-204`) do **not** — `/[Rr]equirements?[:\s|]+(\d{1,2})/` and its siblings match a bare dimension word followed by whitespace, a colon, or a pipe and then digits. So a PLAN-DRAFT written by anything other than `/plan2code-1-plan` Phase 7 must keep both the `%` sign **and** bare `Requirements` / `Feasibility` / `Integration` / `Risk` followed by a number off the page — including innocent table rows like `| Requirements | 11 |`. Otherwise the metrics pipeline records a planning-step confidence that no planning step produced. `/plan2code-0-pathfinder` works around this by hyphenating the labels (`Requirements-clarity 22/25`), which breaks the character class.
|
||||
- **Reference file sizing guideline:** Files in `src/plan2code-*-references/` directories target ~100-200 lines each (soft guideline; evaluate splitting above 300). They are NOT subject to the 11,000 character limit. The pre-commit hook (`validate-char-count.js`) only checks `src/plan2code-*.md` flat files — subdirectory contents are automatically excluded.
|
||||
- **The splitting guideline has a hard ceiling — reference files cannot always be split:** each new reference costs the orchestrator a `Read references/<file>.md` line plus its fallback blockquote (~150-200 chars), and orchestrators near the 11,000 limit have no room to spend. References now stay nested under every generated skill, so the old flat-file path-rewrite constraint no longer applies. When a reference legitimately exceeds 300 lines (e.g. `plan2code-0-pathfinder-references/chart.md`), that is an accepted trade-off, not an oversight.
|
||||
@@ -0,0 +1,103 @@
|
||||
# Development Commands
|
||||
> Part of [AGENTS.md](../AGENTS.md) — project guidance for AI coding agents.
|
||||
|
||||
## Common Commands
|
||||
|
||||
```bash
|
||||
# Install dev dependencies (sets up husky pre-commit hooks)
|
||||
npm install
|
||||
|
||||
# Run the interactive installer
|
||||
node install.js
|
||||
|
||||
# Regenerate skills/ from src/ (non-interactive)
|
||||
npm run build:skills
|
||||
|
||||
# Character-count validation + skills/ drift check
|
||||
npm test
|
||||
|
||||
# Plan2Code Loop
|
||||
cd plan2code-loop && npm install # First time setup
|
||||
cd plan2code-loop && npm run build # Build the CLI
|
||||
|
||||
# Plan2Code Metrics
|
||||
cd plan2code-metrics && npm install # First time setup
|
||||
cd plan2code-metrics && npm run build # Build the CLI
|
||||
```
|
||||
|
||||
## Installer Menu Options
|
||||
|
||||
**Main menu:**
|
||||
|
||||
| Option | Action |
|
||||
|--------|--------|
|
||||
| `I` | Install the Plan2Code skills globally through the skills CLI — **skills only**, no dev tools |
|
||||
| `A` | Everything in `I` plus `plan2code-loop`, `plan2code-bot`, `plan2code-metrics`, and the Claude Code status line |
|
||||
| `U` | Uninstall Plan2Code skills and all dev tools (confirmation required) |
|
||||
| `C` | Open CUSTOM sub-menu |
|
||||
| `Q` | Quit |
|
||||
|
||||
**CUSTOM sub-menu (`C`):**
|
||||
|
||||
| Option | Action |
|
||||
|--------|--------|
|
||||
| `L` | Install the skills into the current project instead of globally |
|
||||
| `O` | Install plan2code-loop CLI only |
|
||||
| `M` | Install plan2code-metrics CLI only |
|
||||
| `S` | Install Claude Code status line only |
|
||||
| `B` | Install plan2code-bot CLI only |
|
||||
| `Q` | Return to main menu |
|
||||
|
||||
## Non-Interactive Flags
|
||||
|
||||
`install.js` takes no arguments for normal use, but exposes two build hooks. Any other argument exits 1 with usage.
|
||||
|
||||
| Flag | Action |
|
||||
|------|--------|
|
||||
| `--build-skills` | Regenerate `skills/` from `src/`, pruning stale skills and reference files |
|
||||
| `--verify-skills` | Compare committed `skills/` with `src/`; exits 1 on drift and is run by `npm test` |
|
||||
|
||||
## How the Installer Works
|
||||
|
||||
1. **Builds `skills/` from `src/`** — one Agent Skill per source prompt, including uncommitted source edits.
|
||||
2. **Checks the skills CLI is reachable** with `npx --yes skills --version`.
|
||||
3. **Sweeps pre-2.2 install paths** and removes installed `plan2code-*` skills so renamed or retired prompts cannot survive as orphans.
|
||||
4. **Delegates installation** to `npx --yes skills add "<repo>/skills" -g -s <skill names> -y`.
|
||||
|
||||
The skills CLI owns distribution from step 4 onward. It stores canonical skills under `~/.agents/skills/` and links them into agents that maintain their own skill directory. Plan2Code no longer maintains platform-specific output formats.
|
||||
|
||||
**Invocation choices:**
|
||||
|
||||
| Choice | Reason |
|
||||
|--------|--------|
|
||||
| Explicit space-separated `-s <names>` | Avoids shell expansion and prevents unrelated directories under `skills/` from being installed. |
|
||||
| No `-a` / `--agent '*'` | Uses the CLI's supported default agent set instead of requesting incompatible scope/agent combinations. |
|
||||
| Captured output | Suppresses the CLI's duplicated banners while preserving real failures; unsupported-scope noise is filtered. |
|
||||
|
||||
## Skill Format
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Path | `skills/<skill-name>/SKILL.md` |
|
||||
| Skill name | `generateSkillName(prompt)`, such as `plan2code-1-plan` or `plan2code-init` |
|
||||
| Frontmatter | `name`, `description`, `disable-model-invocation: true` |
|
||||
| Reference files | `skills/<skill-name>/references/<file>.md` |
|
||||
|
||||
`disable-model-invocation: true` is unconditional because these workflows are user-initiated. Agents that do not recognize the field ignore it.
|
||||
|
||||
## Editing Workflow Prompts
|
||||
|
||||
`src/` is the source of truth; `skills/` is a committed build artifact.
|
||||
|
||||
1. Edit the source file under `src/`.
|
||||
2. Run `npm run build:skills`.
|
||||
3. Test the workflow in an AI tool.
|
||||
4. Commit regenerated `skills/` beside the source change; `npm test` fails on drift.
|
||||
5. Never edit `skills/` by hand because the next build overwrites it.
|
||||
|
||||
## Adding a New Workflow Prompt / Skill
|
||||
|
||||
1. Create `src/plan2code-<name>.md` with body content only and keep it under 11,000 characters.
|
||||
2. Register it in `SOURCE_PROMPTS` in `install.js`; add a matching `generateStepLabel()` case when needed.
|
||||
3. Update command inventories in `README.md`, `QUICK-REFERENCE.md`, `.agents-docs/AGENTS-architecture.md`, and `CHANGELOG.md`. Update `docs/index.html` only for core pipeline steps.
|
||||
4. Run `npm run build:skills` and `npm test`.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Plan2Code Loop
|
||||
> Part of [AGENTS.md](../AGENTS.md) — project guidance for AI coding agents.
|
||||
|
||||
A separate Node.js CLI tool that autonomously implements specs by looping through tasks.
|
||||
|
||||
## Loop Architecture
|
||||
|
||||
The loop uses an **LLM-driven discovery** approach:
|
||||
- Node app just orchestrates iterations and parses completion markers
|
||||
- The LLM reads spec files (`overview.md`, `phase-X.md`) to discover tasks
|
||||
- The LLM finds unchecked checkboxes, implements ONE task per iteration, marks it complete
|
||||
- No regex parsing of markdown in Node - the AI handles all task discovery
|
||||
|
||||
## Loop Commands
|
||||
|
||||
```bash
|
||||
# Build the loop CLI
|
||||
cd plan2code-loop && npm run build
|
||||
|
||||
# Run the loop (after linking) - fully interactive
|
||||
plan2code-loop
|
||||
```
|
||||
|
||||
The CLI auto-detects specs in `./specs/`, prompts for selection if multiple found, and handles session continuation interactively. Session state is stored per-spec in `specs/<feature>/.plan2code-loop/`.
|
||||
|
||||
## Loop Modes
|
||||
|
||||
The CLI asks users to choose a loop mode:
|
||||
- **One task per loop** (default) - Each agent invocation implements exactly one task. The Node controller handles git commits.
|
||||
- **One phase per loop** - Each agent invocation implements all remaining tasks in the current phase. The LLM handles git commits (with JIRA ticket ID if provided). The controller parses multiple completion markers from a single iteration.
|
||||
|
||||
## Completion Markers
|
||||
|
||||
The LLM must output one of these formats:
|
||||
- `TASK_COMPLETE: 1.1 - Task description` - Task done successfully
|
||||
- `TASK_BLOCKED: 1.1 - Reason` - Cannot complete task
|
||||
- `PHASE_COMPLETE` - Current phase finished (phase mode only)
|
||||
- `LOOP_COMPLETE` - All phases finished
|
||||
|
||||
## Key Source Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `plan2code-loop/src/controller.ts` | Main loop orchestrator |
|
||||
| `plan2code-loop/src/prompt/templates.ts` | Prompt templates for both loop modes |
|
||||
| `plan2code-loop/src/utils/git.ts` | `createTaskCommit()` — handles task-mode commits with footer |
|
||||
| `plan2code-loop/src/cli.ts` | Interactive CLI entry point |
|
||||
@@ -0,0 +1,87 @@
|
||||
# Plan2Code Metrics
|
||||
> Part of [AGENTS.md](../AGENTS.md) — project guidance for AI coding agents.
|
||||
|
||||
A recursive self-improvement toolchain for plan2code contributors. Collects run metrics, aggregates by prompt generation, diagnoses weak steps via AI, and proposes surgical prompt edits.
|
||||
|
||||
## Metrics Data Flow
|
||||
|
||||
```
|
||||
Collect → Aggregate → Analyze → Improve → Apply
|
||||
```
|
||||
|
||||
1. **Collector** reads project artifacts (`specs/<feature>/`) → writes `RunMetrics` JSON per run
|
||||
2. **Aggregator** groups runs by prompt SHA fingerprint (cohorts) → `aggregated.json`
|
||||
3. **Analyzer** invokes AI with aggregated metrics + prompt contents → diagnosis markdown
|
||||
4. **Improver** invokes AI with diagnosis → validated `PromptEdit[]` proposals (char limit + verbatim checks)
|
||||
5. **Applier** shows interactive diffs → patches `src/plan2code-*.md` files
|
||||
|
||||
## Metrics Commands
|
||||
|
||||
```bash
|
||||
cd plan2code-metrics && npm run build # Build the CLI
|
||||
plan2code-metrics # Run (fully interactive, no flags)
|
||||
```
|
||||
|
||||
## Metrics CLI Menu
|
||||
|
||||
| Option | Action |
|
||||
|--------|--------|
|
||||
| Collect | Read spec artifacts → run JSON |
|
||||
| Import | Copy run JSON from another project |
|
||||
| View | Display cohort metrics with health indicators |
|
||||
| Analyze | AI diagnosis of weak metrics |
|
||||
| Propose | AI improvement proposals with validation |
|
||||
| Apply | Interactive diff review + file patching |
|
||||
| Fetch community submissions | List/parse/import open community-feedback GitHub issues from jparkerweb/plan2code, close on success |
|
||||
|
||||
Community submissions arrive as GitHub issues labeled `community-feedback` on `jparkerweb/plan2code`, created by the finalize prompt's post-Step-6 submission flow; the "Fetch community submissions" option requires an authenticated `gh` CLI to list/close them.
|
||||
|
||||
## Key Source Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `types.ts` | All interfaces (`RunMetrics`, `UserFeedback`, `CohortMetrics`, etc.) + `METRIC_TARGETS` |
|
||||
| `collector.ts` | Reads project artifacts → run JSON (parses plan drafts, overview.md, loop logs) |
|
||||
| `aggregator.ts` | Merges runs by prompt generation (SHA cohort) → `aggregated.json` |
|
||||
| `community.ts` | Lists/parses/closes `community-feedback`-labeled GitHub issues via `gh` CLI |
|
||||
| `analyzer.ts` | AI diagnosis via `prompts/analyze.md` template |
|
||||
| `improver.ts` | AI proposals via `prompts/improve.md` + validation (char count, old_text match) |
|
||||
| `applier.ts` | Interactive diff review + file patching |
|
||||
| `cli.ts` | Menu-driven interactive CLI (100% prompts, no flags) |
|
||||
| `invoke-llm.ts` | Unified LLM interface (Claude Code, GitHub Copilot CLI, or Devin CLI) |
|
||||
|
||||
## User Feedback
|
||||
|
||||
The collector parses an optional `## User Feedback` table from `overview.md`:
|
||||
|
||||
```markdown
|
||||
## User Feedback
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Rating | 8 |
|
||||
| Reason | Smooth workflow |
|
||||
| Went Well | Planning was thorough |
|
||||
| Went Poorly | Some tasks unclear |
|
||||
```
|
||||
|
||||
Feedback is collected during finalize (Step 5) or retroactively via the CLI. Pipe characters in values are escaped as `\|`. The aggregator computes `avg_user_rating` and `feedback_count` per cohort.
|
||||
|
||||
## Supported AI Agents
|
||||
|
||||
- **Claude Code** (recommended): `claude` CLI with `--inputFile` for prompt delivery
|
||||
- **GitHub Copilot CLI**: `copilot` CLI with stdin prompt delivery
|
||||
- **Devin CLI**: `devin` CLI with `--print --prompt-file <file> --permission-mode dangerous`
|
||||
|
||||
## Metric Targets
|
||||
|
||||
| Metric | Target | Direction |
|
||||
|--------|--------|-----------|
|
||||
| `avg_confidence` | ≥ 90 | higher is better |
|
||||
| `avg_task_completion_rate` | ≥ 0.95 | higher is better |
|
||||
| `avg_blocker_count` | ≤ 1.5 | lower is better |
|
||||
| `avg_completion_marker_success_rate` | ≥ 0.95 | higher is better |
|
||||
| `avg_verification_failures_found` | ≤ 1.0 | lower is better |
|
||||
| `archival_success_rate` | ≥ 0.99 | higher is better |
|
||||
| `avg_user_rating` | ≥ 7.0 | higher is better |
|
||||
|
||||
Data stored in `.plan2code-metrics/` (runs/, aggregated.json, proposals/).
|
||||
@@ -0,0 +1,149 @@
|
||||
---
|
||||
name: plan2code-changelog
|
||||
description: "Validate and fix the CHANGELOG.md version number before opening a PR. Reads main branch to determine the current latest version, classifies changes on the current branch, and proposes the correct next semver. Use this skill when the user mentions changelog, version number, preparing a PR, release version, semver check, or says 'check the changelog', 'what version should this be', 'prepare for PR', or 'fix the version'. Also use proactively when you notice a CHANGELOG entry that may have an incorrect version number."
|
||||
---
|
||||
|
||||
# Plan2Code Changelog Validator
|
||||
|
||||
Ensure the CHANGELOG.md entry for the current branch has the correct semver version before a PR is opened. This skill exists because parallel branches independently pick version numbers that collide or leap-frog when merged — this validates against main's actual state right before the PR.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1 — Gather state
|
||||
|
||||
Run these commands to understand the current situation:
|
||||
|
||||
```bash
|
||||
# 1. Current latest version on main
|
||||
git show main:CHANGELOG.md | head -20
|
||||
|
||||
# 2. Current branch name (for ticket ID extraction)
|
||||
git branch --show-current
|
||||
|
||||
# 3. What this branch changed (commit subjects)
|
||||
git log main...HEAD --oneline
|
||||
|
||||
# 4. Files changed on this branch
|
||||
git diff main...HEAD --name-only
|
||||
```
|
||||
|
||||
Extract from main's CHANGELOG:
|
||||
- The **latest version number** (first `## vX.Y.Z` line)
|
||||
|
||||
Note: on Windows, PowerShell's console encoding mangles the emoji in the `###` headings to `?`. Read `CHANGELOG.md` with the file-read or grep tool rather than `Get-Content` / `Select-String` when you need to see them.
|
||||
|
||||
Extract from the branch:
|
||||
- The **list of changed files** to classify the change type
|
||||
- The **commit messages** for changelog entry content
|
||||
|
||||
### Step 2 — Classify the change
|
||||
|
||||
Determine the change type by examining what was modified on this branch:
|
||||
|
||||
| Signal | Classification | Version Bump | Heading |
|
||||
|--------|---------------|--------------|---------|
|
||||
| An install target removed, or an existing workflow's contract broken | Breaking change | **Major** (X.0.0) | `### 💥 Breaking` |
|
||||
| New workflow prompt (`.md` file under `src/`) | New workflow | **Minor** (x.Y.0) | `### ✨ Added` |
|
||||
| New capability added to an existing prompt, or a new `.claude/skills/` skill | New capability | **Patch** (x.y.Z) | `### ✨ Added` |
|
||||
| Behavioral changes to existing prompt(s), installer, or docs | Behavior change | **Patch** (x.y.Z) | `### 🔧 Changed` |
|
||||
| Bug fix to existing prompt(s) or tooling | Bug fix | **Patch** (x.y.Z) | `### 🐛 Fixed` |
|
||||
| A prompt, target, or file deleted | Removal | **Patch** (x.y.Z) | `### 🗑️ Removed` |
|
||||
| README / `.readme/` / docs-site only | Documentation | **Patch** (x.y.Z) | `### 📚 Documentation` |
|
||||
| Mix of the above | Use the **highest** bump (major > minor > patch) | Combine headings |
|
||||
|
||||
Use only the headings in this table — the CHANGELOG has historical one-off variants (`🎁 Added`, `📦 Updated`, `📝 Documentation`, `🏎️ Improved`, `🧪 Testing`) that should not be introduced in new entries.
|
||||
|
||||
### Step 3 — Compute the correct version
|
||||
|
||||
Starting from main's latest version:
|
||||
- **Major bump:** increment the first number, reset the rest (e.g., `1.16.1` → `2.0.0`)
|
||||
- **Minor bump:** increment the middle number, reset patch to 0 (e.g., `2.0.0` → `2.1.0`)
|
||||
- **Patch bump:** increment the last number (e.g., `2.1.0` → `2.1.1`)
|
||||
|
||||
### Step 4 — Check the current branch's CHANGELOG
|
||||
|
||||
Read the current `CHANGELOG.md` on the branch. Look for:
|
||||
|
||||
0. **You are on `main` with no diff** — there is no branch to validate. Instead, compare the top CHANGELOG version against `git log` since the commit that released it: if commits have landed on `main` without a CHANGELOG entry, treat those commits as the change set and continue from Step 2. Say so explicitly rather than reporting "nothing to do."
|
||||
|
||||
1. **No entry exists yet for this branch's work** — the branch hasn't added a version entry above main's latest. Proceed to Step 5 to draft one.
|
||||
|
||||
2. **An entry exists but the version is wrong** — the branch has a version entry, but it doesn't match the computed correct version (common when branches were rebased or other PRs merged first). Report the discrepancy:
|
||||
|
||||
```
|
||||
Version check for branch: {branch-name}
|
||||
|
||||
Main is at: {main-version}
|
||||
Branch claims: {branch-version}
|
||||
Correct version: {computed-version} ({classification})
|
||||
|
||||
The version needs to be updated: {branch-version} → {computed-version}
|
||||
```
|
||||
|
||||
Ask: "Update the version to {computed-version}? (yes / no)"
|
||||
|
||||
3. **An entry exists and the version is correct** — report success:
|
||||
|
||||
```
|
||||
Version check for branch: {branch-name}
|
||||
|
||||
Main is at: {main-version}
|
||||
Branch version: {branch-version} ({classification})
|
||||
|
||||
Version is correct. CHANGELOG is ready for PR.
|
||||
```
|
||||
|
||||
Stop here unless the user asks for content changes.
|
||||
|
||||
### Step 5 — Draft or fix the CHANGELOG entry
|
||||
|
||||
**If no entry exists**, draft a new one based on the commits and changed files. Match this repo's house format exactly — a bare `## vX.Y.Z` heading with **no date**, emoji `###` headings from the Step 2 table, and a blank line between bullets:
|
||||
|
||||
```markdown
|
||||
## {computed-version-with-v-prefix}
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **{prompt-or-area}** — {what changed, and why it matters to someone installing it}
|
||||
|
||||
### 🔧 Changed
|
||||
|
||||
- **{prompt-or-area}** — {what changed}
|
||||
```
|
||||
|
||||
Conventions to follow, drawn from existing entries:
|
||||
|
||||
- Bold lead-in naming the prompt, file, or area, then an em dash (`—`), then the description.
|
||||
- Reference prompts by their command (`/plan2code-1-plan`) or path (`src/plan2code-init.md`), not by informal name.
|
||||
- One paragraph per bullet is fine — this CHANGELOG favours substantive entries over terse one-liners, and a bullet may carry extra indented paragraphs for detail.
|
||||
- A release with a big theme may open with a one-line summary paragraph directly under the `## vX.Y.Z` heading, before the first `###`.
|
||||
|
||||
Insert the new section directly below the `All notable changes...` line and above the previous version's heading.
|
||||
|
||||
Present the draft and ask for approval before writing.
|
||||
|
||||
**If the version is wrong**, update only the version number — preserve the existing content unless the user asks for content changes too.
|
||||
|
||||
After any changes, show the final CHANGELOG entry for confirmation.
|
||||
|
||||
### Step 6 — Sync `package.json` and `version.json` versions
|
||||
|
||||
After writing or updating the CHANGELOG entry, update the `version` field in `package.json` at the repo root to match the computed version:
|
||||
|
||||
1. Read `package.json` and `version.json` then check the current `version` value.
|
||||
2. If it already matches the computed version, skip — no change needed.
|
||||
3. If it differs, update the `"version"` field in both files to the computed version (e.g., `"version": "2.1.1"`).
|
||||
4. Set `releaseDate` in `version.json` to today's date in `YYYY-MM-DD`. This is the only place a date is recorded — the CHANGELOG headings carry no date.
|
||||
5. Include `package.json` and `version.json` in the same commit as the CHANGELOG changes.
|
||||
|
||||
This keeps `package.json`, `version.json`, and `CHANGELOG.md` in lockstep so `npm pkg get version` always reflects the latest release.
|
||||
|
||||
## Rules
|
||||
|
||||
- Never create a version entry without checking main first — the whole point is to derive the version from main's current state
|
||||
- Always present changes before writing — the user should see and approve the CHANGELOG entry
|
||||
- Match the existing file's format — `## vX.Y.Z` with no date, emoji `###` headings. Do not introduce Keep a Changelog's `## [x.y.z] - date` style
|
||||
- Write for someone reading the release notes, not for someone reading the diff — say what the change lets them do
|
||||
- If multiple change types exist (Added + Changed), use multiple headings under the same version
|
||||
- `version.json`'s `releaseDate` is today's date, i.e. when the release is being prepared, not when the work started
|
||||
- If the branch has no meaningful changes vs main (e.g., only non-shipping files changed), say so and ask if a CHANGELOG entry is actually needed
|
||||
@@ -0,0 +1,159 @@
|
||||
---
|
||||
name: plan2code-publish
|
||||
description: "Publish a GitHub Release for jparkerweb/plan2code whenever CHANGELOG.md's top version is ahead of the latest published release on GitHub — after verifying CHANGELOG.md, version.json, and package.json all agree on the version. Use this skill when the user says 'publish a release', 'create a GitHub release', 'cut a release', 'tag a release', 'tag and release', 'is the changelog published', or otherwise mentions publishing/releasing/tagging this repo."
|
||||
---
|
||||
|
||||
# Plan2Code Release Publisher
|
||||
|
||||
Publish a GitHub Release for `jparkerweb/plan2code` whenever `CHANGELOG.md`'s top version is ahead of the latest published release on GitHub. This turns the merged CHANGELOG entry on `main` into an actual GitHub Release (which also creates the `vX.Y.Z` git tag).
|
||||
|
||||
**Repo-local by design.** This skill lives in the repo's `.claude/skills/` and is intentionally NOT wired into `install.js` — it is a maintainer dev tool, not part of the shipped product, so it is never installed to `~/.claude/skills/`. Do **not** copy it there: the uninstaller (`uninstallFiles()` in `install.js`) and every re-install's pre-copy cleanup in `install()` both delete every entry matching `/^plan2code-/` under `~/.claude/skills/`, so a copy placed there would be silently removed.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1 — Preflight: clean working tree
|
||||
|
||||
Run `git status --porcelain`. If the output is non-empty, stop immediately — do not switch branches or take any other action. Tell the user:
|
||||
|
||||
> Working tree has uncommitted changes. Commit or stash them, then re-run this skill.
|
||||
|
||||
### Step 2 — Switch to main and pull
|
||||
|
||||
If the working tree is clean, switch to `main` and pull latest. Run these as two separate, non-chained commands (never `&&`/`;`-chain `git`/`gh` commands — Windows PowerShell 5.1 rejects `&&`):
|
||||
|
||||
```bash
|
||||
git checkout main
|
||||
git pull origin main
|
||||
```
|
||||
|
||||
### Step 3 — Read the three version sources
|
||||
|
||||
Read all three files at the repo root and extract each version:
|
||||
|
||||
- `CHANGELOG.md` — parse the first `## vX.Y.Z` heading. Strip the leading `v` → `$CHANGELOG_VERSION`. Note: plan2code CHANGELOG headings are `## vX.Y.Z` (v-prefixed, **no** date and **no** brackets) — different from a `## [x.y.z] - YYYY-MM-DD` format.
|
||||
- `version.json` — the `version` field → `$VERSION_JSON`. Also read its `releaseDate` field → `$RELEASE_DATE` (informational only — shown at the confirm step, never part of the sync gate; use `(none)` if absent).
|
||||
- `package.json` (root) → the `version` field → `$PACKAGE_JSON`.
|
||||
|
||||
Also capture from `CHANGELOG.md` the **section body** for the top version: everything from the `## vX.Y.Z` heading line itself (heading **included**) up to — but not including — the next `## v` heading, or end-of-file if there is none. Call this `$SECTION`. Keep the `## vX.Y.Z` heading in `$SECTION`; plan2code release bodies include it.
|
||||
|
||||
### Step 4 — Version-sync preflight (STOP on mismatch)
|
||||
|
||||
All three versions must be identical. This enforces the repo's documented invariant (`.agents-docs/AGENTS-code-style.md` → "Version sync"): `CHANGELOG.md`, `version.json`, and `package.json` must always show the same version number.
|
||||
|
||||
If `$CHANGELOG_VERSION`, `$VERSION_JSON`, and `$PACKAGE_JSON` are **not** all equal, **stop** — do not read the release, do not publish. Report exactly which files disagree:
|
||||
|
||||
> ⚠️ Version files are out of sync — refusing to publish. The repo requires `CHANGELOG.md`, `version.json`, and `package.json` to match.
|
||||
>
|
||||
> - **CHANGELOG.md:** $CHANGELOG_VERSION
|
||||
> - **version.json:** $VERSION_JSON
|
||||
> - **package.json:** $PACKAGE_JSON
|
||||
>
|
||||
> Fix the mismatch first, then re-run this skill. To realign: pick the intended version (normally the highest / newest CHANGELOG entry) and update the other two files to match — see the "Version sync" gotcha in `.agents-docs/AGENTS-code-style.md`.
|
||||
|
||||
This skill never modifies these files — it only reads and compares them.
|
||||
|
||||
### Step 5 — Determine the highest published release
|
||||
|
||||
List every published (non-draft) release and take the numerically-highest semver tag. Do **not** rely on `gh release view` / GitHub's "Latest" flag: that flag returns whatever release is *marked* latest — normally the newest semver, but a maintainer can manually pin it to an older release, which would make the Step 6 ahead-comparison misfire.
|
||||
|
||||
```bash
|
||||
gh release list --repo jparkerweb/plan2code --limit 100 --json tagName,isDraft -q '.[] | select(.isDraft==false) | .tagName'
|
||||
```
|
||||
|
||||
Strip any leading `v` from each returned tag, compare them as numeric `(major, minor, patch)` tuples (the same rule as Step 6), and take the maximum → `$RELEASE_VERSION`. If the command returns no tags or exits non-zero for **any** reason (no releases yet, transient error, etc.), set `$RELEASE_VERSION = "0.0.0"` — no error-text matching is needed.
|
||||
|
||||
### Step 6 — Compare versions
|
||||
|
||||
Compare `$CHANGELOG_VERSION` vs `$RELEASE_VERSION` as a numeric `(major, minor, patch)` tuple. Never do a plain string/lexicographic compare — e.g. `"1.9.0" > "1.10.0"` is true as strings but wrong numerically.
|
||||
|
||||
### Step 7 — Not ahead: no-op
|
||||
|
||||
If `$CHANGELOG_VERSION` ≤ `$RELEASE_VERSION`, print a simple status message showing both versions and stop:
|
||||
|
||||
> CHANGELOG top version ($CHANGELOG_VERSION) is not ahead of the latest published release ($RELEASE_VERSION). Nothing to publish.
|
||||
|
||||
No error is raised and no release is created.
|
||||
|
||||
### Step 8 — Ahead: compute and confirm
|
||||
|
||||
If `$CHANGELOG_VERSION` > `$RELEASE_VERSION`, compute:
|
||||
|
||||
- `tag = "v$CHANGELOG_VERSION"`
|
||||
- `title = "v$CHANGELOG_VERSION"` (plan2code keeps the `v` prefix in release titles)
|
||||
- `notes` = the header line `# What's New 🎉`, then one blank line, then `$SECTION` verbatim (`$SECTION` already starts with the `## vX.Y.Z` heading). Build this as a real multi-line string with **actual newlines** — the `\n\n` shorthand shown elsewhere means "a blank line," never the literal two-character sequence `\` + `n`. Getting this wrong would run the header and the first CHANGELOG heading together with a stray `\n\n` in the published body.
|
||||
|
||||
Present all three to the user and wait for an explicit answer before any write. Offer the optional decorative title suffix — a plain `vX.Y.Z` title is the default, but a release may append one (e.g. `v1.14.0 - 🔍 Review workflow`):
|
||||
|
||||
> 🚀 [Publish Plan2Code Release]
|
||||
>
|
||||
> CHANGELOG is ahead of the latest published release:
|
||||
> - **Current release:** $RELEASE_VERSION
|
||||
> - **CHANGELOG top version:** $CHANGELOG_VERSION
|
||||
> - **version.json releaseDate:** $RELEASE_DATE (informational — read from `version.json`)
|
||||
>
|
||||
> Proposed release:
|
||||
> - **Tag:** $tag
|
||||
> - **Title:** $title
|
||||
> - **Notes:**
|
||||
> ```
|
||||
> $notes
|
||||
> ```
|
||||
>
|
||||
> Publish this release? Reply **yes** to publish as-is, **no** to cancel, or provide a decorative suffix to append to the title (e.g. `⇢ 🎆 Feature Name` or `- 🔍 Feature Name`).
|
||||
|
||||
If the user supplies a suffix, set `title = "v$CHANGELOG_VERSION " + <suffix>` (single space join) and proceed to publish. The tag and notes are unaffected by the suffix.
|
||||
|
||||
### Step 9 — Publish (on approval)
|
||||
|
||||
On approval, write `$notes` to a temp file — never pass multiline text inline via `--notes`, that regresses into a quoting bug — then create the release targeting `main`. Two requirements for the temp file: `$notes` must already hold **real newlines** (per Step 8) because `printf '%s'` / `WriteAllText` write it byte-for-byte — a literal `\n` in the string lands literally in the release body; and it MUST be **UTF-8** because the notes contain emoji (`🎉`, `🐛`, `✨`, `🔧`).
|
||||
|
||||
**bash (preferred in this environment):**
|
||||
|
||||
```bash
|
||||
NOTES_FILE=$(mktemp)
|
||||
printf '%s' "$notes" > "$NOTES_FILE"
|
||||
gh release create "$tag" --repo jparkerweb/plan2code --title "$title" --notes-file "$NOTES_FILE" --target main
|
||||
rm -f "$NOTES_FILE"
|
||||
```
|
||||
|
||||
**PowerShell:** do NOT use `Set-Content` — under Windows PowerShell 5.1 it writes ANSI/UTF-16 by default and mangles the emoji into `??`. Write UTF-8 **without BOM** (a BOM would leak into the release body):
|
||||
|
||||
```powershell
|
||||
$NotesFile = [System.IO.Path]::GetTempFileName()
|
||||
[System.IO.File]::WriteAllText($NotesFile, $notes, [System.Text.UTF8Encoding]::new($false))
|
||||
gh release create "$tag" --repo jparkerweb/plan2code --title "$title" --notes-file "$NotesFile" --target main
|
||||
Remove-Item -Path $NotesFile
|
||||
```
|
||||
|
||||
Always delete the temp file afterward, regardless of whether `gh release create` succeeded or failed.
|
||||
|
||||
**Failure handling — already-exists classification:** if `gh release create` exits non-zero, inspect the error text.
|
||||
|
||||
- If and only if it contains the substring `already exists` (real output: `HTTP 422: Validation Failed` / `Release.tag_name already exists`), report this to the user as already published, not as a raw CLI error:
|
||||
|
||||
> This version ($CHANGELOG_VERSION) was already published as a release — nothing more to do.
|
||||
|
||||
- Every other failure (auth, network, permissions, etc.) must be surfaced to the user verbatim. Never silently reclassify a genuine failure as "already published."
|
||||
|
||||
### Step 10 — Verify
|
||||
|
||||
Confirm the release now exists and report its URL:
|
||||
|
||||
```bash
|
||||
gh release view "$tag" --repo jparkerweb/plan2code
|
||||
```
|
||||
|
||||
Report the release URL to the user.
|
||||
|
||||
## Rules
|
||||
|
||||
- **Repo-local only** — this skill is not part of the installed product; never add it to `install.js`, and never copy it to `~/.claude/skills/` (the uninstaller deletes `plan2code-*` entries there).
|
||||
- **Read-only on version files** — never modify `CHANGELOG.md`, `version.json`, or `package.json`; this skill only reads and compares them.
|
||||
- **Version-sync gate is hard** (Step 4) — if the three version sources disagree, stop and report; do not publish a release from an inconsistent repo.
|
||||
- **Never run a local `git tag` or `git push`** — tag creation is delegated entirely to `gh release create --target main`.
|
||||
- **Always use `--notes-file`**, never inline multiline `--notes`, and always write the notes file as UTF-8 (no BOM) so emoji survive.
|
||||
- **Always clean up the temp notes file**, even on a mid-run error.
|
||||
- **Always get explicit approval before any write** (Step 8) — no release is created without a yes (or a yes-with-suffix).
|
||||
- **Derive the current version from the highest published semver tag** (Step 5), never from GitHub's manually-pinnable "Latest" flag. On an empty or failed release list, treat it as "no prior release" (baseline `0.0.0`) — no error-text matching needed.
|
||||
- **On a `gh release create` failure** (Step 9), only reclassify as already-published when the error contains `already exists` — every other failure must be shown verbatim, never swallowed.
|
||||
- **Idempotent and safe to re-run** at any time — re-running after a successful publish hits the Step 7 no-op; re-running after a race-lost publish hits the Step 9 already-exists handling.
|
||||
@@ -0,0 +1,58 @@
|
||||
---
|
||||
name: sync-repo
|
||||
description: "Run the encrypted, password-protected repository sync workflow for this project. The real instructions are stored encrypted at rest and are only revealed in-session after you supply the correct password. Invoke explicitly with /sync-repo."
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
# sync-repo (password-protected)
|
||||
|
||||
The real instructions for this skill are encrypted at rest in `sync-repo.enc`
|
||||
and are NOT readable without the password. Do not guess, reconstruct, or invent
|
||||
the workflow. Follow this launcher exactly.
|
||||
|
||||
## What you (the agent) must do
|
||||
|
||||
1. **Ask the user for the password.** Request the decryption passphrase (via
|
||||
AskUserQuestion or a plain prompt). Do NOT proceed without it. Tell the user
|
||||
it will be passed to a local script via an environment variable, never written
|
||||
to disk, and warn them that — because you must run the command — the password
|
||||
will appear in this session's local transcript. (It never enters the repo.)
|
||||
|
||||
2. **Decrypt to STDOUT only.** Run the decrypt script with the password supplied
|
||||
through the `SKILL_PASSWORD` environment variable — **never** as a command-line
|
||||
argument. From the repo root:
|
||||
|
||||
- **Windows PowerShell:**
|
||||
```powershell
|
||||
$env:SKILL_PASSWORD='<password the user gave you>'; node .\.claude\skills\sync-repo\decrypt.mjs .\.claude\skills\sync-repo\sync-repo.enc; Remove-Item Env:\SKILL_PASSWORD
|
||||
```
|
||||
- **bash / macOS / Linux:**
|
||||
```bash
|
||||
SKILL_PASSWORD='<password the user gave you>' node ./.claude/skills/sync-repo/decrypt.mjs ./.claude/skills/sync-repo/sync-repo.enc
|
||||
```
|
||||
|
||||
3. **Handle the result.**
|
||||
- If decryption **succeeds**, the script prints the real workflow instructions
|
||||
to STDOUT. Treat that STDOUT as the authoritative instructions for this
|
||||
skill for the rest of this session, and carry them out.
|
||||
- If decryption **fails** (exit code 1, message "Decryption failed: wrong
|
||||
password or corrupted data."), the password was wrong or the file is
|
||||
corrupt. Tell the user, ask them to re-enter the password, and retry. Do
|
||||
NOT attempt to reconstruct the instructions from anything else.
|
||||
|
||||
## Hard rules
|
||||
|
||||
- **Never write the decrypted plaintext to a file.** Read it from STDOUT only.
|
||||
`decrypt.mjs` intentionally has no file-output mode.
|
||||
- **Never echo the password back** into the conversation, and never put it in a
|
||||
CLI argument or in a persisted env export.
|
||||
- After decrypting, always clear the variable (`Remove-Item Env:\SKILL_PASSWORD`
|
||||
on PowerShell; the inline form on bash never persists it).
|
||||
|
||||
## Security note (be honest with the user)
|
||||
|
||||
This protects the workflow body **only at rest in the repository**. Once
|
||||
decrypted, the plaintext enters this session's context and may be written to the
|
||||
Claude Code transcript/logs and be visible on a screen-share. It is
|
||||
obfuscation-grade confidentiality, not runtime secrecy or access control —
|
||||
anyone with both the repo and the password can read the body.
|
||||
@@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env node
|
||||
// decrypt.mjs — verifies GCM auth tag, prints plaintext to STDOUT only.
|
||||
// Password from $SKILL_PASSWORD. Exits 1 on wrong password / tamper, leaking nothing.
|
||||
// Usage: SKILL_PASSWORD=... node decrypt.mjs <file.enc>
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { scryptSync, createDecipheriv } from 'node:crypto';
|
||||
|
||||
const MAGIC = Buffer.from('SENC', 'ascii');
|
||||
const VERSION = 0x01;
|
||||
const SCRYPT = { N: 1 << 17, r: 8, p: 1, maxmem: 256 * 1024 * 1024 };
|
||||
const KEYLEN = 32, SALTLEN = 16, IVLEN = 12, TAGLEN = 16;
|
||||
const HEADER = MAGIC.length + 1; // 5
|
||||
|
||||
const password = process.env.SKILL_PASSWORD;
|
||||
if (!password) { console.error('ERROR: set SKILL_PASSWORD env var.'); process.exit(2); }
|
||||
|
||||
const encPath = process.argv[2];
|
||||
if (!encPath) { console.error('Usage: node decrypt.mjs <file.enc>'); process.exit(2); }
|
||||
|
||||
try {
|
||||
const blob = Buffer.from(readFileSync(encPath, 'utf8').trim(), 'base64');
|
||||
if (blob.length < HEADER + SALTLEN + IVLEN + TAGLEN) throw new Error('truncated');
|
||||
if (!blob.subarray(0, MAGIC.length).equals(MAGIC)) throw new Error('bad magic');
|
||||
if (blob[MAGIC.length] !== VERSION) throw new Error('unsupported version');
|
||||
|
||||
let off = HEADER;
|
||||
const salt = blob.subarray(off, off += SALTLEN);
|
||||
const iv = blob.subarray(off, off += IVLEN);
|
||||
const authTag = blob.subarray(off, off += TAGLEN);
|
||||
const ciphertext = blob.subarray(off);
|
||||
|
||||
const key = scryptSync(password, salt, KEYLEN, SCRYPT);
|
||||
const decipher = createDecipheriv('aes-256-gcm', key, iv);
|
||||
decipher.setAuthTag(authTag);
|
||||
// final() throws here if the tag does not verify (wrong password or tampering).
|
||||
const plaintext = Buffer.concat([decipher.update(ciphertext), decipher.final()]);
|
||||
process.stdout.write(plaintext); // STDOUT only — never written to disk.
|
||||
} catch (err) {
|
||||
// Generic message: do not echo crypto internals or any plaintext.
|
||||
console.error('Decryption failed: wrong password or corrupted data.');
|
||||
process.exit(1);
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
#!/usr/bin/env node
|
||||
// encrypt.mjs — AES-256-GCM + scrypt. Password from $SKILL_PASSWORD (never argv).
|
||||
// Usage: SKILL_PASSWORD=... node encrypt.mjs <plaintextFile|-> <outFile.enc>
|
||||
// (pass '-' or omit the input path to read plaintext from STDIN)
|
||||
import { readFileSync, writeFileSync } from 'node:fs';
|
||||
import { scryptSync, randomBytes, createCipheriv } from 'node:crypto';
|
||||
|
||||
const MAGIC = Buffer.from('SENC', 'ascii');
|
||||
const VERSION = 0x01;
|
||||
const SCRYPT = { N: 1 << 17, r: 8, p: 1, maxmem: 256 * 1024 * 1024 };
|
||||
const KEYLEN = 32, SALTLEN = 16, IVLEN = 12;
|
||||
|
||||
const password = process.env.SKILL_PASSWORD;
|
||||
if (!password) { console.error('ERROR: set SKILL_PASSWORD env var.'); process.exit(2); }
|
||||
|
||||
const inPath = process.argv[2];
|
||||
const outPath = process.argv[3];
|
||||
if (!outPath) { console.error('Usage: node encrypt.mjs <plaintextFile|-> <outFile.enc>'); process.exit(2); }
|
||||
|
||||
// readFileSync(0) reads STDIN; use it when no input file (or '-') is given.
|
||||
const plaintext = (!inPath || inPath === '-') ? readFileSync(0) : readFileSync(inPath);
|
||||
|
||||
const salt = randomBytes(SALTLEN);
|
||||
const iv = randomBytes(IVLEN);
|
||||
const key = scryptSync(password, salt, KEYLEN, SCRYPT);
|
||||
const cipher = createCipheriv('aes-256-gcm', key, iv);
|
||||
const ciphertext = Buffer.concat([cipher.update(plaintext), cipher.final()]);
|
||||
const authTag = cipher.getAuthTag(); // 16 bytes
|
||||
|
||||
const blob = Buffer.concat([MAGIC, Buffer.from([VERSION]), salt, iv, authTag, ciphertext]);
|
||||
writeFileSync(outPath, blob.toString('base64') + '\n');
|
||||
console.error(`Wrote ${outPath} (${blob.length} raw bytes, base64-encoded).`);
|
||||
@@ -0,0 +1,30 @@
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Line endings
|
||||
#
|
||||
# Git stores LF, and every text file is checked out as LF on all platforms.
|
||||
# This is not cosmetic: scripts/validate-char-count.js measures the characters
|
||||
# actually on disk, so a CRLF checkout adds ~1 character per line. The workflow
|
||||
# prompts in src/plan2code-*.md run close to their 11,000 character budget
|
||||
# (several sit above 10,800), and a CRLF working tree pushes them over — turning
|
||||
# `npm test` into a check that passes or fails depending on how the repo was
|
||||
# cloned. Pinning eol=lf makes the count reproducible everywhere.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
* text=auto eol=lf
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Binary — never line-ending-converted, never diffed as text
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
*.png binary
|
||||
*.jpg binary
|
||||
*.jpeg binary
|
||||
*.gif binary
|
||||
*.ico binary
|
||||
*.mp4 binary
|
||||
*.webm binary
|
||||
*.woff binary
|
||||
*.woff2 binary
|
||||
|
||||
# The encrypted sync-repo blob is base64 text but must never have its bytes
|
||||
# altered by line-ending normalization. Treat it as binary so autocrlf/eol
|
||||
# settings can never corrupt the ciphertext.
|
||||
*.enc binary
|
||||
@@ -1,10 +1,17 @@
|
||||
specs/
|
||||
specs--completed/
|
||||
CLAUDE.md
|
||||
dist/
|
||||
plan2code-loop/dist
|
||||
plan2code-loop/node_modules
|
||||
plan2code-loop/package-lock.json
|
||||
plan2code-metrics/dist
|
||||
plan2code-metrics/node_modules
|
||||
plan2code-metrics/package-lock.json
|
||||
.plan2code-loop
|
||||
.plan2code-metrics
|
||||
nul
|
||||
.cognition/
|
||||
handoffs/
|
||||
node_modules/
|
||||
package-lock.json
|
||||
SYNC.md
|
||||
@@ -8,7 +8,7 @@ specs--completed/
|
||||
.gitignore
|
||||
CLAUDE.md
|
||||
|
||||
# Exclude generated files (installer generates dist/ dynamically)
|
||||
# Exclude stale generated output from pre-2.2 checkouts
|
||||
dist/
|
||||
|
||||
# Exclude dependencies (installer will run npm install if needed)
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# Autonomous Loop
|
||||
|
||||
`plan2code-loop` is a CLI that works through your spec's tasks on its own, one agent call at a
|
||||
time. It is an **alternative to Step 3**, not a replacement — the four-step workflow and the specs it
|
||||
produces are unchanged.
|
||||
|
||||
← [Back to README](../README.md)
|
||||
|
||||
---
|
||||
|
||||
## When to use it instead of Step 3
|
||||
|
||||
| Approach | Best for |
|
||||
|----------|----------|
|
||||
| `/plan2code-3-implement` | Interactive control, reviewing each phase, logic that needs your judgment |
|
||||
| `plan2code-loop` | Straightforward implementations, batch work, overnight runs |
|
||||
|
||||
The loop reads the same `overview.md` and phase files. You can start with the loop and finish by
|
||||
hand, or the reverse — the checkboxes are the only handoff.
|
||||
|
||||
---
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
# From the plan2code root directory
|
||||
node install.js # A (everything + dev tools) — or C → O (loop only)
|
||||
```
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
plan2code-loop # fully interactive
|
||||
```
|
||||
|
||||
It will:
|
||||
|
||||
1. Find specs in `./specs/`
|
||||
2. Let you pick one if there are several
|
||||
3. Offer to resume an existing session
|
||||
4. Ask for a JIRA ticket ID, which agent to drive, the loop mode, and a max iteration count
|
||||
|
||||
Then, per iteration: read `overview.md` and the phase files, find the first unchecked task (or
|
||||
phase), implement it, mark the checkbox, repeat — until everything is done or it hits the iteration
|
||||
cap.
|
||||
|
||||
---
|
||||
|
||||
## Loop modes
|
||||
|
||||
| Mode | Each agent call | Git commits | Best for |
|
||||
|------|-----------------|-------------|----------|
|
||||
| **One task per loop** (default) | Implements a single task | The Node controller commits after each task | Smaller models, cautious execution |
|
||||
| **One phase per loop** | Implements every task in a phase | The agent commits after each task, with the JIRA ID | Larger context windows, tightly related tasks |
|
||||
|
||||
Session state lives per-spec in `specs/<feature>/.plan2code-loop/`, so each feature's progress
|
||||
stays isolated.
|
||||
|
||||
---
|
||||
|
||||
## Full documentation
|
||||
|
||||
Architecture, completion markers, agent adapters, and configuration:
|
||||
[`plan2code-loop/README.md`](../plan2code-loop/README.md)
|
||||
@@ -0,0 +1,59 @@
|
||||
# Metrics & Self-Improvement
|
||||
|
||||
`plan2code-metrics` closes the loop on the workflow itself: it collects data from your finished
|
||||
specs, aggregates it across runs and prompt generations, then uses AI to diagnose which step is
|
||||
underperforming and propose edits to the workflow prompts.
|
||||
|
||||
Aimed at **contributors and heavy users** — you don't need it to use Plan2Code.
|
||||
|
||||
← [Back to README](../README.md)
|
||||
|
||||
---
|
||||
|
||||
## The habit
|
||||
|
||||
One thing to remember: **collect after every finished spec.** Everything else is on demand.
|
||||
|
||||
```bash
|
||||
# Install once, from the plan2code root
|
||||
node install.js # A (everything + dev tools) — or C → M (metrics only)
|
||||
|
||||
# After finishing a spec (steps 1–4)
|
||||
cd your-project
|
||||
plan2code-metrics # → "Collect metrics" → pick the spec dir → done, ~5 seconds
|
||||
```
|
||||
|
||||
Then, when you're curious or have a few runs banked:
|
||||
|
||||
```bash
|
||||
plan2code-metrics # → "View metrics status" the dashboard
|
||||
# → "Run analysis" AI diagnosis of weak steps
|
||||
# → "Generate improvement proposal" concrete prompt edits
|
||||
# → "Review and apply" patch src/plan2code-*.md
|
||||
```
|
||||
|
||||
## How much data you need
|
||||
|
||||
| Runs | What you get |
|
||||
|------|--------------|
|
||||
| **1** | Raw data and a basic dashboard. Start here. |
|
||||
| **3+** | AI analysis unlocks. Pattern detection starts working. |
|
||||
| **5–10+** | Averages stabilise; generation-over-generation comparisons become meaningful. |
|
||||
|
||||
You're looking for trends, not individual scores.
|
||||
|
||||
---
|
||||
|
||||
## Sending feedback upstream
|
||||
|
||||
`/plan2code-4-finalize` can submit an anonymised metrics payload to the maintainers as a
|
||||
`community-feedback` issue on the repo. Community runs are cohorted by the Plan2Code version that
|
||||
produced them, so your data improves the prompts everyone installs — without displacing the
|
||||
maintainer's own measurements.
|
||||
|
||||
---
|
||||
|
||||
## Full documentation
|
||||
|
||||
Data model, aggregation, cohorts, analysis prompts, and the ingestion flow:
|
||||
[`plan2code-metrics/README.md`](../plan2code-metrics/README.md)
|
||||
@@ -0,0 +1,54 @@
|
||||
# Claude Code Status Line
|
||||
|
||||
A persistent three-line status bar for Claude Code: model, project, git branch, uncommitted diff
|
||||
stats, session duration and cost, context-window usage, and plan quota.
|
||||
|
||||
It reads everything from the JSON Claude Code already sends on stdin — **no API calls, no auth, no
|
||||
background processes.** Optional, and unrelated to the workflow itself.
|
||||
|
||||
← [Back to README](../README.md)
|
||||
|
||||
---
|
||||
|
||||
## What it looks like
|
||||
|
||||
On Pro / Max / Teams accounts, where rate limits are available:
|
||||
|
||||
```
|
||||
◦ ◦ Opus 5 / high │ plan2code │ feature/PCWEB-11702-pathfinder │ +12 -3
|
||||
╭●╮ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
|
||||
├■┤ 3h 5m ($4.62) │ ▰▰▰▰▰▱▱▱▱▱▱▱ 42% (84K) │ 5h: 28% · 7d: 61%
|
||||
```
|
||||
|
||||
On Enterprise / Bedrock / Vertex / pay-as-you-go, where they aren't, the last segment becomes session
|
||||
token counts instead:
|
||||
|
||||
```
|
||||
◦ ◦ Sonnet 5 │ plan2code │ main │ +12 -3
|
||||
╭●╮ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
|
||||
├■┤ 2m ($0.18) │ ▰▰▱▱▱▱▱▱▱▱▱▱ 18% │ 88k in · 3k out
|
||||
```
|
||||
|
||||
The icons down the left are Planny, the project mascot.
|
||||
|
||||
---
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
node install.js # A (everything + dev tools) — or C → S (status line only)
|
||||
```
|
||||
|
||||
That copies the script to `~/.claude/plan2code-statusline.js`, writes a default config to
|
||||
`~/.claude/statusline-config.json` (an existing config is preserved), and registers it in
|
||||
`~/.claude/settings.json`.
|
||||
|
||||
If you already have a custom `statusLine` entry, the installer asks before replacing it and backs the
|
||||
old one up. Uninstalling removes the script and the settings entry but leaves your config file alone.
|
||||
|
||||
---
|
||||
|
||||
## Full documentation
|
||||
|
||||
Config options, compact mode, thresholds, and troubleshooting:
|
||||
[`src/statusline-claude/README.md`](../src/statusline-claude/README.md)
|
||||
@@ -0,0 +1,46 @@
|
||||
# Workflow Test Bot
|
||||
|
||||
`plan2code-bot` drives the whole workflow end to end with no human in the loop — init, plan,
|
||||
document, implement, finalize — to test that the prompts still hold together.
|
||||
|
||||
Built for **maintainers**. If you're using Plan2Code to ship features, you don't need this.
|
||||
|
||||
← [Back to README](../README.md)
|
||||
|
||||
---
|
||||
|
||||
## Two modes, auto-detected
|
||||
|
||||
| Condition | Mode | What it does |
|
||||
|-----------|------|--------------|
|
||||
| No `AGENTS.md` in the working directory | **New project** | Invents an app idea, creates a subdirectory, writes `IDEA.md`, runs init, then all four steps |
|
||||
| `AGENTS.md` present | **Enhancement** | Reads the existing codebase, proposes a realistic enhancement, writes `IDEA.md`, then runs plan → finalize |
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
node install.js # I (everything) — or C → B (bot only)
|
||||
```
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
# New project mode — start from an empty directory
|
||||
mkdir /tmp/test-bot && cd /tmp/test-bot
|
||||
plan2code-bot
|
||||
|
||||
# Enhancement mode — start from a project that has an AGENTS.md
|
||||
cd my-project
|
||||
plan2code-bot
|
||||
|
||||
# Or seed the idea generator yourself
|
||||
plan2code-bot --idea "web app that renders the current weather as vector images"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Full documentation
|
||||
|
||||
Agent SDK wiring, run evaluation, and scoring:
|
||||
[`plan2code-bot/README.md`](../plan2code-bot/README.md) ·
|
||||
[`plan2code-bot/EVALUATION-SYSTEM.md`](../plan2code-bot/EVALUATION-SYSTEM.md)
|
||||
@@ -0,0 +1,103 @@
|
||||
# Walkthrough: one feature, start to finish
|
||||
|
||||
A REST API for task management, from a sentence to archived specs. Every block below is a **separate
|
||||
conversation** — that's the whole trick.
|
||||
|
||||
← [Back to README](../README.md)
|
||||
|
||||
---
|
||||
|
||||
## Session 1 — Plan
|
||||
|
||||
```
|
||||
You: /plan2code-1-plan
|
||||
I want to build a REST API for a task management application.
|
||||
|
||||
AI: 🤔 [REQUIREMENTS ANALYSIS]
|
||||
Six questions. Four of them I need answered before I can size this.
|
||||
...
|
||||
|
||||
AI: 🤔 [TECH STACK]
|
||||
Proposing Fastify + Postgres + Zod. I don't pick this alone — sign off?
|
||||
|
||||
You: approved
|
||||
|
||||
AI: 🤔 [ARCHITECTURE DESIGN]
|
||||
...
|
||||
|
||||
AI: 🤔 [TRANSITION DECISION]
|
||||
Confidence: 92% (gate is 90%). Assumptions written to the draft: 3.
|
||||
→ specs/task-api/PLAN-DRAFT-20260804.md
|
||||
```
|
||||
|
||||
Two files land: the draft, and a `PLAN-CONVERSATION-*.md` log of how you got there.
|
||||
|
||||
> **Started foggy instead?** Run `/plan2code-0-pathfinder` first. When its map clears it writes a
|
||||
> `PLAN-DRAFT` that Step 1 picks up at Phase 4, with requirements and scope already answered.
|
||||
|
||||
---
|
||||
|
||||
## Session 2 — Document
|
||||
|
||||
```
|
||||
You: /plan2code-2-document
|
||||
specs/task-api/PLAN-DRAFT-20260804.md
|
||||
|
||||
AI: 📝 [DOCUMENTATION]
|
||||
Creating specs/task-api/overview.md
|
||||
Creating specs/task-api/Phase 1.md Project setup (6 tasks)
|
||||
Creating specs/task-api/Phase 2.md Data model (8 tasks)
|
||||
Creating specs/task-api/Phase 3.md API endpoints (11 tasks)
|
||||
Creating specs/task-api/Phase 4.md Authentication (7 tasks)
|
||||
|
||||
Parallel execution groups: Phase 3 and Phase 4 don't share files.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Sessions 3…N — Implement, one phase each
|
||||
|
||||
```
|
||||
You: /plan2code-3-implement
|
||||
specs/task-api/overview.md
|
||||
|
||||
AI: ⚡ [PHASE 1: Project setup]
|
||||
(Phase 1 is the next unchecked phase.)
|
||||
...
|
||||
✓ 6/6 tasks. Checkboxes updated in Phase 1.md and overview.md.
|
||||
Next: Phase 2 — Data model (8 tasks). New conversation.
|
||||
```
|
||||
|
||||
Repeat. New conversation each time. It always finds the next open phase itself.
|
||||
|
||||
When you reach a parallel group, it offers you the choice — open a second agent, take the other
|
||||
phase, and the `[/]` marks keep them from colliding.
|
||||
|
||||
**Optional, any time:** `/plan2code-review` for an independent second opinion on what just landed.
|
||||
|
||||
---
|
||||
|
||||
## Final session — Finalize
|
||||
|
||||
```
|
||||
You: /plan2code-4-finalize
|
||||
specs/task-api/overview.md
|
||||
|
||||
AI: 🧹 [VALIDATION]
|
||||
All 32 tasks verified against the phase specs. 2 gaps found and fixed.
|
||||
|
||||
AI: 🧹 [DOCUMENTATION REVIEW]
|
||||
README needs the new /tasks endpoints. AGENTS.md is current.
|
||||
|
||||
AI: 🧹 [SPEC CLEANUP]
|
||||
Moved specs/task-api/ → specs--completed/task-api/
|
||||
|
||||
Implementation complete.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## If requirements move mid-build
|
||||
|
||||
Don't patch the code and hope the specs catch up. Run `/plan2code-1b-revise-plan` — it edits the
|
||||
specs (and only the specs), so the drawing and the build stay in agreement.
|
||||
@@ -1,205 +1,58 @@
|
||||
# AGENTS.md
|
||||
|
||||
This file provides guidance to AI coding agents like Claude Code (claude.ai/code), Cursor AI, Codex, Gemini CLI, GitHub Copilot, and other AI coding assistants when working with code in this repository.
|
||||
This file provides guidance to AI coding agents working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
Plan2Code is a structured 4-step workflow methodology for AI-assisted software development. It provides prompt templates that can be installed globally or per-project for various AI coding tools (Claude Code, Cursor, Copilot, Continue, Windsurf, Codeium).
|
||||
Plan2Code is a structured 4-step workflow methodology for AI-assisted software development. It provides prompt templates that can be installed globally or per-project for various AI coding tools (Claude Code, Cursor, Copilot, Continue, Windsurf, Codeium, Devin, Zed).
|
||||
|
||||
**Version:** Check `version.json` for current version
|
||||
**Author:** Justin Parker
|
||||
**License:** MIT
|
||||
|
||||
## How to Use This File
|
||||
|
||||
This file is an index — each section below contains a brief summary and a link to a detail file in `.agents-docs/`. Read only the sections relevant to your current task. Full details (commands, tables, file lists) are in the linked files. The sections "Project Overview", "How to Use This File", "Mascot", and "Keeping this file current" / "Failure log" are fully inline here and are never split into `.agents-docs/`.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
plan2code/
|
||||
├── src/ # Source workflow prompts (8 markdown files)
|
||||
├── plan2code-loop/ # Autonomous loop CLI tool (Node.js/TypeScript)
|
||||
│ ├── src/ # TypeScript source
|
||||
│ └── dist/ # Built output (tsup)
|
||||
├── scripts/ # Development scripts
|
||||
│ └── validate-char-count.js # Pre-commit character count validator
|
||||
├── dist/ # Generated distribution files (auto-generated)
|
||||
│ ├── global-commands/ # For global installation (~/.claude/, etc.)
|
||||
│ └── local-commands/ # For per-project installation (.claude/, etc.)
|
||||
├── .husky/ # Git hooks (husky)
|
||||
│ └── pre-commit # Runs character count validation
|
||||
├── docs/ # Documentation and assets
|
||||
├── specs/ # Feature specs (if any in-progress)
|
||||
├── install.js # Interactive installer (Node.js)
|
||||
├── package.json # Root package (husky only, private: true)
|
||||
├── version.json # Version metadata
|
||||
└── README.md # User documentation
|
||||
```
|
||||
High-level directory structure, key files, workflow prompt inventory, and file naming conventions.
|
||||
|
||||
## Key Files
|
||||
Details: [Architecture](./.agents-docs/AGENTS-architecture.md)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `install.js` | Main installer - generates and installs workflow files to AI tool directories |
|
||||
| `src/plan2code-*.md` | Source workflow prompts (the "source of truth") |
|
||||
| `scripts/validate-char-count.js` | Pre-commit validator ensuring all source prompts ≤ 11,000 chars |
|
||||
| `version.json` | Version metadata (name, version, description) |
|
||||
| `QUICK-REFERENCE.md` | User quick-reference card |
|
||||
## Plan2Code Loop
|
||||
|
||||
## Workflow Prompts (in `src/`)
|
||||
Autonomous CLI tool (`plan2code-loop/`) that implements specs by looping through tasks. Covers loop architecture, modes (one-task vs one-phase), completion markers, and key source files.
|
||||
|
||||
| File | Step | Purpose |
|
||||
|------|------|---------|
|
||||
| `plan2code---init.md` | Init | Generate AGENTS.md for projects |
|
||||
| `plan2code---init-update.md` | Update | Update AGENTS.md with learnings |
|
||||
| `plan2code---quick-task.md` | 0 | Lightweight planning for small tasks |
|
||||
| `plan2code-1--plan.md` | 1 | Requirements analysis & architecture |
|
||||
| `plan2code-1b--revise-plan.md` | 1b | Mid-implementation revisions |
|
||||
| `plan2code-2--document.md` | 2 | Create implementation specs |
|
||||
| `plan2code-3--implement.md` | 3 | Execute implementation (phase by phase) |
|
||||
| `plan2code-4--finalize.md` | 4 | Validate, summarize, archive |
|
||||
Details: [Plan2Code Loop](./.agents-docs/AGENTS-plan2code-loop.md)
|
||||
|
||||
## Plan2Code Loop (`plan2code-loop/`)
|
||||
## Plan2Code Metrics
|
||||
|
||||
A separate Node.js CLI tool that autonomously implements specs by looping through tasks.
|
||||
Recursive self-improvement toolchain (`plan2code-metrics/`) for collecting run metrics, aggregating by prompt generation, diagnosing weak steps, and proposing prompt edits.
|
||||
|
||||
### Loop Architecture
|
||||
Details: [Plan2Code Metrics](./.agents-docs/AGENTS-plan2code-metrics.md)
|
||||
|
||||
The loop uses an **LLM-driven discovery** approach:
|
||||
- Node app just orchestrates iterations and parses completion markers
|
||||
- The LLM reads spec files (`overview.md`, `phase-X.md`) to discover tasks
|
||||
- The LLM finds unchecked checkboxes, implements ONE task per iteration, marks it complete
|
||||
- No regex parsing of markdown in Node - the AI handles all task discovery
|
||||
## Plan2Code Status Line (Claude CLI)
|
||||
|
||||
### Loop Commands
|
||||
Optional CLI status bar (`src/statusline-claude/`) for Claude Code. Displays model, project, branch, context usage, usage stats (rate limits or token counts), git diff stats, and duration. Reads data directly from Claude Code's stdin JSON — no API calls, no auth, no background processes. Included in `Install All + dev tools` (`A`); also available via `install.js` Custom → S (opt-in).
|
||||
|
||||
```bash
|
||||
# Build the loop CLI
|
||||
cd plan2code-loop && npm run build
|
||||
|
||||
# Run the loop (after linking) - fully interactive
|
||||
plan2code-loop
|
||||
```
|
||||
|
||||
The CLI auto-detects specs in `./specs/`, prompts for selection if multiple found, and handles session continuation interactively. Session state is stored per-spec in `specs/<feature>/.plan2code-loop/`.
|
||||
|
||||
### Loop Modes
|
||||
|
||||
The CLI asks users to choose a loop mode:
|
||||
- **One task per loop** (default) - Each agent invocation implements exactly one task. The Node controller handles git commits.
|
||||
- **One phase per loop** - Each agent invocation implements all remaining tasks in the current phase. The LLM handles git commits (with JIRA ticket ID if provided). The controller parses multiple completion markers from a single iteration.
|
||||
|
||||
### Completion Markers
|
||||
|
||||
The LLM must output one of these formats:
|
||||
- `TASK_COMPLETE: 1.1 - Task description` - Task done successfully
|
||||
- `TASK_BLOCKED: 1.1 - Reason` - Cannot complete task
|
||||
- `PHASE_COMPLETE` - Current phase finished (phase mode only)
|
||||
- `LOOP_COMPLETE` - All phases finished
|
||||
Details: [Architecture](./.agents-docs/AGENTS-architecture.md) (see Status Line section)
|
||||
|
||||
## Development Commands
|
||||
|
||||
```bash
|
||||
# Install dev dependencies (sets up husky pre-commit hooks)
|
||||
npm install
|
||||
Build commands, installer menu options, non-interactive skill-build verification, skills CLI delegation, skill format, and how to edit workflow prompts.
|
||||
|
||||
# Run the interactive installer
|
||||
node install.js
|
||||
Details: [Development Commands](./.agents-docs/AGENTS-development-commands.md)
|
||||
|
||||
# Install to specific platform only
|
||||
node install.js --platform claude
|
||||
node install.js --platform cursor
|
||||
node install.js --platform copilot
|
||||
node install.js --platform continue
|
||||
node install.js --platform windsurf
|
||||
node install.js --platform codeium
|
||||
node install.js --platform vscode-copilot
|
||||
## Code Style & Gotchas
|
||||
|
||||
# Preview changes without installing
|
||||
node install.js --dry-run
|
||||
Language/toolchain conventions for `install.js` vs TypeScript packages, and pitfalls to avoid (version sync, `.gitignore` pre-flight, character limits, User Feedback table format).
|
||||
|
||||
# Show local (per-project) installation instructions
|
||||
node install.js --local
|
||||
|
||||
# Uninstall all Plan2Code files
|
||||
node install.js --uninstall
|
||||
|
||||
# Plan2Code Loop
|
||||
cd plan2code-loop && npm install # First time setup
|
||||
cd plan2code-loop && npm run build # Build the CLI
|
||||
```
|
||||
|
||||
### Installer Menu Options
|
||||
|
||||
| Option | Action |
|
||||
|--------|--------|
|
||||
| `A` | Install to ALL platforms + build & link loop CLI |
|
||||
| `O` | Build & link plan2code-loop CLI only |
|
||||
| `U` | Uninstall prompts from all platforms + unlink loop CLI |
|
||||
| `L` | Show local (per-project) install instructions |
|
||||
| `1-7` | Install to specific platform |
|
||||
|
||||
## How the Installer Works
|
||||
|
||||
1. **Reads source prompts** from `src/plan2code-*.md`
|
||||
2. **Generates platform-specific files** with appropriate headers (YAML frontmatter for some platforms)
|
||||
3. **Writes to `dist/`** subdirectories organized by destination type
|
||||
4. **Copies to target directories** (global: `~/.claude/commands/`, etc.)
|
||||
|
||||
### Platform-Specific File Formats
|
||||
|
||||
| Platform | Extension | Header |
|
||||
|----------|-----------|--------|
|
||||
| Claude Code | `.md` | None |
|
||||
| Cursor | `.md` | None |
|
||||
| Copilot CLI | `.md` | YAML frontmatter |
|
||||
| Continue | `.prompt.md` | YAML frontmatter |
|
||||
| Windsurf | `.md` | YAML frontmatter |
|
||||
| VS Code Copilot | `.prompt.md` | YAML frontmatter |
|
||||
| Codeium | `.md` | YAML frontmatter |
|
||||
|
||||
## Naming Convention
|
||||
|
||||
Workflow files follow a strict naming pattern:
|
||||
- **Utilities:** `plan2code---<name>.md` (triple dash)
|
||||
- **Numbered steps:** `plan2code-<N>--<name>.md` (single dash, number, double dash)
|
||||
|
||||
Examples:
|
||||
- `plan2code---init.md` (utility)
|
||||
- `plan2code-1--plan.md` (step 1)
|
||||
- `plan2code-1b--revise-plan.md` (step 1b)
|
||||
|
||||
## Editing Workflow Prompts
|
||||
|
||||
When modifying workflow prompts in `src/`:
|
||||
|
||||
1. Edit the source file in `src/`
|
||||
2. Run `node install.js` to regenerate distribution files
|
||||
3. Test the workflow in your AI tool of choice
|
||||
4. The `dist/` folder is regenerated automatically - don't edit files there directly
|
||||
|
||||
## Code Style
|
||||
|
||||
- **JavaScript:** CommonJS modules (Node.js built-ins only - no external dependencies)
|
||||
- **Console output:** Uses ANSI color codes via the `COLORS` constant
|
||||
- **User interaction:** Readline-based interactive prompts
|
||||
- **File operations:** Synchronous fs operations for simplicity
|
||||
|
||||
## Gotchas/Pitfalls
|
||||
|
||||
- **Version sync:** When adding a new version to `CHANGELOG.md`, also update `version.json` to match. The installer displays the version from `version.json` in its header.
|
||||
- **Loop `.gitignore` setup:** `ensureGitignore()` runs at startup in `Controller.run()` as a pre-flight step, not just inside `createTaskCommit()`. This is critical for phase mode where the Node controller doesn't handle commits — without it, `git add -A` would stage spec files.
|
||||
- **Workflow file character limit:** All `src/plan2code-*.md` files must be ≤ 11,000 characters. A husky pre-commit hook enforces this. The 11,000 limit leaves buffer for platform-specific YAML headers (106-142 chars) to stay under Windsurf's 12,000 char limit.
|
||||
|
||||
## Git Commit Messages
|
||||
|
||||
- **AI Assisted footer:** All git commit messages must include `AI Assisted` as the final line, separated from the message body by a blank line
|
||||
- **Commit paths:** This is enforced across all commit surfaces:
|
||||
- Loop task mode: `createTaskCommit()` in `plan2code-loop/src/utils/git.ts` appends the footer automatically
|
||||
- Loop phase mode: Prompt template instructs the LLM to add `-m "AI Assisted"` as the final flag
|
||||
- Implement mode: User-facing commit suggestions in `src/plan2code-3--implement.md` include the footer
|
||||
- **Init workflow:** `src/plan2code---init.md` generates AGENTS.md files with a Git Commit Messages section that includes this convention by default
|
||||
Details: [Code Style & Gotchas](./.agents-docs/AGENTS-code-style.md)
|
||||
|
||||
## Mascot
|
||||
|
||||
The project has a mascot - an ASCII art robot that appears in installer output and workflow prompts. Mascot variants are defined in `MASCOT` constant in `install.js` and appear in workflow markdown files.
|
||||
The project has a mascot called "Planny" — an ASCII art robot that appears in installer output and workflow prompts. Mascot variants are defined in the `MASCOT` constant in `install.js` and appear in workflow markdown files.
|
||||
|
||||
```
|
||||
╭───╮
|
||||
@@ -207,3 +60,18 @@ The project has a mascot - an ASCII art robot that appears in installer output a
|
||||
│ ◡ │
|
||||
╰───╯
|
||||
```
|
||||
|
||||
## Keeping this file current
|
||||
|
||||
The `Failure log` section below is a recording of mistakes made by previous AI Agents while working with this code base.
|
||||
|
||||
When you make a mistake, get corrected, or discover something about this codebase that wasn't written down:
|
||||
|
||||
1. Add one line to the `Failure log` below, in the imperative, describing the correct behaviour.
|
||||
2. Keep it specific to this repo. General advice belongs nowhere.
|
||||
3. If this fix is a workflow rather than a rule, put it in `.claude/skills/` and link it from here.
|
||||
4. Include the change in the same commit and mention it in your summary.
|
||||
|
||||
## Failure log
|
||||
|
||||
- Do not audit `CHANGELOG.md` headings through PowerShell — the emoji come back as `?`. See the gotcha for the correct approach.
|
||||
|
||||
@@ -2,6 +2,443 @@
|
||||
|
||||
All notable changes to Plan2Code will be documented in this file.
|
||||
|
||||
## v2.2.0
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **Committed Agent Skills build** — each workflow under `src/` now builds to `skills/<skill-name>/SKILL.md`, with companion references nested under `references/`. `npm run build:skills` regenerates the artifact, and `npm test` verifies that committed skills have not drifted from their source prompts.
|
||||
|
||||
- **Project-scoped skill installation** — Custom → `L` now installs Plan2Code directly into the current project through the skills CLI instead of printing manual copy instructions.
|
||||
|
||||
### 🔧 Changed
|
||||
|
||||
- **Installation now delegates to the skills CLI** — Plan2Code ships one canonical Agent Skill format instead of maintaining separate command, prompt, workflow, and skill outputs for individual tools. The installer builds `skills/`, checks `npx --yes skills`, removes stale Plan2Code skills, and runs `skills add` with an explicit workflow list. Installation now requires Node.js 18+ and network access; installed skills can be updated with `npx skills update -g`.
|
||||
|
||||
- **Legacy installation cleanup is automatic** — install and uninstall sweep files written by earlier per-tool installers so old commands cannot shadow the canonical skills. This includes the Gemini CLI files retained exclusively for uninstall compatibility.
|
||||
|
||||
- **Default installation is skills-only** — main-menu option `I` no longer installs `plan2code-loop`. The loop remains part of `A` and is still available separately through Custom → `O`.
|
||||
|
||||
- **One frontmatter contract serves every agent** — generated `SKILL.md` files always include `name`, `description`, and `disable-model-invocation: true`. Reference files remain nested under their owning skill, eliminating the old flat-file path rewrite and its column-position constraint.
|
||||
|
||||
## v2.1.1
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **Failure Log convention in the init prompts** — `/plan2code-init` now generates a `## Keeping this file current` / `## Failure log` pair at the end of `AGENTS.md`, so agents record repo-specific corrections as one imperative line each instead of relearning them. The section is always-inline (never split to `.agents-docs/`) and is listed in the generated `CLAUDE.md` pointer block. `/plan2code-init-update` gains a matching **Failure Log Audit** step that flags the section as missing on existing `AGENTS.md` files and offers to add it.
|
||||
|
||||
- **`plan2code-changelog` skill** (`.claude/skills/plan2code-changelog/`) — validates the CHANGELOG version before a PR is opened. Reads `main` for the current latest version, classifies the branch's changes to pick minor vs patch, and keeps `CHANGELOG.md`, `package.json`, and `version.json` in lockstep. Exists because parallel branches pick colliding version numbers independently.
|
||||
|
||||
### 🔧 Changed
|
||||
|
||||
- **`AGENTS.md` preamble shortened** in the init template — the generated header no longer enumerates specific agents (Claude Code, Cursor, Codex, Copilot, Devin, Zed) and reads "AI coding agents working with code in this repository."
|
||||
|
||||
- **`CLAUDE.md` added at the repo root** — a pointer file to `AGENTS.md`, matching what `/plan2code-init` now tells agents to generate.
|
||||
|
||||
- **`CLAUDE.md` template reconciled between the two init prompts** — `/plan2code-init` emitted the `CRITICAL — MANDATORY FIRST STEP` directive followed by a seven-item bullet list, while `/plan2code-init-update`'s Step 7 reference emitted the directive followed by a single prose line. Running one workflow after the other rewrote `CLAUDE.md` back and forth. `src/plan2code-init-update-references/ai-agent-file-sync.md` now carries the bullet-list form for both the `CLAUDE.md` and the generic reference template.
|
||||
|
||||
### 🐛 Fixed
|
||||
|
||||
- **Typos in the Failure Log template** — the template shipped in `src/plan2code-init.md` and `src/plan2code-init-update.md` said `AGETNS.md`, `mistage`, and "mistakes make by". Since agents copy this block verbatim into generated `AGENTS.md` files, the errors propagated into every project initialized with it.
|
||||
|
||||
## v2.1.0
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **GitHub Issues backend for Pathfinder** — `/plan2code-0-pathfinder` no longer assumes local files. Chart Step 1 now asks, HITL and never self-picked, where the map should live: **local** (the default — files under gitignored `specs/<idea>/pathfinder/`, private and solo) or **github** (a `pathfinder:map` issue whose decision questions are its sub-issues, driven by the `gh` CLI). The pick is recorded as the first `## Ground rules` bullet, never re-asked and never switched mid-map, and Auto-Discovery resolves either backend — an issue URL or number as the argument routes straight to `github`.
|
||||
|
||||
On `github`, the local model maps onto the tracker's own primitives rather than being simulated in issue bodies: a question is a **sub-issue** (`sub_issues` endpoint), blocking is GitHub's **native issue dependencies** (`dependencies/blocked_by`, so the frontier renders in GitHub's UI without opening the map), the claim is the **assignee**, `Type:` becomes a single `pathfinder:<type>-<mode>` label so type and mode cannot drift, `Locked: yes` becomes `pathfinder:locked`, and resolution is an `## Answer` comment followed by a close — `completed` for a decision, `not planned` for a question ruled out of scope. Both wiring calls key on the issue's **database id**, not its `#number`. The map body therefore carries no question checklist at all: the frontier is a live query, which removes the single largest source of drift in the local backend.
|
||||
|
||||
Three things stay on local disk whatever the backend: runnable sketches (`specs/<idea>/pathfinder/sketch-NN/`), anything secret, and the `PLAN-DRAFT-<date>.md` — `/plan2code-1-plan` discovers its input with `ls specs/` and has no notion of a tracker, so a draft that existed only as an issue would be invisible to the rest of the pipeline.
|
||||
|
||||
Guardrails carried over from the local backend's assumptions: `github` is only offered after a five-check preflight (`gh` present, authenticated, GitHub remote, issues enabled, push access), and the offer must name the repo's **visibility** in the same breath, because a map on a public tracker publishes the destination, the rejected alternatives, and the codebase recon. The Step 3 recon is held in-session and published at Step 6, so the Step 4 no-fog off-ramp leaves no litter on a shared tracker. A `gh` failure mid-session stops the session rather than falling back to local files, which would fork the map.
|
||||
|
||||
Depth lives in a seventh reference file, `src/plan2code-0-pathfinder-references/github-issues.md` — preflight, label set, the local↔GitHub equivalence table, create-then-wire charting, the frontier query, resolve and out-of-scope flows, reconcile, the trail footer, handoff, and a failure-mode table. Adapted from the GitHub tracker doc behind Matt Pocock's [`wayfinder`](https://github.com/mattpocock/skills/tree/main/skills/engineering/wayfinder) skill (MIT).
|
||||
|
||||
### 🔧 Changed
|
||||
|
||||
- **`src/plan2code-0-pathfinder.md` recompressed** to absorb the new `## Backend` section within the 11,000-character workflow-file limit — duplication between the orchestrator and its reference files was removed (the local layout and marker legend now live only in `questions.md`; Form A/B footer detail only in `trail.md`), and step text tightened. No behaviour was dropped.
|
||||
|
||||
## v2.0.0
|
||||
|
||||
Ports upstream v1.17.0 and v2.0.0 into Plan2Code.
|
||||
|
||||
### 💥 Breaking
|
||||
|
||||
- **Gemini CLI is no longer an install target.** The `.gemini/commands/*.toml` surface is removed, along with the build-time machinery it required: `generateTomlContent()`, `writeTomlToDestination()`, `inlineReferenceContent()`, and the `dest.type === 'toml'` branch in `syncPrompts()`. Every remaining target resolves `Read references/*.md` directives at runtime, so reference content no longer needs inlining at build time.
|
||||
|
||||
**Uninstall still cleans up Gemini files.** The `.gemini/commands` entry is deliberately retained in the uninstall target list (labelled `Gemini CLI (legacy — uninstall only)`) behind a new `uninstallOnly` flag, so `.toml` files written by v1.x installs can still be removed. Do not add it back to `LOCAL_DESTINATIONS` / `GLOBAL_DESTINATIONS`.
|
||||
|
||||
All other platforms are unaffected — Claude Code (skills *and* commands), Cursor, Windsurf, Continue, Codeium, GitHub Copilot, VS Code Copilot, Pi, Crush, Amp, Devin, OpenCode, and Zed all install exactly as before.
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **Pathfinder workflow** — new `/plan2code-0-pathfinder` command, an optional Step 0 for an idea too big and unclear to plan yet: where you can feel the shape of the work but can't write it down as requirements. Adapted from Matt Pocock's [`wayfinder`](https://github.com/mattpocock/skills/tree/main/skills/engineering/wayfinder) skill (MIT), reworked for Plan2Code's local `specs/` workflow.
|
||||
|
||||
Pathfinder names a **destination**, then charts the way to it as a map of decision **questions** under `specs/<idea>/pathfinder/` — `map.md` as the index plus one `questions/NN-<slug>.md` file per decision. It resolves **one question per session** (research excepted), and each resolution clears the fog ahead of it, graduating whatever became specifiable into fresh questions. When nothing is left to decide, it writes `specs/<idea>/PLAN-DRAFT-<date>.md` carrying the status string `/plan2code-1-plan` already recognizes, so planning resumes at Phase 4 in the same folder with Requirements, System Context, and Scope pre-answered.
|
||||
|
||||
**Grilling is batched** — up to three *independent* probes per turn instead of one probe per round trip, delivered either through the environment's structured question tool or as numbered prose Q blocks, chosen per batch by a detail test. Probes are written in plain English, and any probe you skip is re-asked rather than quietly dropped. It **plans, it never builds**: four question types — `grill` (HITL, the default), `research` (AFK, resolved by background subagents in parallel), `sketch` (HITL), and `legwork`.
|
||||
|
||||
Map state uses the house checkbox vocabulary — `[ ]` open (the frontier), `[/]` claimed, `[x]` resolved, `[!]` blocked, `[-]` out of scope. `questions/` is ground truth and `map.md` is a rebuildable index: every Work session reconciles the two before choosing, which self-heals drift and recovers claims left by a crashed session. Every response ends with a **Trail Footer** — a one-line path from `START` to the `⚑` destination, a numbered legend, a plain-English confidence line, and exactly one closer chosen by turn type (a turn that asks you something never emits a resume command).
|
||||
|
||||
Depth lives in six new reference files under `src/plan2code-0-pathfinder-references/` (`chart`, `grilling`, `questions`, `resolve`, `handoff`, `trail`), so the orchestrator stays a dispatcher and the skill has no external skill dependencies.
|
||||
|
||||
- **Community feedback submission** — `/plan2code-4-finalize` gains STEP 6.5: after archival, assembles a METRICS_JSON payload from the completed run and submits it as a `community-feedback`-labeled GitHub issue on `jparkerweb/plan2code`, with a tiered fallback (`gh` CLI issue create → browser-opened prefilled issue → printed URL) for environments without `gh`. Payload schema and submission tiers live in the new `plan2code-4-finalize-references/community-feedback-submission.md`. Step 5 now asks for explicit submission consent and skips straight to Step 6 when declined.
|
||||
|
||||
- **Community submission ingestion in plan2code-metrics** — new "Fetch community submissions" CLI flow (`community.ts`) lists open feedback issues via `gh`, validates and parses each `METRICS_JSON` payload (type-only validation; malformed submissions are skipped and logged, not fixed up), imports them into the local run store deduped by `run_id`, re-aggregates, and closes each imported issue. Matches an open issue by the `community-feedback` label OR the `[Feedback]` title prefix OR the `METRICS_JSON` marker, so browser/print-tier submissions from outside contributors are still picked up; paginates fully (`--limit 1000`); and closes issues idempotently even on the duplicate path.
|
||||
|
||||
- **Community runs cohort by Plan2Code version** — ingested community runs are keyed into cohorts by their `plan2code_version` rather than by a prompt-file fingerprint, since community submissions carry the installed, platform-transformed prompts and an LLM-generated payload. Runs now carry a `source` (`local`/`community`) tag, and `current_cohort_key` prefers local cohorts so ingested feedback never displaces the maintainer's current prompt generation.
|
||||
|
||||
- **Devin CLI as an AI backend** for both `plan2code-metrics` (`invoke-llm.ts`) and `plan2code-loop` (`agents/devin-cli.ts`) — `devin --print --prompt-file <file> --permission-mode dangerous`. Unlike upstream, which replaced GitHub Copilot CLI with Devin, Plan2Code keeps **both**: Claude Code, GitHub Copilot CLI, and Devin CLI are all selectable. Existing Copilot CLI selections keep working.
|
||||
|
||||
### 🔧 Changed
|
||||
|
||||
- **README rebuilt** around a shorter, task-first structure, with the deep material split into a new `.readme/` folder: `walkthrough.md`, `autonomous-loop.md`, `status-line.md`, `metrics.md`, `test-bot.md`.
|
||||
- **Docs site and README redesigned** around an "airmail" postcard theme — a fixed four-sided airmail-chevron page frame, sticky header, and the workflow presented as six posted letters, with Pathfinder and the optional Review step both surfaced. Adds a postage-stamp favicon set (`favicon.svg` / `.ico` / `.png` / `apple-touch-icon.png`) and a new README banner; removes three orphaned images (`desk.jpg`, `install-script.jpg`, `plan2code.jpg`).
|
||||
- **`/plan2code-4-finalize` archives `pathfinder/` with the spec** — STEP 6 now names `pathfinder/` in the move list and no longer describes the cleanup target as "research or scratch files," wording that pointed an agent straight at `pathfinder/questions/`. The map is the rationale record behind the plan, in the same class as `PLAN-CONVERSATION-*.md`.
|
||||
- **`/plan2code-1b-revise-plan` no longer deletes `pathfinder/`** — its Step 6 cleanup had the same "research or scratch files" wording.
|
||||
- **`/plan2code-quick-task` is no longer labelled "Step 0"** — pathfinder now owns step 0, and quick-task was never a pipeline step. It registers as a utility (like `init`, `review`, and `handoff`), so its generated description reads `Plan2Code Quick Task: Quick Task Mode`. Filename, skill name, and command path are unchanged.
|
||||
- **`/plan2code-handoff` asks where to save** — the OS temp directory is now the default, with `./handoffs/` or any other path available on request. Adds a spec-awareness section: when the session worked inside `specs/<feature>/`, the handoff cites the in-progress `phase-X.md` and its actual checkbox state rather than relying on conversation memory.
|
||||
- **`/plan2code-init-update` Step 7 offloaded to a reference file** — the AI Agent File Sync detail moves to `plan2code-init-update-references/ai-agent-file-sync.md` with an inline fallback. The `CLAUDE.md` MANDATORY-FIRST-STEP template is unchanged.
|
||||
- **`/plan2code-review` Session End offloaded to a reference file** — next-step routing moves to `plan2code-review-references/session-end.md` with an inline fallback.
|
||||
- **Status line: context-bar token count suppressed on token-usage accounts** — the bar's `(84k)` reads the same `context_window.total_input_tokens` the `in`/`out` usage segment already shows on Enterprise/Bedrock/Vertex/PAYG accounts. It now renders only on Pro/Max/Teams (rate-limit) accounts, where no other segment carries an absolute token count. The `items.contextTokens` flag still turns it off entirely.
|
||||
- **`aggregator.ts` refactor** — extracted `writeRunFile()` (dedup-by-`run_id` write) out of `importRun()` so the community ingestion path can reuse it without a source file path; `collector.ts` now exports `extractMetricsJson()` for the same reason.
|
||||
- **`.agents-docs/AGENTS-code-style.md`** documents a metrics gotcha: when a `PLAN-DRAFT-*.md` carries no `METRICS_JSON` comment, `collector.ts` scrapes it by regex, and the four confidence-*breakdown* patterns match a bare dimension word plus a number **without** requiring a `%` — so even a table row like `| Requirements | 11 |` gets ingested as a planning confidence score.
|
||||
- **`.agents-docs/AGENTS-architecture.md`** documents the column-0 requirement for `Read references/*.md` directives — `install.js` anchors its flat-file path-rewrite regex at `^`, so an indented `Read` line is silently skipped.
|
||||
|
||||
### 🐛 Fixed
|
||||
|
||||
- Broken review-command row and column alignment in `QUICK-REFERENCE.md`.
|
||||
- `plan2code-loop` banner misspelled the mascot as "Plany".
|
||||
|
||||
## v1.16.1
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **Status line: git worktree awareness** — a session running in a linked git worktree now renders the project segment as `repo ⑂ worktree` (e.g. `plan2code ⑂ spike`) instead of only the worktree's directory name, which previously made the session look like an unrelated project
|
||||
- Repo identity resolved from `git rev-parse --git-common-dir`, so it is correct regardless of how the worktree directory was named (bare `<name>.git` main repos included)
|
||||
- A leading repo prefix is stripped from the worktree name (`plan2code-user-auth` → `user-auth`), and the name collapses to a bare `⑂` when it merely restates the branch already on screen — matched across `/ _ . -` separators and type prefixes like `feature/`, and only when the branch is actually displayed
|
||||
- Toggleable via the new `items.worktree` config flag (default on); one extra timeout-bounded `git` call, skipped outside git repos
|
||||
|
||||
## v1.16.0
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **`/plan2code-handoff` skill** — compacts the current conversation into a self-contained handoff document (written to gitignored `./handoffs/<timestamp>-handoff.md`) so a fresh session or another agent can resume the work
|
||||
- Always captures a confirmed **Next task**: infers a candidate from context and requires the user to confirm or fill it in before the file is written
|
||||
- References plan specs, logs, and files by path rather than copying them; strips secrets; suggests follow-on skills and verification steps
|
||||
- Repo-safe: checks `git check-ignore` and warns (without silently editing `.gitignore`) when `handoffs/` isn't ignored in an arbitrary repo
|
||||
- **Repo-local release publisher skill** — new `/plan2code-publish` maintainer skill in `.claude/skills/` cuts a GitHub Release from the top `CHANGELOG.md` entry once `CHANGELOG.md`, `version.json`, and `package.json` agree and the version is ahead of the latest published release. Dev tooling only — deliberately excluded from `install.js`, never installed to `~/.claude/skills/`.
|
||||
|
||||
### 🐛 Fixed
|
||||
|
||||
- **Review workflow next-step suggestion made context-aware** — `/plan2code-review` Session End now reconciles three signals: session context (what preceded the review in the conversation), the user's review intent, and on-disk spec state gathered shell-agnostically — a file-search tool's empty result is never treated as proof that no specs exist. Suggestions render only at actual session end, cite their evidence and its source, and conflicting signals ask one targeted question instead of guessing.
|
||||
|
||||
## v1.15.4
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **Status line: reasoning effort + context token count** — model segment now appends the current reasoning effort level (e.g. `Sonnet 5 | High`, hidden when the model doesn't support an effort parameter); context bar now shows raw input tokens used alongside the percentage (e.g. `42% (84k)`), independently toggleable via new `items.effort` / `items.contextTokens` config flags
|
||||
|
||||
## v1.15.3
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **Session-end summaries across workflows** — consistent closing context in implement, document, finalize, init-update, quick-task, review, and revise-plan
|
||||
- Implement: work summary + upcoming-phases table (task counts, goals) after each phase approval
|
||||
- Document: phase-overview table at close to help plan sessions and review gates
|
||||
- **Sync & Maintain in init-update** — new unified doc-surface sync option (menu item #9)
|
||||
- Covers `AGENTS.md`, `.agents-docs/`, active `specs/`, README, and human docs; tier-voice routing, never duplicates across tiers
|
||||
- Adaptive to repo conventions — detects the human-docs tree, never assumes
|
||||
- **Research steps in plan** — domain research (Phase 1), tech-options research (Phase 4), and an investigate-to-close-gaps rule on the 90% confidence gate
|
||||
|
||||
### 🔧 Changed
|
||||
|
||||
- **Spec auto-discovery hardened** — replaced Glob (silently fails on gitignored `specs/`) with explicit shell `ls` across document, implement, finalize, init-update, and revise-plan
|
||||
- **Revision-mode guardrails** — `1b-revise-plan` edits restricted to `specs/` paths only; execution-shaped language removed; cleanup step added
|
||||
- **Commit-message enforcement in implement** — subject ≤100 chars, exactly three `-m` flags, no body
|
||||
- **Finalize documentation review expanded** — audits `AGENTS.md` + `.agents-docs/`, mandates corrections (not just additions), routes facts per tier voice
|
||||
- **Quality language pass** — plan demands edge cases/failure modes and measurable criteria; document and finalize role statements sharpened
|
||||
|
||||
### 🐛 Fixed
|
||||
|
||||
- **Review workflow next-step guard** — Review mode no longer suggests `/plan2code-3-implement` when `overview.md` and `phase-*.md` files don't exist. If the document step hasn't been run yet, it now correctly directs users to `/plan2code-2-document` first.
|
||||
|
||||
## v1.15.2
|
||||
|
||||
### 🔧 Changed
|
||||
|
||||
- **Review workflow renamed** — `/plan2code-3b-review` → `/plan2code-review`
|
||||
- Removed `3b` step-number prefix; review is now a standalone utility workflow (like `init` and `quick-task`)
|
||||
- Source file: `src/plan2code-review.md` (was `plan2code-3b-review.md`)
|
||||
- Reference directory: `src/plan2code-review-references/` (was `plan2code-3b-review-references/`)
|
||||
- Updated all docs, installer config, architecture docs, and cross-references
|
||||
|
||||
## v1.15.1
|
||||
|
||||
### 🔧 Changed
|
||||
|
||||
- **Zed agent support** — Added Zed to all agent support documentation
|
||||
- `README.md` Supported Platforms list
|
||||
- `AGENTS.md` intro and Project Overview
|
||||
- `.agents-docs/AGENTS-development-commands.md` Platform-Specific File Formats table
|
||||
- `install.js` Agent Skills platform label
|
||||
- `src/plan2code-init.md` AGENTS.md template text
|
||||
|
||||
## v1.15.0
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **Claude CLI status line** — Persistent three-line status bar for Claude Code, displaying model, project, git branch, uncommitted diff stats, session duration, context window usage bar, and plan/quota usage. Reads all data from Claude Code's stdin JSON — no API calls, no auth, no background processes.
|
||||
- `src/statusline-claude/statusline.js` — self-contained: stdin parsing, config loader, ANSI formatting, orchestration
|
||||
- `src/statusline-claude/statusline-config.json` — distributed default config
|
||||
- `src/statusline-claude/README.md` — user docs: install, config, usage modes, troubleshooting
|
||||
- **Planny mascot icons** on each line (`╭─╮`, `│★│`, `╰─╯`) in brand colors, with monochrome fallback
|
||||
- **Context window bar** (12-cell `▰▱`) with configurable `autocompactBuffer` (default 33000 tokens) so the percentage reflects *usable* context, not the raw window
|
||||
- **Adaptive plan usage segment** — auto-detects data shape:
|
||||
- Pro/Max/Teams (rate_limits present) → `5h: NN% · 7d: NN%`
|
||||
- Bedrock / Vertex / PAYG (no rate_limits) → `NNk in · NNk out` session tokens
|
||||
- Segment hidden when neither is available
|
||||
- **Real uncommitted diff stats** via `git diff HEAD --numstat` → `+NN -NN`
|
||||
- **Color-coded thresholds** — green / yellow / red for context bar and rate limits
|
||||
- **Separator line** rendered at fixed 55-character width for consistent alignment
|
||||
- **Session cost display** — Estimated session cost shown next to duration (e.g. `49m ($4.62)`) via new `items.sessionCost` config (default on); reads `cost.total_cost_usd` from Claude Code stdin; hidden when zero or unavailable
|
||||
- **Compact mode redesigned** — `"compact": true` now renders two content lines (no mascot icons, no separator) instead of one cramped single line. The single-line variant truncated in narrow terminals — the exact case compact mode was meant to help. Default remains `false`.
|
||||
- **Silent failure** on all errors — never crashes, never blocks the CLI; 1.5s per-call timeouts on git operations; non-git workspaces short-circuit without spawning subprocesses
|
||||
- **Installer integration** — Status line included in `Install All + dev tools` (`A`); also available via `Custom → S` (opt-in).
|
||||
- Copies `statusline.js` verbatim to `~/.claude/plan2code-statusline.js` (single-file design, no bundling)
|
||||
- Registers in `~/.claude/settings.json` under `statusLine` via atomic temp-file + rename write
|
||||
- Preserves existing `statusline-config.json` on reinstall
|
||||
- Detects non-plan2code custom `statusLine` configs — prompts before replacing; auto-backs up to `statusline-previous.json`
|
||||
- Uninstall (`U`) removes bundled script + `settings.json` entry; preserves `statusline-config.json`. Only removes `settings.statusLine` if it points to the plan2code bundle — non-plan2code entries are left intact.
|
||||
|
||||
## v1.14.0
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **Review workflow (Step 3b)** — New `/plan2code-3b-review` command for comprehensive post-implementation code review
|
||||
- 5-step process: scope detection, context analysis, 11-dimension review, spec/test assessment, summary with fix options
|
||||
- Adaptive scope: focused (named files), branch (git diff), or full (subsystem) — auto-detected with user override
|
||||
- 3 severity levels (Critical, Warning, Suggestion) with High-confidence-only findings
|
||||
- Reference file architecture: orchestrator (≤11k chars) + 3 companion reference files loaded via Read directives
|
||||
- Deep verification protocol with use-case tracing for architectural findings and adversarial self-check
|
||||
- Detailed dimension checklists (8 non-obvious items per dimension with anti-patterns and "don't flag" guidance)
|
||||
- False-positive catalog with detection shortcuts to prevent common false findings
|
||||
- Finding verification gate — re-reads source at each cited line before presenting
|
||||
- Mnemonic fix options: `H` (high-priority), `A` (all), `S` (specify by number)
|
||||
- Post-fix Plan/Apply/Verify pipeline — enterprise-grade fix quality with full validation
|
||||
- Doc review mode — >70% doc changes triggers editorial critique
|
||||
- Graceful degradation for agents that can't read external files
|
||||
- Spec-aware when `specs/` exists; works standalone for any codebase
|
||||
- Context-aware session end with pipeline state detection
|
||||
- Registered in installer for all 14+ platform destinations
|
||||
- Installer copies reference directories for all 13+ platform targets
|
||||
- Reference content inlined into TOML output (Gemini CLI) so platforms that cannot resolve runtime file reads still get full workflow depth
|
||||
- Uninstall now removes orphaned `*-references/` directories even when prompt files were already removed manually
|
||||
- Implement workflow session-end now suggests review after each phase
|
||||
|
||||
## v1.13.0
|
||||
|
||||
### 🔧 Changed
|
||||
|
||||
- **Simplified workflow naming convention** — All workflow and skill files unified to single-dash naming, eliminating double-dash (`--`) and triple-dash (`---`) conventions
|
||||
- Renamed 8 source prompt files in `src/` (e.g., `plan2code---init.md` → `plan2code-init.md`, `plan2code-1--plan.md` → `plan2code-1-plan.md`)
|
||||
- Updated `install.js` `SOURCE_PROMPTS` metadata and `generateFilename()` to emit single-dash names
|
||||
- Updated all cross-references inside workflow prompt markdown content
|
||||
- Updated documentation: `README.md`, `QUICK-REFERENCE.md`, `AGENTS.md`, `.agents-docs/AGENTS-architecture.md`
|
||||
- Updated tooling references in `plan2code-loop/`, `plan2code-bot/`, and `plan2code-metrics/`
|
||||
- Fixed pre-existing broken test assertion in `plan2code-bot` step-instructions test
|
||||
- Repaired Windows-1252 / U+FFFD encoding artifacts (em-dashes) in `plan2code-loop` and `plan2code-bot` source files
|
||||
|
||||
### 🎁 Added
|
||||
|
||||
- **Devin platform support** — Added Devin to supported platforms list across `README.md`, `AGENTS.md`, `install.js` Agent Skills targets, `.agents-docs/AGENTS-development-commands.md`, and the `/plan2code-init` prompt
|
||||
- **CLAUDE.md MANDATORY FIRST STEP template** — `/plan2code-init` and `/plan2code-init-update` now generate a CLAUDE.md template containing a `CRITICAL — MANDATORY FIRST STEP` directive that forces Claude Code to read AGENTS.md before responding to any user message
|
||||
|
||||
## v1.12.0
|
||||
|
||||
### 🔧 Changed
|
||||
|
||||
- **Resilient code references in workflow prompts** — Workflow prompts now explicitly guide AI agents to use semantic anchors (function names, class names, code patterns) instead of line numbers, which become stale as tasks modify files during implementation
|
||||
- **Document mode** (`plan2code-2-document.md`) — New "Code references" block in Task Writing section lists four preferred anchor types with examples; line numbers allowed only as supplemental context
|
||||
- **Revise-plan mode** (`plan2code-1b-revise-plan.md`) — Matching code reference rule added to Step 3 (Execute Revisions) so revised and new tasks follow the same convention
|
||||
- **Implement mode** (`plan2code-3-implement.md`) — New "Verify locations" row in Code Consistency Rules table instructs agents to treat line numbers as approximate and locate by function/symbol name
|
||||
|
||||
## v1.11.1
|
||||
|
||||
### 🔧 Changed
|
||||
|
||||
- **Task complexity check in Document workflow** — Enhanced task writing guidance with lightweight cognitive complexity heuristics (inspired by PR #26)
|
||||
- "Time-boxed" criterion now includes explicit split triggers: 5+ logic branches, 2+ integration points, or shared interface mutation
|
||||
- New "Complexity check" prompt: before finalizing each task, LLM considers logic branches, distinct behaviors, integration points, shared interface impact, and error/edge cases
|
||||
- Tasks complex on 3+ signals must be split; adjacent trivial tasks forming a cohesive unit should be combined
|
||||
- Process step 7 updated to reinforce the complexity check during phase file authoring
|
||||
|
||||
## v1.11.0
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **plan2code-bot** — New autonomous workflow test runner (`plan2code-bot/`) that uses the Claude Agent SDK to simulate a human running the entire plan2code workflow end-to-end
|
||||
- Two auto-detected modes: **new-project** (generates an app idea, creates a subdirectory, runs init through finalize) and **enhancement** (scans existing codebase, proposes a realistic enhancement)
|
||||
- `--idea` flag to seed the idea generator with a specific concept
|
||||
- `--resume` flag to continue incomplete runs from saved state — skips previously succeeded steps and restores idea, config, and implement pass counter
|
||||
- State file (`.plan2code-bot-state.json`) saved after each step; automatically deleted on full success, preserved on failure for later resume
|
||||
- Resume auto-detects state files in current directory (enhancement mode) or immediate subdirectories (new-project mode)
|
||||
- Artifact validation after each step (aborts on missing expected outputs)
|
||||
- **LLM-as-judge evaluation system** — Always-on quality assessment that transforms bot from "yes-man" to authentic QA agent
|
||||
- **Intelligent decision making**: Uses LLM to answer `AskUserQuestion` prompts based on current observations (tools used, files created, errors) instead of hardcoded keyword matching
|
||||
- **Post-step evaluation**: Comprehensive quality assessment after each step using step-specific criteria (score 0-100, strengths, weaknesses, suggestions, critical issues)
|
||||
- **Observation tracking**: Full execution history captured (tools, files, messages, errors, questions with LLM reasoning)
|
||||
- **Quality gate**: Blocks finalization if average score < 60, ensuring minimum quality standards
|
||||
- **Evaluation artifacts**: Creates `BOT-EVALUATION.md` (quality assessments) and `BOT-NOTES.md` (execution observations) for metrics analysis
|
||||
- **Color-coded output**: Green (≥85), yellow (70-84), red (<70) score display in console
|
||||
- **Honest scoring**: Evaluation criteria emphasize realistic assessment (most work scores 70-85, not inflated)
|
||||
- **Metrics-ready data**: Structured `EvaluationResult` and `ExecutionObservation` in state file for `plan2code-metrics` analysis
|
||||
- Bot-friendly skill installation (copies plan2code skills with `disable-model-invocation` stripped)
|
||||
- Implement step loops up to 10 passes until all phases are complete
|
||||
- Installer integration: `C > B` menu option to install bot CLI only
|
||||
|
||||
## v1.10.0
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **Retry logic with escalating timeouts in loop controller** — Timed-out iterations now retry automatically instead of silently continuing
|
||||
- Default base timeout reduced from 30 minutes to 3 minutes per attempt
|
||||
- Up to 5 retry attempts per iteration (configurable via `maxRetries`)
|
||||
- Each retry escalates timeout by +30 seconds (attempt 0 = base, attempt 1 = base + 30s, etc.)
|
||||
- New `executeWithRetry()` method wraps `executeIteration()` with retry loop
|
||||
- Fatal timeout (all attempts exhausted) stops the loop cleanly with a logged error
|
||||
- Spinner displays elapsed seconds during each attempt; retries show attempt count
|
||||
- `maxRetries` field added to `SessionConfig` and `DEFAULT_CONFIG`
|
||||
|
||||
### 🔧 Changed
|
||||
|
||||
- **Append-only scratchpad enforcement in loop prompt templates** — Both task-mode and phase-mode templates now explicitly prohibit editing or reorganizing existing scratchpad content
|
||||
- Instruction changed from "append to scratchpad.md" to "add a new entry at the **bottom**"
|
||||
- Added rule: "Never edit, reorganize, or insert into existing content — only append new entries to the end of the file"
|
||||
- **AGENTS.md restructured into progressive discovery format** — AGENTS.md converted to a lightweight index with summaries and markdown links; full detail moved to `.agents-docs/` section files
|
||||
- `.agents-docs/AGENTS-architecture.md` — Architecture overview and key design decisions
|
||||
- `.agents-docs/AGENTS-code-style.md` — Code style conventions
|
||||
- `.agents-docs/AGENTS-development-commands.md` — Development commands and setup
|
||||
- `.agents-docs/AGENTS-plan2code-loop.md` — Loop CLI architecture and commands
|
||||
- `.agents-docs/AGENTS-plan2code-metrics.md` — Metrics toolchain details
|
||||
- `init-update` prompt updated with `.agents-docs/` detection in pre-flight check
|
||||
- **Agents mode emoji updated** — Loop controller agents-mode indicator changed from wheel to hammer
|
||||
|
||||
## v1.9.1
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **Pi (pi.dev) platform support** — New target for the Pi terminal-based coding agent
|
||||
- Local prompt templates installed to `.pi/prompts/` (flat `.md` with YAML `description` frontmatter)
|
||||
- Global prompt templates installed to `~/.pi/agent/prompts/`
|
||||
- Pi users already get skill support via existing `.agents/skills/` target; this adds native slash command access
|
||||
- No new helper functions needed — Pi uses the same flat-file-with-YAML pattern as Windsurf, Copilot CLI, and Codeium
|
||||
|
||||
### 📚 Documentation
|
||||
|
||||
- `AGENTS.md` Platform-Specific File Formats table updated with Pi row
|
||||
- `README.md` Supported Platforms list updated with Pi (pi.dev)
|
||||
## v1.9.0
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **Progressive discovery for AGENTS.md** — Init and init-update prompts now generate AGENTS.md as a lightweight index with summaries and markdown links, plus `.agents-docs/` section files containing full detail
|
||||
- **Init prompt** — New `## Progressive Discovery` section defines index format, always-inline sections (Project Overview, Git Commit Messages, How to Use This File), `.agents-docs/` directory setup, grouping heuristics, and opt-in restructure offer for existing single-file AGENTS.md
|
||||
- **Init-update prompt** — `.agents-docs/` detection in pre-flight check, legacy migration offer, edit routing (inline sections → AGENTS.md, detailed sections → `.agents-docs/` files), section file lifecycle (create, delete, orphan cleanup), enhanced summary with file count, new update rule 9 (route edits to correct file)
|
||||
- **Reference templates** updated in both prompts to include `.agents-docs/` in the bullet list
|
||||
- **"How to Use This File"** added as a required content section in generated AGENTS.md files
|
||||
|
||||
## v1.8.2
|
||||
|
||||
### 🐛 Fixed
|
||||
|
||||
- **Inflated metrics task counts** — Metrics collector regex now matches only `**Task X.N:**` checkbox items instead of all checkboxes, fixing ~100-200% count inflation from prerequisite and acceptance criteria checkboxes
|
||||
- `collectStep2` (`tasks_per_phase`) uses Task-pattern-only regex
|
||||
- `collectStep3` (`phaseTotal`, `phaseCompleted`, `blockerCount`) all use Task-pattern-only regex
|
||||
- Step 4 overview fallback left unchanged (correctly counts Phase Checklist checkboxes)
|
||||
|
||||
### 🔧 Changed
|
||||
|
||||
- **Document workflow** — Checkbox format `- [ ]` now restricted to Task items only; Prerequisites, Acceptance Criteria, and Success Criteria use plain bullet lists (no checkboxes)
|
||||
- **Implement workflow** — Prerequisite verification changed from checkbox-based (`[x]`/`[?]`/`[!]`) to inline annotation approach (`VERIFIED`/`ASSUMED: [reason]`/`BLOCKED: [reason]`); added clarifying note that checkbox states apply to Task items and Phase Checklist only
|
||||
- **Finalize workflow** — Task completion audit now specifies counting only `**Task X.N:**` checkbox items
|
||||
- **Loop prompt templates** — Both task-mode and phase-mode templates updated: prerequisites use plain bullets with inline annotations instead of checkboxes; Checkbox States section scoped to "Task items only"
|
||||
|
||||
## v1.8.1
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **User feedback collection** — Optional 1-10 rating with reason, what went well, and what went poorly
|
||||
- Finalize prompt (Step 5) asks for optional feedback before archival, writes structured table to `overview.md`
|
||||
- Collector parses `## User Feedback` table from `overview.md` into `RunMetrics.user_feedback`
|
||||
- Aggregator computes `avg_user_rating` and `feedback_count` per cohort
|
||||
- CLI offers interactive feedback collection if none found during metrics collection
|
||||
- Analysis and improvement prompts reference `avg_user_rating` metric target (≥ 7.0)
|
||||
- `UserFeedback` type exported from public API
|
||||
- **Pipe-safe feedback parsing** — User text containing `|` characters is escaped on write and correctly unescaped on parse using negative lookbehind regex
|
||||
|
||||
### 🐛 Fixed
|
||||
|
||||
- **Duplicate run files** — Interactive feedback no longer creates a second run JSON; the original is deleted before re-collecting
|
||||
- **Finalize step ordering** — Feedback collection moved to Step 5 (before archival at Step 6), ensuring `overview.md` is written while still in the active spec directory
|
||||
|
||||
## v1.8.0
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **plan2code-metrics** — New recursive self-improvement toolchain for plan2code contributors (`plan2code-metrics/`)
|
||||
- Fully interactive menu-driven CLI — no flags, all inputs collected via prompts
|
||||
- **Collect** metrics from completed project specs (plan, document, implement, finalize steps)
|
||||
- **Import** run data from other projects for cross-project aggregation
|
||||
- **View** metrics status with health indicators and generation-over-generation deltas
|
||||
- **Analyze** weak steps via AI-powered diagnosis (Claude Code or GitHub Copilot CLI)
|
||||
- **Generate** surgical improvement proposals with automatic validation (char count limits, edit verification)
|
||||
- **Review and apply** proposals with interactive diff review
|
||||
- Cohort-based aggregation groups runs by prompt generation (SHA fingerprint of prompt files)
|
||||
- Supports both Claude Code and GitHub Copilot CLI as AI backends
|
||||
- Standalone TypeScript package with tsup build (ESM), installed via `npm link`
|
||||
|
||||
### 🔧 Changed
|
||||
|
||||
- **plan2code-4--finalize.md** — Added "Metrics Capture (Contributors)" note in Step 6 directing contributors to run `plan2code-metrics` after finalization
|
||||
- **plan2code-loop index.ts** — Added dim hint "run plan2code-metrics" after session summary
|
||||
|
||||
## v1.7.0
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **4 new platform targets** — Gemini CLI, Crush, Amp, and OpenCode now supported
|
||||
- Gemini CLI installs as TOML commands (`.gemini/commands/plan2code-*.toml`)
|
||||
- Crush installs as skill subdirs (`~/.config/crush/skills/` on Unix, `%LOCALAPPDATA%\crush\skills\` on Windows)
|
||||
- Amp and OpenCode covered via shared Agent Skills target (`.agents/skills/`)
|
||||
- **Claude Code Skills format** — Migrated from flat `.claude/commands/*.md` to `.claude/skills/<skill-name>/SKILL.md` with `disable-model-invocation: true` frontmatter
|
||||
- **Agent Skills cross-tool target** — Single `.agents/skills/` install covers Amp, Gemini CLI, and OpenCode simultaneously
|
||||
- **Legacy cleanup** — Old `.claude/commands/plan2code-*.md` files automatically removed on install and uninstall
|
||||
- **TOML generation** — New `generateTomlContent()` produces Gemini CLI command files using TOML literal multi-line strings
|
||||
|
||||
### 🔧 Changed
|
||||
|
||||
- `AGENTS.md` Platform-Specific File Formats table expanded to 5 columns with 4 new platform rows
|
||||
- `docs/index.html` hero section updated with 4 new platform pills
|
||||
- `README.md` Supported Platforms list updated with 4 new platforms
|
||||
|
||||
## v1.6.2
|
||||
|
||||
### 🔧 Changed
|
||||
|
||||
- **Installer menu simplified** — Replaced the 7-platform picker with a clean 4-option menu (I/U/C/Q)
|
||||
- `I` — Install Plan2Code for all platforms + loop CLI
|
||||
- `U` — Uninstall (with confirmation prompt)
|
||||
- `C` — CUSTOM sub-menu: `L` (local install instructions), `O` (loop CLI only), `Q` (back)
|
||||
- `Q` — Quit
|
||||
- Any CLI arguments (e.g. `--dry-run`, `--platform`) are now silently ignored; installer always runs interactively
|
||||
- **README installation section** — npx install method promoted to primary recommended install path; updated menu example
|
||||
|
||||
### 🗑️ Removed
|
||||
|
||||
- CLI flags `--dry-run`, `--platform`, `--local`, `--uninstall`, `--help`, `--loop`, `--uninstall-loop` (all removed; installer is always interactive)
|
||||
- `displayHelp()` function removed from `install.js`
|
||||
|
||||
## v1.6.1
|
||||
|
||||
### ✨ Added
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# CLAUDE.md
|
||||
|
||||
**CRITICAL — MANDATORY FIRST STEP: You MUST read [AGENTS.md](./AGENTS.md) before responding to ANY user message, including simple questions. Do NOT skip this step regardless of how trivial the request appears. No exceptions.**
|
||||
|
||||
See AGENTS.md for complete project documentation including:
|
||||
- Development commands and setup
|
||||
- Architecture overview
|
||||
- Workflow prompt reference
|
||||
- Plan2Code Loop CLI details
|
||||
- Code style and gotchas
|
||||
- Keeping this file current / Failure log
|
||||
- Section details in .agents-docs/
|
||||
|
||||
This file exists for Claude Code auto-loading. All AI coding agents should reference AGENTS.md.
|
||||
@@ -1,23 +1,30 @@
|
||||
# Plam2Code Quick Reference
|
||||
# Plan2Code Quick Reference
|
||||
|
||||
## Commands
|
||||
|
||||
| Step | Command | Input | Output |
|
||||
| ------ | ----------------------------- | --------------- | ----------------------------------- |
|
||||
| Init | /plan2code---init | None | AGENTS.md file |
|
||||
| Update | /plan2code---init-update | AGENTS.md | Updated AGENTS.md |
|
||||
| 0 | /plan2code---quick-task | Requirements | Conversational plan |
|
||||
| 1 | /plan2code-1--plan | Requirements | PLAN-CONVERSATION-<date>.md + PLAN-DRAFT-<date>.md |
|
||||
| 1b | /plan2code-1b--revise-plan | Specs + changes | Updated specs |
|
||||
| 2 | /plan2code-2--document | PLAN-DRAFT.md | overview.md + Phase files |
|
||||
| 3 | /plan2code-3--implement | overview.md | Implemented code |
|
||||
| 4 | /plan2code-4--finalize | overview.md | Archived specs |
|
||||
| ------- | ----------------------------- | --------------- | -------------------------------------------------- |
|
||||
| Init | /plan2code-init | None | AGENTS.md file |
|
||||
| Update | /plan2code-init-update | AGENTS.md | Updated AGENTS.md |
|
||||
| 0 | /plan2code-0-pathfinder | A foggy idea | pathfinder/map.md *or* GitHub Issues + PLAN-DRAFT-<date>.md |
|
||||
| quick | /plan2code-quick-task | Requirements | Conversational plan (standalone — not a pipeline step) |
|
||||
| review | /plan2code-review | Scope guidance | Review findings + fixes |
|
||||
| 1 | /plan2code-1-plan | Requirements | PLAN-CONVERSATION-<date>.md + PLAN-DRAFT-<date>.md |
|
||||
| 1b | /plan2code-1b-revise-plan | Specs + changes | Updated specs |
|
||||
| 2 | /plan2code-2-document | PLAN-DRAFT.md | overview.md + Phase files |
|
||||
| 3 | /plan2code-3-implement | overview.md | Implemented code |
|
||||
| 4 | /plan2code-4-finalize | overview.md | Archived specs |
|
||||
| handoff | /plan2code-handoff | Conversation | Self-contained handoff doc in handoffs/ |
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
specs/
|
||||
└── <feature-name>/
|
||||
├── pathfinder/ # From Step 0 (optional, if charted locally)
|
||||
│ ├── map.md # the map: destination, decisions, fog
|
||||
│ └── questions/NN-<slug>.md # one decision question per file
|
||||
│ # (GitHub Issues backend: map issue + sub-issues instead)
|
||||
├── PLAN-DRAFT-<date>.md # From Step 1 (verified plan)
|
||||
├── PLAN-CONVERSATION-<date>.md # From Step 1 (conversation log)
|
||||
├── overview.md # From Step 2
|
||||
@@ -51,7 +58,7 @@ When phases have no file conflicts or dependencies, they can run simultaneously:
|
||||
|
||||
1. Documentation Mode auto-detects parallel-eligible phases
|
||||
2. Implementation Mode shows selection UI with status for each phase
|
||||
3. Run multiple `/plan2code-3--implement` instances on different phases
|
||||
3. Run multiple `/plan2code-3-implement` instances on different phases
|
||||
4. `[/]` status shows which phases are actively being worked on
|
||||
|
||||
## Quick Troubleshooting
|
||||
@@ -60,31 +67,35 @@ When phases have no file conflicts or dependencies, they can run simultaneously:
|
||||
| ---------------------- | ------------------------------------------------- |
|
||||
| Lost context mid-phase | Attach spec files, say "resume from Task X.Y" |
|
||||
| Wrong phase started | Say "abort", start correct phase |
|
||||
| Need to change plan | Use `/plan2code-1b--revise-plan` |
|
||||
| Need to change plan | Use `/plan2code-1b-revise-plan` |
|
||||
| Multiple spec folders | Specify which: "Continue with specs/user-auth/" |
|
||||
| Need AGENTS.md file | Use `/plan2code---init` to generate one |
|
||||
| Update AGENTS.md | Use `/plan2code---init-update` after sessions |
|
||||
| Need AGENTS.md file | Use `/plan2code-init` to generate one |
|
||||
| Update AGENTS.md | Use `/plan2code-init-update` after sessions |
|
||||
| Run phases in parallel | Check Parallel Execution Groups in overview.md |
|
||||
|
||||
## Workflow Decision
|
||||
|
||||
```
|
||||
New to a project?
|
||||
└── /plan2code---init → Generate AGENTS.md for project-specific guidance
|
||||
└── /plan2code-init → Generate AGENTS.md for project-specific guidance
|
||||
|
||||
Learned something during a session?
|
||||
└── /plan2code---init-update → Add learnings to AGENTS.md
|
||||
└── /plan2code-init-update → Add learnings to AGENTS.md
|
||||
|
||||
Too unclaer to plan? (big idea, don't yet know what the questions are)
|
||||
└── /plan2code-0-pathfinder → chart it, clear one decision per session
|
||||
└── then → /plan2code-1-plan (resumes at Phase 4)
|
||||
|
||||
Is it a quick, small task?
|
||||
├── Yes → /plan2code---quick-task (standalone)
|
||||
└── No → /plan2code-1--plan (full workflow)
|
||||
├── /plan2code-2--document
|
||||
├── /plan2code-3--implement (repeat per phase)
|
||||
├── Yes → /plan2code-quick-task (standalone)
|
||||
└── No → /plan2code-1-plan (full workflow)
|
||||
├── /plan2code-2-document
|
||||
├── /plan2code-3-implement (repeat per phase)
|
||||
│ └── OR: plan2code-loop (autonomous alternative)
|
||||
└── /plan2code-4--finalize
|
||||
└── /plan2code-4-finalize
|
||||
|
||||
Need to revise mid-implementation?
|
||||
└── /plan2code-1b--revise-plan
|
||||
└── /plan2code-1b-revise-plan
|
||||
```
|
||||
|
||||
## Autonomous Loop (Alternative)
|
||||
@@ -93,7 +104,7 @@ The `plan2code-loop` CLI is an **alternative** to Step 3, not a replacement.
|
||||
|
||||
| Approach | Use When |
|
||||
|----------|----------|
|
||||
| `/plan2code-3--implement` | You want interactive control per phase |
|
||||
| `/plan2code-3-implement` | You want interactive control per phase |
|
||||
| `plan2code-loop` | You want hands-off autonomous execution |
|
||||
|
||||
```bash
|
||||
|
||||
@@ -1,505 +1,315 @@
|
||||
# Plan2Code: AI-Assisted Software Development Workflow
|
||||
|
||||
A structured 4-step workflow for developing features and projects with AI assistance. This methodology emphasizes thorough planning before implementation, ensuring well-documented, maintainable code.
|
||||
# Plan2Code
|
||||
|
||||
<img src="docs/plan2code.jpg" alt="Plan2Code Workflow" height="400">
|
||||
<img src="docs/banner.png" alt="banner" style="max-width:1024px;">
|
||||
|
||||
## Overview
|
||||
**A spec-driven workflow for AI coding agents. Send the plan — the build follows.**
|
||||
|
||||
```
|
||||
🤔 📝 ⚡ 🧹
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ Step 1 │ │ Step 2 │ │ Step 3 │ │ Step 4 │
|
||||
│ PLAN │ --> │ DOCUMENT │ --> │ IMPLEMENT │ --> │ FINALIZE │
|
||||
└─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘
|
||||
New Chat New Chat New Chat (per phase) New Chat
|
||||
```
|
||||
An AI agent is a fine builder and a terrible client. Plan2Code stops making it both: you approve a
|
||||
plan, the plan becomes a set of phase documents in your repo, and the agent builds to those documents
|
||||
one phase at a time. Progress lives in files instead of chat history — so the next session, the next
|
||||
agent, and the next engineer all start from the same specs.
|
||||
|
||||
| Command | Use When |
|
||||
|---------|----------|
|
||||
| `/plan2code---init` | Generate AGENTS.md file for new/existing projects |
|
||||
| `/plan2code---init-update` | Update AGENTS.md with new learnings from coding sessions |
|
||||
| `/plan2code---quick-task` | Small, quick tasks that don't need full workflow |
|
||||
| `/plan2code-1--plan` | Starting a new feature (full planning) |
|
||||
| `/plan2code-1b--revise-plan` | Requirements change mid-implementation |
|
||||
| `/plan2code-2--document` | After planning, create implementation specs |
|
||||
| `/plan2code-3--implement` | Execute implementation (one phase per conversation) |
|
||||
| `/plan2code-4--finalize` | All phases complete, ready to archive |
|
||||
Six commands, each posted separately. Two of them are optional.
|
||||
|
||||
**Key Rules:**
|
||||
- Start NEW conversation for each step (and each implementation phase)
|
||||
- ONE phase per conversation
|
||||
- Reply "approved" to complete phases
|
||||
- 90% confidence required before planning completes
|
||||
|
||||
See [QUICK-REFERENCE.md](QUICK-REFERENCE.md) for full reference card.
|
||||
Version 2.2.0 · MIT · 📖 [plan2code.jparkerweb.com](https://plan2code.jparkerweb.com)
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
## Install
|
||||
|
||||
Plan2Code includes an interactive installer that generates and installs workflow files for all major AI coding assistants.
|
||||
Requires [Node.js](https://nodejs.org/) 18 or later and network access — installation runs through
|
||||
the [skills CLI](https://skills.sh). Re-run any time to update.
|
||||
|
||||
<img src="docs/install-script.jpg" width="582">
|
||||
|
||||
### Prerequisites
|
||||
|
||||
The install script requires **Node.js** (v14 or later). If you don't have Node.js installed:
|
||||
|
||||
1. Download from [nodejs.org](https://nodejs.org/)
|
||||
2. Or use a package manager:
|
||||
- **macOS:** `brew install node`
|
||||
- **Windows:** `winget install OpenJS.NodeJS` or `choco install nodejs`
|
||||
- **Linux:** `sudo apt install nodejs` (Debian/Ubuntu) or `sudo dnf install nodejs` (Fedora)
|
||||
|
||||
### Supported Platforms
|
||||
|
||||
- Claude Code
|
||||
- Cursor
|
||||
- Windsurf
|
||||
- Continue
|
||||
- Codeium (IntelliJ)
|
||||
- GitHub Copilot CLI
|
||||
- VS Code GitHub Copilot
|
||||
|
||||
### Quick Start
|
||||
|
||||
#### No Clone Required
|
||||
|
||||
Run the interactive installer directly using `npx` with your preferred GitHub authentication method:
|
||||
|
||||
**If you use SSH keys:**
|
||||
```bash
|
||||
npx git+ssh://git@github.com/jparkerweb/plan2code.git
|
||||
npx --allow-git=all git+https://github.com/jparkerweb/plan2code.git
|
||||
```
|
||||
|
||||
**If you use HTTPS authentication:**
|
||||
```bash
|
||||
npx git+https://github.com/jparkerweb/plan2code.git
|
||||
This fetches the installer to a temp directory, builds the workflow as Agent Skills, delegates
|
||||
installation to `skills add`, and cleans up after itself. The installed skills work independently
|
||||
from then on.
|
||||
|
||||
Either route lands you on the same menu:
|
||||
|
||||
```
|
||||
╔═════════════════════════════════════════════════════════╗
|
||||
║ INSTALL PLAN2CODE ║
|
||||
╠═════════════════════════════════════════════════════════╣
|
||||
║ I. INSTALL Install Plan2Code skills everywhere ║
|
||||
║ A. ALL Install Plan2Code + dev tools ║
|
||||
║ U. UNINSTALL Remove Plan2Code skills and dev tools ║
|
||||
║ C. CUSTOM Advanced options ║
|
||||
║ Q. QUIT Exit ║
|
||||
╚═════════════════════════════════════════════════════════╝
|
||||
```
|
||||
|
||||
This downloads the installer to a temporary location, runs it, installs the workflow files to your machine, and cleans up automatically. The installed workflows remain on your system and work independently. To update or reinstall, simply run the command again.
|
||||
**Supported tools:** every agent supported by the skills CLI, including Claude Code · Cursor ·
|
||||
GitHub Copilot · Windsurf · Codex · Continue · Codeium · Zed · Amp · OpenCode · Devin · Crush · Pi ·
|
||||
Gemini CLI · Cline · Roo · Kilo · Goose · Trae · Qwen Code.
|
||||
|
||||
#### Standard Installation (Clone Method)
|
||||
The installer keeps one canonical copy of each skill under `~/.agents/skills/` and links it into
|
||||
agents that maintain their own directory. Update later with `npx skills update -g`.
|
||||
|
||||
Use the installer rather than calling `skills add` against the repository root: recursive discovery
|
||||
would also find maintainer-only skills under `.claude/skills/`. The installer targets `skills/`
|
||||
explicitly.
|
||||
|
||||
<details>
|
||||
<summary>Prefer to clone?</summary>
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/plan/plan2code.git
|
||||
git clone https://github.com/jparkerweb/plan2code.git
|
||||
cd plan2code
|
||||
|
||||
# Run the interactive installer
|
||||
node install.js
|
||||
|
||||
# OPTIONAL: Install dev dependencies (ONLY if you plan to modify/contribute to Plan2Code)
|
||||
npm install
|
||||
# Only if you plan to modify or contribute to Plan2Code itself
|
||||
npm install && npx husky
|
||||
```
|
||||
|
||||
The installer will display an interactive menu:
|
||||
|
||||
```
|
||||
Available platforms:
|
||||
|
||||
1. Claude Code (~/.claude/commands/)
|
||||
2. Copilot CLI (~/.copilot/agents/)
|
||||
3. Cursor (~/.cursor/commands/)
|
||||
4. Continue (~/.continue/prompts/)
|
||||
5. Windsurf (~/.codeium/windsurf/global_workflows/)
|
||||
6. Codeium (IJ) (~/.codeium/global_workflows/)
|
||||
7. VS Code Copilot (%APPDATA%\Code\User\prompts\)
|
||||
|
||||
A. Install ALL platforms + loop CLI
|
||||
O. Build/link plan2code-loop CLI only
|
||||
L. Show local (project) install instructions
|
||||
U. Uninstall Plan2Code files + unlink loop CLI
|
||||
Q. Quit
|
||||
|
||||
Enter choice (1-7, A, O, L, U, Q, or comma-separated like 1,3,5):
|
||||
```
|
||||
|
||||
For per-project installation or additional options, run `node install.js --help`.
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Important: Start Fresh Conversations
|
||||
## The workflow
|
||||
|
||||
**Start a new conversation/chat session before each step.** This includes:
|
||||
```
|
||||
┌╴╴╴╴╴╴╴╴╴╴╴╴┐
|
||||
╎0 PATHFINDER╎ optional · new in 2.0 · for an idea too big or unclear to plan
|
||||
└╴╴╴╴╴╴┬╴╴╴╴╴┘
|
||||
▼
|
||||
┌────────────┐ ┌────────────┐ ┌────────────┐ ┌╴╴╴╴╴╴╴╴╴╴╴╴┐ ┌────────────┐
|
||||
│ 1 PLAN │─>│ 2 DOCUMENT │─>│3 IMPLEMENT │─>╎ REVIEW ╎─>│ 4 FINALIZE │
|
||||
│decide what │ │ draw it as │ │build to the│ ╎ optional ╎ │verify, sum,│
|
||||
│ to build │ │phase specs │ │ drawing │ ╎ any time ╎ │ archive │
|
||||
└────────────┘ └────────────┘ └────────────┘ └╴╴╴╴╴╴╴╴╴╴╴╴┘ └────────────┘
|
||||
new chat new chat new chat/phase new chat new chat
|
||||
├◀─────────────── one feature, start to archive ──────────────────▶┤
|
||||
```
|
||||
|
||||
- Step 1: New conversation
|
||||
- Step 2: New conversation
|
||||
- Step 3: New conversation **for each phase** (Phase 1, Phase 2, etc.)
|
||||
- Step 4: New conversation
|
||||
Every box is its own conversation. That is not a style preference — planning context leaking into
|
||||
implementation is where most agent drift starts.
|
||||
|
||||
Fresh conversations prevent context pollution and ensure the AI focuses on the current task with the relevant specifications.
|
||||
| Command | Use it when |
|
||||
|---------|-------------|
|
||||
| `/plan2code-0-pathfinder` | The idea is too big and unclear to plan. Charts it as decisions, clears one per session, hands a hot plan draft to Step 1 |
|
||||
| `/plan2code-1-plan` | Starting a feature. Full requirements → architecture pass |
|
||||
| `/plan2code-2-document` | Planning is done. Turn the plan into phase specs |
|
||||
| `/plan2code-3-implement` | Build the next phase (one per conversation) |
|
||||
| `/plan2code-review` | Independent second opinion on local changes, then optional fixes |
|
||||
| `/plan2code-4-finalize` | All phases done. Validate, summarize, archive |
|
||||
| `/plan2code-init` | Generate this repo's `AGENTS.md` so every agent starts informed |
|
||||
| `/plan2code-init-update` | Fold what you learned this session back into `AGENTS.md` |
|
||||
| `/plan2code-quick-task` | A small change that doesn't warrant the full sequence |
|
||||
| `/plan2code-1b-revise-plan` | Requirements moved mid-build. Revise the specs, not the code |
|
||||
| `/plan2code-handoff` | Compact this conversation into a doc the next one resumes from |
|
||||
|
||||
---
|
||||
|
||||
## The Workflow Steps
|
||||
## The four rules that do most of the work
|
||||
|
||||
### Step 1: Planning Mode 🤔
|
||||
**1 · A fresh conversation for each step, and each implementation phase.**
|
||||
Step 3 gets a new chat per phase, not one chat for all of them.
|
||||
|
||||
**Purpose:** Thoroughly analyze requirements and design the solution architecture before writing any code.
|
||||
**2 · No code until the plan hits 90% confidence.**
|
||||
Step 1 will not finalize below the threshold. Under it, the agent keeps asking and keeps reading your
|
||||
code — and writes every assumption down where you can argue with it.
|
||||
|
||||
**AI Role:** Senior software architect and technical product manager
|
||||
**3 · Checkboxes are the state, not the chat.**
|
||||
Progress lives in the spec files. Any agent, any session, resumes cold from them.
|
||||
|
||||
**Phases (completed one at a time):**
|
||||
|
||||
1. **Requirements Analysis** - Extract functional/non-functional requirements, identify ambiguities
|
||||
2. **System Context Examination** - Review existing codebase, identify integration points
|
||||
3. **Tech Stack** - Recommend and confirm all technologies (requires user sign-off)
|
||||
4. **Architecture Design** - Propose patterns, define components, design interfaces/schemas
|
||||
5. **Technical Specification** - Break down implementation phases, identify risks
|
||||
6. **Transition Decision** - Finalize plan when confidence reaches 90%+
|
||||
|
||||
**Output:** `specs/<feature-name>/PLAN-DRAFT-<date>.md` and `specs/<feature-name>/PLAN-CONVERSATION-<date>.md` (date format: YYYYMMDD)
|
||||
|
||||
**Key Behaviors:**
|
||||
|
||||
- AI stops after each phase for clarification
|
||||
- Must reach 90% confidence before finalizing
|
||||
- All assumptions are documented
|
||||
- User must approve tech stack decisions
|
||||
**4 · Reply `approved` to close a phase.**
|
||||
Nothing advances on a guess about what you meant.
|
||||
|
||||
---
|
||||
|
||||
### Step 2: Documentation Mode 📝
|
||||
|
||||
**Purpose:** Transform the planning output into structured, actionable implementation documents.
|
||||
|
||||
**Required Context:** Attach or reference the `specs/<feature-name>/PLAN-DRAFT-<date>.md` from Step 1 (or provide the planning conversation).
|
||||
|
||||
**Output Structure:**
|
||||
|
||||
```
|
||||
specs/
|
||||
└── <feature-name>/
|
||||
├── overview.md # High-level overview with phase checkboxes and parallel groups
|
||||
├── Phase 1.md # Detailed tasks for Phase 1
|
||||
├── Phase 2.md # Detailed tasks for Phase 2
|
||||
└── Phase N.md # ...additional phases
|
||||
```
|
||||
|
||||
The `overview.md` includes a "Parallel Execution Groups" section that identifies which phases can be run simultaneously in separate agent instances.
|
||||
|
||||
**Document Format:**
|
||||
|
||||
- Each phase file contains detailed one-story-point tasks
|
||||
- All tasks have checkboxes `[ ]` for progress tracking
|
||||
- Each phase is self-contained (developer needs no prior context)
|
||||
- Unit/E2E testing excluded unless explicitly requested
|
||||
|
||||
---
|
||||
|
||||
### Step 3: Implementation Mode ⚡
|
||||
|
||||
**Purpose:** Execute the implementation following the documented specifications.
|
||||
|
||||
**AI Role:** Senior software engineer
|
||||
|
||||
**Required Context:** Provide the path to `specs/<feature-name>/overview.md`. The command will auto-detect the next uncompleted phase and read the corresponding `Phase X.md` file automatically.
|
||||
|
||||
**Workflow:**
|
||||
|
||||
1. Identify the next uncompleted phase (unchecked in `overview.md`)
|
||||
2. Check for parallel execution options (if phases can run simultaneously)
|
||||
3. Implement ALL tasks in that phase exactly as specified
|
||||
4. Update `Phase X.md` checkboxes as tasks complete `[x]`
|
||||
5. Update `overview.md` phase checkbox when phase completes
|
||||
6. Perform code review to ensure nothing was missed
|
||||
7. Add completion summary to the phase document
|
||||
|
||||
**Parallel Execution:** If the next phase is part of a parallel-eligible group, you'll be prompted to choose which phase to implement. This allows running multiple agent instances simultaneously on different phases that don't conflict with each other.
|
||||
|
||||
**Key Rules:**
|
||||
|
||||
- **Start a new conversation for EACH phase**
|
||||
- Work on ONE phase per conversation (unless told otherwise)
|
||||
- Follow specifications EXACTLY as documented
|
||||
- Keep checkboxes updated (enables progress tracking across sessions)
|
||||
- Do NOT run tests unless specified in phase tasks
|
||||
|
||||
---
|
||||
|
||||
### Step 4: Finalization Mode 🧹
|
||||
|
||||
**Purpose:** Validate implementation, create summaries, and archive documentation.
|
||||
|
||||
**Required Context:** Attach or reference the `specs/<feature-name>/` directory contents.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. **Validation** - Verify all tasks implemented correctly, check for issues
|
||||
2. **Summary** - Document what was built and list all modified/created files
|
||||
3. **Documentation Review** - Identify any needed README/CHANGELOG updates
|
||||
4. **Spec Cleanup** - Move completed specs to `specs--completed/<implementation-name>/`
|
||||
5. **Final Confirmation** - Confirm completion
|
||||
|
||||
---
|
||||
|
||||
## How to Use
|
||||
|
||||
After running `node install.js`, use the slash commands directly in your AI tool:
|
||||
|
||||
```
|
||||
/plan2code-1--plan # Start planning a new feature
|
||||
/plan2code-2--document # Create implementation docs from plan
|
||||
/plan2code-3--implement # Begin/continue implementation
|
||||
/plan2code-4--finalize # Wrap up after all phases complete
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Complete Workflow Example
|
||||
|
||||
### Starting a New Project
|
||||
|
||||
**Session 1 - Planning (New Chat):**
|
||||
|
||||
```
|
||||
User: [Paste or invoke Step 1 prompt]
|
||||
I want to build a REST API for a task management application.
|
||||
|
||||
AI: 🤔 [REQUIREMENTS ANALYSIS]
|
||||
... asks clarifying questions, works through phases ...
|
||||
|
||||
AI: 🤔 [TRANSITION DECISION]
|
||||
Confidence: 92%. Creating specs/task-api/PLAN-DRAFT-20250204.md...
|
||||
```
|
||||
|
||||
**Session 2 - Documentation (New Chat):**
|
||||
|
||||
```
|
||||
User: [Paste or invoke Step 2 prompt]
|
||||
[Attach: specs/task-api/PLAN-DRAFT-20250204.md]
|
||||
|
||||
AI: 📝 [DOCUMENTATION]
|
||||
Creating specs/task-api/overview.md...
|
||||
Creating specs/task-api/Phase 1.md...
|
||||
Creating specs/task-api/Phase 2.md...
|
||||
...
|
||||
```
|
||||
|
||||
**Session 3 - Implementation Phase 1 (New Chat):**
|
||||
|
||||
```
|
||||
User: [Paste or invoke Step 3 prompt]
|
||||
[Provide: specs/task-api/overview.md]
|
||||
|
||||
AI: ⚡ [PHASE 1: Project Setup]
|
||||
(Auto-detected Phase 1 as next uncompleted phase)
|
||||
Implementing tasks...
|
||||
✓ Phase 1 complete. Updated checkboxes in Phase 1.md and overview.md.
|
||||
```
|
||||
|
||||
**Session 4 - Implementation Phase 2 (New Chat):**
|
||||
|
||||
```
|
||||
User: [Paste or invoke Step 3 prompt]
|
||||
[Provide: specs/task-api/overview.md]
|
||||
|
||||
AI: ⚡ [PHASE 2: Database Models]
|
||||
(Auto-detected Phase 2 as next uncompleted phase)
|
||||
Implementing tasks...
|
||||
✓ Phase 2 complete. Updated checkboxes in Phase 2.md and overview.md.
|
||||
```
|
||||
|
||||
**Sessions 5-N - Continue Implementation (New Chat for each phase):**
|
||||
|
||||
```
|
||||
... repeat for each remaining phase ...
|
||||
```
|
||||
|
||||
**Final Session - Finalization (New Chat):**
|
||||
|
||||
```
|
||||
User: [Paste or invoke Step 4 prompt]
|
||||
[Provide: specs/task-api/overview.md]
|
||||
|
||||
AI: 🧹 [VALIDATION]
|
||||
Verifying implementation...
|
||||
|
||||
AI: 🧹 [SPEC CLEANUP]
|
||||
Moving to specs--completed/task-api/
|
||||
|
||||
Implementation complete!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Progress Tracking
|
||||
|
||||
The checkbox system enables seamless progress tracking across multiple sessions:
|
||||
|
||||
**Phase Status (overview.md):**
|
||||
|
||||
| Checkbox | Status | Meaning |
|
||||
|----------|--------|---------|
|
||||
| `[ ]` | Pending | Not yet started |
|
||||
| `[/]` | In Progress | Agent actively working (or paused/aborted) |
|
||||
| `[x]` | Complete | Finished and approved |
|
||||
|
||||
```markdown
|
||||
## Phases
|
||||
|
||||
- [x] Phase 1: Project Setup
|
||||
- [x] Phase 2: Database Models
|
||||
- [/] Phase 3: API Endpoints <- In progress (agent working)
|
||||
- [ ] Phase 4: Authentication <- Next available
|
||||
```
|
||||
|
||||
**Task Status (phase-X.md):**
|
||||
|
||||
```markdown
|
||||
## Tasks
|
||||
|
||||
- [x] Create routes file
|
||||
- [x] Implement GET /tasks
|
||||
- [ ] Implement POST /tasks <- Current task
|
||||
- [ ] Implement PUT /tasks/:id
|
||||
- [ ] Implement DELETE /tasks/:id
|
||||
```
|
||||
|
||||
The `[/]` status enables parallel execution - multiple agents can work on different phases simultaneously, and you can see which phases are actively being worked on.
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Start fresh conversations** - New chat for each step and each implementation phase
|
||||
2. **Always attach specs** - The AI needs the spec files to understand the current state
|
||||
3. **Don't skip planning** - The upfront investment prevents costly rework later
|
||||
4. **Confirm tech stack** - Ensure AI gets explicit approval before architecture design
|
||||
5. **One phase at a time** - Keeps conversations focused and manageable
|
||||
6. **Update checkboxes immediately** - Maintains accurate progress state
|
||||
7. **Review phase output** - Verify each phase before moving to the next
|
||||
8. **Keep spec files** - The completed folder serves as project documentation
|
||||
|
||||
---
|
||||
|
||||
## What to Attach at Each Step
|
||||
|
||||
| Step | Required Input |
|
||||
| ------------------ | ---------------------------------------------------- |
|
||||
| Step 1 (Plan) | None (describe your feature/project) |
|
||||
| Step 2 (Document) | `specs/<feature>/PLAN-DRAFT-<date>.md` or planning conversation |
|
||||
| Step 3 (Implement) | `specs/<feature>/overview.md` (auto-detects phase) |
|
||||
| Step 4 (Finalize) | `specs/<feature>/overview.md` |
|
||||
|
||||
---
|
||||
|
||||
## Autonomous Loop (Alternative to Step 3)
|
||||
|
||||
For hands-off implementation, Plan2Code includes an optional autonomous loop CLI that iterates through your spec tasks automatically.
|
||||
|
||||
> **Note:** The loop is an **alternative** to `/plan2code-3--implement`, not a replacement. Use the manual Step 3 workflow when you want direct control over each phase, or use the loop when you prefer autonomous execution.
|
||||
|
||||
### When to Use Each
|
||||
|
||||
| Approach | Best For |
|
||||
|----------|----------|
|
||||
| `/plan2code-3--implement` | Interactive control, reviewing each phase, complex logic requiring human judgment |
|
||||
| `plan2code-loop` | Straightforward implementations, batch processing, overnight runs |
|
||||
|
||||
### Installing the Loop
|
||||
|
||||
```bash
|
||||
# From the plan2code root directory:
|
||||
|
||||
# Option 1: Install everything (recommended)
|
||||
node install.js # Select option A
|
||||
|
||||
# Option 2: Install loop only
|
||||
node install.js # Select option O
|
||||
```
|
||||
|
||||
### Using the Loop
|
||||
|
||||
```bash
|
||||
# Run the loop - fully interactive
|
||||
plan2code-loop
|
||||
```
|
||||
|
||||
The CLI will:
|
||||
1. Auto-detect specs in `./specs/` directory
|
||||
2. Let you select a spec if multiple are found
|
||||
3. Prompt to continue if an existing session is found
|
||||
4. Ask for JIRA ticket ID, agent selection, loop mode, and max iterations
|
||||
|
||||
### Loop Modes
|
||||
|
||||
| Mode | Behavior | Git Commits | Best For |
|
||||
|------|----------|-------------|----------|
|
||||
| **One task per loop** (default) | Each agent call implements one task | Node controller commits after each task | Smaller models, cautious execution |
|
||||
| **One phase per loop** | Each agent call implements all tasks in a phase | LLM commits after each task (with JIRA ID) | Smart models with larger context windows, related tasks |
|
||||
|
||||
Session state is stored per-spec in `specs/<feature>/.plan2code-loop/`, keeping each feature's progress isolated.
|
||||
|
||||
The loop will:
|
||||
1. Read your `overview.md` and phase files
|
||||
2. Find the first unchecked task (or phase, in phase mode)
|
||||
3. Implement it and mark the checkbox complete
|
||||
4. Repeat until all tasks are done or max iterations reached
|
||||
|
||||
See [plan2code-loop/](plan2code-loop/) for full documentation.
|
||||
|
||||
---
|
||||
|
||||
## File Structure After Complete Implementation
|
||||
## What lands in your repo
|
||||
|
||||
```
|
||||
your-project/
|
||||
├── specs/
|
||||
│ └── another-feature/ # In-progress feature
|
||||
│ ├── overview.md
|
||||
│ └── Phase 1.md
|
||||
│ └── task-api/ ← in progress
|
||||
│ ├── pathfinder/ ← only if you charted it in Step 0
|
||||
│ │ ├── map.md the destination, the decisions, the fog
|
||||
│ │ └── questions/NN-<slug>.md one decision per file
|
||||
│ ├── PLAN-DRAFT-20260804.md ← Step 1: the verified plan
|
||||
│ ├── PLAN-CONVERSATION-*.md ← Step 1: how you got there
|
||||
│ ├── overview.md ← Step 2: phase list + parallel groups
|
||||
│ └── Phase 1.md … Phase N.md ← Step 2: one-point tasks, self-contained
|
||||
├── specs--completed/
|
||||
│ └── feature-name/
|
||||
│ ├── overview.md # Archived with completion summary
|
||||
│ ├── Phase 1.md # All checkboxes marked [x]
|
||||
│ ├── Phase 2.md
|
||||
│ └── ...
|
||||
├── your project files...
|
||||
└── README.md
|
||||
│ └── auth-refresh/ ← Step 4 files finished work here
|
||||
└── ...your code
|
||||
```
|
||||
|
||||
`specs/` is gitignored by default — it's your working drawing, not a deliverable. Share a folder
|
||||
deliberately with `git add -f` when you want to.
|
||||
|
||||
### Progress marks
|
||||
|
||||
| Mark | Status | Meaning |
|
||||
|------|--------|---------|
|
||||
| `[ ]` | Open | Unclaimed. Any agent picks it up cold. |
|
||||
| `[/]` | In progress | Claimed right now — which is how two agents run parallel phases without colliding. |
|
||||
| `[x]` | Done | Built, self-reviewed against the spec, approved by you. |
|
||||
|
||||
```markdown
|
||||
## Phases
|
||||
|
||||
- [x] Phase 1: Project setup
|
||||
- [x] Phase 2: Data model
|
||||
- [/] Phase 3: API endpoints ← an agent is on this now
|
||||
- [ ] Phase 4: Authentication ← next available
|
||||
```
|
||||
|
||||
Step 2 marks which phases don't share files. Open a second agent on one of those, and the `[/]` marks
|
||||
keep the two out of each other's way.
|
||||
|
||||
---
|
||||
|
||||
## Customization
|
||||
## The six steps in detail
|
||||
|
||||
Feel free to modify these prompts to fit your workflow:
|
||||
Each one travels on its own — a fresh conversation, opened and closed, with the specs on disk as the
|
||||
only thing carried between them.
|
||||
|
||||
- **Add testing phases** - Uncomment/add testing requirements in Step 2
|
||||
- **Adjust confidence threshold** - Change the 90% threshold in Step 1
|
||||
- **Modify output structure** - Customize the specs folder organization
|
||||
- **Add code review steps** - Enhance Step 3 with additional review gates
|
||||
### 0 · Pathfinder 🧭 — optional, new in 2.0
|
||||
|
||||
Some ideas are too big and unclear to plan: you can feel the shape of the work but you can't write
|
||||
it as requirements, so planning would just invent the answers. Pathfinder finds the *way* to the
|
||||
destination; Step 1 then walks it.
|
||||
|
||||
1. **Name the destination** — one or two lines fixing what this effort is finding its way to. Settled
|
||||
first, because it fixes scope. It also asks where the map should live: **local files** under
|
||||
gitignored `specs/` (private, solo — the default), or **GitHub Issues** (a map issue with one
|
||||
sub-issue per decision, native blocking, so your team can see and work the frontier in the tracker).
|
||||
2. **Chart the map** — a breadth-first grilling surfaces the open decisions. Anything you can phrase
|
||||
*sharply* becomes a question file; anything you can only sense stays listed as fog.
|
||||
3. **Clear one question per session** — resolving a question burns off the fog behind it, graduating
|
||||
whatever just became sharp into new questions.
|
||||
4. **Hand off** — when nothing is left to decide, it writes a `PLAN-DRAFT` that
|
||||
`/plan2code-1-plan` resumes from at Phase 4, with requirements, context, and scope already
|
||||
answered.
|
||||
|
||||
**Question types:** `grill` (a decision only you can make — the default) · `research` (a fact gates
|
||||
it; background agents resolve these, several in parallel) · `sketch` (you need something concrete to
|
||||
react to) · `legwork` (manual work that has to happen before a decision is possible).
|
||||
|
||||
It never answers its own questions, and it **plans, it never builds.** When the urge to just build it
|
||||
arrives, the map is done. Skip Step 0 entirely when you already know what you're building.
|
||||
|
||||
**Out:** `specs/<feature>/pathfinder/map.md` + `questions/` (or a `pathfinder:map` issue and its
|
||||
sub-issues) → `PLAN-DRAFT-<date>.md`. The draft is always a local file — that is what Step 1 reads.
|
||||
|
||||
### 1 · Plan 🤔
|
||||
|
||||
The agent works as a senior architect through six phases, stopping for you after each: requirements
|
||||
analysis · system context (reading your actual codebase) · tech stack (needs your explicit sign-off) ·
|
||||
architecture design · technical specification · transition decision.
|
||||
|
||||
It won't finalize below **90% confidence**, and every assumption it makes is written into the draft.
|
||||
|
||||
**In:** a description of the feature. **Out:** `PLAN-DRAFT-<date>.md` + `PLAN-CONVERSATION-<date>.md`
|
||||
|
||||
### 2 · Document 📝
|
||||
|
||||
The plan becomes the drawing. One `overview.md` with the phase checklist, plus one file per phase of
|
||||
one-story-point tasks. Each phase is **self-contained** — an agent opening `Phase 3.md` cold needs
|
||||
nothing else to build it. Unit and E2E tests are excluded unless you ask for them.
|
||||
|
||||
The overview also identifies the **parallel execution groups**: phases with no shared files or
|
||||
dependencies, safe to run in separate agents at once.
|
||||
|
||||
**In:** the `PLAN-DRAFT`. **Out:** `overview.md` + `Phase 1…N.md`
|
||||
|
||||
### 3 · Implement ⚡
|
||||
|
||||
Point it at `overview.md` and it does the rest: finds the next unchecked phase, implements every task
|
||||
exactly as specified, ticks tasks off as they land, then reviews its own work against the spec and
|
||||
writes a completion summary.
|
||||
|
||||
One phase per conversation. It won't run tests unless the phase says to.
|
||||
|
||||
**In:** `specs/<feature>/overview.md`. **Out:** working code, and updated checkboxes.
|
||||
|
||||
### Review 🔬 — optional, any time
|
||||
|
||||
An independent second opinion, not a rubber stamp. It figures out its own scope (conversation
|
||||
context, your instruction, or the git diff as a fallback), analyses across 11 dimensions, and ranks
|
||||
findings Critical / Warning / Suggestion. Every finding cites a file and a line, or it gets dropped —
|
||||
and the review pass is read-only. It fixes things only if you ask, and verifies each fix afterwards.
|
||||
|
||||
Spec-aware when `specs/` exists, and works fine without it. Most useful right after a planning or
|
||||
implementation step, but there's no wrong time to run it.
|
||||
|
||||
### 4 · Finalize 🧹
|
||||
|
||||
Validates every task against its phase spec, writes the summary and the list of files touched, flags
|
||||
the docs that drifted (`README`, `CHANGELOG`, `AGENTS.md`), then archives the whole spec folder —
|
||||
`pathfinder/` included — to `specs--completed/`. That folder is the record of *why* the code looks
|
||||
like this.
|
||||
|
||||
**In:** `specs/<feature>/overview.md`. **Out:** archived specs.
|
||||
|
||||
---
|
||||
|
||||
## What to bring to each step
|
||||
|
||||
| Step | Required input |
|
||||
|------|----------------|
|
||||
| 0 · Pathfinder | Nothing to start — just describe the idea. To continue: the feature name; it finds its own map |
|
||||
| 1 · Plan | Nothing — describe the feature |
|
||||
| 2 · Document | `specs/<feature>/PLAN-DRAFT-<date>.md`, or the planning conversation |
|
||||
| 3 · Implement | `specs/<feature>/overview.md` — it detects the phase itself |
|
||||
| Review | Scope guidance, e.g. "the last two phases", "just the auth module", "the whole PR". Auto-detects if you give none |
|
||||
| 4 · Finalize | `specs/<feature>/overview.md` |
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Slash commands/workflows not recognized:**
|
||||
**The skills aren't recognised.** Re-run the installer and restart your AI tool. Confirm the global
|
||||
install with `npx skills list -g`. For a project install, check the generated directories aren't
|
||||
gitignored.
|
||||
|
||||
- Ensure you ran `node install.js` and selected the appropriate platform
|
||||
- Restart your AI tool after installation
|
||||
- For per-project installation, ensure the directory isn't in `.gitignore`
|
||||
**Your tool doesn't read Agent Skills.** Since v2.2.0, Plan2Code ships only as skills. If your tool
|
||||
has no skill support, paste the relevant `src/plan2code-*.md` manually or point it at the installed
|
||||
copy under `~/.agents/skills/`.
|
||||
|
||||
**AI jumps ahead to implementation during planning:**
|
||||
**The agent starts coding during planning.** The prompts forbid it, but models drift. Say: "Stay in
|
||||
planning mode. Do not write code yet."
|
||||
|
||||
- The prompts explicitly forbid this, but if it happens, remind the AI: "Stay in planning mode. Do not write code yet."
|
||||
**The agent doesn't know what to implement.** Give it the path to `overview.md` — it reads the phase
|
||||
file itself from there.
|
||||
|
||||
**AI doesn't know what to implement:**
|
||||
**You lost track between sessions.** `overview.md` has the phase status; the phase files have the
|
||||
task status. That's the whole state.
|
||||
|
||||
- Make sure you provided the path to `overview.md`
|
||||
- The AI will auto-detect the next phase and read the corresponding `Phase X.md` file
|
||||
**The agent isn't following the spec.** Point at the specific phase document and tell it to re-read
|
||||
the requirements.
|
||||
|
||||
**Lost progress between sessions:**
|
||||
**Too many or too few phases.** Fix it in Step 2 — a phase should be a logical grouping of work, not
|
||||
a fixed size.
|
||||
|
||||
- Check `overview.md` for phase status
|
||||
- Review individual phase files for task completion status
|
||||
---
|
||||
|
||||
**AI not following spec exactly:**
|
||||
## Customizing
|
||||
|
||||
- Reference the specific phase document and ask it to re-read the requirements
|
||||
The prompts are yours to edit. Common changes: add testing requirements in Step 2, move the 90%
|
||||
confidence threshold in Step 1, restructure the `specs/` layout, or add review gates to Step 3.
|
||||
Source files live in `src/`; run `npm run build:skills`, then re-run `node install.js` to push your
|
||||
edits out through the skills CLI.
|
||||
|
||||
**Too many/few phases:**
|
||||
---
|
||||
|
||||
- Adjust during Step 2 (Documentation) - phases should represent logical groupings of work
|
||||
## Dive deeper
|
||||
|
||||
Core reference:
|
||||
|
||||
- **[QUICK-REFERENCE.md](QUICK-REFERENCE.md)** — the one-page card: commands, inputs, outputs, decision tree
|
||||
- **[.readme/walkthrough.md](.readme/walkthrough.md)** — one feature from a sentence to archived specs, session by session
|
||||
- **[AGENTS.md](AGENTS.md)** — architecture and contributor guide for this repo
|
||||
- **[CHANGELOG.md](CHANGELOG.md)** — what changed, and why
|
||||
|
||||
Optional tooling — none of it is required to use the workflow:
|
||||
|
||||
- **[.readme/autonomous-loop.md](.readme/autonomous-loop.md)** — `plan2code-loop`, a hands-off alternative to Step 3
|
||||
- **[.readme/status-line.md](.readme/status-line.md)** — three-line Claude Code status bar: model, context, quota, diff
|
||||
- **[.readme/metrics.md](.readme/metrics.md)** — `plan2code-metrics`, measuring and improving the prompts themselves
|
||||
- **[.readme/test-bot.md](.readme/test-bot.md)** — `plan2code-bot`, maintainer harness that runs the whole workflow unattended
|
||||
|
||||
|
After Width: | Height: | Size: 6.1 KiB |
|
After Width: | Height: | Size: 127 KiB |
|
Before Width: | Height: | Size: 185 KiB |
|
Before Width: | Height: | Size: 24 KiB After Width: | Height: | Size: 1.9 KiB |
|
Before Width: | Height: | Size: 2.4 KiB After Width: | Height: | Size: 584 B |
@@ -0,0 +1,33 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" width="64" height="64" role="img" aria-label="Plan2Code postage stamp">
|
||||
<defs>
|
||||
<!-- Perforated stamp silhouette: white keeps, black bites out.
|
||||
Few, deep notches so the scalloped edge still reads at 16px. -->
|
||||
<mask id="perf">
|
||||
<rect width="64" height="64" fill="#000"/>
|
||||
<rect x="2" y="2" width="60" height="60" fill="#fff"/>
|
||||
<g fill="#000">
|
||||
<circle cx="2" cy="2" r="5"/><circle cx="14" cy="2" r="5"/><circle cx="26" cy="2" r="5"/>
|
||||
<circle cx="38" cy="2" r="5"/><circle cx="50" cy="2" r="5"/><circle cx="62" cy="2" r="5"/>
|
||||
<circle cx="2" cy="62" r="5"/><circle cx="14" cy="62" r="5"/><circle cx="26" cy="62" r="5"/>
|
||||
<circle cx="38" cy="62" r="5"/><circle cx="50" cy="62" r="5"/><circle cx="62" cy="62" r="5"/>
|
||||
<circle cx="2" cy="14" r="5"/><circle cx="2" cy="26" r="5"/>
|
||||
<circle cx="2" cy="38" r="5"/><circle cx="2" cy="50" r="5"/>
|
||||
<circle cx="62" cy="14" r="5"/><circle cx="62" cy="26" r="5"/>
|
||||
<circle cx="62" cy="38" r="5"/><circle cx="62" cy="50" r="5"/>
|
||||
</g>
|
||||
</mask>
|
||||
</defs>
|
||||
|
||||
<g mask="url(#perf)">
|
||||
<!-- printed stamp -->
|
||||
<rect x="2" y="2" width="60" height="60" fill="#D33A38"/>
|
||||
<!-- the 2, set as a franking-machine numeral -->
|
||||
<g fill="#FBFAF6">
|
||||
<rect x="16" y="13" width="32" height="8"/>
|
||||
<rect x="40" y="13" width="8" height="22"/>
|
||||
<rect x="16" y="27" width="32" height="8"/>
|
||||
<rect x="16" y="27" width="8" height="24"/>
|
||||
<rect x="16" y="43" width="32" height="8"/>
|
||||
</g>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 1.6 KiB |
|
Before Width: | Height: | Size: 147 KiB |
|
Before Width: | Height: | Size: 95 KiB |
@@ -1,12 +1,14 @@
|
||||
{
|
||||
"name": "plan2code",
|
||||
"version": "1.6.1",
|
||||
"version": "2.2.0",
|
||||
"private": true,
|
||||
"bin": {
|
||||
"plan2code": "./install.js"
|
||||
},
|
||||
"scripts": {
|
||||
"prepare": "husky"
|
||||
"prepare": "husky",
|
||||
"build:skills": "node install.js --build-skills",
|
||||
"test": "node scripts/validate-char-count.js && node install.js --verify-skills"
|
||||
},
|
||||
"devDependencies": {
|
||||
"husky": "^9.0.0"
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
# Changelog
|
||||
|
||||
## 1.1.0
|
||||
|
||||
### Resume support
|
||||
- Add `--resume` flag to continue incomplete runs from saved state
|
||||
- Skip previously succeeded steps when resuming (init, plan, document, implement, finalize)
|
||||
- Restore idea name, description, project directory, and implement pass counter from state
|
||||
- Auto-detect state files in current directory (enhancement mode) or subdirectories (new-project mode)
|
||||
- Delete state file automatically after a fully successful run
|
||||
- Preserve state file on failure for later resume
|
||||
- Add `deleteState()` and `findExistingState()` utilities to bot-state module
|
||||
- Add bot-state unit tests (saveState, loadState, deleteState, findExistingState)
|
||||
|
||||
### Idea generation improvements
|
||||
- Expand idea categories from binary CLI/web-app coin flip to 12 diverse categories (games, dashboards, browser extensions, desktop utilities, etc.)
|
||||
- Add guidance to avoid defaulting to developer-centric tools (git analyzers, code formatters)
|
||||
- Strengthen `--idea` seed clause so the LLM stays aligned with the user's theme instead of ignoring it
|
||||
- Update system prompt to encourage creative, cross-domain ideas
|
||||
|
||||
### Init step overhaul (new projects)
|
||||
- Init now creates a minimal AGENTS.md stub (name, description, status) instead of running the full `/plan2code-init` skill
|
||||
- Prevents hallucinated architecture, commands, and `.agents-docs/` files before the plan step runs
|
||||
- Init evaluation criteria updated to reward minimalism and penalize premature detail
|
||||
|
||||
### Implement step overhaul
|
||||
- Implement step now works directly with Read/Write/Edit/Glob/Grep tools instead of delegating to Skill sub-session
|
||||
- Inlined step-by-step process: find specs, pick phase, implement tasks, mark checkboxes
|
||||
- Fixes issue where Skill sub-sessions did all work invisibly, causing zero tool observations
|
||||
|
||||
### Observation tracking fix
|
||||
- Capture `tool_use` blocks from the assistant message stream in session-runner as a fallback when `canUseTool` callback doesn't fire
|
||||
- Add deduplication in ObservationCollector to prevent double-counting from both sources
|
||||
- Fixes all steps reporting 0 tools used / 0 files created in BOT-NOTES and evaluations
|
||||
|
||||
### Evaluator improvements
|
||||
- Increase evaluator `maxTurns` from 3 to 30 so it has room for tool calls before producing the scored response
|
||||
- Add warning log when evaluation parser can't find SCORE in output (was silently defaulting to 50)
|
||||
|
||||
## 1.0.0
|
||||
|
||||
- Initial release
|
||||
- Two auto-detected modes: new-project and enhancement
|
||||
- `--idea` flag to seed the idea generator
|
||||
- Full workflow execution: init → plan → document → implement → finalize
|
||||
- Artifact validation after each step
|
||||
- State persistence to `.plan2code-bot-state.json`
|
||||
- Auto-responder for autonomous Claude Agent SDK sessions
|
||||
- Bot-friendly skill installation (strips `disable-model-invocation`)
|
||||
@@ -0,0 +1,234 @@
|
||||
# LLM-as-Judge Evaluation System
|
||||
|
||||
## Overview
|
||||
|
||||
The plan2code-bot now includes an **always-on LLM-as-judge evaluation system** that transforms it from a "yes-man" into an authentic QA agent. This provides realistic quality signals for `plan2code-metrics` to analyze and drive recursive self-improvement.
|
||||
|
||||
## Key Features
|
||||
|
||||
### 1. Intelligent Decision Making (Real-Time)
|
||||
|
||||
**What:** During execution, when `AskUserQuestion` is called, the bot uses an LLM to make thoughtful decisions based on current observations.
|
||||
|
||||
**How it works:**
|
||||
- Collects observations up to the current point (tools used, files created, errors)
|
||||
- Queries LLM with context: "Given what you've seen, should you approve this plan?"
|
||||
- LLM inspects current artifacts using Read/Glob/Grep
|
||||
- Returns evidence-based answer with reasoning
|
||||
- All decisions are recorded for metrics analysis
|
||||
|
||||
**Example:**
|
||||
```
|
||||
Question: "Approve this plan?"
|
||||
Observations: Created PLAN-DRAFT.md, 3 phases, 42s duration, no errors
|
||||
LLM reads PLAN-DRAFT.md, evaluates quality
|
||||
LLM decides: "Yes, approve - phases are well-scoped and realistic"
|
||||
```
|
||||
|
||||
### 2. Post-Step Evaluation
|
||||
|
||||
**What:** After each step completes, the bot evaluates quality using step-specific criteria.
|
||||
|
||||
**How it works:**
|
||||
- Collects complete execution observations
|
||||
- Queries LLM with evaluation criteria for the step
|
||||
- LLM inspects final artifacts
|
||||
- Returns structured evaluation (score, strengths, weaknesses, suggestions)
|
||||
- Writes `specs/<feature>/BOT-EVALUATION.md` and `specs/<feature>/BOT-NOTES.md` (falls back to project root if no spec folder exists yet, e.g. during `init`)
|
||||
|
||||
**Example output:**
|
||||
```markdown
|
||||
# Evaluation: plan Step
|
||||
|
||||
**Score:** 78/100
|
||||
|
||||
## Strengths
|
||||
- Clear phase breakdown with realistic scope
|
||||
- Tech stack choices appropriate
|
||||
|
||||
## Weaknesses
|
||||
- Phase 3 description too vague
|
||||
- No testing strategy mentioned
|
||||
|
||||
## Suggestions
|
||||
- Expand Phase 3 with concrete tasks
|
||||
- Add explicit testing phase
|
||||
```
|
||||
|
||||
### 3. Quality Gate
|
||||
|
||||
**What:** Before finalize, checks that average quality score is acceptable.
|
||||
|
||||
**How it works:**
|
||||
- Calculates average score across all evaluated steps
|
||||
- If average < 60, blocks finalization
|
||||
- Displays clear message about quality issues
|
||||
- User must review `specs/<feature>/BOT-EVALUATION.md` and fix problems
|
||||
|
||||
## Files Created
|
||||
|
||||
### New Files
|
||||
|
||||
1. **`src/observation-collector.ts`**
|
||||
- Tracks execution details (tools, files, messages, errors, questions)
|
||||
- Provides snapshots for real-time decisions
|
||||
- Captures complete history for evaluation
|
||||
|
||||
2. **`src/intelligent-responder.ts`**
|
||||
- Replaces hardcoded auto-responder
|
||||
- Uses LLM to answer AskUserQuestion prompts
|
||||
- Provides reasoning for all decisions
|
||||
- Falls back gracefully if LLM unavailable
|
||||
|
||||
3. **`src/prompts/evaluation-criteria.ts`**
|
||||
- Step-specific evaluation criteria (init, plan, document, implement, finalize)
|
||||
- Quality checks, common pitfalls, scoring guidance
|
||||
- Emphasizes honest scoring (most work should score 70-85)
|
||||
|
||||
4. **`src/evaluator.ts`**
|
||||
- Post-step evaluation using LLM-as-judge
|
||||
- Queries LLM with observations and criteria
|
||||
- Parses structured evaluation output
|
||||
- Writes `specs/<feature>/BOT-EVALUATION.md` and `specs/<feature>/BOT-NOTES.md`
|
||||
|
||||
### Modified Files
|
||||
|
||||
1. **`src/types.ts`**
|
||||
- Added interfaces: `ToolObservation`, `QuestionContext`, `ExecutionObservation`, `EvaluationResult`
|
||||
- Extended `StepResult` with `evaluation` and `observations` fields
|
||||
|
||||
2. **`src/session-runner.ts`**
|
||||
- Added `collector` parameter to `SessionOptions`
|
||||
- Returns `observations` in `SessionResult`
|
||||
- Records all messages for observation tracking
|
||||
- Uses intelligent responder instead of auto-responder
|
||||
|
||||
3. **`src/cli.ts`**
|
||||
- Creates `ObservationCollector` for each step
|
||||
- Always runs evaluation after successful steps
|
||||
- Displays scores with color coding (green/yellow/red)
|
||||
- Implements quality gate before finalize
|
||||
- Shows evaluation summary in step output
|
||||
|
||||
4. **`src/bin/plan2code-bot.ts`**
|
||||
- Updated help text to mention LLM-as-judge evaluation
|
||||
- No new CLI flags (evaluation is always on)
|
||||
|
||||
### Deleted Files
|
||||
|
||||
1. **`src/auto-responder.ts`** - Replaced by intelligent-responder.ts
|
||||
2. **`src/auto-responder.test.ts`** - No longer needed
|
||||
|
||||
## Output Files (Created During Execution)
|
||||
|
||||
Both files are written to `specs/<feature>/` so they stay co-located with the feature they describe. If no spec folder exists yet (e.g. during `init`), they fall back to the project root.
|
||||
|
||||
### BOT-EVALUATION.md
|
||||
|
||||
Contains evaluation results for each step:
|
||||
- Score (0-100)
|
||||
- Strengths identified
|
||||
- Weaknesses found
|
||||
- Suggestions for improvement
|
||||
- Critical issues (if any)
|
||||
- Full reasoning from LLM
|
||||
|
||||
### BOT-NOTES.md
|
||||
|
||||
Contains execution observations:
|
||||
- Duration, tool counts, file changes
|
||||
- Questions asked and LLM reasoning for answers
|
||||
- Tool usage timeline
|
||||
- Files created/modified
|
||||
- Assistant output summary
|
||||
|
||||
## Data Structure for Metrics
|
||||
|
||||
All evaluation data is structured in `StepResult`:
|
||||
|
||||
```typescript
|
||||
{
|
||||
step: 'plan',
|
||||
success: true,
|
||||
duration: 42000,
|
||||
evaluation: {
|
||||
score: 78,
|
||||
strengths: ["Clear phases", "Realistic scope"],
|
||||
weaknesses: ["Phase 3 too vague"],
|
||||
suggestions: ["Add specific tasks to Phase 3"],
|
||||
criticalIssues: [],
|
||||
reasoning: "...",
|
||||
timestamp: 1234567890,
|
||||
evaluatorModel: 'claude-sonnet-4-5'
|
||||
},
|
||||
observations: {
|
||||
tools: [{ toolName, input, output, timestamp }, ...],
|
||||
questionsAsked: [
|
||||
{
|
||||
question: "Approve plan?",
|
||||
selectedAnswer: "Yes, approve",
|
||||
llmReasoning: "Phases are well-scoped...",
|
||||
timestamp: 1234567890
|
||||
}
|
||||
],
|
||||
filesCreated: [...],
|
||||
filesModified: [...],
|
||||
errors: []
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Benefits for Recursive Improvement
|
||||
|
||||
1. **Authentic Signals:** Real quality scores identify actual problem areas
|
||||
2. **Detailed Context:** Observations + reasoning explain WHY failures happen
|
||||
3. **Correlation Analysis:** Link patterns (tool usage, duration, errors) to quality
|
||||
4. **Continuous Loop:** Better metrics → improved workflows → higher scores → repeat
|
||||
|
||||
## Usage
|
||||
|
||||
No special flags needed - evaluation is always on:
|
||||
|
||||
```bash
|
||||
# New project
|
||||
plan2code-bot --idea "todo app"
|
||||
|
||||
# Enhancement
|
||||
cd my-project && plan2code-bot
|
||||
|
||||
# Resume with evaluation data preserved
|
||||
plan2code-bot --resume
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
After running the bot, check:
|
||||
|
||||
1. **`specs/<feature>/BOT-EVALUATION.md`** - Should show realistic scores (not all 100s)
|
||||
2. **`specs/<feature>/BOT-NOTES.md`** - Should show LLM reasoning for decisions
|
||||
3. **Console output** - Should display color-coded scores after each step
|
||||
4. **State file** (`.plan2code-bot-state.json`) - Should include evaluation data
|
||||
|
||||
## Trade-offs
|
||||
|
||||
### Latency
|
||||
- Adds ~2-3s per AskUserQuestion call (~15-20s total per run)
|
||||
- Worth it for authentic evaluation
|
||||
|
||||
### Token Cost
|
||||
- ~20-26K tokens per run (~$0.60 with Opus 4.6)
|
||||
- Investment pays off through metrics-driven improvement
|
||||
|
||||
### Determinism
|
||||
- LLM decisions vary between runs (non-deterministic)
|
||||
- Realistic - humans vary too
|
||||
- Metrics average over many runs
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
Potential improvements:
|
||||
- Model selection per step (use Haiku for simple decisions)
|
||||
- Configurable quality gate threshold
|
||||
- Historical score tracking across runs
|
||||
- Comparison with previous evaluations
|
||||
- More sophisticated scoring (weighted by step importance)
|
||||
@@ -0,0 +1,111 @@
|
||||
# plan2code-bot
|
||||
|
||||
Autonomous workflow test runner for plan2code. Uses the Claude Agent SDK to simulate a human running through the entire plan2code workflow (init, plan, document, implement, finalize) end-to-end.
|
||||
|
||||
## Two Modes (Auto-Detected)
|
||||
|
||||
1. **New Project Mode** — No `AGENTS.md` in cwd: generates an app idea, creates a subdirectory, writes IDEA.md, runs init, then all 4 steps.
|
||||
2. **Enhancement Mode** — `AGENTS.md` exists in cwd: scans the existing codebase and proposes a realistic enhancement, writes IDEA.md, then runs plan through finalize.
|
||||
|
||||
## Installation
|
||||
|
||||
From the plan2code root:
|
||||
|
||||
```bash
|
||||
node install.js
|
||||
# Select C > B to install bot only, or I to install everything
|
||||
```
|
||||
|
||||
Or manually:
|
||||
|
||||
```bash
|
||||
cd plan2code-bot
|
||||
npm install
|
||||
npm run build
|
||||
npm link
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# New project mode (run from an empty directory)
|
||||
mkdir /tmp/test-bot && cd /tmp/test-bot
|
||||
plan2code-bot
|
||||
|
||||
# Enhancement mode (run from an existing project with AGENTS.md)
|
||||
cd my-project
|
||||
plan2code-bot
|
||||
|
||||
# Seed the idea generator with a specific concept
|
||||
plan2code-bot --idea "web app that displays the current weather as vector images"
|
||||
|
||||
# Resume a previous incomplete run
|
||||
plan2code-bot --resume
|
||||
```
|
||||
|
||||
### `--idea`
|
||||
|
||||
Pass a quoted string after `--idea` to seed the idea generator with a specific concept. The AI will use it as inspiration rather than generating a completely random idea. Wrap the value in double quotes so the shell treats it as a single argument.
|
||||
|
||||
```bash
|
||||
# Specific app concept
|
||||
plan2code-bot --idea "web app that displays the current weather as vector images"
|
||||
|
||||
# Short keyword to nudge the category
|
||||
plan2code-bot --idea "markdown editor"
|
||||
|
||||
# Detailed constraint
|
||||
plan2code-bot --idea "CLI tool that converts CSV files to SQLite databases with type inference"
|
||||
|
||||
# Works in enhancement mode too — guides what kind of enhancement to propose
|
||||
cd my-existing-project
|
||||
plan2code-bot --idea "add dark mode support"
|
||||
```
|
||||
|
||||
Without `--idea`, the bot picks a random category (CLI tool or web app) and invents something on its own.
|
||||
|
||||
### `--resume`
|
||||
|
||||
Resume a previous incomplete run. The bot searches for a `.plan2code-bot-state.json` file in the current directory (enhancement mode) or in immediate subdirectories (new-project mode). If found, it restores the idea, config, and progress — skipping steps that already succeeded and continuing from where it left off.
|
||||
|
||||
```bash
|
||||
# A run failed at the implement step — resume it
|
||||
plan2code-bot --resume
|
||||
|
||||
# Can combine with --idea (idea is ignored when resuming since it's restored from state)
|
||||
plan2code-bot --resume --idea "ignored when state exists"
|
||||
```
|
||||
|
||||
If no state file is found, the bot starts a fresh run.
|
||||
|
||||
## How It Works
|
||||
|
||||
1. Detects mode based on presence of `AGENTS.md`
|
||||
2. Generates an idea (new app or enhancement) via Claude, optionally guided by `--idea` seed
|
||||
3. Writes `IDEA.md` to the project directory
|
||||
4. Installs bot-friendly copies of plan2code skills (strips `disable-model-invocation` so sessions can invoke them)
|
||||
5. Runs each workflow step as a separate Claude Agent SDK session:
|
||||
- **init** — generates `AGENTS.md` (new-project mode only)
|
||||
- **plan** — creates plan draft in `specs/<feature>/`
|
||||
- **document** — produces `overview.md` and `phase-*.md` files
|
||||
- **implement** — loops until all phases are complete (max 10 passes)
|
||||
- **finalize** — validates and archives to `specs--completed/`
|
||||
6. Validates expected artifacts after each step (aborts on missing artifacts)
|
||||
7. Moves `IDEA.md` into `specs/<feature>/` after the plan step so it stays with its feature
|
||||
8. Auto-responds to `AskUserQuestion` prompts (approvals, testing gates, name questions)
|
||||
9. Saves state to `.plan2code-bot-state.json` after each step
|
||||
|
||||
## State File
|
||||
|
||||
After each step, the bot saves its state to `.plan2code-bot-state.json` in the project directory. This includes the config, all step results, and progress tracking.
|
||||
|
||||
- **On full success** — the state file is automatically deleted (clean finish)
|
||||
- **On failure/incomplete** — the state file is preserved so you can `--resume` later
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
npm run build # Build with tsup
|
||||
npm run dev # Watch mode
|
||||
npm test # Run tests (vitest)
|
||||
```
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"name": "plan2code-bot",
|
||||
"version": "1.1.0",
|
||||
"description": "Plan2Code Bot - Autonomous workflow runner for testing plan2code end-to-end",
|
||||
"type": "module",
|
||||
"main": "dist/index.js",
|
||||
"bin": {
|
||||
"plan2code-bot": "./dist/bin/plan2code-bot.js"
|
||||
},
|
||||
"files": [
|
||||
"dist"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18.0.0"
|
||||
},
|
||||
"scripts": {
|
||||
"build": "tsup",
|
||||
"dev": "tsup --watch",
|
||||
"start": "node dist/bin/plan2code-bot.js",
|
||||
"test": "vitest run",
|
||||
"prepublishOnly": "npm run build"
|
||||
},
|
||||
"dependencies": {
|
||||
"@anthropic-ai/claude-agent-sdk": "^0.2.63",
|
||||
"chalk": "^5.6.2",
|
||||
"fs-extra": "^11.3.3",
|
||||
"ora": "^9.0.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/fs-extra": "^11.0.4",
|
||||
"@types/node": "^25.0.3",
|
||||
"tsup": "^8.5.1",
|
||||
"typescript": "^5.9.3",
|
||||
"vitest": "^4.0.18"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
import { runCLI } from '../cli.js';
|
||||
|
||||
function showHelp(): void {
|
||||
console.log(`
|
||||
+----------------------------------------------------------------+
|
||||
— PLAN2CODEDE-BOT —
|
||||
—----------------------------------------------------------------—
|
||||
— Autonomous workflow test runner foplan2codede —
|
||||
— Features LLM-as-judge for honest quality evaluation —
|
||||
+----------------------------------------------------------------+
|
||||
|
||||
Usage:
|
||||
plan2code-bot [options]
|
||||
|
||||
Options:
|
||||
--help Show this help message
|
||||
--idea <string> Seed the idea generator with a specific concept
|
||||
Example: --idea "web app for weather"
|
||||
Example: --idea="CLI tool for CSV conversion"
|
||||
--resume Resume a previous incomplete run
|
||||
|
||||
Modes:
|
||||
— New Project Mode - Run from empty directory
|
||||
The bot generates an app idea, creates a subdirectory, writes
|
||||
IDEA.md, runs init, then all 4 workflow steps.
|
||||
|
||||
— Enhancement Mode - Run from directory with AGENTS.md
|
||||
The bot scans the existing codebase, proposes an enhancement,
|
||||
writes IDEA.md, then runs plan through finalize.
|
||||
|
||||
Evaluation:
|
||||
The bot acts as an authentic QA agent, using LLM-based decision
|
||||
making during execution and providing honest quality assessments
|
||||
after each step. Results are written to specs/<feature>/BOT-EVALUATION.md
|
||||
and specs/<feature>/BOT-NOTES.md for metrics analysis.
|
||||
|
||||
Examples:
|
||||
# New project (from empty directory)
|
||||
plan2code-bot
|
||||
|
||||
# Enhancement (from existing project)
|
||||
cd my-project && plan2code-bot
|
||||
|
||||
# With specific idea
|
||||
plan2code-bot --idea "markdown editor with live preview"
|
||||
|
||||
# Resume incomplete run
|
||||
plan2code-bot --resume
|
||||
|
||||
Documentation:
|
||||
https://github.com/jparkerweb/plan2code
|
||||
`);
|
||||
}
|
||||
|
||||
function stripQuotes(str: string): string {
|
||||
// Remove surrounding quotes if present (both single and double)
|
||||
if ((str.startsWith('"') && str.endsWith('"')) ||
|
||||
(str.startsWith("'") && str.endsWith("'"))) {
|
||||
return str.slice(1, -1);
|
||||
}
|
||||
return str;
|
||||
}
|
||||
|
||||
function parseArgs(): { idea?: string; resume?: boolean; help?: boolean } {
|
||||
const args = process.argv.slice(2);
|
||||
|
||||
// Check for --help
|
||||
if (args.includes('--help') || args.includes('-h')) {
|
||||
return { help: true };
|
||||
}
|
||||
|
||||
// Parse --idea (supports both --idea="value" and --idea "value")
|
||||
let idea: string | undefined;
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const arg = args[i];
|
||||
|
||||
// Format: --idea="value"
|
||||
if (arg.startsWith('--idea=')) {
|
||||
idea = stripQuotes(arg.substring('--idea='.length));
|
||||
break;
|
||||
}
|
||||
|
||||
// Format: --idea "value"
|
||||
if (arg === '--idea' && i + 1 < args.length) {
|
||||
idea = stripQuotes(args[i + 1]);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Parse --resume
|
||||
const resume = args.includes('--resume');
|
||||
|
||||
return { idea, resume: resume || undefined };
|
||||
}
|
||||
|
||||
async function main() {
|
||||
try {
|
||||
const { idea, resume, help } = parseArgs();
|
||||
|
||||
if (help) {
|
||||
showHelp();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
await runCLI({ idea, resume });
|
||||
process.exit(0);
|
||||
} catch (err) {
|
||||
if (err instanceof Error && err.message.includes('User force closed')) {
|
||||
process.exit(0);
|
||||
}
|
||||
console.error(err instanceof Error ? err.message : String(err));
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
main();
|
||||
@@ -0,0 +1,121 @@
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
|
||||
import fs from 'fs-extra';
|
||||
import path from 'path';
|
||||
import os from 'os';
|
||||
import { saveState, loadState, deleteState, findExistingState } from './bot-state.js';
|
||||
import type { BotState } from './types.js';
|
||||
|
||||
function makeTmpDir(): string {
|
||||
return fs.mkdtempSync(path.join(os.tmpdir(), 'bot-state-test-'));
|
||||
}
|
||||
|
||||
function makeState(projectDir: string): BotState {
|
||||
return {
|
||||
config: {
|
||||
workDir: path.dirname(projectDir),
|
||||
projectDir,
|
||||
ideaName: 'test-idea',
|
||||
ideaDescription: 'A test idea',
|
||||
mode: 'new-project',
|
||||
},
|
||||
steps: [],
|
||||
currentStep: null,
|
||||
implementPasses: 0,
|
||||
allPhasesComplete: false,
|
||||
};
|
||||
}
|
||||
|
||||
describe('bot-state', () => {
|
||||
let tmpDir: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tmpDir = makeTmpDir();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
fs.removeSync(tmpDir);
|
||||
});
|
||||
|
||||
describe('saveState', () => {
|
||||
it('writes valid JSON', () => {
|
||||
const projectDir = path.join(tmpDir, 'project');
|
||||
fs.ensureDirSync(projectDir);
|
||||
const state = makeState(projectDir);
|
||||
|
||||
saveState(state);
|
||||
|
||||
const filePath = path.join(projectDir, '.plan2code-bot-state.json');
|
||||
expect(fs.existsSync(filePath)).toBe(true);
|
||||
const parsed = fs.readJsonSync(filePath);
|
||||
expect(parsed.config.ideaName).toBe('test-idea');
|
||||
});
|
||||
});
|
||||
|
||||
describe('loadState', () => {
|
||||
it('returns state from file', () => {
|
||||
const projectDir = path.join(tmpDir, 'project');
|
||||
fs.ensureDirSync(projectDir);
|
||||
const state = makeState(projectDir);
|
||||
saveState(state);
|
||||
|
||||
const loaded = loadState(projectDir);
|
||||
expect(loaded).not.toBeNull();
|
||||
expect(loaded!.config.ideaName).toBe('test-idea');
|
||||
expect(loaded!.implementPasses).toBe(0);
|
||||
});
|
||||
|
||||
it('returns null when file does not exist', () => {
|
||||
const result = loadState(path.join(tmpDir, 'nonexistent'));
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('deleteState', () => {
|
||||
it('removes the file', () => {
|
||||
const projectDir = path.join(tmpDir, 'project');
|
||||
fs.ensureDirSync(projectDir);
|
||||
const state = makeState(projectDir);
|
||||
saveState(state);
|
||||
|
||||
const filePath = path.join(projectDir, '.plan2code-bot-state.json');
|
||||
expect(fs.existsSync(filePath)).toBe(true);
|
||||
|
||||
deleteState(projectDir);
|
||||
expect(fs.existsSync(filePath)).toBe(false);
|
||||
});
|
||||
|
||||
it('is a no-op when file does not exist', () => {
|
||||
// Should not throw
|
||||
deleteState(path.join(tmpDir, 'nonexistent'));
|
||||
});
|
||||
});
|
||||
|
||||
describe('findExistingState', () => {
|
||||
it('finds state in workDir (enhancement mode)', () => {
|
||||
const state = makeState(tmpDir);
|
||||
state.config.projectDir = tmpDir;
|
||||
state.config.mode = 'enhancement';
|
||||
saveState(state);
|
||||
|
||||
const found = findExistingState(tmpDir);
|
||||
expect(found).not.toBeNull();
|
||||
expect(found!.config.ideaName).toBe('test-idea');
|
||||
});
|
||||
|
||||
it('finds state in a subdirectory (new-project mode)', () => {
|
||||
const projectDir = path.join(tmpDir, 'my-app');
|
||||
fs.ensureDirSync(projectDir);
|
||||
const state = makeState(projectDir);
|
||||
saveState(state);
|
||||
|
||||
const found = findExistingState(tmpDir);
|
||||
expect(found).not.toBeNull();
|
||||
expect(found!.config.projectDir).toBe(projectDir);
|
||||
});
|
||||
|
||||
it('returns null when no state exists', () => {
|
||||
const found = findExistingState(tmpDir);
|
||||
expect(found).toBeNull();
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,48 @@
|
||||
import fs from 'fs-extra';
|
||||
import path from 'path';
|
||||
import type { BotState } from './types.js';
|
||||
|
||||
const STATE_FILE = '.plan2code-bot-state.json';
|
||||
|
||||
export function saveState(state: BotState): void {
|
||||
const filePath = path.join(state.config.projectDir, STATE_FILE);
|
||||
fs.writeJsonSync(filePath, state, { spaces: 2 });
|
||||
}
|
||||
|
||||
export function loadState(projectDir: string): BotState | null {
|
||||
const filePath = path.join(projectDir, STATE_FILE);
|
||||
if (!fs.existsSync(filePath)) {
|
||||
return null;
|
||||
}
|
||||
return fs.readJsonSync(filePath) as BotState;
|
||||
}
|
||||
|
||||
export function deleteState(projectDir: string): void {
|
||||
const filePath = path.join(projectDir, STATE_FILE);
|
||||
if (fs.existsSync(filePath)) {
|
||||
fs.removeSync(filePath);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Search for an existing state file in workDir (enhancement mode)
|
||||
* or in immediate subdirectories (new-project mode).
|
||||
*/
|
||||
export function findExistingState(workDir: string): BotState | null {
|
||||
// Enhancement mode: state is in workDir directly
|
||||
const direct = loadState(workDir);
|
||||
if (direct) return direct;
|
||||
|
||||
// New-project mode: state is in a subdirectory
|
||||
try {
|
||||
for (const entry of fs.readdirSync(workDir, { withFileTypes: true })) {
|
||||
if (entry.isDirectory()) {
|
||||
const sub = loadState(path.join(workDir, entry.name));
|
||||
if (sub) return sub;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// workDir not readable — ignore
|
||||
}
|
||||
return null;
|
||||
}
|
||||
@@ -0,0 +1,195 @@
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
|
||||
import fs from 'fs-extra';
|
||||
import os from 'os';
|
||||
import path from 'path';
|
||||
import { validateStepArtifacts, installSkillsForBot } from './cli.js';
|
||||
|
||||
let tmpDir: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'p2c-cli-test-'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
fs.removeSync(tmpDir);
|
||||
});
|
||||
|
||||
// ── validateStepArtifacts ───────────────────────────────────────────
|
||||
|
||||
describe('validateStepArtifacts', () => {
|
||||
describe('init', () => {
|
||||
it('valid when AGENTS.md exists', () => {
|
||||
fs.writeFileSync(path.join(tmpDir, 'AGENTS.md'), '# Agents');
|
||||
const result = validateStepArtifacts('init', tmpDir);
|
||||
expect(result.valid).toBe(true);
|
||||
expect(result.missing).toEqual([]);
|
||||
});
|
||||
|
||||
it('missing when AGENTS.md does not exist', () => {
|
||||
const result = validateStepArtifacts('init', tmpDir);
|
||||
expect(result.valid).toBe(false);
|
||||
expect(result.missing).toContain('AGENTS.md');
|
||||
});
|
||||
});
|
||||
|
||||
describe('plan', () => {
|
||||
it('valid when specs/feature/PLAN-DRAFT-*.md exists', () => {
|
||||
const specDir = path.join(tmpDir, 'specs', 'my-feature');
|
||||
fs.ensureDirSync(specDir);
|
||||
fs.writeFileSync(path.join(specDir, 'PLAN-DRAFT-v1.md'), '# Plan');
|
||||
const result = validateStepArtifacts('plan', tmpDir);
|
||||
expect(result.valid).toBe(true);
|
||||
});
|
||||
|
||||
it('missing when specs/ is absent', () => {
|
||||
const result = validateStepArtifacts('plan', tmpDir);
|
||||
expect(result.valid).toBe(false);
|
||||
expect(result.missing).toContain('specs/ directory');
|
||||
});
|
||||
|
||||
it('missing when specs/ has dirs but no plan draft files', () => {
|
||||
const specDir = path.join(tmpDir, 'specs', 'my-feature');
|
||||
fs.ensureDirSync(specDir);
|
||||
fs.writeFileSync(path.join(specDir, 'notes.md'), '# Notes');
|
||||
const result = validateStepArtifacts('plan', tmpDir);
|
||||
expect(result.valid).toBe(false);
|
||||
expect(result.missing).toContain('specs/*/PLAN-DRAFT-*.md');
|
||||
});
|
||||
});
|
||||
|
||||
describe('document', () => {
|
||||
it('valid when overview.md and phase-*.md exist', () => {
|
||||
const specDir = path.join(tmpDir, 'specs', 'my-feature');
|
||||
fs.ensureDirSync(specDir);
|
||||
fs.writeFileSync(path.join(specDir, 'overview.md'), '# Overview');
|
||||
fs.writeFileSync(path.join(specDir, 'phase-1.md'), '# Phase 1');
|
||||
const result = validateStepArtifacts('document', tmpDir);
|
||||
expect(result.valid).toBe(true);
|
||||
});
|
||||
|
||||
it('missing overview.md when absent', () => {
|
||||
const specDir = path.join(tmpDir, 'specs', 'my-feature');
|
||||
fs.ensureDirSync(specDir);
|
||||
fs.writeFileSync(path.join(specDir, 'phase-1.md'), '# Phase 1');
|
||||
const result = validateStepArtifacts('document', tmpDir);
|
||||
expect(result.valid).toBe(false);
|
||||
expect(result.missing).toContain('specs/*/overview.md');
|
||||
});
|
||||
|
||||
it('missing phase files when absent', () => {
|
||||
const specDir = path.join(tmpDir, 'specs', 'my-feature');
|
||||
fs.ensureDirSync(specDir);
|
||||
fs.writeFileSync(path.join(specDir, 'overview.md'), '# Overview');
|
||||
const result = validateStepArtifacts('document', tmpDir);
|
||||
expect(result.valid).toBe(false);
|
||||
expect(result.missing).toContain('specs/*/phase-*.md files');
|
||||
});
|
||||
|
||||
it('accepts phase files inside phases/ subdirectory', () => {
|
||||
const specDir = path.join(tmpDir, 'specs', 'my-feature');
|
||||
const phasesDir = path.join(specDir, 'phases');
|
||||
fs.ensureDirSync(phasesDir);
|
||||
fs.writeFileSync(path.join(specDir, 'overview.md'), '# Overview');
|
||||
fs.writeFileSync(path.join(phasesDir, 'phase-1.md'), '# Phase 1');
|
||||
const result = validateStepArtifacts('document', tmpDir);
|
||||
expect(result.valid).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('finalize', () => {
|
||||
it('valid when specs--completed/ has a subdirectory', () => {
|
||||
fs.ensureDirSync(path.join(tmpDir, 'specs--completed', 'my-feature'));
|
||||
const result = validateStepArtifacts('finalize', tmpDir);
|
||||
expect(result.valid).toBe(true);
|
||||
});
|
||||
|
||||
it('missing when specs--completed/ is absent', () => {
|
||||
const result = validateStepArtifacts('finalize', tmpDir);
|
||||
expect(result.valid).toBe(false);
|
||||
expect(result.missing).toContain('specs--completed/ directory');
|
||||
});
|
||||
|
||||
it('missing when specs--completed/ is empty', () => {
|
||||
fs.ensureDirSync(path.join(tmpDir, 'specs--completed'));
|
||||
const result = validateStepArtifacts('finalize', tmpDir);
|
||||
expect(result.valid).toBe(false);
|
||||
expect(result.missing).toContain('archived spec in specs--completed/');
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
// ── installSkillsForBot ─────────────────────────────────────────────
|
||||
|
||||
describe('installSkillsForBot', () => {
|
||||
let fakeHome: string;
|
||||
let projectDir: string;
|
||||
let origHome: string | undefined;
|
||||
let origUserProfile: string | undefined;
|
||||
|
||||
beforeEach(() => {
|
||||
fakeHome = fs.mkdtempSync(path.join(os.tmpdir(), 'p2c-home-'));
|
||||
projectDir = fs.mkdtempSync(path.join(os.tmpdir(), 'p2c-proj-'));
|
||||
origHome = process.env.HOME;
|
||||
origUserProfile = process.env.USERPROFILE;
|
||||
process.env.HOME = fakeHome;
|
||||
process.env.USERPROFILE = fakeHome;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
process.env.HOME = origHome;
|
||||
process.env.USERPROFILE = origUserProfile;
|
||||
fs.removeSync(fakeHome);
|
||||
fs.removeSync(projectDir);
|
||||
});
|
||||
|
||||
it('copies skills from user dir to project .claude/skills/', () => {
|
||||
const srcDir = path.join(fakeHome, '.claude', 'skills', 'plan2code-init');
|
||||
fs.ensureDirSync(srcDir);
|
||||
fs.writeFileSync(path.join(srcDir, 'SKILL.md'), '---\nname: init\n---\nSome content');
|
||||
|
||||
installSkillsForBot(projectDir);
|
||||
|
||||
const dest = path.join(projectDir, '.claude', 'skills', 'plan2code-init', 'SKILL.md');
|
||||
expect(fs.existsSync(dest)).toBe(true);
|
||||
expect(fs.readFileSync(dest, 'utf-8')).toContain('Some content');
|
||||
});
|
||||
|
||||
it('strips disable-model-invocation: true from copied SKILL.md', () => {
|
||||
const srcDir = path.join(fakeHome, '.claude', 'skills', 'plan2code-1-plan');
|
||||
fs.ensureDirSync(srcDir);
|
||||
fs.writeFileSync(
|
||||
path.join(srcDir, 'SKILL.md'),
|
||||
'disable-model-invocation: true\n---\nname: plan\n---\nPlan content',
|
||||
);
|
||||
|
||||
installSkillsForBot(projectDir);
|
||||
|
||||
const dest = path.join(projectDir, '.claude', 'skills', 'plan2code-1-plan', 'SKILL.md');
|
||||
const content = fs.readFileSync(dest, 'utf-8');
|
||||
expect(content).not.toContain('disable-model-invocation');
|
||||
expect(content).toContain('Plan content');
|
||||
});
|
||||
|
||||
it('preserves rest of skill content', () => {
|
||||
const srcDir = path.join(fakeHome, '.claude', 'skills', 'plan2code-2-document');
|
||||
fs.ensureDirSync(srcDir);
|
||||
const original = 'disable-model-invocation: true\n---\nname: doc\n---\nLine 1\nLine 2\nLine 3';
|
||||
fs.writeFileSync(path.join(srcDir, 'SKILL.md'), original);
|
||||
|
||||
installSkillsForBot(projectDir);
|
||||
|
||||
const dest = path.join(projectDir, '.claude', 'skills', 'plan2code-2-document', 'SKILL.md');
|
||||
const content = fs.readFileSync(dest, 'utf-8');
|
||||
expect(content).toContain('Line 1');
|
||||
expect(content).toContain('Line 2');
|
||||
expect(content).toContain('Line 3');
|
||||
expect(content).toContain('name: doc');
|
||||
});
|
||||
|
||||
it('skips gracefully when source skill does not exist', () => {
|
||||
// No skills in fakeHome — should not throw
|
||||
expect(() => installSkillsForBot(projectDir)).not.toThrow();
|
||||
// No .claude/skills/ created in project
|
||||
expect(fs.existsSync(path.join(projectDir, '.claude', 'skills'))).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,587 @@
|
||||
import fs from 'fs-extra';
|
||||
import path from 'path';
|
||||
import chalk from 'chalk';
|
||||
import ora from 'ora';
|
||||
import { generateNewAppIdea, generateEnhancementIdea } from './idea-generator.js';
|
||||
import { runSession } from './session-runner.js';
|
||||
import { buildStepPrompt } from './prompts/step-instructions.js';
|
||||
import { checkAllPhasesComplete } from './step-detector.js';
|
||||
import { saveState, loadState, deleteState, findExistingState } from './bot-state.js';
|
||||
import { ObservationCollector } from './observation-collector.js';
|
||||
import { evaluateStep } from './evaluator.js';
|
||||
import type { BotConfig, BotMode, BotState, StepName, StepResult } from './types.js';
|
||||
|
||||
const BANNER = `
|
||||
╔══════════════════════════════════════╗
|
||||
║ plan2code-bot v1.1.0 ║
|
||||
║ Autonomous Workflow Test Runner ║
|
||||
╚══════════════════════════════════════╝
|
||||
`;
|
||||
|
||||
const MAX_IMPLEMENT_PASSES = 10;
|
||||
|
||||
/** Skill name mapping: bot step name → installed skill directory name */
|
||||
const SKILL_MAP: Record<StepName, string> = {
|
||||
init: 'plan2code-init',
|
||||
plan: 'plan2code-1-plan',
|
||||
document: 'plan2code-2-document',
|
||||
implement: 'plan2code-3-implement',
|
||||
finalize: 'plan2code-4-finalize',
|
||||
};
|
||||
|
||||
/**
|
||||
* Install bot-friendly copies of plan2code skills into the project's .claude/skills/.
|
||||
* The global skills have `disable-model-invocation: true` which prevents the autonomous
|
||||
* session from invoking them via the Skill tool. We copy them with that flag removed
|
||||
* so the session can invoke the real workflow prompts instead of guessing.
|
||||
*/
|
||||
export function installSkillsForBot(projectDir: string): void {
|
||||
const userSkillsDir = path.join(
|
||||
process.env.HOME || process.env.USERPROFILE || '',
|
||||
'.claude',
|
||||
'skills',
|
||||
);
|
||||
const projectSkillsDir = path.join(projectDir, '.claude', 'skills');
|
||||
|
||||
for (const skillName of Object.values(SKILL_MAP)) {
|
||||
const srcFile = path.join(userSkillsDir, skillName, 'SKILL.md');
|
||||
if (!fs.existsSync(srcFile)) continue;
|
||||
|
||||
const content = fs.readFileSync(srcFile, 'utf-8');
|
||||
// Remove the disable-model-invocation line so the bot session can invoke the skill
|
||||
const patched = content.replace(/^disable-model-invocation:\s*true\n?/m, '');
|
||||
|
||||
const destDir = path.join(projectSkillsDir, skillName);
|
||||
fs.ensureDirSync(destDir);
|
||||
fs.writeFileSync(path.join(destDir, 'SKILL.md'), patched);
|
||||
}
|
||||
}
|
||||
|
||||
export function validateStepArtifacts(step: StepName, projectDir: string): { valid: boolean; missing: string[] } {
|
||||
const missing: string[] = [];
|
||||
|
||||
switch (step) {
|
||||
case 'init': {
|
||||
const agentsPath = path.join(projectDir, 'AGENTS.md');
|
||||
if (!fs.existsSync(agentsPath)) missing.push('AGENTS.md');
|
||||
break;
|
||||
}
|
||||
case 'plan': {
|
||||
const specsDir = path.join(projectDir, 'specs');
|
||||
if (!fs.existsSync(specsDir)) {
|
||||
missing.push('specs/ directory');
|
||||
} else {
|
||||
const entries = fs.readdirSync(specsDir, { withFileTypes: true });
|
||||
const specDirs = entries.filter((e) => e.isDirectory());
|
||||
const hasPlanDraft = specDirs.some((d) => {
|
||||
const files = fs.readdirSync(path.join(specsDir, d.name));
|
||||
return files.some((f) => /plan[-_]?draft/i.test(f));
|
||||
});
|
||||
if (!hasPlanDraft) missing.push('specs/*/PLAN-DRAFT-*.md');
|
||||
}
|
||||
break;
|
||||
}
|
||||
case 'document': {
|
||||
const specsDir = path.join(projectDir, 'specs');
|
||||
if (!fs.existsSync(specsDir)) {
|
||||
missing.push('specs/ directory');
|
||||
} else {
|
||||
const entries = fs.readdirSync(specsDir, { withFileTypes: true });
|
||||
const specDirs = entries.filter((e) => e.isDirectory());
|
||||
let foundOverview = false;
|
||||
let foundPhaseFile = false;
|
||||
for (const d of specDirs) {
|
||||
const specPath = path.join(specsDir, d.name);
|
||||
const files = fs.readdirSync(specPath);
|
||||
if (files.some((f) => f === 'overview.md')) foundOverview = true;
|
||||
if (files.some((f) => /^phase[-_]?\d+.*\.md$/i.test(f))) foundPhaseFile = true;
|
||||
// Also check phases/ subdirectory
|
||||
const phasesSubdir = path.join(specPath, 'phases');
|
||||
if (fs.existsSync(phasesSubdir)) {
|
||||
const subFiles = fs.readdirSync(phasesSubdir);
|
||||
if (subFiles.some((f) => /phase/i.test(f))) foundPhaseFile = true;
|
||||
}
|
||||
}
|
||||
if (!foundOverview) missing.push('specs/*/overview.md');
|
||||
if (!foundPhaseFile) missing.push('specs/*/phase-*.md files');
|
||||
}
|
||||
break;
|
||||
}
|
||||
case 'finalize': {
|
||||
const completedDir = path.join(projectDir, 'specs--completed');
|
||||
if (!fs.existsSync(completedDir)) {
|
||||
missing.push('specs--completed/ directory');
|
||||
} else {
|
||||
const entries = fs.readdirSync(completedDir, { withFileTypes: true });
|
||||
const specDirs = entries.filter((e) => e.isDirectory());
|
||||
if (specDirs.length === 0) missing.push('archived spec in specs--completed/');
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
return { valid: missing.length === 0, missing };
|
||||
}
|
||||
|
||||
function detectMode(workDir: string): BotMode {
|
||||
const agentsPath = path.join(workDir, 'AGENTS.md');
|
||||
return fs.existsSync(agentsPath) ? 'enhancement' : 'new-project';
|
||||
}
|
||||
|
||||
function formatDuration(ms: number): string {
|
||||
const seconds = Math.floor(ms / 1000);
|
||||
const minutes = Math.floor(seconds / 60);
|
||||
const remainingSeconds = seconds % 60;
|
||||
if (minutes > 0) {
|
||||
return `${minutes}m ${remainingSeconds}s`;
|
||||
}
|
||||
return `${seconds}s`;
|
||||
}
|
||||
|
||||
async function runStep(
|
||||
step: StepName,
|
||||
config: BotConfig,
|
||||
state: BotState,
|
||||
): Promise<StepResult> {
|
||||
const spinner = ora({
|
||||
text: chalk.cyan(`Running ${step} step...`),
|
||||
spinner: 'dots',
|
||||
}).start();
|
||||
|
||||
const prompt = buildStepPrompt(step, config);
|
||||
|
||||
try {
|
||||
// Create observation collector
|
||||
const collector = new ObservationCollector(step);
|
||||
|
||||
const result = await runSession({
|
||||
prompt,
|
||||
config,
|
||||
step,
|
||||
maxTurns: step === 'implement' ? 80 : 50,
|
||||
collector,
|
||||
});
|
||||
|
||||
if (result.success) {
|
||||
spinner.succeed(
|
||||
chalk.green(`${step} completed in ${formatDuration(result.duration)}`)
|
||||
);
|
||||
|
||||
// Run evaluation
|
||||
spinner.text = chalk.cyan('Evaluating step quality...');
|
||||
spinner.start();
|
||||
|
||||
const evaluation = await evaluateStep(step, result.observations, config.projectDir);
|
||||
|
||||
spinner.succeed(
|
||||
chalk.cyan(`Evaluation complete: ${formatScore(evaluation.score)}`)
|
||||
);
|
||||
|
||||
// Display evaluation summary
|
||||
console.log(chalk.dim(` Score: ${formatScore(evaluation.score)}`));
|
||||
if (evaluation.strengths.length > 0) {
|
||||
console.log(chalk.green(` ✓ ${evaluation.strengths[0]}`));
|
||||
}
|
||||
if (evaluation.weaknesses.length > 0) {
|
||||
console.log(chalk.yellow(` ⚠ ${evaluation.weaknesses[0]}`));
|
||||
}
|
||||
|
||||
const stepResult: StepResult = {
|
||||
step,
|
||||
success: result.success,
|
||||
sessionId: result.sessionId,
|
||||
duration: result.duration,
|
||||
error: null,
|
||||
evaluation,
|
||||
observations: result.observations,
|
||||
};
|
||||
|
||||
return stepResult;
|
||||
} else {
|
||||
spinner.fail(chalk.red(`${step} failed after ${formatDuration(result.duration)}`));
|
||||
|
||||
return {
|
||||
step,
|
||||
success: false,
|
||||
sessionId: result.sessionId,
|
||||
duration: result.duration,
|
||||
error: 'Session failed',
|
||||
observations: result.observations,
|
||||
};
|
||||
}
|
||||
} catch (error) {
|
||||
const errorMsg = error instanceof Error ? error.message : String(error);
|
||||
spinner.fail(chalk.red(`${step} error: ${errorMsg}`));
|
||||
return {
|
||||
step,
|
||||
success: false,
|
||||
sessionId: null,
|
||||
duration: 0,
|
||||
error: errorMsg,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function formatScore(score: number): string {
|
||||
if (score >= 85) return chalk.green(`${score}/100`);
|
||||
if (score >= 70) return chalk.yellow(`${score}/100`);
|
||||
return chalk.red(`${score}/100`);
|
||||
}
|
||||
|
||||
export interface CLIOptions {
|
||||
idea?: string;
|
||||
resume?: boolean;
|
||||
}
|
||||
|
||||
/** Check if a step completed successfully in a saved state */
|
||||
function stepSucceeded(state: BotState, step: StepName): boolean {
|
||||
return state.steps.some((s) => s.step === step && s.success);
|
||||
}
|
||||
|
||||
export async function runCLI(options: CLIOptions = {}): Promise<void> {
|
||||
console.log(chalk.cyan(BANNER));
|
||||
|
||||
const workDir = process.cwd();
|
||||
|
||||
// Check for resumable state
|
||||
let resuming = false;
|
||||
let savedState: BotState | null = null;
|
||||
|
||||
if (options.resume) {
|
||||
savedState = findExistingState(workDir);
|
||||
if (savedState) {
|
||||
resuming = true;
|
||||
console.log(chalk.yellow('Resuming previous incomplete run...'));
|
||||
} else {
|
||||
console.log(chalk.dim('No previous state found, starting fresh.'));
|
||||
}
|
||||
}
|
||||
|
||||
const mode = resuming ? savedState!.config.mode : detectMode(workDir);
|
||||
|
||||
console.log(chalk.dim(`Working directory: ${workDir}`));
|
||||
console.log(chalk.dim(`Mode: ${mode === 'new-project' ? 'New Project' : 'Enhancement'}`));
|
||||
if (options.idea) {
|
||||
console.log(chalk.dim(`Idea seed: ${options.idea}`));
|
||||
}
|
||||
console.log('');
|
||||
|
||||
let ideaName: string;
|
||||
let ideaDescription: string;
|
||||
let projectDir: string;
|
||||
|
||||
if (resuming) {
|
||||
// Restore from saved state
|
||||
ideaName = savedState!.config.ideaName;
|
||||
ideaDescription = savedState!.config.ideaDescription;
|
||||
projectDir = savedState!.config.projectDir;
|
||||
console.log(chalk.green(`Restored idea: ${ideaName}`));
|
||||
console.log(chalk.dim(` ${ideaDescription}`));
|
||||
console.log('');
|
||||
} else {
|
||||
// Step 1: Generate idea
|
||||
const ideaSpinner = ora({
|
||||
text: chalk.cyan('Generating idea...'),
|
||||
spinner: 'dots',
|
||||
}).start();
|
||||
|
||||
try {
|
||||
if (mode === 'new-project') {
|
||||
const idea = await generateNewAppIdea(options.idea);
|
||||
ideaName = idea.name;
|
||||
ideaDescription = idea.description;
|
||||
} else {
|
||||
const idea = await generateEnhancementIdea(workDir, options.idea);
|
||||
ideaName = idea.name;
|
||||
ideaDescription = idea.description;
|
||||
}
|
||||
ideaSpinner.succeed(chalk.green(`Idea generated: ${ideaName}`));
|
||||
} catch (error) {
|
||||
ideaSpinner.fail(chalk.red('Failed to generate idea'));
|
||||
throw error;
|
||||
}
|
||||
|
||||
console.log(chalk.dim(` ${ideaDescription}`));
|
||||
console.log('');
|
||||
|
||||
// Determine project directory
|
||||
projectDir = mode === 'new-project'
|
||||
? path.join(workDir, ideaName)
|
||||
: workDir;
|
||||
|
||||
// Create project directory for new projects
|
||||
if (mode === 'new-project') {
|
||||
fs.ensureDirSync(projectDir);
|
||||
}
|
||||
}
|
||||
|
||||
// Install bot-friendly skills (without disable-model-invocation)
|
||||
installSkillsForBot(projectDir);
|
||||
console.log(chalk.dim('Installed plan2code skills for bot sessions'));
|
||||
|
||||
// Write IDEA.md (only if not resuming past plan step, since it gets moved)
|
||||
if (!resuming || !stepSucceeded(savedState!, 'plan')) {
|
||||
const ideaContent = `# ${ideaName}\n\n${ideaDescription}\n`;
|
||||
fs.writeFileSync(path.join(projectDir, 'IDEA.md'), ideaContent);
|
||||
console.log(chalk.dim(`Wrote IDEA.md to ${projectDir}`));
|
||||
}
|
||||
console.log('');
|
||||
|
||||
// Build config
|
||||
const config: BotConfig = {
|
||||
workDir,
|
||||
projectDir,
|
||||
ideaDescription,
|
||||
ideaName,
|
||||
mode,
|
||||
};
|
||||
|
||||
// Initialize or restore state
|
||||
const state: BotState = resuming
|
||||
? { ...savedState!, config }
|
||||
: {
|
||||
config,
|
||||
steps: [],
|
||||
currentStep: null,
|
||||
implementPasses: 0,
|
||||
allPhasesComplete: false,
|
||||
};
|
||||
|
||||
// Step 2: Run init (new project only)
|
||||
if (mode === 'new-project') {
|
||||
if (resuming && stepSucceeded(savedState!, 'init')) {
|
||||
console.log(chalk.dim('--- Init (skipped — previously succeeded) ---'));
|
||||
} else {
|
||||
console.log(chalk.bold('--- Init ---'));
|
||||
state.currentStep = 'init';
|
||||
const initResult = await runStep('init', config, state);
|
||||
state.steps.push(initResult);
|
||||
saveState(state);
|
||||
|
||||
if (!initResult.success) {
|
||||
console.log(chalk.red('\nInit failed. Aborting.'));
|
||||
printSummary(state, workDir, false);
|
||||
return;
|
||||
}
|
||||
const initValidation = validateStepArtifacts('init', projectDir);
|
||||
if (!initValidation.valid) {
|
||||
console.log(chalk.yellow(` ⚠ Missing artifacts: ${initValidation.missing.join(', ')}`));
|
||||
initResult.success = false;
|
||||
initResult.error = `Missing artifacts: ${initValidation.missing.join(', ')}`;
|
||||
console.log(chalk.red('\nInit artifacts missing. Aborting.'));
|
||||
printSummary(state, workDir, false);
|
||||
return;
|
||||
}
|
||||
}
|
||||
console.log('');
|
||||
}
|
||||
|
||||
// Step 3: Plan
|
||||
if (resuming && stepSucceeded(savedState!, 'plan')) {
|
||||
console.log(chalk.dim('--- Plan (skipped — previously succeeded) ---'));
|
||||
} else {
|
||||
console.log(chalk.bold('--- Plan ---'));
|
||||
state.currentStep = 'plan';
|
||||
const planResult = await runStep('plan', config, state);
|
||||
state.steps.push(planResult);
|
||||
saveState(state);
|
||||
|
||||
if (!planResult.success) {
|
||||
console.log(chalk.red('\nPlan step failed. Aborting.'));
|
||||
printSummary(state, workDir, false);
|
||||
return;
|
||||
}
|
||||
const planValidation = validateStepArtifacts('plan', projectDir);
|
||||
if (!planValidation.valid) {
|
||||
console.log(chalk.yellow(` ⚠ Missing artifacts: ${planValidation.missing.join(', ')}`));
|
||||
planResult.success = false;
|
||||
planResult.error = `Missing artifacts: ${planValidation.missing.join(', ')}`;
|
||||
console.log(chalk.red('\nPlan artifacts missing. Aborting.'));
|
||||
printSummary(state, workDir, false);
|
||||
return;
|
||||
}
|
||||
|
||||
// Move IDEA.md into the spec directory so it stays with its feature
|
||||
const ideaPath = path.join(projectDir, 'IDEA.md');
|
||||
if (fs.existsSync(ideaPath)) {
|
||||
const specEntries = fs.readdirSync(path.join(projectDir, 'specs'), { withFileTypes: true });
|
||||
const firstSpecDir = specEntries.find((e) => e.isDirectory());
|
||||
if (firstSpecDir) {
|
||||
const dest = path.join(projectDir, 'specs', firstSpecDir.name, 'IDEA.md');
|
||||
fs.moveSync(ideaPath, dest, { overwrite: true });
|
||||
console.log(chalk.dim(`Moved IDEA.md → specs/${firstSpecDir.name}/IDEA.md`));
|
||||
}
|
||||
}
|
||||
}
|
||||
console.log('');
|
||||
|
||||
// Step 4: Document
|
||||
if (resuming && stepSucceeded(savedState!, 'document')) {
|
||||
console.log(chalk.dim('--- Document (skipped — previously succeeded) ---'));
|
||||
} else {
|
||||
console.log(chalk.bold('--- Document ---'));
|
||||
state.currentStep = 'document';
|
||||
const docResult = await runStep('document', config, state);
|
||||
state.steps.push(docResult);
|
||||
saveState(state);
|
||||
|
||||
if (!docResult.success) {
|
||||
console.log(chalk.red('\nDocument step failed. Aborting.'));
|
||||
printSummary(state, workDir, false);
|
||||
return;
|
||||
}
|
||||
const docValidation = validateStepArtifacts('document', projectDir);
|
||||
if (!docValidation.valid) {
|
||||
console.log(chalk.yellow(` ⚠ Missing artifacts: ${docValidation.missing.join(', ')}`));
|
||||
docResult.success = false;
|
||||
docResult.error = `Missing artifacts: ${docValidation.missing.join(', ')}`;
|
||||
console.log(chalk.red('\nDocument artifacts missing. Aborting.'));
|
||||
printSummary(state, workDir, false);
|
||||
return;
|
||||
}
|
||||
}
|
||||
console.log('');
|
||||
|
||||
// Step 5: Implement (loop until all phases complete)
|
||||
if (resuming && savedState!.allPhasesComplete) {
|
||||
console.log(chalk.dim('--- Implement (skipped — all phases already complete) ---'));
|
||||
} else {
|
||||
console.log(chalk.bold('--- Implement ---'));
|
||||
while (state.implementPasses < MAX_IMPLEMENT_PASSES) {
|
||||
state.implementPasses++;
|
||||
state.currentStep = 'implement';
|
||||
|
||||
console.log(chalk.dim(` Pass ${state.implementPasses}/${MAX_IMPLEMENT_PASSES}`));
|
||||
const implResult = await runStep('implement', config, state);
|
||||
state.steps.push(implResult);
|
||||
saveState(state);
|
||||
|
||||
if (!implResult.success) {
|
||||
console.log(chalk.yellow(`\nImplement pass ${state.implementPasses} failed. Continuing...`));
|
||||
}
|
||||
|
||||
// Check if all phases are complete
|
||||
if (checkAllPhasesComplete(projectDir)) {
|
||||
state.allPhasesComplete = true;
|
||||
saveState(state);
|
||||
console.log(chalk.green(' All phases complete!'));
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (!state.allPhasesComplete && state.implementPasses >= MAX_IMPLEMENT_PASSES) {
|
||||
console.log(chalk.yellow(`\nMax implement passes (${MAX_IMPLEMENT_PASSES}) reached.`));
|
||||
}
|
||||
}
|
||||
console.log('');
|
||||
|
||||
// Step 6: Finalize
|
||||
if (resuming && stepSucceeded(savedState!, 'finalize')) {
|
||||
console.log(chalk.dim('--- Finalize (skipped — previously succeeded) ---'));
|
||||
} else {
|
||||
console.log(chalk.bold('--- Finalize ---'));
|
||||
|
||||
// Check quality gate: average score must be >= 60
|
||||
const evaluatedSteps = state.steps.filter((s) => s.evaluation);
|
||||
if (evaluatedSteps.length > 0) {
|
||||
const avgScore =
|
||||
evaluatedSteps.reduce((sum, s) => sum + (s.evaluation?.score ?? 0), 0) /
|
||||
evaluatedSteps.length;
|
||||
|
||||
console.log(chalk.dim(` Average quality score: ${formatScore(Math.round(avgScore))}`));
|
||||
|
||||
if (avgScore < 60) {
|
||||
console.log(
|
||||
chalk.red(
|
||||
'\n⚠ Quality gate failed: Average score is below 60. Please review and fix issues before finalizing.'
|
||||
)
|
||||
);
|
||||
console.log(chalk.dim(' Check specs/<feature>/BOT-EVALUATION.md for detailed feedback.'));
|
||||
printSummary(state, workDir, false);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
state.currentStep = 'finalize';
|
||||
const finalizeResult = await runStep('finalize', config, state);
|
||||
state.steps.push(finalizeResult);
|
||||
saveState(state);
|
||||
|
||||
if (finalizeResult.success) {
|
||||
const finalValidation = validateStepArtifacts('finalize', projectDir);
|
||||
if (!finalValidation.valid) {
|
||||
console.log(chalk.yellow(` ⚠ Missing artifacts: ${finalValidation.missing.join(', ')}`));
|
||||
finalizeResult.success = false;
|
||||
finalizeResult.error = `Missing artifacts: ${finalValidation.missing.join(', ')}`;
|
||||
saveState(state);
|
||||
}
|
||||
}
|
||||
}
|
||||
console.log('');
|
||||
|
||||
// Determine overall success
|
||||
const allSucceeded = state.steps.length > 0 && state.steps.every((s) => s.success);
|
||||
printSummary(state, workDir, allSucceeded);
|
||||
|
||||
// Clean up state file on full success
|
||||
if (allSucceeded) {
|
||||
deleteState(projectDir);
|
||||
console.log(chalk.dim('Cleaned up state file (run succeeded)'));
|
||||
} else {
|
||||
console.log(chalk.dim('State file preserved for resume (run incomplete)'));
|
||||
}
|
||||
console.log('');
|
||||
}
|
||||
|
||||
function printSummary(state: BotState, workDir: string, allSucceeded: boolean): void {
|
||||
const { ideaName, mode, projectDir } = state.config;
|
||||
|
||||
console.log(chalk.cyan('═══════════════════════════════════════'));
|
||||
console.log(chalk.bold(' Bot Run Summary'));
|
||||
console.log(chalk.cyan('═══════════════════════════════════════'));
|
||||
console.log(chalk.dim(` Project: ${ideaName}`));
|
||||
console.log(chalk.dim(` Mode: ${mode}`));
|
||||
console.log(chalk.dim(` Directory: ${projectDir}`));
|
||||
console.log('');
|
||||
|
||||
const totalDuration = state.steps.reduce((sum, s) => sum + s.duration, 0);
|
||||
const successCount = state.steps.filter((s) => s.success).length;
|
||||
|
||||
for (const step of state.steps) {
|
||||
const icon = step.success ? chalk.green('✓') : chalk.red('✗');
|
||||
const scoreText = step.evaluation
|
||||
? ` [${formatScore(step.evaluation.score)}]`
|
||||
: '';
|
||||
console.log(
|
||||
` ${icon} ${step.step.padEnd(12)} ${formatDuration(step.duration)}${scoreText}`
|
||||
);
|
||||
}
|
||||
|
||||
console.log('');
|
||||
|
||||
// Display average quality score if available
|
||||
const evaluatedSteps = state.steps.filter((s) => s.evaluation);
|
||||
if (evaluatedSteps.length > 0) {
|
||||
const avgScore =
|
||||
evaluatedSteps.reduce((sum, s) => sum + (s.evaluation?.score ?? 0), 0) /
|
||||
evaluatedSteps.length;
|
||||
console.log(
|
||||
chalk.dim(` Average quality: ${formatScore(Math.round(avgScore))}`)
|
||||
);
|
||||
}
|
||||
|
||||
console.log(
|
||||
chalk.dim(
|
||||
` Total: ${successCount}/${state.steps.length} steps succeeded in ${formatDuration(totalDuration)}`
|
||||
)
|
||||
);
|
||||
console.log(chalk.dim(` Implement passes: ${state.implementPasses}`));
|
||||
if (!allSucceeded) {
|
||||
console.log(
|
||||
chalk.dim(
|
||||
` State saved to: ${path.relative(workDir, projectDir)}/.plan2code-bot-state.json`
|
||||
)
|
||||
);
|
||||
}
|
||||
console.log('');
|
||||
}
|
||||
@@ -0,0 +1,311 @@
|
||||
import { query } from '@anthropic-ai/claude-agent-sdk';
|
||||
import fs from 'fs-extra';
|
||||
import path from 'path';
|
||||
import type { EvaluationResult, ExecutionObservation, StepName } from './types.js';
|
||||
import { getCriteriaForStep } from './prompts/evaluation-criteria.js';
|
||||
|
||||
/**
|
||||
* Evaluates a completed step using LLM-as-judge.
|
||||
* Returns honest quality assessment based on observations and artifacts.
|
||||
*/
|
||||
export async function evaluateStep(
|
||||
step: StepName,
|
||||
observations: ExecutionObservation,
|
||||
projectDir: string
|
||||
): Promise<EvaluationResult> {
|
||||
const criteria = getCriteriaForStep(step);
|
||||
const prompt = buildEvaluationPrompt(step, observations, criteria, projectDir);
|
||||
|
||||
try {
|
||||
// Query LLM with ability to inspect artifacts
|
||||
const session = query({
|
||||
prompt,
|
||||
options: {
|
||||
maxTurns: 30,
|
||||
cwd: projectDir,
|
||||
permissionMode: 'bypassPermissions',
|
||||
allowDangerouslySkipPermissions: true,
|
||||
allowedTools: ['Read', 'Glob', 'Grep'],
|
||||
systemPrompt:
|
||||
'You are a QA engineer evaluating completed work. Be thorough, honest, and constructive.',
|
||||
},
|
||||
});
|
||||
|
||||
let output = '';
|
||||
for await (const message of session) {
|
||||
if (message.type === 'assistant') {
|
||||
for (const block of message.message.content) {
|
||||
if (block.type === 'text') output += block.text;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const evaluation = parseEvaluationOutput(output, step);
|
||||
|
||||
// Write evaluation files
|
||||
await writeEvaluationFiles(evaluation, observations, projectDir);
|
||||
|
||||
return evaluation;
|
||||
} catch (error) {
|
||||
console.warn('Evaluation failed, using fallback:', error);
|
||||
return createFallbackEvaluation(step, observations);
|
||||
}
|
||||
}
|
||||
|
||||
function buildEvaluationPrompt(
|
||||
step: StepName,
|
||||
observations: ExecutionObservation,
|
||||
criteria: ReturnType<typeof getCriteriaForStep>,
|
||||
projectDir: string
|
||||
): string {
|
||||
const duration = observations.endTime - observations.startTime;
|
||||
const durationSec = Math.floor(duration / 1000);
|
||||
|
||||
const toolsSummary = observations.tools
|
||||
.map((t) => `- ${t.toolName} (${new Date(t.timestamp).toISOString()})`)
|
||||
.join('\n');
|
||||
|
||||
const filesSummary = [
|
||||
...observations.filesCreated.map((f) => `CREATED: ${f}`),
|
||||
...observations.filesModified.map((f) => `MODIFIED: ${f}`),
|
||||
].join('\n');
|
||||
|
||||
const questionsSummary = observations.questionsAsked
|
||||
.map(
|
||||
(q) =>
|
||||
`Q: ${q.question}\nA: ${q.selectedAnswer}\nReasoning: ${q.llmReasoning}`
|
||||
)
|
||||
.join('\n\n');
|
||||
|
||||
return `You are a QA engineer evaluating the quality of a completed workflow step.
|
||||
|
||||
## Step Information
|
||||
- Step: ${step}
|
||||
- Duration: ${durationSec}s
|
||||
- Tools used: ${observations.tools.length}
|
||||
- Files created: ${observations.filesCreated.length}
|
||||
- Files modified: ${observations.filesModified.length}
|
||||
- Errors: ${observations.errors.length}
|
||||
|
||||
## Execution Details
|
||||
|
||||
### Tools Used
|
||||
${toolsSummary || '(none)'}
|
||||
|
||||
### Files Changed
|
||||
${filesSummary || '(none)'}
|
||||
|
||||
${observations.questionsAsked.length > 0 ? `### Questions & Decisions\n${questionsSummary}` : ''}
|
||||
|
||||
${observations.errors.length > 0 ? `### Errors Encountered\n${observations.errors.join('\n')}` : ''}
|
||||
|
||||
## Evaluation Criteria
|
||||
|
||||
**Key Artifacts Expected:**
|
||||
${criteria.keyArtifacts.map((a) => `- ${a}`).join('\n')}
|
||||
|
||||
**Quality Checks:**
|
||||
${criteria.qualityChecks.map((c) => `- ${c}`).join('\n')}
|
||||
|
||||
**Common Pitfalls to Watch For:**
|
||||
${criteria.commonPitfalls.map((p) => `- ${p}`).join('\n')}
|
||||
|
||||
**Scoring Guidance:**
|
||||
${criteria.scoringGuidance}
|
||||
|
||||
## Your Task
|
||||
|
||||
Evaluate this step honestly and thoroughly:
|
||||
|
||||
1. **Inspect the artifacts** using Read, Glob, and Grep tools
|
||||
2. **Check against quality criteria** listed above
|
||||
3. **Identify strengths and weaknesses** based on actual evidence
|
||||
4. **Provide constructive suggestions** for improvement
|
||||
5. **Assign an honest score** (0-100) following the guidance
|
||||
|
||||
Be critical but fair. Most work scores 70-85. Don't inflate scores.
|
||||
|
||||
## Response Format
|
||||
|
||||
SCORE: <number 0-100>
|
||||
|
||||
STRENGTHS:
|
||||
- <strength 1>
|
||||
- <strength 2>
|
||||
- <strength 3>
|
||||
|
||||
WEAKNESSES:
|
||||
- <weakness 1>
|
||||
- <weakness 2>
|
||||
|
||||
SUGGESTIONS:
|
||||
- <suggestion 1>
|
||||
- <suggestion 2>
|
||||
|
||||
CRITICAL_ISSUES:
|
||||
- <critical issue 1 (or "None")>
|
||||
|
||||
REASONING:
|
||||
<1-2 paragraphs explaining your evaluation, referencing specific files/evidence>
|
||||
|
||||
Provide honest, evidence-based evaluation.`;
|
||||
}
|
||||
|
||||
function parseEvaluationOutput(
|
||||
output: string,
|
||||
step: StepName
|
||||
): EvaluationResult {
|
||||
// Extract score
|
||||
const scoreMatch = output.match(/SCORE:\s*(\d+)/i);
|
||||
if (!scoreMatch) {
|
||||
console.warn(`Evaluation parser: no SCORE found in output (${output.length} chars). Defaulting to 50.`);
|
||||
}
|
||||
const score = scoreMatch ? parseInt(scoreMatch[1], 10) : 50;
|
||||
|
||||
// Extract sections
|
||||
const strengthsMatch = output.match(
|
||||
/STRENGTHS:\s*((?:- .+\n?)+)/i
|
||||
);
|
||||
const weaknessesMatch = output.match(
|
||||
/WEAKNESSES:\s*((?:- .+\n?)+)/i
|
||||
);
|
||||
const suggestionsMatch = output.match(
|
||||
/SUGGESTIONS:\s*((?:- .+\n?)+)/i
|
||||
);
|
||||
const criticalMatch = output.match(
|
||||
/CRITICAL_ISSUES:\s*((?:- .+\n?)+)/i
|
||||
);
|
||||
const reasoningMatch = output.match(/REASONING:\s*(.+?)(?=\n\n|$)/is);
|
||||
|
||||
const parseList = (text: string | undefined): string[] => {
|
||||
if (!text) return [];
|
||||
return text
|
||||
.split('\n')
|
||||
.map((line) => line.replace(/^-\s*/, '').trim())
|
||||
.filter((line) => line.length > 0 && !line.toLowerCase().includes('none'));
|
||||
};
|
||||
|
||||
return {
|
||||
step,
|
||||
score: Math.max(0, Math.min(100, score)),
|
||||
strengths: parseList(strengthsMatch?.[1]),
|
||||
weaknesses: parseList(weaknessesMatch?.[1]),
|
||||
suggestions: parseList(suggestionsMatch?.[1]),
|
||||
criticalIssues: parseList(criticalMatch?.[1]),
|
||||
timestamp: Date.now(),
|
||||
evaluatorModel: 'claude-sonnet-4-5',
|
||||
reasoning: reasoningMatch?.[1]?.trim() || 'No reasoning provided',
|
||||
};
|
||||
}
|
||||
|
||||
function createFallbackEvaluation(
|
||||
step: StepName,
|
||||
observations: ExecutionObservation
|
||||
): EvaluationResult {
|
||||
return {
|
||||
step,
|
||||
score: 50,
|
||||
strengths: ['Step completed'],
|
||||
weaknesses: ['Evaluation failed - using fallback'],
|
||||
suggestions: ['Re-run with evaluation enabled'],
|
||||
criticalIssues: ['Evaluation system unavailable'],
|
||||
timestamp: Date.now(),
|
||||
evaluatorModel: 'fallback',
|
||||
reasoning: 'Evaluation failed, using fallback. Cannot provide detailed assessment.',
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the active spec subdirectory (e.g. specs/<feature>/).
|
||||
* Falls back to projectDir if no spec folder exists yet (e.g. during init).
|
||||
*/
|
||||
function resolveOutputDir(projectDir: string): string {
|
||||
const specsDir = path.join(projectDir, 'specs');
|
||||
if (fs.existsSync(specsDir)) {
|
||||
const entries = fs.readdirSync(specsDir, { withFileTypes: true });
|
||||
const firstSpec = entries.find((e) => e.isDirectory());
|
||||
if (firstSpec) {
|
||||
return path.join(specsDir, firstSpec.name);
|
||||
}
|
||||
}
|
||||
return projectDir;
|
||||
}
|
||||
|
||||
async function writeEvaluationFiles(
|
||||
evaluation: EvaluationResult,
|
||||
observations: ExecutionObservation,
|
||||
projectDir: string
|
||||
): Promise<void> {
|
||||
const outputDir = resolveOutputDir(projectDir);
|
||||
|
||||
// Write BOT-EVALUATION.md
|
||||
const evalContent = formatEvaluationMarkdown(evaluation);
|
||||
const evalPath = path.join(outputDir, 'BOT-EVALUATION.md');
|
||||
|
||||
if (fs.existsSync(evalPath)) {
|
||||
// Append to existing file
|
||||
const existing = fs.readFileSync(evalPath, 'utf-8');
|
||||
fs.writeFileSync(evalPath, existing + '\n\n---\n\n' + evalContent);
|
||||
} else {
|
||||
fs.writeFileSync(evalPath, evalContent);
|
||||
}
|
||||
|
||||
// Write BOT-NOTES.md
|
||||
const notesContent = formatObservationsMarkdown(observations);
|
||||
const notesPath = path.join(outputDir, 'BOT-NOTES.md');
|
||||
|
||||
if (fs.existsSync(notesPath)) {
|
||||
const existing = fs.readFileSync(notesPath, 'utf-8');
|
||||
fs.writeFileSync(notesPath, existing + '\n\n---\n\n' + notesContent);
|
||||
} else {
|
||||
fs.writeFileSync(notesPath, notesContent);
|
||||
}
|
||||
}
|
||||
|
||||
function formatEvaluationMarkdown(evaluation: EvaluationResult): string {
|
||||
return `# Evaluation: ${evaluation.step} Step
|
||||
|
||||
**Score:** ${evaluation.score}/100
|
||||
**Timestamp:** ${new Date(evaluation.timestamp).toISOString()}
|
||||
**Evaluator:** ${evaluation.evaluatorModel}
|
||||
|
||||
## Strengths
|
||||
${evaluation.strengths.map((s) => `- ${s}`).join('\n') || '(none)'}
|
||||
|
||||
## Weaknesses
|
||||
${evaluation.weaknesses.map((w) => `- ${w}`).join('\n') || '(none)'}
|
||||
|
||||
## Suggestions for Improvement
|
||||
${evaluation.suggestions.map((s) => `- ${s}`).join('\n') || '(none)'}
|
||||
|
||||
${evaluation.criticalIssues.length > 0 ? `## Critical Issues\n${evaluation.criticalIssues.map((i) => `- ${i}`).join('\n')}\n` : ''}
|
||||
|
||||
## Reasoning
|
||||
${evaluation.reasoning}`;
|
||||
}
|
||||
|
||||
function formatObservationsMarkdown(observations: ExecutionObservation): string {
|
||||
const duration = observations.endTime - observations.startTime;
|
||||
const durationSec = Math.floor(duration / 1000);
|
||||
|
||||
return `# Observations: ${observations.step} Step
|
||||
|
||||
**Duration:** ${durationSec}s
|
||||
**Tools Used:** ${observations.tools.length}
|
||||
**Files Created:** ${observations.filesCreated.length}
|
||||
**Files Modified:** ${observations.filesModified.length}
|
||||
**Errors:** ${observations.errors.length}
|
||||
|
||||
${observations.questionsAsked.length > 0 ? `## Questions Asked & Answers\n\n${observations.questionsAsked.map((q) => `### Question: "${q.question}"\n**Selected:** "${q.selectedAnswer}"\n**Reasoning:** ${q.llmReasoning}`).join('\n\n')}\n` : ''}
|
||||
|
||||
## Tool Usage
|
||||
${observations.tools.map((t) => `- ${t.toolName}`).join('\n')}
|
||||
|
||||
## Files Created
|
||||
${observations.filesCreated.map((f) => `- ${f}`).join('\n') || '(none)'}
|
||||
|
||||
## Files Modified
|
||||
${observations.filesModified.map((f) => `- ${f}`).join('\n') || '(none)'}
|
||||
|
||||
${observations.assistantMessages.length > 0 ? `## Assistant Output Summary\n${observations.assistantMessages.slice(0, 3).map((m) => `> ${m.substring(0, 100)}...`).join('\n')}\n` : ''}`;
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
import { query } from '@anthropic-ai/claude-agent-sdk';
|
||||
|
||||
interface IdeaResult {
|
||||
name: string;
|
||||
description: string;
|
||||
}
|
||||
|
||||
function parseIdea(text: string): IdeaResult {
|
||||
const nameMatch = text.match(/NAME:\s*(.+)/i);
|
||||
const descMatch = text.match(/DESCRIPTION:\s*([\s\S]+?)(?:\n\n|$)/i);
|
||||
|
||||
const name = nameMatch?.[1]?.trim() ?? 'auto-project';
|
||||
const description = descMatch?.[1]?.trim() ?? text.trim();
|
||||
|
||||
// Ensure kebab-case
|
||||
const kebabName = name
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9]+/g, '-')
|
||||
.replace(/^-|-$/g, '');
|
||||
|
||||
return { name: kebabName, description };
|
||||
}
|
||||
|
||||
export async function generateNewAppIdea(seed?: string): Promise<IdeaResult> {
|
||||
const categories = [
|
||||
'CLI tool',
|
||||
'single-page web app',
|
||||
'REST API service',
|
||||
'browser extension',
|
||||
'interactive data visualization dashboard',
|
||||
'terminal-based game',
|
||||
'real-time web app (using WebSockets)',
|
||||
'static site generator or theme',
|
||||
'browser-based game',
|
||||
'desktop utility (using Electron or Tauri)',
|
||||
'chat bot or conversational tool',
|
||||
'automation script or workflow tool',
|
||||
];
|
||||
const category = categories[Math.floor(Math.random() * categories.length)];
|
||||
|
||||
const seedClause = seed
|
||||
? `\n\nThe user provided this seed for inspiration. Stay closely aligned with the theme and intent of the seed — build on it, don't ignore it:\n"${seed}"`
|
||||
: '';
|
||||
|
||||
const prompt = `Generate a random, creative idea for a ${category}. The project should be achievable in a single coding session (1-2 hours) and should be interesting but not overly complex.
|
||||
|
||||
IMPORTANT: Be creative and diverse with your ideas. Avoid defaulting to developer-centric tools (git analyzers, code formatters, repo scanners, etc.) unless the category specifically calls for it. Think about ideas that would appeal to a broad audience — productivity, entertainment, education, health, finance, art, music, social, cooking, travel, fitness, etc.${seedClause}
|
||||
|
||||
Respond in EXACTLY this format (no other text):
|
||||
NAME: <kebab-case-project-name>
|
||||
DESCRIPTION: <2-3 sentence description of what the app does, its key features, and the tech stack to use>`;
|
||||
|
||||
const session = query({
|
||||
prompt,
|
||||
options: {
|
||||
maxTurns: 1,
|
||||
systemPrompt: 'You are a wildly creative project idea generator. You come up with surprising, fun, and diverse software project ideas spanning many domains — not just developer tools. Respond only in the exact format requested.',
|
||||
},
|
||||
});
|
||||
|
||||
let output = '';
|
||||
for await (const message of session) {
|
||||
if (message.type === 'assistant') {
|
||||
for (const block of message.message.content) {
|
||||
if (block.type === 'text') {
|
||||
output += block.text;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return parseIdea(output);
|
||||
}
|
||||
|
||||
export async function generateEnhancementIdea(projectDir: string, seed?: string): Promise<IdeaResult> {
|
||||
const seedClause = seed
|
||||
? `\n\nUse this as inspiration for the enhancement: "${seed}"`
|
||||
: '';
|
||||
|
||||
const prompt = `You are in a project directory. Scan the existing codebase to understand what it does, then propose a realistic enhancement (new feature, refactor, improvement, or extension).
|
||||
|
||||
Use the Read, Glob, and Grep tools to explore the project. Look at:
|
||||
- Package.json or similar config files for project info
|
||||
- Source files for current functionality
|
||||
- README or docs for context${seedClause}
|
||||
|
||||
Then respond in EXACTLY this format (no other text):
|
||||
NAME: <kebab-case-enhancement-name>
|
||||
DESCRIPTION: <2-3 sentence description of the enhancement, what it adds/changes, and why it would be valuable>`;
|
||||
|
||||
const session = query({
|
||||
prompt,
|
||||
options: {
|
||||
maxTurns: 8,
|
||||
cwd: projectDir,
|
||||
tools: { type: 'preset', preset: 'claude_code' },
|
||||
allowedTools: ['Read', 'Glob', 'Grep'],
|
||||
systemPrompt: { type: 'preset', preset: 'claude_code' },
|
||||
},
|
||||
});
|
||||
|
||||
let output = '';
|
||||
for await (const message of session) {
|
||||
if (message.type === 'assistant') {
|
||||
for (const block of message.message.content) {
|
||||
if (block.type === 'text') {
|
||||
output += block.text;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return parseIdea(output);
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
export { runCLI } from './cli.js';
|
||||
export type { BotConfig, BotMode, BotState, StepName, StepResult } from './types.js';
|
||||
@@ -0,0 +1,243 @@
|
||||
import { query } from '@anthropic-ai/claude-agent-sdk';
|
||||
import type { BotConfig, ExecutionObservation, StepName } from './types.js';
|
||||
import type { ObservationCollector } from './observation-collector.js';
|
||||
|
||||
interface AskUserQuestionInput {
|
||||
questions: Array<{
|
||||
question: string;
|
||||
options: Array<{ label: string; description: string }>;
|
||||
multiSelect?: boolean;
|
||||
}>;
|
||||
}
|
||||
|
||||
interface IntelligentAnswers {
|
||||
answers: Record<string, string>;
|
||||
reasoning: Record<string, string>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates an intelligent responder that uses LLM-as-judge for ALL decisions.
|
||||
* Replaces the old hardcoded auto-responder logic.
|
||||
*/
|
||||
export function createIntelligentResponder(
|
||||
config: BotConfig,
|
||||
step: StepName,
|
||||
collector: ObservationCollector
|
||||
) {
|
||||
return async (
|
||||
toolName: string,
|
||||
input: Record<string, unknown>
|
||||
): Promise<{ behavior: 'allow'; updatedInput?: Record<string, unknown> } | { behavior: 'deny'; message: string }> => {
|
||||
// Record every tool invocation for observations
|
||||
collector.recordToolUse(toolName, input, undefined, true);
|
||||
|
||||
// Handle AskUserQuestion with LLM-generated answers
|
||||
if (toolName === 'AskUserQuestion') {
|
||||
const askInput = input as unknown as AskUserQuestionInput;
|
||||
|
||||
// Get current observations to provide context to LLM
|
||||
const observations = collector.getSnapshot();
|
||||
|
||||
// Generate answers using LLM-as-judge
|
||||
const answers = await generateIntelligentAnswers(
|
||||
askInput,
|
||||
observations,
|
||||
config,
|
||||
step
|
||||
);
|
||||
|
||||
// Record each question/answer pair for metrics
|
||||
for (const q of askInput.questions) {
|
||||
const answer = answers.answers[q.question];
|
||||
const reasoning = answers.reasoning[q.question] || 'No reasoning provided';
|
||||
collector.recordQuestion(q.question, q.options, answer, reasoning);
|
||||
}
|
||||
|
||||
return {
|
||||
behavior: 'allow',
|
||||
updatedInput: { ...input, answers: answers.answers },
|
||||
};
|
||||
}
|
||||
|
||||
// Allow all other tools
|
||||
return { behavior: 'allow' };
|
||||
};
|
||||
}
|
||||
|
||||
async function generateIntelligentAnswers(
|
||||
askInput: AskUserQuestionInput,
|
||||
observations: ExecutionObservation,
|
||||
config: BotConfig,
|
||||
step: StepName
|
||||
): Promise<IntelligentAnswers> {
|
||||
const prompt = buildDecisionPrompt(askInput, observations, config, step);
|
||||
|
||||
try {
|
||||
// Query LLM for decision (single turn, read-only tools)
|
||||
const session = query({
|
||||
prompt,
|
||||
options: {
|
||||
maxTurns: 1,
|
||||
cwd: config.projectDir,
|
||||
permissionMode: 'bypassPermissions',
|
||||
allowDangerouslySkipPermissions: true,
|
||||
allowedTools: ['Read', 'Glob', 'Grep'],
|
||||
systemPrompt: 'You are a QA engineer reviewing work-in-progress. Be thoughtful and honest.',
|
||||
},
|
||||
});
|
||||
|
||||
let output = '';
|
||||
for await (const message of session) {
|
||||
if (message.type === 'assistant') {
|
||||
for (const block of message.message.content) {
|
||||
if (block.type === 'text') output += block.text;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return parseDecisionOutput(output, askInput);
|
||||
} catch (error) {
|
||||
console.warn('LLM decision failed, using fallback logic:', error);
|
||||
// Fallback to reasonable defaults if LLM fails
|
||||
return generateFallbackAnswers(askInput, step);
|
||||
}
|
||||
}
|
||||
|
||||
function buildDecisionPrompt(
|
||||
askInput: AskUserQuestionInput,
|
||||
observations: ExecutionObservation,
|
||||
config: BotConfig,
|
||||
step: StepName
|
||||
): string {
|
||||
const duration = observations.endTime - observations.startTime;
|
||||
const toolSummary = observations.tools
|
||||
.map((t) => `- ${t.toolName}`)
|
||||
.join('\n');
|
||||
const filesSummary = [
|
||||
...observations.filesCreated.map((f) => `CREATED: ${f}`),
|
||||
...observations.filesModified.map((f) => `MODIFIED: ${f}`),
|
||||
].join('\n');
|
||||
|
||||
const questionsText = askInput.questions
|
||||
.map((q, i) => {
|
||||
const optionsText = q.options
|
||||
.map((o, j) => ` ${j + 1}. ${o.label} - ${o.description}`)
|
||||
.join('\n');
|
||||
return `QUESTION ${i + 1}: ${q.question}\nOptions:\n${optionsText}`;
|
||||
})
|
||||
.join('\n\n');
|
||||
|
||||
return `You are a QA engineer reviewing a workflow step in progress.
|
||||
|
||||
## Context
|
||||
- Step: ${step}
|
||||
- Mode: ${config.mode}
|
||||
- Duration so far: ${Math.floor(duration / 1000)}s
|
||||
- Tools used: ${observations.tools.length}
|
||||
- Errors encountered: ${observations.errors.length}
|
||||
|
||||
## What's Happened So Far
|
||||
|
||||
### Tools Used
|
||||
${toolSummary || '(none yet)'}
|
||||
|
||||
### Files Changed
|
||||
${filesSummary || '(none yet)'}
|
||||
|
||||
${observations.errors.length > 0 ? `### Errors\n${observations.errors.join('\n')}` : ''}
|
||||
|
||||
## Questions to Answer
|
||||
|
||||
${questionsText}
|
||||
|
||||
## Your Task
|
||||
|
||||
You need to answer these questions as a thoughtful QA engineer would:
|
||||
1. Use Read, Glob, and Grep tools to inspect the current state of artifacts if needed
|
||||
2. Consider what you've observed (tools used, files created, errors)
|
||||
3. For each question, select the most appropriate answer
|
||||
4. Provide brief reasoning for your choice
|
||||
|
||||
Respond in this format:
|
||||
|
||||
QUESTION 1:
|
||||
ANSWER: <option label>
|
||||
REASONING: <1-2 sentences explaining your choice>
|
||||
|
||||
QUESTION 2:
|
||||
ANSWER: <option label>
|
||||
REASONING: <1-2 sentences>
|
||||
|
||||
Be honest. If work looks incomplete or problematic, don't approve it.
|
||||
If tests should be run but haven't been, don't skip them without good reason.
|
||||
Act like a real developer who cares about quality.`;
|
||||
}
|
||||
|
||||
function parseDecisionOutput(
|
||||
output: string,
|
||||
askInput: AskUserQuestionInput
|
||||
): IntelligentAnswers {
|
||||
const answers: Record<string, string> = {};
|
||||
const reasoning: Record<string, string> = {};
|
||||
|
||||
// Parse structured output
|
||||
const questionBlocks = output.split(/QUESTION \d+:/i).slice(1);
|
||||
|
||||
askInput.questions.forEach((q, i) => {
|
||||
const block = questionBlocks[i] || '';
|
||||
|
||||
const answerMatch = block.match(/ANSWER:\s*(.+?)(?=\n|$)/i);
|
||||
const reasoningMatch = block.match(/REASONING:\s*(.+?)(?=\n\n|$)/is);
|
||||
|
||||
const selectedLabel = answerMatch?.[1]?.trim() || '';
|
||||
|
||||
// Find matching option by label (case-insensitive partial match)
|
||||
const matchedOption = q.options.find(
|
||||
(opt) =>
|
||||
opt.label.toLowerCase().includes(selectedLabel.toLowerCase()) ||
|
||||
selectedLabel.toLowerCase().includes(opt.label.toLowerCase())
|
||||
);
|
||||
|
||||
answers[q.question] = matchedOption?.label || q.options[0].label;
|
||||
reasoning[q.question] = reasoningMatch?.[1]?.trim() || 'No reasoning provided';
|
||||
});
|
||||
|
||||
return { answers, reasoning };
|
||||
}
|
||||
|
||||
function generateFallbackAnswers(
|
||||
askInput: AskUserQuestionInput,
|
||||
step: StepName
|
||||
): IntelligentAnswers {
|
||||
// Simple fallback: pick first option for most questions
|
||||
// For approval questions, approve; for testing, skip
|
||||
const answers: Record<string, string> = {};
|
||||
const reasoning: Record<string, string> = {};
|
||||
|
||||
for (const q of askInput.questions) {
|
||||
const questionLower = q.question.toLowerCase();
|
||||
|
||||
if (questionLower.includes('approve') || questionLower.includes('proceed')) {
|
||||
const approveOption = q.options.find(
|
||||
(o) =>
|
||||
o.label.toLowerCase().includes('approve') ||
|
||||
o.label.toLowerCase().includes('yes')
|
||||
);
|
||||
answers[q.question] = approveOption?.label || q.options[0].label;
|
||||
reasoning[q.question] = 'Fallback approval (LLM unavailable)';
|
||||
} else if (questionLower.includes('test')) {
|
||||
const skipOption = q.options.find(
|
||||
(o) =>
|
||||
o.label.toLowerCase().includes('skip') ||
|
||||
o.label.toLowerCase().includes('none')
|
||||
);
|
||||
answers[q.question] = skipOption?.label || q.options[0].label;
|
||||
reasoning[q.question] = 'Fallback skip (LLM unavailable)';
|
||||
} else {
|
||||
answers[q.question] = q.options[0].label;
|
||||
reasoning[q.question] = 'Fallback first option (LLM unavailable)';
|
||||
}
|
||||
}
|
||||
|
||||
return { answers, reasoning };
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
import type { ExecutionObservation, StepName } from './types.js';
|
||||
|
||||
/**
|
||||
* Collects detailed observations during step execution.
|
||||
* Tracks tool usage, messages, errors, file changes, and questions asked.
|
||||
*/
|
||||
export class ObservationCollector {
|
||||
private observations: ExecutionObservation;
|
||||
private recentToolKeys: Set<string> = new Set();
|
||||
|
||||
constructor(step: StepName) {
|
||||
this.observations = {
|
||||
step,
|
||||
startTime: Date.now(),
|
||||
endTime: 0,
|
||||
tools: [],
|
||||
assistantMessages: [],
|
||||
errors: [],
|
||||
filesCreated: [],
|
||||
filesModified: [],
|
||||
questionsAsked: [],
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Records a tool invocation with its input and output.
|
||||
* Deduplicates if the same tool+input is recorded from both canUseTool and the message stream.
|
||||
*/
|
||||
recordToolUse(
|
||||
toolName: string,
|
||||
input: Record<string, unknown>,
|
||||
output: unknown,
|
||||
allowed: boolean
|
||||
): void {
|
||||
// Deduplicate based on tool name + serialized input (within a short time window)
|
||||
const key = `${toolName}:${JSON.stringify(input)}`;
|
||||
if (this.recentToolKeys.has(key)) {
|
||||
return;
|
||||
}
|
||||
this.recentToolKeys.add(key);
|
||||
// Clean up old keys periodically to avoid unbounded growth
|
||||
if (this.recentToolKeys.size > 500) {
|
||||
const entries = [...this.recentToolKeys];
|
||||
this.recentToolKeys = new Set(entries.slice(entries.length - 250));
|
||||
}
|
||||
|
||||
this.observations.tools.push({
|
||||
toolName,
|
||||
input,
|
||||
output,
|
||||
timestamp: Date.now(),
|
||||
allowed,
|
||||
autoAnswered: toolName === 'AskUserQuestion',
|
||||
});
|
||||
|
||||
// Extract file paths from common tools
|
||||
if (toolName === 'Write' && input.file_path) {
|
||||
this.observations.filesCreated.push(input.file_path as string);
|
||||
}
|
||||
if (toolName === 'Edit' && input.file_path) {
|
||||
this.observations.filesModified.push(input.file_path as string);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Records messages from the session (assistant text, errors).
|
||||
*/
|
||||
recordMessage(message: any): void {
|
||||
if (message.type === 'assistant') {
|
||||
for (const block of message.message.content) {
|
||||
if (block.type === 'text') {
|
||||
this.observations.assistantMessages.push(block.text);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (message.type === 'error') {
|
||||
this.observations.errors.push(message.error?.message ?? 'Unknown error');
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Records a question that was asked and the LLM-generated answer.
|
||||
*/
|
||||
recordQuestion(
|
||||
question: string,
|
||||
options: Array<{ label: string; description: string }>,
|
||||
selectedAnswer: string,
|
||||
llmReasoning: string
|
||||
): void {
|
||||
this.observations.questionsAsked.push({
|
||||
question,
|
||||
options,
|
||||
llmReasoning,
|
||||
selectedAnswer,
|
||||
timestamp: Date.now(),
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Gets a snapshot of current observations (for real-time decision making).
|
||||
*/
|
||||
getSnapshot(): ExecutionObservation {
|
||||
return { ...this.observations, endTime: Date.now() };
|
||||
}
|
||||
|
||||
/**
|
||||
* Finalizes observations and returns the complete record.
|
||||
*/
|
||||
finalize(): ExecutionObservation {
|
||||
this.observations.endTime = Date.now();
|
||||
return { ...this.observations };
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,158 @@
|
||||
import type { StepName } from '../types.js';
|
||||
|
||||
export interface StepEvaluationCriteria {
|
||||
step: StepName;
|
||||
keyArtifacts: string[];
|
||||
qualityChecks: string[];
|
||||
commonPitfalls: string[];
|
||||
scoringGuidance: string;
|
||||
}
|
||||
|
||||
export const EVALUATION_CRITERIA: Record<StepName, StepEvaluationCriteria> = {
|
||||
init: {
|
||||
step: 'init',
|
||||
keyArtifacts: ['AGENTS.md', 'IDEA.md'],
|
||||
qualityChecks: [
|
||||
'AGENTS.md exists with project name and description',
|
||||
'AGENTS.md includes a brief intro line and a Status section indicating the project is in planning phase',
|
||||
'AGENTS.md does NOT contain hallucinated architecture, commands, file structures, or tech stack details',
|
||||
'No .agents-docs/ directory was created (too early for detail files)',
|
||||
'No project scaffolding (package.json, dependencies, src/) was created',
|
||||
'IDEA.md exists with the project idea',
|
||||
],
|
||||
commonPitfalls: [
|
||||
'Hallucinating architecture or tech stack details before the plan step',
|
||||
'Creating .agents-docs/ detail files with invented content',
|
||||
'Scaffolding project files or installing dependencies prematurely',
|
||||
],
|
||||
scoringGuidance: `
|
||||
100 = Perfect: AGENTS.md stub with intro line, project name/description, and status section. IDEA.md present. Nothing else created.
|
||||
85-95 = Good but minor extra content beyond the expected stub format (e.g., an extra placeholder heading)
|
||||
70-84 = AGENTS.md exists but includes some hallucinated details (e.g., assumed tech stack or commands)
|
||||
50-69 = Significant hallucination (e.g., .agents-docs/ created with invented content, project scaffolded)
|
||||
<50 = Major problems (e.g., AGENTS.md missing, full project structure hallucinated)
|
||||
|
||||
The expected AGENTS.md format is: intro line, Project Overview (name + description), and a Status section. This is the target for a 100 score.
|
||||
`,
|
||||
},
|
||||
|
||||
plan: {
|
||||
step: 'plan',
|
||||
keyArtifacts: ['specs/*/PLAN-DRAFT-*.md', 'IDEA.md'],
|
||||
qualityChecks: [
|
||||
'Plan breaks work into clear, achievable phases',
|
||||
'Each phase has specific goals and deliverables',
|
||||
'Technical approach is appropriate',
|
||||
'Scope is realistic for the idea',
|
||||
'Dependencies between phases are identified',
|
||||
],
|
||||
commonPitfalls: [
|
||||
'Phases too vague ("polish the app")',
|
||||
'Missing specific tasks within phases',
|
||||
'No testing strategy mentioned',
|
||||
'Overly ambitious scope',
|
||||
'Missing file paths or specific actions',
|
||||
],
|
||||
scoringGuidance: `
|
||||
100 = Exceptional plan: detailed phases, realistic scope, clear tasks, testing included
|
||||
85-95 = Good plan with minor improvements possible (e.g., one phase could be more specific)
|
||||
70-84 = Acceptable but has vague sections or missing testing strategy
|
||||
50-69 = Significant issues (e.g., multiple vague phases, unrealistic scope)
|
||||
<50 = Major problems (e.g., no clear phases, plan doesn't match idea)
|
||||
|
||||
Most plans should score 70-85. Be critical of vague language.
|
||||
`,
|
||||
},
|
||||
|
||||
document: {
|
||||
step: 'document',
|
||||
keyArtifacts: ['specs/*/overview.md', 'specs/*/phase-*.md files'],
|
||||
qualityChecks: [
|
||||
'overview.md provides clear project summary',
|
||||
'Each phase file has specific tasks with checkboxes',
|
||||
'File paths are explicit (not generic)',
|
||||
'Dependencies between tasks are identified',
|
||||
'Technical details are specific',
|
||||
'Acceptance criteria are clear',
|
||||
],
|
||||
commonPitfalls: [
|
||||
'Tasks too generic ("implement feature X")',
|
||||
'Missing file paths',
|
||||
'No checkboxes or unclear task structure',
|
||||
'Missing dependencies',
|
||||
'Overly verbose or lacking specifics',
|
||||
],
|
||||
scoringGuidance: `
|
||||
100 = Exceptional documentation: specific tasks, explicit file paths, clear dependencies
|
||||
85-95 = Good documentation with minor vagueness in one or two tasks
|
||||
70-84 = Acceptable but multiple tasks lack specifics or file paths
|
||||
50-69 = Significant issues (e.g., many generic tasks, missing file paths)
|
||||
<50 = Major problems (e.g., tasks don't match plan, fundamentally vague)
|
||||
|
||||
Most documentation should score 70-85. Penalize generic language heavily.
|
||||
`,
|
||||
},
|
||||
|
||||
implement: {
|
||||
step: 'implement',
|
||||
keyArtifacts: ['actual code files', 'checked-off tasks in phase files'],
|
||||
qualityChecks: [
|
||||
'Phase tasks are being completed',
|
||||
'Code files are actually created/modified',
|
||||
'Implementation follows the documented plan',
|
||||
'No major errors blocking progress',
|
||||
'Tests are written (if applicable)',
|
||||
],
|
||||
commonPitfalls: [
|
||||
'Tasks marked complete but files not actually changed',
|
||||
'Implementation deviates significantly from plan',
|
||||
'Errors not addressed',
|
||||
'Skipping tests without justification',
|
||||
'Working on wrong phase',
|
||||
],
|
||||
scoringGuidance: `
|
||||
100 = Exceptional implementation: all tasks complete, code works, tests pass
|
||||
85-95 = Good implementation with minor issues or incomplete tests
|
||||
70-84 = Acceptable but some tasks incomplete or code has issues
|
||||
50-69 = Significant issues (e.g., many tasks incomplete, code doesn't work)
|
||||
<50 = Major problems (e.g., wrong phase, no actual work done)
|
||||
|
||||
Implementation scoring depends heavily on actual progress. Be realistic.
|
||||
`,
|
||||
},
|
||||
|
||||
finalize: {
|
||||
step: 'finalize',
|
||||
keyArtifacts: ['specs--completed/', 'README or docs', 'final code state'],
|
||||
qualityChecks: [
|
||||
'All phases are marked complete',
|
||||
'Spec moved to specs--completed/',
|
||||
'Documentation is updated',
|
||||
'Code is in working state',
|
||||
'No obvious loose ends',
|
||||
],
|
||||
commonPitfalls: [
|
||||
'Incomplete phases',
|
||||
'Missing specs--completed/ move',
|
||||
'Documentation not updated',
|
||||
'Code broken or incomplete',
|
||||
'Unrealistic self-assessment',
|
||||
],
|
||||
scoringGuidance: `
|
||||
100 = Exceptional finalization: everything complete, polished, documented
|
||||
85-95 = Good finalization with minor issues
|
||||
70-84 = Acceptable but some loose ends or documentation gaps
|
||||
50-69 = Significant issues (e.g., incomplete phases, broken code)
|
||||
<50 = Major problems (e.g., work not actually done, fundamentally incomplete)
|
||||
|
||||
Finalize scores should reflect overall project quality. Be honest.
|
||||
`,
|
||||
},
|
||||
};
|
||||
|
||||
/**
|
||||
* Gets evaluation criteria for a specific step.
|
||||
*/
|
||||
export function getCriteriaForStep(step: StepName): StepEvaluationCriteria {
|
||||
return EVALUATION_CRITERIA[step];
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { buildStepPrompt } from './step-instructions.js';
|
||||
import type { BotConfig } from '../types.js';
|
||||
|
||||
const config: BotConfig = {
|
||||
workDir: '/tmp/work',
|
||||
projectDir: '/tmp/work/my-app',
|
||||
ideaDescription: 'A todo app with drag-and-drop',
|
||||
ideaName: 'drag-todo',
|
||||
mode: 'new-project',
|
||||
};
|
||||
|
||||
describe('buildStepPrompt', () => {
|
||||
it('each step includes the autonomous preamble', () => {
|
||||
const steps = ['init', 'plan', 'document', 'implement', 'finalize'] as const;
|
||||
for (const step of steps) {
|
||||
const prompt = buildStepPrompt(step, config);
|
||||
expect(prompt).toContain('running autonomously');
|
||||
}
|
||||
});
|
||||
|
||||
it('init prompt includes /plan2code-init skill invocation', () => {
|
||||
const prompt = buildStepPrompt('init', config);
|
||||
expect(prompt).toContain('/plan2code-init');
|
||||
});
|
||||
|
||||
it('plan prompt includes /plan2code-1-plan skill invocation', () => {
|
||||
const prompt = buildStepPrompt('plan', config);
|
||||
expect(prompt).toContain('/plan2code-1-plan');
|
||||
});
|
||||
|
||||
it('document prompt includes /plan2code-2-document skill invocation', () => {
|
||||
const prompt = buildStepPrompt('document', config);
|
||||
expect(prompt).toContain('/plan2code-2-document');
|
||||
});
|
||||
|
||||
it('implement prompt instructs direct tool usage without skill invocation', () => {
|
||||
const prompt = buildStepPrompt('implement', config);
|
||||
expect(prompt).toContain('Do NOT use the Skill tool');
|
||||
});
|
||||
|
||||
it('finalize prompt includes /plan2code-4-finalize skill invocation', () => {
|
||||
const prompt = buildStepPrompt('finalize', config);
|
||||
expect(prompt).toContain('/plan2code-4-finalize');
|
||||
});
|
||||
|
||||
it('plan prompt includes project name and description', () => {
|
||||
const prompt = buildStepPrompt('plan', config);
|
||||
expect(prompt).toContain('drag-todo');
|
||||
expect(prompt).toContain('A todo app with drag-and-drop');
|
||||
});
|
||||
|
||||
it('init prompt includes project name and description', () => {
|
||||
const prompt = buildStepPrompt('init', config);
|
||||
expect(prompt).toContain('drag-todo');
|
||||
expect(prompt).toContain('A todo app with drag-and-drop');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,131 @@
|
||||
import type { BotConfig, StepName } from '../types.js';
|
||||
|
||||
const AUTONOMOUS_PREAMBLE = `You are running autonomously as part of an automated test pipeline. Do NOT pause for human input. When you encounter questions or approval gates, make reasonable decisions and proceed. If asked for confirmation, approve. If asked to choose, pick the most reasonable option. Complete the entire step without stopping.`;
|
||||
|
||||
export function buildStepPrompt(step: StepName, config: BotConfig): string {
|
||||
switch (step) {
|
||||
case 'init':
|
||||
return buildInitPrompt(config);
|
||||
case 'plan':
|
||||
return buildPlanPrompt(config);
|
||||
case 'document':
|
||||
return buildDocumentPrompt(config);
|
||||
case 'implement':
|
||||
return buildImplementPrompt(config);
|
||||
case 'finalize':
|
||||
return buildFinalizePrompt(config);
|
||||
}
|
||||
}
|
||||
|
||||
function buildInitPrompt(config: BotConfig): string {
|
||||
return `${AUTONOMOUS_PREAMBLE}
|
||||
|
||||
This is a brand new project with no existing code. Create a minimal stub AGENTS.md file and an IDEA.md file. Do NOT run the /plan2code-init skill — there is no codebase to analyze yet.
|
||||
|
||||
The project idea is: ${config.ideaDescription}
|
||||
The project name is: ${config.ideaName}
|
||||
|
||||
## What to create
|
||||
|
||||
### AGENTS.md
|
||||
Create a minimal stub with ONLY the following — do NOT invent architecture, tech stack details, commands, or file structures:
|
||||
|
||||
\`\`\`markdown
|
||||
# AGENTS.md
|
||||
|
||||
This file provides guidance to AI coding agents when working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
**Name:** ${config.ideaName}
|
||||
**Description:** ${config.ideaDescription}
|
||||
|
||||
## Status
|
||||
This project is in the planning phase. Architecture, commands, and detailed documentation will be added after the plan and document steps are complete.
|
||||
\`\`\`
|
||||
|
||||
### IDEA.md
|
||||
If IDEA.md does not already exist, create it with the project name and description.
|
||||
|
||||
## Rules
|
||||
- Do NOT create .agents-docs/ or any detail files — there is nothing to document yet
|
||||
- Do NOT hallucinate architecture, dependencies, file structures, or tech stack choices
|
||||
- Do NOT install dependencies or scaffold project files
|
||||
- ONLY create the two files above`;
|
||||
}
|
||||
|
||||
function buildPlanPrompt(config: BotConfig): string {
|
||||
return `${AUTONOMOUS_PREAMBLE}
|
||||
|
||||
Run the /plan2code-1-plan skill to create a plan for this project.
|
||||
|
||||
Read the IDEA.md file first to understand the project. The project is: ${config.ideaDescription}
|
||||
|
||||
When making decisions during planning:
|
||||
- Set confidence levels reasonably high (85-95%)
|
||||
- Accept the generated tech stack without revision
|
||||
- Do not request additional reference files
|
||||
- Approve the plan when asked for sign-off
|
||||
- Keep scope small and achievable (3-4 phases max)
|
||||
- Use the project name: ${config.ideaName}`;
|
||||
}
|
||||
|
||||
function buildDocumentPrompt(config: BotConfig): string {
|
||||
return `${AUTONOMOUS_PREAMBLE}
|
||||
|
||||
Run the /plan2code-2-document skill to transform the plan into implementation specs.
|
||||
|
||||
Read the existing plan output in specs/ first to understand what was planned.
|
||||
|
||||
When making decisions:
|
||||
- Accept all generated documentation
|
||||
- Approve phase breakdowns and task lists
|
||||
- Do not request changes to the generated docs`;
|
||||
}
|
||||
|
||||
function buildImplementPrompt(config: BotConfig): string {
|
||||
return `${AUTONOMOUS_PREAMBLE}
|
||||
|
||||
You are a senior software engineer implementing a project phase. Do NOT use the Skill tool — implement directly using Read, Write, Edit, Glob, and Grep tools.
|
||||
|
||||
## Process
|
||||
|
||||
1. **Find the spec**: Read \`specs/*/overview.md\` to find the phase checklist
|
||||
2. **Pick the next phase**: Find the first phase marked \`[ ]\` (pending) or \`[/]\` (in-progress)
|
||||
3. **Read the phase file**: Read the corresponding \`phase-X.md\` from the same directory
|
||||
4. **Mark phase in-progress**: Update \`[ ]\` to \`[/]\` in overview.md
|
||||
5. **Implement each task sequentially**:
|
||||
- Read the task specification completely
|
||||
- Write the code using Write or Edit tools — create real files, not code blocks
|
||||
- Mark the task \`[x]\` in the phase file immediately after completing it
|
||||
6. **Complete the phase**: After all tasks, fill in the "Phase Completion Summary" in the phase file
|
||||
7. **Mark phase complete**: Update \`[/]\` to \`[x]\` in overview.md
|
||||
|
||||
## Rules
|
||||
|
||||
- Follow AGENTS.md if it exists
|
||||
- Implement specs EXACTLY — no creative additions or unsolicited improvements
|
||||
- Write task completion status (\`[x]\`) to disk immediately after each task — never batch
|
||||
- Only create files mentioned in the spec tasks
|
||||
- Use the specified file paths, function names, and structures from the spec
|
||||
- No placeholder code — fully implement every function
|
||||
- Match existing codebase conventions
|
||||
- Do NOT run git commands
|
||||
- Skip running tests unless explicitly listed as a phase task
|
||||
|
||||
## Project info
|
||||
- Project: ${config.ideaName}
|
||||
- Description: ${config.ideaDescription}
|
||||
- Project directory: ${config.projectDir}`;
|
||||
}
|
||||
|
||||
function buildFinalizePrompt(config: BotConfig): string {
|
||||
return `${AUTONOMOUS_PREAMBLE}
|
||||
|
||||
Run the /plan2code-4-finalize skill to validate and archive the completed project.
|
||||
|
||||
When making decisions:
|
||||
- Approve all documentation updates
|
||||
- Accept the completion summary
|
||||
- If asked for a rating or feedback, give 8/10 and positive feedback
|
||||
- Complete the archival process fully`;
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
import { describe, it, expect, vi } from 'vitest';
|
||||
|
||||
// Mock the SDK before importing session-runner
|
||||
vi.mock('@anthropic-ai/claude-agent-sdk', () => ({
|
||||
query: vi.fn(),
|
||||
}));
|
||||
|
||||
import { query } from '@anthropic-ai/claude-agent-sdk';
|
||||
import { runSession } from './session-runner.js';
|
||||
import { ObservationCollector } from './observation-collector.js';
|
||||
import type { BotConfig } from './types.js';
|
||||
|
||||
const config: BotConfig = {
|
||||
workDir: '/tmp/work',
|
||||
projectDir: '/tmp/work/my-app',
|
||||
ideaDescription: 'test app',
|
||||
ideaName: 'test-app',
|
||||
mode: 'new-project',
|
||||
};
|
||||
|
||||
describe('runSession', () => {
|
||||
it('returns success: false when output is empty', async () => {
|
||||
// Simulate a session that yields an assistant message with empty text
|
||||
const mockQuery = vi.mocked(query);
|
||||
mockQuery.mockReturnValue(
|
||||
(async function* () {
|
||||
yield {
|
||||
type: 'assistant' as const,
|
||||
session_id: 'sess-1',
|
||||
message: { content: [{ type: 'text' as const, text: '' }] },
|
||||
};
|
||||
yield {
|
||||
type: 'result' as const,
|
||||
session_id: 'sess-1',
|
||||
};
|
||||
})() as any,
|
||||
);
|
||||
|
||||
const collector = new ObservationCollector('init');
|
||||
const result = await runSession({
|
||||
prompt: 'do something',
|
||||
config,
|
||||
step: 'init',
|
||||
collector,
|
||||
});
|
||||
|
||||
expect(result.success).toBe(false);
|
||||
expect(result.output.trim()).toBe('');
|
||||
expect(result.observations).toBeDefined();
|
||||
});
|
||||
|
||||
it('returns success: true when output has content', async () => {
|
||||
const mockQuery = vi.mocked(query);
|
||||
mockQuery.mockReturnValue(
|
||||
(async function* () {
|
||||
yield {
|
||||
type: 'assistant' as const,
|
||||
session_id: 'sess-2',
|
||||
message: { content: [{ type: 'text' as const, text: 'AGENTS.md has been created successfully' }] },
|
||||
};
|
||||
yield {
|
||||
type: 'result' as const,
|
||||
session_id: 'sess-2',
|
||||
};
|
||||
})() as any,
|
||||
);
|
||||
|
||||
const collector = new ObservationCollector('init');
|
||||
const result = await runSession({
|
||||
prompt: 'do something',
|
||||
config,
|
||||
step: 'init',
|
||||
collector,
|
||||
});
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
expect(result.output).toContain('AGENTS.md');
|
||||
expect(result.sessionId).toBe('sess-2');
|
||||
expect(result.observations).toBeDefined();
|
||||
expect(result.observations.step).toBe('init');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,77 @@
|
||||
import { query } from '@anthropic-ai/claude-agent-sdk';
|
||||
import { createIntelligentResponder } from './intelligent-responder.js';
|
||||
import type { BotConfig, ExecutionObservation, StepName } from './types.js';
|
||||
import type { ObservationCollector } from './observation-collector.js';
|
||||
|
||||
export interface SessionOptions {
|
||||
prompt: string;
|
||||
config: BotConfig;
|
||||
step: StepName;
|
||||
maxTurns?: number;
|
||||
collector: ObservationCollector;
|
||||
}
|
||||
|
||||
export interface SessionResult {
|
||||
sessionId: string | null;
|
||||
output: string;
|
||||
success: boolean;
|
||||
duration: number;
|
||||
observations: ExecutionObservation;
|
||||
}
|
||||
|
||||
export async function runSession(options: SessionOptions): Promise<SessionResult> {
|
||||
const { prompt, config, step, maxTurns = 50, collector } = options;
|
||||
const startTime = Date.now();
|
||||
let output = '';
|
||||
let sessionId: string | null = null;
|
||||
|
||||
try {
|
||||
const session = query({
|
||||
prompt,
|
||||
options: {
|
||||
cwd: config.projectDir,
|
||||
maxTurns,
|
||||
permissionMode: 'bypassPermissions',
|
||||
allowDangerouslySkipPermissions: true,
|
||||
canUseTool: createIntelligentResponder(config, step, collector),
|
||||
systemPrompt: { type: 'preset', preset: 'claude_code' },
|
||||
settingSources: ['project'],
|
||||
},
|
||||
});
|
||||
|
||||
for await (const message of session) {
|
||||
// Record all messages for observations
|
||||
collector.recordMessage(message);
|
||||
|
||||
if (message.type === 'assistant') {
|
||||
sessionId = message.session_id ?? sessionId;
|
||||
for (const block of message.message.content) {
|
||||
if (block.type === 'text') {
|
||||
output += block.text + '\n';
|
||||
} else if (block.type === 'tool_use') {
|
||||
// Capture tool invocations from the message stream as a fallback
|
||||
// in case canUseTool doesn't fire (e.g., Skill sub-sessions)
|
||||
collector.recordToolUse(
|
||||
block.name,
|
||||
block.input as Record<string, unknown>,
|
||||
undefined,
|
||||
true
|
||||
);
|
||||
}
|
||||
}
|
||||
} else if (message.type === 'result') {
|
||||
sessionId = message.session_id ?? sessionId;
|
||||
}
|
||||
}
|
||||
|
||||
const duration = Date.now() - startTime;
|
||||
const hasOutput = output.trim().length > 0;
|
||||
const observations = collector.finalize();
|
||||
return { sessionId, output, success: hasOutput, duration, observations };
|
||||
} catch (error) {
|
||||
const duration = Date.now() - startTime;
|
||||
const errorMsg = error instanceof Error ? error.message : String(error);
|
||||
const observations = collector.finalize();
|
||||
return { sessionId, output, success: false, duration, observations };
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
|
||||
import fs from 'fs-extra';
|
||||
import os from 'os';
|
||||
import path from 'path';
|
||||
import { checkAllPhasesComplete, detectStepCompletion } from './step-detector.js';
|
||||
|
||||
let tmpDir: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'p2c-test-'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
fs.removeSync(tmpDir);
|
||||
});
|
||||
|
||||
// ── checkAllPhasesComplete ──────────────────────────────────────────
|
||||
|
||||
describe('checkAllPhasesComplete', () => {
|
||||
it('returns false when no specs/ directory exists', () => {
|
||||
expect(checkAllPhasesComplete(tmpDir)).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false when specs/ has no subdirectories', () => {
|
||||
fs.ensureDirSync(path.join(tmpDir, 'specs'));
|
||||
expect(checkAllPhasesComplete(tmpDir)).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false when spec dir exists but has no overview.md and no phase files', () => {
|
||||
// This is the false-positive bug we fixed — an empty spec dir should NOT be "complete"
|
||||
fs.ensureDirSync(path.join(tmpDir, 'specs', 'my-feature'));
|
||||
expect(checkAllPhasesComplete(tmpDir)).toBe(false);
|
||||
});
|
||||
|
||||
it('returns true when overview.md has all phases checked [x]', () => {
|
||||
const specDir = path.join(tmpDir, 'specs', 'my-feature');
|
||||
fs.ensureDirSync(specDir);
|
||||
fs.writeFileSync(
|
||||
path.join(specDir, 'overview.md'),
|
||||
`# Overview\n- [x] Phase 1: Setup\n- [x] Phase 2: Core\n- [x] Phase 3: Polish\n`,
|
||||
);
|
||||
expect(checkAllPhasesComplete(tmpDir)).toBe(true);
|
||||
});
|
||||
|
||||
it('returns false when overview.md has unchecked [ ] phases', () => {
|
||||
const specDir = path.join(tmpDir, 'specs', 'my-feature');
|
||||
fs.ensureDirSync(specDir);
|
||||
fs.writeFileSync(
|
||||
path.join(specDir, 'overview.md'),
|
||||
`# Overview\n- [x] Phase 1: Setup\n- [ ] Phase 2: Core\n- [ ] Phase 3: Polish\n`,
|
||||
);
|
||||
expect(checkAllPhasesComplete(tmpDir)).toBe(false);
|
||||
});
|
||||
|
||||
it('returns true via fallback: phase-*.md files with all checked, no overview.md', () => {
|
||||
const specDir = path.join(tmpDir, 'specs', 'my-feature');
|
||||
fs.ensureDirSync(specDir);
|
||||
fs.writeFileSync(
|
||||
path.join(specDir, 'phase-1.md'),
|
||||
`# Phase 1\n- [x] Task A\n- [x] Task B\n`,
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.join(specDir, 'phase-2.md'),
|
||||
`# Phase 2\n- [x] Task C\n`,
|
||||
);
|
||||
expect(checkAllPhasesComplete(tmpDir)).toBe(true);
|
||||
});
|
||||
|
||||
it('returns false via fallback: phase-*.md with unchecked items', () => {
|
||||
const specDir = path.join(tmpDir, 'specs', 'my-feature');
|
||||
fs.ensureDirSync(specDir);
|
||||
fs.writeFileSync(
|
||||
path.join(specDir, 'phase-1.md'),
|
||||
`# Phase 1\n- [x] Task A\n- [ ] Task B\n`,
|
||||
);
|
||||
expect(checkAllPhasesComplete(tmpDir)).toBe(false);
|
||||
});
|
||||
|
||||
it('returns true via nested phases/ subdirectory fallback with all checked', () => {
|
||||
const specDir = path.join(tmpDir, 'specs', 'my-feature');
|
||||
const phasesDir = path.join(specDir, 'phases');
|
||||
fs.ensureDirSync(phasesDir);
|
||||
fs.writeFileSync(
|
||||
path.join(phasesDir, 'phase-1.md'),
|
||||
`# Phase 1\n- [x] Task A\n- [x] Task B\n`,
|
||||
);
|
||||
expect(checkAllPhasesComplete(tmpDir)).toBe(true);
|
||||
});
|
||||
|
||||
it('returns false via nested phases/ with unchecked items', () => {
|
||||
const specDir = path.join(tmpDir, 'specs', 'my-feature');
|
||||
const phasesDir = path.join(specDir, 'phases');
|
||||
fs.ensureDirSync(phasesDir);
|
||||
fs.writeFileSync(
|
||||
path.join(phasesDir, 'phase-1.md'),
|
||||
`# Phase 1\n- [x] Task A\n- [ ] Task B\n`,
|
||||
);
|
||||
expect(checkAllPhasesComplete(tmpDir)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// ── detectStepCompletion ────────────────────────────────────────────
|
||||
|
||||
describe('detectStepCompletion', () => {
|
||||
it('init: completed when output mentions agents.md created', () => {
|
||||
const result = detectStepCompletion('AGENTS.md has been created successfully', 'init');
|
||||
expect(result.completed).toBe(true);
|
||||
expect(result.nextStep).toBe('plan');
|
||||
});
|
||||
|
||||
it('init: not completed for unrelated output', () => {
|
||||
const result = detectStepCompletion('Hello world, nothing happened', 'init');
|
||||
expect(result.completed).toBe(false);
|
||||
expect(result.nextStep).toBeNull();
|
||||
});
|
||||
|
||||
it('plan: completed when plan is saved', () => {
|
||||
const result = detectStepCompletion('The plan has been saved and finalized', 'plan');
|
||||
expect(result.completed).toBe(true);
|
||||
expect(result.nextStep).toBe('document');
|
||||
});
|
||||
|
||||
it('plan: not completed for unrelated output', () => {
|
||||
const result = detectStepCompletion('Reading the codebase...', 'plan');
|
||||
expect(result.completed).toBe(false);
|
||||
expect(result.nextStep).toBeNull();
|
||||
});
|
||||
|
||||
it('document: completed when overview.md is mentioned', () => {
|
||||
const result = detectStepCompletion('Created overview.md with all phases', 'document');
|
||||
expect(result.completed).toBe(true);
|
||||
expect(result.nextStep).toBe('implement');
|
||||
});
|
||||
|
||||
it('document: not completed for unrelated output', () => {
|
||||
const result = detectStepCompletion('Thinking about the design...', 'document');
|
||||
expect(result.completed).toBe(false);
|
||||
expect(result.nextStep).toBeNull();
|
||||
});
|
||||
|
||||
it('implement: completed when phase is done', () => {
|
||||
const result = detectStepCompletion('Phase 1 is now complete!', 'implement');
|
||||
expect(result.completed).toBe(true);
|
||||
expect(result.nextStep).toBe('finalize');
|
||||
});
|
||||
|
||||
it('implement: not completed for unrelated output', () => {
|
||||
const result = detectStepCompletion('Working on some files', 'implement');
|
||||
expect(result.completed).toBe(false);
|
||||
expect(result.nextStep).toBeNull();
|
||||
});
|
||||
|
||||
it('finalize: completed when finalize is done', () => {
|
||||
const result = detectStepCompletion('Finalize step is complete and archived', 'finalize');
|
||||
expect(result.completed).toBe(true);
|
||||
expect(result.nextStep).toBeNull(); // last step
|
||||
});
|
||||
|
||||
it('finalize: not completed for unrelated output', () => {
|
||||
const result = detectStepCompletion('Just starting...', 'finalize');
|
||||
expect(result.completed).toBe(false);
|
||||
expect(result.nextStep).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,151 @@
|
||||
import fs from 'fs-extra';
|
||||
import path from 'path';
|
||||
import type { StepName } from './types.js';
|
||||
|
||||
export interface DetectionResult {
|
||||
completed: boolean;
|
||||
nextStep: StepName | null;
|
||||
needsAnotherImplementPass: boolean;
|
||||
}
|
||||
|
||||
const STEP_ORDER: StepName[] = ['init', 'plan', 'document', 'implement', 'finalize'];
|
||||
|
||||
export function detectStepCompletion(output: string, step: StepName): DetectionResult {
|
||||
const lowerOutput = output.toLowerCase();
|
||||
|
||||
let completed = false;
|
||||
|
||||
switch (step) {
|
||||
case 'init':
|
||||
completed = lowerOutput.includes('agents.md') && (
|
||||
lowerOutput.includes('created') ||
|
||||
lowerOutput.includes('generated') ||
|
||||
lowerOutput.includes('written')
|
||||
);
|
||||
break;
|
||||
|
||||
case 'plan':
|
||||
completed = lowerOutput.includes('plan') && (
|
||||
lowerOutput.includes('complete') ||
|
||||
lowerOutput.includes('approved') ||
|
||||
lowerOutput.includes('finalized') ||
|
||||
lowerOutput.includes('saved')
|
||||
);
|
||||
break;
|
||||
|
||||
case 'document':
|
||||
completed = lowerOutput.includes('overview.md') || (
|
||||
lowerOutput.includes('document') && lowerOutput.includes('complete')
|
||||
);
|
||||
break;
|
||||
|
||||
case 'implement':
|
||||
completed = lowerOutput.includes('phase') && (
|
||||
lowerOutput.includes('complete') ||
|
||||
lowerOutput.includes('done') ||
|
||||
lowerOutput.includes('finished')
|
||||
);
|
||||
break;
|
||||
|
||||
case 'finalize':
|
||||
completed = lowerOutput.includes('finalize') && (
|
||||
lowerOutput.includes('complete') ||
|
||||
lowerOutput.includes('archived') ||
|
||||
lowerOutput.includes('done')
|
||||
);
|
||||
break;
|
||||
}
|
||||
|
||||
// Determine next step
|
||||
const currentIdx = STEP_ORDER.indexOf(step);
|
||||
const nextStep = currentIdx < STEP_ORDER.length - 1 ? STEP_ORDER[currentIdx + 1] : null;
|
||||
|
||||
return {
|
||||
completed,
|
||||
nextStep: completed ? nextStep : null,
|
||||
needsAnotherImplementPass: false,
|
||||
};
|
||||
}
|
||||
|
||||
export function checkAllPhasesComplete(projectDir: string): boolean {
|
||||
const specsDir = path.join(projectDir, 'specs');
|
||||
|
||||
if (!fs.existsSync(specsDir)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const entries = fs.readdirSync(specsDir, { withFileTypes: true });
|
||||
const specDirs = entries.filter((e) => e.isDirectory());
|
||||
|
||||
if (specDirs.length === 0) {
|
||||
return false;
|
||||
}
|
||||
|
||||
let foundPhaseTracking = false;
|
||||
|
||||
for (const dir of specDirs) {
|
||||
const specPath = path.join(specsDir, dir.name);
|
||||
|
||||
// Try overview.md first
|
||||
const overviewPath = path.join(specPath, 'overview.md');
|
||||
if (fs.existsSync(overviewPath)) {
|
||||
foundPhaseTracking = true;
|
||||
if (hasUncheckedPhases(fs.readFileSync(overviewPath, 'utf-8'))) {
|
||||
return false;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
// Fallback: look for phase-*.md or PHASE-*.md in the spec dir
|
||||
const phaseFiles = findPhaseFiles(specPath);
|
||||
if (phaseFiles.length > 0) {
|
||||
foundPhaseTracking = true;
|
||||
for (const pf of phaseFiles) {
|
||||
if (hasUncheckedPhases(fs.readFileSync(pf, 'utf-8'))) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
// Fallback: look inside a phases/ subdirectory
|
||||
const phasesSubdir = path.join(specPath, 'phases');
|
||||
if (fs.existsSync(phasesSubdir)) {
|
||||
const subPhaseFiles = findPhaseFiles(phasesSubdir);
|
||||
if (subPhaseFiles.length > 0) {
|
||||
foundPhaseTracking = true;
|
||||
for (const pf of subPhaseFiles) {
|
||||
if (hasUncheckedPhases(fs.readFileSync(pf, 'utf-8'))) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Only return true if we positively confirmed all phases are checked off
|
||||
return foundPhaseTracking;
|
||||
}
|
||||
|
||||
function hasUncheckedPhases(content: string): boolean {
|
||||
const lines = content.split('\n');
|
||||
|
||||
const phaseLines = lines.filter((line) =>
|
||||
line.match(/^[-*]\s*\[[ x]\]/i) && line.toLowerCase().includes('phase')
|
||||
);
|
||||
|
||||
if (phaseLines.length > 0) {
|
||||
return phaseLines.some((line) => line.includes('[ ]'));
|
||||
}
|
||||
|
||||
// No phase-specific checkboxes — check for any unchecked boxes
|
||||
return lines.some((line) => /^[-*]\s*\[ \]/.test(line));
|
||||
}
|
||||
|
||||
function findPhaseFiles(dir: string): string[] {
|
||||
if (!fs.existsSync(dir)) return [];
|
||||
const entries = fs.readdirSync(dir);
|
||||
return entries
|
||||
.filter((name) => /^phase[-_]?\d+.*\.md$/i.test(name))
|
||||
.map((name) => path.join(dir, name));
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
export type BotMode = 'new-project' | 'enhancement';
|
||||
|
||||
export interface BotConfig {
|
||||
workDir: string;
|
||||
projectDir: string;
|
||||
ideaDescription: string;
|
||||
ideaName: string;
|
||||
mode: BotMode;
|
||||
}
|
||||
|
||||
export type StepName = 'init' | 'plan' | 'document' | 'implement' | 'finalize';
|
||||
|
||||
export interface ToolObservation {
|
||||
toolName: string;
|
||||
input: Record<string, unknown>;
|
||||
output?: unknown;
|
||||
timestamp: number;
|
||||
allowed: boolean;
|
||||
autoAnswered?: boolean;
|
||||
}
|
||||
|
||||
export interface QuestionContext {
|
||||
question: string;
|
||||
options: Array<{ label: string; description: string }>;
|
||||
llmReasoning: string;
|
||||
selectedAnswer: string;
|
||||
timestamp: number;
|
||||
}
|
||||
|
||||
export interface ExecutionObservation {
|
||||
step: StepName;
|
||||
startTime: number;
|
||||
endTime: number;
|
||||
tools: ToolObservation[];
|
||||
assistantMessages: string[];
|
||||
errors: string[];
|
||||
filesCreated: string[];
|
||||
filesModified: string[];
|
||||
questionsAsked: QuestionContext[];
|
||||
}
|
||||
|
||||
export interface EvaluationResult {
|
||||
step: StepName;
|
||||
score: number;
|
||||
strengths: string[];
|
||||
weaknesses: string[];
|
||||
suggestions: string[];
|
||||
criticalIssues: string[];
|
||||
timestamp: number;
|
||||
evaluatorModel: string;
|
||||
reasoning: string;
|
||||
}
|
||||
|
||||
export interface StepResult {
|
||||
step: StepName;
|
||||
success: boolean;
|
||||
sessionId: string | null;
|
||||
duration: number;
|
||||
error: string | null;
|
||||
evaluation?: EvaluationResult;
|
||||
observations?: ExecutionObservation;
|
||||
}
|
||||
|
||||
export interface BotState {
|
||||
config: BotConfig;
|
||||
steps: StepResult[];
|
||||
currentStep: StepName | null;
|
||||
implementPasses: number;
|
||||
allPhasesComplete: boolean;
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "bundler",
|
||||
"lib": ["ES2022"],
|
||||
"outDir": "dist",
|
||||
"rootDir": ".",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"resolveJsonModule": true,
|
||||
"declaration": true,
|
||||
"declarationMap": true,
|
||||
"sourceMap": true
|
||||
},
|
||||
"include": ["src/**/*"],
|
||||
"exclude": ["node_modules", "dist"]
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
import { defineConfig } from 'tsup';
|
||||
|
||||
export default defineConfig({
|
||||
entry: {
|
||||
'bin/plan2code-bot': 'src/bin/plan2code-bot.ts',
|
||||
index: 'src/index.ts',
|
||||
},
|
||||
format: ['esm'],
|
||||
dts: false,
|
||||
clean: true,
|
||||
sourcemap: true,
|
||||
banner: {
|
||||
js: '#!/usr/bin/env node',
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,7 @@
|
||||
import { defineConfig } from 'vitest/config';
|
||||
|
||||
export default defineConfig({
|
||||
test: {
|
||||
include: ['src/**/*.test.ts'],
|
||||
},
|
||||
});
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
An autonomous CLI tool that implements Plan2Code specs by looping through tasks automatically.
|
||||
|
||||
> **Note:** This is an **alternative** to `/plan2code-3--implement`, not a replacement. Use the manual Step 3 workflow when you want interactive control over each phase, or use this loop when you prefer hands-off autonomous execution.
|
||||
> **Note:** This is an **alternative** to `/plan2code-3-implement`, not a replacement. Use the manual Step 3 workflow when you want interactive control over each phase, or use this loop when you prefer hands-off autonomous execution.
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -109,6 +109,7 @@ The scratchpad is managed by the LLM itself - after each task, the AI appends no
|
||||
|-------|--------|
|
||||
| Claude Code | Supported |
|
||||
| GitHub Copilot CLI | Supported |
|
||||
| Devin CLI | Supported |
|
||||
|
||||
The loop uses your configured default model for each agent.
|
||||
|
||||
@@ -119,7 +120,7 @@ $ plan2code-loop
|
||||
|
||||
╭──────────────────────────────────────╮
|
||||
│ │
|
||||
│ 🔮 Plany's Loop │
|
||||
│ 🔮 Planny's Loop │
|
||||
│ Autonomous Implementation │
|
||||
│ │
|
||||
╰──────────────────────────────────────╯
|
||||
@@ -196,7 +197,8 @@ src/
|
||||
├── cli.ts # Interactive prompts
|
||||
├── agents/ # Agent implementations
|
||||
│ ├── claude-code.ts
|
||||
│ └── copilot-cli.ts
|
||||
│ ├── copilot-cli.ts
|
||||
│ └── devin-cli.ts
|
||||
├── prompt/ # Prompt building
|
||||
│ ├── templates.ts
|
||||
│ └── builder.ts
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "plan2code-loop",
|
||||
"version": "1.6.1",
|
||||
"version": "1.6.2",
|
||||
"description": "Plan2Code Loop - Autonomous spec-driven implementation CLI",
|
||||
"type": "module",
|
||||
"main": "dist/index.js",
|
||||
@@ -20,6 +20,7 @@
|
||||
"automation",
|
||||
"claude",
|
||||
"copilot",
|
||||
"devin",
|
||||
"spec-driven"
|
||||
],
|
||||
"license": "MIT",
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
import type { Agent, AgentConfig, AgentExecutionOptions, AgentExecutionResult } from './types.js';
|
||||
import { executeCommand } from '../utils/process.js';
|
||||
import { writeFileSync, unlinkSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
|
||||
const devinCliConfig: AgentConfig = {
|
||||
name: 'devin-cli',
|
||||
displayName: 'Devin CLI',
|
||||
command: 'devin',
|
||||
models: [
|
||||
{ value: 'default', label: 'Default (use Devin config)' },
|
||||
],
|
||||
defaultModel: 'default',
|
||||
flags: {
|
||||
prompt: '--print',
|
||||
promptFile: '--prompt-file',
|
||||
model: '--model',
|
||||
skipPermissions: '--permission-mode',
|
||||
},
|
||||
};
|
||||
|
||||
class DevinCliAgent implements Agent {
|
||||
readonly config = devinCliConfig;
|
||||
|
||||
async execute(options: AgentExecutionOptions): Promise<AgentExecutionResult> {
|
||||
// Devin CLI takes the prompt via --prompt-file rather than stdin
|
||||
const tempFile = join(tmpdir(), `plan2code-prompt-${Date.now()}.txt`);
|
||||
writeFileSync(tempFile, options.prompt, 'utf-8');
|
||||
|
||||
try {
|
||||
const args: string[] = [
|
||||
this.config.flags.prompt, // --print for non-interactive mode
|
||||
this.config.flags.promptFile!, tempFile, // --prompt-file <path>
|
||||
this.config.flags.skipPermissions, 'dangerous', // --permission-mode dangerous (auto-approve all tools)
|
||||
];
|
||||
|
||||
// Only add --model if not using default
|
||||
if (options.model && options.model !== 'default') {
|
||||
args.push(this.config.flags.model, options.model);
|
||||
}
|
||||
|
||||
const result = await executeCommand({
|
||||
command: this.config.command,
|
||||
args,
|
||||
cwd: options.cwd,
|
||||
timeout: options.timeout,
|
||||
signal: options.signal,
|
||||
});
|
||||
|
||||
return {
|
||||
stdout: result.stdout,
|
||||
stderr: result.stderr,
|
||||
exitCode: result.exitCode,
|
||||
timedOut: result.timedOut,
|
||||
cancelled: result.cancelled,
|
||||
duration: result.duration,
|
||||
};
|
||||
} finally {
|
||||
// Clean up temp file
|
||||
try {
|
||||
unlinkSync(tempFile);
|
||||
} catch {
|
||||
// Ignore cleanup errors
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async isAvailable(): Promise<boolean> {
|
||||
// Run devin --version to verify it's actually installed and working
|
||||
const result = await executeCommand({
|
||||
command: this.config.command,
|
||||
args: ['--version'],
|
||||
cwd: process.cwd(),
|
||||
timeout: 5000,
|
||||
});
|
||||
return result.exitCode === 0;
|
||||
}
|
||||
}
|
||||
|
||||
export const devinCliAgent = new DevinCliAgent();
|
||||
@@ -9,11 +9,14 @@ export type {
|
||||
export { agentRegistry } from './registry.js';
|
||||
export { claudeCodeAgent } from './claude-code.js';
|
||||
export { copilotCliAgent } from './copilot-cli.js';
|
||||
export { devinCliAgent } from './devin-cli.js';
|
||||
|
||||
// Register all agents
|
||||
import { agentRegistry } from './registry.js';
|
||||
import { claudeCodeAgent } from './claude-code.js';
|
||||
import { copilotCliAgent } from './copilot-cli.js';
|
||||
import { devinCliAgent } from './devin-cli.js';
|
||||
|
||||
agentRegistry.register(claudeCodeAgent);
|
||||
agentRegistry.register(copilotCliAgent);
|
||||
agentRegistry.register(devinCliAgent);
|
||||
|
||||
@@ -14,6 +14,7 @@ export interface AgentConfig {
|
||||
model: string;
|
||||
skipPermissions: string;
|
||||
silent?: string;
|
||||
promptFile?: string;
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -23,11 +23,11 @@ async function selectSpec(cwd: string = process.cwd()): Promise<string | null> {
|
||||
logger.info('');
|
||||
logger.info('To get started:');
|
||||
logger.info('');
|
||||
logger.info('1. Create a spec using `plan2code-1--plan`');
|
||||
logger.info('1. Create a spec using `plan2code-1-plan`');
|
||||
logger.info(' command in our AI Agent');
|
||||
logger.info('');
|
||||
logger.info('2. Come back here and run `plan2code-loop`');
|
||||
logger.info(' as an alternative to `plan2code-3--implement`');
|
||||
logger.info(' as an alternative to `plan2code-3-implement`');
|
||||
logger.info('');
|
||||
return null;
|
||||
}
|
||||
@@ -176,7 +176,7 @@ async function handleExistingSession(
|
||||
export async function setupSession(
|
||||
stateManager: StateManager
|
||||
): Promise<SessionSetupResult | null> {
|
||||
// Show welcome
|
||||
// Show Planny welcome
|
||||
logger.welcome();
|
||||
|
||||
// Select spec
|
||||
@@ -224,7 +224,8 @@ export async function setupSession(
|
||||
model: 'default',
|
||||
maxIterations,
|
||||
specPath,
|
||||
timeout: 30,
|
||||
timeout: 3,
|
||||
maxRetries: 5,
|
||||
verbose: false,
|
||||
startedAt: new Date().toISOString(),
|
||||
currentIteration: 0,
|
||||
|
||||
@@ -64,9 +64,82 @@ export class Controller {
|
||||
});
|
||||
}
|
||||
|
||||
private async executeIteration(prompt: string): Promise<AgentExecutionResult> {
|
||||
const timeoutMs = this.config.timeout * 60 * 1000;
|
||||
private computeTimeoutMs(attempt: number): number {
|
||||
const baseMs = this.config.timeout * 60 * 1000;
|
||||
return baseMs + (attempt * 30 * 1000);
|
||||
}
|
||||
|
||||
private formatDuration(ms: number): string {
|
||||
const totalSec = Math.round(ms / 1000);
|
||||
const min = Math.floor(totalSec / 60);
|
||||
const sec = totalSec % 60;
|
||||
if (min === 0) return `${sec}s`;
|
||||
if (sec === 0) return `${min}m`;
|
||||
return `${min}m ${sec}s`;
|
||||
}
|
||||
|
||||
private async executeWithRetry(prompt: string, iterNum: number): Promise<AgentExecutionResult | 'fatal_timeout'> {
|
||||
const maxAttempts = (this.config.maxRetries ?? 5) + 1;
|
||||
|
||||
for (let attempt = 0; attempt < maxAttempts; attempt++) {
|
||||
const timeoutMs = this.computeTimeoutMs(attempt);
|
||||
|
||||
if (attempt > 0) {
|
||||
const prevTimeoutMs = this.computeTimeoutMs(attempt - 1);
|
||||
logger.warning(
|
||||
`Timed out after ${this.formatDuration(prevTimeoutMs)}, retrying (${attempt}/${maxAttempts - 1})...`
|
||||
);
|
||||
}
|
||||
|
||||
const spinnerBase = attempt > 0
|
||||
? `Waiting for AI Agent response (retry ${attempt}/${maxAttempts - 1})`
|
||||
: 'Waiting for AI Agent response (please be patient)';
|
||||
const spinner = logger.spinner(spinnerBase);
|
||||
const startTime = Date.now();
|
||||
|
||||
const elapsedInterval = setInterval(() => {
|
||||
const elapsed = Math.round((Date.now() - startTime) / 1000);
|
||||
spinner.text = `${spinnerBase} ... (${elapsed}s)`;
|
||||
}, 1000);
|
||||
|
||||
const result = await this.executeIteration(prompt, timeoutMs);
|
||||
clearInterval(elapsedInterval);
|
||||
spinner.stop();
|
||||
|
||||
// Cancelled or interrupted — return immediately, don't retry
|
||||
if (result.cancelled || this.interrupted) {
|
||||
return result;
|
||||
}
|
||||
|
||||
// Completed (success or error exit code) — return to caller
|
||||
if (!result.timedOut) {
|
||||
return result;
|
||||
}
|
||||
|
||||
// Timed out — retry if attempts remain, otherwise fatal
|
||||
if (attempt < maxAttempts - 1) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// All attempts exhausted
|
||||
logger.error(
|
||||
`Iteration ${iterNum} timed out on all ${maxAttempts} attempt${maxAttempts === 1 ? '' : 's'}. Stopping loop.`
|
||||
);
|
||||
const entry: IterationLogEntry = {
|
||||
iteration: iterNum,
|
||||
timestamp: new Date().toISOString(),
|
||||
duration: result.duration,
|
||||
exitCode: -1,
|
||||
status: 'timeout',
|
||||
};
|
||||
await this.stateManager.appendIterationLog(entry);
|
||||
return 'fatal_timeout';
|
||||
}
|
||||
|
||||
return 'fatal_timeout'; // unreachable, satisfies TS
|
||||
}
|
||||
|
||||
private async executeIteration(prompt: string, timeoutMs: number): Promise<AgentExecutionResult> {
|
||||
// Create new AbortController for this iteration
|
||||
this.abortController = new AbortController();
|
||||
|
||||
@@ -179,19 +252,21 @@ export class Controller {
|
||||
// Build prompt - simple, just spec path and iteration info
|
||||
const prompt = await this.buildPrompt();
|
||||
|
||||
const spinner = logger.spinner('Waiting for AI Agent response (please be patient)');
|
||||
const startTime = Date.now();
|
||||
|
||||
// Update spinner with elapsed time every second
|
||||
const elapsedInterval = setInterval(() => {
|
||||
const elapsed = Math.round((Date.now() - startTime) / 1000);
|
||||
spinner.text = `Waiting for AI Agent response (please be patient) ... (${elapsed}s)`;
|
||||
}, 1000);
|
||||
|
||||
try {
|
||||
const result = await this.executeIteration(prompt);
|
||||
clearInterval(elapsedInterval);
|
||||
spinner.stop();
|
||||
const retryResult = await this.executeWithRetry(prompt, iterNum);
|
||||
|
||||
if (retryResult === 'fatal_timeout') {
|
||||
return {
|
||||
completed: false,
|
||||
iterations: this.config.currentIteration,
|
||||
exitReason: 'error',
|
||||
tasksCompleted: this.tasksCompleted,
|
||||
prereqsCompleted: this.prereqsCompleted,
|
||||
error: new Error(`Iteration ${iterNum} failed after all retry attempts`),
|
||||
};
|
||||
}
|
||||
|
||||
const result = retryResult;
|
||||
|
||||
// Check if cancelled
|
||||
if (result.cancelled || this.interrupted) {
|
||||
@@ -390,13 +465,8 @@ export class Controller {
|
||||
await this.stateManager.updateSpecHash(this.config.specPath);
|
||||
this.config.currentIteration++;
|
||||
|
||||
// Handle timeout
|
||||
if (result.timedOut) {
|
||||
logger.warning(`Iteration ${iterNum} timed out, continuing...`);
|
||||
}
|
||||
|
||||
// Handle error (but continue - LLM might recover)
|
||||
if (result.exitCode !== 0 && !result.timedOut) {
|
||||
if (result.exitCode !== 0) {
|
||||
// In phase mode, check if any tasks completed despite error exit code
|
||||
const hasCompletions = isPhaseMode
|
||||
? checkForAllCompletions(result.stdout + result.stderr).tasks.length > 0
|
||||
|
||||
@@ -23,18 +23,19 @@ Find the FIRST unchecked task, implement ONLY that task, then STOP and report.
|
||||
2. Find the FIRST phase with an unchecked checkbox (\`- [ ]\` or \`- [/]\`)
|
||||
3. Read that phase's file (e.g., \`phase-1.md\`)
|
||||
4. Check the \`## Prerequisites\` section FIRST
|
||||
5. Find the FIRST unchecked prerequisite (\`- [ ]\`)
|
||||
- If found, that is your task for this iteration
|
||||
- Verify/complete it, then mark \`[x]\` or \`[?]\`
|
||||
6. Only if ALL prerequisites are complete (\`[x]\` or \`[?]\`), find the FIRST unchecked task
|
||||
5. Find the FIRST unverified prerequisite (no "VERIFIED" or "ASSUMED" annotation)
|
||||
- If found, verify/complete it, then annotate "VERIFIED" or "ASSUMED: [reason]" inline
|
||||
6. Only if ALL prerequisites are verified or assumed, find the FIRST unchecked task (\`- [ ]\`)
|
||||
7. That is your ONE task - implement ONLY that task
|
||||
|
||||
## Checkbox States
|
||||
## Checkbox States (Task items only)
|
||||
- \`[ ]\` = incomplete/pending (do the FIRST one you find)
|
||||
- \`[x]\` = complete (skip)
|
||||
- \`[?]\` = assumed complete, couldn't verify (skip)
|
||||
- \`[!]\` = blocked (skip)
|
||||
|
||||
Prerequisites use plain bullets with inline annotations, not checkboxes.
|
||||
|
||||
## Implementation Steps
|
||||
1. Read and understand the single task
|
||||
2. Implement it completely
|
||||
@@ -66,7 +67,10 @@ Use only when ALL phases in overview.md are marked complete.
|
||||
|
||||
## Scratchpad Management
|
||||
|
||||
After completing each task, append to \`{{specPath}}/.plan2code-loop/scratchpad.md\`:
|
||||
After completing each task, add a new entry at the **bottom** of \`{{specPath}}/.plan2code-loop/scratchpad.md\`.
|
||||
Never edit, reorganize, or insert into existing content — only append new entries to the end of the file.
|
||||
|
||||
Each entry should include:
|
||||
- Task completed and Phase item reference
|
||||
- Key decisions made and reasoning
|
||||
- Files changed
|
||||
@@ -90,6 +94,7 @@ export const LOOP_PROMPT_TEMPLATE_PHASE = `# PLAN2CODE-LOOP: Autonomous Phase Im
|
||||
**IMPLEMENT ALL REMAINING TASKS IN THE CURRENT PHASE.**
|
||||
Find the first incomplete phase, then implement every remaining task in that phase before stopping.
|
||||
Complete each task fully before moving to the next task within the phase.
|
||||
|
||||
## Project Information
|
||||
- **Project Root:** \`{{projectRoot}}\`
|
||||
- **Spec Location:** \`{{specPath}}\`
|
||||
@@ -108,17 +113,19 @@ Complete each task fully before moving to the next task within the phase.
|
||||
2. Find the FIRST phase with an unchecked checkbox (\`- [ ]\` or \`- [/]\`)
|
||||
3. Read that phase's file (e.g., \`phase-1.md\`)
|
||||
4. Check the \`## Prerequisites\` section FIRST
|
||||
5. Complete ALL unchecked prerequisites (\`- [ ]\`) first, in order
|
||||
- Verify/complete each, then mark \`[x]\` or \`[?]\`
|
||||
6. Once ALL prerequisites are complete, implement ALL unchecked tasks in order
|
||||
5. Verify ALL unverified prerequisites first, in order
|
||||
- Annotate each "VERIFIED" or "ASSUMED: [reason]" inline
|
||||
6. Once ALL prerequisites are verified, implement ALL unchecked tasks in order
|
||||
7. Continue until every task in the phase is marked \`[x]\`
|
||||
|
||||
## Checkbox States
|
||||
## Checkbox States (Task items only)
|
||||
- \`[ ]\` = incomplete/pending
|
||||
- \`[x]\` = complete (skip)
|
||||
- \`[?]\` = assumed complete, couldn't verify (skip)
|
||||
- \`[!]\` = blocked (skip, note in scratchpad)
|
||||
|
||||
Prerequisites use plain bullets with inline annotations, not checkboxes.
|
||||
|
||||
## Implementation Steps (repeat for EACH task in the phase)
|
||||
1. Read and understand the task
|
||||
2. Implement it completely
|
||||
@@ -139,9 +146,13 @@ git commit -m "<commit message>"
|
||||
\`\`\`
|
||||
|
||||
**Commit message format:**
|
||||
- With JIRA ticket: Use \`-m "Task X.Y: description" -m "{{jiraTicketId}}" -m "AI Assisted"\` (three \`-m\` flags)
|
||||
- Without JIRA ticket: \`-m "Task X.Y: description" -m "AI Assisted"\` (two \`-m\` flags)
|
||||
- ALWAYS include the "AI Assisted" footer as the final \`-m\` flag
|
||||
\`\`\`
|
||||
git add -A
|
||||
git commit -m "Task X.Y: description" -m "{{jiraTicketId}}" -m "AI Assisted"
|
||||
\`\`\`
|
||||
- With JIRA ticket: three \`-m\` flags (description, ticket ID, AI Assisted)
|
||||
- Without JIRA ticket: two \`-m\` flags (description, AI Assisted)
|
||||
- ALWAYS include "AI Assisted" as the final \`-m\` flag
|
||||
|
||||
Replace X.Y with the actual task ID and description with a concise summary of what was implemented.
|
||||
|
||||
@@ -167,7 +178,10 @@ After ALL tasks in the phase are complete (or blocked), output:
|
||||
|
||||
## Scratchpad Management
|
||||
|
||||
After completing each task, append to \`{{specPath}}/.plan2code-loop/scratchpad.md\`:
|
||||
After completing each task, add a new entry at the **bottom** of \`{{specPath}}/.plan2code-loop/scratchpad.md\`.
|
||||
Never edit, reorganize, or insert into existing content — only append new entries to the end of the file.
|
||||
|
||||
Each entry should include:
|
||||
- Task completed and Phase item reference
|
||||
- Key decisions made and reasoning
|
||||
- Files changed
|
||||
|
||||
@@ -1,10 +1,11 @@
|
||||
export type LoopMode = 'task' | 'phase';
|
||||
|
||||
export interface SessionConfig {
|
||||
agent: string; // "claude-code" | "copilot-cli"
|
||||
agent: string; // "claude-code" | "copilot-cli" | "devin-cli"
|
||||
model: string; // Selected model
|
||||
maxIterations: number; // 5-50
|
||||
timeout: number; // Minutes per iteration
|
||||
timeout: number; // Base timeout in minutes per iteration attempt
|
||||
maxRetries: number; // Max retry attempts per iteration (timeout increments by 30s each retry)
|
||||
verbose: boolean;
|
||||
specPath: string; // Path to the spec directory
|
||||
startedAt: string; // ISO timestamp
|
||||
@@ -26,7 +27,8 @@ export type SessionState = 'new' | 'continue' | 'changed';
|
||||
|
||||
export const DEFAULT_CONFIG: Partial<SessionConfig> = {
|
||||
maxIterations: 100,
|
||||
timeout: 30,
|
||||
timeout: 3,
|
||||
maxRetries: 5,
|
||||
verbose: false,
|
||||
currentIteration: 0,
|
||||
loopMode: 'task',
|
||||
|
||||
@@ -40,7 +40,7 @@ export async function ensureGitRepo(cwd: string): Promise<boolean> {
|
||||
/**
|
||||
* Required entries for the .gitignore file
|
||||
*/
|
||||
const REQUIRED_GITIGNORE_ENTRIES = ['specs/', 'specs--completed/', 'nul', 'node_modules/'];
|
||||
const REQUIRED_GITIGNORE_ENTRIES = ['specs/', 'specs--completed/', '.plan2code-loop', '.plan2code-metrics', 'nul', 'node_modules/'];
|
||||
|
||||
/**
|
||||
* Ensure .gitignore exists with required entries
|
||||
@@ -109,7 +109,9 @@ export async function createTaskCommit(options: GitCommitOptions): Promise<boole
|
||||
// Build commit message
|
||||
let commitMessage = taskName;
|
||||
if (jiraTicketId) {
|
||||
commitMessage = `${taskName}\n\n${jiraTicketId}`;
|
||||
commitMessage = `${taskName}\n\n${jiraTicketId}\nAI Assisted`;
|
||||
} else {
|
||||
commitMessage = `${taskName}\n\nAI Assisted`;
|
||||
}
|
||||
|
||||
// Create the commit
|
||||
|
||||
@@ -117,7 +117,7 @@ export const logger = {
|
||||
console.log(chalk.green.bold('═'.repeat(50)));
|
||||
console.log();
|
||||
console.log(chalk.cyan.bold('Next Step:'));
|
||||
console.log(chalk.white(' Return to your AI Agent and run the'), chalk.yellow.bold('/plan2code-4--finalize'), chalk.white('step.'));
|
||||
console.log(chalk.white(' Return to your AI Agent and run the'), chalk.yellow.bold('/plan2code-4-finalize'), chalk.white('step.'));
|
||||
console.log(chalk.dim(' This will ensure quality, completeness, and proper documentation.'));
|
||||
console.log();
|
||||
},
|
||||
|
||||
@@ -6,7 +6,7 @@ export default defineConfig({
|
||||
index: 'src/index.ts',
|
||||
},
|
||||
format: ['esm'],
|
||||
dts: true,
|
||||
dts: false,
|
||||
clean: true,
|
||||
sourcemap: true,
|
||||
banner: {
|
||||
|
||||
@@ -0,0 +1,292 @@
|
||||
# plan2code-metrics
|
||||
|
||||
Recursive self-improvement toolchain for plan2code contributors. Collects metrics from completed project specs, aggregates them across runs and prompt generations, then uses AI to diagnose weak steps and propose targeted edits to the workflow prompts.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
```bash
|
||||
# Install (one-time, from the plan2code root)
|
||||
node install.js # → "A" (Install All + dev tools) includes metrics
|
||||
# → or "M" (Metrics only) under the CUSTOM sub-menu
|
||||
|
||||
# After finishing any project spec (steps 1-4):
|
||||
cd your-project
|
||||
plan2code-metrics # → "Collect metrics" → pick your spec dir → done (5 sec)
|
||||
|
||||
# When you're curious or have 3+ runs:
|
||||
plan2code-metrics # → "View metrics status" to see the dashboard
|
||||
# → "Run analysis" for AI diagnosis of weak spots
|
||||
# → "Generate improvement proposal" for prompt edits
|
||||
# → "Review and apply" to patch src/plan2code-*.md
|
||||
```
|
||||
|
||||
**One habit:** collect after every finished spec. Everything else is on-demand.
|
||||
|
||||
### How Many Runs Do I Need?
|
||||
|
||||
| Runs | What You Get |
|
||||
|------|-------------|
|
||||
| **1** | Raw data and basic dashboard. Start here. |
|
||||
| **3+** | AI analysis unlocks (warns below 3 that results may be unreliable). Pattern detection starts working. |
|
||||
| **5-10+** | Averages stabilize. Generation-over-generation comparisons become meaningful. |
|
||||
|
||||
More is always better — each run adds a data point. You're looking for trends, not individual scores.
|
||||
|
||||
### The Feedback Loop
|
||||
|
||||
```
|
||||
Use plan2code on a project (steps 1-4)
|
||||
|
|
||||
v
|
||||
Collect metrics from the finished spec <-- do this every time
|
||||
|
|
||||
v
|
||||
Aggregate across runs (automatic)
|
||||
|
|
||||
v
|
||||
Analyze weak spots (AI-powered, 3+ runs)
|
||||
|
|
||||
v
|
||||
Generate prompt improvements
|
||||
|
|
||||
v
|
||||
Apply edits to src/plan2code-*.md
|
||||
|
|
||||
v
|
||||
Prompts have new SHA hashes = new "generation"
|
||||
|
|
||||
v
|
||||
Use improved prompts on next project
|
||||
|
|
||||
v
|
||||
Collect again, compare generations <-- the loop closes
|
||||
```
|
||||
|
||||
After applying edits, the prompt file SHA hashes change. The next collected run falls into a **new cohort** (generation). The dashboard then shows generation-over-generation deltas — "Did confidence go up? Did blockers decrease?" — so you can measure whether your edits actually helped.
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
Requires Node.js >= 18.
|
||||
|
||||
### Via the plan2code installer (recommended)
|
||||
|
||||
From the plan2code repo root:
|
||||
|
||||
```bash
|
||||
node install.js
|
||||
```
|
||||
|
||||
Choose **A** (Install All + dev tools) to install prompts, loop CLI, bot, metrics, and status line together. Or choose **C** (Custom) then **M** (Metrics) to install just the metrics CLI.
|
||||
|
||||
The installer handles `npm install`, `npm run build`, and global linking automatically. After install, `plan2code-metrics` is available from any directory.
|
||||
|
||||
### Manual install
|
||||
|
||||
If you prefer to install manually or are developing on the metrics package:
|
||||
|
||||
```bash
|
||||
cd plan2code-metrics
|
||||
npm install
|
||||
npm run build
|
||||
npm link
|
||||
```
|
||||
|
||||
To run without linking:
|
||||
|
||||
```bash
|
||||
npm start
|
||||
# or
|
||||
node dist/bin/plan2code-metrics.js
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
Run `plan2code-metrics` from inside (or near) a plan2code repository. The CLI is fully interactive — no flags, all inputs collected via prompts.
|
||||
|
||||
```
|
||||
plan2code-metrics
|
||||
```
|
||||
|
||||
### Main Menu
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| **Collect metrics** | Parse a completed project spec and extract step-by-step metrics into a run JSON |
|
||||
| **Import run data** | Copy a run JSON from another project into the local metrics store |
|
||||
| **View metrics status** | Show aggregated metrics with health indicators and generation deltas |
|
||||
| **Run analysis** | AI-powered diagnosis of weak steps (requires Claude Code, GitHub Copilot CLI, or Devin CLI) |
|
||||
| **Generate improvement proposal** | AI generates surgical prompt edits based on a diagnosis |
|
||||
| **Review and apply** | Interactive diff review to accept/reject individual edits |
|
||||
| **Fetch community submissions** | List open community-feedback GitHub issues, parse + validate their METRICS_JSON payload, import into the local run store, and close them |
|
||||
|
||||
### Each Time You Finish a Spec
|
||||
|
||||
After completing Step 4 (finalize) on any project:
|
||||
|
||||
```bash
|
||||
cd your-project
|
||||
plan2code-metrics
|
||||
```
|
||||
|
||||
Pick **"Collect metrics"** and point it at your spec directory (`specs/<feature>/` or `specs--completed/<feature>/`). It reads your artifacts, extracts ~30 metrics, and writes a run JSON. Takes about 5 seconds.
|
||||
|
||||
The data stays local to that project in `.plan2code-metrics/runs/`. To aggregate across multiple projects, use **"Import"** to copy run files into one central location.
|
||||
|
||||
### Cross-Project Aggregation
|
||||
|
||||
If you use plan2code across several repos, you can consolidate all run data in one place:
|
||||
|
||||
```bash
|
||||
# From your central repo (e.g., the plan2code repo itself):
|
||||
plan2code-metrics
|
||||
# → "Import run data"
|
||||
# → Paste path to: /path/to/other-project/.plan2code-metrics/runs/run-xxx.json
|
||||
```
|
||||
|
||||
Imported runs are copied locally and included in all future aggregations, analyses, and comparisons.
|
||||
|
||||
## What It Measures
|
||||
|
||||
Each collection scrapes your spec files and extracts:
|
||||
|
||||
### Step 1 (Plan) — from `PLAN-DRAFT-*.md` and `PLAN-CONVERSATION-*.md`
|
||||
|
||||
- **Confidence score** — overall planning confidence percentage
|
||||
- **Confidence breakdown** — requirements, feasibility, integration, risk sub-scores
|
||||
- **Clarification rounds** — how many rounds of Q&A occurred
|
||||
- **Verification gaps found** — missing requirements caught during verification
|
||||
- **Functional/non-functional requirement counts** — FR- and NFR- headings
|
||||
- **Risk count** — risk table entries
|
||||
- **Phase count** — number of implementation phases planned
|
||||
|
||||
### Step 2 (Document) — from `overview.md` and `phase-*.md`
|
||||
|
||||
- **Total tasks** — checkbox count across all phases
|
||||
- **Tasks per phase** — distribution of work
|
||||
- **Phase count** — number of phase files
|
||||
- **Parallel groups** — parallel execution groups identified
|
||||
- **Requirement coverage** — coverage percentage if present
|
||||
- **Verification items** — verification checklist items added
|
||||
|
||||
### Step 3 (Implement) — from `.plan2code-loop/` data (loop mode only)
|
||||
|
||||
- **Task completion rate** — completed / total tasks
|
||||
- **Blocker count** — tasks marked TASK_BLOCKED
|
||||
- **Blocker categories** — normalized reasons for blocks
|
||||
- **Total iterations** — loop iteration count
|
||||
- **Avg iteration duration** — mean time per iteration
|
||||
- **Exit code distribution** — process exit codes
|
||||
- **Completion marker success rate** — valid markers / total iterations
|
||||
|
||||
### Step 4 (Finalize) — from `overview.md` and directory structure
|
||||
|
||||
- **Completion rate at audit** — tasks done at finalization time
|
||||
- **Verification failures** — tasks marked with `[!]`
|
||||
- **Documentation updates needed** — TODO/FIXME/update markers
|
||||
- **Archival success** — whether spec was moved to `specs--completed/`
|
||||
|
||||
### Health Targets
|
||||
|
||||
The dashboard compares key metrics against targets:
|
||||
|
||||
| Metric | Target | Direction |
|
||||
|--------|--------|-----------|
|
||||
| avg_confidence | 90 | >= |
|
||||
| avg_clarification_rounds | 2.0 | <= |
|
||||
| avg_verification_gaps_found | 2.0 | <= |
|
||||
| avg_parallel_groups | 0.5 | >= |
|
||||
| avg_task_completion_rate | 0.95 | >= |
|
||||
| avg_blocker_count | 1.5 | <= |
|
||||
| avg_completion_marker_success_rate | 0.95 | >= |
|
||||
| avg_verification_failures_found | 1.0 | <= |
|
||||
| archival_success_rate | 0.99 | >= |
|
||||
|
||||
Green checkmark = meeting target. Red X = below target (with the target shown).
|
||||
|
||||
## How It Works
|
||||
|
||||
### 1. Collect
|
||||
|
||||
Reads completed project artifacts from `specs/` or `specs--completed/` and extracts metrics for each workflow step. Output: `.plan2code-metrics/runs/run-<timestamp>.json`
|
||||
|
||||
### 2. Aggregate
|
||||
|
||||
Merges all run files into `aggregated.json`, grouped by **prompt generation** — a SHA fingerprint of the 8 workflow prompt files. Two runs that used identical prompt files belong to the same cohort.
|
||||
|
||||
### 3. Analyze
|
||||
|
||||
Sends aggregated metrics to an AI model with an analyst prompt. The AI produces a diagnosis identifying the weakest workflow steps and root causes. No edits are proposed at this stage.
|
||||
|
||||
Output: `.plan2code-metrics/proposals/<timestamp>-diagnosis.md`
|
||||
|
||||
### 4. Improve
|
||||
|
||||
Sends the diagnosis plus current prompt file contents to an AI model. The AI proposes surgical `old_text -> new_text` edits, each validated against:
|
||||
|
||||
- The `old_text` actually exists in the target file
|
||||
- The edited file stays under the 11,000 character limit
|
||||
|
||||
Output: `.plan2code-metrics/proposals/prop-<timestamp>.json`
|
||||
|
||||
### 5. Apply
|
||||
|
||||
Interactive diff review for each proposed edit. Accept, reject, or skip individual changes. Applied edits are patched directly into `src/plan2code-*.md`.
|
||||
|
||||
## Data Directory
|
||||
|
||||
All metrics data lives in `.plan2code-metrics/` (add to `.gitignore` if desired):
|
||||
|
||||
```
|
||||
.plan2code-metrics/
|
||||
├── runs/ # Individual run JSONs
|
||||
│ ├── run-20260215-120000-a1b2.json
|
||||
│ └── run-20260220-090000-c3d4.json
|
||||
├── aggregated.json # Merged cohort data
|
||||
└── proposals/ # Diagnoses and improvement proposals
|
||||
├── 20260220-diagnosis.md
|
||||
└── prop-20260220.json
|
||||
```
|
||||
|
||||
## AI Backends
|
||||
|
||||
The analysis and improvement steps require an AI agent. Two backends are supported:
|
||||
|
||||
| Backend | Command | Notes |
|
||||
|---------|---------|-------|
|
||||
| **Claude Code** | `claude` | Uses `--print` mode. Recommended. |
|
||||
| **GitHub Copilot CLI** | `copilot` | Uses stdin piping with `--allow-all-tools -s`. |
|
||||
| **Devin CLI** | `devin` | Uses `--print --prompt-file <file> --permission-mode dangerous`. |
|
||||
|
||||
Model selection is interactive — choose from available models when prompted.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
src/
|
||||
├── bin/plan2code-metrics.ts # Entry point
|
||||
├── cli.ts # Interactive menu (main loop)
|
||||
├── types.ts # RunMetrics, PromptProposal, CohortMetrics, etc.
|
||||
├── collector.ts # Reads project artifacts → run JSON
|
||||
├── aggregator.ts # Merges runs → aggregated.json (cohort grouping)
|
||||
├── analyzer.ts # AI diagnosis via LLM invocation
|
||||
├── improver.ts # AI improvement proposal + validation
|
||||
├── applier.ts # Interactive diff review + file patching
|
||||
├── community.ts # Community-feedback issue parsing + ingestion
|
||||
├── invoke-llm.ts # Unified LLM invocation (Claude / Copilot / Devin)
|
||||
├── index.ts # Public API exports
|
||||
└── prompts/
|
||||
├── analyze.md # AI prompt template for diagnosis
|
||||
└── improve.md # AI prompt template for improvement proposals
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
npm run dev # Watch mode (rebuilds on change)
|
||||
npm run build # Production build
|
||||
npm test # Run unit tests
|
||||
npm run test:watch # Watch mode tests
|
||||
npm start # Run the CLI
|
||||
```
|
||||
@@ -0,0 +1,46 @@
|
||||
{
|
||||
"name": "plan2code-metrics",
|
||||
"version": "1.0.0",
|
||||
"description": "Plan2Code Metrics - Recursive self-improvement toolchain for plan2code contributors",
|
||||
"type": "module",
|
||||
"main": "dist/index.js",
|
||||
"bin": {
|
||||
"plan2code-metrics": "./dist/bin/plan2code-metrics.js"
|
||||
},
|
||||
"files": [
|
||||
"dist"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18.0.0"
|
||||
},
|
||||
"keywords": [
|
||||
"cli",
|
||||
"ai",
|
||||
"metrics",
|
||||
"plan2code",
|
||||
"contributor-tooling"
|
||||
],
|
||||
"license": "MIT",
|
||||
"scripts": {
|
||||
"build": "tsup",
|
||||
"dev": "tsup --watch",
|
||||
"start": "node dist/bin/plan2code-metrics.js",
|
||||
"test": "vitest run",
|
||||
"test:watch": "vitest",
|
||||
"prepublishOnly": "npm run build"
|
||||
},
|
||||
"dependencies": {
|
||||
"@inquirer/prompts": "^8.1.0",
|
||||
"chalk": "^5.6.2",
|
||||
"execa": "^9.6.1",
|
||||
"fs-extra": "^11.3.3",
|
||||
"ora": "^9.0.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/fs-extra": "^11.0.4",
|
||||
"@types/node": "^25.0.3",
|
||||
"tsup": "^8.5.1",
|
||||
"typescript": "^5.9.3",
|
||||
"vitest": "^3.1.1"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,253 @@
|
||||
import fs from 'fs';
|
||||
import os from 'os';
|
||||
import path from 'path';
|
||||
import { describe, it, expect, afterEach } from 'vitest';
|
||||
import { avg, rate, buildCohortKey, backfillPromptVersions, cohortKeyForRun, aggregate } from './aggregator.js';
|
||||
import type { PromptVersions, RunMetrics } from './types.js';
|
||||
|
||||
// ── avg() ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
describe('avg', () => {
|
||||
it('returns null for empty array', () => {
|
||||
expect(avg([])).toBeNull();
|
||||
});
|
||||
|
||||
it('returns null for all-null/undefined values', () => {
|
||||
expect(avg([null, undefined, null])).toBeNull();
|
||||
});
|
||||
|
||||
it('computes correct average for valid numbers', () => {
|
||||
expect(avg([10, 20, 30])).toBe(20);
|
||||
});
|
||||
|
||||
it('filters out null/undefined/NaN from mixed arrays', () => {
|
||||
expect(avg([10, null, 20, undefined, NaN, 30])).toBe(20);
|
||||
});
|
||||
});
|
||||
|
||||
// ── rate() ────────────────────────────────────────────────────────────────────
|
||||
|
||||
describe('rate', () => {
|
||||
it('returns null for empty array', () => {
|
||||
expect(rate([])).toBeNull();
|
||||
});
|
||||
|
||||
it('returns null for all-null values', () => {
|
||||
expect(rate([null, null])).toBeNull();
|
||||
});
|
||||
|
||||
it('returns 1.0 for all-true', () => {
|
||||
expect(rate([true, true, true])).toBe(1.0);
|
||||
});
|
||||
|
||||
it('returns 0.0 for all-false', () => {
|
||||
expect(rate([false, false, false])).toBe(0.0);
|
||||
});
|
||||
|
||||
it('computes correct rate for mixed true/false', () => {
|
||||
expect(rate([true, false, true, false])).toBe(0.5);
|
||||
});
|
||||
|
||||
it('filters out null/undefined from mixed arrays', () => {
|
||||
expect(rate([true, null, false, undefined])).toBe(0.5);
|
||||
});
|
||||
});
|
||||
|
||||
// ── backfillPromptVersions() ──────────────────────────────────────────────────
|
||||
|
||||
const FULL_VERSIONS: PromptVersions = {
|
||||
plan: 'sha256:aaa',
|
||||
revise_plan: 'sha256:bbb',
|
||||
document: 'sha256:ccc',
|
||||
implement: 'sha256:ddd',
|
||||
finalize: 'sha256:eee',
|
||||
init: 'sha256:fff',
|
||||
init_update: 'sha256:ggg',
|
||||
quick_task: 'sha256:hhh',
|
||||
};
|
||||
|
||||
describe('backfillPromptVersions', () => {
|
||||
it('returns all 8 fields with sentinels when given empty-ish object', () => {
|
||||
const result = backfillPromptVersions({} as PromptVersions);
|
||||
expect(Object.keys(result)).toHaveLength(8);
|
||||
for (const val of Object.values(result)) {
|
||||
expect(val).toBe('sha256:missing');
|
||||
}
|
||||
});
|
||||
|
||||
it('preserves existing values, fills missing with sha256:missing', () => {
|
||||
const partial = { plan: 'sha256:aaa', implement: 'sha256:ddd' } as PromptVersions;
|
||||
const result = backfillPromptVersions(partial);
|
||||
expect(result.plan).toBe('sha256:aaa');
|
||||
expect(result.implement).toBe('sha256:ddd');
|
||||
expect(result.revise_plan).toBe('sha256:missing');
|
||||
expect(result.document).toBe('sha256:missing');
|
||||
expect(result.finalize).toBe('sha256:missing');
|
||||
expect(result.init).toBe('sha256:missing');
|
||||
expect(result.init_update).toBe('sha256:missing');
|
||||
expect(result.quick_task).toBe('sha256:missing');
|
||||
});
|
||||
|
||||
it('returns unchanged object when all 8 fields present', () => {
|
||||
const result = backfillPromptVersions(FULL_VERSIONS);
|
||||
expect(result).toEqual(FULL_VERSIONS);
|
||||
});
|
||||
});
|
||||
|
||||
// ── buildCohortKey() ──────────────────────────────────────────────────────────
|
||||
|
||||
describe('buildCohortKey', () => {
|
||||
it('returns 12-char hex string', () => {
|
||||
const key = buildCohortKey(FULL_VERSIONS);
|
||||
expect(key).toMatch(/^[0-9a-f]{12}$/);
|
||||
});
|
||||
|
||||
it('is deterministic (same input → same output)', () => {
|
||||
const key1 = buildCohortKey(FULL_VERSIONS);
|
||||
const key2 = buildCohortKey(FULL_VERSIONS);
|
||||
expect(key1).toBe(key2);
|
||||
});
|
||||
|
||||
it('old 4-field run with backfill sentinels produces same key as raw 4-field object', () => {
|
||||
// Simulate an old run that only had 4 fields
|
||||
const oldRun = {
|
||||
plan: 'sha256:aaa',
|
||||
implement: 'sha256:ddd',
|
||||
document: 'sha256:ccc',
|
||||
finalize: 'sha256:eee',
|
||||
} as PromptVersions;
|
||||
|
||||
// After backfill, the missing fields get 'sha256:missing'
|
||||
const backfilled = backfillPromptVersions(oldRun);
|
||||
|
||||
// buildCohortKey filters out 'sha256:missing', so both should match
|
||||
const keyDirect = buildCohortKey(oldRun);
|
||||
const keyBackfilled = buildCohortKey(backfilled);
|
||||
expect(keyDirect).toBe(keyBackfilled);
|
||||
});
|
||||
|
||||
it('different prompt versions → different keys', () => {
|
||||
const altered = { ...FULL_VERSIONS, plan: 'sha256:zzz' };
|
||||
expect(buildCohortKey(FULL_VERSIONS)).not.toBe(buildCohortKey(altered));
|
||||
});
|
||||
|
||||
it('key is independent of field insertion order', () => {
|
||||
const ordered: PromptVersions = {
|
||||
plan: 'sha256:aaa',
|
||||
revise_plan: 'sha256:bbb',
|
||||
document: 'sha256:ccc',
|
||||
implement: 'sha256:ddd',
|
||||
finalize: 'sha256:eee',
|
||||
init: 'sha256:fff',
|
||||
init_update: 'sha256:ggg',
|
||||
quick_task: 'sha256:hhh',
|
||||
};
|
||||
const reversed: PromptVersions = {
|
||||
quick_task: 'sha256:hhh',
|
||||
init_update: 'sha256:ggg',
|
||||
init: 'sha256:fff',
|
||||
finalize: 'sha256:eee',
|
||||
implement: 'sha256:ddd',
|
||||
document: 'sha256:ccc',
|
||||
revise_plan: 'sha256:bbb',
|
||||
plan: 'sha256:aaa',
|
||||
};
|
||||
expect(buildCohortKey(ordered)).toBe(buildCohortKey(reversed));
|
||||
});
|
||||
});
|
||||
|
||||
// ── Run fixtures for cohort keying / aggregation ──────────────────────────────
|
||||
|
||||
function makeRun(overrides: Partial<RunMetrics> = {}): RunMetrics {
|
||||
return {
|
||||
schema_version: '1.0',
|
||||
run_id: 'run-20260101-000000-0000',
|
||||
plan2code_version: '1.17.0',
|
||||
prompt_versions: { ...FULL_VERSIONS },
|
||||
project: { name: 'proj', started_at: null, completed_at: null },
|
||||
step1_plan: { present: false, final_confidence: null, confidence_breakdown: null, clarification_rounds: null, tech_stack_revision_rounds: null, verification_gaps_found: null, functional_requirements_count: null, non_functional_requirements_count: null, risk_count: null, phase_count: null },
|
||||
step2_document: { present: false, total_tasks: null, tasks_per_phase: null, phase_count: null, parallel_groups_identified: null, requirement_coverage_percent: null, verification_items_added: null },
|
||||
step3_implement: { present: false, task_completion_rate: null, tasks_completed: null, tasks_total: null, blocker_count: null },
|
||||
step4_finalize: { present: false, completion_rate_at_audit: null, verification_failures_found: null, documentation_updates_needed: null, archival_succeeded: null },
|
||||
user_feedback: null,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
// ── cohortKeyForRun() ─────────────────────────────────────────────────────────
|
||||
|
||||
describe('cohortKeyForRun', () => {
|
||||
it('keys local runs by the prompt-version hash (unchanged from buildCohortKey)', () => {
|
||||
const run = makeRun({ source: 'local' });
|
||||
expect(cohortKeyForRun(run)).toBe(buildCohortKey(run.prompt_versions));
|
||||
});
|
||||
|
||||
it('treats a run with no source as local', () => {
|
||||
const run = makeRun();
|
||||
delete run.source;
|
||||
expect(cohortKeyForRun(run)).toBe(buildCohortKey(run.prompt_versions));
|
||||
});
|
||||
|
||||
it('keys community runs by plan2code_version, ignoring prompt fingerprints', () => {
|
||||
const run = makeRun({ source: 'community', plan2code_version: '1.17.0' });
|
||||
expect(cohortKeyForRun(run)).toBe('community:v1.17.0');
|
||||
});
|
||||
|
||||
it('groups two community runs of the same version together regardless of prompt fingerprint', () => {
|
||||
const a = makeRun({ source: 'community', plan2code_version: '1.17.0', prompt_versions: { ...FULL_VERSIONS } });
|
||||
const b = makeRun({ source: 'community', plan2code_version: '1.17.0', prompt_versions: backfillPromptVersions({} as PromptVersions) });
|
||||
expect(cohortKeyForRun(a)).toBe(cohortKeyForRun(b));
|
||||
});
|
||||
|
||||
it('separates community runs from different versions', () => {
|
||||
const a = makeRun({ source: 'community', plan2code_version: '1.17.0' });
|
||||
const b = makeRun({ source: 'community', plan2code_version: '1.18.0' });
|
||||
expect(cohortKeyForRun(a)).not.toBe(cohortKeyForRun(b));
|
||||
});
|
||||
});
|
||||
|
||||
// ── aggregate() cohort separation ─────────────────────────────────────────────
|
||||
|
||||
describe('aggregate', () => {
|
||||
let tmpDir: string;
|
||||
|
||||
afterEach(() => {
|
||||
if (tmpDir) fs.rmSync(tmpDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function writeRuns(runs: RunMetrics[]): { runsDir: string; outPath: string } {
|
||||
tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'plan2code-agg-test-'));
|
||||
const runsDir = path.join(tmpDir, 'runs');
|
||||
fs.mkdirSync(runsDir, { recursive: true });
|
||||
for (const run of runs) {
|
||||
fs.writeFileSync(path.join(runsDir, `${run.run_id}.json`), JSON.stringify(run), 'utf8');
|
||||
}
|
||||
return { runsDir, outPath: path.join(tmpDir, 'aggregated.json') };
|
||||
}
|
||||
|
||||
it('places local and community runs of the same version in separate cohorts', () => {
|
||||
const local = makeRun({ run_id: 'run-20260101-000001-0001', source: 'local' });
|
||||
const community = makeRun({ run_id: 'run-20260101-000002-0002', source: 'community' });
|
||||
const { runsDir, outPath } = writeRuns([local, community]);
|
||||
|
||||
const result = aggregate(runsDir, outPath);
|
||||
|
||||
expect(result.total_runs).toBe(2);
|
||||
expect(result.cohorts).toHaveLength(2);
|
||||
const communityCohort = result.cohorts.find(c => c.source === 'community');
|
||||
const localCohort = result.cohorts.find(c => c.source === 'local');
|
||||
expect(communityCohort?.cohort_key).toBe('community:v1.17.0');
|
||||
expect(localCohort?.cohort_key).toBe(buildCohortKey(local.prompt_versions));
|
||||
});
|
||||
|
||||
it('never selects a community cohort as current when a local cohort exists', () => {
|
||||
// Community run sorts last by run_id, but current must stay on the local cohort.
|
||||
const local = makeRun({ run_id: 'run-20260101-000001-0001', source: 'local' });
|
||||
const community = makeRun({ run_id: 'run-29991231-235959-9999', source: 'community' });
|
||||
const { runsDir, outPath } = writeRuns([local, community]);
|
||||
|
||||
const result = aggregate(runsDir, outPath);
|
||||
|
||||
expect(result.current_cohort_key).toBe(buildCohortKey(local.prompt_versions));
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,253 @@
|
||||
/**
|
||||
* aggregator.ts
|
||||
* Merges all run JSON files into aggregated.json, grouped by prompt generation (SHA fingerprint).
|
||||
*/
|
||||
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import crypto from 'crypto';
|
||||
import type { RunMetrics, AggregatedMetrics, CohortMetrics, PromptVersions } from './types.js';
|
||||
|
||||
// ── Helpers ───────────────────────────────────────────────────────────────────
|
||||
|
||||
export function avg(values: (number | null | undefined)[]): number | null {
|
||||
const valid = values.filter((v): v is number => v != null && !isNaN(v));
|
||||
if (valid.length === 0) return null;
|
||||
return valid.reduce((a, b) => a + b, 0) / valid.length;
|
||||
}
|
||||
|
||||
export function rate(values: (boolean | null | undefined)[]): number | null {
|
||||
const valid = values.filter((v): v is boolean => v != null);
|
||||
if (valid.length === 0) return null;
|
||||
return valid.filter(v => v).length / valid.length;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a stable cohort key from the sorted prompt_versions object.
|
||||
* Two runs with identical prompt files get the same cohort key.
|
||||
*/
|
||||
export function buildCohortKey(promptVersions: PromptVersions): string {
|
||||
// Filter out missing sentinels so old runs (4 fields) keep their original key
|
||||
const entries = Object.entries(promptVersions)
|
||||
.filter(([, v]) => v !== 'sha256:missing')
|
||||
.sort(([a], [b]) => a.localeCompare(b));
|
||||
const str = JSON.stringify(Object.fromEntries(entries));
|
||||
return crypto.createHash('sha256').update(str).digest('hex').slice(0, 12);
|
||||
}
|
||||
|
||||
/**
|
||||
* Cohort key for a single run, chosen by the run's origin.
|
||||
*
|
||||
* Local runs are keyed by their prompt-file SHA-256 fingerprint, which captures
|
||||
* in-development prompt edits that share one unreleased version.
|
||||
*
|
||||
* Community submissions can't reproduce that byte-exact hash: they carry the
|
||||
* installed, platform-transformed prompts (not the raw src/*.md the local
|
||||
* collector hashes), and the payload is LLM-generated. They are keyed instead
|
||||
* by `plan2code_version` -- a reliable, byte-comparison-free identifier, since
|
||||
* every released version ships a fixed set of prompts. This keeps community
|
||||
* cohorts free of the CRLF/whitespace fragility a cross-machine hash would have.
|
||||
*/
|
||||
export function cohortKeyForRun(run: RunMetrics): string {
|
||||
if (run.source === 'community') {
|
||||
return `community:v${run.plan2code_version}`;
|
||||
}
|
||||
return buildCohortKey(run.prompt_versions);
|
||||
}
|
||||
|
||||
/**
|
||||
* Backfill missing PromptVersions fields for old run files (pre-v1.1).
|
||||
*/
|
||||
export function backfillPromptVersions(pv: PromptVersions): PromptVersions {
|
||||
return {
|
||||
plan: pv.plan ?? 'sha256:missing',
|
||||
revise_plan: pv.revise_plan ?? 'sha256:missing',
|
||||
document: pv.document ?? 'sha256:missing',
|
||||
implement: pv.implement ?? 'sha256:missing',
|
||||
finalize: pv.finalize ?? 'sha256:missing',
|
||||
init: pv.init ?? 'sha256:missing',
|
||||
init_update: pv.init_update ?? 'sha256:missing',
|
||||
quick_task: pv.quick_task ?? 'sha256:missing',
|
||||
};
|
||||
}
|
||||
|
||||
// ── Load run files ────────────────────────────────────────────────────────────
|
||||
|
||||
export function loadRunFiles(runsDir: string): RunMetrics[] {
|
||||
if (!fs.existsSync(runsDir)) return [];
|
||||
|
||||
const files = fs.readdirSync(runsDir)
|
||||
.filter(f => f.endsWith('.json') && f.startsWith('run-'))
|
||||
.sort();
|
||||
|
||||
const runs: RunMetrics[] = [];
|
||||
for (const file of files) {
|
||||
try {
|
||||
const content = fs.readFileSync(path.join(runsDir, file), 'utf8');
|
||||
const data = JSON.parse(content) as RunMetrics;
|
||||
data.prompt_versions = backfillPromptVersions(data.prompt_versions);
|
||||
runs.push(data);
|
||||
} catch (err) {
|
||||
console.warn(`Warning: could not parse ${file}: ${err}`);
|
||||
}
|
||||
}
|
||||
return runs;
|
||||
}
|
||||
|
||||
// ── Build cohort metrics ──────────────────────────────────────────────────────
|
||||
|
||||
function buildCohort(runs: RunMetrics[], cohortKey: string): CohortMetrics {
|
||||
const runIds = runs.map(r => r.run_id);
|
||||
const timestamps = runs
|
||||
.flatMap(r => [r.project.started_at, r.project.completed_at])
|
||||
.filter((t): t is string => t != null)
|
||||
.sort();
|
||||
|
||||
// Step 1 averages
|
||||
const avgConfidence = avg(runs.map(r => r.step1_plan.final_confidence));
|
||||
const avgClarification = avg(runs.map(r => r.step1_plan.clarification_rounds));
|
||||
const avgVerifGaps = avg(runs.map(r => r.step1_plan.verification_gaps_found));
|
||||
const avgFRCount = avg(runs.map(r => r.step1_plan.functional_requirements_count));
|
||||
const avgNFRCount = avg(runs.map(r => r.step1_plan.non_functional_requirements_count));
|
||||
const avgRiskCount = avg(runs.map(r => r.step1_plan.risk_count));
|
||||
const avgPhaseStep1 = avg(runs.map(r => r.step1_plan.phase_count));
|
||||
|
||||
// Step 2 averages
|
||||
const avgTotalTasks = avg(runs.map(r => r.step2_document.total_tasks));
|
||||
const avgPhaseStep2 = avg(runs.map(r => r.step2_document.phase_count));
|
||||
const avgParallelGroups = avg(runs.map(r => r.step2_document.parallel_groups_identified));
|
||||
const avgReqCoverage = avg(runs.map(r => r.step2_document.requirement_coverage_percent));
|
||||
const avgVerifItems = avg(runs.map(r => r.step2_document.verification_items_added));
|
||||
|
||||
// Step 3 averages
|
||||
const avgCompletionRate = avg(runs.map(r => r.step3_implement.task_completion_rate));
|
||||
const avgBlockerCount = avg(runs.map(r => r.step3_implement.blocker_count));
|
||||
|
||||
// Step 4 averages
|
||||
const avgCompletionAtAudit = avg(runs.map(r => r.step4_finalize.completion_rate_at_audit));
|
||||
const avgVerifFailures = avg(runs.map(r => r.step4_finalize.verification_failures_found));
|
||||
const avgDocUpdates = avg(runs.map(r => r.step4_finalize.documentation_updates_needed));
|
||||
const archivalRate = rate(runs.map(r => r.step4_finalize.archival_succeeded));
|
||||
|
||||
// User feedback
|
||||
const feedbackRuns = runs.filter(r => r.user_feedback != null);
|
||||
const avgUserRating = avg(feedbackRuns.map(r => r.user_feedback!.overall_rating));
|
||||
const feedbackCount = feedbackRuns.length;
|
||||
|
||||
return {
|
||||
cohort_key: cohortKey,
|
||||
source: runs[0].source ?? 'local',
|
||||
prompt_versions: runs[0].prompt_versions,
|
||||
run_count: runs.length,
|
||||
run_ids: runIds,
|
||||
first_seen: timestamps[0] ?? runs[0].run_id,
|
||||
last_seen: timestamps[timestamps.length - 1] ?? runs[runs.length - 1].run_id,
|
||||
|
||||
avg_confidence: avgConfidence,
|
||||
avg_clarification_rounds: avgClarification,
|
||||
avg_verification_gaps_found: avgVerifGaps,
|
||||
avg_functional_requirements_count: avgFRCount,
|
||||
avg_non_functional_requirements_count: avgNFRCount,
|
||||
avg_risk_count: avgRiskCount,
|
||||
avg_phase_count_step1: avgPhaseStep1,
|
||||
|
||||
avg_total_tasks: avgTotalTasks,
|
||||
avg_phase_count_step2: avgPhaseStep2,
|
||||
avg_parallel_groups: avgParallelGroups,
|
||||
avg_requirement_coverage_percent: avgReqCoverage,
|
||||
avg_verification_items_added: avgVerifItems,
|
||||
|
||||
avg_task_completion_rate: avgCompletionRate,
|
||||
avg_blocker_count: avgBlockerCount,
|
||||
|
||||
avg_completion_rate_at_audit: avgCompletionAtAudit,
|
||||
avg_verification_failures_found: avgVerifFailures,
|
||||
avg_documentation_updates_needed: avgDocUpdates,
|
||||
archival_success_rate: archivalRate,
|
||||
|
||||
avg_user_rating: avgUserRating,
|
||||
feedback_count: feedbackCount,
|
||||
};
|
||||
}
|
||||
|
||||
// ── Main aggregator ───────────────────────────────────────────────────────────
|
||||
|
||||
export function aggregate(runsDir: string, outputPath: string): AggregatedMetrics {
|
||||
const runs = loadRunFiles(runsDir);
|
||||
|
||||
// Group by cohort key
|
||||
const cohortMap = new Map<string, RunMetrics[]>();
|
||||
for (const run of runs) {
|
||||
const key = cohortKeyForRun(run);
|
||||
if (!cohortMap.has(key)) cohortMap.set(key, []);
|
||||
cohortMap.get(key)!.push(run);
|
||||
}
|
||||
|
||||
// Sort cohorts by first seen
|
||||
const cohorts: CohortMetrics[] = [];
|
||||
for (const [key, cohortRuns] of cohortMap) {
|
||||
cohorts.push(buildCohort(cohortRuns, key));
|
||||
}
|
||||
cohorts.sort((a, b) => a.first_seen.localeCompare(b.first_seen));
|
||||
|
||||
// Determine current cohort (most recent). Prefer local cohorts so an
|
||||
// ingested community submission never becomes the maintainer's "current
|
||||
// generation" for self-improvement; fall back to all cohorts if there are
|
||||
// no local runs yet.
|
||||
const localCohorts = cohorts.filter(c => c.source !== 'community');
|
||||
const currentPool = localCohorts.length > 0 ? localCohorts : cohorts;
|
||||
const currentCohortKey = currentPool.length > 0
|
||||
? currentPool[currentPool.length - 1].cohort_key
|
||||
: null;
|
||||
|
||||
const aggregated: AggregatedMetrics = {
|
||||
schema_version: '1.0',
|
||||
last_updated: new Date().toISOString(),
|
||||
total_runs: runs.length,
|
||||
cohorts,
|
||||
current_cohort_key: currentCohortKey,
|
||||
};
|
||||
|
||||
// Write to output path
|
||||
const dir = path.dirname(outputPath);
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
fs.writeFileSync(outputPath, JSON.stringify(aggregated, null, 2), 'utf8');
|
||||
|
||||
return aggregated;
|
||||
}
|
||||
|
||||
export function loadAggregated(outputPath: string): AggregatedMetrics | null {
|
||||
try {
|
||||
const content = fs.readFileSync(outputPath, 'utf8');
|
||||
return JSON.parse(content) as AggregatedMetrics;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Write a run to the local runs dir, deduped by run_id filename.
|
||||
* Returns true if written, false if a file for that run_id already existed.
|
||||
*/
|
||||
export function writeRunFile(run: RunMetrics, runsDir: string): boolean {
|
||||
const destPath = path.join(runsDir, `${run.run_id}.json`);
|
||||
|
||||
if (fs.existsSync(destPath)) {
|
||||
return false; // Already imported
|
||||
}
|
||||
|
||||
fs.mkdirSync(runsDir, { recursive: true });
|
||||
fs.writeFileSync(destPath, JSON.stringify(run, null, 2), 'utf8');
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Import a single run JSON from another project into the local runs dir.
|
||||
* Returns true if imported, false if already present.
|
||||
*/
|
||||
export function importRun(runJsonPath: string, runsDir: string): boolean {
|
||||
const content = fs.readFileSync(runJsonPath, 'utf8');
|
||||
const run = JSON.parse(content) as RunMetrics;
|
||||
run.prompt_versions = backfillPromptVersions(run.prompt_versions);
|
||||
return writeRunFile(run, runsDir);
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
/**
|
||||
* analyzer.ts
|
||||
* Reads aggregated.json + prompt file contents, builds analysis prompt, invokes AI.
|
||||
*/
|
||||
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import type { AggregatedMetrics } from './types.js';
|
||||
import { invokeLLM, type AgentType } from './invoke-llm.js';
|
||||
|
||||
const ANALYZE_PROMPT_PATH = new URL('../src/prompts/analyze.md', import.meta.url).pathname
|
||||
.replace(/^\/([A-Za-z]:)/, '$1'); // Fix Windows path
|
||||
|
||||
// ── Helpers ───────────────────────────────────────────────────────────────────
|
||||
|
||||
function readPromptFiles(plan2codeRoot: string): Record<string, string> {
|
||||
const srcDir = path.join(plan2codeRoot, 'src');
|
||||
const promptFiles = [
|
||||
'plan2code-1-plan.md',
|
||||
'plan2code-1b-revise-plan.md',
|
||||
'plan2code-2-document.md',
|
||||
'plan2code-3-implement.md',
|
||||
'plan2code-4-finalize.md',
|
||||
'plan2code-init.md',
|
||||
'plan2code-init-update.md',
|
||||
'plan2code-quick-task.md',
|
||||
];
|
||||
|
||||
const contents: Record<string, string> = {};
|
||||
for (const file of promptFiles) {
|
||||
try {
|
||||
contents[file] = fs.readFileSync(path.join(srcDir, file), 'utf8');
|
||||
} catch {
|
||||
contents[file] = '(file not found)';
|
||||
}
|
||||
}
|
||||
return contents;
|
||||
}
|
||||
|
||||
function interpolate(template: string, vars: Record<string, string>): string {
|
||||
let result = template;
|
||||
for (const [key, value] of Object.entries(vars)) {
|
||||
result = result.replaceAll(`{{${key}}}`, value);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
function generateDiagnosisId(): string {
|
||||
const now = new Date();
|
||||
const ts = now.toISOString().replace(/[-:T.Z]/g, '').slice(0, 14);
|
||||
return `diag-${ts}`;
|
||||
}
|
||||
|
||||
// ── Main analyzer ─────────────────────────────────────────────────────────────
|
||||
|
||||
export interface AnalyzerOptions {
|
||||
aggregatedPath: string; // Path to aggregated.json
|
||||
plan2codeRoot: string; // Path to plan2code repo root
|
||||
proposalsDir: string; // Where to save diagnosis output
|
||||
model?: string; // AI model to use (default: agent's default)
|
||||
agent?: AgentType; // Agent to use (default: claude-code)
|
||||
}
|
||||
|
||||
export async function runAnalysis(opts: AnalyzerOptions): Promise<string> {
|
||||
const { aggregatedPath, plan2codeRoot, proposalsDir, model = 'default', agent = 'claude-code' } = opts;
|
||||
|
||||
// Load aggregated metrics
|
||||
let aggregated: AggregatedMetrics | null = null;
|
||||
try {
|
||||
aggregated = JSON.parse(fs.readFileSync(aggregatedPath, 'utf8')) as AggregatedMetrics;
|
||||
} catch {
|
||||
throw new Error(`Could not read aggregated metrics at ${aggregatedPath}. Run "Collect metrics" and "Import run data" first.`);
|
||||
}
|
||||
|
||||
if (aggregated.total_runs === 0) {
|
||||
throw new Error('No runs in aggregated metrics. Collect and import some runs first.');
|
||||
}
|
||||
|
||||
// Read prompt files
|
||||
const promptContents = readPromptFiles(plan2codeRoot);
|
||||
|
||||
// Format for interpolation
|
||||
const promptContentsStr = Object.entries(promptContents)
|
||||
.map(([file, content]) => `## ${file}\n\n${content}`)
|
||||
.join('\n\n---\n\n');
|
||||
|
||||
const aggregatedStr = JSON.stringify(aggregated, null, 2);
|
||||
|
||||
// Load analyze prompt template
|
||||
let analyzeTemplate: string;
|
||||
try {
|
||||
analyzeTemplate = fs.readFileSync(ANALYZE_PROMPT_PATH, 'utf8');
|
||||
} catch {
|
||||
// Fallback: look relative to cwd
|
||||
const altPath = path.join(process.cwd(), 'src', 'prompts', 'analyze.md');
|
||||
analyzeTemplate = fs.readFileSync(altPath, 'utf8');
|
||||
}
|
||||
|
||||
const fullPrompt = interpolate(analyzeTemplate, {
|
||||
aggregatedMetrics: aggregatedStr,
|
||||
promptContents: promptContentsStr,
|
||||
});
|
||||
|
||||
// Invoke Claude
|
||||
console.log(`\nInvoking AI analysis (agent: ${agent}, model: ${model === 'default' ? 'user default' : model})...`);
|
||||
console.log('This may take a minute...\n');
|
||||
|
||||
let diagnosisContent: string;
|
||||
try {
|
||||
diagnosisContent = await invokeLLM({
|
||||
prompt: fullPrompt,
|
||||
model,
|
||||
agent,
|
||||
timeout: 300_000,
|
||||
});
|
||||
} catch (err) {
|
||||
throw new Error(`AI invocation failed: ${err instanceof Error ? err.message : String(err)}\n\nMake sure the ${agent} CLI is installed and authenticated.`);
|
||||
}
|
||||
|
||||
// Save diagnosis
|
||||
const diagId = generateDiagnosisId();
|
||||
fs.mkdirSync(proposalsDir, { recursive: true });
|
||||
const diagPath = path.join(proposalsDir, `${diagId}-diagnosis.md`);
|
||||
fs.writeFileSync(diagPath, diagnosisContent, 'utf8');
|
||||
|
||||
console.log(`Diagnosis saved to: ${diagPath}`);
|
||||
return diagPath;
|
||||
}
|
||||
@@ -0,0 +1,201 @@
|
||||
/**
|
||||
* applier.ts
|
||||
* Interactive review of a PromptProposal: display colored diff per edit,
|
||||
* user approves/rejects each edit, apply approved edits to src files.
|
||||
*/
|
||||
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import { confirm } from '@inquirer/prompts';
|
||||
import chalk from 'chalk';
|
||||
import type { PromptEdit, PromptProposal } from './types.js';
|
||||
import { validateEdit } from './improver.js';
|
||||
|
||||
const CHAR_LIMIT = 11_000;
|
||||
|
||||
// ── Diff display ──────────────────────────────────────────────────────────────
|
||||
|
||||
function displayEditDiff(edit: PromptEdit, index: number, total: number): void {
|
||||
console.log();
|
||||
console.log(chalk.bold(`─── Edit ${index + 1} of ${total} ───────────────────────────────────`));
|
||||
console.log(chalk.cyan(`File: ${edit.file}`));
|
||||
console.log(chalk.gray(`Rationale: ${edit.rationale}`));
|
||||
console.log(chalk.gray(`Expected impact: ${edit.expected_metric_impact}`));
|
||||
console.log(chalk.gray(`Char impact: ${edit.char_count_before} → ${edit.char_count_after} (${edit.char_count_delta >= 0 ? '+' : ''}${edit.char_count_delta})`));
|
||||
|
||||
// Show char count status
|
||||
if (edit.char_count_after > CHAR_LIMIT) {
|
||||
console.log(chalk.red(`⚠ WARNING: Would exceed ${CHAR_LIMIT} char limit!`));
|
||||
} else {
|
||||
const headroom = CHAR_LIMIT - edit.char_count_after;
|
||||
console.log(chalk.green(`✓ Char count OK (${headroom} chars headroom after edit)`));
|
||||
}
|
||||
|
||||
console.log();
|
||||
console.log(chalk.bold('── REMOVED (old_text) ──'));
|
||||
|
||||
// Show old text with line-level context
|
||||
const oldLines = edit.old_text.split('\n');
|
||||
for (const line of oldLines) {
|
||||
console.log(chalk.red('- ') + chalk.red(line));
|
||||
}
|
||||
|
||||
console.log();
|
||||
console.log(chalk.bold('── ADDED (new_text) ──'));
|
||||
|
||||
const newLines = edit.new_text.split('\n');
|
||||
for (const line of newLines) {
|
||||
console.log(chalk.green('+ ') + chalk.green(line));
|
||||
}
|
||||
|
||||
console.log();
|
||||
}
|
||||
|
||||
// ── Apply edit to file ────────────────────────────────────────────────────────
|
||||
|
||||
function applyEdit(edit: PromptEdit, plan2codeRoot: string): boolean {
|
||||
// Reject path traversal attempts
|
||||
if (edit.file.includes('..') || path.isAbsolute(edit.file)) {
|
||||
console.error(chalk.red(`✗ Rejected: "${edit.file}" contains path traversal or absolute path`));
|
||||
return false;
|
||||
}
|
||||
|
||||
const filePath = path.join(plan2codeRoot, 'src', edit.file);
|
||||
try {
|
||||
let content = fs.readFileSync(filePath, 'utf8');
|
||||
if (!content.includes(edit.old_text)) {
|
||||
console.error(chalk.red(`✗ Cannot apply: old_text not found in ${edit.file} (may have been modified by a previous edit)`));
|
||||
return false;
|
||||
}
|
||||
content = content.replace(edit.old_text, edit.new_text);
|
||||
|
||||
// Post-apply char count check
|
||||
if (content.length > CHAR_LIMIT) {
|
||||
console.error(chalk.red(`✗ Cannot apply: would exceed ${CHAR_LIMIT} char limit (${content.length} chars)`));
|
||||
return false;
|
||||
}
|
||||
|
||||
fs.writeFileSync(filePath, content, 'utf8');
|
||||
console.log(chalk.green(`✓ Applied edit to ${edit.file} (now ${content.length} chars)`));
|
||||
return true;
|
||||
} catch (err) {
|
||||
console.error(chalk.red(`✗ Failed to apply edit: ${err instanceof Error ? err.message : String(err)}`));
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// ── Main applier ──────────────────────────────────────────────────────────────
|
||||
|
||||
export interface ApplierOptions {
|
||||
proposalPath: string;
|
||||
plan2codeRoot: string;
|
||||
proposalsDir: string;
|
||||
}
|
||||
|
||||
export interface ApplierResult {
|
||||
approved: number;
|
||||
rejected: number;
|
||||
applied: number;
|
||||
failed: number;
|
||||
}
|
||||
|
||||
export async function reviewAndApply(opts: ApplierOptions): Promise<ApplierResult> {
|
||||
const { proposalPath, plan2codeRoot, proposalsDir } = opts;
|
||||
|
||||
// Load proposal
|
||||
let proposal: PromptProposal;
|
||||
try {
|
||||
proposal = JSON.parse(fs.readFileSync(proposalPath, 'utf8')) as PromptProposal;
|
||||
} catch {
|
||||
throw new Error(`Could not read proposal at ${proposalPath}`);
|
||||
}
|
||||
|
||||
if (proposal.proposals.length === 0) {
|
||||
console.log(chalk.yellow('No edits in this proposal.'));
|
||||
return { approved: 0, rejected: 0, applied: 0, failed: 0 };
|
||||
}
|
||||
|
||||
console.log();
|
||||
console.log(chalk.bold.cyan('=== Plan2Code Prompt Improvement Review ==='));
|
||||
console.log(chalk.gray(`Proposal: ${proposal.proposal_id}`));
|
||||
console.log(chalk.gray(`Created: ${proposal.created_at}`));
|
||||
console.log(chalk.gray(`Based on: ${proposal.based_on_runs.length} run(s)`));
|
||||
console.log(chalk.gray(`Edits: ${proposal.proposals.length}`));
|
||||
|
||||
// Re-validate all edits against current file state
|
||||
const promptContents: Record<string, string> = {};
|
||||
const srcDir = path.join(plan2codeRoot, 'src');
|
||||
for (const edit of proposal.proposals) {
|
||||
if (!promptContents[edit.file]) {
|
||||
try {
|
||||
promptContents[edit.file] = fs.readFileSync(path.join(srcDir, edit.file), 'utf8');
|
||||
} catch {
|
||||
promptContents[edit.file] = '';
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let approved = 0;
|
||||
let rejected = 0;
|
||||
let applied = 0;
|
||||
let failed = 0;
|
||||
|
||||
for (let i = 0; i < proposal.proposals.length; i++) {
|
||||
const edit = proposal.proposals[i];
|
||||
|
||||
// Re-validate
|
||||
const validation = validateEdit(edit, promptContents);
|
||||
if (!validation.valid) {
|
||||
console.log();
|
||||
console.log(chalk.red(`✗ Edit ${i + 1} is no longer valid (files may have changed):`));
|
||||
for (const err of validation.errors) {
|
||||
console.log(chalk.red(` - ${err}`));
|
||||
}
|
||||
rejected++;
|
||||
continue;
|
||||
}
|
||||
|
||||
displayEditDiff(edit, i, proposal.proposals.length);
|
||||
|
||||
const approve = await confirm({
|
||||
message: `Apply this edit to ${edit.file}?`,
|
||||
default: true,
|
||||
});
|
||||
|
||||
if (!approve) {
|
||||
console.log(chalk.gray('Skipped.'));
|
||||
rejected++;
|
||||
continue;
|
||||
}
|
||||
|
||||
approved++;
|
||||
const success = applyEdit(edit, plan2codeRoot);
|
||||
if (success) {
|
||||
applied++;
|
||||
// Update in-memory content to reflect the edit
|
||||
promptContents[edit.file] = promptContents[edit.file].replace(edit.old_text, edit.new_text);
|
||||
} else {
|
||||
failed++;
|
||||
}
|
||||
}
|
||||
|
||||
// Update proposal status
|
||||
proposal.status = applied > 0 ? 'applied' : 'rejected';
|
||||
fs.writeFileSync(proposalPath, JSON.stringify(proposal, null, 2), 'utf8');
|
||||
|
||||
// Summary
|
||||
console.log();
|
||||
console.log(chalk.bold('─── Review Complete ──────────────────────────────────'));
|
||||
console.log(`Approved: ${chalk.green(String(approved))} Rejected: ${chalk.red(String(rejected))} Applied: ${chalk.green(String(applied))} Failed: ${chalk.red(String(failed))}`);
|
||||
|
||||
if (applied > 0) {
|
||||
console.log();
|
||||
console.log(chalk.bold.cyan('Next steps — create a PR with your changes:'));
|
||||
console.log(chalk.gray(' git add src/'));
|
||||
console.log(chalk.gray(` git commit -m "metrics: apply prompt improvements (${proposal.proposal_id})"`));
|
||||
console.log(chalk.gray(' git push -u origin HEAD'));
|
||||
console.log(chalk.gray(' gh pr create --title "metrics: apply gen N improvements"'));
|
||||
}
|
||||
|
||||
return { approved, rejected, applied, failed };
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
import { runCLI } from '../cli.js';
|
||||
|
||||
async function main() {
|
||||
try {
|
||||
await runCLI();
|
||||
process.exit(0);
|
||||
} catch (err) {
|
||||
if (err instanceof Error && err.message.includes('User force closed')) {
|
||||
// Ctrl+C — exit cleanly
|
||||
process.exit(0);
|
||||
}
|
||||
console.error(err instanceof Error ? err.message : String(err));
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
main();
|
||||
@@ -0,0 +1,917 @@
|
||||
/**
|
||||
* cli.ts
|
||||
* 100% interactive menu-driven CLI for plan2code-metrics.
|
||||
* No flags — all inputs collected via @inquirer/prompts.
|
||||
*/
|
||||
|
||||
import fs from 'fs';
|
||||
import os from 'os';
|
||||
import path from 'path';
|
||||
import { select, input, confirm } from '@inquirer/prompts';
|
||||
import chalk from 'chalk';
|
||||
import ora from 'ora';
|
||||
import { execa } from 'execa';
|
||||
import { collectRun } from './collector.js';
|
||||
import { aggregate, loadAggregated, importRun, loadRunFiles, writeRunFile } from './aggregator.js';
|
||||
import { runAnalysis } from './analyzer.js';
|
||||
import { generateImprovement } from './improver.js';
|
||||
import { reviewAndApply } from './applier.js';
|
||||
import type { AggregatedMetrics, CohortMetrics } from './types.js';
|
||||
import { METRIC_TARGETS } from './types.js';
|
||||
import { AGENTS, type AgentType } from './invoke-llm.js';
|
||||
import { listCommunityIssues, closeIssue, ingestCommunityIssues } from './community.js';
|
||||
|
||||
// ── Session state (set at startup via interactive prompts) ───────────────────
|
||||
|
||||
let resolvedPlan2CodeRoot = '';
|
||||
let resolvedProjectRoot = '';
|
||||
|
||||
// ── Persisted user config (~/.plan2code-metrics.json) ──────────────────────
|
||||
|
||||
const USER_CONFIG_PATH = path.join(os.homedir(), '.plan2code-metrics.json');
|
||||
|
||||
interface UserConfig {
|
||||
plan2codeRepoPath?: string;
|
||||
}
|
||||
|
||||
function loadUserConfig(): UserConfig {
|
||||
try {
|
||||
return JSON.parse(fs.readFileSync(USER_CONFIG_PATH, 'utf8'));
|
||||
} catch {
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
function saveUserConfig(config: UserConfig): void {
|
||||
try {
|
||||
fs.writeFileSync(USER_CONFIG_PATH, JSON.stringify(config, null, 2), 'utf8');
|
||||
} catch { /* best-effort */ }
|
||||
}
|
||||
|
||||
// ── Defaults ──────────────────────────────────────────────────────────────────
|
||||
|
||||
const DEFAULT_METRICS_DIR = '.plan2code-metrics';
|
||||
const DEFAULT_RUNS_SUBDIR = 'runs';
|
||||
const DEFAULT_AGGREGATED_FILE = 'aggregated.json';
|
||||
const DEFAULT_PROPOSALS_SUBDIR = 'proposals';
|
||||
|
||||
function getMetricsDirs(baseDir = resolvedProjectRoot) {
|
||||
const metricsDir = path.join(baseDir, DEFAULT_METRICS_DIR);
|
||||
return {
|
||||
metricsDir,
|
||||
runsDir: path.join(metricsDir, DEFAULT_RUNS_SUBDIR),
|
||||
aggregatedPath: path.join(metricsDir, DEFAULT_AGGREGATED_FILE),
|
||||
proposalsDir: path.join(metricsDir, DEFAULT_PROPOSALS_SUBDIR),
|
||||
};
|
||||
}
|
||||
|
||||
// Detect plan2code root (either this directory or parent directories)
|
||||
function detectPlan2CodeRoot(): string | null {
|
||||
let dir = process.cwd();
|
||||
for (let i = 0; i < 5; i++) {
|
||||
const srcDir = path.join(dir, 'src');
|
||||
if (
|
||||
fs.existsSync(path.join(srcDir, 'plan2code-1-plan.md')) &&
|
||||
fs.existsSync(path.join(srcDir, 'plan2code-2-document.md'))
|
||||
) {
|
||||
return dir;
|
||||
}
|
||||
const parent = path.dirname(dir);
|
||||
if (parent === dir) break;
|
||||
dir = parent;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// Detect plan2code version
|
||||
function detectPlan2CodeVersion(plan2codeRoot: string): string {
|
||||
try {
|
||||
const versionPath = path.join(plan2codeRoot, 'version.json');
|
||||
const content = JSON.parse(fs.readFileSync(versionPath, 'utf8'));
|
||||
return content.version ?? '0.0.0';
|
||||
} catch {
|
||||
try {
|
||||
const pkg = JSON.parse(fs.readFileSync(path.join(plan2codeRoot, 'package.json'), 'utf8'));
|
||||
return pkg.version ?? '0.0.0';
|
||||
} catch {
|
||||
return '0.0.0';
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── Metric health helpers ─────────────────────────────────────────────────────
|
||||
|
||||
function metricStatus(key: string, value: number | null): string {
|
||||
if (value == null) return chalk.gray('N/A');
|
||||
const target = METRIC_TARGETS[key as keyof typeof METRIC_TARGETS];
|
||||
if (!target) return chalk.white(String(Math.round(value * 100) / 100));
|
||||
|
||||
const ok = target.direction === 'gte' ? value >= target.target : value <= target.target;
|
||||
const formatted = Number.isInteger(value) ? String(value) : value.toFixed(2);
|
||||
return ok ? chalk.green(`✓ ${formatted}`) : chalk.red(`✗ ${formatted} (target: ${target.direction === 'gte' ? '≥' : '≤'}${target.target})`);
|
||||
}
|
||||
|
||||
// ── Agent + model selection ───────────────────────────────────────────────────
|
||||
|
||||
async function selectAgentAndModel(): Promise<{ agent: AgentType; model: string }> {
|
||||
const agentChoice = await select({
|
||||
message: 'Which AI agent?',
|
||||
choices: Object.values(AGENTS).map(a => ({ name: a.displayName, value: a.name })),
|
||||
});
|
||||
|
||||
return { agent: agentChoice, model: AGENTS[agentChoice].defaultModel };
|
||||
}
|
||||
|
||||
// ── Flow: Collect metrics ─────────────────────────────────────────────────────
|
||||
|
||||
async function flowCollect(): Promise<void> {
|
||||
console.log();
|
||||
console.log(chalk.bold.cyan('── Collect Metrics ──'));
|
||||
|
||||
const sourceChoice = await select({
|
||||
message: 'Where is the completed project spec?',
|
||||
choices: [
|
||||
{ name: 'Archived spec (specs--completed/<feature-name>/)', value: 'archived' },
|
||||
{ name: 'Active spec directory (specs/<feature-name>/)', value: 'active' },
|
||||
{ name: 'Custom path', value: 'custom' },
|
||||
],
|
||||
});
|
||||
|
||||
let specDir: string;
|
||||
let projectName: string;
|
||||
|
||||
if (sourceChoice === 'active' || sourceChoice === 'archived') {
|
||||
const baseSubdir = sourceChoice === 'active' ? 'specs' : 'specs--completed';
|
||||
const baseDir = path.join(resolvedProjectRoot, baseSubdir);
|
||||
|
||||
if (!fs.existsSync(baseDir)) {
|
||||
console.log(chalk.red(`No ${baseSubdir}/ directory found in ${resolvedProjectRoot}`));
|
||||
return;
|
||||
}
|
||||
|
||||
const entries = fs.readdirSync(baseDir)
|
||||
.filter(f => fs.statSync(path.join(baseDir, f)).isDirectory());
|
||||
|
||||
if (entries.length === 0) {
|
||||
console.log(chalk.red(`No spec directories found in ${baseSubdir}/`));
|
||||
return;
|
||||
}
|
||||
|
||||
const chosen = await select({
|
||||
message: 'Select spec directory:',
|
||||
choices: entries.map(e => ({ name: e, value: e })),
|
||||
});
|
||||
|
||||
specDir = path.join(baseDir, chosen);
|
||||
projectName = chosen;
|
||||
} else {
|
||||
specDir = await input({
|
||||
message: 'Enter full path to spec directory:',
|
||||
validate: (v) => fs.existsSync(v) ? true : 'Directory not found',
|
||||
});
|
||||
projectName = await input({
|
||||
message: 'Project name (for metrics label):',
|
||||
default: path.basename(specDir),
|
||||
});
|
||||
}
|
||||
|
||||
// Detect plan2code root
|
||||
const plan2codeRoot = resolvedPlan2CodeRoot;
|
||||
const plan2codeVersion = detectPlan2CodeVersion(plan2codeRoot);
|
||||
|
||||
const { runsDir, aggregatedPath } = getMetricsDirs();
|
||||
|
||||
const spinner = ora('Collecting metrics from project artifacts...').start();
|
||||
try {
|
||||
const metrics = await collectRun({
|
||||
specDir,
|
||||
projectName,
|
||||
plan2codeRoot,
|
||||
plan2codeVersion,
|
||||
outputDir: runsDir,
|
||||
});
|
||||
|
||||
spinner.succeed(`Metrics collected: ${metrics.run_id}`);
|
||||
console.log(chalk.gray(` Saved to: ${runsDir}/${metrics.run_id}.json`));
|
||||
|
||||
// Re-aggregate
|
||||
aggregate(runsDir, aggregatedPath);
|
||||
console.log(chalk.gray(' Aggregated metrics updated.'));
|
||||
|
||||
// Show summary
|
||||
console.log();
|
||||
console.log(chalk.bold('Collected:'));
|
||||
console.log(` Step 1 (Plan): ${metrics.step1_plan.present ? chalk.green('✓') : chalk.gray('—')}`);
|
||||
console.log(` Step 2 (Document): ${metrics.step2_document.present ? chalk.green('✓') : chalk.gray('—')}`);
|
||||
console.log(` Step 3 (Implement): ${metrics.step3_implement.present ? chalk.green('✓') : chalk.gray('—')}`);
|
||||
console.log(` Step 4 (Finalize): ${metrics.step4_finalize.present ? chalk.green('✓') : chalk.gray('—')}`);
|
||||
console.log(` User Feedback: ${metrics.user_feedback ? chalk.green(`✓ (${metrics.user_feedback.overall_rating}/10)`) : chalk.gray('—')}`);
|
||||
|
||||
// Offer to collect feedback interactively if not present
|
||||
if (!metrics.user_feedback) {
|
||||
console.log();
|
||||
const wantFeedback = await confirm({
|
||||
message: 'No user feedback found in overview.md. Would you like to provide feedback now?',
|
||||
default: false,
|
||||
});
|
||||
|
||||
if (wantFeedback) {
|
||||
const rating = await input({
|
||||
message: 'Overall rating (1-10):',
|
||||
validate: (v) => {
|
||||
const n = parseInt(v, 10);
|
||||
return (n >= 1 && n <= 10) ? true : 'Must be a number between 1 and 10';
|
||||
},
|
||||
});
|
||||
const reason = await input({ message: 'Rating reason:' });
|
||||
const wentWell = await input({ message: 'What went well?' });
|
||||
const wentPoorly = await input({ message: 'What went poorly?' });
|
||||
|
||||
// Write the feedback table into overview.md
|
||||
const overviewPath = path.join(specDir, 'overview.md');
|
||||
let overviewContent = '';
|
||||
try {
|
||||
overviewContent = fs.readFileSync(overviewPath, 'utf8');
|
||||
} catch { /* empty */ }
|
||||
|
||||
// Escape pipe characters in user text to prevent table parsing issues
|
||||
const esc = (s: string) => s.replace(/\|/g, '\\|');
|
||||
|
||||
const feedbackTable = [
|
||||
'',
|
||||
'## User Feedback',
|
||||
'| Field | Value |',
|
||||
'|-------|-------|',
|
||||
`| Rating | ${rating} |`,
|
||||
`| Reason | ${esc(reason)} |`,
|
||||
`| Went Well | ${esc(wentWell)} |`,
|
||||
`| Went Poorly | ${esc(wentPoorly)} |`,
|
||||
'',
|
||||
].join('\n');
|
||||
|
||||
fs.writeFileSync(overviewPath, overviewContent + feedbackTable, 'utf8');
|
||||
console.log(chalk.green('✓ Feedback written to overview.md'));
|
||||
|
||||
// Delete the original run (without feedback) before re-collecting
|
||||
const originalRunPath = path.join(runsDir, `${metrics.run_id}.json`);
|
||||
try { fs.unlinkSync(originalRunPath); } catch { /* ignore */ }
|
||||
|
||||
// Re-collect and re-aggregate
|
||||
const reSpinner = ora('Re-collecting metrics with feedback...').start();
|
||||
const updatedMetrics = await collectRun({
|
||||
specDir,
|
||||
projectName,
|
||||
plan2codeRoot,
|
||||
plan2codeVersion,
|
||||
outputDir: runsDir,
|
||||
});
|
||||
aggregate(runsDir, aggregatedPath);
|
||||
reSpinner.succeed(`Updated: ${updatedMetrics.run_id} (rating: ${updatedMetrics.user_feedback?.overall_rating}/10)`);
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
spinner.fail(`Collection failed: ${err instanceof Error ? err.message : String(err)}`);
|
||||
}
|
||||
}
|
||||
|
||||
// ── Flow: Import run data ─────────────────────────────────────────────────────
|
||||
|
||||
async function flowImport(): Promise<void> {
|
||||
console.log();
|
||||
console.log(chalk.bold.cyan('── Import Run Data ──'));
|
||||
console.log(chalk.gray('Copy a run JSON from another project into this repo for aggregation.'));
|
||||
console.log();
|
||||
|
||||
const sourcePath = await input({
|
||||
message: 'Path to run JSON file (e.g., /path/to/project/.plan2code-metrics/runs/run-xxx.json):',
|
||||
validate: (v) => {
|
||||
if (!fs.existsSync(v)) return 'File not found';
|
||||
if (!v.endsWith('.json')) return 'Must be a .json file';
|
||||
return true;
|
||||
},
|
||||
});
|
||||
|
||||
const { runsDir, aggregatedPath } = getMetricsDirs();
|
||||
|
||||
try {
|
||||
const imported = importRun(sourcePath, runsDir);
|
||||
if (!imported) {
|
||||
console.log(chalk.yellow('Run already imported (same run_id exists).'));
|
||||
} else {
|
||||
console.log(chalk.green(`✓ Imported ${path.basename(sourcePath)}`));
|
||||
}
|
||||
|
||||
const spinner = ora('Re-aggregating...').start();
|
||||
const aggregated = aggregate(runsDir, aggregatedPath);
|
||||
spinner.succeed(`Aggregated metrics updated (${aggregated.total_runs} total runs)`);
|
||||
} catch (err) {
|
||||
console.error(chalk.red(`Import failed: ${err instanceof Error ? err.message : String(err)}`));
|
||||
}
|
||||
}
|
||||
|
||||
// ── Flow: View metrics status ─────────────────────────────────────────────────
|
||||
|
||||
async function flowViewStatus(): Promise<void> {
|
||||
console.log();
|
||||
console.log(chalk.bold.cyan('── Metrics Status & History ──'));
|
||||
|
||||
const { runsDir, aggregatedPath } = getMetricsDirs();
|
||||
const aggregated = loadAggregated(aggregatedPath);
|
||||
|
||||
if (!aggregated || aggregated.total_runs === 0) {
|
||||
console.log(chalk.yellow('No metrics collected yet. Use "Collect metrics" to get started.'));
|
||||
return;
|
||||
}
|
||||
|
||||
const localCount = aggregated.cohorts.filter(c => c.source !== 'community').length;
|
||||
const communityCount = aggregated.cohorts.length - localCount;
|
||||
|
||||
console.log();
|
||||
console.log(chalk.bold(
|
||||
`Total runs: ${aggregated.total_runs} | Generations: ${localCount}` +
|
||||
(communityCount > 0 ? ` | Community cohorts: ${communityCount}` : '')
|
||||
));
|
||||
console.log(chalk.gray(`Last updated: ${aggregated.last_updated}`));
|
||||
|
||||
// Community runs carry no wall-clock timestamps, so their first_seen/last_seen
|
||||
// fall back to the run_id string; only render a Period line for real ISO dates.
|
||||
const isIsoDate = (s?: string): boolean => !!s && /^\d{4}-\d{2}-\d{2}/.test(s);
|
||||
|
||||
for (let i = 0; i < aggregated.cohorts.length; i++) {
|
||||
const cohort = aggregated.cohorts[i];
|
||||
const isCurrent = cohort.cohort_key === aggregated.current_cohort_key;
|
||||
const label = isCurrent ? chalk.bold.green('[CURRENT]') : '';
|
||||
const isCommunity = cohort.source === 'community';
|
||||
|
||||
console.log();
|
||||
if (isCommunity) {
|
||||
// cohort_key is already `community:v<version>` — no sha: prefix.
|
||||
console.log(chalk.bold(`Community feedback (${cohort.cohort_key}) — ${cohort.run_count} run(s) ${label}`));
|
||||
} else {
|
||||
console.log(chalk.bold(`Generation ${i + 1} (sha:${cohort.cohort_key}) — ${cohort.run_count} run(s) ${label}`));
|
||||
}
|
||||
if (isIsoDate(cohort.first_seen)) {
|
||||
const end = isIsoDate(cohort.last_seen) ? cohort.last_seen.slice(0, 10) : cohort.first_seen.slice(0, 10);
|
||||
console.log(chalk.gray(` Period: ${cohort.first_seen.slice(0, 10)} → ${end}`));
|
||||
}
|
||||
|
||||
// Step 1 metrics
|
||||
if (cohort.avg_confidence != null || cohort.avg_clarification_rounds != null) {
|
||||
console.log(chalk.bold(' Step 1 (Plan):'));
|
||||
if (cohort.avg_confidence != null)
|
||||
console.log(` avg_confidence: ${metricStatus('avg_confidence', cohort.avg_confidence)}`);
|
||||
if (cohort.avg_clarification_rounds != null)
|
||||
console.log(` avg_clarification_rounds: ${metricStatus('avg_clarification_rounds', cohort.avg_clarification_rounds)}`);
|
||||
if (cohort.avg_verification_gaps_found != null)
|
||||
console.log(` avg_verification_gaps: ${metricStatus('avg_verification_gaps_found', cohort.avg_verification_gaps_found)}`);
|
||||
}
|
||||
|
||||
// Step 2 metrics
|
||||
if (cohort.avg_total_tasks != null || cohort.avg_parallel_groups != null) {
|
||||
console.log(chalk.bold(' Step 2 (Document):'));
|
||||
if (cohort.avg_total_tasks != null)
|
||||
console.log(` avg_total_tasks: ${chalk.white(cohort.avg_total_tasks.toFixed(1))}`);
|
||||
if (cohort.avg_parallel_groups != null)
|
||||
console.log(` avg_parallel_groups: ${metricStatus('avg_parallel_groups', cohort.avg_parallel_groups)}`);
|
||||
if (cohort.avg_verification_items_added != null)
|
||||
console.log(` avg_verification_items: ${metricStatus('avg_verification_items_added', cohort.avg_verification_items_added)}`);
|
||||
}
|
||||
|
||||
// Step 3 metrics
|
||||
if (cohort.avg_task_completion_rate != null || cohort.avg_blocker_count != null) {
|
||||
console.log(chalk.bold(' Step 3 (Implement):'));
|
||||
if (cohort.avg_task_completion_rate != null)
|
||||
console.log(` avg_task_completion_rate: ${metricStatus('avg_task_completion_rate', cohort.avg_task_completion_rate)}`);
|
||||
if (cohort.avg_blocker_count != null)
|
||||
console.log(` avg_blocker_count: ${metricStatus('avg_blocker_count', cohort.avg_blocker_count)}`);
|
||||
}
|
||||
|
||||
// Step 4 metrics
|
||||
if (cohort.avg_completion_rate_at_audit != null || cohort.archival_success_rate != null) {
|
||||
console.log(chalk.bold(' Step 4 (Finalize):'));
|
||||
if (cohort.avg_completion_rate_at_audit != null)
|
||||
console.log(` avg_completion_at_audit: ${chalk.white(cohort.avg_completion_rate_at_audit.toFixed(2))}`);
|
||||
if (cohort.avg_verification_failures_found != null)
|
||||
console.log(` avg_verif_failures: ${metricStatus('avg_verification_failures_found', cohort.avg_verification_failures_found)}`);
|
||||
if (cohort.archival_success_rate != null)
|
||||
console.log(` archival_success_rate: ${metricStatus('archival_success_rate', cohort.archival_success_rate)}`);
|
||||
}
|
||||
|
||||
// User feedback
|
||||
if (cohort.feedback_count > 0) {
|
||||
console.log(chalk.bold(' User Feedback:'));
|
||||
console.log(` avg_user_rating: ${metricStatus('avg_user_rating', cohort.avg_user_rating)}`);
|
||||
console.log(` feedback_count: ${chalk.white(String(cohort.feedback_count))}`);
|
||||
}
|
||||
|
||||
// Compare with the previous cohort -- but only within the same population.
|
||||
// Community and local cohorts measure different things; a cross-source
|
||||
// delta (e.g. a community cohort vs the last local generation) is noise.
|
||||
if (i > 0 && aggregated.cohorts[i - 1].source === cohort.source) {
|
||||
const prev = aggregated.cohorts[i - 1];
|
||||
const deltas: string[] = [];
|
||||
if (cohort.avg_confidence != null && prev.avg_confidence != null) {
|
||||
const d = cohort.avg_confidence - prev.avg_confidence;
|
||||
deltas.push(`confidence ${d >= 0 ? chalk.green(`▲${d.toFixed(1)}`) : chalk.red(`▼${Math.abs(d).toFixed(1)}`)}`);
|
||||
}
|
||||
if (cohort.avg_task_completion_rate != null && prev.avg_task_completion_rate != null) {
|
||||
const d = cohort.avg_task_completion_rate - prev.avg_task_completion_rate;
|
||||
deltas.push(`completion ${d >= 0 ? chalk.green(`▲${(d * 100).toFixed(1)}%`) : chalk.red(`▼${(Math.abs(d) * 100).toFixed(1)}%`)}`);
|
||||
}
|
||||
if (deltas.length > 0) {
|
||||
console.log(chalk.gray(` ${isCommunity ? 'vs prior version' : `vs Gen ${i}`}: ${deltas.join(' ')}`));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Run list
|
||||
const runs = loadRunFiles(runsDir);
|
||||
if (runs.length > 0) {
|
||||
console.log();
|
||||
console.log(chalk.bold(`Run files (${runs.length}):`));
|
||||
for (const run of runs.slice(-10)) {
|
||||
console.log(chalk.gray(` ${run.run_id} ${run.project.name} v${run.plan2code_version}`));
|
||||
}
|
||||
if (runs.length > 10) {
|
||||
console.log(chalk.gray(` ... and ${runs.length - 10} more`));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── Flow: Run analysis ────────────────────────────────────────────────────────
|
||||
|
||||
async function flowRunAnalysis(): Promise<string | null> {
|
||||
console.log();
|
||||
console.log(chalk.bold.cyan('── Run Analysis (Diagnose Weak Steps) ──'));
|
||||
|
||||
const { aggregatedPath, proposalsDir } = getMetricsDirs();
|
||||
const plan2codeRoot = resolvedPlan2CodeRoot;
|
||||
|
||||
const aggregated = loadAggregated(aggregatedPath);
|
||||
if (!aggregated || aggregated.total_runs === 0) {
|
||||
console.log(chalk.yellow('No aggregated metrics found. Collect and import runs first.'));
|
||||
return null;
|
||||
}
|
||||
|
||||
if (aggregated.total_runs < 3) {
|
||||
const proceed = await confirm({
|
||||
message: `Only ${aggregated.total_runs} run(s) available (≥3 recommended for reliable analysis). Proceed anyway?`,
|
||||
default: false,
|
||||
});
|
||||
if (!proceed) return null;
|
||||
}
|
||||
|
||||
const { agent, model } = await selectAgentAndModel();
|
||||
|
||||
try {
|
||||
const diagPath = await runAnalysis({
|
||||
aggregatedPath,
|
||||
plan2codeRoot,
|
||||
proposalsDir,
|
||||
model,
|
||||
agent,
|
||||
});
|
||||
|
||||
console.log();
|
||||
console.log(chalk.green('✓ Analysis complete'));
|
||||
console.log(chalk.gray(`Diagnosis: ${diagPath}`));
|
||||
|
||||
const viewNow = await confirm({
|
||||
message: 'Open diagnosis in console?',
|
||||
default: true,
|
||||
});
|
||||
|
||||
if (viewNow) {
|
||||
const content = fs.readFileSync(diagPath, 'utf8');
|
||||
console.log();
|
||||
console.log(chalk.dim('─'.repeat(60)));
|
||||
console.log(content);
|
||||
console.log(chalk.dim('─'.repeat(60)));
|
||||
}
|
||||
|
||||
return diagPath;
|
||||
} catch (err) {
|
||||
console.error(chalk.red(`Analysis failed: ${err instanceof Error ? err.message : String(err)}`));
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// ── Flow: Generate improvement proposal ──────────────────────────────────────
|
||||
|
||||
async function flowGenerateProposal(): Promise<void> {
|
||||
console.log();
|
||||
console.log(chalk.bold.cyan('── Generate Improvement Proposal ──'));
|
||||
|
||||
const { aggregatedPath, proposalsDir, runsDir } = getMetricsDirs();
|
||||
const plan2codeRoot = resolvedPlan2CodeRoot;
|
||||
|
||||
// Find diagnosis files
|
||||
let diagFiles: string[] = [];
|
||||
if (fs.existsSync(proposalsDir)) {
|
||||
diagFiles = fs.readdirSync(proposalsDir)
|
||||
.filter(f => f.endsWith('-diagnosis.md'))
|
||||
.sort()
|
||||
.reverse(); // most recent first
|
||||
}
|
||||
|
||||
let diagnosisPath: string;
|
||||
|
||||
if (diagFiles.length === 0) {
|
||||
console.log(chalk.yellow('No diagnosis files found. Running analysis first...'));
|
||||
const diagPath = await flowRunAnalysis();
|
||||
if (!diagPath) return;
|
||||
diagnosisPath = diagPath;
|
||||
} else {
|
||||
const choice = await select({
|
||||
message: 'Select diagnosis to base proposal on:',
|
||||
choices: [
|
||||
...diagFiles.map(f => ({ name: f, value: path.join(proposalsDir, f) })),
|
||||
{ name: '(run new analysis first)', value: '__new__' },
|
||||
],
|
||||
});
|
||||
|
||||
if (choice === '__new__') {
|
||||
const diagPath = await flowRunAnalysis();
|
||||
if (!diagPath) return;
|
||||
diagnosisPath = diagPath;
|
||||
} else {
|
||||
diagnosisPath = choice;
|
||||
}
|
||||
}
|
||||
|
||||
const { agent, model } = await selectAgentAndModel();
|
||||
|
||||
try {
|
||||
const result = await generateImprovement({
|
||||
diagnosisPath,
|
||||
plan2codeRoot,
|
||||
proposalsDir,
|
||||
runsDir,
|
||||
model,
|
||||
agent,
|
||||
});
|
||||
|
||||
console.log();
|
||||
console.log(chalk.green(`✓ Proposal generated: ${result.proposal.proposal_id}`));
|
||||
console.log(chalk.gray(` Valid edits: ${result.validEditCount}`));
|
||||
console.log(chalk.gray(` Invalid edits: ${result.invalidEditCount} (rejected)`));
|
||||
console.log(chalk.gray(` Saved to: ${result.proposalPath}`));
|
||||
|
||||
if (result.validEditCount > 0) {
|
||||
const reviewNow = await confirm({
|
||||
message: 'Review and apply edits now?',
|
||||
default: true,
|
||||
});
|
||||
|
||||
if (reviewNow) {
|
||||
await reviewAndApply({
|
||||
proposalPath: result.proposalPath,
|
||||
plan2codeRoot,
|
||||
proposalsDir,
|
||||
});
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
console.error(chalk.red(`Proposal generation failed: ${err instanceof Error ? err.message : String(err)}`));
|
||||
}
|
||||
}
|
||||
|
||||
// ── Flow: Review and apply proposal ──────────────────────────────────────────
|
||||
|
||||
async function flowReviewAndApply(): Promise<void> {
|
||||
console.log();
|
||||
console.log(chalk.bold.cyan('── Review and Apply a Proposal ──'));
|
||||
|
||||
const { proposalsDir } = getMetricsDirs();
|
||||
const plan2codeRoot = resolvedPlan2CodeRoot;
|
||||
|
||||
if (!fs.existsSync(proposalsDir)) {
|
||||
console.log(chalk.yellow('No proposals directory found. Generate a proposal first.'));
|
||||
return;
|
||||
}
|
||||
|
||||
const proposalFiles = fs.readdirSync(proposalsDir)
|
||||
.filter(f => /^prop-\d+\.json$/.test(f))
|
||||
.sort()
|
||||
.reverse(); // most recent first
|
||||
|
||||
if (proposalFiles.length === 0) {
|
||||
console.log(chalk.yellow('No proposal files found. Generate a proposal first.'));
|
||||
return;
|
||||
}
|
||||
|
||||
const chosen = await select({
|
||||
message: 'Select proposal to review:',
|
||||
choices: proposalFiles.map(f => {
|
||||
try {
|
||||
const p = JSON.parse(fs.readFileSync(path.join(proposalsDir, f), 'utf8'));
|
||||
const label = `${f} [${p.status}] ${p.proposals?.length ?? 0} edits`;
|
||||
return { name: label, value: path.join(proposalsDir, f) };
|
||||
} catch {
|
||||
return { name: f, value: path.join(proposalsDir, f) };
|
||||
}
|
||||
}),
|
||||
});
|
||||
|
||||
await reviewAndApply({
|
||||
proposalPath: chosen,
|
||||
plan2codeRoot,
|
||||
proposalsDir,
|
||||
});
|
||||
}
|
||||
|
||||
// ── Flow: Delete metrics data ─────────────────────────────────────────────────
|
||||
|
||||
async function flowDelete(): Promise<void> {
|
||||
console.log();
|
||||
console.log(chalk.bold.cyan('── Delete Metrics Data ──'));
|
||||
|
||||
const { metricsDir, runsDir, aggregatedPath, proposalsDir } = getMetricsDirs();
|
||||
|
||||
if (!fs.existsSync(metricsDir)) {
|
||||
console.log(chalk.yellow('No metrics data found (.plan2code-metrics/ does not exist).'));
|
||||
return;
|
||||
}
|
||||
|
||||
const scope = await select({
|
||||
message: 'What do you want to delete?',
|
||||
choices: [
|
||||
{ name: 'Delete specific run(s)', value: 'select' },
|
||||
{ name: 'Delete all runs and aggregated data', value: 'runs' },
|
||||
{ name: 'Delete everything (runs, aggregated data, proposals)', value: 'all' },
|
||||
{ name: 'Cancel', value: 'cancel' },
|
||||
],
|
||||
});
|
||||
|
||||
if (scope === 'cancel') return;
|
||||
|
||||
if (scope === 'select') {
|
||||
const runs = loadRunFiles(runsDir);
|
||||
if (runs.length === 0) {
|
||||
console.log(chalk.yellow('No run files found.'));
|
||||
return;
|
||||
}
|
||||
|
||||
const choices = runs.map(r => ({
|
||||
name: `${r.run_id} ${r.project.name} v${r.plan2code_version}`,
|
||||
value: r.run_id,
|
||||
}));
|
||||
|
||||
// Select runs one at a time since @inquirer/prompts select is single-choice
|
||||
const toDelete: string[] = [];
|
||||
let selecting = true;
|
||||
while (selecting) {
|
||||
const remaining = choices.filter(c => !toDelete.includes(c.value));
|
||||
if (remaining.length === 0) break;
|
||||
|
||||
const chosen = await select({
|
||||
message: `Select a run to delete (${toDelete.length} selected so far):`,
|
||||
choices: [
|
||||
...remaining,
|
||||
{ name: toDelete.length > 0 ? `Done selecting (delete ${toDelete.length})` : 'Cancel', value: '__done__' },
|
||||
],
|
||||
});
|
||||
|
||||
if (chosen === '__done__') {
|
||||
selecting = false;
|
||||
} else {
|
||||
toDelete.push(chosen);
|
||||
console.log(chalk.gray(` + ${chosen}`));
|
||||
}
|
||||
}
|
||||
|
||||
if (toDelete.length === 0) return;
|
||||
|
||||
const confirmed = await confirm({
|
||||
message: `Delete ${toDelete.length} run(s)? This cannot be undone.`,
|
||||
default: false,
|
||||
});
|
||||
if (!confirmed) return;
|
||||
|
||||
let deleted = 0;
|
||||
for (const runId of toDelete) {
|
||||
const filePath = path.join(runsDir, `${runId}.json`);
|
||||
try {
|
||||
fs.unlinkSync(filePath);
|
||||
deleted++;
|
||||
} catch {
|
||||
console.log(chalk.yellow(` Could not delete ${runId}.json`));
|
||||
}
|
||||
}
|
||||
|
||||
console.log(chalk.green(`✓ Deleted ${deleted} run(s).`));
|
||||
|
||||
// Re-aggregate with remaining runs
|
||||
const remainingRuns = loadRunFiles(runsDir);
|
||||
if (remainingRuns.length > 0) {
|
||||
aggregate(runsDir, aggregatedPath);
|
||||
console.log(chalk.gray(` Aggregated metrics updated (${remainingRuns.length} runs remaining).`));
|
||||
} else {
|
||||
try { fs.unlinkSync(aggregatedPath); } catch { /* ignore */ }
|
||||
console.log(chalk.gray(' No runs remaining — aggregated data removed.'));
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// scope === 'runs' or 'all'
|
||||
const label = scope === 'all'
|
||||
? 'ALL metrics data (runs, aggregated data, and proposals)'
|
||||
: 'all runs and aggregated data';
|
||||
|
||||
const confirmed = await confirm({
|
||||
message: `Delete ${label}? This cannot be undone.`,
|
||||
default: false,
|
||||
});
|
||||
if (!confirmed) return;
|
||||
|
||||
// Delete run files
|
||||
let runCount = 0;
|
||||
if (fs.existsSync(runsDir)) {
|
||||
const files = fs.readdirSync(runsDir).filter(f => f.endsWith('.json'));
|
||||
for (const f of files) {
|
||||
try { fs.unlinkSync(path.join(runsDir, f)); runCount++; } catch { /* ignore */ }
|
||||
}
|
||||
}
|
||||
|
||||
// Delete aggregated file
|
||||
try { fs.unlinkSync(aggregatedPath); } catch { /* ignore */ }
|
||||
|
||||
console.log(chalk.green(`✓ Deleted ${runCount} run(s) and aggregated data.`));
|
||||
|
||||
if (scope === 'all') {
|
||||
// Delete proposals
|
||||
let proposalCount = 0;
|
||||
if (fs.existsSync(proposalsDir)) {
|
||||
const files = fs.readdirSync(proposalsDir);
|
||||
for (const f of files) {
|
||||
try { fs.unlinkSync(path.join(proposalsDir, f)); proposalCount++; } catch { /* ignore */ }
|
||||
}
|
||||
try { fs.rmdirSync(proposalsDir); } catch { /* ignore */ }
|
||||
}
|
||||
console.log(chalk.green(`✓ Deleted ${proposalCount} proposal file(s).`));
|
||||
}
|
||||
}
|
||||
|
||||
// ── Flow: Fetch community submissions ────────────────────────────────────────
|
||||
|
||||
const COMMUNITY_REPO = 'jparkerweb/plan2code';
|
||||
|
||||
async function flowFetchCommunitySubmissions(): Promise<void> {
|
||||
console.log();
|
||||
console.log(chalk.bold.cyan('── Fetch Community Submissions ──'));
|
||||
|
||||
try {
|
||||
await execa('gh', ['auth', 'status']);
|
||||
} catch {
|
||||
console.log(chalk.red('`gh` CLI not found or not authenticated — install/auth `gh` to use this feature.'));
|
||||
return;
|
||||
}
|
||||
|
||||
let issues;
|
||||
try {
|
||||
issues = await listCommunityIssues(COMMUNITY_REPO);
|
||||
} catch (err) {
|
||||
console.log(chalk.red(`Failed to list community-feedback issues: ${err instanceof Error ? err.message : String(err)}`));
|
||||
return;
|
||||
}
|
||||
|
||||
if (issues.length === 0) {
|
||||
console.log(chalk.yellow('No open community-feedback issues found.'));
|
||||
return;
|
||||
}
|
||||
|
||||
const { runsDir, aggregatedPath } = getMetricsDirs();
|
||||
|
||||
const tally = await ingestCommunityIssues(issues, COMMUNITY_REPO, runsDir, { writeRunFile, closeIssue });
|
||||
|
||||
for (const num of tally.malformedIssues) {
|
||||
console.log(chalk.yellow(` Skipping issue #${num}: malformed or missing METRICS_JSON payload.`));
|
||||
}
|
||||
for (const num of tally.closeFailedIssues) {
|
||||
console.log(chalk.yellow(` Imported issue #${num} but failed to close it (still open; will retry next fetch).`));
|
||||
}
|
||||
|
||||
if (tally.imported > 0) {
|
||||
aggregate(runsDir, aggregatedPath);
|
||||
}
|
||||
|
||||
console.log();
|
||||
console.log(chalk.green(
|
||||
`✓ Imported: ${tally.imported} Skipped (duplicate): ${tally.skippedDuplicate} Skipped (malformed): ${tally.skippedMalformed} Closed: ${tally.closed} Close failed: ${tally.closeFailed}`
|
||||
));
|
||||
}
|
||||
|
||||
// ── Main menu ─────────────────────────────────────────────────────────────────
|
||||
|
||||
export async function runCLI(): Promise<void> {
|
||||
console.log();
|
||||
console.log(chalk.bold.white('plan2code-metrics'));
|
||||
console.log(chalk.gray('Recursive self-improvement toolchain for plan2code contributors'));
|
||||
console.log();
|
||||
|
||||
// ── Prompt for paths ────────────────────────────────────────────────────────
|
||||
|
||||
const userConfig = loadUserConfig();
|
||||
const savedRoot = userConfig.plan2codeRepoPath && fs.existsSync(userConfig.plan2codeRepoPath)
|
||||
? userConfig.plan2codeRepoPath
|
||||
: null;
|
||||
const detectedRoot = savedRoot ?? detectPlan2CodeRoot();
|
||||
|
||||
const s2cPath = await input({
|
||||
message: 'Path to plan2code repo:',
|
||||
default: detectedRoot ?? undefined,
|
||||
validate: (v) => {
|
||||
if (!v.trim()) return 'Path is required';
|
||||
const resolved = path.resolve(v.trim());
|
||||
if (!fs.existsSync(resolved)) return 'Directory not found';
|
||||
return true;
|
||||
},
|
||||
});
|
||||
resolvedPlan2CodeRoot = path.resolve(s2cPath.trim());
|
||||
|
||||
// Persist the path for next run
|
||||
if (resolvedPlan2CodeRoot !== userConfig.plan2codeRepoPath) {
|
||||
saveUserConfig({ ...userConfig, plan2codeRepoPath: resolvedPlan2CodeRoot });
|
||||
}
|
||||
|
||||
// Warn if the path doesn't look like a plan2code repo
|
||||
const hasSrcPrompts =
|
||||
fs.existsSync(path.join(resolvedPlan2CodeRoot, 'src', 'plan2code-1-plan.md')) &&
|
||||
fs.existsSync(path.join(resolvedPlan2CodeRoot, 'src', 'plan2code-2-document.md'));
|
||||
if (!hasSrcPrompts) {
|
||||
console.log(chalk.yellow('⚠ No src/plan2code-*.md prompts found at that path. Hashing and analysis may be limited.'));
|
||||
}
|
||||
|
||||
const projPath = await input({
|
||||
message: 'Path to project repo (metrics source):',
|
||||
default: process.cwd(),
|
||||
validate: (v) => {
|
||||
if (!v.trim()) return 'Path is required';
|
||||
const resolved = path.resolve(v.trim());
|
||||
if (!fs.existsSync(resolved)) return 'Directory not found';
|
||||
return true;
|
||||
},
|
||||
});
|
||||
resolvedProjectRoot = path.resolve(projPath.trim());
|
||||
|
||||
// Display resolved paths
|
||||
const version = detectPlan2CodeVersion(resolvedPlan2CodeRoot);
|
||||
console.log();
|
||||
console.log(chalk.gray(`plan2code repo: ${resolvedPlan2CodeRoot} (v${version})`));
|
||||
console.log(chalk.gray(`project repo: ${resolvedProjectRoot}`));
|
||||
console.log();
|
||||
|
||||
let continueLoop = true;
|
||||
while (continueLoop) {
|
||||
const action = await select({
|
||||
message: 'What would you like to do?',
|
||||
choices: [
|
||||
{ name: 'Collect metrics for a completed project', value: 'collect' },
|
||||
{ name: 'Import run data from another project', value: 'import' },
|
||||
{ name: 'View metrics status and history', value: 'view' },
|
||||
{ name: 'Run analysis (diagnose weak steps)', value: 'analyze' },
|
||||
{ name: 'Generate improvement proposal', value: 'propose' },
|
||||
{ name: 'Review and apply a proposal', value: 'apply' },
|
||||
{ name: 'Fetch community submissions', value: 'fetch-community' },
|
||||
{ name: chalk.red('Delete metrics data'), value: 'delete' },
|
||||
{ name: 'Exit', value: 'exit' },
|
||||
],
|
||||
});
|
||||
|
||||
switch (action) {
|
||||
case 'collect':
|
||||
await flowCollect();
|
||||
break;
|
||||
case 'import':
|
||||
await flowImport();
|
||||
break;
|
||||
case 'view':
|
||||
await flowViewStatus();
|
||||
break;
|
||||
case 'analyze':
|
||||
await flowRunAnalysis();
|
||||
break;
|
||||
case 'propose':
|
||||
await flowGenerateProposal();
|
||||
break;
|
||||
case 'apply':
|
||||
await flowReviewAndApply();
|
||||
break;
|
||||
case 'fetch-community':
|
||||
await flowFetchCommunitySubmissions();
|
||||
break;
|
||||
case 'delete':
|
||||
await flowDelete();
|
||||
break;
|
||||
case 'exit':
|
||||
continueLoop = false;
|
||||
break;
|
||||
}
|
||||
|
||||
if (continueLoop && action !== 'exit') {
|
||||
console.log();
|
||||
}
|
||||
}
|
||||
|
||||
console.log(chalk.gray('Goodbye.'));
|
||||
}
|
||||
@@ -0,0 +1,634 @@
|
||||
/**
|
||||
* collector.ts
|
||||
* Reads finished project artifacts and writes a RunMetrics JSON file.
|
||||
* Zero dependency on plan2code-loop internals — reads files directly.
|
||||
*/
|
||||
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import crypto from 'crypto';
|
||||
import type {
|
||||
RunMetrics,
|
||||
PromptVersions,
|
||||
Step1PlanMetrics,
|
||||
Step2DocumentMetrics,
|
||||
Step3ImplementMetrics,
|
||||
Step4FinalizeMetrics,
|
||||
UserFeedback,
|
||||
} from './types.js';
|
||||
|
||||
// ── Helpers ──────────────────────────────────────────────────────────────────
|
||||
|
||||
function sha256File(filePath: string): string {
|
||||
try {
|
||||
const content = fs.readFileSync(filePath, 'utf8');
|
||||
return 'sha256:' + crypto.createHash('sha256').update(content).digest('hex');
|
||||
} catch {
|
||||
return 'sha256:missing';
|
||||
}
|
||||
}
|
||||
|
||||
function readFileSafe(filePath: string): string | null {
|
||||
try {
|
||||
return fs.readFileSync(filePath, 'utf8');
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract METRICS_JSON HTML comment blocks from a file.
|
||||
* Format: <!-- METRICS_JSON {"key": value, ...} -->
|
||||
* A file may contain multiple blocks (e.g., overview.md has document + finalize).
|
||||
* If `stepFilter` is provided, returns only the block with matching "step" field.
|
||||
* Returns parsed object or null if not found / invalid.
|
||||
*/
|
||||
export function extractMetricsJson(content: string, stepFilter?: string): Record<string, unknown> | null {
|
||||
const re = /<!--\s*METRICS_JSON\s+(\{[\s\S]*?\})\s*-->/g;
|
||||
let match: RegExpExecArray | null;
|
||||
while ((match = re.exec(content)) !== null) {
|
||||
try {
|
||||
const parsed = JSON.parse(match[1]) as Record<string, unknown>;
|
||||
if (!stepFilter || parsed['step'] === stepFilter) {
|
||||
return parsed;
|
||||
}
|
||||
} catch {
|
||||
// try next match
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Strip the "## Success Criteria" section so its checkboxes aren't counted as tasks. */
|
||||
function stripSuccessCriteriaSection(overview: string): string {
|
||||
return overview.replace(/^## Success Criteria\s*\n[\s\S]*?(?=\n## |\n*$)/m, '');
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract canonical task counts from the Completion Summary section.
|
||||
* Supports two formats found in real specs:
|
||||
* - Bold text: "**Completion Rate:** 100% (5/5 tasks)"
|
||||
* - Table row: "| Total Tasks | 28 (28/28 complete — 100%) |"
|
||||
* Returns null when no parseable counts are found.
|
||||
*/
|
||||
function parseCompletionSummaryTaskCount(overview: string): { completed: number; total: number } | null {
|
||||
const summaryMatch = overview.match(/## Completion Summary\s*\n([\s\S]*?)(?=\n## |\n*$)/);
|
||||
if (!summaryMatch) return null;
|
||||
const summary = summaryMatch[1];
|
||||
|
||||
// "Completion Rate: X% (Y/Z tasks)" — handles bold markdown and various separators
|
||||
const rateMatch = summary.match(/\**Completion(?:\s+Rate)?\**[:\s*]+\d{1,3}%\s*\((\d+)\/(\d+)\s*tasks?\)/i);
|
||||
if (rateMatch) return { completed: parseInt(rateMatch[1], 10), total: parseInt(rateMatch[2], 10) };
|
||||
|
||||
// "Total Tasks | 28 (28/28 complete"
|
||||
const tableMatch = summary.match(/Total Tasks\s*\|\s*(\d+)\s*\((\d+)\/(\d+)\s*complete/i);
|
||||
if (tableMatch) return { completed: parseInt(tableMatch[2], 10), total: parseInt(tableMatch[3], 10) };
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function generateRunId(): string {
|
||||
const now = new Date();
|
||||
const ts = now.toISOString().replace(/[-:T]/g, '').slice(0, 14);
|
||||
const rand = crypto.randomBytes(2).toString('hex');
|
||||
return `run-${ts.slice(0, 8)}-${ts.slice(8, 14)}-${rand}`;
|
||||
}
|
||||
|
||||
// ── Prompt version hashing ────────────────────────────────────────────────────
|
||||
|
||||
export function collectPromptVersions(plan2codeRoot: string): PromptVersions {
|
||||
const srcDir = path.join(plan2codeRoot, 'src');
|
||||
return {
|
||||
plan: sha256File(path.join(srcDir, 'plan2code-1-plan.md')),
|
||||
revise_plan: sha256File(path.join(srcDir, 'plan2code-1b-revise-plan.md')),
|
||||
document: sha256File(path.join(srcDir, 'plan2code-2-document.md')),
|
||||
implement: sha256File(path.join(srcDir, 'plan2code-3-implement.md')),
|
||||
finalize: sha256File(path.join(srcDir, 'plan2code-4-finalize.md')),
|
||||
init: sha256File(path.join(srcDir, 'plan2code-init.md')),
|
||||
init_update: sha256File(path.join(srcDir, 'plan2code-init-update.md')),
|
||||
quick_task: sha256File(path.join(srcDir, 'plan2code-quick-task.md')),
|
||||
};
|
||||
}
|
||||
|
||||
// ── Step 1: Plan ──────────────────────────────────────────────────────────────
|
||||
|
||||
export function collectStep1(specDir: string): Step1PlanMetrics {
|
||||
// Find PLAN-DRAFT-*.md files
|
||||
let draftFiles: string[] = [];
|
||||
let convFiles: string[] = [];
|
||||
try {
|
||||
const files = fs.readdirSync(specDir);
|
||||
draftFiles = files
|
||||
.filter(f => f.startsWith('PLAN-DRAFT-') && f.endsWith('.md'))
|
||||
.map(f => path.join(specDir, f));
|
||||
convFiles = files
|
||||
.filter(f => f.startsWith('PLAN-CONVERSATION-') && f.endsWith('.md'))
|
||||
.map(f => path.join(specDir, f));
|
||||
} catch {
|
||||
return { present: false, final_confidence: null, confidence_breakdown: null,
|
||||
clarification_rounds: null, tech_stack_revision_rounds: null,
|
||||
verification_gaps_found: null, functional_requirements_count: null,
|
||||
non_functional_requirements_count: null, risk_count: null, phase_count: null };
|
||||
}
|
||||
|
||||
if (draftFiles.length === 0) {
|
||||
return { present: false, final_confidence: null, confidence_breakdown: null,
|
||||
clarification_rounds: null, tech_stack_revision_rounds: null,
|
||||
verification_gaps_found: null, functional_requirements_count: null,
|
||||
non_functional_requirements_count: null, risk_count: null, phase_count: null };
|
||||
}
|
||||
|
||||
// Use the latest draft file
|
||||
draftFiles.sort();
|
||||
const latestDraft = readFileSafe(draftFiles[draftFiles.length - 1]) ?? '';
|
||||
|
||||
// ── Primary source: METRICS_JSON HTML comment ──
|
||||
// Format: <!-- METRICS_JSON {"confidence": 95, "clarification_rounds": 0, ...} -->
|
||||
const metricsJson = extractMetricsJson(latestDraft);
|
||||
if (metricsJson) {
|
||||
const num = (key: string): number | null => {
|
||||
const v = metricsJson[key];
|
||||
return typeof v === 'number' ? v : null;
|
||||
};
|
||||
const bd = metricsJson['confidence_breakdown'] as Record<string, number> | undefined;
|
||||
return {
|
||||
present: true,
|
||||
final_confidence: num('confidence'),
|
||||
confidence_breakdown: bd ? {
|
||||
requirements: bd.requirements ?? null,
|
||||
feasibility: bd.feasibility ?? null,
|
||||
integration: bd.integration ?? null,
|
||||
risk: bd.risk ?? null,
|
||||
} : null,
|
||||
clarification_rounds: num('clarification_rounds'),
|
||||
tech_stack_revision_rounds: num('tech_stack_revision_rounds'),
|
||||
verification_gaps_found: num('verification_gaps_found'),
|
||||
functional_requirements_count: num('functional_requirements_count'),
|
||||
non_functional_requirements_count: num('non_functional_requirements_count'),
|
||||
risk_count: num('risk_count'),
|
||||
phase_count: num('phase_count'),
|
||||
};
|
||||
}
|
||||
|
||||
// ── Fallback: structured ## Planning Metrics appendix ──
|
||||
// Handles both plain (confidence: 95) and bold (**confidence:** 95) field formats
|
||||
// Planning Metrics is typically the last section, so we capture greedily to end,
|
||||
// but stop at the next ## heading or --- if present.
|
||||
const metricsSection = latestDraft.match(/^##\s+Planning\s+Metrics\s*\n([\s\S]+?)(?:\n##\s|\n---)/m)
|
||||
?? latestDraft.match(/^##\s+Planning\s+Metrics\s*\n([\s\S]+)/m);
|
||||
const metricsBlock = metricsSection?.[1] ?? '';
|
||||
const parseMetricField = (field: string): number | null => {
|
||||
// Match plain "field: N" or bold "**field:** N"
|
||||
const m = metricsBlock.match(new RegExp(`^\\**${field}\\**[:\\s*]+?(\\d+)`, 'm'));
|
||||
return m ? parseInt(m[1], 10) : null;
|
||||
};
|
||||
|
||||
// ── Confidence ──
|
||||
let finalConfidence = parseMetricField('confidence');
|
||||
if (finalConfidence === null) {
|
||||
// Fallback: prose scraping — handle bold markdown like **Confidence:** 95%
|
||||
const confMatch = latestDraft.match(/(?:Overall|Final|Total)?\s*\**[Cc]onfidence\**[:\s*]+(\d{1,3})%/);
|
||||
if (confMatch) {
|
||||
finalConfidence = parseInt(confMatch[1], 10);
|
||||
} else {
|
||||
const tableMatch = latestDraft.match(/[|]\s*[Cc]onfidence\s*[|]\s*(\d{1,3})/);
|
||||
if (tableMatch) finalConfidence = parseInt(tableMatch[1], 10);
|
||||
}
|
||||
}
|
||||
|
||||
// Confidence breakdown (requirements, feasibility, integration, risk)
|
||||
let confidenceBreakdown: Step1PlanMetrics['confidence_breakdown'] = null;
|
||||
const reqMatch = latestDraft.match(/[Rr]equirements?[:\s|]+(\d{1,2})/);
|
||||
const feasMatch = latestDraft.match(/[Ff]easibility[:\s|]+(\d{1,2})/);
|
||||
const intMatch = latestDraft.match(/[Ii]ntegration[:\s|]+(\d{1,2})/);
|
||||
const riskMatch = latestDraft.match(/[Rr]isk[:\s|]+(\d{1,2})/);
|
||||
if (reqMatch || feasMatch || intMatch || riskMatch) {
|
||||
confidenceBreakdown = {
|
||||
requirements: reqMatch ? parseInt(reqMatch[1], 10) : null,
|
||||
feasibility: feasMatch ? parseInt(feasMatch[1], 10) : null,
|
||||
integration: intMatch ? parseInt(intMatch[1], 10) : null,
|
||||
risk: riskMatch ? parseInt(riskMatch[1], 10) : null,
|
||||
};
|
||||
}
|
||||
|
||||
// ── Clarification rounds ──
|
||||
let clarificationRounds = parseMetricField('clarification_rounds');
|
||||
|
||||
// ── Verification gaps ──
|
||||
let verificationGaps = parseMetricField('verification_gaps_found');
|
||||
|
||||
// ── Functional / non-functional requirements ──
|
||||
let frCount = parseMetricField('functional_requirements_count');
|
||||
if (frCount === null) {
|
||||
frCount = (latestDraft.match(/###\s+FR-/g) ?? []).length || null;
|
||||
}
|
||||
let nfrCount = parseMetricField('non_functional_requirements_count');
|
||||
if (nfrCount === null) {
|
||||
nfrCount = (latestDraft.match(/###\s+NFR-/g) ?? []).length || null;
|
||||
}
|
||||
|
||||
// ── Risk count ──
|
||||
let riskCount = parseMetricField('risk_count');
|
||||
if (riskCount === null) {
|
||||
const riskRows = latestDraft.match(/^\s*[|][^|]*(?:High|Medium|Low|Critical)[^|]*[|]/gm) ?? [];
|
||||
riskCount = riskRows.length || null;
|
||||
}
|
||||
|
||||
// ── Phase count ──
|
||||
let phaseCount = parseMetricField('phase_count');
|
||||
if (phaseCount === null) {
|
||||
phaseCount = (latestDraft.match(/^##\s+Phase\s+\d/gm) ?? []).length || null;
|
||||
}
|
||||
|
||||
// ── Tech stack revision rounds (conversation file only) ──
|
||||
let techStackRevisions: number | null = null;
|
||||
|
||||
// Fall back to conversation file scraping for fields not found in appendix
|
||||
if (convFiles.length > 0) {
|
||||
convFiles.sort();
|
||||
const latestConv = readFileSafe(convFiles[convFiles.length - 1]) ?? '';
|
||||
if (clarificationRounds === null) {
|
||||
const clarRounds = (latestConv.match(/^##\s+(?:Clarification|Round)\s+\d/gm) ?? []).length;
|
||||
clarificationRounds = clarRounds || null;
|
||||
}
|
||||
const techRounds = (latestConv.match(/^##\s+(?:Tech\s+Stack|Technology)\s+Revision/gmi) ?? []).length;
|
||||
techStackRevisions = techRounds || null;
|
||||
if (verificationGaps === null) {
|
||||
const gapMatches = latestConv.match(/(?:verification\s+gap|gap\s+found|missing\s+requirement)/gi) ?? [];
|
||||
verificationGaps = gapMatches.length || null;
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
present: true,
|
||||
final_confidence: finalConfidence,
|
||||
confidence_breakdown: confidenceBreakdown,
|
||||
clarification_rounds: clarificationRounds,
|
||||
tech_stack_revision_rounds: techStackRevisions,
|
||||
verification_gaps_found: verificationGaps,
|
||||
functional_requirements_count: frCount,
|
||||
non_functional_requirements_count: nfrCount,
|
||||
risk_count: riskCount,
|
||||
phase_count: phaseCount,
|
||||
};
|
||||
}
|
||||
|
||||
// ── Step 2: Document ──────────────────────────────────────────────────────────
|
||||
|
||||
export function collectStep2(specDir: string): Step2DocumentMetrics {
|
||||
const overviewPath = path.join(specDir, 'overview.md');
|
||||
const overview = readFileSafe(overviewPath);
|
||||
|
||||
if (!overview) {
|
||||
return { present: false, total_tasks: null, tasks_per_phase: null,
|
||||
phase_count: null, parallel_groups_identified: null,
|
||||
requirement_coverage_percent: null, verification_items_added: null };
|
||||
}
|
||||
|
||||
// ── Primary source: METRICS_JSON in overview.md ──
|
||||
const step2Json = extractMetricsJson(overview, 'document');
|
||||
if (step2Json) {
|
||||
const num = (key: string): number | null => {
|
||||
const v = step2Json[key];
|
||||
return typeof v === 'number' ? v : null;
|
||||
};
|
||||
const tpp = step2Json['tasks_per_phase'];
|
||||
return {
|
||||
present: true,
|
||||
total_tasks: num('total_tasks'),
|
||||
tasks_per_phase: Array.isArray(tpp) ? tpp : null,
|
||||
phase_count: num('phase_count'),
|
||||
parallel_groups_identified: num('parallel_groups_identified') ?? 0,
|
||||
requirement_coverage_percent: num('requirement_coverage_percent'),
|
||||
verification_items_added: num('verification_items_added'),
|
||||
};
|
||||
}
|
||||
|
||||
// ── Fallback: regex scraping ──
|
||||
|
||||
// Detect Parallel Execution Groups table
|
||||
const parallelGroups = (overview.match(/Parallel\s+Execution\s+Group/gi) ?? []).length;
|
||||
|
||||
// Count phase files (scanned first so we can use sum as a total_tasks fallback)
|
||||
let phaseCount = 0;
|
||||
let tasksPerPhase: number[] = [];
|
||||
try {
|
||||
const files = fs.readdirSync(specDir);
|
||||
const phaseFiles = files
|
||||
.filter(f => /^phase-\d+\.md$/i.test(f))
|
||||
.sort();
|
||||
phaseCount = phaseFiles.length;
|
||||
for (const pf of phaseFiles) {
|
||||
const content = readFileSafe(path.join(specDir, pf)) ?? '';
|
||||
const count = (content.match(/^\s*-\s+\[[ x!/?]\]\s+\*\*Task\s+\d/gm) ?? []).length;
|
||||
tasksPerPhase.push(count);
|
||||
}
|
||||
} catch {
|
||||
// leave empty
|
||||
}
|
||||
|
||||
// total_tasks priority:
|
||||
// 1. Completion Summary canonical count
|
||||
// 2. Sum of phase file task counts
|
||||
// 3. Stripped overview checkboxes (excluding Success Criteria)
|
||||
const summaryCount = parseCompletionSummaryTaskCount(overview);
|
||||
const phaseSum = tasksPerPhase.length > 0 ? tasksPerPhase.reduce((a, b) => a + b, 0) : 0;
|
||||
const strippedOverview = stripSuccessCriteriaSection(overview);
|
||||
const strippedCheckboxCount = (strippedOverview.match(/^\s*-\s+\[[ x!/?]\]/gm) ?? []).length;
|
||||
|
||||
let totalTasks: number;
|
||||
if (summaryCount) {
|
||||
totalTasks = summaryCount.total;
|
||||
} else if (phaseSum > 0) {
|
||||
totalTasks = phaseSum;
|
||||
} else {
|
||||
totalTasks = strippedCheckboxCount;
|
||||
}
|
||||
|
||||
// Requirement coverage: look for coverage percentage in overview
|
||||
let reqCoverage: number | null = null;
|
||||
const covMatch = overview.match(/[Cc]overage[:\s]+(\d{1,3})%/);
|
||||
if (covMatch) reqCoverage = parseInt(covMatch[1], 10);
|
||||
|
||||
// Verification items added (look for verification checklist items)
|
||||
const verifItems = (overview.match(/(?:verify|verification|test|check):\s*\[[ x]\]/gi) ?? []).length;
|
||||
|
||||
return {
|
||||
present: true,
|
||||
total_tasks: totalTasks || null,
|
||||
tasks_per_phase: tasksPerPhase.length > 0 ? tasksPerPhase : null,
|
||||
phase_count: phaseCount || null,
|
||||
parallel_groups_identified: parallelGroups || 0,
|
||||
requirement_coverage_percent: reqCoverage,
|
||||
verification_items_added: verifItems || null,
|
||||
};
|
||||
}
|
||||
|
||||
// ── Step 3: Implement ──────────────────────────────────────────────────────────
|
||||
|
||||
export function collectStep3(specDir: string): Step3ImplementMetrics {
|
||||
// Discover phase-*.md files
|
||||
let phaseFiles: string[] = [];
|
||||
try {
|
||||
const files = fs.readdirSync(specDir);
|
||||
phaseFiles = files
|
||||
.filter(f => /^phase-\d+\.md$/i.test(f))
|
||||
.sort();
|
||||
} catch {
|
||||
// leave empty
|
||||
}
|
||||
|
||||
// Read overview.md for Completion Summary
|
||||
const overviewPath = path.join(specDir, 'overview.md');
|
||||
const overview = readFileSafe(overviewPath);
|
||||
const summaryCount = overview ? parseCompletionSummaryTaskCount(overview) : null;
|
||||
|
||||
// ── Primary source: METRICS_JSON in overview.md (from finalize step) ──
|
||||
if (overview) {
|
||||
const step3Json = extractMetricsJson(overview, 'finalize');
|
||||
if (step3Json) {
|
||||
const num = (key: string): number | null => {
|
||||
const v = step3Json[key];
|
||||
return typeof v === 'number' ? v : null;
|
||||
};
|
||||
const total = num('tasks_total');
|
||||
const completed = num('tasks_completed');
|
||||
return {
|
||||
present: true,
|
||||
task_completion_rate: total && total > 0 && completed != null ? completed / total : null,
|
||||
tasks_completed: completed,
|
||||
tasks_total: total,
|
||||
blocker_count: num('blocker_count') ?? 0,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// ── Fallback: regex scraping ──
|
||||
|
||||
// present = true if phase files exist OR overview has completion data
|
||||
const present = phaseFiles.length > 0 || summaryCount != null;
|
||||
if (!present) {
|
||||
return { present: false, task_completion_rate: null, tasks_completed: null,
|
||||
tasks_total: null, blocker_count: null };
|
||||
}
|
||||
|
||||
// Count checkboxes in phase files
|
||||
let phaseTotal = 0;
|
||||
let phaseCompleted = 0;
|
||||
let blockerCount = 0;
|
||||
for (const pf of phaseFiles) {
|
||||
const content = readFileSafe(path.join(specDir, pf)) ?? '';
|
||||
phaseTotal += (content.match(/^\s*-\s+\[[ x!/?]\]\s+\*\*Task\s+\d/gm) ?? []).length;
|
||||
phaseCompleted += (content.match(/^\s*-\s+\[x\]\s+\*\*Task\s+\d/gm) ?? []).length;
|
||||
blockerCount += (content.match(/^\s*-\s+\[!\]\s+\*\*Task\s+\d/gm) ?? []).length;
|
||||
}
|
||||
|
||||
// Count checkboxes in overview (stripped of Success Criteria)
|
||||
let overviewTotal = 0;
|
||||
let overviewCompleted = 0;
|
||||
if (overview) {
|
||||
const stripped = stripSuccessCriteriaSection(overview);
|
||||
overviewTotal = (stripped.match(/^\s*-\s+\[[ x!/?]\]/gm) ?? []).length;
|
||||
overviewCompleted = (stripped.match(/^\s*-\s+\[x\]/gm) ?? []).length;
|
||||
}
|
||||
|
||||
// Task counts priority:
|
||||
// 1. Completion Summary in overview.md
|
||||
// 2. Checkbox counting in phase files
|
||||
// 3. Checkbox counting in overview.md (stripped of Success Criteria)
|
||||
let tasksTotal: number | null = null;
|
||||
let tasksCompleted: number | null = null;
|
||||
if (summaryCount) {
|
||||
tasksTotal = summaryCount.total;
|
||||
tasksCompleted = summaryCount.completed;
|
||||
} else if (phaseTotal > 0) {
|
||||
tasksTotal = phaseTotal;
|
||||
tasksCompleted = phaseCompleted;
|
||||
} else if (overviewTotal > 0) {
|
||||
tasksTotal = overviewTotal;
|
||||
tasksCompleted = overviewCompleted;
|
||||
}
|
||||
|
||||
const taskCompletionRate = tasksTotal && tasksTotal > 0 && tasksCompleted != null
|
||||
? tasksCompleted / tasksTotal
|
||||
: null;
|
||||
|
||||
return {
|
||||
present: true,
|
||||
task_completion_rate: taskCompletionRate,
|
||||
tasks_completed: tasksCompleted,
|
||||
tasks_total: tasksTotal,
|
||||
blocker_count: blockerCount,
|
||||
};
|
||||
}
|
||||
|
||||
// ── Step 4: Finalize ──────────────────────────────────────────────────────────
|
||||
|
||||
export function collectStep4(specDir: string): Step4FinalizeMetrics {
|
||||
// Check if spec was archived (specs--completed exists at parent level)
|
||||
const specDirName = path.basename(specDir);
|
||||
const parentDir = path.dirname(specDir);
|
||||
const completedPath = path.join(parentDir, '..', 'specs--completed', specDirName);
|
||||
const archivalSucceeded = fs.existsSync(completedPath);
|
||||
|
||||
// Read overview for completion metrics
|
||||
const overviewPath = path.join(specDir, 'overview.md');
|
||||
|
||||
// If archived, try from archived location
|
||||
const effectiveOverviewPath = archivalSucceeded
|
||||
? path.join(completedPath, 'overview.md')
|
||||
: overviewPath;
|
||||
|
||||
const overview = readFileSafe(effectiveOverviewPath) ?? readFileSafe(overviewPath);
|
||||
if (!overview) {
|
||||
return { present: false, completion_rate_at_audit: null,
|
||||
verification_failures_found: null, documentation_updates_needed: null,
|
||||
archival_succeeded: archivalSucceeded };
|
||||
}
|
||||
|
||||
// ── Primary source: METRICS_JSON in overview.md ──
|
||||
const step4Json = extractMetricsJson(overview, 'finalize');
|
||||
if (step4Json) {
|
||||
const num = (key: string): number | null => {
|
||||
const v = step4Json[key];
|
||||
return typeof v === 'number' ? v : null;
|
||||
};
|
||||
return {
|
||||
present: true,
|
||||
completion_rate_at_audit: num('completion_rate_at_audit'),
|
||||
verification_failures_found: num('verification_failures_found') ?? 0,
|
||||
documentation_updates_needed: num('documentation_updates_needed'),
|
||||
archival_succeeded: archivalSucceeded,
|
||||
};
|
||||
}
|
||||
|
||||
// ── Fallback: regex scraping ──
|
||||
|
||||
// Completion rate: prefer Completion Summary, fall back to stripped overview checkboxes
|
||||
let completionRate: number | null = null;
|
||||
const summaryCount = parseCompletionSummaryTaskCount(overview);
|
||||
if (summaryCount) {
|
||||
completionRate = summaryCount.total > 0 ? summaryCount.completed / summaryCount.total : null;
|
||||
} else {
|
||||
const stripped = stripSuccessCriteriaSection(overview);
|
||||
const totalTasks = (stripped.match(/^\s*-\s+\[[ x!/?]\]/gm) ?? []).length;
|
||||
const completedTasks = (stripped.match(/^\s*-\s+\[x\]/gm) ?? []).length;
|
||||
completionRate = totalTasks > 0 ? completedTasks / totalTasks : null;
|
||||
}
|
||||
|
||||
// Verification failures: look for [!] tasks or "verification failed" text
|
||||
const blockedTasks = (overview.match(/^\s*-\s+\[!\]/gm) ?? []).length;
|
||||
|
||||
// Documentation updates needed (look for TODO or "update" markers)
|
||||
const docUpdates = (overview.match(/(?:TODO|FIXME|update\s+(?:README|CHANGELOG|docs))/gi) ?? []).length;
|
||||
|
||||
return {
|
||||
present: true,
|
||||
completion_rate_at_audit: completionRate,
|
||||
verification_failures_found: blockedTasks || 0,
|
||||
documentation_updates_needed: docUpdates || null,
|
||||
archival_succeeded: archivalSucceeded,
|
||||
};
|
||||
}
|
||||
|
||||
// ── User Feedback ─────────────────────────────────────────────────────────
|
||||
|
||||
export function collectUserFeedback(specDir: string): UserFeedback | null {
|
||||
// Try both active and archived locations for overview.md
|
||||
const overviewPath = path.join(specDir, 'overview.md');
|
||||
const specDirName = path.basename(specDir);
|
||||
const parentDir = path.dirname(specDir);
|
||||
const completedPath = path.join(parentDir, '..', 'specs--completed', specDirName, 'overview.md');
|
||||
|
||||
const overview = readFileSafe(overviewPath) ?? readFileSafe(completedPath);
|
||||
if (!overview) return null;
|
||||
|
||||
// Look for ## User Feedback section with a markdown table
|
||||
const feedbackMatch = overview.match(
|
||||
/## User Feedback\s*\n\s*\|[^\n]*\|\s*\n\s*\|[-| ]+\|\s*\n([\s\S]*?)(?=\n##\s|\n*$)/
|
||||
);
|
||||
if (!feedbackMatch) return null;
|
||||
|
||||
const tableBody = feedbackMatch[1];
|
||||
|
||||
// Parse table rows: | Field | Value |
|
||||
// Use regex to split only on unescaped pipes, preserving whitespace in values
|
||||
const rows = tableBody.split('\n').filter(l => l.trim().startsWith('|'));
|
||||
const fields: Record<string, string> = {};
|
||||
for (const row of rows) {
|
||||
// Split on unescaped pipes (not preceded by backslash)
|
||||
const cells = row.split(/(?<!\\)\|/).map(c => c.trim()).filter(c => c.length > 0);
|
||||
if (cells.length >= 2) {
|
||||
const value = cells.slice(1).join('|').replace(/\\\|/g, '|');
|
||||
fields[cells[0].toLowerCase()] = value;
|
||||
}
|
||||
}
|
||||
|
||||
const rating = parseInt(fields['rating'] ?? '', 10);
|
||||
if (isNaN(rating) || rating < 1 || rating > 10) return null;
|
||||
|
||||
return {
|
||||
overall_rating: rating,
|
||||
rating_reason: fields['reason'] ?? '',
|
||||
what_went_well: fields['went well'] ?? '',
|
||||
what_went_poorly: fields['went poorly'] ?? '',
|
||||
};
|
||||
}
|
||||
|
||||
// ── Main collector ────────────────────────────────────────────────────────────
|
||||
|
||||
export interface CollectorOptions {
|
||||
specDir: string; // Path to the spec directory (specs/<feature-name>)
|
||||
projectName: string; // Human-readable project name
|
||||
plan2codeRoot: string; // Path to the plan2code repo (for prompt hashing)
|
||||
plan2codeVersion: string; // e.g. "1.7.0"
|
||||
outputDir: string; // Where to write <run-id>.json
|
||||
}
|
||||
|
||||
export async function collectRun(opts: CollectorOptions): Promise<RunMetrics> {
|
||||
const {
|
||||
specDir,
|
||||
projectName,
|
||||
plan2codeRoot,
|
||||
plan2codeVersion,
|
||||
outputDir,
|
||||
} = opts;
|
||||
|
||||
const runId = generateRunId();
|
||||
|
||||
// Get started_at / completed_at from spec directory mtime / iteration.log
|
||||
let startedAt: string | null = null;
|
||||
let completedAt: string | null = null;
|
||||
try {
|
||||
const stat = fs.statSync(specDir);
|
||||
startedAt = stat.birthtime.toISOString();
|
||||
completedAt = stat.mtime.toISOString();
|
||||
} catch {
|
||||
// leave null
|
||||
}
|
||||
|
||||
const metrics: RunMetrics = {
|
||||
schema_version: '1.0',
|
||||
run_id: runId,
|
||||
plan2code_version: plan2codeVersion,
|
||||
source: 'local',
|
||||
prompt_versions: collectPromptVersions(plan2codeRoot),
|
||||
project: {
|
||||
name: projectName,
|
||||
started_at: startedAt,
|
||||
completed_at: completedAt,
|
||||
},
|
||||
step1_plan: collectStep1(specDir),
|
||||
step2_document: collectStep2(specDir),
|
||||
step3_implement: collectStep3(specDir),
|
||||
step4_finalize: collectStep4(specDir),
|
||||
user_feedback: collectUserFeedback(specDir),
|
||||
};
|
||||
|
||||
// Write to output dir
|
||||
fs.mkdirSync(outputDir, { recursive: true });
|
||||
const outPath = path.join(outputDir, `${runId}.json`);
|
||||
fs.writeFileSync(outPath, JSON.stringify(metrics, null, 2), 'utf8');
|
||||
|
||||
return metrics;
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
import fs from 'fs';
|
||||
import os from 'os';
|
||||
import path from 'path';
|
||||
import { describe, it, expect, afterEach } from 'vitest';
|
||||
import { parseSubmissionPayload, ingestCommunityIssues } from './community.js';
|
||||
import { writeRunFile } from './aggregator.js';
|
||||
import type { RunMetrics } from './types.js';
|
||||
|
||||
function issueBody(payload: Record<string, unknown>): string {
|
||||
return `Some issue text.\n\n<!-- METRICS_JSON ${JSON.stringify(payload)} -->\n`;
|
||||
}
|
||||
|
||||
const VALID_PAYLOAD = {
|
||||
schema_version: '1.0',
|
||||
run_id: 'run-20260715-143000-a1b2',
|
||||
plan2code_version: '1.15.3',
|
||||
prompt_versions_short: {
|
||||
plan: 'abc123def456', revise_plan: 'a', document: 'b', implement: 'c',
|
||||
finalize: 'd', init: 'e', init_update: 'f', quick_task: 'g',
|
||||
},
|
||||
step1: {
|
||||
final_confidence: 95,
|
||||
confidence_breakdown: { requirements: 24, feasibility: 23, integration: 24, risk: 22 },
|
||||
clarification_rounds: 0,
|
||||
tech_stack_revision_rounds: 0,
|
||||
verification_gaps_found: 0,
|
||||
functional_requirements_count: 8,
|
||||
non_functional_requirements_count: 6,
|
||||
risk_count: 7,
|
||||
phase_count: 4,
|
||||
},
|
||||
step2: {
|
||||
total_tasks: 28,
|
||||
phase_count: 4,
|
||||
parallel_groups_identified: 1,
|
||||
requirement_coverage_percent: 100,
|
||||
verification_items_added: 3,
|
||||
},
|
||||
step3: {
|
||||
task_completion_rate: 0.96,
|
||||
tasks_completed: 27,
|
||||
tasks_total: 28,
|
||||
blocker_count: 1,
|
||||
},
|
||||
step4: {
|
||||
completion_rate_at_audit: 0.96,
|
||||
verification_failures_found: 1,
|
||||
documentation_updates_needed: 2,
|
||||
},
|
||||
user_feedback: {
|
||||
overall_rating: 8,
|
||||
rating_reason: 'good stuff',
|
||||
what_went_well: 'well',
|
||||
what_went_poorly: 'poorly',
|
||||
},
|
||||
};
|
||||
|
||||
// ── parseSubmissionPayload() ─────────────────────────────────────────────────
|
||||
|
||||
describe('parseSubmissionPayload', () => {
|
||||
it('parses a fully valid payload into a correctly-shaped RunMetrics', () => {
|
||||
const result = parseSubmissionPayload(issueBody(VALID_PAYLOAD));
|
||||
expect(result).not.toBeNull();
|
||||
expect(result!.run_id).toBe('run-20260715-143000-a1b2');
|
||||
expect(result!.schema_version).toBe('1.0');
|
||||
expect(result!.plan2code_version).toBe('1.15.3');
|
||||
expect(result!.source).toBe('community');
|
||||
expect(result!.step1_plan.present).toBe(true);
|
||||
expect(result!.step1_plan.final_confidence).toBe(95);
|
||||
expect(result!.step2_document.present).toBe(true);
|
||||
expect(result!.step2_document.total_tasks).toBe(28);
|
||||
expect(result!.step3_implement.present).toBe(true);
|
||||
expect(result!.step3_implement.tasks_completed).toBe(27);
|
||||
expect(result!.step4_finalize.present).toBe(true);
|
||||
expect(result!.step4_finalize.completion_rate_at_audit).toBe(0.96);
|
||||
expect(result!.user_feedback).toEqual({
|
||||
overall_rating: 8,
|
||||
rating_reason: 'good stuff',
|
||||
what_went_well: 'well',
|
||||
what_went_poorly: 'poorly',
|
||||
});
|
||||
});
|
||||
|
||||
it('returns null when run_id is missing', () => {
|
||||
const { run_id, ...withoutRunId } = VALID_PAYLOAD;
|
||||
const result = parseSubmissionPayload(issueBody(withoutRunId));
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
|
||||
it('returns null when user_feedback.overall_rating is a string instead of a number', () => {
|
||||
const badPayload = {
|
||||
...VALID_PAYLOAD,
|
||||
user_feedback: { ...VALID_PAYLOAD.user_feedback, overall_rating: 'eight' },
|
||||
};
|
||||
const result = parseSubmissionPayload(issueBody(badPayload));
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
|
||||
it('returns null when schema_version is not exactly "1.0"', () => {
|
||||
const badPayload = { ...VALID_PAYLOAD, schema_version: '2.0' };
|
||||
const result = parseSubmissionPayload(issueBody(badPayload));
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
|
||||
it('sets all four steps present:false when only user_feedback is included', () => {
|
||||
const minimalPayload = {
|
||||
schema_version: '1.0',
|
||||
run_id: 'run-20260715-150000-c3d4',
|
||||
plan2code_version: '1.15.3',
|
||||
user_feedback: VALID_PAYLOAD.user_feedback,
|
||||
};
|
||||
const result = parseSubmissionPayload(issueBody(minimalPayload));
|
||||
expect(result).not.toBeNull();
|
||||
expect(result!.step1_plan.present).toBe(false);
|
||||
expect(result!.step2_document.present).toBe(false);
|
||||
expect(result!.step3_implement.present).toBe(false);
|
||||
expect(result!.step4_finalize.present).toBe(false);
|
||||
});
|
||||
|
||||
it('backfills missing prompt_versions_short keys with the sha256:missing sentinel', () => {
|
||||
const partialPayload = {
|
||||
...VALID_PAYLOAD,
|
||||
prompt_versions_short: { plan: 'abc123def456' },
|
||||
};
|
||||
const result = parseSubmissionPayload(issueBody(partialPayload));
|
||||
expect(result).not.toBeNull();
|
||||
expect(result!.prompt_versions.plan).toBe('abc123def456');
|
||||
expect(result!.prompt_versions.revise_plan).toBe('sha256:missing');
|
||||
expect(result!.prompt_versions.document).toBe('sha256:missing');
|
||||
expect(result!.prompt_versions.implement).toBe('sha256:missing');
|
||||
expect(result!.prompt_versions.finalize).toBe('sha256:missing');
|
||||
expect(result!.prompt_versions.init).toBe('sha256:missing');
|
||||
expect(result!.prompt_versions.init_update).toBe('sha256:missing');
|
||||
expect(result!.prompt_versions.quick_task).toBe('sha256:missing');
|
||||
});
|
||||
});
|
||||
|
||||
// ── writeRunFile() ────────────────────────────────────────────────────────────
|
||||
|
||||
describe('writeRunFile', () => {
|
||||
let tmpDir: string;
|
||||
|
||||
afterEach(() => {
|
||||
if (tmpDir) fs.rmSync(tmpDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
const RUN: RunMetrics = {
|
||||
schema_version: '1.0',
|
||||
run_id: 'run-20260715-160000-e5f6',
|
||||
plan2code_version: '1.15.3',
|
||||
prompt_versions: {
|
||||
plan: 'sha256:missing', revise_plan: 'sha256:missing', document: 'sha256:missing',
|
||||
implement: 'sha256:missing', finalize: 'sha256:missing', init: 'sha256:missing',
|
||||
init_update: 'sha256:missing', quick_task: 'sha256:missing',
|
||||
},
|
||||
project: { name: '', started_at: null, completed_at: null },
|
||||
step1_plan: { present: false, final_confidence: null, confidence_breakdown: null, clarification_rounds: null, tech_stack_revision_rounds: null, verification_gaps_found: null, functional_requirements_count: null, non_functional_requirements_count: null, risk_count: null, phase_count: null },
|
||||
step2_document: { present: false, total_tasks: null, tasks_per_phase: null, phase_count: null, parallel_groups_identified: null, requirement_coverage_percent: null, verification_items_added: null },
|
||||
step3_implement: { present: false, task_completion_rate: null, tasks_completed: null, tasks_total: null, blocker_count: null },
|
||||
step4_finalize: { present: false, completion_rate_at_audit: null, verification_failures_found: null, documentation_updates_needed: null, archival_succeeded: null },
|
||||
user_feedback: null,
|
||||
};
|
||||
|
||||
it('writes a new run_id to an empty runsDir and returns true', () => {
|
||||
tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'plan2code-metrics-test-'));
|
||||
const result = writeRunFile(RUN, tmpDir);
|
||||
expect(result).toBe(true);
|
||||
expect(fs.existsSync(path.join(tmpDir, `${RUN.run_id}.json`))).toBe(true);
|
||||
});
|
||||
|
||||
it('returns false and does not overwrite when the same run_id already exists', () => {
|
||||
tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'plan2code-metrics-test-'));
|
||||
writeRunFile(RUN, tmpDir);
|
||||
const modified = { ...RUN, plan2code_version: '9.9.9' };
|
||||
const result = writeRunFile(modified, tmpDir);
|
||||
expect(result).toBe(false);
|
||||
const onDisk = JSON.parse(fs.readFileSync(path.join(tmpDir, `${RUN.run_id}.json`), 'utf8'));
|
||||
expect(onDisk.plan2code_version).toBe('1.15.3');
|
||||
});
|
||||
});
|
||||
|
||||
// ── ingestCommunityIssues() ───────────────────────────────────────────────────
|
||||
|
||||
describe('ingestCommunityIssues', () => {
|
||||
const goodBody = issueBody(VALID_PAYLOAD);
|
||||
const badBody = 'an issue with no METRICS_JSON payload';
|
||||
|
||||
it('imports a new run and closes its issue', async () => {
|
||||
const closed: number[] = [];
|
||||
const tally = await ingestCommunityIssues(
|
||||
[{ number: 1, body: goodBody }],
|
||||
'owner/repo',
|
||||
'/runs',
|
||||
{ writeRunFile: () => true, closeIssue: async (_r, n) => { closed.push(n); } },
|
||||
);
|
||||
expect(tally.imported).toBe(1);
|
||||
expect(tally.skippedDuplicate).toBe(0);
|
||||
expect(tally.closed).toBe(1);
|
||||
expect(closed).toEqual([1]);
|
||||
});
|
||||
|
||||
it('closes an already-imported (duplicate) issue instead of skipping the close', async () => {
|
||||
const closed: number[] = [];
|
||||
const tally = await ingestCommunityIssues(
|
||||
[{ number: 7, body: goodBody }],
|
||||
'owner/repo',
|
||||
'/runs',
|
||||
{ writeRunFile: () => false, closeIssue: async (_r, n) => { closed.push(n); } },
|
||||
);
|
||||
expect(tally.imported).toBe(0);
|
||||
expect(tally.skippedDuplicate).toBe(1);
|
||||
expect(tally.closed).toBe(1); // the close-retry: duplicates are still closed
|
||||
expect(closed).toEqual([7]);
|
||||
});
|
||||
|
||||
it('records a close failure without throwing and leaves the issue for a later retry', async () => {
|
||||
const tally = await ingestCommunityIssues(
|
||||
[{ number: 9, body: goodBody }],
|
||||
'owner/repo',
|
||||
'/runs',
|
||||
{ writeRunFile: () => true, closeIssue: async () => { throw new Error('network'); } },
|
||||
);
|
||||
expect(tally.imported).toBe(1);
|
||||
expect(tally.closed).toBe(0);
|
||||
expect(tally.closeFailed).toBe(1);
|
||||
expect(tally.closeFailedIssues).toEqual([9]);
|
||||
});
|
||||
|
||||
it('skips and reports a malformed issue without writing or closing it', async () => {
|
||||
let wrote = false;
|
||||
let closeCalled = false;
|
||||
const tally = await ingestCommunityIssues(
|
||||
[{ number: 3, body: badBody }],
|
||||
'owner/repo',
|
||||
'/runs',
|
||||
{ writeRunFile: () => { wrote = true; return true; }, closeIssue: async () => { closeCalled = true; } },
|
||||
);
|
||||
expect(tally.skippedMalformed).toBe(1);
|
||||
expect(tally.malformedIssues).toEqual([3]);
|
||||
expect(wrote).toBe(false);
|
||||
expect(closeCalled).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,276 @@
|
||||
/**
|
||||
* community.ts
|
||||
* Ingestion side of the community feedback flow: list/parse/close
|
||||
* `community-feedback`-labeled GitHub issues on jparkerweb/plan2code.
|
||||
*/
|
||||
|
||||
import { execa } from 'execa';
|
||||
import type {
|
||||
RunMetrics,
|
||||
PromptVersions,
|
||||
UserFeedback,
|
||||
Step1PlanMetrics,
|
||||
Step2DocumentMetrics,
|
||||
Step3ImplementMetrics,
|
||||
Step4FinalizeMetrics,
|
||||
} from './types.js';
|
||||
import { extractMetricsJson } from './collector.js';
|
||||
import { backfillPromptVersions } from './aggregator.js';
|
||||
|
||||
// ── GitHub interaction (via gh CLI) ──────────────────────────────────────────
|
||||
|
||||
export interface CommunityIssue {
|
||||
number: number;
|
||||
body: string;
|
||||
}
|
||||
|
||||
interface RawIssue {
|
||||
number: number;
|
||||
title: string;
|
||||
body: string;
|
||||
labels: { name: string }[];
|
||||
}
|
||||
|
||||
const COMMUNITY_LABEL = 'community-feedback';
|
||||
const FEEDBACK_TITLE_PREFIX = '[Feedback]';
|
||||
const METRICS_JSON_MARKER = /<!--\s*METRICS_JSON\s+\{/;
|
||||
|
||||
export async function listCommunityIssues(repo: string): Promise<CommunityIssue[]> {
|
||||
// Fetch open issues broadly rather than by label alone. Browser/print-tier
|
||||
// submissions from outside contributors can lose the `community-feedback`
|
||||
// label: GitHub only honors the `labels=` query param on issues/new for
|
||||
// users with triage/push access, so the label is silently dropped for
|
||||
// community members without `gh`. We therefore also match by the `[Feedback]`
|
||||
// title prefix and the METRICS_JSON marker. Only OPEN issues are considered
|
||||
// (closed/done submissions are already processed); parseSubmissionPayload is
|
||||
// the final gate that rejects anything without a valid payload.
|
||||
const result = await execa('gh', [
|
||||
'issue', 'list',
|
||||
'--repo', repo,
|
||||
'--state', 'open',
|
||||
'--limit', '1000',
|
||||
'--json', 'number,title,body,labels',
|
||||
]);
|
||||
const raw = JSON.parse(result.stdout) as RawIssue[];
|
||||
return raw
|
||||
.filter((issue) =>
|
||||
issue.labels.some((l) => l.name === COMMUNITY_LABEL) ||
|
||||
issue.title.startsWith(FEEDBACK_TITLE_PREFIX) ||
|
||||
METRICS_JSON_MARKER.test(issue.body)
|
||||
)
|
||||
.map((issue) => ({ number: issue.number, body: issue.body }));
|
||||
}
|
||||
|
||||
export async function closeIssue(repo: string, issueNumber: number): Promise<void> {
|
||||
await execa('gh', ['issue', 'close', String(issueNumber), '--repo', repo]);
|
||||
}
|
||||
|
||||
// ── Ingestion control flow (I/O injected so it is unit-testable) ──────────────
|
||||
|
||||
export interface IngestionDeps {
|
||||
writeRunFile: (run: RunMetrics, runsDir: string) => boolean;
|
||||
closeIssue: (repo: string, issueNumber: number) => Promise<void>;
|
||||
}
|
||||
|
||||
export interface IngestionTally {
|
||||
imported: number;
|
||||
skippedDuplicate: number;
|
||||
skippedMalformed: number;
|
||||
closed: number;
|
||||
closeFailed: number;
|
||||
malformedIssues: number[];
|
||||
closeFailedIssues: number[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Process a batch of community issues: parse each payload, write new runs
|
||||
* (deduped by run_id), and close every open issue idempotently.
|
||||
*
|
||||
* The close is attempted on the duplicate path too: a submission that imported
|
||||
* on an earlier run but failed to close would otherwise be seen as a duplicate
|
||||
* forever and never closed again, leaving the issue open and reprocessed on
|
||||
* every fetch. Malformed issues are reported (not fixed up) and left open.
|
||||
*
|
||||
* I/O (writeRunFile/closeIssue) is injected so the control flow can be unit
|
||||
* tested without a live `gh`. Returns a tally; the caller owns all logging.
|
||||
*/
|
||||
export async function ingestCommunityIssues(
|
||||
issues: CommunityIssue[],
|
||||
repo: string,
|
||||
runsDir: string,
|
||||
deps: IngestionDeps,
|
||||
): Promise<IngestionTally> {
|
||||
const tally: IngestionTally = {
|
||||
imported: 0, skippedDuplicate: 0, skippedMalformed: 0,
|
||||
closed: 0, closeFailed: 0, malformedIssues: [], closeFailedIssues: [],
|
||||
};
|
||||
|
||||
for (const issue of issues) {
|
||||
const run = parseSubmissionPayload(issue.body);
|
||||
if (!run) {
|
||||
tally.skippedMalformed++;
|
||||
tally.malformedIssues.push(issue.number);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (deps.writeRunFile(run, runsDir)) {
|
||||
tally.imported++;
|
||||
} else {
|
||||
tally.skippedDuplicate++;
|
||||
}
|
||||
|
||||
try {
|
||||
await deps.closeIssue(repo, issue.number);
|
||||
tally.closed++;
|
||||
} catch {
|
||||
tally.closeFailed++;
|
||||
tally.closeFailedIssues.push(issue.number);
|
||||
}
|
||||
}
|
||||
|
||||
return tally;
|
||||
}
|
||||
|
||||
// ── Payload parsing (type-only validation, per NFR-5) ────────────────────────
|
||||
|
||||
function isString(v: unknown): v is string {
|
||||
return typeof v === 'string';
|
||||
}
|
||||
|
||||
function isNumber(v: unknown): v is number {
|
||||
return typeof v === 'number';
|
||||
}
|
||||
|
||||
function numOrNull(v: unknown): number | null {
|
||||
return isNumber(v) ? v : null;
|
||||
}
|
||||
|
||||
/** Type-check `keys` off `raw` (object or not) into a { [key]: number | null } map. */
|
||||
function pickNumbers<K extends string>(raw: Record<string, unknown>, keys: readonly K[]): Record<K, number | null> {
|
||||
const result = {} as Record<K, number | null>;
|
||||
for (const key of keys) result[key] = numOrNull(raw[key]);
|
||||
return result;
|
||||
}
|
||||
|
||||
const STEP1_ABSENT: Step1PlanMetrics = {
|
||||
present: false, final_confidence: null, confidence_breakdown: null,
|
||||
clarification_rounds: null, tech_stack_revision_rounds: null,
|
||||
verification_gaps_found: null, functional_requirements_count: null,
|
||||
non_functional_requirements_count: null, risk_count: null, phase_count: null,
|
||||
};
|
||||
|
||||
function parseStep1(raw: unknown): Step1PlanMetrics {
|
||||
if (raw == null || typeof raw !== 'object') return STEP1_ABSENT;
|
||||
const step1 = raw as Record<string, unknown>;
|
||||
const bdRaw = step1['confidence_breakdown'];
|
||||
const breakdown = bdRaw != null && typeof bdRaw === 'object'
|
||||
? pickNumbers(bdRaw as Record<string, unknown>, ['requirements', 'feasibility', 'integration', 'risk'])
|
||||
: null;
|
||||
return {
|
||||
present: true,
|
||||
confidence_breakdown: breakdown,
|
||||
...pickNumbers(step1, [
|
||||
'final_confidence', 'clarification_rounds', 'tech_stack_revision_rounds',
|
||||
'verification_gaps_found', 'functional_requirements_count',
|
||||
'non_functional_requirements_count', 'risk_count', 'phase_count',
|
||||
]),
|
||||
};
|
||||
}
|
||||
|
||||
const STEP2_ABSENT: Step2DocumentMetrics = {
|
||||
present: false, total_tasks: null, tasks_per_phase: null,
|
||||
phase_count: null, parallel_groups_identified: null,
|
||||
requirement_coverage_percent: null, verification_items_added: null,
|
||||
};
|
||||
|
||||
function parseStep2(raw: unknown): Step2DocumentMetrics {
|
||||
if (raw == null || typeof raw !== 'object') return STEP2_ABSENT;
|
||||
const step2 = raw as Record<string, unknown>;
|
||||
return {
|
||||
present: true,
|
||||
tasks_per_phase: null,
|
||||
...pickNumbers(step2, [
|
||||
'total_tasks', 'phase_count', 'parallel_groups_identified',
|
||||
'requirement_coverage_percent', 'verification_items_added',
|
||||
]),
|
||||
};
|
||||
}
|
||||
|
||||
const STEP3_ABSENT: Step3ImplementMetrics = {
|
||||
present: false, task_completion_rate: null, tasks_completed: null, tasks_total: null, blocker_count: null,
|
||||
};
|
||||
|
||||
function parseStep3(raw: unknown): Step3ImplementMetrics {
|
||||
if (raw == null || typeof raw !== 'object') return STEP3_ABSENT;
|
||||
const step3 = raw as Record<string, unknown>;
|
||||
return {
|
||||
present: true,
|
||||
...pickNumbers(step3, ['task_completion_rate', 'tasks_completed', 'tasks_total', 'blocker_count']),
|
||||
};
|
||||
}
|
||||
|
||||
const STEP4_ABSENT: Step4FinalizeMetrics = {
|
||||
present: false, completion_rate_at_audit: null, verification_failures_found: null,
|
||||
documentation_updates_needed: null, archival_succeeded: null,
|
||||
};
|
||||
|
||||
function parseStep4(raw: unknown): Step4FinalizeMetrics {
|
||||
if (raw == null || typeof raw !== 'object') return STEP4_ABSENT;
|
||||
const step4 = raw as Record<string, unknown>;
|
||||
const archivalRaw = step4['archival_succeeded'];
|
||||
return {
|
||||
present: true,
|
||||
archival_succeeded: typeof archivalRaw === 'boolean' ? archivalRaw : null,
|
||||
...pickNumbers(step4, ['completion_rate_at_audit', 'verification_failures_found', 'documentation_updates_needed']),
|
||||
};
|
||||
}
|
||||
|
||||
function parsePromptVersionsShort(raw: unknown): PromptVersions {
|
||||
const partial: Partial<PromptVersions> = {};
|
||||
if (raw != null && typeof raw === 'object') {
|
||||
const pv = raw as Record<string, unknown>;
|
||||
for (const key of ['plan', 'revise_plan', 'document', 'implement', 'finalize', 'init', 'init_update', 'quick_task'] as const) {
|
||||
const v = pv[key];
|
||||
if (isString(v)) partial[key] = v;
|
||||
}
|
||||
}
|
||||
return backfillPromptVersions(partial as PromptVersions);
|
||||
}
|
||||
|
||||
export function parseSubmissionPayload(body: string): RunMetrics | null {
|
||||
const parsed = extractMetricsJson(body);
|
||||
if (!parsed) return null;
|
||||
|
||||
if (parsed['schema_version'] !== '1.0') return null;
|
||||
if (!isString(parsed['run_id'])) return null;
|
||||
if (!isString(parsed['plan2code_version'])) return null;
|
||||
|
||||
const feedbackRaw = parsed['user_feedback'];
|
||||
if (feedbackRaw == null || typeof feedbackRaw !== 'object') return null;
|
||||
const feedback = feedbackRaw as Record<string, unknown>;
|
||||
if (!isNumber(feedback['overall_rating'])) return null;
|
||||
if (!isString(feedback['rating_reason'])) return null;
|
||||
if (!isString(feedback['what_went_well'])) return null;
|
||||
if (!isString(feedback['what_went_poorly'])) return null;
|
||||
|
||||
const userFeedback: UserFeedback = {
|
||||
overall_rating: feedback['overall_rating'],
|
||||
rating_reason: feedback['rating_reason'],
|
||||
what_went_well: feedback['what_went_well'],
|
||||
what_went_poorly: feedback['what_went_poorly'],
|
||||
};
|
||||
|
||||
return {
|
||||
schema_version: '1.0',
|
||||
run_id: parsed['run_id'],
|
||||
plan2code_version: parsed['plan2code_version'],
|
||||
source: 'community',
|
||||
prompt_versions: parsePromptVersionsShort(parsed['prompt_versions_short']),
|
||||
project: { name: '', started_at: null, completed_at: null },
|
||||
step1_plan: parseStep1(parsed['step1']),
|
||||
step2_document: parseStep2(parsed['step2']),
|
||||
step3_implement: parseStep3(parsed['step3']),
|
||||
step4_finalize: parseStep4(parsed['step4']),
|
||||
user_feedback: userFeedback,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,139 @@
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { validateEdit, parseProposalFromResponse } from './improver.js';
|
||||
import type { PromptEdit } from './types.js';
|
||||
|
||||
// ── Shared fixtures ───────────────────────────────────────────────────────────
|
||||
|
||||
const PROMPT_CONTENTS: Record<string, string> = {
|
||||
'plan2code-1-plan.md': 'This is the plan prompt content. It has some text here.',
|
||||
'plan2code-2-document.md': 'Document prompt with repeated text. repeated text. Done.',
|
||||
'plan2code-3-implement.md': 'Implement prompt content.',
|
||||
};
|
||||
|
||||
function makeEdit(overrides: Partial<PromptEdit> = {}): PromptEdit {
|
||||
return {
|
||||
file: 'plan2code-1-plan.md',
|
||||
rationale: 'test rationale',
|
||||
expected_metric_impact: 'test impact',
|
||||
char_count_before: PROMPT_CONTENTS['plan2code-1-plan.md'].length,
|
||||
char_count_after: PROMPT_CONTENTS['plan2code-1-plan.md'].length,
|
||||
char_count_delta: 0,
|
||||
old_text: 'some text',
|
||||
new_text: 'better text',
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
// ── validateEdit() ────────────────────────────────────────────────────────────
|
||||
|
||||
describe('validateEdit', () => {
|
||||
it('valid edit passes with no errors', () => {
|
||||
const result = validateEdit(makeEdit(), PROMPT_CONTENTS);
|
||||
expect(result.valid).toBe(true);
|
||||
expect(result.errors).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('rejects path traversal (../ in file path)', () => {
|
||||
const result = validateEdit(makeEdit({ file: '../etc/passwd' }), PROMPT_CONTENTS);
|
||||
expect(result.valid).toBe(false);
|
||||
expect(result.errors[0]).toMatch(/path traversal/i);
|
||||
});
|
||||
|
||||
it('rejects absolute paths', () => {
|
||||
const result = validateEdit(makeEdit({ file: '/etc/passwd' }), PROMPT_CONTENTS);
|
||||
expect(result.valid).toBe(false);
|
||||
expect(result.errors[0]).toMatch(/path traversal|absolute/i);
|
||||
});
|
||||
|
||||
it('errors when file not found in promptContents', () => {
|
||||
const result = validateEdit(makeEdit({ file: 'nonexistent.md' }), PROMPT_CONTENTS);
|
||||
expect(result.valid).toBe(false);
|
||||
expect(result.errors[0]).toMatch(/not found/i);
|
||||
});
|
||||
|
||||
it('errors when old_text not found in file content', () => {
|
||||
const result = validateEdit(makeEdit({ old_text: 'hallucinated text' }), PROMPT_CONTENTS);
|
||||
expect(result.valid).toBe(false);
|
||||
expect(result.errors[0]).toMatch(/not found verbatim/i);
|
||||
});
|
||||
|
||||
it('warns when old_text appears multiple times', () => {
|
||||
const edit = makeEdit({
|
||||
file: 'plan2code-2-document.md',
|
||||
old_text: 'repeated text',
|
||||
char_count_before: PROMPT_CONTENTS['plan2code-2-document.md'].length,
|
||||
char_count_after: PROMPT_CONTENTS['plan2code-2-document.md'].length,
|
||||
});
|
||||
const result = validateEdit(edit, PROMPT_CONTENTS);
|
||||
expect(result.valid).toBe(true);
|
||||
expect(result.warnings.some(w => /appears.*times/i.test(w))).toBe(true);
|
||||
});
|
||||
|
||||
it('errors when edit would exceed 11,000 char limit', () => {
|
||||
const bigText = 'x'.repeat(12_000);
|
||||
const edit = makeEdit({ new_text: bigText });
|
||||
const result = validateEdit(edit, PROMPT_CONTENTS);
|
||||
expect(result.valid).toBe(false);
|
||||
expect(result.errors.some(e => /exceed.*11.?000/i.test(e))).toBe(true);
|
||||
});
|
||||
|
||||
it('warns when reported char counts diverge from actual (>10 chars off)', () => {
|
||||
const edit = makeEdit({
|
||||
char_count_before: 999,
|
||||
char_count_after: 999,
|
||||
});
|
||||
const result = validateEdit(edit, PROMPT_CONTENTS);
|
||||
expect(result.valid).toBe(true);
|
||||
expect(result.warnings.some(w => /char_count_before.*differs/i.test(w))).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
// ── parseProposalFromResponse() ───────────────────────────────────────────────
|
||||
|
||||
describe('parseProposalFromResponse', () => {
|
||||
const sampleEdit = {
|
||||
file: 'test.md',
|
||||
rationale: 'r',
|
||||
expected_metric_impact: 'e',
|
||||
char_count_before: 100,
|
||||
char_count_after: 110,
|
||||
char_count_delta: 10,
|
||||
old_text: 'old',
|
||||
new_text: 'new',
|
||||
};
|
||||
|
||||
it('parses JSON from markdown code block (```json ... ```)', () => {
|
||||
const response = `Here is my proposal:\n\n\`\`\`json\n${JSON.stringify([sampleEdit])}\n\`\`\`\n\nDone.`;
|
||||
const result = parseProposalFromResponse(response);
|
||||
expect(result).toHaveLength(1);
|
||||
expect(result![0].file).toBe('test.md');
|
||||
});
|
||||
|
||||
it('parses JSON from bare code block (``` ... ```)', () => {
|
||||
const response = `Proposal:\n\n\`\`\`\n${JSON.stringify([sampleEdit])}\n\`\`\``;
|
||||
const result = parseProposalFromResponse(response);
|
||||
expect(result).toHaveLength(1);
|
||||
expect(result![0].old_text).toBe('old');
|
||||
});
|
||||
|
||||
it('parses bare JSON array with old_text field', () => {
|
||||
const response = `Some preamble\n${JSON.stringify([sampleEdit])}\nSome postamble`;
|
||||
const result = parseProposalFromResponse(response);
|
||||
expect(result).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('returns null for non-JSON response', () => {
|
||||
const result = parseProposalFromResponse('No changes needed at this time.');
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
|
||||
it('returns null for malformed JSON', () => {
|
||||
const result = parseProposalFromResponse('```json\n{broken json]\n```');
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
|
||||
it('returns null for empty response', () => {
|
||||
const result = parseProposalFromResponse('');
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,288 @@
|
||||
/**
|
||||
* improver.ts
|
||||
* Reads diagnosis + prompt files, invokes AI, parses PromptEdit[] from response.
|
||||
* Validates: old_text verbatim match, char count limits.
|
||||
*/
|
||||
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import type { PromptEdit, PromptProposal } from './types.js';
|
||||
import { invokeLLM, type AgentType } from './invoke-llm.js';
|
||||
|
||||
const CHAR_LIMIT = 11_000;
|
||||
const IMPROVE_PROMPT_PATH = new URL('../src/prompts/improve.md', import.meta.url).pathname
|
||||
.replace(/^\/([A-Za-z]:)/, '$1'); // Fix Windows path
|
||||
|
||||
// ── Helpers ───────────────────────────────────────────────────────────────────
|
||||
|
||||
function interpolate(template: string, vars: Record<string, string>): string {
|
||||
let result = template;
|
||||
for (const [key, value] of Object.entries(vars)) {
|
||||
result = result.replaceAll(`{{${key}}}`, value);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
function generateProposalId(): string {
|
||||
const now = new Date();
|
||||
const ts = now.toISOString().replace(/[-:T.Z]/g, '').slice(0, 14);
|
||||
return `prop-${ts}`;
|
||||
}
|
||||
|
||||
function readPromptFiles(plan2codeRoot: string): Record<string, string> {
|
||||
const srcDir = path.join(plan2codeRoot, 'src');
|
||||
const promptFiles = [
|
||||
'plan2code-1-plan.md',
|
||||
'plan2code-1b-revise-plan.md',
|
||||
'plan2code-2-document.md',
|
||||
'plan2code-3-implement.md',
|
||||
'plan2code-4-finalize.md',
|
||||
'plan2code-init.md',
|
||||
'plan2code-init-update.md',
|
||||
'plan2code-quick-task.md',
|
||||
];
|
||||
|
||||
const contents: Record<string, string> = {};
|
||||
for (const file of promptFiles) {
|
||||
try {
|
||||
contents[file] = fs.readFileSync(path.join(srcDir, file), 'utf8');
|
||||
} catch {
|
||||
contents[file] = '';
|
||||
}
|
||||
}
|
||||
return contents;
|
||||
}
|
||||
|
||||
// ── Edit validation ───────────────────────────────────────────────────────────
|
||||
|
||||
export interface ValidationResult {
|
||||
valid: boolean;
|
||||
errors: string[];
|
||||
warnings: string[];
|
||||
}
|
||||
|
||||
export function validateEdit(
|
||||
edit: PromptEdit,
|
||||
promptContents: Record<string, string>,
|
||||
): ValidationResult {
|
||||
const errors: string[] = [];
|
||||
const warnings: string[] = [];
|
||||
|
||||
// Reject path traversal attempts
|
||||
if (edit.file.includes('..') || path.isAbsolute(edit.file)) {
|
||||
errors.push(`Rejected: "${edit.file}" contains path traversal or absolute path.`);
|
||||
return { valid: false, errors, warnings };
|
||||
}
|
||||
|
||||
// Check target file exists
|
||||
const fileContent = promptContents[edit.file];
|
||||
if (fileContent === undefined) {
|
||||
errors.push(`Target file "${edit.file}" not found. Valid files: ${Object.keys(promptContents).join(', ')}`);
|
||||
return { valid: false, errors, warnings };
|
||||
}
|
||||
|
||||
// Check old_text exists verbatim in the file
|
||||
if (!fileContent.includes(edit.old_text)) {
|
||||
errors.push(`old_text not found verbatim in "${edit.file}". The AI may have hallucinated text.`);
|
||||
} else {
|
||||
// Warn if old_text appears more than once (ambiguous match)
|
||||
const occurrences = fileContent.split(edit.old_text).length - 1;
|
||||
if (occurrences > 1) {
|
||||
warnings.push(`old_text appears ${occurrences} times in "${edit.file}". Only the first occurrence will be replaced.`);
|
||||
}
|
||||
}
|
||||
|
||||
// Check char count after edit
|
||||
const afterContent = fileContent.replace(edit.old_text, edit.new_text);
|
||||
if (afterContent.length > CHAR_LIMIT) {
|
||||
errors.push(`Edit would cause "${edit.file}" to exceed ${CHAR_LIMIT} char limit (would be ${afterContent.length} chars).`);
|
||||
}
|
||||
|
||||
// Verify reported char counts match reality
|
||||
const actualBefore = fileContent.length;
|
||||
const actualAfter = afterContent.length;
|
||||
if (Math.abs(edit.char_count_before - actualBefore) > 10) {
|
||||
warnings.push(`Reported char_count_before (${edit.char_count_before}) differs from actual (${actualBefore}).`);
|
||||
}
|
||||
if (Math.abs(edit.char_count_after - actualAfter) > 10) {
|
||||
warnings.push(`Reported char_count_after (${edit.char_count_after}) differs from actual (${actualAfter}).`);
|
||||
}
|
||||
|
||||
return { valid: errors.length === 0, errors, warnings };
|
||||
}
|
||||
|
||||
// ── AI response parsing ───────────────────────────────────────────────────────
|
||||
|
||||
export function parseProposalFromResponse(response: string): PromptEdit[] | null {
|
||||
// Look for JSON code block containing PromptEdit[]
|
||||
const jsonBlockMatch = response.match(/```(?:json)?\s*(\[[\s\S]*?\])\s*```/);
|
||||
if (!jsonBlockMatch) {
|
||||
// Try bare JSON array
|
||||
const bareMatch = response.match(/(\[[\s\S]*"old_text"[\s\S]*\])/);
|
||||
if (!bareMatch) return null;
|
||||
try {
|
||||
return JSON.parse(bareMatch[1]) as PromptEdit[];
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
return JSON.parse(jsonBlockMatch[1]) as PromptEdit[];
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// ── Main improver ─────────────────────────────────────────────────────────────
|
||||
|
||||
export interface ImproverOptions {
|
||||
diagnosisPath: string; // Path to diagnosis markdown file
|
||||
plan2codeRoot: string; // Path to plan2code repo root
|
||||
proposalsDir: string; // Where to save proposal JSON
|
||||
runsDir: string; // For tracking which runs this is based on
|
||||
model?: string;
|
||||
agent?: AgentType; // Agent to use (default: claude-code)
|
||||
}
|
||||
|
||||
export interface ImproverResult {
|
||||
proposalPath: string;
|
||||
proposal: PromptProposal;
|
||||
validationResults: Array<{ edit: PromptEdit; result: ValidationResult }>;
|
||||
validEditCount: number;
|
||||
invalidEditCount: number;
|
||||
}
|
||||
|
||||
export async function generateImprovement(opts: ImproverOptions): Promise<ImproverResult> {
|
||||
const { diagnosisPath, plan2codeRoot, proposalsDir, runsDir, model = 'default', agent = 'claude-code' } = opts;
|
||||
|
||||
// Load diagnosis
|
||||
let diagnosisContent: string;
|
||||
try {
|
||||
diagnosisContent = fs.readFileSync(diagnosisPath, 'utf8');
|
||||
} catch {
|
||||
throw new Error(`Could not read diagnosis file at ${diagnosisPath}`);
|
||||
}
|
||||
|
||||
// Read prompt files
|
||||
const promptContents = readPromptFiles(plan2codeRoot);
|
||||
const srcDir = path.join(plan2codeRoot, 'src');
|
||||
|
||||
// Build char counts for each file
|
||||
const charCounts = Object.entries(promptContents)
|
||||
.map(([file, content]) => `| ${file} | ${content.length} | ${CHAR_LIMIT} | ${CHAR_LIMIT - content.length} headroom |`)
|
||||
.join('\n');
|
||||
|
||||
const promptContentsStr = Object.entries(promptContents)
|
||||
.map(([file, content]) => `## ${file} (${content.length} chars)\n\n${content}`)
|
||||
.join('\n\n---\n\n');
|
||||
|
||||
// Load improve prompt template
|
||||
let improveTemplate: string;
|
||||
try {
|
||||
improveTemplate = fs.readFileSync(IMPROVE_PROMPT_PATH, 'utf8');
|
||||
} catch {
|
||||
const altPath = path.join(process.cwd(), 'src', 'prompts', 'improve.md');
|
||||
improveTemplate = fs.readFileSync(altPath, 'utf8');
|
||||
}
|
||||
|
||||
const fullPrompt = interpolate(improveTemplate, {
|
||||
diagnosisContent,
|
||||
promptContents: promptContentsStr,
|
||||
charCounts: `| File | Current Chars | Limit | Headroom |\n|------|--------------|-------|----------|\n${charCounts}`,
|
||||
});
|
||||
|
||||
// Invoke Claude
|
||||
console.log(`\nInvoking AI improvement proposal (agent: ${agent}, model: ${model === 'default' ? 'user default' : model})...`);
|
||||
console.log('This may take a minute...\n');
|
||||
|
||||
let aiResponse: string;
|
||||
try {
|
||||
aiResponse = await invokeLLM({
|
||||
prompt: fullPrompt,
|
||||
model,
|
||||
agent,
|
||||
timeout: 300_000,
|
||||
});
|
||||
} catch (err) {
|
||||
throw new Error(`AI invocation failed: ${err instanceof Error ? err.message : String(err)}`);
|
||||
}
|
||||
|
||||
// Parse edits
|
||||
const rawEdits = parseProposalFromResponse(aiResponse);
|
||||
if (!rawEdits || rawEdits.length === 0) {
|
||||
throw new Error('Could not parse PromptEdit[] from AI response. The AI may not have produced a valid JSON block.');
|
||||
}
|
||||
|
||||
// Enforce max edits per cycle
|
||||
const MAX_EDITS = 5;
|
||||
if (rawEdits.length > MAX_EDITS) {
|
||||
console.warn(`\n⚠ AI generated ${rawEdits.length} edits (max is ${MAX_EDITS}). Truncating to first ${MAX_EDITS}.`);
|
||||
rawEdits.length = MAX_EDITS;
|
||||
}
|
||||
|
||||
// Validate each edit
|
||||
const validationResults: ImproverResult['validationResults'] = [];
|
||||
const validEdits: PromptEdit[] = [];
|
||||
|
||||
for (const edit of rawEdits) {
|
||||
const result = validateEdit(edit, promptContents);
|
||||
validationResults.push({ edit, result });
|
||||
|
||||
if (result.valid) {
|
||||
// Compute accurate char counts
|
||||
const fileContent = promptContents[edit.file] ?? '';
|
||||
const afterContent = fileContent.replace(edit.old_text, edit.new_text);
|
||||
edit.char_count_before = fileContent.length;
|
||||
edit.char_count_after = afterContent.length;
|
||||
edit.char_count_delta = afterContent.length - fileContent.length;
|
||||
validEdits.push(edit);
|
||||
} else {
|
||||
console.warn(`\n⚠ Edit rejected for "${edit.file}":`);
|
||||
for (const err of result.errors) {
|
||||
console.warn(` - ${err}`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const warn of result.warnings) {
|
||||
console.warn(` Warning: ${warn}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Get run IDs that contributed to this analysis
|
||||
const runIds: string[] = [];
|
||||
try {
|
||||
const files = fs.readdirSync(runsDir)
|
||||
.filter(f => f.startsWith('run-') && f.endsWith('.json'));
|
||||
runIds.push(...files.map(f => f.replace('.json', '')));
|
||||
} catch { /* no runs dir */ }
|
||||
|
||||
// Build proposal
|
||||
const proposalId = generateProposalId();
|
||||
const proposal: PromptProposal = {
|
||||
proposal_id: proposalId,
|
||||
created_at: new Date().toISOString(),
|
||||
based_on_runs: runIds,
|
||||
analyst_model: model,
|
||||
proposals: validEdits,
|
||||
status: 'pending',
|
||||
diagnosis_file: path.basename(diagnosisPath),
|
||||
};
|
||||
|
||||
// Save proposal JSON
|
||||
fs.mkdirSync(proposalsDir, { recursive: true });
|
||||
const proposalPath = path.join(proposalsDir, `${proposalId}.json`);
|
||||
fs.writeFileSync(proposalPath, JSON.stringify(proposal, null, 2), 'utf8');
|
||||
|
||||
// Also save raw AI response alongside
|
||||
const rawPath = path.join(proposalsDir, `${proposalId}-raw.md`);
|
||||
fs.writeFileSync(rawPath, aiResponse, 'utf8');
|
||||
|
||||
return {
|
||||
proposalPath,
|
||||
proposal,
|
||||
validationResults,
|
||||
validEditCount: validEdits.length,
|
||||
invalidEditCount: rawEdits.length - validEdits.length,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
// Public API for plan2code-metrics
|
||||
export { collectRun, collectPromptVersions } from './collector.js';
|
||||
export { aggregate, loadAggregated, loadRunFiles, importRun } from './aggregator.js';
|
||||
export { runAnalysis } from './analyzer.js';
|
||||
export { generateImprovement, validateEdit, parseProposalFromResponse } from './improver.js';
|
||||
export { reviewAndApply } from './applier.js';
|
||||
export { invokeLLM, AGENTS } from './invoke-llm.js';
|
||||
export type { AgentType, InvokeLLMOptions } from './invoke-llm.js';
|
||||
export { runCLI } from './cli.js';
|
||||
export { METRIC_TARGETS } from './types.js';
|
||||
export type {
|
||||
RunMetrics,
|
||||
UserFeedback,
|
||||
PromptVersions,
|
||||
PromptEdit,
|
||||
PromptProposal,
|
||||
AggregatedMetrics,
|
||||
CohortMetrics,
|
||||
} from './types.js';
|
||||
@@ -0,0 +1,114 @@
|
||||
/**
|
||||
* invoke-llm.ts
|
||||
* Unified LLM invocation for plan2code-metrics.
|
||||
* Supports Claude Code (temp file → stdin), Copilot CLI (stdin string), and
|
||||
* Devin CLI (temp prompt file).
|
||||
* Mirrors the agent pattern from plan2code-loop.
|
||||
*/
|
||||
|
||||
import { execa } from 'execa';
|
||||
import { writeFileSync, unlinkSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
|
||||
// ── Agent definitions ────────────────────────────────────────────────────────
|
||||
|
||||
export type AgentType = 'claude-code' | 'copilot-cli' | 'devin-cli';
|
||||
|
||||
export interface AgentDef {
|
||||
name: AgentType;
|
||||
displayName: string;
|
||||
command: string;
|
||||
defaultModel: string;
|
||||
}
|
||||
|
||||
export const AGENTS: Record<AgentType, AgentDef> = {
|
||||
'claude-code': {
|
||||
name: 'claude-code',
|
||||
displayName: 'Claude Code',
|
||||
command: 'claude',
|
||||
defaultModel: 'default',
|
||||
},
|
||||
'copilot-cli': {
|
||||
name: 'copilot-cli',
|
||||
displayName: 'GitHub Copilot CLI',
|
||||
command: 'copilot',
|
||||
defaultModel: 'claude-sonnet-4',
|
||||
},
|
||||
'devin-cli': {
|
||||
name: 'devin-cli',
|
||||
displayName: 'Devin CLI',
|
||||
command: 'devin',
|
||||
defaultModel: 'default',
|
||||
},
|
||||
};
|
||||
|
||||
// ── Invocation ───────────────────────────────────────────────────────────────
|
||||
|
||||
export interface InvokeLLMOptions {
|
||||
prompt: string;
|
||||
model: string;
|
||||
agent: AgentType;
|
||||
timeout?: number;
|
||||
}
|
||||
|
||||
export async function invokeLLM(opts: InvokeLLMOptions): Promise<string> {
|
||||
const { prompt, model, agent, timeout = 300_000 } = opts;
|
||||
const def = AGENTS[agent];
|
||||
|
||||
if (agent === 'claude-code') {
|
||||
// Write prompt to temp file — more reliable than stdin on Windows
|
||||
const tempFile = join(tmpdir(), `plan2code-metrics-prompt-${Date.now()}.txt`);
|
||||
writeFileSync(tempFile, prompt, 'utf-8');
|
||||
|
||||
try {
|
||||
const args: string[] = [
|
||||
'--print',
|
||||
'--dangerously-skip-permissions',
|
||||
];
|
||||
// Only add --model if not using default
|
||||
if (model && model !== 'default') {
|
||||
args.push('--model', model);
|
||||
}
|
||||
|
||||
const result = await execa(def.command, args, {
|
||||
inputFile: tempFile,
|
||||
timeout,
|
||||
});
|
||||
return result.stdout;
|
||||
} finally {
|
||||
try { unlinkSync(tempFile); } catch { /* ignore cleanup errors */ }
|
||||
}
|
||||
} else if (agent === 'devin-cli') {
|
||||
// Devin CLI: load prompt from a temp file, run single-turn, auto-approve tool calls
|
||||
const tempFile = join(tmpdir(), `plan2code-metrics-prompt-${Date.now()}.txt`);
|
||||
writeFileSync(tempFile, prompt, 'utf-8');
|
||||
|
||||
try {
|
||||
const args: string[] = ['--print', '--prompt-file', tempFile, '--permission-mode', 'dangerous'];
|
||||
// Only add --model if not using default
|
||||
if (model && model !== 'default') {
|
||||
args.push('--model', model);
|
||||
}
|
||||
|
||||
const result = await execa(def.command, args, { timeout });
|
||||
return result.stdout;
|
||||
} finally {
|
||||
try { unlinkSync(tempFile); } catch { /* ignore cleanup errors */ }
|
||||
}
|
||||
} else {
|
||||
// Copilot CLI: pipe prompt via stdin string
|
||||
const args: string[] = [];
|
||||
// Only add --model if not using default
|
||||
if (model && model !== 'default') {
|
||||
args.push('--model', model);
|
||||
}
|
||||
args.push('--allow-all-tools', '-s');
|
||||
|
||||
const result = await execa(def.command, args, {
|
||||
input: prompt,
|
||||
timeout,
|
||||
});
|
||||
return result.stdout;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
# PLAN2CODE METRICS ANALYSIS REQUEST
|
||||
|
||||
You are a senior AI systems analyst specializing in prompt engineering quality assessment. Your role is to diagnose weaknesses in the plan2code workflow prompts by examining aggregated run metrics.
|
||||
|
||||
**IMPORTANT:** Do NOT propose specific edits in this response. Diagnosis only. The improvement step is separate.
|
||||
|
||||
---
|
||||
|
||||
## Aggregated Run Metrics
|
||||
|
||||
The following JSON contains metrics aggregated from real plan2code project runs, grouped by prompt "generation" (a cohort is identified by the SHA fingerprint of the src/plan2code-*.md prompt files at collection time):
|
||||
|
||||
```json
|
||||
{{aggregatedMetrics}}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Current Prompt File Contents
|
||||
|
||||
The following are the current contents of the plan2code workflow prompt files being evaluated:
|
||||
|
||||
{{promptContents}}
|
||||
|
||||
---
|
||||
|
||||
## Metric Targets Reference
|
||||
|
||||
| Metric | Target | Direction |
|
||||
|--------|--------|-----------|
|
||||
| avg_confidence (Step 1) | ≥ 90 | higher is better |
|
||||
| avg_clarification_rounds (Step 1) | ≤ 2.0 | lower is better |
|
||||
| avg_verification_gaps_found (Step 1) | ≤ 2.0 | lower is better |
|
||||
| avg_parallel_groups (Step 2) | ≥ 0.5 | higher is better |
|
||||
| avg_verification_items_added (Step 2) | ≤ 1.5 | lower is better |
|
||||
| avg_task_completion_rate (Step 3) | ≥ 0.95 | higher is better |
|
||||
| avg_blocker_count (Step 3) | ≤ 1.5 | lower is better |
|
||||
| avg_verification_failures_found (Step 4) | ≤ 1.0 | lower is better |
|
||||
| archival_success_rate (Step 4) | ≥ 0.99 | higher is better |
|
||||
| avg_user_rating (Feedback) | ≥ 7.0 | higher is better (1-10 scale, null if no feedback) |
|
||||
|
||||
---
|
||||
|
||||
## Analysis Instructions
|
||||
|
||||
1. Treat the aggregated JSON as the authoritative source of truth about run quality.
|
||||
2. Compare each metric against its target. Calculate delta (actual − target).
|
||||
3. For metrics that miss their target, identify the specific section of the relevant prompt file most likely responsible.
|
||||
4. Acknowledge provisional confidence explicitly when N < 5 runs in a cohort.
|
||||
5. If 2+ generations exist, compare them to identify trend direction (improving/degrading/flat).
|
||||
6. Root cause hypotheses must name a specific file AND a specific section within that file.
|
||||
7. Do not invent metrics not present in the JSON. If a metric is null, note it as "insufficient data."
|
||||
|
||||
---
|
||||
|
||||
## Required Output Format
|
||||
|
||||
Produce EXACTLY the following sections in order. Use these exact headers — they are parsed by machine:
|
||||
|
||||
# PLAN2CODE METRICS DIAGNOSIS
|
||||
|
||||
### Metrics Summary
|
||||
|
||||
A markdown table with columns: Step | Metric | Target | Actual | Delta | Status (✓/✗/—)
|
||||
|
||||
Include ALL metrics listed in the targets table. Use "—" for null values.
|
||||
|
||||
### Step Health Assessment
|
||||
|
||||
For each step (1–4), provide:
|
||||
- **Grade:** A–F
|
||||
- **Key signals:** 2–4 bullet points with specific metric values
|
||||
- **Assessment:** 1–2 sentence diagnosis
|
||||
|
||||
### Root Cause Hypotheses
|
||||
|
||||
Numbered list. For each underperforming metric:
|
||||
1. **Metric:** [metric name] | **Value:** [actual] | **Target:** [target]
|
||||
- **File:** [plan2code-X-name.md]
|
||||
- **Section:** [specific heading or section name]
|
||||
- **Hypothesis:** [specific gap in the prompt that would explain the metric miss]
|
||||
- **Confidence:** [High/Medium/Low] — [reason for confidence level]
|
||||
|
||||
### Recommended Improvement Targets
|
||||
|
||||
Ordered list (highest estimated impact first). For each:
|
||||
- **File:** [filename]
|
||||
- **Section:** [section name]
|
||||
- **Why:** [link to specific metric being addressed]
|
||||
- **Priority:** [High/Medium/Low]
|
||||
|
||||
### Generation Comparison
|
||||
|
||||
If 2+ generations exist: A comparison table showing before/after for each metric per generation, with trend arrows (▲/▼/→).
|
||||
|
||||
If fewer than 2 generations: "Insufficient generation data for comparison. Current generation: [cohort_key], [N] runs."
|
||||
|
||||
---
|
||||
|
||||
End of analysis request.
|
||||
@@ -0,0 +1,92 @@
|
||||
# PLAN2CODE PROMPT IMPROVEMENT REQUEST
|
||||
|
||||
You are a senior AI prompt engineer. Your role is to propose surgical, targeted edits to the plan2code workflow prompt files based on a metrics diagnosis. You make precise, minimal changes — NOT rewrites.
|
||||
|
||||
---
|
||||
|
||||
## Metrics Diagnosis
|
||||
|
||||
The following diagnosis was produced by the analysis step:
|
||||
|
||||
{{diagnosisContent}}
|
||||
|
||||
---
|
||||
|
||||
## Current Prompt File Contents (with char counts)
|
||||
|
||||
{{promptContents}}
|
||||
|
||||
---
|
||||
|
||||
## Character Count Status
|
||||
|
||||
{{charCounts}}
|
||||
|
||||
---
|
||||
|
||||
## Hard Constraints — ALL must be satisfied:
|
||||
|
||||
1. **Char limit:** Each target file MUST stay under 11,000 characters after your edit is applied. This is enforced by code — edits that violate it will be automatically rejected.
|
||||
2. **Maximum 5 edits per cycle.** Focus on the highest-impact changes only.
|
||||
3. **Each edit must cite a specific metric** in its `expected_metric_impact` field (e.g., "avg_task_completion_rate", "avg_blocker_count").
|
||||
4. **`old_text` must be verbatim** from the file. Copy-paste exactly — include surrounding whitespace/newlines as they appear. Edits with mismatched old_text will be automatically rejected.
|
||||
5. **Do NOT modify Role sections** (lines starting with "You are" at the top of each file) or step headings (lines starting with `#`).
|
||||
6. **Each edit must be independently applicable** — no edit should depend on another edit being applied first.
|
||||
7. **Prefer additive guidance over deletions.** Adding clarifying instructions or examples is safer than removing existing text.
|
||||
8. **Do not change the overall structure** or flow of any prompt file.
|
||||
|
||||
---
|
||||
|
||||
## Edit Strategy Guidelines
|
||||
|
||||
- Target the specific sections identified in "Recommended Improvement Targets" from the diagnosis.
|
||||
- For high `avg_clarification_rounds`: Add more upfront specification examples or decision criteria to Step 1.
|
||||
- For high `avg_blocker_count`: Add blocker-recovery guidance or prerequisite check instructions.
|
||||
- For low `avg_confidence`: Strengthen the confidence calculation instructions with clearer rubrics.
|
||||
- For low `avg_parallel_groups`: Add explicit guidance for identifying parallel tasks in Step 2.
|
||||
- For low `avg_user_rating`: Review user feedback themes (what_went_well, what_went_poorly) for systemic issues.
|
||||
- Keep each `new_text` as short as possible while still addressing the root cause.
|
||||
|
||||
---
|
||||
|
||||
## Required Output Format
|
||||
|
||||
Produce EXACTLY the following sections. The JSON block is parsed by machine — it must be syntactically valid.
|
||||
|
||||
# PLAN2CODE PROMPT IMPROVEMENT PROPOSAL
|
||||
|
||||
### Improvement Rationale
|
||||
|
||||
2–3 paragraphs explaining:
|
||||
1. Which metrics are being addressed and why they matter
|
||||
2. The specific prompt gaps identified in the diagnosis that you are targeting
|
||||
3. Why the proposed edits are expected to improve those metrics
|
||||
|
||||
### Proposed Edits
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"file": "plan2code-X-name.md",
|
||||
"rationale": "One sentence explaining what this edit fixes",
|
||||
"expected_metric_impact": "avg_metric_name: expected direction and magnitude",
|
||||
"char_count_before": 0,
|
||||
"char_count_after": 0,
|
||||
"char_count_delta": 0,
|
||||
"old_text": "exact verbatim text from the file to replace",
|
||||
"new_text": "replacement text"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Set `char_count_before`, `char_count_after`, and `char_count_delta` to your best estimate (the system will verify and correct these automatically).
|
||||
|
||||
### Character Count Verification
|
||||
|
||||
A table with columns: File | Before | Projected After | Delta | Limit | Status (✓/✗)
|
||||
|
||||
Verify that NO file exceeds 11,000 characters after your proposed edits.
|
||||
|
||||
---
|
||||
|
||||
End of improvement request.
|
||||
@@ -0,0 +1,169 @@
|
||||
// All TypeScript interfaces for plan2code-metrics
|
||||
|
||||
export interface PromptVersions {
|
||||
plan: string; // sha256:... plan2code-1-plan.md
|
||||
revise_plan: string; // plan2code-1b-revise-plan.md
|
||||
document: string; // plan2code-2-document.md
|
||||
implement: string; // plan2code-3-implement.md
|
||||
finalize: string; // plan2code-4-finalize.md
|
||||
init: string; // plan2code-init.md
|
||||
init_update: string; // plan2code-init-update.md
|
||||
quick_task: string; // plan2code-quick-task.md
|
||||
}
|
||||
|
||||
export interface Step1PlanMetrics {
|
||||
present: boolean;
|
||||
final_confidence: number | null;
|
||||
confidence_breakdown: {
|
||||
requirements: number | null;
|
||||
feasibility: number | null;
|
||||
integration: number | null;
|
||||
risk: number | null;
|
||||
} | null;
|
||||
clarification_rounds: number | null;
|
||||
tech_stack_revision_rounds: number | null;
|
||||
verification_gaps_found: number | null;
|
||||
functional_requirements_count: number | null;
|
||||
non_functional_requirements_count: number | null;
|
||||
risk_count: number | null;
|
||||
phase_count: number | null;
|
||||
}
|
||||
|
||||
export interface Step2DocumentMetrics {
|
||||
present: boolean;
|
||||
total_tasks: number | null;
|
||||
tasks_per_phase: number[] | null;
|
||||
phase_count: number | null;
|
||||
parallel_groups_identified: number | null;
|
||||
requirement_coverage_percent: number | null;
|
||||
verification_items_added: number | null;
|
||||
}
|
||||
|
||||
export interface Step3ImplementMetrics {
|
||||
present: boolean;
|
||||
task_completion_rate: number | null;
|
||||
tasks_completed: number | null;
|
||||
tasks_total: number | null;
|
||||
blocker_count: number | null;
|
||||
}
|
||||
|
||||
export interface Step4FinalizeMetrics {
|
||||
present: boolean;
|
||||
completion_rate_at_audit: number | null;
|
||||
verification_failures_found: number | null;
|
||||
documentation_updates_needed: number | null;
|
||||
archival_succeeded: boolean | null;
|
||||
}
|
||||
|
||||
export interface UserFeedback {
|
||||
overall_rating: number; // 1-10
|
||||
rating_reason: string;
|
||||
what_went_well: string;
|
||||
what_went_poorly: string;
|
||||
}
|
||||
|
||||
export interface RunMetrics {
|
||||
schema_version: '1.0';
|
||||
run_id: string;
|
||||
plan2code_version: string;
|
||||
// Origin of the run. Local runs are collected on the maintainer's machine;
|
||||
// community runs are ingested from GitHub feedback issues. Absent on pre-v1.17
|
||||
// run files, which are treated as 'local'. Drives cohort keying (see
|
||||
// cohortKeyForRun in aggregator.ts).
|
||||
source?: 'local' | 'community';
|
||||
prompt_versions: PromptVersions;
|
||||
project: {
|
||||
name: string;
|
||||
started_at: string | null;
|
||||
completed_at: string | null;
|
||||
};
|
||||
step1_plan: Step1PlanMetrics;
|
||||
step2_document: Step2DocumentMetrics;
|
||||
step3_implement: Step3ImplementMetrics;
|
||||
step4_finalize: Step4FinalizeMetrics;
|
||||
user_feedback: UserFeedback | null;
|
||||
}
|
||||
|
||||
export interface PromptEdit {
|
||||
file: string;
|
||||
rationale: string;
|
||||
expected_metric_impact: string;
|
||||
char_count_before: number;
|
||||
char_count_after: number;
|
||||
char_count_delta: number;
|
||||
old_text: string;
|
||||
new_text: string;
|
||||
}
|
||||
|
||||
export interface PromptProposal {
|
||||
proposal_id: string;
|
||||
created_at: string;
|
||||
based_on_runs: string[];
|
||||
analyst_model: string;
|
||||
proposals: PromptEdit[];
|
||||
status: 'pending' | 'applied' | 'rejected';
|
||||
diagnosis_file: string | null;
|
||||
}
|
||||
|
||||
// Aggregated metrics schema
|
||||
export interface CohortMetrics {
|
||||
cohort_key: string; // local: hash of sorted prompt_versions; community: `community:v<version>`
|
||||
source?: 'local' | 'community';
|
||||
prompt_versions: PromptVersions;
|
||||
run_count: number;
|
||||
run_ids: string[];
|
||||
first_seen: string;
|
||||
last_seen: string;
|
||||
|
||||
// Step 1 averages
|
||||
avg_confidence: number | null;
|
||||
avg_clarification_rounds: number | null;
|
||||
avg_verification_gaps_found: number | null;
|
||||
avg_functional_requirements_count: number | null;
|
||||
avg_non_functional_requirements_count: number | null;
|
||||
avg_risk_count: number | null;
|
||||
avg_phase_count_step1: number | null;
|
||||
|
||||
// Step 2 averages
|
||||
avg_total_tasks: number | null;
|
||||
avg_phase_count_step2: number | null;
|
||||
avg_parallel_groups: number | null;
|
||||
avg_requirement_coverage_percent: number | null;
|
||||
avg_verification_items_added: number | null;
|
||||
|
||||
// Step 3 averages
|
||||
avg_task_completion_rate: number | null;
|
||||
avg_blocker_count: number | null;
|
||||
|
||||
// Step 4 averages
|
||||
avg_completion_rate_at_audit: number | null;
|
||||
avg_verification_failures_found: number | null;
|
||||
avg_documentation_updates_needed: number | null;
|
||||
archival_success_rate: number | null;
|
||||
|
||||
// User feedback
|
||||
avg_user_rating: number | null;
|
||||
feedback_count: number;
|
||||
}
|
||||
|
||||
export interface AggregatedMetrics {
|
||||
schema_version: '1.0';
|
||||
last_updated: string;
|
||||
total_runs: number;
|
||||
cohorts: CohortMetrics[];
|
||||
current_cohort_key: string | null;
|
||||
}
|
||||
|
||||
// Metric targets for health assessment
|
||||
export const METRIC_TARGETS = {
|
||||
avg_confidence: { target: 90, direction: 'gte' as const },
|
||||
avg_clarification_rounds: { target: 2.0, direction: 'lte' as const },
|
||||
avg_verification_gaps_found: { target: 2.0, direction: 'lte' as const },
|
||||
avg_parallel_groups: { target: 0.5, direction: 'gte' as const },
|
||||
avg_verification_items_added: { target: 1.5, direction: 'lte' as const },
|
||||
avg_task_completion_rate: { target: 0.95, direction: 'gte' as const },
|
||||
avg_blocker_count: { target: 1.5, direction: 'lte' as const },
|
||||
avg_verification_failures_found: { target: 1.0, direction: 'lte' as const },
|
||||
archival_success_rate: { target: 0.99, direction: 'gte' as const },
|
||||
avg_user_rating: { target: 7.0, direction: 'gte' as const },
|
||||
} as const;
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "bundler",
|
||||
"lib": ["ES2022"],
|
||||
"outDir": "dist",
|
||||
"rootDir": ".",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"resolveJsonModule": true,
|
||||
"declaration": true,
|
||||
"declarationMap": true,
|
||||
"sourceMap": true
|
||||
},
|
||||
"include": ["src/**/*"],
|
||||
"exclude": ["node_modules", "dist"]
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
import { defineConfig } from 'tsup';
|
||||
|
||||
export default defineConfig({
|
||||
entry: {
|
||||
'bin/plan2code-metrics': 'src/bin/plan2code-metrics.ts',
|
||||
index: 'src/index.ts',
|
||||
},
|
||||
format: ['esm'],
|
||||
dts: true,
|
||||
clean: true,
|
||||
sourcemap: true,
|
||||
banner: {
|
||||
js: '#!/usr/bin/env node',
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,7 @@
|
||||
import { defineConfig } from 'vitest/config';
|
||||
|
||||
export default defineConfig({
|
||||
test: {
|
||||
include: ['src/**/*.test.ts'],
|
||||
},
|
||||
});
|
||||
@@ -6,6 +6,9 @@ const path = require('path');
|
||||
const CHAR_LIMIT = 11000;
|
||||
const srcDir = path.join(__dirname, '..', 'src');
|
||||
|
||||
// Note: readdirSync is non-recursive, so files in subdirectories like
|
||||
// plan2code-review-references/ are automatically excluded from
|
||||
// the character limit check. Reference files have no char limit.
|
||||
const files = fs.readdirSync(srcDir)
|
||||
.filter(f => f.startsWith('plan2code-') && f.endsWith('.md'))
|
||||
.sort();
|
||||
|
||||
@@ -0,0 +1,157 @@
|
||||
---
|
||||
name: plan2code-0-pathfinder
|
||||
description: "Plan2Code Step 0: Pathfinder Mode - user-initiated workflow step. Do not invoke autonomously."
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
# 🧭 PATHFINDER MODE
|
||||
|
||||
Start all PATHFINDER MODE responses with '🧭 [PATHFINDER: Chart - Step X: Name]' or '🧭 [PATHFINDER: Work - Step X: Name]'.
|
||||
|
||||
## Role
|
||||
|
||||
Pathfinder, not architect. An idea has arrived too big or unclear to plan. Chart the way as a map of decision **questions**, then clear them ONE PER SESSION until nothing is left to decide. Hand off to `/plan2code-1-plan`.
|
||||
|
||||
Read references/grilling.md
|
||||
|
||||
> Fallback: ≤3 independent probes/turn, each with a recommendation, re-ask any skipped; structured tool first, prose only on a detail-test trip; plain English, no jargon; facts you look up, decisions are the human's.
|
||||
|
||||
## Backend
|
||||
|
||||
The map lives in ONE of two places — the human's pick at Chart Step 1, never yours:
|
||||
|
||||
- **local** (default) — files under `specs/<idea>/pathfinder/`. Private, gitignored, solo.
|
||||
- **github** — a `pathfinder:map` issue whose questions are sub-issues, driven by `gh`. Shared, visible in the tracker UI, parallel.
|
||||
|
||||
Read references/github-issues.md — REQUIRED on `github`, skip it on `local`.
|
||||
|
||||
> Fallback: map = issue labelled `pathfinder:map` titled `Map: <idea>`; questions = its sub-issues, labelled `pathfinder:<type>-<mode>`; blocking = native issue dependencies; claim = assign `@me`; resolve = `## Answer` comment, then close.
|
||||
|
||||
Recorded as the first `## Ground rules` bullet (`**Backend:** local|github`), never re-asked, never switched. Either way the PLAN-DRAFT lands in local `specs/<idea>/` — downstream steps read files, not issues.
|
||||
|
||||
## Project Context
|
||||
|
||||
Load `./AGENTS.md` if it exists — its conventions govern; never re-ask what it answers. If missing, do NOT ask here; fold it into the Step 0 gate batch: *"No `AGENTS.md`. Pathfinder can chart without it. Continue, or run `plan2code-init` first?"* Record it in `## Ground rules` so no later session re-asks.
|
||||
|
||||
## Rules
|
||||
|
||||
- **Plan, don't do.** Every question resolves a DECISION. The pull to just build it is the edge of the map — hand off.
|
||||
- **Confirm before creating anything.** No files, no issues, until the Intent Gate (Step 0) and backend pick (Step 1) return.
|
||||
- **One question per session** (`research` excepted — parallel subagents).
|
||||
- **Refer by name.** "[Export format](<link>)", never "02" or "#42" in prose. Bare ids belong on `Blocked by:` lines and in commands.
|
||||
- **HITL questions are never self-answered.** Ask and wait. An agent that answers its own grill has broken the skill.
|
||||
- **Questions are ground truth; the map is a rebuildable index.** A filled `## Answer` beats any state marker; detail lives in one place.
|
||||
- **Never write implementation code** into the project. Sketches are throwaway, living only under `specs/<idea>/pathfinder/sketch-NN/`.
|
||||
- **Reserved names — never create inside `pathfinder/`:** `overview.md`, `phase-<N>.md`, `PLAN-DRAFT-*.md`, `PLAN-CONVERSATION-*.md`.
|
||||
- **Never emit the loop's completion tokens under `specs/`** — `TASK_COMPLETE`, `PHASE_COMPLETE`, `ALL_TASKS_COMPLETE`, `IMPLEMENTATION_COMPLETE`, `SPEC_COMPLETE`, `WORK_COMPLETE`. It scans for them.
|
||||
- **No `- [ ]` checkboxes inside question files**, and no `METRICS_JSON` anywhere. Pathfinder is not a metered step.
|
||||
|
||||
## Auto-Discovery and Mode Selection
|
||||
|
||||
⚠️ `specs/` is gitignored — NEVER use Glob (silently fails). Shell only: `ls specs/` (Bash) or `Get-ChildItem specs/` (PS).
|
||||
|
||||
**Identify the target idea FIRST** (from the argument, or ask), then evaluate for THAT idea — first match wins. An issue URL or number as the argument means `github`; else look for a local map, then `gh issue list --label pathfinder:map` for `Map: <idea>`.
|
||||
|
||||
| Condition | Route |
|
||||
|---|---|
|
||||
| No map, but `specs/<idea>/overview.md` exists | Documented — offer `/plan2code-3-implement`. STOP |
|
||||
| No map in either backend | MODE A, Step 0 (Intent Gate) |
|
||||
| Map `**Status:** Charting` | MODE A, resume at Step 6 |
|
||||
| Map `**Status:** Working` | MODE B |
|
||||
| Map `**Status:** Cleared` | Point at the PLAN-DRAFT and `/plan2code-1-plan`. STOP |
|
||||
| Map exists, `**Status:**` unreadable | MODE B — Step 2 rebuilds and sets it |
|
||||
|
||||
Each idea has its own map. Never chart two in one session.
|
||||
|
||||
## Questions
|
||||
|
||||
Read references/questions.md — the `local` format. On `github` the backend playbook's equivalence table replaces it, and there is no checklist: the frontier is a live query.
|
||||
|
||||
> Fallback (`local`): `map.md` indexes; `questions/NN-<slug>.md` hold the decisions, `00` is codebase context, five `Key: value` schema lines each. Markers, rebuilt from the files each session: `[ ]` open — **the frontier** · `[/]` claimed · `[x]` resolved · `[!]` blocked · `[-]` out of scope.
|
||||
|
||||
## MODE A: Chart
|
||||
|
||||
Read references/chart.md
|
||||
|
||||
> Fallback: confirm the outcome with the human FIRST; only then grill the destination, then breadth-first; write the map and one question per sharp decision.
|
||||
|
||||
0. `[Step 0: Intent Gate]` **Before creating anything**, ask which outcome and WAIT: **chart a map** (foggy — Step 1), **`/plan2code-1-plan`** (clear — STOP), **`/plan2code-quick-task`** (tiny — STOP). HITL, never self-select "chart".
|
||||
1. `[Step 1: Name and backend]` Only after the gate returns "chart." Confirm the kebab-case idea name, then ask — HITL, never self-picked — **local files or GitHub Issues?** Recommend `local` for solo work; offer `github` only if its preflight passes, naming the repo's visibility. THEN the first write.
|
||||
2. `[Step 2: Destination]` Grill until it is one or two lines. It fixes scope — settle it first.
|
||||
3. `[Step 3: Recon]` Explore the codebase; record codebase context, resolved on the spot, `legwork · AFK`. On `github` hold it until Step 6 so a Step 4 off-ramp leaves no litter.
|
||||
4. `[Step 4: Map the frontier]` Grill again **breadth-first**: fan out, never deep on one thread. Surface the open decisions and what is takeable now.
|
||||
5. `[Step 5: Create the map]` `**Status:** Charting`, Destination, Ground rules (backend first), an empty index, the fog in `## Not yet specified`. Say once where it lives and who can see it.
|
||||
6. `[Step 6: Write the questions]` One per decision you can phrase sharply NOW, dependency order, `Blocked by:` filled the same pass — on `github`, create them all first, wire the edges second. The rest stays fog. Always include a `grill · HITL` testing-posture question; `/plan2code-1-plan` Phase 1 needs it.
|
||||
7. `[Step 7: Index]` Fill `## Question Checklist` from the files (`local` only). Set `**Status:** Working`.
|
||||
8. `[Step 8: Fire research]` One subagent per `research` question, in parallel. Each reads primary sources, writes to that question's `## Evidence` — never decides. Then Session End.
|
||||
|
||||
**No fog at Step 4?** Small enough to plan directly: do NOT create the map, keep the recon as a local file, attach it to `/plan2code-1-plan`, STOP. Charting resolves nothing by hand — stop at Step 8.
|
||||
|
||||
## MODE B: Work
|
||||
|
||||
Read references/resolve.md
|
||||
|
||||
> Fallback: resolve by type — research reads sources, sketch makes something concrete, grill interviews the human, legwork does the manual work.
|
||||
|
||||
Assume NO memory of any prior session.
|
||||
|
||||
1. `[Step 1: Load]` Read the map whole. No question yet.
|
||||
2. `[Step 2: Reconcile]` **Always.** Read every question. `## Answer` written but the state disagrees? The answer wins. Claimed with no `## Answer`? A crash: release it, say so. Rebuild every marker from the questions.
|
||||
3. `[Step 3: Frontier]` Every question open, unclaimed, and unblocked. First in order.
|
||||
4. `[Step 4: Choose and claim]` The question the user named, else first on the frontier. Mark it claimed on the question and the map, **saved before any work.** Frontier empty but questions remain? All blocked — report the chain, STOP. Stranded on an `out-of-scope` blocker? Re-frame or rule out, re-run Step 3. Nothing open? Go to The Clearing Gate.
|
||||
5. `[Step 5: Zoom]` Read the claimed question in full, plus any closed question it references. Obey `## Ground rules`.
|
||||
6. `[Step 6: Resolve]` Route by type per the resolve playbook. HITL needs the human's own words.
|
||||
7. `[Step 7: Record]` Write `## Answer`: the decision, what was rejected and why, consequences, a one-line `**Gist:**`. Sources under `## Evidence`. Mark it resolved, index the gist on the map, bump `**Updated:**`.
|
||||
8. `[Step 8: Graduate]` Fog now sharp? Write those questions, delete the graduated bullets. Past the destination? Rule it out of scope, one line in `## Out of scope`. Invalidated? Re-frame or rule out.
|
||||
9. `[Step 9: Gate]` Run The Clearing Gate, then Session End.
|
||||
|
||||
## The Clearing Gate
|
||||
|
||||
Read references/handoff.md
|
||||
|
||||
> Fallback: write `specs/<idea>/PLAN-DRAFT-<YYYYMMDD>.md` from the map, Status `Phase 3 Complete - Resume at Phase 4`, then route to `/plan2code-1-plan`.
|
||||
|
||||
The map clears only when ALL hold:
|
||||
|
||||
1. Nothing open, claimed, or blocked
|
||||
2. `## Not yet specified` is EMPTY
|
||||
3. The destination is reachable with nothing left to decide
|
||||
4. Every confidence dimension (Requirements, Feasibility, Integration, Risk) scores ≥ 18/25
|
||||
|
||||
Any failing: name it, keep working. All passing: follow the handoff playbook, set `**Status:** Cleared`, stop. The PLAN-DRAFT is always a local file — `/plan2code-1-plan` cannot read a tracker.
|
||||
|
||||
## Trail Footer
|
||||
|
||||
Read references/trail.md
|
||||
|
||||
> Fallback: once the map exists, close every response with a one-line path of markers (`●` done · `◉` here · `○` open · `⊘` blocked · `⊝` out of scope) from `START` to `⚑`, a numbered legend of question names, plus a plain-English confidence note.
|
||||
|
||||
Once the map exists the trail closes EVERY response, then ONE closer by turn type, not map status. Asking the human anything → `WAITING ON YOU · answer here, in this conversation:` and the open items; never a resume command. Ending the session → `NEXT STEP · start a new conversation and run:` plus `/plan2code-0-pathfinder specs/<idea>/pathfinder` (the map issue URL on `github`), or `/plan2code-1-plan` once `Cleared`.
|
||||
|
||||
## Session End
|
||||
|
||||
Report the question resolved (by name), its gist, what graduated from the fog, what's still open. Nothing to commit — a `local` map is gitignored, a `github` map is already on the tracker. Then the mascot, then the Trail Footer.
|
||||
|
||||
```
|
||||
⋅
|
||||
╭───╮
|
||||
│ ★ │
|
||||
│ ◡ │ One more decision down. The fog is thinner!
|
||||
╰───╯
|
||||
```
|
||||
|
||||
**When the map cleared**, the mascot says `The way is clear! Time to plan!` and the footer routes to `/plan2code-1-plan` — or `/plan2code-init` FIRST if `## Ground rules` records `AGENTS.md` absent.
|
||||
|
||||
## Abort / Recovery
|
||||
|
||||
| Issue | Action |
|
||||
|---|---|
|
||||
| Session stops mid-question, or the map drifted | Release the claim, note why. Work Step 2 repairs the map; the questions always win. |
|
||||
| Frontier empty, fog remains | Not sharp yet. Grill it into a question, or clear the map |
|
||||
| Reference file missing | Use the fallback blockquote under its `Read` line |
|
||||
| `gh` fails mid-session on a `github` map | Report it and STOP. Falling back to local forks the map |
|
||||
| User wants to skip to planning | Their call. Say what is undecided, route to `/plan2code-1-plan` |
|
||||
|
||||
## Learning Capture
|
||||
|
||||
If charting surfaced project-specific insights, suggest `/plan2code-init-update` to capture them in `AGENTS.md`.
|
||||
@@ -0,0 +1,663 @@
|
||||
# Chart Playbook
|
||||
> Part of plan2code-0-pathfinder — loaded at the top of MODE A (Chart). Expands the numbered Chart steps.
|
||||
>
|
||||
> **Backend note.** Steps 0, 2, and 4 — the gates and the grills — are identical either way, and so is every judgement call below (fog vs question, in scope vs out, the destination test). What differs is where Steps 3 and 5-7 put the bytes: on `**Backend:** github`, `github-issues.md` overrides the `map.md` and question-file templates here, the single-pass rule under *Step 6: Numbering and dependency order*, and the timing of the recon. Read it alongside this file, not instead of it.
|
||||
|
||||
Charting produces a map and a set of question files. It resolves nothing by hand. Every judgement below serves one goal: put a sharp question on the map for everything you can phrase now, and leave everything else honestly in the fog.
|
||||
|
||||
---
|
||||
|
||||
## Step 0: The intent gate
|
||||
|
||||
Pathfinder builds an apparatus — a directory, a map, a file per decision. That apparatus earns its keep only when the way to the destination is genuinely foggy. For a small or already-clear ask it is pure overhead, and creating it before the human has agreed to it is the fastest way to make Pathfinder feel heavy and get in the way. So the gate runs **before the first byte hits disk**.
|
||||
|
||||
You already have the idea name from Auto-Discovery. Do NOT create the directory yet. Say, in substance:
|
||||
|
||||
> "This is Pathfinder. Nothing exists for `<idea>` yet. Pathfinder charts a map of the open decisions when an idea is big or unclear to plan — but that is overhead if this is small or already clear. Three ways to go:
|
||||
> - **Chart it** — I map the open decisions, one per session, then hand a draft to `/plan2code-1-plan`.
|
||||
> - **Straight to `/plan2code-1-plan`** — the way looks clear enough to plan now.
|
||||
> - **`/plan2code-quick-task`** — small enough to just do.
|
||||
>
|
||||
> My read: `<recommendation with a one-line reason>`. Which?"
|
||||
|
||||
Rules for the gate:
|
||||
|
||||
- **It is HITL.** You recommend; the human chooses. Never self-select "chart" and start creating files because it is the default path — that is exactly the failure this gate exists to stop.
|
||||
- **Read the request honestly.** A one-line bugfix, or a change with no open decisions, is not a charting job — recommend an off-ramp and mean it. Reserve "chart" for real fog: several unsettled decisions, unclear scope, or competing designs.
|
||||
- **No disk writes.** Naming the idea and talking is free. Creating `specs/<idea>/pathfinder/` is not — it waits for an explicit "chart."
|
||||
- **On an off-ramp, route and STOP.** Point at `/plan2code-1-plan` or `/plan2code-quick-task`, create nothing, end the session. If `AGENTS.md` is absent, mention `/plan2code-init` first, as with any handoff.
|
||||
|
||||
This gate and the Step 4 no-fog off-ramp are the same instinct at two moments: the gate is the human's call before any work; the off-ramp is your call once the breadth-first grill has proven there was no fog after all. Either one ending the session without a map is a success, not a failure.
|
||||
|
||||
---
|
||||
|
||||
## Step 2: The destination grill
|
||||
|
||||
The destination is settled **first** because it fixes scope. Every later judgement — is this a question or fog, is this in scope or past the edge, is this map cleared — is measured against it. A vague destination makes all three unanswerable, and you will spend the rest of the map arguing about boundaries instead of decisions.
|
||||
|
||||
A destination is **one or two lines** describing what exists when the map clears. It is not a feature description. It is not a value proposition. It names the artifact and draws the edge.
|
||||
|
||||
### The script
|
||||
|
||||
Recommend an answer with each probe — the human corrects faster than they compose. Follow the grilling playbook for tone, cadence, and the batching rules; this is the content.
|
||||
|
||||
**Six probes, two batches.** They do not all pass the independence test, so they split:
|
||||
|
||||
| Batch | Probes | Why they go together |
|
||||
|---|---|---|
|
||||
| First | 1 (artifact), 2 (person), 6 (forcing function) | Each stands alone. None reads differently under the others' answers. |
|
||||
| Second | 3 (sacrificial boundary), 4 (smallest arrival), 5 (arrival signal) | All three presuppose an artifact and an actor. Sending them before batch 1 lands asks the human to draw an edge around something unnamed. |
|
||||
|
||||
Two round trips, not six. Three probes each — exactly the cap, so neither batch needs splitting.
|
||||
|
||||
Batch 2 bends the independence test on purpose. The arrival signal (5) can shift under the smallest arrival (4), so strictly it should be held back — but holding it costs a third round trip to catch a conflict that is rare and cheap to spot. The trade is to send them together and reconcile at the recap: if the smallest arrival comes back materially smaller than the artifact you were told about, re-check the arrival signal against it before writing the destination. A knowing trade here, not a licence to batch dependent probes elsewhere.
|
||||
|
||||
**Both batches go out as numbered Q blocks — this grill is the other standing exception to the tool-first rule.** Probes 2, 3, 4, and 5 need the human's own phrasing — the destination is written into `map.md` verbatim as agreed, so a clicked option label is not something you can write down. That is the detail test's first row, four times over. Probe 6 names categories but the category is the worthless half of the answer: "deadline" changes nothing, "Q3 close, and the SEC audit lands Nov 1" changes the delivery question, the testing posture, and the out-of-scope line at once. Only probe 1 would survive a picker on its own, and it rides in a Q block anyway, because one tripping probe downgrades the whole batch. Do not reach for the structured question tool here.
|
||||
|
||||
**Probe 1 — the artifact**
|
||||
|
||||
> "When this map is cleared, what exists that does not exist now: a plan you hand to `/plan2code-1-plan`, a decision locked before anyone plans, or a change already made in the codebase? My guess: a plan."
|
||||
|
||||
*Fishing for:* the shape of the destination. Push back if the answer is "the feature working" — that is past the edge of every pathfinder map. Say so plainly: "That is the build. The map ends at the plan for the build."
|
||||
|
||||
**Probe 2 — the person on the other end**
|
||||
|
||||
> "Who uses the result, and what do they do with it the day it lands?"
|
||||
|
||||
*Fishing for:* the actor and the moment of use. Vague actors ("users", "the business") produce vague scope. Push until you get a role someone could name in an approval — compliance officer, on-call SRE, tenant admin.
|
||||
|
||||
**Probe 3 — the sacrificial boundary**
|
||||
|
||||
> "Name one thing a reasonable person would assume is part of this that you are willing to say is NOT part of it."
|
||||
|
||||
*Fishing for:* the first `## Out of scope` bullet. This probe does more work than any other. A destination nobody has excluded anything from has not been thought about. If the human cannot name one, offer two candidates and make them reject one.
|
||||
|
||||
**Probe 4 — the smallest arrival**
|
||||
|
||||
> "What is the smallest version that would still count as arriving? If only that existed, would you call it done or would you feel cheated?"
|
||||
|
||||
*Fishing for:* the difference between the destination and the wish list. Everything above the smallest arrival is a candidate for out of scope or for a later effort.
|
||||
|
||||
**Probe 5 — the arrival signal**
|
||||
|
||||
> "How do you know you have arrived — what do you look at?"
|
||||
|
||||
*Fishing for:* a checkable condition. "It feels right" is not one. "Every open decision has an answer and I can hand the draft to planning without re-litigating format" is one.
|
||||
|
||||
**Probe 6 — the forcing function**
|
||||
|
||||
> "What made this surface now? A deadline, an incident, an audit, a customer?"
|
||||
|
||||
*Fishing for:* constraints that will shape half the questions and that nobody volunteers unprompted. A regulatory deadline changes the delivery question, the testing posture question, and the out-of-scope line all at once.
|
||||
|
||||
**The probe names above are internal labels, not headings the human reads.** Head each Q block plainly — *What you end up with*, *Who uses it*, *What's not included*, *Smallest version that counts*, *How you know it's done*, *Why now* — and keep the probe text itself as plain as the quotes above. "The sacrificial boundary" and "the arrival signal" mean something to this playbook and nothing to the person answering. Full rule in the grilling playbook, *Say it in plain English*.
|
||||
|
||||
### Worked example — same idea, two destinations
|
||||
|
||||
**Idea:** "we should let people export audit logs"
|
||||
|
||||
**BAD destination**
|
||||
|
||||
> Let users export audit logs so they have their data.
|
||||
|
||||
Why it fails, concretely:
|
||||
|
||||
| Failure | Consequence downstream |
|
||||
|---|---|
|
||||
| No artifact named | Nobody knows whether the map clears at a plan or at shipped code |
|
||||
| "Users" is not a role | The authorization question cannot even be phrased |
|
||||
| No edge | Live streaming, SIEM push, and a schema redesign all argue their way in |
|
||||
| No arrival signal | The Clearing Gate has nothing to check against |
|
||||
| "their data" is a rationale, not a boundary | Every fog bullet reads as in scope |
|
||||
|
||||
**GOOD destination**
|
||||
|
||||
> A locked implementation plan for a compliance officer to export a filtered range of audit events from the admin UI and receive them as a single downloadable file. The map ends at the plan, not at shipped code. Continuous streaming to external systems is not on the route.
|
||||
|
||||
Three sentences, two lines of substance: artifact (a plan), actor (compliance officer), trigger surface (admin UI), shape of the result (one downloadable file), and an explicit edge (no streaming). Every one of those clauses will be cited later when you decide whether something is a question, fog, or out of scope.
|
||||
|
||||
**Write the destination into `map.md` verbatim as agreed.** Do not improve it afterwards. If it needs changing, change it with the human present — a silently redrawn destination invalidates every scope call already made.
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Codebase recon
|
||||
|
||||
Recon is `legwork · AFK` — you do it alone, and you write it down **already resolved**. It exists so that no later session re-reads the same directories, and so that the handoff carries the ground truth `/plan2code-1-plan` Phase 2 (System Context Examination) would otherwise have to rediscover.
|
||||
|
||||
`questions/00-codebase-context.md` is always `00`. It always exists. It is created with `State: resolved` and a filled `## Answer` in the same write.
|
||||
|
||||
### What to explore
|
||||
|
||||
| Area | What to establish | Where to look |
|
||||
|---|---|---|
|
||||
| **Directory structure** | The map of the repo at the depth that matters for this destination — not every folder, the ones the work will touch | Top-level listing, then two levels into the relevant subtrees |
|
||||
| **Key components** | The modules that would be read, changed, or called. Name, path, responsibility | Entry points, route/controller registries, service layers |
|
||||
| **Patterns and conventions** | How this codebase does the thing you are about to plan: error handling, validation, config, module layout, naming, async style | Two or three recent files in the target area, plus `AGENTS.md` |
|
||||
| **Integration points** | External systems, queues, storage, auth providers, feature-flag services the work will cross | Config files, environment variable references, client wrappers |
|
||||
| **Technical debt in the blast radius** | Only debt the destination would collide with. Not a repo-wide audit | Long files in the target area, duplicated helpers, stale TODO markers with no owner |
|
||||
| **System boundaries** | What this effort owns versus what it merely calls. Where the change stops | Package boundaries, ownership files, API contracts |
|
||||
|
||||
Two disciplines keep this file useful:
|
||||
|
||||
- **Verify behaviour against actual code, never against a filename.** A file called `auditLogger.ts` may log nothing.
|
||||
- **Scope the recon to the destination.** A recon of the whole repo is unreadable and stale in a week. If a subtree cannot plausibly be touched by the destination, say so in one line and move on.
|
||||
|
||||
Record what you could **not** determine as an explicit gap. Gaps at recon time are often the first real fog bullets.
|
||||
|
||||
### The literal file
|
||||
|
||||
```markdown
|
||||
> Pathfinder planning note - decisions, not implementation work. Archive with the spec; do not delete.
|
||||
|
||||
# Codebase context
|
||||
|
||||
Type: legwork · AFK
|
||||
State: resolved
|
||||
Blocked by: none
|
||||
Claimed: 2026-08-03 09:12
|
||||
Locked: no
|
||||
|
||||
## Question
|
||||
|
||||
What does this codebase already provide, constrain, and forbid for an operator-initiated
|
||||
audit-log export? Establish structure, components, conventions, integrations, debt in the
|
||||
blast radius, and boundaries — enough that no later session re-reads the same ground and
|
||||
enough to hand to `/plan2code-1-plan` as its system context.
|
||||
|
||||
## Answer
|
||||
|
||||
### Directory structure
|
||||
|
||||
- `src/api/` — Express routers, one file per resource. `src/api/admin/` is the admin surface.
|
||||
- `src/services/` — business logic; the only layer allowed to touch `src/db/`.
|
||||
- `src/db/` — Knex query builders and migrations. `audit_events` lives here.
|
||||
- `src/jobs/` — BullMQ workers. Existing precedent for long-running work.
|
||||
- `src/web/admin/` — React admin UI, TanStack Query, colocated route components.
|
||||
- `test/` — Vitest, mirroring `src/` one-to-one.
|
||||
|
||||
Untouched by this destination: `src/billing/`, `src/web/marketing/`.
|
||||
|
||||
### Key components
|
||||
|
||||
| Component | Path | Responsibility |
|
||||
|---|---|---|
|
||||
| `auditEvents.record()` | `src/services/auditEvents.ts` | Sole writer of `audit_events`; called from 31 sites |
|
||||
| `adminRouter` | `src/api/admin/index.ts` | Mounts admin routes; applies `requireAdmin` |
|
||||
| `requireAdmin` | `src/api/middleware/auth.ts` | Session check plus role check; no per-tenant scoping today |
|
||||
| `reportQueue` | `src/jobs/reportQueue.ts` | BullMQ queue used by the existing billing report export |
|
||||
| `signedUrl()` | `src/services/storage.ts` | S3 pre-signed URL helper, fixed 15-minute expiry |
|
||||
|
||||
### Patterns and conventions
|
||||
|
||||
- Errors: typed error classes thrown from services, mapped to HTTP by `errorHandler`. Never raw `res.status(500)`.
|
||||
- Validation: Zod schema per route, exported next to the handler.
|
||||
- Config: everything through `src/config.ts`; no direct `process.env` reads outside it.
|
||||
- Async: `async`/`await` throughout. No callback style remains.
|
||||
- Long-running work: enqueue to BullMQ, return `202` with a job id. Established by billing reports.
|
||||
- Tests: Vitest, colocated fixtures, no shared mutable state between cases.
|
||||
|
||||
### Integration points
|
||||
|
||||
- **Postgres 15** via Knex. `audit_events` is ~180M rows, partitioned monthly.
|
||||
- **Redis** backing BullMQ.
|
||||
- **S3** for generated artifacts; the billing export already writes there.
|
||||
- **SES** for transactional mail; templates in `src/mail/templates/`.
|
||||
- No SIEM, log-shipping, or streaming integration exists today.
|
||||
|
||||
### Technical debt in the blast radius
|
||||
|
||||
- `audit_events` has an index on `(tenant_id, created_at)` but none on `actor_id`. Any
|
||||
actor-filtered export will sequential-scan a partition.
|
||||
- `requireAdmin` does not scope by tenant — a platform admin currently sees all tenants.
|
||||
Any authorization decision here inherits that gap.
|
||||
- The billing export writes CSV by hand-rolled string concatenation with no escaping.
|
||||
Do not copy it.
|
||||
|
||||
### System boundaries
|
||||
|
||||
Owned by this effort: a read path over `audit_events`, an admin UI surface, an artifact
|
||||
written to S3, and a delivery notification. Not owned: the write path (`auditEvents.record()`
|
||||
is untouched), the audit event schema, tenancy semantics in `requireAdmin`.
|
||||
|
||||
### Gaps
|
||||
|
||||
- Retention policy for `audit_events` is not expressed anywhere in code. Someone outside
|
||||
engineering owns it.
|
||||
- No load figures exist for the largest tenant's monthly event count.
|
||||
|
||||
**Gist:** Node/Express/Knex/React with an established BullMQ-to-S3 export precedent from
|
||||
billing; `audit_events` is 180M rows partitioned monthly with no `actor_id` index, and
|
||||
`requireAdmin` has no per-tenant scoping.
|
||||
|
||||
## Evidence
|
||||
|
||||
- `src/api/admin/index.ts`, `src/api/middleware/auth.ts`
|
||||
- `src/services/auditEvents.ts`, `src/services/storage.ts`
|
||||
- `src/jobs/reportQueue.ts` and the billing export job it drives
|
||||
- `src/db/migrations/20240914_partition_audit_events.js`
|
||||
- `AGENTS.md` (conventions section)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4: The breadth-first frontier grill
|
||||
|
||||
The destination grill went **deep on one thing**. This grill goes **wide on everything**. They are different activities and mixing them is the most common way to produce a bad map.
|
||||
|
||||
| | Destination grill (Step 2) | Frontier grill (Step 4) |
|
||||
|---|---|---|
|
||||
| Goal | One or two settled lines | An inventory of open decisions |
|
||||
| Movement | Drill until it is precise | Fan out until you stop finding new areas |
|
||||
| Follow-ups | Chase every hedge | One clarifier at most, then move on |
|
||||
| Success | The human commits to a boundary | You can name the areas, sharp and unsharp alike |
|
||||
| Failure mode | Accepting a slogan | Solving a question instead of finding the next one |
|
||||
|
||||
**You are not resolving anything here.** You are taking inventory. The instant an answer feels satisfying, you are probably going deep.
|
||||
|
||||
### Recognising that you have gone deep
|
||||
|
||||
Watch for these. Any one of them means stop and pull back:
|
||||
|
||||
- You have asked three consecutive probes about the same area.
|
||||
- You are discussing an implementation detail (a column type, a library, a retry count) rather than a decision.
|
||||
- You are proposing a design instead of asking what has to be decided.
|
||||
- The human is enjoying it. Depth is more fun than breadth; that is exactly why it steals the session.
|
||||
- You have written something down that reads like an answer.
|
||||
|
||||
### Pulling back
|
||||
|
||||
Say it out loud so the human tracks the move, then jump:
|
||||
|
||||
> "Good — that is one for the map, not for now. Parking it as *Export format*. Different corner: who is allowed to run an export at all?"
|
||||
|
||||
Two mechanics keep the fan-out honest:
|
||||
|
||||
1. **Round-robin the areas.** Before you start, list the axes you intend to cross: data, surface, permissions, volume, delivery, failure, operations, testing. Take one probe per axis before any second probe on any axis.
|
||||
2. **Ask for the axis you have not touched.** Near the end: "What have I not asked about that would embarrass us to discover in week three?"
|
||||
|
||||
**Breadth-first is the ideal batch.** One probe per axis means the probes are independent by construction — that is what breadth-first *means* — so this grill should run as batches of three, not as a stream of singles. Seven axes is three turns. If you catch yourself wanting to batch two probes on the same axis, that is depth wearing a batch's clothes; pull back.
|
||||
|
||||
### Sample breadth probes
|
||||
|
||||
Each opens a different axis. Send 1-3 as one batch and 4-6 as the next, then probe 7 alongside the "what have I not asked about" closer above; note each answer and move.
|
||||
|
||||
**They go out as numbered Q blocks — this grill is one of the two standing exceptions to the tool-first rule.** Several of the probes do name alternatives, so they would pass the detail test on its own terms, and that is exactly the trap: the output of this grill is not a decision, it is a *sort* into sharp question or fog, and sorting takes the elaboration around the answer. A clicked label leaves you nothing to sort with. The structured question tool earns its keep in MODE B, where a claimed question already has named alternatives and the sorting is long done.
|
||||
|
||||
1. **Data** — "What is the smallest and largest thing an operator could reasonably ask for in one export? Give me both ends."
|
||||
2. **Surface** — "Where does this start: a button in the admin UI, a scheduled thing, an API call someone scripts?"
|
||||
3. **Permissions** — "Who is allowed to run one, and can they export events about people other than themselves?"
|
||||
4. **Volume and time** — "If an export takes four minutes, is that fine, bad, or a redesign?"
|
||||
5. **Delivery and failure** — "The export succeeds but the download link expires before they click it. What should have happened?"
|
||||
6. **Operations** — "Six months from now someone asks who exported what. Does this feature audit itself?"
|
||||
7. **Testing** — "What would you need to see pass before you would let this near a customer's compliance data?" *(This one always runs — see the mandatory testing-posture question below.)*
|
||||
|
||||
Record each answer as one line in your working notes with an area label. At the end of the grill you will have two piles: lines you can turn into a sharp question, and lines you cannot. The second pile is the fog.
|
||||
|
||||
---
|
||||
|
||||
## The fog-vs-question test
|
||||
|
||||
This is the single most important judgement in the skill. Get it wrong toward questions and the map fills with unanswerable stubs that block the frontier. Get it wrong toward fog and the map has nothing takeable on it.
|
||||
|
||||
> **The test is whether you can STATE the question precisely now — not whether you can ANSWER it now.**
|
||||
|
||||
- **Write a question file** when the question is already sharp — *even if it is blocked and nobody can act on it yet*. Blocked questions are real questions; they get `Blocked by:` and a `[!]` row and they wait. Blocked is not the same as unformed.
|
||||
- **Leave it in `## Not yet specified`** when you cannot yet phrase it that sharply. You can see there is something there; you cannot say what is being asked.
|
||||
|
||||
**Do not pre-slice the fog.** A fog patch is deliberately coarser than a question. One patch may graduate into three questions, or one, or none once the frontier reaches it. Splitting fog into question-shaped fragments before it is sharp invents a structure that the answers will contradict, and it costs a later session the work of deleting your guesses. Write the patch as loosely as the view allows.
|
||||
|
||||
A useful forcing check: **could a different person, reading only this line, know what a good answer looks like?** If yes, it is a question. If they would have to ask you what you meant, it is fog.
|
||||
|
||||
### Worked examples
|
||||
|
||||
| Candidate | Verdict | Reasoning |
|
||||
|---|---|---|
|
||||
| "CSV or JSONL for the export file?" | **Question** | Two named options, one decision, one sitting. A reader knows what an answer looks like. `grill · HITL`. |
|
||||
| "Which roles may export events about other users?" | **Question**, blocked | Sharp today, but it depends on the tenant-scoping decision. Write it, set `Blocked by:`, mark the row `[!]`. Blockedness never demotes a sharp question to fog. |
|
||||
| "Something about how big exports behave" | **Fog** | "Big" has no meaning until the volume ceiling lands. You cannot say whether the question is about pagination, streaming, timeouts, or refusal. One line in `## Not yet specified`. |
|
||||
| "There is probably something about PII redaction" | **Fog** | The area is visible, the question is not. Once Legal answers, this may graduate into *which fields are redacted*, *who configures it*, and *does redaction apply to the actor or the subject* — or into nothing, if the answer is "export raw." Slicing it now guesses all three. |
|
||||
| "How should we architect the export pipeline?" | **Neither — split it** | No single answer closes it; it bundles at least four decisions (sync vs queued, storage target, artifact lifetime, notification). Ask what the parts are. The sharp parts become questions, the rest becomes fog. A candidate no single answer closes is not a question. |
|
||||
|
||||
**What never belongs in `## Not yet specified`:** anything already decided (it is a resolved question with a gist on its row), anything that already has a question file, and anything past the destination (that is out of scope).
|
||||
|
||||
---
|
||||
|
||||
## Out of scope versus fog
|
||||
|
||||
Fog gathers **only toward the destination**. The destination fixes the scope, so work beyond it is not dim — it is *excluded*. It is not fog, and it must never sit in `## Not yet specified`, where a later session would try to graduate it.
|
||||
|
||||
**The distinction is SCOPE, not sharpness.** This is the part people get wrong. An out-of-scope item can be perfectly sharp — "should the export push to Splunk on a schedule?" is a crisp question with a crisp answer. It is still out of scope, because the destination said the map ends at an operator-initiated export. Sharpness decides *fog versus question*. Position relative to the destination decides *in scope versus out*.
|
||||
|
||||
| | Fog (`## Not yet specified`) | Out of scope (`## Out of scope`) |
|
||||
|---|---|---|
|
||||
| Position | Before the destination | Past the destination |
|
||||
| Why it is not a question | Cannot be phrased sharply yet | Could be phrased perfectly — it just is not ours |
|
||||
| Future | Graduates into questions as the frontier advances | Never graduates |
|
||||
| Reopening | Automatic, as answers land | Only if the destination is redrawn — and then as a fresh effort, not a resumption |
|
||||
| The act | An admission of ignorance | A scoping decision |
|
||||
|
||||
Ruling something out of scope is a **scoping act, not a step on the route**. When a question you already created turns out to sit past the destination — mis-scoped in during charting, or exposed by a later answer — set `State: out-of-scope`, mark its row `[-]`, and leave one line in `## Out of scope` giving the gist and the reason, linking the question by name. It does not get an `## Answer` and it is not a decision the route walked.
|
||||
|
||||
Watch for **stranded** questions: a live question whose `Blocked by:` names something now out of scope will never unblock. Re-frame its `## Question` to drop the dependency, or rule it out too. Never leave it sitting.
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Numbering and dependency order
|
||||
|
||||
Upstream wayfinder creates every unit first and wires the blocking edges in a **second pass**, because a server-side tracker assigns ids and nothing can reference a sibling until it has one. On `**Backend:** local` that constraint does not exist — **you choose `NN` yourself**, so charting is a **single pass**: decide the order, then write each file complete, `Blocked by:` filled at the moment of writing.
|
||||
|
||||
(On `**Backend:** github` the constraint comes back, and so does the two-pass shape. Rules 1, 6, 7, and 8 below still hold — they are about dependency reasoning, not about ids. Rules 2-5, which are about `NN`, are replaced by sub-issue order; see `github-issues.md`.)
|
||||
|
||||
The rules:
|
||||
|
||||
1. **Sort by dependency before you write anything.** Sketch the edges on paper first: which decisions must land before which others can even be discussed.
|
||||
2. **Blockers get lower numbers.** If *Row-count ceiling* blocks *Delivery channel*, the ceiling is `02` and delivery is `05`. This makes `Blocked by: 02` readable at a glance and makes "lowest `NN` first" on the frontier a sane traversal order.
|
||||
3. **`00` is always the codebase context.** Never anything else.
|
||||
4. **`NN` is never reused and never renumbered.** Not when a question is ruled out of scope, not when one is deleted, not to close a gap in the sequence. Links and `Blocked by:` lines would rot silently. Gaps in the numbering are normal and harmless.
|
||||
5. **The next number is max + 1**, computed from the directory listing, not from the map.
|
||||
6. **Only depend on what genuinely gates the question.** A `Blocked by:` chain that is really a preference for reading order strangles the frontier. Ask: could this question be answered — badly but honestly — without the blocker? If yes, it is not blocked.
|
||||
7. **Cycles are a phrasing bug.** If A blocks B and B blocks A, the two are one decision. Merge them or re-frame one to drop the edge.
|
||||
8. **Refer by name in prose.** Bare numbers appear only on `Blocked by:` lines.
|
||||
|
||||
A worked ordering for the audit-log export map:
|
||||
|
||||
| `NN` | Name | Type | Blocked by | Why here |
|
||||
|---|---|---|---|---|
|
||||
| `00` | Codebase context | `legwork · AFK` | none | Always first, always resolved |
|
||||
| `01` | Export format | `grill · HITL` | none | Nothing gates it; it gates the artifact shape |
|
||||
| `02` | Row-count ceiling | `research · AFK` | none | A fact about the data, independent of every preference |
|
||||
| `03` | Export authorization | `grill · HITL` | none | Independent axis; can be argued today |
|
||||
| `04` | Testing posture | `grill · HITL` | none | Mandatory; independent of everything else |
|
||||
| `05` | Delivery channel | `grill · HITL` | `02` | Synchronous download versus queued link turns entirely on volume |
|
||||
| `06` | SIEM push connector | `grill · HITL` | none | Created, then immediately ruled out of scope during charting — `State: out-of-scope`, no `## Answer` |
|
||||
|
||||
---
|
||||
|
||||
## The `map.md` template
|
||||
|
||||
Below is a complete, realistic map for the audit-log export effort **partway through MODE B**, after two questions have resolved — it shows every marker in use. At Chart Step 5 the same file carries `**Status:** Charting`, an empty `## Question Checklist`, and no `[x]` row except `00`. Copy the structure exactly, including the HTML comments — they are written **for the next session**, which has no memory of this one.
|
||||
|
||||
`**Status:**` is `Charting` while Step 5 and Step 6 run, becomes `Working` at Step 7, and becomes `Cleared` only at handoff. It is how the orchestrator routes a fresh session, so it must be accurate before you stop.
|
||||
|
||||
Set `**Confidence:**` honestly at Step 5 and re-score it every time a question resolves. Charting scores are low by construction — that is the point. The Clearing Gate needs every dimension at 18/25 or better. Never inflate to make the gate pass.
|
||||
|
||||
Write the four dimension labels **hyphenated exactly as shown** — `Requirements-clarity`, `Feasibility-technical`, `Integration-points`, `Risk-assessment` — in the `NN/25` form, with no percent symbol anywhere in the file. The metrics collector scrapes planning documents by regex for a bare dimension word followed by whitespace, a colon, or a pipe and then digits; the hyphen breaks that match. A percent sign or a bare `Requirements 18` would be ingested as a completed planning step's confidence score that no planning step ever produced.
|
||||
|
||||
Say once, at Step 5: *"This map lives in gitignored `specs/` — local to you, not shared. `git add -f` it to track it."*
|
||||
|
||||
```markdown
|
||||
> Pathfinder planning note - decisions, not implementation work. Archive with the spec; do not delete.
|
||||
|
||||
# Map: audit-log-export
|
||||
|
||||
**Status:** Working
|
||||
**Updated:** 2026-08-03
|
||||
**Confidence:** Requirements-clarity 18/25 · Feasibility-technical 14/25 · Integration-points 16/25 · Risk-assessment 14/25
|
||||
|
||||
<!-- Status: Charting while the map is being built (Chart Steps 1-6) -> Working once the
|
||||
checklist is indexed (Chart Step 7) -> Cleared only when the Clearing Gate passes.
|
||||
A fresh session routes on this line, so it must be correct before the session ends. -->
|
||||
|
||||
<!-- Confidence: four dimensions, each scored out of 25, re-scored at every resolution.
|
||||
The Clearing Gate requires all four at 18/25 or better. Score against evidence. -->
|
||||
|
||||
## Destination
|
||||
|
||||
A locked implementation plan for a compliance officer to export a filtered range of audit
|
||||
events from the admin UI and receive them as a single downloadable file. The map ends at the
|
||||
plan, not at shipped code. Continuous streaming to external systems is not on the route.
|
||||
|
||||
<!-- Settled at Chart Step 2 and quoted as agreed. Every question is measured against it:
|
||||
in scope or past the edge, still needed or now moot. Change it only with the human
|
||||
present — a silent redraw invalidates every scope call already made. -->
|
||||
|
||||
## Ground rules
|
||||
|
||||
<!-- Standing constraints for every session on this map. Read before choosing a question,
|
||||
obeyed while resolving it. Nothing here is re-asked. -->
|
||||
|
||||
- `AGENTS.md` exists and governs. Its conventions are not re-litigated by any question here.
|
||||
- One question _file_ per session. `research` questions may run as parallel subagents.
|
||||
- Grill probes are batched per the grilling playbook — at most three per turn, through the structured question tool unless the detail test forces prose Q blocks.
|
||||
- Questions are put to the human in plain English. Technical terms only where the term is the decision.
|
||||
- HITL questions are answered by the human in their own words. Never self-answered.
|
||||
- No new runtime dependency is assumed without a `research` question backing it.
|
||||
- Compliance language is reviewed by Dana before anything user-facing is finalised.
|
||||
- Sketches are throwaway and live only under `pathfinder/sketch-NN/`.
|
||||
|
||||
## Glossary
|
||||
|
||||
<!-- Terms this effort uses precisely. Prevents two sessions meaning different things by
|
||||
the same word — the cheapest correctness win on the whole map. -->
|
||||
|
||||
| Term | Meaning here | Avoid |
|
||||
|---|---|---|
|
||||
| Audit event | One row in `audit_events`: actor, tenant, action, target, timestamp, payload | log line, activity record |
|
||||
| Compliance officer | Tenant-scoped role that reviews activity; not a platform admin | admin, auditor |
|
||||
| Export | One operator-initiated request producing one artifact for one filtered range | download, dump, extract |
|
||||
| Retention window | How far back `audit_events` is queryable; owned outside engineering | archive period |
|
||||
| Signed artifact | The generated file plus a checksum a recipient can verify independently | signed file, bundle |
|
||||
|
||||
## Question Checklist
|
||||
|
||||
<!-- Rebuilt from questions/ every session — the files are ground truth, this is an index.
|
||||
[ ] open (the frontier) · [/] claimed · [x] resolved · [!] open but blocked
|
||||
[-] out of scope. Resolved rows carry the one-line gist from the question's Answer. -->
|
||||
|
||||
- [x] [Codebase context](./questions/00-codebase-context.md) — Node/Express/Knex/React with a BullMQ-to-S3 export precedent; `audit_events` is 180M rows partitioned monthly, no `actor_id` index, `requireAdmin` has no tenant scoping.
|
||||
- [x] [Export format](./questions/01-export-format.md) — CSV with a UTF-8 BOM and RFC 4180 quoting, plus a sidecar SHA-256 manifest; JSONL rejected because recipients open these in Excel.
|
||||
- [/] [Row-count ceiling](./questions/02-row-count-ceiling.md)
|
||||
- [ ] [Export authorization](./questions/03-export-authorization.md)
|
||||
- [ ] [Testing posture](./questions/04-testing-posture.md)
|
||||
- [!] [Delivery channel](./questions/05-delivery-channel.md) — Blocked by 02
|
||||
- [-] [SIEM push connector](./questions/06-siem-push-connector.md) — out of scope, see below
|
||||
|
||||
## Not yet specified
|
||||
|
||||
<!-- The fog: in-scope areas you can see but cannot yet phrase as a question. Graduates into
|
||||
question files as answers land, and the graduated bullet is deleted from here.
|
||||
Do NOT pre-slice these into question-sized pieces — one bullet may become three
|
||||
questions, or none. Nothing already decided, already a question, or out of scope. -->
|
||||
|
||||
- How far back an export may reach. There is a retention answer somewhere outside engineering
|
||||
and nobody has it yet; until then we cannot say whether the question is about a hard limit,
|
||||
a warning, or a per-tenant setting.
|
||||
- Redaction of event payloads. Legal may say "export raw", in which case this evaporates —
|
||||
or it may become several decisions about which fields, who configures them, and whether the
|
||||
rule follows the actor or the subject. Revisit after Export authorization.
|
||||
- What happens when an export range straddles a monthly partition that was migrated mid-range.
|
||||
Cannot phrase this sharply until Export format is applied to a real query plan.
|
||||
- Whether the export feature audits itself, and if so at what granularity. Suspect this is one
|
||||
small question but it may turn on the authorization model.
|
||||
|
||||
## Out of scope
|
||||
|
||||
<!-- Work consciously ruled past the destination. Never graduates; returns only if the
|
||||
destination is redrawn, and then as a fresh effort. One line each: gist plus why. -->
|
||||
|
||||
- [SIEM push connector](./questions/06-siem-push-connector.md) — scheduled push to Splunk or
|
||||
similar. The destination ends at an operator-initiated export; anything continuous is a
|
||||
different effort with a different owner.
|
||||
- Redesign of the `audit_events` schema. The write path is untouched by this destination;
|
||||
changing it would pull in all 31 call sites of `auditEvents.record()`.
|
||||
- Adding the missing `actor_id` index. Real, and it will hurt, but it is a database change
|
||||
with its own review path. Recorded here so the plan can reference it as a dependency
|
||||
rather than absorb it.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## The question-file template
|
||||
|
||||
Five contiguous `Key: value` lines after the H1. Not YAML. No frontmatter delimiters. No `- [ ]` checkboxes anywhere inside a question file — use plain bullets, including for legwork checklists.
|
||||
|
||||
`## Question` is written at charting. `## Answer` is appended only when the question resolves. `## Evidence` holds sources, links, and artifacts, and a `research` subagent writes into it during Chart Step 8 without deciding anything.
|
||||
|
||||
### An open question
|
||||
|
||||
```markdown
|
||||
> Pathfinder planning note - decisions, not implementation work. Archive with the spec; do not delete.
|
||||
|
||||
# Export authorization
|
||||
|
||||
Type: grill · HITL
|
||||
State: open
|
||||
Blocked by: none
|
||||
Claimed: none
|
||||
Locked: yes
|
||||
|
||||
## Question
|
||||
|
||||
Who may run an audit-log export, and over whose events?
|
||||
|
||||
Three sub-decisions, all of which must land together because any two of them constrain the third:
|
||||
|
||||
- Which role gates the export action — the existing `admin` role, a new `compliance` role, or
|
||||
a per-tenant grant?
|
||||
- May an exporter include events where they are the actor, or must self-events be excluded to
|
||||
keep the export usable as evidence?
|
||||
- `requireAdmin` currently does not scope by tenant, so a platform admin sees every tenant's
|
||||
events. Does the export inherit that, or does it enforce a tenant scope the rest of the
|
||||
admin surface does not?
|
||||
|
||||
Recommended answer to react to: a new tenant-scoped `compliance` role; self-events included
|
||||
but flagged in a column; the export enforces tenant scope even though the surrounding admin
|
||||
surface does not.
|
||||
|
||||
Marked `Locked: yes` — the third sub-decision creates a precedent that later admin features
|
||||
will follow, and reversing it later means re-auditing every export already delivered.
|
||||
|
||||
## Evidence
|
||||
|
||||
- `src/api/middleware/auth.ts` — `requireAdmin` checks session and role, no tenant predicate.
|
||||
- Codebase context records the same gap under technical debt.
|
||||
```
|
||||
|
||||
### A resolved question
|
||||
|
||||
```markdown
|
||||
> Pathfinder planning note - decisions, not implementation work. Archive with the spec; do not delete.
|
||||
|
||||
# Export format
|
||||
|
||||
Type: grill · HITL
|
||||
State: resolved
|
||||
Blocked by: none
|
||||
Claimed: 2026-08-03 10:41
|
||||
Locked: yes
|
||||
|
||||
## Question
|
||||
|
||||
What file format does an export produce, and what does a recipient need in order to trust the
|
||||
file has not been altered?
|
||||
|
||||
## Answer
|
||||
|
||||
**Decision.** CSV, UTF-8 with a byte-order mark, RFC 4180 quoting, one header row, timestamps
|
||||
in ISO 8601 UTC. Alongside it a sidecar `.sha256` manifest listing the artifact filename and
|
||||
its digest.
|
||||
|
||||
**Rejected.**
|
||||
|
||||
- *JSONL* — better for nested payloads and trivially streamable, but every named recipient
|
||||
opens these in Excel and would need a conversion step during an audit. The people who
|
||||
prefer JSONL are not the people receiving the file.
|
||||
- *XLSX* — solves the Excel encoding problems outright, but adds a generation library and
|
||||
makes byte-level verification of the artifact meaningfully harder.
|
||||
- *Detached signature instead of a checksum* — real integrity guarantees, but requires key
|
||||
management nobody has scoped, and no recipient has asked to verify a signature.
|
||||
|
||||
**Consequences.**
|
||||
|
||||
- Nested `payload` is flattened to one JSON string column. Anyone needing structure parses
|
||||
that column.
|
||||
- The BOM is required or Excel mangles non-ASCII actor names. This must be an explicit test.
|
||||
- Do not reuse the billing export's CSV writer — it concatenates strings with no escaping.
|
||||
A quoting-correct writer is now in scope for the plan.
|
||||
- The checksum makes the artifact self-verifying, which lets Delivery channel consider a
|
||||
short-lived link without weakening the integrity story.
|
||||
|
||||
**Gist:** CSV with a UTF-8 BOM and RFC 4180 quoting, plus a sidecar SHA-256 manifest; JSONL
|
||||
rejected because recipients open these in Excel.
|
||||
|
||||
## Evidence
|
||||
|
||||
- RFC 4180, sections 2.5-2.7 (quoting and embedded delimiters).
|
||||
- `src/jobs/billingExport.ts` — the hand-rolled writer that must not be copied.
|
||||
- Dana confirmed on 2026-08-03 that external auditors accept a published checksum.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4: The no-fog off-ramp
|
||||
|
||||
If the breadth-first grill surfaces **no fog** — every area you opened produced either a settled answer or a question you could phrase immediately, and `## Not yet specified` would be empty — then the way to the destination is already visible. The journey is small enough to plan directly and a map would be pure overhead.
|
||||
|
||||
Do this, in order:
|
||||
|
||||
1. **Keep `questions/00-codebase-context.md`.** This is the important part. It is exactly what `/plan2code-1-plan` Phase 2 (System Context Examination) has to produce anyway, and you have already produced it. Throwing the recon away to "clean up" wastes the most valuable artifact of the session.
|
||||
2. **Do not create `map.md`.** The off-ramp fires at Step 4 and the map is not written until Step 5, so in the normal flow there is nothing to delete — just stop before writing it. (If you reached Step 4 with a `map.md` already on disk, delete it: a map with empty fog and no open questions will confuse the next session into resuming something that does not exist.)
|
||||
3. **Tell the user plainly what happened and what to do:**
|
||||
|
||||
> "No fog surfaced — the way to the destination is already visible, so this does not need a map. I kept the codebase recon at `specs/audit-log-export/pathfinder/questions/00-codebase-context.md`; attach that file to a `/plan2code-1-plan` session and it covers Phase 2 outright."
|
||||
|
||||
4. **Stop.** Do not chart anyway "just in case", do not create questions, do not start planning in this session.
|
||||
|
||||
Be honest about the trigger. If two areas are genuinely unformed, that is fog and the map earns its keep. The off-ramp is for the case where you fanned out across every axis and kept landing on solid ground.
|
||||
|
||||
---
|
||||
|
||||
## Step 6: The mandatory testing-posture question
|
||||
|
||||
**Every map includes a `grill · HITL` question on testing posture.** No exceptions, including maps where testing feels obvious.
|
||||
|
||||
The reason is mechanical: `/plan2code-1-plan` Phase 1 asks for three things — testing types, whether tests run after each implementation phase, and a coverage target — and `/plan2code-2-document` **string-matches** that answer, either appending a testing block to every phase, creating a dedicated final testing phase, or omitting testing entirely. A map that clears without this answer hands the human a planning session that stalls on its first question. Ask it while there is still someone in the room.
|
||||
|
||||
Record the answer in the literals downstream matches on, not in paraphrase. ("Phase" here names `/plan2code-1-plan`'s implementation phases and is a downstream contract string — it is never a pathfinder unit of work.)
|
||||
|
||||
It is almost never blocked. Give it whatever number the dependency ordering leaves free, and expect it to sit on the frontier from day one.
|
||||
|
||||
```markdown
|
||||
> Pathfinder planning note - decisions, not implementation work. Archive with the spec; do not delete.
|
||||
|
||||
# Testing posture
|
||||
|
||||
Type: grill · HITL
|
||||
State: open
|
||||
Blocked by: none
|
||||
Claimed: none
|
||||
Locked: no
|
||||
|
||||
## Question
|
||||
|
||||
What testing does this work carry, so `/plan2code-1-plan` Phase 1 can be answered without
|
||||
stalling? Three parts, all needed:
|
||||
|
||||
- **Types** — unit, integration, end-to-end, some combination, or none.
|
||||
- **Phase testing** — record one of the two literals `/plan2code-2-document` matches:
|
||||
`Run after each phase` (a testing block closes every implementation phase) or
|
||||
`Dedicated phase only` (one final testing phase).
|
||||
- **Coverage target** — record one of Phase 1's three literals: `Critical paths`,
|
||||
`Moderate (~60-80%)`, or `Comprehensive (>80%)`.
|
||||
|
||||
Recommended answer to react to: unit plus integration; `Run after each phase`;
|
||||
`Critical paths`. Rationale — this touches compliance data, so the correctness of the
|
||||
CSV writer and the authorization predicate must be pinned by tests, but the admin UI is thin
|
||||
enough that end-to-end coverage would cost more than it catches.
|
||||
|
||||
Two specific cases worth naming in the answer regardless of the general posture, because
|
||||
Codebase context shows both are easy to get wrong here:
|
||||
|
||||
- The UTF-8 BOM survives Excel round-tripping for non-ASCII actor names.
|
||||
- The tenant-scope predicate actually excludes other tenants' events, asserted against seeded
|
||||
cross-tenant data rather than a mock.
|
||||
|
||||
Note for whoever resolves this: the coverage target is a number the human owns. Do not infer
|
||||
it from the codebase's current coverage, and do not soften it to whatever the repo already
|
||||
achieves.
|
||||
|
||||
## Evidence
|
||||
|
||||
- `AGENTS.md` records Vitest as the runner with fixtures colocated under `test/`.
|
||||
- Codebase context: no end-to-end harness exists today; adding one is a real cost, not a flag.
|
||||
```
|
||||
@@ -0,0 +1,449 @@
|
||||
# GitHub Issues Backend
|
||||
|
||||
> Part of plan2code-0-pathfinder — loaded at the top of EVERY session whose map lives on GitHub Issues. It re-expresses the local-file model in issue terms: where the map lives, where a question lives, how blocking, claiming, and resolving are done, and what stays on local disk regardless.
|
||||
>
|
||||
> **Local-file maps never load this file.** If `## Ground rules` says `**Backend:** local`, close it and use `questions.md`.
|
||||
|
||||
Everything the skill says about *judgement* is unchanged by the backend: the fog-vs-question test, the destination grill, one question per session, HITL is never self-answered, the Clearing Gate rubric. This file changes only *where the bytes go*.
|
||||
|
||||
---
|
||||
|
||||
## Why a second backend exists
|
||||
|
||||
Local files are private scratch — `specs/` is gitignored, so the map is yours alone and nobody else can see it, comment on it, or resolve a question in parallel. That is the right default for a solo effort.
|
||||
|
||||
A map on GitHub Issues buys three things local files cannot:
|
||||
|
||||
| | Local files | GitHub Issues |
|
||||
|---|---|---|
|
||||
| Visibility | One machine, one person | Anyone with repo access, in a UI they already have open |
|
||||
| Blocking | A `Blocked by:` line only an agent reads | Native issue dependencies — GitHub greys out blocked issues in its own UI |
|
||||
| Concurrency | One session at a time by construction | Several people can work unblocked questions at once; the assignee is a real lock |
|
||||
|
||||
It costs three things too, and the human must know all three before they pick it:
|
||||
|
||||
1. **Issues on a public repo are public.** The destination, the rejected alternatives, the codebase recon, the technical debt in the blast radius — all of it is world-readable the moment it is written. Never chart to a public repo's tracker anything that would embarrass the project or leak a customer.
|
||||
2. **It writes to shared state.** A local map costs nothing to abandon. Twelve stale issues labelled `pathfinder:grill-hitl` on a team's tracker is litter someone has to clean.
|
||||
3. **It needs `gh`, auth, and issues enabled.** More that can break, in a step whose whole job is to remove friction.
|
||||
|
||||
---
|
||||
|
||||
## Preflight — before the first write
|
||||
|
||||
Run these once, at Chart Step 1, **before** offering GitHub as an option. Any failure means GitHub is not offered at all; say why in one line and continue with local files.
|
||||
|
||||
| # | Check | Command | On failure |
|
||||
|---|---|---|---|
|
||||
| 1 | `gh` is installed | `gh --version` | Not offered — "no `gh` on this machine" |
|
||||
| 2 | Authenticated | `gh auth status` | Not offered — "`gh` is not logged in" |
|
||||
| 3 | Inside a repo with a GitHub remote | `gh repo view --json nameWithOwner,visibility,hasIssuesEnabled` | Not offered — "no GitHub remote here" |
|
||||
| 4 | Issues are enabled | same call, `hasIssuesEnabled` | Not offered — "issues are disabled on this repo" |
|
||||
| 5 | Write access | `gh api repos/<owner>/<repo> --jq .permissions.push` | Not offered — read-only access cannot chart |
|
||||
|
||||
Record `nameWithOwner` and `visibility` from check 3 — **`visibility` is not optional detail.** If it is `PUBLIC`, the offer must say so in the same breath, e.g. *"GitHub Issues — note `jparkerweb/plan2code` is public, so the whole map is world-readable."*
|
||||
|
||||
Once a map exists, preflight shrinks to checks 1 and 2. A session that cannot reach `gh` cannot work a GitHub map: say so and stop, rather than silently starting a local one.
|
||||
|
||||
### Labels
|
||||
|
||||
Create the label set at **Chart Step 5**, with the map issue — never during preflight, which must stay read-only until the human has actually picked `github`. `--force` makes it idempotent, so it is safe to re-run every session:
|
||||
|
||||
```bash
|
||||
gh label create "pathfinder:map" --color 5319E7 --description "Pathfinder map" --force
|
||||
gh label create "pathfinder:grill-hitl" --color 1D76DB --description "Decision only the human can make" --force
|
||||
gh label create "pathfinder:research-afk" --color 0E8A16 --description "Fact-finding, agent alone" --force
|
||||
gh label create "pathfinder:sketch-hitl" --color FBCA04 --description "Human reacts to something concrete" --force
|
||||
gh label create "pathfinder:legwork-hitl" --color D93F0B --description "Manual work needing a human" --force
|
||||
gh label create "pathfinder:legwork-afk" --color D93F0B --description "Manual work the agent can do" --force
|
||||
gh label create "pathfinder:locked" --color B60205 --description "Hard to reverse; consequences recorded" --force
|
||||
gh label create "pathfinder:out-of-scope" --color CFD3D7 --description "Ruled past the destination" --force
|
||||
```
|
||||
|
||||
**Type and mode share one label** — `grill-hitl`, not `grill` plus `hitl` — for the same reason the local `Type:` line is one token: two labels can drift apart, and a `research` question that has quietly become HITL is a question nobody is driving.
|
||||
|
||||
---
|
||||
|
||||
## The equivalence table
|
||||
|
||||
This is the whole mapping. Everything below expands a row.
|
||||
|
||||
| Local file model | GitHub Issues model |
|
||||
|---|---|
|
||||
| `specs/<idea>/pathfinder/map.md` | One issue, labelled `pathfinder:map`, titled `Map: <idea>` |
|
||||
| `questions/NN-<slug>.md` | A **sub-issue** of the map, titled with the question name |
|
||||
| `NN` ordering | The map's sub-issue order — the order they were created, which is dependency order |
|
||||
| `## Question` in the file | The issue body |
|
||||
| `## Answer` appended | A comment on the issue, opening `## Answer` |
|
||||
| `## Evidence` | A comment opening `## Evidence` (a research subagent writes its own) |
|
||||
| `Type:` line | The `pathfinder:<type>-<mode>` label |
|
||||
| `State: open` | Issue open, **no assignee** |
|
||||
| `State: claimed` | Issue open, **assigned** |
|
||||
| `State: resolved` | Issue **closed as completed**, with an `## Answer` comment |
|
||||
| `State: out-of-scope` | Issue **closed as not planned**, labelled `pathfinder:out-of-scope`, no `## Answer` |
|
||||
| `Blocked by: 02, 04` | Native issue dependencies (`dependencies/blocked_by`) |
|
||||
| `Locked: yes` | The `pathfinder:locked` label |
|
||||
| `Claimed: <timestamp>` | GitHub's own assignment event in the timeline |
|
||||
| `## Question Checklist` in `map.md` | **Nothing** — the frontier is a live query, not a written list |
|
||||
| `## Not yet specified`, `## Out of scope`, `## Ground rules`, `## Destination`, `## Glossary` | The same sections, in the map issue body |
|
||||
| `sketch-NN/` | Still local disk — see *What stays on local disk* |
|
||||
| `PLAN-DRAFT-<YYYYMMDD>.md` | Still local disk — see *Handoff* |
|
||||
|
||||
### The checklist is deleted, not ported
|
||||
|
||||
In local mode `map.md` carries a `## Question Checklist` because a directory of files has no queryable state. GitHub has queryable state, so **the map issue body carries no checklist at all.** Closed questions get one line each under `## Decisions so far`; open questions are not listed anywhere.
|
||||
|
||||
This kills the single largest source of drift in the local backend — a checklist that disagrees with the files — and it is why Work Step 2's reconcile pass is much shorter here.
|
||||
|
||||
---
|
||||
|
||||
## Refer by name
|
||||
|
||||
Unchanged, and harder to get right here because GitHub hands you a number for everything. In prose the human reads, write `[Export format](https://github.com/o/r/issues/42)` — never `#42`, never "issue 42", never a bare number. A wall of `#42, #43, #44` is illegible; names read at a glance.
|
||||
|
||||
Bare `#<n>` appears in exactly two places: inside a fallback `Blocked by:` body line when native dependencies are unavailable, and inside a `gh` command.
|
||||
|
||||
---
|
||||
|
||||
## Chart Steps 3-4 — hold the recon, protect the off-ramp
|
||||
|
||||
In `local` mode Step 3 writes `questions/00-codebase-context.md` the moment the recon is done, because a file in gitignored scratch costs nothing if the session then takes the Step 4 off-ramp. **On a shared tracker it costs something**: a stray issue nobody asked for, on a repo other people are watching.
|
||||
|
||||
So on `github`, Step 3 does the recon and **holds it in the session**. It becomes an issue at Step 6, alongside the other questions.
|
||||
|
||||
If the Step 4 breadth-first grill surfaces **no fog**, the off-ramp fires before anything has been created:
|
||||
|
||||
1. Write the recon to `specs/<idea>/pathfinder/questions/00-codebase-context.md` — a **local file**, exactly as the local backend would. It is what `/plan2code-1-plan` Phase 2 needs, and it is too valuable to throw away.
|
||||
2. Create **nothing** on the tracker. No map issue, no question issues, no labels.
|
||||
3. Tell the user plainly and STOP.
|
||||
|
||||
The tracker only ever sees an effort that earned a map.
|
||||
|
||||
---
|
||||
|
||||
## Creating the map (Chart Step 5)
|
||||
|
||||
Title is `Map: <idea>` — the kebab-case idea name, verbatim, so `gh issue list --label pathfinder:map` reads as an index of efforts.
|
||||
|
||||
```bash
|
||||
gh issue create --label "pathfinder:map" --title "Map: audit-log-export" --body-file - <<'EOF'
|
||||
> Pathfinder planning note - decisions, not implementation work. Archive with the spec; do not delete.
|
||||
|
||||
**Status:** Charting
|
||||
**Updated:** 2026-08-08
|
||||
**Confidence:** Requirements-clarity 8/25 · Feasibility-technical 6/25 · Integration-points 6/25 · Risk-assessment 5/25
|
||||
|
||||
<!-- Status: Charting (Chart Steps 5-6) -> Working (Chart Step 7) -> Cleared at the gate.
|
||||
A fresh session routes on this line, so it must be correct before the session ends. -->
|
||||
|
||||
## Destination
|
||||
|
||||
A locked implementation plan for a compliance officer to export a filtered range of audit
|
||||
events from the admin UI and receive them as a single downloadable file. The map ends at the
|
||||
plan, not at shipped code. Continuous streaming to external systems is not on the route.
|
||||
|
||||
## Ground rules
|
||||
|
||||
- **Backend:** github — this issue is the map; questions are its sub-issues.
|
||||
- `AGENTS.md` exists and governs. Its conventions are not re-litigated by any question here.
|
||||
- One question _issue_ per session. `research` questions may run as parallel subagents.
|
||||
- HITL questions are answered by the human in their own words. Never self-answered.
|
||||
- Sketches are throwaway and live on local disk only, under `specs/audit-log-export/pathfinder/sketch-<issue>/`.
|
||||
|
||||
## Glossary
|
||||
|
||||
| Term | Meaning here | Avoid |
|
||||
|---|---|---|
|
||||
| Audit event | One row in `audit_events`: actor, tenant, action, target, timestamp, payload | log line |
|
||||
|
||||
## Decisions so far
|
||||
|
||||
<!-- The index — one line per CLOSED question: enough to judge relevance, then open the
|
||||
issue for the detail. Open questions are NOT listed; they are open sub-issues. -->
|
||||
|
||||
## Not yet specified
|
||||
|
||||
<!-- The fog: in-scope areas you can see but cannot yet phrase as a question. Graduates into
|
||||
sub-issues as answers land, and the graduated bullet is deleted from here. -->
|
||||
|
||||
## Out of scope
|
||||
|
||||
<!-- Work consciously ruled past the destination. Never graduates. One line each: gist plus why. -->
|
||||
EOF
|
||||
```
|
||||
|
||||
Two things that must be exact:
|
||||
|
||||
- **`**Status:**` is still a literal line in the body.** It is how a fresh session routes, exactly as in local mode. `Charting` → `Working` → `Cleared`.
|
||||
- **`**Backend:** github` is the first `## Ground rules` bullet.** It is how a fresh session knows to load this file at all. Without it, a session that opens the map issue has no way to know which playbook it is in.
|
||||
|
||||
Confidence keeps the hyphenated `Requirements-clarity 8/25` form for the same reason it does in local mode — the metrics collector scrapes bare dimension words followed by digits, and would ingest a planning confidence nobody scored.
|
||||
|
||||
**Say once, at Step 5:** *"The map lives on `<owner>/<repo>`'s issue tracker — `<PUBLIC or PRIVATE>`, so `<world-readable / visible to anyone with repo access>`. Everything charted here is visible there."*
|
||||
|
||||
---
|
||||
|
||||
## Creating the questions (Chart Step 6) — two passes, not one
|
||||
|
||||
The local backend charts in a **single pass** because you choose `NN` yourself and can write `Blocked by: 02` into a file before `02` exists. **On GitHub that is impossible** — an issue has no id until the server assigns one, and a dependency edge needs the blocker's id. So charting here reverts to upstream's shape:
|
||||
|
||||
**Pass 1 — create every question issue, in dependency order.** Blockers first. The creation order becomes the sub-issue order, which becomes the reading order for the frontier and the trail, so it is doing the job `NN` does locally. Capture each new issue's number *and* database id as you go.
|
||||
|
||||
```bash
|
||||
# Create, capturing the URL; the number is its last path segment.
|
||||
gh issue create --label "pathfinder:grill-hitl" --title "Export format" --body-file - <<'EOF'
|
||||
> Pathfinder planning note - decisions, not implementation work. Archive with the spec; do not delete.
|
||||
|
||||
## Question
|
||||
|
||||
What file format does an export produce, and what does a recipient need in order to trust
|
||||
the file has not been altered?
|
||||
|
||||
Recommended answer to react to: CSV with a UTF-8 BOM plus a sidecar SHA-256 manifest.
|
||||
EOF
|
||||
|
||||
# The database id — needed for BOTH wiring steps below. Not the #number, not the node_id.
|
||||
gh api repos/<owner>/<repo>/issues/<number> --jq .id
|
||||
```
|
||||
|
||||
**Pass 2 — wire the structure.** Two edges per question, both keyed on **database ids**:
|
||||
|
||||
```bash
|
||||
# a) Attach as a sub-issue of the map. sub_issue_id is the CHILD's database id.
|
||||
gh api --method POST repos/<owner>/<repo>/issues/<map-number>/sub_issues \
|
||||
-F sub_issue_id=<child-db-id>
|
||||
|
||||
# b) Add each blocking edge. issue_id is the BLOCKER's database id.
|
||||
gh api --method POST repos/<owner>/<repo>/issues/<blocked-number>/dependencies/blocked_by \
|
||||
-F issue_id=<blocker-db-id>
|
||||
```
|
||||
|
||||
**The database id is the single most common failure in this backend.** `gh api repos/o/r/issues/42 --jq .id` returns something like `2716143027`. The `42` is the *number*; `I_kwDO...` is the *node id*. The node id is rejected outright. The *number* is worse: a small integer like `42` is itself a perfectly valid database id — of some unrelated issue created years ago — so the call can succeed and silently attach the wrong thing. Fetch `.id` for every issue you are about to reference, and never hand-assemble one.
|
||||
|
||||
Charting still writes `00-codebase-context`'s equivalent — the Step 3 recon you held. Create it **first**, labelled `pathfinder:legwork-afk`, post the recon as an `## Answer` comment, and close it as completed in the same pass. It is resolved on arrival, exactly as in local mode, and it is what the handoff's `## System Context` is built from.
|
||||
|
||||
### If the endpoints are unavailable
|
||||
|
||||
Sub-issues and dependencies are recent GitHub features. On an instance that rejects either endpoint, fall back in the body — and say plainly, once, that the frontier will not render in GitHub's UI:
|
||||
|
||||
| Missing | Fallback |
|
||||
|---|---|
|
||||
| Sub-issues | Put `Part of #<map>` on the first line of each question body, and a task list of the questions in the map body |
|
||||
| Dependencies | Put `Blocked by: #12, #14` on its own line at the top of the question body |
|
||||
|
||||
Prefer the native mechanisms every time they work. The whole reason to pay GitHub's costs is that the human sees the frontier in the UI without opening the map.
|
||||
|
||||
---
|
||||
|
||||
## The frontier query (Work Step 3)
|
||||
|
||||
The frontier is every question that is **open, unblocked, and unassigned**. Lowest position in sub-issue order wins — the same traversal `lowest NN first` gives locally.
|
||||
|
||||
```bash
|
||||
# 1. The map's children, in order, with the state you need to filter on.
|
||||
gh api repos/<owner>/<repo>/issues/<map-number>/sub_issues \
|
||||
--jq '.[] | {number, title, state, assignee: .assignee.login,
|
||||
blocked: .issue_dependencies_summary.blocked_by,
|
||||
labels: [.labels[].name]}'
|
||||
```
|
||||
|
||||
Then, in order:
|
||||
|
||||
1. Drop anything `state: closed` — that is resolved or out of scope.
|
||||
2. Drop anything with an `assignee` — claimed by another session.
|
||||
3. Drop anything still blocked.
|
||||
4. The first survivor is the next question.
|
||||
|
||||
For step 3, `issue_dependencies_summary.blocked_by` counts **open** blockers, which is exactly the live gate — a blocker that closes drops the count without anyone editing anything. **Treat it as a fast pre-filter, not the authority.** If it comes back `null` or absent from the list response, or you need to *name* the blockers for the trail footer or a fully-blocked report, ask the endpoint that owns the answer:
|
||||
|
||||
```bash
|
||||
gh api repos/<owner>/<repo>/issues/<n>/dependencies/blocked_by --jq '.[] | {number, title, state, reason: .state_reason}'
|
||||
```
|
||||
|
||||
A question is unblocked when every blocker listed there is closed. That call is also the only way to see the next trap:
|
||||
|
||||
**A blocker closed as `not planned` is out of scope and will never resolve.** Its dependent is not merely blocked, it is *stranded* — the same trap as locally. Re-frame the dependent's body to drop the dependency, cut the edge, or rule it out too. Never leave it sitting: the summary count cannot tell you the difference, so this check is on you.
|
||||
|
||||
```bash
|
||||
# Cut a dependency edge. The blocker's database id goes in the PATH here, not the body.
|
||||
gh api --method DELETE \
|
||||
repos/<owner>/<repo>/issues/<blocked-number>/dependencies/blocked_by/<blocker-db-id>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Claim (Work Step 4)
|
||||
|
||||
```bash
|
||||
gh issue edit <n> --add-assignee "@me"
|
||||
```
|
||||
|
||||
**The session's first write, before any work.** The assignee *is* the claim — GitHub timestamps it for you, so there is no `Claimed:` line to maintain. An open, unassigned question is unclaimed; that is the whole protocol.
|
||||
|
||||
Unlike the local backend, other people may genuinely be working this map at the same time. Re-read the issue immediately after assigning; if someone else's login is on it, you lost the race — release yours and take the next frontier item.
|
||||
|
||||
---
|
||||
|
||||
## Resolve (Work Step 7)
|
||||
|
||||
Three writes, in this order. The order matters: the answer must exist before the issue closes, or a crash between them leaves a closed question with no decision in it.
|
||||
|
||||
```bash
|
||||
# 1. The answer, as a comment. Same anatomy as a local ## Answer:
|
||||
# decision, rejected alternatives with reasons, consequences, one-line **Gist:**.
|
||||
gh issue comment <n> --body-file - <<'EOF'
|
||||
## Answer
|
||||
|
||||
**Decision.** CSV, UTF-8 with a byte-order mark, RFC 4180 quoting, one header row.
|
||||
Alongside it a sidecar `.sha256` manifest.
|
||||
|
||||
**Rejected.**
|
||||
|
||||
- *JSONL* — trivially streamable, but every named recipient opens these in Excel.
|
||||
- *XLSX* — fixes Excel encoding, but adds a library and makes byte-level verification harder.
|
||||
|
||||
**Consequences.**
|
||||
|
||||
- Do not reuse the billing export's CSV writer; it concatenates strings with no escaping.
|
||||
- Makes [Delivery channel](https://github.com/o/r/issues/45) sharper — the artifact is
|
||||
self-verifying, so a short-lived link no longer weakens the integrity story.
|
||||
|
||||
**Gist:** CSV with a UTF-8 BOM and RFC 4180 quoting, plus a sidecar SHA-256 manifest;
|
||||
JSONL rejected because recipients open these in Excel.
|
||||
EOF
|
||||
|
||||
# 2. Close as completed.
|
||||
gh issue close <n> --reason completed
|
||||
|
||||
# 3. Append the gist to the map's Decisions so far (read body, edit, write back).
|
||||
gh issue view <map-number> --json body --jq .body > /tmp/map.md
|
||||
# ...append: - [Export format](<issue-url>) — <gist>
|
||||
gh issue edit <map-number> --body-file /tmp/map.md
|
||||
```
|
||||
|
||||
Then bump `**Updated:**` and re-score `**Confidence:**` in the same map edit.
|
||||
|
||||
**Never edit the question body to hold the answer.** The body is the question as asked; the comment is the answer. Editing the body rewrites history and destroys the record of what was actually put to the human — which is half of why the answer is defensible three weeks later.
|
||||
|
||||
### Editing the map body safely
|
||||
|
||||
Every map mutation is read-modify-write on a body other sessions may be editing concurrently. Read it fresh immediately before the edit, apply your change to *that* text, and write it straight back. Never edit from a copy you read at the top of the session — you will silently revert whatever landed in between.
|
||||
|
||||
---
|
||||
|
||||
## Ruling a question out of scope (Work Step 8)
|
||||
|
||||
```bash
|
||||
gh issue edit <n> --add-label "pathfinder:out-of-scope"
|
||||
gh issue close <n> --reason "not planned"
|
||||
```
|
||||
|
||||
Then one line under the map's `## Out of scope`, giving the name as a link plus the reason. **No `## Answer` comment** — there is no decision here, only a scope boundary. Add one comment saying why it is out, so the closed issue explains itself.
|
||||
|
||||
`not planned` versus `completed` is the load-bearing distinction: it is how a later session tells a decision that was made from a question that was ruled off the route, and it is what GitHub's UI shows at a glance. Getting it backwards puts a scope boundary into the Provenance table of the plan.
|
||||
|
||||
---
|
||||
|
||||
## Reconcile (Work Step 2)
|
||||
|
||||
Much shorter here — the tracker holds the state, so there is no checklist to rebuild. Three checks:
|
||||
|
||||
| Check | Symptom | Repair |
|
||||
|---|---|---|
|
||||
| Crashed mid-answer | Open, assigned, and an `## Answer` comment already exists | The comment wins. Close as completed, add the gist to Decisions so far, say so. |
|
||||
| Stale claim | Open, assigned, no `## Answer`, and the assignee is you from a dead session | Unassign, say so, put it back on the frontier. **If it is someone else's login, leave it** — that is a live session, not a crash. |
|
||||
| Index drift | A closed, completed question with no line under `## Decisions so far` | Read its `## Answer` comment, append the gist. |
|
||||
|
||||
Then re-read `## Not yet specified` in full — that part is identical to local mode, and the bullet left behind after its question exists is just as corrosive here.
|
||||
|
||||
---
|
||||
|
||||
## What stays on local disk
|
||||
|
||||
Three things never move to the tracker, whatever the backend:
|
||||
|
||||
| Artifact | Where | Why |
|
||||
|---|---|---|
|
||||
| Runnable sketches | `specs/<idea>/pathfinder/sketch-<issue-number>/` | Throwaway code has no business in an issue, and the quarantine rule (never in the project's own source tree) is unchanged. Link the path from the issue and note the reader needs the repo checked out. |
|
||||
| `PLAN-DRAFT-<YYYYMMDD>.md` | `specs/<idea>/` | `/plan2code-1-plan` reads a **file**. This is a hard downstream contract — see Handoff. |
|
||||
| Anything secret | Nowhere | Credentials, tokens, customer data. A `legwork` checklist says *where* a credential lives, never what it is — and on a public tracker that rule stops being a convention and starts being an incident. |
|
||||
|
||||
Sketch directories are named for the issue number rather than a local `NN`, so `sketch-42` belongs to the question at `#42`. Same rules otherwise: throwaway, one command to run, never merged.
|
||||
|
||||
Research subagents work the same way with one substitution: the brief carries the **issue URL** instead of a file path, and the instruction is to post findings as a comment opening `## Evidence` via `gh issue comment` — and to decide nothing. `## Answer` and the close are still written by the session that fired it.
|
||||
|
||||
---
|
||||
|
||||
## The trail footer
|
||||
|
||||
Identical in shape; the inputs come from the query instead of the checklist.
|
||||
|
||||
- **Heading** — `🧭 <idea> · <Status> · <closed>/<total> cleared`, where `<total>` excludes anything labelled `pathfinder:out-of-scope`.
|
||||
- **Glyph order** — sub-issue order, the same order Chart Step 6 created them in.
|
||||
- **Glyphs** — `●` closed as completed · `◉` open and assigned to you · `○` open, unassigned, unblocked · `⊘` open with `blocked_by > 0` · `⊝` closed as not planned.
|
||||
- **Named legend** — names, never `#numbers`. `(blocked:<name>)` names the blocker rather than numbering it, since there is no stable `NN` to point at.
|
||||
- **Confidence** — the plain-English line, from the map body's `**Confidence:**`.
|
||||
|
||||
Form A's resume command changes, because there is no local path to resume from:
|
||||
|
||||
```
|
||||
NEXT STEP · start a new conversation and run:
|
||||
`/plan2code-0-pathfinder https://github.com/<owner>/<repo>/issues/<map-number>`
|
||||
```
|
||||
|
||||
Form B is unchanged — a turn that asks the human something still says `WAITING ON YOU`, still names the outstanding probes, and still emits no resume command.
|
||||
|
||||
---
|
||||
|
||||
## Handoff (The Clearing Gate)
|
||||
|
||||
The gate's four dimensions, the 18/25 bar, the hard caps, and the honesty rules are unchanged. Only the preflight and the plumbing differ.
|
||||
|
||||
**Preflight, GitHub form:**
|
||||
|
||||
| # | Check |
|
||||
|---|---|
|
||||
| 1 | Reconcile pass run (above) |
|
||||
| 2 | Zero open sub-issues — `gh api .../sub_issues --jq '[.[] \| select(.state=="open")] \| length'` returns `0` |
|
||||
| 3 | `## Not yet specified` in the map body is empty |
|
||||
| 4 | Every completed question has an `## Answer` comment carrying a `**Gist:**` |
|
||||
| 5 | `ls specs/<idea>/` shows no existing `PLAN-DRAFT-*.md` |
|
||||
| 6 | No `## Answer` defers a choice to "whoever implements this" |
|
||||
|
||||
**The draft is written to local disk**, at `specs/<idea>/PLAN-DRAFT-<YYYYMMDD>.md`, with the byte-exact status line `**Status:** Phase 3 Complete - Resume at Phase 4`. This is not a preference. `/plan2code-1-plan` discovers its input with `ls specs/`; it has no notion of an issue tracker, and a draft that exists only as an issue is a draft the rest of plan2code cannot see. Create `specs/<idea>/` if charting never needed it.
|
||||
|
||||
Three substitutions inside the template:
|
||||
|
||||
| Local | GitHub |
|
||||
|---|---|
|
||||
| `**Planning record:** specs/<idea>/pathfinder/map.md` | `**Planning record:** <map issue URL>` |
|
||||
| `[Export format](./pathfinder/questions/01-export-format.md)` | `[Export format](https://github.com/o/r/issues/42)` |
|
||||
| `## Out of scope` copied line for line, only the link prefix changing | Copied line for line, links already absolute — **nothing changes at all** |
|
||||
|
||||
Everything else — the mapping table, Section 5 left empty, the two load-bearing `**Next:**` bullets, no scrapable confidence numbers — is unchanged.
|
||||
|
||||
**Freeze the map:**
|
||||
|
||||
1. Set `**Status:** Cleared` in the map issue body.
|
||||
2. Bump `**Updated:**`.
|
||||
3. Add `**Plan:** specs/<idea>/PLAN-DRAFT-<YYYYMMDD>.md` under the status line.
|
||||
4. `gh issue close <map-number> --reason completed`.
|
||||
5. **Close nothing else, delete nothing, edit no answers.** Every question issue stays exactly as it is — it is the rationale record behind the plan.
|
||||
|
||||
Closing the map is the one addition over local mode, and it earns its place: `gh issue list --label pathfinder:map --state open` then reads as *the efforts still being charted*, which is the question a person scanning the tracker actually has.
|
||||
|
||||
A later session that opens a map issue reading `Status: Cleared` must not resume it. Point at the PLAN-DRAFT and `/plan2code-1-plan`, and stop. A redrawn destination is a fresh effort with a fresh map issue.
|
||||
|
||||
---
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---|---|---|
|
||||
| `Not Found` from a `sub_issues` or `dependencies` POST | An issue *number* was passed where a database *id* is required | `gh api repos/o/r/issues/<n> --jq .id`, retry with that |
|
||||
| The wrong issue got attached | A number from another repo happened to be a valid id | Detach, re-fetch `.id` from the right repo, re-attach |
|
||||
| Frontier is empty but open questions remain | Every one is blocked, or every one is assigned | Name the chain and stop — or, if the assignees are stale claims from your own dead sessions, reconcile first |
|
||||
| A blocked question never unblocks | Its blocker was closed as `not planned` | Stranded. Re-frame to drop the dependency and cut the edge, or rule it out too |
|
||||
| Two sessions resolved the same question | The claim was written after the work, not before | Claim is the *first* write. Merge the two answers into one comment, keep one close |
|
||||
| A map edit lost someone's line | The body was edited from a stale copy read earlier in the session | Re-read the body immediately before every map write |
|
||||
| `/plan2code-1-plan` finds nothing to resume | The PLAN-DRAFT was posted as an issue instead of written to `specs/<idea>/` | Write the file. The draft is always local |
|
||||
| Secrets in the tracker | A `legwork` answer pasted a credential | Rotate the credential first, then delete the comment. Editing it is not enough — GitHub keeps the edit history |
|
||||
@@ -0,0 +1,402 @@
|
||||
# Grilling Playbook
|
||||
> Part of plan2code-0-pathfinder — loaded once per session, used by Chart and Work alike.
|
||||
|
||||
Grilling is how a `grill · HITL` question resolves, how Chart Step 2 names the destination, and how Chart Step 4 maps the frontier. It is also the fallback for any question whose type gives you no better route. The output of a grill is a decision in the human's own words — never a decision you made on their behalf.
|
||||
|
||||
**Probe ≠ question file.** A *probe* is one turn of the interrogation; a *question file* is one `questions/NN-<slug>.md` on the map. Batching applies to probes only. **One question file per session still holds** — resolving three question files in one sitting is not what this is.
|
||||
|
||||
## The interview protocol
|
||||
|
||||
**Grilling is batched.** Put up to **three independent probes** to the human per turn, then wait. Never more than three, and never two probes where one's wording depends on the other's answer.
|
||||
|
||||
The old rule here was one probe per turn. It was safe and it was unusably slow: a charted map carries a dozen open questions, and a decision that costs one round trip per probe is a decision the human abandons half-finished. Batching is the default now; the discipline moved from *ask one* to *prove they are independent, then send three*.
|
||||
|
||||
| Rule | Why |
|
||||
|---|---|
|
||||
| Up to 3 probes per turn, never more | Past three the human skims, and a skimmed answer is worse than none. Three is a ceiling, not a quota — send two if only two are independent. |
|
||||
| Only batch mutually independent probes | The independence test below. A probe whose wording or recommendation shifts based on another probe's answer waits for the next turn. |
|
||||
| Reach for the structured question tool first | It is the intended channel, not the leftover bin. Shape the batch so it fits — three probes, plain headers, options a description can carry — and fall back to prose only when the detail test genuinely trips. |
|
||||
| A probe that needs detail goes in prose, never in options | The detail test below. Batching buys round trips; it must never buy them by shrinking a decision to fit a picker. |
|
||||
| Wait for the whole batch before sending the next | Their answers reshape what comes next. Pre-writing turn 2 wastes it. |
|
||||
| Recommend an answer with every probe | A bare question makes the human do all the work. A recommendation gives them something to push against, which is faster and sharper. |
|
||||
| Write it in plain English; keep the technical word only where that word IS the decision | A probe the human has to decode is a probe they answer approximately. See *Say it in plain English*. |
|
||||
| Track the batch; re-ask what came back unanswered | This is what buys the batch. Dropped probes going unnoticed was the entire case for asking one at a time. |
|
||||
| Walk one branch of the decision tree at a time | Batch across the branch's width, never down its depth. Do not ask about export scheduling before you know whether exports exist. |
|
||||
| Do not act until they confirm shared understanding | Recap, get the confirmation, then write. No `## Answer` before the confirmation. |
|
||||
| Never write implementation code during a grill | Grilling produces decisions. If you feel the pull to build, you have reached the edge of the map — say so and hand off. |
|
||||
|
||||
Shape of a single probe, batched or not:
|
||||
|
||||
```
|
||||
Question — one decision, stated so it can be answered in a sentence.
|
||||
Why it — one line: what it blocks, what breaks if it goes the other way.
|
||||
matters
|
||||
Recommend — your pick, with the reason. One line.
|
||||
Options — the genuine alternatives, if there are more than two.
|
||||
```
|
||||
|
||||
Worked probe, from a grill on `[Export format](./questions/04-export-format.md)`:
|
||||
|
||||
> **Q:** When a custodian exports a conversation that includes a 40 MB video attachment, does the export bundle the file or link to it?
|
||||
> **Why it matters:** Bundling sets the size ceiling on an export job and decides whether exports can stream; linking makes the export useless once retention expires the blob.
|
||||
> **Recommendation:** Bundle, with a per-job size cap of 2 GB and an automatic split into part files above that — reviewers open exports offline in tools that cannot follow links.
|
||||
> **Alternatives:** Link-only (smaller, breaks offline); hybrid by MIME type (two code paths, two failure modes).
|
||||
|
||||
Batching does not shrink a probe. Three probes means three of these, each with its own why-it-matters and its own recommendation. Three bare questions in a list is not a batch, it is a form to fill in.
|
||||
|
||||
### Say it in plain English
|
||||
|
||||
Every probe gets read once, by a busy human, in a terminal. Write it the way you would say it out loud to a colleague who knows the product but has never read this skill. Plain is not vague — the two failures are opposite and both cost you the decision: woolly wording gets a woolly answer, dense wording gets a guessed one.
|
||||
|
||||
**Default to everyday words.** Short sentences, concrete nouns. "What should happen when the export is too big to email?" beats "what are the failure semantics of the artifact delivery path under a size-limit violation?" Same decision, one of them answerable on the first read.
|
||||
|
||||
**Keep the technical term where that term IS the decision.** A format name, a real file path, a column name, a version number, a limit, a `## Glossary` term the map already settled — those are load-bearing, and softening them makes the probe unanswerable. `.eml`-in-a-ZIP versus NDJSON *is* the choice; "normal email files versus one big machine-readable stream" is the gloss you put beside it, never the replacement for it.
|
||||
|
||||
> The test: would swapping the term for a plain phrase lose information the human needs in order to choose? Lose information — keep the term and gloss it. Lose nothing — cut it.
|
||||
|
||||
**Gloss an unavoidable term once, inline, then use it freely:** "…stored under Object Lock (S3's write-once mode — once it is set, even we cannot delete early)." Once per session, not once per probe. Re-explaining a term to the person who owns the system is its own insult.
|
||||
|
||||
**Never put Pathfinder's machinery in front of the human.** They are deciding something about their product; the vocabulary below is internal bookkeeping and buys them nothing:
|
||||
|
||||
| Do not say | Say |
|
||||
|---|---|
|
||||
| "this is a `grill · HITL`" | "this one is yours to call" |
|
||||
| "the frontier holds two takeable questions" | "two things we can decide right now" |
|
||||
| "graduating this out of the fog" | "this is sharp enough to write down as a real question now" |
|
||||
| "the sacrificial boundary" | "name one thing people would assume is included that you are willing to cut" |
|
||||
| "shall I set `Locked: yes`?" | "worth recording why we picked this, so nobody re-opens it in six months?" |
|
||||
| "Q3 is blocked by 02" | "the export format question has to land before this one" |
|
||||
| "this batch trips the detail test" | nothing — that call is yours, not theirs |
|
||||
|
||||
**Refer to questions by name, never by number** — "[Export format](./questions/04-export-format.md)", not "04". The number means something to the file system and to nobody else.
|
||||
|
||||
**No metaphor where a fact fits.** Maps, fog, and trails belong in the footer and the mascot. Inside a probe they cost a translation step: "three things here are still undecided" beats "the fog is thick in this quarter of the map."
|
||||
|
||||
The same discipline covers everything else the human reads — the recap turn in *Landing the grill*, the option labels and descriptions in the structured tool, the sketch probes, and the HITL checklists in the resolution playbook. Plain in the question, precise in the term that carries the decision.
|
||||
|
||||
### The independence test
|
||||
|
||||
A probe may join the current batch only if **all three** hold:
|
||||
|
||||
| Test | Fails when |
|
||||
|---|---|
|
||||
| Its wording would not change under any answer to another probe in the batch | "How do we name the part files?" reads differently if the format turns out to be a single stream |
|
||||
| Its recommendation would not change either | You would recommend a 2 GB cap under ZIP parts and no cap under NDJSON |
|
||||
| It does not presuppose another probe's answer | "How often do scheduled exports run?" assumes scheduled exports exist |
|
||||
|
||||
In doubt, hold it back. A held probe costs one extra round trip. A dependent probe sent early costs a wrong answer recorded as a decision, and you will not find out until the plan contradicts itself.
|
||||
|
||||
Independent probes are usually the ones that came from **different areas** — data, interface, security, operations, testing. Dependent probes are usually consecutive steps down one thread.
|
||||
|
||||
### Delivering a batch: choosing the channel
|
||||
|
||||
Two channels — the environment's structured question tool, or numbered Q blocks in prose. **Choose before you write a word, and choose per batch, not per probe.** One channel per turn: a batch split across a tool popup and a loose prose question loses the prose half every time, because the human answers in the tool and never scrolls back.
|
||||
|
||||
**The tool is where you start.** Assemble the batch for it — three probes, a plain two-or-three-word header each, alternatives a sentence or two of description can carry — and only then run the detail test to see whether anything forces you out. Prose is the exception you fall back to, not the safe default you retreat to. Two things make the tool worth the effort: a skipped probe comes back *visibly* skipped, and a picker is answerable in one pass by a human who has thirty seconds. Neither survives the move to prose.
|
||||
|
||||
The two failure directions are opposite and both real. Retreating to prose out of caution costs you the visible skip and the fast reply. Forcing a genuinely gnarly decision into a picker costs you the reasoning, which is worse. The detail test below is where that line sits — run it honestly in both directions.
|
||||
|
||||
**Whichever channel you pick, the turn closes with the waiting footer.** A turn that sends a batch is a **Form B turn** in `trail.md`: the Trail Footer under it names the outstanding probes after `WAITING ON YOU` and carries **no** resume command. Emitting "start a NEW conversation" above an unanswered batch tells the human to leave the session you are sitting in — they walk, and the batch you built to save round trips costs you the whole decision instead. Same for the recap turn below, which is also waiting on them.
|
||||
|
||||
#### The detail test — the only things that force you out of the tool
|
||||
|
||||
Numbered Q blocks are **required, not merely permitted**, if *any* probe in the batch trips *any* row below. One tripping probe downgrades the whole batch.
|
||||
|
||||
These four rows are the whole list. Nothing else forces prose — not a long question, not a hard decision, not a `Locked: yes`, not your discomfort with the widget.
|
||||
|
||||
| Trip | Looks like | Not this |
|
||||
|---|---|---|
|
||||
| The answer must be composed, not picked | "Name one thing a reasonable person would assume is in scope that you are willing to cut." There is no option set, because inventing one puts words in their mouth. | A decision with genuine named alternatives, however weighty. Write the options. |
|
||||
| The probe needs an artifact inline to be answerable | A state table, a fake request/response pair, an ASCII UI, a worked example with real numbers — effectively every `sketch` probe | A probe that merely *mentions* a file path, a format, or a number. Those go in the question text. |
|
||||
| An option cannot be conveyed even in its description | Each alternative needs a worked paragraph before it means anything — a migration path, a failure sequence, a schema | An alternative that needs one or two sentences of trade-off. That is what the description field is for. |
|
||||
| The alternatives themselves are unknown to you | You cannot name the losing options at all, because the frame is theirs — a contract, an old incident, an org politics fact | You can name them but cannot say why each loses. Name them, recommend one, and let the recap turn supply the reasoning. |
|
||||
|
||||
Three things that look like trips and are not:
|
||||
|
||||
- **One label bundling several decisions** — "authentic counts, one-use per attempt, restored on death" is three answers wearing one coat. The fix is to **split it into separate probes**, not to write prose. Three separated probes is exactly one batch.
|
||||
- **Two probes colliding on the tool's short header limit** (16 characters in Claude Code) — `Export scope` twice is unanswerable, but the fix is to rename them (`Date range`, `Who can run`) or to hold one for the next turn. Reword before you retreat.
|
||||
- **A hybrid is possible** — the free-text escape hatch takes "the header from B with the list from C" fine. Trip only when you can already predict the answer *will* be a composition, which is the first row.
|
||||
|
||||
**`Locked: yes` on its own does not trip the test.** A lock's `## Answer` owes every alternative and the reason each lost — but if *you* can already name the alternatives, you have written the options, and the recap turn turns the pick into words the human said. A lock trips only on the fourth row, where you cannot name them at all. Treating every lock as an automatic downgrade sends almost every MODE B decision worth grilling to prose, which defeats the point — MODE B is exactly where a claimed question already has named alternatives and the tool earns its keep.
|
||||
|
||||
**Nothing tripped? Use the structured tool.** Not "may" — do. It is the intended channel, and it is where the visible-skip guarantee behind the partial-answer discipline below comes from.
|
||||
|
||||
**Never reshape a probe to fit the tool.** Reaching for the tool first is not licence to shrink a decision into it. The failure mode is not that the tool rejects a gnarly probe — it is that it *accepts* one. You compress a decision with real texture into three tidy options, the human clicks the least-bad one, and you have recorded a decision with no reasoning behind it. That answer cannot satisfy `## Answer`'s obligation to name what was rejected and why, and nobody finds out until handoff, when the PLAN-DRAFT's Architecture section turns out to have nothing to say. Splitting a bundled probe or renaming a colliding header is reshaping the *batch* and is always right. Cutting a real alternative, or thinning a description until the trade-off disappears, is reshaping the *decision* and is always wrong. When the honest choice is between paragraphs and dishonest options, write the paragraphs.
|
||||
|
||||
#### The structured tool
|
||||
|
||||
The default channel, and the one you build the batch for. One question object per probe, up to three in a single call:
|
||||
|
||||
- **Header** — the decision in two or three plain words (`Export format`, `Size cap`). Not a type, not a marker, not a number.
|
||||
- **Question** — the probe, with its why-it-matters. This is prose and it is not rationed; the same sentences you would have written in a Q block go here.
|
||||
- **Options** — the genuine alternatives, each described by its trade-off, with the recommended one named as such in its description. Two to four; the free-text escape hatch covers the rest. Label plainly, then let the description carry the precise term: `One file per message` labelling the `.eml`-in-a-ZIP option, with `.eml` named in the description.
|
||||
|
||||
A short *label* is not a short *decision*. The label is a handle — `Fixed tick count` — and the description carries the trade-off that makes it choosable. A probe only trips the third detail-test row when even that description cannot hold the option. A label bundling several independent answers is not that row — it is a probe that wants splitting.
|
||||
|
||||
**A click is a decision, not a sentence.** The HITL rule wants an `## Answer` traceable to something the human actually said, and a selected option label is thin evidence on its own. What makes tool-delivered answers legitimate is the recap turn in *Landing the grill* — you play the choices back in prose and they confirm or correct in their own words. Never skip the recap on the grounds that the tool already captured the answer; the tool captured the *pick*, and the recap captures the *agreement*.
|
||||
|
||||
**If a reply comes back thinner than the decision** — a bare click on something you now realise carries weight — do not paper over it. Fold the why into the recap turn as one more probe before writing the `## Answer`.
|
||||
|
||||
#### Numbered Q blocks
|
||||
|
||||
The mandatory channel for anything the detail test catches, and the fallback anywhere the structured tool does not exist. Give each probe the room the tool would have denied it.
|
||||
|
||||
**Copy this shape exactly.** The blank lines are load-bearing, not decoration:
|
||||
|
||||
````markdown
|
||||
Three independent decisions are open. Answer in any order, skip any you want to punt — "Q2: b, Q3: the hybrid" is a perfectly good reply.
|
||||
|
||||
---
|
||||
|
||||
**Q1 — Export format**
|
||||
|
||||
When a custodian exports a year of a channel, what do they get back?
|
||||
|
||||
*Why it matters:* fixes the size ceiling, decides whether exports can stream, and determines whether the review vendors can ingest without a conversion step.
|
||||
|
||||
*Recommendation:* **(a)** — both vendors named in Codebase context read `.eml` natively.
|
||||
|
||||
- **a)** ZIP of one `.eml` per message, plus `manifest.csv`
|
||||
- **b)** A single NDJSON stream — compact and streamable, but nobody downstream parses it
|
||||
- **c)** PST — what Legal asked for by name, but single-writer with a ~50 GB ceiling
|
||||
|
||||
---
|
||||
|
||||
**Q2 — What's not included**
|
||||
|
||||
Name one thing a reasonable person would assume is part of this that you are willing to say is NOT part of it.
|
||||
|
||||
*Why it matters:* this becomes the first thing written down as out of scope, and every later "is that in or out?" call is measured against it. A destination nobody has excluded anything from has not been thought about.
|
||||
|
||||
*Recommendation:* none — this one is yours. If nothing comes to mind, I will offer two candidates and you reject one.
|
||||
|
||||
*(free text — no options on this one)*
|
||||
````
|
||||
|
||||
Number them, keep the numbers stable across turns and sessions, and say out loud that partial answers are welcome — the invitation is what makes the skip visible instead of silent.
|
||||
|
||||
#### Formatting rules for a Q block
|
||||
|
||||
A batch is only worth sending if the human can read it. These are mechanical, and getting them wrong turns three careful probes into one unreadable paragraph:
|
||||
|
||||
| Rule | Why |
|
||||
|---|---|
|
||||
| **A blank line between every element** — the `**Qn — Name**` line, the question, *Why it matters*, *Recommendation*, and the option list | Markdown joins consecutive non-blank lines into a single paragraph. Without blank lines the entire batch renders as a wall of text and the human skims it, which is the failure the three-probe cap exists to prevent. |
|
||||
| **Never hard-wrap a sentence across source lines** | The wrap is invisible to the renderer, so it buys nothing and costs you the paragraph break. Write each sentence as one logical line however long it is; the terminal wraps it. |
|
||||
| **Options are a bullet list, one option per bullet** — `- **a)** …` | Indented continuation lines are the specific thing that collapses: under four spaces the indent is stripped, at four or more it becomes a code block. A bullet list survives every renderer and keeps the options scannable. |
|
||||
| **`---` between probes** | Three probes run together is one block of text. The rule gives the eye a stop and makes "answer Q2 and Q3" easy to aim at. |
|
||||
| **The question itself gets its own line, not a run-on with the heading** | `**Q1 — Export format.** When a custodian…` buries the decision inside a paragraph. Name it, break, then ask it. |
|
||||
| **Never use spaces to convey structure** | Whatever hierarchy you indent by hand disappears on render. Structure comes from blank lines, bullets, and bold — nothing else. |
|
||||
|
||||
The same applies to the recap turn in *Landing the grill*: it is prose the human has to check line by line, so give each recapped decision its own bullet.
|
||||
|
||||
### When answers come back partial
|
||||
|
||||
Assume they will. The human answers two and drops one, and the dropped one is often the hardest and most valuable.
|
||||
|
||||
1. **Diff what came back against what you sent.** Skipped, answered with "Other: skip", or silently omitted all count as unanswered.
|
||||
2. **Lead the next turn with the unanswered probes**, at their original numbers, restated in full. Not "you missed Q3" — the whole probe again, with its recommendation, because they have lost the context by now. Unanswered probes come *before* any new probe, and they count against the cap of three.
|
||||
3. **Skipped twice, stop pushing.** Record it under `## Evidence` as an open probe with your recommendation verbatim, then either narrow it into something answerable or spin it out — a fresh question file if you can phrase it sharply, a `## Not yet specified` line if you cannot.
|
||||
4. **Never promote your own recommendation into `## Answer`.** A probe the human declined twice is unanswered, not decided. Writing it up as decided is self-answering a HITL question, which breaks the skill.
|
||||
|
||||
Never let a dropped probe fall off the end of the session unrecorded.
|
||||
|
||||
## Facts you look up, decisions you ask
|
||||
|
||||
This is the single rule that keeps a grill from feeling like an interrogation. If a **fact** can be found by exploring the environment — filesystem, codebase, tools, docs, config, git history, the map's own resolved questions — go find it. The **decisions** are the human's; put each one to them and wait.
|
||||
|
||||
| You look it up | You ask |
|
||||
|---|---|
|
||||
| Which Postgres version the app runs against | Whether the export index is allowed to add a new table |
|
||||
| Whether `ExportJob` already has a `status` column | What states that column is allowed to hold |
|
||||
| How the current retention sweep is scheduled | Whether exports must survive a retention sweep |
|
||||
| Whether the repo uses Vitest or Jest | Whether these paths get unit tests, integration tests, or neither |
|
||||
| What the S3 bucket lifecycle rule is today | Whether we are allowed to change it |
|
||||
| What `AGENTS.md` says about naming conventions | Anything `AGENTS.md` does not already answer |
|
||||
|
||||
Look first, then ask. A question that opens with "I checked `src/export/job.ts` — it already has a `status` enum with `queued | running | failed`. Does a partial success need a fourth state?" is worth three of "how should export status work?"
|
||||
|
||||
**Never ask what `AGENTS.md`, the map's `## Ground rules`, or a resolved question already answers.** Re-asking a settled decision reopens it by accident and costs you the human's trust for the rest of the session.
|
||||
|
||||
**When the lookup is expensive**, that is not a grill — it is a `research · AFK` or `legwork` question. Say so, note it, and keep grilling the decisions you can still put to the human.
|
||||
|
||||
## The HITL rule, stated hard
|
||||
|
||||
An agent that answers its own grill has broken the skill.
|
||||
|
||||
A `grill · HITL` question resolves **only** through live exchange with the human. Not from the codebase, not from a plausible default, not from "the obvious industry standard," not from what you would have picked. The whole value of the question is that a human with context you do not have chose one branch over another.
|
||||
|
||||
Signs you are about to break it:
|
||||
|
||||
- You wrote a recommendation and then wrote the `## Answer` without a reply in between.
|
||||
- You wrote "assuming the user would want X" anywhere.
|
||||
- You resolved a question in a session where the human said nothing but "go".
|
||||
- The `## Answer` contains no sentence traceable to something the human actually said.
|
||||
|
||||
**When the human goes unreachable mid-grill:**
|
||||
|
||||
1. The claim **stays**. `State: claimed` and `Claimed:` are left exactly as they are.
|
||||
2. Append to `## Evidence` — never `## Answer` — the exchange so far: the probes already answered, **every probe of the last batch still outstanding**, and your recommendation for each, verbatim.
|
||||
3. End the session. Report the question by name and say it is mid-grill and waiting on the human. This one IS a session end, so the Trail Footer takes **Form A** — the pathed resume command, not `WAITING ON YOU`.
|
||||
4. Invent nothing. No provisional answer, no "pending confirmation" answer, no default recorded as a decision.
|
||||
|
||||
The next session picks the claim back up and re-sends the outstanding batch. If the session was truly abandoned rather than paused, Work Step 2 reconciliation resets it to `open` on its own — that is its job, not yours.
|
||||
|
||||
## The four disciplines
|
||||
|
||||
Run all four continuously during any grill. They are not stages; they fire whenever the trigger appears in what the human just said.
|
||||
|
||||
### Challenge against the glossary
|
||||
|
||||
When a term conflicts with the map's `## Glossary`, call it out immediately, mid-sentence if necessary.
|
||||
|
||||
> "The glossary defines **Export** as a completed archive file delivered to a custodian. You just used it for the background job that builds one. Which do you mean — or do we need a second term?"
|
||||
|
||||
### Sharpen fuzzy or overloaded language
|
||||
|
||||
When a term is vague or carries two meanings, propose a precise canonical term and get a ruling.
|
||||
|
||||
> "You keep saying 'account.' Sometimes you mean the organization paying us, sometimes the individual login. Those are a **Customer** and a **User** and they have different retention rules. Which one owns the export quota?"
|
||||
|
||||
### Stress-test relationships with concrete invented scenarios
|
||||
|
||||
Do not ask abstractly whether a relationship holds. Invent a specific scenario that probes the edge and force a precise boundary.
|
||||
|
||||
> "A custodian leaves the company on the 3rd. Their retention policy expires their messages on the 5th. Legal opens a hold on the 4th. On the 6th, does the export still contain those messages?"
|
||||
|
||||
Invent the numbers, names, and dates. Vague scenarios get vague answers.
|
||||
|
||||
### Cross-reference claims against the actual code
|
||||
|
||||
When the human states how something works, check whether the code agrees, and surface contradictions instead of quietly picking a side.
|
||||
|
||||
> "You said retention deletes rows. `RetentionSweep.run()` sets `deleted_at` and leaves the row in place — a soft delete. Which is the behavior we are designing against?"
|
||||
|
||||
A contradiction is a finding, not an embarrassment. Surface it in the same turn you found it.
|
||||
|
||||
## The glossary
|
||||
|
||||
Domain modeling would write a `CONTEXT.md`. Pathfinder does **not** — plan2code owns `AGENTS.md`, and a competing root glossary file would collide with it. The map's `## Glossary` section is the one place resolved terms live.
|
||||
|
||||
Entry format — one row in the map's `## Glossary` table: the term, a one-or-two-sentence definition, and the rejected synonyms in the Avoid column:
|
||||
|
||||
```
|
||||
| Term | Meaning here | Avoid |
|
||||
|---|---|---|
|
||||
| Export | A completed, immutable archive file delivered to a custodian. Always the artifact, never the process that produces it. | download, extract, dump |
|
||||
| Export Job | The background unit of work that produces an Export. Has states; an Export does not. | export run |
|
||||
| Custodian | The person whose communications an Export contains. Not necessarily the person who requested it. | user, owner, subject |
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
| Rule | Detail |
|
||||
|---|---|
|
||||
| Update inline, never defer | The moment a term is resolved, write it into `## Glossary` and save. A term you meant to add at the end of the session is a term you lost. |
|
||||
| Be opinionated | When several words compete, pick one and put the rest in the Avoid column. A glossary that lists synonyms as equals has decided nothing. |
|
||||
| Keep definitions tight | One or two sentences. Define what it **is**, not what it does. |
|
||||
| Only project-specific terms | "Retention Policy" belongs. "Timeout", "retry", "DTO" do not, however heavily the project uses them. |
|
||||
| It is a glossary and nothing else | No implementation details, no open questions, no scratch notes, no decisions. Decisions live in question files. |
|
||||
| Group under sub-bullets only when clusters emerge | A flat list is fine for one cohesive area. |
|
||||
|
||||
Every term you resolve is a term no later session re-litigates. That is the whole return on the discipline.
|
||||
|
||||
## The Lock test
|
||||
|
||||
Domain modeling would offer an ADR here. Pathfinder sets `Locked: yes` on the question instead — the decision record and the question are the same file.
|
||||
|
||||
Offer `Locked: yes` **only** when all three hold:
|
||||
|
||||
| Test | Meaning | Fails when |
|
||||
|---|---|---|
|
||||
| Hard to reverse | Changing your mind later costs real time or migration | You could flip it in an afternoon |
|
||||
| Surprising without context | A future reader will ask "why did they do it this way?" | It is the obvious choice anyone would make |
|
||||
| A real trade-off | There were genuine alternatives and one was picked for specific reasons | There was only ever one option |
|
||||
|
||||
If any one is missing, skip it. An easy-to-reverse decision will just get reversed. An unsurprising one raises no questions to answer. One with no alternative records nothing beyond "we did the obvious thing."
|
||||
|
||||
**What earns a lock:** architectural shape ("the export index is a materialized view, not a table"); integration patterns between components ("retention and export communicate by event, never by direct call"); technology choices carrying lock-in (database, message bus, auth provider — not every library, just the ones that would take a quarter to swap); boundary and scope decisions, including the explicit no-s; deliberate deviations from the obvious path ("hand-written SQL here, not the ORM, because the ORM cannot express the retention join"); constraints invisible in the code ("no cross-region replication — the data residency contract forbids it"); and non-obvious rejections ("we considered and rejected GraphQL, for reasons someone will otherwise re-propose in six months").
|
||||
|
||||
**What `Locked: yes` obliges the `## Answer` to contain:**
|
||||
|
||||
1. The decision itself, in one or two sentences.
|
||||
2. **Every alternative genuinely considered, and why each was rejected.** This is the part that makes a lock worth having. "We rejected X" with no reason is not a lock.
|
||||
3. The consequences a later reader would not guess.
|
||||
4. The one-line `**Gist:**` that Work Step 7 requires, same as any answer.
|
||||
|
||||
Offer it, do not impose it: *"This one looks hard to reverse and the reasoning will not be obvious in six months. Lock it?"* The human decides.
|
||||
|
||||
**Where locks go at handoff:** the handoff playbook lifts every `Locked: yes` question into the PLAN-DRAFT's **Architecture** section, and their rejected alternatives and standing constraints into **Assumptions**. Unlocked answers still inform the draft, but locks are the ones that survive verbatim into planning. Grill them harder for that reason.
|
||||
|
||||
## The one grill every map must resolve, and the lens applied to all of them
|
||||
|
||||
### 1. Testing posture — `grill · HITL`
|
||||
|
||||
Chart Step 6 requires this question. It exists because `/plan2code-1-plan` Phase 1 asks for exactly three things and stalls without them: testing types, whether tests run after each phase, and the coverage target. A map that clears without answering them hands the human a plan session that immediately re-asks.
|
||||
|
||||
The first three probes below pass the independence test against each other — none reads differently under another's answer — so **send all three as one batch**. This is the canonical worked example of a full batch, and it is the canonical case for the structured tool: every one of the three has named alternatives you can already write, the answers are fixed literals rather than prose, and nothing in the batch trips the detail test.
|
||||
|
||||
| Probe | Recommend by default |
|
||||
|---|---|
|
||||
| Which types are in scope — unit, integration, E2E, or none? | Unit plus integration; E2E only where a real browser or real broker is the only honest test |
|
||||
| Does the suite run after each implementation phase, or once at the end? | `Run after each phase` — work that cannot be verified when it lands cannot be signed off |
|
||||
| Coverage target: critical paths, moderate (~60-80%), or comprehensive (>80%)? | Critical paths, named explicitly, rather than a percentage nobody defends |
|
||||
| What is deliberately not tested, and why? | **Fourth probe, and it does not ride in the batch.** It has no option set — the explicit no-s have to be composed — and it reads differently once the types are settled. Fold it into the recap turn, where you are already waiting on them. |
|
||||
| What is already there — runner, fixtures, CI wiring? | **Not a probe.** Look it up before you send the batch, and cite it in the probes above |
|
||||
|
||||
Record the answer in the exact literals `/plan2code-2-document` string-matches: the types; `Run after each phase` or `Dedicated phase only`; and `Critical paths`, `Moderate (~60-80%)`, or `Comprehensive (>80%)`. Not prose — a paraphrase matches no branch downstream.
|
||||
|
||||
### 2. Test seams and verifiability — a lens, not a question file
|
||||
|
||||
For each major decision on the map, ask the same question: **how will anyone know it works?** A decision nobody can verify is a decision that silently rots.
|
||||
|
||||
This is not a question of its own and never gets a file or an `NN`. Run it inside whatever grill is claimed; anything it surfaces that needs deciding separately becomes a new question at Work Step 8.
|
||||
|
||||
These six all interrogate one decision from different sides, so they batch cleanly — but they ride along inside the claimed grill rather than owning a turn. Fold the two or three that bite into the batch you were already sending; never spend a whole turn on all six.
|
||||
|
||||
| Probe | What it flushes out |
|
||||
|---|---|
|
||||
| What observable behavior changes if this decision is implemented correctly? | Decisions with no observable effect — usually a sign the question was about implementation, not design |
|
||||
| What is the cheapest thing that fails when it breaks? | The seam. If the answer is "a customer complains," there is no seam yet |
|
||||
| Can this be tested without a live third-party account, a real S3 bucket, a wall-clock sleep? | Untestable-by-construction designs, while they are still cheap to change |
|
||||
| Where does the boundary go so a test can stand at it — an interface, a queue, an HTTP edge, a pure function? | The seam the plan will need to name |
|
||||
| What does the failure look like in production — log line, metric, alert, dead-letter queue? | Verifiability after ship, not just in CI |
|
||||
| If we get this wrong, how long before we find out? | Decisions that need a canary or a feature flag rather than a test |
|
||||
|
||||
If a decision survives all six with no answer, it is not ready to leave the map. Either re-frame the question, or add the seam as a constraint in the `## Answer` so the plan inherits it.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
| Failure mode | What it looks like | Fix |
|
||||
|---|---|---|
|
||||
| Batching **dependent** probes | "What format, what do we name the part files, and how big is a part?" | Only the first is independent. Send it; hold the other two — they are unanswerable until format lands. |
|
||||
| Drip-feeding one probe at a time | Twelve open questions on the map, one probe per response, the human gives up on session four | Batch up to three independent probes. On a charted map the human's round trips are the scarce resource, not your token budget. |
|
||||
| Losing a probe the human skipped | Sent three, got two back, moved on and never mentioned the third | Diff the batch. Lead the next turn with what came back empty, restated in full. |
|
||||
| A batch of naked questions | Three one-liners with no recommendations and no why-it-matters | Every probe in a batch carries its own recommendation and its own stake. Otherwise you have offloaded the thinking, not the round trips. |
|
||||
| Sending a batch as a wall of text | Three probes hard-wrapped across source lines with their options indented, all collapsing into one paragraph on render | Blank line between every element, options as a bullet list, `---` between probes. The three-probe cap exists so the human reads all three; an unreadable batch gets skimmed, and a skimmed answer is worse than none. |
|
||||
| Flattening a gnarly probe into a picker | An architectural decision reduced to three option labels because the batch was already going through the structured tool | Run the detail test. One tripping probe sends the whole batch to numbered Q blocks. A clicked option records no reasoning, and the `## Answer` needs reasoning. |
|
||||
| Defaulting to prose when nothing tripped | A clean three-probe batch written as Q blocks "to be safe", or downgraded just because the question will be `Locked: yes` | The detail test is a test, not a preference, and it is four rows long. Nothing tripped means the tool: you gain visibly skipped probes, and the recap turn still captures the reasoning a lock needs. |
|
||||
| Retreating to prose over a fixable batch | Two probes collided on the 16-character header, or one option label was bundling three answers, so the whole batch went to Q blocks | Neither is a detail-test trip. Rename the headers; split the bundled probe. Reshape the batch, never the decision. |
|
||||
| Sending four probes because the tool accepts four | A fourth probe added to a clean batch of three because there was room in the call | The cap is three regardless of what the environment allows. The ceiling is the human's attention, not the tool's schema. |
|
||||
| Grilling in Pathfinder's own vocabulary | "The frontier has one takeable `grill · HITL` — shall we graduate 02 out of the fog and lock it?" | Plain English. Markers, types, `NN` numbers, and fog are your bookkeeping; the human is deciding about their product. |
|
||||
| Plain-washing the load-bearing term | "Do you want the friendly file or the compact one?" where the real choice is `.eml`-in-a-ZIP versus NDJSON | Plain wording, precise nouns. Name the formats and gloss them; a decision made on a euphemism cannot be written into `## Answer`. |
|
||||
| Asking what the codebase already answers | "Do you use Postgres or MySQL?" | Look. Every avoidable question spends trust you need for the hard ones. |
|
||||
| Accepting a vague answer and moving on | "Handle it sensibly" → recorded as the decision | Push once more, concretely: "Sensibly meaning we drop the attachment, or fail the whole job?" A vague answer is not an answer. |
|
||||
| Leading the human to your preferred answer | "You'd want Postgres here, right?" | Recommend openly, then present the real alternatives with their real merits. A recommendation invites a fight; a leading question suppresses one. |
|
||||
| Grilling past the decision into implementation | "Should the retry helper take a callback or return a promise?" | That is the plan's job, or the implementer's. Stop at the decision. The pull to keep going is the edge of the map. |
|
||||
| Drifting off the claimed question | Claimed `[Export format](./questions/04-export-format.md)`, forty minutes later deep in auth | Name the drift out loud, capture the new thread as a fresh question or as a line in `## Not yet specified`, and return. One question _file_ per session. |
|
||||
| Self-answering a HITL question | An `## Answer` with no words the human said | Delete it. Reopen the question. See the HITL rule. |
|
||||
| Recording the decision but not the rejections | "We chose event-driven." | Rejections are half the record — and mandatory when `Locked: yes`. Ask what else was on the table before you close. |
|
||||
| Grilling a fact | "How long does the retention sweep take?" | If it is measurable, measure it — or make it a `research · AFK` question. Do not make the human guess at their own system. |
|
||||
| Letting the glossary go stale | Three terms resolved, none written down | Write each one the moment it lands. Deferring loses them. |
|
||||
| Closing without confirmation | Answer written straight after the last reply | Recap the whole chain of decisions, get the explicit confirm, then write. |
|
||||
|
||||
## Landing the grill
|
||||
|
||||
When the branch is walked out. **Steps 1-3 are ONE turn, not three** — recap, confirmation request, and lock offer go out together, because a lock offer sent after a separate confirmation costs a round trip to ask a yes/no the human could have answered alongside the recap.
|
||||
|
||||
1. Recap the decisions in order, in the human's own terms, using glossary vocabulary. Include anything a structured-tool reply left implicit, so the confirmation covers the reasoning and not just the picks.
|
||||
2. Ask for the confirmation. Do not skip this — the recap is where the human catches the one thing you misheard, and where a clicked option becomes words they said.
|
||||
3. Apply the Lock test in the same message. Offer, do not impose.
|
||||
4. Write `## Answer` per Work Step 7: the decision, what was rejected and why, consequences, and a one-line `**Gist:**`. Evidence, links, and transcript fragments go under `## Evidence`.
|
||||
5. Anything the grill surfaced that belongs to a different question goes to the map — a fresh question if you can phrase it sharply, `## Not yet specified` if you cannot, `## Out of scope` if it sits past the destination.
|
||||
@@ -0,0 +1,415 @@
|
||||
# Handoff Playbook — Clearing the Map into a PLAN-DRAFT
|
||||
|
||||
Loaded at The Clearing Gate. Turns a cleared map into `specs/<idea>/PLAN-DRAFT-<YYYYMMDD>.md` that `/plan2code-1-plan` resumes from at Phase 4, then freezes `pathfinder/` as the rationale record.
|
||||
|
||||
**Backend note.** The scoring rubric, the hard caps, the honesty rules, the template, and the mapping table are the same either way — and the draft is written to local disk either way, because `/plan2code-1-plan` reads a file, not a tracker. On `**Backend:** github`, `github-issues.md` replaces only the preflight table, the question links (issue URLs, already absolute), and the freeze steps.
|
||||
|
||||
Nothing here is creative. The gate is scored, the mapping is fixed, the template is literal. Follow it exactly or the resuming plan session silently loses work.
|
||||
|
||||
## Preflight — before scoring anything
|
||||
|
||||
| # | Check | If it fails |
|
||||
|---|---|---|
|
||||
| 1 | Re-run the reconcile pass: read every file in `questions/`, rebuild every map marker from the files | Fix the map first. Markers are derived, never authored. |
|
||||
| 2 | Zero `[ ]`, zero `[/]`, zero `[!]` rows in `## Question Checklist` | Not cleared. Return to the frontier. |
|
||||
| 3 | `## Not yet specified` is empty | Not cleared. Graduate the fog into questions, or admit it is out of scope. |
|
||||
| 4 | Every `[x]` row's file has a real `## Answer` with a `**Gist:**` | The file wins over the marker. Repair, then re-check. |
|
||||
| 5 | `ls specs/<idea>/` shows no existing `PLAN-DRAFT-*.md` | One already exists — read it. Update it in place; never add a second dated draft. |
|
||||
| 6 | The destination is reachable with nothing left to decide — no `## Answer` defers a choice to "whoever implements this" | Not cleared. Name the open decision and graduate it into a question. |
|
||||
|
||||
Only after all six pass do you score the four dimensions.
|
||||
|
||||
## The clearing-gate scoring rubric
|
||||
|
||||
Four dimensions, 0-25 each, scored against **written evidence in `questions/`** — never against your recollection of the conversation. The gate needs **every dimension at 18/25 or better**. There is no averaging: 25/25/25/14 fails.
|
||||
|
||||
### Band scale (applies to all four)
|
||||
|
||||
| Band | Meaning |
|
||||
|---|---|
|
||||
| 23-25 | Decided, written down, and consequences recorded. A developer could act on it without asking a question. |
|
||||
| 18-22 | Decided and written down. Residual detail remains, but it is *specification* detail that Phase 5/6 settles — not a decision anyone still has to make. |
|
||||
| 12-17 | A real decision is still open, or an answer exists with no evidence behind it. **Gate fails.** |
|
||||
| 0-11 | The area was never charted. **Gate fails**, and the map was cleared prematurely. |
|
||||
|
||||
The 18-boundary is the honest line between *"needs designing"* and *"needs deciding"*. Pathfinder owns deciding. If someone still has to decide, you are not done.
|
||||
|
||||
### What each dimension scores
|
||||
|
||||
| Dimension | Scores | Raises it | Lowers it |
|
||||
|---|---|---|---|
|
||||
| **Requirements Clarity** | Are the requirements unambiguous? | Every resolved `grill` answer states the decision AND what was rejected; the testing-posture answer names types, cadence, and coverage | Answers phrased as preferences ("probably NDJSON") instead of decisions; a requirement that only exists in the map gist and not in a question file |
|
||||
| **Technical Feasibility** | Do we know HOW to build each component? | Resolved `sketch` questions with a real artifact under `sketch-NN/`; `research` answers citing primary sources under `## Evidence`; `00-codebase-context.md` naming the actual files that change | A mechanism nobody has exercised in this codebase; a research answer whose `## Evidence` is empty or cites only a blog post |
|
||||
| **Integration Points** | Are all external dependencies identified? | Every system named in `## Destination` has a resolved question touching it; auth, quota, and failure behavior named per integration | An integration mentioned in an answer but never questioned; a config store that was read but never written to during a sketch |
|
||||
| **Risk Assessment** | Are blockers documented with mitigations? | Answers that record consequences; `Locked: yes` answers that say what breaks if reversed; ceilings with a stated behavior at the ceiling | A recorded limit with no decided behavior past it; a `Locked: yes` answer with no consequences section |
|
||||
|
||||
### Hard caps
|
||||
|
||||
A cap overrides your judgment. While a cap condition holds, the dimension **cannot** exceed 17, so the gate cannot pass.
|
||||
|
||||
| Cap | Condition |
|
||||
|---|---|
|
||||
| Requirements ≤ 17 | The testing-posture question is not `resolved`, or its answer omits any of types / cadence / coverage |
|
||||
| Feasibility ≤ 17 | Any `research` question is `resolved` with an empty `## Evidence` |
|
||||
| Integration ≤ 17 | A system named in `## Destination` has no resolved question touching it |
|
||||
| Risk ≤ 17 | Any `Locked: yes` answer records no consequences |
|
||||
|
||||
### Honesty rules
|
||||
|
||||
- Score the **written record**, not the conversation. If the human agreed to something in a session and nobody wrote it into a `## Answer`, it does not exist and it does not earn points.
|
||||
- A filled `## Answer` is not automatically 25. An answer that decides but records no consequences tops out around 20.
|
||||
- Never round up to clear the gate. A 17 that "feels like an 18" is the exact case the gate exists to catch.
|
||||
- Never move a decision to `## Out of scope` to raise a score. Out-of-scope is a scoping act with a reason; scope-cutting to pass a gate is score inflation with extra steps.
|
||||
- If two dimensions are borderline, write the one-line justification for each score into the Session End report. Justifications that cannot be written are scores that cannot be defended.
|
||||
|
||||
### Worked example — idea `audit-log-s3-export`
|
||||
|
||||
Destination: *"A spec a developer can implement: nightly export of tenant audit logs to customer-owned S3 buckets, with a signed manifest per run."*
|
||||
|
||||
Resolved questions: [Codebase context](./questions/00-codebase-context.md), [Export format](./questions/01-export-format.md), [Scheduling model](./questions/02-scheduling-model.md), [Destination auth](./questions/03-destination-auth.md), [Retention and replay](./questions/04-retention-and-replay.md), [Testing posture](./questions/05-testing-posture.md), [Throughput ceiling](./questions/06-throughput-ceiling.md). Ruled out: [Failure notification](./questions/07-failure-notification.md).
|
||||
|
||||
**First scoring pass:**
|
||||
|
||||
| Dimension | Score | Justification against evidence |
|
||||
|---|---|---|
|
||||
| Requirements Clarity | 22/25 | Four `grill` answers state decisions and rejections (NDJSON chosen, CSV rejected for nested actor payloads). Testing posture settled: integration + unit, run after each phase, moderate coverage. Minus 3: the manifest's exact field list is unspecified — a Phase 5 spec detail, not an open decision. |
|
||||
| Technical Feasibility | 21/25 | `sketch-02` uploaded a 400 MB multipart object to a real bucket end to end. `00-codebase-context.md` names `AuditExportJob` and the existing Hangfire registration as the extension points. Minus 4: no component in this codebase has ever assumed a cross-account IAM role; the pattern is documented in AWS docs cited under `## Evidence` but unexercised here. |
|
||||
| Integration Points | 23/25 | Three integrations, each with a resolved question: S3 (Destination auth), Hangfire (Scheduling model), tenant config store (Codebase context). Auth and quota named per integration. Minus 2: the tenant config store's write path was read but never exercised by a sketch. |
|
||||
| Risk Assessment | **16/25** | Consequences recorded on Export format and Destination auth. But Throughput ceiling establishes 200 MB per tenant per day at p95 and **records no decided behavior above it** — truncate, spill to the next run, or fail the run is still undecided. |
|
||||
|
||||
**Gate result: FAIL** on Risk Assessment (16 < 18), and the orchestrator third clearing condition — the destination reachable with nothing left to decide — fails with it: a decision is genuinely still open. Do not write a PLAN-DRAFT. Name the failure, graduate `questions/08-overflow-behavior.md` (`grill · HITL`, `Blocked by: none`) from the gap, and end the session on the frontier.
|
||||
|
||||
**Second scoring pass, one session later**, with [Overflow behavior](./questions/08-overflow-behavior.md) resolved (spill to the next run, alarm at three consecutive spills):
|
||||
|
||||
| Dimension | Score |
|
||||
|---|---|
|
||||
| Requirements Clarity | 22/25 |
|
||||
| Technical Feasibility | 21/25 |
|
||||
| Integration Points | 23/25 |
|
||||
| Risk Assessment | 20/25 |
|
||||
|
||||
All four at 18 or better. Gate passes. Proceed to the mechanism.
|
||||
|
||||
## The mechanism, stated plainly
|
||||
|
||||
Pathfinder writes `specs/<idea>/PLAN-DRAFT-<YYYYMMDD>.md` with the header line:
|
||||
|
||||
`**Status:** Phase 3 Complete - Resume at Phase 4`
|
||||
|
||||
`/plan2code-1-plan` **already recognises that exact string.** Its "Check for Existing Progress" block, which runs before Phase 1, reads:
|
||||
|
||||
`- Status "Phase 3 Complete - Resume at Phase 4": Resume at Phase 4`
|
||||
|
||||
That is the whole handoff. The string is the contract, and it needs **zero changes to the planning skill** — pathfinder is impersonating the Large-project Context Checkpoint that 1-plan's own Phase 3 performs, which writes the same status for the same reason.
|
||||
|
||||
Consequences of that being a literal string match:
|
||||
|
||||
- Copy it byte for byte. Plain ASCII hyphen-minus surrounded by single spaces. An en dash, a colon, or "Phase 3 complete" in lower case breaks the match and 1-plan starts over at Phase 1 — throwing away every decision the map holds.
|
||||
- It goes on its own `**Status:**` line in the header block, not buried in prose.
|
||||
- The file must be named `PLAN-DRAFT-<YYYYMMDD>.md` and live directly in `specs/<idea>/`. `PLAN-DRAFT-*.md` is a reserved name inside `pathfinder/` — never write it there.
|
||||
- Get the date from the shell (`date +%Y%m%d` in Bash, `Get-Date -Format yyyyMMdd` in PowerShell). Do not guess it.
|
||||
|
||||
**Discovery is shell-only.** `specs/` is gitignored, so Glob silently returns nothing and every downstream skill would report "no PLAN-DRAFT found". Use `ls specs/` and `ls specs/<idea>/` (Bash) or `Get-ChildItem specs/` (PowerShell) — the same rule 1-plan and `/plan2code-2-document` follow when they look for the file you are about to write.
|
||||
|
||||
**Do not append `## Planning Metrics` or any metrics comment.** Pathfinder is not a metered step. 1-plan's Phase 7 owns that block and will add it when it finishes the plan. Do not emit any of the loop's completion tokens listed in the skill's Rules anywhere under `specs/`.
|
||||
|
||||
## Map to PLAN-DRAFT mapping
|
||||
|
||||
Everything in the draft traces to something on the map. Nothing is invented at handoff time — if a section has no source, that is a gate failure you missed, not a paragraph to write from imagination.
|
||||
|
||||
| Source on the cleared map | Becomes |
|
||||
|---|---|
|
||||
| `## Destination` | Section 1 Executive Summary (2-3 sentences, present tense) and Section 7 Success Criteria (the destination restated as checkable outcomes) |
|
||||
| Resolved `grill` answers describing behavior | Section 2.1 Functional Requirements, one `FR-N` per decided behavior |
|
||||
| Resolved `grill` answers describing performance, security, scale, operability | Section 2.2 Non-Functional Requirements, one `NFR-N` each |
|
||||
| `## Out of scope` | Section 2.3 Out of Scope — copied line for line, wording and order intact. The ONLY permitted change is the link prefix: `./questions/` becomes `./pathfinder/questions/`, because the draft sits one level above the map. Do not re-word, re-order, or summarise; a re-worded scope boundary is a re-litigated one. |
|
||||
| The testing-posture question's answer | Section 2.4 Testing Strategy table (Types / Phase Testing / Coverage) |
|
||||
| Resolved `research` answers and their `## Evidence` | Section 3 Tech Stack — the cited source becomes the Justification cell |
|
||||
| `questions/00-codebase-context.md` | The `## System Context` section, and the components table in 4.3 |
|
||||
| Answers with `Locked: yes` | Section 4 Architecture (4.1 Pattern rationale, 4.4 Data Model, 4.5 API Design) **and** Section 9 Assumptions — a locked decision is an assumption downstream work is allowed to rely on |
|
||||
| Consequences recorded across all `## Answer` sections | Section 6 Risks and Mitigations — the consequence is the Risk, the decision that bounds it is the Mitigation |
|
||||
| `## Ground rules` | The `AGENTS.md` line in `## System Context`; conventions the plan must not violate |
|
||||
| The shape of the map (question count, integrations touched, components named) | The `## Scope Assessment` section |
|
||||
| Resolved `sketch` questions and their artifacts | Section 3 Justification cells and Section 6 Mitigation cells ("proven by `pathfinder/sketch-02/`") |
|
||||
| Every resolved question, by name | `## Pathfinder Provenance` |
|
||||
| — | **Section 5 Implementation Phases stays empty.** 1-plan Phase 6 breaks the work into phases. Pathfinder decides; it does not slice. Leave the placeholder note in place and do not put implementation checkboxes there. |
|
||||
|
||||
Questions ruled `out-of-scope` never appear in Provenance and never become requirements. Their one line in `## Out of scope` is their only trace — that is the point of the marker.
|
||||
|
||||
## The PLAN-DRAFT template
|
||||
|
||||
Write this literally, substituting real content. Keep the section numbering exactly as shown — 1-plan and `/plan2code-2-document` both address sections by number.
|
||||
|
||||
````markdown
|
||||
> Pathfinder planning note - decisions, not implementation work. Archive with the spec; do not delete.
|
||||
|
||||
# Audit Log Export - Implementation Plan
|
||||
|
||||
**Created:** 2026-08-03
|
||||
**Status:** Phase 3 Complete - Resume at Phase 4
|
||||
**Charted by:** `/plan2code-0-pathfinder` over 9 sessions
|
||||
**Planning record:** `specs/audit-log-s3-export/pathfinder/map.md` (no PLAN-CONVERSATION - this plan was charted, not conversed)
|
||||
**Confidence (pathfinder):** Requirements-clarity 22/25 · Feasibility-technical 21/25 · Integration-points 23/25 · Risk-assessment 20/25
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
Tenants can have their audit logs exported nightly to an S3 bucket they own, with a
|
||||
signed manifest per run so they can prove completeness. Export runs on the existing
|
||||
Hangfire schedule, writes NDJSON, and assumes a customer-provided cross-account IAM
|
||||
role with an external ID. Runs that exceed the per-tenant daily ceiling spill into the
|
||||
next run rather than truncating.
|
||||
|
||||
## 2. Requirements
|
||||
|
||||
### 2.1 Functional Requirements
|
||||
|
||||
- [ ] **FR-1:** Export each tenant's prior-day audit events as newline-delimited JSON, one object per event, UTF-8, no BOM
|
||||
- [ ] **FR-2:** Write a per-run manifest listing object keys, event counts, byte counts, and a SHA-256 per object
|
||||
- [ ] **FR-3:** Sign the manifest with the platform export key; publish the public key at a stable URL
|
||||
- [ ] **FR-4:** Assume the tenant-configured IAM role with the tenant's external ID; never use platform-owned credentials against a customer bucket
|
||||
- [ ] **FR-5:** Allow an operator to replay any run within a 7-day window without duplicating manifest sequence numbers
|
||||
- [ ] **FR-6:** Spill events above the per-tenant daily ceiling into the next scheduled run, oldest first
|
||||
- [ ] **FR-7:** Raise an alarm after three consecutive spilling runs for the same tenant
|
||||
|
||||
### 2.2 Non-Functional Requirements
|
||||
|
||||
- [ ] **NFR-1:** Sustain 200 MB per tenant per day at p95 without extending the nightly window past 04:00 UTC
|
||||
- [ ] **NFR-2:** Never log tenant event bodies, bucket names, or assumed-role ARNs above debug level
|
||||
- [ ] **NFR-3:** A failed run must leave no partial objects visible in the customer bucket
|
||||
- [ ] **NFR-4:** Export must add no schema changes to the audit event write path
|
||||
|
||||
### 2.3 Out of Scope
|
||||
|
||||
<!-- copied line for line from pathfinder/map.md ## Out of scope; only the link prefix changes -->
|
||||
|
||||
- [Failure notification](./pathfinder/questions/07-failure-notification.md) — email/webhook delivery of run failures belongs to the platform alerting effort, not this export. The alarm in FR-7 is raised, not delivered.
|
||||
- **On-demand export from the tenant UI** — the destination is the scheduled export. A user-triggered export is a separate effort with its own map.
|
||||
- **Log formats other than NDJSON** — Parquet was raised and ruled past the destination.
|
||||
|
||||
### 2.4 Testing Strategy
|
||||
|
||||
| Aspect | Decision |
|
||||
|---|---|
|
||||
| Types | Unit + Integration |
|
||||
| Phase Testing | Run after each phase |
|
||||
| Coverage | Moderate (~60-80%) |
|
||||
|
||||
## System Context
|
||||
|
||||
**Project type:** Existing codebase — .NET 8 service, `src/Platform.Audit/`
|
||||
|
||||
| Aspect | Finding | Source |
|
||||
|---|---|---|
|
||||
| Entry points to change | `AuditExportJob`, registered in `HangfireStartup.ConfigureRecurringJobs()` | [Codebase context](./pathfinder/questions/00-codebase-context.md) |
|
||||
| Existing patterns to follow | Jobs resolve tenant scope via `ITenantScopeFactory`; no job reads config directly | [Codebase context](./pathfinder/questions/00-codebase-context.md) |
|
||||
| Integration surfaces | S3 (customer-owned), Hangfire scheduler, `TenantConfigStore` | [Destination auth](./pathfinder/questions/03-destination-auth.md), [Scheduling model](./pathfinder/questions/02-scheduling-model.md) |
|
||||
| Technical debt in the path | `AuditQuery` materialises full result sets; streaming reader needed before FR-1 | [Codebase context](./pathfinder/questions/00-codebase-context.md) |
|
||||
| System boundaries | Read-only against the audit store; writes only to customer buckets and the run-log table | [Retention and replay](./pathfinder/questions/04-retention-and-replay.md) |
|
||||
| Conventions in force | `AGENTS.md` present and read; its logging and DI conventions govern | `pathfinder/map.md` ## Ground rules |
|
||||
|
||||
## Scope Assessment
|
||||
|
||||
**Assessment: Medium** — 11 requirements across 3 integrations, 5 components touched. No Large threshold is met.
|
||||
|
||||
| Indicator | Value |
|
||||
|---|---|
|
||||
| Requirements decided | 11 (7 FR + 4 NFR) |
|
||||
| Components | 5 (`AuditExportJob`, `NdjsonWriter`, `ManifestSigner`, `S3RoleAssumer`, `ExportRunLog`) |
|
||||
| Integrations | 3 (S3, Hangfire, `TenantConfigStore`) |
|
||||
| Decisions charted | 8 resolved, 1 ruled out of scope |
|
||||
|
||||
## 3. Tech Stack
|
||||
|
||||
<!-- Phase 4 completes this table. Rows below are decided; do not re-open them. -->
|
||||
|
||||
| Category | Technology | Version | Justification |
|
||||
|---|---|---|---|
|
||||
| Serialization | `System.Text.Json` NDJSON writer | .NET 8 | [Export format](./pathfinder/questions/01-export-format.md) — no new dependency; CSV rejected for nested actor payloads |
|
||||
| Object storage | `AWSSDK.S3` multipart upload | 3.7.x | [Throughput ceiling](./pathfinder/questions/06-throughput-ceiling.md) — proven in `pathfinder/sketch-02/` against a real bucket at 400 MB |
|
||||
| Cross-account auth | STS `AssumeRole` + external ID | — | [Destination auth](./pathfinder/questions/03-destination-auth.md) — AWS confused-deputy guidance cited in that file's `## Evidence` |
|
||||
| Scheduling | Existing Hangfire recurring job | in-repo | [Scheduling model](./pathfinder/questions/02-scheduling-model.md) — a new scheduler was rejected |
|
||||
|
||||
## 4. Architecture
|
||||
|
||||
### 4.1 Pattern
|
||||
|
||||
Pipeline inside the existing job host: query → stream → chunk → upload → manifest → sign.
|
||||
Chosen because the audit store is the only source and the export is strictly one-way.
|
||||
[Locked] A separate export microservice was rejected — see [Scheduling model](./pathfinder/questions/02-scheduling-model.md).
|
||||
|
||||
### 4.2 System Context Diagram
|
||||
|
||||
<!-- Phase 5 refines. Boundaries above are settled. -->
|
||||
|
||||
### 4.3 Components
|
||||
|
||||
| Component | Responsibility | Inputs | Outputs | Depends on |
|
||||
|---|---|---|---|---|
|
||||
| `AuditExportJob` | Orchestrates one tenant-run | Tenant id, run date | Run result | `TenantConfigStore` |
|
||||
| `NdjsonWriter` | Streams events to chunked NDJSON | Event stream | Byte stream, counts | — |
|
||||
| `S3RoleAssumer` | Assumes the tenant role, returns a scoped client | Role ARN, external ID | `IAmazonS3` | STS |
|
||||
| `ManifestSigner` | Builds and signs the run manifest | Object metadata | Signed manifest | Platform export key |
|
||||
| `ExportRunLog` | Records runs for replay and spill detection | Run result | Run rows | Platform DB |
|
||||
|
||||
### 4.4 Data Model
|
||||
|
||||
Manifest sequence numbers are per tenant, monotonic, and reused on replay.
|
||||
[Locked] See [Retention and replay](./pathfinder/questions/04-retention-and-replay.md).
|
||||
|
||||
### 4.5 API Design
|
||||
|
||||
<!-- Phase 5 fills. No public API surface was decided during pathfinding. -->
|
||||
|
||||
## 5. Implementation Phases
|
||||
|
||||
<!-- Intentionally empty. Phase 6 of /plan2code-1-plan breaks the requirements
|
||||
above into implementation phases. Pathfinder decides; it does not slice. -->
|
||||
|
||||
## 6. Risks and Mitigations
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|---|---|---|---|
|
||||
| Cross-account role assumption is unexercised in this codebase | Medium | High | Spike `S3RoleAssumer` against a second AWS account before any other component |
|
||||
| A tenant exceeds the daily ceiling indefinitely | Medium | Medium | Spill oldest-first plus a three-run alarm — [Overflow behavior](./pathfinder/questions/08-overflow-behavior.md) |
|
||||
| Partial objects visible after a failed run | Low | High | Upload to a run-scoped prefix, publish the manifest last — the manifest is the commit point |
|
||||
| `AuditQuery` materialises full result sets | High | High | Streaming reader is a prerequisite, not an optimisation |
|
||||
| Customer revokes the role mid-run | Low | Medium | Fail the run whole; replay window covers recovery |
|
||||
|
||||
## 7. Success Criteria
|
||||
|
||||
- [ ] A tenant with a configured role receives NDJSON and a signed manifest for the prior day, nightly
|
||||
- [ ] The published public key verifies the manifest signature
|
||||
- [ ] A 250 MB tenant-day completes without extending the window past 04:00 UTC
|
||||
- [ ] A replay inside 7 days reproduces the run without a new sequence number
|
||||
- [ ] A failed run leaves nothing visible in the customer bucket
|
||||
|
||||
## 8. Open Questions
|
||||
|
||||
<!-- 1-plan's template removes this section when empty; pathfinder keeps it with "None" so a
|
||||
resuming session can see the map cleared clean, rather than that the section was forgotten. -->
|
||||
|
||||
None. The map cleared with zero open questions.
|
||||
|
||||
## 9. Assumptions
|
||||
|
||||
- Tenants can create an IAM role in their own account — [Destination auth](./pathfinder/questions/03-destination-auth.md) [Locked]
|
||||
- The nightly Hangfire window remains available and is not contended by other jobs — [Scheduling model](./pathfinder/questions/02-scheduling-model.md) [Locked]
|
||||
- Manifest sequence reuse on replay is acceptable to tenant compliance teams — [Retention and replay](./pathfinder/questions/04-retention-and-replay.md) [Locked]
|
||||
- Audit events are immutable once written, so a replay reproduces byte-identical output
|
||||
|
||||
## Pathfinder Provenance
|
||||
|
||||
Charted over 9 sessions. Each requirement above traces to a decision below; open the
|
||||
question for what was rejected, why, and what it costs.
|
||||
|
||||
| Question | Gist |
|
||||
|---|---|
|
||||
| [Codebase context](./pathfinder/questions/00-codebase-context.md) | `AuditExportJob` and `HangfireStartup` are the extension points; `AuditQuery` needs a streaming reader first |
|
||||
| [Export format](./pathfinder/questions/01-export-format.md) | NDJSON with a per-run signed manifest; CSV rejected for nested actor payloads |
|
||||
| [Scheduling model](./pathfinder/questions/02-scheduling-model.md) | Reuse the existing Hangfire recurring job; a dedicated export service was rejected |
|
||||
| [Destination auth](./pathfinder/questions/03-destination-auth.md) | Customer-owned bucket via assumed role plus external ID; no platform-held customer credentials |
|
||||
| [Retention and replay](./pathfinder/questions/04-retention-and-replay.md) | 7-day replay window, sequence numbers reused on replay |
|
||||
| [Testing posture](./pathfinder/questions/05-testing-posture.md) | Unit + integration, run after each phase, moderate coverage |
|
||||
| [Throughput ceiling](./pathfinder/questions/06-throughput-ceiling.md) | 200 MB per tenant per day at p95; multipart upload proven in `pathfinder/sketch-02/` |
|
||||
| [Overflow behavior](./pathfinder/questions/08-overflow-behavior.md) | Spill oldest-first into the next run; alarm after three consecutive spills |
|
||||
|
||||
Ruled out of scope: [Failure notification](./pathfinder/questions/07-failure-notification.md) — recorded in 2.3.
|
||||
|
||||
---
|
||||
|
||||
**Next:** Resume with `/plan2code-1-plan` at Phase 4 (Tech Stack).
|
||||
|
||||
- **Phase 7 verification:** sections 1, 2, System Context and Scope Assessment are already settled — their source of truth is `specs/audit-log-s3-export/pathfinder/map.md`, not this conversation. Verify sections 3-7 only.
|
||||
- **Phase 7:** replace THIS file in place. Do not create a second PLAN-DRAFT in this folder.
|
||||
````
|
||||
|
||||
### Two things in that template that are not optional
|
||||
|
||||
**No scrapable confidence numbers anywhere in the file.** Write the confidence as `Requirements-clarity 22/25 · Feasibility-technical 21/25 · Integration-points 23/25 · Risk-assessment 20/25` — hyphenated dimension labels, sub-scores over 25, no total, no percent sign.
|
||||
|
||||
The reason is exact. When a PLAN-DRAFT carries no `METRICS_JSON` comment, the metrics collector falls back to scraping it by regex: an overall-confidence pattern that requires a literal `%`, and four breakdown patterns that match a bare `Requirements` / `Feasibility` / `Integration` / `Risk` followed directly by whitespace, a colon, or a pipe and then digits. **The breakdown patterns do not require a percent sign.** A pathfinder-written draft always lacks that comment until `/plan2code-1-plan` Phase 7 appends one, so both the percent sign *and* the bare dimension words have to be kept off the page — otherwise the pipeline records a planning-step confidence that no planning step ever produced. The hyphen in `Requirements-clarity` breaks the match; a table row reading `| Requirements | 11 |` does not, which is why the Scope Assessment row is labelled `Requirements decided`.
|
||||
|
||||
**The `**Next:**` footer must ship with both bullets.** Pathfinder cannot edit the planning skill, so those two instructions travel inside the artifact:
|
||||
|
||||
- *Without the verification bullet*, 1-plan's Phase 7 does exactly what it is told to do — "re-read conversation as source of truth" — finds a fresh conversation that starts at Phase 4 and contains no requirements discussion at all, concludes sections 1 and 2 are unsupported, and silently drops the requirements that N pathfinder sessions produced. The bullet redirects the source of truth for the settled sections to `map.md`.
|
||||
- *Without the replace-in-place bullet*, 1-plan's Phase 7 creates `PLAN-DRAFT-<its own date>.md` alongside yours. `/plan2code-2-document` then finds two drafts in the folder, hits its "Multiple found: List all, ask which to document" branch, and asks the user to disambiguate between a pathfinder draft and a plan draft that partially supersedes it.
|
||||
|
||||
Never drop the footer to make the file tidier. It is load-bearing.
|
||||
|
||||
## System Context and Scope Assessment — why they buy you Phase 4
|
||||
|
||||
These two named, unnumbered sections are what make "Resume at Phase 4" legitimate rather than a shortcut. They stand in for the phases pathfinder already did the work of:
|
||||
|
||||
| Draft section | Satisfies | Because pathfinder already |
|
||||
|---|---|---|
|
||||
| Sections 1 and 2 (including 2.4 Testing Strategy) | 1-plan **Phase 1: Requirements Analysis** | Grilled every functional and non-functional decision, and always charted a testing-posture question — that question exists specifically so Phase 1's testing prompt is already answered |
|
||||
| `## System Context` | 1-plan **Phase 2: System Context Examination** | Wrote `questions/00-codebase-context.md` at Chart Step 3: directory structure, key components verified against actual code, patterns and conventions, integration points, technical debt, boundaries — Phase 2's own checklist, item for item |
|
||||
| `## Scope Assessment` | 1-plan **Phase 3: Scope Assessment** | Produced the counts Phase 3 measures — requirements, components, integrations — as a byproduct of charting. Map the totals onto Phase 3's Small / Medium / Large table and state the verdict |
|
||||
|
||||
Populate `## System Context` from `00-codebase-context.md` and nothing else. It is the one question guaranteed to exist on every map, it was resolved on the spot with the codebase open, and it is a `legwork · AFK` answer — factual, not preferential. Cite it in the Source column so a skeptical reader can check the finding against the file.
|
||||
|
||||
Populate `## Scope Assessment` from the shape of the map. Count resolved questions that produced requirements (not `00-codebase-context.md`, not out-of-scope ones), count distinct components named across the answers, count distinct external systems. Apply Phase 3's thresholds honestly: Large if **any** threshold is met. Score the counts, never the session count — a map can take nine sessions to clear and still be Medium, and the `Phase 3 Complete - Resume at Phase 4` status string works regardless of the verdict, so there is nothing to gain by inflating it. Pathfinder cannot count implementation phases (Section 5 is deliberately left empty), so assess on requirements, components, and integrations only.
|
||||
|
||||
If the charting session found `AGENTS.md` absent, say so in the `Conventions in force` row rather than leaving it blank. The plan session needs to know the conventions were never available, not guess that they were checked.
|
||||
|
||||
## Freeze the map
|
||||
|
||||
Once the PLAN-DRAFT is written and saved:
|
||||
|
||||
1. Set `**Status:** Cleared` in `map.md`.
|
||||
2. Bump `**Updated:**` to today.
|
||||
3. Add a plan pointer line under the status: `**Plan:** ../PLAN-DRAFT-20260803.md`.
|
||||
4. Leave **everything** under `pathfinder/` exactly where it is — `map.md`, every file in `questions/`, every `sketch-NN/` directory.
|
||||
|
||||
**Never delete `pathfinder/`.** It is the rationale record behind the plan: what was decided, what was rejected, why, and what it costs. It sits in the same class as `PLAN-CONVERSATION-*.md` — the transcript a plan is defensible against — and `/plan2code-4-finalize` archives it alongside `PLAN-DRAFT.md` and `PLAN-CONVERSATION-*.md` into `specs--completed/<idea>/`. Deleting it turns every locked decision in the plan into an unexplained constraint six months from now.
|
||||
|
||||
Do not tidy it either. Do not collapse resolved questions into the map, do not prune `## Evidence`, do not remove sketch directories because the code is throwaway. The sketch is the proof behind a feasibility score.
|
||||
|
||||
**A later session that finds `Status: Cleared` must not resume work on it.** The map is finished; there is nothing left to decide inside it. Point at the PLAN-DRAFT and `/plan2code-1-plan`, and stop. If the destination has been redrawn — the scope grew, an out-of-scope item came back, the goal changed — that is a **fresh effort with a fresh map**, not a resumption: a new kebab-case idea name, a new `specs/<new-idea>/pathfinder/`, charting from Step 1. The frontier stops at the destination, so a new destination gets a new frontier. Reopening a cleared map silently invalidates the PLAN-DRAFT that was built from it, and nothing downstream would notice.
|
||||
|
||||
## What to tell the user
|
||||
|
||||
Session End for the cleared case reports six things, in this order:
|
||||
|
||||
| # | Report |
|
||||
|---|---|
|
||||
| 1 | The destination, restated — what the map was finding its way to, now reached |
|
||||
| 2 | How many decisions were made, and the headline ones **by name** |
|
||||
| 3 | What was ruled out of scope, and why (one line each) |
|
||||
| 4 | The four confidence scores, with a one-line justification for any below 21 |
|
||||
| 5 | The PLAN-DRAFT path |
|
||||
| 6 | The next command — carried by the Trail Footer, which at `Cleared` routes to `/plan2code-1-plan` |
|
||||
|
||||
Example:
|
||||
|
||||
> **Destination reached:** a spec a developer can implement for nightly audit-log export to customer-owned S3 buckets with a signed manifest.
|
||||
>
|
||||
> **8 decisions made** across 9 sessions. The load-bearing ones: [Export format](./questions/01-export-format.md) settled on NDJSON with a signed manifest; [Destination auth](./questions/03-destination-auth.md) settled on assumed roles with an external ID so we never hold customer credentials; [Overflow behavior](./questions/08-overflow-behavior.md) settled on spilling rather than truncating.
|
||||
>
|
||||
> **Ruled out of scope:** failure-notification delivery (belongs to the platform alerting effort), on-demand export from the tenant UI (its own effort), Parquet output (past the destination).
|
||||
>
|
||||
> **Confidence:** Requirements-clarity 22/25 · Feasibility-technical 21/25 · Integration-points 23/25 · Risk-assessment 20/25. Feasibility is held at 21 because cross-account role assumption is unexercised in this codebase — it is a documented pattern, not a proven one here, and it is the first thing to spike.
|
||||
>
|
||||
> **Written:** `specs/audit-log-s3-export/PLAN-DRAFT-20260803.md`. It resumes planning at Phase 4 — sections 1, 2, System Context and Scope Assessment are already settled. The reasoning behind every one of them stays in `specs/audit-log-s3-export/pathfinder/`; do not delete it.
|
||||
|
||||
Then the closing block from the skill's Session End — the mascot with the message *The way is clear! Time to plan!* followed by the Trail Footer, whose trail shows every stop walked to the `⚑` destination and whose command is `/plan2code-1-plan`.
|
||||
|
||||
**One branch.** If `## Ground rules` records `AGENTS.md` as **absent**, recommend `/plan2code-init` FIRST, and offer `questions/00-codebase-context.md` as its input:
|
||||
|
||||
> Before planning: this project has no `AGENTS.md`, and `/plan2code-1-plan` blocks on that. Run `/plan2code-init` first and attach `specs/audit-log-s3-export/pathfinder/questions/00-codebase-context.md` — the recon pass already established the structure, conventions, and integration points it asks for. Then `/plan2code-1-plan`.
|
||||
|
||||
Nothing to commit — `specs/` is gitignored. Say so once, then stop.
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---|---|---|
|
||||
| 1-plan starts at Phase 1 and re-asks for requirements | The status string does not match byte for byte | Compare against the quoted line above; watch for en dashes and casing |
|
||||
| 1-plan cannot find the draft at all | Glob was used to discover `specs/` | Shell only: `ls specs/` |
|
||||
| `/plan2code-2-document` asks which of two drafts to use | The replace-in-place bullet was dropped from the footer | Merge the two drafts into the pathfinder-dated one, delete the other, restore the footer |
|
||||
| The finished plan is missing requirements the map decided | The verification bullet was dropped from the footer | Re-derive 2.1 and 2.2 from the resolved answers, restore the footer |
|
||||
| Metrics report a planning confidence nobody scored | A percent sign, or a bare dimension word followed by a number, reached the file | Hyphenate the dimension labels and drop the percent sign |
|
||||
| A locked decision in the plan has no visible reason | `pathfinder/` was deleted or pruned | Unrecoverable. This is why the freeze step exists |
|
||||
| Gate passes but the first implementation session immediately hits an undecided question | A dimension was rounded up | The gate was the check. Score the written record, not the feeling |
|
||||
@@ -0,0 +1,41 @@
|
||||
# Questions & Map Format
|
||||
> Part of plan2code-0-pathfinder — the on-disk format both modes share: directory layout, `NN` numbering, the question-file schema, `Type:` vocabulary, and the marker / blocking rules. The main file keeps only the marker legend and a layout gist; the authority is here.
|
||||
>
|
||||
> **This file describes the `local` backend only.** If `## Ground rules` says `**Backend:** github`, the equivalence table in `github-issues.md` replaces every rule below — there are no files, no `NN`, no schema lines, and no checklist.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
specs/<idea>/
|
||||
├── pathfinder/
|
||||
│ ├── map.md <- the index
|
||||
│ ├── questions/NN-<slug>.md <- 00-codebase-context.md always exists
|
||||
│ └── sketch-NN/ <- optional runnable sketch, throwaway
|
||||
└── PLAN-DRAFT-<YYYYMMDD>.md <- written ONLY when the map clears
|
||||
```
|
||||
|
||||
`questions/` is ground truth; `map.md` is a rebuildable index that gists and links. A filled `## Answer` beats any `State:` line. Detail lives in exactly one place — the question file.
|
||||
|
||||
## Numbering
|
||||
|
||||
`NN` is zero-padded from `00`, assigned in dependency order (blockers lower), **never reused or renumbered** — links and `Blocked by:` would rot silently. Next = max + 1. `00` is always `00-codebase-context.md`, never anything else. Gaps in the sequence are normal and harmless.
|
||||
|
||||
## The five schema lines
|
||||
|
||||
Each question file carries five contiguous `Key: value` lines after its H1 — NOT YAML frontmatter, no `---` delimiters:
|
||||
|
||||
| Line | Values |
|
||||
|---|---|
|
||||
| `Type:` | `grill · HITL` \| `research · AFK` \| `sketch · HITL` \| `legwork · HITL` \| `legwork · AFK` — one token, so type and mode cannot drift |
|
||||
| `State:` | `open` \| `claimed` \| `resolved` \| `out-of-scope` |
|
||||
| `Blocked by:` | `none` \| `02, 04` |
|
||||
| `Claimed:` | `none` \| `<YYYY-MM-DD HH:mm>` |
|
||||
| `Locked:` | `yes` only when hard to reverse AND surprising without context AND a real trade-off |
|
||||
|
||||
**Type meanings.** **grill** (default) — a decision only the human can make. **research** — a fact outside this directory gates it. **sketch** — the human needs something concrete to react to. **legwork** — manual work that must happen before a decision is possible.
|
||||
|
||||
## Markers and blocking
|
||||
|
||||
**Map markers**, rebuilt from the files every session: `[ ]` open — **these rows ARE the frontier** · `[/]` claimed · `[x]` resolved · `[!]` open but blocked · `[-]` out of scope.
|
||||
|
||||
**Unblocked** ⟺ every `NN` in `Blocked by:` is `resolved`. **Stranded:** a blocker gone `out-of-scope` never resolves — the question is not merely blocked. Re-frame its `## Question` to drop the dependency, or rule it out too. Never leave it sitting.
|
||||