plan2code-bot: replace auto-responder with LLM-as-judge evaluation

Bot now acts as an authentic QA agent instead of rubber-stamping every
question and step.

- intelligent-responder answers AskUserQuestion via LLM using current
  observations (tools used, files created, errors) instead of keyword
  matching; auto-responder removed.
- evaluator runs after each step with step-specific criteria
  (prompts/evaluation-criteria.ts), produces 0-100 score plus
  strengths/weaknesses/critical issues. Avg <60 blocks finalization.
- observation-collector captures tool_use, file writes, errors, and
  question reasoning; session-runner falls back to scraping tool_use
  blocks when canUseTool doesn't fire and dedupes both sources.
- Bot writes BOT-EVALUATION.md and BOT-NOTES.md for metrics analysis.
- Idea generator: 12 categories instead of CLI/web-app coin flip,
  stronger seed adherence, less developer-tool bias.
- Init step now writes a minimal AGENTS.md stub instead of running
  the full init skill, avoiding hallucinated architecture before plan.
- Implement step uses Read/Write/Edit/Glob/Grep directly instead of
  a Skill sub-session that produced no visible tool observations.
- bin: add --help, accept --idea="value" form, strip surrounding
  quotes; evaluator maxTurns 3 -> 30; warn when parser misses SCORE.
This commit is contained in:
2026-05-11 13:14:28 -07:00
parent 9d1104d105
commit c2579a7b6a
15 changed files with 1434 additions and 254 deletions
+57 -12
View File
@@ -20,15 +20,37 @@ export function buildStepPrompt(step: StepName, config: BotConfig): string {
function buildInitPrompt(config: BotConfig): string {
return `${AUTONOMOUS_PREAMBLE}
Run the /plan2code-init skill to generate an AGENTS.md file for this project.
This is a brand new project with no existing code. Create a minimal stub AGENTS.md file and an IDEA.md file. Do NOT run the /plan2code-init skill there is no codebase to analyze yet.
The project idea is: ${config.ideaDescription}
The project name is: ${config.ideaName}
When asked questions:
- For the project description, use the idea above
- Accept all defaults and approve all gates
- Complete the init process fully`;
## What to create
### AGENTS.md
Create a minimal stub with ONLY the following do NOT invent architecture, tech stack details, commands, or file structures:
\`\`\`markdown
# AGENTS.md
This file provides guidance to AI coding agents when working with code in this repository.
## Project Overview
**Name:** ${config.ideaName}
**Description:** ${config.ideaDescription}
## Status
This project is in the planning phase. Architecture, commands, and detailed documentation will be added after the plan and document steps are complete.
\`\`\`
### IDEA.md
If IDEA.md does not already exist, create it with the project name and description.
## Rules
- Do NOT create .agents-docs/ or any detail files there is nothing to document yet
- Do NOT hallucinate architecture, dependencies, file structures, or tech stack choices
- Do NOT install dependencies or scaffold project files
- ONLY create the two files above`;
}
function buildPlanPrompt(config: BotConfig): string {
@@ -63,14 +85,37 @@ When making decisions:
function buildImplementPrompt(config: BotConfig): string {
return `${AUTONOMOUS_PREAMBLE}
Run the /plan2code-3-implement skill to implement the next available phase.
You are a senior software engineer implementing a project phase. Do NOT use the Skill tool implement directly using Read, Write, Edit, Glob, and Grep tools.
When making decisions:
- Pick the first available/unchecked phase
- Write working code that fulfills the spec tasks
- Mark tasks as complete as you finish them
- Approve and sign off when the phase is done
- Skip running tests if asked (or select the simplest test option)`;
## Process
1. **Find the spec**: Read \`specs/*/overview.md\` to find the phase checklist
2. **Pick the next phase**: Find the first phase marked \`[ ]\` (pending) or \`[/]\` (in-progress)
3. **Read the phase file**: Read the corresponding \`phase-X.md\` from the same directory
4. **Mark phase in-progress**: Update \`[ ]\` to \`[/]\` in overview.md
5. **Implement each task sequentially**:
- Read the task specification completely
- Write the code using Write or Edit tools create real files, not code blocks
- Mark the task \`[x]\` in the phase file immediately after completing it
6. **Complete the phase**: After all tasks, fill in the "Phase Completion Summary" in the phase file
7. **Mark phase complete**: Update \`[/]\` to \`[x]\` in overview.md
## Rules
- Follow AGENTS.md if it exists
- Implement specs EXACTLY no creative additions or unsolicited improvements
- Write task completion status (\`[x]\`) to disk immediately after each task never batch
- Only create files mentioned in the spec tasks
- Use the specified file paths, function names, and structures from the spec
- No placeholder code fully implement every function
- Match existing codebase conventions
- Do NOT run git commands
- Skip running tests unless explicitly listed as a phase task
## Project info
- Project: ${config.ideaName}
- Description: ${config.ideaDescription}
- Project directory: ${config.projectDir}`;
}
function buildFinalizePrompt(config: BotConfig): string {