plan2code-bot: replace auto-responder with LLM-as-judge evaluation

Bot now acts as an authentic QA agent instead of rubber-stamping every
question and step.

- intelligent-responder answers AskUserQuestion via LLM using current
  observations (tools used, files created, errors) instead of keyword
  matching; auto-responder removed.
- evaluator runs after each step with step-specific criteria
  (prompts/evaluation-criteria.ts), produces 0-100 score plus
  strengths/weaknesses/critical issues. Avg <60 blocks finalization.
- observation-collector captures tool_use, file writes, errors, and
  question reasoning; session-runner falls back to scraping tool_use
  blocks when canUseTool doesn't fire and dedupes both sources.
- Bot writes BOT-EVALUATION.md and BOT-NOTES.md for metrics analysis.
- Idea generator: 12 categories instead of CLI/web-app coin flip,
  stronger seed adherence, less developer-tool bias.
- Init step now writes a minimal AGENTS.md stub instead of running
  the full init skill, avoiding hallucinated architecture before plan.
- Implement step uses Read/Write/Edit/Glob/Grep directly instead of
  a Skill sub-session that produced no visible tool observations.
- bin: add --help, accept --idea="value" form, strip surrounding
  quotes; evaluator maxTurns 3 -> 30; warn when parser misses SCORE.
This commit is contained in:
2026-05-11 13:14:28 -07:00
parent 9d1104d105
commit c2579a7b6a
15 changed files with 1434 additions and 254 deletions
+20 -4
View File
@@ -22,13 +22,29 @@ function parseIdea(text: string): IdeaResult {
}
export async function generateNewAppIdea(seed?: string): Promise<IdeaResult> {
const category = Math.random() > 0.5 ? 'CLI tool' : 'small web app';
const categories = [
'CLI tool',
'single-page web app',
'REST API service',
'browser extension',
'interactive data visualization dashboard',
'terminal-based game',
'real-time web app (using WebSockets)',
'static site generator or theme',
'browser-based game',
'desktop utility (using Electron or Tauri)',
'chat bot or conversational tool',
'automation script or workflow tool',
];
const category = categories[Math.floor(Math.random() * categories.length)];
const seedClause = seed
? `\n\nUse this as inspiration for the idea: "${seed}"`
? `\n\nThe user provided this seed for inspiration. Stay closely aligned with the theme and intent of the seed — build on it, don't ignore it:\n"${seed}"`
: '';
const prompt = `Generate a random, creative idea for a ${category} that a developer might build as a side project. The project should be achievable in a single coding session (1-2 hours) and should be interesting but not overly complex.${seedClause}
const prompt = `Generate a random, creative idea for a ${category}. The project should be achievable in a single coding session (1-2 hours) and should be interesting but not overly complex.
IMPORTANT: Be creative and diverse with your ideas. Avoid defaulting to developer-centric tools (git analyzers, code formatters, repo scanners, etc.) unless the category specifically calls for it. Think about ideas that would appeal to a broad audience — productivity, entertainment, education, health, finance, art, music, social, cooking, travel, fitness, etc.${seedClause}
Respond in EXACTLY this format (no other text):
NAME: <kebab-case-project-name>
@@ -38,7 +54,7 @@ DESCRIPTION: <2-3 sentence description of what the app does, its key features, a
prompt,
options: {
maxTurns: 1,
systemPrompt: 'You are a creative software project idea generator. Respond only in the exact format requested.',
systemPrompt: 'You are a wildly creative project idea generator. You come up with surprising, fun, and diverse software project ideas spanning many domains — not just developer tools. Respond only in the exact format requested.',
},
});