Fix LLM output format inconsistency through prompt engineering

[cubxxw]

🐛 Problem Description

LLM outputs are generating inconsistent JSON formats, causing frontend display issues:

Current Problematic Formats

// Format 1: recommendations wrapper (current issue)
{"recommendations": [{"type": "deepen", "title": "...", "description": "..."}]}

// Format 2: options wrapper  
{"type": "deepen", "options": [{"title": "...", "description": "..."}]}

// Format 3: direct array
[{"type": "deepen", "title": "...", "description": "..."}]

Expected Format

{"type": "deepen", "content": "...", "describe": "..."}
{"type": "next", "content": "...", "describe": "..."}

🔍 Root Cause Analysis

Technical Analysis Confirms:

• ✅ Parsing logic works correctly (tested with all formats)
• ✅ Data flow from splitContentAndOptions → mergeOptions → setOptions is functional
• ❌ Root Issue: Inconsistent LLM output formats requiring continuous technical fixes

📋 Proposed Solution: Prompt Engineering Standardization

Strategy: Fix at the Source

Rather than maintaining complex parsing logic for every possible format variation, standardize the LLM output through improved prompt engineering.

Implementation Plan

1. Update nextStepJsonl.system.zh.j2

• Add strict format constraints
• Remove ambiguous instructions
• Provide clear positive/negative examples
• Enforce field name consistency

2. Enhanced Validation

• Add prompt validation examples
• Test with multiple LLM models
• Ensure consistent output quality

3. Gradual Rollout

• Test new prompt extensively
• Maintain backward compatibility during transition
• Monitor success rates

✅ Expected Benefits

• Stability: Eliminate format inconsistency at source
• Maintainability: Reduce technical debt from format-specific parsing
• Performance: Simpler parsing logic = faster processing
• Reliability: Predictable output format = fewer edge cases

🔧 Technical Implementation

Files to Update

• src/prompt/nextStepJsonl.system.zh.j2 - Primary prompt template
• src/prompt/nextStepJsonl.system.en.j2 - English equivalent
• Add validation examples and test cases

Validation Criteria

• ✅ LLM outputs exactly 6 lines of valid JSONL
• ✅ Each line is a complete JSON object with correct fields
• ✅ No wrapper objects or arrays
• ✅ Consistent field names (content/describe, not title/description)

🎯 Success Metrics

• Zero format parsing failures after implementation
• Consistent 6 options output per generation
• No need for format-specific technical patches

📅 Timeline

• Phase 1: Update prompt templates (2-3 hours)
• Phase 2: Extensive testing with multiple models (1 day)
• Phase 3: Production deployment and monitoring (1 day)

🔗 Related Issues

This addresses the recurring pattern of LLM format inconsistencies and provides a sustainable long-term solution.

---

Priority: High
Type: Enhancement/Bug Fix
Component: Prompt Engineering, LLM Integration

[cubxxw]

✅ Solution B Implementation Complete

Prompt Engineering Standardization has been successfully implemented to solve LLM output format inconsistencies at the source.

🔧 Changes Made

Enhanced Prompt Templates

• Files Updated:
  • src/prompt/nextStepJsonl.system.zh.j2
  • src/prompt/nextStepJsonl.system.en.j2

Strict Format Enforcement

• ✅ Correct Format Examples: Clear visual demonstration of expected JSONL output
• ❌ Prohibited Formats: Explicit examples of banned wrapper patterns
• 🎯 Exact Specifications: Must output exactly 6 lines (3 deepen + 3 next)
• 🚨 Field Name Constraints: Only "content" and "describe" allowed (no title/description)
• ⚡ Pure Output: No wrappers, arrays, code blocks, or explanatory text

📋 Specific Improvements

Visual Format Constraints

✅ CORRECT:
{"type": "deepen", "content": "Part 1: Core Analysis", "describe": "Detailed description..."}

❌ PROHIBITED:
{"recommendations": [{"type": "deepen", "title": "...", "description": "..."}]}
{"type": "deepen", "options": [{"title": "...", "description": "..."}]}
[{"type": "deepen", "title": "...", "description": "..."}]

Enforcement Mechanisms

• Multiple constraint layers with emoji emphasis for visibility
• Explicit prohibition statements for problematic patterns
• Clear final output requirements (6 lines, pure JSONL)
• Enhanced template examples with correct field names

🎯 Expected Results

With these strict prompt constraints, LLM should now consistently output:

• Exactly 6 lines of pure JSONL
• Correct field names (content/describe)
• No wrapper objects or arrays
• No format variations requiring technical parsing fixes

📊 Implementation Status

• ✅ Prompt Templates: Updated both Chinese and English versions
• ✅ Build Verification: Passes without errors
• ✅ Format Validation: Clear examples and constraints added
• ✅ Documentation: Comprehensive format guidance provided
• ✅ Deployment Ready: Changes committed to PR Fix LLM output parsing and ResizeObserver error suppression #47

🔄 Next Steps

1. Testing Phase: Monitor LLM outputs after deployment to confirm format consistency
2. Validation: Verify zero parsing failures with new prompt constraints
3. Monitoring: Track success rate of standardized JSONL generation
4. Optimization: Fine-tune prompt language if any edge cases emerge

The solution addresses the root cause by constraining LLM behavior rather than expanding technical parsing logic, providing a more sustainable long-term fix.

Commit: bb396cb
PR: #47

[cubxxw]

done