Greenroom

Finally, AI that reads the room because you show it how

Comment

Inspiration

AI assistants are used by salespeople, teachers, pastors, therapists, consultants, and more, but they always feel all or nothing—either the AI takes full control, or the user has to micromanage every detail. There’s no middle ground where AI can think ahead while remaining adaptable.

Thinking ahead while talking is hard—whether you’re a teacher guiding a discussion, a doctor explaining options to a patient, or a consultant steering a meeting. AI should help, but most systems either force rigid automation or demand constant manual input. We wanted a solution where:

  • AI plans ahead dynamically but lets users override and steer when needed.
  • AI’s thought process is visible instead of hidden inside a black-box model.
  • The interface feels intuitive and real-time, without requiring perfect prompts.

What It Does

Greenroom is a real-time, visual AI conversation control system. Instead of just responding to input, the AI:

  • Plans ahead based on the conversation flow.
  • Visualizes its own thought process on a whiteboard (tldraw).
  • Lets the user intervene—adjusting tone, context, and direction on the fly.
  • Updates dynamically when users make changes, adapting in real-time.

For example, if a therapist is guiding a session, they can see the AI’s planned direction and adjust if the conversation needs a different tone. If a salesperson realizes the client needs a new approach, they can steer the AI’s strategy mid-call. The AI doesn’t just react—it actively co-directs the conversation while remaining flexible.

How We Built It

  • Frontend: React, TypeScript, Vite, Tailwind
  • Backend: Bun.
  • AI & Voice Systems:
    • OpenAI GPT models for conversation planning and response generation.
    • ElevenLabs API for real-time voice synthesis.
    • WebSockets for smooth, lag-free media streaming.
    • Web Speech API for live speech recognition.
  • Whiteboard Logic: - We love tldraw!!!
    • AI-generated “thought nodes” show conversation direction.
    • Users can adjust tone, context, and structure dynamically.
    • The AI updates its strategy in response to changes.

Challenges We Ran Into

  1. Keeping AI flexible without making it unpredictable—it needs to plan ahead but stay responsive when the user intervenes.
  2. Real-time synchronization—AI updates, voice output, and whiteboard visuals all needed to stay in sync without lag.
  3. Designing an intuitive UI—people don’t naturally think in prompts, so we had to make sure the canvas felt effortless to use.
  4. Managing AI-generated pacing—AI needed to anticipate conversation flow without forcing rigid structure.

Accomplishments That We’re Proud Of

  • Visual AI control system—we moved beyond text prompts into spatial, interactive control.
  • Seamless real-time updates—AI adjusts its responses and thought process instantly when the user makes a change.
  • Fluid integration of speech, text, and visual planning—all conversation elements work together naturally.
  • Balance between automation and control—AI directs, but users always have visibility and intervention power.

What We Learned

  • Real-time AI planning is hard—it needs to think ahead without overcommitting to a single path.
  • Users don’t just want control—they want visibility. Even if they don’t intervene, seeing the AI’s plan builds trust and understanding.
  • Text-based prompting has limits—a whiteboard approach makes AI adjustments much more intuitive.
  • Tone and pacing matter more than expected—people don’t just want the right response, they want the right delivery.

What’s Next for Greenroom

  • Integrating more knowledge sources—allowing users to bring in external docs, research, and structured data.
  • More granular AI steering—expanding how users can guide tone, pacing, and structure.
  • Collaborative mode—multiple users being able to co-direct a conversation in a team setting.
  • Expanding AI personas—letting users train the AI on specific conversational styles.
  • Beyond conversations—exploring applications in scriptwriting, guided learning, and AI-assisted interviews.

Greenroom isn’t just about fixing AI assistants—it’s about rethinking how we interact with AI in real-time.

Built With

Share this project:

Updates