Hey, Chef!

Cooking, Conversationally

Comment

Inspiration

As college students, cooking isn't just about making more delicious meals. Learning to cook helps students improve their nutrition and prepare themselves for learning.

"Students who eat regular, healthy meals are less likely to be tired, are more attentive in class, and retain more information" — (NYT, 2017)

We wanted to help new cooks find their footing in the kitchen by building a conversational cooking assistant that would be more responsive and attentive.

What it does

Hey, Chef is a conversational voice agent built to be a great companion for the kitchen. Chef also has the standard functionality of traditional conversational agents: Web Search, human-like conversational capability, interruptibility.

Visual and Audio data is collected and Chef converts the multimodal input into text, building a consistent "understanding" of its environment.

With realtime understanding, Chef is helpful in realtime. Users can focus on the task before them, with Chef as a supportive assistant that proactively aids in decision-making, task execution, and environment interaction. Chef’s goal is to seamlessly blend into the user’s workflow.

How we built it

Hey, Chef is composed of 3 key components.

  1. We ingest realtime video and audio, enabling multimodal context and understanding.
  2. Our agent is powered by LiveKit which serves as our conversational layer and gives us industry-standard speech capabilities (interruptibility, conversation)
  3. We build a semantic understanding of the world by using the OpenAI API to analyze multimodal data.

Architecture Diagram

Challenges we ran into

  1. Working with new frameworks was tough
  2. Designing an architecture for more declarative events

Accomplishments that we're proud of

  1. Pushed in all the push/pop bubble fidget toys
  2. Integrated function calling and external services

What we learned

  1. Working with conversational AI
  2. Velveeta is pretty sticky
  3. Blue yerba is not as good as green yerba

What's next for hey-chef

  1. We hope to further extend its capability with better reasoning and processing speeds
  2. Improve the long term memory as well with external providers.

Built With

Share this project:

Updates