Multimodal AI Chat App with Dynamic Routing

VT.ai is a multimodal AI chat application designed to simplify interaction with different AI models through a unified interface. It employs vector-based semantic routing to direct queries to the most suitable model, eliminating the need to switch between multiple applications and interfaces.

Documentation

• Multi-Provider Integration: Unified access to models from OpenAI (o1/o3/4o), Anthropic (Claude), Google (Gemini), DeepSeek, Llama, Cohere, and local models via Ollama
• Semantic Routing System: Vector-based classification automatically routes queries to appropriate models using FastEmbed embeddings, removing the need for manual model selection
• Multimodal Capabilities: Comprehensive support for text, image, and audio inputs with advanced vision analysis
• Image Generation: GPT-Image-1 integration with support for transparent backgrounds, multiple formats, and customizable quality parameters
• Web Search Integration: Real-time information retrieval with source attribution via Tavily API
• Voice Processing: Advanced speech-to-text and text-to-speech functionality with configurable voice options and silence detection
• Reasoning Visualization: Step-by-step model reasoning visualization with the <think> tag for transparent AI decision processes

Multiple installation methods are available depending on requirements:

# Standard PyPI installation
uv pip install vtai

# Zero-installation experience with uvx
export OPENAI_API_KEY='your-key-here'
uvx vtai

# Development installation
git clone https://github.com/vinhnx/VT.ai.git
cd VT.ai
uv venv
source .venv/bin/activate  # Linux/Mac
uv pip install -e ".[dev]"  # Install with development dependencies

Configure API keys to enable specific model capabilities:

# Command-line configuration
vtai --api-key openai=sk-your-key-here

# Environment variable configuration
export OPENAI_API_KEY='sk-your-key-here'  # For OpenAI models
export ANTHROPIC_API_KEY='sk-ant-your-key-here'  # For Claude models
export GEMINI_API_KEY='your-key-here'  # For Gemini models

API keys are securely stored in ~/.config/vtai/.env for future use.

The application provides a clean, intuitive interface with the following capabilities:

1. Dynamic Conversations: The semantic router automatically selects the most appropriate model for each query
2. Image Generation: Create images using prompts like "generate an image of..." or "draw a..."
3. Visual Analysis: Upload or provide URLs to analyze visual content
4. Reasoning Visualization: Add <think> to prompts to observe step-by-step reasoning
5. Voice Interaction: Use the microphone feature for speech input and text-to-speech output

Detailed usage instructions are available in the Getting Started Guide.

The documentation is organized into sections designed for different user needs:

• User Guide: Installation, configuration, and feature documentation
• Developer Guide: Architecture details, extension points, and implementation information
• API Reference: Comprehensive API documentation for programmatic usage

VT.ai offers two distinct implementations:

• Python Implementation: Full-featured reference implementation with complete support for all capabilities
• Rust Implementation: High-performance alternative with optimized memory usage and native compiled speed

The implementation documentation provides a detailed comparison of both options.

The Models Documentation provides detailed information about model-specific capabilities and configuration options.

VT.ai leverages several open-source projects to deliver its functionality:

• Chainlit: Modern chat interface framework
• LiteLLM: Unified model abstraction layer
• SemanticRouter: Intent classification system
• FastEmbed: Efficient embedding generation
• Tavily: Web search capabilities

The application architecture follows a clean, modular design:

• Entry Point: vtai/app.py - Main application logic
• Routing Layer: vtai/router/ - Semantic classification system
• Assistants: vtai/assistants/ - Specialized handlers for different query types
• Tools: vtai/tools/ - Web search, file operations, and other integrations

Contributions to VT.ai are welcome. The project accepts various types of contributions:

• Bug Reports: Submit detailed GitHub issues for any bugs encountered
• Feature Requests: Propose new functionality through GitHub issues
• Pull Requests: Submit code improvements and bug fixes
• Documentation: Enhance documentation or add examples
• Feedback: Share user experiences to help improve the project

Follow these steps to set up a local development environment:

• Python 3.11 or higher
• uv package manager (recommended) or pip
• Git

1. Clone the repository

   git clone https://github.com/vinhnx/VT.ai.git
   cd VT.ai

2. Set up Python virtual environment

   # Using uv (recommended)
   uv venv
   
   # Or using standard Python venv
   python -m venv .venv

3. Activate the virtual environment

   # On macOS/Linux
   source .venv/bin/activate
   
   # On Windows
   .venv\Scripts\activate

4. Install development dependencies

   # Install the package in editable mode with development dependencies
   uv pip install -e ".[dev]"
   
   # Or if using pip
   pip install -e ".[dev]"

5. Set up environment variables Copy the .env.example file to create your own .env file:

   cp .env.example .env

   Then edit .env to add your API keys:

   # Edit the .env file with your preferred editor
   nano .env
   # or
   code .env  # if using VS Code

6. Run the application in development mode

   # Using chainlit (recommended for development)
   chainlit run vtai/app
   
   # The application will be available at http://localhost:8000

7. Run tests to verify your setup

   # Run all tests
   pytest
   
   # Run tests with coverage
   pytest --cov=vtai
   
   # Run specific test categories
   pytest tests/unit/
   pytest tests/integration/

For convenience, you can use the run_app.py script which provides an all-in-one solution for running the application:

# Direct Python execution (will initialize but not start server)
python run_app.py

# With chainlit for full interactive development
chainlit run run_app.py -w

# Or simply use the main app module
chainlit run vtai/app

The run_app.py script serves as a wrapper that:

• Sets up the proper Python path
• Initializes the application with all necessary components
• Provides a single entry point for development
• Handles environment setup automatically

If you prefer to use pip instead of uv:

# Clone and navigate to the project
git clone https://github.com/vinhnx/VT.ai.git
cd VT.ai

# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate  # On macOS/Linux
# .venv\Scripts\activate  # On Windows

# Install in development mode
pip install -e ".[dev]"

# Run the application
chainlit run vtai/app

Common commands you'll use during development:

• chainlit run vtai/app - Run the development server
• pytest - Run all tests
• pytest -x - Run tests and stop on first failure
• pytest --cov=vtai - Run tests with coverage report
• ruff check . - Check code for linting issues
• ruff format . - Format code according to project standards
• uv pip install -e ".[dev]" - Reinstall after dependency changes

Issue: ModuleNotFoundError when running the application

• Solution: Make sure you've activated your virtual environment and installed the package in editable mode:

  source .venv/bin/activate
  pip install -e .

Issue: Permission denied when creating virtual environment

• Solution: Make sure you have write permissions in the project directory:

  chmod 755 .
  uv venv

Issue: Chainlit not found

• Solution: Install chainlit separately or make sure you've installed all dependencies:

  pip install chainlit
  # or
  pip install -e ".[dev]"

Issue: API keys not being recognized

• Solution: Verify your .env file is in the correct location and contains properly formatted keys:

  # Check if .env file exists in project root
  ls -la .env
  
  # Verify content format
  cat .env
  # Should contain: OPENAI_API_KEY=your_actual_key_here

Issue: Application fails to start with port binding errors

• Solution: Check if another process is using the default port:

  # Find processes using port 8000
  lsof -ti:8000
  # Kill the process if needed
  kill $(lsof -ti:8000)

Issue: Slow startup times

• Solution: You can enable fast startup mode by setting an environment variable:

  export VT_FAST_START=1
  chainlit run vtai/app

Issue: Dependency conflicts

• Solution: Create a fresh virtual environment and reinstall dependencies:

  rm -rf .venv
  python -m venv .venv
  source .venv/bin/activate
  pip install -e ".[dev]"

Quality is maintained through comprehensive testing:

# Run the test suite
pytest

# Run with coverage reporting
pytest --cov=vtai

# Run specific test categories
pytest tests/unit/
pytest tests/integration/

VT.ai is available under the MIT License - See LICENSE for details.