How we built it We built XChannel as a comprehensive VR football game with AI voice control and hardware-integrated foot tracking using Unity 3D. The core architecture consists of:
- VR Framework: Built on Unity with BNG VR Interaction Framework for hand tracking and VR controls
- Hardware Integration: Custom IMU sensor shoes (FoottrollerNet/FoottrollerCtrl) providing real foot movement tracking including: • Right/Left foot heading and tilt angles • Touch state detection (TSLF/TSRF) for kick detection • UDP-based real-time communication between hardware and game
- Network Multiplayer: Implemented using Mirror networking framework for real-time multiplayer football gameplay
- AI Voice Assistant: Integrated Firebase AI (Gemini 2.5-flash) for natural language processing of voice commands
- Audio Pipeline: Custom audio recording system that captures WAV files and sends them to AI for analysis
- World System: Cesium for Unity integration allowing players to teleport to real-world locations
- Physics-Based Football: Combination of VR hand controls and IMU foot tracking for realistic football mechanics The system works by combining physical foot movements detected by IMU sensors with VR hand controls, voice commands processed through AI, and multiplayer networking for a complete football simulation. Challenges we ran into
- Hardware-Software Integration: Synchronizing IMU sensor data from physical shoes with Unity's physics system required custom UDP networking protocols
- Real-time Sensor Data Processing: Managing continuous streams of heading, tilt, and touch state data while maintaining 60fps VR performance
- Multi-Input Coordination: Balancing input from IMU shoes, VR controllers, and voice commands without conflicts or latency
- Network Synchronization: Ensuring foot tracking data, ball physics, and player interactions stay synchronized across multiple VR clients
- Sensor Calibration: Implementing reliable calibration systems for IMU sensors to account for different player orientations and play styles
- UDP Communication Reliability: Managing connection timeouts, reconnection logic, and heartbeat protocols for stable hardware communication
- AI Response Integration: Creating seamless integration between voice AI responses and hardware-controlled gameplay actions What we learned
- Hardware-VR Integration: Gained deep experience in bridging physical sensor hardware with immersive VR environments
- IMU Data Processing: Understanding sensor fusion, angle calculations, and real-time motion tracking in game contexts
- UDP Networking: Mastered low-latency communication protocols for time-sensitive hardware input
- Multi-Modal Interaction Design: Designing intuitive interfaces that combine voice, hand, and foot inputs seamlessly
- Sensor Calibration Algorithms: Developing user-friendly calibration systems for varying physical setups
- Real-time Data Streaming: Optimizing continuous sensor data processing without impacting VR performance
- Cross-Platform Hardware Support: Managing hardware compatibility across different VR platforms and operating systems What's next for XChannel
- Advanced Foot Mechanics: Implement more sophisticated kick detection algorithms using machine learning on IMU data
- Haptic Feedback Integration: Add tactile feedback to the shoes for better immersion during ball contact
- Player Biometrics: Incorporate additional sensors for heart rate and fatigue monitoring during gameplay
- AI Coaching: Use foot tracking data combined with AI to provide personalized football technique coaching
- Tournament Analytics: Advanced statistical analysis of player movements and performance metrics
- Wireless Optimization: Develop more efficient wireless communication protocols for reduced latency
- Multi-Sport Expansion: Extend the foot tracking system to other sports like soccer, basketball, and martial arts What languages, frameworks, platforms, cloud services, databases, APIs, or other technologies did you use? Language: C# Frameworks: • Unity Mirror (networking) • BNG VR Interaction Framework (VR controls) • Firebase AI SDK (voice processing) • Custom UDP Communication Framework (hardware integration) Hardware: • IMU Sensor Shoes (custom hardware) • Accelerometer and Gyroscope sensors • Touch/pressure sensors for kick detection • Wireless communication modules Platforms: • Unity 3D Engine • Oculus VR Platform • .NET Framework 4.7.1 • Custom embedded systems (shoe controllers) Cloud Services: • Firebase AI (Google Gemini 2.5-flash) • Google Cloud Vertex AI Databases/APIs: • Cesium 3D Tiles (3D world data) • Cesium for Unity (geospatial API) Communication Protocols: • UDP networking (real-time sensor data) • TCP networking (Mirror multiplayer) • Wireless sensor communication Other Technologies: • Mirror Networking (multiplayer) • Unity Audio System • PCM Audio Encoding • Real-time motion tracking • IMU sensor fusion algorithms • Physics-based ball mechanics • Geospatial coordinate systems • Hardware-software bridge systems
