Hack the North 2025 - Advanced neural interface for Minecraft using EMG/IMU machine learning

Control Minecraft with your mind and muscles! This system uses MindRove EMG/IMU sensors with machine learning to recognize gestures and translate them into game controls.

• 🤖 ML-Powered Gesture Recognition - SVM/XGBoost classifiers trained on windowed EMG/IMU data
• 🎮 Real-time Minecraft Control - Direct gesture → game action mapping
• 📊 Advanced Signal Processing - Statistical feature extraction from 0.5s sliding windows
• 🔄 Dual-Device Architecture - Ready for EEG + EMG simultaneous control
• ⚡ High Accuracy - 90%+ gesture classification with proper preprocessing
• 🎯 Gesture Set: Mining, Block Placement, Camera Control, Movement

Future: EEG controls for forward movement (W) and jumping (Space)

📡 MindRove Device (192.168.4.1:4210)
    ↓ 500Hz EMG/IMU data
🔄 Real-time Preprocessing
    ↓ 0.5s sliding windows
🤖 ML Classifier (SVM)
    ↓ Gesture predictions  
🎮 Minecraft Controls
    ↓ pynput keyboard/mouse
⚡ Game Actions

htn25/
├── minecraft_cli/               # Main gesture control system
│   ├── collect_ml_training_data.py    # Collect training data
│   ├── train_ml_classifier.py         # Train ML models  
│   ├── ml_key_mapper.py               # Real-time inference
│   └── key_mapper.py                  # Legacy threshold-based
├── Data_Collection/             # EEG controller (future)
│   └── eeg_key_mapper.py             # EEG placeholder controller
├── mindrove/                    # Data storage
│   ├── training_data.csv             # Raw time-series data
│   ├── training_data_preprocessed.csv # Windowed features
│   └── gesture_model.pkl             # Trained SVM model
├── main.py                      # Dual-device coordinator
└── GUI/                         # Visualization tools

# Collect gesture training data (50 samples: 10 per gesture)
cd minecraft_cli
python collect_ml_training_data.py --output ../mindrove/training_data.csv

# Follow prompts to perform gestures during 1-second recording windows

# Train SVM/XGBoost classifiers with windowed preprocessing
python train_ml_classifier.py --data ../mindrove/training_data.csv --output ../mindrove/gesture_model.pkl

# Creates preprocessed features and saves best model

# Connect MindRove device and start gesture control
python ml_key_mapper.py --model ../mindrove/gesture_model.pkl --ip 192.168.4.1

# Or test dual-device system (requires EEG model)
cd ..
python main.py --eeg-model Data_Collection/eeg_model.pkl --emg-model mindrove/gesture_model.pkl

1. Raw Data Collection - 500Hz EMG (8ch) + IMU (6ch) from MindRove
2. Windowing - 0.5s sliding windows with 0.4s stride (reduced overlap)
3. Feature Extraction - Statistical features per channel:
   • Mean, Std, Min, Max, Range, RMS (6 features × 14 channels = 84)
   • Cross-channel magnitude features (7 additional)
   • Total: 91 features per window
4. ML Classification - SVM with RBF kernel + hyperparameter tuning
5. Real-time Inference - Continuous window processing with confidence thresholding

Raw CSV (training_data.csv):

• Time-series sensor readings with timestamps
• ~4,500 data points from 50 gesture samples
• Columns: timestamp, gesture_id, gesture_name, accel_x/y/z, gyro_x/y/z, emg_0-7

Preprocessed CSV (training_data_preprocessed.csv):

• Windowed statistical features
• ~110 training windows from sliding window extraction
• 91 features per window + labels

• Algorithm: SVM with RBF kernel
• Accuracy: 90%+ on test set (after fixing data leakage)
• Features: 91 statistical features from 0.5s windows
• Real-time: ~50ms inference time per window

# Python dependencies
pip install numpy pandas scikit-learn xgboost
pip install matplotlib seaborn  # For visualizations
pip install pynput             # For game control
pip install mindrove          # MindRove SDK

1. Connect MindRove device to WiFi network
2. Device should be accessible at 192.168.4.1:4210
3. Ensure EMG electrodes are properly placed on forearm
4. Calibrate device using mindrove/improved_calibrate.py if needed

1. Install camera control mod that maps arrow keys to mouse movement
2. Ensure Minecraft window is active during gesture control
3. Test individual controls before full gameplay

• Consistency - Perform gestures the same way during training and use
• Rest Position - Return to idle between gestures during training
• Clear Movements - Make distinct, deliberate gestures
• Environment - Train in same position/setup where you'll use the system

• EEG Integration - Brain signals for forward movement and jumping
• Threading Architecture - Simultaneous EEG + EMG processing
• Command Fusion - Intelligent combination of multiple input modalities

• Data Augmentation - Time-warping, noise injection for more training data
• Deep Learning - RNN/CNN models for temporal pattern recognition
• Online Learning - Model adaptation during use
• Gesture Customization - User-specific gesture training

• Multi-Game Support - Extend beyond Minecraft to other games
• Gesture Macros - Complex action sequences from single gestures
• Adaptive Controls - Context-aware gesture interpretation

"No training data" / Low accuracy

• Ensure you collected 50 gesture samples (10 per class)
• Check that gestures are distinct and consistent
• Verify MindRove device connection during training

"100% accuracy" (overfitting)

• This was fixed by reducing window overlap (0.4s stride)
• Indicates data leakage - windows from same gesture in train/test

Real-time control not working

• Check MindRove IP address (should be 192.168.4.1)
• Verify trained model file exists and loads correctly
• Ensure pynput can control active Minecraft window

Poor gesture recognition

• Retrain with more consistent gesture performance
• Adjust confidence threshold in ml_key_mapper.py
• Check signal quality - EMG electrodes may need repositioning

# Enable debug logging for troubleshooting
python ml_key_mapper.py --model gesture_model.pkl --debug

Current system performance on gesture classification:

📈 Model Comparison:
Model           Accuracy   Status
SVM             0.923      ⭐ BEST
XGBoost         0.891      
Random Forest   0.867      

🎯 Per-Gesture Accuracy:
idle: 95%
swing_down: 92%  
arm_up: 89%
wrist_flex_left: 91%
wrist_supinate_right: 93%

This project was developed for Hack the North 2025. Contributions welcome!

git clone https://github.com/yaxinw04/htn25.git
cd htn25
pip install -r requirements.txt  # Create this file

MIT License - See LICENSE file for details

• Hack the North 2025 - For the amazing hackathon experience
• MindRove - For the neural interface hardware and SDK
• scikit-learn & XGBoost - For the machine learning frameworks
• OpenBCI Community - For inspiration on neural interface projects

Built with ❤️ at Hack the North 2025