IoT Wiimote | Devpost

Inspiration

My initial inspiration came from how the Nintendo Wii is controlled with its Wiimote motion controller. I imagined having a Wiimote that could always tell where it is pointing, as opposed to the requirement of a sensor bar with the original Wiimote, and how it could allow you to easily control devices like lights and speakers with simple gestures. That's when I realized that the highly precise sensors and cameras that modern phones have could allow for that kind of position tracking.

What it does

The user is able to scan rooms with their phone and then manually mark the positions of IoT devices. After that has been completed, they can point their phone at the IoT devices and make gestures with the phone, such as tilting it up or down. When the program has detected the phone pointing at the device for a second, it starts attempting to interpret what gesture is being performed with the phone. Each device is configured with specific gestures that correspond to specific actions, such as a downwards tilt Lowering the brightness of a light. When the device stops pointing at the device, the program returns to waiting until the phone points at a device again.

How we built it

The program was never finished, but the goal was to have a MATLAB program take in inertial sensor measurements and photos from the cameras being streamed from a phone using the MATLAB Mobile app. The program would use this data with a viSLAM (visual-inertial simultaneous localization and mapping) algorithm that would be able to determine the exact location and orientation of the phone. The program would keep track of where the phone is pointing at all times, and it would detect when it is pointed at the prespecified position of a IoT device for more than a second. At that point the program would begin to try classifying the gestures being made with the phone and apply them to the devices to control them.