Tinklebot | Devpost

Mapping Image 1
Mapping Image 2
Mapping Image 3
Mapping Image 4

Inspiration

Amazon warehouses can be busy places, and can often be constantly changing environments with lots of people, packages, and other robots passing through. We created Tinklebot as a centralized system capable of mapping out a warehouse environment and its obstacles and managing a collection of robots that can dynamically change their pathing to avoid collisions in real time.

What it does

Our system relies on overhead cameras to pinpoint moving robots and their destinations while constantly assessing the environment to optimize paths while avoiding new and existing obstacles. To do this, we use fiducial markers called AprilTags on the robots and at the destination to determine the robot's positioning and rotation. This helps plot the best path using breadth-first search and custom buffer zones around obstacles to avoid collisions. The obstacle detection works through a custom image processing pipeline that first detects start and end locations, then filters out anything matching the color of the background or floor, which are then marked as free areas. It then undergoes two more rounds of filtering to extract noise, and Gaussian blur to eliminate any stray pixels. Finally we obtain a binary classification of free areas and obstacles. The optimal path and movement vectors are calculated on our main server and then the ESP32 directly connected to our server wirelessly communicates them via ESP-Now protocol to the ESP32 attached to the robot.

How we built it

Hardware: We made sure to closely simulate hardware-scaled technologies for our project. Our system involved 3 main parts. First was the overhead camera system. Designed to be compatible with high-quality cameras, we used an iPhone 14 Pro camera, held overhead to the field simulating the warehouse. Next was the wireless ESP32 communication system. This served as the middleman between the camera and car systems, sending messages between the two. Lastly was the most complex system, the car. We chose a mecanum-drivebase for its omni-directional movement which provided practical advantages with its strafing and lack of need to change heading, allowing for quick and flexible movement within a tightly constricted warehouse. The drivetrain was powered by 4 DC motors, and controlled by a ULN2003 motor driver and ESP32 chip.

Software: We used a Python backend with the openCV and aprilTags libraries to detect fiducial markers and generate the image processing pipeline to detect obstacles. Additionally, we used C to program our ESPs to connect to the server, transmit data, and handle motor powering logic.

Challenges we ran into

Our biggest issue was the connection between the server and the robots, as we did not know how to transmit the Python server data. However, we leveraged the ESP-Now connection protocol between ESPs to transfer the movement vector data. We initially experienced difficulty detecting obstacles and AprilTags in our environment due to inconsistent lighting conditions such as shadows, but eventually developed an image processing pipeline capable of calibrating to any environment.

Accomplishments that we're proud of

We are proud of our pathfinding algorithm and obstacle avoidance, as we can change our path in real time in response to new obstacles in the way. The system can even detect rolling objects that appeared in frame for a split second in real time! In conjunction with our server that sends vector data to the robots, we have developed a prototype for an efficient and safe solution to warehouse delivery that can be quickly scaled up in the future.

What we learned

With this hackathon being the first time we worked with an ESP32, we learned much about its peer-to-peer communication protocol, and physical limitations to controlling hardware components like DC motors. Additionally, working with the April Tag fiducials, we initially thought our camera would be able to recognize them from far distances and very accurately. However, we learned about the necessity to implement image processing and filtering to make viewing the fiducials easy, and to single out obstacles for the car to avoid.

What's next for Tinklebot

Our current prototype demonstrates Tinklebot’s capabilities with one robot, but this idea can be developed into a swarm of warehouse bots working in conjunction with one another to deliver packages while simultaneously avoiding each other. Additionally, Tinklebot can be scaled up in size from its prototype to carry larger packages and operate within a larger environment, using a network of cameras rather than one singular central camera.

Note: Submission for Amazon Sponsor and Beginner Tracks