MelaNone | Devpost

Inspiration

One of us recently went through an airport body scanner - the kind where you stand with your arms up and the machine rotates around you. It's really good at detecting even the smallest things we forgot in a pocket so we thought: why isn't this kind of scanning used to help detect melanoma?

What it does

MelaNone is a limb-scanning system designed to help detect melanoma early on. A patient places their arm or leg into the scan chamber, where cameras capture photos of the limb from multiple angles. These images are then stitched together to form a 3D object of the limb.

At the same time, smaller segments of the captured images are streamed to a PyTorch AI melanoma model that detects skin spots then classifies them as likely benign or potentially concerning. Once the analysis has completed, the results are summarized and displayed on our website.

How we built it

Hardware: We resurrected a broken, scrapped, and honestly very sad 3D printer and repurposed it as our scanning platform. The scan chamber was designed on Solidworks then 3D printed. 3 webcams are mounted on the chamber, which is then driven by the stepper. An ESP32 microcontroller both controls the motion and communicates with our server.

Firmware: The ESP32 is coded in C. It sends pulses to the stepper driver to step the motor. The code also communicates error messages and responses to the software through predetermined serial codes.

Software: The website is built using React and Flask then packaged with Vite. The backend is mainly coded in Python. We use PySerial to communicate with our firmware.

The detection system leverages OpenCV for mesh generation and image detection. Subsections of images are passed to a melanoma detection model trained with PyTorch.

Process: To begin the scanning, the user starts the process on the frontend, calling our internal start function. This will send commands over serial to our firmware, prompting it to restart its position, and send updates on each of the predetermined locations. At these location, our software captures several images, before proceeding. The resulting images are combined into OpenCV to generate a 3D mesh for visualization, as well as a flattened UV mesh. To predict potential skin cancer spots, this UV map is passed through a pretrained PyTorch model for melanoma prediction. The predicted results, potential concerns, and generated 3D mesh are passed to the frontend for visualization.

Challenges we ran into

Cameras take a long time to initialize. Some also wouldn't initialize properly at the start, resulting in some.... interesting meshes
The 3D printer we found in the trash bin was a lot of missing parts and had hard-to-find schematics

Accomplishments that we're proud of

We brought the dead back to life! And gave it an even cooler job ;)

What we learned

It's okay to dream big and be really ambitious for a 24 hour hackathon. Even though there were many features that we didn't end up implementing, we had fun and learned a lot along the way.

What's next for MelaNone

We want to reduce the number of cameras needed by mounting a single camera on a gear system. This would decrease the cost whilst also increasing the initializing speed of our program. We also plan to expand our dataset to improve detection accuracy. Long term, we'd love to scale MelaNone into a full-body scanning system for medical clinics.