MindScape | Devpost

Inspiration

Our project, MindScape, draws inspiration from the ancient Greek memory technique known as the "Mind Palace," which takes advantage of our natural spatial awareness to store and recall unfamiliar information. This technique is normally hard to share and communicate but with the latest advancements in 3D reconstruction, it's now possible to teach this powerful technique in a much more digestible way. We help users create a virtual "memory palace" by having them take videos of familiar rooms which we then give to our machine-learning models to generate Gaussian Splats. They can use this to retain whatever piece of information they want. We then have a built-in LLM that evaluates whether you remembered what you set out to store in memory.

I used to think that I had a poor memory, and I realized that mentality made me feel insecure about learning. But discovering these mnemonic techniques, which take advantage of what we humans are already good at remembering, helped me get past that. It was quite empowering to see memory champions, pull off insane things like memorizing the name of an entire of everyone in a conference hall or the ordering of an entire deck of cards in less than a minute, and then it turning out that anyone can learn to do it. But it's difficult to have a specific technique like mind palace scale, without having a technology like this, which is why it's exciting to make it possible.

What it does

MindScape allows users to capture 3D scans of physical environments and associate them with real-world locations. These 3D environments, built with Gaussian Splatting, serve as virtual memory palaces where users can embed personal memories or information using the Memory Palace technique. The user can then define what they're learning and have it associated with different objects in the room. And during test time, we have an LLM that evaluates whether the user remembered what they set out to remember.

How we built it

3D Reconstruction with Gaussian Splatting: We captured video footage of the environment and applied Gaussian Splatting to generate sparse point cloud data. Using Google Cloud Platform's Compute Engine with an L4 GPU and CUDA-accelerated runtimes, we transformed real-world environments into detailed 3D models.
Map Integration: Our project connects these 3D models with real-world locations using geolocation services, allowing users to scan QR codes at physical locations and view their corresponding 3D versions.
Memory Palace Functionality: We implemented an overlay of the Mind Palace technique, where users can add memories, notes, or information within these 3D environments, turning real-world places into interactive memory storage spaces.
User Interface: We designed a simple yet functional interface to ensure that users, even without technical expertise, can easily capture, explore, and embed memories within the 3D spaces.

Challenges we ran into

We faced several challenges throughout the development process:
Long training time even with L4 GPU. Building the data pipeline from the server collecting the result from VM, and writing scripts to programmatically control the virtual machine.
Rendering a 3D model iPhone: due to performance optimization, limited processing power, and ensuring compatibility across various iOS devices.
Backend to Frontend Loading: Loading successfully rendered models from the backend to the frontend took a considerable amount of time, causing delays in delivering a smooth user experience.

Accomplishments that we're proud of

We successfully created a platform that combines advanced 3D rendering with an intuitive interface for capturing and interacting with memory palaces. Our integration of geolocation and QR code functionality to trigger immersive 3D environments in real-world locations was a significant achievement, providing a unique and engaging experience.

What we learned

Through this project, we deepened our understanding of 3D modeling, rendering techniques like Gaussian Splatting with using VM to speed up training time, and how to combine spatial memory with interactive technologies. We also learned a lot about designing user-friendly interfaces that can make complex technology accessible to all.

What's next for MindScape

We plan to continue improving the resolution and accuracy of the 3D models while optimizing the system for mobile devices. We also aim to expand customization features for users when creating and organizing their memory palaces. Additionally, we’re exploring potential partnerships with museums, historical sites, and educational institutions to create interactive, immersive experiences for learning and exploration.