Magicmath

Open-source, more powerful Apple Intelligence using a projector. Good for learning Maths!

Comment

Mum! I want Apple Intelligence™ We have Apple Intelligence™ at home!

Inspiration

Apple Intelligence, specifically 'Math Notes' seems like a cool and useful concept, but doesn't support more advanced concepts such as Summations (Σ) and particularly doesn't sit with computer science mathematics, such as λ-calculus. We wanted to make a free, open-source alternative!

What it does

We’ve created an augmented projection layer to enhance the experience of working with whiteboards for mathematics. Think autocomplete, but for maths and logic! Whilst Apple Intelligence can only solve basic expressions, Magicmath is able to understand expressions, and programatically calculate answers for a variety of notations (see our Gallery for some cool examples!).

To ensure this is understandable, and not just for being lazy, we wrap generative, explainable outputs in a TTS layer and play this back to the user as the answer appears.

How we built it

The projector was £10 from facebook marketplace! We also have a USB webcam we point at the projected image. There is a huge diagram explaining the technical process in detail in the gallery, and how we actually accomplish this.

Challenges we ran into

Working with projectors is extremely hard, we did not know what homography matrices were before today! Toby had never used Computer Vision before, so it was a learning curve to get used to opencv and symbolic image pre-processing.

  • Doing image registration in a Hackathon! - Masking what we're projecting to the board from what the user has written was the most difficult part of this project, we ended up using homography matricies (which can help transform irregular trapeziums and parallelograms into rectangles) with masking and waypoints to accomplish this! See our diagram.
  • Segmenting the elements in an expression from the background then joining these to form fuller expressions was a challenge, particularly in vertical cases such as Summations. This was increasingly difficult with the venue change (and the lighting that brought!)
  • Rendering LaTeX to the screen is difficult, and no game engine really supports this easily (that we could find). We build a custom game engine for projector-style-games on top of Tkinter with a basic draw/think cycle.
  • Integrating all of this together under some general architecture was also a challenge in of itself, to get this working we had to use a variety of multithreading techniques.

Accomplishments that we're proud of

We set out to get it to work with a variety of problems, and are proud to achieve this. It works for a variety of notations including lambda-calculus and summations (see our gallery!). We are also proud to have integrated an LLM voice explanation (optional) for each solution - which we think makes it extremely useful! The whole thing is actually really robust, particularly as the webcam dynamically adjusts for the relative position of the projector.

What we learned

Countless, countless things, these are too long to go over in a paragraph so I'll list them here.

  • First and foremost, that projectors are not just weird monitors.
  • Image registration & homography matricies (related to above)
  • Image segmentation and pre-processing
  • Contouring (not the makeup kind), erosion and dilating!
  • Runtime interpreters in python (exec!) and how this can be used with LLMs for runtime code solutions, it's pretty nifty for this, and it's something I want to mess around with in the future.
  • Just generally how hard parsing certain maths problems can be, weird notations like integrals aren't as easy as you'd think, and there aren't really existing models out there.

What's next for Magicmath

Apple takeover.

Built With

Share this project:

Updates