Hintify | Devpost

Front Page
Solution to one of our example problems

Inspiration

Now that we are in high school, we usually get the answers to our homework already. The problem is, sometimes, or even most of the time, they don't provide the full solution, so you are left wondering why your work is wrong.

What it does

This is why we decided to make this project, which uses OCR technology to read the math problem, and provide a way to solve the problem.

How we built it

We decided to use more advanced OCR techniques by using computer vision models as classical methods can't read math properly. The python library we chose uses multiple models like yolov8 in the back in order very accurately read text and mathematical equations. We also decided to use GPT3.5 to provide the solution to our problem, as in our experiences, even if GPT is bad at calculating, it can consistently provide the correct steps to solving the problem.

Challenges we ran into

At first, we wanted to use Wolfram Alpha's API to solve the problems, but it's lack of ability to solve word problems made us switch to GPT. Finding a good OCR library was also very hard, as an open source library that can accurately read equations is very rare. We did eventually find one, which is also not very well documented.

The biggest challenge we ran into was displaying the result. We tried many different ways to do so, like converting the markdown/LaTeX to pdf and to png, then uploading the image to the website. That idea was scrapped as the library that we relied on for the pdf transformation was unreliable. We ended up using MathJax to parse the template and render LaTeX equations.

Another issue we had was with GPT formatting, which didn't always use the correct format for LaTeX. This was solved by tuning the prompt.

Accomplishments that we're proud of

We were very proud of finishing this complete and usable product in 12 hours, we tested it thoroughly and it is very consistent with responses.

What we learned

We learned to don't always look at the same path when blocked, always look for different solutions, you might even just find an easier one. We also learned to manage our time while working on a project efficiently.

What's next for Hintify

Firstly, we can use a more advanced OCR model to support more languages, then we can add a follow up prompt option to get further explanation of the problem. We can also filter the input to detect if text actually exists in the image, and then we could always improve on visuals.