Inspiration
Back at the start of Fall Quarter, in-person classes were on the cusp of fully returning, yet many classes were still adopting a hybrid format. Professors were lecturing in person, while many students attended online or watched recordings. However, those recordings required extra staff or personnel to operate cameras and technology. This meant that not every classroom was eligible for hybrid lectures due to a severe shortage of staff. Fishbowl is an all-in-one solution that allows professors to seamlessly record their in-person lectures with just their phones.
What it does
Fishbowl is a web app that can be used on the professor's computer or laptop to create virtual "lecture rooms" for each of their classes. They can connect their smartphone to their laptop to use a camera which can be placed in the back of the classroom. Students can connect to this web app themselves to add classes to their schedule and plan their workload with a dynamic calendar. They can enter virtual lecture rooms set up by the professors to view the live lecture, as well as engage in an interactive chat with all the other students. Students can upvote questions they think are important and want to raise attention to.
How we built it
The tracking software with the phone's camera is built using Python and OpenCV, utilizing Histograms of Oriented Gradients to detect people within the frame. A box is drawn around the subject, and by using their relative position among the frame, the camera is panned to place them in the center. All of these calculations were done using Numpy in Python.
The web app was initially designed using Figma to layout the designs and workflow. Firebase was used in the backend to set up user authentication and store user accounts. React and React Router were utilized to implement the proposed designs and create a seamless experience for users. The Ably API allowed us to create a live chat in our web app so that multiple users could connect to it and interact with each other.
Challenges we ran into
One of the first challenges we ran into was the fact that our outputted video steam was only around 10-15 FPS. This made the algorithm extremely inefficient and the tracking was quite slow as there was simply less data to use as input. To fix this, we implemented multithreading using the "imutils" Python library to speed up the computation significantly. We also made several other optimizations such as converting the input video to greyscale, compressing it, and only performing calculations every three seconds so the computations aren't as intensive. All of these changes gave us an average FPS of 40.
Another issue we had was the implementation of our actual tracking algorithm. Initially, the camera immediately panned to the next frame with no transition. This resulted in a very jarring switch from one frame to the next. To combat this, we incremented the pixels in tiny adjustments until the subject reached the center of the frame. This allowed for much smoother transitions and the video was much clearer to follow.
The biggest challenge with the front-end application was trying to embed the Python outputted video into the web app. In order to do this, we screen captured the frame in Python and put it on a YouTube live stream, which was then embedded into the website.
Accomplishments that we're proud of
We’re extremely proud of creating an object tracking algorithm to smoothly detect the movements of a person, and allowing professors and students to have an effective, hybrid learning experience. With this, we hope that this opens more opportunities and pathways to optimize online learning.
It was also our first experience working with OpenCV, so connecting the object tracking software with our React web app was a huge task.
What we learned
We learned how to work with OpenCV and gained a better understanding of Python programming. Additionally, we learned how to connect React web apps to different libraries, such as OpenCV, in order to develop more dynamic and versatile applications.
What's next for Fish Bowl
In the future, we hope to add more interactive elements to better resemble a more complete classroom environment. This includes adding polls, being able to see written answers to questions, and built-in quizzes.
We also envision developing a mobile app to generate a QR code that allows users to connect their phone camera wirelessly.
Log in or sign up for Devpost to join the conversation.