ZoomPlus

ZoomPlus is an algorithm that improves video quality in real-time, especially in a classroom setting where the content on the chalkboard sometimes get blurred due to Internet connection.

Inspiration

One of the dramatic changes we have seen during COVID-19 is millions of people rely on video conferences for daily meetings and communications. However, there are many concerns and challenges raised regarding the usage of online conference platforms. For example, the low image quality of handwriting notes on the chalkboard has bothered many students while watching professors' lectures. Similar issues happened for people working from home as well, unable to see colleagues' notes or handwriting when screen share was disabled.

What it is

ZoomPlus is an algorithmic model for improving video quality for online conference platforms. It is simple to use: users only need to click to run the program while they are watching the videos, ZoomPlus will automatically improve the video quality on your screen. Comparison in the Image Gallery (as well as in the video) shows that ZoomPlus can make a significant improvement on reconstructing the writing on the chalkboard.

How we built it

ZoomPlus is constructed based on the Residual Dense Network (RDN) and Image Sharpening. The RDN model is for image super-resolution, which takes a low-resolution input image and reconstructs it to a high resolution output image. For the RDN model, we use transfer learning; this is done by fine-tuning (using a Navidia Telsa P100 GPU on Google Cloud) a pretrained RDN model on our self-made dataset, so that the model can be more capable of reconstructing content from the chalkboard. The dataset consists of 4,000 png files of clear handwriting on chalkboard obtained from online open courses. Every original high-resolution image will be the expected output, and the corresponding 2x smaller downsampled image will be the corresponding input. 3600 pairs were randomly selected as the training set and the remaining 400 were used as the validation set. The Image Sharpening is implemented by using OpenCV filter2D function.

Challenges we ran into

We learned through the development of ZoomPlus a way to see the big picture. An opportunity to experience the complete workflow of program design, it challenged us to think comprehensively at each stage. During preliminary design, we faced the difficulty of finding the balance between functionality and feasibility. Many ideas came through the project brainstorm, and after careful consideration of each, we finally set our sights on ZoomPlus, as it could be a practical and useful tool that enhances video calls' efficiency and customer experience. More challenges evolved during program design and implementation. Breaking down the work to several tasks, we managed to expand our knowledge in solving each and eventually integrate them back into one. This hands-on experience on both high and low-level program designs prepared us better for future tasks of a larger scale.