LifeLens

Inspiration

Most technology is designed for humans—but every species shares the same planet, and every perspective matters. We wanted to create something that extends innovation beyond humanity, letting people see through animal eyes and rethink how we design and learn. By visualizing animal perception, our system helps design environments that respect biodiversity, reduce harmful lighting, and support animal welfare—guiding smarter choices in agriculture, pet products, and environmental design. It also makes biology and neuroscience visible, turning abstract knowledge into immersive learning for students and researchers alike. Technology shouldn’t just serve humans—it should help all life thrive.

What it does

Given an input image, video, or literally live webcam video, our project shows you how this vision looks like from the eyes of 30+ unique animals. Provide built-in AI feature that explains how and why individual animals see the world in this way.

How we built it

for dichromatic animals vision (contributed by Tina and Lumiere)

Most of the dichromatic animals are done through:

remapping the pixels from human's persepective to specific animals', using the following matrix multiplication: $$ \begin{bmatrix} a &1-a &0 \ a &1-a &0 \ 0 &0 &s \end{bmatrix} \begin{bmatrix} L \ M \ S \end{bmatrix} = \begin{bmatrix} L_{animal} &M_{animal} &S_{animal} \end{bmatrix} $$ (Note the leftmost matrix is a 3x3 matrix instead of a 1x9 matrix, just displayed incorrectly)

where $a$ and $s$ are values based on the two wavelengths which animals are sensitive to

adding a new acuity blur or anisotropic acuity blur with streak on top to mimic animals that have a more blur vision than humans
adding extra functionalities dedicated for that animal (eg for rat, the B cones are more sensitive on the top part of the sight, which means blues are easier to be perceived from the above of rat)

For UV required animal vision (contributed by Kevin)

For UV animals, we converted the input RGB image (3 wavelengths) into hyperspectral images (31 wavelengths), before estimating UV from those 31 bands. Then, we map these UV as well as different reflectances into different receptors of each animals uniquely.

for frontend (contributed by Yoshixi)

We built with a simple frontend that used progressive web app to have access to mobile microhpone and camera. We use websockets in socket.io to enable two-way communication for video streaming. Gemini was also used to provide tips.

Challenges we ran into

Represent UV colour for "human" to see

This is problematic because a LOT of animals can see UV light and skipping this just means we have to give up a good chunk of nice animals to mimic. We managed to overcome this challenge by running a ML model that converting the RGB (3 wavelengths) into hyperspectral images (31 wavelengths). This allow us to better extrapolate into UV by leveraging spectral local similarity from hyperspectral images. First, we attempted to use a transformer based MST++ model. However, even in its ONNX runtime, it still take 30seconds for an inference, which makes it unrealistic for usage as real-time conversion. Thus, we ended up using a GPU-based gaussian estimation on primary spectral decomposition (Mallet's 2019 algorithm), which allow us to reduce the inference up to 2seconds.

FOV modeling

Another major difficulty involved field-of-view (FOV) modeling. Many animals have panoramic or asymmetrical vision—some exceeding 320°—which is fundamentally different from human camera perspectives. To approximate these geometries without extra spatial data, we developed a reverse-scaling crop-and-zoom algorithm: We dynamically crop around the image’s diagonal center, then apply a controlled zoom-out transformation to emulate expanded peripheral vision. This approach required balancing biological realism, computational efficiency, and perceptual coherence and ultimately became one of the most defining technical components of our project.

Accomplishments that we're proud of

Literally everything we built so far! One day + night without sleep allowed us to achieve in 30+ different animals for the user to view, a nice front end that works great with both mobile app and computer, and our decent fps for best of user's experience (through camera)