YouTube pose dream

• 

  Workflow

• 

  WorkflowExtended

Inspiration

The project is inspired by Winamp. There is the possibility to visualize music files.

I was wondering how such a visualization for videos could look like and how AI could support it. Finally I woke up one morning and knew that I want to build a Chrome extension that does exactly that. Visualize YouTube videos.

So my subconscious is the true creator of this creation and my source of inspiration.

The whole thing should be available as an extension to reach a large user base. Besides, I personally like to watch YouTube videos.

What it does

The Chrome extension becomes active when you watch a YouTube video. I use TensorFlow.js to detect human posture. Then I overlay visualizations defined by me on top of the original image. I use the output of the AI to arrange the elements.

Currently, only single-person recognition is implemented. I don't check key point accuracy scores from the AI output. This means that something is always displayed. And you can view the AI's interpretation of the video.

How we built it

TensorfFlow.js, JavaScript, HTML, CSS, Chrome extension V3, Proton (for particle animation)

Challenges we ran into

There were many stones along the way. Proton Framework (particle animation) was unknown to me. But thanks to their great examples I could quickly understand the system and adapt it for me. And I had to evaluate a lot of things and check if this whole idea is possible at all.

Another "problem" is that through this research I have fed my subconscious with new food and that now constantly comes up with new ideas. :-)

Accomplishments that we're proud of

You can try my Chrome extension.

What we learned

• TensorFlow.js
• Did my first Chrome extension
• Now I know or try out a lot of JavaScript Frameworks for animation/visualization

What's next for YouTube pose dream

The GitHub repository has a list.

Technical changes are listed there, but also more far-reaching goals. Since the whole thing is open source, I hope for input from many people. There are so many beautiful minds out there. And I hope that some will like the idea and collaborate to create a new video experience.

Built With

• css
• googlechromeextension
• html
• javascript
• movenet
• proton
• tensorflow.js