GeneReel AI

Inspiration

Teenagers are most likely to be influenced by social medias than articles, we believe to make a change for the future, we need to influence them in a good way, which led us to build an add ons for Adobe Express to make content creators' working flow faster and more productive.

What it does

We built an Adobe Express add-on that helps user make educational or awareness reels/shorts faster by generating half of the work for them.

How we built it

The UI is the Adobe Add-on, and the Backend is Flask. We implemented different model for AI generator, we have Gemini by Brady, CloudFlare by Alex and Will was in charge of Fetch.ai. We have both Image and Text Generation, we also integrated GIPHY to generate GIFs based on keywords found in the transcript for the short and finally, we use Eleven Labs for text-to-speech for the video.

Challenges we ran into

Rava:

• The Adobe Ad On really gave hard times somehow but stood with it to the end xD
• The Cloudflare is not ideal for text generation so after trying to fix the prompt and response for about an hour, we decided to use it for Image generation instead

Brady:

• I'm new to hackathons, so it was hard to get used to all these new APIs and tools that I have never used.

Alex:

• I had trouble matching input requirements for Workers AI requests. I eventually figured it out by converting input to json format.

Will: I was the member of the team selected to integrate fetch.ai agents into our code. We had two goals with fetch.ai, 1) call OpenAI to generate a ~1 minute script (~200 words) from an initial prompt and 2) feed the output from 1 into something that uses Text-to-Speech software to generate a voice-over for our video. I ran into lots of challenges when trying to create, host, and integrate fetch.ai agents into our project, the least of which hosting agents on agentverse. I could not find pre-existing fetch.ai agents in the agent marketplace for either of our problems, so I tried to build and host my own, but that gave me my own suite of problems. Initially I ran into lots of errors from trying to add standard REST API functionality (GET, POST, etc...) to the OpenAI agent, but kept getting "Agent objects has no attribute" errors. I tried this on my windows machine, on a personalized environment inside my windows machine, on Windows Subsystem for Linux, and even on Jupyter Notebook but was unable to resolve this problem. I went to fetch.ai office hours for this multiple times and was showed to use a proxy as recommended in their official documentation examples but even that did not work as intended. Sidenote: I figured out how to run Text-to-Speech software with fetch.ai, just not in time to host

Accomplishments that we're proud of

Will:

• using cloudflare ai

Alex:

• video generation

Brady:

• Captioning

What we learned

Rava:

• Learned how to use Adobe Ad On
• Generated the GIFs based on the keyword received
• Classify text to get keyword from each sentence

Alex:

• Learned how to make API requests
• Chain LLM output to create text and speech media

Brady:

• How to use Gemini API and Moviepy

What's next for Reel

• Improve video generation
• Better prompting
• Better captioning (highlight current word?)

Built With

• adobe-ad-on
• ai
• cloudflare
• flask
• gemini
• javascript
• tts