Inspiration
Our inspiration for this project came from our personal experience with classes, with information being lengthy and difficult to retain at times. There's no convenient visual way to study and remember the necessary concepts. In our personal lives, this is also especially true with remembering food recipes when cooking. Additionally, much of the information conveyed in auditory and written form is not as accessible to all individuals.
What it does
This add-on allows for an instructional PDF on any topic to be conveyed with images and a caption for each step. Each image and caption is placed on the project canopy, essentially acting as a visual instructional aid for the user.
How we built it
We used React.js for frontend and Node.js for backend/server functionality. We parse through the text of the input instructional PDF using OpenAI's API. From there, we extract individual steps for generating images using the Flash 2.0 API via Google Gemini. We also extract the individual steps for displaying as a caption for each photo on the project canopy.
Challenges we ran into
We ran into challenges with including non-Adobe-sourced images on our project programmatically, especially when it came to resizing and repositioning the images on the canopy. We also ran into concerns regarding textbox resizing and reformatting (i.e., reducing font size) to depict the text neatly. A third significant challenge we faced included linking the sandbox/canopy addition to the add-on frontend to show output on the canopy.
Accomplishments that we're proud of
We are proud of the fact that we were able to get image generation working, adding the AI-generated images from the JSON output from Flash 2.0. This response included a b64 source for the image and we converted the source into a tangible image object, uploading it to an image hosting server to get a public URL to display on our Adobe Express project. We are also proud of how we were able to figure out how to control the positioning of the AI-generated images.
What we learned
We learnt how to work with and develop with Adobe Express, understanding its limitations and abilities along the way. We also learned how to connect a React.js frontend with Node.js backend services in an add-on environment, and how to parse and extract structured instructional content using the OpenAI API.
We gained hands-on experience in integrating generative AI APIs (such as Flash 2.0/Gemini) to create dynamic, step-by-step visualizations based on instructional text, including converting base64 image data to usable image assets. This also gave us practice in handling CORS, cross-origin communication, and securely transferring data between services.
On the UI/UX side, we learned how to programmatically add, resize, and position elements (images, captions, shapes, and text) directly onto the Adobe Express project canvas, and how to iterate on design for clarity and accessibility.
What's next for ScriptToScene
We will look towards adding voice narration for each step, making our product even more accessible. We will also look into creating GIFs instead of images for every step.
Log in or sign up for Devpost to join the conversation.