Background music plays a crucial role in social media videos, yet finding the right music remains a challenge for video creators. These creators, often not music experts, struggle to describe their musical goals and compare options. AI text-to-music generation presents an opportunity to address these challenges by allowing users to generate music through text prompts; however, these models often require musical expertise and are difficult to control. In this work, we explore how to incorporate music generation into video editing workflows. A formative study with video creators revealed challenges in articulating and iterating on musical preferences, as creators described music as “vibes” rather than with explicit musical vocabulary. Guided by these insights, we developed a creative assistant for music generation using editable vibe-based recommendations and structured refinement of music output. A user study showed that the assistant supports exploration, while direct prompting is more effective for precise goals. Our findings offer design recommendations for AI music tools for video creators.

Publications:

"It's more of a vibe I'm going for": Designing Text-to-Music Generation Interfaces for Video Creators. DIS '25.

Collaborators:

Noor Hammad, Erik Harpstead, Jessica Hammer, Mira Dontcheva.

Project Blink is an AI-powered web-based video editing app. It is the result of a multi-year cross-disciplinary effort led by Adobe Research. With Project Blink, users can edit videos like they would edit text; by copying, pasting, and deleting text from the video transcript. Users can also search the video for objects, sounds, emotions, speakers, and more, to find when they appear and pull out key moments. Project Blink automatically trims clips for smooth transitions, and makes it easy to remove unnecessary “ums,” “ahs,” and pauses. We released Project Blink as a public beta in October 2022, and it was used by over 1500 people.

Watch our demo videos of Project Blink on Adobe Labs.

Read more about Project Blink and the team behind it on Adobe Research News.

This work investigates how multimodal interaction can make in-task help-seeking easier and faster. We introduce ReMap, a multimodal search interface that helps users find video assistance while using desktop and web applications. Users can speak search queries, add application-specific terms deictically (e.g., “how to erase this”), and navigate search results via speech, all without taking their hands (or mouse) off their current task. Thirteen participants who used ReMap in the lab found that it helped them stay focused on their task while simultaneously searching for and using learning videos.

Publications:

ReMap: Lowering the Barrier to Help-Seeking with Multimodal Search. UIST '20.

ReMap: Multimodal Help-Seeking. UIST '19 adjunct. (Demo, Poster)

Collaborators:

Julia Markel, James Basa, Mira Dontcheva, Scott Klemmer.

Complex activities often require people to work across multiple software applications, each with its own set of conventions, tools, and features. To make it easier for people to find relevant help while they work, the RePlay system introduces an application-independent approach for contextually presenting video learning resources. RePlay uses accessibility APIs to gather context about the user’s activity. It leverages an existing search engine to present relevant videos and highlights key segments within them using video captions. A week-long field study (n=7) and a lab study (n=24) showed that contextual assistance helps people spend less time away from their task than web video search and replaces current video navigation strategies.

Publications:

RePlay: Contextually Presenting Learning Videos Across Software Applications. CHI '19. Honorable Mention.

Software videos: Rich content and learning potential, but a challenge for sensemaking. CHI '18 Sensemaking Workshop. (Workshop Paper)

Collaborators:

Tricia Ngoon, Mira Dontcheva, Scott Klemmer.

Many artists livestream their creative process, allowing viewers to learn and be inspired from the decisions – and mistakes – they make along the way. To understand current practices and challenges in creative livestream communities, we have conducted a content analysis of livestream archives, interviews with streamers and viewers, and online surveys with viewers. Most recently, we have developed a semi-automatic approach for generating table of contents for creative live stream archives to make them easier to navigate and browse.

Publications:

Temporal Segmentation of Creative Live Streams. CHI '20.

How Live Streaming Does (and Doesn't) Change Creative Practices. interactions Jan/Feb 2020.

Sharing the Studio: How Creative Livestreaming can Inspire, Educate, and Engage. Creativity & Cognition '19.

Creative livestreaming: How sharing one’s process can inspire, educate, and engage. CHI '19 Live Streaming Workshop. (Workshop Paper)

Collaborators:

Mira Dontcheva, Joy Kim, Joel Brandt, Valentina Shin, Alison Thornsberry, Scott Klemmer.

Good feedback is critical to creativity and learning, yet rare. Many people do not know how to actually provide effective feedback. There is increasing demand for quality feedback – and thus feedback givers – in learning and professional settings. The CritiqueKit system presents two interactive techniques – reusable suggestions and adaptive guidance – that improve novice feedback on creative work. Two real-world deployment studies and two controlled experiments with CritiqueKit found that adaptively-presented suggestions improve the quality of feedback from novice reviewers.

Publications:

Interactive Guidance Techniques for Improving Creative Feedback. CHI '18. Honorable Mention.

CritiqueKit: A Mixed-Initiative, Real-Time Interface For Improving Feedback. UIST ’17 Adjunct. (Demo, Poster)

Collaborators:

Tricia Ngoon, Ariel Weingarten, Mira Dontcheva, Scott Klemmer.

Physical construction and assembly tasks are often carried out by groups of collocated workers, and they can be difficult to coordinate. Group members must spend time deciding how to split up the task, how to assign subtasks to each other, and in what order subtasks should be completed. WeBuild is a task distribution system that dynamically assigns subtasks to workers in a group, taking into account factors such as the dependencies between subtasks and the skills of each group member. Each worker views personalized step-by-step instructions on a mobile phone, while a dashboard visualizes the entire process.

Publication:

WeBuild: Automatically Distributing Assembly Tasks Among Collocated Workers to Improve Coordination. CHI '17.

Collaborators:

Tovi Grossman, George Fitzmaurice.

Complex software offers power for experts, yet overwhelms new users. Novices often do not know how to execute tasks, what they want to achieve, or even what is possible. To address this, we introduce the DiscoverySpace interface for executable action suggestions. DiscoverySpace is a prototype extension panel for Adobe Photoshop that suggests task-level action macros to apply to photographs based on visual features. DiscoverySpace harvests these one-click actions from the online Photoshop user community. A between-subjects study indicated that action suggestions may help novices maintain confidence, accomplish tasks, and discover features. This work demonstrates how interfaces can leverage user-generated content to help novices navigate complex software.

Publications:

DiscoverySpace: Suggesting Actions in Complex Software. DIS ’16.

DiscoverySpace: Crowdsourced Suggestions Onboard Novices in Complex Software. CSCW ’16 Companion. (Extended Abstract) -- Demo

Collaborators:

Mira Dontcheva, Holger Winnemöller, Sheryl Ehrlich, Scott Klemmer.

Framing of a peer review task can significantly impact reviews. In this 3-part study, subtle feature changes in rubrics, task structure, and artifact representation resulted in reviews that were significantly different in both the quality and focus of reviewer feedback.

As part of this project, I built a website to host our first study, where graduate school applicants were invited to submit their application essay and give feedback to each other using PeerStudio.

See the Framing Feedback Project Website.

Publications:

Framing Feedback: Choosing Review Environment Features that Support High Quality Peer Assessment. CHI ’16.

Do numeric ratings impact peer reviewers? Learning at Scale. (Extended Abstract) -- Poster

Collaborators:

Catherine Hicks, Vineet Pandey, Purvi Desai, Scott Klemmer.