A few notes before we start:

• The complete source code for this article, including future updates, is available in this notebook under the Apache 2.0 license.
• All new images in this article were generated with Gemini Nano Banana using the explored proof-of-concept. All source images are either in the public domain or free to use (reference links are provided in the code output).
• You can experiment with Gemini models for free in Google AI Studio. For programmatic API access, please note that while a free tier is available for some models (i.e., you can perform object detection), image generation is a pay-as-you-go service.

✨ Overview

Traditional computer vision models are typically trained to detect a fixed set of object classes, like “person”, “cat”, or “car”. If you want to detect something specific that wasn’t in the training set, such as an “illustration” in a book photograph, you usually have to gather a dataset, label it manually, and train a custom model, which can take hours or even days.

In this exploration, we’ll test a different approach using Gemini. We will leverage its spatial understanding capabilities to perform open-vocabulary object detection. This allows us to find objects based solely on a natural language description, without any training.

Once the visual objects are detected, we’ll extract them and then use Gemini’s image editing capabilities (specifically the Nano Banana models) to restore and creatively transform them.

🔥 Challenge

We are dealing with unstructured data: photos of books, magazines, and objects in the wild. These images present several difficulties for traditional computer vision:

• Variety: The objects we want to find (illustrations, engravings, and any visuals in general) vary wildly in style and content.
• Distortion: Pages are curved, photos are taken at angles, and lighting is uneven.
• Noise: Old books have stains, paper grain, and text bleeding through from the other side.

Our challenge is to build a robust pipeline that can detect these objects despite the distortions, extract them cleanly, and edit them to look like high-quality digital assets… all using simple text prompts.

🏁 Setup

🐍 Python packages

We’ll use the following packages:

• google-genai: the Google Gen AI Python SDK lets us call Gemini with a few lines of code
• pillow for image management
• matplotlib for result visualization

We’ll also use these packages (dependencies of google-genai):

• pydantic for data management
• tenacity for request management

pip install --quiet "google-genai>=1.64.0" "pillow>=11.3.0" "matplotlib>=3.10.0"

🤖 Gen AI SDK

To send Gemini requests, create a google.genai client:

from google import genai

check_environment()

client = genai.Client()

check_configuration(client)

Using the Vertex AI API with project "..." in location "europe-west9"

🖼️ Image test suite

Let’s define a list of images for our tests:

from dataclasses import dataclass
from enum import StrEnum

Url = str


class Source(StrEnum):
    incunable = "https://tile.loc.gov/image-services/iiif/service:rbc:rbc0001:2014:2014rosen0487:0165/full/pct:25/0/default.jpg"
    engravings = "https://tile.loc.gov/image-services/iiif/service:gdc:gdcscd:00:34:07:66:92:1:00340766921:0121/full/pct:50/0/default.jpg"
    museum_guidebook = "https://tile.loc.gov/image-services/iiif/service:rbc:rbc0001:2014:2014gen34181:0033/full/pct:75/0/default.jpg"
    denver_illustrated = "https://tile.loc.gov/image-services/iiif/service:gdc:gdclccn:rc:01:00:04:94:rc01000494:0051/full/pct:50/0/default.jpg"
    physics_textbook = "https://tile.loc.gov/image-services/iiif/service:gdc:gdcscd:00:03:64:87:31:8:00036487318:0103/full/pct:50/0/default.jpg"
    portrait_miniatures = "https://tile.loc.gov/image-services/iiif/service:rbc:rbc0001:2024:2024rosen013592v02:0249/full/pct:50/0/default.jpg"
    wizard_of_oz_drawings = "https://tile.loc.gov/image-services/iiif/service:rbc:rbc0001:2006:2006gen32405:0048/full/pct:25/0/default.jpg"
    paintings = "https://images.unsplash.com/photo-1714146681164-f26fed839692?h=1440"
    alice_drawing = "https://images.unsplash.com/photo-1630595011903-689853b04ee2?h=800"
    book = "https://images.unsplash.com/photo-1643451533573-ee364ba6e330?h=800"
    manual = "https://images.unsplash.com/photo-1623666936367-a100f62ba9b7?h=800"
    electronics = "https://images.unsplash.com/photo-1757397584789-8b2c5bfcdbc3?h=1440"


@dataclass
class SourceMetadata:
    title: str
    webpage_url: Url
    credit_line: str


LOC = "Library of Congress"
LOC_RARE_BOOKS = "Library of Congress, Rare Book and Special Collections Division"
LOC_MEETING_FRONTIERS = "Library of Congress, Meeting of Frontiers"

metadata_by_source: dict[Source, SourceMetadata] = {
    Source.incunable: SourceMetadata(
        "Vergaderinge der historien van Troy (1485)",
        "https://www.loc.gov/resource/rbc0001.2014rosen0487/?sp=165",
        LOC_RARE_BOOKS,
    ),
    Source.engravings: SourceMetadata(
        "Harper's illustrated catalogue (1847)",
        "https://www.loc.gov/resource/gdcscd.00340766921/?sp=121",
        LOC,
    ),
    Source.museum_guidebook: SourceMetadata(
        "Barnum's American Museum illustrated (1850)",
        "https://www.loc.gov/resource/rbc0001.2014gen34181/?sp=33",
        LOC_RARE_BOOKS,
    ),
    Source.denver_illustrated: SourceMetadata(
        "Denver illustrated (1893)",
        "https://www.loc.gov/resource/gdclccn.rc01000494/?sp=51",
        LOC_MEETING_FRONTIERS,
    ),
    Source.physics_textbook: SourceMetadata(
        "Lessons in physics (1916)",
        "https://www.loc.gov/resource/gdcscd.00036487318/?sp=103",
        LOC,
    ),
    Source.portrait_miniatures: SourceMetadata(
        "The history of portrait miniatures (1904)",
        "https://www.loc.gov/resource/rbc0001.2024rosen013592v02/?sp=249",
        LOC_RARE_BOOKS,
    ),
    Source.wizard_of_oz_drawings: SourceMetadata(
        "The wonderful Wizard of Oz (1899)",
        "https://www.loc.gov/resource/rbc0001.2006gen32405/?sp=48",
        LOC_RARE_BOOKS,
    ),
    Source.paintings: SourceMetadata(
        "Open book showing paintings by Vincent van Gogh",
        "https://unsplash.com/photos/9hD7qrxICag",
        "Photo by Trung Manh cong on Unsplash",
    ),
    Source.alice_drawing: SourceMetadata(
        "Open book showing an illustration and text from Alice's Adventures in Wonderland",
        "https://unsplash.com/photos/bewzr_Q9u2o",
        "Photo by Brett Jordan on Unsplash",
    ),
    Source.book: SourceMetadata(
        "Open book showing two botanical illustrations",
        "https://unsplash.com/photos/4IDqcNj827I",
        "Photo by Ranurte on Unsplash",
    ),
    Source.manual: SourceMetadata(
        "Open user manual for vintage camera",
        "https://unsplash.com/photos/aaFU96eYASk",
        "Photo by Annie Spratt on Unsplash",
    ),
    Source.electronics: SourceMetadata(
        "Circuit board with electronic components",
        "https://unsplash.com/photos/Aqa1pHQ57pw",
        "Photo by Albert Stoynov on Unsplash",
    ),
}

print("✅ Test images defined")

🧠 Gemini models

Gemini comes in different versions. We can currently use the following models:

• For object detection: Gemini 2.5 or Gemini 3, each available in Flash or Pro versions.
• For object editing: Gemini 2.5 Flash Image, Gemini 3 Pro Image, or Gemini 3.1 Flash Image (also known as Nano Banana 🍌, Nano Banana Pro 🍌, and Nano Banana 2 🍌).

🛠️ Helpers

Now, let’s add core helper classes and functions:

from enum import auto
from pathlib import Path
from typing import Any, cast

import IPython.display
import matplotlib.pyplot as plt
import pydantic
import tenacity
from google.genai.errors import ClientError
from google.genai.types import (
    FinishReason,
    GenerateContentConfig,
    GenerateContentResponse,
    PIL_Image,
    ThinkingConfig,
    ThinkingLevel,
)


# Multimodal models with spatial understanding and structured outputs
class MultimodalModel(StrEnum):
    # Generally Available (GA)
    GEMINI_2_5_FLASH = "gemini-2.5-flash"
    GEMINI_2_5_PRO = "gemini-2.5-pro"
    # Preview
    GEMINI_3_FLASH_PREVIEW = "gemini-3-flash-preview"
    GEMINI_3_1_PRO_PREVIEW = "gemini-3.1-pro-preview"
    # Default model used for object detection
    DEFAULT = GEMINI_3_FLASH_PREVIEW


# Image generation and editing models (Nano Banana 🍌 models)
class ImageModel(StrEnum):
    # Generally Available (GA)
    GEMINI_2_5_FLASH_IMAGE = "gemini-2.5-flash-image"  # Nano Banana
    # Preview
    GEMINI_3_PRO_IMAGE_PREVIEW = "gemini-3-pro-image-preview"  # Nano Banana Pro
    GEMINI_3_1_FLASH_IMAGE_PREVIEW = "gemini-3.1-flash-image-preview"  # Nano Banana 2
    # Default model used for image editing
    DEFAULT = GEMINI_2_5_FLASH_IMAGE


Model = MultimodalModel | ImageModel


def generate_content(
    contents: list[Any],
    model: Model,
    config: GenerateContentConfig | None,
    should_display_response_info: bool = False,
) -> GenerateContentResponse | None:
    response = None
    client = check_client_for_model(model)

    for attempt in get_retrier():
        with attempt:
            response = client.models.generate_content(
                model=model.value,
                contents=contents,
                config=config,
            )
    if should_display_response_info:
        display_response_info(response, config)

    return response


def check_client_for_model(model: Model) -> genai.Client:
    if (
        model.value.endswith("-preview")
        and client.vertexai
        and client._api_client.location != "global"
    ):
        # Preview models are only available on the "global" location
        return genai.Client(location="global")

    return client


def display_response_info(
    response: GenerateContentResponse | None,
    config: GenerateContentConfig | None,
) -> None:
    if response is None:
        print("❌ No response")
        return

    if usage_metadata := response.usage_metadata:
        if usage_metadata.prompt_token_count:
            print(f"Input tokens   : {usage_metadata.prompt_token_count:9,d}")
        if usage_metadata.candidates_token_count:
            print(f"Output tokens  : {usage_metadata.candidates_token_count:9,d}")
        if usage_metadata.thoughts_token_count:
            print(f"Thoughts tokens: {usage_metadata.thoughts_token_count:9,d}")

    if (
        config is not None
        and config.response_mime_type == "application/json"
        and response.parsed is None
    ):
        print("❌ Could not parse the JSON response")
        return
    if not response.candidates:
        print("❌ No `response.candidates`")
        return
    if (finish_reason := response.candidates[0].finish_reason) != FinishReason.STOP:
        print(f"❌ {finish_reason = }")
    if not response.text:
        print("❌ No `response.text`")
        return


def generate_image(
    sources: list[PIL_Image],
    prompt: str,
    model: ImageModel,
    config: GenerateContentConfig | None = None,
) -> PIL_Image | None:
    contents = [*sources, prompt.strip()]

    response = generate_content(contents, model, config)

    return check_get_output_image_from_response(response)


def check_get_output_image_from_response(
    response: GenerateContentResponse | None,
) -> PIL_Image | None:
    if response is None:
        print("❌ No `response`")
        return None
    if not response.candidates:
        print("❌ No `response.candidates`")
        if response.prompt_feedback:
            if block_reason := response.prompt_feedback.block_reason:
                print(f"{block_reason = :s}")
            if block_reason_message := response.prompt_feedback.block_reason_message:
                print(f"{block_reason_message = }")
        return None
    if not (content := response.candidates[0].content):
        print("❌ No `response.candidates[0].content`")
        return None
    if not (parts := content.parts):
        print("❌ No `response.candidates[0].content.parts`")
        return None

    output_image: PIL_Image | None = None
    for part in parts:
        if part.text:
            display_markdown(part.text)
            continue
        sdk_image = part.as_image()
        assert sdk_image is not None
        output_image = sdk_image._pil_image
        assert output_image is not None
        break  # There should be a single image

    return output_image


def get_thinking_config(model: Model) -> ThinkingConfig | None:
    match model:
        case MultimodalModel.GEMINI_2_5_FLASH:
            return ThinkingConfig(thinking_budget=0)
        case MultimodalModel.GEMINI_2_5_PRO:
            return ThinkingConfig(thinking_budget=128, include_thoughts=False)
        case MultimodalModel.GEMINI_3_FLASH_PREVIEW:
            return ThinkingConfig(thinking_level=ThinkingLevel.MINIMAL)
        case MultimodalModel.GEMINI_3_1_PRO_PREVIEW:
            return ThinkingConfig(thinking_level=ThinkingLevel.LOW)
        case _:
            return None  # Default


def display_markdown(markdown: str) -> None:
    IPython.display.display(IPython.display.Markdown(markdown))


def display_image(image: PIL_Image) -> None:
    IPython.display.display(image)


def get_retrier() -> tenacity.Retrying:
    return tenacity.Retrying(
        stop=tenacity.stop_after_attempt(7),
        wait=tenacity.wait_incrementing(start=10, increment=1),
        retry=should_retry_request,
        reraise=True,
    )


def should_retry_request(retry_state: tenacity.RetryCallState) -> bool:
    if not retry_state.outcome:
        return False
    err = retry_state.outcome.exception()
    if not isinstance(err, ClientError):
        return False
    print(f"❌ ClientError {err.code}: {err.message}")

    retry = False
    match err.code:
        case 400 if err.message is not None and " try again " in err.message:
            # Workshop: first time access to Cloud Storage (service agent provisioning)
            retry = True
        case 429:
            # Workshop: temporary project with 1 QPM quota
            retry = True
    print(f"🔄 Retry: {retry}")

    return retry


print("✅ Helpers defined")

🔍 Detecting visual objects

To perform visual object detection, craft the prompt to indicate what you’d like to detect and how results should be returned. In the same request, it’s possible to also extract additional information about each detected object. This can be virtually anything, from labels such as “furniture”, “table”, or “chair”, to more precise classifications like “mammals” or “reptiles”, or to contextual data such as captions, colors, shapes, etc.

For the next tests, we’ll experiment with detecting illustrations within book photos. Here’s a possible prompt:

OBJECT_DETECTION_PROMPT = """
Detect every illustration within the book photo and extract the following data for each:
- `box_2d`: Bounding box coordinates of the illustration only (ignoring any caption).
- `caption`: Verbatim caption or legend such as "Figure 1". Use "" if not found.
- `label`: Single-word label describing the illustration. Use "" if not found.
"""

Notes:

• Bounding boxes are very useful for locating or extracting the detected objects.
• Typically, for Gemini models, a box_2d bounding box represents coordinates normalized to a (0, 0, 1000, 1000) space for a (0, 0, width, height) input image.
• We’re also requesting to extract captions (metadata often present in reference books) and labels (dynamic metadata).

To automate response processing, it’s convenient to define a Pydantic class that matches the prompt, such as:

class DetectedObject(pydantic.BaseModel):
    box_2d: list[int]
    caption: str
    label: str

DetectedObjects = list[DetectedObject]

Then, request a structured output with config fields response_mime_type and response_schema:

config = GenerateContentConfig(
    # …,
    response_mime_type="application/json",
    response_schema=DetectedObjects,
    # …,
)

This will generate a JSON response which the SDK can parse automatically, letting us directly use object instances:

detected_objects = cast(DetectedObjects, response.parsed)

Let’s add a few object-detection-specific classes and functions:

import io
import urllib.request
from collections.abc import Iterator
from dataclasses import field
from datetime import datetime

import PIL.Image
from google.genai.types import Part, PartMediaResolutionLevel
from PIL.PngImagePlugin import PngInfo

OBJECT_DETECTION_PROMPT = """
Detect every illustration within the book photo and extract the following data for each:
- `box_2d`: Bounding box coordinates of the illustration only (ignoring any caption).
- `caption`: Verbatim caption or legend such as "Figure 1". Use "" if not found.
- `label`: Single-word label describing the illustration. Use "" if not found.
"""

# Margin added to detected/cropped objects, giving more context for a better understanding of spatial distortions
CROP_MARGIN_PX = 10

# Set to True to save each generated image
SAVE_GENERATED_IMAGES = False
OUTPUT_IMAGES_PATH = Path("./object_detection_and_editing")


# Matching class for structured output generation
class DetectedObject(pydantic.BaseModel):
    box_2d: list[int]
    caption: str
    label: str


# Misc data classes
InputImage = Path | Url
DetectedObjects = list[DetectedObject]
WorkflowStepImages = list[PIL_Image]


class WorkflowStep(StrEnum):
    SOURCE = auto()
    CROPPED = auto()
    RESTORED = auto()
    COLORIZED = auto()
    CINEMATIZED = auto()


@dataclass
class VisualObjectWorkflow:
    source_image: PIL_Image
    detected_objects: DetectedObjects
    images_by_step: dict[WorkflowStep, WorkflowStepImages] = field(default_factory=dict)

    def __post_init__(self) -> None:
        denormalize_bounding_boxes(self)


workflow_by_image: dict[InputImage, VisualObjectWorkflow] = {}


def denormalize_bounding_boxes(self: VisualObjectWorkflow) -> None:
    """Convert the box_2d coordinates.
    - Before: [y1, x1, y2, x2] normalized to 0-1000, as returned by Gemini
    - After:  [x1, y1, x2, y2] in source_image coordinates, as used in Pillow
    """

    def to_image_coord(coord: int, dim: int) -> int:
        return int(coord * dim / 1000 + 0.5)

    w, h = self.source_image.size
    for obj in self.detected_objects:
        y1, x1, y2, x2 = obj.box_2d
        x1, x2 = to_image_coord(x1, w), to_image_coord(x2, w)
        y1, y2 = to_image_coord(y1, h), to_image_coord(y2, h)
        obj.box_2d = [x1, y1, x2, y2]


def detect_objects(
    image: InputImage,
    prompt: str = OBJECT_DETECTION_PROMPT,
    model: MultimodalModel = MultimodalModel.DEFAULT,
    config: GenerateContentConfig | None = None,
    media_resolution: PartMediaResolutionLevel | None = None,
    display_results: bool = True,
) -> None:
    display_image_source_info(image)
    pil_image, content_part = get_pil_image_and_part(image, model, media_resolution)
    prompt = prompt.strip()
    contents = [content_part, prompt]
    config = config or get_object_detection_config(model)

    response = generate_content(contents, model, config)

    if response is not None and response.parsed is not None:
        detected_objects = cast(DetectedObjects, response.parsed)
    else:
        detected_objects = DetectedObjects()

    workflow = VisualObjectWorkflow(pil_image, detected_objects)
    workflow_by_image[image] = workflow
    add_cropped_objects(workflow, image, prompt)

    if display_results:
        display_detected_objects(workflow)


def get_pil_image_and_part(
    image: InputImage,
    model: MultimodalModel,
    media_resolution: PartMediaResolutionLevel | None,
) -> tuple[PIL_Image, Part]:
    if isinstance(image, Path):
        image_bytes = image.read_bytes()
    else:
        headers = {"User-Agent": "Mozilla/5.0"}
        req = urllib.request.Request(image, headers=headers)
        with urllib.request.urlopen(req, timeout=10) as response:
            image_bytes = response.read()

    pil_image = PIL.Image.open(io.BytesIO(image_bytes))
    mime_type = f"image/{pil_image.format.lower()}" if pil_image.format else "image/*"
    content_part = Part.from_bytes(
        data=image_bytes,
        mime_type=mime_type,
        media_resolution=media_resolution,
    )

    return pil_image, content_part


def get_object_detection_config(model: Model) -> GenerateContentConfig:
    # Low randomness for more determinism
    return GenerateContentConfig(
        temperature=0.0,
        top_p=0.0,
        seed=42,
        response_mime_type="application/json",
        response_schema=DetectedObjects,
        thinking_config=get_thinking_config(model),
    )


def add_cropped_objects(
    workflow: VisualObjectWorkflow,
    input: InputImage,
    prompt: str,
    crop_margin: int = CROP_MARGIN_PX,
) -> None:
    cropped_images: list[PIL_Image] = []
    obj_count = len(workflow.detected_objects)
    for obj_order, obj in enumerate(workflow.detected_objects, 1):
        cropped_image, _ = extract_object_image(workflow.source_image, obj, crop_margin)
        cropped_images.append(cropped_image)
        save_workflow_image(
            WorkflowStep.SOURCE,
            WorkflowStep.CROPPED,
            input,
            obj_order,
            obj_count,
            cropped_image,
            dict(prompt=prompt, crop_margin=str(crop_margin)),
        )
    workflow.images_by_step[WorkflowStep.CROPPED] = cropped_images


def extract_object_image(
    image: PIL_Image,
    obj: DetectedObject,
    margin: int = 0,
) -> tuple[PIL_Image, tuple[int, int, int, int]]:
    def clamp(coord: int, dim: int) -> int:
        return min(max(coord, 0), dim)

    x1, y1, x2, y2 = obj.box_2d
    w, h = image.size
    if margin != 0:
        x1, x2 = clamp(x1 - margin, w), clamp(x2 + margin, w)
        y1, y2 = clamp(y1 - margin, h), clamp(y2 + margin, h)

    box = (x1, y1, x2, y2)
    object_image = image.crop(box)

    return object_image, box


def save_workflow_image(
    source_step: WorkflowStep,
    target_step: WorkflowStep,
    input_image: InputImage,
    obj_order: int,
    obj_count: int,
    target_image: PIL_Image | None,
    image_info: dict[str, str] | None = None,
) -> None:
    if not SAVE_GENERATED_IMAGES or target_image is None:
        return
    OUTPUT_IMAGES_PATH.mkdir(parents=True, exist_ok=True)
    time_str = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
    try:
        filename = f"{Source(input_image).name}_"
    except ValueError:
        filename = ""
    filename += f"{obj_order}o{obj_count}_{source_step}_{target_step}_{time_str}.png"
    image_path = OUTPUT_IMAGES_PATH.joinpath(filename)
    params = {}
    if image_info:
        png_info = PngInfo()
        for k, v in image_info.items():
            png_info.add_text(k, v)
        params.update(pnginfo=png_info)
    target_image.save(image_path, **params)


# Matplotlib
FIGURE_FG_COLOR = "#F1F3F4"
FIGURE_BG_COLOR = "#202124"
EDGE_COLOR = "#80868B"
rcParams = {
    "figure.dpi": 300,
    "text.color": FIGURE_FG_COLOR,
    "figure.facecolor": FIGURE_BG_COLOR,
    "figure.edgecolor": FIGURE_FG_COLOR,
    "axes.titlecolor": FIGURE_FG_COLOR,
    "axes.edgecolor": EDGE_COLOR,
    "xtick.color": FIGURE_FG_COLOR,
    "ytick.color": FIGURE_FG_COLOR,
    "xtick.bottom": False,
    "xtick.top": False,
    "ytick.left": False,
    "ytick.right": False,
    "xtick.labelbottom": False,
    "ytick.labelleft": False,
}
plt.rcParams.update(rcParams)


def display_image_source_info(image: InputImage) -> None:
    def get_image_info_md() -> str:
        if image not in Source:
            return f"[[Source Image]({image})]"
        source = Source(image)
        metadata = metadata_by_source.get(source)
        if not metadata:
            return f"[[Source Image]({source.value})]"
        parts = [
            f"[Source Image]({source.value})",
            f"[Source Page]({metadata.webpage_url})",
            metadata.title,
            metadata.credit_line,
        ]
        separator = "•"
        inner_info = f" {separator} ".join(parts)
        return f"{separator} {inner_info} {separator}"

    def yield_md_rows() -> Iterator[str]:
        horizontal_line = "---"
        image_info = get_image_info_md()
        yield horizontal_line
        yield f"_{image_info}_"
        yield horizontal_line

    display_markdown(f"{chr(10)}{chr(10)}".join(yield_md_rows()))


def display_detected_objects(workflow: VisualObjectWorkflow) -> None:
    source_image = workflow.source_image
    detected_objects = PIL.Image.new("RGB", source_image.size, "white")
    for obj in workflow.detected_objects:
        obj_image, box = extract_object_image(source_image, obj)
        detected_objects.paste(obj_image, (box[0], box[1]))

    _, (ax1, ax2) = plt.subplots(1, 2, layout="compressed")
    ax1.imshow(source_image)
    ax2.imshow(detected_objects)

    disable_colab_cell_scrollbar()
    plt.show()


print("✅ Object detection helpers defined")

🧪 Let’s start simple: can we detect the single illustration in this incunable from 1485?

detect_objects(Source.incunable)

💡 This works nicely. The bounding box is very precise, enclosing the hand-colored woodcut illustration very tightly.

🧪 Now, let’s check the detection of the multiple visuals in this museum guidebook:

detect_objects(Source.museum_guidebook)

💡 Remarks:

• The bounding boxes are again very precise.
• The results are perfect: there are no false positives and no false negatives.
• The captions below the visuals are not enclosed within the bounding boxes, which was specifically requested. The bounding box granularity can be controlled by changing the prompt.

🧪 What about slightly warped visuals?

detect_objects(Source.paintings)

💡 This doesn’t make a difference. Notice how the bottom-right painting is partially covered by the orange bookmark. We’ll try to fix that in the restoration step.

🧪 What about the tilted visuals in this book about the architecture in Denver?

detect_objects(Source.denver_illustrated)

💡 Each visual is perfectly detected: spatial understanding covers tilted objects.

🧪 Finally, let’s check the detection on this significantly warped book page from Alice’s Adventures in Wonderland:

detect_objects(Source.alice_drawing)

💡 Page curvature and other distortions don’t prevent non-rectangular objects from being detected. In fact, spatial understanding works at the pixel level, which explains this precision for warped objects. If you’d like to work at a lower level, you can also ask for a “segmentation mask” in the prompt and you’ll get a base64-encoded PNG (each pixel giving the 0–255 probability it belongs to the object within the bounding box). See the segmentation doc for more details.

🏷️ Text extraction and dynamic labeling

On top of localizing each object with its bounding box, our prompt requested to extract a verbatim caption and to assign a single-word label, when possible.

Let’s add a simple function to display the detection data in a table:

from collections import defaultdict


def display_detection_data(source: Source, show_consolidated: bool = False) -> None:
    def string_with_visible_linebreaks(s: str) -> str:
        return f'''"{s.replace(chr(10), "↩️")}"'''

    def yield_md_rows_consolidated(workflow: VisualObjectWorkflow) -> Iterator[str]:
        yield "| label | count | captions |"
        yield "| :--- | ---: | :--- |"
        stats = defaultdict(list)
        for obj in workflow.detected_objects:
            stats[obj.label].append(string_with_visible_linebreaks(obj.caption))
        for label, captions in stats.items():
            count = len(captions)
            label_captions = " • ".join(sorted(captions))
            yield f"| {label} | {count} | {label_captions} |"

    def yield_md_rows_with_bbox(workflow: VisualObjectWorkflow) -> Iterator[str]:
        yield "| box_2d | label | caption |"
        yield "| :--- | :--- | :--- |"
        for obj in workflow.detected_objects:
            yield f"| {obj.box_2d} | {obj.label} | {string_with_visible_linebreaks(obj.caption)} |"

    workflow = workflow_by_image.get(source)
    if workflow is None:
        print(f'❌ No detection for source "{source.name}"')
        return
    md_rows = list(
        yield_md_rows_consolidated(workflow)
        if show_consolidated
        else yield_md_rows_with_bbox(workflow)
    )
    display_image_source_info(source)
    display_markdown(chr(10).join(md_rows))

In the museum guidebook, the dynamic labeling is precise according to the context, and the captions below each illustration are perfectly extracted:

display_detection_data(Source.museum_guidebook)

| box_2d                   | label     | caption                   |
| :----------------------- | :-------- | :------------------------ |
| [954, 629, 1338, 1166]   | beetle    | "The Horned Beetle."      |
| [265, 984, 464, 1504]    | armor     | "Armor of a Man."         |
| [737, 984, 915, 1328]    | armor     | "Horse Armor."            |
| [1225, 1244, 1589, 1685] | beetle    | "The Goliath Beetle."     |
| [264, 1766, 431, 2006]   | mask      | "The Mask."               |
| [937, 1769, 1260, 2087]  | butterfly | "Painted Lady Butterfly." |
| [1325, 2170, 1581, 2468] | butterfly | "The Lady Butterfly."     |

In the book photo showing four paintings, this is perfect too:

display_detection_data(Source.paintings)

| box_2d                | label    | caption                                                                                                                                            |
| :-------------------- | :------- | :------------------------------------------------------------------------------------------------------------------------------------------------- |
| [378, 203, 837, 575]  | painting | "Hái Ô-liu (Olive Picking), tháng 12 năm 1889, sơn dầu trên toan, 28 3/4 x 35 in. [73 x 89 cm]"                                                    |
| [913, 207, 1380, 563] | painting | "Hẻm núi Les Peiroulets (Les Peiroulets Ravine), tháng 10 năm 1889, sơn dầu trên toan, 28 3/4 x 36 1/4 in. [73 x 92 cm]"                           |
| [387, 596, 845, 978]  | painting | "Trưa: Nghỉ ngơi (phỏng theo Millet) (Noon: Rest from Work [after Millet]), tháng 1 năm 1890, sơn dầu trên toan, 28 3/4 x 35 7/8 in. [73 x 91 cm]" |
| [921, 611, 1397, 982] | painting | "Hoa hạnh đào (Almond Blossom), tháng 2 năm 1890, sơn dầu trên toan, 28 3/8 x 36 1/4 in. [73 x 92 cm]"                                             |

In the Denver architecture book, the four captions are assigned to the correct illustrations, which was not an obvious task:

display_detection_data(Source.denver_illustrated)

| box_2d                 | label    | caption                        |
| :--------------------- | :------- | :----------------------------- |
| [203, 224, 741, 839]   | building | "ERNEST AND CRANMER BUILDING." |
| [743, 73, 1192, 758]   | building | "PEOPLE'S BANK BUILDING."      |
| [1185, 211, 1787, 865] | building | "BOSTON BUILDING."             |
| [699, 754, 1238, 1203] | building | "COOPER BUILDING."             |

💡 If you have a closer look at the input image, it’s hard to tell which caption belongs to which illustration at a glance. Most of us would need to think about it (and might be wrong). Asking Gemini reveals that the results are intentional and not pure luck: Deciphering vintage layouts can feel a bit like a puzzle, but there is usually a “reading-order” logic at play. In this specific case, the captions are arranged to correspond with the images in a clockwise or Z-pattern starting from the top left.

In the “Alice’s Adventures in Wonderland” book page, there was a single illustration accompanying the story text. As expected, the caption is empty (i.e., no false positive):

display_detection_data(Source.alice_drawing)

| box_2d                | label        | caption |
| :-------------------- | :----------- | :------ |
| [111, 146, 1008, 593] | illustration | ""      |

🔭 Generalizing object detection

We can use the same principles for other object types. We’ll generally keep requesting bounding boxes to identify object positions within images. Without changing our current output structure (i.e., no code change), we can use captions and labels to extract different object metadata depending on the input type.

🧪 See how we can detect electronic components by adapting the prompt while keeping the exact same code and output structure:

ELECTRONIC_COMPONENT_DETECTION_PROMPT = """
Exhaustively detect all the individual electronic components in the image and provide the following data for each:
- `box_2d`: bounding box coordinates.
- `caption`: Verbatim alphanumeric text visible on the component (including original line breaks), or "" if no text is present.
- `label`: Specific type of component.
"""

detect_objects(
    Source.electronics,
    ELECTRONIC_COMPONENT_DETECTION_PROMPT,
    media_resolution=PartMediaResolutionLevel.MEDIA_RESOLUTION_ULTRA_HIGH,
)

💡 Remarks:

• Large and tiny components are detected, thanks to the specific instruction “exhaustively detect…”.
• By using the ultra-high media resolution, we ensure more details are tokenized and the “P” component (a visual outlier) gets detected.

Here’s a consolidated view of the detected components:

display_detection_data(Source.electronics, show_consolidated=True)

| label              | count | captions                                        |
| :----------------- | ----: | :---------------------------------------------- |
| integrated circuit |     3 | "49240↩️020S6K" • "8105↩️0:35" • "P4010↩️9NA0" |
| resistor           |     4 | "" • "" • "105" • "R020"                        |
| inductor           |     1 | "n1W"                                           |
| diode              |     3 | "K" • "L" • "P"                                 |
| capacitor          |     6 | "" • "" • "" • "" • "" • ""                     |
| transistor         |     1 | "41"                                            |
| connector          |     1 | ""                                              |

💡 Remarks:

• Components are detected along with their text markings, despite the three different text orientations (upright, sideways, and upside down), the blur, and the photo noise.
• We removed the degree of freedom for multi-line text by specifying the inclusion of “original line breaks” in the prompt: responses now consistently include the line breaks for the three integrated circuits (displayed with the ↩️ emoji for better visibility).
• The last degree of freedom lies in the labeling. While most components have been properly labeled, it is unclear whether the “P” component is a diode, a resistor, or a fuse. Making the instructions more specific (e.g., listing the possible labels, using an enum for the label field in the Pydantic class, or providing guidelines and more details about the expected circuit boards) will make the prompt more "closed" and the results more deterministic and accurate.
  It's also possible to enable/update the thinking_config configuration, which will trigger a chain of thought before generating the final answer. In all the detections performed, our code used ThinkingLevel.MINIMAL, which didn't consume any thought tokens (with Gemini 3 Flash). Updating the parameter to ThinkingLevel.LOW, ThinkingLevel.MEDIUM, or ThinkingLevel.HIGH will use thought tokens and can lead to better outputs in complex cases.

This demonstrates the versatility of the approach. Without retraining a model, we switched from detecting 15th-century woodcuts and illustrations with vintage layouts to identifying modern electronics just by changing the prompt. Such detections, including caption and label metadata, could be used to auto-crop components for a parts catalog, verify assembly lines, or create interactive schematics… all without a single labeled training image.

🪄 Editing visual objects

Now that we can detect visual objects, we can envision an automation workflow to extract and reuse them. For this, we’ll use Gemini 2.5 Flash Image (also known as Nano Banana 🍌) by default, a state-of-the-art image generation and editing model.

Our object editing functions will follow the same template, taking one step as input and generating an edited image for the output step. Let’s define core helpers for this:

from typing import Protocol


class ObjectEditingFunction(Protocol):
    def __call__(
        self,
        image: InputImage,
        prompt: str | None = None,
        model: ImageModel | None = None,
        config: GenerateContentConfig | None = None,
        display_results: bool = True,
    ) -> None: ...


SourceTargetSteps = tuple[WorkflowStep, WorkflowStep]
registered_functions: dict[SourceTargetSteps, ObjectEditingFunction] = {}

DEFAULT_EDITING_CONFIG = GenerateContentConfig(response_modalities=["IMAGE"])
EMPTY_IMAGE = PIL.Image.new("1", (1, 1), "white")


def object_editing_function(
    default_prompt: str,
    source_step: WorkflowStep,
    target_step: WorkflowStep,
    default_model: ImageModel = ImageModel.DEFAULT,
    default_config: GenerateContentConfig = DEFAULT_EDITING_CONFIG,
) -> ObjectEditingFunction:
    def editing_function(
        image: InputImage,
        prompt: str | None = default_prompt,
        model: ImageModel | None = default_model,
        config: GenerateContentConfig | None = default_config,
        display_results: bool = True,
    ) -> None:
        workflow, source_images = get_workflow_and_step_images(image, source_step)
        if prompt is None:
            prompt = default_prompt
        prompt = prompt.strip()
        if model is None:
            model = default_model
        # Note: "config is None" is valid and will use the model endpoint default config

        target_images: list[PIL_Image] = []
        display_image_source_info(image)
        obj_count = len(source_images)
        for obj_order, source_image in enumerate(source_images, 1):
            target_image = generate_image([source_image], prompt, model, config)
            save_workflow_image(
                source_step,
                target_step,
                image,
                obj_order,
                obj_count,
                target_image,
                dict(prompt=prompt),
            )
            target_images.append(target_image if target_image else EMPTY_IMAGE)

        workflow.images_by_step[target_step] = target_images
        if display_results:
            display_sources_and_targets(workflow, source_step, target_step)

    registered_functions[(source_step, target_step)] = editing_function

    return editing_function


def get_workflow_and_step_images(
    image: InputImage,
    step: WorkflowStep,
) -> tuple[VisualObjectWorkflow, list[PIL_Image]]:
    # Objects detected?
    if image not in workflow_by_image:
        detect_objects(image, display_results=False)
    workflow = workflow_by_image.get(image)
    assert workflow is not None

    # Workflow step objects? (single level, could be extended to a dynamical graph)
    operation = (WorkflowStep.CROPPED, step)
    if step not in workflow.images_by_step and operation in registered_functions:
        source_function = registered_functions[operation]
        source_function(image, display_results=False)

    # Source images
    source_images = workflow.images_by_step.get(step)
    assert source_images is not None

    return workflow, source_images


def display_sources_and_targets(
    workflow: VisualObjectWorkflow,
    source_step: WorkflowStep,
    target_step: WorkflowStep,
) -> None:
    source_images = workflow.images_by_step[source_step]
    target_images = workflow.images_by_step[target_step]
    if not source_images:
        print("❌ No images to display")
        return

    fig = plt.figure(layout="compressed")
    if horizontal := (len(source_images) >= 2):
        rows, cols = 2, len(source_images)
    else:
        rows, cols = len(source_images), 2
    gs = fig.add_gridspec(rows, cols)

    for i, (source_image, target_image) in enumerate(
        zip(source_images, target_images, strict=True)
    ):
        for dim, image in enumerate([source_image, target_image]):
            grid_spec = gs[dim, i] if horizontal else gs[i, dim]
            ax = fig.add_subplot(grid_spec)
            ax.set_axis_off()
            ax.imshow(image)

    disable_colab_cell_scrollbar()
    plt.show()


print("✅ Object editing helpers defined")

Now, let’s define a first editing step to restore the detected objects that can contain many real-life artifacts…

✨ Restoring visual objects

For this restoration step, we need to craft a prompt that is generic enough (to cover most use cases) but also specific enough (to take into account restoration needs).

An image editing prompt is based on natural language, typically using imperative or declarative instructions. With an imperative prompt, you describe the actions to perform on the input, while with a declarative prompt, you describe the expected output. Both are possible and will provide equivalent results. Your choice is really a matter of preference, as long as the prompt makes sense.

Our test suite is mostly composed of book photos, which can contain various photographic and paper artifacts. The Nano Banana models understand these subtleties and can edit images accordingly, which simplifies the prompt.

Here is a possible restoration function using an imperative prompt:

RESTORATION_PROMPT = """
- Isolate and straighten the visual on a pure white background, excluding any surrounding text.
- Clean up all physical artifacts and noise while preserving every original detail.
- Center the result and scale it to fit the canvas with minimal, symmetrical margins, ensuring no distortion or cropping.
"""

# Default config with low randomness for more deterministic restoration outputs
RESTORATION_CONFIG = GenerateContentConfig(
    temperature=0.0,
    top_p=0.0,
    seed=42,
    response_modalities=["IMAGE"],
)

restore_objects = object_editing_function(
    RESTORATION_PROMPT,
    WorkflowStep.CROPPED,
    WorkflowStep.RESTORED,
    default_config=RESTORATION_CONFIG,
)

print("✅ Restoration function defined")

🧪 Let’s try to restore the illustration from the 1485 incunable:

restore_objects(Source.incunable)

💡 We now have a nice restoration of the hand-colored woodcut illustration. Note that our prompt is generic ( “clean up all physical artifacts”) and could be made more specific to remove more or fewer artifacts. In this example, there are remaining artifacts, such as the paper discoloration in the sword or the bleeding ink in the armor. We’ll see if we can fix these in the colorization step.

🧪 What about the illustrations from the museum guidebook?

restore_objects(Source.museum_guidebook)

💡 All good!

🧪 What about the slightly warped visuals?

restore_objects(Source.paintings)

💡 Remarks:

• Notice how, on the last painting, the orange bookmark is properly removed and the hidden part inpainted to complete the painting.
• We requested to “fit the canvas with minimal, symmetrical margins, without distortion or cropping”. Depending on the aspect ratio and type of the visual, this degree of freedom can result in different white margins.
• This example shows famous paintings by Vincent Van Gogh. Nano Banana does not fetch any reference images and only uses the provided input. If these were photos of private paintings, they would be restored in the same way.

In the Denver architecture book, the illustrations can be tilted, which our generic prompt does not fully take into account. When several geometric transformations are involved, it can be challenging to craft an imperative prompt that details all the operations to perform. Instead, a descriptive prompt can be more straightforward by directly describing the expected output.

🧪 Here’s an example of a descriptive prompt focusing on the restoration of tilted visuals:

tilted_visual_prompt = """
An upright, high-fidelity rendition of the visual isolated against a pure white background, filling the canvas with minimal uniform margins. The output is clean, sharp, and free of physical artifacts.
"""

restore_objects(Source.denver_illustrated, tilted_visual_prompt)

💡 Remarks:

• To get these results, the prompt focuses on requesting an “upright” visual “filling the canvas”, which proves more straightforward to write than trying to account for all possible geometric corrections.
• The native visual understanding automatically identifies the content type (photo, illustration, etc.) and the different artifacts (photographic, paper, printing, scanning…), allowing for precise restorations out of the box.
• Notice how the consistency is preserved: the last visual is restored as an illustration, while the first visuals maintain their photographic style.
• The results, with this rather generic prompt, are impressive. It is, of course, possible to be more specific and request particular lighting, styles, colors…

In this last test, the input visual has distortions not only from the page curvature but also from the photo perspective.

🧪 Here’s an example of a descriptive prompt focusing on restoring warped illustrations:

warped_visual_prompt = """
An edge-to-edge digital extraction of the illustration from the provided book photo, excluding any peripheral text. All page curvature and perspective distortions are corrected, resulting in an image framed in a perfect rectangle, on a pure white canvas with minimal margins.
"""

restore_objects(Source.alice_drawing, warped_visual_prompt)

💡 It is really impressive that such a restoration can be performed in a single step. Note that this prompt is not stable and can generate less optimal results (it would benefit from being more precise). If you have complex transformations, test descriptive prompts iteratively, using precise and concise instructions, and you might be pleasantly surprised. In the worst case, it’s also possible to process the transformations in successive, easier steps.

Now, let’s add a colorization step…

🎨 Colorization

Our restoration step respected the original styles of the input images. Recent image editing models excel at transforming image styles, starting with colors. This can generally be performed directly with a simple, precise instruction.

Here is a possible colorization function using an imperative prompt:

COLORIZATION_PROMPT = """
Colorize this image in a modern book illustration style, maintaining all original details without any additions.
"""

colorize = object_editing_function(
    COLORIZATION_PROMPT,
    WorkflowStep.RESTORED,
    WorkflowStep.COLORIZED,
)

print("✅ Colorization function defined")

🧪 Let’s modernize our 1485 illustration:

colorize(Source.incunable)

💡 All details are preserved, as requested in the prompt. Notice how the colorization can naturally fix some remaining artifacts (e.g., the paper discoloration in the sword or the bleeding ink in the armor).

🧪 Let’s colorize our museum guidebook illustrations:

colorize(Source.museum_guidebook)

💡 Our prompt is very open as it only specifies “modern book illustration style”. This can generate very creative colorizations, but they all seem to make perfect sense.

🧪 What about our Denver buildings?

colorize(Source.denver_illustrated)

💡 As requested, they all look like modern illustrations, including the first visuals (originating from noisy photos).

It’s possible to go further by not only “colorizing” but also “transforming” the image into a significantly different one.

🧪 Let’s make our “Alice’s Adventures in Wonderland” drawing into a watercolor painting:

watercolor_prompt = """
Transform this visual into a warm, watercolor painting.
"""

colorize(Source.alice_drawing, watercolor_prompt)

🧪 What about making it a traditional painting?

painting_prompt = """
Transform this visual into a traditional painting.
"""

colorize(Source.alice_drawing, painting_prompt)

We can also change image compositions. Depending on the context, some compositions are more or less implied by default. For example, illustrations often have margins, while photos generally have edge-to-edge (full-bleed in the printing world) compositions. When possible, it’s interesting to refer to a type of visual (which intrinsically brings a lot of semantics to the context) and adjust the instructions accordingly.

🧪 Let’s see how we can detect engravings in this 1847 book, restore them, and transform them into modern digital graphics:

detect_objects(Source.engravings)

restore_objects(Source.engravings)

visual_to_digital_graphic_prompt = """
Transform this visual into a full-color, flat digital graphic, extending the content for a full-bleed effect.
"""

colorize(Source.engravings, visual_to_digital_graphic_prompt)

🧪 We can also transform the same engravings into photos with a very simple prompt:

visual_to_photo_prompt = """
Transform this visual into a high-end, modern camera photograph.
"""

colorize(Source.engravings, visual_to_photo_prompt)

💡 As photos are generally full-bleed, the prompt does not need to specify a composition.

It’s really up to our imagination, as Nano Banana seems to grasp every aspect of the visual semantics.

Let’s add a final step to see how far we can go, reimagining images as cinematic movie stills…

🎞️ Cinematization

We’ve used rather “closed” prompts so far, crafting specific instructions and constraints to control the outputs. It’s possible to go even further with “open” prompts and generate images in full creative mode. Notably, it can be interesting to refer to photographic or cinematographic terminology as it encompasses many visual techniques.

Here is a possible generic cinematization function to reimagine images as movie stills:

CINEMATIZATION_PROMPT = """
Reimagine this image as a joyful, modern live-action cinematic movie still featuring professional lighting and composition.
"""

cinematize = object_editing_function(
    CINEMATIZATION_PROMPT,
    WorkflowStep.RESTORED,
    WorkflowStep.CINEMATIZED,
)

🧪 Let’s cinematize the “Alice’s Adventures in Wonderland” drawing:

cinematize(Source.alice_drawing)

💡 This looks like a high-budget movie still. There are lots of degrees of freedom in the prompt, but you’re likely to get foreground figures in sharp focus, a gradual background blur, “golden hour” lighting (a magical ingredient for many cinematographers), and detailed textures. Such compositions really evoke different atmospheres compared to the photos generated in the previous test.

🧪 Let’s test the workflow on a page from the Wonderful Wizard of Oz containing three drawings:

detect_objects(Source.wizard_of_oz_drawings)

restore_objects(Source.wizard_of_oz_drawings)

cinematize(Source.wizard_of_oz_drawings)

💡 The cast for a new movie is ready 😉

Cinematic images have various use cases:

• These cinematized stills can be perfect “reference images” for video generation models like Veo. See Generate Veo videos from reference images.
• As they are photorealistic representations, they can also be a source for generating 2D or 3D visuals, in any style, with realistic figures, perfect proportions, advanced lighting, enhanced compositions…
• You can use them in many professional contexts or for high-end products: presentations, magazines, posters, storyboards, brainstorming sessions…

🏁 Conclusion

• Gemini’s native spatial understanding enables the detection of specific visual objects based on a single prompt in natural language.
• We tested the detection of illustrations in book photos, which traditional machine learning (ML) models usually miss, as they are typically trained to detect people, animals, vehicles, food, and a finite set of physical object classes.
• We tested the detection of straight, tilted, and even significantly warped illustrations, and they were always precisely identified.
• The core implementation was straightforward, requiring minimal code using the Python SDK and customized prompts. By comparison, fine-tuning a traditional object detection model is time-consuming: it involves assembling an image dataset, labeling objects, and managing training jobs.
• This solution is very flexible: we could switch from detecting illustrations to electronic components, by adapting the prompt, while keeping the code unchanged.
• Using structured outputs (with a JSON schema or Pydantic classes, and the Python SDK) makes the code both easy to implement and ready to deploy to production.
• Then, Nano Banana allows editing these visual objects in virtually any way imaginable.
• We tested a workflow with restoration, colorization, and even cinematization steps, using imperative and descriptive prompts.
• The possibilities seem really endless, and the principles in this exploration can be reused in different contexts.

➕ More!

• Try it yourself: Use the companion notebook to reproduce the results in this article.
• Read more: Read Generating Consistent Imagery to explore another image generation use case.
• Get inspired: Check out the Nano Banana recipes notebook for more practical examples.
• Follow me: Connect with me (@PicardParis) on LinkedIn or Twitter-X for more cloud, applied AI, and Python explorations…

Thanks for reading. Let me know if you create something cool!

Originally published at https://towardsdatascience.com on February 26, 2026.

Detecting and Editing Visual Objects with Gemini was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

“Gemini 3 Pro Image” (aka Nano Banana Pro 🍌) just launched (in preview) and is the new state-of-the-art image generation/editing model.

What’s new:

• Advanced reasoning
• Advanced text rendering
• Real-world grounding with Google Search
• High-resolution outputs: 1K, 2K, and 4K
• Up to 14 reference input images

Let’s switch our code directly from 1K to 4K, and check how it behaves compared to the previous test…

🪄 Character sheet

source_ids = [AssetId.ARCHIVE]
prompt = """
- Scene: Robot character sheet.
- Left half: Direct flat front view of the robot.
- Right half: Direct flat back view of the robot (seamless back).
- In both views, the robot wears a same small, brown-felt backpack, with a tiny polished-brass buckle and simple straps.
- Background: Pure white.
- Top text: Caption the image "ROBOT CHARACTER SHEET".
- Bottom text: Caption the views "FRONT VIEW" and "BACK VIEW".
"""
new_id = AssetId.ROBOT

generate_image(source_ids, prompt, new_id)

🔼️💡 Perfect, every time, not missing the backpack straps, in both views, and in 4K!

✨ First scene

source_ids = [AssetId.ROBOT]
prompt = """
- Image 1: Robot character sheet.
- Scene: Macro photography of a beautifully crafted miniature diorama.
- Foreground: A medium-gray felt cliff is visible in the bottom-left corner. The robot stands on the edge of the cliff, viewed from a 3/4 back angle, looking out over a sea of clouds (made of white cotton).
- Background: A range of infinite, interspersed, large dome-like felt mountains, in various shades of medium blue/green, with curvy white snowcaps.
- Angle: Eye level.
- Lighting: Studio, clean and soft.
"""
new_id = AssetId.MOUNTAINS

generate_image(source_ids, prompt, new_id)

🔼️💡 More mountains can be generated. The “infinite” concept is understood.

✨ Successive scenes

source_ids = [AssetId.ROBOT, AssetId.MOUNTAINS]
prompt = """
- Image 1: Robot character sheet.
- Image 2: Previous scene.
- The robot has descended from the cliff to a gray felt valley. It stands in the center, seen directly from the back. It is holding/reading a felt map with outstretched arms.
- Large smooth, round, felt rocks in various beige/gray shades are visible on the sides.
- Background: The distant mountain range. A thin layer of clouds obscures its base and the end of the valley.
- Lighting: Golden hour light, soft and diffused.
"""
new_id = AssetId.VALLEY

generate_image(source_ids, prompt, new_id)

🔼️💡 Beautiful mix and lighting!

source_ids = [AssetId.ROBOT, AssetId.VALLEY]
prompt = """
- Image 1: Robot character sheet.
- Image 2: Previous scene.
- The robot walks to the end of the valley and reaches an infinite forest wall filling the entire background.
- The forest is a dense cluster of tall, slender, simple-upright-elongated-cone-with-rounded-tip-and-no-trunk trees.
- The trees are made from various shades of light/medium/dark green felt.
- The robot is on the right, viewed from a 3/4 rear angle, no longer holding the map, with both hands clasped to its ears in despair.
- On the left & right bottom sides, rocks (similar to image 2) are partially visible.
"""
new_id = AssetId.FOREST

generate_image(source_ids, prompt, new_id)

🔼️💡 It was not possible to generate so many trees before. Nano Banana Pro can “manage many more objects” (if that means anything).

source_ids = [AssetId.ROBOT, AssetId.FOREST]
prompt = """
- Image 1: Robot character sheet.
- Image 2: Previous scene.
- The robot goes through the dense forest and emerges into a clearing.
- The robot is in the center, front-facing the camera.
- The ground is made of green felt, with flat patches of white felt snow. Rocks are no longer visible.
"""
new_id = AssetId.CLEARING

generate_image(source_ids, prompt, new_id)

🔼️💡 Magnificent clearing! Many consistent trees. Footsteps were not specified but are a nice addition to the scene indeed!

source_ids = [AssetId.ROBOT, AssetId.MOUNTAINS]
prompt = """
- Image 1: Robot character sheet.
- Image 2: Previous scene.
- The robot is now climbing the peak of a medium-green mountain, reaching its summit.
- The mountain is in the center of the image, with the robot on its left, viewed from a 3/4 rear angle.
- The robot has both feet on the mountain and is using two felt ice axes (brown handles, gray heads), reaching the snowcap.
- Horizon: Filled by the distant mountain range.
"""
new_id = AssetId.ASCENSION

generate_image(source_ids, prompt, new_id)

🔼️💡 Excellent! Nice focus on the foreground and soft-focus on the background.

source_ids = [AssetId.ROBOT, AssetId.ASCENSION]
prompt = """
- Image 1: Robot character sheet.
- Image 2: Previous scene.
- The robot reaches the top and stands on the summit (facing the camera).
- It is no longer holding the ice axes, which are planted upright in the snow on each side.
- It has both arms raised in sign of victory.
"""
new_id = AssetId.SUMMIT

generate_image(source_ids, prompt, new_id)

🔼️💡 Perfect! Even nicer when displayed in 4K.

source_ids = [AssetId.ROBOT, AssetId.SUMMIT]
prompt = """
- Image 1: Robot character sheet.
- Image 2: Previous scene.
- Remove the ice axes.
- Move the center mountain to the left side of the image and add a slightly taller medium-blue mountain to the right side.
- Suspend a stylized felt bridge between the two mountains: Its deck is made of thick felt planks in various uniform wood shades.
- Place the robot on the center of the bridge with one arm pointing toward the blue mountain.
- Keep the horizon filled by the distant mountain range.
"""
new_id = AssetId.BRIDGE

generate_image(source_ids, prompt, new_id)

🔼️💡 Excellent prompt adherence! You can even specify that “the robot walks toward the blue mountain” or more complex compositions.

source_ids = [AssetId.ROBOT, AssetId.BRIDGE]
prompt = """
- Image 1: Robot character sheet.
- Image 2: Previous scene.
- The robot is sleeping peacefully (both eyes changed into a "closed" state) in a large comfortable brown-and-tan tartan hammock that has replaced the bridge.
- The backpack lays at the robot's feet.
"""
new_id = AssetId.HAMMOCK

generate_image(source_ids, prompt, new_id)

🔼️💡 This is one of the most complex prompts: Nano Banana Pro replaces the bridge with a hammock, and never misses to have “both eyes changed into a ‘closed’ state”. You can even move the “backpack at the robot’s feet” (the character sheet doesn’t have the robot without the backpack)!

🚀 Conclusion

In this limited quick test, we could see the following:

• Advanced reasoning: Consistent results on complex prompts and complex prompts can even be enriched with more constraints.
• Advanced composition: Many more objects can be generated, which can lead to even more breathtaking images.
• Advanced text rendering: The cover image for this article was generated with the following additional instruction: The robot is holding a light-beige crocheted sign in its hands that says, over 3 lines, “I love” “Nano Banana” “Pro 🍌” with navy blue letters, and the emoji in yellow.
• It takes less iterations when trying new scenes, thanks to the advanced prompt understanding and adherence!
• With everything in 4K resolution!

We now have two state-of-the-art image generation/editing models in our toolbox:

• Gemini 2.5 Flash Image for speed and cost-effectiveness
• Gemini 3 Pro Image for advanced reasoning capabilities + 2K + 4K, and a lot more (we just scratched the surface)

➕ More!

• Dive in: Check out the documentation.
• Experiment: Run the getting-started notebook.
• Get inspired: Check out the Nano Banana recipes notebook for more practical examples.
• Explore: See additional use cases in the Vertex AI Prompt Gallery.
• Stay updated: Follow the Vertex AI Release Notes.
• Follow me: Connect with me (@PicardParis) on LinkedIn or Twitter-X for more cloud, applied AI, and Python explorations…

🍌 Testing Gemini 3 Pro Image was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

A few notes before we dive in:

• The complete source code for this article, including future updates, is available in this notebook under the Apache 2.0 license.
• All new images in this article were generated with Gemini Nano Banana using the proof-of-concept generation pipeline explored here.
• This article was originally published at towardsdatascience.com.
• You can experiment with Gemini for free in Google AI Studio. Please note that programmatic API access to Nano Banana is a pay-as-you-go service.
• 2025–10–02: Nano Banana got generally available for production and added support for ten aspect ratios. Notebook & article were updated accordingly.

🔥 Challenge

We all have existing images worth reusing in different contexts. This would generally imply modifying the images, a complex (if not impossible) task requiring very specific skills and tools. This explains why our archives are full of forgotten or unused treasures. State-of-the-art vision models have evolved so much that we can reconsider this problem.

So, can we breathe new life into our visual archives?

Let’s try to complete this challenge with the following steps:

• 1️⃣ Start from an archive image we’d like to reuse
• 2️⃣ Extract a character to create a brand-new reference image
• 3️⃣ Generate a series of images to illustrate the character’s journey, using only prompts and the new assets

For this, we’ll explore the capabilities of “Gemini 2.5 Flash Image”, also known as “Nano Banana” 🍌.

🏁 Setup

🐍 Python packages

We’ll use the following packages:

• google-genai: The Google Gen AI Python SDK lets us call Gemini with a few lines of code
• networkx for graph management

We’ll also use the following dependencies:

• pillow and matplotlib for data visualization
• tenacity for request management

%pip install --quiet "google-genai>=1.40.0" "networkx[default]"

🤖 Gen AI SDK

Create a google.genai client:

from google import genai

check_environment()

client = genai.Client()

Check your configuration:

check_configuration(client)

Using the Vertex AI API with project "..." in location "europe-west1"

🧠 Gemini model

For this challenge, we’ll select the latest Gemini 2.5 Flash Image model:

GEMINI_2_5_FLASH_IMAGE = "gemini-2.5-flash-image"

🛠️ Helpers

Define some helper functions to generate and display images:

import IPython.display
import tenacity
from google.genai.errors import ClientError
from google.genai.types import GenerateContentConfig, ImageConfig, PIL_Image

GEMINI_2_5_FLASH_IMAGE = "gemini-2.5-flash-image"

# You can add the "TEXT" modality for potential textual feedback (or in iterative chat mode)
RESPONSE_MODALITIES = ["IMAGE"]

# Supported aspect ratios: "1:1", "2:3", "3:2", "3:4", "4:3", "4:5", "5:4", "9:16", "16:9", and "21:9"
ASPECT_RATIO = "16:9"

GENERATION_CONFIG = GenerateContentConfig(
    response_modalities=RESPONSE_MODALITIES,
    image_config=ImageConfig(aspect_ratio=ASPECT_RATIO),
)


def generate_content(sources: list[PIL_Image], prompt: str) -> PIL_Image | None:
    prompt = prompt.strip()
    contents = [*sources, prompt] if sources else prompt

    response = None
    for attempt in get_retrier():
        with attempt:
            response = client.models.generate_content(
                model=GEMINI_2_5_FLASH_IMAGE,
                contents=contents,
                config=GENERATION_CONFIG,
            )

    if not response or not response.candidates:
        return None
    if not (content := response.candidates[0].content):
        return None
    if not (parts := content.parts):
        return None

    image: PIL_Image | None = None
    for part in parts:
        if part.text:
            display_markdown(part.text)
            continue
        assert (sdk_image := part.as_image())
        assert (image := sdk_image._pil_image)
        display_image(image)

    return image


def get_retrier() -> tenacity.Retrying:
    return tenacity.Retrying(
        stop=tenacity.stop_after_attempt(7),
        wait=tenacity.wait_incrementing(start=10, increment=1),
        retry=should_retry_request,
        reraise=True,
    )


def should_retry_request(retry_state: tenacity.RetryCallState) -> bool:
    if not retry_state.outcome:
        return False
    err = retry_state.outcome.exception()
    if not isinstance(err, ClientError):
        return False
    print(f"❌ ClientError {err.code}: {err.message}")

    retry = False
    match err.code:
        case 400 if err.message is not None and " try again " in err.message:
            # Workshop: Cloud Storage accessed for the first time (service agent provisioning)
            retry = True
        case 429:
            # Workshop: temporary project with 1 QPM quota
            retry = True
    print(f"🔄 Retry: {retry}")

    return retry


def display_markdown(markdown: str) -> None:
    IPython.display.display(IPython.display.Markdown(markdown))


def display_image(image: PIL_Image) -> None:
    IPython.display.display(image)

🖼️ Assets

Let’s define the assets for our character’s journey and the functions to manage them:

import enum
from collections.abc import Sequence
from dataclasses import dataclass


class AssetId(enum.StrEnum):
    ARCHIVE = "0_archive"
    ROBOT = "1_robot"
    MOUNTAINS = "2_mountains"
    VALLEY = "3_valley"
    FOREST = "4_forest"
    CLEARING = "5_clearing"
    ASCENSION = "6_ascension"
    SUMMIT = "7_summit"
    BRIDGE = "8_bridge"
    HAMMOCK = "9_hammock"


@dataclass
class Asset:
    id: str
    source_ids: Sequence[str]
    prompt: str
    pil_image: PIL_Image


class Assets(dict[str, Asset]):
    def set_asset(self, asset: Asset) -> None:
        # Note: This replaces any existing asset (if needed, add guardrails to auto-save or keep all versions)
        self[asset.id] = asset


def generate_image(source_ids: Sequence[str], prompt: str, new_id: str = "") -> None:
    sources = [assets[source_id].pil_image for source_id in source_ids]
    prompt = prompt.strip()
    image = generate_content(sources, prompt)
    if image and new_id:
        assets.set_asset(Asset(new_id, source_ids, prompt, image))


assets = Assets()

📦 Reference archive

We can now fetch our reference archive and make it our first asset:

import urllib.request

import PIL.Image

ARCHIVE_URL = "https://storage.googleapis.com/github-repo/generative-ai/gemini/use-cases/media-generation/consistent_imagery_generation/0_archive.png"


def load_archive() -> None:
    image = get_image_from_url(ARCHIVE_URL)
    assets.set_asset(Asset(AssetId.ARCHIVE, [], "", image))
    display_image(image)


def get_image_from_url(image_url: str) -> PIL_Image:
    with urllib.request.urlopen(image_url) as response:
        return PIL.Image.open(response)

load_archive()

This archive image was generated in July 2024 with a beta version of Imagen 3, prompted with “On white background, a small hand-felted toy of blue robot. The felt is soft and cuddly…”. The result looked really good but, at the time, there was absolutely no determinism and no consistency. As a result, this was a nice one-shot image generation but the cute little robot seemed gone forever…

⛏️ Asset extraction

Let’s try to extract our little robot:

source_ids = [AssetId.ARCHIVE]
prompt = "Extract the robot in a clean cutout on a solid white fill."

generate_image(source_ids, prompt)

⚠️ The robot is perfectly extracted, but this is essentially a good background removal, which many models can perform. This prompt uses terms from graphics software, whereas we can now reason in terms of image composition. It’s also not necessarily a good idea to try to use traditional binary masks, as object edges and shadows convey significant details about shapes, textures, positions, and lighting.

Let’s go back to our archive to perform an advanced extraction instead, and directly generate a character sheet…

🪄 Character sheet

Gemini has spatial understanding, so it’s able to provide different views while preserving visual features. Let’s generate a front/back character sheet and, as our little robot will go on a journey, also add a backpack at the same time:

source_ids = [AssetId.ARCHIVE]
prompt = """
- Scene: Robot character sheet.
- Left: Front view of the extracted robot.
- Right: Back view of the extracted robot (seamless back).
- The robot wears a same small, brown-felt backpack, with a tiny polished-brass buckle and simple straps in both views. The backpack straps are visible in both views.
- Background: Pure white.
- Text: On the top, caption the image "ROBOT CHARACTER SHEET" and, on the bottom, caption the views "FRONT VIEW" and "BACK VIEW".
"""
new_id = AssetId.ROBOT

generate_image(source_ids, prompt, new_id)

💡 A few remarks:

• Our prompt focuses on the composition of the scene, a common practice in media studios.
• Successive generated images will be consistent, preserving all robot features visible in the provided image. However, since we only specified some features of the backpack (e.g., a single buckle) and left others unspecified, we’ll get slightly different backpacks.
• For simplicity, we directly included the backpack in the character sheet. In a real production pipeline, we would likely make it part of a separate accessory sheet.
• To control the backpack’s exact shape and design, we could also use a reference photo of a real backpack and instruct Gemini to “transform the backpack into a stylized felt version”.
• Gemini can generate 1024 × 1024images (1:1 aspect ratio) or equivalent resolutions (token-wise) for the other supported aspect ratios (2:3, 3:2, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9, and 21:9).
• In the request configuration, we specified aspect_ratio="16:9", which generates images at 1344 × 768 pixels. If this parameter is omitted, Gemini uses the aspect ratio of the input image (the last one if multiple are provided) to select the closest supported aspect ratio.

This new asset can now serve as a design reference in our future image generations.

✨ First scene

Let’s get started with a mountain scenery:

source_ids = [AssetId.ROBOT]
prompt = """
- Image 1: Robot character sheet.
- Scene: Macro photography of a beautifully crafted miniature diorama.
- Background: Soft-focus of a panoramic range of interspersed, dome-like felt mountains, in various shades of medium blue/green, with curvy white snowcaps, extending over the entire horizon.
- Foreground: In the bottom-left, the robot stands on the edge of a medium-gray felt cliff, viewed from a 3/4 back angle, looking out over a sea of clouds (made of white cotton).
- Lighting: Studio, clean and soft.
"""
new_id = AssetId.MOUNTAINS

generate_image(source_ids, prompt, new_id)

💡 The mountain shape is specified as “dome-like” so our character can stand on one of the summits later on.

It’s important to spend some time on this first scene as, in a cascading effect, it will define the overall look of our story. Take some time to refine the prompt or try a couple of times to get the best variation.

From now on, our generation inputs will be both the character sheet and a reference scene…

✨ Successive scenes

Let’s get the robot down a valley:

source_ids = [AssetId.ROBOT, AssetId.MOUNTAINS]
prompt = """
- Image 1: Robot character sheet.
- Image 2: Previous scene.
- The robot has descended from the cliff to a gray felt valley. It stands in the center, seen directly from the back. It is holding/reading a felt map with outstretched arms.
- Large smooth, round, felt rocks in various beige/gray shades are visible on the sides.
- Background: The distant mountain range. A thin layer of clouds obscures its base and the end of the valley.
- Lighting: Golden hour light, soft and diffused.
"""
new_id = AssetId.VALLEY

generate_image(source_ids, prompt, new_id)

💡 A few notes:

• The provided specifications about our input images ("Image 1:...", "Image 2:...") are important. Without them, "the robot" could refer to any of the 3 robots in the input images (2 in the character sheet, 1 in the previous scene). With them, we indicate that it's the same robot. In case of confusion, we can be more specific with "the [entity] from image [number]".
• On the other hand, since we didn’t provide a precise description of the valley, successive requests will give different, interesting, and creative results (we can pick our favorite or make the prompt more precise for more determinism).
• Here, we also tested a different lighting, which significantly changes the whole scene.

Then, we can move forward into this scene:

source_ids = [AssetId.ROBOT, AssetId.VALLEY]
prompt = """
- Image 1: Robot character sheet.
- Image 2: Previous scene.
- The robot goes on and faces a dense, infinite forest of simple, giant, thin trees, that fills the entire background.
- The trees are made from various shades of light/medium/dark green felt.
- The robot is on the right, viewed from a 3/4 rear angle, no longer holding the map, with both hands clasped to its ears in despair.
- On the left & right bottom sides, rocks (similar to image 2) are partially visible.
"""
new_id = AssetId.FOREST

generate_image(source_ids, prompt, new_id)

💡 Of interest:

• We could position the character, change its point of view, and even “animate” its arms for more expressivity.
• The “no longer holding the map” precision prevents the model from trying to keep it from the previous scene in a meaningful way (e.g., the robot dropped the map on the floor).
• We didn’t provide lighting details: The lighting source, quality, and direction have been kept from the previous scene.

Let’s go through the forest:

source_ids = [AssetId.ROBOT, AssetId.FOREST]
prompt = """
- Image 1: Robot character sheet.
- Image 2: Previous scene.
- The robot goes through the dense forest and emerges into a clearing, pushing aside two tree trunks.
- The robot is in the center, now seen from the front view.
- The ground is made of green felt, with flat patches of white felt snow. Rocks are no longer visible.
"""
new_id = AssetId.CLEARING

generate_image(source_ids, prompt, new_id)

💡 We changed the ground but didn’t provide additional details for the view and the forest: The model will generally preserve most of the trees.

Now that the valley-forest sequence is over, we can journey up to the mountains, using the original mountain scene as our reference to return to that environment:

source_ids = [AssetId.ROBOT, AssetId.MOUNTAINS]
prompt = """
- Image 1: Robot character sheet.
- Image 2: Previous scene.
- Close-up of the robot now climbing the peak of a medium-green mountain and reaching its summit.
- The mountain is right in the center, with the robot on its left slope, viewed from a 3/4 rear angle.
- The robot has both feet on the mountain and is using two felt ice axes (brown handles, gray heads), reaching the snowcap.
- Horizon: The distant mountain range.
"""
new_id = AssetId.ASCENSION

generate_image(source_ids, prompt, new_id)

💡 The mountain close-up, inferred from the blurred background, is pretty impressive.

Let’s climb to the summit:

source_ids = [AssetId.ROBOT, AssetId.ASCENSION]
prompt = """
- Image 1: Robot character sheet.
- Image 2: Previous scene.
- The robot reaches the top and stands on the summit, seen in the front view, in close-up.
- It is no longer holding the ice axes, which are planted upright in the snow on each side.
- It has both arms raised in sign of victory.
"""
new_id = AssetId.SUMMIT

generate_image(source_ids, prompt, new_id)

💡 This is a logical follow-up but also a nice, different view.

Now, let’s try something different to significantly recompose the scene:

source_ids = [AssetId.ROBOT, AssetId.SUMMIT]
prompt = """
- Image 1: Robot character sheet.
- Image 2: Previous scene.
- Remove the ice axes.
- Move the center mountain to the left edge of the image and add a slightly taller medium-blue mountain to the right edge.
- Suspend a stylized felt bridge between the two mountains: Its deck is made of thick felt planks in various wood shades.
- Place the robot on the center of the bridge with one arm pointing toward the blue mountain.
- View: Close-up.
"""
new_id = AssetId.BRIDGE

generate_image(source_ids, prompt, new_id)

💡 Of interest:

• This imperative prompt composes the scene in terms of actions. It’s sometimes easier than descriptions.
• A new mountain is added as instructed, and it is both different and consistent.
• The bridge attaches to the summits in very plausible ways and seems to obey the laws of physics.
• The “Remove the ice axes” instruction is here for a reason. Without it, it’s as if we were prompting “do whatever you can with the ice axes from the previous scene: leave them where they are, don’t let the robot leave without them, or anything else”, leading to random results.
• It’s also possible to get the robot to walk on the bridge, seen from the side (which we never generated before), but it’s hard to have it consistently walk from left to right. Adding left and right views in the character sheet should fix this.

Let’s generate a final scene and let the robot get some well-deserved rest:

source_ids = [AssetId.ROBOT, AssetId.BRIDGE]
prompt = """
- Image 1: Robot character sheet.
- Image 2: Previous scene.
- The robot is sleeping peacefully (both eyes changed into a "closed" state), in a comfortable brown-and-tan tartan hammock that has replaced the bridge.
"""
new_id = AssetId.HAMMOCK

generate_image(source_ids, prompt, new_id)

💡 Of interest:

• This time, the prompt is descriptive, and it works as well as the previous imperative prompt.
• The bridge-hammock transformation is really nice and preserves the attachments on the mountain summits.
• The robot transformation is also impressive, as it hasn’t been seen in this position before.
• The closed eyes are the most difficult detail to get consistently (may require a couple of attempts), probably because we’re accumulating many different transformations at once (and diluting the model’s attention). For full control and more deterministic results, we can focus on significant changes over iterative steps, or create various character sheets upfront.

We have illustrated our story with 9 new consistent images! Let’s take a step back to understand what we’ve built…

🗺️ Graph visualization

We now have a collection of image assets, from archives to brand-new generated assets.

Let’s add some data visualization to get a better sense of the steps completed…

🔗 Directed graph

Our new assets are all related, connected by one or more “generated from” links. From a data structure point of view, this is a directed graph.

We can build the corresponding directed graph using the networkx library:

import networkx as nx


def build_graph(assets: Assets) -> nx.DiGraph:
    graph = nx.DiGraph(assets=assets)
    # Nodes
    for asset in assets.values():
        graph.add_node(asset.id, asset=asset)
    # Edges
    for asset in assets.values():
        for source_id in asset.source_ids:
            graph.add_edge(source_id, asset.id)
    return graph


asset_graph = build_graph(assets)
print(asset_graph)

DiGraph with 10 nodes and 16 edges

Let’s place the most used asset in the center and display the other assets around:

import matplotlib.pyplot as plt


def display_basic_graph(graph: nx.Graph) -> None:
    pos = compute_node_positions(graph)
    color = "#4285F4"
    options = dict(
        node_color=color,
        edge_color=color,
        arrowstyle="wedge",
        with_labels=True,
        font_size="small",
        bbox=dict(ec="black", fc="white", alpha=0.7),
    )
    nx.draw(graph, pos, **options)
    plt.show()


def compute_node_positions(graph: nx.Graph) -> dict[str, tuple[float, float]]:
    # Put the most connected node in the center
    center_node = most_connected_node(graph)
    edge_nodes = set(graph) - {center_node}
    pos = nx.circular_layout(graph.subgraph(edge_nodes))
    pos[center_node] = (0.0, 0.0)
    return pos


def most_connected_node(graph: nx.Graph) -> str:
    if not graph.nodes():
        return ""
    centrality_by_id = nx.degree_centrality(graph)
    return max(centrality_by_id, key=lambda s: centrality_by_id.get(s, 0.0))

display_basic_graph(asset_graph)

That’s a correct summary of our different steps. It’d be nice if we could also visualize our assets…

🌟 Asset graph

Let’s add custom matplotlib functions to render the graph nodes with the assets in a more visually appealing way:

import typing
from collections.abc import Iterator
from io import BytesIO
from pathlib import Path

import PIL.Image
import PIL.ImageDraw
from google.genai.types import PIL_Image
from matplotlib.axes import Axes
from matplotlib.backends.backend_agg import FigureCanvasAgg
from matplotlib.figure import Figure
from matplotlib.image import AxesImage
from matplotlib.patches import Patch
from matplotlib.text import Annotation
from matplotlib.transforms import Bbox, TransformedBbox


@enum.unique
class ImageFormat(enum.StrEnum):
    # Matches PIL.Image.Image.format
    WEBP = enum.auto()
    PNG = enum.auto()
    GIF = enum.auto()


def yield_generation_graph_frames(
    graph: nx.DiGraph,
    animated: bool,
) -> Iterator[PIL_Image]:
    def get_fig_ax() -> tuple[Figure, Axes]:
        factor = 1.0
        figsize = (16 * factor, 9 * factor)
        fig, ax = plt.subplots(figsize=figsize)
        fig.tight_layout(pad=3)
        handles = [
            Patch(color=COL_OLD, label="Archive"),
            Patch(color=COL_NEW, label="Generated"),
        ]
        ax.legend(handles=handles, loc="lower right")
        ax.set_axis_off()
        return fig, ax

    def prepare_graph() -> None:
        arrows = nx.draw_networkx_edges(graph, pos, ax=ax)
        for arrow in arrows:
            arrow.set_visible(False)

    def get_box_size() -> tuple[float, float]:
        xlim_l, xlim_r = ax.get_xlim()
        ylim_t, ylim_b = ax.get_ylim()
        factor = 0.08
        box_w = (xlim_r - xlim_l) * factor
        box_h = (ylim_b - ylim_t) * factor
        return box_w, box_h

    def add_axes() -> Axes:
        xf, yf = tr_figure(pos[node])
        xa, ya = tr_axes([xf, yf])
        x_y_w_h = (xa - box_w / 2.0, ya - box_h / 2.0, box_w, box_h)
        a = plt.axes(x_y_w_h)
        a.set_title(
            asset.id,
            loc="center",
            backgroundcolor="#FFF8",
            fontfamily="monospace",
            fontsize="small",
        )
        a.set_axis_off()
        return a

    def draw_box(color: str, image: bool) -> AxesImage:
        if image:
            result = pil_image.copy()
        else:
            result = PIL.Image.new("RGB", image_size, color="white")
        xy = ((0, 0), image_size)
        # Draw box outline
        draw = PIL.ImageDraw.Draw(result)
        draw.rounded_rectangle(xy, box_r, outline=color, width=outline_w)
        # Make everything outside the box outline transparent
        mask = PIL.Image.new("L", image_size, 0)
        draw = PIL.ImageDraw.Draw(mask)
        draw.rounded_rectangle(xy, box_r, fill=0xFF)
        result.putalpha(mask)
        return a.imshow(result)

    def draw_prompt() -> Annotation:
        text = f"Prompt:\n{asset.prompt}"
        margin = 2 * outline_w
        image_w, image_h = image_size
        bbox = Bbox([[0, margin], [image_w - margin, image_h - margin]])
        clip_box = TransformedBbox(bbox, a.transData)
        return a.annotate(
            text,
            xy=(0, 0),
            xytext=(0.06, 0.5),
            xycoords="axes fraction",
            textcoords="axes fraction",
            verticalalignment="center",
            fontfamily="monospace",
            fontsize="small",
            linespacing=1.3,
            annotation_clip=True,
            clip_box=clip_box,
        )

    def draw_edges() -> None:
        STYLE_STRAIGHT = "arc3"
        STYLE_CURVED = "arc3,rad=0.15"
        for parent in graph.predecessors(node):
            edge = (parent, node)
            color = COL_NEW if assets[parent].prompt else COL_OLD
            style = STYLE_STRAIGHT if center_node in edge else STYLE_CURVED
            nx.draw_networkx_edges(
                graph,
                pos,
                [edge],
                width=2,
                edge_color=color,
                style="dotted",
                ax=ax,
                connectionstyle=style,
            )

    def get_frame() -> PIL_Image:
        canvas = typing.cast(FigureCanvasAgg, fig.canvas)
        canvas.draw()
        image_size = canvas.get_width_height()
        image_bytes = canvas.buffer_rgba()
        return PIL.Image.frombytes("RGBA", image_size, image_bytes).convert("RGB")

    COL_OLD = "#34A853"
    COL_NEW = "#4285F4"
    assets = graph.graph["assets"]
    center_node = most_connected_node(graph)
    pos = compute_node_positions(graph)
    fig, ax = get_fig_ax()
    prepare_graph()
    box_w, box_h = get_box_size()
    tr_figure = ax.transData.transform  # Data → display coords
    tr_axes = fig.transFigure.inverted().transform  # Display → figure coords

    for node, data in graph.nodes(data=True):
        if animated:
            yield get_frame()
        # Edges and sub-plot
        asset = data["asset"]
        pil_image = asset.pil_image
        image_size = pil_image.size
        box_r = min(image_size) * 25 / 100  # Radius for rounded rect
        outline_w = min(image_size) * 5 // 100
        draw_edges()
        a = add_axes()  # a is used in sub-functions
        # Prompt
        if animated and asset.prompt:
            box = draw_box(COL_NEW, image=False)
            prompt = draw_prompt()
            yield get_frame()
            box.set_visible(False)
            prompt.set_visible(False)
        # Generated image
        color = COL_NEW if asset.prompt else COL_OLD
        draw_box(color, image=True)

    plt.close()
    yield get_frame()


def draw_generation_graph(
    graph: nx.DiGraph,
    format: ImageFormat,
) -> BytesIO:
    frames = list(yield_generation_graph_frames(graph, animated=False))
    assert len(frames) == 1
    frame = frames[0]

    params: dict[str, typing.Any] = dict()
    match format:
        case ImageFormat.WEBP:
            params.update(lossless=True)

    image_io = BytesIO()
    frame.save(image_io, format, **params)

    return image_io


def draw_generation_graph_animation(
    graph: nx.DiGraph,
    format: ImageFormat,
) -> BytesIO:
    frames = list(yield_generation_graph_frames(graph, animated=True))
    assert 1 <= len(frames)

    if format == ImageFormat.GIF:
        # Dither all frames with the same palette to optimize the animation
        # The animation is cumulative, so most colors are in the last frame
        method = PIL.Image.Quantize.MEDIANCUT
        palettized = frames[-1].quantize(method=method)
        frames = [frame.quantize(method=method, palette=palettized) for frame in frames]

    # The animation will be played in a loop: start cycling with the most complete frame
    first_frame = frames[-1]
    next_frames = frames[:-1]
    INTRO_DURATION = 3000
    FRAME_DURATION = 1000
    durations = [INTRO_DURATION] + [FRAME_DURATION] * len(next_frames)
    params: dict[str, typing.Any] = dict(
        save_all=True,
        append_images=next_frames,
        duration=durations,
        loop=0,
    )
    match format:
        case ImageFormat.GIF:
            params.update(optimize=False)
        case ImageFormat.WEBP:
            params.update(lossless=True)

    image_io = BytesIO()
    first_frame.save(image_io, format, **params)

    return image_io


def display_generation_graph(
    graph: nx.DiGraph,
    format: ImageFormat | None = None,
    animated: bool = False,
    save_image: bool = False,
) -> None:
    if format is None:
        format = ImageFormat.WEBP if running_in_colab_env else ImageFormat.PNG
    if animated:
        image_io = draw_generation_graph_animation(graph, format)
    else:
        image_io = draw_generation_graph(graph, format)

    image_bytes = image_io.getvalue()
    IPython.display.display(IPython.display.Image(image_bytes))

    if save_image:
        stem = "graph_animated" if animated else "graph"
        Path(f"./{stem}.{format.value}").write_bytes(image_bytes)

We can now display our generation graph:

display_generation_graph(asset_graph)

🚀 Challenge completed

We managed to generate a full set of new consistent images with Nano Banana and learned a few things along the way:

• Images prove again that they are worth a thousand words: It’s now a lot easier to generate new images from existing ones and simple instructions.
• We can create or edit images just in terms of composition (letting us all become artistic directors).
• We can use descriptive or imperative instructions.
• The model’s spatial understanding allows 3D manipulations.
• We can add text in our outputs (character sheet) and also refer to text in our inputs (front/back views).
• Consistency can be preserved at very different levels: character, scene, texture, lighting, camera angle/type…
• The generation process can still be iterative but it feels like 10x-100x faster for reaching better-than-hoped-for results.
• It’s now possible to breathe new life into our archives!

Possible next steps:

• The process we followed is essentially a generation pipeline. It can be industrialized for automation (e.g., changing a node regenerates its descendants) or for the generation of different variations in parallel (e.g., the same set of images could be generated for different aesthetics, audiences, or simulations).
• For the sake of simplicity and exploration, the prompts are intentionally simple. In a production environment, they could have a fixed structure with a systematic set of parameters.
• We described scenes as if in a photo studio. Virtually any other imaginable artistic style is possible (photorealistic, abstract, 2D…).
• Our assets could be made self-sufficient by saving prompts and ancestors in the image metadata (e.g., in PNG chunks), allowing for full local storage and retrieval (no database needed and no more lost prompts!). For details, see the “asset metadata” section in the notebook (link below).

As a bonus, let’s end with an animated version of our journey, with the generation graph also showing a glimpse of our instructions:

display_generation_graph(asset_graph, animated=True)

➕ More!

Want to go deeper?

• Try it yourself: Use the article’s accompanying notebook to reproduce the results or generate your own images. If you “run all” the notebook cells, this will generate 10 images for a total cost of $0.39.
• Get inspired: Check out the Nano Banana recipes notebook for more practical examples.
• Explore: See additional use cases in the Vertex AI Prompt Gallery.
• Stay updated: Follow the Vertex AI Release Notes.
• Follow me: Connect with me (@PicardParis) on LinkedIn or Twitter-X for more cloud, applied AI, and Python explorations…

Thanks for reading. I look forward to seeing what you create!

Generating Consistent Imagery with Gemini Nano Banana 🍌 was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Unlocking Multimodal Video Transcription with Gemini

1. 🔥 Challenge
2. 🛠️ Setup
3. 🧪 Prototyping
4. 🏗️ Prompt Crafting
5. 🚀 Finalization
6. ✅ Challenge Complete
7. ⚖️ Analysis, Tips & Optimizations
8. 🏁 Conclusion ◀️

🏁 Conclusion

Multimodal video transcription, which requires the complex synthesis of audio and visual data, is a true challenge for ML practitioners, without mainstream solutions. A traditional approach, involving an elaborate pipeline of specialized models, would be engineering-intensive without any guarantee of success. In contrast, Gemini proved to be a versatile toolbox for reaching a powerful and straightforward solution based on a single prompt:

We managed to address this complex problem with the following techniques:

• Prototyping with open prompts to develop intuition about Gemini’s natural strengths
• Taking into account how LLMs work under the hood
• Crafting increasingly specific prompts using a tabular extraction strategy
• Generating structured outputs to move towards production-ready code
• Adding data visualization for easier interpretation of responses and smoother iterations
• Adapting default parameters to optimize the results
• Conducting more tests, iterating, and even enriching the extracted data

These principles should apply to many other data extraction domains and allow you to solve your own complex problems. Have fun and happy solving!

➕ More!

• Run this notebook to reproduce the results from this article and transcribe your own videos
• Experiment for free in Google AI Studio and get an API key to call Gemini programmatically
• Explore additional use cases in the Vertex AI Prompt Gallery
• Stay updated by following the Vertex AI Release Notes
• Follow me on LinkedIn or Twitter / X for more cloud, applied AI, and Python explorations…
• UPDATE: It was so much fun to generate the intro images for this eight-part series that I had to share it with you. See my next article: Generating Consistent Imagery with Gemini Nano Banana 🍌

Unlocking Multimodal Video Transcription with Gemini — Part 8: 🏁 Conclusion was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Unlocking Multimodal Video Transcription with Gemini

1. 🔥 Challenge
2. 🛠️ Setup
3. 🧪 Prototyping
4. 🏗️ Prompt Crafting
5. 🚀 Finalization
6. ✅ Challenge Complete
7. ⚖️ Analysis, Tips & Optimizations ◀️
8. 🏁 Conclusion

⚖️ Strengths & weaknesses

👍 Strengths

Overall, Gemini is capable of generating excellent transcriptions that surpass human-generated ones in these aspects:

• Consistency of the transcription
• Impressive semantic understanding
• Highly accurate grammar and punctuation
• No typos or transcription system mistakes
• Exhaustivity (every audible word is transcribed)

💡 As you know, a single incorrect/missing word (or even letter) can completely change the meaning. These strengths help ensure high-quality transcriptions and reduce the risk of misunderstandings.

If we compare YouTube’s user-provided transcriptions (sometimes by professional caption vendors) to our auto-generated ones, we can observe some significant differences. Here are some examples from the last test:

👎 Weaknesses

The current prompt isn’t perfect, though. It focuses first on the audio for transcription and then on all cues for speaker data extraction. Though Gemini natively ensures a very high consolidation from the context, the prompt can lead to these side effects:

• Sensitivity to speakers’ pronunciation or accent
• Misspellings for proper nouns
• Inconsistencies between the transcription and a perfectly identified speaker’s name

Here are examples from the same test:

We’ll stop our exploration here and leave it as an exercise, but here are possible ways to fix these errors, in order of simplicity/cost:

• Update the prompt to use visual cues for proper nouns, such as “Ensure all proper nouns (people, companies, products, etc.) are spelled correctly and consistently. Prioritize on-screen text for reference.”
• Enrich the prompt with an additional preliminary table to extract the proper nouns and use them explicitly in the context
• Add available video context metadata in the prompt
• Split the prompt into two successive requests

📈 Tips & optimizations

🔧 Model selection

Each model can differ in terms of performance, speed, and cost.

Here’s a practical summary based on the model specifications, our video test suite, and the current prompt:

🔧 Video segment

You don’t always need to analyze videos from start to finish. You can indicate a video segment with start and/or end offsets in the VideoMetadata structure.

In this example, Gemini will only analyze the 30:00–50:00 segment of the video:

video_metadata = VideoMetadata(
    start_offset="1800.0s",
    end_offset="3000.0s",
    …
)

🔧 Media resolution

In our test suite, the videos are fairly standard. We got excellent results by using a “low” media resolution (“medium” being the default), specified with the GenerateContentConfig.media_resolution parameter.

💡 This provides faster and cheaper inferences, while also enabling the analysis of videos that are three times as long.

We used a simple heuristic based on video duration, but you might want to make it dynamic on a per-video basis:

def get_media_resolution_for_video(video: Video) -> MediaResolution | None:
    if not (video_duration := get_video_duration(video)):
        return None  # Default

    # For testing purposes, this is based on video duration, as our short videos tend to be more detailed
    less_than_five_minutes = video_duration < timedelta(minutes=5)
    if less_than_five_minutes:
        media_resolution = MediaResolution.MEDIA_RESOLUTION_MEDIUM
    else:
        media_resolution = MediaResolution.MEDIA_RESOLUTION_LOW

    return media_resolution

⚠️ If you select a “low” media resolution and experience an apparent loss of understanding, you might be losing important details in the sampled video frames. This is easy to fix: switch back to the default media resolution.

🔧 Sampling frame rate

The default sampling frame rate of 1 FPS worked fine in our tests. You might want to customize it for each video:

SamplingFrameRate = float

def get_sampling_frame_rate_for_video(video: Video) -> SamplingFrameRate | None:
    sampling_frame_rate = None  # Default (1 FPS for current models)

    # [Optional] Define a custom FPS: 0.0 < sampling_frame_rate <= 24.0

    return sampling_frame_rate

💡 You can mix the parameters. In this extreme example, assuming the input video has a 24fps frame rate, all frames will be sampled for a 10s segment:

video_metadata = VideoMetadata(
    start_offset="42.0s",
    end_offset="52.0s",
    fps=24.0,
)

⚠️ If you use a higher sampling rate, this multiplies the number of frames (and tokens) accordingly, increasing latency and cost. As 10s × 24fps = 240 frames = 4×60s × 1fps, this 10-second analysis at 24 FPS is equivalent to a 4-minute default analysis at 1 FPS.

🎯 Precision vs recall

The prompt can influence the precision and recall of our data extractions, especially when using explicit versus implicit wording. If you want more qualitative results, favor precision using explicit wording; if you want more quantitative results, favor recall using implicit wording:

Here are examples that can lead to subtly different results:

💡 Different models can also behave differently for the same prompt. In particular, more performant models might seem more “confident” and make more implicit inferences or consolidations.

💡 As an example, in this AlphaFold video, at the 04:57 timecode, “Spring 2020” is first displayed as context. Then, a short declaration from “The Prime Minister” is heard in the background (“You must stay at home”) without any other hints. When asked to “identify” (rather than “extract”) the speaker, Gemini is likely to infer more and attribute the voice to “Boris Johnson”. There’s absolutely no explicit mention of Boris Johnson; his identity is correctly inferred from the context (“UK”, “Spring 2020”, and “The Prime Minister”).

🏷️ Metadata

In our current tests, Gemini only uses audio and frame tokens, tokenized from sources on Google Cloud Storage or YouTube. If you have additional video metadata, this can be a goldmine; try to add it to your prompt and enrich the video context for better results upfront.

Potentially helpful metadata:

• Video description: This can provide a better understanding of where and when the video was shot.
• Speaker info: This can help auto-correct names that are only heard and not obvious to spell.
• Entity info: Overall, this can help get better transcriptions for custom or private data.

💡 For YouTube videos, no additional metadata or transcript is fetched. Gemini only receives the raw audio and video streams. You can check this yourself by comparing your results with YouTube’s automatic captioning (no punctuation, audio only) or user-provided transcripts (cleaned up), when available.

💡 If you know your video concerns a team or a company, adding internal data in the context can help correct or complete the requested speaker names (provided there are no homonyms in the same context), companies, and job titles.

💡 In this French reportage, in the 06:16–06:31 segment, there are two dogs: Arnold and Rio. “Arnold” is clearly audible, repeated three times, and correctly transcribed. “Rio” is called only once, audible for a fraction of a second in a noisy environment, and the audio transcription can vary. Providing the names of the whole team (owners & dogs, even if they are not all in the video) can help in transcribing this short name consistently.

💡 It should also be possible to ground the results with Google Search, Google Maps, or your own RAG system. See Grounding overview.

🔬 Debugging & evidence

Iterating through successive prompts and debugging LLM outputs can be challenging, especially when trying to understand the reasons for the results.

It’s possible to ask Gemini to provide evidence in the response. In our video transcription solution, we could request a timecoded “evidence” for each speaker’s identified name, company, or role. This enables linking results to their sources, discovering and understanding unexpected insights, checking potential false positives…

💡 In the tested videos, when trying to understand where the insights came from, requesting evidence yielded very insightful explanations, for example:

- Person names could be extracted from various sources (video conference captions, badges, unseen participants introducing themselves when asking questions during a conference panel…)

- Company names could be found from text on uniforms, backpacks, vehicles…

💡 In a document data extraction solution, we could request to provide an “excerpt” as evidence, including page number, chapter number, or any other relevant location information.

🐘 Verbose JSON

The JSON format is currently the most common way to generate structured outputs with LLMs. However, JSON is a rather verbose data format, as field names are repeated for each object. For example, an output can look like the following, with many repeated underlying tokens:

{
  "task1_transcripts": [
    { "start": "00:02", "text": "We've…", "voice": 1 },
    { "start": "00:07", "text": "But we…", "voice": 1 }
    // …
  ],
  "task2_speakers": [
    {
      "voice": 1,
      "name": "John Moult",
      "company": "University of Maryland",
      "position": "Co-Founder, CASP",
      "role_in_video": "Expert"
    },
    // …
    {
      "voice": 3,
      "name": "Demis Hassabis",
      "company": "DeepMind",
      "position": "Founder and CEO",
      "role_in_video": "Team Leader"
    }
    // …
  ]
}

To optimize output size, an interesting possibility is to ask Gemini to generate an XML block containing a CSV for each of your tabular extractions. The field names are specified once in the header, and by using tab separators, for example, we can achieve more compact outputs like the following:

<TASK1_TRANSCRIPT_CSV>
start  text     voice
00:02  We've…   1
00:07  But we…  1
…
</TASK1_TRANSCRIPT_CSV>
<TASK2_SPEAKER_CSV>
voice  name            company                 position          role_in_video
1      John Moult      University of Maryland  Co-Founder, CASP  Expert
…
3      Demis Hassabis  DeepMind                Founder and CEO   Team Leader
…
</TASK2_SPEAKER_CSV>

💡 Gemini excels at patterns and formats. Depending on your needs, feel free to experiment with JSON, XML, CSV, YAML, and any custom structured formats. It’s likely that the industry will evolve to allow even more elaborate structured outputs.

🐿️ Context caching

Context caching optimizes the cost and the latency of repeated requests using the same base inputs.

There are two ways requests can benefit from context caching:

• Implicit caching: By default, upon the first request, input tokens are cached, to accelerate responses for subsequent requests with the same base inputs. This is fully automated and no code change is required.
• Explicit caching: You place specific inputs into the cache and reuse this cached content as a base for your requests. This provides full control but requires managing the cache manually.

Example of implicit caching:

model_id = "gemini-2.0-flash"
video_file_data = FileData(
    file_uri="gs://bucket/path/to/my-video.mp4",
    mime_type="video/mp4",
)
video = Part(file_data=video_file_data)
prompt_1 = "List the people visible in the video."
prompt_2 = "Summarize what happens to John Smith."

# ✅ Request A1: static data (video) placed first
response = client.models.generate_content(
    model=model_id,
    contents=[video, prompt_1],
)

# ✅ Request A2: likely cache hit for the video tokens
response = client.models.generate_content(
    model=model_id,
    contents=[video, prompt_2],
)

💡 Implicit caching can be disabled at the project level (see data governance).

Implicit caching is prefix-based, meaning it only works if you put static data first and variable data last.

Example of requests preventing implicit caching:

# ❌ Request B1: variable input placed first
response = client.models.generate_content(
    model=model_id,
    contents=[prompt_1, video],
)

# ❌ Request B2: no cache hit
response = client.models.generate_content(
    model=model_id,
    contents=[prompt_2, video],
)

💡 This explains why the data-plus-instructions input order is preferred, for performance (not LLM-related) reasons.

Cost-wise, the input tokens retrieved with a cache hit benefit from a 90% discount in the following cases:

• Implicit caching: With all Gemini models, cache hits are automatically discounted (without any control on the cache or cache-hit guarantee).
• Explicit caching: With all Gemini models and supported models in Model Garden, you control your cached inputs and their lifespans to ensure cache hits.

Example of explicit caching:

from google.genai.types import (
    Content,
    CreateCachedContentConfig,
    FileData,
    GenerateContentConfig,
    Part,
)

model_id = "gemini-2.0-flash-001"

# Input video
video_file_data = FileData(
    file_uri="gs://cloud-samples-data/video/JaneGoodall.mp4",
    mime_type="video/mp4",
)
video_part = Part(file_data=video_file_data)
video_contents = [Content(role="user", parts=[video_part])]

# Video explicitly put in cache, with time-to-live (TTL) before automatic deletion
cached_content = client.caches.create(
    model=model_id,
    config=CreateCachedContentConfig(
        ttl="1800s",
        display_name="video-cache",
        contents=video_contents,
    ),
)
if cached_content.usage_metadata:
    print(f"Cached tokens: {cached_content.usage_metadata.total_token_count or 0:,}")
    # Cached tokens: 46,171
    # ✅ Video tokens are cached (standard tokenization rate + storage cost for TTL duration)

cache_config = GenerateContentConfig(cached_content=cached_content.name)

# Request #1
response = client.models.generate_content(
    model=model_id,
    contents="List the people mentioned in the video.",
    config=cache_config,
)
if response.usage_metadata:
    print(f"Input tokens : {response.usage_metadata.prompt_token_count or 0:,}")
    print(f"Cached tokens: {response.usage_metadata.cached_content_token_count or 0:,}")
    # Input tokens : 46,178
    # Cached tokens: 46,171
    # ✅ Cache hit (90% discount)

# Request #i (within the TTL period)
# …

# Request #n (within the TTL period)
response = client.models.generate_content(
    model=model_id,
    contents="List all the timecodes when Jane Goodall is mentioned.",
    config=cache_config,
)
if response.usage_metadata:
    print(f"Input tokens : {response.usage_metadata.prompt_token_count or 0:,}")
    print(f"Cached tokens: {response.usage_metadata.cached_content_token_count or 0:,}")
    # Input tokens : 46,182
    # Cached tokens: 46,171
    # ✅ Cache hit (90% discount)

💡 Explicit caching needs a specific model version (like ...-001 in this example) to ensure the cache remains valid and is not affected by a model update.

ℹ️ Learn more about Context caching.

⏳ Batch prediction

If you need to process a large volume of videos and don’t need synchronous responses, you can use a single batch request and reduce your cost.

💡 Batch requests for Gemini models get a 50% discount compared to standard requests.

ℹ️ Learn more about Batch prediction.

♾️ To production… and beyond

A few additional notes:

• The current prompt is not perfect and can be improved. It has been preserved in its current state to illustrate its development starting with Gemini 2.0 Flash and a simple video test suite.
• The Gemini 2.5 models are more capable and intrinsically provide a better video understanding. However, the current prompt has not been optimized for them. Writing optimal prompts for different models is another challenge.
• If you test transcribing your own videos, especially different types of videos, you may run into new or specific issues. They can probably be addressed by enriching the prompt.
• Future models will likely support more output features. This should allow for richer structured outputs and simpler prompts.
• As models keep learning, it’s also possible that multimodal video transcription will become a one-liner prompt.
• Gemini’s image and audio tokenizers are truly impressive and enable many other use cases. To fully grasp the extent of the possibilities, you can run unit tests on images or audio files.
• We constrained our challenge to using a single request, which optimizes the solution both for speed and cost.
• For applications demanding the absolute highest transcription accuracy, we could isolate the audio-only transcription in a first request before performing speaker identification on the video frames in a second request. It might produce many more voice identifiers than actual speakers, but it should minimize false positives. In the second step, we’d reinject the transcription to focus on extracting and consolidating speaker data from the video frames. This two-step approach would also be a viable strategy to process very long videos, even those several hours in duration.

▶️ Next up: 🏁 Conclusion

Unlocking Multimodal Video Transcription with Gemini — Part 7: ⚖️ Analysis, Tips & Optimizations was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Unlocking Multimodal Video Transcription with Gemini

1. 🔥 Challenge
2. 🛠️ Setup
3. 🧪 Prototyping
4. 🏗️ Prompt Crafting
5. 🚀 Finalization
6. ✅ Challenge Complete ◀️
7. ⚖️ Analysis, Tips & Optimizations
8. 🏁 Conclusion

✅ Challenge completed

🎬 Short video

This video is a trailer for the Google DeepMind podcast. It features a fast-paced montage of 6 interviews. The multimodal transcription is excellent:

transcribe_video(TestVideo.GDM_PODCAST_TRAILER_PT59S)

Video (source)

----------------- GDM_PODCAST_TRAILER_PT59S / gemini-2.0-flash -----------------
Input tokens   :    16,917
Output tokens  :       989

Speakers (6)

Transcripts (13)

🎬 Narrator-only video

This video is a documentary that takes viewers on a virtual tour of the Gombe National Park in Tanzania. There’s no visible speaker. Jane Goodall is correctly detected as the narrator, her name is extracted from the credits:

transcribe_video(TestVideo.JANE_GOODALL_PT2M42S)

Video (source)

------------------- JANE_GOODALL_PT2M42S / gemini-2.0-flash --------------------
Input tokens   :    46,324
Output tokens  :       717

Speakers (1)

Transcripts (14)

💡 Over the past few years, I have regularly used this video to test specialized ML models and these tests consistently resulted in various types of errors. Gemini’s transcription, including punctuation, is perfect.

🎬 French video

This French reportage combines on-the-ground footage of a specialized team using trained dogs to detect leaks in underground drinking water pipes. The recording takes place entirely outdoors in a rural setting. The interviewed workers are introduced with on-screen text overlays. The audio, captured live on location, includes ambient noise. There are also some off-screen or unidentified speakers. This video is rather complex. The multimodal transcription provides excellent results with no false positives:

transcribe_video(TestVideo.BRUT_FR_DOGS_WATER_LEAK_PT8M28S)

Video (source)

-------------- BRUT_FR_DOGS_WATER_LEAK_PT8M28S / gemini-2.0-flash --------------
Input tokens   :    46,514
Output tokens  :     4,924

Speakers (14)

Transcripts (61)

💡 Our prompt was crafted and tested with English videos, but it works without modification with this French video. It should also work for videos in these 100+ different languages.

💡 In a multilingual solution, we might ask to translate our transcriptions into any of those 100+ languages and even perform text cleanup. This can be done in a second request, as the multimodal transcription is complex enough by itself.

💡 Gemini’s audio tokenizer detects more than speech. If you try to list non-speech sounds on audio tracks only (to ensure the response doesn’t benefit from any visual cues), you’ll see it can detect sounds such as “dog bark”, “music”, “sound effect”, “footsteps”, “laughter”, “applause”…

💡 In our data visualization tables, colored rows are inference positives (speakers identified by the model), while gray rows correspond to negatives (unidentified speakers). This makes it easier to understand the results. As the prompt we crafted favors accuracy over recall, colored rows are generally correct, and gray rows correspond either to unnamed/unidentifiable speakers (true negatives) or to speakers that should have been identified (false negatives).

🎬 Complex video

This Google DeepMind video is quite complex:

• It is highly edited and very dynamic
• Speakers are often off-screen and other people can be visible instead
• The researchers are often in groups and it’s not always obvious who’s speaking
• Some video shots were taken 2 years apart: the same speakers can sound and look different!

Gemini 2.0 Flash generates an excellent transcription. However, the complexity of the video can lead to some missed consolidations. Gemini 2.5 Pro shows a deeper inference and manages to consolidate the differently-looking-and-sounding speakers:

transcribe_video(
    TestVideo.GDM_ALPHAFOLD_PT7M54S,
    model=Model.GEMINI_2_5_PRO,
)

Video (source)

-------------------- GDM_ALPHAFOLD_PT7M54S / gemini-2.5-pro --------------------
Input tokens   :    43,354
Output tokens  :     4,861
Thoughts tokens:        80

Speakers (11)

Transcripts (81)

🎬 Long transcription

The total length of the transcribed text can quickly reach the maximum number of output tokens. With our current JSON response schema, we can reach 8,192 output tokens (supported by Gemini 2.0) with transcriptions of ~25min videos. Gemini 2.5 models support up to 65,536 output tokens (8x more) and let us transcribe longer videos.

For this 54-minute panel discussion, Gemini 2.5 Pro uses only ~30–35% of the input/output token limits:

transcribe_video(
    TestVideo.GDM_AI_FOR_SCIENCE_FRONTIER_PT54M23S,
    model=Model.GEMINI_2_5_PRO,
)

Video (source)

------------ GDM_AI_FOR_SCIENCE_FRONTIER_PT54M23S / gemini-2.5-pro -------------
Input tokens   :   297,153
Output tokens  :    22,896
Thoughts tokens:        65

Speakers (14)

Transcripts (593)

💡 In this long video, the five panelists are correctly transcribed, diarized, and identified. In the second half of the video, unseen attendees ask questions to the panel. They are correctly identified as audience members and, though their names and companies are never written on the screen, Gemini correctly extracts and even consolidates the information from the audio cues.

🎬 1h+ video

In the latest Google I/O keynote video (1h 10min):

• ~30–35%% of the token limit is used (383k/1M in, 20/64k out)
• The dozen speakers are nicely identified, including the demo “AI Voices” (esp. “Casey”)
• Speaker names are extracted from slanted text on the background screen for the live keynote speakers (e.g., Josh Woodward at 0:07) and from lower-third on-screen text in the DolphinGemma reportage (e.g., Dr. Denise Herzing at 1:05:28)

transcribe_video(
    TestVideo.GOOGLE_IO_DEV_KEYNOTE_PT1H10M03S,
    model=Model.GEMINI_2_5_PRO,
)

Video (source)

-------------- GOOGLE_IO_DEV_KEYNOTE_PT1H10M03S / gemini-2.5-pro ---------------
Input tokens   :   382,699
Output tokens  :    19,772
Thoughts tokens:        75

Speakers (14)

Transcripts (201)

🎬 40 speaker video

In this 1h 40min Google Cloud Next keynote video:

• ~50–70% of the token limit is used (547k/1M in, 45/64k out)
• 40 distinct voices are diarized
• 29 speakers are identified, connected to their 21 respective companies or divisions
• The transcription takes up to 8 minutes (approximately 4 minutes with video tokens cached), which is 13 to 23 times faster than watching the entire video without pauses.

transcribe_video(
    TestVideo.GOOGLE_CLOUD_NEXT_PT1H40M03S,
    model=Model.GEMINI_2_5_PRO,
)

Video (source)

---------------- GOOGLE_CLOUD_NEXT_PT1H40M03S / gemini-2.5-pro -----------------
Input tokens   :   546,590
Output tokens  :    45,398
Thoughts tokens:        74

Speakers (40)

Transcripts (853)

▶️ Next up: ⚖️ Analysis, Tips & Optimizations

Unlocking Multimodal Video Transcription with Gemini — Part 6: ✅ Challenge Complete was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Unlocking Multimodal Video Transcription with Gemini

1. 🔥 Challenge
2. 🛠️ Setup
3. 🧪 Prototyping
4. 🏗️ Prompt Crafting
5. 🚀 Finalization ◀️
6. ✅ Challenge Complete
7. ⚖️ Analysis, Tips & Optimizations
8. 🏁 Conclusion

🚀 Finalization

🧩 Structured output

We’ve iterated towards a precise and concise prompt. Now, we can focus on Gemini’s response:

• The response is plain text containing fenced code blocks
• Instead, we’d like a structured output, to receive consistently formatted responses
• Ideally, we’d also like to avoid having to parse the response, which can be a maintenance burden

Getting structured outputs is an LLM feature also called “controlled generation”. Since we’ve already crafted our prompt in terms of data tables and JSON fields, this is now a formality.

In our request, we can add the following parameters:

• response_mime_type="application/json"
• response_schema="YOUR_JSON_SCHEMA" (docs)

In Python, this gets even easier:

• Use the pydantic library
• Reflect your output structure with classes derived from pydantic.BaseModel

We can simplify the prompt by removing the output specification parts:

Generate a JSON object with keys `task1_transcripts` and `task2_speakers` for the following tasks.
…
- The `task1_transcripts` value is a JSON array where each object has the following fields:
  - `start`
  - `text`
  - `voice`
…
- The `task2_speakers` value is a JSON array where each object has the following fields:
  - `voice`
  - `name`

… to move them to matching Python classes instead:

import pydantic

class Transcript(pydantic.BaseModel):
    start: str
    text: str
    voice: int

class Speaker(pydantic.BaseModel):
    voice: int
    name: str

class VideoTranscription(pydantic.BaseModel):
    task1_transcripts: list[Transcript] = pydantic.Field(default_factory=list)
    task2_speakers: list[Speaker] = pydantic.Field(default_factory=list)

… and request a structured response:

response = client.models.generate_content(
    # …
    config=GenerateContentConfig(
        # …
        response_mime_type="application/json",
        response_schema=VideoTranscription,
        # …
    ),
)

Finally, retrieving the objects from the response is also direct:

if isinstance(response.parsed, VideoTranscription):
    video_transcription = response.parsed
else:
    video_transcription = VideoTranscription()  # Empty transcription

The interesting aspects of this approach are the following:

• The prompt focuses on the logic and the classes focus on the output
• It’s easier to update and maintain typed classes
• The JSON schema is automatically generated by the Gen AI SDK from the class provided in response_schema and dispatched to Gemini
• The response is automatically parsed by the Gen AI SDK and deserialized into the corresponding Python objects

⚠️ If you keep output specifications in your prompt, ensure there are no contradictions between the prompt and the schema (e.g., same field names and order), as this can negatively impact the quality of the responses.

💡 It’s possible to have more structural information directly in the schema (e.g., detailed field definitions). See Controlled generation.

✨ Implementation

Let’s finalize our code. In addition, now that we have a stable prompt, we can even enrich our solution to extract each speaker’s company, position, and role_in_video:

import re

import pydantic
from google.genai.types import MediaResolution, ThinkingConfig

SamplingFrameRate = float
NOT_FOUND = "?"
VIDEO_TRANSCRIPTION_PROMPT = f"""
**Task 1 - Transcripts**

- Watch the video and listen carefully to the audio.
- Identify the distinct voices using a `voice` ID (1, 2, 3, etc.).
- Transcribe the video's audio verbatim with voice diarization.
- Include the `start` timecode ({{timecode_spec}}) for each speech segment.

**Task 2 - Speakers**

- For each `voice` ID from Task 1, extract information about the corresponding speaker.
- Use visual and audio cues.
- If a piece of information cannot be found, use `{NOT_FOUND}` as the value.
"""


class Transcript(pydantic.BaseModel):
    start: str
    text: str
    voice: int


class Speaker(pydantic.BaseModel):
    voice: int
    name: str
    company: str
    position: str
    role_in_video: str


class VideoTranscription(pydantic.BaseModel):
    task1_transcripts: list[Transcript] = pydantic.Field(default_factory=list)
    task2_speakers: list[Speaker] = pydantic.Field(default_factory=list)


def get_generate_content_config(model: Model, video: Video) -> GenerateContentConfig:
    media_resolution = get_media_resolution_for_video(video)
    thinking_config = get_thinking_config(model)

    return GenerateContentConfig(
        temperature=DEFAULT_CONFIG.temperature,
        top_p=DEFAULT_CONFIG.top_p,
        seed=DEFAULT_CONFIG.seed,
        response_mime_type="application/json",
        response_schema=VideoTranscription,
        media_resolution=media_resolution,
        thinking_config=thinking_config,
    )


def get_video_duration(video: Video) -> timedelta | None:
    # For testing purposes, video duration is statically specified in the enum name
    # Suffix (ISO 8601 based): _PT[<h>H][<m>M][<s>S]
    # For production,
    # - fetch durations dynamically or store them separately
    # - take into account video VideoMetadata.start_offset & VideoMetadata.end_offset
    regex = r"_PT(?:(\d+)H)?(?:(\d+)M)?(?:(\d+)S)?$"
    if not (match := re.search(regex, video.name)):
        print(f"⚠️ No duration info in {video.name}. Will use defaults.")
        return None

    h_str, m_str, s_str = match.groups()
    return timedelta(
        hours=int(h_str or 0), minutes=int(m_str or 0), seconds=int(s_str or 0)
    )

def get_media_resolution_for_video(video: Video) -> MediaResolution | None:
    if not (video_duration := get_video_duration(video)):
        return None  # Default

    # For testing purposes, this is based on video duration, as our short videos tend to be more detailed
    less_than_five_minutes = video_duration < timedelta(minutes=5)
    if less_than_five_minutes:
        media_resolution = MediaResolution.MEDIA_RESOLUTION_MEDIUM
    else:
        media_resolution = MediaResolution.MEDIA_RESOLUTION_LOW

    return media_resolution


def get_sampling_frame_rate_for_video(video: Video) -> SamplingFrameRate | None:
    sampling_frame_rate = None  # Default (1 FPS for current models)

    # [Optional] Define a custom FPS: 0.0 < sampling_frame_rate <= 24.0

    return sampling_frame_rate


def get_timecode_spec_for_model_and_video(model: Model, video: Video) -> str:
    timecode_spec = "MM:SS"  # Default

    match model:
        case Model.GEMINI_2_0_FLASH:  # Supports MM:SS
            pass
        case Model.GEMINI_2_5_FLASH | Model.GEMINI_2_5_PRO:  # Support MM:SS and H:MM:SS
            duration = get_video_duration(video)
            one_hour_or_more = duration is not None and timedelta(hours=1) <= duration
            if one_hour_or_more:
                timecode_spec = "MM:SS or H:MM:SS"
        case _:
            raise NotImplementedError(f"Undefined timecode spec for {model.name}.")

    return timecode_spec


def get_thinking_config(model: Model) -> ThinkingConfig | None:
    # Examples of thinking configurations (Gemini 2.5 models)
    match model:
        case Model.GEMINI_2_5_FLASH:  # Thinking disabled
            return ThinkingConfig(thinking_budget=0, include_thoughts=False)
        case Model.GEMINI_2_5_PRO:  # Minimum thinking budget and no summarized thoughts
            return ThinkingConfig(thinking_budget=128, include_thoughts=False)
        case _:
            return None  # Default


def get_video_transcription_from_response(
    response: GenerateContentResponse,
) -> VideoTranscription:
    if isinstance(response.parsed, VideoTranscription):
        return response.parsed
    
    print("❌ Could not parse the JSON response")
    return VideoTranscription()  # Empty transcription


def get_video_transcription(
    video: Video,
    video_segment: VideoSegment | None = None,
    fps: float | None = None,
    prompt: str | None = None,
    model: Model | None = None,
) -> VideoTranscription:
    model = model or Model.DEFAULT
    model_id = model.value

    fps = fps or get_sampling_frame_rate_for_video(video)
    video_part = get_video_part(video, video_segment, fps)
    if not video_part:  # Unsupported source, return an empty transcription
        return VideoTranscription()
    if prompt is None:
        timecode_spec = get_timecode_spec_for_model_and_video(model, video)
        prompt = VIDEO_TRANSCRIPTION_PROMPT.format(timecode_spec=timecode_spec)
    contents = [video_part, prompt.strip()]

    config = get_generate_content_config(model, video)

    print(f" {video.name} / {model_id} ".center(80, "-"))
    response = None
    for attempt in get_retrier():
        with attempt:
            response = client.models.generate_content(
                model=model_id,
                contents=contents,
                config=config,
            )
            display_response_info(response)

    assert isinstance(response, GenerateContentResponse)
    return get_video_transcription_from_response(response)

Test it:

def test_structured_video_transcription(video: Video) -> None:
    transcription = get_video_transcription(video)

    print("-" * 80)
    print(f"Transcripts : {len(transcription.task1_transcripts):3d}")
    print(f"Speakers    : {len(transcription.task2_speakers):3d}")
    for speaker in transcription.task2_speakers:
        print(f"- {speaker}")


test_structured_video_transcription(TestVideo.GDM_PODCAST_TRAILER_PT59S)

----------------- GDM_PODCAST_TRAILER_PT59S / gemini-2.0-flash -----------------
Input tokens   :    16,917
Output tokens  :       989
--------------------------------------------------------------------------------
Transcripts :  13
Speakers    :   6
- voice=1 name='Professor Hannah Fry' company='Google DeepMind' position='Host' role_in_video='Host'
- voice=2 name='Demis Hassabis' company='Google DeepMind' position='Co-Founder & CEO' role_in_video='Interviewee'
- voice=3 name='Anca Dragan' company='?' position='Director, AI Safety & Alignment' role_in_video='Interviewee'
- voice=4 name='Pushmeet Kohli' company='?' position='VP Science & Strategic Initiatives' role_in_video='Interviewee'
- voice=5 name='Jeff Dean' company='?' position='Chief Scientist' role_in_video='Interviewee'
- voice=6 name='Douglas Eck' company='?' position='Senior Research Director' role_in_video='Interviewee'

📊 Data visualization

We started prototyping in natural language, crafted a prompt, and generated a structured output. Since reading raw data can be cumbersome, we can now present video transcriptions in a more visually appealing way.

Here’s a possible orchestrator function:

def transcribe_video(video: Video, …) -> None:
    display_video(video)
    transcription = get_video_transcription(video, …)
    display_speakers(transcription)
    display_transcripts(transcription)

Let’s add some data visualization functions…

import itertools
from collections.abc import Callable, Iterator

from pandas import DataFrame, Series
from pandas.io.formats.style import Styler
from pandas.io.formats.style_render import CSSDict

BGCOLOR_COLUMN = "bg_color"  # Hidden column to store row background colors


def yield_known_speaker_color() -> Iterator[str]:
    PAL_40 = ("#669DF6", "#EE675C", "#FCC934", "#5BB974")
    PAL_30 = ("#8AB4F8", "#F28B82", "#FDD663", "#81C995")
    PAL_20 = ("#AECBFA", "#F6AEA9", "#FDE293", "#A8DAB5")
    PAL_10 = ("#D2E3FC", "#FAD2CF", "#FEEFC3", "#CEEAD6")
    PAL_05 = ("#E8F0FE", "#FCE8E6", "#FEF7E0", "#E6F4EA")
    return itertools.cycle([*PAL_40, *PAL_30, *PAL_20, *PAL_10, *PAL_05])


def yield_unknown_speaker_color() -> Iterator[str]:
    GRAYS = ["#80868B", "#9AA0A6", "#BDC1C6", "#DADCE0", "#E8EAED", "#F1F3F4"]
    return itertools.cycle(GRAYS)


def get_color_for_voice_mapping(speakers: list[Speaker]) -> dict[int, str]:
    known_speaker_color = yield_known_speaker_color()
    unknown_speaker_color = yield_unknown_speaker_color()

    mapping: dict[int, str] = {}
    for speaker in speakers:
        if speaker.name != NOT_FOUND:
            color = next(known_speaker_color)
        else:
            color = next(unknown_speaker_color)
        mapping[speaker.voice] = color

    return mapping


def get_table_styler(df: DataFrame) -> Styler:
    def join_styles(styles: list[str]) -> str:
        return ";".join(styles)

    table_css = [
        "color: #202124",
        "background-color: #BDC1C6",
        "border: 0",
        "border-radius: 0.5rem",
        "border-spacing: 0px",
        "outline: 0.5rem solid #BDC1C6",
        "margin: 1rem 0.5rem",
    ]
    th_css = ["background-color: #E8EAED"]
    th_td_css = ["text-align:left", "padding: 0.25rem 1rem"]
    table_styles = [
        CSSDict(selector="", props=join_styles(table_css)),
        CSSDict(selector="th", props=join_styles(th_css)),
        CSSDict(selector="th,td", props=join_styles(th_td_css)),
    ]

    return df.style.set_table_styles(table_styles).hide()


def change_row_bgcolor(row: Series) -> list[str]:
    style = f"background-color:{row[BGCOLOR_COLUMN]}"
    return [style] * len(row)


def display_table(yield_rows: Callable[[], Iterator[list[str]]]) -> None:
    data = yield_rows()
    df = DataFrame(columns=next(data), data=data)
    styler = get_table_styler(df)
    styler.apply(change_row_bgcolor, axis=1)
    styler.hide([BGCOLOR_COLUMN], axis="columns")

    html = styler.to_html()
    IPython.display.display(IPython.display.HTML(html))


def display_speakers(transcription: VideoTranscription) -> None:
    def sanitize_field(s: str, symbol_if_unknown: str) -> str:
        return symbol_if_unknown if s == NOT_FOUND else s

    def yield_rows() -> Iterator[list[str]]:
        yield ["voice", "name", "company", "position", "role_in_video", BGCOLOR_COLUMN]

        color_for_voice = get_color_for_voice_mapping(transcription.task2_speakers)
        for speaker in transcription.task2_speakers:
            yield [
                str(speaker.voice),
                sanitize_field(speaker.name, NOT_FOUND),
                sanitize_field(speaker.company, NOT_FOUND),
                sanitize_field(speaker.position, NOT_FOUND),
                sanitize_field(speaker.role_in_video, NOT_FOUND),
                color_for_voice.get(speaker.voice, "red"),
            ]

    display_markdown(f"### Speakers ({len(transcription.task2_speakers)})")
    display_table(yield_rows)


def display_transcripts(transcription: VideoTranscription) -> None:
    def yield_rows() -> Iterator[list[str]]:
        yield ["start", "speaker", "transcript", BGCOLOR_COLUMN]

        color_for_voice = get_color_for_voice_mapping(transcription.task2_speakers)
        speaker_for_voice = {
            speaker.voice: speaker for speaker in transcription.task2_speakers
        }
        previous_voice = None
        for transcript in transcription.task1_transcripts:
            current_voice = transcript.voice
            speaker_label = ""
            if speaker := speaker_for_voice.get(current_voice, None):
                if speaker.name != NOT_FOUND:
                    speaker_label = speaker.name
                elif speaker.position != NOT_FOUND:
                    speaker_label = f"[voice {current_voice}][{speaker.position}]"
                elif speaker.role_in_video != NOT_FOUND:
                    speaker_label = f"[voice {current_voice}][{speaker.role_in_video}]"
            if not speaker_label:
                speaker_label = f"[voice {current_voice}]"
            yield [
                transcript.start,
                speaker_label if current_voice != previous_voice else '"',
                transcript.text,
                color_for_voice.get(current_voice, "red"),
            ]
            previous_voice = current_voice

    display_markdown(f"### Transcripts ({len(transcription.task1_transcripts)})")
    display_table(yield_rows)


def transcribe_video(
    video: Video,
    video_segment: VideoSegment | None = None,
    fps: float | None = None,
    prompt: str | None = None,
    model: Model | None = None,
) -> None:
    display_video(video)
    transcription = get_video_transcription(video, video_segment, fps, prompt, model)
    display_speakers(transcription)
    display_transcripts(transcription)

▶️ Next up: ✅ Challenge Complete

Unlocking Multimodal Video Transcription with Gemini — Part 5: 🚀 Finalization was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Unlocking Multimodal Video Transcription with Gemini

1. 🔥 Challenge
2. 🛠️ Setup
3. 🧪 Prototyping
4. 🏗️ Prompt Crafting ◀️
5. 🚀 Finalization
6. ✅ Challenge Complete
7. ⚖️ Analysis, Tips & Optimizations
8. 🏁 Conclusion

🏗️ Prompt crafting

🪜 Methodology

Prompt crafting, also called prompt engineering, is a relatively new field. It involves designing and refining text instructions to guide LLMs towards generating desired outputs. Like writing, it is both an art and a science, a skill that everyone can develop with practice.

We can find countless reference materials about prompt crafting. Some prompts can be very long, complex, and even scary. Crafting prompts with a high-performing LLM like Gemini is much simpler. Here are three key adjectives to keep in mind:

• iterative
• precise
• concise

Iterative

Prompt crafting is typically an iterative process. Here are some recommendations:

• Craft your prompt step by step
• Keep track of your successive iterations
• At every iteration, make sure to measure what’s working versus what’s not
• If you reach a regression, backtrack to a successful iteration

Precise

Precision is key:

• Use words as specific as possible
• Words with multiple meanings can introduce variability, so use precise expressions
• Precision will influence probabilities in your favor

Concise

Concision has additional advantages:

• A short prompt is easier for us developers to understand (and maintain!)
• The longer your prompt is, the more likely you are to introduce inconsistencies or even contradictions, which results in variable interpretations of your instructions
• Test and trust the LLM’s knowledge: this knowledge acts as an implicit context and can make your prompt shorter and clearer

Overall, though this may seem contradictory, if you take the time to be iterative, precise, and concise, you are likely to save a lot of time.

💡 If you want to explore this topic, check out Prompting strategies (Google Cloud reference) and Prompt engineering (68-page PDF by Lee Boonstra).

📚 Terminology

We’re not experts in video transcription (yet!) but we want Gemini to behave as one. Consequently, we’d like to write prompts as specific as possible for this use case. While LLMs process instructions based on their training knowledge, they can also share this knowledge with us.

We can learn a lot by directly asking Gemini:

prompt = """
What is the terminology used for video transcriptions?
Please show a typical output example.
"""
generate_content(prompt, show_as=ShowAs.MARKDOWN)

------------------------------- gemini-2.0-flash -------------------------------
Input tokens   :        18
Output tokens  :     1,064
------------------------------ start of response -------------------------------

## Terminology Used for Video Transcriptions

Here's a breakdown of common terminology used when discussing video transcriptions:

*   **Transcription:** The general term for converting audio or video content into written text.
*   **Verbatim Transcription:** A transcription that captures every spoken word, including filler words ("um," "ah"), false starts, stutters, and background noises.  It aims for a highly accurate representation of the audio.
*   **Clean Verbatim Transcription (or Edited Transcription):** A transcription that removes filler words, false starts, and stutters to create a more readable and polished text. It focuses on conveying the core message clearly.
*   **Intelligent Verbatim Transcription:** Similar to clean verbatim, but may also correct minor grammatical errors or rephrase sentences slightly for clarity, while still preserving the speaker's intent.
*   **Timecoding (or Timestamping):** Adding timestamps to the transcription to indicate when specific words or phrases were spoken in the video.  This is often used for indexing, subtitling, or finding specific sections of the video.
*   **Speaker Identification:** Identifying and labeling each speaker in the transcription (e.g., Speaker 1, Speaker 2, or using names if known).
*   **Subtitles/Captions:** Textual versions of the video's dialogue displayed on the screen, often used for accessibility or translation.  While related to transcription, they are specifically formatted for on-screen display.
*   **Closed Captions:** Captions that can be turned on or off by the viewer.
*   **Open Captions:** Captions that are permanently embedded in the video and cannot be turned off.
*   **SRT (SubRip Subtitle) file:** A common file format for subtitles and captions, containing the text and timestamps.
*   **VTT (Video Text Tracks) file:** A more advanced subtitle format that supports styling and other features.
*   **Burned-in Captions:** Captions that are permanently embedded into the video file.
*   **Transcription Software:** Software used to assist in the transcription process, often including features like automatic speech recognition (ASR), playback controls, and text editing tools.
*   **Automatic Speech Recognition (ASR):** Technology that automatically converts speech to text.  While improving, ASR often requires human editing for accuracy.
*   **Human Transcription:** Transcription performed by a human transcriber, generally considered more accurate than ASR alone.

## Typical Output Examples

Here are examples of different transcription styles:

**1. Verbatim Transcription:**

```
Speaker 1: Um, so, like, I think the, uh, the main point is, ah, we need to, um, focus on, you know, the, the marketing strategy.
Speaker 2: Yeah, I, I agree.  It's, it's, uh, really important.  And, um, we should, like, consider, um, different, uh, channels.
```

**2. Clean Verbatim Transcription:**

```
Speaker 1: So, I think the main point is we need to focus on the marketing strategy.
Speaker 2: Yeah, I agree. It's really important. And we should consider different channels.
```

**3. Intelligent Verbatim Transcription:**

```
Speaker 1: I think the main point is that we need to focus on our marketing strategy.
Speaker 2: Yes, I agree. It's very important. We should consider different marketing channels.
```

**4. Transcription with Timecoding (Clean Verbatim):**

```
00:00:05 Speaker 1: So, I think the main point is we need to focus on the marketing strategy.
00:00:12 Speaker 2: Yeah, I agree. It's really important. And we should consider different channels.
```

**5. Transcription with Speaker Identification and Timecoding (Clean Verbatim):**

```
00:00:05 John: So, I think the main point is we need to focus on the marketing strategy.
00:00:12 Mary: Yeah, I agree. It's really important. And we should consider different channels.
```

**6. Example of an SRT file:**

```
1
00:00:05,000 --> 00:00:12,000
So, I think the main point is
we need to focus on the marketing strategy.

2
00:00:12,000 --> 00:00:18,000
Yeah, I agree. It's really important.
And we should consider different channels.
```

The best type of transcription depends on the specific needs of the project.  For example, legal proceedings often require verbatim transcriptions, while marketing videos might benefit from clean verbatim or intelligent verbatim transcriptions for clarity and readability.  Subtitles and captions require specific formatting and timing to synchronize with the video.

------------------------------- end of response --------------------------------

📝 Tabular extraction

So far, we’ve seen the following:

• We didn’t manage to get the full transcription with identified speakers all at once
• Order matters (because a generated token influences the probabilities for subsequent tokens)

To tackle our challenge, we need Gemini to infer from the following multimodal information:

• text (our instructions + what may be written in the video)
• audio cues (everything said or audible in the video’s audio)
• visual cues (everything visible in the video)
• time (when things happen)

That is quite a mixture of information types!

As video transcription is a data extraction use case, if we think about the final result as a database, our final goal can be seen as the generation of two related tables (transcripts and speakers). If we write it down, our initial three sub-problems now look decoupled:

💡 In computer science, data decoupling enhances data locality, often yielding improved performance across areas such as cache utilization, data access, semantic understanding, or system maintenance. Within the LLM Transformer architecture, core performance relies heavily on the attention mechanism. Nonetheless, the attention pool is finite and tokens compete for attention. Researchers sometimes refer to “attention dilution” for long-context, million-token-scale benchmarks. While we cannot directly debug LLMs as users, intuitively, data decoupling may improve the model’s focus, leading to a better attention span.

Since Gemini is extremely good with patterns, it can automatically generate identifiers to link our tables. In addition, since we eventually want an automated workflow, we can start reasoning in terms of data and fields:

Let’s call this approach “tabular extraction”, split our instructions into two tasks (tables), still in a single request, and arrange them in a meaningful order…

💬 Transcripts

First of all, let’s focus on getting the audio transcripts:

• Gemini has proven to be natively good at audio transcription
• This requires less inference than image analysis
• It is central and independent information

💡 Generating an output that starts with correct answers should help to achieve an overall correct output.

We’ve also seen what a typical transcription entry can look like:

00:02 speaker_1: Welcome!

But, right away, there can be some ambiguities in our multimodal use case:

• What is a speaker?
• Is it someone we see/hear?
• What if the person visible in the video is not the one speaking?
• What if the person speaking is never seen in the video?

How do we unconsciously identify who is speaking in a video?

• First, probably by identifying the different voices on the fly?
• Then, probably by consolidating additional audio and visual cues?

Can Gemini understand voice characteristics?

prompt = """
Using only the video's audio, list the following audible characteristics:
- Voice tones
- Voice pitches
- Languages
- Accents
- Speaking styles
"""
video = TestVideo.GDM_PODCAST_TRAILER_PT59S

generate_content(prompt, video, show_as=ShowAs.MARKDOWN)

----------------- GDM_PODCAST_TRAILER_PT59S / gemini-2.0-flash -----------------
Input tokens   :    16,730
Output tokens  :       168
------------------------------ start of response -------------------------------

Okay, here's a breakdown of the audible characteristics in the video's audio:

- **Voice Tones:** The tones range from conversational and friendly to more serious and thoughtful, depending on the speaker and the topic. There are also moments of excitement and humor.
- **Voice Pitches:** There's a mix of high and low pitches, reflecting the different speakers (both male and female).
- **Languages:** The primary language is English.
- **Accents:** There are a variety of accents, including British and American.
- **Speaking Styles:** The speaking styles vary from casual and conversational to more formal and academic, depending on the speaker and the context.

------------------------------- end of response --------------------------------

What about a French video?

video = TestVideo.BRUT_FR_DOGS_WATER_LEAK_PT8M28S

generate_content(prompt, video, show_as=ShowAs.MARKDOWN)

-------------- BRUT_FR_DOGS_WATER_LEAK_PT8M28S / gemini-2.0-flash --------------
Input tokens   :   144,055
Output tokens  :       147
------------------------------ start of response -------------------------------

Here is a list of the audible characteristics of the video's audio:

- **Voice tones:** Conversational, informative, enthusiastic, serious, humorous
- **Voice pitches:** Varying, from low to high
- **Languages:** French
- **Accents:** Standard French
- **Speaking styles:** Clear, articulate, professional, casual

------------------------------- end of response --------------------------------

⚠️ We have to be cautious here: responses can consolidate multimodal information or even general knowledge. For example, if a person is famous, their name is most likely part of the LLM’s knowledge. If they are known to be from the UK, a possible inference is that they have a British accent. This is why we made our prompt more specific by including “using only the video’s audio”.

💡 If you conduct more tests, for example on private audio files (i.e., not part of common knowledge and with no additional visual cues), you’ll see that Gemini’s audio tokenizer performs exceptionally well and extracts semantic speech information!

After a few iterations, we can arrive at a transcription prompt focusing on the audio and voices:

prompt = """
Task:
- Watch the video and listen carefully to the audio.
- Identify the distinct voices using a `voice` ID (1, 2, 3, etc.).
- Transcribe the video's audio verbatim with voice diarization.
- Include the `start` timecode (MM:SS) for each speech segment.
- Output a JSON array where each object has the following fields:
  - `start`
  - `text`
  - `voice`
"""
video = TestVideo.GDM_PODCAST_TRAILER_PT59S

generate_content(prompt, video, show_as=ShowAs.MARKDOWN)

----------------- GDM_PODCAST_TRAILER_PT59S / gemini-2.0-flash -----------------
Input tokens   :    16,800
Output tokens  :       635
------------------------------ start of response -------------------------------

[
  {
    "start": "00:00",
    "text": "Do I have to call you Sir Demis now?",
    "voice": 1
  },
  {
    "start": "00:01",
    "text": "Oh, you don't. Absolutely not.",
    "voice": 2
  },
  {
    "start": "00:03",
    "text": "Welcome to Google Deep Mind the podcast with me, your host Professor Hannah Fry.",
    "voice": 1
  },
  {
    "start": "00:06",
    "text": "We want to take you to the heart of where these ideas are coming from. We want to introduce you to the people who are leading the design of our collective future.",
    "voice": 1
  },
  {
    "start": "00:19",
    "text": "Getting the safety right is probably, I'd say, one of the most important challenges of our time. I want safe and capable.",
    "voice": 3
  },
  {
    "start": "00:26",
    "text": "I want a bridge that will not collapse.",
    "voice": 3
  },
  {
    "start": "00:30",
    "text": "just give these scientists a superpower that they had not imagined earlier.",
    "voice": 4
  },
  {
    "start": "00:34",
    "text": "autonomous vehicles. It's hard to fathom that when you're working on a search engine.",
    "voice": 5
  },
  {
    "start": "00:38",
    "text": "We may see entirely new genre or entirely new forms of art come up. There may be a new word that is not music, painting, photography, movie making, and that AI will have helped us create it.",
    "voice": 6
  },
  {
    "start": "00:48",
    "text": "You really want AGI to be able to peer into the mysteries of the universe.",
    "voice": 1
  },
  {
    "start": "00:51",
    "text": "Yes, quantum mechanics, string theory, well, and the nature of reality.",
    "voice": 2
  },
  {
    "start": "00:55",
    "text": "Ow.",
    "voice": 1
  },
  {
    "start": "00:56",
    "text": "the magic of AI.",
    "voice": 6
  }
]

------------------------------- end of response --------------------------------

This is looking good! And if you test these instructions on more complex videos, you’ll get similarly promising results.

Notice how the prompt reuses cherry-picked terms from the terminology previously provided by Gemini, while aiming for precision and concision:

• verbatim is unambiguous (unlike "spoken words")
• 1, 2, 3, etc. is an ellipsis (Gemini can infer the pattern)
• timecode is specific (timestamp has more meanings)
• MM:SS clarifies the timecode format

💡 Gemini 2.0 was trained to understand the specific MM:SS timecode format. Gemini 2.5 also supports the H:MM:SS format for longer videos. For the latest updates, refer to the video understanding documentation.

We’re halfway there. Let’s complete our database generation with a second task…

🧑 Speakers

The second task is pretty straightforward: we want to extract speaker information into a second table. The two tables are logically linked by the voice ID.

After a few iterations, we can reach a two-task prompt like the following:

prompt = """
Generate a JSON object with keys `task1_transcripts` and `task2_speakers` for the following tasks.

**Task 1 - Transcripts**

- Watch the video and listen carefully to the audio.
- Identify the distinct voices using a `voice` ID (1, 2, 3, etc.).
- Transcribe the video's audio verbatim with voice diarization.
- Include the `start` timecode (MM:SS) for each speech segment.
- The `task1_transcripts` value is a JSON array where each object has the following fields:
  - `start`
  - `text`
  - `voice`

**Task 2 - Speakers**

- For each `voice` ID from Task 1, extract the name of the corresponding speaker.
- Use visual and audio cues.
- If a speaker's name cannot be found, use `?` as the value.
- The `task2_speakers` value is a JSON array where each object has the following fields:
  - `voice`
  - `name`

JSON:
"""
video = TestVideo.GDM_PODCAST_TRAILER_PT59S

generate_content(prompt, video, show_as=ShowAs.MARKDOWN)

----------------- GDM_PODCAST_TRAILER_PT59S / gemini-2.0-flash -----------------
Input tokens   :    16,920
Output tokens  :       806
------------------------------ start of response -------------------------------

{
  "task1_transcripts": [
    {
      "start": "00:00",
      "text": "Do I have to call you Sir Demis now?",
      "voice": 1
    },
    {
      "start": "00:01",
      "text": "Oh, you don't. Absolutely not.",
      "voice": 2
    },
    {
      "start": "00:04",
      "text": "Welcome to Google Deep Mind the podcast with me, your host Professor Hannah Fry.",
      "voice": 1
    },
    {
      "start": "00:06",
      "text": "We want to take you to the heart of where these ideas are coming from. We want to introduce you to the people who are leading the design of our collective future.",
      "voice": 1
    },
    {
      "start": "00:19",
      "text": "Getting the safety right is probably, I'd say, one of the most important challenges of our time. I want safe and capable.",
      "voice": 3
    },
    {
      "start": "00:26",
      "text": "I want a bridge that will not collapse.",
      "voice": 3
    },
    {
      "start": "00:30",
      "text": "That just give these scientists a superpower that they had not imagined earlier.",
      "voice": 4
    },
    {
      "start": "00:34",
      "text": "autonomous vehicles. It's hard to fathom that when you're working on a search engine.",
      "voice": 5
    },
    {
      "start": "00:38",
      "text": "We may see entirely new genre or entirely new forms of art come up. There may be a new word that is not music, painting, photography, movie making, and that AI will have helped us create it.",
      "voice": 6
    },
    {
      "start": "00:48",
      "text": "You really want AGI to be able to peer into the mysteries of the universe.",
      "voice": 1
    },
    {
      "start": "00:51",
      "text": "Yes, quantum mechanics, string theory, well, and the nature of reality.",
      "voice": 2
    },
    {
      "start": "00:55",
      "text": "Ow.",
      "voice": 1
    },
    {
      "start": "00:56",
      "text": "the magic of AI.",
      "voice": 6
    }
  ],
  "task2_speakers": [
    {
      "voice": 1,
      "name": "Professor Hannah Fry"
    },
    {
      "voice": 2,
      "name": "Demis Hassabis"
    },
    {
      "voice": 3,
      "name": "Anca Dragan"
    },
    {
      "voice": 4,
      "name": "Pushmeet Kohli"
    },
    {
      "voice": 5,
      "name": "Jeff Dean"
    },
    {
      "voice": 6,
      "name": "Douglas Eck"
    }
  ]
}

------------------------------- end of response --------------------------------

Test this prompt on more complex videos: it’s still looking good!

▶️ Next up: 🚀 Finalization

Unlocking Multimodal Video Transcription with Gemini — Part 4: 🏗️ Prompt Crafting was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Unlocking Multimodal Video Transcription with Gemini

1. 🔥 Challenge
2. 🛠️ Setup
3. 🧪 Prototyping ◀️
4. 🏗️ Prompt Crafting
5. 🚀 Finalization
6. ✅ Challenge Complete
7. ⚖️ Analysis, Tips & Optimizations
8. 🏁 Conclusion

🧪 Prototyping

🌱 Natural behavior

Before diving any deeper, it’s interesting to see how Gemini responds to simple instructions, to develop some intuition about its natural behavior.

Let’s first see what we get with minimalistic prompts and a short English video:

video = TestVideo.GDM_PODCAST_TRAILER_PT59S
display_video(video)

prompt = "Transcribe the video's audio with time information."
generate_content(prompt, video)

Video (source)

----------------- GDM_PODCAST_TRAILER_PT59S / gemini-2.0-flash -----------------
Input tokens   :    16,708
Output tokens  :       421
------------------------------ start of response -------------------------------
[00:00:00] Do I have to call you Sir Demis now?
[00:00:01] Oh, you don't.
[00:00:02] Absolutely not.
[00:00:04] Welcome to Google DeepMind the podcast with me, your host Professor Hannah Fry.
[00:00:06] We want to take you to the heart of where these ideas are coming from.
[00:00:12] We want to introduce you to the people who are leading the design of our collective future.
[00:00:19] Getting the safety right is probably, I'd say, one of the most important challenges of our time.
[00:00:25] I want safe and capable.
[00:00:27] I want a bridge that will not collapse.
[00:00:30] just give these scientists a superpower that they had not imagined earlier.
[00:00:34] autonomous vehicles.
[00:00:35] It's hard to fathom that when you're working on a search engine.
[00:00:38] We may see entirely new genre or entirely new forms of art come up.
[00:00:42] There may be a new word that is not music, painting, photography, movie making, and that AI will have helped us create it.
[00:00:48] You really want AGI to be able to peer into the mysteries of the universe.
[00:00:51] Yes, quantum mechanics, string theory, well, and the nature of reality.
[00:00:55] Ow.
[00:00:57] the magic of AI.
------------------------------- end of response --------------------------------

Results:

• Gemini naturally outputs a list of [time] transcript lines.
• That’s Speech-to-Text in one line!
• It looks like we can answer “1️⃣ What was said and when?”.

Now, what about “2️⃣ Who are the speakers?”

prompt = "List the speakers identifiable in the video."
generate_content(prompt, video)

----------------- GDM_PODCAST_TRAILER_PT59S / gemini-2.0-flash -----------------
Input tokens   :    16,705
Output tokens  :        46
------------------------------ start of response -------------------------------
Here are the speakers identifiable in the video:

*   Professor Hannah Fry
*   Demis Hassabis
*   Anca Dragan
*   Pushmeet Kohli
*   Jeff Dean
*   Douglas Eck
------------------------------- end of response --------------------------------

Results:

• Gemini can consolidate the names visible on title cards during the video.
• That’s OCR + entity extraction in one line!
• “2️⃣ Who are the speakers?” looks solved too!

⏩ Not so fast!

The natural next step is to jump to the final instructions, to solve our problem once and for all.

prompt = """
Transcribe the video's audio including speaker names (use `?` if not found).

Format example:
[00:02] John Doe - Hello Alice!
"""
generate_content(prompt, video)

----------------- GDM_PODCAST_TRAILER_PT59S / gemini-2.0-flash -----------------
Input tokens   :    16,732
Output tokens  :       378
------------------------------ start of response -------------------------------
Here is the audio transcription of the video:

[00:00] ? - Do I have to call you Sir Demis now?
[00:01] Demis Hassabis - Oh, you don't. Absolutely not.
[00:03] Professor Hannah Fry - Welcome to Google DeepMind the podcast with me, your host, Professor Hannah Fry.
[00:06] Professor Hannah Fry - We want to take you to the heart of where these ideas are coming from. We want to introduce you to the people who are leading the design of our collective future.
[00:19] Anca Dragan - Getting the safety right is probably, I'd say, one of the most important challenges of our time. I want safe and capable. I want a bridge that will not collapse.
[00:29] Pushmeet Kohli - Just give these scientists a superpower that they had not imagined earlier.
[00:34] Jeff Dean - Autonomous vehicles. It's hard to fathom that when you're working on a search engine.
[00:38] Douglas Eck - We may see entirely new genre or entirely new forms of art come up. There may be a new word that is not music, painting, photography, movie making, and that AI will have helped us create it.
[00:48] Professor Hannah Fry - You really want AGI to be able to peer into the mysteries of the universe.
[00:51] Demis Hassabis - Yes, quantum mechanics, string theory, well, and the nature of reality.
[00:55] Professor Hannah Fry - Ow!
[00:56] Douglas Eck - The magic of AI.
------------------------------- end of response --------------------------------

This is almost correct. The first segment is not attributed to the host (who is only introduced a bit later), but everything else looks correct.

Nonetheless, these are not real-world conditions:

• The video is very short (less than a minute)
• The video is also rather simple (speakers are clearly introduced with on-screen title cards)

Let’s try with this 8-minute (and more complex) video:

generate_content(prompt, TestVideo.GDM_ALPHAFOLD_PT7M54S)

------------------- GDM_ALPHAFOLD_PT7M54S / gemini-2.0-flash -------------------
Input tokens   :   134,177
Output tokens  :     2,689
------------------------------ start of response -------------------------------
[00:02] ? - We've discovered more about the world than any other civilization before us.
[00:08] ? - But we have been stuck on this one problem.
[00:11] ? - How do proteins fold up?
[00:13] ? - How do proteins go from a string of amino acids to a compact shape that acts as a machine and drives life?
[00:22] ? - When you find out about proteins, it is very exciting.
[00:25] ? - You could think of them as little biological nano machines.
[00:28] ? - They are essentially the fundamental building blocks that power everything living on this planet.
[00:34] ? - If we can reliably predict protein structures using AI, that could change the way we understand the natural world.
[00:46] ? - Protein folding is one of these holy grail type problems in biology.
[00:50] Demis Hassabis - We've always hypothesized that AI should be helpful to make these kinds of big scientific breakthroughs more quickly.
[00:58] ? - And then I'll probably be looking at little tunings that might make a difference.
[01:02] ? - It should be creating a histogram on and a background skill.
[01:04] ? - We've been working on our system AlphaFold really hard now for over two years.
[01:08] ? - Rather than having to do painstaking experiments, in the future biologists might be able to instead rely on AI methods to directly predict structures quickly and efficiently.
[01:17] Kathryn Tunyasuvunakool - Generally speaking, biologists tend to be quite skeptical of computational work, and I think that skepticism is healthy and I respect it, but I feel very excited about what AlphaFold can achieve.
[01:28] Andrew Senior - CASP is when we, we say, look, DeepMind is doing protein folding.
[01:31] Andrew Senior - This is how good we are, and maybe it's better than everybody else, maybe it isn't.
[01:37] ? - We decided to enter CASP competition because it represented the Olympics of protein folding.
[01:44] John Moult - CASP, we started to try and speed up the solution to the protein folding problem.
[01:50] John Moult - When we started CASP in 1994, I certainly was naive about how hard this was going to be.
[01:58] ? - It was very cumbersome to do that because it took a long time.
[02:01] ? - Let's see what, what, what are we doing still to improve?
[02:03] ? - Typically 100 different groups from around the world participate in CASP, and we take a set of 100 proteins and we ask the groups to send us what they think the structures look like.
[02:15] ? - We can reach 57.9 GDT on CASP 12 ground truth.
[02:19] John Jumper - CASP has a metric on which you will be scored, which is this GDT metric.
[02:25] John Jumper - On a scale of zero to 100, you would expect a GDT over 90 to be a solution to the problem.
[02:33] ? - If we do achieve this, this has incredible medical relevance.
[02:37] Pushmeet Kohli - The implications are immense, from how diseases progress, how you can discover new drugs.
[02:45] Pushmeet Kohli - It's endless.
[02:46] ? - I wanted to make a, a really simple system and the results have been surprisingly good.
[02:50] ? - The team got some results with a new technique, not only is it more accurate, but it's much faster than the old system.
[02:56] ? - I think we'll substantially exceed what we're doing right now.
[02:59] ? - This is a game, game changer, I think.
[03:01] John Moult - In CASP 13, something very significant had happened.
[03:06] John Moult - For the first time, we saw the effective application of artificial intelligence.
[03:11] ? - We've advanced the state of the art in the field, so that's fantastic, but we still got a long way to go before we've solved it.
[03:18] ? - The shapes were now approximately correct for many of the proteins, but the details, exactly where each atom sits, which is really what we would call a solution, we're not yet there.
[03:29] ? - It doesn't help if you have the tallest ladder when you're going to the moon.
[03:33] ? - We hit a little bit of a brick wall, um, since we won CASP, then it was back to the drawing board and like what are our new ideas?
[03:41] ? - Um, and then it's taken a little while, I would say, for them to get back to where they were, but with the new ideas.
[03:51] ? - They can go further, right?
[03:52] ? - So, um, so that's a really important moment.
[03:55] ? - I've seen that moment so many times now, but I know what that means now, and I know this is the time now to press.
[04:02] ? - We need to double down and go as fast as possible from here.
[04:05] ? - I think we've got no time to lose.
[04:07] ? - So the intention is to enter CASP again.
[04:09] ? - CASP is deeply stressful.
[04:12] ? - There's something weird going on with, um, the learning because it is learning something that's correlated with GDT, but it's not calibrated.
[04:18] ? - I feel slightly uncomfortable.
[04:20] ? - We should be learning this, you know, in the blink of an eye.
[04:23] ? - The technology advancing outside DeepMind is also doing incredible work.
[04:27] Richard Evans - And there's always the possibility another team has come somewhere out there field that we don't even know about.
[04:32] ? - Someone asked me, well, should we panic now?
[04:33] ? - Of course, we should have been panicking before.
[04:35] ? - It does seem to do better, but still doesn't do quite as well as the best model.
[04:39] ? - Um, so it looks like there's room for improvement.
[04:42] ? - There's always a risk that you've missed something, and that's why blind assessments like CASP are so important to validate whether our results are real.
[04:49] ? - Obviously, I'm excited to see how CASP 14 goes.
[04:51] ? - My expectation is we get our heads down, we focus on the full goal, which is to solve the whole problem.
[05:14] ? - We were prepared for CASP to start on April 15th because that's when it was originally scheduled to start, and it's been delayed by a month due to coronavirus.
[05:24] ? - I really miss everyone.
[05:25] ? - No, I struggled a little bit just kind of getting into a routine, especially, uh, my wife, she came down with the, the virus.
[05:32] ? - I mean, luckily it didn't turn out too serious.
[05:34] ? - CASP started on Monday.
[05:37] Demis Hassabis - Can I just check this diagram you've got here, John, this one where we ask ground truth.
[05:40] Demis Hassabis - Is this one we've done badly on?
[05:42] ? - We're actually quite good on this region.
[05:43] ? - If you imagine that we hadn't have said it came around this way, but had put it in.
[05:47] ? - Yeah, and that instead.
[05:48] ? - Yeah.
[05:49] ? - One of the hardest proteins we've gotten in CASP thus far is a SARS-CoV-2 protein, uh, called Orf8.
[05:55] ? - Orf8 is a coronavirus protein.
[05:57] ? - We tried really hard to improve our prediction, like really, really hard, probably the most time that we have ever spent on a single target.
[06:05] ? - So we're about two-thirds of the way through CASP, and we've gotten three answers back.
[06:11] ? - We now have a ground truth for Orf8, which is one of the coronavirus proteins.
[06:17] ? - And it turns out we did really well in predicting that.
[06:20] Demis Hassabis - Amazing job, everyone, the whole team.
[06:23] Demis Hassabis - It's been an incredible effort.
[06:24] John Moult - Here what we saw in CASP 14 was a group delivering atomic accuracy off the bat, essentially solving what in our world is two problems.
[06:34] John Moult - How do you look to find the right solution, and then how do you recognize you've got the right solution when you're there?
[06:41] ? - All right, are we, are we mostly here?
[06:46] ? - I'm going to read an email.
[06:48] ? - Uh, I got this from John Moult.
[06:50] ? - Now I'll just read it.
[06:51] ? - It says, John, as I expect you know, your group has performed amazingly well in CASP 14, both relative to other groups and in absolute model accuracy.
[07:02] ? - Congratulations on this work.
[07:05] ? - It is really outstanding.
[07:07] Demis Hassabis - AlphaFold represents a huge leap forward that I hope will really accelerate drug discovery and help us to better understand disease.
[07:13] John Moult - It's pretty mind-blowing.
[07:16] John Moult - You know, these results were, for me, having worked on this problem so long, after many, many stops and starts and will this ever get there, suddenly this is a solution.
[07:28] John Moult - We've solved the problem.
[07:29] John Moult - This gives you such excitement about the way science works, about how you can never see exactly or even approximately what's going to happen next.
[07:37] John Moult - There are always these surprises, and that really, as a scientist, is what keeps you going.
[07:41] John Moult - What's going to be the next surprise?
------------------------------- end of response --------------------------------

This falls apart: Most segments have no identified speaker!

As we are trying to solve a new complex problem, LLMs haven’t been trained on any known solution. This is likely why direct instructions do not yield the expected answer.

At this stage:

• We might conclude that we can’t solve the problem with real-world videos.
• Persevering by trying more and more elaborate prompts for this unsolved problem might result in a waste of time.

Let’s take a step back and think about what happens under the hood…

⚛️ Under the hood

Modern LLMs are mostly based on the Transformer architecture, a new neural network design detailed in a 2017 paper by Google researchers titled Attention Is All You Need. The paper introduced the self-attention mechanism, a key innovation that fundamentally changed the way machines process language.

🪙 Tokens

Tokens are the LLM building blocks. We can consider a token to represent a piece of information.

Examples of Gemini multimodal tokens (with default parameters):

🎞️ Sampling frame rate

By default, video frames are sampled at 1 frame per second (1 FPS). These frames are included in the context with their corresponding timecodes.

You can use a custom sampling frame rate with the Part.video_metadata.fps parameter:

💡 For 1.0 < fps, Gemini was trained to understand MM:SS.sss and H:MM:SS.sss timecodes.

🔍 Media resolution

By default, each sampled frame is represented with 258 tokens.

You can specify a medium or low media resolution with the GenerateContentConfig.media_resolution parameter:

💡 The “media resolution” can be seen as the “image token resolution”: the number of tokens used to represent an image.

🧮 Probabilities all the way down

The ability of LLMs to communicate in flawless natural language is very impressive, but it’s easy to get carried away and make incorrect assumptions.

Keep in mind how LLMs work:

• An LLM is trained on a massive tokenized dataset, which represents its knowledge (its long-term memory)
• During the training, its neural network learns token patterns
• When you send a request to an LLM, your inputs are transformed into tokens (tokenization)
• To answer your request, the LLM predicts, token by token, the next likely tokens
• Overall, LLMs are exceptional statistical token prediction machines that seem to mimic how some parts of our brain work

This has a few consequences:

• LLM outputs are just statistically likely follow-ups to your inputs
• LLMs show some forms of reasoning: they can match complex patterns but have no actual deep understanding
• LLMs have no consciousness: they are designed to generate tokens and will do so based on your instructions
• Order matters: Tokens that are generated first will influence tokens that are generated next

For the next step, some methodical prompt crafting might help…

▶️ Next up: 🏗️ Prompt Crafting

Unlocking Multimodal Video Transcription with Gemini — Part 3: 🧪 Prototyping was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Unlocking Multimodal Video Transcription with Gemini

1. 🔥 Challenge
2. 🛠️ Setup ◀️
3. 🧪 Prototyping
4. 🏗️ Prompt Crafting
5. 🚀 Finalization
6. ✅ Challenge Complete
7. ⚖️ Analysis, Tips & Optimizations
8. 🏁 Conclusion

🏁 Setup

🐍 Python packages

We’ll use the following packages:

• google-genai: the Google Gen AI Python SDK lets us call Gemini with a few lines of code
• pandas for data visualization

We’ll also use these packages (dependencies of google-genai):

• pydantic for data management
• tenacity for request management

pip install --quiet "google-genai>=1.49.0" "pandas[output-formatting]"

🔗 Gemini API

We have two main options to send requests to Gemini:

• Vertex AI: Build enterprise-ready projects on Google Cloud
• Google AI Studio: Experiment, prototype, and deploy small projects

The Google Gen AI SDK provides a unified interface to these APIs and we can use environment variables for the configuration.

Option A — Gemini API via Vertex AI

Requirement:

• A Google Cloud project
• The Vertex AI API must be enabled for this project

Gen AI SDK environment variables:

• GOOGLE_GENAI_USE_VERTEXAI="True"
• GOOGLE_CLOUD_PROJECT="<PROJECT_ID>"
• GOOGLE_CLOUD_LOCATION="<LOCATION>" (see Google model endpoint locations)

Learn more about setting up a project and a development environment.

Option B — Gemini API via Google AI Studio

Requirement:

• A Gemini API key

Gen AI SDK environment variables:

• GOOGLE_GENAI_USE_VERTEXAI="False"
• GOOGLE_API_KEY="<API_KEY>"

Learn more about getting a Gemini API key from Google AI Studio.

💡 You can store your environment configuration outside of the source code:

Define the following environment detection functions. You can also define your configuration manually if needed:

# @title {display-mode: "form"}

import os
import sys
from collections.abc import Callable

from google import genai

# Manual setup (leave unchanged if setup is environment-defined)

# @markdown **Which API: Vertex AI or Google AI Studio?**
GOOGLE_GENAI_USE_VERTEXAI = True  # @param {type: "boolean"}

# @markdown **Option A - Google Cloud project [+location]**
GOOGLE_CLOUD_PROJECT = ""  # @param {type: "string"}
GOOGLE_CLOUD_LOCATION = "global"  # @param {type: "string"}

# @markdown **Option B - Google AI Studio API key**
GOOGLE_API_KEY = ""  # @param {type: "string"}


def check_environment() -> bool:
    check_colab_user_authentication()
    return check_manual_setup() or check_vertex_ai() or check_colab() or check_local()


def check_manual_setup() -> bool:
    return check_define_env_vars(
        GOOGLE_GENAI_USE_VERTEXAI,
        GOOGLE_CLOUD_PROJECT.strip(),  # Might have been pasted with line return
        GOOGLE_CLOUD_LOCATION,
        GOOGLE_API_KEY,
    )


def check_vertex_ai() -> bool:
    # Workbench and Colab Enterprise
    match os.getenv("VERTEX_PRODUCT", ""):
        case "WORKBENCH_INSTANCE":
            pass
        case "COLAB_ENTERPRISE":
            if not running_in_colab_env():
                return False
        case _:
            return False

    return check_define_env_vars(
        True,
        os.getenv("GOOGLE_CLOUD_PROJECT", ""),
        os.getenv("GOOGLE_CLOUD_REGION", ""),
        "",
    )


def check_colab() -> bool:
    if not running_in_colab_env():
        return False

    # Colab Enterprise was checked before, so this is Colab only
    from google.colab import auth as colab_auth  # type: ignore

    colab_auth.authenticate_user()

    # Use Colab Secrets (🗝️ icon in left panel) to store the environment variables
    # Secrets are private, visible only to you and the notebooks that you select
    # - Vertex AI: Store your settings as secrets
    # - Google AI: Directly import your Gemini API key from the UI
    vertexai, project, location, api_key = get_vars(get_colab_secret)

    return check_define_env_vars(vertexai, project, location, api_key)


def check_local() -> bool:
    vertexai, project, location, api_key = get_vars(os.getenv)

    return check_define_env_vars(vertexai, project, location, api_key)


def running_in_colab_env() -> bool:
    # Colab or Colab Enterprise
    return "google.colab" in sys.modules


def check_colab_user_authentication() -> None:
    if running_in_colab_env():
        from google.colab import auth as colab_auth  # type: ignore

        colab_auth.authenticate_user()


def get_colab_secret(secret_name: str, default: str) -> str:
    from google.colab import errors, userdata  # type: ignore

    try:
        return userdata.get(secret_name)
    except errors.SecretNotFoundError:
        return default


def get_vars(getenv: Callable[[str, str], str]) -> tuple[bool, str, str, str]:
    # Limit getenv calls to the minimum (may trigger UI confirmation for secret access)
    vertexai_str = getenv("GOOGLE_GENAI_USE_VERTEXAI", "")
    if vertexai_str:
        vertexai = vertexai_str.lower() in ["true", "1"]
    else:
        vertexai = bool(getenv("GOOGLE_CLOUD_PROJECT", ""))

    project = getenv("GOOGLE_CLOUD_PROJECT", "") if vertexai else ""
    location = getenv("GOOGLE_CLOUD_LOCATION", "") if project else ""
    api_key = getenv("GOOGLE_API_KEY", "") if not project else ""

    return vertexai, project, location, api_key


def check_define_env_vars(
    vertexai: bool,
    project: str,
    location: str,
    api_key: str,
) -> bool:
    match (vertexai, bool(project), bool(location), bool(api_key)):
        case (True, True, _, _):
            # Vertex AI - Google Cloud project [+location]
            location = location or "global"
            define_env_vars(vertexai, project, location, "")
        case (True, False, _, True):
            # Vertex AI - API key
            define_env_vars(vertexai, "", "", api_key)
        case (False, _, _, True):
            # Google AI Studio - API key
            define_env_vars(vertexai, "", "", api_key)
        case _:
            return False

    return True


def define_env_vars(vertexai: bool, project: str, location: str, api_key: str) -> None:
    os.environ["GOOGLE_GENAI_USE_VERTEXAI"] = str(vertexai)
    os.environ["GOOGLE_CLOUD_PROJECT"] = project
    os.environ["GOOGLE_CLOUD_LOCATION"] = location
    os.environ["GOOGLE_API_KEY"] = api_key


def check_configuration(client: genai.Client) -> None:
    service = "Vertex AI" if client.vertexai else "Google AI Studio"
    print(f"Using the {service} API", end="")

    if client._api_client.project:
        print(f' with project "{client._api_client.project[:7]}…"', end="")
        print(f' in location "{client._api_client.location}"')
    elif client._api_client.api_key:
        api_key = client._api_client.api_key
        print(f' with API key "{api_key[:5]}…{api_key[-5:]}"', end="")
        print(f" (in case of error, make sure it was created for {service})")

🤖 Gen AI SDK

To send Gemini requests, create a google.genai client:

from google import genai

check_environment()

client = genai.Client()

Check your configuration:

check_configuration(client)

Using the Vertex AI API with project "lpdemo-..." in location "europe-west9"

🧠 Gemini model

Gemini comes in different versions.

Let’s get started with Gemini 2.0 Flash, as it offers both high performance and low latency:

• GEMINI_2_0_FLASH = "gemini-2.0-flash"

💡 We select Gemini 2.0 Flash intentionally. The Gemini 2.5 model family is generally available and even more capable, but we want to experiment and understand Gemini’s core multimodal behavior. If we complete our challenge with 2.0, this should also work with newer models.

⚙️ Gemini configuration

Gemini can be used in different ways, ranging from factual to creative mode. The problem we’re trying to solve is a data extraction use case. We want results as factual and deterministic as possible. For this, we can change the content generation parameters.

We’ll set the temperature, top_p, and seed parameters to minimize randomness:

• temperature=0.0
• top_p=0.0
• seed=42 (arbitrary fixed value)

🎞️ Video sources

Here are the main video sources that Gemini can analyze:

⚠️ Important notes

• Our video test suite primarily uses public YouTube videos. This is for simplicity.
• When analyzing YouTube sources, Gemini receives raw audio/video streams without any additional metadata, exactly as if processing the corresponding video files from Cloud Storage.
• YouTube does offer caption/subtitle/transcript features (user-provided or auto-generated). However, these features focus on word-level speech-to-text and are limited to 40+ languages. Gemini does not receive any of this data and you’ll see that a multimodal transcription with Gemini provides additional benefits.
• Furthermore, our challenge also involves identifying speakers and extracting speaker data, a unique new capability.

🛠️ Helpers

Define our helper functions and data:

import enum
from dataclasses import dataclass
from datetime import timedelta
from typing import Any

import IPython.display
import tenacity
from google.genai.errors import ClientError
from google.genai.types import (
    FileData,
    FinishReason,
    GenerateContentConfig,
    GenerateContentResponse,
    Part,
    VideoMetadata,
)


class Model(enum.Enum):
    # Generally Available (GA)
    GEMINI_2_0_FLASH = "gemini-2.0-flash"
    GEMINI_2_5_FLASH = "gemini-2.5-flash"
    GEMINI_2_5_PRO = "gemini-2.5-pro"
    # Default model
    DEFAULT = GEMINI_2_0_FLASH


# Default configuration for more deterministic outputs
DEFAULT_CONFIG = GenerateContentConfig(
    temperature=0.0,
    top_p=0.0,
    seed=42,  # Arbitrary fixed value
)

YOUTUBE_URL_PREFIX = "https://www.youtube.com/watch?v="
CLOUD_STORAGE_URI_PREFIX = "gs://"


def url_for_youtube_id(youtube_id: str) -> str:
    return f"{YOUTUBE_URL_PREFIX}{youtube_id}"


class Video(enum.Enum):
    pass


class TestVideo(Video):
    # For testing purposes, video duration is statically specified in the enum name
    # Suffix (ISO 8601 based): _PT[<h>H][<m>M][<s>S]

    # Google DeepMind | The Podcast | Season 3 Trailer | 59s
    GDM_PODCAST_TRAILER_PT59S = url_for_youtube_id("0pJn3g8dfwk")
    # Google Maps | Walk in the footsteps of Jane Goodall | 2min 42s
    JANE_GOODALL_PT2M42S = "gs://cloud-samples-data/video/JaneGoodall.mp4"
    # Google DeepMind | AlphaFold | The making of a scientific breakthrough | 7min 54s
    GDM_ALPHAFOLD_PT7M54S = url_for_youtube_id("gg7WjuFs8F4")
    # Brut | French reportage | 8min 28s
    BRUT_FR_DOGS_WATER_LEAK_PT8M28S = url_for_youtube_id("U_yYkb-ureI")
    # Google DeepMind | The Podcast | AI for science | 54min 23s
    GDM_AI_FOR_SCIENCE_FRONTIER_PT54M23S = url_for_youtube_id("nQKmVhLIGcs")
    # Google I/O 2025 | Developer Keynote | 1h 10min 03s
    GOOGLE_IO_DEV_KEYNOTE_PT1H10M03S = url_for_youtube_id("GjvgtwSOCao")
    # Google Cloud | Next 2025 | Opening Keynote | 1h 40min 03s
    GOOGLE_CLOUD_NEXT_PT1H40M03S = url_for_youtube_id("Md4Fs-Zc3tg")
    # Google I/O 2025 | Keynote | 1h 56min 35s
    GOOGLE_IO_KEYNOTE_PT1H56M35S = url_for_youtube_id("o8NiE3XMPrM")


class ShowAs(enum.Enum):
    DONT_SHOW = enum.auto()
    TEXT = enum.auto()
    MARKDOWN = enum.auto()


@dataclass
class VideoSegment:
    start: timedelta
    end: timedelta


def generate_content(
    prompt: str,
    video: Video | None = None,
    video_segment: VideoSegment | None = None,
    model: Model | None = None,
    config: GenerateContentConfig | None = None,
    show_as: ShowAs = ShowAs.TEXT,
) -> None:
    prompt = prompt.strip()
    model = model or Model.DEFAULT
    config = config or DEFAULT_CONFIG

    model_id = model.value
    if video:
        if not (video_part := get_video_part(video, video_segment)):
            return
        contents = [video_part, prompt]
        caption = f"{video.name} / {model_id}"
    else:
        contents = prompt
        caption = f"{model_id}"
    print(f" {caption} ".center(80, "-"))

    for attempt in get_retrier():
        with attempt:
            response = client.models.generate_content(
                model=model_id,
                contents=contents,
                config=config,
            )
            display_response_info(response)
            display_response(response, show_as)


def get_video_part(
    video: Video,
    video_segment: VideoSegment | None = None,
    fps: float | None = None,
) -> Part | None:
    video_uri: str = video.value

    if not client.vertexai:
        video_uri = convert_to_https_url_if_cloud_storage_uri(video_uri)
        if not video_uri.startswith(YOUTUBE_URL_PREFIX):
            print("Google AI Studio API: Only YouTube URLs are currently supported")
            return None

    file_data = FileData(file_uri=video_uri, mime_type="video/*")
    video_metadata = get_video_part_metadata(video_segment, fps)

    return Part(file_data=file_data, video_metadata=video_metadata)


def get_video_part_metadata(
    video_segment: VideoSegment | None = None,
    fps: float | None = None,
) -> VideoMetadata:
    def offset_as_str(offset: timedelta) -> str:
        return f"{offset.total_seconds()}s"

    if video_segment:
        start_offset = offset_as_str(video_segment.start)
        end_offset = offset_as_str(video_segment.end)
    else:
        start_offset = None
        end_offset = None

    return VideoMetadata(start_offset=start_offset, end_offset=end_offset, fps=fps)


def convert_to_https_url_if_cloud_storage_uri(uri: str) -> str:
    if uri.startswith(CLOUD_STORAGE_URI_PREFIX):
        return f"https://storage.googleapis.com/{uri.removeprefix(CLOUD_STORAGE_URI_PREFIX)}"

    return uri


def get_retrier() -> tenacity.Retrying:
    return tenacity.Retrying(
        stop=tenacity.stop_after_attempt(7),
        wait=tenacity.wait_incrementing(start=10, increment=1),
        retry=should_retry_request,
        reraise=True,
    )


def should_retry_request(retry_state: tenacity.RetryCallState) -> bool:
    if not retry_state.outcome:
        return False
    err = retry_state.outcome.exception()
    if not isinstance(err, ClientError):
        return False
    print(f"❌ ClientError {err.code}: {err.message}")

    retry = False
    match err.code:
        case 400 if err.message is not None and " try again " in err.message:
            # Workshop: project accessing Cloud Storage for the first time (service agent provisioning)
            retry = True
        case 429:
            # Workshop: temporary project with 1 QPM quota
            retry = True
    print(f"🔄 Retry: {retry}")

    return retry


def display_response_info(response: GenerateContentResponse) -> None:
    if usage_metadata := response.usage_metadata:
        if usage_metadata.prompt_token_count:
            print(f"Input tokens   : {usage_metadata.prompt_token_count:9,d}")
        if usage_metadata.candidates_token_count:
            print(f"Output tokens  : {usage_metadata.candidates_token_count:9,d}")
        if usage_metadata.thoughts_token_count:
            print(f"Thoughts tokens: {usage_metadata.thoughts_token_count:9,d}")
    if not response.candidates:
        print("❌ No `response.candidates`")
        return
    if (finish_reason := response.candidates[0].finish_reason) != FinishReason.STOP:
        print(f"❌ {finish_reason = }")
    if not response.text:
        print("❌ No `response.text`")
        return


def display_response(
    response: GenerateContentResponse,
    show_as: ShowAs,
) -> None:
    if show_as == ShowAs.DONT_SHOW:
        return
    if not (response_text := response.text):
        return
    response_text = response.text.strip()

    print(" start of response ".center(80, "-"))
    match show_as:
        case ShowAs.TEXT:
            print(response_text)
        case ShowAs.MARKDOWN:
            display_markdown(response_text)
    print(" end of response ".center(80, "-"))


def display_markdown(markdown: str) -> None:
    IPython.display.display(IPython.display.Markdown(markdown))


def display_video(video: Video) -> None:
    video_url = convert_to_https_url_if_cloud_storage_uri(video.value)
    assert video_url.startswith("https://")

    video_width = 600
    if video_url.startswith(YOUTUBE_URL_PREFIX):
        youtube_id = video_url.removeprefix(YOUTUBE_URL_PREFIX)
        # Add referrerpolicy to fix video player configuration error 153
        extras = ['referrerpolicy="strict-origin-when-cross-origin"']
        ipython_video = IPython.display.YouTubeVideo(
            youtube_id, width=video_width, extras=extras
        )
    else:
        ipython_video = IPython.display.Video(video_url, width=video_width)

    display_markdown(f"### Video ([source]({video_url}))")
    IPython.display.display(ipython_video)

▶️ Next up: 🧪 Prototyping

Unlocking Multimodal Video Transcription with Gemini — Part 2: 🛠️ Setup was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.