feat: gemini audio video support#574
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
Adds Gemini chat support for audio/video inputs and TTS audio outputs, plus more resilient “URL → base64 inline” conversion behavior across Gemini/Anthropic/OpenAI image handling.
Changes:
- Extend OpenAI-compatible message/content models to represent
input_audioandvideo_url, and add responseaudiopayload support. - Update Gemini adaptor to (a) build media parts for audio/video, (b) auto-download audio/video URLs to inline base64 (configurable), and (c) configure TTS models with audio modality + speech config.
- Make URL→base64 conversions best-effort (log + keep original URL) instead of failing the whole request/response path in several adaptors.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| core/relay/model/gemini.go | Adds Gemini speech/voice config structures for TTS generation config. |
| core/relay/model/constant.go | Introduces video_url content type constant. |
| core/relay/model/completions.go | Adds request/response audio structs and OpenAI content modeling for input audio + video URL. |
| core/relay/adaptor/openai/image.go | Keeps original URL when URL→b64 conversion fails in image responses. |
| core/relay/adaptor/gemini/openai.go | Implements Gemini audio/video part building, URL auto-base64 conversion, and TTS config behavior. |
| core/relay/adaptor/gemini/openai_test.go | Adds coverage for TTS config, audio/video part conversion, URL auto-base64 behavior, and response audio mapping. |
| core/relay/adaptor/gemini/export_test.go | Exposes additional Gemini adaptor internals for testing. |
| core/relay/adaptor/gemini/config.go | Adds config toggles for auto base64 of audio/video URLs. |
| core/relay/adaptor/gemini/claude.go | Updates callsite for expanded buildContents signature. |
| core/relay/adaptor/gemini/adaptor.go | Surfaces new Gemini channel config options in adaptor metadata. |
| core/relay/adaptor/anthropic/openai.go | Changes image URL→base64 patching to best-effort (log + keep URL). |
| core/relay/adaptor/anthropic/openai_test.go | Adds test ensuring failures keep original image URL. |
| core/relay/adaptor/anthropic/main.go | Makes AST-based image URL→base64 conversion best-effort with logging. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+788
to
803
| var wg sync.WaitGroup | ||
|
|
||
| for _, task := range imageTasks { | ||
| if task.FileData == nil || task.FileData.FileURI == "" { | ||
| continue | ||
| } | ||
|
|
||
| wg.Go(func() { | ||
| _ = sem.Acquire(ctx, 1) | ||
| if err := sem.Acquire(ctx, 1); err != nil { | ||
| log.Warnf("convert gemini image url to base64 skipped, keep original url: %v", err) | ||
| return | ||
| } | ||
| defer sem.Release(1) | ||
|
|
||
| mimeType, data, err := image.GetImageFromURL(ctx, task.FileData.FileURI) | ||
| if err != nil { |
Comment on lines
+832
to
+856
| var wg sync.WaitGroup | ||
|
|
||
| for _, task := range mediaTasks { | ||
| if task.FileData == nil || task.FileData.FileURI == "" { | ||
| continue | ||
| } | ||
|
|
||
| wg.Go(func() { | ||
| if err := sem.Acquire(ctx, 1); err != nil { | ||
| log.Warnf( | ||
| "convert gemini %s url to base64 skipped, keep original url: %v", | ||
| mediaType, | ||
| err, | ||
| ) | ||
|
|
||
| return | ||
| } | ||
| defer sem.Release(1) | ||
|
|
||
| mimeType, data, err := getGeminiMediaFromURL( | ||
| ctx, | ||
| task.FileData.FileURI, | ||
| mediaType, | ||
| task.FileData.MimeType, | ||
| ) |
Comment on lines
300
to
333
| @@ -334,12 +333,6 @@ func batchPatchImage2Base64(ctx context.Context, imageTasks []*relaymodel.Claude | |||
| } | |||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.