fix: gemini image video some error and converts#592
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR improves Gemini image/video relay behavior by adding a dedicated Gemini file-download mode and enhancing video operation handling (including URI rewriting and RAI-filtered failure surfacing). It also refactors/standardizes error handling across several adaptors and introduces a DEFAULT_HOST configuration used for generating public-facing URLs.
Changes:
- Add
/v1beta/files/*...routing plus a newmode.GeminiFilesto proxy Gemini-generated media downloads. - Enhance Gemini video operation handling: store generated file URIs, rewrite operation responses to proxy URLs, treat RAI-filtered results as failures, and improve async usage error reporting.
- Improve Gemini image empty-response error messaging and standardize upstream error parsing for Coze/Cohere/Ali.
Reviewed changes
Copilot reviewed 40 out of 40 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| core/router/relay.go | Registers the new /v1beta/files/*model endpoint to reach Gemini file proxying. |
| core/router/relay_test.go | Asserts the new files route is registered. |
| core/relay/utils/testreq.go | Updates test request payloads (image prompt; Gemini video parameters removal). |
| core/relay/utils/testreq_test.go | Updates/extends tests to validate image prompt and video request JSON. |
| core/relay/model/gemini_video.go | Aligns Gemini video operation error type and adds RAI filtering fields to the response model. |
| core/relay/model/errors.go | Ensures GeminiFiles mode uses Gemini-style error wrapping. |
| core/relay/mode/define.go | Introduces GeminiFiles mode. |
| core/relay/mode/define_test.go | Locks in the new persisted mode ID for GeminiFiles. |
| core/relay/meta/meta.go | Adds FileID to relay metadata plus a helper option. |
| core/relay/adaptor/vertexai/gemini/adapter.go | Passes store into Gemini video operation handler (needed for file store saving). |
| core/relay/adaptor/vertexai/adaptor_test.go | Adjusts VertexAI tests to set the correct channel type. |
| core/relay/adaptor/gemini/video.go | Implements Gemini file proxy plumbing (URI rewrite, file store save/load, file download handler), RAI-filter failure handling, and numberOfVideos stripping logic. |
| core/relay/adaptor/gemini/video_test.go | Adds broad test coverage for new video/file behaviors (URI rewrite, stored downloads, RAI filtering behavior, etc.). |
| core/relay/adaptor/gemini/image.go | Improves the error message when Gemini returns an empty image response. |
| core/relay/adaptor/gemini/image_test.go | Tests the enhanced empty-image error message content. |
| core/relay/adaptor/gemini/export_test.go | Refactors test helper to reuse the new shared conversion helper. |
| core/relay/adaptor/gemini/error.go | Splits Gemini error parsing from wrapper creation and adds OpenAI-video error conversion helper. |
| core/relay/adaptor/gemini/async_usage.go | Treats RAI-filtered operations as completed failures in async usage fetching. |
| core/relay/adaptor/gemini/adaptor.go | Adds GeminiFiles to supported modes and routes it to the new file handler. |
| core/relay/adaptor/coze/main.go | Switches Coze adaptor to use a local error handler. |
| core/relay/adaptor/coze/error.go | Adds Coze-specific upstream error parsing into OpenAI error format. |
| core/relay/adaptor/coze/error_test.go | Tests Coze error parsing. |
| core/relay/adaptor/cohere/main.go | Switches Cohere adaptor to use a local error handler. |
| core/relay/adaptor/cohere/error.go | Adds Cohere-specific upstream error parsing into OpenAI error format. |
| core/relay/adaptor/cohere/error_test.go | Tests Cohere error parsing. |
| core/relay/adaptor/ali/error.go | Refactors Ali error handling to parse response bodies directly (not via OpenAI adaptor). |
| core/relay/adaptor/ali/adaptor_test.go | Adds coverage for Ali top-level error format parsing. |
| core/model/yaml_integration.go | Allows YAML model type names to map to GeminiFiles. |
| core/model/store.go | Adds a GeminiFileStoreID store key helper. |
| core/model/option.go | Adds/updates runtime options for DefaultHost and MCP host behavior. |
| core/middleware/distributor.go | Adds request-model resolution and ctx metadata plumbing for Gemini file downloads. |
| core/middleware/distributor_gemini_video_test.go | Ensures GeminiFiles is accepted for GeminiVideo model mode routing. |
| core/middleware/ctxkey.go | Adds a new context key for file_id. |
| core/controller/relay.go | Updates GeminiByPath routing to detect file download paths and select GeminiFiles mode. |
| core/controller/mcp/groupmcp.go | Uses configured default host behavior for MCP endpoints (path mode). |
| core/controller/mcp/endpoint_test.go | Adds tests for default-host behavior in MCP endpoint generation. |
| core/common/consume/consume.go | Skips consumption recording on successful GeminiFiles reads (like other stored-content reads). |
| core/common/consume/consume_record_test.go | Updates tests for consume-record skipping to include GeminiFiles. |
| core/common/config/config.go | Introduces DEFAULT_HOST and makes MCP host optionally fall back to it. |
| core/common/config/config_test.go | Tests DefaultMCPHost fallback/override semantics. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
140
to
146
| case mode.ImagesGenerations: | ||
| return gemini.ImageHandler(meta, c, resp) | ||
| case mode.GeminiVideo: | ||
| return gemini.NativeVideoHandler(meta, store, c, resp) | ||
| case mode.GeminiVideoOperations: | ||
| return gemini.NativeVideoOperationHandler(meta, c, resp) | ||
| return gemini.NativeVideoOperationHandler(meta, store, c, resp) | ||
| case mode.VideoGenerationsJobs: |
Comment on lines
+1816
to
+1834
| func geminiFileProxyURL(c *gin.Context, fileID string) string { | ||
| if fileID == "" || c == nil || c.Request == nil { | ||
| return "" | ||
| } | ||
|
|
||
| scheme := firstNonEmpty(c.Request.Header.Get("X-Forwarded-Proto"), c.Request.URL.Scheme) | ||
| if scheme == "" { | ||
| if c.Request.TLS != nil { | ||
| scheme = "https" | ||
| } else { | ||
| scheme = "http" | ||
| } | ||
| } | ||
|
|
||
| host := firstNonEmpty(c.Request.Header.Get("X-Forwarded-Host"), c.Request.Host) | ||
| if defaultHost := config.GetDefaultHost(); defaultHost != "" { | ||
| host = defaultHost | ||
| } | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.