📺 feat: Google URL Context Param with Native YouTube Video Understanding#13924
Conversation
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f61235d831
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -780,6 +793,7 @@ const googleConfig: SettingsConfiguration = [ | |||
| google.thinkingBudget, | |||
| google.thinkingLevel, | |||
| google.web_search, | |||
| google.url_context, | |||
There was a problem hiding this comment.
When this toggle is shown for the normal Google conversation/preset settings, the description promises YouTube links are handled via native video understanding, but the only call to appendYouTubeVideoParts is in api/server/controllers/agents/client.js; the regular Google endpoint path never adds the fileData media parts. In non-Agent Google chats, enabling URL Context and sending a YouTube URL will therefore still only give urlContext an unsupported YouTube URL instead of the promised video-understanding input.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Regular Google conversations and presets already go through this path, so the injection covers them. Non-Agent endpoints do not use a separate GoogleClient: they route through the same agents framework as ephemeral agents. api/server/routes/ exposes only agents, and api/server/services/Endpoints/agents/build.js assigns agent_id = Constants.EPHEMERAL_AGENT_ID for non-agent endpoints. So AgentClient.buildMessages runs for normal Google chats too, and appendYouTubeVideoParts fires whenever the resolved urlContext tool is present on the ephemeral Google/Vertex agent. There is no non-agent Google chat path that bypasses it.
| const existing = collectFileUris(baseParts); | ||
| const newParts = urls | ||
| .filter((url) => !existing.has(url)) | ||
| .map((fileUri) => ({ type: 'media', mimeType: undefined, fileUri }) as MessageContentComplex); |
There was a problem hiding this comment.
For Vertex agents with url_context enabled, this constructs every YouTube fileUri part with mimeType: undefined, which is then omitted from the serialized fileData. Vertex video requests require a mimeType whenever fileUri is used, so a public YouTube link that should be routed through video understanding can be rejected before the model runs.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 468553d8. YouTube fileData parts now carry mimeType: 'video/mp4' on Vertex (matching the Vertex samples, which always set it) via the new resolveYouTubeInjectionConfig. The Gemini Developer API path still omits the mimeType, matching its own docs/examples.
| */ | ||
|
|
||
| /** Default cap on auto-injected YouTube video parts per message (Gemini practical limits). */ | ||
| export const DEFAULT_MAX_YOUTUBE_PARTS = 5; |
There was a problem hiding this comment.
This default is used unconditionally when AgentClient calls appendYouTubeVideoParts, but Vertex only supports one YouTube video URL per request. With a Vertex agent and url_context enabled, a user message containing two YouTube links can inject multiple fileData video parts and cause the provider request to be rejected instead of analyzing the videos.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 468553d8. The per-request cap is now provider/model-aware via resolveYouTubeInjectionConfig: Vertex is capped at 1 YouTube URL, and the Gemini Developer API caps pre-2.5 models at 1 while allowing the default for 2.5+ (which accept up to 10). A multi-link message can no longer exceed the provider limit.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 468553d80e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (enableUrlContext) { | ||
| tools.push({ urlContext: {} }); |
There was a problem hiding this comment.
When url_context is enabled this unconditionally adds the native urlContext tool for any Google/Vertex model, including existing gemini-1.5-* and gemini-2.0-* selections (the new tests even cover 1.5/2.0). Google's URL Context supported-model table currently lists only Gemini 2.5+ and 3.x models (https://ai.google.dev/gemini-api/docs/url-context#supported_models), so enabling the new switch on older configured models will turn otherwise valid chats into provider-side 400s instead of hiding/ignoring the option for unsupported models.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed: URL Context is Gemini 2.5+/3.x only per the supported-models table. This currently mirrors the existing search-grounding wiring, which likewise does not model-gate googleSearch (the provider validates and errors on unsupported models). Gating url_context to 2.5+ is reasonable, but it deviates from that grounding parity and interacts with the YouTube companion: native YouTube video understanding is supported on 2.0+, so gating the tool would also disable YouTube on 2.0-2.4 unless decoupled. Deferring the gate-vs-mirror-grounding decision to the maintainer and will follow up based on that call.
There was a problem hiding this comment.
Resolved in 7213cd16 per maintainer decision. The urlContext tool is now gated to Gemini 2.5+/3.x in getGoogleConfig (debug-log + no-op on earlier models), matching the supported-models list. The YouTube video-understanding injection keys off the resolved urlContext tool, so it is gated to 2.5+ as well. Added llm.spec/config.google.spec coverage for the gate.
| google.web_search, | ||
| google.url_context, |
There was a problem hiding this comment.
This setting is added to the normal Google endpoint controls, but the only call site I found for appendYouTubeVideoParts is AgentClient.buildMessages; direct Google chats still go through initializeGoogle/getGoogleConfig and send only the text URL plus urlContext. Since Google's URL Context limitations explicitly exclude YouTube videos (https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/url-context#supported_and_unsupported_content_types), users enabling this switch on the standard Google endpoint will not get the advertised native YouTube video understanding path.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
There is no separate non-agent Google chat client, so this path is covered. api/server/routes/ exposes only agents, and api/server/services/Endpoints/agents/build.js assigns agent_id = Constants.EPHEMERAL_AGENT_ID for non-agent endpoints, so the standard Google endpoint runs through AgentClient.buildMessages as an ephemeral agent, where appendYouTubeVideoParts fires. initializeGoogle/getGoogleConfig is the provider-config step invoked by that same agents flow (getProviderConfig -> getOptions), not a separate chat path. (api/strategies/googleStrategy.js is passport OAuth login, unrelated.) So direct Google chats do get the native YouTube path.
| /** @endpoints google */ | ||
| url_context: true, |
There was a problem hiding this comment.
For custom endpoints configured with customParams.defaultParamsEndpoint: 'google', the parameters panel renders the Google settings (including this new switch), but buildEndpointOption still parses endpoint: custom with openAISchema, which does not pick url_context. I checked that path through parseCompactConvo, so a user's per-chat URL Context toggle on a Google-compatible custom endpoint is stripped before getOpenAIConfig; only admin-level addParams/defaultParams can enable it.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 1e60e409. Added url_context to openAIBaseSchema so the per-chat value round-trips for Google-compatible custom endpoints, matching how web_search is already picked there. As with web_search on that path, the tool is enabled via admin addParams/defaultParams (the transform intentionally only preserves explicitly-added native tools).
There was a problem hiding this comment.
Follow-up: reverted the openAIBaseSchema change in 754440ae. It caused url_context to leak to non-Google OpenAI endpoints (separate thread), and it only added inert per-chat persistence anyway: the transform intentionally only enables native tools added via admin addParams/defaultParams, so per-chat toggles do not enable the tool for custom endpoints (identical to web_search). So url_context on Google-compatible custom endpoints is admin-configured by design.
| /** Preserve urlContext if url_context was explicitly enabled */ | ||
| if (urlContextExplicitlyEnabled && toolKeys.includes('urlContext')) { | ||
| return true; |
There was a problem hiding this comment.
When a Google-compatible custom endpoint enables addParams.url_context, this preserves { urlContext: {} } in the Chat Completions tools array. Google's OpenAI-compatibility docs route Gemini-only chat features through extra_body and do not define urlContext as an OpenAI chat tool (OpenAI tools require a typed tool shape), so this admin-supported path can send an invalid tool payload instead of enabling URL Context.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
This mirrors the existing, shipped behavior for googleSearch: the OpenAI-config transform already preserves { googleSearch: {} } in the Chat Completions tools array when web_search is explicitly enabled for a Google-compatible custom endpoint (covered in config.google.spec.ts). url_context is handled identically, by design. If Google's OpenAI-compat layer needs these Gemini-native tools routed via extra_body instead of tools, that is a pre-existing concern affecting googleSearch too and should be fixed uniformly rather than diverging url_context from grounding here.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7213cd1681
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| /** @endpoints google-compatible custom endpoints (defaultParamsEndpoint: 'google') */ | ||
| url_context: true, |
There was a problem hiding this comment.
Including url_context in the shared openAIBaseSchema means the flag is preserved for normal OpenAI/Azure/OpenRouter/custom-OpenAI requests as well as Google-compatible custom endpoints. In the non-Google branch, getOpenAILLMConfig only special-cases web_search; url_context remains in modelOptions and is copied into llmConfig, so a conversation or preset that previously enabled Google URL Context can send an unsupported url_context parameter after switching to an OpenAI-style endpoint. Keep this out of the shared OpenAI schema or explicitly drop it unless defaultParamsEndpoint === 'google'.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 754440ae. Reverted the openAIBaseSchema addition (from an earlier round): url_context is Google-only and should not ride the shared OpenAI schema, where it would leak into OpenAI/Azure/OpenRouter requests. On Google-compatible custom endpoints, url_context is enabled via admin addParams/defaultParams (handled by getOpenAIConfig -> getGoogleConfig + the transform), which is the same way it functions for web_search there.
| if ( | ||
| latestOrdered?.isCreatedByUser === true && | ||
| (provider === Providers.GOOGLE || provider === Providers.VERTEXAI) && | ||
| hasUrlContextTool(this.options.agent?.tools) |
There was a problem hiding this comment.
The YouTube workaround is only called in this AgentClient branch, but the same change exposes url_context on the regular Google endpoint settings and describes YouTube links as handled via native video understanding. In a normal Google/Vertex chat outside the agents endpoint, enabling URL Context with a YouTube URL never reaches appendYouTubeVideoParts; the request only contains the unsupported URL in text for the urlContext tool, so the advertised video-understanding path is skipped. Add the same injection to the standard Google message formatting path or scope the setting/description to agents.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
There is no non-agent Google chat path. The only chat route registered is /api/agents/chat (api/server/index.js:268-269) -> routes.agents -> AgentClient; there is no /api/ask or /api/edit route and no GoogleClient class anywhere. For endpoint google, api/server/services/Endpoints/agents/build.js assigns agent_id = Constants.EPHEMERAL_AGENT_ID and the request runs through AgentClient.buildMessages, where appendYouTubeVideoParts fires. initializeGoogle/getGoogleConfig is the provider-config step that same agents flow calls (getProviderConfig -> getOptions); it is not a separate message-formatting path. So a normal Google/Vertex chat does reach the injection. (Confirmed there are no Ask/Edit controllers wired and no GoogleClient in the tree.)
| if (newParts.length === 0) { | ||
| return content; | ||
| } | ||
| return [...baseParts, ...newParts]; |
There was a problem hiding this comment.
This appends a Gemini video part but leaves the original text part, including the YouTube URL, in the same prompt. Since the URL Context tool discovers URLs from prompt text and is limited to 20 URLs per request, a message with YouTube links plus ordinary URLs still sends those unsupported YouTube URLs through urlContext, which can consume the URL budget or trigger unsupported-content handling instead of routing YouTube solely through video understanding. Remove or rewrite matched YouTube URLs in the LLM text payload when adding the media parts.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 754440ae. When YouTube video parts are injected, the matched YouTube URLs are now stripped from the prompt text (full URL token, including trailing query/fragment), so the urlContext tool no longer sees them and cannot spend its URL budget on unsupported YouTube links. Non-YouTube URLs are left intact for the tool, and a text part that becomes empty after stripping is dropped. Covered by new youtube.spec cases.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 754440aef8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (newParts.length === 0) { | ||
| return content; | ||
| } | ||
| return [...stripYouTubeFromTextParts(baseParts), ...newParts]; |
There was a problem hiding this comment.
When max caps the injected videos (notably Vertex always passes max: 1), extractYouTubeUrls(text, max) only appends the first N videos, but this call strips every YouTube URL from the text parts. A prompt with two YouTube links on Vertex therefore sends only the first video as media and silently removes the second link from the prompt entirely, so the model cannot see or reason about it.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 15e0c2d2. Stripping is now id-aware: only the YouTube links actually routed to video parts (the capped + deduped set) are removed from the text. Over-limit links (e.g. the 2nd link on Vertex with max=1) stay in the prompt text so the model can still see and reason about them. New youtube.spec case covers it.
| const urlContextExplicitlyEnabled = | ||
| addParams?.url_context === true || defaultParams?.url_context === true; |
There was a problem hiding this comment.
For a custom OpenAI-compatible Gemini endpoint configured with customParams.defaultParamsEndpoint: 'google', the parameter panel/parseCompactConvo path sends the user's url_context switch in modelOptions, and getGoogleConfig turns that into { urlContext: {} }. This transform only preserves urlContext when it came from admin addParams or defaultParams, so the subsequent filter drops the tool and the user-facing URL Context toggle has no effect on those custom endpoints.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
This is intentional parity with web_search/googleSearch on custom endpoints: the transform only enables native Google tools when explicitly set via admin addParams/defaultParams, and filters per-chat (modelOptions) toggles (covered by config.google.spec.ts for googleSearch). Honoring per-chat modelOptions for url_context here would diverge url_context from web_search on the same path. On the native Google/Vertex endpoint (not the OpenAI-compat custom path), the per-chat toggle works directly via getGoogleConfig. Making custom-endpoint per-chat native tools work would be a separate change applying to web_search too.
| continue; | ||
| } | ||
| seen.add(videoId); | ||
| urls.push(`https://www.youtube.com/watch?v=${videoId}`); |
There was a problem hiding this comment.
Timestamped YouTube links such as youtu.be/<id>?t=90 or watch?v=<id>&t=90 are normalized here to a bare watch URL, and the original URL token is later stripped from the prompt text. In prompts like “what happens here ”, the model receives the full video with no timestamp cue, so it can answer about the wrong moment instead of the user-selected segment.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 15e0c2d2. Timestamped YouTube links (?t= / &start=) are now kept in the prompt text rather than stripped, so the moment cue survives for prompts like "what happens here " while the video still flows through video understanding. (Full segment focusing would use Gemini's videoMetadata.start_offset, which the agents content formatter doesn't expose yet; preserving the textual cue is the LibreChat-side fix.) Covered by a new youtube.spec case.
| } | ||
| const major = Number(match[1]); | ||
| const minor = Number(match[2] ?? '0'); | ||
| return major > 2 || (major === 2 && minor >= 5); |
There was a problem hiding this comment.
This check treats every gemini-3*/2.5* name as URL Context-capable, including modality-specific models such as image, live, and TTS variants that the rest of the agent code already treats as non-text for Google tool combinations. When a user enables url_context on one of those models, getGoogleConfig will still send { urlContext: {} }, producing a provider-side unsupported-tool error instead of skipping the tool as intended.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 15e0c2d2. supportsUrlContext now excludes non-text modality variants (image/live/tts) using the same modality regex the Google tool-combination path already uses, so url_context is skipped on e.g. gemini-2.5-flash-image / -live / -tts instead of sending an unsupported tool. New llm.spec case covers it.
| @@ -780,6 +793,7 @@ const googleConfig: SettingsConfiguration = [ | |||
| google.thinkingBudget, | |||
| google.thinkingLevel, | |||
| google.web_search, | |||
| google.url_context, | |||
There was a problem hiding this comment.
Adding this setting to the regular Google parameter set exposes the same URL Context toggle on non-agent Google chats, but the only YouTube-to-fileData rewrite added in this change lives in AgentClient.buildMessages. In standard Google endpoint conversations, a YouTube URL is still sent only through urlContext even though this change notes that tool cannot read YouTube, so the advertised native video handling does not run for those users.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
There is no non-agent Google chat path, so this is covered (same as the earlier threads). The only chat route is /api/agents/chat (api/server/index.js:268-269); there is no /api/ask or /api/edit route and no GoogleClient class. For endpoint google, build.js sets agent_id = Constants.EPHEMERAL_AGENT_ID and the chat runs through AgentClient.buildMessages, where appendYouTubeVideoParts fires. So standard Google endpoint conversations do get native YouTube handling.
| const { max, mimeType } = resolveYouTubeInjectionConfig({ | ||
| provider, | ||
| model: this.options.agent?.model, |
There was a problem hiding this comment.
When an agent's model is overridden through model_parameters or the current request options, initialization resolves the actual model onto agent.model_parameters.model (also stored as this.model), while agent.model can remain the saved base value. Passing the stale base model here misclassifies Gemini 2.5/3 runs as older or unknown and caps automatic YouTube video injection to one video, so additional YouTube links are not routed through video understanding for those overridden-model runs.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 15e0c2d2. The injection now resolves the run model via this.options.agent?.model_parameters?.model ?? this.options.agent?.model, so a model_parameters/request override is classified correctly (2.5/3 runs no longer fall back to the 1-video cap).
| '(?:(?:[a-z0-9-]+\\.)*youtube\\.com\\/(?:watch\\?(?:\\S*?&)?v=|shorts\\/|live\\/|embed\\/|v\\/)|youtu\\.be\\/)' + | ||
| '([A-Za-z0-9_-]{11})(?![A-Za-z0-9_-])', |
There was a problem hiding this comment.
Privacy-enhanced embeds commonly use youtube-nocookie.com/embed/<id>, which is still a YouTube video URL users paste from embed snippets. Because this matcher only accepts youtube.com or youtu.be, those links bypass the media-part path and are left for urlContext, which this change notes cannot read YouTube, so native video understanding does not run for that embed form.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 15e0c2d2. The matcher now recognizes youtube-nocookie.com/embed/<id> and normalizes it to the canonical watch URL, so privacy-enhanced embed links route through video understanding. New youtube.spec case covers it.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 15e0c2d2d0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -780,6 +793,7 @@ const googleConfig: SettingsConfiguration = [ | |||
| google.thinkingBudget, | |||
| google.thinkingLevel, | |||
| google.web_search, | |||
| google.url_context, | |||
There was a problem hiding this comment.
Adding google.url_context to the regular Google settings exposes this behavior on normal Google conversations, but the YouTube-to-video conversion is only called from api/server/controllers/agents/client.js (the only appendYouTubeVideoParts caller in the repo). In a non-agent Google chat with URL Context enabled, a pasted YouTube link stays only in the text, so the URL Context tool still cannot read it despite the new UI description saying YouTube links are handled via native video understanding. Either apply the injection in the Google endpoint path as well, or avoid advertising/exposing that YouTube behavior there.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Non-agent Google chats run through this same AgentClient path, so the injection does apply. The only chat route is /api/agents/chat (api/server/index.js:268-269); there is no /api/ask or /api/edit route and no GoogleClient class, and build.js assigns EPHEMERAL_AGENT_ID for non-agent endpoints, so a standard Google conversation reaches AgentClient.buildMessages -> appendYouTubeVideoParts. Verified at the route level across several prior threads.
| if (YOUTUBE_TIMESTAMP_REGEX.test(trailing)) { | ||
| return match; | ||
| } |
There was a problem hiding this comment.
When a valid YouTube watch URL puts the timestamp before v (for example https://www.youtube.com/watch?t=90&v=dQw4w9WgXcQ), the strip regex consumes t=90& as part of the prefix and trailing contains only the text after the id, so this check misses the timestamp and removes the whole URL from the prompt. That loses the user's requested time cue even though timestamped links are meant to remain visible; test the full matched URL (or capture the pre-v query) before stripping.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 619b474. The timestamp check now tests the full matched URL rather than only the post-id trailing, so a time before v= (e.g. watch?t=90&v=<id>) is detected and the link is kept in the prompt text. New youtube.spec case covers the pre-id form.
| @@ -135,6 +136,30 @@ function isGemini35Flash(model: string) { | |||
| return modelId === GEMINI_3_5_FLASH || modelId.startsWith(`${GEMINI_3_5_FLASH}-`); | |||
| } | |||
|
|
|||
| const urlContextModelRegex = /gemini-(\d+)(?:\.(\d+))?/i; | |||
| const urlContextExcludedModalityRegex = /(?:^|-)image(?:-|$)|(?:^|-)live(?:-|$)|(?:^|-)tts(?:-|$)/; | |||
There was a problem hiding this comment.
When url_context is enabled for an audio-specific Gemini model such as gemini-2.5-flash-preview-native-audio-dialog, this exclusion still lets it through because it only rejects image/live/TTS names. supportsUrlContext then adds { urlContext: {} } based solely on the gemini-2.5 version match, so audio-model requests fail at the provider instead of being skipped like the other modality-specific variants this guard is meant to filter.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 619b474. Added audio to the modality exclusion (now /(?:^|-)(?:image|live|tts|audio)(?:-|$)/), so audio-only variants like gemini-2.5-flash-preview-native-audio-dialog skip the urlContext tool instead of 400ing. Added to the modality-exclusion test.
| latestOrdered?.isCreatedByUser === true && | ||
| (provider === Providers.GOOGLE || provider === Providers.VERTEXAI) && | ||
| hasUrlContextTool(this.options.agent?.tools) |
There was a problem hiding this comment.
When an agent uses a custom OpenAI-compatible Gemini endpoint (customParams.defaultParamsEndpoint: 'google'), the new OpenAI transform can pass through { urlContext: {} }, but initializeAgent normalizes that provider to OpenAI, so this Google/Vertex-only gate skips the YouTube video-part injection. In that custom Gemini-agent configuration, YouTube links remain only as text and the URL Context tool still cannot read them even though URL Context is enabled; gate this on the resolved urlContext tool (or the custom Google defaultParamsEndpoint) instead of only agent.provider.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Gating on agent.provider here is intentional. YouTube parts are { type: 'media', fileUri } blocks that only the agents Google content formatter (convertBaseMessagesToContent -> messageContentMedia) converts to Gemini fileData. A custom OpenAI-compatible Gemini endpoint resolves to the OpenAI provider and uses the OpenAI message formatter, which would not convert a media part to fileData — injecting there would produce malformed content. So YouTube-as-video is intentionally native-Google/Vertex-only; the url_context tool still passes through for custom Gemini (handled by the transform). Gating on hasUrlContextTool alone would inject parts the OpenAI path can't render.
| if (key === 'url_context') { | ||
| if (typeof value === 'boolean') { | ||
| enableUrlContext = value; | ||
| } | ||
| continue; |
There was a problem hiding this comment.
This new special case consumes every addParams.url_context value and then continues, but only booleans actually update enableUrlContext. The config schema currently validates non-boolean shapes only for web_search, so a config like addParams: { url_context: { enabled: true } } can pass validation and then be silently ignored here rather than rejected or forwarded, leaving admins with a valid-looking config that does not enable the tool. Mirror the web_search validation/coercion for url_context.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
url_context already mirrors web_search exactly here: both are z.boolean().optional() in the zod schema (so a non-boolean shape fails validation identically), and both use the same typeof value === 'boolean' guard in getGoogleConfig. There is no extra validation/coercion on web_search to mirror — the handlers are line-for-line equivalent. So a non-boolean url_context is rejected/ignored the same way web_search is.
…tanding Mirror the web_search grounding wiring for a new Google/Gemini `url_context` model param (resolves to the native `urlContext` tool). When enabled, YouTube URLs in the latest user message are injected as Gemini video parts (fileData), since the URL Context tool does not support YouTube.
Address Codex review on the YouTube video-understanding path: - Cap injected YouTube parts per request by provider/model (Vertex: 1; Gemini Developer API: 10 on 2.5+, 1 on earlier models) so multi-link messages cannot exceed the provider limit and get rejected. - Set a video/mp4 mimeType on Vertex YouTube fileData (matching Vertex samples); the Developer API still omits it.
Add url_context to openAIBaseSchema so the per-chat value persists for custom endpoints configured with customParams.defaultParamsEndpoint: 'google', matching how web_search is already picked there.
Per Google's URL Context supported-models list (2.5+/3.x only), skip the native urlContext tool on earlier models (debug-log + no-op) instead of sending it and triggering a provider 400. This also gates the coupled YouTube video-understanding injection to 2.5+, since it keys off the resolved urlContext tool.
… of OpenAI schema - Remove url_context from the shared openAIBaseSchema (revert): it is Google-only and would otherwise leak as an unsupported param to OpenAI/Azure/OpenRouter requests. On Google-compatible custom endpoints url_context is enabled via admin addParams/defaultParams, same as web_search. - When injecting YouTube video parts, strip the matched YouTube URLs from the prompt text so the urlContext tool (which reads URLs from text and cannot fetch YouTube) does not consume its URL budget on them. Non-YouTube URLs are left intact.
Address Codex round 4: - Exclude non-text modality variants (image/live/tts) from URL Context support, mirroring the Google tool-combination modality exclusion. - Use the resolved run model (model_parameters.model) for YouTube injection limits instead of the saved base model. - Strip only the YouTube links actually routed to video (id-aware); keep over-limit links in the text so the model can still reason about them. - Keep timestamped YouTube links (?t=/&start=) in the text so the moment cue survives. - Recognize youtube-nocookie.com/embed links.
15e0c2d to
585f83c
Compare
…tamps Address Codex round 5: - Add `audio` to the url_context modality exclusion so audio-only Gemini variants (e.g. gemini-2.5-flash-preview-native-audio-dialog) skip the tool instead of 400ing. - Detect YouTube timestamps anywhere in the matched URL (incl. before `v=`, e.g. watch?t=90&v=<id>), so timestamped links are kept in the prompt text as intended.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 619b47489f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| google.thinkingBudget, | ||
| google.thinkingLevel, | ||
| google.web_search, | ||
| google.url_context, |
There was a problem hiding this comment.
Adding google.url_context to googleConfig exposes the toggle on the normal Google endpoint, but the YouTube-to-fileData workaround is only invoked from api/server/controllers/agents/client.js in AgentClient.buildMessages (appendYouTubeVideoParts has no non-agent call sites). For a regular Google chat with url_context enabled and a YouTube URL, the request still sends the YouTube link only as text to the urlContext tool, which this change documents as unable to fetch YouTube, so those prompts fail or ignore the video even though the UI says YouTube is handled separately. Wire the same transformation into the regular Google message path or avoid exposing/promising that behavior there.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Same as the prior threads on this point: the regular Google endpoint IS served by AgentClient, so this path runs. /api/agents/chat is the only chat route (api/server/index.js:268-269); non-agent endpoints (incl. google) get agent_id = Constants.EPHEMERAL_AGENT_ID in api/server/services/Endpoints/agents/build.js and execute through AgentClient.buildMessages, where appendYouTubeVideoParts is called. There is no separate non-agent Google chat client (no GoogleClient class, no /api/ask or /api/edit route). So a normal Google chat with url_context + a YouTube link does get the fileData injection. Confirmed at the route level across rounds 1-5.
…ing (danny-avila#13924) * ✨ feat: Add Google url_context Param with Native YouTube Video Understanding Mirror the web_search grounding wiring for a new Google/Gemini `url_context` model param (resolves to the native `urlContext` tool). When enabled, YouTube URLs in the latest user message are injected as Gemini video parts (fileData), since the URL Context tool does not support YouTube. * 🎞️ fix: Provider-aware YouTube injection limits for url_context Address Codex review on the YouTube video-understanding path: - Cap injected YouTube parts per request by provider/model (Vertex: 1; Gemini Developer API: 10 on 2.5+, 1 on earlier models) so multi-link messages cannot exceed the provider limit and get rejected. - Set a video/mp4 mimeType on Vertex YouTube fileData (matching Vertex samples); the Developer API still omits it. * 🧩 fix: Round-trip url_context for Google-compatible custom endpoints Add url_context to openAIBaseSchema so the per-chat value persists for custom endpoints configured with customParams.defaultParamsEndpoint: 'google', matching how web_search is already picked there. * 🚦 fix: Gate url_context tool to Gemini 2.5+ models Per Google's URL Context supported-models list (2.5+/3.x only), skip the native urlContext tool on earlier models (debug-log + no-op) instead of sending it and triggering a provider 400. This also gates the coupled YouTube video-understanding injection to 2.5+, since it keys off the resolved urlContext tool. * ✂️ fix: Strip YouTube URLs from urlContext text; keep url_context out of OpenAI schema - Remove url_context from the shared openAIBaseSchema (revert): it is Google-only and would otherwise leak as an unsupported param to OpenAI/Azure/OpenRouter requests. On Google-compatible custom endpoints url_context is enabled via admin addParams/defaultParams, same as web_search. - When injecting YouTube video parts, strip the matched YouTube URLs from the prompt text so the urlContext tool (which reads URLs from text and cannot fetch YouTube) does not consume its URL budget on them. Non-YouTube URLs are left intact. * 🎯 fix: Refine url_context model gating and YouTube injection edges Address Codex round 4: - Exclude non-text modality variants (image/live/tts) from URL Context support, mirroring the Google tool-combination modality exclusion. - Use the resolved run model (model_parameters.model) for YouTube injection limits instead of the saved base model. - Strip only the YouTube links actually routed to video (id-aware); keep over-limit links in the text so the model can still reason about them. - Keep timestamped YouTube links (?t=/&start=) in the text so the moment cue survives. - Recognize youtube-nocookie.com/embed links. * 🎚️ fix: Exclude audio Gemini variants + preserve pre-id YouTube timestamps Address Codex round 5: - Add `audio` to the url_context modality exclusion so audio-only Gemini variants (e.g. gemini-2.5-flash-preview-native-audio-dialog) skip the tool instead of 400ing. - Detect YouTube timestamps anywhere in the matched URL (incl. before `v=`, e.g. watch?t=90&v=<id>), so timestamped links are kept in the prompt text as intended.
…ing (danny-avila#13924) * ✨ feat: Add Google url_context Param with Native YouTube Video Understanding Mirror the web_search grounding wiring for a new Google/Gemini `url_context` model param (resolves to the native `urlContext` tool). When enabled, YouTube URLs in the latest user message are injected as Gemini video parts (fileData), since the URL Context tool does not support YouTube. * 🎞️ fix: Provider-aware YouTube injection limits for url_context Address Codex review on the YouTube video-understanding path: - Cap injected YouTube parts per request by provider/model (Vertex: 1; Gemini Developer API: 10 on 2.5+, 1 on earlier models) so multi-link messages cannot exceed the provider limit and get rejected. - Set a video/mp4 mimeType on Vertex YouTube fileData (matching Vertex samples); the Developer API still omits it. * 🧩 fix: Round-trip url_context for Google-compatible custom endpoints Add url_context to openAIBaseSchema so the per-chat value persists for custom endpoints configured with customParams.defaultParamsEndpoint: 'google', matching how web_search is already picked there. * 🚦 fix: Gate url_context tool to Gemini 2.5+ models Per Google's URL Context supported-models list (2.5+/3.x only), skip the native urlContext tool on earlier models (debug-log + no-op) instead of sending it and triggering a provider 400. This also gates the coupled YouTube video-understanding injection to 2.5+, since it keys off the resolved urlContext tool. * ✂️ fix: Strip YouTube URLs from urlContext text; keep url_context out of OpenAI schema - Remove url_context from the shared openAIBaseSchema (revert): it is Google-only and would otherwise leak as an unsupported param to OpenAI/Azure/OpenRouter requests. On Google-compatible custom endpoints url_context is enabled via admin addParams/defaultParams, same as web_search. - When injecting YouTube video parts, strip the matched YouTube URLs from the prompt text so the urlContext tool (which reads URLs from text and cannot fetch YouTube) does not consume its URL budget on them. Non-YouTube URLs are left intact. * 🎯 fix: Refine url_context model gating and YouTube injection edges Address Codex round 4: - Exclude non-text modality variants (image/live/tts) from URL Context support, mirroring the Google tool-combination modality exclusion. - Use the resolved run model (model_parameters.model) for YouTube injection limits instead of the saved base model. - Strip only the YouTube links actually routed to video (id-aware); keep over-limit links in the text so the model can still reason about them. - Keep timestamped YouTube links (?t=/&start=) in the text so the moment cue survives. - Recognize youtube-nocookie.com/embed links. * 🎚️ fix: Exclude audio Gemini variants + preserve pre-id YouTube timestamps Address Codex round 5: - Add `audio` to the url_context modality exclusion so audio-only Gemini variants (e.g. gemini-2.5-flash-preview-native-audio-dialog) skip the tool instead of 400ing. - Detect YouTube timestamps anywhere in the matched URL (incl. before `v=`, e.g. watch?t=90&v=<id>), so timestamped links are kept in the prompt text as intended.
1 participant
Summary
I added URL Context as a first-class Google/Gemini parameter, mirroring the existing search grounding wiring, and implemented a LibreChat-native enhancement: when URL Context is enabled, YouTube links in the user's message are routed to Gemini video understanding (
fileDataparts), since the Gemini URL Context tool does not support YouTube. All other URLs keep flowing through the nativeurlContexttool.url_contexthandling ingetGoogleConfig(packages/api/src/endpoints/google/llm.ts): it honorsdefaultParams/addParams/dropParamsand pushes the native{ urlContext: {} }tool, identical to howweb_searchresolves{ googleSearch: {} }.urlContexttool through the OpenAI-compatible transform when explicitly enabled, and filtered it otherwise (packages/api/src/endpoints/openai/transform.ts).url_contextacross the data-provider zod schemas and the data-schemas Mongoose conversation/preset layer, so the toggle persists and round-trips exactly likeweb_search.url_contextswitch to the Google parameter settings (conversation, preset, and agent panels) with new English i18n keys.extractYouTubeUrlsandappendYouTubeVideoParts(packages/api/src/endpoints/google/youtube.ts) and wired them into the agentsbuildMessages, gated to Google/Vertex with the resolvedurlContexttool present. YouTube links inject asmedia/fileDatavideo parts on the LLM payload only, never into the persisted message or memory copy.watch?v=<id>form, de-duplicated by video id, and capped per message. Look-alike hosts (notyoutube.com,evil-youtube.com,youtube.com.evil.com) are rejected.Notes:
@librechat/agentschange is required:@langchain/google-common's tool type already acceptsurlContext, and it passes through serialization the same waygoogleSearchdoes.url_contexttool itself also survives the OpenAI-compatible custom-endpoint path.Change Type
Testing
I exercised the new behavior with unit tests across the affected layers and verified the full build, typecheck, and lint pass.
packages/api/src/endpoints/google/youtube.spec.ts: URL extraction (watch / youtu.be / shorts / live / embed / subdomains / query params), de-duplication, capping, look-alike rejection, and the injection logic (enabled vs disabled, mixed URLs, string vs array content, dedupe against existing media parts).packages/api/src/endpoints/google/llm.spec.ts:urlContexttool added via direct option,defaultParams, andaddParams; removed when off or viadropParams; co-exists withgoogleSearch; works on Vertex; never leaks intollmConfig.packages/api/src/endpoints/openai/config.google.spec.ts:urlContextpreserved when explicitly enabled, filtered otherwise, through the OpenAI-compat transform.packages/api/src/agents/__tests__/initialize.test.ts:urlContexttreated as a native provider tool and preserved alongside LibreChatweb_search.Test Configuration:
cd packages/api && npx jest src/endpoints/google src/endpoints/openai/config.google.spec.ts src/agents/__tests__/initialize.test.ts(215 pass)cd api && npx jest server/controllers/agents/client.test.js(85 pass)npm run build:data-provider && npm run build:data-schemas && npm run build:api;tsc --noEmitand ESLint clean across all changed files.Checklist