🧵 feat: Enable Anthropic Tool Argument Streaming#12962
Conversation
danny-avila
changed the title
|
@codex review |
There was a problem hiding this comment.
Pull request overview
Enables Anthropic fine-grained tool argument streaming by ensuring the fine-grained-tool-streaming-2025-05-14 beta value is present in the outgoing anthropic-beta header while preserving any existing beta header values (e.g., prompt-cache related betas).
Changes:
- Added
appendAnthropicBetaHeader()+FINE_GRAINED_TOOL_STREAMING_BETAhelper exports for composinganthropic-betavalues. - Updated Anthropic
getLLMConfig()to always append the fine-grained tool streaming beta intoclientOptions.defaultHeaders. - Updated Anthropic endpoint unit tests to assert the new default/combined
anthropic-betaheader behavior.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| packages/api/src/endpoints/anthropic/llm.ts | Appends the fine-grained tool streaming beta header into Anthropic client default headers. |
| packages/api/src/endpoints/anthropic/llm.spec.ts | Updates assertions to expect the new default and combined anthropic-beta header values. |
| packages/api/src/endpoints/anthropic/helpers.ts | Introduces shared helper + constant for safely composing anthropic-beta header values. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bbcb2583de
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
GitNexus: 🚀 deployedThe |
|
@codex review |
1 similar comment
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ab427791f3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
GitNexus: 🚀 deployedThe |
|
@codex review |
|
Codex Review: Didn't find any major issues. Hooray! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
GitNexus: 🚀 deployedThe |
|
@codex review |
|
Codex Review: Didn't find any major issues. Bravo. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
GitNexus: 🚀 deployedThe |
* fix: Enable Anthropic Tool Argument Streaming * fix: Honor Anthropic clientOptions drops * fix: Preserve custom Anthropic beta headers * fix: Enable Bedrock Anthropic Tool Streaming
* fix: Enable Anthropic Tool Argument Streaming * fix: Honor Anthropic clientOptions drops * fix: Preserve custom Anthropic beta headers * fix: Enable Bedrock Anthropic Tool Streaming
Upstream v0.8.6 (danny-avila#12962) auto-applies the `fine-grained-tool-streaming-2025-05-14` anthropic-beta to all Claude 4+ Bedrock requests. With this header set, the @librechat/agents handler fails to reassemble the streamed JSON-arg chunks into proper `tool_use` content blocks for some turns — the raw <function_calls> XML lands in a plain `text` block in the saved message, and the frontend (correctly) renders it as text instead of as a tool-call widget. Observed on cBioDBAgentBeta (us.anthropic.claude-sonnet-4-5-20250929-v1:0) within minutes of deploying v0.8.6-rc1-custom-v1; tool_call blocks were the reliable pattern on v0.8.3-rc1-custom-v8 (~10k of them in MongoDB, two-week sample showed ~190 tool_call blocks vs only 1 XML-in-text). Workaround: gate the auto-applied beta behind an env var (default off). Set BEDROCK_FINE_GRAINED_TOOL_STREAMING=true to re-enable once the agents-SDK reassembly is fixed upstream.
* fix: Enable Anthropic Tool Argument Streaming * fix: Honor Anthropic clientOptions drops * fix: Preserve custom Anthropic beta headers * fix: Enable Bedrock Anthropic Tool Streaming
Summary
I enabled Anthropic fine-grained tool argument streaming by appending the
fine-grained-tool-streaming-2025-05-14beta header to Anthropic client headers.clientOptions.defaultHeaders.Change Type
Testing
I validated the behavior with a live Anthropic stream probe for a long single tool argument and confirmed the header causes
input_json_delta.partial_jsonchunks to stream instead of emitting only an empty delta beforemax_tokens.I also ran the focused Anthropic endpoint test suite:
NODE_PATH=/Users/danny/Projects/LibreChat/node_modules /Users/danny/Projects/LibreChat/node_modules/.bin/jest --config packages/api/jest.config.mjs packages/api/src/endpoints/anthropic/llm.spec.ts --runInBandTest Configuration:
claude-haiku-4-5-20251001packages/api/src/endpoints/anthropic/llm.spec.tsChecklist