🖼️ fix: Correct ToolMessage Response Format for Agent-Mode Image Tools#12310
Merged
Conversation
…leDiffusion classes - Added logic to set `responseFormat` to 'content_and_artifact' when `isAgent` is true in DALLE3.js, FluxAPI.js, and StableDiffusion.js.
…ent.spec.js - Introduced a new test suite for DALLE3, FluxAPI, and StableDiffusion classes to verify that the invoke() method returns a ToolMessage with base64 in artifact.content, ensuring it is not serialized into content. - Validated that responseFormat is set to 'content_and_artifact' when isAgent is true, and confirmed the correct handling of base64 data in the response.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an agent-mode regression guard for image generation tools to ensure LangChain Tool.invoke() returns a ToolMessage with image base64 placed in artifact.content (instead of being JSON-stringified into content, which bloats token counting / context).
Changes:
- Set
responseFormat = 'content_and_artifact'for DALLE3 / FluxAPI / StableDiffusion tools whenisAgentis true. - Add Jest regression tests validating
invoke()produces aToolMessagewhere base64 only appears in the artifact payload.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| api/app/clients/tools/structured/specs/imageTools-agent.spec.js | Adds regression tests ensuring agent-mode invoke() keeps base64 out of ToolMessage.content and in artifact.content. |
| api/app/clients/tools/structured/StableDiffusion.js | Sets responseFormat for agent-mode Stable Diffusion tool invocations. |
| api/app/clients/tools/structured/FluxAPI.js | Sets responseFormat for agent-mode Flux tool invocations. |
| api/app/clients/tools/structured/DALLE3.js | Sets responseFormat for agent-mode DALL·E 3 tool invocations. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
+52
to
+58
| describe('image tools - agent mode ToolMessage format', () => { | ||
| beforeEach(() => { | ||
| jest.clearAllMocks(); | ||
| process.env.DALLE_API_KEY = 'test-key'; | ||
| process.env.FLUX_API_KEY = 'test-key'; | ||
| process.env.SD_WEBUI_URL = 'http://localhost:7860'; | ||
| }); |
- StableDiffusion._call() was returning a raw string on API error, bypassing returnValue() and breaking the content_and_artifact contract when isAgent is true - FluxAPI.generateFinetunedImage() had no isAgent branch; it would call processFileURL (unset in agent context) instead of fetching and returning the base64 image as an artifact tuple - Add JSDoc to all three responseFormat assignments clarifying why LangChain requires this property for correct ToolMessage construction
- Add env var save/restore in beforeEach/afterEach to prevent test pollution - Add error path tests for all three tools verifying ToolMessage content and artifact are correctly populated when the upstream API fails - Add generate_finetuned action test for FluxAPI covering the new agent branch in generateFinetunedImage
danny-avila
changed the title
jcbartle
pushed a commit
to jcbartle/LibreChat
that referenced
this pull request
May 11, 2026
danny-avila#12310) * fix: Set response format for agent tools in DALLE3, FluxAPI, and StableDiffusion classes - Added logic to set `responseFormat` to 'content_and_artifact' when `isAgent` is true in DALLE3.js, FluxAPI.js, and StableDiffusion.js. * test: Add regression tests for image tool agent mode in imageTools-agent.spec.js - Introduced a new test suite for DALLE3, FluxAPI, and StableDiffusion classes to verify that the invoke() method returns a ToolMessage with base64 in artifact.content, ensuring it is not serialized into content. - Validated that responseFormat is set to 'content_and_artifact' when isAgent is true, and confirmed the correct handling of base64 data in the response. * fix: handle agent error paths and generateFinetunedImage in image tools - StableDiffusion._call() was returning a raw string on API error, bypassing returnValue() and breaking the content_and_artifact contract when isAgent is true - FluxAPI.generateFinetunedImage() had no isAgent branch; it would call processFileURL (unset in agent context) instead of fetching and returning the base64 image as an artifact tuple - Add JSDoc to all three responseFormat assignments clarifying why LangChain requires this property for correct ToolMessage construction * test: expand image tool agent mode regression suite - Add env var save/restore in beforeEach/afterEach to prevent test pollution - Add error path tests for all three tools verifying ToolMessage content and artifact are correctly populated when the upstream API fails - Add generate_finetuned action test for FluxAPI covering the new agent branch in generateFinetunedImage * chore: fix lint errors in FluxAPI and imageTools-agent spec * chore: fix import ordering in imageTools-agent spec
ThomasVuNguyen
pushed a commit
to ThomasVuNguyen/LibreChat
that referenced
this pull request
Jul 15, 2026
danny-avila#12310) * fix: Set response format for agent tools in DALLE3, FluxAPI, and StableDiffusion classes - Added logic to set `responseFormat` to 'content_and_artifact' when `isAgent` is true in DALLE3.js, FluxAPI.js, and StableDiffusion.js. * test: Add regression tests for image tool agent mode in imageTools-agent.spec.js - Introduced a new test suite for DALLE3, FluxAPI, and StableDiffusion classes to verify that the invoke() method returns a ToolMessage with base64 in artifact.content, ensuring it is not serialized into content. - Validated that responseFormat is set to 'content_and_artifact' when isAgent is true, and confirmed the correct handling of base64 data in the response. * fix: handle agent error paths and generateFinetunedImage in image tools - StableDiffusion._call() was returning a raw string on API error, bypassing returnValue() and breaking the content_and_artifact contract when isAgent is true - FluxAPI.generateFinetunedImage() had no isAgent branch; it would call processFileURL (unset in agent context) instead of fetching and returning the base64 image as an artifact tuple - Add JSDoc to all three responseFormat assignments clarifying why LangChain requires this property for correct ToolMessage construction * test: expand image tool agent mode regression suite - Add env var save/restore in beforeEach/afterEach to prevent test pollution - Add error path tests for all three tools verifying ToolMessage content and artifact are correctly populated when the upstream API fails - Add generate_finetuned action test for FluxAPI covering the new agent branch in generateFinetunedImage * chore: fix lint errors in FluxAPI and imageTools-agent spec * chore: fix import ordering in imageTools-agent spec
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixed a context-exhaustion bug where DALL-E 3, FluxAPI, and Stable Diffusion image tools, when invoked in agent mode, were serializing the full base64 image payload directly into
ToolMessage.contentinstead of routing it toartifact.content.this.responseFormat = 'content_and_artifact'in the constructors ofDALLE3,FluxAPI, andStableDiffusionAPIwhenisAgentistrue, so LangChain'sTool.invoke()correctly maps the[content, artifact]tuple returned by_call()onto the respectiveToolMessagefields rather thanJSON.stringify-ing the entire tuple intocontent.responseFormatassignment explaining why LangChain requires the property for correctToolMessageconstruction.FluxAPI.generateFinetunedImage(), which had noisAgentbranch and would unconditionally callthis.processFileURL— a method that is never set in agent context — causing a silentTypeErrorand a failed generation with no image returned.StableDiffusionAPI._call()error path, which was returning a raw string onaxiosfailure, bypassingreturnValue()and breaking thecontent_and_artifacttuple contract whenisAgentistrue.imageTools-agent.spec.jswith 13 regression tests coveringresponseFormatassignment, happy-pathinvoke()for all three tools (including thegenerate_finetunedFluxAPI action), and error-pathinvoke()for all three tools, verifying theToolMessagestructure in every case.Change Type
Testing
Verified via the new
api/app/clients/tools/structured/specs/imageTools-agent.spec.jsregression suite. Run from theapiworkspace:Test Configuration:
axios,node-fetch, andundiciare all mocked.responseFormatset/not-set on construction,invoke()returning aToolMessagewith base64 confined toartifact.contenton success,invoke()returning a well-formedToolMessagewith an error string incontenton upstream API failure, and thegenerate_finetunedFluxAPI action path in agent mode.Checklist