docs: add model capabilities guide and update provider documentation with tool_use#6901

bdougie · 2025-07-30T23:27:50Z

Description

Add new model-capabilities.mdx deep dive guide explaining how to configure tool and image support
Update OpenRouter provider documentation with capability configuration examples
Update Ollama provider documentation with tool_use support details
Enhance reference documentation with clearer capability guidance
Improve troubleshooting documentation for capability-related issues

[ What changed? Feel free to be brief. ]

Checklist

I've read the contributing guide
The relevant docs, if any, have been updated or created
The relevant tests, if any, have been updated or created

Screen recording or screenshot

[ When applicable, please include a short screen recording or screenshot - this makes it much easier for us as contributors to review and understand your changes. See this PR as a good example. ]

There is also a compatibility matrix added. I did generate this, but expect to get feedback on actual compatibilities in review as this is not my expertise.

Tests

[ What tests were added or updated to ensure the changes work as expected? ]

View the new compatibility matrix - https://continue-docs-bdougie-con-2740.mintlify.app/customize/deep-dives/model-capabilities.

Summary by cubic

Added a new guide on model capabilities and updated provider documentation to help users configure tool and image support for custom and proxy models.

Docs Updates
- Added a deep dive guide explaining model capabilities and manual overrides.
- Updated OpenRouter and Ollama docs with capability configuration examples.
- Improved reference and troubleshooting docs with clearer guidance on capability issues.

- Add new model-capabilities.mdx deep dive guide - Update OpenRouter provider documentation - Update Ollama provider documentation - Update reference documentation - Update troubleshooting documentation

- Add model-capabilities.mdx to deep-dives section in docs.json

- Add comprehensive compatibility matrix for popular models - Include OpenAI, Anthropic, Google, Mistral, DeepSeek, and Ollama models - Document feature support for Chat, Apply, Edit, Tool Use, and Image Input - Add GitHub edit link for community contributions

- Remove migration section reference to non-existent page - Align table headers with left alignment for better readability - Clean up document structure by removing broken cross-reference

- Enhanced model capabilities documentation with more detailed explanations - Added model setup information across agent, autocomplete, chat, and edit features - Improved documentation structure and clarity for better user understanding

Co-authored-by: BekahHW <34313413+BekahHW@users.noreply.github.com>

BekahHW

LGTM 🚀

RomneyDa · 2025-08-04T23:05:47Z

+You can explicitly set the capabilities you want the model to use. This disables autodetection for any other capabilities. 
+
+<Note>
+ You cannot configure a model to use no capabilities.


to have no capabilities? Maybe say you cannot override autodetection

RomneyDa · 2025-08-04T23:07:57Z

+
+Continue automatically detects capabilities based on your provider and model name. For example:
+
+- **OpenAI**: GPT-4 models have tool support, GPT-3.5 doesn't


This is interesting, gpt-3.5 turbo models actually support function calling, we may want to change this support but at least this example might confuse users who know that

RomneyDa · 2025-08-04T23:09:31Z

+<Info>
+  The `tool_use` capability is for native tool/function calling support. The
+  model must actually support tools for this to work. If your model doesn't
+  support native tools, Continue will automatically use system message tools


The wording on this is a bit confusing. System message tools are currently experimental and will not be automatically used as a fallback. capabilities only applies to native tools, it does not impact system message tool use whether it is set or not. An exception is openrouter, which we did a hot fix for

added a warn callout

RomneyDa · 2025-08-04T23:15:46Z

+| GPT-4o        | ✅       | ✅          |
+| GPT-4 Turbo   | ✅       | ✅          |
+| GPT-4         | ✅       | ❌          |
+| GPT-3.5 Turbo | ❌       | ❌          |


RomneyDa · 2025-08-04T23:17:03Z

+
+### Notes
+
+- **Tool Use**: Function calling for Agent mode (required for Agent mode)


The tool_use property is not required for agent mode (tools are), wording is a bit confusing

RomneyDa · 2025-08-04T23:17:44Z

+### Notes
+
+- **Tool Use**: Function calling for Agent mode (required for Agent mode)
+- **Image Input**: Processing screenshots and images


I think "screenshots and images" is confusing because a screenshot is an image and feels oddly specific here

RomneyDa · 2025-08-04T23:21:17Z

+### Recommended Models
+
+For the best Agent mode experience, we recommend models with strong reasoning and instruction-following capabilities:
+


I might recommend against using these models with system message tools because they are trained to use native tools and very effective with them. Besides deepseek

RomneyDa · 2025-08-04T23:22:07Z

+2. Ensure your model actually supports vision (e.g., gpt-4-vision, claude-3)
+3. Check that your provider passes through image data
+
+#### Override capabilities


Maybe *add capabilities, since there's no override functionality

bdougie · 2025-08-05T18:13:04Z

Hey @BekahHW and @RomneyDa, the requested changes have been implemented in the latest commits:

Clarified model capabilities configuration and detection
Updated the compatibility matrix columns as discussed

Please let me know if there's anything else needed!

Generated with cn

RomneyDa

Looks good!

RomneyDa · 2025-08-04T23:25:56Z

 - `roles`: An array specifying the roles this model can fulfill, such as `chat`, `autocomplete`, `embed`, `rerank`, `edit`, `apply`, `summarize`. The default value is `[chat, edit, apply, summarize]`. Note that the `summarize` role is not currently used.

- `capabilities`: Array of strings denoting model capabilities, which will overwrite Continue's autodetection based on provider and model. Supported capabilities include `tool_use` and `image_input`.
+- `capabilities`: Array of strings denoting model capabilities, which will overwrite Continue's autodetection based on provider and model. Supported capabilities include:


tool_use does not overwrite autodetection, as noted above

sestinj · 2025-08-18T20:28:12Z

🎉 This PR is included in version 1.5.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

sestinj · 2025-08-18T20:34:27Z

🎉 This PR is included in version 1.7.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

docs: add model capabilities guide and update provider documentation

d2cd9b3

- Add new model-capabilities.mdx deep dive guide - Update OpenRouter provider documentation - Update Ollama provider documentation - Update reference documentation - Update troubleshooting documentation

github-project-automation bot added this to Issues and PRs Jul 30, 2025

github-project-automation bot moved this to Todo in Issues and PRs Jul 30, 2025

mintlify bot deployed to staging - docs July 30, 2025 23:28 View deployment

docs: add model-capabilities to navigation menu

ced7170

- Add model-capabilities.mdx to deep-dives section in docs.json

mintlify bot deployed to staging - docs July 30, 2025 23:35 View deployment

mintlify bot deployed to staging - docs July 30, 2025 23:42 View deployment

bdougie force-pushed the bdougie/con-2740 branch from 6008d3c to 2de4a4c Compare July 30, 2025 23:43

mintlify bot deployed to staging - docs July 30, 2025 23:43 View deployment

bdougie changed the title ~~docs: add model capabilities guide and update provider documentation~~ docs: add model capabilities guide and update provider documentation with tool_use Jul 31, 2025

docs: refactor troubleshooting content for model capabilities

53cc34d

mintlify bot deployed to staging - docs July 31, 2025 00:10 View deployment

bdougie commented Jul 31, 2025

View reviewed changes

Comment thread docs/customize/deep-dives/model-capabilities.mdx Outdated

bdougie commented Jul 31, 2025

View reviewed changes

Comment thread docs/customize/model-providers/more/openrouter.mdx

docs: improve model capabilities guide formatting and structure

5572f29

- Remove migration section reference to non-existent page - Align table headers with left alignment for better readability - Clean up document structure by removing broken cross-reference

mintlify bot deployed to staging - docs July 31, 2025 00:54 View deployment

fix: remove css fix

d781605

mintlify bot deployed to staging - docs July 31, 2025 01:00 View deployment

mintlify bot deployed to staging - docs July 31, 2025 19:05 View deployment

bdougie force-pushed the bdougie/con-2740 branch from 9a75f31 to 9c3bcc5 Compare July 31, 2025 19:06

mintlify bot deployed to staging - docs July 31, 2025 19:06 View deployment

bdougie force-pushed the bdougie/con-2740 branch from 9c3bcc5 to 5f7b99e Compare July 31, 2025 19:19

mintlify bot deployed to staging - docs July 31, 2025 19:20 View deployment

bdougie force-pushed the bdougie/con-2740 branch from 5f7b99e to a23cd28 Compare July 31, 2025 19:22

mintlify bot deployed to staging - docs July 31, 2025 19:23 View deployment

bdougie marked this pull request as ready for review July 31, 2025 21:31

bdougie requested a review from a team as a code owner July 31, 2025 21:31

bdougie removed the request for review from a team July 31, 2025 21:31

mintlify bot deployed to staging - docs August 1, 2025 18:18 View deployment

BekahHW reviewed Aug 4, 2025

View reviewed changes

Comment thread docs/customize/model-providers/top-level/ollama.mdx Outdated

BekahHW reviewed Aug 4, 2025

View reviewed changes

Comment thread docs/customize/deep-dives/model-capabilities.mdx Outdated

BekahHW reviewed Aug 4, 2025

View reviewed changes

Comment thread docs/customize/deep-dives/model-capabilities.mdx

BekahHW reviewed Aug 4, 2025

View reviewed changes

Comment thread docs/customize/deep-dives/model-capabilities.mdx Outdated

bdougie and others added 3 commits August 4, 2025 15:36

Update docs/customize/deep-dives/model-capabilities.mdx

0adbf9c

Co-authored-by: BekahHW <34313413+BekahHW@users.noreply.github.com>

Update docs/customize/deep-dives/model-capabilities.mdx

f778c40

Co-authored-by: BekahHW <34313413+BekahHW@users.noreply.github.com>

Update docs/customize/model-providers/top-level/ollama.mdx

d21cca6

Co-authored-by: BekahHW <34313413+BekahHW@users.noreply.github.com>

mintlify bot deployed to staging - docs August 4, 2025 22:37 View deployment

fix: remove note

ed19565

mintlify bot deployed to staging - docs August 4, 2025 22:47 View deployment

BekahHW approved these changes Aug 4, 2025

View reviewed changes

bdougie requested a review from RomneyDa August 4, 2025 22:57

RomneyDa reviewed Aug 4, 2025

View reviewed changes

bdougie added 2 commits August 5, 2025 10:02

docs: clarify model capabilities configuration and detection

cd27c8a

fix: update columns

3280391

mintlify bot deployed to staging - docs August 5, 2025 17:03 View deployment

RomneyDa approved these changes Aug 5, 2025

View reviewed changes

github-project-automation bot moved this from Todo to In Progress in Issues and PRs Aug 5, 2025

dosubot bot added the lgtm This PR has been approved by a maintainer label Aug 5, 2025

bdougie merged commit 33fa558 into main Aug 5, 2025
38 checks passed

bdougie deleted the bdougie/con-2740 branch August 5, 2025 18:16

github-project-automation bot moved this from In Progress to Done in Issues and PRs Aug 5, 2025

github-actions bot locked and limited conversation to collaborators Aug 5, 2025

sestinj added the released label Aug 18, 2025


		Continue automatically detects capabilities based on your provider and model name. For example:

		- OpenAI: GPT-4 models have tool support, GPT-3.5 doesn't


		### Notes

		- Tool Use: Function calling for Agent mode (required for Agent mode)

		### Recommended Models

		For the best Agent mode experience, we recommend models with strong reasoning and instruction-following capabilities:

Conversation

bdougie commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Screen recording or screenshot

Tests

Summary by cubic

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BekahHW left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RomneyDa Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RomneyDa Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bdougie commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RomneyDa left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sestinj commented Aug 18, 2025

Uh oh!

sestinj commented Aug 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bdougie commented Jul 30, 2025 •

edited

Loading

RomneyDa Aug 4, 2025 •

edited

Loading

RomneyDa Aug 4, 2025 •

edited

Loading

bdougie commented Aug 5, 2025 •

edited

Loading