Update the default Sourcegraph-supplied LLM models#64281
Conversation
Just FYI, there is also some silly business going on in the clients where we leverage percentage-user-feature-flags to override 'fireworks/starcoder' to 'deepseek V2'... I haven't traced that code path, but something we'll need to figure out. I think it only applies to dotcom, but not 100% positive. |
aramaraju
left a comment
aramaraju
left a comment
There was a problem hiding this comment.
A few small nits that popped up to my eyes
| newModel( | ||
| modelIdentity{ | ||
| MRef: mRef(fireworksV1, "starcoder"), | ||
| // NOTE: THis model name is virutalized. |
There was a problem hiding this comment.
| // NOTE: THis model name is virutalized. | |
| // NOTE: This model name is virtualized. |
cmd/cody-gateway-config/main.go
Outdated
| // | ||
| // See internal/version/version.go for reference. | ||
| Revision: "0.0.0+dev", | ||
| Revision: "0.0.`0+dev", |
There was a problem hiding this comment.
That's a typo, good catch 😅
@slimsag not sure about the specific, as there are several places where we could "virtualize" the model name. And/or pick something different via feature flag. But this is what first comes to mind: |
| { | ||
| "schemaVersion": "1.0", | ||
| "revision": "0.0.0+dev", | ||
| "revision": "0.0.`0+dev", |
|
@chrsmith here's a thread to pull on: https://sourcegraph.com/github.com/sourcegraph/cody/-/blob/vscode/src/completions/providers/create-provider.ts?L194-240 (again, no idea what is going on there specifically) |
| Chat: types.ModelRef("anthropic::2023-06-01::claude-3-sonnet"), | ||
| CodeCompletion: types.ModelRef("anthropic::2023-06-01::claude-3-sonnet"), | ||
| FastChat: types.ModelRef("anthropic::2023-06-01::claude-3-sonnet"), | ||
| Chat: types.ModelRef("anthropic::2023-06-01::claude-3.5-sonnet"), |
There was a problem hiding this comment.
I see Claude 3.5 Sonnet called anthropic/claude-3-5-sonnet-20240620, according to Gateway telemetry. Flagging in case it's worth making these names consistent (3-5 vs 3.5)
There was a problem hiding this comment.
That's a good concern to call out, but the difference is 100% A-OK in this case.
Here we are using the Model ID claude-3.5-sonnet. This will refer to an object with a Model Name of claude-3-5-sonnet-20240620. One of the benefits of using this whole "model config" system is that we are no longer conflating the two. And can treat them as separate values.
So claude-3.5-sonnet can later refer to claude-3-5-sonnet-20240805 without needing to make as many changes across the codebase.
chenkc805
left a comment
chenkc805
left a comment
There was a problem hiding this comment.
Re: Cody Free/Pro using a super-shady hack:
When will we address this? Because my understanding of the server-side model selection is that it'll be easier to set up new models, so easy in fact that even a PM can do it.
If these experiences bifurcate like what's being described here, will we have achieved this goal? If not, do we have a follow up planned?
@chenkc805 I will update the tickets in Linear this week to call out the sequence of steps needed so we can make Cody Pro "just work" with the new model config system. (It's not a lot of work, but we do need to be careful when we roll out the changes to avoid any disruptions.) |
|
@aramaraju PTAL |
Looks good to me, Chris! Thank you 🙇 |
5 participants
This PR sets the defaults for "Sourcegraph supplied LLM models".
When will these "defaults" be used?
These models will only be used IFF the Sourcegraph instance is explicitly using the newer "modelConfiguration" site configuration data. (And opts into using Sourcegraph-supplied LLM models.)
If the Sourcegraph instance is using the older "completions" configuration blob, then only the user-supplied models will be used. (Or, based on the specific defaults defined in the code for the completions provider.)
What about Cody Free or Cody Pro?
😬 yeah, we're going to need to deal with that later. Currently Sourcegraph.com is not using the newer "modelConfiguration" site configuration, and instead we have some hacks in the code to ignore the internal modelconfig. See this "super-shady hack":
https://github.com/sourcegraph/sourcegraph/blob/e5178a6bc0e0f2f5208f9dedb0c226661c881d98/cmd/frontend/internal/httpapi/completions/get_model.go#L425-L455
So we are just erring on the side of having Cody Free / Cody Pro "do whatever they do now", and this PR won't have any impact on that.
We do want Sourcegraph.com to only return this data, but there are a few things we need to get straightened out first. (e.g. Cody Gateway being ware of mrefs, and having Cody Clients no longer using
dotcom.tsto hard-code Cody Pro LLM models.)What does this PR actually do?
cmd/cody-gateway-configso that it will produce a new "supported-models.json" file.internal/modelconfig/embedded/models.json.For any Sourcegraph releases after this PR gets merged, the "Sourcegraph supplied LLM models" will be the newer set defined in
models.json. (i.e. having these new defaults, and including "fireworks::v1::starcoder".)Test plan
I tested things locally, and unfortunately it doesn't look like any clients are filtering based on the model capabilities. So "StarCoder" is showing up in the Cody Web UI, despite failing at runtime.Update: This was a problem on my end. This isn't an issue.
Changelog
NA?