Skip to content
This repository was archived by the owner on Sep 30, 2024. It is now read-only.

fix(Cody Gateway): add model field to Google completion request#63113

Merged
abeatrix merged 1 commit intomainfrom
bee/fix-model
Jun 5, 2024
Merged

fix(Cody Gateway): add model field to Google completion request#63113
abeatrix merged 1 commit intomainfrom
bee/fix-model

Conversation

@abeatrix
Copy link
Contributor

@abeatrix abeatrix commented Jun 5, 2024

Add the missing Model field to the googleRequest struct to include the model name in the request to the Cody Gateway completion API.

This change ensures that the model name is properly included in the request, which is necessary for Cody Gateway to function correctly.

Test plan

Verified connection to Gemini API directly works:

❯ curl -X 'POST' -d '{"messages":[{"speaker":"human","text":"Who are you?"}],"maxTokensToSample":30,"temperature":0,"stopSequences":[],"timeoutMs":5000,"stream":true}' -H 'Accept: application/json' -H 'Authorization: token $LOCAL_SG_TOKEN' -H 'Content-Type: application/json' 'https://sourcegraph.test:3443/.api/completions/stream?client-name=web'

Response:

event: completion
data: {"completion":"I","stopReason":"STOP"}

event: completion
data: {"completion":"I am a large language model, trained by Google. \n\nHere are some key","stopReason":"STOP"}

event: completion
data: {"completion":"I am a large language model, trained by Google. \n\nHere are some key things to know about me:\n\n* **I'm a","stopReason":"MAX_TOKENS"}

event: done
data: {}

Verified connection to Cody Gateway endpoint works:

❯ curl  -H  "Authorization: Bearer $CODY_GATEWAY_TOKEN" -H  "X-Sourcegraph-Feature: chat_completions"  -d '{"model":"gemini-1.5-flash-latest","contents":[{"parts":[{"text":"You are Cody"}],"role":"user"},{"parts":[{"text":"Ok I am Cody"}],"role":"model"},{"parts":[{"text":"What is your name?"}],"role":"user"}],"max_tokens":100}' https://cody-gateway.sourcegraph.com/v1/completions/google

Response:

data: {"candidates": [{"content": {"parts": [{"text": "As"}],"role": "model"},"finishReason": "STOP","index": 0}],"usageMetadata": {"promptTokenCount": 15,"candidatesTokenCount": 1,"totalTokenCount": 16}}
data: {"candidates": [{"content": {"parts": [{"text": " a large language model, I don't have a name in the traditional sense"}],"role": "model"},"finishReason": "STOP","index": 0,"safetyRatings": [{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE"}]}],"usageMetadata": {"promptTokenCount": 15,"candidatesTokenCount": 17,"totalTokenCount": 32}}

data: {"candidates": [{"content": {"parts": [{"text": ". You can call me Cody, or any other name you like!  What"}],"role": "model"},"finishReason": "STOP","index": 0,"safetyRatings": [{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE"}]}],"usageMetadata": {"promptTokenCount": 15,"candidatesTokenCount": 33,"totalTokenCount": 48}}

data: {"candidates": [{"content": {"parts": [{"text": " do you prefer? \n"}],"role": "model"},"finishReason": "STOP","index": 0,"safetyRatings": [{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE"},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE"}]}],"usageMetadata": {"promptTokenCount": 15,"candidatesTokenCount": 37,"totalTokenCount": 52}}

curl-ing the Cody Gateway endpoint returns the correct model name google/gemini-model:

curl -X 'POST' -d '{"messages":[{"speaker":"human","text":"Who are you?"}],"maxTokensToSample":30,"temperature":0,"stopSequences":[],"timeoutMs":5000,"stream":true}' -H 'Accept: application/json' -H 'Authorization: token $LOCAL_TOKEN' -H 'Content-Type: application/json' 'https://sourcegraph.test:3443/.api/completions/stream'

Response:

data: {"error":"Sourcegraph Cody Gateway: unexpected status code 400: {\"error\":\"model \\\"google/gemini-1.5-flash-latest\\\" is not allowed, allowed: []\"}\n"}

Before

curl-ing the Cody Gateway endpoint returns the following error because Model is missing, that's why model only contains the provider (google) name:

curl -X 'POST' -d '{"messages":[{"speaker":"human","text":"Who are you?"}],"maxTokensToSample":30,"temperature":0,"stopSequences":[],"timeoutMs":5000,"stream":true,"model":"google/gemini-1.5-pro-latest"}' -H 'Accept: application/json' -H 'Authorization: token $DOT_COM_TOKEN' -H 'Content-Type: application/json' 'https://sourcegraph.com/.api/completions/stream'

Response:

"Sourcegraph Cody Gateway: unexpected status code 400: {\"error\":\"model \\\"google/\\\" is not allowed, allowed: [google/gemini-1.5-pro-latest, google/gemini-1.5-flash-latest]\"}

No Changelog is required for unreleased change.

Add the `Model` field to the `googleRequest` struct to include the model name in the request to the Google completion API.

This change ensures that the model name is properly included in the request, which is necessary for Cody Gateway to function correctly.

## Changelog

- Added `Model` field to `googleRequest` struct to include the model name in the request to the Google completion API.
@cla-bot cla-bot bot added the cla-signed label Jun 5, 2024
@abeatrix abeatrix requested review from a team and chrsmith June 5, 2024 22:17
@abeatrix abeatrix merged commit c9bfc65 into main Jun 5, 2024
@abeatrix abeatrix deleted the bee/fix-model branch June 5, 2024 22:32
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants