GitLab AI Gateway

Configure the GitLab AI Gateway to run locally in GDK.

Prerequisites

• Access to Google Cloud.
• Access to Anthropic API.

Set up the AI Gateway

1. Run setup_ai_development from your GDK root directory:

   gdk rake setup_ai_development

   This Rake task sets up a complete AI development environment by executing the following Rake tasks:

   • setup_ai_services: Sets up AI services (the GitLab AI Gateway and GitLab Duo Workflow Service).

   • setup_duo_project: Creates an Ultimate group and project with GitLab Duo.

   • onboard_dap: Onboards the GitLab Duo Agent Platform.

   • setup_runner: Configures GitLab Runner.

2. Respond to the prompts:

   1. Enter your Anthropic API key: The key used to authenticate requests to Anthropic and is essential for accessing their API and using their AI models.
   2. Enter your Fireworks API key: The key used to authenticate requests to the Fireworks API services. This key is available in 1Password, in the Engineering folder, under Fireworks development key.
   3. Set additional environment variables for debugging: Set additional environment variables that provide more detailed logs and information for troubleshooting and development.
   4. Enable hot reload: Enables hot reloading, which allows the application to update in real-time as you make code changes, without requiring a full restart.

   Note

   If you update steps or documentation for setting up AI Gateway, check and update this Rake task as well if necessary.

   To check if your monolith is using the correct URL after restarting, run bundle exec rake cache:clear and then visit http://gdk.test:3000/help/instance_configuration#ai_gateway_url.

   Note

   When you access the AI Gateway URL directly, you’ll see a {"error":"No authorization header presented"} error message. This is expected and doesn’t affect the usage of AI features locally in GDK. You can bypass authentication by modifying the AI Gateway’s environment configuration, but this should only be done for using the OpenAPI playground. Make sure to revert any authentication bypass changes before pushing to production.

3. Set the DEVELOPMENT_AI_GATEWAY_URLand FETCH_MODEL_SELECTION_DATA_FROM_LOCAL environment variables in your <GDK-root>/gdk.yml file. Replace http://gdk.test with your custom hostname.

   env:
     DEVELOPMENT_AI_GATEWAY_URL: "http://gdk.test:5052"
     FETCH_MODEL_SELECTION_DATA_FROM_LOCAL: "1"

4. Go to the AI Gateway OpenAPI playground to verify that your local AI Gateway started successfully.

Additional AI Gateway troubleshooting and configuration

Change the AI Gateway URL

You must tell your local GitLab instance to talk to your local AI Gateway. Otherwise, your instance tries to talk to the production AI Gateway at cloud.gitlab.com, which results in an A1001 error. By default, the AI Gateway is accessible at gdk.test:5052/docs.

You can host the AI Gateway at a different URL by updating the following values in the application settings file:

# <GDK-root>/gitlab-ai-gateway/.env

AIGW_FASTAPI__API_HOST=0.0.0.0
AIGW_FASTAPI__API_PORT=5052

To check if your setup is using the correct AI Gateway, go to http://gdk.test:3000/help/instance_configuration#ai_gateway_url. This value is cached so you might need to run bundle exec rake cache:clear to see the latest value.

Set up Google Cloud Platform in AI Gateway

If you do not set up Google Cloud Platform (GCP) correctly, you might not be able to boot AI Gateway because AI Gateway checks the GCP credentials and access at boot time.

Prerequisites:

• If you haven’t signed up for GCP before using your GitLab account, follow these instructions.
• The gcloud CLI. If you’re using mise for runtime version management, gcloud should already be installed automatically.
• Run gcloud auth application-default login --disable-quota-project.

To set up GCP, you can do either of the following:

• Use the existing project.
• Create a sandbox project.

Use the existing project

You can use the existing ai-enablement-dev-69497ba7 Google Cloud project.

You should use this project:

• Because it has Vertex APIs and Vertex AI Search already enabled.
• If you are a GitLab team member. Members from the Engineering and Product divisions should already have access to this project.

Even though you will have your own GCP project when you sign up on GCP, for AI features, you can use the existing shared project ai-enablement-dev-69497ba7. To check if you have access to this existing project:

1. Go to the Google Cloud console.
2. Toward the left of the top navigation bar, select the project picker and then select ALL. If you have access to the existing project, you should see ai-enablement-dev-69497ba7 in the list.

If you do not have access, complete the GCP access request template.

Create a sandbox Google Cloud project

To create a sandbox Google Cloud project:

1. Authenticate locally with Google Cloud using gcloud auth application-default login.

2. Update the application settings file in AI Gateway:

   # <GDK-root>/gitlab-ai-gateway/.env
   
   # PROJECT_ID = "ai-enablement-dev-69497ba7" for GitLab team members with access
   # to the shared project. This should be set by default
   
   # PROJECT_ID = "your-google-cloud-project-name" for those with their own sandbox
   # Google Cloud project.
   
   AIGW_GOOGLE_CLOUD_PLATFORM__PROJECT='PROJECT_ID'

3. Create a sandbox project.

If you are using an individual Google Cloud project, because some of the GitLab Duo features use the Vertex AI API, you might also have to enable the Vertex AI API:

1. Go to the Google Cloud welcome page.
2. Choose your project (for example: jdoe-5d23dpe).
3. Select APIs & Services > Enabled APIs & services.
4. Select Enable APIs and Services.
5. Search for Vertex AI API.
6. Select Vertex AI API, then select Enable.
7. Authenticate locally with Google Cloud:
   • Use the gcloud auth application-default login command if your Application Default Credentials (ADC) account has the serviceusage.services.use permission on the quota project from GCloud’s context.
   • Use the gcloud auth application-default login --disable-quota-project command if you:
   • Do not have a project with the serviceusage.services.use permission.
   • Want to always bill the project owning the resources.

Set up Anthropic in the AI Gateway

You must set up Anthropic because some GitLab Duo features use Anthropic models.

1. Complete an access request.

2. Sign up for an Anthropic account and create an API key.

3. Update the application settings file in AI Gateway:

   # <GDK-root>/gitlab-ai-gateway/.env
   
   ANTHROPIC_API_KEY='<your-anthropic-api-key>'

Optional: Enable logging in AI Gateway

Logging makes it easier to debug any issues with GitLab Duo requests.

To enable logging, update the application settings file in AI Gateway:

# <GDK-root>/gitlab-ai-gateway/.env

AIGW_LOGGING__LEVEL=debug
AIGW_LOGGING__FORMAT_JSON=false
AIGW_LOGGING__TO_FILE='./ai-gateway.log'

For example, you can watch the log file with the following command when in the gitlab-ai-gateway directory:

# <GDK-root>/gitlab-ai-gateway

tail -f ai-gateway.log | fblog -a prefix -a suffix -a current_file_name -a suggestion -a language -a input -a parameters -a score -a exception

Optional: Run a different branch of AI Gateway and Duo Agent Platform Service

The AI Gateway repository is cloned into <gdk-dir>/gitlab-ai-gateway.

To configure GDK to run a specific branch, use either the branch name or SHA:

gdk config set gitlab_ai_gateway.version <branch-name-or-SHA>
gdk reconfigure

Error: Activated Python version 3.XX.X is not supported

The Rake task calls a makefile, Make.gitlab-ai-gateway.mk, that installs dependencies defined in the AI Gateway tool version file. The makefile will install Python version 3.10, but the system may return an error message if a different version of Python has been installed from a separate source:

The currently activated Python version 3.XX.X is not supported by the project (~3.10.0).

To resolve this issue, you should install the following dependency versions with mise:

mise install python 3.10.14