As organizations expand their cloud strategy, they often manage workloads across multiple Kubernetes clusters, sometimes spanning different cloud providers. Migrating workloads between clouds — especially to Google Kubernetes Engine (GKE) — requires careful planning to ensure seamless networking, security, and service availability.

This article explores how Istio Service Mesh can be leveraged to connect two Kubernetes clusters — one running in another cloud provider and the other in Google Cloud (GKE).

We will cover the following:

• Challenges of Multi-Cloud Networking — Managing inter-cluster traffic, security, service discovery, and latency.
• Why Istio? — How Istio facilitates secure, observable, and controlled traffic routing between clusters.
• Setting Up Istio Across Multi-Cloud Clusters — Deploying Istio to enable cross-cluster networking between an external cloud and GKE.
• Service-to-Service Communication Between Clouds — Enabling seamless connectivity between workloads across both clusters.
• Live Migration to GKE — gradually migrate services from another cloud provider to Google Cloud without downtime.
• Testing & Validating Migration — Ensuring smooth traffic flow and validating the migration success.

By the end of this guide, you’ll have a multi-cloud Istio service mesh that connects an external cloud provider to Google Kubernetes Engine (GKE). More importantly, you’ll learn how to gradually shift traffic and migrate workloads to GKE without service disruption.

Before getting started with the setup, Istio offers the following types of connectivity setups:

• Install Multi-Primary on same network
• Install Primary-Remote on same network
• Install Multi-Primary on different networks
• Install Primary-Remote on different networks

For our use-case we shall be using multi primary setup on different networks, as there are two completely different cloud and networks in place.

To demonstrate this setup, I’ll be using EKS as the remote cluster, from where the workloads will be migrated to the GKE. Supposedly can be done for Azure as well!

Here’s the command used to create the AWS EKS Cluster:

eksctl create cluster --name eks-cluster --region ap-south-1 --nodegroup-name nodegroup-1 --node-type t3.medium --nodes 3 --nodes-min 1 --nodes-max 5 --managed

3 node EKS cluster up and running

For the GKE cluster to be up and running, I’ve used the following gcloud command to spin up a 3 node cluster:

gcloud container  clusters create "gke-cluster" --zone "asia-south1-a" --tier "standard"  --machine-type "n2-standard-2" --disk-type "pd-balanced" --disk-size "50"

3 node GKE Cluster Up and Running

We already have workloads running in EKS which are required to be replicated in GKE, and then eventually the traffic shall be shifted completely to GKE.

Here’s the overview of the sample application which happens to be a Animal Album.

This happens to be a very simple application with the following workload services:

• ui-service: Built with React, responsible for the user interface.
• data-service: Built with FastAPI, responsible for retrieving data from the database.
• image-service: Built with FastAPI, responsible for retrieving images from the object storage.
• mongo: NoSQL database to store the metadata of the objects.
• minio: Object storage to store images of the animals.

As per the current setup, the complete application is served from EKS. once we connect both the cluster, we’ll start to migrate the workloads.

Deployed workloads on EKS

Preparing the clusters

Configuring the cluster include assigning the cluster with their unique name for cluster and network, and setting up east-west gateway which will act as the entry point of traffic from/to the respective cluster’s traffic.

🛠️Configuring Istio in EKS Cluster

We’ll start the preparation for migration from EKS, by configuring istio to consider EKS cluster as cluster1 in network1 as part of the mesh1. I’ll be using istioctl command line tool to configure the clusters, the YAML for the same here:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster1
      network: network1

# Command to install istio control plane component (istiod & ingressgateway)
istioctl install --context="${CTX_CLUSTER1}" -f cluster1.yaml

🌐Installing East-West Gateway in EKS

To enable cross-cluster communication, install the east-west gateway:

# Have used the istio scripts that come as part of installing istioctl
cd istio-<version>
samples/multicluster/gen-eastwest-gateway.sh \
    --network network1 | \
    istioctl --context="${CTX_CLUSTER1}" install -y -f -

📡Exposing the services in EKS cluster

Since the clusters are on separate networks, we need to expose all services (*.local) on the east-west gateway in both clusters.

While this gateway is public on the Internet, services behind it can only be accessed by services with a trusted mTLS certificate and workload ID, just as if they were on the same network.

kubectl --context="${CTX_CLUSTER1}" apply -n istio-system -f \
    samples/multicluster/expose-services.yaml

🔐 Building Trust Between GKE and EKS Istio Meshes

With Istio successfully configured in our EKS cluster, we now move on to setting it up in the GKE cluster. But before diving in, there’s one important prerequisite: establishing mutual trust between the two service meshes.

🔑 Sharing the CA (Certificate Authority) Between Clusters

To enable secure cross-cluster communication and DNS-based service discovery, both clusters must trust the same root certificate authority (CA). While a production setup should use a secure and automated method for certificate management, for the purpose of this demo, we’ll manually reuse the CA secrets from the EKS cluster in the GKE cluster.

📥 Step 1: Export CA Secret from EKS

# Getting ca secrets from EKS cluster to copy to GKE
kubectl get secrets --context="${CTX_CLUSTER1}"  -n istio-system \
istio-ca-secret -oyaml > istio-ca-secrets.yaml

⚠️ Important Note

Copying Kubernetes Service Account (SA) secrets or CA secrets across clusters is not recommended in production environments. This approach is acceptable only in experimental or demo scenarios.

For production use cases, it is highly recommended to use cert-manager or Istio’s integration with external CAs to securely and automatically manage certificates across clusters.

🛠️ Configuring Istio in the GKE Cluster

Now, let’s set up Istio in GKE. First, we’ll pre-create the namespace and apply the copied CA secret:

kubectl create ns istio-system --context $CTX_CLUSTER2
kubectl apply -f istio-ca-secrets.yaml --context $CTX_CLUSTER2

Then configure Istio with a shared mesh ID, and unique cluster/network identifiers:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster2
      network: network2

Install Istio in the GKE cluster:

istioctl install --context="${CTX_CLUSTER2}" -f cluster2.yaml

🌐Installing East-West Gateway in GKE

To enable cross-cluster communication, install the east-west gateway:

samples/multicluster/gen-eastwest-gateway.sh \
    --network network2 | \
    istioctl --context="${CTX_CLUSTER2}" install --set components.cni.enabled=false -y -f -

📡Exposing the services in GKE cluster

kubectl --context="${CTX_CLUSTER2}" apply -n istio-system -f \
    samples/multicluster/expose-services.yaml

🔐Enable Endpoint Discovery

Install a remote secret in cluster2 that provides access to cluster1’s API server.

istioctl create-remote-secret \
  --context="${CTX_CLUSTER1}" \
  --name=cluster1 | \
  kubectl apply -f - --context="${CTX_CLUSTER2}"

Install a remote secret in cluster1 that provides access to cluster2’s API server.

istioctl create-remote-secret \
  --context="${CTX_CLUSTER2}" \
  --name=cluster2 | \
  kubectl apply -f - --context="${CTX_CLUSTER1}"

✅ Verifying the Multicluster Istio Setup

To validate that Istiod can successfully communicate with the Kubernetes control plane of the remote cluster:

For EKS run:

istioctl remote-clusters --context="${CTX_CLUSTER1}"

For GKE run:

istioctl remote-clusters --context="${CTX_CLUSTER2}"

Both clusters are connected using the east-west gateway, and this connectivity has been verified via the previously shared snapshots.

Next, we deploy the image-service component in the GKE cluster. Once deployed, we access it via the web UI.

# Creating and labeling namespace
kubectl create ns animal-album --context $CTX_CLUSTER2
kubectl label ns animal-album istio-injection=enabled --context $CTX_CLUSTER2

# Deploying the image service
kubectl apply -k image-service/ --context $CTX_CLUSTER2

Here’s the web-ui of the sample application that I’ve used for the demo purpose.

📈 Observing Cross-Cluster Traffic

If the configuration is correct (and it absolutely is 😉), we’ll observe traffic reaching the image-service pods running in the GKE cluster.

As expected, the traffic is flowing seamlessly across both image-service workloads deployed in GKE and EKS clusters — confirming that our multi-cluster Istio setup is working flawlessly.

🎉 Success!

Now, we can scale down the image-service in the EKS cluster completely and eventually migrating all the

With this setup in place, we can now migrate and run any application — stateless or stateful — between clusters with zero downtime. This significantly boosts our flexibility and resilience as we shift our workloads toward Google Kubernetes Engine. From unified traffic management to robust observability and secure service-to-service communication, Istio continues to prove its value in modern, distributed cloud-native applications.

By integrating Istio with Google Kubernetes Engine (GKE), we’ve unlocked the ability to manage service-to-service communication with fine-grained control, built-in security, and powerful observability — all while spanning multiple clusters across cloud providers like GKE and EKS.

This multicluster setup demonstrates how Istio can extend a single service mesh across environments, giving us the flexibility to deploy, scale, and migrate workloads without disruption. GKE’s seamless integration with Istio further simplifies the management of complex architectures, making it easier to adopt zero-trust security, advanced traffic routing, and multi-cloud resilience.

This is just the beginning — I’m looking forward to leveraging more of Istio’s capabilities to enhance reliability, observability, and security across our distributed applications running on GKE and beyond.

When Google Cloud Next 2025 dropped the curtain on the Agent SDK (a.k.a. ADK) — their shiny new toolset for building conversational agents — I knew two things instantly:

1. I had to try it immediately.
2. I was going to make it talk to my GKE cluster.

Fast forward a few coffees and command-line sessions later, and I now have a working conversational agent that can talk to, analyze, and interact with my Kubernetes workloads — all wrapped with a clean API, thanks to none other than FastAPI, my favorite framework (and the one Google used under the hood of ADK!).

Let’s dive into how I built it, what this SDK actually does, and why this might just be the future of cloud-native operations.

🚀 What Is Google’s Agent SDK (ADK)?

In short: a framework for building AI-powered agents that can understand user intents, execute tools, and hold context-aware conversations.

It combines the power of Google Cloud’s Gemini models, vector memory, and native tool-calling via FastAPI — all packaged in a dev-friendly kit.

What’s inside ADK?

• Built-in Gemini Pro support (via Vertex AI)
• FastAPI-based tool integration
• Natural multi-turn conversation support
• Easy-to-deploy agent runtimes

And yes — FastAPI is baked into the SDK as the underlying server tech. Google knows what’s up 😎.

🛠️ Why Make a GKE Chatbot?

Because kubectl is great, but natural language is better.

Imagine asking:

“How many pods are in my production namespace?”
or
“Scale the frontend deployment to 5 replicas.”

…and getting instant results. No context switching, no syntax memorization. Just chat.

I’ve uploaded a simple yet powerful implementation of this GKE chatbot to GitHub, designed to showcase just the tip of the iceberg of what Google’s ADK is capable of. It uses a handful of straightforward tools and functions — but even with this minimal setup, the possibilities are already exciting.

This is just the beginning. I’m looking forward to expanding the bot’s capabilities — adding more advanced tools, deeper GKE integrations, and perhaps even multi-cluster or policy-driven workflows. With the ADK’s flexibility and FastAPI’s developer-friendliness, the sky’s the limit. 🚀

Here’s a quick look at how the agent is declared along with the tool functions I’ve registered.

Honestly, it’s just so damn fabulous how the ADK abstracts away almost all the complexity involved in building a conversational agent. And the cherry on top? The built-in web UI — powered by none other than FastAPI — is chef’s kiss. Here’s the snap of the same:

The UI is phenomenal. It not only allows you to experiment with different inputs but also provides rich, step-by-step details for each action the agent performs. You can visually trace every decision, every tool execution, and every intermediate step the model takes — it’s like watching the brain of your agent in action.

For this demo, I kept it simple with two tools:

• One to list all namespaces
• Another to list all pods and deployments within a given namespace

Asked the agent for listing all the pods in all the namespaces.

What’s truly impressive is how the agent chains the tools intelligently. In the example below, you can see how it:

1. First calls the list_namespaces() function.
2. Parses the result and picks a namespace.
3. Then calls list_pods(namespace) using that result—all without me hardcoding a workflow!

The tool execution graph visualizes this beautifully, showing the flow of information and how one tool’s output becomes another tool’s input. Every action is logged in detail, making it incredibly easy to debug or improve the logic.

🛠️ Want to Try It Yourself?
I’ve published the full implementation of the Kubernetes Assistant Agent on GitHub:
👉 sayed-imran/google-adk-kubernetes-bot

This project is a lightweight yet powerful example of using Google’s new Agent Development Kit (ADK) to interact with Kubernetes clusters via natural language. Here’s what’s inside:

🔍 Overview

A conversational agent built with FastAPI (via ADK) and backed by Gemini models, designed to understand your prompts and perform Kubernetes operations accordingly.

⚙️ Features

The assistant currently supports:

• Listing all namespaces in your cluster
• Viewing deployments, pods, and services within any namespace
• Retrieving all resources in a given namespace with a single query

📦 Installation

It’s super easy to set up. You’ll need Python 3.10+, an API key for Gemini, and a working kubeconfig. Just clone the repo, install dependencies with pip, set your API key, and you’re good to go.

🚀 Usage

Run the assistant locally with:

adk run

This spins up an interactive web UI where you can start chatting with your cluster right away.

Feel free to fork it, extend it with more tools, or plug it into your own DevOps workflows. Let’s explore how conversational agents can become a part of our daily ops toolkit. 💬⚙️

This, again, is just a minimal implementation, but it already feels like magic. Imagine what we could do with a few more tools — triggering rollouts, managing autoscaling, debugging failing pods, or even syncing with GitOps pipelines!

If you’re reading this and already thinking of ideas — you’re my kind of person.
Let’s build this out together. Fork the repo, add a new tool, or connect it to your DevOps workflows. The possibilities with Google’s ADK are nearly endless.

Let’s give our clusters a voice — because DevOps should be more than just YAML and kubectl. 🧠💬

Istio has become the de facto standard for managing service-to-service communication in Kubernetes, offering powerful features like traffic management, security, and observability. However, when it comes to securing external traffic using Istio Gateway on Google Kubernetes Engine (GKE), managing TLS certificates can be a challenge. While cert-manager simplifies certificate issuance in Kubernetes, it currently does not support auto-provisioning certificates based on annotations for Istio Gateway.

This limitation leaves teams with the manual task of creating Certificate and Issuer resources, which can lead to inconsistencies, misconfigurations, and increased operational overhead. To address this gap, we can leverage a Kubernetes Admission Controller to automate certificate provisioning seamlessly.

Why This Matters for GKE Users ?

For teams running Istio on GKE, managing certificates manually is inefficient, especially in large-scale environments with multiple microservices. Google Kubernetes Engine provides built-in integrations with Google-managed certificates, but many teams prefer cert-manager for its flexibility and support for different Certificate Authorities (CAs). This article will provide a solution that ensures cert-manager works seamlessly with Istio Gateway on GKE, automating certificate provisioning while maintaining security and reliability.

Tech Stack

To build the admission controller, we will be leveraging the following technologies:

• FastAPI — A high-performance web framework for building asynchronous, HTTP-based service APIs in Python. This will serve as the backbone of our webhook service.
• Uvicorn — A minimal and lightning-fast ASGI server, perfect for running FastAPI applications in production.
• Kubernetes Python SDK — Enables interaction with Kubernetes API resources, allowing our webhook to automatically create certificates upon request.
• Docker — Used to containerize our webhook service, making deployment seamless across environments.
• Google Kubernetes Engine (GKE) — Our target Kubernetes cluster where we will deploy and test the admission controller in action.

Here’s the link to the github repo: Istio-Cert Admission Controller

Before diving into implementation, let’s first understand Kubernetes webhooks and their role in automating resource management.

Kubernetes Webhooks: Validating vs. Mutating

Admission webhooks in Kubernetes allow us to modify or validate resources before they are persisted to the cluster. They provide an efficient way to enforce policies, modify objects dynamically, and automate resource management. Kubernetes supports two main types of webhooks:

1. Validating Webhook

A validating webhook intercepts API requests and ensures they meet specific requirements before the resource is created or modified. If the request does not comply with defined rules, the webhook can reject it.

💡 Use Case Example: Ensuring that every Istio Gateway resource contains a valid host field before it is admitted into the cluster.

2. Mutating Webhook

A mutating webhook goes a step further by modifying the request before it is persisted in the cluster. This is particularly useful for injecting default values or adding annotations dynamically.

💡 Use Case Example: Automatically adding cert-manager.io/issuer annotations to Istio Gateway resources so that certificates are provisioned without manual intervention.

For our admission controller, we will be building a validating webhook that dynamically creates a certificate from the mentioned issuer/cluster-issuer in the annotation of the Istio Gateway, also, if the same doesn’t exist will respond with the same message that the mentioned issuer/cluster-issuer doesn’t exist.

Going into the details about the code can make things lengthy and become complex, I’ll just brief about what concept and the implementation.

As the above diagram shows, there are three primary layers of this admission controller as follows:

API Layer

The API layer serves as the entry point to the admission controller. It receives the intercepted Kubernetes API requests related to the admission of Istio Gateway resources. Once a request is received, it is parsed and the JSON payload is forwarded to the handler layer for validation and further processing.

This modular design keeps the controller clean and extensible. Below is a snapshot of the API layer in action:

Handler Layer

The handler layer is responsible for validating the presence and correctness of specific annotations within the intercepted Istio Gateway resource. It primarily checks for the following annotations:

• cert-manager.io/issuer: <issuer-name>
• cert-manager.io/cluster-issuer: <clusterissuer-name>

Based on the presence or absence of these annotations, it follows one of two paths:

Case 1: Annotations Not Present

If neither of the required annotations is found in the request, the handler bypasses further validation and immediately returns an AdmissionReview response indicating the request is allowed. The response looks like this:

{
    "apiVersion": "admission.k8s.io/v1", 
    "kind": "AdmissionReview", 
    "response": {
        "allowed": true, 
        "uid": "c51f1429-48e9-47b7-bd3b-659b025b74b9",
        "status":{
            "message": "Validation passed"
        }
    }
}

NOTE: The response must strictly adhere to this format for Kubernetes to accept the result. The uid field in the response must exactly match the uid from the intercepted request to ensure that the response maps correctly to the original AdmissionReview.

Sample Intercepted Request

Below is an example of an incoming AdmissionReview request received by the admission controller. It includes metadata and annotations related to the creation of an Istio Gateway:

{
    "kind": "AdmissionReview",
    "apiVersion": "admission.k8s.io/v1",
    "request": {
        "uid": "c51f1429-48e9-47b7-bd3b-659b025b74b9",
        "kind": {
            "group": "networking.istio.io",
            "version": "v1",
            "kind": "Gateway"
        },
        "resource": {
            "group": "networking.istio.io",
            "version": "v1",
            "resource": "gateways"
        },
        "requestKind": {
            "group": "networking.istio.io",
            "version": "v1",
            "kind": "Gateway"
        },
        "requestResource": {
            "group": "networking.istio.io",
            "version": "v1",
            "resource": "gateways"
        },
        "name": "istio-gateway",
        "namespace": "istio-system",
        "operation": "CREATE",
        "userInfo": {
            "username": "",
            "uid": "",
            "groups": [
                "system:masters",
                "system:authenticated"
            ],
            "extra": {}
        },
        "object": {
            "apiVersion": "networking.istio.io/v1",
            "kind": "Gateway",
            "metadata": {
                "annotations": {
                    "cert-manager.io/cluster-issuer": "istio-cluster-issuer",
                },
                "creationTimestamp": "2025-04-05T07:25:12Z",
                "generation": 1,
                "managedFields": [
                    {
                        "apiVersion": "networking.istio.io/v1",
                        "fieldsType": "FieldsV1",
                        "fieldsV1": {
                            "f:metadata": {
                                "f:annotations": {
                                    ".": {},
                                    "f:cert-manager.io/cluster-issuer": {},
                                    "f:kubectl.kubernetes.io/last-applied-configuration": {}
                                }
                            },
                            "f:spec": {
                                ".": {},
                                "f:selector": {
                                    ".": {},
                                    "f:istio": {}
                                },
                                "f:servers": {}
                            }
                        },
                        "manager": "kubectl-client-side-apply",
                        "operation": "Update",
                        "time": "2025-04-05T07:25:12Z"
                    }
                ],
                "name": "istio-gateway",
                "namespace": "istio-system",
                "uid": "b658440c-9eff-4579-bbdc-81a137129981"
            },
            "spec": {
                "selector": {
                    "istio": "ingressgateway"
                },
                "servers": [
                    {
                        "hosts": [
                            "grpc-test.example.com"
                        ],
                        "port": {
                            "name": "https",
                            "number": 443,
                            "protocol": "HTTPS"
                        },
                        "tls": {
                            "credentialName": "istio-ingress-tls",
                            "mode": "SIMPLE"
                        }
                    }
                ]
            }
        },
        "oldObject": null,
        "dryRun": false,
        "options": {
            "kind": "CreateOptions",
            "apiVersion": "meta.k8s.io/v1",
            "fieldManager": "kubectl-client-side-apply",
            "fieldValidation": "Strict"
        }
    }
}

NOTE: The uid inside the request block (c51f1429-48e9-47b7-bd3b-659b025b74b9) is the one that must be returned in the admission controller's response to ensure accurate validation of the original request.

Case 2: Annotations Are Present

If the required annotations are present, the handler proceeds to verify whether the specified Issuer or ClusterIssuer exists within the cluster:

• If the Issuer/ClusterIssuer is found: The request is allowed to proceed as usual.
• If the Issuer/ClusterIssuer is not found: The request is denied, and the admission controller returns a response with allowed: false, along with an error message stating that the referenced issuer does not exist.

This safeguards the cluster from deploying gateways referencing non-existent certificate managers, which could result in misconfigurations or failures in TLS termination.

Kubernetes Utility Layer

Once all validations are successfully passed, control is handed over to the Kubernetes Utility Layer. Unlike standard validation webhooks that only accept or reject requests, this layer goes a step further — it actively creates Kubernetes resources. This is essential to our core goal: auto-provisioning TLS certificates for Istio Gateway resources.

This layer interacts directly with the Kubernetes API server to create the required Certificate resources based on metadata and annotations extracted from the intercepted Gateway object.

Certificate Creation and Ownership

A central task of this layer is the creation of a Certificate resource when the referenced Issuer or ClusterIssuer exists in the cluster.

The certificate is initialized with default values for fields such as duration and renewBefore. However, these can be customized via annotations on the Gateway resource, giving users granular control without modifying the controller itself.

Another key aspect is the automatic injection of an ownerReference into the generated Certificate. This reference links the Certificate to its parent Gateway resource, ensuring that when the Gateway is deleted, the associated Certificate is also cleaned up by Kubernetes' garbage collector.

This automation replaces what would otherwise be a manual and error-prone process:

1. Manually creating the Gateway.
2. Retrieving its metadata.
3. Manually crafting a matching Certificate resource with correct references.

By handling all of this internally, the admission controller drastically reduces operational complexity and ensures seamless, lifecycle-aware certificate management.

Wiring It All Together: ValidatingWebhookConfiguration

Once the admission controller is deployed and running, the final step is to ensure that relevant API requests are intercepted and routed through it. This is achieved by creating a ValidatingWebhookConfiguration.

This Kubernetes resource registers the webhook with the API server, specifying what kind of resources and operations should trigger the admission controller. In our case, it’s configured to intercept CREATE and UPDATE operations on Istio Gateway resources.

Here’s the manifest for the ValidatingWebhookConfiguration that ties everything together:

🛡️ NOTE: Admission controllers must be served over HTTPS, as the Kubernetes API server only communicates with webhooks via secure channels. While the details of setting up TLS for the controller are outside the scope of this article, it’s an essential step to ensure the controller functions properly in a production environment.

Conclusion

By extending the functionality of a standard admission controller to include resource creation, we’ve built a streamlined and automated solution for certificate provisioning tied to Istio Gateways. This not only eliminates manual overhead but also ensures lifecycle-aware cleanup using owner references — all while integrating seamlessly into the Kubernetes admission workflow.

While this approach is tailored for Istio and cert-manager, the underlying pattern can be extended to other dynamic resource scenarios as well. With the right validations, ownership models, and security considerations in place, admission controllers can go beyond validation to become powerful automation tools within your Kubernetes ecosystem.

Seamless TLS for Istio on GKE: Auto-Provisioning Certificates with a Custom Admission Controller was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

In my previous article on Getting Started with Writing Operator, I built a simple operator which launched a Kubernetes Deployment and exposed it via Kubernetes Service for the custom resource Microservice.

In this part of the article, I’ll be moving on from development mode to deployment, and will be running the operator as a Kubernetes Deployment in the Kubernetes itself. I’ll be using AWS EKS as the Kubernetes environment.

Kubernetes Operator operates in the same method irrespective of the Kubernetes Environment.

The following steps will be involved to deploy the operator to Kubernetes:

1. Adding authentication logic for Operator
2. Creating Dockerfile for the Operator.
3. Building and pushing the image to Docker Hub.
4. Launching an EKS Cluster.
5. Creating necessary resources, CRD and the Operator.
6. Deploying Microservice to test the operator.

Authentication Logic for Operator

@kopf.on.login()
def custom_login_fn(**kwargs):
    if EnvConfig.ENV == "dev":
        return kopf.login_with_kubeconfig(**kwargs)
    else:
        return kopf.login_with_service_account(**kwargs)

I’ve modified the code and folder structures to make it more readable, you may find the code here: Microservice Operator.

Every time the operator starts, it performs an initial authentication process with the Kubernetes API server, to determine the whereabouts of the Kubernetes to be used. This above function overrides the default authentication flow, which in my case is environment driven.

Depending upon the value passed for the environment variable, operator will be able to authenticate from either using kubeconfig (for development purpose) or from the service account (for production).

Dockerfile for Operator

The following Dockerfile is used for creating the image of the operator.

FROM python:3.11-slim-buster

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY . .

ENTRYPOINT ["kopf", "run", "main.py"]

The requiremets.txt file contains the python packages used for the operator.

Building and Pushing Image

I’ve used my personal docker image registry to push the image.

Command to build the image:

docker build -t sayedimran/microservice-operator:v1.0.0 .

Command to push the image to the registry:

docker push sayedimran/microservice-operator:v1.0.0

Launching an EKS Cluster

I’ve created an EKS cluster with all the default options as shown in the below image.

Created a Node Group with all default options with Amazon Linux 2 Image with 2 nodes of type t2.medium and 20 GiB of storage for each:

Once the cluster is ready, to access the EKS via kubectl, I’ll be the using the aws CLI in AWS CloudShell.

Command to generate kubeconfig:

aws eks --region ap-south-1 update-kubeconfig --name aws-cluster-1

Checking for the kube-system namespace pods

Cluster is now ready for the operator to be deployed.

Creating resources & CRD for Operator

As of now the operator has been bundled as image and pushed to the registry which now can be pulled into the Kubernetes when deployed.

Before deploying the operator, the following resources are the pre-requisites for the operator to function in the cluster:

• ClusterRole
• ClusterRoleBinding
• Role
• RoleBinding
• ServiceAccount

All the above resources are basically the RBAC permissions to be provided to the operator so as to provision resources on launch of the custom resources.

Link of the CRD, resources & operator deployment can be found here:

• Microservice CRD [crd.yaml]
• Microservice Operator Resources and Deployment [deploy.yaml]

Command to deploy the CRD, resources and operator:

kubectl apply -f https://raw.githubusercontent.com/sayed-imran/microservice-operator/v1.0.0/crd.yml
kubectl apply -f https://raw.githubusercontent.com/sayed-imran/microservice-operator/v1.0.0/deploy.yml

Testing the Microservice Resource

Once the pod for operator goes to running state, we can deploy our custom resource i.e. Microservice.

Deploying a sample microservice application, link for the same can be found here: [sample-microservice.yaml]

Command to deploy the custom resource:

kubectl apply -f https://raw.githubusercontent.com/sayed-imran/microservice-operator/v1.0.0/cr.yml

As we can see the resource has been deployed successfully, we can also check for the child resources i.e., pods of the deployment and the service.

This marks the completion of the deployment of the Kubernetes Operator in the EKS.

Looking forward to add further more advanced logic to the Operator, do let me know if you want me to add anything relevant and please do give your feedbacks in the comment, would love to hear from you.

#kubernetes #operator #aws #eks #automation #python #kopf #docker #devops #cloud #containerization #sdk #python #go #community #learning #dev

In my previous article, Introduction to Operators (Part-1) I briefed about the various aspects of a Kubernetes Operator. This part of the series will be getting started with writing a basic operator.

As mentioned earlier, we’ll be using Kopf (Kubernetes Operator Pythonic Framework), for building our first basic operator.

Apart from that, we need the following things to get started:

• pykube-ng ( a lightweight Python client library for Kubernetes which is preferred when developing a Operator.)
• minikube or any running kubernetes cluster with accessible kubeconfig file. (I’ll be using minikube)

We’ll be creating a operator for a custom resource called Microservice, which basically will launch a deployment as a child resource for the same.

Custom Resource Definition for the Operator and Resource

Any Custom Resource (CR) comes with a Custom Resource Definition (CRD), which defines the schema of the CR and other necessary fields. Here’s the CRD to be used for the resource that we will be creating.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: microservices.imran.dev.io
spec:
  group: imran.dev.io
  scope: Namespaced
  names:
    plural: microservices
    singular: microservice
    kind: Microservice
    shortNames:
    - ms
  versions:
  - name: v1alpha
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              replicas:
                type: integer
                minimum: 1
              image:
                type: string
              labels:
                type: object
                additionalProperties:
                  type: string
              env:
                type: array
                items:
                  type: object
                  properties:
                    name:
                      type: string
                    value:
                      type: string
            required:
            - replicas
            - image
          status:
            type: object
            properties:
              ready:
                type: string
    additionalPrinterColumns:
    - name: Replicas
      type: integer
      description: Number of replicas for the microservice
      jsonPath: .spec.replicas
    - name: Image
      type: string
      description: Image used for the microservice
      jsonPath: .spec.image

The metadata field contains the name of the CRD in the format <plural>.<group>. Here, it's microservices.imran.dev.io.

The spec field contains the details of the custom resource:

• group is the name of the API group that this CRD belongs to. In this case, the group is imran.dev.io. API groups are a way of organizing related kinds in the Kubernetes API.
• scope defines whether the custom resource is cluster-wide or namespace-specific. Here, the scope is Namespaced, meaning instances of this custom resource can be created within specific namespaces in the Kubernetes cluster.
• names defines the naming conventions for the custom resource. The plural and singular fields are used in the URL of the API endpoint and in command-line interactions, respectively. The kind field is the name of the custom resource that you're defining with this CRD. The shortNames field provides a shorter alias for the resource, which can be used in command-line interactions.

The versions field is an array that defines one or more versions for the custom resource. Each version can have its own schema and additional printer columns. Here, we have one version named v1alpha which is both served and stored.

The schema of the custom resource is defined under properties. Here, the custom resource has properties replicas, image, labels, and env.

• replicas is a required field that specifies the number of instances of a microservice that should be running.
• image is another required field that specifies the Docker image to use for the microservice.
• labels is another required field that is an object with additional properties of type string. This is typically used for specifying metadata.
• env is an array of objects, each having name and value properties. This is used for setting environment variables for the microservice.

The additionalPrinterColumns field is used to customize the output when using kubectl to list these resources. Here, two additional columns are defined: Replicas and Image, which display the number of replicas and the image used for the microservice, respectively. The jsonPath specifies the path to the field in the custom resource's Spec that should be used for the column's value.

Now, applying the CRD to the Kubernetes:

Kubernetes Operator Code Snippet

Here’s the code snippet to the operator that will drive the CR lifecycle:

import kopf
import pykube


@kopf.on.create("imran.dev.io", "v1alpha1", "microservices")
def create_fn(spec, **kwargs):
    kube_api = pykube.HTTPClient(pykube.KubeConfig.from_file())
    deployment = pykube.Deployment(kube_api,{
        'apiVersion': 'apps/v1',
        'kind': 'Deployment',
        'spec': {
            'replicas': spec['replicas'],
            'selector': {
                'matchLabels': spec['labels'],
            },
            'template': {
                'metadata': {
                    'labels': spec['labels'],
                },
                'spec': {
                    'containers': [
                        {
                            'name': kwargs['body']['metadata']['name'],
                            'image': spec['image'],
                            'env': spec.get('env', []),
                        },
                    ],
                },
            },
        },
    })
    kopf.adopt(deployment)
    deployment.create()
    return {'message': 'Deployment created'}

The Operator listens for the creation of custom resources of type microservices in the imran.dev.io group and v1alpha1 version, and creates a corresponding Kubernetes Deployment.

The @kopf.on.create decorator registers a function to be called whenever a microservices custom resource is created. The function parameters spec and kwargs contain the details of the custom resource.

Inside the function, a pykube.HTTPClient is created to interact with the Kubernetes API. The pykube.KubeConfig.from_file() function reads the kubeconfig file from its default location and uses it to authenticate with the Kubernetes API.

A pykube.Deployment object is then created. The deployment specification is built using the spec from the custom resource. The number of replicas, the labels, the container image, and the environment variables are all taken from the custom resource's spec.

The kopf.adopt(deployment) function is called to set the owner references of the Deployment to the custom resource. This means that when the custom resource is deleted, the Deployment will be automatically deleted as well.

Command to run the Operator: kopf run main.py --verbose

YAML for the CR:

apiVersion: imran.dev.io/v1alpha1
kind: Microservice
metadata:
  name: micron
spec:
  labels:
    app: microservice-sample
    env: dev
  replicas: 3
  image: sayedimran/fastapi-sample-app:v4
  env:
    - name: ENV
      value: test
    - name: LOG_LEVEL
      value: debug

Now, applying the YAML to the Cluster

So, now if check for the resource:

Our custom resource has been created, and the deployment & pods for the same has been created as well, which can be verified:

This marks the end of part-2 of the series, where we created a simple operator to get an overall understanding of the concepts, we can definitely add more functionality to it such as: support for having secretKeyRef, configMapKey, volumes and volumeMounts, etc.

I’ve been adding more and more logic to the operator, you may find the GitHub Repo Link: microservice-operator for the same.

Currently I’ve added support for service creation, which is helpful in exposing the deployment, that gets deployed as a child resource as well.

In the next part of series, we’ll work on deploying this operator to the Kubernetes Operator as a deployment.

Do let me know if you’ve got any doubts or any type of feedbacks in the comments.

#kubernetes #operator #crd #devops #automation #containerization #cloud #aws #googlecloud #azure #containers #automation #sdk #python #go #community #learning #dev