What is Domain Redirection?

A domain redirect is a technique for automatically redirecting users from one web address to another. It is also known as URL redirection, which forwards one domain or URL to another. When users attempt to access a specific domain, the server redirects them to a different domain.

Types of Redirections

• 301 Redirect: This is a permanent redirection, indicating that the original URL has permanently moved to a new location. Search engines recognize this redirection type and transfer the page rank to the new URL.
• 302 Redirect: This is a temporary redirection. It indicates that the move is temporary, and search engines typically don’t transfer page rank to the new URL.
• Meta Refresh: This method of redirection is executed at the HTML level. It involves placing a meta tag in the HTML header of a web page, which automatically redirects the browser to a different URL after a specified amount of time.
• JavaScript Redirect: Similar to Meta Refresh, JavaScript can redirect users to a different URL after a certain amount of time or immediately.

Use Cases

1. Changing Domain Names: If you’ve changed your website’s domain name, you can redirect the old domain to the new one so that visitors who use the old URL are automatically directed to the new one.
2. Consolidating Multiple Domains: If you have multiple domains that you want to point to a single website, you can set up redirection to all lead to the same destination.
3. Correcting Misspellings: If users frequently misspell your domain name, you can redirect the misspelled versions to the correct ones to ensure they still reach your site.
4. Managing Traffic: Redirection can also be used to manage traffic flow. For example, during high-traffic periods, you might redirect visitors to a different server or a cached version of your site to improve performance.

Problem Statement

An organization deployed its product application on the Google Kubernetes Engine, enabled with ASM (Anthos Service Mesh), and recently renewed its brand name. To continue serving older customers using the initial domain name, they have set Nginx Redirection.

The challenge here is to set up the Redirection using Native Istio Service Mesh instead of Nginx, which adds another layer of Nginx management overhead.

Solution

Istio supports various kinds of domain redirections and URL rewrites. We can use it to handle domain redirection using the Istio native approach instead of adding additional redirection services or Ingress.

Prerequisites

1. Google Cloud Platform Account
2. Google Kubernetes Engine (GKE) Cluster with ASM or Istio

We have already set up a Google Kubernetes Engine cluster with Anthos Service Mesh (ASM), which was installed as part of the prerequisites.

Anthos Service Mesh is a managed service mesh, based on Istio, that provides a security-enhanced, observable, and standardized communication layer for applications.

• About Anthos Service Mesh | Google Cloud
• Anthos Service Mesh prerequisites | Google Cloud

Let’s deploy one sample microservice application as part of our use case. We use the Bank of Anthos Application provided by GCP for demo purposes.

Create a separate namespace to deploy all the Bank of Anthos Application microservices.

kubectl create ns anthos-app

You can refer to the deployment Quickstart below provided by GCP for the BOA application deployment.

GitHub - GoogleCloudPlatform/bank-of-anthos: Retail banking sample application showcasing Kubernetes and Google Cloud

ASM Ingress Gateway Deployment

An ingress Gateway describes a load balancer operating at the edge of the mesh that receives incoming HTTP/TCP connections. Unlike Kubernetes Ingress Resources, it does not include any traffic routing configuration.

kubectl apply -f asm-ingressgateway-deployment.yaml

After the Ingress Gateway deployment, we need to create the ASM Ingress Gateway deployment service to integrate it with GKE Ingress and expose the BOA application to external traffic.

Note: The difference between the routing of a regular GKE Ingress and an ASM Ingress is that the routing decisions are performed inside the cluster (e.g., inside namespace/asm-ingress/deployment/asm-ingressgateway).

kubectl apply -f ingressgateway-service.yaml

GKE Ingress Deployment

Frontend Istio VirtualService Configuration

A VirtualService is a high-level configuration resource that controls how requests are routed to services in your cluster.

Below is the frontend virtual service configuration for routing traffic to the BOA frontend application.

Domain Redirection Istio VirtualService Configuration

The provided manifest is an Istio VirtualService configuration used to redirect HTTP traffic from one domain to another domain.

• Host: Matches incoming requests to demo.example.com.
• Gateway: Uses the Istio Gateway example-ingressgateway under the asm-ingress namespace.
• Redirect:
  When a request comes to demo.example.comIstio will redirect the authority (host) to app.demo.example.com.
  This typically results in an HTTP 301/302 redirect.

Note: If you also want to redirect the path or scheme (e.g., HTTP to HTTPS), you can extend the rule like this:

http:
  — match:
      — uri:
          prefix: /
    redirect:
      authority: app.demo.example.com
      scheme: https

Testing the domain redirection

1. If we curl to the old domain URL as shown below, Istio redirects it to the new domain as configured in the Istio Virtual Service.

2. If we curl to the old domain URL with any prefix with it as shown below, Istio redirects it to the new domain as configured in the Istio Virtual Service.

3. Upon hitting the Old domain URL from the browser, you can see it redirecting to the new domain URL successfully.

Conclusion

Istio’s powerful traffic management capabilities allow you to implement clean and efficient HTTP redirection policies through its VirtualService configuration. By simply leveraging the redirect block, you can seamlessly route users to a new domain, subdomain, or even enforce HTTPS—all without changing your application code.

This is especially useful during domain migrations, enforcing canonical URLs, or setting up SEO-friendly redirects. Combined with Istio Gateways, you gain fine-grained control over ingress behavior in a secure and declarative way.

Whether you’re managing a microservices architecture or just introducing smart routing at the edge, Istio makes redirection easy, scalable, and cloud-native.

🔗 References:

1. Sample BOA application source code: https://github.com/GoogleCloudPlatform/bank-of-anthos
2. Istio Service Mesh Documentation: https://istio.io/latest/docs/

🤔 Questions?

If you have any questions, I’ll be happy to read them in the comments. Follow me on Medium or LinkedIn for more interesting content.

😃 Try out the Istio Native Redirection & let me know your experience or use cases in the comments.

🚀 Let’s connect and embark on a transformative cloud journey together!

Streamlining Domain Redirects: Unlocking Efficiency with Istio was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

What is Google Infrastructure Manager?

Infrastructure Manager is a GCP-managed service that automates the deployment and management of Google Cloud infrastructure resources.

Infrastructure is defined using Terraform and deployed onto Google Cloud by Infra Manager, enabling you to manage GCP resources using Terraform as a service.

Infra Manager for IAC Lifecycle Management

1. You can keep the Terraform code or configuration, either in a public Git repository, in a Cloud Storage bucket, or in a local machine.
2. Infra Manager keeps the versioning of every terraform deployment by creating revisions of the deployments.
3. For every revision, Infra Manager stores the Logs, terraform source code, Artifacts, and state files in a cloud storage bucket.

Infra Manager Workflow

1. Creating Infra Manager deployment using gcloud CLI to deploy resources from terraform code.
2. Infra Manager takes the request and creates a cloud build trigger to perform terraform init | validate | plan and apply stages.

Infra Manager Terraform Constraints

1. Templating or generation of Terraform is not supported.
2. The configuration should be using Terraform version 1.2.3.
3. Backend blocks shouldn’t be defined.
4. Provisioners are not supported.

Infra Manager Pricing

Infrastructure Manager charges for Cloud Build execution and Cloud Storage of source code + logs blueprints.

Infra Manager In Action

In this article, we will be deploying sample GCS buckets using Google Infra Manager.

Prerequisite

1. Google Cloud Project

As a part of the prerequisite, we already have an active GCP Project where we can use the GCP Infra manager to provision the GCP resources.

Enabling Infra Manager Service API

Before getting started with GCP Infra Manager we need to enable the Infra Manager service API for the GCP Project.

gcloud services enable config.googleapis.com

Authentication for Infra Manager Deployments

In order to create Infra Manager deployment the service account used for deploying the resources defined in the terraform code needs to have a config agent role.

1. Create the service account (If you do not have one already)

gcloud iam service-accounts create SERVICE_ACCOUNT_NAME

2. Grant the roles/config.agent IAM role to the service account:

gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com" --role=roles/config.agent

3. Grant the additional required permissions for deploying the resources described in the terraform code.

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member=serviceAccount:SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com \
    --role=storage.Admin // for creating GCS buckets

Creating Infra Manager Deployment

Below are the respective commands for creating Infra Manager deployments based on source code location.

1. Deploy a Terraform configuration stored in a Cloud Storage bucket

gcloud alpha infra-manager deployments apply projects/PROJECT_ID/locations/LOCATION/deployments/DEPLOYMENT_ID \
    --service-account projects/SERVICE_ACCOUNT_PROJECT_ID/serviceAccounts/SERVICE_ACCOUNT \
    --gcs-source gs://BUCKET_NAME/OBJECT_NAME \
    --input-values=INPUT_1_NAME=VALUE,INPUT_2_NAME=VALUE

2. Deploy a Terraform configuration stored in a public Git repository

gcloud alpha infra-manager deployments apply projects/PROJECT_ID/locations/LOCATION/deployments/DEPLOYMENT_ID \
    --service-account projects/SERVICE_ACCOUNT_PROJECT_ID/serviceAccounts/SERVICE_ACCOUNT \
    --git-source-repo="GIT_REPO" \
    --git-source-directory="DIRECTORY" \
    --git-source-ref="REF" \
    --input-values=INPUT_1_NAME=VALUE,INPUT_2_NAME=VALUE

3. Deploy a Terraform configuration stored on your local machine

gcloud alpha infra-manager deployments apply projects/PROJECT_ID/locations/LOCATION/deployments/DEPLOYMENT_ID \
    --service-account projects/SERVICE_ACCOUNT_PROJECT_ID/serviceAccounts/SERVICE_ACCOUNT \
    --local-source="LOCAL_DIRECTORY" \
    --input-values=INPUT_1_NAME=VALUE,INPUT_2_NAME=VALUE

In our case, we have the terraform source code (root+calling modules) stored in a public git repository.

Note: Currently Infra Manager only supports public git repositories.

We can also see the cloud build trigger created by Infra Manager at the backend for deploying the terraform code.

After Infra Manager terraform deployment we can see that the sample GCS bucket has been provisioned successfully.

Infra Manager creates a storage bucket in the project and location where Infra Manager is run. Currently, the supported locations include asia-east1, europe-west1, and us-central1.

After the deployment is created, the Infrastructure Manager artifacts are in this storage bucket which has the name in the format: gs://PROJECT_ID-LOCATION-blueprint-config.

The artifacts in the blueprint storage bucket include:

• Cloud Build logs.
• Terraform logs.
• A copy of the Terraform configuration.

Infra Manager Deployment Revisions

For every terraform run Infra Manager deployment creates revisions to keep versioning.

When you initially create a deployment, this deployment is also a revision and has the revision ID r-0.

If you update the deployment(Terraform code modification), then Infrastructure Manager creates a new revision with an identifier(revision ID)r-1. For every new revision, the identifier increases by one.

Now, Let's create a new GCS bucket named gcp-infra-manager-bucket-2 using Infra Manager deployment. This will create a new revision of Infra manager deployment named gcs-deployment.

gcloud alpha infra-manager deployments apply projects/onkar-17-cloud/locations/us-central1/deployments/gcs-deployment\
   --service-account projects/onkar-17-cloud/serviceAccounts/infra-manager-gcp@onkar-17-cloud.iam.gserviceaccount.com\
    --git-source-repo=https://github.com/Onkar179/google-infra-manager-repo \
    --git-source-directory=gcs-buckets \
    --git-source-ref=main \
    --input-values=name=gcp-infra-manager-bucket-2

You can also check & view the deployments, revisions & state of the resources that have been deployed by the Infra Manager.

Listing Infra Manager deployment

gcloud alpha infra-manager deployments list --project PROJECT_ID --location "LOCATION"

Checking Infra Manager deployment state

The Infra Manager deployments have several states from which the status of deployment can be checked.

REST Resource: projects.locations.deployments | Infrastructure Manager | Google Cloud

gcloud alpha infra-manager deployments describe projects/PROJECT_ID/locations/LOCATION/deployments/DEPLOYMENT_ID

Listing revisions in a deployment

gcloud alpha infra-manager revisions list --deployment=projects/PROJECT_ID/locations/LOCATION/deployments/DEPLOYMENT_ID

Checking the State of a revision in a deployment

gcloud alpha infra-manager revisions describe projects/PROJECT_ID/locations/LOCATION/deployments/DEPLOYMENT_ID/revisions/REVISION_ID

Listing the resources provisioned by deployment under a particular revision

Similar to deployments each revision has several states given below from which the status of resource provisioning can be checked.

STATE_UNSPECIFIED, PLANNED, IN_PROGRESS, RECONCILED, FAILED

gcloud alpha infra-manager resources list --revision=projects/PROJECT_ID/locations/LOCATION/deployments/DEPLOYMENT_ID/revisions/REVISION_ID

Checking details of deployed resources

The below command requires Resource ID which you can get from the above command while listing out the resources.

gcloud alpha infra-manager resources describe projects/PROJECT_ID/locations/LOCATION/deployments/DEPLOYMENT_ID/revisions/REVISION_ID/resources/RESOURCE_ID

Managing Terraform states with Infra Manager

Google Infra Manager does support terraform state modification & inspection for each deployment & revision by which we can do state mutation or import the states of manually created resources for managing those resources using Google Infra Manager.

For importing the terraform state of any manually created resource into the Infra Manager managed terraform state, first we need to lock the Infra Manager deployment and download the state file locally.

1. Lock the deployment

Lock the deployment to prevent any changes to the deployment while you mutate the state file. The deployment must be locked to be able to download the state file.

gcloud alpha infra-manager deployments lock DEPLOYMENT_ID --project PROJECT_ID --location LOCATION

2. Download the state file

Downloading state files requires a blueprint cloud storage bucket signed URL.

SIGNED_STATE_DOWNLOAD_URL=$(gcloud alpha infra-manager deployments export-statefile DEPLOYMENT_ID --project PROJECT_ID --location LOCATION --format="get(signedUri)")

curl -s -X GET --output terraform.tfstate ${SIGNED_STATE_DOWNLOAD_URL}

Once the state file is downloaded you can place it under local terraform code workspace and perform state import.

Here we have one bucket named onkar-datathat is manually created in our GCP project. Going forward, In order to manage it using Infra Manager we need to import the state of that bucket in the Infra Manager deployment state that we downloaded above.

Importing the terraform state

After the successful state import, if you have made changes to terraform code files locally then have to upload the modified terraform code to the storage bucket, or public git repository that you are using as the source to deploy the configuration.

In our case, it is a public git repository so have to push the updated terraform code for onkar-data GCS bucket, so that Infra Manager will sync the updated source code with the updated state in order to manage new resources.

Upload back the terraform state file to the Infra Manager storage bucket

1. Getting the lock ID of Infra Manager deployment to unlock

LOCK_ID=$(gcloud alpha infra-manager deployments export-lock DEPLOYMENT_ID --project PROJECT_ID --location LOCATION --format="get(lockId)"

2. Uploading the modified/imported state file

SIGNED_STATE_UPLOAD_URL=$(gcloud alpha infra-manager deployments import-statefile DEPLOYMENT_ID --project PROJECT_ID --location LOCATION --lock-id ${LOCK_ID} --format="get(signedUri)")

curl -s -X PUT --upload-file terraform.tfstate $SIGNED_STATE_UPLOAD_URL

3. Unlock the Infra Manager deployment

LOCK_ID=$(gcloud alpha infra-manager deployments export-lock DEPLOYMENT_ID --project PROJECT_ID --location LOCATION --format="get(lockId)")

gcloud alpha infra-manager deployments unlock DEPLOYMENT_ID --project PROJECT_ID --location LOCATION --lock-id ${LOCK_ID}

Once the deployment is unlocked & after successful terraform state import, the Infra Manager also manages the manually created bucket onkar-data along with gcp-infra-manager-bucket-2 .

Deleting the Infra Manager Deployment

1. Delete the deployment and provisioned resources

gcloud alpha infra-manager deployments delete projects/PROJECT_ID/locations/LOCATION/deployments/DEPLOYMENT_ID

2. Delete the deployment by keeping resources undeleted

Google Infra Manager also supports deletion of deployment by keeping the provisioned resources undeleted.

This can be useful in scenarios where you want to manage the resources manually not using IAC for which you can only delete the deployment.

gcloud infra-manager deployments delete projects/PROJECT_ID/locations/LOCATION/deployments/DEPLOYMENT_ID \
    --delete-policy=abandon

Why Google Infra Manager? Why not just run Terraform directly?

Google Infrastructure Manager Vs Terraform

Google Infra Manager provides the below features in addition to using Terraform locally, which makes it more useable as Terraform as a service.

1. Provides history or traceability of all deployments
2. Role-based access control for each deployment & revisions
3. Governance and auditing controls
4. Secured storage of terraform state files (SIGNED URLs)
5. UI and Observability

Conclusion

With Infrastructure Manager, users can create templates, version their infrastructure configurations, and deploy resources consistently, promoting scalability, agility, and resource optimization.

This tool streamlines infrastructure management and enhances the overall efficiency and reliability of operations on Google Cloud.

🔗 References

1. Demo Source Code: https://github.com/Onkar179/google-infra-manager-repo.git
2. Infra Manager Documentation: https://cloud.google.com/infrastructure-manager/docs/

🤔 Questions?

If you have any questions, I’ll be happy to read them in the comments. Follow me on Medium or LinkedIn for more interesting content.

😃 Try out the Infra Manager deployments & let me know your experience or use cases in the comments.

🚀 Let’s connect and embark on a transformative cloud journey together!

Terraform As A Service: Google Infrastructure Manager was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

This Article is based on how to secure Microservices on GKE using Ambassador Edge Stack API Gateway Authentication.

Problem Statement

Multiple microservices are deployed on the GKE Cluster and exposed to the public world using GKE Ingress. The challenge here is setting up authentication for users accessing various application microservices via GKE Ingress.

Use Case

Let's assume a scenario where we have applications microservices deployed on the GKE cluster and expose them using GKE Ingress to the public users. All incoming user requests to GKE Ingress must be authenticated before routing them to the actual application or backend microservices.

Solution

The solutions to the use case are either having the authentication preconfigured within the application code itself or else having a separate application for authentication only. The requirement here is to have a particular application that does the authentication which can be achieved by using the Ambassador Edge Stack API Gateway solution which handles all the routing and authentication parts more efficiently in microservices.

Prerequisites

1. Google Cloud Platform Account
2. Google Kubernetes Engine (GKE) Cluster

What is Ambassador Edge Stack?

Ambassador Edge Stack is a Kubernetes-native API Gateway that delivers scalability, security, and simplicity for Kubernetes microservices.

The Ambassador Edge Stack makes securing microservices easy with comprehensive security functionality, including automatic TLS, authentication, rate limiting, WAF integration, and fine-grained access control.

As part of the prerequisites, we have already set up a zonal Google Kubernetes Engine cluster for deploying the Ambassador Edge Stack and sample quote microservice application.

Creating a zonal cluster | Google Kubernetes Engine (GKE) | Google Cloud

Ambassador Edge Stack Installation on GKE Cluster

Installing Edge Stack | Edge Stack

You can install Ambassador Edge Stack using various options like Helm charts, Kubernetes YAML, or Docker Image.

We are using the Kubernetes YAML Installation method for Edge Stack.

1. Creating Namespace for Ambassador Edge Stack

kubectl create namespace ambassador

2. Installing Ambassador Edge Stack

For installing the Ambassador Edge Stack first we need to install the Ambassador Edge Stack-related CRDs in order to deploy various edge stack-specific resources on GKE.

kubectl apply -f https://app.getambassador.io/yaml/edge-stack/3.1.0/aes-crds.yaml && \

kubectl wait --timeout=90s --for=condition=available deployment emissary-apiext -n emissary-system && \

After installing the Edge Stack Kubernetes CRDs we can begin the installation of the Ambassador Edge Stack on GKE.

Note: By default, the service type for edge-stack service is Node Port. As GKE Ingress only supports ClusterIP services as the backend, you need to change this service type for edge-stack service to Cluster IP before installing the ambassador edge stack on GKE.

kubectl apply -f https://app.getambassador.io/yaml/edge-stack/3.1.0/aes.yaml && \

kubectl -n ambassador wait --for condition=available --timeout=90s deploy -lproduct=aes

kubectl get pods -n ambassador

kubectl get service -n ambassador

kubectl get all -n emissary-system

We have successfully installed Ambassador Edge Stack on the GKE Cluster.

As part of our use case, let's deploy one sample microservice application. For demo purposes, we use a sample quote application provided by Ambassador.

GitHub - datawire/quote: The Quote of the Moment Service... now with 100% more Go.

1. Sample Quote Application Deployment

This basic configuration creates the quote deployment and service to expose that deployment on port 80.

kubectl apply -f quote.yaml

quote.yaml :

2. Routing traffic to Quote Application from Ambassador Edge Stack

In order to route traffic from the Ambassador Edge Stack service (gateway) to the Quote application, we need to attach the quote application as the backend to the Ambassador Edge Stack service.

Ambassador Edge Stack mainly provides two CRDs (Listener and Mapping) for routing traffic to specific applications from the edge stack.

Listener CRD tells Ambassador Edge Stack what port to listen on, and the Mapping CRD tells Ambassador Edge Stack how to route incoming requests by host and URL path from the edge of your cluster to Kubernetes services.

Listener for HTTP on port 8080 & HTTPS on port 8443 :

listener.yaml :

kubectl get listener -n ambassador

Mapping for Quote Application service :

kubectl apply -f quote-mapping.yaml

quote-mapping.yaml :

kubectl get mappings -n ambassador

Note: If you are using Ambassador Edge Stack in the Private GKE Cluster you might get the below error while creating the mapping for backend services.

The error while creating the mapping for the quote application :

Error from server: error when creating "quote-mapping.yaml": conversion webhook for getambassador.io/v3alpha1, Kind=Mapping failed: Post "https://emissary-apiext.emissary-system.svc:443/webhooks/crd-convert?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

The issue is that the control plane (API Server) is trying to call emissary-apiext.emissary-system.svc:443 but the pods behind it are listening on port 8443.

To solve this issue you need to edit the preconfigured GKE Control Plane firewall rule gke-<cluster_name>-xxxxx-master and add the 8443 TCP port.

We have successfully deployed the sample quote microservice application in the ambassador namespace.

Now let's create GKE Ingress for the Quote Application for L7 application layer load balancing features and expose it to the public users.

GKE Ingress for HTTP(S) Load Balancing | Google Kubernetes Engine (GKE) | Google Cloud

Optional: Here we are configuring SSL termination on GKE Ingress using Google Managed Certificates.

Using Google-managed SSL certificates | Google Kubernetes Engine (GKE) | Google Cloud

kubectl apply -f managed-cert.yaml

managed-cert.yaml :

Before creating the Ingress we need to configure health checks (backend config) for the edge-stack service so that GKE Ingress will mark the backend as healthy without failure.

kubectl apply -f backendconfig.yaml

backendconfig.yaml :

kubectl annote service edge-stack -n ambassador
cloud.google.com/backend-config: '{"default": "ambassador-hc-config"}'

Creating GKE Ingress for quote microservice application service :

Here we are placing the ambassador edge stack service as a backend to the GKE Ingress. Ambassador Edge Stack service routes the traffic coming from GKE Ingress to its mapped backends i.e quote application microservice.

kubetl apply -f ingress.yaml

After creating GKE Ingress successfully let's access the quote microservice application mapped as the backend to edge-stack service by sending curl requests to the configured domain for GKE Ingress and the path on which the quote application running.

curl <GKE_INGRESS_DOMAIN>/backend/

GKE Ingress → edge-stack service → quote application service

You can see that now all public users can able to access the quote application without any kind of authentication.

The requirement is to have authentication in place for authenticating all user's requests before routing them to the main application.

Here you can use the ambassador edge stack authentication feature which gives you options to configure and Integrate the authentication with various third-party authentication providers.

Ambassador Edge Stack can authenticate incoming requests before routing them to a backing service.

As per our use case, we are configuring Ambassador Edge Stack to use an external third-party authentication service. Here we are using a sample authentication service application for implementing the authentication on the GKE Ingress level using Ambassador Edge Stack for the quote application we deployed earlier.

Deploying the Authentication Service Application

GitHub - datawire/ambassador-auth-service: Example auth service for Ambassador

Ambassador Edge Stack delegates the actual authentication logic to a third-party authentication service which :

• listens for requests on port 3000
• expects all URLs to begin with /extauth/
• performs HTTP Basic Auth only for the URLs starting with /backend/get-quote/ (other URLs are always permitted without authentication)
• accepts only user username, password password
• makes sure that the x-qotm-session the header is present, generating a new one if needed

Ambassador Edge Stack routes all requests through the authentication service. It relies on the auth service to distinguish between requests that need authentication and those that do not.

This sample authentication service takes all requests from the ambassador edge stack service but only performs or asks for authentication on /backend/get-quote/ path requests. Rest all requests on other paths will be permitted by auth service without authentication.

If the request gets unauthenticated due to an incorrect username or password then it will return a 404 User Unauthorised message.

If Ambassador Edge Stack cannot contact the auth service, it will return a 503 for the request.

It is important to have the auth service running before configuring Ambassador Edge Stack to use it.

kubectl apply -f auth-service.yaml

Configure Ambassador Edge Stack authentication

Once the auth service is running, we need to tell Ambassador Edge Stack about it.

The easiest way to tell Ambassador Edge Stack to use the authentication service that we deployed is using ambassador CRD called Filters and Filter Policy.

Filters and authentication | Edge Stack

Filters

Filters are used to extend the Ambassador Edge Stack to modify or intercept a request before sending it to your backend service. The most common use case for Filters is authentication, and Edge Stack includes a number of built-in filters for this purpose. Edge Stack also supports developing custom filters.

There are various types of Filters, as per our solution we are using External Filter.

The External Filter calls out to an external service speaking the ext_authZ protocol, providing a highly flexible interface to plug in your own authentication, authorization, and filtering logic.

kubectl apply -f filter.yaml

This configuration tells Ambassador Edge Stack about the Filter which has the type External which includes auth_service reference to the authentication service running on port 3000.

Filter Policy

Filters are managed using a FilterPolicy resource. The FilterPolicy resource specifies a particular host or URL to match, along with a set of filters to run when a request matches the host/URL.

The following filter policy would enable you to Filter requests to all hosts and paths.

The authentication service that we are using is only configured to perform authentication on requests to /backend/get-quote/ .

kubectl apply -f filter-policy.yaml

kubectl get filter && kubectl get filterpolicy

Test authentication

1. If we curl to a protected URL (/backend/get-quote/). We get a 401 since we haven’t authenticated by username and password.
2. Similarly for authenticating by the wrong username or password we get 401 unauthorized responses.

curl <GKE_INGRESS_DOMAIN>/backend/get-quote/  
                             //without passing username or password

curl -Lv -u username:password123 <GKE_INGRESS_DOMAIN>/backend/get-quote    
                              //passing wrong username or password

You can also test the API Authentication using Postman Tool.

Using Postman API Console :

Postman: The World's Leading API Platform | Sign Up for Free

3. If you authenticate to the protected URL (/backend/get-quote/) with the correct username and password, you will able to access the quote application microservice.

Note: The only valid credentials for the sample authentication service (example-auth:3000) we deployed are username:password.

ambassador-auth-service/server.js at master · datawire/ambassador-auth-service

curl -Lv -u username:password <GKE_INGRESS_DOMAIN>/backend/get-quote/ 

Using Postman API Console :

Accessing the quote application from the browser with a protected URL :

<GKE_INGRESS_DOMAIN>/backend/get-quote/

As the user hit the protected URL for accessing the quote application behind the scene ambassador first intelligently routes this request to the authentication service (example-auth:3000) and it will prompt the user for authentication.

After entering the correct username and password, the authentication service (example-auth) validates it with the username & password configured in its database and routes the request to quote application microservice after completion of end-user authentication.

But if any requests come on a path other than /backend/get-quote/ the requests are permitted without authentication to quote application as the authentication service which we deployed as example-auth only performs authentication on /backend/get-quote/ path.

curl <GKE_INGRESS_DOMAIN>/backend/

Conclusion

In the world of DevOps automation, many organizations moving their applications to microservice architecture. Managing the security for these applications includes multiple factors in DevOps such as Authentication or Authorization etc.

Using Ambassador API Gateway — Edge Stack solution we can set up Authentication on Kubernetes microservices APIs for end users by Integrating it with various kinds of third-party authentication providers such as SSO with Google, Keycloak, OIDC, Azure AD, and many more.

Ambassador API Gateway authentication makes applications or microservices unaware of the authentication part so that developers don't need to include authentication logic into the application itself as it's handled by a separate application or third-party authentication providers Integrated with edge stack API Gateway.

For AWS cloud-based deployments, AWS Provides the feature called API Gateway Lambda authorizers under AWS Lambda which can be extended according to the use case for authentication and authorization for microservices.

Use AWS Lambda authorizers with a third-party identity provider to secure Amazon API Gateway REST APIs | Amazon Web Services

Questions?

If you have any questions, I’ll be happy to read them in the comments.

For more Interesting Cloud & DevOps content follow me on Medium or LinkedIn.

Microservices Authentication Using Ambassador API Gateway on GKE was originally published in Searce on Medium, where people are continuing the conversation by highlighting and responding to this story.

This Article is based on how to implement Chaos Engineering Experiments Using Gremlin on Google Cloud.

Prerequisites

1. Google Cloud Platform Account
2. Gremlin Application Account
3. GKE Cluster

What Is Chaos Engineering?

Chaos engineering is a method of testing distributed software that deliberately introduces failure and faulty scenarios to verify its resilience in the face of random disruptions.

Chaos Engineering lets you compare what you think will happen to what actually happens in your systems. You literally “break things on purpose” to learn how to build more resilient systems.

What Is Gremlin?

Gremlin is a simple, safe, and secure way to improve the resilience of your systems by using Chaos Engineering to identify and fix failure modes.

Gremlin is a cloud-native platform that runs in any environment. Gremlin supports all public cloud environments — AWS, Azure, and GCP — and runs on Linux, Windows, and containerized environments like Kubernetes, and bare metal.

In 2010, Netflix introduced a technology to switch production software instances off randomly — like setting a monkey loose in a server room — to test how the cloud handled its services. Thus, the tool Chaos Monkey was born.

Chaos engineering matured at organizations such as Netflix, and gave rise to technologies such as Gremlin (2016), becoming more targeted and knowledge-based.

Now let's start Gremlin Chaos Engineering Experiments on Google Cloud as follows :

1. Gremlin Shutdown Attack on GKE Pods to validate the High Availability of the application
2. Gremlin CPU Attack on GKE Pods to validate GKE Horizontal Pod Autoscaling
3. Gremlin Blackhole Attack on GKE Load Balancer to validate High Availability of web application on GKE

As the part of prerequisites, we already signed up for the Gremlin Web App account.

To signup for a Gremlin Web App account → https://app.gremlin.com/signup

In order to perform various chaos experiments on GKE, we need to install the Gremlin Kubernetes Client in GKE so that we can create various attacks on GKE from the Gremlin web application.

Installing the Gremlin Client Agent on GKE Cluster

To install the Gremlin Kubernetes client, you will need your Gremlin Team ID and Secret Key. If you don’t know what your Team ID and Secret Key are, you can get them from the Gremlin web app.

Visit the Teams page in Gremlin, and then click on your team’s name in the list.

On the Teams screen click on Configuration.

If you don’t know your Secret Key, you will need to reset it. Click the Reset button. You’ll get a popup reminding you that any running clients using the current Secret Key will need to be configured with the new key. Hit Continue.

Next, you’ll see a popup screen that will show you the new Secret Key. Make a note of it.

Install the Gremlin Client with Helm

Getting Started > Installing Gremlin on Kubernetes with Helm | Gremlin Docs

Add the Gremlin Helm chart:

helm repo add gremlin https://helm.gremlin.com

Create a namespace for the Gremlin Kubernetes client:

kubectl create namespace gremlin

Next, you will run the helm command to install the Gremlin client. In this command, there are three placeholder variables that you will need to replace with real data. Replace $GREMLIN_TEAM_ID with your Team ID, and replace $GREMLIN_TEAM_SECRET with your Secret Key. Replace $GREMLIN_CLUSTER_ID with the name of GKE Cluster.

helm install gremlin gremlin/gremlin 
--namespace gremlin \
--set gremlin.hostPID=true \
--set gremlin.secret.managed=true \
--set gremlin.container.driver=docker-runc \
--set gremlin.secret.type=secret \
--set gremlin.secret.teamID=$GREMLIN_TEAM_ID \ 
--set gremlin.secret.clusterID=$GREMLIN_CLUSTER_ID \
--set gremlin.secret.teamSecret=$GREMLIN_TEAM_SECRET

Note: Here in the above helm installation gremlin. container. driver value may vary as per your cluster node container runtime. In my case, I have used GKE Nodes with the Ubuntu-docker image type.

To verify that the installation was successful, run this command:

kubectl get pods -n gremlin

The output should show one chao pod and one gremlin pod for each node in your cluster. These should all be in the Running state:

You can also check the status of the client from Gremlin Web Application

Setting Up Slack Integration with Gremlin Web App

We can also Integrate our Slack app with Gremlin for getting Attack reports and Notifications on the slack channel.

For setting up Slack Integration with Gremlin we have to add the slack channel In Gremlin Integration we created for getting Gremlin attacks notifications.

Now just select your slack channel where you want to receive Gremlin Notifications and Click Allow.

That’s all your Slack Integration with Gremlin Is now Ready.🎉🎉

Now we have the Gremlin Agent Installed on the GKE cluster with slack app notifications enabled.

Let us deploy one sample Nginx web application on the GKE cluster.

Deploying Nginx Application on GKE Cluster with HPA

1. Create a separate namespace for Nginx deployment.

kubectl create ns app

2. Deploying Nginx deployment in app namespace.

kubectl apply -f nginx.yaml

3. Exposing Nginx Deployment with GKE Load Balancer.

kubectl expose deployment myapp --type=LoadBalancer --port=80

4. Deploying Horizontal Pod Autoscalar for Nginx Deployment.

kubectl apply -f hpa.yaml

5. Check all resources deployed correctly or not using the below command.

kubectl get all -n app

We can now start performing various chaos experiments or attacks on Nginx sample deployment to validate various things like monitoring, alerting, scaling and availability, etc working or not.

Gremlin Attacks

An attack is a method of injecting failure into a system in a simple, safe, and secure way. Gremlin provides a range of attacks that you can run against your infrastructure. This includes impacting system resources, delaying or dropping network traffic, shutting down hosts, and more. In addition to running one-time attacks, you can also schedule regular or recurring attacks, create attack templates, and view attack reports.

Gremlin provides three categories of attacks:

• Resource attacks: test against sudden changes in consumption of computing resources
• Network attacks: test against unreliable network conditions
• State attacks: test against unexpected changes in your environment such as power outages, node failures, clock drift, or application crashes

We are going to perform one attack from each of these types of attacks on the Nginx Deployment to validate various aspects of our system.

Validating High Availability with Gremlin Shutdown Attack

Shutdown attacks let teams build resilience to host failures by testing how their applications and systems behave when an instance is no longer running.

Configuring Gremlin Attacks for Nginx Deployment

Go to Attacks Menu in Gremlin Web App Console → Select Kubernetes → Choose a Cluster and Namespace ( where the target app is present )

Select the target Deployment on which Gremlin performs the attack and define the blast radius for the attack.

Blast Radius: Number of hosts included or amount of target part in the attack

For Shutdown, Attacks choose the type of attack as Shutdown attack from State Attacks.

Click on unleash Gremlin to Run the particular Attack.

Running Gremlin Shutdown Attack

Click on attack details to see the attack details and output of the attack which Gremlin is running on Nginx Deployment Pods.

We can see that the Nginx Deployment is highly available even while running the shutdown attack due to more replicas.

kubectl get pods -n app -w

A shutdown Attack can answers questions such as:

• How long does it take for an instance to restart? Does my application successfully restart when the instance comes back online?
• Does my load balancer automatically reroute requests away from the failed instance? Do I have other instances available to handle these requests?
• If a user has an active session on an instance that fails, does the session gracefully continue on a different instance?
• Is there any data loss? Are ongoing processing jobs restarted?

Validating GKE Horizontal Pod Autoscaling with Gremlin

CPU attacks help you ensure that your application behaves as expected even when CPU capacity is limited or exhausted. CPU attacks can also help test and validate automatic remediation processes, such as auto-scaling and load balancing.

For CPU Attack choose the type of attack as the CPU attack from Resource Attacks and configure more details for the attack like attack duration time, target CPU capacity going to be impacted, etc.

Running Gremlin CPU Attack

Click on attack details to see the attack details and output of the attack which Gremlin is running on Nginx Deployment Pods.

we can validate whether the Horizontal Pod Autoscaling defined for CPU Utilization by Nginx Pods working or not during the attack.

kubectl get pods -n app -w

We can see that GKE HPA scaling up the Nginx Pods to count 4 as Max Count we defined is 4.

kubectl get hpa

In this way, by running CPU or Memory attacks we can validate the HPA defined for the application.

CPU attacks can answer questions such as:

• How is the user experience impacted when CPU resources are exhausted?
• Do I have monitoring and alerting in place to detect CPU spikes?
• Do I have quotas configured to limit CPU by application or process or container?
• Do I have cleanup scripts to get rid of corrupted threads?

Validating Nginx Web App High Availability with Gremlin Blackhole Attack on GKE Load Balancer

Blackhole attacks let you simulate these outages by dropping network traffic between services. This lets you uncover hard dependencies, test fallback, and failover mechanisms, and prepare your applications for unreliable networks.

For Blackhole Attack choose the type of attack as the Blackhole attack from Network Attacks and configure more details for the attack like Port details etc.

In our case, we have Nginx Deployment running on port 80.

Running Gremlin Blackhole Attack

Click on attack details to see the attack details and output of the attack which Gremlin is running on Nginx Deployment Pods.

We can check the Nginx web page status from the GCP Load Balancer External IP. There is no latency or blockage in terms of web experience due to high availability and GCP Load Balancing between multiple backend pods.

Blackhole attacks can answer questions such as:

• Where do dependencies exist within our system?
• Do we have monitoring in place to alert on the unavailability of each service?
• Does our application gracefully degrade if a dependency is unavailable?
• Is the user experience negatively affected when a downstream dependency is unavailable?
• Do we have dependencies that we think are non-critical, but can actually bring down our entire application?

For the notification part, we have configured the slack Integration with the Gremlin web app, by which we get all the notifications in the slack channel regarding the status of each Gremlin attack we perform on GKE.

Conclusion

The overarching goal of Chaos Engineering is to improve the reliability of our applications and systems by testing how they handle failure. To do this, we need to take a structured and well-organized approach with clearly defined objectives and key performance indicators (KPIs).

Running random experiments without any direction or oversight won’t yield actionable results and will put our systems at unnecessary risk.

To meet the high-level goal of improved reliability, we need to guide our Chaos Engineering adoption process using more granular objectives.

Questions?

If you have any questions, I’ll be happy to read them in the comments. Follow me on medium or LinkedIn.

Gremlin Chaos Engineering On Google Cloud was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

This Article is based on how to set up Jenkins Distributed Cluster in GKE with Dynamic Build Agents on Kubernetes Pods.

Problem Statement

We have our Jenkins Master setup built on a GKE Cluster environment from where the CICD Pipeline Jobs build and deploy the applications on another GKE Application cluster. The challenge over here is how we set up the CI environment for Jenkins to run its Jobs more efficiently.

Use Case

Let’s assume a scenario where we have the Jenkins Master deployed on GKE Cluster and applications are being deployed on another GKE Cluster which served as Application Cluster. By default, the Jenkins runs all its pipeline Jobs on the Master Node which results in more compute power consumption and less isolation between multiple Jobs as they running on the same environment.

Solution

The solution to the use case is building a Distributed Jenkins Cluster using Dynamic Agents. The use of Jenkins agents provides clustering functionality to Jenkins so that Jenkins can distribute its jobs to various slave nodes or agents on a need basis.

Prerequisites

1. Google Cloud Platform Account
2. Google Kubernetes Engine Clusters

What Is Distributed Jenkins Cluster?

Jenkins is one of the most important tools in DevOps. Jenkins is used in both Continuous Integration and Continous Deployment or Delivery stage of DevOps.

In many cases, you require testing your build jobs in different environments, which is not possible to handle with a single Jenkins server. The creation of a heavy project also requires more than a single server, because a single server can’t handle its load.

Jenkins Distributed Cluster allows you to add multiple Jenkins slaves under a central (main) Jenkins server. This Jenkins server is also known as Jenkins Master. These Jenkins slaves perform different tasks according to the instructions given by the Jenkins server. It also allows you to run your build jobs on a different operating system like Windows, macOS, Ubuntu, etc.

There are mainly two parts we need to configure according to our use case for setting up the Distributed Jenkins Cluster.

1. Configure Nodes → slave node configurations like windows, Linux, mac machine, etc.
2. Configure Clouds → dynamic agents configurations like Docker, Kubernetes, etc.

As the part of prerequisite, we have already set up two zonal Google Kubernetes Engine clusters for Jenkins and applications respectively as management and application clusters on GCP.

Creating a zonal cluster | Google Kubernetes Engine (GKE) | Google Cloud

1. Jenkins Management Cluster → For Jenkins Cluster Deployment

2. Application Cluster → For Application Deployments

Jenkins Server Deployment On Management GKE Cluster

There are typically two scenarios for Jenkins server deployment as follows :

1. Jenkins master and slave running inside the same Kubernetes cluster.
2. Jenkins master and slave running in separate Kubernetes cluster.

In our use case, we have both Jenkins Master and Slave or Agents running in the same GKE Cluster.

Let us start the setting up the Jenkins Server on the GKE management cluster as follows :

1. Create a namespace called jenkins .
2. Manifest stack for Jenkins Server Deployment :

Jenkins Deployment Manifest :

Jenkins Service Manifest :

Jenkins Ingress Manifest :

Note: Ensure that your Kubernetes cluster setup supports persistent volumes. If you deploy Jenkins without a persistent volume, you will lose the Jenkins data on every restart or pod deletion.

Jenkins Storage Class Manifest :

Jenkins PVC Manifest :

3. Apply all the above manifests in order to get the Jenkins Master running.

kubectl apply -f storage-class.yaml
kubectl apply -f pvc.yaml
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
kubectl apply -f ingress.yaml

You can also use the Jenkins helm chart to deploy the Jenkins using the helm stack on Kubernetes.

jenkins 4.1.8 · jenkins/jenkinsci

Now let's check to access the Jenkins Master or Server using Jenkins UI by the IP Address of Jenkins Ingress.

The Initial Admin password for Jenkins can be found in the Jenkins pods logs. To check the same run the below command :

kubectl logs <jenkins_pod_name> -n jenkins

Copy and paste the Initial Admin password from the above logs. Then Jenkins will prompt for new user creation and suggested plugin installation.

The Jenkins UI configuration part is now done and we are now good to use Jenkins CICD Tool.

Before moving to the configuration of Distributed Jenkins Cluster or Dynamic Agents on Kubernetes Pods let's understand how the workflow works.

Jenkins Dynamic Kubernetes Agents Workflow

1. Whenever the Jenkins job is triggered, the Jenkins Kubernetes plugin will make an API call to Kubernetes API Server to create a Kubernetes agent pod.
2. Then, the Jenkins agent pod gets deployed in the Kubernetes for running the Jenkins Job. As soon as the job has completed the agent pod got terminated automatically.
3. When the agent pod comes up, it uses the details in its environment variables and talks back to Jenkins using the JNLP method.
4. Using JNLP Protocol the slave gets connected to the Jenkins Master Node and the resultant architecture serves as Distributed Jenkins Cluster.

Now let us start the Jenkins Dynamic Agents or Slaves configuration for Kubernetes.

Jenkins Dynamic Kubernetes Pods Build Agents Configuration

Step 1: Install Jenkins Plugin for Kubernetes

Kubernetes

Go to Manage Jenkins –> Manage Plugins, search for Kubernetes Plugin in the available tab and install it.

After installing the plugin you need to restart the Jenkins server in order to enable the plugin.

Step 2: Kubernetes Cloud Configuration

Go to Manage Jenkins → Manage Nodes and Cloud → Configure Clouds → Add a new cloud then select Kubernetes.

Select Kubernetes Cloud Details.

Here we have two scenarios of configurations:

1. Jenkins master and slave running inside the same Kubernetes cluster.
2. Jenkins master and slave running in separate Kubernetes cluster.

Since we have Jenkins Master inside the same Kubernetes cluster with a default service account since no need to specify Kubernetes URL or Kubernetes Server Certificate Key.

If you have Jenkins Master outside the Kubernetes cluster, you have to specify Kubernetes URL or Kubernetes Server Certificate Key and add the respective service account token secret in the credentials section.

Note: Here we are using the default service account for Jenkins Master Pod. For Jenkins to communicate with the Kubernetes cluster, we need a service account token with permissions to deploy dynamic build agent pods in the Jenkins namespace.

If your service account doesn't have appropriate permissions you will get the below error for Jenkins master connection to the Jenkins namespace.

In order to resolve the above error, we need to create cluster-role-binding for the default service account attached to the Jenkins Master pod by granting a cluster-admin role. So that now Jenkins can able to launch Dynamic Agent pods in the Jenkins Namespace.

kubectl create clusterrolebinding jenkins — clusterrole cluster-admin — serviceaccount=jenkins:default

To validate the connection using the service account, use the Test connection button as shown below. It should show a connected message if the Jenkins pod can connect to the Kubernetes master API.

Step 3: Configure the Jenkins URL Details

For Jenkins master running inside the cluster, you can use the Service endpoint of the Kubernetes cluster as the Jenkins URL because agents pods can connect to the cluster via internal service DNS.

Note: If the Jenkins master runs outside the Kubernetes cluster, you can use the Jenkins IP or DNS in the Jenkins URL configuration.

Step 5: Create POD and Container Template

For the POD and container templates, you need to provide basic details for the Jenkins worker or slave build agent pods which will be dynamically launched by Jenkins Master in order to run the pipeline jobs.

The label slave will be used in the job as an identifier to pick this pod as the build agent.

Next, we need to add a container template for Dynamic Jenkins agent pods. Here specify the container name as JNLP only and the docker image URL for the agent image on which the Jenkins runs its jobs.

Note: If you don’t add a container template, the Jenkins Kubernetes plugin will use the default JNLP image from the Docker hub to spin up the agents. ie, Jenkins/inbound-agent

You can also define POD and Container Template in Jenkinsfile also as shown below as an example :

agent {
        kubernetes {
            yaml '''
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: shell
    image: ${image_name}
    command: ["sleep", "infinity"]
    env:
    - name: JAVA_HOME
      value: /root/.sdkman/candidates/java/current
    volumeMounts: 
    - name: "workspace-volume"
      mountPath: /
    - name: "volume-0"
      mountPath: /var/run/docker.sock
      readOnly: false
    securityContext:
      privileged: true
      runAsGroup: 0
      runAsUser: 0
  serviceAccountName: "jenkins-sa"
  volumes:
  - hostPath:
      path: "/var/run/docker.sock"
    name: "volume-0"
    
'''
            defaultContainer 'shell'
        }
    }

For our use case, we are going to execute a complete CICD Pipeline for the sample application from the build stage to deploy it to the application cluster. Since we need to build a custom docker image for Jenkins Build Agent which has preinstalled stack for all the dependencies of the CICD Pipeline

Dockerfile for Jenkins Dynamic Build Agent Pod :

Building the custom docker image for Jenkins Dynamic Build Agents :

docker build . -t asia-south-1-  docker.pkg.dev/PROJECT_ID/jenkins/jenkins-slave:v1

Pushing the custom docker image for Jenkins Dynamic Build Agents to Google Artifact Registry :

Push and pull images | Artifact Registry documentation | Google Cloud

gcloud auth configure-docker asia-south1-docker.pkg.dev

docker push docker.pkg.dev/PROJECT_ID/jenkins/jenkins-slave:v1

We can also add multiple container templates to the POD template and use them in the pipeline.

After the container template, we need to add volume details for the docker container of Jenkins dynamic agent pod as it requires docker inside docker because our pipeline has build and push application image stages.

Click Add Volume → Select Host Path Volume

For running docker inside docker we need to mount base Kubernetes node docker daemon socket on the Jenkins agent pod so that the agents can able to run docker-related jobs like building/pushing and creating containers etc.

For both Host and Mount Path provide the value as /var/run/docker.sock.

After the volume details, we need to provide service account details for Jenkins dynamic agent pods by which it can able to communicate with external services like Google Artifact Registry for pushing the images or GKE for deploying the applications, etc.

As part of Google's best practices instead of creating the service account keys and adding in Jenkins Credentials to attach and use it for agent pods, we can make use of workload identity.

Configuring service account for dynamic agent pods using Workload Identity :

Authenticate to Google Cloud APIs from GKE workloads | Google Kubernetes Engine (GKE)

1. Create a Kubernetes service account for the agent pods.

kubectl create serviceaccount jenkins-sa --namespace jenkins

2. Create an IAM service account for your agent pods or use an existing IAM service account instead.

gcloud iam service-accounts create jenkins-sa --project=PROJECT_ID

3. Ensure that your IAM service account has the roles you need. You can grant additional roles using the following command.

Note: In our use case we need two roles as for pushing the application images to Google Artifact Registry and deploying it to GKE Cluster.

gcloud projects add-iam-policy-binding PROJECT_ID --member "serviceAccount:jenkins-sa@PROJECT_ID.iam.gserviceaccount.com" --role "ROLE_NAME"

4. Allow the Kubernetes service account to impersonate the IAM service account by adding an IAM policy binding between the two service accounts. This binding allows the Kubernetes service account to act as the IAM service account.

gcloud iam service-accounts add-iam-policy-binding jenkins-
sa@PROJECT_ID.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member
"serviceAccount:PROJECT_ID.svc.id.goog[jenkins/jenkins-sa]"

5. Annotate the Kubernetes service account with the email address of the IAM service account.

kubectl annotate serviceaccount jenkins-sa --namespace jenkins
iam.gke.io/gcp-service-account=jenkins-sa@PROJECT_ID-
v1.iam.gserviceaccount.com

After the creation of the service account put the service account name and User ID as 0 in order to run Jenkins dynamic agent pod container as the root user to avoid any permission issues for running docker inside docker.

At last, click apply and save the POD and Container Template configuration. This is the basic cloud configuration required in order to work with Jenkins dynamic agent pods on Kubernetes.

Jenkins CICD Pipeline with Jenkins Distributed Cluster using Dynamic Build Agents

The proposed Jenkins CICD Pipeline is consist of multiple stages as follows:

1. Git Checkout → Application Code
2. Build and Tag Application Docker Image
3. Push the Application Docker Image to Google Artifact Registry
4. Updating the Deployment Manifest with New Application Image Tags
5. Deploy the Application to Application GKE Cluster

The groovy script for the Pipeline As A Code is as follows:

You can refer to the syntax of Jenkinsfile from the official documentation provided by the Jenkins community.

Pipeline Syntax

In the pipeline code first, we need to define the agent on which Jenkins master can run its jobs.

If you don't have an agent or you are running the Jobs on the Jenkins master itself then you can define as an agent none.

In our use case, we are going to use Kubernetes dynamic build agent pods to run the pipeline jobs. In order to define the same use Kubernetes as the agent and provide the label of the agent pod that we have set in the POD template as a slave.

For the rest of the pipeline code, we have defined the stages for various jobs.

Jenkinsfile

Application Code Stack

The sample application code being deployed on app GKE Cluster by Jenkins CICD is as follows:

main. go

Application Dockerfile

Application Deployment Manifests:

deployment.yaml

service.yaml

Let's start configuring Jenkins CICD Pipeline Job for the sample application deployment.

1. Click on Create a job

2. Select Type of Project as Pipeline and Enter the Project Name.

Note: If you don't have a pipeline in place and want to configure the Job using UI options you can choose Freestyle Project.

3. For the SCM Tool configuration of the Pipeline we need to connect Jenkins to the location where the application code is present eg Github so that Jenkins can able to clone or check out the code from the location.

In order to do so, we need to configure credentials for the application Github Repo by using which Jenkins can able to clone or check out the application code into agent pods.

Click Manage Jenkins → Manage Credentials → Add Credentials

Choose the Kind of credential as Username and Password. Before this, you need to create an access token for your Github which becomes your password for the application repo.

Note: If you have SSH Keys configured for Github then you can choose a kind of secret file.

For the username and password enter your Github username and access token as passwords respectively.

Set ID of the credentials by your preferred naming. In the pipeline, we call the credentials using its ID.

Click Save.

3. click on Configure to configure the pipeline.

In the last script section, we can pass the Jenkins Pipeline Code from the Jenkinsfile.

Here, if you have the Jenkinsfile already on your application git repo, you just need to give the path to Jenkinsfile.

We are going to add the Jenkinsfile Pipeline code in the script section itself as follows:

4. After configuring the Jenkins Job we can now run it. For running the Job Click on Build Now.

As soon as the Job starts building or running we can see the same in build history.

As you can see below our Job started building first time as build number 1.

5. Let's check the output of the Job by clicking on Job Build #1 and console output.

We can see that Jenkins has launched one build agent or slave Kubernetes pod dynamically using the Kubernetes plugin to run the pipeline.

In the output, we can see Jenkins showing the manifest that is used for the build agent pod from the POD template we configured earlier. It populates all the details we configured in the POD template in the build agent pods like environmental variables etc.

6. Meanwhile the Job started running we can quickly check the dynamic agent pod launched by Jenkins on the management GKE cluster in the Jenkins namespace.

Below you can see Jenkins has successfully launched the dynamic build agent Jenkins-slave for running the pipeline code.

As soon as the pipeline is executed completely, Jenkins will terminate the agent pod.

7. We can check the status of the pipeline briefly using the stage view option in Jenkins UI.

As you can see the Jenkins pipeline has been successfully executed using the power of dynamic build agent pods with all the defined stages.

8. Let's check the output of the defined pipeline stages.

1. Pushing Application Docker Image to Google Artifact Registry

2. Deploying the Application stack on application GKE Cluster ( in platform namespace )

kubectl get all -n platform

Let's check whether the application is working or not by accessing through LoadBalancer IP from the service that we have deployed.

The application got successfully deployed on the app cluster by Jenkins Dynamic Build Agent pods.

Conclusion

If you are using Jenkins & Kubernetes, you should definitely use the approach of dynamic build agent pods.

Running Jenkins agents on the Kubernetes pods will give you good build isolation for different application versions.

Also, an ephemeral Kubernetes pod-based Jenkins agent is a great way of reducing the cost of the CI environment as Jenkins agents get spun up only if there is a build request.

Scaling the Jenkins build agents with the Kubernetes cluster gives you much more flexibility in reducing the overhead of running out of resources for the Jenkins as this gives us distributed Jenkins cluster.

You can also use a static build VM as the Jenkins slave there also you can get options for dynamic builds but it's much slower than the dynamic build agents like containers.

Questions?

If you have any questions, I’ll be happy to read them in the comments. Follow me on medium or LinkedIn.

Jenkins Distributed Cluster Using Dynamic Build Agents On GKE was originally published in Searce on Medium, where people are continuing the conversation by highlighting and responding to this story.

Problem Statement

We have our Infrastructure setup built on a multi-cloud or hybrid cloud environment where we need to establish a networking stack that includes private endpoints connections from one cloud or on-premises server to another cloud server. The challenge over here is how to achieve this private DNS endpoint resolution from another network.

Use Case

Let’s assume a scenario where we have the Infrastructure setup built on AWS and GCP Cloud. There are some private GCP Endpoints managed in GCP Cloud DNS Private Zone and the applications deployed in AWS Cloud need access to this endpoint. By default, the AWS Route53 is not able to resolve GCP Cloud DNS Private endpoints.

Solution

The solution to such use cases is the approach of setting up DNS Query forwarders so that the DNS Servers of another cloud can able to resolve the private endpoints residing in different clouds. To solve the above scenario we can use AWS Outbound Endpoint service by AWS Route53.

Prerequisites

1. Amazon Web Services (AWS) Account
2. Google Cloud Platform (GCP) Account
3. High Availability Virtual Private Network (VPN) between GCP VPC and AWS VPC

As the part of prerequisite we have already set up High Availability VPN Setup with BGP Sessions configured between AWS Cloud and GCP Cloud as follow :

Create an HA VPN gateway to a peer VPN gateway | Google Cloud

Now let us create the private DNS zone in GCP for managing private endpoint DNS records.

What Is Google Cloud DNS?

Google Cloud DNS offers both public zones and privately managed DNS zones. A public zone is visible to the public internet, while a private zone is visible only from one or more Virtual Private Cloud (VPC) networks that you specify.

Cloud DNS overview | Google Cloud

Creating A Managed Google Cloud DNS Private Zone

1. In the Cloud Console, go to the Create a DNS zone page.
2. Go to Create a DNS zone.
3. For the Zone type, select Private type.
4. Enter a Zone name as per your choice.
5. Enter a DNS name suffix for the DNS private zone.
6. Optional: Add a description.
7. Under Options, select the Default (private) option.
8. Select the Virtual Private Cloud (VPC) networks to which the private zone must be visible. Only the VPC networks that you select are authorized to query records in the zone.
9. Finally Click Create.

To create a recordset, follow these steps:

1. In the Google Cloud Console, go to the Cloud DNS zones page.
2. Click the name of the GCP DNS managed zone that you want to add the record to.
3. On the Zone details page, click Add recordset option.
4. On the Create recordset page, in the DNS name field, enter the subdomain of the DNS zone. The trailing dot is automatically added at the end.
5. Select the Resource record type — for example, A.
6. In the TTL field, enter a numeric value for the resource record’s time to live, which is the amount of time that it can be cached. This value must be a positive integer.
7. From the TTL unit menu, select the unit of time.
8. Depending on the resource record type that you have selected, populate the remaining fields.
9. To enter additional information, click Add item.
10. Finally Click Create.

Now let us check if the AWS Route53 is able to resolve the Google Cloud DNS endpoints or not. For this, we have created one EC2 Instance In AWS VPC for testing this DNS connectivity working or not.

nslookup <domain_name>

As expected you can see that AWS Route53 DNS is unable to resolve the Google Cloud DNS endpoints as they are managed via a private zone.

Since for AWS Side, Route 53 Resolver automatically uses a Resolver on the VPC to answer DNS queries for local Amazon VPC domain names for EC2 instances and records in private hosted zones. For all other domain names, Resolver performs recursive lookups against public name servers.

To solve this problem and to make AWS Route53 able to resolve private endpoints from Google Cloud DNS we can use the AWS Route53 Resolver Outbound Endpoint Service.

What Is Amazon Route53 Resolver Endpoint?

Amazon Route53 Resolver helps to resolve AWS Private DNS endpoints or DNS records residing in the private hosted zone from the on-premise network or another cloud network using the feature of Route53 Resolver Endpoints.

Resolving DNS queries between VPCs and your network

There are two types of Route53 Resolver Endpoints:

1. Inbound Endpoint :

DNS resolvers on your network can forward DNS queries to Route 53 Resolver via this endpoint

This allows your DNS resolvers to easily resolve domain names for AWS resources such as EC2 instances or records in a Route 53 private hosted zone.

2. Outbound Endpoint:

Resolver conditionally forwards queries to resolvers on your network via this endpoint

This would enable your AWS resources to resolve the domain names of your on-prem network resources using resolver rules, which would forward selected queries to the on-prem DNS resolvers.

Before you start to forward queries, you need to create Resolver inbound and/or outbound endpoints in the connected VPC. These endpoints provide a path for inbound or outbound queries.

For our use case, we have to resolve Google Cloud DNS Private endpoints from AWS Route53. In order to resolve this, we can make use of Amazon Route53 Outbound Endpoint to start forwarding our Cloud DNS queries to Google Cloud DNS.

Before configuring the AWS Route53 Outbound Endpoint we have to configure the Inbound Server Policy for Google Cloud DNS to accept the queries from AWS Network.

What Is Google Cloud DNS Server Policy?

You can configure one DNS server policy for each Virtual Private Cloud (VPC) network. The policy can specify inbound DNS forwarding, outbound DNS forwarding, or both. Inbound server policy refers to a policy that permits inbound DNS forwarding. Outbound server policy refers to one possible method for implementing outbound DNS forwarding.

DNS server policies | Google Cloud

Google Cloud DNS Inbound server policy

By default, a VPC network’s name resolution services — through its name resolution order — are only available to that VPC network itself. We can create an inbound server policy in your VPC network to make these name resolution services available to an on-premises network that is connected using Cloud VPN or Cloud Interconnect.

When we create an inbound server policy, Cloud DNS takes an internal IP address from the primary IP address range of each subnet that your VPC network uses.

Creating Google Cloud DNS Inbound Server Policy

1. In the Google Cloud Console, go to the Cloud DNS zones page.
2. Switch to the DNS Server Policies section and click Create Policy.
3. Enter DNS Policy Name.
4. Optional: Add a description.
5. For the Logs select on or off as per your use case.
6. Set Inbound query forwarding on.
7. Optional: Add the alternate DNS Servers.
8. Add the VPC Network for which the DNS Server policy is to be configured.
9. Click Create.

Inbound Query Forwarding IP

Apply Cloud DNS server policies | Google Cloud

The Inbound Query Forwarding IP is nothing but the Entrypoint for the remote networks to resolve the Google Cloud DNS private endpoints.

When an inbound server policy applies to a VPC network, Cloud DNS creates a set of regional internal IP addresses that serve as destinations to which your on-premises systems or name resolvers can send DNS requests.

Configuring AWS Route53 Resolver Outbound Endpoint

Forwarding outbound DNS queries to your network

1. Open the Route 53 console.
2. In the navigation pane, choose Outbound endpoints.
3. On the navigation bar, choose the Region for the VPC where you want to create the outbound endpoint.
4. Choose to Create an outbound endpoint.
5. On the Create outbound endpoint page, complete the General settings for the outbound endpoint section. Choose a Security group that allows outbound TCP and UDP connectivity to the IP addresses and the ports that the resolvers use for DNS queries on your remote network.
6. Complete the IP addresses section. You can let the Resolver choose IP addresses for you from the available IP addresses in the subnet, or specify IP addresses yourself.

• Route tables that include routes to the IP addresses of the DNS resolvers on your remote network using AWS Direct Connect, a VPN connection, or a network address translation (NAT) gateway.
• Network access control lists (ACLs) that allow both UDP and TCP traffic to the IP addresses and the ports that the resolvers use for DNS queries on your remote network and from resolvers on destination port range 1024–65535.

7. (Optional) Complete the Tags section.

8. Finally, Choose Submit.

Now to tell the Route53 Resolver Outbound Endpoint their Destination/Target DNS Server ( In our case it is Google Cloud DNS ) to resolve the private endpoints or records in Google Cloud we need to configure the Resolver Rule for the same.

Managing forwarding rules

Configure a Resolver Rule

1. Open the Route 53 console.
2. Choose Rules from the Route 53 navigation pane.
3. On the navigation bar, choose the Region where the newly created outbound endpoint exists.
4. Choose Create rule.
5. On the Create rule page, complete the Rule for outbound traffic sections. For Rule Type, configure a Forward rule and associate it to the VPC where you’ll forward DNS queries to your remote network from. For the Outbound endpoint, choose the outbound endpoint that you just created.
   Note: The VPC that you associate this rule to doesn’t need to be the same VPC where you created the outbound endpoint.
6. Complete the IP addresses section. For IP addresses, specify the IP addresses of the DNS resolvers on your remote network. For Port, specify the ports that these resolvers use for DNS queries.
   Note: Resolver forwards any DNS query that matches this rule and originates from a VPC associated with this rule to the referenced outbound endpoint. As a result, these queries are forwarded to the target IP addresses you specify here.
7. Finally, Choose Submit.

In our case, the Target IP Address is Google Cloud Inbound Query Forwarding IP as shown below.

Now we have our complete DNS Query Forwarding Setup ready. Let us check now if Amazon Route53 is able to resolve the Google Cloud Private Endpoints or records In Google Cloud private hosted zone.

nslookup <domain_name>

Amazon Route53 is now able to resolve Google Cloud DNS Private Records or Endpoints by forwarding these DNS queries to Google Cloud DNS Using AWS Route53 Resolver Outbound Endpoint.

Creating an outbound endpoint doesn’t change the behavior of Resolver, it just provides a path to a location outside the AWS network to Resolver.

If you have a use case of building the Hybrid Networking between Google Cloud and AWS as part of DNS Forwarding from Google Cloud to AWS Cloud then you can go through PART-I of the Multi-Cloud Private DNS Forwarding Series as follow.

Multi Cloud Private DNS Forwarding

Conclusion

You can set up a Hybrid kind of Private DNS forwarding for the multi-cloud infrastructure or hybrid cloud environment where multiple applications or services need to connect private endpoints from different networks.

Questions?

If you have any questions, I’ll be happy to read them in the comments. Follow me on medium or LinkedIn.

Multi-Cloud Private DNS Forwarding PART II was originally published in Searce on Medium, where people are continuing the conversation by highlighting and responding to this story.

This Article is based on how to setup Multi Cloud Private DNS Forwarding between Google Cloud and AWS Cloud.

Problem Statement

We have our Infrastructure setup built on a multi-cloud or hybrid cloud environment where we need to establish a networking stack that includes private endpoints connections from one cloud or on-premises server to another cloud server. The challenge over here is how to achieve this private DNS endpoint resolution from another network.

Use Case

Let's assume a scenario where we have the Infrastructure setup built on AWS and GCP Cloud. There are some private AWS Endpoints managed in AWS Route 53 private hosted zone and the applications deployed in GCP Cloud need access to this endpoint. By default, the GCP Cloud DNS is not able to resolve AWS Route53 Private endpoints.

Solution

The solution to such use cases is the approach of setting up DNS Query forwarders so that the DNS Servers of another cloud can able to resolve the private endpoints residing in different clouds. To solve the above scenario we can use AWS Inbound endpoint service by AWS Route53.

Prerequisites

1. Amazon Web Services Account
2. Google Cloud Platform Account
3. High Availability Virtual Private Network (VPN) between GCP VPC and AWS VPC

What Is DNS Query Forwarding?

DNS forwarding is the process by which particular sets of DNS queries are forwarded to a designated server for resolution according to the DNS domain name in the query rather than being handled by the initial server that was contacted by the client. This process improves the network’s performance and resilience. It provides a way to resolve name queries both inside and outside of the network by passing on namespaces or resource records that aren’t contained in the zone of a local DNS server to a remote DNS server for resolution.

When a DNS server is configured to use a forwarder, if it can’t resolve a name query by using its local primary zone, secondary zone, or cache, it forwards the request to the designated forwarder instead of attempting to resolve it by using root hints.

As the part of prerequisite we have already set up High Availability VPN Setup with BGP Sessions configured between AWS Cloud and GCP Cloud as follow :

Create an HA VPN gateway to a peer VPN gateway | Google Cloud

1. AWS Cloud VPC

2. AWS Virtual Private Gateway

3. AWS Customer Gateway

4. AWS Site to Site VPN Connection

5. GCP Cloud VPC

6. GCP Cloud HA VPN Gateway with VPN Tunnels

7. GCP Peer VPN Gateway

8. GCP Cloud Router with BGP Sessions Configuration

Now let us create the private hosted zone in AWS Route53 for managing private endpoint DNS records.

What is Amazon Route53?

Amazon Route 53 is a highly available and scalable Domain Name System (DNS) web service. You can use Route 53 to perform three main functions in any combination: domain registration, DNS routing, and health checking.

What is Amazon Route 53?

What is a Private hosted zone in AWS Route 53?

A private hosted zone is a container that holds information about how you want Amazon Route 53 to respond to DNS queries for a domain and its subdomains within one or more VPCs that you create with the Amazon VPC service.

Working with private hosted zones

Creating Private Hosted Zone In Amazon Route53

Creating a private hosted zone

1. Sign in to the AWS Management Console.
2. If you’re new to Route 53, choose the Get started option
3. If you’re already using Route 53, choose Hosted zones in the navigation pane.
4. Click Create hosted zone.
5. In the Create private hosted zone pane, enter a domain name and, optionally, a comment.
6. In the Type list, choose the Private hosted zone.
7. In the VPC ID list, choose the VPC that you want to associate with the hosted zone.
8. Choose to Create hosted zone.

So we have created the Private Hosted Zone and attached it to our AWS VPC as follow :

Creating DNS Records In Amazon Route53 Private hosted zone

You can follow the below-given link which explains the procedure to create records using Amazon Route53 Console:

Creating records by using the Amazon Route 53 console

Let's create a DNS record (aws.onkar.dns) of type A mapped to AWS EC2 Instance Private IP that resides in AWS VPC as follow:

Now let us check if the GCP Cloud DNS is able to resolve the AWS Route53 DNS endpoints or not. For this, we have created one GCP Compute VM in GCP VPC for testing this DNS connectivity working or not.

nslookup <domain_name>

As expected you can see that GCP Cloud DNS is unable to resolve the AWS Route53 endpoints as they are managed via a private hosted zone.

Since for AWS Side, Route 53 Resolver automatically uses a Resolver on the VPC to answer DNS queries for local Amazon VPC domain names for EC2 instances and records in private hosted zones. For all other domain names, Resolver performs recursive lookups against public name servers.

To solve this problem and to make GCP Cloud DNS able to resolve private endpoints from Amazon Route53 we can use the approach of DNS Query Forwarding.

In order to implement DNS Query forwarding, we need to set up DNS Forwarders so that our DNS queries are forwarded to an intended destination DNS server which results in private DNS endpoints resolution.

To set up the DNS Forwarders AWS Route53 has the feature of Resolver Endpoints.

What Is Amazon Route53 Resolver Endpoint?

Amazon Route53 Resolver helps to resolve AWS Private DNS endpoints or DNS records residing in the private hosted zone from the on-premise network or another cloud network using the feature of Route53 Resolver Endpoints.

Resolving DNS queries between VPCs and your network

There are two types of Route53 Resolver Endpoints:

1. Inbound Endpoint :

DNS resolvers on your network can forward DNS queries to Route 53 Resolver via this endpoint

This allows your DNS resolvers to easily resolve domain names for AWS resources such as EC2 instances or records in a Route 53 private hosted zone.

2. Outbound Endpoint:

Resolver conditionally forwards queries to resolvers on your network via this endpoint

This would enable your AWS resources to resolve the domain names of your on-prem network resources using resolver rules, which would forward selected queries to the on-prem DNS resolvers.

Before you start to forward queries, you need to create Resolver inbound and/or outbound endpoints in the connected VPC. These endpoints provide a path for inbound or outbound queries.

For our use case, we have to resolve AWS Route53 Private endpoints from Google Cloud DNS. In order to resolve this, we can make use of Amazon Route53 Inbound Endpoint to start forwarding our Cloud DNS queries to Amazon Route53.

Before we create Amazon Route53 Inbound Endpoints, make sure you have the following prerequisites in place:

• Ensure that there’s network connectivity between your on-prem network and AWS via a VPN connection or Direct Connect.
• Enable DNS hostnames and resolutions in the DNS support attributes in the VPC in which you would create the endpoint.
• There’s at least a private hosted zone with the records you would like to resolve created there and attached to the VPCs with the resources that the records point to.
• For the inbound endpoint, you would need a security group with inbound rules that allow incoming traffic from the Google Cloud DNS IP addresses via the following port:
• TCP/UDP 53

Creating Amazon Route53 Resolver Inbound Endpoint

1. Go to the Route 53 console and click Inbound endpoints.
2. Click Create inbound endpoint.
3. Then enter a name for the endpoint and select a VPC through which all the inbound DNS queries will flow on the way to the Resolver. Then set the security group, which is mentioned in the prerequisites.

Note: The VPC in which you would create the endpoint should be attached to the private hosted zone.

4. Afterward, specify the IP addresses of the endpoint. To improve reliability, Resolver requires that you specify two IP addresses for DNS queries. It is recommended to span them across different availability zones.

Note: You can add more than two IPs if you wish.

5. We also need to keep in mind while choosing the subnet that has route tables that include routes to the IP addresses of the DNS resolvers on the GCP network in the VPN connection (Since we have the BGP session it is automatically done but if not we will need to modify the routes).

The Google Cloud DNS has an IP Range of 35.199.192.0/19 .

What Is Google Cloud DNS?

Google Cloud DNS offers both public zones and private managed DNS zones. A public zone is visible to the public internet, while a private zone is visible only from one or more Virtual Private Cloud (VPC) networks that you specify.

Cloud DNS overview | Google Cloud

Creating A Managed Google Cloud DNS Private Forwarding Zone

1. In the Cloud Console, go to the Create a DNS zone page.
2. Go to Create a DNS zone option.
3. For the Zone type, select Private type.
4. Enter a Zone name as per your choice.
5. Enter a DNS name suffix for the private zone. Note: The DNS name must be the same as the privately hosted zone name in AWS’s Route 53 (onkar.dns)
6. Under Options, select Forward queries to another server.
7. Select the networks to which the private zone must be visible.

8. To add the IPv4 addresses of a forwarding target, click Add item. Here we have to add the IP address that we got from Amazon Route53 Resolver Inbound Endpoint as the Destination DNS Server for Google Cloud DNS.

9. To force private routing to the forwarding target, under Private forwarding, select the Enable checkbox.

10. Finally click create and the Google Cloud DNS is now configured to forward the queries to Amazon Route53 Resolver Inbound Endpoint which results in AWS Private Endpoints resolution by Google Cloud DNS.

After creating Google Cloud DNS Private Forwarding Zone we have to modify Cloud router Routes as by default they have default advertisement mode of IP Ranges which includes IP Ranges of GCP VPC Network and subnet.

By changing the IP Ranges advertisement mode of Cloud Router that we have configured with BGP Sessions dynamically propagates the Google Cloud DNS IP Range Route in AWS VPC Route Tables.

Advertising Google Cloud DNS IP Range In Cloud Router

Advertising custom IP ranges | Cloud Router | Google Cloud

1. In the Google Cloud Console, go to the Cloud Routers page.
2. Go to Cloud Routers
3. Select the Cloud Router that contains the BGP session to update.
4. On the Router details page, select the BGP session to update.
5. On the BGP session details page, click edit.
6. For Routes, select Create custom routes.
7. Select the Advertise all subnets visible to the Cloud Router.
8. Select Add custom route to add an advertised route.
9. Configure the route advertisement:

• Source: Select Custom IP range.
• IP address range: Google Cloud DNS IP Range (35.199. 192.0/19)
• Description: Add a description for the custom route purpose, and then click Done.

10. After you finish adding custom routes, click Save.

In AWS Cloud under VPC Section we can see that the custom route that we have configured above in Google Cloud Router BGP Sessions dynamically propogated in AWS VPC Route table as follow:

Now we have our complete DNS Query Forwarding Setup ready. Let us check now if the Google Cloud DNS is able to resolve the AWS Private Endpoints or records In Route53 private hosted zone.

nslookup  <domain_name>

Google Cloud DNS is now able to resolve AWS Route53 Private Records or Endpoints by forwarding these DNS queries to Amazon Route53 Resolver Inbound Endpoint.

How AWS Route53 Resolver resolves DNS queries that originate from Google Cloud Network?

1. A web browser or another application on the Google Cloud network submits a DNS query for a domain name that you forwarded to Resolver.
2. Google Cloud DNS forwards the query to the IP addresses in your inbound endpoint.
3. The inbound endpoint forwards the query to AWS Route53 Resolver.
4. Resolver gets the applicable value for the domain name in the DNS query, either internally or by performing a recursive lookup against public name servers.
5. The resolver returns the value (typically an IPv4 IP address) to the inbound endpoint.
6. The inbound endpoint returns the value to the Google Cloud DNS.
7. The Google Cloud DNS Resolver returns the value to the application running on the Google Cloud Platform.

Creating an inbound endpoint doesn’t change the behavior of Resolver, it just provides a path from a location outside the AWS network to Resolver.

Conclusion

You can set up a Hybrid kind of Private DNS forwarding for the multi-cloud infrastructure or hybrid cloud environment where multiple applications or services need to connect private endpoints from different networks.

Questions?

If you have any questions, I’ll be happy to read them in the comments. Follow me on medium or LinkedIn.

Multi Cloud Private DNS Forwarding was originally published in Searce on Medium, where people are continuing the conversation by highlighting and responding to this story.

🔷 This Article is based on Terraform Automation for Google Kubernetes Engine Cluster Monitoring using Elastic Cloud Deployment for Elastic Search & Kibana 🔷

Prerequisites

1. An Elastic Cloud Account
2. Google Cloud Platform Account

What Is Elastic Cloud?

Elastic Cloud is a family of Elasticsearch SaaS offerings — including hosted Elasticsearch, hosted app search, and hosted site search. Elastic Cloud offers multiple managed services including Elastic Stack which includes Hosted Elasticsearch Cluster & Kibana as a Deployment on various public clouds like AWS | GCP, and Azure.

As Elastic Cloud provides managed Elasticsearch + Kibana Stack since we don’t require to deploy and Manage our own Elastic Stack. We just have to set up the Data shipping agents that ship the target system logs or metrics to the Elasticserach Cluster hosted on Elastic Cloud.

You can open your Elastic Cloud account free tier account which comes with 14 days free trial.

Elastic - The Search AI Company

What is Elastic Search?

Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. Elasticsearch is the central component of the Elastic Stack, a set of free and open tools for data ingestion, enrichment, storage, analysis, and visualization. Commonly referred to as the ELK Stack (after Elasticsearch, Logstash, and Kibana), the Elastic Stack now includes a rich collection of lightweight shipping agents known as Beats for sending data to Elasticsearch.

Elasticsearch: The Official Distributed Search & Analytics Engine | Elastic

What Is Kibana?

Kibana is a free and open frontend application that provides search and data visualization capabilities for data indexed in Elasticsearch. It is commonly included in Elastic Stack which is normally called ELK Stack which includes Elasticsearch, Logstash, and Kibana. Kibana also acts as the user interface for monitoring, managing, and securing an Elastic Stack cluster — as well as the centralized hub for built-in solutions developed on the Elastic Stack.

Kibana: Explore, Visualize, Discover Data | Elastic

What Is Beat or Data Shipping Agent?

Beats are open source data shippers that you install as agents on your servers to send operational data to Elasticsearch. Elastic provides Beats for capturing:

1. Auditbeat for Audit Data
2. Filebeat for log files
3. Metricbeat for metrics

Beats can send data directly to Elasticsearch or via Logstash, where you can further process and enhance the data, before visualizing it in Kibana.

Here I have used Filebeat as a Data Shipping Agent which collects the GKE cluster logs from the log files of respective resources and sends them to Elasticsearch deployed on Elastic Cloud and Visualized using Kibana. This is termed as EFK Stack ( Elasticsearch + Filebeat + Kibana ).

⚡ Logs → Filebeat → Elasticsearch → Kibana ⚡

You can also use Logstash | Fluented etc as a Data Shipping Agent Instead of Filebeat. Using these Lightweight Data shippers we can collect and send logs or metrics of various systems to Elastic Cloud.

Now after creating the account on Elastic Cloud we have to create Deployment which includes ( Elastic Search + Kibana ) Services which are deployed on Compute Virtual Machines of the chosen public cloud provider. We can configure the deployment according to our requirements we can set resources for Elastic Cloud Deployment.

Here I am using Terraform for Provisioning Elastic Cloud Deployment on Elastic Cloud.

As Elastic Cloud user has its own credentials for the elastic cloud deployment which are required for login and provisioning the deployment. Since I have used GCP Secret Manager to store the credentials of elastic cloud securely.

You can also use different secret managers like Hashicorp Vault.

Secret Manager documentation | Google Cloud

There is a total of 3 secrets being created for storing Elastic Cloud API key | Elastic Cloud Deployment Password & Elastic Cloud ID.

Terraform Code for creating the secrets | setting up accessors for the secret and storing the credentials of Elastic Cloud by creating secret versions →

providers.tf

backend.tf

main.tf

variables.tf

Now run the Terraform code by basic steps of executing Terraform Code

terraform init

terraform plan

terraform apply

We can see that secrets created successfully by Terraform.

Terraform Code for provisioning Elastic Cloud Deployment →

providers.tf

backend.tf

main.tf

Here we are accessing the API key required for Elastic Cloud Login from the GCP Secret we created using Terraform Data resource.

variables.tf

Now run the Terraform code by basic steps of executing Terraform Code

terraform init

terraform plan

terraform apply

The Elastic Cloud Deployment successfully created by Terraform. We can check through Elastic Cloud Console.

Elastic Cloud Deployment

Elastic Cloud Deployment comes with various services managed by Elastic Cloud which includes Elasticsearch Cluster | Kibana & Enterprise Search with lots of various features.

Kibana Dashboard

Elasticsearch Cluster

Elasticsearch API Console

Elasticsearch Database records can be accessed using API but Elasticcloud provides Elasticsearch Cluster with an API console through which we can easily use API queries for Elasticsearch.

REST APIs | Elasticsearch Guide [8.11] | Elastic

When you create the Deployment manually in Elastic Cloud you get a username (elastic) and password for Elasticsearch which you can download. Now we have created the Deployment using Terraform so you need to reset the Elasticsearch password through the console and store that password in the GCP secret we created using Terraform.

Reset the elastic user password | Elasticsearch Service Documentation | Elastic

Google Kubernetes Engine

Kubernetes - Google Kubernetes Engine (GKE) | Google Cloud

For connecting Google Kubernetes Engine GKE Cluster with Elastic Cloud we need to Deploy any of the Data Shipping Agent which can send the GKE cluster logs to Elasticsearch on Elastic Cloud.

Here I have used Filebeat as Data Shipping Agent which can collect the containers logs ( Kubernetes Resources ) from the Log File and send them to Elasticsearch on Elastic Cloud.

Filebeat Reference

For Monitoring GKE Cluster we have to Deploy Filebeat as a Daemon set that collects logs from each GKE Nodes and sends them to Elasticsearch.

Filebeat Manifest →

Filebeat Daemon set

Filebeat ConfigMap

Filebeat Service Account

Filebeat Role

Filebeat Role Binding

Terraform Code for Provisioning GKE Cluster & Deploying Filebeat as a Daemon set for Monitoring GKE Using Elastic Cloud

providers.tf

backend.tf

main.tf

Using Data Resource provided by Terraform we can access the secret value i.e Elastic Cloud ID & Password and that passed using sed command to filebeat manifests which updates the Elasticsearch cluster details to Filebeat deployment.

variables.tf

Now run the Terraform code by basic steps of executing Terraform Code

terraform init

terraform plan

terraform apply

Here we can see that terraform updating the kubeconfig file in the local system by GKE Cluster credentials to connect through local system & Deploying Filebeat Daemon Set on GKE Cluster.

GCP Infrastructure Provisioned by Terraform →

GCP VPC

GCP Subnet

GCP IAP Firewall

Google Kubernetes Engine GKE Cluster

GKE Node Pool

Filebeat Daemon Set on GKE Cluster

Connecting to GKE Cluster from Local system using Updated kubeconfig by Terraform

Here I am using kubectx command to switch the kubeconfig context between multiple clusters. Using kubectx we can switch to multiple clusters by updating the context of kubeconfig done by kubectx.

GitHub - ahmetb/kubectx: Faster way to switch between clusters and namespaces in kubectl

kubectx

You can see that kubectx showing the kubeconfig active cluster as ‘my-test-k8s’ cluster provisioned using Terraform which clear that Terraform successfully updated the kubeconfig file.

$ kubectl get nodes

$ kubectl get ds -n kube-system

$ kubectl describe ds filebeat -n kube-system

Terraform has successfully updated the Login Details taken from GCP Secret Manager i.e Password & Cloud ID of Elasticsearch in Filebeat configmap.

$ELASTICSERACH_PASSWORD & $ELASTIC_CLOUD_ID values updated as shown below in Filebeat Daemonset Description

As shown above, the Filebeat configmap is mounted on the /var/log directory of GKE Nodes such that Filebeat is now configured for Log file as an Input from GKE Cluster.

filebeat.inputs:   
- type: container     
  paths:        
    - /var/log/containers/*.log      
  processors:        
    - add_kubernetes_metadata:          
        host: ${NODE_NAME}          
        matchers:          
          - logs_path:              
               logs_path: "/var/log/containers/"

We can also check that Filebeat taking Input from the/var/log/containers directory from GKE Nodes. For that, I have set up IAP Firewall Rule so that we can SSH into GKE Nodes and can see the Filebeat Input.

After going inside GKE Node we can see the Filebeat Input as .log files listed inside /var/log/containers.

Filebeat takes all these log files as Input and sends them to Elasticsearch hosted on Elastic Cloud.

$ kubectl get pods -n kube-system

$ kubectl logs <filebeat-pod-name> -n kube-system

Filebeat logs showing that the Data Pipeline has started and filebeat successfully connected to Elasticsearch on Elastic Cloud and started collecting and sending logs to Elasticsearch.

As filebeat now started sending GKE Cluster Logs to Elastic Cloud. To visualize GKE Cluster Logs Using Kibana first we need to create Elasticsearch Index Pattern by going on Elastic Cloud Console.

Create an index pattern | Kibana Guide [7.14] | Elastic

Provide Index pattern name as filebeat-* which matches the index sources already created in Elasticsearch which includes GKE logs sent by Filebeat.

Now filebeat Index Pattern created for Kibana by which Kibana can access the GKE Logs from Elasticsearch and visualize it.

We can also verify that there is Index created for Filebeat in Elasticsearch Database. Using API Console we can query and check the Filebeat Index.

GKE Cluster Logs Visualization Using Kibana →

In the Kibana Console go inside the Observability section then select filebeat index pattern there, you can see that Live data Kibana fetching live data from Elasticsearch.

GKE Cluster Logs →

You can also use different queries to access data from Elasticsearch in Kibana using Kibana Query Language KQL.

Using Visualize Library provided by Kibana we can visualize GKE Cluster logs in detail.

Now for testing the Monitoring of the GKE Cluster by Elastic Cloud I have launched one Nginx pod In GKE Cluster and check whether the logs of the Nginx Pod coming to Elastic Cloud or Not.

$ kubectl run myapp --image=nginx

Filebeat has successfully collected and sent ‘myapp’ pod logs to Elastic Cloud as shown below.

GKE Nodes as a Hosts In Kibana →

In this way, all K8S resources that launched inside GKE Cluster are monitored by Elastic Cloud which is Integrated with GKE Cluster Using Terraform Automation.

This complete Integrated Infrastructure of GKE Cluster and Elastic Cloud is termed as ⚡ EFK Stack ( Elasticsearch + Filebeat + Kibana ) ⚡

✨ GKE Cluster → Filebeat → Elastic Cloud → Elasticsearch + Kibana ✨

Conclusion

You can also Integrate Elastic Cloud with any platform by using Elastic Cloud Agents provided by Elastic Cloud. Elastic Cloud also gives you the facility of Uploading your Data directly to Elasticsearch Cluster which can be further be visualized Using Kibana.

Reference Links

Elastic Cloud: https://www.elastic.co/guide/index.html

Elasticsearch: https://www.elastic.co/elasticsearch/service

Filebeat: https://www.elastic.co/guide/en/beats/filebeat/current/index.html

Kibana: https://www.elastic.co/guide/en/kibana/current/get-started.html

Terraform: https://registry.terraform.io/providers/elastic/ec/latest/docs

GCP Docs: https://cloud.google.com/docs

🔷 Terraform Code for the Complete Infrastructure from GKE Cluster to Elastic Cloud Integration Using Filebeat Deployment →

GitHub - Onkar179/EFK-Stack-with-Elastic-Cloud-and-Integrating-with-GKE-Using-Terraform

Questions?

If you have any questions, I’ll be happy to read them in the comments. Follow me on medium or LinkedIn.

Automated Logging Solution Using EFK Stack on GCP was originally published in Searce on Medium, where people are continuing the conversation by highlighting and responding to this story.