Pssst! I started my own blog!
You can read this very same article over there too!
No paywall, no ad, no Javascript — no bullshit, just pure content.
See you there!

Saying goodbye to old systems can sound scary, but for us at EGYM, migrating from Elastic Container Service (ECS) to Elastic Kubernetes Service (EKS) was a journey of growth and excitement.

🎬 Hi there, I’m Jean!

In this blog post, I’ll share the inside scoop on our successful migration from ECS to EKS, why we did it and how, complete with the magic formula for zero downtime!

Starting with the Why

Let’s kick it off with a famous phrase in engineering:

If it ain’t broke, don’t fix it.

Although this holds true in most cases, there are some situations where the costs of keeping the lights on for “what isn’t broken”, can no longer be justified.

Originally, EGYM’s entire infrastructure was hosted on a single Cloud, Google Cloud Platform (GCP), and all of our workloads were running in Google Kubernetes Engine (GKE). This held true until EGYM acquired a company, which made us embrace multi-Cloud by adding 3 ECS clusters hosted on Amazon Web Services (AWS).

While the merger itself was more or less successful from a business perspective (depending who you ask), maintaining, coaching and hiring engineers with the necessary skills to work with 2 Clouds, fundamentally different yet doing pretty much the same thing, became challenging and most importantly: costly.

With Kubernetes being Cloud-agnostic by design, we soon realized there could be potential for porting most of the core components we already used in GKE, to EKS. This fact alone made the case for moving away from ECS towards EKS much easier to sell to upper management.

Very soon, we all unanimously agreed in the SRE team that migrating to EKS was the way forward, and that we needed to evaluate its feasibility.

Evaluating Feasibility with a Proof of Concept (PoC)

One of the main goals of this PoC, was to design an EKS cluster from the ground up with all the core infrastructure components that our micro-services would need to serve traffic reliably (load balancing, auto-scaling, monitoring, etc).

From all the technical choices we had to make, deciding which Load Balancer to pick was by far the most challenging. Ingress NGINX seemed like an obvious candidate for its Cloud-agnostic design, yet AWS Load Balancer Controller also caught our eye for its native integration with AWS Load Balancers.

AWS Load Balancer Controller is not a Load Balancer per se, but the Controller that provisions them. One can either provision a Network Load Balancer (NLB) using the Service annotations or an Application Load Balancer (ALB) using the Ingress annotations.

Unless explicitly configured otherwise, AWS Load Balancer Controller will provision an AWS Load Balancer per Service/Ingress resource, which can be a driver for higher costs.

Ingress NGINX is fundamentally different in the fact that it does not only act as a Controller, but also as an L7 Load Balancer: it is responsible for both satisfying Ingress rules and load balancing HTTP traffic to the corresponding Services.

While AWS Load Balancer Controller can be used to provision both ALBs (L7 Load Balancers) and NLBs (L4 Load Balancers), Ingress NGINX needs to be itself load balanced by an L4 Load Balancer (NLB), before taking over and load balance traffic on the application layer.

Testing Staging Workloads in EKS

As soon as our EKS clusters architecture was designed and showcased in a PoC, it was time to port the staging workloads to a new shiny EKS test cluster and make sure they function equally well in ECS and EKS.

While this part was mostly smooth sailing, here are a few issues that we faced along the way:

• Many of our micro-services hosted on AWS are Java services, which are using the AWS SDK for Java to interact with AWS services. On paper this should not be a problem however, the authentication method used in EKS (AssumeRoleWithWebIdentity) is different than in ECS (AmazonEC2ContainerServiceforEC2Role) and sometimes, based on how the SDK was utilized or whether the AWS STS package was found in the classpath, the default authentication chain provider would choose the wrong authentication method in EKS and fail to authenticate.
• Some of our micro-services perform asynchronous processing (also known as event messaging), which meant we had to ensure each event consumption was done in an idempotent operation or else be at risk of data duplication/corruption during the migration.

Once the staging workloads were successfully deployed and tested, it was time to migrate the production workloads to EKS as well!

Deploying Production Workloads to EKS

To ensure maximum availability during the migration, we decided to run all our workloads twice: one deployment remained in the existing ECS production cluster and a new replica set was deployed in the new EKS production cluster.

This approach had the following advantages:

• Isolated Testing: Since our ECS cluster was already serving user traffic, it was easy to deploy and test all production services via a separate domain and Ingress in the EKS cluster without affecting users.
• Blue-Green Deployment: By far the biggest advantage of running the EKS cluster along side the ECS one, was the ability to route user traffic to EKS gradually and to quickly roll it back to ECS in case things went south.

However, running things twice also had some drawbacks:

• Added Costs: Duplicating the replica sets also meant doubling the costs for a short period of time, which was a conscious investment we made to guarantee high availability during the entirety of the migration.
• Increased Connections: With twice more workloads, twice more connections were opened to our data infrastructure such as MySQL databases, Redis instances and ActiveMQ brokers, which meant raising the connection limit temporarily in some cases.

Eventually, all EGYM’s production workloads were deployed to EKS and tested successfully, in isolation from the services deployed in ECS.

The next step was to execute our rollout plan, which consisted of 3 phases:

1. Enabling Asynchronous Processing in EKS
2. Routing User Traffic Gradually to EKS
3. Scaling down Workloads in ECS

Enabling Asynchronous Processing in EKS

While deploying EGYM’s micro-services to EKS, we took the precautionary measure of disabling asynchronous processing (PubSub, SQS/SNS, etc.) for all workloads deployed in EKS.

This way, we ensured all upstream connections were ready before we would enable event processing gradually, giving us plenty of time to observe and ensure events were processed properly.

Once asynchronous processing was enabled on all replica sets in EKS, it was finally time for the most exciting yet also treacherous part of the migration: routing user traffic.

Routing User Traffic Gradually to EKS

In order to gradually pour traffic in the new EKS cluster, we took advantage of 2 routing mechanisms:

• CloudFront Staging Distribution
  With most of our AWS traffic being served on CloudFront’s edges, one easy and powerful way of routing traffic to EKS gradually, was using CloudFront Staging Distribution.
  Simply put, a Staging Distribution is like a blue-green deployment but for Content Delivery Networks (CDN): when deploying a new proxy configuration, one can choose to deploy it only to a subset of users and roll it out gradually to observe the change while minimizing risks.
  The advantage of a Staging Distribution is that it is directly deployed on the edges, meaning that rolling back in case of an incident is almost instant.
  The drawback on the other hand, is that it is currently only possible to route up to 15% of user traffic to a Staging Distribution before promoting it to a Production Distribution.
• Route 53 Weighted Routing
  This is where Route 53 Weighted Routing comes in: for routing more than 15% of user traffic to EKS, we had to temporarily route user traffic away from the CDN, and directly towards the NLB.
  With Weighted Routing, one has complete control of how much traffic is sent to either one of the origins however, the Time To Live (TTL) of the record (60 seconds for A records) must be taken into account when rolling back with DNS.

By exploiting both CloudFront Staging Distribution and Route 53 Weighted Routing routing mechanisms, we were able to gradually transfer traffic from ECS to EKS while benefiting from the incredible rollback speed of CloudFront for the first 15% of routed traffic, thus ensuring the highest availability possible.

We proceeded with a gradual rollout of roughly 2 weeks, starting at 1%, followed by 5%, 10%, 15%, 25%, 35%, 50%, 65%, 80% and finally 100%.

For the first 15%, traffic routed to EKS was entirely served on CloudFront’s edges, and above the 15%, weighted routing was handled on DNS level via Route 53.

Once arrived at 80% of traffic routed to EKS, the last bump to 100% was achieved, not by routing traffic exclusively from Route 53, but by promoting the CloudFront Staging Distribution to Production Distribution instead and transfer all user traffic back to CloudFront.

As more traffic entered the EKS cluster, we were able to cautiously observe the Horizontal Pod Autoscalers and ensure all deployments were scaling properly as the load increased.

Ultimately, with the EKS cluster reporting stable and the availability of the services left intact, we were ready to proceed with the ECS scale-down.

Scaling down Workloads in ECS

Sunsetting workloads in ECS was the final moment of truth, leading to the fateful question: did we miss anything?

As we scaled down each service in ECS one by one, we paid special attention to asynchronous processing and event messaging until…

“The last ECS Task, a digital phoenix battling resource constraints, choked on its final heartbeat, its containerized spirit evaporating into the cloud.”
Gemini, February 2024

Acknowledgements

I would like to thank Kolawole for helping me in carrying this project to its completion, Marty for trusting me in leading the project, as well as Viktor, Serhii, Roma and Eugene for their continued support throughout the migration.

That’s it! I hope this experience was valuable to you, if so:

1. Let me know in the comments below! 👇
2. Don’t forget to hit that subscribe button! ✅
3. Follow me on Twitter, I’ll be happy to answer any of your questions and you’ll be the first ones to know when a new article comes out! 👌

Bye-bye! 👋

Farewell, ECS! Hello, EKS: A Kubernetes Migration Story was originally published in EGYM Software Development on Medium, where people are continuing the conversation by highlighting and responding to this story.

Pssst! I started my own blog!
You can read this very same article over there too!
No paywall, no ad, no Javascript — no bullshit, just pure content.
See you there!

In Kubernetes, we usually think about the Horizontal Pod Autoscaler (HPA) when referring to autoscaling. In most cases, it will be the preferred way of scaling services, based on CPU usage, memory usage, or custom metrics.

If you haven’t already, go read Horizontal Pod Autoscaler in Kubernetes (Part 1) — Simple Autoscaling using Metrics Server and learn how to implement a Horizontal Pod Autoscaler using Metrics Server!

However, while HPA can scale up and down replicas based on the current load, it is not capable of optimizing resource usage over the long term: This is where the Vertical Pod Autoscaler (VPA) comes in.

The VPA can be leveraged to optimize resource usage over time, based on mid to long-term observation.

Please note that to avoid a race condition, the VPA should only be used together with HPAs that are based on custom metrics. In addition, the VPA should not be used with JVM-based services due to limited visibility into the actual memory usage of the workload (learn more about its limitations here).

If you haven’t already, go read Horizontal Pod Autoscaler in Kubernetes (Part 2) — Advanced Autoscaling using Prometheus Adapter and learn how to implement a Horizontal Pod Autoscaler using Prometheus Adapter!

🎬 Hi there, I’m Jean!

In this article, we’re going to learn how to use Vertical Pod Autoscaler (VPA) to vertically scale services in Kubernetes automatically based on resource metrics! 💪

Requirements

Before we start, make sure you have the following tools installed:

• Kind
• Kubectl
• Helm
• K6

Note: for MacOS users or Linux users using Homebrew, simply run: brew install kind kubectl helm k6

All set? Let’s go! 🏁

Creating Kind Cluster

Kind is a tool for running local Kubernetes clusters using Docker container “nodes”. It was primarily designed for testing Kubernetes itself, but may be used for local development or CI.

I don’t expect you to have a demo project in handy, so I built one for you.

git clone https://github.com/jhandguy/vertical-pod-autoscaler.git

cd vertical-pod-autoscaler

Alright, let’s spin up our Kind cluster! 🚀

➜ kind create cluster --image kindest/node:v1.27.3 --config=kind/cluster.yaml

Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾

Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

Installing cert-manager

cert-manager is a Kubernetes addon that automates the management and issuance of TLS certificates from various issuing sources. It ensures certificates are valid and up to date periodically, and attempts to renew certificates at an appropriate time before expiry.

cert-manager can be installed via its Helm chart.

helm repo add jetstack https://charts.jetstack.io

helm install jetstack/cert-manager --name-template cert-manager --create-namespace -n cert-manager --values kind/cert-manager-values.yaml --version 1.13.2 --wait

If everything went fine, you should be able to see three newly spawned Deployments with the READY state!

➜ kubectl get deploy -n cert-manager
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
cert-manager              1/1     1            1           6m27m
cert-manager-cainjector   1/1     1            1           6m27m
cert-manager-webhook      1/1     1            1           6m27m

Installing NGINX Ingress Controller

NGINX Ingress Controller is one of the many available Kubernetes Ingress Controllers, which acts as a load balancer and satisfies routing rules specified in Ingress resources, using the NGINX reverse proxy.

NGINX Ingress Controller can be installed via its Helm chart.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

helm install ingress-nginx/ingress-nginx --name-template ingress-nginx --create-namespace -n ingress-nginx --values kind/ingress-nginx-values.yaml --version 4.8.3 --wait

Now, if everything goes according to plan, you should be able to see the ingress-nginx-controller Deployment running.

➜ kubectl get deploy -n ingress-nginx
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx-controller   1/1     1            1           4m35s

Installing Metrics Server

Metrics Server is a source of container resource metrics, which collects them from Kubelets and exposes them in Kubernetes API server through Metrics API for use by Horizontal Pod Autoscaler and Vertical Pod Autoscaler.

Metrics Server can be installed via its Helm chart.

helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server

helm install metrics-server/metrics-server --name-template metrics-server --create-namespace -n metrics-server --values kind/metrics-server-values.yaml --version 3.11.0 --wait

Now, if everything goes well, you should see a metrics-server Deployment running.

➜ kubectl get deploy -n metrics-server
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   1/1     1            1           3m21s

Installing Vertical Pod Autoscaler

Vertical Pod Autoscaler (VPA) is a component of the Kubernetes Autoscaler that frees users from the necessity of setting up-to-date resource limits and requests for the containers in their pods.

When configured, it will set the requests automatically based on usage and thus allow proper scheduling onto nodes so that the appropriate resource amount is available for each pod. It will also maintain ratios between limits and requests that were specified in the initial container configuration.

It can both down-scale pods that are over-requesting resources, and also up-scale pods that are under-requesting resources based on their usage over time.

Note: VPA is still in its beta phase, using it for production is at your own risk.

As of this writing, Kubernetes does not provide an official Helm chart, so I went ahead and built one!

helm install helm-chart --name-template vertical-pod-autoscaler --create-namespace -n vertical-pod-autoscaler --wait

If everything goes fine, you should eventually see three Deployments with the READY state!

➜ kubectl get deploy -n vertical-pod-autoscaler
NAME                               READY UP-TO-DATE AVAILABLE AGE
vert...scaler-admission-controller 1/1   1          1         2m32s
vert...scaler-recommender          1/1   1          1         2m32s
vert...scaler-updater              1/1   1          1         2m32s

As you can observe, the VPA is split into 3 separate components:

• The Recommender computes the recommended resource requests for pods based on historical and current usage of the resources. The current recommendations are then put in the status of the VPA resource, where they can be inspected;
• The Updater decides which pods should be restarted based on resource allocation recommendations calculated by Recommender. If a pod should be updated, Updater will try to evict the pod. It respects the pod disruption budget, by using the Eviction API to evict pods. Updater does not perform the actual resources update but relies on Admission Controller to update pod resources when the pod is recreated after eviction.
• The Admission Controller will get a request from the API server for each pod creation and will either decide there’s no matching VPA configuration or find the corresponding one and use the current recommendation to set resource requests in the pod.

Configuring Vertical Pod Autoscaler

Now that the Vertical Pod Autoscaler is up and running, let’s get to it, shall we? 🧐

helm install sample-app/helm-chart --name-template sample-app --create-namespace -n sample-app --wait

If everything goes fine, you should eventually see one Deployment with the READY state.

➜ kubectl get deploy -n sample-app
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
sample-app   3/3     3            3           58s

Alright, now let’s have a look at the VPA!

➜ kubectl describe vpa -n sample-app
...
Spec:
  Resource Policy:
    Container Policies:
      Container Name:  sample-app
      Controlled Resources:
        cpu
        memory
      Max Allowed:
        Cpu:     100m
        Memory:  200Mi
      Min Allowed:
        Cpu:     10m
        Memory:  20Mi
  Target Ref:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         sample-app
  Update Policy:
    Update Mode:  Auto

As you can see, this VPA is configured to scale the service based on its CPU and memory resources. Its spec states that the minimum allowed CPU/memory is 10m/20Mi and the maximum is 100m/200Mi.

Finally, its update policy is in “Auto” mode, meaning that VPA assigns resource requests on pod creation as well as updates them on existing pods using the preferred update mechanism.

Currently, both the resource requests and limits are matching the VPA’s minimum allowance.

➜ kubectl get pods -n sample-app -o yaml | grep -A 6 'resources:'
      resources:
        limits:
          cpu: 10m
          memory: 20Mi
        requests:
          cpu: 10m
          memory: 20Mi
--
      resources:
        limits:
          cpu: 10m
          memory: 20Mi
        requests:
          cpu: 10m
          memory: 20Mi
--
      resources:
        limits:
          cpu: 10m
          memory: 20Mi
        requests:
          cpu: 10m
          memory: 20Mi

Now, let’s give some load to our service and see what happens!

For Load Testing, I really recommend k6 from the Grafana Labs team. It is a dead-simple yet super powerful tool with very extensive documentation.

See for yourself!

k6 run k6/script.js

While k6 is gradually increasing strain on the pods' CPU usage, let’s watch out for any EvictedByVPA events in a second tab: eventually, you should see all 3 pods get evicted simultaneously!

➜ kubectl get events -n sample-app -w | grep EvictedByVPA
... Pod was evicted by VPA Updater to apply resource recommendation.
... Pod was evicted by VPA Updater to apply resource recommendation.
... Pod was evicted by VPA Updater to apply resource recommendation.

As soon as this happens, have a look at the updated pods’ resource requests and limits: the CPU/memory requests/limits should have doubled in value (CPU from 10m to 20m and memory from 10Mi to 20Mi).

➜ kubectl get pods -n sample-app -o yaml | grep -A 6 'resources:'
      resources:
        limits:
          cpu: 20m
          memory: "20971520"
        requests:
          cpu: 20m
          memory: "20971520"
--
      resources:
        limits:
          cpu: 20m
          memory: "20971520"
        requests:
          cpu: 20m
          memory: "20971520"
--
      resources:
        limits:
          cpu: 20m
          memory: "20971520"
        requests:
          cpu: 20m
          memory: "20971520"

This means Vertical Pod Autoscaler successfully evicted the pods in order to increase their resource requests and limits! 🎉

Once k6 is done, have a look at the Load Test summary and the result of the status code counter metric in particular.

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: k6/script.js
     output: -

  scenarios: (100.00%) 1 scenario, 200 max VUs, 10m30s max duration (incl. graceful stop):
           * load: Up to 40.00 iterations/s for 10m0s over 3 stages (maxVUs: 200, gracefulStop: 30s)

     ✗ status code is 200
      ↳  97% — ✓ 17724 / ✗ 365
     ✗ node is kind-control-plane
      ↳  97% — ✓ 17724 / ✗ 365
     ✗ namespace is sample-app
      ↳  97% — ✓ 17724 / ✗ 365
     ✗ pod is sample-app-*
      ↳  97% — ✓ 17724 / ✗ 365

   ✓ checks.........................: 97.98% ✓ 70896     ✗ 1460
     data_received..................: 4.2 MB 7.1 kB/s
     data_sent......................: 2.1 MB 3.5 kB/s
     http_req_blocked...............: avg=18.39µs  min=2µs   med=8µs    max=2.93ms  p(90)=17µs     p(95)=20µs
     http_req_connecting............: avg=5.71µs   min=0s    med=0s     max=2.74ms  p(90)=0s       p(95)=0s
   ✓ http_req_duration..............: avg=188.13ms min=491µs med=4.57ms max=59.99s  p(90)=269.43ms p(95)=646.78ms
       { expected_response:true }...: avg=129.55ms min=491µs med=4.62ms max=7.3s    p(90)=261.24ms p(95)=602.03ms
     http_req_failed................: 2.01%  ✓ 365       ✗ 17724
     http_req_receiving.............: avg=98.96µs  min=0s    med=75µs   max=4.27ms  p(90)=159µs    p(95)=209µs
     http_req_sending...............: avg=49.92µs  min=7µs   med=34µs   max=14.39ms p(90)=72µs     p(95)=93µs
     http_req_tls_handshaking.......: avg=0s       min=0s    med=0s     max=0s      p(90)=0s       p(95)=0s
     http_req_waiting...............: avg=187.99ms min=451µs med=4.38ms max=59.99s  p(90)=269.31ms p(95)=646.69ms
     http_reqs......................: 18089  30.147771/s
     iteration_duration.............: avg=188.59ms min=666µs med=5.11ms max=1m0s    p(90)=270.13ms p(95)=647.49ms
     iterations.....................: 18089  30.147771/s
     vus............................: 0      min=0       max=145
     vus_max........................: 200    min=200     max=200

running (10m00.0s), 000/200 VUs, 18089 complete and 0 interrupted iterations
load ✓ [======================================] 000/200 VUs  10m0s  00.41 iters/s

Uh-oh… It looks like we had some downtime! 😱

Thankfully, our service was able to restart relatively fast and 365 out of 17724 requests failed. But for a service with a slower startup time, this could have led to an incident! 🚨

This is due to the fact that vertical scaling, in essence, cannot happen without a restart: a pod’s CPU and/or memory cannot be increased in place. Instead, the pod must be terminated and a new one created with increased resources.

So how do we ensure the availability of our service during vertical autoscaling then?

This is where the Pod Disruption Budget (PDB) comes in!

We’ll get to that in a minute, let’s uninstall our Helm release first! (we won’t be needing this one anymore)

helm uninstall sample-app -n sample-app

Configuring Pod Disruption Budget

To prevent downtimes during pod disruption such as the one we previously experienced, a Pod Disruption Budget (PDB) can be configured.

A PDB limits the number of pods that are down simultaneously from voluntary disruptions. It can be configured to sustain either a minimum amount of available pods (minAvailable), or a maximum amount of unavailable pods (maxUnavailable).

Let’s see what happens if we try to autoscale vertically the same application but with a Pod Disruption Budget with maxUnavailable: 1.

helm install sample-app/helm-chart --name-template sample-app --set podDisruptionBudget.enabled=true --create-namespace -n sample-app --wait

Once again, you should eventually see one Deployment with the READY state.

➜ kubectl get deploy -n sample-app
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
sample-app   3/3     3            3           32s

Alright, now let’s have a look at the PDB!

➜ kubectl describe pdb -n sample-app
Name:             sample-app
Namespace:        sample-app
Max unavailable:  1
Selector:         app=sample-app
Status:
    Allowed disruptions:  1
    Current:              3
    Desired:              2
    Total:                3

As you can see, this PDB is configured to prevent more than 1 pod to be unavailable during a voluntary pod disruption (such as pod eviction by VPA).

Now, let’s see how our service is going to behave under load with a PDB!

k6 run k6/script.js

Once again, while k6 is gradually increasing strain on the pods’ CPU usage, let’s watch out for any EvictedByVPA events in a second tab: eventually, you should see all 3 pods get evicted but this time, only one by one!

➜ kubectl get events -n sample-app -w | grep EvictedByVPA
... Pod was evicted by VPA Updater to apply resource recommendation.
... Pod was evicted by VPA Updater to apply resource recommendation.
... Pod was evicted by VPA Updater to apply resource recommendation.

Once k6 is done, have a look at the Load Test summary and the result of the status code counter metric in particular.

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: k6/script.js
     output: -

  scenarios: (100.00%) 1 scenario, 200 max VUs, 10m30s max duration (incl. graceful stop):
           * load: Up to 40.00 iterations/s for 10m0s over 3 stages (maxVUs: 200, gracefulStop: 30s)

     ✓ status code is 200
     ✓ node is kind-control-plane
     ✓ namespace is sample-app
     ✓ pod is sample-app-*

   ✓ checks.........................: 100.00% ✓ 72356     ✗ 0
     data_received..................: 4.2 MB  7.0 kB/s
     data_sent......................: 2.1 MB  3.5 kB/s
     http_req_blocked...............: avg=19.7µs   min=2µs      med=7µs    max=2.95ms  p(90)=16µs     p(95)=21µs
     http_req_connecting............: avg=6.86µs   min=0s       med=0s     max=2.24ms  p(90)=0s       p(95)=0s
   ✓ http_req_duration..............: avg=103.43ms min=452µs    med=6.4ms  max=5.29s   p(90)=259.88ms p(95)=484.22ms
       { expected_response:true }...: avg=103.43ms min=452µs    med=6.4ms  max=5.29s   p(90)=259.88ms p(95)=484.22ms
     http_req_failed................: 0.00%   ✓ 0         ✗ 18089
     http_req_receiving.............: avg=99.34µs  min=8µs      med=77µs   max=5.42ms  p(90)=166µs    p(95)=212µs
     http_req_sending...............: avg=51.86µs  min=9µs      med=33µs   max=18.76ms p(90)=71µs     p(95)=96.59µs
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s     max=0s      p(90)=0s       p(95)=0s
     http_req_waiting...............: avg=103.28ms min=418µs    med=6.2ms  max=5.29s   p(90)=259.8ms  p(95)=483.99ms
     http_reqs......................: 18089   30.148343/s
     iteration_duration.............: avg=103.9ms  min=625.62µs med=6.98ms max=5.29s   p(90)=260.24ms p(95)=485.04ms
     iterations.....................: 18089   30.148343/s
     vus............................: 0       min=0       max=69
     vus_max........................: 200     min=200     max=200

running (10m00.0s), 000/200 VUs, 18089 complete and 0 interrupted iterations
load ✓ [======================================] 000/200 VUs  10m0s  00.41 iters/s

Yay! 🎉
This time, our service handled pod disruption beautifully and not a single request failed!

Thanks to the Pod Disruption Budget, a pod can only be evicted if all other pods are up, ensuring that at least 2 pods are available to handle the traffic.

This is what we call: High availability! 🚀

Wrapping up

That’s it! You can now stop and delete your Kind cluster.

kind delete cluster

To summarize, using Vertical Pod Autoscaler (VPA) we were able to:

• Autoscale vertically our service, based on resource metrics;
• Prevent downtime during pod eviction thanks to Pod Disruption Budget.

Was it worth it? Did that help you understand how to implement Vertical Pod Autoscaler in Kubernetes?

If so:

1. Let me know in the comments below! 👇
2. Don’t forget to hit that subscribe button! ✅
3. Follow me on Twitter, I’ll be happy to answer any of your questions and you’ll be the first ones to know when a new article comes out! 👌

Bye-bye! 👋

Vertical Pod Autoscaler in Kubernetes was originally published in EGYM Software Development on Medium, where people are continuing the conversation by highlighting and responding to this story.

Learn how to use Prometheus Adapter to horizontally scale services in Kubernetes automatically based on Prometheus metrics.

Pssst! I started my own blog!
You can read this very same article over there too!
No paywall, no ad, no Javascript — no bullshit, just pure content.
See you there!

The Horizontal Pod Autoscaler (HPA) is a fundamental feature of Kubernetes. It enables automatic scale-up and scale-down of containerized applications based on CPU usage, memory usage, or custom metrics.

Traditionally, when scaling software, we first think of vertical scaling: the CPU and the RAM are increased so the application consuming them can perform better. While this seems like a flawless mechanism on paper, it actually comes with many drawbacks.

First, upgrading the CPU or RAM on a physical machine (or VM) requires downtime and unless a Pod Disruption Budget (PDB) is used to handle disruptions, all pods will be evicted and recreated in the new resized node.

Nodes resource usage is also not optimized, as scaling vertically means requiring sufficient resources in a single node, while horizontal scaling may have the same amount of resources distributed across multiple nodes.

Additionally, vertical scaling is not as resilient as horizontal scaling, as fewer replicas mean higher risks of disruptions in case of node failure.

Finally, reaching a certain threshold, scaling only vertically becomes very expensive and most importantly, isn’t limitless. In fact, there is only so much CPU and RAM a physical machine(or VM) alone can handle.
This is where horizontal scaling comes into play!

Eventually, it is more efficient to duplicate an instance, than increase its resources.

🎬 Hi there, I’m Jean!

In this 2 parts series, we’re going to explore several ways to scale services horizontally in Kubernetes, and the second one is…
🥁
… using Prometheus Adapter! 🎊

Requirements

Before we start, make sure you have the following tools installed:

• Kind
• Kubectl
• Helm
• K6

Note: for MacOS users or Linux users using Homebrew, simply run: brew install kind kubectl helm k6

All set? Let’s go! 🏁

Creating Kind Cluster

Kind is a tool for running local Kubernetes clusters using Docker container “nodes”. It was primarily designed for testing Kubernetes itself, but may be used for local development or CI.

I don’t expect you to have a demo project in handy, so I built one for you.

git clone https://github.com/jhandguy/horizontal-pod-autoscaler.git

cd horizontal-pod-autoscaler

Alright, let’s spin up our Kind cluster! 🚀

➜ kind create cluster --image kindest/node:v1.27.3 --config=kind/cluster.yaml

Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾

Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

Installing NGINX Ingress Controller

NGINX Ingress Controller is one of the many available Kubernetes Ingress Controllers, which acts as a load balancer and satisfies routing rules specified in Ingress resources, using the NGINX reverse proxy.

NGINX Ingress Controller can be installed via its Helm chart.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

helm install ingress-nginx/ingress-nginx --name-template ingress-nginx --create-namespace -n ingress-nginx --values kind/ingress-nginx-values.yaml --version 4.8.3 --wait

Now, if everything goes according to plan, you should be able to see the ingress-nginx-controller Deployment running.

➜ kubectl get deploy -n ingress-nginx
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx-controller   1/1     1            1           4m35s

Installing Prometheus

Prometheus can be installed via its community Helm chart, which also provides Grafana out of the box.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm install prometheus-community/kube-prometheus-stack --name-template prometheus --create-namespace -n prometheus --version 54.2.2 --wait

If everything went fine, you should be able to see three newly spawned deployments with the READY state!

➜ kubectl get deploy -n prometheus
NAME                                  READY   UP-TO-DATE   AVAILABLE   
prometheus-grafana                    1/1     1            1           
prometheus-kube-prometheus-operator   1/1     1            1           
prometheus-kube-state-metrics         1/1     1            1

Installing Prometheus Adapter

Prometheus Adapter is a source of custom metrics, which collects them from Prometheus and exposes them in Kubernetes API server through Metrics API for use by Horizontal Pod Autoscaler and Vertical Pod Autoscaler.

Unlike Metrics Server which is limited to resource metrics, Prometheus Adapter exposes custom metrics measurable from within a Pod’s container, such as memory usage, GC duration, but also request throughput, latency, etc.

If you haven’t already, go read Horizontal Pod Autoscaler in Kubernetes (Part 1) — Simple Autoscaling using Metrics Server and learn how to implement a Horizontal Pod Autoscaler using Metrics Server!

Prometheus Adapter can be installed via its Helm chart.

helm install prometheus-community/prometheus-adapter --name-template prometheus-adapter --create-namespace -n prometheus-adapter --values kind/prometheus-adapter-values.yaml --version 4.9.0 --wait

Now, if all goes well, you should see the prometheus-adapter Deployment running with the READY state.

➜ kubectl get deploy -n prometheus-adapter
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
prometheus-adapter   1/1     1            1           52s

Autoscaling Services based on Prometheus Metrics

Now that Prometheus Adapter is up and running, let’s get to it, shall we? 🧐

helm install golang-sample-app/helm-chart --name-template sample-app --create-namespace -n sample-app --set prometheus.enabled=true --wait

If everything goes fine, you should eventually see one Deployment with the READY state.

➜ kubectl get deploy -n sample-app
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
sample-app   2/2     2            2           28s

Alright, now let’s have a look at the HPA!

➜ kubectl describe hpa -n sample-app
...
Metrics: ( current / target )
  "golang_sample_app_requests_per_second" on pods:  <unknown> / 10
Min replicas:                                       2
Max replicas:                                       8

As you can see, this HPA is configured to scale the service based on requests per second (rps), with an average of 10.

Note: as you probably have noticed, the current value for the request throughput is unknown, this is expected as the service hasn’t served any requests yet, thus no timeseries for this metric are present in Prometheus.

This means that as soon as the request throughput breaches the 10rps threshold, the HPA will trigger an upscale.

Under minimal load, the HPA will still retain a replica count of 2, while the maximum amount of Pods the HPA is allowed to spin up under high load is 8.

Note: in a production environment, it is recommended to have a minimum replica count of at least 3, to guarantee maintained availability in the case of Pod Disruption.

Now, this is the moment you’ve certainly expected… It’s Load Testing time! 😎

For Load Testing, I really recommend k6 from the Grafana Labs team. It is a dead-simple yet super powerful tool with very extensive documentation.

See for yourself!

k6 run k6/script.js

While the load test is running, I suggest watching the HPA in a separate tab.

kubectl get hpa -n sample-app -w

As the load test progresses and the 2 starting Pods start to handle more than 10rps each, you should see the Prometheus metric’s target increasing, and ultimately, the replica count reaching its maximum!

Deployment/sample-app   <unknown>/10   2         8         2
Deployment/sample-app   4360m/10       2         8         2
Deployment/sample-app   6236m/10       2         8         2
Deployment/sample-app   12471m/10      2         8         2
Deployment/sample-app   15577m/10      2         8         3
Deployment/sample-app   21231m/10      2         8         3
Deployment/sample-app   21231m/10      2         8         5
Deployment/sample-app   16752m/10      2         8         5
Deployment/sample-app   18362m/10      2         8         5
Deployment/sample-app   15921m/10      2         8         6
Deployment/sample-app   15543m/10      2         8         6
Deployment/sample-app   16530m/10      2         8         8
Deployment/sample-app   15397m/10      2         8         8
Deployment/sample-app   14428m/10      2         8         8
Deployment/sample-app   11897m/10      2         8         8
Deployment/sample-app   12370m/10      2         8         8
Deployment/sample-app   12423m/10      2         8         8
Deployment/sample-app   12414m/10      2         8         8
Deployment/sample-app   12423m/10      2         8         8
Deployment/sample-app   10962m/10      2         8         8

Now, let’s quickly have a look at the Load Test summary and the result of the http_req_duration metric in particular!

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: k6/script.js
     output: -

  scenarios: (100.00%) 1 scenario, 100 max VUs, 5m30s max duration (incl. graceful stop):
           * load: Up to 100.00 iterations/s for 5m0s over 2 stages (maxVUs: 100, gracefulStop: 30s)

     ✓ status code is 200
     ✓ node is kind-control-plane
     ✓ namespace is sample-app
     ✓ pod is sample-app-*

   ✓ checks.........................: 100.00% ✓ 60356     ✗ 0
     data_received..................: 3.5 MB  12 kB/s
     data_sent......................: 1.7 MB  5.8 kB/s
     http_req_blocked...............: avg=18.43µs min=1µs     med=8µs    max=3.9ms  p(90)=17µs    p(95)=20µs
     http_req_connecting............: avg=5.41µs  min=0s      med=0s     max=3.7ms  p(90)=0s      p(95)=0s
   ✓ http_req_duration..............: avg=16.74ms min=498µs   med=2.77ms max=1.48s  p(90)=14.52ms p(95)=64.78ms
       { expected_response:true }...: avg=16.74ms min=498µs   med=2.77ms max=1.48s  p(90)=14.52ms p(95)=64.78ms
     http_req_failed................: 0.00%   ✓ 0         ✗ 15089
     http_req_receiving.............: avg=97.15µs min=9µs     med=74µs   max=3.33ms p(90)=151µs   p(95)=197µs
     http_req_sending...............: avg=48.11µs min=6µs     med=34µs   max=3.12ms p(90)=69µs    p(95)=86µs
     http_req_tls_handshaking.......: avg=0s      min=0s      med=0s     max=0s     p(90)=0s      p(95)=0s
     http_req_waiting...............: avg=16.59ms min=452µs   med=2.61ms max=1.48s  p(90)=14.32ms p(95)=64.69ms
     http_reqs......................: 15089   50.297179/s
     iteration_duration.............: avg=17.18ms min=610.5µs med=3.21ms max=1.48s  p(90)=15.08ms p(95)=65.33ms
     iterations.....................: 15089   50.297179/s
     vus............................: 0       min=0       max=18
     vus_max........................: 100     min=100     max=100

running (5m00.0s), 000/100 VUs, 15089 complete and 0 interrupted iterations
load ✓ [======================================] 000/100 VUs  5m0s  000.65 iters/s

As you can observe, our service has performed very well under heavy load, with a Success Share of 100%, a median latency of ~3ms, and a 95th percentile latency of ~50ms!

We have the HPA to thank for that, as it scaled the Deployment from 2 to 8 Pods swiftly and automatically, based on request throughput!

We definitely would not have had the same results without an HPA… Actually, why don’t you try it yourself? 😉

Just delete the HPA (kubectl delete hpa sample-app -n sample-app), run the load test again (k6 run k6/script.js) and see what happens! (spoiler alert: it’s not pretty 😬)

Wrapping up

That’s it! You can now stop and delete your Kind cluster.

kind delete cluster

To summarize, using Prometheus Adapter we were able to:

• Autoscale horizontally our service, based on Prometheus metrics.

Was it worth it? Did that help you understand how to implement Horizontal Pod Autoscaler in Kubernetes using Prometheus Adapter?

If so:

1. Let me know in the comments below! 👇
2. Don’t forget to hit that subscribe button! ✅
3. Follow me on Twitter, I’ll be happy to answer any of your questions and you’ll be the first ones to know when a new article comes out! 👌

Bye-bye! 👋

Horizontal Pod Autoscaler in Kubernetes (Part 2) — Advanced Autoscaling using Prometheus Adapter was originally published in EGYM Software Development on Medium, where people are continuing the conversation by highlighting and responding to this story.

Learn how to use Metrics Server to horizontally scale native and JVM services in Kubernetes automatically based on resource metrics.

Pssst! I started my own blog!
You can read this very same article over there too!
No paywall, no ad, no Javascript — no bullshit, just pure content.
See you there!

The Horizontal Pod Autoscaler (HPA) is a fundamental feature of Kubernetes. It enables automatic scale-up and scale-down of containerized applications based on CPU usage, memory usage, or custom metrics.

Traditionally, when scaling software, we first think of vertical scaling: the CPU and the RAM are increased so the application consuming them can perform better. While this seems like a flawless mechanism on paper, it actually comes with many drawbacks.

First, upgrading the CPU or RAM on a physical machine (or VM) requires downtime and unless a Pod Disruption Budget (PDB) is used to handle disruptions, all pods will be evicted and recreated in the new resized node.

Nodes resource usage is also not optimized, as scaling vertically means requiring sufficient resources in a single node, while horizontal scaling may have the same amount of resources distributed across multiple nodes.

Additionally, vertical scaling is not as resilient as horizontal scaling, as fewer replicas mean higher risks of disruptions in case of node failure.

Finally, reaching a certain threshold, scaling only vertically becomes very expensive and most importantly, isn’t limitless. In fact, there is only so much CPU and RAM a physical machine(or VM) alone can handle.
This is where horizontal scaling comes into play!

Eventually, it is more efficient to duplicate an instance, than increase its resources.

🎬 Hi there, I’m Jean!

In this 2 parts series, we’re going to explore several ways to scale services horizontally in Kubernetes, and the first one is…
🥁
… using Metrics Server! 🎊

Requirements

Before we start, make sure you have the following tools installed:

• Kind
• Kubectl
• Helm
• K6

Note: for MacOS users or Linux users using Homebrew, simply run: brew install kind kubectl helm k6

All set? Let’s go! 🏁

Creating Kind Cluster

Kind is a tool for running local Kubernetes clusters using Docker container “nodes”. It was primarily designed for testing Kubernetes itself, but may be used for local development or CI.

I don’t expect you to have a demo project in handy, so I built one for you.

git clone https://github.com/jhandguy/horizontal-pod-autoscaler.git

cd horizontal-pod-autoscaler

Alright, let’s spin up our Kind cluster! 🚀

➜ kind create cluster --image kindest/node:v1.27.3 --config=kind/cluster.yaml

Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾

Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

Installing NGINX Ingress Controller

NGINX Ingress Controller is one of the many available Kubernetes Ingress Controllers, which acts as a load balancer and satisfies routing rules specified in Ingress resources, using the NGINX reverse proxy.

NGINX Ingress Controller can be installed via its Helm chart.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

helm install ingress-nginx/ingress-nginx --name-template ingress-nginx --create-namespace -n ingress-nginx --values kind/ingress-nginx-values.yaml --version 4.8.3 --wait

Now, if everything goes according to plan, you should be able to see the ingress-nginx-controller Deployment running.

➜ kubectl get deploy -n ingress-nginx
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx-controller   1/1     1            1           4m35s

Installing Metrics Server

Metrics Server is a source of container resource metrics, which collects them from Kubelets and exposes them in Kubernetes API server through Metrics API for use by Horizontal Pod Autoscaler and Vertical Pod Autoscaler.

Metrics Server can be installed via its Helm chart.

helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server

helm install metrics-server/metrics-server --name-template metrics-server --create-namespace -n metrics-server --values kind/metrics-server-values.yaml --version 3.11.0 --wait

Now, if everything goes according to plan, you should be able to see the metrics-server Deployment running.

➜ kubectl get deploy -n metrics-server
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   1/1     1            1           38s

Horizontal Pod Autoscaler

Before we dive in, let’s quickly remind ourselves of what a Horizontal Pod Autoscaler in Kubernetes actually is:

A HorizontalPodAutoscaler (HPA for short) automatically updates a workload resource (such as a Deployment or StatefulSet), with the aim of automatically scaling the workload to match demand.

Horizontal scaling means that the response to increased load is to deploy more Pods. This is different from vertical scaling, which for Kubernetes would mean assigning more resources (for example: memory or CPU) to the Pods that are already running for the workload.

If the load decreases, and the number of Pods is above the configured minimum, the HorizontalPodAutoscaler instructs the workload resource (the Deployment, StatefulSet, or other similar resource) to scale back down.

Now that we know what an HPA is, let’s get started, shall we? 🧐

Autoscaling Native Services based on CPU and Memory Usage

A native service is a piece of software that does not require a virtual environment in order to run across different OSs and CPU architectures. This is the case for C/C++, Golang, Rust, and more: those are languages that compile into a binary, that is directly executable by the Pod.
This means that native services can utilize all of the CPU and memory available from the Pod they run in, without an intermediary environment.

Let’s try it out with a Golang service!

helm install golang-sample-app/helm-chart --name-template sample-app --create-namespace -n sample-app --wait

If everything goes fine, you should eventually see one Deployment with the READY state.

➜ kubectl get deploy -n sample-app
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
sample-app   2/2     2            2           44s

Once the Pods are running, Metrics Server will start collecting the Pods resource metrics from the node’s Kubelet and expose them in Kubernetes through Metrics API for use by Horizontal Pod Autoscaler and Vertical Pod Autoscaler.

Let’s see what the resource usage for those Pods currently is!

➜ kubectl top pods -n sample-app
NAME                          CPU(cores)   MEMORY(bytes)
sample-app-6bcbfc8b49-j6xmq   1m           1Mi
sample-app-6bcbfc8b49-wtd8g   1m           1Mi

Pretty low, right? 🤔
This is obviously expected since our Go service currently isn’t handling any load.

Alright, now let’s have a look at the HPA!

➜ kubectl describe hpa -n sample-app
...
Metrics: ( current / target )
  resource memory on pods (as a percentage of request):  8% / 50%
  resource cpu on pods (as a percentage of request):     10% / 50%
Min replicas:                                            2
Max replicas:                                            8
...

As you can see, this HPA is configured to scale the service based on both CPU and memory, with average utilization of 50% each.

This means that as soon as either the CPU or memory utilization breaches the 50% threshold, the HPA will trigger an upscale.

Under minimal load, the HPA will still retain a replica count of 2, while the maximum amount of Pods the HPA is allowed to spin up under high load is 8.

Note: in a production environment, it is recommended to have a minimum replica count of at least 3, to guarantee maintained availability in the case of Pod Disruption.

Now, this is the moment you’ve certainly expected… It’s Load Testing time! 😎

For Load Testing, I really recommend k6 from the Grafana Labs team. It is a dead-simple yet super powerful tool with very extensive documentation.

See for yourself!

k6 run k6/script.js

While the load test is running, I suggest watching the HPA in a separate tab.

kubectl get hpa -n sample-app -w

As the load test progresses and the 2 starting Pods struggle to handle incoming requests, you should see both CPU and memory targets increasing, and ultimately, the replica count reaching its maximum!

Deployment/sample-app   16%/50%, 10%/50%   2         8         2
Deployment/sample-app   16%/50%, 15%/50%   2         8         2
Deployment/sample-app   17%/50%, 40%/50%   2         8         2
Deployment/sample-app   18%/50%, 50%/50%   2         8         2
Deployment/sample-app   19%/50%, 60%/50%   2         8         2
Deployment/sample-app   22%/50%, 75%/50%   2         8         3
Deployment/sample-app   27%/50%, 85%/50%   2         8         3
Deployment/sample-app   24%/50%, 80%/50%   2         8         4
Deployment/sample-app   27%/50%, 80%/50%   2         8         5
Deployment/sample-app   22%/50%, 72%/50%   2         8         5
Deployment/sample-app   23%/50%, 70%/50%   2         8         6
Deployment/sample-app   25%/50%, 64%/50%   2         8         7
Deployment/sample-app   24%/50%, 61%/50%   2         8         7
Deployment/sample-app   25%/50%, 61%/50%   2         8         7
Deployment/sample-app   27%/50%, 60%/50%   2         8         8
Deployment/sample-app   28%/50%, 60%/50%   2         8         8
Deployment/sample-app   27%/50%, 57%/50%   2         8         8

When relying on multiple targets for a single HPA, you can find out which of those have triggered the up/downscale by consulting Kubernetes events.

➜ kubectl get events -n sample-app
...
New size: 3; reason: cpu resource utilization (percentage of request) above target
Scaled up replica set sample-app-6bcbfc8b49 to 3
New size: 4; reason: cpu resource utilization (percentage of request) above target
Scaled up replica set sample-app-6bcbfc8b49 to 4
New size: 5; reason: cpu resource utilization (percentage of request) above target
Scaled up replica set sample-app-6bcbfc8b49 to 5
New size: 6; reason: cpu resource utilization (percentage of request) above target
Scaled up replica set sample-app-6bcbfc8b49 to 6
New size: 7; reason: cpu resource utilization (percentage of request) above target
Scaled up replica set sample-app-6bcbfc8b49 to 7
New size: 8; reason: cpu resource utilization (percentage of request) above target
Scaled up replica set sample-app-6bcbfc8b49 to 8
...

Now, let’s quickly have a look at the Load Test summary and the result of the http_req_duration metric in particular!

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: k6/script.js
     output: -

  scenarios: (100.00%) 1 scenario, 100 max VUs, 5m30s max duration (incl. graceful stop):
           * load: Up to 100.00 iterations/s for 5m0s over 2 stages (maxVUs: 100, gracefulStop: 30s)

     ✓ status code is 200
     ✓ node is kind-control-plane
     ✓ namespace is sample-app
     ✓ pod is sample-app-*

   ✓ checks.........................: 100.00% ✓ 60360    ✗ 0
     data_received..................: 3.5 MB  12 kB/s
     data_sent......................: 1.7 MB  5.8 kB/s
     http_req_blocked...............: avg=17.57µs  min=1µs      med=9µs    max=5.48ms  p(90)=17µs    p(95)=21µs
     http_req_connecting............: avg=4.36µs   min=0s       med=0s     max=5.26ms  p(90)=0s      p(95)=0s
   ✓ http_req_duration..............: avg=17.63ms  min=496µs    med=3.16ms max=1.76s   p(90)=11.38ms p(95)=51.18ms
       { expected_response:true }...: avg=17.63ms  min=496µs    med=3.16ms max=1.76s   p(90)=11.38ms p(95)=51.18ms
     http_req_failed................: 0.00%   ✓ 0        ✗ 15090
     http_req_receiving.............: avg=107.72µs min=10µs     med=78µs   max=7.62ms  p(90)=164µs   p(95)=214µs
     http_req_sending...............: avg=53.87µs  min=5µs      med=35µs   max=15.33ms p(90)=73µs    p(95)=95µs
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s     max=0s      p(90)=0s      p(95)=0s
     http_req_waiting...............: avg=17.47ms  min=423µs    med=2.99ms max=1.76s   p(90)=11.08ms p(95)=51.02ms
     http_reqs......................: 15090   50.29863/s
     iteration_duration.............: avg=18.11ms  min=626.66µs med=3.64ms max=1.76s   p(90)=12.03ms p(95)=51.78ms
     iterations.....................: 15090   50.29863/s
     vus............................: 0       min=0      max=18
     vus_max........................: 100     min=100    max=100

running (5m00.0s), 000/100 VUs, 15090 complete and 0 interrupted iterations
load ✓ [======================================] 000/100 VUs  5m0s  000.65 iters/s

As you can observe, our Golang service has performed very well under heavy load, with a Success Share of 100%, a median latency of ~3ms, and a 95th percentile latency of ~50ms!

We have the HPA to thank for that, as it scaled the Deployment from 2 to 8 Pods swiftly and automatically, based on the Pods resource usage!

We definitely would not have had the same results without an HPA… Actually, why don’t you try it yourself? 😉

Just delete the HPA (kubectl delete hpa sample-app -n sample-app), run the load test again (k6 run k6/script.js) and see what happens! (spoiler alert: it’s not pretty 😬)

Once you are done, don’t forget to uninstall the Helm release! (we won’t be needing this one anymore)

helm uninstall sample-app -n sample-app

Autoscaling JVM Services based on CPU Usage

While native services run as executable binaries, JVM services need an extra environment layer in order to run on various OSs and CPU architectures: the Java Virtual Machine (JVM).

This creates an issue for Pod Autoscaling, as the JVM pre-allocates more memory than it actually needs from the Pod’s resources, for its Garbage Collector (GC). This makes memory altogether an unreliable metric to use for autoscaling a JVM-based service in Kubernetes via Metrics Server.

Thus, in the case of Java or other JVM-based services, when utilizing Metrics Server for HPA, one can only rely on the CPU metric for autoscaling.

Let’s experience it with a Kotlin/JVM service!

helm install kotlin-sample-app/helm-chart --name-template sample-app --create-namespace -n sample-app --wait

If everything goes fine, you should eventually see one Deployment with the READY state.

➜ kubectl get deploy -n sample-app
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
sample-app   2/2     2            2           52s

Let’s see what the resource usage for those Pods running a JVM currently is!

➜ kubectl top pods -n sample-app
NAME                         CPU(cores)   MEMORY(bytes)
sample-app-8df8cfcd4-lg9s8   7m           105Mi
sample-app-8df8cfcd4-r9fjh   7m           105Mi

Interesting! As you can see, while being idle, both pods consume ~100Mi (~104Mb) of memory, which is almost 50% of the Pods memory limit! 😱
As previously stated, this is due to the JVM pre-allocating memory for its Garbage Collector (GC).

Alright, now let’s have a look at the HPA!

➜ kubectl describe hpa -n sample-app
...
Metrics: ( current / target )
  resource cpu on pods (as a percentage of request):  10% / 50%
Min replicas:                                         2
Max replicas:                                         8
...

As announced, this time the HPA only relies on one resource metric: the CPU.

Alright, let’s give our favorite Load Testing tool another go! 🚀

k6 run k6/script.js

As previously mentioned, I suggest watching the HPA in a separate tab.

kubectl get hpa -n sample-app -w

As the load test progresses and the 2 starting Pods struggle to handle incoming requests, you should see the CPU target increasing, and ultimately, the replica count reaching its maximum!

Deployment/sample-app   10%/50%   2         8         2
Deployment/sample-app   15%/50%   2         8         2
Deployment/sample-app   36%/50%   2         8         2
Deployment/sample-app   64%/50%   2         8         2
Deployment/sample-app   37%/50%   2         8         3
Deployment/sample-app   41%/50%   2         8         3
Deployment/sample-app   51%/50%   2         8         3
Deployment/sample-app   99%/50%   2         8         3
Deployment/sample-app   56%/50%   2         8         6
Deployment/sample-app   50%/50%   2         8         6
Deployment/sample-app   76%/50%   2         8         6
Deployment/sample-app   74%/50%   2         8         8
Deployment/sample-app   61%/50%   2         8         8
Deployment/sample-app   58%/50%   2         8         8

Now, let’s quickly have a look at the Load Test summary and the result of the http_req_duration metric in particular!

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: k6/script.js
     output: -

  scenarios: (100.00%) 1 scenario, 100 max VUs, 5m30s max duration (incl. graceful stop):
           * load: Up to 100.00 iterations/s for 5m0s over 2 stages (maxVUs: 100, gracefulStop: 30s)

     ✓ status code is 200
     ✓ node is kind-control-plane
     ✓ namespace is sample-app
     ✓ pod is sample-app-*

   ✓ checks.........................: 100.00% ✓ 60360     ✗ 0
     data_received..................: 3.3 MB  11 kB/s
     data_sent......................: 1.7 MB  5.8 kB/s
     http_req_blocked...............: avg=18.56µs  min=1µs    med=9µs    max=2.37ms p(90)=17µs    p(95)=20µs
     http_req_connecting............: avg=5.52µs   min=0s     med=0s     max=1.65ms p(90)=0s      p(95)=0s
   ✓ http_req_duration..............: avg=13.4ms   min=864µs  med=3.7ms  max=1.96s  p(90)=11.43ms p(95)=43.16ms
       { expected_response:true }...: avg=13.4ms   min=864µs  med=3.7ms  max=1.96s  p(90)=11.43ms p(95)=43.16ms
     http_req_failed................: 0.00%   ✓ 0         ✗ 15090
     http_req_receiving.............: avg=101.68µs min=10µs   med=79µs   max=5.31ms p(90)=167µs   p(95)=217µs
     http_req_sending...............: avg=47.68µs  min=4µs    med=37µs   max=5.87ms p(90)=70µs    p(95)=88µs
     http_req_tls_handshaking.......: avg=0s       min=0s     med=0s     max=0s     p(90)=0s      p(95)=0s
     http_req_waiting...............: avg=13.25ms  min=803µs  med=3.54ms max=1.96s  p(90)=11.25ms p(95)=43.05ms
     http_reqs......................: 15090   50.306331/s
     iteration_duration.............: avg=13.84ms  min=1.06ms med=4.17ms max=1.96s  p(90)=12.02ms p(95)=43.55ms
     iterations.....................: 15090   50.306331/s
     vus............................: 0       min=0       max=13
     vus_max........................: 100     min=100     max=100

running (5m00.0s), 000/100 VUs, 15090 complete and 0 interrupted iterations
load ✓ [======================================] 000/100 VUs  5m0s  000.65 iters/s

As you can observe, our Kotlin/JVM service has performed very well under heavy load, with a Success Share of 100%, a median latency of ~3ms, and a 95th percentile latency of ~50ms!

Once again, the HPA was able to scale the Deployment from 2 to 8 Pods swiftly and automatically, based on the Pods CPU usage alone!

Note: if you keep the Deployment idle for a few minutes, you should see the HPA gradually scaling back down to 2 Pods, due to low CPU usage.

Wrapping up

That’s it! You can now stop and delete your Kind cluster.

kind delete cluster

To summarize, using Metrics Server we were able to:

• Autoscale horizontally our native service written in Golang, based on both CPU and memory usage;
• Autoscale horizontally our JVM service written in Kotlin/JVM, based on CPU usage.

Was it worth it? Did that help you understand how to implement Horizontal Pod Autoscaler in Kubernetes using Metrics Server?

If so:

1. Let me know in the comments below! 👇
2. Don’t forget to hit that subscribe button! ✅
3. Follow me on Twitter, I’ll be happy to answer any of your questions and you’ll be the first ones to know when a new article comes out! 👌

See you next month, for Part 2 of my series Horizontal Pod Autoscaler in Kubernetes!

Bye-bye! 👋

Horizontal Pod Autoscaler in Kubernetes (Part 1) — Simple Autoscaling using Metrics Server was originally published in EGYM Software Development on Medium, where people are continuing the conversation by highlighting and responding to this story.

Pssst! I started my own blog!
You can read this very same article over there too!
No paywall, no ad, no Javascript — no bullshit, just pure content.
See you there!

Mobile force updates occur when old versions of an app are no longer compatible with the APIs they consume. Until the app is updated to the required version, the UI blocks further usage. This is usually materialized as a system popup that will redirect users to the respective Store and disappear only once they have updated to the latest version. This is often considered bad practice as it deteriorates the UX of an app drastically, yet there are situations (mostly breaking API changes) when it is unavoidable.

Such updates are quite risky, as once an app is updated in the stores, there is no going back unless the previous version of the app is released again under a different version number. Web applications do not face such risks as they are easily rolled back. Also, just like for backend APIs, a web app can incrementally roll out such changes using a canary deployment.

If you haven’t already, go read Canary Deployment in Kubernetes (Part 1) — Simple Canary Deployment using Ingress NGINX and learn how to implement a Simple Canary Deployment using Ingress NGINX!

However, this doesn’t mean that there are no solutions. In fact, it is possible to leverage backend canary deployments in order to incrementally roll out a mobile force update. Ingress NGINX, for instance, only provides traffic splitting based on canary-weight but it also features traffic routing based on canary-by-header which can be used for this purpose.

What if we could route user traffic based on the version of the app they are using?

🎬 Hi there, I’m Jean!

In this article, we’re going to learn how to leverage a simple canary deployment using Ingress NGINX and Firebase Remote Config to safely conduct an incremental mobile force update! 💪

Smart Traffic Routing with Firebase Remote Config

Firebase Remote Config is a powerful Cloud service from Google that enables frontend applications to store their configuration properties in Google’s Cloud and access them remotely via a dedicated Firebase SDK (available in iOS, Android, C++, and Javascript).

While editing those configuration properties would normally require a frontend release, with Firebase Remote Config, developers can simply change the properties values in Firebase’s UI and the frontend will update itself automatically on the fly.

The most powerful feature of Firebase Remote Config though is that configuration properties can have dynamic values, based on app data (or even user data from Google Analytics). 🤯

For instance, say we must make a non-backward-compatible change to our Backend API, this means all our mobile apps must conduct a force update. Thus, if users are using version v1, they will be prompted to update their app to version v2 so that they can keep on consuming the Backend API.
But what if this non-backward-compatible change is a faulty one? How do we revert such a change? Well, in fact, we can’t: once a user has updated their mobile app, there is no going back. So how do we solve this then?

Well, to solve the force update problem, we could theoretically define a Remote Config called UseCanary and give it a dynamic value based on the app version.

Specifically:

• If AppVersion < 2.0.0, UseCanary = never;
• If AppVersion ≥ 2.0.0, UseCanary = always.

Then, when the app fetches its Remote Config, it can add to its requests the famous X-Canary header we talked about in Simple Canary Deployment using Ingress NGINX (remember? 😉) like so: X-Canary: <UseCanary>.

This will result in:

• If AppVersion < 2.0.0, request header = X-Canary: never;
• If AppVersion ≥ 2.0.0, request header = X-Canary: always.

In Firebase Remote Config, this is called a condition. A condition is a logical block allowing Remote Config properties (called parameters) to have dynamic values based on various app or user data, such as the app version.

Then, creating a dynamic parameter simply boils down to specifying its name, condition(s), and associated values.

Now we made sure that only users using the v2 version of the app get routed to the canary deployment, we can roll out our non-backward-compatible change to canary and start doing some smoke tests, in complete isolation from the rest of the userbase!

Incrementally Rolling Out Mobile Releases

Once ready to go live, the mobile engineering team can start incrementally rolling out their new v2 version of the app in the stores:

• In iOS, using the Phased Release feature of the App Store ;
• In Android, using the Staged Rollout feature of the Play Store.

As users update their app, traffic to the canary deployment will start coming in and if things suddenly go south, the Phased Release or Staged Rollout can be easily paused, until a fix is deployed to the canary deployment.

Once the rollout of the non-backward-compatible change in canary is considered successful, the mobile engineering team can proceed with the actual force update and the stable deployment will soon stop receiving traffic altogether. This means that all users have now migrated to the v2 version of the app, the canary rollout is therefore complete and can be rolled over to the stable deployment!

Wrapping up

To summarize, using Ingress NGINX in tandem with Firebase Remote Config, we were able to:

• Route user traffic to a canary deployment based on the version of the app they are using;
• Incrementally rollout a non-backward-compatible change to minimize risk before conducting a force update.

Was it worth it? Did that help you understand how to incrementally conduct a mobile force update using Ingress NGINX and Firebase Remote Config?

If so:

1. Let me know in the comments below! 👇
2. Don’t forget to hit that subscribe button! ✅
3. Follow me on Twitter, I’ll be happy to answer any of your questions and you’ll be the first ones to know when a new article comes out! 👌

Bye-bye! 👋

Incremental Mobile Force Update using Ingress NGINX and Firebase Remote Config was originally published in EGYM Software Development on Medium, where people are continuing the conversation by highlighting and responding to this story.

Learn how to use Argo Rollouts with Prometheus to automate the detection and rollback of faulty deployments using AnalysisTemplates.

Pssst! I started my own blog!
You can read this very same article over there too!
No paywall, no ad, no Javascript — no bullshit, just pure content.
See you there!

Deploying to production in Kubernetes can be quite stressful. Even after meaningful and reliable automated tests have successfully passed, there is still room for things to go wrong and lead to a nasty incident when pressing the final button.

Thankfully, Kubernetes is made to be resilient to this kind of scenario, and rolling back is a no-brainer. But still, rolling back means that, at least for some time, all of the users were negatively impacted by the faulty change…

What if we could smoke test our change in production before it actually hits real users? What if we could roll out a change incrementally to some users instead of all of them at once? What if we could detect a faulty deployment and roll it back automatically?
Well, that, my friend, is what Canary Deployment is all about!

Minimizing the impact on real users while deploying a risky change to production.

🎬 Hi there, I’m Jean!

In this 3 parts series, we’re going to explore several ways to do Canary Deployment in Kubernetes, and the last one is…
🥁
… using Argo Rollouts and Prometheus! 🎊

Requirements

Before we start, make sure you have the following tools installed:

• Kind
• Kubectl
• Argo Rollouts Kubectl Plugin
• Helm
• K6

Note: for MacOS users or Linux users using Homebrew, simply run:
brew install kind kubectl argoproj/tap/kubectl-argo-rollouts helm k6

All set? Let’s go! 🏁

Creating Kind Cluster

Kind is a tool for running local Kubernetes clusters using Docker container “nodes”. It was primarily designed for testing Kubernetes itself, but may be used for local development or CI.

I don’t expect you to have a demo project in handy, so I built one for you.

git clone https://github.com/jhandguy/canary-deployment.git

cd canary-deployment

Alright, let’s spin up our Kind cluster! 🚀

➜ kind create cluster --image kindest/node:v1.27.3 --config=kind/cluster.yaml

Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾

Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

Using Argo Rollouts with Prometheus

Argo Rollouts is a Kubernetes controller and set of CRDs which provide advanced deployment capabilities to Kubernetes such as blue-green, canary, canary analysis, experimentation, and progressive delivery.

In combination with Prometheus, Argo Rollouts can automatically roll back a Canary Deployment based on Prometheus metrics, which means it can theoretically handle an incremental rollout without human intervention.

Installing NGINX Ingress Controller

If you haven’t already, go read Canary Deployment in Kubernetes (Part 1) — Simple Canary Deployment using Ingress NGINX and learn how to implement a Simple Canary Deployment using Ingress NGINX!

NGINX Ingress Controller is one of the many available Kubernetes Ingress Controllers, which acts as a load balancer and satisfies routing rules specified in Ingress resources, using the NGINX reverse proxy.

NGINX Ingress Controller can be installed via its Helm chart.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

helm install ingress-nginx/ingress-nginx --name-template ingress-nginx --create-namespace -n ingress-nginx --values kind/ingress-nginx-values.yaml --version 4.8.3 --wait

Now, if everything goes according to plan, you should be able to see the ingress-nginx-controller Deployment running.

➜ kubectl get deploy -n ingress-nginx
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx-controller   1/1     1            1           4m35s

Installing Argo Rollouts

If you haven’t already, go read Canary Deployment in Kubernetes (Part 2) — Automated Canary Deployment using Argo Rollouts and learn how to implement a Canary Deployment using Argo Rollouts!

Argo Rollouts can be installed via its Helm chart.

helm repo add argo https://argoproj.github.io/argo-helm

helm install argo/argo-rollouts --name-template argo-rollouts --create-namespace -n argo-rollouts --set dashboard.enabled=true --version 2.32.5 --wait

If all goes well, you should see two newly spawned deployments with the READY state!

➜ kubectl get deploy -n argo-rollouts
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
argo-rollouts             1/1     1            1           13m
argo-rollouts-dashboard   1/1     1            1           13m

Installing Prometheus

Prometheus can be installed via its community Helm chart, which also provides Grafana out of the box.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm install prometheus-community/kube-prometheus-stack --name-template prometheus --create-namespace -n prometheus --version 54.2.2 --wait

If everything went fine, you should be able to see three newly spawned deployments with the READY state!

➜ kubectl get deploy -n prometheus
NAME                                  READY   UP-TO-DATE   AVAILABLE   
prometheus-grafana                    1/1     1            1           
prometheus-kube-prometheus-operator   1/1     1            1           
prometheus-kube-state-metrics         1/1     1            1           

Automating Rollbacks using AnalysisTemplates

In Part 2, we learned how to configure a Rollout and how to automate the Traffic Routing increments via Rollout steps.
Yet another amazing feature of Argo Rollouts is the ability to leverage Prometheus metrics in order to detect faulty deployments (i.e. based on request Success Share or Latency).

To this end, Argo Rollouts provides another Custom Resource called AnalysisTemplate.
Alright! Let’s explore what this is!

helm install sample-app/helm-charts/argo-rollouts --name-template sample-app --create-namespace -n sample-app --set prometheus.enabled=true --wait

If everything goes fine, you should eventually see one Rollout with the READY state.

➜ kubectl get rollout sample-app -n sample-app
NAME         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE
sample-app   1         1         1            1

Alright, let’s have a look at the analysistemplate.yaml and the rollout.yaml inside the templates folder!

➜ ls -1 sample-app/helm-charts/argo-rollouts/templates
analysistemplate.yaml
canary
ingress.yaml
rollout.yaml
service.yaml
serviceaccount.yaml
servicemonitor.yaml

An AnalysisTemplate is a template spec that defines how to perform a canary analysis. It consists of a Prometheus metric which is being evaluated, at a given interval, against a given success condition.

In this example, the Prometheus metric is a Success Share, with a minimum threshold of 99% and an interval of 1 minute. The failure limit is 0, meaning that as soon as it fails, the Rollout will be aborted. Since it is the canary Deployment that we are trying to evaluate, the PromQL query must target it specifically via the service label.

---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
...
spec:
  metrics:
    - name: success-share
      interval: 1m
      successCondition: len(result) == 0 || result[0] >= 0.99
      failureLimit: 0
      provider:
        prometheus:
          address: {{ .Values.prometheus.address }}
          query: |
            sum(rate(
              sample_app_requests_count{service="{{ .Release.Name }}-canary", success="true"}[1m])
            ) by (service)
            /
            sum(rate(
              sample_app_requests_count{service="{{ .Release.Name }}-canary"}[1m])
            ) by (service)
            unless sum(rate(
              sample_app_requests_count{service="{{ .Release.Name }}-canary"}[1m])
            ) by (service) == 0

The instantiation of an AnalysisTemplate is called an AnalysisRun. It is like a Job, meaning it eventually completes, with a result of either Successful, Failed, or Inconclusive. If successful, the rollout continues, if failed, it is aborted and if inconclusive, it is paused.

There are two ways to define an AnalysisTemplate:

1. Background Analysis
   This Analysis runs in the background, while the Rollout is progressing through its steps. If the AnalysisRun fails, the Rollout is aborted. On the other hand, if the AnalysisRun succeeds or all the Rollout steps are finished before it completes, the Rollout is considered successful.
2. Inline Analysis
   This Analysis runs as part of the Rollout steps. This means that the Rollout will wait for the AnalysisRun to complete, before proceeding to the next step, or aborting the rollout depending on the outcome.

In our case, we will be using the Background Analysis, as it will detect a faulty deployment sooner than an Inline Analysis would.

---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
...
spec:
  ...
  strategy:
    canary:
      ...
      steps:
        - setWeight: 25
        - pause: {}
        - setWeight: 50
        - pause:
            duration: 5m
        - setWeight: 75
        - pause:
            duration: 5m
      analysis:
        templates:
          - templateName: {{ .Release.Name }}
        startingStep: 2
      ...
  ...

As you can see, the Analysis has been scheduled to start after step 2.
Meaning, once the Rollout is promoted, the canary-weight will be set to 50 and an AnalysisRun will start. It will determine later if the Rollout succeeds or not: if the Prometheus metric falls below 99% Success Share, the Rollout will be aborted immediately. On the other hand, if the Success Share stays above 99% for 10 minutes, the Rollout will complete successfully.

Detecting and Aborting Rollouts Automatically

Now that we’ve seen how an AnalysisRun works in theory, let’s see it in practice, shall we?! 🧐

kubectl argo rollouts dashboard -n argo-rollouts &

If you now head to http://localhost:3100/rollout/sample-app, you should see a shiny dashboard showing the state of the sample-app Rollout.

Cool! So far, we can observe only one revision labeled as stable, serving 100% of the traffic.

Now, let’s set a new image for the container!

kubectl argo rollouts set image sample-app sample-app=ghcr.io/jhandguy/canary-deployment/sample-app:latest -n sample-app

Tada! The Rollout has just completed step 1: setting the canary-weight to 25%.

As you can see, the AnalysisRun has not yet started, this will only happen once we start step 2. Let’s proceed then!

kubectl argo rollouts promote sample-app -n sample-app

Awesome! The canary-weight is now at 50% and an AnalysisRun has just started.

➜ kubectl get analysisrun -n sample-app
NAME                      STATUS
sample-app-5c9fc8b7d4-2   Running

Let’s have a deeper look at it!

➜ kubectl describe analysisrun -n sample-app
Name:         sample-app-5c9fc8b7d4-2
Namespace:    sample-app
Labels:       rollout-type=Background
              rollouts-pod-template-hash=5c9fc8b7d4
Annotations:  rollout.argoproj.io/revision: 2
API Version:  argoproj.io/v1alpha1
Kind:         AnalysisRun
...
Spec:
  Metrics:
    Failure Limit:  0
    Interval:       1m
    Name:           success-share
    Provider:
      Prometheus:
        Address:  http://prometheus-operated.prometheus.svc.cluster.local:9090
        Query:    sum(rate(
                    sample_app_requests_count{service="sample-app-canary", success="true"}[1m])
                  ) by (service)
                  /
                  sum(rate(
                    sample_app_requests_count{service="sample-app-canary"}[1m])
                  ) by (service)
                  unless sum(rate(
                    sample_app_requests_count{service="sample-app-canary"}[1m])
                  ) by (service) == 0
    Success Condition:  len(result) == 0 || result[0] >= 0.99
Status:
  Metric Results:
    Count:  1
    Measurements:
      Finished At:  2022-01-29T18:37:49Z
      Phase:        Successful
      Started At:   2022-01-29T18:37:49Z
      Value:        []
    Name:           success-share
    Phase:          Running
    Successful:     1
  Phase:            Running
  Started At:       2022-01-29T18:37:49Z
Events:             <none>

As expected, the Spec part of the AnalysisRun is the same one we’ve specified in the AnalysisTemplate.
To monitor the progress of the Rollout, however, it is the Status and the Events that we will be looking at.

Right now, the measurement’s value is empty ([]), this is because the service has not recorded any metrics yet. Let’s change that and send some successful requests for Prometheus to scrape! 🚀

Since the PromQL query targets the canary Service specifically, we must make sure to always land in the canary Deployment using the Ingress canary-by-header annotation (learn more about it in Part 1).

curl localhost/success -H "Host: sample.app" -H "X-Canary: always"

After firing some requests and waiting about a minute for Prometheus to scrape the metrics and the AnalysisRun to measure the success-share metric, you should see one of the measurements with a value of [1], meaning that the Success Share has been successfully measured at 100%! 💯

➜ kubectl describe analysisrun -n sample-app
...
Status:
  Metric Results:
    Count:  5
    Measurements:
      ...
      Finished At:  2022-01-29T18:41:49Z
      Phase:        Successful
      Started At:   2022-01-29T18:41:49Z
      Value:        [1]
    Name:           success-share
    Phase:          Running
    Successful:     5
  Phase:            Running
  Started At:       2022-01-29T18:37:49Z
Events:             <none>

The best part of this is, that while all of this was happening, the Rollout kept on progressing and has probably already reached the next step: increasing the canary-weight to 75%.

That’s great! But let’s make things more interesting: what happens once the Success Share falls below 99%? 😱

Let’s go ahead and simulate a 50% Success Share by sending several requests and alternating between successes and errors!

curl localhost/success -H "Host: sample.app" -H "X-Canary: always"
curl localhost/error -H "Host: sample.app" -H "X-Canary: always"

After a short while, you should notice that something quite magical has happened… 🪄

➜ kubectl describe analysisrun -n sample-app
...
Status:
  Message:  metric "success-share" assessed Failed due to failed (1) > failureLimit (0)
  Metric Results:
    Count:   9
    Failed:  1
    Measurements:
      ...
      Finished At:  2022-01-29T18:45:49Z
      Phase:        Failed
      Started At:   2022-01-29T18:45:49Z
      Value:        [0.5]
    Name:           success-share
    Phase:          Failed
    Successful:     8
  Phase:            Failed
  Started At:       2022-01-29T18:37:49Z
Events:
  Type     Reason             Age   From                 Message
  ----     ------             ----  ----                 -------
  Warning  MetricFailed       17s   rollouts-controller  metric 'success-share' completed Failed
  Warning  AnalysisRunFailed  17s   rollouts-controller  analysis completed Failed

As soon as the AnalysisRun recorded a measurement breaching below the 99% Success Share threshold, the Rollout was immediately aborted.

Let’s confirm that from the Rollout’s status real quick!

➜ kubectl argo rollouts status sample-app -n sample-app
Degraded
Error: The rollout is in a degraded state with message: RolloutAborted: Rollout aborted update to revision 2: metric "success-share" assessed Failed due to failed (1) > failureLimit (0)

Indeed, even the Dashboard highlights that the Rollout is in a Degraded state, and consequently the canary-weight has been taken down to 0%.

This would, in practice, give us time to investigate the issue, and once fixed, retry the Rollout, either via the CLI or the Dashboard’s RETRY button.

kubectl argo rollouts retry rollout sample-app -n sample-app
kubectl argo rollouts promote sample-app -n sample-app

Now that the Rollout has restarted and reached step 2, a new AnalysisRun has started and the incremental rollout is back on track! 🛤

Load Testing with k6

Instead of running some curl commands like we previously did, how about we execute a Load Test instead?

For Load Testing, I really recommend k6 from the Grafana Labs team. It is a dead-simple yet super powerful tool with very extensive documentation.

See for yourself!

k6 run k6/script.js

After about 1 minute, k6 should be done executing the load test and show you the results.

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: k6/script.js
     output: -

  scenarios: (100.00%) 1 scenario, 20 max VUs, 1m30s max duration (incl. graceful stop):
           * load: Up to 20.00 iterations/s for 1m0s over 2 stages (maxVUs: 20, gracefulStop: 30s)

     ✓ status code is 200
     ✓ node is kind-control-plane
     ✓ namespace is sample-app
     ✓ pod is sample-app-*
     ✓ deployment is stable or canary

   ✓ checks.........................: 100.00% ✓ 3095      ✗ 0
     data_received..................: 157 kB  2.6 kB/s
     data_sent......................: 71 kB   1.2 kB/s
     http_req_blocked...............: avg=41.24µs min=3µs    med=8µs    max=3.39ms  p(90)=19µs   p(95)=55.19µs
     http_req_connecting............: avg=21.96µs min=0s     med=0s     max=2.71ms  p(90)=0s     p(95)=0s
   ✓ http_req_duration..............: avg=3.75ms  min=940µs  med=2.96ms max=17.22ms p(90)=6.99ms p(95)=9.37ms
       { expected_response:true }...: avg=3.75ms  min=940µs  med=2.96ms max=17.22ms p(90)=6.99ms p(95)=9.37ms
     http_req_failed................: 0.00%   ✓ 0         ✗ 619
     http_req_rate..................: 50.00%  ✓ 619       ✗ 619
     ✓ { deployment:canary }........: 49.91%  ✓ 309       ✗ 310
     ✓ { deployment:stable }........: 50.08%  ✓ 310       ✗ 309
     http_req_receiving.............: avg=107.1µs min=24µs   med=88µs   max=2.09ms  p(90)=165µs  p(95)=189.09µs
     http_req_sending...............: avg=50.78µs min=14µs   med=36µs   max=837µs   p(90)=77µs   p(95)=114.19µs
     http_req_tls_handshaking.......: avg=0s      min=0s     med=0s     max=0s      p(90)=0s     p(95)=0s
     http_req_waiting...............: avg=3.6ms   min=857µs  med=2.78ms max=17.09ms p(90)=6.81ms p(95)=9.1ms
     http_reqs......................: 619     10.316653/s
     iteration_duration.............: avg=4.31ms  min=1.09ms med=3.53ms max=21.84ms p(90)=7.82ms p(95)=10.34ms
     iterations.....................: 619     10.316653/s
     vus............................: 0       min=0       max=0
     vus_max........................: 20      min=20      max=20

running (1m00.0s), 00/20 VUs, 619 complete and 0 interrupted iterations
load ✓ [======================================] 00/20 VUs  1m0s  00.71 iters/s

That sounds about right!
Depending on which step the Rollout is at, the Traffic Distribution shown in k6’s output should be either 50/50 or 75/25.

Same as before, if you now describe the AnalysisRun, you should see some successful measurements were measured.

➜ kubectl describe analysisrun -n sample-app
...
Status:
  Metric Results:
    Count:  5
    Measurements:
      ...
      Phase:        Successful
      Started At:   2022-01-30T14:34:07Z
      Value:        [1]
    Name:           success-share
    Phase:          Running
    Successful:     5
  Phase:            Running
  Started At:       2022-01-30T14:32:07Z
Events:             <none>

Eventually, the Rollout should complete successfully and the new revision should become the new stable Deployment! 🎉

Wrapping up

That’s it! You can now stop and delete your Kind cluster!

kind delete cluster

To summarize, using Argo Rollouts and Prometheus we were able to:

• Detect when a Prometheus metric breaches a given threshold and abort the faulty Rollout automatically, thanks to the AnalysisRun CRD;
• Restart the aborted Rollout effortlessly once the issue is fixed.

Was it worth it? Did that help you understand how to implement Canary Deployment in Kubernetes using Argo Rollouts and Prometheus?

If so:

1. Let me know in the comments below! 👇
2. Don’t forget to hit that subscribe button! ✅
3. Follow me on Twitter, I’ll be happy to answer any of your questions and you’ll be the first ones to know when a new article comes out! 👌

Bye-bye! 👋

Canary Deployment in Kubernetes (Part 3) — Smart Canary Deployment using Argo Rollouts and… was originally published in EGYM Software Development on Medium, where people are continuing the conversation by highlighting and responding to this story.

Learn how to use Argo Rollouts to improve a Canary Deployment setup using Ingress NGINX and automate incremental rollouts.

Pssst! I started my own blog!
You can read this very same article over there too!
No paywall, no ad, no Javascript — no bullshit, just pure content.
See you there!

Deploying to production in Kubernetes can be quite stressful. Even after meaningful and reliable automated tests have successfully passed, there is still room for things to go wrong and lead to a nasty incident when pressing the final button.

Thankfully, Kubernetes is made to be resilient to this kind of scenario, and rolling back is a no-brainer. But still, rolling back means that, at least for some time, all of the users were negatively impacted by the faulty change…

What if we could smoke test our change in production before it actually hits real users? What if we could roll out a change incrementally to some users instead of all of them at once? What if we could detect a faulty deployment and roll it back automatically?
Well, that, my friend, is what Canary Deployment is all about!

Minimizing the impact on real users while deploying a risky change to production.

🎬 Hi there, I’m Jean!

In this 3 parts series, we’re going to explore several ways to do Canary Deployment in Kubernetes, and the second one is…
🥁
… using Argo Rollouts! 🎊

Requirements

Before we start, make sure you have the following tools installed:

• Kind
• Kubectl
• Argo Rollouts Kubectl Plugin
• Helm
• K6

Note: for MacOS users or Linux users using Homebrew, simply run: brew install kind kubectl argoproj/tap/kubectl-argo-rollouts helm k6

All set? Let’s go! 🏁

Creating Kind Cluster

Kind is a tool for running local Kubernetes clusters using Docker container “nodes”. It was primarily designed for testing Kubernetes itself, but may be used for local development or CI.

I don’t expect you to have a demo project in handy, so I built one for you.

git clone https://github.com/jhandguy/canary-deployment.git

cd canary-deployment

Alright, let’s spin up our Kind cluster! 🚀

➜ kind create cluster --image kindest/node:v1.27.3 --config=kind/cluster.yaml

Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾

Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

Using Argo Rollouts with NGINX Ingress Controller

Argo Rollouts is a Kubernetes controller and set of CRDs which provide advanced deployment capabilities to Kubernetes such as blue-green, canary, canary analysis, experimentation, and progressive delivery.

In combination with the NGINX Ingress Controller, Argo Rollouts achieves the same as a Simple Canary Deployment using Ingress NGINX, but better.

Installing NGINX Ingress Controller

If you haven’t already, go read Canary Deployment in Kubernetes (Part 1) — Simple Canary Deployment using Ingress NGINX and learn how to implement a Simple Canary Deployment using Ingress NGINX!

NGINX Ingress Controller is one of the many available Kubernetes Ingress Controllers, which acts as a load balancer and satisfies routing rules specified in Ingress resources, using the NGINX reverse proxy.

NGINX Ingress Controller can be installed via its Helm chart.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

helm install ingress-nginx/ingress-nginx --name-template ingress-nginx --create-namespace -n ingress-nginx --values kind/ingress-nginx-values.yaml --version 4.8.3 --wait

Now, if everything goes according to plan, you should be able to see the ingress-nginx-controller Deployment running.

➜ kubectl get deploy -n ingress-nginx
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx-controller   1/1     1            1           4m35s

Installing Argo Rollouts

Argo Rollouts can be installed via its Helm chart.

helm repo add argo https://argoproj.github.io/argo-helm

helm install argo/argo-rollouts --name-template argo-rollouts --create-namespace -n argo-rollouts --set dashboard.enabled=true --version 2.32.5 --wait

If all goes well, you should see two newly spawned Deployments with the READY state.

➜ kubectl get deploy -n argo-rollouts
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
argo-rollouts             1/1     1            1           13m
argo-rollouts-dashboard   1/1     1            1           13m

Configuring Rollout Resources

One of the limitations of using Ingress NGINX alone is that it requires the duplication of quite a few Kubernetes resources. If you remember in Part 1, the Deployment, Service, and Ingress resources were almost identically cloned.
With Argo Rollouts, the duplication of Kubernetes resources is reduced to only one: the Service resource.

Another limitation of using plain Ingress NGINX is that the incremental rollout of the Canary Deployment (via canary-weight) has to be done manually by annotating the canary Ingress.
Argo Rollouts created its own Custom Resource to remediate that problem: the Rollout resource.

Alrighty then! Let’s get to it! 🧐

helm install sample-app/helm-charts/argo-rollouts --name-template sample-app --create-namespace -n sample-app --wait

If everything goes fine, you should eventually see one Rollout with the READY state.

➜ kubectl get rollout sample-app -n sample-app
NAME         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE
sample-app   1         1         1            1

Alright, let’s take a look at what’s under all this!

➜ ls -1 sample-app/helm-charts/argo-rollouts/templates
analysistemplate.yaml
canary
ingress.yaml
rollout.yaml
service.yaml
serviceaccount.yaml
servicemonitor.yaml

➜ ls -1 sample-app/helm-charts/argo-rollouts/templates/canary
service.yaml

Note: ignore the analysistemplate.yaml and the servicemonitor.yaml for now and focus on the rest of the resources.

As you can see, there is only one resource that has been duplicated in the canary folder: the service.yaml.

Now you might wonder, where did the Deployment resource go?! 🤔

In Argo Rollouts, the Deployment resource has been replaced with the Rollout resource. It is pretty much the same though, except it contains some extra specifications.

In fact, if you compare the deployment.yaml from Part 1 with the rollout.yaml, you will find that they are very similar, except for the strategy spec which is very unique to the Rollout resource.

---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
...
spec:
  ...
  strategy:
    canary:
      canaryService: {{ .Release.Name }}-canary
      stableService: {{ .Release.Name }}
      canaryMetadata:
        labels:
          deployment: canary
      stableMetadata:
        labels:
          deployment: stable
      trafficRouting:
        nginx:
          stableIngress: {{ .Release.Name }}
          additionalIngressAnnotations:
            canary-by-header: X-Canary
      steps:
        - setWeight: 25
        - pause: {}
        - setWeight: 50
        - pause:
            duration: 5m
        - setWeight: 75
        - pause:
            duration: 5m
  ...

Now, let’s break down what is happening here:

• Argo Rollouts Controller needs to know which Service is the canaryService and which is the stableService so it can update them to select their corresponding pods on the fly;
• In order for us to know which pods are being selected by which service, we can specify a label or annotation to be attached to the Pods metadata via canaryMetadata and stableMetadata;
• Argo Rollouts Controller also needs to know which Ingress is the stableIngress so it can create automatically the canary Ingress from it;
• Finally, in order to avoid us from manually editing the Ingress’s canary-weight annotation, the Argo Rollouts Controller will be able to do it for us automatically, thanks to the steps specification.

A step can be one of three things:

1. pause: this will put the Rollout in a Paused state. It can either be a timed pause (i.e. duration: 5m) or an indefinite pause, until it is manually resumed;
2. setWeight: this will increase the canary Ingress’s canary-weight annotation to the desired weight (i.e. setWeight: 25);
3. setCanaryScale: this will increase the number of replicas or pods which are selected by the canary Service (i.e. setCanaryScale: 4).

Automating Incremental Rollouts

Awesome! Now that our Rollout is up and we understand how it works under the hood, let’s take it for a spin! 🚗

But first, let’s have a look at the dashboard!

kubectl argo rollouts dashboard -n argo-rollouts &

If you now head to http://localhost:3100/rollout/sample-app, you should see a shiny dashboard showing the state of the sample-app Rollout.

Cool! So far, we can observe only one revision labeled as stable, serving 100% of the traffic.

Now, let’s set a new image for the container and see what happens!

kubectl argo rollouts set image sample-app sample-app=ghcr.io/jhandguy/canary-deployment/sample-app:latest -n sample-app

Aha! A new revision has been created and was labeled as canary. The weight for the canary Ingress has been set to 25% and the Rollout has now entered the Paused state.

Let’s verify this real quick!

➜ kubectl get ingress -n sample-app
NAME                           CLASS    HOSTS        ADDRESS     ...   
sample-app                     <none>   sample.app   localhost   ...  
sample-app-sample-app-canary   <none>   sample.app   localhost   ...

➜ kubectl describe ingress sample-app-sample-app-canary -n sample-app
...
Annotations:  kubernetes.io/ingress.class: nginx
              nginx.ingress.kubernetes.io/canary: true
              nginx.ingress.kubernetes.io/canary-by-header: X-Canary
              nginx.ingress.kubernetes.io/canary-weight: 25
...

Indeed! An extra Ingress resource was automatically created, and given a canary-weight of 25.

Let’s see if it works!

curl localhost/success -H "Host: sample.app"

As you can see, out of 4 requests, one came back from the canary pod.

➜ curl localhost/success -H "Host: sample.app"
{"node":"kind-control-plane","namespace":"sample-app","pod":"sample-app-6bf6dfdbb4-xmqdw","deployment":"stable"}

➜ curl localhost/success -H "Host: sample.app"
{"node":"kind-control-plane","namespace":"sample-app","pod":"sample-app-6bf6dfdbb4-xmqdw","deployment":"stable"}

➜ curl localhost/success -H "Host: sample.app"
{"node":"kind-control-plane","namespace":"sample-app","pod":"sample-app-6bf6dfdbb4-xmqdw","deployment":"stable"}

➜ curl localhost/success -H "Host: sample.app"
{"node":"kind-control-plane","namespace":"sample-app","pod":"sample-app-5c9fc8b7d4-wdrck","deployment":"canary"}

And, the same way as in Part 1, we can easily Smoke Test our change in isolation by leveraging the Ingress canary-by-header annotation.

curl localhost/success -H "Host: sample.app" -H "X-Canary: always"

Aha! We are now consistently hitting the canary pod.

➜ curl localhost/success -H "Host: sample.app" -H "X-Canary: always"
{"node":"kind-control-plane","namespace":"sample-app","pod":"sample-app-5c9fc8b7d4-wdrck","deployment":"canary"}

➜ curl localhost/success -H "Host: sample.app" -H "X-Canary: always"
{"node":"kind-control-plane","namespace":"sample-app","pod":"sample-app-5c9fc8b7d4-wdrck","deployment":"canary"}

➜ curl localhost/success -H "Host: sample.app" -H "X-Canary: always"
{"node":"kind-control-plane","namespace":"sample-app","pod":"sample-app-5c9fc8b7d4-wdrck","deployment":"canary"}

Load Testing with k6

Now we’ve made a few requests, let’s see how it behaves under load and whether or not it indeed guarantees a rough 25/75 traffic split!

For Load Testing, I really recommend k6 from the Grafana Labs team. It is a dead-simple yet super powerful tool with very extensive documentation.

See for yourself!

k6 run k6/script.js

After about 1 minute, k6 should be done executing the load test and show you the results.

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: k6/script.js
     output: -

  scenarios: (100.00%) 1 scenario, 20 max VUs, 1m30s max duration (incl. graceful stop):
           * load: Up to 20.00 iterations/s for 1m0s over 2 stages (maxVUs: 20, gracefulStop: 30s)

     ✓ status code is 200
     ✓ node is kind-control-plane
     ✓ namespace is sample-app
     ✓ pod is sample-app-*
     ✓ deployment is stable or canary

   ✓ checks.........................: 100.00% ✓ 3100      ✗ 0
     data_received..................: 157 kB  2.6 kB/s
     data_sent......................: 71 kB   1.2 kB/s
     http_req_blocked...............: avg=42.6µs   min=2µs    med=15µs   max=2.44ms  p(90)=23µs   p(95)=36.29µs
     http_req_connecting............: avg=20.73µs  min=0s     med=0s     max=1.49ms  p(90)=0s     p(95)=0s
   ✓ http_req_duration..............: avg=4.73ms   min=829µs  med=4.09ms max=36.81ms p(90)=7.89ms p(95)=10.14ms
       { expected_response:true }...: avg=4.73ms   min=829µs  med=4.09ms max=36.81ms p(90)=7.89ms p(95)=10.14ms
     http_req_failed................: 0.00%   ✓ 0         ✗ 620
     http_req_rate..................: 50.00%  ✓ 620       ✗ 620
     ✓ { deployment:canary }........: 28.70%  ✓ 178       ✗ 442
     ✓ { deployment:stable }........: 71.29%  ✓ 442       ✗ 178
     http_req_receiving.............: avg=129.17µs min=21µs   med=128µs  max=2.05ms  p(90)=186µs  p(95)=214.04µs
     http_req_sending...............: avg=65.53µs  min=11µs   med=62µs   max=1.96ms  p(90)=92.1µs p(95)=104.04µs
     http_req_tls_handshaking.......: avg=0s       min=0s     med=0s     max=0s      p(90)=0s     p(95)=0s
     http_req_waiting...............: avg=4.53ms   min=774µs  med=3.88ms max=36.6ms  p(90)=7.7ms  p(95)=9.73ms
     http_reqs......................: 620     10.333024/s
     iteration_duration.............: avg=5.4ms    min=1.07ms med=4.81ms max=38.46ms p(90)=8.76ms p(95)=10.76ms
     iterations.....................: 620     10.333024/s
     vus............................: 0       min=0       max=1
     vus_max........................: 20      min=20      max=20

running (1m00.0s), 00/20 VUs, 620 complete and 0 interrupted iterations
load ✓ [======================================] 00/20 VUs  1m0s  00.71 iters/s

That sounds about right!
Out of 620 requests, 178 (28%) were served by the canary Deployment while 442 (72%) were served by the stable one. Pretty good!

Promoting Rollouts

Once feeling confident enough to proceed to the next step, we can now promote our Rollout, either from the dashboard via the PROMOTE button, or via the Kubectl Plugin.

kubectl argo rollouts promote sample-app -n sample-app

Note: while the rollout is ongoing, keep firing some curl requests or run the k6 Load Test script again to observe the change in traffic distribution!

And if feeling so confident that we want to skip all the Rollout steps and get straight to 100% canary traffic, then we can fully promote the Rollout, also either from the dashboard via the PROMOTE-FULL button, or from the command line.

kubectl argo rollouts promote sample-app -n sample-app --full

But if on the contrary, we found an issue while in the middle of rolling out a change, then we can easily abort the Rollout and return to 100% stable traffic, either from the dashboard via the ABORT button, or from the CLI.

kubectl argo rollouts abort sample-app -n sample-app

Finally, once the Rollout is complete and if something were to go wrong, just like one would do with a standard Deployment, we can easily undo a Rollout and roll it back to the previous revision, also either from the dashboard via the ROLLBACK button, or via Kubectl.

kubectl argo rollouts undo sample-app -n sample-app

Wrapping up

That’s it! You can now stop and delete your Kind cluster.

kind delete cluster

To summarize, using Argo Rollouts we were able to:

• Improve our Canary Deployment setup, by leveraging the Rollout custom resource, without which duplicated Deployment and Ingress resources would have to be created;
• Automate the Traffic Routing increments via Rollout steps, which would otherwise have to be done manually by editing the canary Ingress annotation directly.

Was it worth it? Did that help you understand how to implement Canary Deployment in Kubernetes using Argo Rollouts?

If so:

1. Let me know in the comments below! 👇
2. Don’t forget to hit that subscribe button! ✅
3. Follow me on Twitter, I’ll be happy to answer any of your questions and you’ll be the first ones to know when a new article comes out! 👌

See you next week, for Part 3 of my series Canary Deployment in Kubernetes!

Bye-bye! 👋

Canary Deployment in Kubernetes (Part 2) — Automated Canary Deployment using Argo Rollouts was originally published in EGYM Software Development on Medium, where people are continuing the conversation by highlighting and responding to this story.

Learn how to use Ingress NGINX to set up a Canary Deployment allowing isolated Smoke Testing and incremental rollout to minimize risk.

Pssst! I started my own blog!
You can read this very same article over there too!
No paywall, no ad, no Javascript — no bullshit, just pure content.
See you there!

Deploying to production in Kubernetes can be quite stressful. Even after meaningful and reliable automated tests have successfully passed, there is still room for things to go wrong and lead to a nasty incident when pressing the final button.

Thankfully, Kubernetes is made to be resilient to this kind of scenario, and rolling back is a no-brainer. But still, rolling back means that, at least for some time, all of the users were negatively impacted by the faulty change…

What if we could smoke test our change in production before it actually hits real users? What if we could roll out a change incrementally to some users instead of all of them at once? What if we could detect a faulty deployment and roll it back automatically?
Well, that, my friend, is what Canary Deployment is all about!

Minimizing the impact on real users while deploying a risky change to production.

🎬 Hi there, I’m Jean!

In this 3 parts series, we’re going to explore several ways to do Canary Deployment in Kubernetes, and the first one is…
🥁
… using Ingress NGINX! 🎊

Requirements

Before we start, make sure you have the following tools installed:

• Kind
• Kubectl
• Helm
• K6

Note: for MacOS users or Linux users using Homebrew, simply run: brew install kind kubectl helm k6

All set? Let’s go! 🏁

Creating Kind Cluster

Kind is a tool for running local Kubernetes clusters using Docker container “nodes”. It was primarily designed for testing Kubernetes itself, but may be used for local development or CI.

I don’t expect you to have a demo project in handy, so I built one for you.

git clone https://github.com/jhandguy/canary-deployment.git

cd canary-deployment

Alright, let’s spin up our Kind cluster! 🚀

➜ kind create cluster --image kindest/node:v1.27.3 --config=kind/cluster.yaml

Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾

Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

Installing NGINX Ingress Controller

NGINX Ingress Controller is one of the many available Kubernetes Ingress Controllers, which acts as a load balancer and satisfies routing rules specified in Ingress resources, using the NGINX reverse proxy.

NGINX Ingress Controller can be installed via its Helm chart.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

helm install ingress-nginx/ingress-nginx --name-template ingress-nginx --create-namespace -n ingress-nginx --values kind/ingress-nginx-values.yaml --version 4.8.3 --wait

Now, if everything goes according to plan, you should be able to see the ingress-nginx-controller Deployment running.

➜ kubectl get deploy -n ingress-nginx
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx-controller   1/1     1            1           4m35s

Configuring Ingress Canary Annotations

Now that our NGINX Ingress Controller is up and running, let’s get into the thick of it, shall we? 🧐

helm install sample-app/helm-charts/ingress-nginx --name-template sample-app --create-namespace -n sample-app --wait

If everything goes fine, you should eventually see two Deployments with the READY state.

➜ kubectl get deploy -n sample-app
NAME                READY   UP-TO-DATE   AVAILABLE   AGE
sample-app          1/1     1            1           100s
sample-app-canary   1/1     1            1           100s

Alright, let’s take a look at what’s under all this!

➜ ls -1 sample-app/helm-charts/ingress-nginx/templates
canary
deployment.yaml
ingress.yaml
service.yaml
serviceaccount.yaml

➜ ls -1 sample-app/helm-charts/ingress-nginx/templates/canary
deployment.yaml
ingress.yaml
service.yaml

As you can see, most of the resources have been duplicated, except for the serviceaccount.yaml:

• The deployment.yaml and the canary/deployment.yaml are strictly the same, apart from the name and the image tag.
• The service.yaml and the canary/service.yaml are also very much the same, except for the name and the label selector.

This is actually for a simple reason: in the end, the stable and the canary deployments are meant to be very much alike, apart from the image built inside the container, exactly like for a blue/green deployment.

Now, the ingress.yaml and the canary/ingress.yaml are similar, but not very much the same though: you’ll notice some extra annotations in the canary/ingress.yaml.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  ...
  annotations:
    ...
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "{{ .Values.canary.weight }}"
    nginx.ingress.kubernetes.io/canary-by-header: "X-Canary"
...

This is where the magic happens! 🪄

Let’s dive into each of these annotations real quick:

• nginx.ingress.kubernetes.io/canary
  When set to "true", this will let the Ingress Controller know that this is the Ingress routing traffic to the canary Deployment.
• nginx.ingress.kubernetes.io/canary-weight
  When given a value, this will tell the Ingress Controller how to split traffic between the stable and the canary Deployments.
  For instance, a weight of 50 would result in a 50/50 split.
• nginx.ingress.kubernetes.io/canary-by-header
  When given a key, this will allow us to force a request into hitting the canary or the stable deployment, no matter the canary-weight.
  By using the header value always, the request will always land in the canary Deployment, and when using the value never, the request will always land in the stable one.

Right now, the canary-weight is 0 and the canary-by-header is X-Canary. This means that normal requests will never land in the canary Deployment unless a header X-Canary: always is passed to the request.

➜ kubectl describe ingress sample-app-canary -n sample-app
...
Annotations:  ...
              nginx.ingress.kubernetes.io/canary: true
              nginx.ingress.kubernetes.io/canary-by-header: X-Canary
              nginx.ingress.kubernetes.io/canary-weight: 0
...

Let’s give it a try!

curl localhost/success -H "Host: sample.app"

As you can see, no matter how many times you run this command, the pod will always be the same. In fact, this is the stable pod.

➜ curl localhost/success -H "Host: sample.app"
{"node":"kind-control-plane","namespace":"sample-app","pod":"sample-app-6bd9dc6d5d-jstn2","deployment":"stable"}

➜ curl localhost/success -H "Host: sample.app"
{"node":"kind-control-plane","namespace":"sample-app","pod":"sample-app-6bd9dc6d5d-jstn2","deployment":"stable"}

➜ curl localhost/success -H "Host: sample.app"
{"node":"kind-control-plane","namespace":"sample-app","pod":"sample-app-6bd9dc6d5d-jstn2","deployment":"stable"}

This is due to the fact that the current canary-weight is 0 thus 100% of the traffic will normally go to the stable Deployment.

Now let’s try with the X-Canary: always header!

curl localhost/success -H "Host: sample.app" -H "X-Canary: always"

Aha! We are now consistently hitting the canary pod.

➜ curl localhost/success -H "Host: sample.app" -H "X-Canary: always"
{"node":"kind-control-plane","namespace":"sample-app","pod":"sample-app-canary-b559b9d75-gfvxp","deployment":"canary"}

➜ curl localhost/success -H "Host: sample.app" -H "X-Canary: always"
{"node":"kind-control-plane","namespace":"sample-app","pod":"sample-app-canary-b559b9d75-gfvxp","deployment":"canary"}

➜ curl localhost/success -H "Host: sample.app" -H "X-Canary: always"
{"node":"kind-control-plane","namespace":"sample-app","pod":"sample-app-canary-b559b9d75-gfvxp","deployment":"canary"}

Did you pay attention to what we just did? 😉
This has a name: Smoke Testing! 🔥

That’s right, we just smoke-tested a change, in production, while isolating it from the rest of the users as normal requests are routed to the stable Deployment thanks to the canary-weight. However, we bypassed that during Smoke Testing by utilizing the canary-by-header to force the request into hitting the canary Deployment.
Isn’t that awesome?! No more panic attacks when smoke tests fail on Canary, your users aren’t affected. 😌

Rolling-out Canary Deployments Incrementally

So we have solved one problem, yet one remains: How do we expose a change to our real users incrementally, to minimize the impact if something goes south?

Well, you’ve probably guessed it, canary-weight to the rescue! 🚀

We can simply slowly increase the canary-weight in order to let some users being routed to the canary Deployment. Let’s give it a try!

helm upgrade sample-app sample-app/helm-charts/ingress-nginx -n sample-app --reuse-values --set canary.weight=50 --wait

Let’s verify that our canary Ingress now has a canary-weight of 50!

➜ kubectl describe ingress sample-app-canary -n sample-app
...
Annotations:  ...
              nginx.ingress.kubernetes.io/canary: true
              nginx.ingress.kubernetes.io/canary-by-header: X-Canary
              nginx.ingress.kubernetes.io/canary-weight: 50
...

All good, let’s test it out!

curl localhost/success -H "Host: sample.app"

As you can see, traffic is now equally split between the stable and canary Deployments.

➜ curl localhost/success -H "Host: sample.app"
{"node":"kind-control-plane","namespace":"sample-app","pod":"sample-app-6bd9dc6d5d-jstn2","deployment":"stable"}

➜ curl localhost/success -H "Host: sample.app"
{"node":"kind-control-plane","namespace":"sample-app","pod":"sample-app-canary-5754f4bbc7-lvthj","deployment":"canary"}

➜ curl localhost/success -H "Host: sample.app"
{"node":"kind-control-plane","namespace":"sample-app","pod":"sample-app-6bd9dc6d5d-jstn2","deployment":"stable"}

➜ curl localhost/success -H "Host: sample.app"
{"node":"kind-control-plane","namespace":"sample-app","pod":"sample-app-canary-5754f4bbc7-lvthj","deployment":"canary"}

Load Testing with k6

Now we’ve made a few requests, let’s see how it behaves under load and whether or not it indeed guarantees about 50% traffic distribution!

For Load Testing, I really recommend k6 from the Grafana Labs team. It is a dead-simple yet super powerful tool with very extensive documentation.

See for yourself!

k6 run k6/script.js

After about 1 minute, k6 should be done executing the load test and show you the results.

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: k6/script.js
     output: -

  scenarios: (100.00%) 1 scenario, 20 max VUs, 1m30s max duration (incl. graceful stop):
           * load: Up to 20.00 iterations/s for 1m0s over 2 stages (maxVUs: 20, gracefulStop: 30s)

     ✓ status code is 200
     ✓ node is kind-control-plane
     ✓ namespace is sample-app
     ✓ pod is sample-app-*
     ✓ deployment is stable or canary

   ✓ checks.........................: 100.00% ✓ 3095      ✗ 0
     data_received..................: 160 kB  2.7 kB/s
     data_sent......................: 71 kB   1.2 kB/s
     http_req_blocked...............: avg=48.21µs  min=3µs    med=18µs   max=2.38ms  p(90)=23µs   p(95)=30.09µs
     http_req_connecting............: avg=24.56µs  min=0s     med=0s     max=1.48ms  p(90)=0s     p(95)=0s
   ✓ http_req_duration..............: avg=4.87ms   min=800µs  med=4.24ms max=34.52ms p(90)=7.85ms p(95)=9.88ms
       { expected_response:true }...: avg=4.87ms   min=800µs  med=4.24ms max=34.52ms p(90)=7.85ms p(95)=9.88ms
     http_req_failed................: 0.00%   ✓ 0         ✗ 619
     http_req_rate..................: 50.00%  ✓ 619       ✗ 619
     ✓ { deployment:canary }........: 47.65%  ✓ 295       ✗ 324
     ✓ { deployment:stable }........: 52.34%  ✓ 324       ✗ 295
     http_req_receiving.............: avg=132.17µs min=24µs   med=136µs  max=610µs   p(90)=179µs  p(95)=213.09µs
     http_req_sending...............: avg=70.69µs  min=15µs   med=72µs   max=736µs   p(90)=94.2µs p(95)=101µs
     http_req_tls_handshaking.......: avg=0s       min=0s     med=0s     max=0s      p(90)=0s     p(95)=0s
     http_req_waiting...............: avg=4.67ms   min=719µs  med=4.02ms max=34.2ms  p(90)=7.64ms p(95)=9.65ms
     http_reqs......................: 619     10.316511/s
     iteration_duration.............: avg=5.59ms   min=1.08ms med=5.02ms max=41.22ms p(90)=8.68ms p(95)=10.52ms
     iterations.....................: 619     10.316511/s
     vus............................: 0       min=0       max=0
     vus_max........................: 20      min=20      max=20

running (1m00.0s), 00/20 VUs, 619 complete and 0 interrupted iterations
load ✓ [======================================] 00/20 VUs  1m0s  00.71 iters/s

That sounds about right!
Out of 619 requests, 295 (48%) were served by the canary Deployment while 324 (52%) were served by the stable one. Pretty good!

Wrapping up

That’s it! You can now stop and delete your Kind cluster.

kind delete cluster

To summarize, using Ingress NGINX we were able to:

• Smoke test our change in complete isolation from the rest of the users;
• Incrementally expose our change to some users to minimize risk.

Was it worth it? Did that help you understand how to implement Canary Deployment in Kubernetes using Ingress NGINX?

If so:

1. Let me know in the comments below! 👇
2. Don’t forget to hit that subscribe button! ✅
3. Follow me on Twitter, I’ll be happy to answer any of your questions and you’ll be the first ones to know when a new article comes out! 👌

See you next week, for Part 2 of my series Canary Deployment in Kubernetes!

Bye-bye! 👋

Canary Deployment in Kubernetes (Part 1) — Simple Canary Deployment using Ingress NGINX was originally published in EGYM Software Development on Medium, where people are continuing the conversation by highlighting and responding to this story.

Pssst! I started my own blog!
You can read this very same article over there too!
No paywall, no ad, no Javascript — no bullshit, just pure content.
See you there!

For many companies, testing and releasing are still very blurry processes, which don’t seem to work as they should. Testing is mostly manual, slow and error-prone. Releasing is usually also manual and slow, making its frequency hard to maintain on a sprint-basis. Continuous Integration (CI) and Continuous Deployment (CD) are all about automating both testing and releasing, making it possible for a team to release every week or two. At EGYM, we have achieved just that and I am going to tell you all about it!

Testing & Automating is easy, but doing it at scale isn’t.

🎬 Hi there, I’m Jean!

In this article, I want to share with you my experience in our Path to CI / CD Nirvana in iOS, at EGYM! 😃

Continuous Integration (CI)

CI is the process of ensuring that a new change added on top of previous ones won’t break what has been built so far, making developments smooth and flawless. 🏎

That pretty much means most of the time “Testing before Merging”. ✅

Do you remember the testing pyramid? Well, I won’t go through it in detail, but we have pretty much 3 layers, Unit Testing, Integration Testing and finally UI Testing. All of those are crucial for ensuring the quality of your app, but the one, in particular, I want to focus with you today is UI Testing. 📱

We all know it, UI Testing is a pain when it comes to stability and maintainability, but don’t you worry, I might just have what you are looking for! 🧐

UI Testing in iOS - The Ultimate Guide

That’s right, I made a complete guide for you to get started with stable, maintainable and scalable UI testing in iOS! 🎉

🔌 Mocking the Network

The first thing you need to take care of is Mocking the Network. You don’t want to let your network requests hit an actual network and there are mainly two reasons for that.

Isolation. Of course, what if your backend friends break their test server? Well, your UI Tests will fail as well, even though nothing is wrong with the app 😱 So now you’ll say “Well let's deploy a dedicated environment for it then!”. And whose ownership will that be? iOS devs? Or Backend devs? Isn’t that a little bit over-engineering? 🤔 Mocking the Network allows you to avoid unnecessary headaches and “Just re-trigger it!” kind of workflows. 😆

Monitoring. Well yes, if you mock or stub the network, that means you need to predict exactly how many calls are going to be made during your UI test and that’s extremely valuable because you might sometimes find out that your app is actually making unnecessary calls which are eating your user’s data! 😮

Overall, Mocking the Network gives you full control and helps to keep your UI tests stability to a very high standard! 💪

If you want to read more about Mocking the Network, go ahead and check out my dedicated article! 👀

🚦 Stubbing the Navigation

The second thing you want to address is Stubbing the Navigation. One major pain point with iOS UI Testing is that every test will always relaunch the app entirely. This means that you are forced to go through several screens manually before getting to the one you actually want to test. That breaks the isolation principle, making UI tests hard to debug. 😕

One way of solving this is to inject a stub that will route your navigation in the exact screen you want to end up on. Of course, that also means that your navigation logic must be centralised in a dedicated layer like the Coordinator for instance! 🚦

That way, your UI Tests will suddenly become blazingly fast, but also more isolated, stable and easier to maintain! 🚀

If you want to read more about Stubbing the Navigation, go ahead and check out my dedicated article! 👀

📺 Disabling Animations

The third action you might want to take is Disabling Animations. This one is a quick and easy win, simply disabling animations while UI testing will make your tests not only faster but more stable as animations won’t be a potential risk for your assertions. 👍

If you want to read more about Disabling Animations, go ahead and check out my dedicated article! 👀

🔮 Generating Accessibility Identifiers using Reflection

Until now, Accessibility Identifiers have always been assigned manually. This is a pain and most importantly unnecessary noise in your code so you should avoid doing this. Generating Accessibility Identifiers using Reflection is a great way to get rid of all this manual work, but instead, let your code generate it for you at runtime! 🤯

If you want to read more about Generating Accessibility Identifiers using Reflection, go ahead and check out my dedicated article! 👀

🤖 Implementing the Robot Pattern

Finally, UI Tests are not only meant to be verifying that your app still behaves as expected, but they also can serve as implicit documentation of how your app works, what the main features are, for your new hires who do not have a clue about the product yet! 😮

Implementing the Robot Pattern will not only help you separate the What from the How of your UI Tests and make screen assertions reusable across UI Tests, but also provide a more human-readable way of specifying tests! 😎

If you want to read more about Implementing the Robot Pattern, go ahead and check out my dedicated article! 👀

Continuous Deployment (CD)

CD is the process of deploying a new version of the product, for every new change that is made, making deployments incremental meaning, less risky than releases sent-out in big chunks. This process should ideally be fully automated, just like CI. 🚀

Fastlane Swift

When it comes to automation for iOS we, at EGYM, use Fastlane Swift.
It is Fastlane but in Swift! 😱🍾

The choice of Fastlane was simple, we must be able to trigger automated tasks remotely as well as locally. And using Fastlane Swift, you can democratise your CI / CD automation to the whole iOS team, as they don’t need to learn Ruby anymore: their extensive Swift knowledge will do! 👌

Automate Everything

Currently, we have a total of 12 lanes:

✅ Run Tests (Unit / Integration / UI)

Obviously, running tests is the first process you might want to automate. 🤖
In Fastlane Swift, this is easy, simply call therunTests action fromFastfile.swift and you’re all set! 🚀

At EGYM, we run the tests on every Pull Request.

📸 Capture Screenshots

Now, the capture screenshots lane is pretty similar to the run tests one, simply call the action captureScreenshots fromFastfile.swift and if you want to be fancy, also call the frameScreenshots action to add the device’s frame around your screenshots! 🖼

At EGYM, we use this lane to capture, frame and share our localised screenshots with our designers so they can reuse them for marketing purposes, this saves them a lot of time and they’re really grateful for it! 🤗

✈️ Upload To Testflight

Of course, uploading to Testflight is also crucial to your development team. 👨‍✈
It requires a little more logic though, so hold on to your seats! 💺

Here are the actions you need to sequentially use:

• Get the latest Testflight build number usinglatestTestflightBuildNumber;
• Increment the build number usingincrementBuildNumber;
• Build the app usingbuildApp;
• Upload a new build to Testflight usinguploadToTestflight;
• Download.dSYMsusingdownloadDsyms;
• Upload.dSYMsto Crashlytics usinguploadSymbolsToCrashlytics;

At EGYM, we automatically submit a new Testflight build after every Git merge to ourdevelopbranch.

📱 Upload To AppStore

Now, uploading to AppStore is a whole different story. 😆

Overall, here are the steps we follow at EGYM:

• Generate Metadata from ourAppStore.stringslocalization file ;
• Download Screenshots from a Google Cloud Storage bucket ;
• Upload a new build to Testflight ;
• Upload to AppStore the metadata, screenshots and link the Testflight build ;
• Increment the version number (patch, minor or major) ;
• Add, commit and push the changes (build increment) ;
• Create Pull Requests ;
• Track Deployment Frequency.

At EGYM, our AppStore releases are triggered manually from Bitrise, but once triggered, everything else is automated, including sending for Apple Review and releasing using Phased Rollout! 🚀

🐛 Upload To Crashlytics Beta

Unfortunately, AppStoreConnect’s build processing can take quite some time (~20 minutes); therefore, we also like to use Crashlytics Beta to distribute our development builds. 👍

At EGYM, we automatically submit a new Crashlyitics Beta build on every Git push.

📝 Sync Code Signing

Syncing Code Signing Certificates and Profiles is a very trivial process using Fastlane, simply call the actionsyncCodeSigningfromFastfile.swiftand you are good to go! 🚗

At EGYM, we mostly use this lane for downloading development certificates.

🛬 Download Translations

Remembering the exact CLI parameters can be a pain sometimes, that is why we also have a lane for managing our Lokalise translations! 🔡

At EGYM, our Download Translations lane simply calls thelokaliseCLI under the hood.

🛫 Upload Translations

Similarly to Downloading Translations to Lokalise, we also have a lane for uploading them. Same way as the downloading lane, the upload one also uses thelokaliseCLI under the hood. 🔤

🖨 Generate Metadata

Fastlane Metadata is a folder/file representation of what information AppStore Connect expects from your app when releasing. 🚀

At EGYM, we’ve automated the generation of Metadata by creating a Swift script expecting a localisation file(AppStore.strings)as input! 🤖

✨ Update Dependencies

Updating dependencies automatically is very helpful, its main purpose is to let you know when a dependency has a new version, by creating a Pull Request when something has changed in any of your.lock or .resolvedfiles. 🔐

Simply create a lane that executes for instancebundle updateorcarthage updateand submits a Pull Request! 📝

At EGYM, we use it to update ourGemfile.lockandCartfile.resolvedfiles! 👌

🖼 Update Screenshots

As capturing screenshots can take quite some time, doing it as part of the release process might not be very wise. 🤔

Instead, it might be more efficient to upload the screenshots inside a Google Cloud Storage bucket and update them on a weekly / monthly basis! 💡

At EGYM, we currently host about 220 screenshots inside a dedicated Google Cloud Storage bucket, in 2 different languages for 3 different devices! 🤯

🎫 Move Ticket to Ready for Acceptance Testing

As I said earlier, we automate everything. And that includes moving tickets in our Sprint Dashboard! 😆

Whenever we send a build to Testflight after a merge, once the build processing has been completed, we move the corresponding ticket to “Ready for Acceptance Testing” and send a message on Slack to our QA testers! 🚀

I know, that’s babysitting at this point but well, our QA testers love it! 😍

Fully Automated Release - The Holy Grail

That’s right, we made it, the holy grail: a fully automated release! 👑

This was a long journey, but by combining the power of both Bitrise and Fastlane, we’ve managed to submit our app for AppStore release, create version bump Pull Requests and track our Deployment Frequency, with a simple click of a button! 😱

Here is the full sequence of actions we perform automatically:

1. Create branch “release-\(version)” (from develop)
2. Download AppStore.strings localisation file from Lokalise
3. Generate metadata from AppStore.strings localisation file
4. Download screenshots from Google Cloud Storage bucket
5. Upload new build to Testflight
6. Upload DSYM to Crashlytics
7. Release to AppStore with metadata, screenshots and Testflight build
8. Add Git tag “releases/\(version)”
9. Increment Version Number (patch, minor, major)
10. Commit and Push
11. Create Pull Requests (develop, master)
12. Track Deployment Frequency in Deployment Logbook

All we need to do is trigger our UploadToAppStore workflow on Bitrise from the branch develop and a few minutes later, we get a Slack message confirming that the release was successful, along with 2 freshly created Pull Requests, one for the actual release (master) and one for the Version Bump (develop)! 🚀

Conclusion

It’s been a long journey, but definitely worthwhile! 🎉
Our workflows work so well, that we are able to catch bugs early on in our development process, as well as keeping a high Deployment Frequency without compromising on our developments speed! ✈️

Worth a try? Would you like to follow our path to CI / CD Nirvana? If so:

1. Let me know in the comments below how are you doing CI / CD in your company! 👇
2. Don’t forget to hit that subscribe button to be notified of my future articles about Fastlane Swift! ✅
3. Follow me on Twitter, I’ll be happy to answer any of your questions and you’ll be the first ones to know when a new article comes out! 👌

Until next time, happy automating! 🤖

Bye-bye! 👋

A Path to CI / CD Nirvana in iOS was originally published in EGYM Software Development on Medium, where people are continuing the conversation by highlighting and responding to this story.

Pssst! I started my own blog!
You can read this very same article over there too!
No paywall, no ad, no Javascript — no bullshit, just pure content.
See you there!

Most of the time, UI Testing gets abandoned because it becomes harder to maintain over time and one reason for that is: readability. We also tend to forget that UI tests should not only serve the purpose of verification but provide meaningful documentation that everybody (including non-developers) can understand. Alright, let’s get right into it, fellas!

What if we could write UI Tests in a human-readable language, that even a Product Owner could understand?

🎬 Hi there, I’m Jean!

In this article, I am going to introduce a better way to write UI Tests using XCTest! 😃
Note: third-party libraries such as Appium or Calabash won’t be part of this post.

The usual way

Usually, all UI Testing logic is bundled in one test function. This inevitably leads to this…

func uglyUITest() {
    ...
    app.launch()
    let title = app.navigationBars.firstMatch.otherElements.firstMatch
    XCTAssert(title.isHittable)
    XCTAssertEqual(title.label, "U.S. - Politics")
    let stackView = app.otherElements["StoryView.stackView"]
    let multimediaImage = stackView.images["StoryView.multimediaImageView"]
    XCTAssert(multimediaImage.isHittable)
    let titleLabel = stackView.staticTexts["StoryView.titleLabel"]
    XCTAssertEqual(titleLabel.label, "A Trump Endorsement Can Decide a Race. Here’s How to Get One.")
    let abstractLabel = stackView.staticTexts["StoryView.abstractLabel"]
    XCTAssertEqual(abstractLabel.label, "The president’s grip on G.O.P. primary voters is as strong as it has been since he seized the party’s nomination.")
    let bylineLabel = stackView.staticTexts["StoryView.bylineLabel"]
    XCTAssertEqual(bylineLabel.label, "By JONATHAN MARTIN and MAGGIE HABERMAN")
    snapshot("Story")
}

I am sure you didn’t even finish reading that you already jumped to this line.
And to be fair, I’m not angry at you; this UI Test was fairly unreadable! 😵

And the main reason for that is:

Common UI Tests are mixing the What and the How altogether, in one place.

Well, that’s obviously a problem, as this isn’t following the Separation of Concern Clean Code teaches us… 🤔

Beep… Beep… Boop… 🤖

That’s right! Introducing, the Robot Pattern by Jake Wharton! 👏
The idea Jake had for the Android community is to separate in a UI Test the What from the How.
And to achieve this, he introduced a new layer called: Robot.

Transcribed to iOS (Swift), our UI Test now looks like this…

func prettyUITest() {
    ...
    StoryRobot(app)
        .start()
        .checkTitle(contains: "U.S. - Politics")
        .checkStoryImage()
        .checkStoryTitle(contains: "A Trump Endorsement Can Decide a Race. Here’s How to Get One.")
        .checkStoryAbstract(contains: "The president’s grip on G.O.P. primary voters is as strong as it has been since he seized the party’s nomination.")
        .checkStoryByline(contains: "By JONATHAN MARTIN and MAGGIE HABERMAN")
        .takeScreenshot(named: "Story")
}

How about that, didn’t it feel right to read the whole function this time? 😍
Exactly, in the UI Test itself, we are now left only with the What and the How is delegated to the Robot. Meaning: anybody can read and understand it! 👍

The Robot in action!

First and foremost, we are going to implement a base Robot.
This class will take care of setting up Fastlane’s Snapshot, launching the app and providing a nice panel of util functions, such as tapping on elements, making assertions and taking screenshots… 👌

class Robot {

    private static let defaultTimeout: Double = 30

    var app: XCUIApplication

    lazy var navigationBar       = app.navigationBars.firstMatch
    lazy var navigationBarButton = navigationBar.buttons.firstMatch
    lazy var navigationBarTitle  = navigationBar.otherElements.firstMatch

    init(_ app: XCUIApplication) {
        self.app = app
        setupSnapshot(app)
    }

    @discardableResult
    func start(timeout: TimeInterval = Robot.defaultTimeout) -> Self {
        app.launch()
        assert(app, [.exists], timeout: timeout)

        return self
    }

    @discardableResult
    func tap(_ element: XCUIElement, timeout: TimeInterval = Robot.defaultTimeout) -> Self {
        assert(element, [.isHittable], timeout: timeout)
        element.tap()

        return self
    }

    @discardableResult
    func assert(_ element: XCUIElement, _ predicates: [Predicate], timeout: TimeInterval = Robot.defaultTimeout) -> Self {
        let expectation = XCTNSPredicateExpectation(predicate: NSPredicate(format: predicates.map { $0.format }.joined(separator: " AND ")), object: element)
        guard XCTWaiter.wait(for: [expectation], timeout: timeout) == .completed else {
            XCTFail("[\(self)] Element \(element.description) did not fulfill expectation: \(predicates.map { $0.format })")
            return self
        }

        return self
    }

    @discardableResult
    func checkTitle(contains title: String, timeout: TimeInterval = Robot.defaultTimeout) -> Self {
        assert(navigationBar, [.isHittable], timeout: timeout)
        assert(navigationBarTitle, [.contains(title)], timeout: timeout)

        return self
    }

    @discardableResult
    func takeScreenshot(named name: String, timeout: TimeInterval = Robot.defaultTimeout) -> Self {
        snapshot(name, timeWaitingForIdle: timeout)

        return self
    }

    @discardableResult
    func back(timeout: TimeInterval = Robot.defaultTimeout) -> Self {
        tap(navigationBarButton, timeout: timeout)

        return self
    }
}

Simple, right?
Oh, by the way, here is the Predicate enum I am using for making assertions, you’ll find it quite handy! 😎

enum Predicate {
    case contains(String), doesNotContain(String)
    case exists, doesNotExist
    case isHittable, isNotHittable

    var format: String {
        switch self {
        case .contains(let label):
            return "label == '\(label)'"
        case .doesNotContain(let label):
            return "label != '\(label)'"
        case .exists:
            return "exists == true"
        case .doesNotExist:
            return "exists == false"
        case .isHittable:
            return "isHittable == true"
        case .isNotHittable:
            return "isHittable == false"
        }
    }
}

Alright! Now our Base Robot class is ready, let’s inherit from it in the StoryRobot and see how it plays out! 🤖

final class StoryRobot: Robot {

    private lazy var stackView       = app.otherElements["StoryView.stackView"]
    private lazy var multimediaImage = stackView.images["StoryView.multimediaImageView"]
    private lazy var titleLabel      = stackView.staticTexts["StoryView.titleLabel"]
    private lazy var abstractLabel   = stackView.staticTexts["StoryView.abstractLabel"]
    private lazy var bylineLabel     = stackView.staticTexts["StoryView.bylineLabel"]

    @discardableResult
    func checkStoryImage() -> Self {
        assert(multimediaImage, [.exists])

        return self
    }

    @discardableResult
    func checkStoryTitle(contains storyTitle: String) -> Self {
        assert(titleLabel, [.isHittable, .contains(storyTitle)])

        return self
    }

    @discardableResult
    func checkStoryAbstract(contains storyAbstract: String) -> Self {
        assert(abstractLabel, [.isHittable, .contains(storyAbstract)])

        return self
    }

    @discardableResult
    func checkStoryByline(contains storyByline: String) -> Self {
        assert(bylineLabel, [.isHittable, .contains(storyByline)])

        return self
    }
}

That’s it! We’ve just implemented the How of our UI Test! 🎊
What’s also really nice about Robots, is that they can cross-call themselves without breaking the continuity of our UI Tests! 🤯

For instance, let’s say we want to open Safari in the StoryUITest.
All we need to do is create a SafariRobot which will take care of UI Testing the browser and create a function inside the StoryRobot that will open Safari as well as return the SafariRobot! 😮

lazy var urlButton = stackView.buttons["StoryView.urlButton"]

@discardableResult
func openSafari() -> SafariRobot {
    tap(urlButton)

    return SafariRobot(app)
}

And as you can see now, the continuity of our UI Test isn’t disturbed…

final class StoryUITest: XCTestCase {

    private lazy var app = XCUIApplication()

    func testStory() {
        StoryRobot(app)
            .start()
            .checkTitle(contains: "U.S. - Politics")
            .checkStoryImage()
            .checkStoryTitle(contains: "A Trump Endorsement Can Decide a Race. Here’s How to Get One.")
            .checkStoryAbstract(contains: "The president’s grip on G.O.P. primary voters is as strong as it has been since he seized the party’s nomination.")
            .checkStoryByline(contains: "By JONATHAN MARTIN and MAGGIE HABERMAN")
            .takeScreenshot(named: "Story")
            .openSafari()
            .checkURL(contains: "https://www.nytimes.com/2018/08/27/us/politics/trump-endorsements.html")
            .closeSafari()
    }
}

Boom! 💥
We just implemented the Robot Pattern! 🎉
UI Testing is now easy, elegant and readable! 💪

Worth a try? Did you regain faith in UI Testing? If so:

1. Let me know in the comments below! 👇
2. Don’t forget to hit that subscribe button! ✅
3. Follow me on Twitter, I’ll be happy to answer any of your questions and you’ll be the first ones to know when a new article comes out! 👌

Want to read more about UI Testing?
Check out UI Testing in iOS - Generating Accessibility Identifiers using Reflection, as well as my 3 parts series:

1. Painless UI Testing in iOS (Part 1) — Mocking the Network
2. Painless UI Testing in iOS (Part 2) — Stubbing the Navigation
3. Painless UI Testing in iOS (Part 3) — Disabling Animations

If you’d like to see a real-life example of an app doing extensive UI Testing, don’t hesitate to check out my Github repo! 📦

Until next time, happy testing! 🤖

Bye-bye! 👋

UI Testing in iOS - Robot Pattern was originally published in EGYM Software Development on Medium, where people are continuing the conversation by highlighting and responding to this story.