werf’s first commit was on January 22, 2016, and the project is now celebrating its 10th anniversary. To honor its anniversary, we decided to take a look at its key moments, milestones, wins, and future plans.

Here’s a brief timeline:

Below, you can find a more detailed story of how werf evolved.

werf v0 (dapp): January 2016 — December 2018

The project was originally known as dapp, and it started as a tool created in a DevOps agency for building container images within CI/CD pipelines. The focus on efficiency made it special: the incremental approach allowed reusing the results from the previous builds.

The first version was written in Ruby. We used a Vagrantfile-like Dappfile for defining build configurations, which featured an imperative Ruby-based DSL.

dimg 'symfony-demo-app' do
  docker.from 'ubuntu:16.04'

  git do
    add '/' do
      to '/demo'
      stage_dependencies.before_setup 'composer.json', 'composer.lock'
    end
  end

  shell do
    before_install do
      run 'apt-get update',
          'apt-get install -y curl php7.0',
          # add the phpapp user
          'groupadd -g 242 phpapp',
          'useradd -m  -d /home/phpapp -g 242 -u 242 phpapp'
    end
    install do
      run 'apt-get install -y php7.0-sqlite3 php7.0-xml php7.0-zip',
          # install composer
          'curl -LsS https://getcomposer.org/download/1.4.1/composer.phar -o /usr/local/bin/composer',
          'chmod a+x /usr/local/bin/composer'
    end
    before_setup do
      # modify source code permissions and run composer install
      run 'chown phpapp:phpapp -R /demo && cd /demo',
          "su -c 'composer install' phpapp"
    end
    setup do
      # use the current date as the application version
      run 'echo `date` > /demo/version.txt',
          'chown phpapp:phpapp /demo/version.txt'
    end
  end

  # the port must match the port specified in start.sh
  docker.expose 8000
end

The standout feature was its stage-based build model. Every single instruction created a new Docker image (a “stage”). This whole approach became the basis for build orchestration and efficient caching. Even in its first version, werf/dapp supported building any number of images in parallel using a shared configuration.

June 2016: Advanced build debugging tools

Pretty early in the game, dapp has got some build debugging tools. The user could hop into a container interactively at any build stage — before instructions were executed, after they completed, or upon an error. This greatly simplified the development and debugging of complex build scenarios and became one of dapp’s key features at the time.

July 2016: Chef support for describing assembly instructions

As builds grew more complex, it became clear that defining modular logic in a plain shell was a pain. To address this, we introduced support for Chef, a tool widely used in our company at the time.

This way, we could use Chef recipes during the image build process while ensuring no Chef artifacts or its cookbooks cluttered up the final image. So dapp has got a powerful mechanism for modularity and build logic reuse:

dimg do
  docker.from 'ubuntu:16.04'

  git.add('/').to('/app')
  docker.workdir '/app'

  docker.cmd ['/bin/bash', '-lec', 'bundle exec ruby app.rb']
  docker.expose 4567

  chef do
    cookbook 'apt'
    cookbook 'rvm'

    recipe 'ruby'
    recipe 'bundle_gems'
    recipe 'app_config'
  end
end

March 2017: Deploying to Kubernetes via Helm

Next year, dapp gained the capability to deploy applications to Kubernetes. It worked by calling the system’s Helm, and you just had to have a chart in the .helm directory.

But dapp did more than just calling Helm. It introduced linking the build and deploy steps right in the templates, using special Helm functions to handle images. We also added a basic way to track resource statuses during a deployment.

December 2017: Smart cleanup of the container registry

With more and more builds and tags piling up, cleaning up the container registry became a priority. So, at the end of 2017, we added a cleanup feature to dapp. It was a garbage collection system that worked with different policies (like for branches, commits, or tags) and was smart enough to check which images were being used in Kubernetes.

What are the challenges of cleaning up the container registry? Read more about our approach here.

March 2018: Support for YAML configuration and Ansible for describing assembly instructions (Chef gets replaced)

At this point, dapp shifted to a more user-friendly and familiar approach by adding YAML support. Moving from the imperative Ruby DSL to a declarative syntax made configurations easier to read, more predictable, and simpler to manage (particularly in complex projects).

At the same time, we moved away from Chef in favor of Ansible for build instructions. Chef turned out to be too cumbersome and didn’t live up to our expectations. Ansible, on the other hand, let us keep the modular and declarative style we wanted, while making it easier to get started with and more straightforward to run. Here’s how it looked:

dimg: ~
from: alpine:latest
git:
- add: /
  to: /app
  owner: app
  group: app
  excludePaths:
  - public/assets
  - vendor
  - .helm
  stageDependencies:
    install:
    - package.json
    - Bowerfile
    - Gemfile.lock
    - "app/assets/*"
- url: https://github.com/kr/beanstalkd.git
  add: /
  to: /build
ansible:
  beforeInstall:
  - name: "Create non-root main application user"
    user:
      name: app
      comment: "Non-root main application user"
      uid: 7000
      shell: /bin/bash
      home: /app
  - name: "Disable docs and man files installation in dpkg"
    copy:
      content: |
        path-exclude=/usr/share/man/*
        path-exclude=/usr/share/doc/*
      dest: /etc/dpkg/dpkg.cfg.d/01_nodoc
  install:
  - name: "Precompile assets"
    shell: |
      set -e
      export RAILS_ENV=production
      source /etc/profile.d/rvm.sh
      cd /app
      bundle exec rake assets:precompile
    args:
      executable: /bin/bash

October 2018: Spinning off deployment tracking into kubedog

The logic for tracking deployments (including statuses, events, logs, and waiting for resources to become ready) was moved out of dapp and into its own project: kubedog.

Separating kubedog into a standalone library allowed us to isolate this feature, make it reusable for other tools, and evolve its monitoring capabilities independently of the main product. Over time, kubedog became a key component of the werf ecosystem and was also adopted by the community for other projects.

November 2018: Support for secret values and files in deployment configs

dapp has got support for secret values and secret files for deployment configurations. That addressed the challenge of securely storing and using sensitive data like tokens, access keys, TLS certificates, and private keys — without having them stored in plain text in Git (or your Helm charts). Secrets were decrypted only at deployment time for use in Kubernetes templates and manifests

December 2018: Seamless migration from Ruby to Go

With the project’s growth, expanding use cases, and feedback from our users, it became obvious that the Ruby implementation was holding us back. So, we commenced the migration to Go.

The migration was done gradually to ensure a smooth experience for users, allowing us to keep developing the product while preserving its stability. In the end, we got rid of the old limitations and made it easier to integrate with the Kubernetes ecosystem.

werf v1: December 2018 — March 2020

January 2019: A new name: werf

This is when the project got its new name: werf. We allowed everyone in the company and the wider community participate in discussions and voting for the name. We ended up with around 100 suggestions, from sea and pirate themes to more abstract and technical concepts.

After the vote, three front-runners were identified:

• grog — 32 %
• flimb — 29,7 %
• werf — 27 %

Although werf did not end up on the first place in the vote, the team ultimately went with it. It just “clicked” — best reflected the project’s vibe, being associated with a place of assembly and creation — a “shipyard” (“werf” in Dutch) — which organically fit into the brand’s vision and future plans. (By the way, note that we prefer to use a lowercase initial when spelling “werf.”)

January 2019: The werf update manager (multiwerf)

multiwerf implemented our approach to handling versions through update channels. Its job was to auto-update werf and, crucially, to isolate a specific version for the active shell session.

The . $(multiwerf use 1.1 stable --as-file) shell command updated werf automatically in the background from a channel set by the user (stable for v1.1) and “pinned” that version for the active shell session. So you could use multiple werf versions on the same machine without any conflicts.

January 2019: Availability on all major operating systems

This is when werf was made available for all major operating systems, with testing and distribution now covering Linux, macOS, and Windows.

April 2019: Switching to our own fork of Helm

A major architectural choice for v1 was to build Helm right into the werf binary. Both the Helm client and Tiller ran inside the same werf process during deployment, so you didn’t need to install anything extra in your Kubernetes cluster. That approach ensured werf was fully compatible with Helm 2 while getting rid of most of its operational headaches and made things more secure.

Fun fact: Helm didn’t adopt this Tiller-less approach until Helm 3 came out in November 2019.

Having Helm built-in allowed us to level-up our deployment process. Instead of treating Helm like a black box with the --wait flag, werf started keeping a close eye on resources. It could track statuses, print events and logs, and stop the deployment process instantly on an error instead of waiting for a timeout. All the monitoring logic for this came from our kubedog project, which we had already spun off into its own solid library.

August 2019: 3-way merge is implemented in our Helm 2 fork

By this point, werf had begun contributing to the Helm upstream while at the same time developing its own Helm fork to deliver value to users more quickly and to experiment with features the official Helm lacked. One such extension was the 3-way merge for updating Kubernetes resources.

Having 3-way merge in our Helm 2 fork meant we could apply changes to existing resources way more accurately, because we considered their actual state in the cluster. This feature was introduced in werf before the Helm 3 showed up. It significantly improved the predictability and safety of updates.

To enhance reliability, werf also introduced locks to prevent parallel deployments of the same release. This helped prevent race conditions and inconsistent states when multiple deployments happened at once.

August 2019: First 1,000 stars on GitHub

The project surpassed the 1,000-star milestone on GitHub, a clear signal of growing interest and recognition from the Open Source community.

September 2019: Dockerfile support

Adding Dockerfile support was a major step in welcoming more users and making migration easier. werf learned to work with both its own Stapel syntax and classic Dockerfiles.

December 2019: Distributed locking put into a separate project

We pulled out the Go library for distributed locking from werf and made it into a separate tool: lockgate. It supports both local file locks and distributed locking via Kubernetes or an HTTP lock server, making it suitable for different infrastructure scenarios.

The library was well-received by the community, garnering external contributions and seeing adoption in other projects (over 250 ⭐ on GitHub), which proved it was a genuinely useful tool — again, not just for werf!

werf v1.1: March 2020 — November 2020

March 2020: Content-based tagging

Support for content-based tags was introduced. This laid out the basis for the subsequent move away from contextual tagging strategies (based on branches, commits, or CI) and toward a unified approach to image handling — at the configuration, cleanup, and storage levels — and made our use of the container registry way more efficient.

April 2020: Distributed layer storage based on container registry

From this moment, werf took on the task of synchronizing parallel builders that use the same container registry. The mechanism was modeled after how Docker handles its local storage when saving, selecting, and ensuring layer immutability.

The key difference was the scale: Docker synchronized processes that use a single host’s local storage, whereas werf applied this principle to a distributed environment. This way, multiple builders could work with the same container registry at the same time — no conflicts.

werf v1.2: November 2020 — April 2024

December 2020: Bundle support is added

Bundling was a new way to ship a Helm chart and all its container images as a single artifact package. Bundles contain everything you need to deploy an application, and you can distribute them around and use them without relying on the original Git repo.

Published bundles can be deployed with werf or any tool that supports Helm charts in OCI registries (like Helm, Argo CD, Flux). You can copy them between container registries, export them as tar files, use offline, or in air-gapped environments. When creating a bundle, werf automatically includes image details, tags, passed values, and global annotations and labels in the chart. This enables reproducible and standalone deployments — all without being tied to the project’s Git repository.

May 2021: Secure update manager to replace multiwerf (trdl)

multiwerf was replaced by trdl (“true delivery”) — an update manager focused on the secure delivery of binaries from a Git repository to the user’s host. trdl was designed as a secure update channel that eliminates a whole class of risks associated with artifact substitution, compromise, or uncontrolled distribution.

trdl’s security is based on the combination of Git, a TUF repository, and HashiCorp Vault. Such a combination prevents supply chain attacks, verifies the integrity and authenticity of updates, and minimizes potential damage even if individual components are compromised.

trdl is not limited to werf and can be used for the secure release and distribution of any software. The CLI supports a wide range of scenarios, but the core update and usage approach known from multiwerf has been kept intact: . "$(trdl use werf 1.2 stable)" will pull the correct version of werf, runs a cryptographic check on it, and activates it in your current shell. That way, you can safely use different versions of tools on the same machine without any conflicts or global installs.

December 2021: Online tutorial for developers dedicated to Kubernetes and deployment with werf

An online tutorial was launched, aimed at developers and DevOps engineers seeking to master Kubernetes and practical application delivery with werf. It integrated theory with step-by-step practical guides, covering a spectrum from basic concepts to advanced CI/CD scenarios.

The tutorial was tailored to popular languages and frameworks, featuring examples of applications and infrastructure (IaC). It allowed users to choose a familiar technology stack to learn Kubernetes and practice with werf on real-world use cases.

February 2022: Secure image builds; no privileged daemon required

werf got experimental support for Buildah, which enabled secure image builds in a rootless mode without using the Docker daemon. The old Docker-based build workflow was also preserved for situations where it was needed.

At the same time, the werf release process was updated to include ready-to-use images for running builds with Docker as well as right inside a Kubernetes cluster. This streamlined werf’s integration into diverse CI/CD environments and broadened its potential applications.

August 2022: Telemetry was introduced

werf has got telemetry to help analyze how the tool is being used. It collects data on versions, update channels, project activity, runtime environments, CLI command usage, and build metrics.

Telemetry became the basis for understanding how werf is used in the real world, and helped us to assess the stability of releases and identify bottlenecks in the delivery process.

December 2022: werf joined CNCF!

The werf project was accepted into the CNCF (Cloud Native Computing Foundation), marking its official recognition in the Cloud Native space. This confirmed the project’s maturity and openness, signaling its readiness for wider adoption while encouraging greater community involvement in its development and integration with other CNCF tools.

May 2023: Abandoning the Helm fork and launching Nelm

The first commit to Nelm marked a new stage in the evolution of werf’s deployment mechanism. By this point, it had become clear that further development of the custom Helm fork was constrained by Helm’s own architecture. So we decided to abandon the fork and rewrite the deployment subsystem from scratch while maintaining backward compatibility.

2023–2024: Community involvement

During these years, werf was showcased at several offline events:

• at KCD Czech & Slovak 2023 in Bratislava;
• at the CNCF Project Pavilion during KubeCon + CloudNativeCon Europe 2023 in Amsterdam;
• at the CNCF Project Pavilion during KubeCon + CloudNativeCon Europe 2024 in Paris.

werf v2: April 2024 — …

March 2025: Nelm 1.0 released

Nelm 1.0 was released. It is a stable, backward-compatible alternative to Helm 3, designed to work with Helm charts. Nelm comes as a standalone CLI tool and can also be used as a library in other tools.

November 2025: Ongoing Nelm development and how it compares with Helm 4

Nelm adoption is growing, and so does the list of features it provides. After the long-awaited Helm 4 release — its first major update in years, — the community asked whether Nelm was still relevant. Thus, we published a detailed overview of how Nelm continues to evolve and remains a decent alternative with a broader feature set and a dedicated user base.

2024–2025: Community involvement

werf was featured at:

• the CNCF YouTube channel, featuring the “From improving Helm to developing Nelm: the evolution of deployments in werf” webinar;
• the “Specialized Templating” episode of the “You Choose!” YouTube show (Ch. 05, Ep. 05), alongside a diverse set of CNCF tools: Porter, Radius, Score, and PipeCD;
• FOSSASIA Summit 2025 in Bangkok.

January 2026: werf turns 10!

The werf project is celebrating its 10th anniversary. Over this period, it has evolved from a simple experiment in incremental Docker image building to a mature Open Source ecosystem for application delivery in Kubernetes. This ecosystem includes its own tools for building, deploying, and distributing software, as well as a number of standalone projects.

The werf ecosystem in numbers

werf (website + GitHub):

• 4600+ GitHub ⭐
• 1300+ releases
• 18,000+ active projects using werf
• 15,000+ commits
• 60+ contributors
• 6000+ merged pull requests

Other projects include:

• Nelm: 1000+ ⭐, 45 releases, 800+ commits
• trdl (website): 290+ ⭐, 45 releases, 900+ commits
• kubedog: 700+ ⭐, 38 releases, 650+ commits
• lockgate: 250+ ⭐, 1 release, ~100 commits

What’s next for werf

But we’re not stopping here — we have big plans ahead:

• A new build architecture featuring deep integration with Docker BuildKit.
• Enhanced supply chain security, including image signing, verification, SBOMs, and vulnerability scanning.
• A Nelm operator to integrate with tools like Argo CD and Flux, or to be used on its own (Issue #494).
• The ability to patch Helm chart resources (Issue #115).
• An alternative to Helm templates (but not a replacement!): our TypeScript experiment is almost ready for you to try (PR #502, Issue #54).
• …and a whole lot more. Stay tuned!

10 years of werf: The Cloud Native story we made together was originally published in werf blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Helm 4 evolution

Helm 4 introduced a pack of new features for the Cloud Native community. Perhaps the most significant user-facing changes were there adoption of Kubernetes Server-Side Apply (SSA) instead of the 3-Way Merge (to resolve issues with incorrect resource updates) and kstatus-based resource watching. The rest of the new features are mostly focused on reducing technical debt.

While implementing SSA is a noteworthy achievement deserving of its own release, the community seemed to expect more from Helm 4. Among the most popular feature requests were an alternative to Go templates and improved handling of Custom Resource Definition (CRD) deployments.

The pace of Helm’s development accelerated leading up to the new release. However, given the tremendous adoption of Helm in the industry and strict backward compatibility requirements, further significant architectural changes will likely be postponed until the next major Helm release.

How Nelm differs from Helm 4

Nelm is a modern alternative to Helm 4, focusing on introducing new major features while maintaining backward compatibility with Helm charts and releases.

Nelm was created in werf, following the tool’s users’ needs for improved and more powerful deployment. Later, it became a standalone project that can be used on its own (without werf) to deploy Helm charts to Kubernetes. Under the hood, Nelm uses parts of the Helm codebase, but its most troublesome parts, particularly the deployment engine, have been rewritten from scratch.

While Nelm has supported Server-Side Apply for a long time, it had more user-facing features to offer and continued to evolve. For instance, it recently introduced resource lifecycle management via the werf.io/delete-policy, werf.io/ownership, and werf.io/deploy-on annotations. Let’s examine the key differences between Nelm and Helm 4.

1. Deploying CRDs

Helm recommends placing Custom Resource Definitions (CRDs) in the chart’s crds directory. However, resources in this directory cannot be updated and are only deployed during the initial release installation. The crds directory is ignored during subsequent helm upgrade operations.

As a workaround, some users deploy CRDs as regular resources by putting them in the templates directory. However, such an approach makes it harder to maintain the deployment order. On top of that, since CRD manifests are so large, you are risking hitting the release Secret’s size limit.

To get around these issues, some well-known Open Source charts even create a separate subchart just for deploying CRDs.

With Nelm, you just move your CRDs in the crds directory. Nelm features the fully-fledged deployment mechanism for this directory, so CRDs get updated and deployed every time you run an upgrade.

2. Defining deployment order

In Helm, deployment order is typically defined using Helm hooks. This method is adequate for simple Jobs that need to run before or after a rollout.

But what if a Job requires a Deployment to run? Or what if a Job must be run halfway through the release? No standard solutions exist in Helm for these scenarios.

Before each rollout, Nelm builds a graph of operations with the Kubernetes cluster’s resources, which defines their deployment order:

It also provides a simple yet powerful way for setting this order: the werf.io/deploy-dependency annotation. This annotation creates a dependency between operations in the graph, thus defining their rollout sequence. For example, the following configuration:

kind: Deployment
metadata:
  name: backend
  annotations:
    werf.io/deploy-dependency-db: state=ready,kind=StatefulSet,name=postgres

… means that the backend Deployment will only be created or updated after the postgres StatefulSet is created/updated and ready. The graph will look like this:

The werf.io/deploy-dependency annotation works for both regular resources and hooks. We plan to add support for specifying dependencies on entire charts in the future.

As an alternative, Nelm also features the werf.io/weight annotation. It works similarly to helm.sh/hook-weight but applies to both hooks and regular resources.

There’s also the external-dependency.werf.io/resource annotation, which lets you specify a dependency for resources outside of the Helm release, such as a Secret that an operator creates.

Of course, regular Helm hooks and their weights are also supported.

3. Resource lifecycle

Helm lets you prevent a resource from being deleted using helm.sh/resource-policy: keep and control when hooks are deleted using helm.sh/hook-delete-policy. But what if you need to deploy an immutable Job mid-release? Or clean up a regular resource after its deployment? Or manage the same resource across different releases?

We recently added to Nelm a whole new set of features for managing resource lifecycle:

1. The werf.io/delete-policy annotation, which is similar to helm.sh/hook-delete-policy, allows a resource to be recreated instead of updated (before-creation), recreated only upon encountering a “field is immutable” error (before-creation-if-immutable), or deleted after a successful (succeeded) or failed (failed) deployment. This annotation, like all others in Nelm, applies to both hooks and regular resources.
2. The werf.io/ownership annotation enables hook-like behavior for regular resources. Specifically, it prevents applying or validating release annotations for the resource, and it protects the resource from deletion if it has been removed from the chart or if the release itself is being deleted.
3. Another annotation, werf.io/deploy-on, allows rendering a resource only during a release install, upgrade, rollback, or uninstall, similar to what you can already do with Helm hooks. Still, using this annotation does not convert the resource into a hook.

With these annotations, it is possible to replicate the behavior of a Helm hook without formally declaring one. For example, this hook:

metadata:
  annotations:
    helm.sh/hook: pre-install
    helm.sh/hook-delete-policy: before-hook-creation

… is similar to the following non-hook resource:

metadata:
  annotations:
    werf.io/deploy-on: install
    werf.io/delete-policy: before-creation
    werf.io/ownership: anyone

In general, we recommend Nelm users avoid using Helm hooks when authoring charts. This simplifies charts, allows for more flexible resource behavior, and accelerates rollouts by eliminating the separate hook deployment phase. However, using hooks may still be justified if maintaining compatibility with vanilla Helm is a requirement.

4. Advanced resource tracking

Helm 3 included a basic mechanism for waiting for certain regular Kubernetes resources to become ready. Helm 4 replaced it with kstatus, which improved readiness detection accuracy, but did not introduce any fundamental changes.

Nelm features its own advanced resource tracking system. Compared to Helm 4, it:

• is more accurate than kstatus at detecting when a resource is ready;
• can track not just readiness, but also whether a resource exists or not, and can detect and react to errors like failing probes;
• supports readiness tracking for popular Custom Resources with manually defined rule sets;
• determines the readiness of other Custom Resources heuristically, which works for most resources (no false positives);
• displays real-time status, errors, logs, and events for resources in the terminal during deployment.

Tracking requires no initial configuration but can be fine-tuned or disabled via command-line flags and annotations.

5. Encrypting values.yaml and other files

Helm doesn’t have built-in support for encrypted files in a chart; this functionality is provided by the helm-secrets plugin.

Nelm, on the other hand, comes with out-of-the-box support for encrypted values files and any other encrypted files in the chart’s secrets directory. Working with secrets in Nelm is easier than using the helm-secrets plugin.

Generate a secret key and create an encrypted values file:

NELM_SECRET_KEY=$(nelm chart secret key create)
nelm chart secret values-file edit secret-values.yaml

After that, you can use the encrypted values just like any other values:

# templates/secret.yaml
kind: Secret
stringData:
  mySecret: {{ .Values.mySecretValue }}

nelm release install -n foo -r bar

On top of that, Nelm can encrypt arbitrary files within the chart’s secrets directory.

6. Release planning

Nelm natively implements an analog of the helm diff plugin for Helm. The nelm release plan install command accurately displays the changes that will be applied to the Kubernetes cluster’s resources during the next rollout.

The output is precise as it is based on the plan of operations with resources, which is devised before every deployment. On top of that, unlike helm diff, this plan is based on resource updates performed via Server-Side Apply, not a 3-Way Merge.

We’re also working on a way to create and save a plan with a single command (nelm release plan install --save-plan) and then pass it to another command (nelm release install --use-plan). This means you can approve a plan and be certain that Nelm will not perform any unplanned actions. This workflow cannot be implemented with Helm and helm diff.

What’s missing in Nelm

First of all, Nelm does not support Helm 3 CLI plugins. They depend on the Helm CLI, including its command structure, options, and even on the way the logs are rendered. Achieving compatibility would require rewriting a significant portion of the Helm codebase, which is time-consuming and seems to be a pointless task. Instead, we are implementing the functionality of the most popular plugins natively within Nelm (e.g., helm diff and helm secrets).

Secondly, Nelm lacks support for post-renderers. Instead, we’ll introduce a replacement for Go templates (more on that below) and provide out-of-the-box resource patching, eliminating the need to install external plugins or configure anything. The reasoning behind this approach is detailed in issues #54 and #115.

Currently, Nelm cannot be used with Argo CD or Flux. We will address this via a Nelm operator, with its Custom Resources being deployed via Argo CD, Flux, or any other GitOps tool.

Finally, tools like Helmfile and Helmwave are not compatible with Nelm. We will likely resolve this by implementing a native Nelmfile accessible right from the Nelm CLI. The Helmwave project considers switching to Nelm itself.

What’s next for Nelm after the Helm 4.0 release

Nelm serves as the deployment engine for werf, a tool currently used in over 20,000 projects. On top of that, Nelm is actively being integrated into other products, such as the Deckhouse Kubernetes Platform. Being such an essential building block secures a solid future for Nelm, thanks to our commitment to further developing it.

The Helm 4.0 release didn’t really change anything. Thanks to its initial focus on bringing new capabilities to those in need, Nelm is still far ahead in features and improvements, and we expect this lead to grow. Over the past year, we have stabilized Nelm v1, refactored the entire codebase, and added many new features. We are also excited to have two new full-time developers joining the Nelm team very soon and to see an increasing community engagement in the project development.

Future plans

Over the next six months, we intend to release Nelm v2, migrate to the Helm 4 codebase, and release the Nelm operator for Argo CD and Flux integration.

Plans for the next year include an alternative to Go templates (our current proposal involves using TypeScript for that), chart patching, and downloading charts directly from Git.

We will continue to actively develop Nelm, just as we have been developing and supporting werf for nine years. You can learn more about Nelm and try it out in our GitHub repository.

How Nelm compares to Helm 4: Current differences and future plans was originally published in werf blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Canary Deployment Pros and Quirks

Before we dive in, let’s quickly recap why canary deployments are so cool. The core idea is to gradually shift traffic to the new app version. This way, you can test fresh features on a small subgroup of real-world users before releasing it to everyone. Canary deployments have a few other cool perks too:

1. Saving Resources. The transition to the new version is smooth and easy. You can gradually scale up instances running the new version while scaling down instances with the old (stable) version on board. No need to double your resource usage, creating a duplicate, full-sized environment.
2. Easy Rollbacks. If something goes wrong, you just switch the traffic back to the good ol’ stable version.
3. Zero-Downtime Updates. Since user traffic is shifted between versions gradually, users won’t experience any downtime or interruptions.

Keep in mind, however, that with canary deployments, you’re running two different versions of your app at the same time, which can lead to some tricky situations.

If your app uses a database, you have to be careful. During a canary release, you’ll have two different versions of your app hitting the same database. Thus, you have to make sure your DB schema works with both the old and the new version. You can address this issue by, say, doing migrations in steps — first, a migration to get the schema ready for the new version, and then another one to clean up old stuff after the deployment is complete. Another option is to avoid any backward-incompatible changes to the database.

On top of that, you have to make sure that user sessions are “sticky,” meaning a user consistently hits the same version of the application. Otherwise, their requests may bounce between the new and old version. One way to address this is by using Hash / Consistent Hashing.

In this article, I’ll show how to implement a canary deployment using passive health checks with Argo Rollouts and the Istio service mesh. The Ingress NGINX Controller works well if traffic reaches your app through an ingress, but if workloads inside the cluster talk to each other by their Service name, Ingress NGINX won’t do the trick. That’s where Istio comes into play. On top of that, Istio features much more capable observability and security tools.

Since most applications receive traffic from the outside, we’ll first look at how to shift traffic at the Ingress NGINX level. Then, we’ll dive into how the upgrade works when you’re making requests to an internal service.

Getting the Environment Ready

We’re going to set up a canary deployment using the Open Source Deckhouse Kubernetes Platform Community Edition (DKP CE). Here’s what we’ll need:

• Deckhouse Kubernetes Platform CE
• Ingress NGINX Controller
• Istio service mesh
• Argo Rollouts
• Prometheus (specifically, the Deckhouse Prom++ flavor)

To get the DKP CE cluster up and running, follow the quick start guide. Once the platform is ready, there are three more prep steps to take.

The first one is to create an Ingress NGINX Controller so users can reach our app. Make sure to enable the enableIstioSidecar parameter — it puts the NGINX controller under Istio’s control:

apiVersion: deckhouse.io/v1
kind: IngressNginxController
metadata:
  name: nginx
spec:
  ingressClass: nginx
  inlet: HostPort
  enableIstioSidecar: true
  hostPort:
    httpPort: 80
    httpsPort: 443
  nodeSelector:
    node-role.kubernetes.io/worker: ""
  tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/worker
    operator: Exists

Next, enable the Istio module. d8 is the Deckhouse Kubernetes Platform’s CLI manager.

# d8 system module enable istio

Wait for the tasks to finish. In DKP, you can monitor the progress by checking the task queue:

# d8 system queue list
Summary:
- 'main' queue: empty.
- 124 other queues (0 active, 124 empty): 0 tasks.
- no tasks to handle.

Now that the task queue is empty, we’re clear to continue.

Time to install Argo Rollouts. While you’d normally use a GitOps approach to install tools in a real-world scenarios, we’ll keep it simple for now and just apply the manifests manually:

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

Setting Up Canary Deployment

Let’s look at the key components of our setup:

1. NGINX Ingress Controller — An Nginx-based controller for receiving user traffic and routing it to our target application.
2. Istio is our service mesh. We use it to intelligently route traffic between app versions.
3. Argo Rollouts — An operator + set of CRDs (Custom Resource Definitions) for implementing more advanced deployment strategies like canary, blue/green, which you don’t get with vanilla Kubernetes.
4. Deckhouse Prom++ — a built-in DKP solution for collecting metrics.

As for our app, we’ll create a simple Go service featuring three different versions. The first one, v1, is the stable one; it’ll respond with 200 OK to all requests. v2, our “buggy” version, will throw a 500 — Internal Server Error to every other request thus simulating issue with the “new” app. Finally, v3 (the “fixed” version of v2) will return 200 OK to all requests.

Here’s what our final traffic flow and component setup will look like:

Getting the Manifests Ready

Istio handles our user traffic distribution. Let’s apply the manifests needed for our application:

---
apiVersion: v1
kind: Service
metadata:
  name: app
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: backend
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: app-vsvc
spec:
  hosts:
  - app
  http:
  - name: primary
    route:
    - destination:
        host: app
        subset: stable
      weight: 100
    - destination:
        host: app
        subset: canary
      weight: 0

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: app-destrule
spec:
  host: app
  subsets:
  - name: stable
    labels:
      app: backend
  - name: canary
    labels:
      app: backend

Note the DestinationRule manifest. Istio lets you define the so-called subsets — sets of endpoints of the same Deployment selected by labels. In this case, we define a subset named stable for the stable version and another named canary for the new version.

In the VirtualService, we set up two destinations. At the start, the stable version receives 100% of the traffic while the canary gets 0%. As the canary deployment kicks in, the Argo Rollouts operator will tweak these numbers to route more traffic to the new version.

Let’s create the Rollout:

---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: app
spec:
  strategy:
    canary:
      analysis:
        templates:
        - templateName: success-rate
        startingStep: 1
        args:
        - name: service-name
          value: app.default.svc.cluster.local
      trafficRouting:
        istio:
          virtualService:
            name: app-vsvc
            routes:
            - primary
          destinationRule:
            name: app-destrule
            canarySubsetName: canary
            stableSubsetName: stable
      steps:
      - setWeight: 20
      - pause: {duration: 1m}
      - setWeight: 40
      - pause: {duration: 1m}
      - setWeight: 60
      - pause: {duration: 1m}
      - setWeight: 80
      - pause: {duration: 1m}
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
      - name: backend
        image: rinamuka/canary:v1
        ports:
        - name: http
          containerPort: 80
          protocol: TCP
        resources:
          requests:
              memory: 5Mi
              cpu: 5m
            limits:
              memory: 10Mi
              cpu: 10m

A Rollout is an Argo Rollouts’ CRD that acts as a wrapper around a Deployment object, but with extra settings for your deployment strategy. As the deployment proceeds, the operator patches the DestinationRule objects by changing their labels, and the VirtualService to alter how traffic is split between the different subsets.

As you can see, the manifest is similar to a regular vanilla Deployment, except for the canary section. It has three parts:

1. analysis: defines which health check template to use. Note the startingStep parameter — it controls at which step the analysis of the new version (i. e., querying Prometheus) commences. Here, the check runs in the background, but you can also run it “inline” as a separate step.
2. trafficRouting: links our Rollout to a destinationRule and a virtualService.
3. steps: defines the actual canary deployment plan: what percentages of traffic to shift at a time, and how long to wait before moving more traffic to the new version.

To integrate the app into the service mesh, you just need to add the istio-injection: enabled label to the app namespace. Once that’s done, a sidecar container running the Istio agent will be injected into each app pod.

Let’s now create an Ingress object to expose our application to the outside world:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/service-upstream: "true"
    nginx.ingress.kubernetes.io/upstream-vhost: app.default.svc

spec:
  tls:
  - hosts:
    - app.31.184.210.137.sslip.io
    secretName: app-tls
  rules:
    - host: app.31.184.210.137.sslip.io
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app
                port:
                  number: 80

Note the two important Ingress annotations:

1. nginx.ingress.kubernetes.io/service-upstream: “true”: this annotation instructs the Ingress controller to send requests to the service’s ClusterIP instead of directly to the pods. In this case, the istio-proxy sidecar will only intercept traffic to the Service CIDR range, the rest of the requests will be routed directly.
2. nginx.ingress.kubernetes.io/upstream-vhost: app.default.svc. In Istio, all routing relies on the Host header. So, instead of making Istio aware of our public domain (app.31.184.210.137.sslip.io), we just use the internal one it already knows about.

The next step is to create the AnalysisTemplate manifest. This resource defines the process for checking if the new app version is running properly. In our case, it will query the cluster’s Prometheus (which comes with DKP out of the box) and check the percentage of 5xx errors. If it’s more than 5%, the deployment is canceled and all traffic shifted back to the stable app version. If it’s less, we keep sending more traffic to the new app version:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
    - name: service-name
    - name: api-token
      valueFrom:
        secretKeyRef:
          name: rollout-token
          key: token
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.95
      failureLimit: 2
      provider:
        prometheus:
          address: https://prometheus.d8-monitoring:9090
          insecure: true
          headers:
            - key: Authorization
              value: "Bearer {{ args.api-token }}"
          query: |
            sum(irate(istio_requests_total{reporter="source",destination_service=~"{{args.service-name}}",response_code!~"5.*"}[5m])) /
            sum(irate(istio_requests_total{reporter="source",destination_service=~"{{args.service-name}}"}[5m]))

DKP’s built-in Prometheus requires authorization, so you have to add a Kubernetes token to your requests. To get one, we’ll create a ServiceAccount, a Role, and a RoleBinding, and then a Secret to store the token for that ServiceAccount.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rollout
  namespace: default

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: app:prometheus-access
rules:
- apiGroups: ["monitoring.coreos.com"]
  resources: ["prometheuses/http"]
  resourceNames: ["main", "longterm"]
  verbs: ["get","create"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: app:prometheus-access
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: app:prometheus-access
subjects:
- kind: ServiceAccount
  name: rollout
  namespace: default

---
apiVersion: v1
kind: Secret
metadata:
  name: rollout-token
  annotations:
    kubernetes.io/service-account.name: rollout
type: kubernetes.io/service-account-token

Updating the App

Time to update our app. The easiest way to watch the traffic switch over is in Kiali, Istio’s web UI. Let’s open it up and commence the rollout for the new v2 version.

Here is what happens after the tag in the Rollout is changed:

1. A replica with the new image version is created (revision 2).

root@master-0:~# kubectl argo rollouts get rollout app
Name:            app
Namespace:       default
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          1/8
  SetWeight:     20
  ActualWeight:  20
Images:          rinamuka/canary:v1 (stable)
                 rinamuka/canary:v2 (canary)
Replicas:
  Desired:       1
  Current:       2
  Updated:       1
  Ready:         2
  Available:     2

NAME                             KIND        STATUS     AGE    INFO
⟳ app                            Rollout     ॥ Paused   3m28s
├──# revision:2
│  └──⧉ app-655bb4c96c           ReplicaSet  ✔ Healthy  14s    canary
│     └──□ app-655bb4c96c-p2lsv  Pod         ✔ Running  14s    ready:2/2
└──# revision:1
   └──⧉ app-84975c75b            ReplicaSet  ✔ Healthy  3m28s  stable
      └──□ app-84975c75b-tfkqv   Pod         ✔ Running  3m28s  ready:2/2

2. Next, the steps from the Rollout manifest are executed. Argo Rollouts patches the DestinationRule and VirtualService objects.

It adds a subset to the DestinationRule using the ReplicaSet’s hash label. One of the subsets points to the old (stable) version of the app, while the other points to the new (canary) one.

subsets:
    - labels:
        app: backend
        rollouts-pod-template-hash: 84975c75b
      name: stable
    - labels:
        app: backend
        rollouts-pod-template-hash: 655bb4c96c
      name: canary

In the VirtualService, the destination weights get changed:

spec:
    hosts:
    - app
    http:
    - name: primary
      route:
      - destination:
          host: app
          subset: stable
        weight: 80 # Initially, weight had a value of 100
      - destination:
          host: app
          subset: canary
        weight: 20 # Initially, weight had a value of  0

On top of that, the Argo Rollouts operator creates an AnalysisRun object, which checks metrics in Prometheus. You can see the status of these checks by describing the object.

Status:
  Completed At:  2025-08-23T11:59:39Z
  Dry Run Summary:
  Message:  Metric "success-rate" assessed Failed due to failed (3) > failureLimit (2)
  Metric Results:
    Count:   3
    Failed:  3
    Measurements:
      Finished At:  2025-08-23T11:57:39Z
      Phase:        Failed
      Started At:   2025-08-23T11:57:39Z
      Value:        [0.9033333333333333]
      Finished At:  2025-08-23T11:58:39Z
      Phase:        Failed
      Started At:   2025-08-23T11:58:39Z
      Value:        [0.7958333333333333]
      Finished At:  2025-08-23T11:59:39Z
      Phase:        Failed
      Started At:   2025-08-23T11:59:39Z
      Value:        [0.8058333333333334]

Two of the three checks failed, so the proportion of 500 errors was greater than 5%. And in Kiali, you can see exactly how the traffic got switched.

The check failed, so we’re switching back to the stable version:

Normal   RolloutResumed          2m29s                  rollouts-controller  Rollout is resumed
  Normal   Updated VirtualService  2m29s                  rollouts-controller  VirtualService `app-vsvc` set to desiredWeight '40'
  Normal   TrafficWeightUpdated    2m29s                  rollouts-controller  Traffic weight updated from 20 to 40
  Normal   RolloutStepCompleted    2m29s                  rollouts-controller  Rollout step 3/8 completed (setWeight: 40)
  Normal   AnalysisRunRunning      2m29s                  rollouts-controller  Background Analysis Run 'app-655bb4c96c-2' Status New: 'Running' Previous: ''
  Warning  AnalysisRunFailed       29s                    rollouts-controller  Background Analysis Run 'app-655bb4c96c-2' Status New: 'Failed' Previous: 'Running'

Once it rolls back, the canary pod gets deleted, and all traffic goes to the stable version:

We have now covered updating to a buggy release and the subsequent rollback. Let’s now update the application to a good release (v3). Insert the v3 image tag in the Rollout:

Name:            app
Namespace:       default
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          1/8
  SetWeight:     20
  ActualWeight:  20
Images:          rinamuka/canary:v1 (stable)
                 rinamuka/canary:v3 (canary)
Replicas:
  Desired:       1
  Current:       2
  Updated:       1
  Ready:         2
  Available:     2

NAME                             KIND         STATUS        AGE    INFO
⟳ app                            Rollout      ॥ Paused      11m
├──# revision:3
│  └──⧉ app-76dfffd666           ReplicaSet   ✔ Healthy     52s    canary
│     └──□ app-76dfffd666-6tk68  Pod          ✔ Running     51s    ready:2/2
├──# revision:2
│  ├──⧉ app-655bb4c96c           ReplicaSet   • ScaledDown  7m46s  delay:passed
│  └──α app-655bb4c96c-2         AnalysisRun  ✖ Failed      4m38s  ✖ 3
└──# revision:1
   └──⧉ app-84975c75b            ReplicaSet   ✔ Healthy     11m    stable
      └──□ app-84975c75b-tfkqv   Pod          ✔ Running     11m    ready:2/2

Let’s see how the check went:

Status:
  Dry Run Summary:
  Metric Results:
    Consecutive Success:  3
    Count:                3
    Measurements:
      Finished At:  2025-08-23T12:04:33Z
      Phase:        Successful
      Started At:   2025-08-23T12:04:33Z
      Value:        [1]
      Finished At:  2025-08-23T12:05:33Z
      Phase:        Successful
      Started At:   2025-08-23T12:05:33Z
      Value:        [1]
      Finished At:  2025-08-23T12:06:33Z
      Phase:        Successful
      Started At:   2025-08-23T12:06:33Z
      Value:        [1]

All three checks were successful, so traffic will be gradually shifted to the new version:

Name:            app
Namespace:       default
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          5/8
  SetWeight:     60
  ActualWeight:  60
Images:          rinamuka/canary:v1 (stable)
                 rinamuka/canary:v3 (canary)
Replicas:
  Desired:       1
  Current:       2
  Updated:       1
  Ready:         2
  Available:     2

NAME                             KIND         STATUS        AGE    INFO
⟳ app                            Rollout      ॥ Paused      19m
├──# revision:3
│  ├──⧉ app-76dfffd666           ReplicaSet   ✔ Healthy     9m1s   canary
│  │  └──□ app-76dfffd666-6tk68  Pod          ✔ Running     9m     ready:2/2
│  └──α app-76dfffd666-3         AnalysisRun  ◌ Running     5m53s  ✔ 6
├──# revision:2
│  ├──⧉ app-655bb4c96c           ReplicaSet   • ScaledDown  15m    delay:passed
│  └──α app-655bb4c96c-2         AnalysisRun  ✖ Failed      12m    ✖ 3
└──# revision:1
   └──⧉ app-84975c75b            ReplicaSet   ✔ Healthy     19m    stable
      └──□ app-84975c75b-tfkqv   Pod          ✔ Running     19m    ready:2/2

For the final test, let’s try the same thing, but hit the service directly instead of going through Ingress NGINX. This is exactly the kind of situation that Ingress NGINX can’t handle.

Let’s fire up a client pod that will send requests to the service name (app.default) instead of the Ingress host. Traffic switching and load balancing still work here because the Istio sidecar is handling all the routing:

Summary

As you see, it is possible to set up a canary deployment for your Kubernetes application with just a couple of tools, Argo Rollouts and Istio. This article only covered the basic features of Argo Rollouts. On top of the features I talked about, it also integrates with HPA and VPA, handles different load balancers for traffic routing, and pulls metrics from various sources, and so on. So you can really tweak it to your liking.

The cool thing about the approach I described above is that everything runs automatically: switching traffic, checking if the new release works smoothly, and rolling back in case of issues. Plus, you don’t need to mess with your app’s manifests too much, as a Rollout object is basically the same as a standard Deployment. Finally, Istio is a great base to build on later if you need more advanced deployment features.

Just a friendly reminder in the end: each deployment strategy has its own pros and cons. Choose the one that best suits your application.

Useful links

• Argo Rollout documentation
• Istio documentation
• Repository with the source code

Related books

• Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation by David Farley, Jez Humble
• Cloud Native Patterns: Designing change-tolerant software First Edition by Cornelia Davis

Canary Deployment in Kubernetes Using Argo Rollouts and Istio was originally published in Deckhouse blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

My journey to virtualization started with Obsidian, which doesn’t have a built-in sync feature. I have a small rented VPS, but I ran into an issue where running a simple Docker container would spike the server’s CPU, making even an SSH connection lag.

While looking for a solution, I realized I wanted more than just to host a single app; I wanted to easily move and scale my entire infrastructure using a declarative configuration. The idea of building my own Kubernetes cluster was very appealing.

However, containers, despite all their benefits, can introduce security risks. If one of my applications gets compromised, an attacker could potentially break out and access other containers. This may become a huge issue, given that I’m not able to monitor the cluster 24/7.

So I decided I needed full-fledged virtualization. I went with the Deckhouse Virtualization Platform Community Edition (DVP CE), which is an Open Source tool running on top of Deckhouse Kubernetes Platform. By the way, to avoid confusion, I’ll just call Deckhouse Kubernetes Platform and its components “Deckhouse” from here on out.

In this post, I’m going to show you how to build a home virtualization cluster from scratch with DVP CE. We’ll pick the hardware, get it ready, set up the network, install the platform, and launch the first few VMs along with their storage.

Getting the Home Cluster Ready

For my home cluster, I bought three mini-PCs (~$100 a piece) and a gigabit switch ($15) to connect them. The total cost came out to be $345.

PC Specifications:

• Processor: Intel N150 (4 cores, 4 threads, 3.6GHz)
• RAM: 16GB
• Storage: 500GB SSD

Prepping the Cluster

Before deploying the cluster, you have to perform a number of preliminary steps to get the DNS network and SSH access uo and running.

We’re going to install the DVP CE platform on the cluster.

Your nodes will need static IPs on the same subnet so they can talk to each other. We will stick to the “one master node and two worker nodes” scheme. For storage, we’ll use the sds-replicated-volume module from Deckhouse.

Here’s what our cluster would look like conceptually:

To make sure everything works, you’ll need to configure your DNS records. I will be running the installation from a Windows machine, so I need to add those domains to the hosts file and point them to the master node’s IP:

api.homecluster.com
argocd.homecluster.com
dashboard.homecluster.com
documentation.homecluster.com
dex.homecluster.com
grafana.homecluster.com
hubble.homecluster.com
istio.homecluster.com
istio-api-proxy.homecluster.com
kubeconfig.homecluster.com
openvpn-admin.homecluster.com
prometheus.homecluster.com
status.homecluster.com
upmeter.homecluster.com

You'll also need to generate an SSH key on your main computer using ssh-keygen and then copy it over to the authorized_keys file on the master node.

Installing the OS

I went with Ubuntu 24.04 for all the servers in the cluster. You’ll need to install it on each one. I won’t go into detail on this step, but highlight a couple of points.

During installation, I recommend setting up a static IP for the server right away to avoid having to manually configure it later. After installation, the configs will be in /etc/netplan. In my case, the server uses two network interfaces: Wi-Fi for internet access and the gigabit switch for inter-server communication.

For our distributed storage, we will need block devices. Those can be unpartitioned devices or partitions that have been created but not formatted. In my case, the partitions are laid out as follows:

• /boot — 1gb
• / — ext4, 100gb

With the leftover space, just create a partition and pick the “leave unformatted” option. That’s what we’ll use for the sds-replicated-volume storage.

Deploying a Cluster

Deploying a Master Node

Since my PC is running Windows, I’ll use it to intall the platform. First, you need to make sure the master node is accessible over SSH with a key-based authentication.

Next, per the second step of the DVP homecluster quick start guide, you need to enter the domain name template for your cluster. In my case, it’s %s.homecluster.com:

After that, click “Next: Platform installation”, and you will get a ready-made config with the domain name template filled in.

The only thing you have to do is copy and change the internalNetworkCIDRs parameter that specifies the cluster subnet. This is necessary if our servers use more than one network interface. For me, this is 10.0.4.0/24, which is my Ethernet subnet.

Keep in mind the parameters that define the subnets reserved for the cluster’s needs. There must be no overlaps with other server networks. If there are, change either the external settings or these parameters.

Save the resulting config to a file named config.yml.

Then the installation of DVP CE on the master node will commence:

Once the command completes, the master node deployment is finished.

Setting up Worker Nodes

A master node is great for system stuff, but your cluster is pretty much useless without worker nodes. You’ll run your user workloads (pods, VMs, and so on) on those. So I have to set up two more nodes.

First off, let’s create a NodeGroup for our worker nodes:

sudo -i d8 k create -f - << EOF
apiVersion: deckhouse.io/v1
kind: NodeGroup
metadata:
  name: worker
spec:
  nodeType: Static
  staticInstances:
    count: 2
    labelSelector:
      matchLabels:
        role: worker
EOF

Note the count parameter — It tells Deckhouse how many nodes to run in this group.

Next, you need to configure the worker nodes. Deckhouse does the heavy lifting here; you just need to make sure the master and worker nodes can communicate with each other over SSH. To do so, generate an SSH key on the master node with an empty passphrase:

ssh-keygen -t rsa -f /dev/shm/caps-id -C "" -N ""

Next, create the SSHCredentials resource on the cluster:

sudo -i d8 k create -f - <<EOF
apiVersion: deckhouse.io/v1alpha1
kind: SSHCredentials
metadata:
  name: caps
spec:
  user: caps
  privateSSHKey: "`cat /dev/shm/caps-id | base64 -w0`"
EOF

Now, you need to add the public key you just created to the authorized_keys file for the caps user on your worker nodes. Let’s print it out so you can copy it:

cat /dev/shm/caps-id.pub

Next, SSH into each worker node and run these commands as root (paste your public key in place of <SSH-PUBLIC-KEY>). They will create the caps user and add your key:

export KEY='<SSH-PUBLIC-KEY>' # Insert your public SSH key here
useradd -m -s /bin/bash caps
usermod -aG sudo caps
echo 'caps ALL=(ALL) NOPASSWD: ALL' | sudo EDITOR='tee -a' visudo
mkdir /home/caps/.ssh
echo $KEY >> /home/caps/.ssh/authorized_keys
chown -R caps:caps /home/caps
chmod 700 /home/caps/.ssh
chmod 600 /home/caps/.ssh/authorized_keys

To add a node to the Deckhouse cluster, you need to create a definition of the static node (StaticInstance) and make sure the master node can access to the worker nodes over SSH. Let’s do it.

Return to the master node and create a StaticInstance for each worker node. In it, specify the IP address of the taget node (use the IP from the internal node network) and the name of the entity being created (the name parameter):

export NODE=<NODE-IP-ADDRESS> # Enter the IP address of the node to connect to the cluster
sudo -i d8 k create -f - <<EOF
apiVersion: deckhouse.io/v1alpha1
kind: StaticInstance
metadata:
  name: dvp-worker
  labels:
    role: worker
spec:
  address: "$NODE"
  credentialsRef:
    kind: SSHCredentials
    name: caps
EOF

Run the following command to see if StaticInstance resources are ready:

d8 k get staticinstances.deckhouse.io -w

Once new nodes are ready, you will see the following output:

Now, check the nodes with d8 k get no:

Nice, the nodes are up!

Installing Software-Defined Storage

One of the reasons I settled on a three-node configuration is the need to store data reliably. The basic idea here is to store multiple copies of data. Right now, we need to get our replicated storage configured, and we’ll use Deckhouse’s sds-replicated-volume module for that.

First, let’s enable the required modules:

sudo -i d8 k create -f - <<EOF
---
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: sds-node-configurator
spec:
  version: 1
  enabled: true
---
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: sds-replicated-volume
spec:
  version: 1
  enabled: true
EOF

Wait for the sds-replicated-volume module to start:

sudo -i d8 k wait module sds-replicated-volume --for='jsonpath={.status.phase}=Ready' --timeout=1200s

In Deckhouse, pretty much everything — modules, system images, etc. — is managed by the deckhouse deployment running in the d8-system namespace. Whenever you enable or tweak modules, a bunch of hooks run in the background. You can see what’s happening by checking the Deckhouse queue using the d8 platform queue list command. Let’s run watch d8 platform queue list and wait for that list to clear out:

Here is what an empty queue looks like:

Let’s see what block devices we have (use the d8 k get bd command):

sds-replicated-volume features thin and thick pools for data storage. Thick pools occupy the entire allocated space right from the start, while thin pools use only the portion of disk space that is needed at the moment.

Thick pools are faster, but storage provisioning takes more time. On top of that, snapshots don’t work with thick pools. Thin pools save space and provision volumes faster, with the inherent risk the total provisioned space exceeding the actual storage capacity. So you have to monitor the actual disk usage.

Let’s create an LVMVolumeGroup for each node. You’ll need to substitute the node name and the block device name into the following command:

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: LVMVolumeGroup
metadata:
  name: "vg-on-worker-0"
spec:
  type: Local
  local:
    # Replace it with the name of the node for which the volume group is being created 
    nodeName: "worker-0"
  blockDeviceSelector:
    matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: In
        values:
          # Replace with block device names for which the volume group is being created
          - dev-ef4fb06b63d2c05fb6ee83008b55e486aa1161aa
  # The name of the LVM volume group that will include the above block devices on the selected node
  actualVGNameOnTheNode: "vg"
  thinPools:
    - name: thin-pool-0
      size: 70%
EOF

Next, create a thin pool:

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: ReplicatedStoragePool
metadata:
  name: thin-pool
spec:
  type: LVMThin
  lvmVolumeGroups:
    - name: vg-1-on-homecluster0
      thinPoolName: thin-pool-0
    - name: vg-1-on-homecluster1
      thinPoolName: thin-pool-0
    - name: vg-1-on-homecluster2
      thinPoolName: thin-pool-0
EOF

With sds-replicated-volume, the user doesn’t set up a StorageClass manually but instead configures a higher-level entity called ReplicatedStorageClass.

Let’s create the ReplicatedStorageClass (see the documentation to decide which replication option is best for you):

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: ReplicatedStorageClass
metadata:
  name: replicated-storage-class
spec:
  # The name of the storage pool we created earlier
  storagePool: thin-pool
  # What to do when the PVC is being deleted  
  # Can be "Delete" or "Retain"  
  # [More info...](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming)
  reclaimPolicy: Delete
  # Replicas can run on any available node, but only one replica per volume on any single node  
  # Our cluster doesn't have any zones (no nodes with the topology.kubernetes.io/zone label)
  topology: Ignored
  # This mode keeps the volume up for reads and writes even if a replica goes down  
  # Data is stored in three separate copies on different nodes
  replication: ConsistencyAndAvailability
EOF

Double-check that it all got created:

Now, make it the default StorageClass:

DEFAULT_STORAGE_CLASS=replicated-storage-class
sudo -i d8 k patch mc global --type='json' -p='[{"op": "replace", "path": "/spec/settings/defaultClusterStorageClass", "value": "'"$DEFAULT_STORAGE_CLASS"'"}]'

Enabling the Virtualization Module

Alright, the moment of truth. Let’s enable the virtualization module:

sudo -i d8 k create -f - <<EOF
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: virtualization
spec:
  enabled: true
  settings:
    dvcr:
      storage:
        persistentVolumeClaim:
          size: 50G
        type: PersistentVolumeClaim
    virtualMachineCIDRs:
    # Subnets from which to assign IP addresses  to the VMs
    - 10.66.10.0/24
    - 10.66.20.0/24
    - 10.66.30.0/24
  version: 1
EOF

Wait for it to report as Ready:

Next, wait for the Deckhouse queue to become empty. This might take some time:

If you went with a thick pool during the storage setup, make sure that all pods in the 8-virtualization namespace are in the Running state:

Setting Up Ingress and DNS

First, let’s check that the Kruise controller manager pod is up and running:

d8 k -n d8-ingress-nginx get po -l app=kruise

Time to get the Ingress controller installed:

sudo -i d8 k apply -f - <<EOF
# NGINX Ingress controller settings
# https://deckhouse.io/modules/ingress-nginx/cr.html#ingressnginxcontroller
apiVersion: deckhouse.io/v1
kind: IngressNginxController
metadata:
  name: nginx
spec:
  ingressClass: nginx
  # The way external traffic flows into the cluster
  inlet: HostPort
  hostPort:
    httpPort: 80
    httpsPort: 443
  # Defines which nodes the Ingress controller will run on  
  # You may want to change it
  nodeSelector:
    node-role.kubernetes.io/control-plane: ""
  tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
    operator: Exists
EOF

The controller’s pod should now be Running:

d8 k -n d8-ingress-nginx get po -l app=controller

Creating a User and Setting Up Monitoring

Let’s create a user to access the cluster and its web interface:

sudo -i d8 k apply -f - <<"EOF"
apiVersion: deckhouse.io/v1
kind: ClusterAuthorizationRule
metadata:
 name: admin
spec:
 # List of Kubernetes RBAC accounts
 subjects:
 - kind: User
   name: admin@deckhouse.io
 # Preset access level template
 accessLevel: SuperAdmin
 # Allow the user to run kubectl port-forward
 portForwarding: true
---
# Static user parameters
# Version of the Deckhouse API
apiVersion: deckhouse.io/v1
kind: User
metadata:
 name: admin
spec:
 # user e-mail
 email: admin@deckhouse.io
 # The hash for the "password" temporary password
 # Generate your own or use this one for testing:
 # echo "password" | htpasswd -BinC 10 "" | cut -d: -f2
 # You may want to change it
 password: $2y$10$5.7NBl2MtHbQNzpc4/NOGeBU8lO73qDrc1jMjo.DQz8.X.PuZB7Ji
EOF

Now, head over to grafana.homecluster.com:

As you can see, the cluster’s network is running, serving the requested pages. On the screenshot page, you can see the cluster metrics and browse specific dashboards.

Notes:

If you want to access your cluster from the internet, a simple (though not perfect) trick is to use reverse SSH port forwarding with the autossh tool to keep the tunnel up. In this case, change the cluster’s domain name. You can do that by tweaking the .spec.settings.modules.publicDomainTemplate parameter in the mc global entity (just run kubectl edit mc global).

Creating a Project and a Virtual Machine

VMs run in so-called projects, so it’s time to create one and move on to what we’ve been working towards: creating a virtual machine.

Let’s create a test project:

d8 k create -f - <<EOF
apiVersion: deckhouse.io/v1alpha2
kind: Project
metadata:
 name: test-project
spec:
 description: test-project
 projectTemplateName: default
 parameters:
  # Project quotas
  resourceQuota:
   requests:
    cpu: 16
   limits:
    cpu: 16
  networkPolicy: NotRestricted
  # Project admins
  administrators:
   - subject: User
     name: admin
EOF

And the image:

d8 k apply -f - <<EOF
apiVersion: virtualization.deckhouse.io/v1alpha2
kind: VirtualImage
metadata:
  name: ubuntu-22-04
  namespace: test-project
spec:
  # Save the image to DVCR
  storage: ContainerRegistry
  # Image source
  dataSource:
    type: HTTP
    http:
      url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
EOF

Verify that the image has been created and wait for it to become Ready:

d8 k -n test-project get vi -w

Create a virtual disk based on the image:

d8 k apply -f - <<EOF
apiVersion: virtualization.deckhouse.io/v1alpha2
kind: VirtualDisk
metadata:
  name: linux-vm-root
spec:
  # Virtual disk parameters
  persistentVolumeClaim:
    # Set a size larger than the unpacked image
    size: 10Gi
    # Insert your StorageClass name here
    storageClassName: i-sds-replicated-thin-r2
  # Data source to use for the disk
  dataSource:
    type: ObjectRef
    objectRef:
      kind: VirtualImage
      name: ubuntu-22-04
EOF

Our StorageClass’s WaitForFirstConsumer parameter basically means the disk won’t be created until something needs it (our VM). This setting makes sure the disk is created on the same node as the VM, which reduces disk latency. Create a VM:

d8 k apply -f - <<"EOF"
apiVersion: virtualization.deckhouse.io/v1alpha2
kind: VirtualMachine
metadata:
  name: linux-vm
  namespace: test-project
spec:
  # The VM class name
  virtualMachineClassName: generic
  # Scripts for bootstrapping the VM
  provisioning:
    type: UserData
    # A sample cloud-init script that creates a 'cloud' user (password 'cloud') and installs qemu-guest-agent and nginx
    userData: |
      #cloud-config
      package_update: true
      packages:
        - qemu-guest-agent
      run_cmd:
        - systemctl daemon-reload
        - systemctl enable --now qemu-guest-agent.service
      ssh_pwauth: True
      users:
      - name: cloud
        passwd: '$6$rounds=4096$saltsalt$fPmUsbjAuA7mnQNTajQM6ClhesyG0.yyQhvahas02ejfMAq1ykBo1RquzS0R6GgdIDlvS.kbUwDablGZKZcTP/'
        shell: /bin/bash
        sudo: ALL=(ALL) NOPASSWD:ALL
        lock_passwd: False
      final_message: "The system is finally up, after $UPTIME seconds"
  # VM resource settings
  cpu:
    # Number of CPU cores
    cores: 1
    # Request 10% of a single physical core
    coreFraction: 10%
  memory:
    # RAM size
    size: 1Gi
  # A list of the disks and images used by the VM
  blockDeviceRefs:
    # The order here sets the boot priority
    - kind: VirtualDisk
      name: linux-vd
EOF

All you have to do is wait for the VM to start:

 d8 k -n test-project get vm -w

You can now connect to the VM over SSH. I’ll do this using the d8 v tool provided by DVP:

The virtual machine is up and running.

What resources are available for VMs in a cluster like this?

Each of the cluster’s worker nodes provides about 10 GB of RAM and 4 CPU cores for running virtual machines. In total, the cluster features around 20 GB of RAM and 8 CPU cores for VMs.

Master node:

First worker node (a VM was launched and then stopped on it):

Second worker node:

Conclusion

We’ve set up a home virtualization cluster and have taken the first steps in using it — with a declarative approach, monitoring, and data replication. You can also create a flexible and fault-tolerant environment that can be scaled and used for various purposes, such as testing, learning, pet projects, or home services. In total, the cluster installation took about 1.5 hours (not counting the OS setup).

Currently, I’m using this setup for a personal Nextcloud. On top of that, I plan to deploy a personal GitLab server. And the best part? Experimenting with web apps and Telegram bots is no longer a headache, since I don’t have to worry about “where the heck do I host this?” anymore.

How to build a home cluster with VMs running in containers for a couple hundred dollars was originally published in Deckhouse blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

However, some werf users were only interested in the deployment part of werf, without building containers and other non-deployment functionality. For these users we even maintained the werf helm … set of commands, which was basically our Helm 3 fork exposed. As the werf deployment subsystem became more complex, we decided to separate it into the Nelm project, which we initiated at the end of December 2023. And now, with the release of Nelm CLI, Nelm has reached its 1.0 milestone!

What is Nelm?

Nelm is an Open Source CLI tool to manage Helm charts and deploy them to Kubernetes. Nelm is based on the Helm 3 codebase; it does almost everything Helm can do, improves upon it, and even adds some extra functionality. Nelm is backward-compatible with Helm charts and Helm releases, so it will be easy for Helm users to migrate to Nelm. For those familiar with werf, Nelm is werf without giterminism and without building, distributing, and cleaning up container images.

Let’s now dive into the key advantages of Nelm compared to Helm 3.

Advanced resource ordering

First of all, the Helm deployment subsystem in Nelm has been rewritten from scratch. During deployment, Nelm builds a Directed Acyclic Graph (DAG) of all operations we intend to perform in a cluster to do a release, which is then executed. The DAG allowed us to implement advanced resource ordering capabilities, such as:

• werf.io/weight annotation — it is similar to helm.sh/hook-weight, except it also works for non-hook resources, and resources with the same weight are deployed in parallel;
• werf.io/deploy-dependency-<id> annotation that makes Nelm wait for another resource to be ready or merely present in the cluster before deploying the annotated resource. This is the most powerful and efficient way to arrange the order in which resources will be deployed by Nelm;
• <id>.external-dependency.werf.io/resource annotation that makes Nelm wait for the readiness of non-release resources, such as resources created by third-party operators;
• Helm ordering capabilities (i.e., Helm hooks and Helm hook weights), which are also supported.

Server-Side Apply replaces 3-Way Merge

In Nelm, Server-Side Apply (SSA) has taken the place of the problematic Helm 3-Way Merge (3WM).

3WM is a client-side mechanism to make a patch for updating a resource in a cluster. Its issues stem from the fact that it assumes that all previous release manifests were successfully applied to the cluster, which is not always the case. For example, if some resources weren’t updated due to being invalid or if a release was aborted too early, then upon the next release, incorrect 3WM patches might be produced. This results in a seemingly “successful” Helm release with wrong changes silently being applied to the cluster, which is a very serious issue.

In 2019, Kubernetes introduced Server-Side Apply (SSA) for resource updates, which became stable in v1.22 (released in August 2021). With SSA, the patches are made in Kubernetes itself instead of client-side in Helm. SSA effectively resolves the issues associated with 3WM, and it is widely adopted by other deployment tools, like Flux. Unfortunately, it will take a lot of work to replace 3WM with SSA in Helm. However, since in Nelm, the deployment subsystem has been rewritten from scratch, we went SSA-first from the very beginning, thus solving long-standing issues of 3-Way Merge.

Resource state tracking

Nelm has powerful resource tracking built from the ground up:

• Reliable detection of resource readiness, presence, absence, or failures;
• The readiness of Custom Resources is determined heuristically by analyzing their status fields. Works for about half of Custom Resources. No false positives;
• Some dependent resources, like Pods of Deployments, are automatically found and individually tracked;
• The table with the current information (statuses, errors, and more) about the tracked resources is printed every few seconds during the deployment;
• Tracking can be configured on a per-resource basis using annotations.

Printing logs and events during deploy

During deployment, Nelm finds Pods of the release resources being deployed and periodically prints their container logs to your console. On top of that, the werf.io/show-service-messages: "true" annotation lets you print resource events as well. Log/event printing can be tuned with annotations.

Encrypted values and encrypted files

nelm chart secret commands manage encrypted values files such as secret-values.yaml or arbitrary encrypted files like secret/mysecret.txt. Those files are decrypted in-memory during templating and can be referenced in templates as .Values.my.secret.value and {{ werf_secret_file "mysecret.txt" }} respectively.

Release planning

The nelm release plan install command explains exactly what’s going to happen in a cluster during the next release. It shows 100% accurate diffs between what resources in the cluster are right now and what they will be after the next deployment, utilizing robust dry-run Server-Side Apply instead of client-side trickery.

Future plans

Here is a sneak peek at what’s on our roadmap for Nelm:

• Implement an alternative to Helm templating;
• Implement an option to pull charts directly from Git;
• Expose a public Go API for embedding Nelm into third-party software;
• Enhance the CLI experience with new commands and improve the consistency between the reimplemented commands and original Helm commands;
• Overhaul the chart dependency management;
• Migrate the built-in secret management to Mozilla SOPS.

Try it

Install Nelm and follow the Nelm quickstart. Considering migrating from Helm for your deployments? Read more about Helm compatibility.

Let us know what you think! We’d love your feedback.

Nelm 1.0 released: Helm-chart compatible alternative to Helm 3 was originally published in werf blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

A few weeks ago, I was tasked with setting up a seamless application deployment for one of our customers. I explored different approaches for this task and settled on a strategy known as blue-green deployment. Unfortunately, I couldn’t manage to find any practical examples on how to conduct this. The articles I have been fortunate enough to come across only address the general theory aspect of it. So I had to explore the blue-green deployment approach on my own. And now I’m eager to share today the learnings I took away from it.

In this piece, I will guide you through the entire process of deploying an application using the blue-green approach. That being said, I am not going to go over the various deployment strategies and the pros and cons that accompany them. See this article to learn more about blue-green and other deployment strategies.

I’ll split this article into two parts — we’ll start by looking at how blue-green deployment works and then in the next part we’ll discuss how to deploy multiple apps from one repo using werf bundles. Note that there are multiple ways to implement this strategy: e. g., you can use third-party tools such as Service Mesh, Argo CD, etc. In my case, I will use werf for deployment, describe all the resources as Helm templates, and opt for GitLab to initiate the deployment (I assume you are familiar with those technologies). The twist here is that I stick to native Kubernetes entities and mechanisms (such as labels).

For simplicity’s sake, I will refer to the green and blue application instances as “versions”. Also, this article will not cover database migration, although for some applications that might be a necessity.

Basic Blue-Green Deployment

Suppose you would like to deploy a newer version of your application. Here’s how you can do it without experiencing downtime using a “blue-green” approach:

1. First, deploy the application (the deploy_app pipeline) with the corresponding Deployment and Service.
2. Next, prepare everything you will need for version switching. That is, deploy an Ingress with the appropriate Service name (the deploy_ingress pipeline).
3. Here’s what the interface for running these two pipelines would look like in GitLab:

Define a deploy_version variable to substitute into Helm templates, which will be either blue or green (its value will be derived from the GitLab CI). Add the corresponding labels to the Deployment and Service:

{{ $deploy_version := "" }}
{{ if .Values.werf.deploy_version }}
{{ $deploy_version = print "-" .Values.werf.deploy_version }}
{{ end }}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Chart.Name }}{{ $deploy_version }}
  labels:
    app: {{ .Chart.Name }}{{ $deploy_version }}
...
---
apiVersion: v1
kind: Service
metadata:
  name: {{ .Chart.Name }}{{ $deploy_version }}
spec:
  selector:
    app: {{ .Chart.Name }}{{ $deploy_version }}
...

Now, create an Ingress for the traffic to be able to reach the pod. Service with the proper name (either blue or green) will allow you to use the specific application version. The Ingress template in this case would look as follows:

{{ $deploy_version := "" }}
{{ if .Values.werf.deploy_version }}
{{ $deploy_version = print "-" .Values.werf.deploy_version  }}
{{ end }}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
  labels:
    deploy-version: {{ .Values.werf.deploy_version | quote }}
spec:
  ingressClassName: nginx
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: {{ .Chart.Name }}{{ $deploy_version }}
            port:
              name: http
 tls:
  - hosts:
    - example.com
    secretName: {{ .Chart.Name }}-tls

Instead of routing traffic using Ingress, you can do it using Service, forwarding traffic to blue or green Deployment based on labels. However, I don’t recommend doing it this way — you will not be able to reference the Deployment by the Service name in the cluster. This may be a problem if you need to check whether the update went smoothly, since you will not be able to access the new version before all traffic is routed to it.

Another option is to create a second Ingress pointing to the inactive version with a different domain for testing. In that case, you will need to secure it with authorization to restrict access.

Now let’s take a look at the pipeline. When deploying an application, you must set the deploy_version variable to the app version to be deployed. Here’s how you can do that with werf:

werf converge --set "werf.deploy_version=${DEPLOY_VERSION}"

Also, when you’re deploying, you need to check that the version you’re deploying isn’t getting any traffic yet — this way, users won’t be affected during the rollout. To do so, retrieve information as to which Service the Ingress routes traffic to in the cluster, and search for blue or green.

See the complete gitlab-ci.yml file below:

stages:
 - deploy_app
 - deploy_ingress

.check_upstreams: &check_upstreams
 - APP_CURRENT_ACTIVE=$(werf kubectl -n ${WERF_NAMESPACE} get ingress example --output=custom-columns='SVCs:..service.name' --no-headers --ignore-not-found | awk -F '-' {'print $NF'})

.deploy_app:
 stage: deploy_app
 script:
   - *check_upstreams
   - if [[ ${KUBE_CURRENT_ACTIVE} == ${UPSTREAM} ]];
     then
       tput setaf 9 && echo "You are trying to deploy to the active version, the deployment process is halted!" && exit 1;
     else
       werf converge \
         --release example-${UPSTREAM} \
         --set "werf.deploy_version=${UPSTREAM}";
     fi;
 allow_failure: false

.deploy_ingress:
 stage: converge_ingresses
 script:
   - *check_upstreams
   - if [ ${APP_CURRENT_ACTIVE} == ${DEPLOY_VERSION} ];
     then
       tput setaf 9 && echo "You are trying to switch to the active version, the deployment process is halted!" && exit 1;
     else
       werf converge
       --set "werf.deploy_version=${DEPLOY_VERSION}"
     fi;

Deploy to blue:
 extends: .deploy_app
 environment:
   name: production
 variables:
   UPSTREAM: "blue"

Deploy to green:
 extends: .deploy_app
 environment:
   name: production
 variables:
   UPSTREAM: "green"

Switch to blue:
 extends: .deploy_ingress
 environment:
   name: production
 variables:
   DEPLOY_VERSION: "blue"

Switch to green:
 extends: .deploy_ingress
 environment:
   name: production
 variables:
   DEPLOY_VERSION: "green"

So here’s the list of the steps we’ve taken so far:

1. We’ve updated our Helm templates (Deployment, Service, and Ingress) by adding version “color” to each of them.
2. We created a CI that:

• Deploys the application to blue and green.
• Deploys the Ingress to route traffic to the desired version.
• Checks that the version being deployed is not active.

Now, let’s get to the bundle part.

Deploying multiple applications with werf bundles

Why would you need a bundle? Let’s say you need to deploy multiple apps at once. It makes the most sense to keep them all in one repo. The bundle mechanism allows you to publish a chart of an application and deploy it later on — no access to a specific Git repository is required. All you need is access to the container registry where that bundle is stored. That renders delivering app charts a breeze.

The werf bundle is designed to do just that. I won’t get into all the specifics here — just take a look at the documentation for more details and examples of how it works.

Bundles are created in the main application repository. Here, I’ll focus just on the deployment process. In the CI file, specify the application names and the corresponding variables for each application: the repository, the bundle tag, and the Ingress name:

variables:
 FIRST_REPO_BUNDLE: registry.gitlab.awesome.com/frontend/first
 FIRST_TAG: "0.1"
 FIRST_INGRESS: first
...

# apps_for_matrix & apps_for_bash must be the same!

.apps_for_matrix: &apps_for_matrix
 ["FIRST", "SECOND", "THIRD", "FOURTH", "FIFTH"]

.apps_for_bash: &apps_for_bash
 APPLICATIONS=("FIRST", "SECOND", "THIRD", "FOURTH", "FIFTH")

The new pipeline will have three separate stages. When deploying multiple applications from a single repository, you have to make sure that all the applications are in the same state. To do so, let’s create a dedicated job called check_upstream that will check the version states of the applications.

At this stage, the system must ensure that the following basic conditions are met:

• All deployed applications must share the same active version (either blue or green).
• If the application has not yet been deployed to the cluster, it shouldn’t have any active versions.

stages:
 - check_upstreams
 - deploy_apps
 - deploy_ingresses

.base_werf: &base_werf
 - set -x
 - type trdl && source $(trdl use werf 2)
 - werf version
 - type werf && source $(werf ci-env gitlab --verbose --as-file)

.check_upstreams: &check_upstreams
 - *base_werf
 - *apps_for_bash
 - |
   GREEN=false
   BLUE=false
   EMPTY=0

   for APP in ${APPLICATIONS[@]}
   do
     REPOSITORY_INGRESS=${APP}_INGRESS
     APP_CURRENT_ACTIVE=$(werf kubectl -n ${WERF_NAMESPACE} get ingress ${!REPOSITORY_INGRESS} --output=custom-columns='SVCs:..service.name' --no-headers --ignore-not-found | awk -F '-' {'print $NF'})

     EMPTY=$((EMPTY+1))
     if [[ ${APP_CURRENT_ACTIVE} == "green" ]];
       then GREEN=true;
     elif [[ ${APP_CURRENT_ACTIVE} == "blue" ]];
       then BLUE=true;
     elif [[ -z ${APP_CURRENT_ACTIVE} ]];
       then EMPTY=$((EMPTY-1));
     else
       tput setaf 9 && echo "Something is wrong! Version status is invalid" && exit 1;
     fi;
   done

   if [[ ${GREEN} != ${BLUE} ]];
     then
     if [[ ${GREEN} ]]
       COLOR="green"
       then tput setaf 14 && echo "The app version statuses are the same — green — you can proceed with the deployment";
     elif [[ ${BLUE} ]]
       COLOR="blue"
       then tput setaf 14 && echo "The app version statuses are the same — blue — you can proceed with the deployment";
     fi;
   elif [[ ${EMPTY} = 0 ]]
     then tput setaf 14 && echo "No Ingress for these applications is detected in the cluster, you can proceed with the deployment";
   else
     tput setaf 9 && echo "The app version statuses are different, the deployment process is halted!!!" && exit 1;
   fi;

Check_upstreams:
 stage: check_upstreams
 script:
   - *check_upstreams
 environment:
   name: production
 when: always
 allow_failure: false

We will use the bundle mechanism to deploy the application. When running the deployment command, you must pass all the required parameters. Note that the release name for each application must be unique (use the — release parameter to specify it). This is essential, as sharing the same release name will result in the new deployment overwriting the previous one. During the deployment stage, the system will automatically create the necessary number of deployment jobs via the parallel:matrix function.

Your deployment configuration might look something like this:

.deploy_apps: &deploy_apps
 stage: deploy_apps
 before_script:
   - *base_werf
   - REPOSITORY_BUNBLE=${REPOSITORY_NAME}_REPO_BUNDLE
   - REPOSITORY_TAG=${REPOSITORY_NAME}_TAG
   - REPOSITORY_INGRESS=${REPOSITORY_NAME}_INGRESS
   - APP_CURRENT_ACTIVE=$(werf kubectl -n ${WERF_NAMESPACE} get ingress ${!REPOSITORY_INGRESS} --output=custom-columns='SVCs:..service.name' --no-headers --ignore-not-found | awk -F '-' {'print $NF'})
   - |
     if [[ ${APP_CURRENT_ACTIVE} = ${DEPLOY_VERSION} ]];
       then tput setaf 9 && echo "You are trying to deploy to the active version, the deployment process is halted!!!" && exit 1;
     fi;
 script:
   - werf cr login -u nobody -p ${BUNDLE_PULLER_PASSWORD} ${!REPOSITORY_BUNBLE}
   - werf bundle apply
     --release $(echo ${!REPOSITORY_BUNBLE} | cut -d / -f4)-${DEPLOY_VERSION}-${CI_ENVIRONMENT_SLUG}
     --repo ${!REPOSITORY_BUNBLE}
     --tag ${!REPOSITORY_TAG}
     --set "werf.deploy_version=${DEPLOY_VERSION}"
 when: manual

Deploy to Green:
 extends: .deploy_apps
 stage: deploy_apps
 environment:
   name: production
 parallel:
   matrix:
     - REPOSITORY_NAME: *apps_for_matrix
 variables:
   DEPLOY_VERSION: "green"

Congrats: you’ve created a deployment pipeline that allows you to deploy different applications from a single repository using pre-published bundles.

Conclusion

The blue-green approach helps you deploy application updates reliably and quickly. You can test your new version before sending users to it, thereby rendering the entire process smoother. As for bundles, they really shine in cases when you need to deploy several apps at once. This renders application management and updates more visible and centralized, an essential element for large projects.

In this article, we’ve gone through blue-green deployments using GitLab-CI and demonstrated how to deploy multiple apps from a single repository. Hope this makes it easier for you to work with GitLab deployments and write your own CI scripts!

Blue-Green Deployments: a Guide to Deploying One or More Applications was originally published in werf blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

What is Kwasm?

Kwasm is a Kubernetes operator that supports running WebAssembly applications on Kubernetes cluster nodes. Kwasm-node-installer — a component of the operator — installs the containerd binaries and makes the necessary configuration changes. It is run on nodes labeled with kwasm.sh/kwasm-node=true.

The module then downloads the required containerd-shim binaries to the cluster nodes and makes changes to the containerd configuration (refer to this article to learn more about WebAssembly). After that, you can run Wasm applications on those nodes.

Kwasm supports a plethora of cloud platforms as well as local installations ranging from kind to AWS, GCP, and Azure clouds:

Installing Kwasm

Let’s install the operator in a Kubernetes cluster. First, we have to create a cluster. Let’s use the kind tool. Our cluster will consist of three nodes:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker

Create a cluster:

kind create cluster --config=./kind.yaml

Ascertain that the cluster has been created:

kubectl get nodes
NAME                 STATUS   ROLES           AGE   VERSION
kind-control-plane   Ready    control-plane   59s   v1.24.0
kind-worker          Ready    <none>          40s   v1.24.0
kind-worker2         Ready    <none>          40s   v1.24.0

Once the cluster is ready, proceed to install the Kwasm operator:

helm repo add kwasm http://kwasm.sh/kwasm-operator/
helm install -n kwasm --create-namespace kwasm-operator kwasm/kwasm-operator
kubectl annotate node --all kwasm.sh/kwasm-node=true

Make sure that the operator has been installed:

kubectl get pods -n kwasm -o wide
NAME                                       READY   STATUS      RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
kind-control-plane-provision-kwasm-cfbvg   0/1     Completed   0          55s   10.244.0.5   kind-control-plane   <none>           <none>
kind-worker-provision-kwasm-n5c95          0/1     Completed   0          55s   10.244.1.3   kind-worker          <none>           <none>
kind-worker2-provision-kwasm-zqj5z         0/1     Completed   0          55s   10.244.2.2   kind-worker2         <none>           <none>
kwasm-operator-7f7d456678-hxsgx            1/1     Running     0          72s   10.244.1.2   kind-worker          <none>           <none>

Running a sample Wasm application

Once the operator has made changes to the containerd configuration on the nodes, create a separate runtime class to run the WebAssembly containers.

kubectl apply -f -<<EOF
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: wasmedge
handler: wasmedge
EOF

Now you are all set to run the Wasm application. Let’s run the wasi-demo pod with the wasmedge/example-wasi:latest image as an example.

kubectl apply -f -<<EOF
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: wasi-demo
  name: wasi-demo
spec:
  containers:
  - args:
    - /wasi_example_main.wasm
    - "50000000"
    image: wasmedge/example-wasi:latest
    name: wasi-demo
  restartPolicy: Never
  runtimeClassName: wasmedge
EOF

Once the pod has been Completed, take a look at its logs:

kubectl logs wasi-demo
Random number: 63685983
Random bytes: [247, 43, 129, 227, 3, 56, 148, 40, 154, 241, 96, 85, 109, 140, 104, 71, 188, 245, 165, 107, 146, 202, 215, 21, 50, 33, 54, 193, 175, 35, 142, 108, 150, 30, 229, 50, 105, 139, 110, 170, 187, 234, 41, 249, 213, 65, 146, 27, 88, 115, 30, 147, 95, 155, 203, 183, 143, 0, 139, 108, 12, 141, 255, 191, 11, 254, 40, 189, 186, 19, 196, 136, 51, 114, 103, 119, 130, 105, 99, 177, 192, 158, 122, 120, 160, 9, 241, 73, 209, 235, 22, 158, 35, 6, 223, 217, 3, 215, 114, 4, 52, 11, 49, 191, 33, 253, 80, 254, 255, 176, 137, 38, 53, 190, 18, 194, 53, 143, 251, 1, 147, 254, 206, 130, 195, 77, 93, 151]
Printed from wasi: This is from a main function
This is from a main function
The env vars are as follows.
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_SERVICE_HOST: 10.96.0.1
HOSTNAME: wasi-demo
KUBERNETES_PORT: tcp://10.96.0.1:443
KUBERNETES_SERVICE_PORT: 443
KUBERNETES_SERVICE_PORT_HTTPS: 443
KUBERNETES_PORT_443_TCP_ADDR: 10.96.0.1
KUBERNETES_PORT_443_TCP_PROTO: tcp
KUBERNETES_PORT_443_TCP_PORT: 443
KUBERNETES_PORT_443_TCP: tcp://10.96.0.1:443
The args are as follows.
/wasi_example_main.wasm
50000000
File content is This is in a file

Cool, the WebAssembly application works!

Conclusion

As we’ve seen, Kwasm significantly streamlines the deployment of Wasm applications into a Kubernetes cluster. However, it is worth noting that while the operator claims support for a large number of Kubernetes distributions, the developers themselves warn that the tool should only be used for evaluation purposes.

As of now, SpinKube is well worth a good look, which is powered by Kwasm technologies. It’s exhibiting a more rapid pace of development. We’ll definitely cover it in one of our upcoming articles on WebAssembly technologies.

Kwasm review: run WebAssembly apps in Kubernetes clusters was originally published in Deckhouse blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

But running Wasm applications in plain vanilla Kubernetes is not as easy as it sounds, since setting up runtime environments on worker nodes is tricky. The built-in K8s tools are simply not designed to make the node customization process convenient for a casual user. Of course, you can configure a single node on your own. The problem with this approach is that if you need to try out different runtimes or run a large number of applications, you want cluster scaling to be as easy as possible. In this case, managing nodes declaratively also makes perfect sense. So I thought I’d use the Deckhouse Kubernetes Platform (DKP) to try to run a Wasm application. This platform greatly streamlines the deployment and management of Kubernetes clusters.

My name is Yegor Lazarev, I’m a DevOps engineer at Flant. In this article, I will show you how to run Wasm applications in Kubernetes clusters managed by DKP. We will set up an environment, install the necessary components, and run a simple WebAssembly module.

Configuring a NodeGroup

I guess it makes sense to separate the regular workloads and Wasm workloads so that there is a dedicated worker for experiments. To do so, let’s create a NodeGroup that the platform will use to manage individual nodes. When configuring, you should add labels to the NodeGroup nodes. This way, you can use NodeSelector to assign the workloads to the appropriate nodes:

kubectl create -f -<<EOF
apiVersion: deckhouse.io/v1
kind: NodeGroup
metadata:
  name: wasm
spec:
  cloudInstances:
    classReference:
      kind: YandexInstanceClass
      name: worker
    maxPerZone: 1
    minPerZone: 1
    zones:
    - ru-central1-a
  disruptions:
    approvalMode: Automatic
  kubelet:
    containerLogMaxFiles: 4
    containerLogMaxSize: 50Mi
    resourceReservation:
      mode: Auto
  nodeTemplate:
    labels:
      node.deckhouse.io/group: wasm
  nodeType: CloudEphemeral
EOF

Once the NodeGroup is created, DKP will provision one virtual machine in the cloud of the AWSInstanceClass=worker class in the eu-west-1a zone and add the node.deckhouse.io/group=wasm label to it.

Installing the WasmEdge runtime

Kubernetes requires a specialized runtime, WebAssembly System Interface (WASI), to run Wasm applications. In this article, we will use WasmEdge. On top of that, we’ll need to update the containerd configuration to reflect the new runtimes. The NodeGroupConfiguration resource allows you to run bash scripts on nodes, so let’s use it to install WasmEdge and do some additional configuration.

Check if the WASI bin file is available and download it if there isn’t one. Next, use bashbooster to merge the main containerd config with the config from /etc/containerd/conf.d/*.toml. Modifying /etc/containerd/config.toml will result in containerd being restarted as well:

kubectl create -f -<<EOF
apiVersion: deckhouse.io/v1alpha1
kind: NodeGroupConfiguration
metadata:
  name: wasm-additional-shim.sh
spec:
  bundles:
    - '*'
  content: |
    [ -f "/bin/containerd-shim-wasmedge-v1" ] || curl -L https://github.com/containerd/runwasi/releases/download/containerd-shim-wasmedge%2Fv0.3.0/containerd-shim-wasmedge-$(uname -m | sed s/arm64/aarch64/g | sed s/amd64/x86_64/g).tar.gz | tar -xzf - -C /bin

    mkdir -p /etc/containerd/conf.d
    bb-sync-file /etc/containerd/conf.d/additional_shim.toml - containerd-config-changed << "EOF"
    [plugins]
      [plugins."io.containerd.grpc.v1.cri"]
        [plugins."io.containerd.grpc.v1.cri".containerd]
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
            [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmedge]
              runtime_type = "io.containerd.wasmedge.v1"
              [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmedge.options]
                BinaryName = "/bin/containerd-shim-wasmedge-v1"
    EOF
  nodeGroups:
    - "wasm"
  weight: 30
EOF

Defining new RuntimeClasses

Now that WasmEdge is installed, you have to define a new RuntimeClass. This will allow you to specify how to run a particular workload: use the default runtime or another one by explicitly specifying spec.runtimeClassName in the spec.pods:

kubectl apply -f -<<EOF
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: wasmedge
handler: wasmedge
EOF

Running a test Wasm application

First, make sure that platform has finished configuring the node and updated the containerd configuration:

root@test-wasm-75934c42-5956c-l5m7f:~# grep wasm /etc/containerd/config.toml
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmedge]
          runtime_type = "io.containerd.wasmedge.v1"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasmedge.options]
            BinaryName = "/bin/containerd-shim-wasmedge-v1"

Now you can run the test Wasm application. To do so, create a Job with a basic WebAssembly module. In the Job, specify the NodeSelector and the newly created wasmedge RuntimeClass:

kubectl apply -f -<<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: wasm-test
spec:
  template:
    spec:
      containers:
      - image: wasmedge/example-wasi:latest
        name: wasm-test
        resources: {}
      restartPolicy: Never
      runtimeClassName: wasmedge
      nodeSelector:
        node.deckhouse.io/group: wasm
  backoffLimit: 1
EOF

Check the pod’s status and logs to make sure everything is running smoothly:

root@test-master-0:~# kubectl get pods
NAME              READY   STATUS      RESTARTS   AGE
wasm-test-2g5jl   0/1     Completed   0          18s

root@test-master-0:~# kubectl logs wasm-test-2g5jl
Random number: -700610054
Random bytes: [163, 184, 229, 154, 4, 145, 145, 96, 181, 77, 64, 159, 123, 45, 5, 134, 93, 193, 207, 74, 129, 113, 204, 174, 188, 152, 172, 151, 125, 78, 199, 177, 127, 112, 116, 255, 188, 180, 47, 110, 22, 241, 63, 87, 78, 168, 36, 202, 168, 90, 248, 79, 38, 59, 204, 128, 141, 92, 209, 205, 129, 51, 71, 214, 91, 237, 115, 145, 77, 136, 166, 115, 221, 66, 123, 186, 19, 39, 122, 204, 103, 221, 89, 97, 148, 57, 250, 255, 165, 53, 14, 241, 97, 138, 147, 201, 204, 29, 76, 219, 128, 48, 143, 165, 138, 231, 62, 235, 190, 94, 142, 63, 197, 37, 57, 241, 33, 99, 240, 215, 216, 33, 68, 141, 82, 21, 152, 93]
Printed from wasi: This is from a main function
This is from a main function
The env vars are as follows.
KUBERNETES_SERVICE_PORT_HTTPS: 443
KUBERNETES_PORT_443_TCP: tcp://10.222.0.1:443
KUBERNETES_PORT_443_TCP_ADDR: 10.222.0.1
KUBERNETES_PORT_443_TCP_PROTO: tcp
KUBERNETES_SERVICE_PORT: 443
HOSTNAME: wasm-test-2g5jl
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_SERVICE_HOST: 10.222.0.1
KUBERNETES_PORT: tcp://10.222.0.1:443
KUBERNETES_PORT_443_TCP_PORT: 443
The args are as follows.
/wasi_example_main.wasm
File content is This is in a file

If this is the case, the pod will have a status of Completed, which means that the Job has been executed and the pod has finished its operation without errors.

In the logs, you should see a random number and lots of random bytes generated by the application just like it was intended. This also means that the application had access to the environment and file system.

Running a test Wasm application with an init container

Now let’s make things a little more challenging. Quite often there is a need to run init or sidecar containers in pods from regular container images. To do so, you will have to define a different runtime for each container. However, the runtimeClassName is defined at the pod level, not the container level.

Containerd supports container runtime switching, so you will need a tool that can determine which runtime to use for a particular container. The regular runc used by default in cluster doesn’t support this. Fortunately, the beta version of crun supports such a functionality.

First, you will have to build crun yourself, as it does not support WasmEdge if you install it from the official repositories using a package manager. NodeGroupConfiguration can help you with this:

kubectl apply -f -<<EOF
apiVersion: deckhouse.io/v1alpha1
kind: NodeGroupConfiguration
metadata:
  name: crun-install.sh
spec:
  bundles:
  - '*'
  content: |
    if ! [ -x /usr/local/bin/crun ]; then
      apt-get update && apt-get install -y make git gcc build-essential pkgconf libtool libsystemd-dev libprotobuf-c-dev libcap-dev libseccomp-dev libyajl-dev go-md2man autoconf python3 automake
      cd /root
      [ -f "/root/.wasmedge/bin/wasmedge" ] || curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash
      git clone https://github.com/containers/crun && cd crun
      ./autogen.sh
      source /root/.wasmedge/env && ./configure --with-wasmedge
      make
      make install
      cd .. && rm -rf crun
    fi
      echo "crun has been installed"
    mkdir -p /etc/containerd/conf.d
    bb-sync-file /etc/containerd/conf.d/add_crun.toml - containerd-config-changed << "EOF"
    [plugins]
      [plugins."io.containerd.grpc.v1.cri"]
        [plugins."io.containerd.grpc.v1.cri".containerd]
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
            [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.crun]
              runtime_type = "io.containerd.runc.v2"
              pod_annotations = ["*.wasm.*", "wasm.*", "module.wasm.image/*", "*.module.wasm.image", "module.wasm.image/variant.*"]
              [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.crun.options]
                BinaryName = "/usr/local/bin/crun"
    EOF
  nodeGroups:
  - wasm
  weight: 30
EOF

The code snippet above installs WasmEdge (which is different from the WasmEdge runtime we installed earlier in this article), as well as the required dependencies, and builds crun. On top of that, you have to add a new runtime container to the /etc/containerd/config.toml configuration, just like we did earlier for the Wasm one.

Note the pod_annotations: this is a list of annotations to be passed to both the runtime environment and the container’s OCI annotations. I’ll explain why this is necessary in a minute.

Next, create a new RuntimeClass:

kubectl apply -f -<<EOF
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: crun
handler: crun
EOF

Now, try to run your workload:

kubectl apply -f -<<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: wasm-test
spec:
  template:
    metadata:
      annotations:
        module.wasm.image/variant: compat-smart
    spec:
      initContainers:
      - name: hello
        image: busybox:latest
        command: ['sh', '-c', 'echo "Hello, Medium!"']
      containers:
      - image: wasmedge/example-wasi:latest
        name: wasm-test
        resources: {}
      restartPolicy: Never
      runtimeClassName: crun
      nodeSelector:
        node.deckhouse.io/group: wasm
  backoffLimit: 1
EOF

The runtimeClassName: crun parameter indicates that crun, rather than the default runc, is now used for starting containers. On the other hand, the module.wasm.image/variant: compat-smart annotation tells crun which mode to operate in.

For this to work, you’ll have to add the following OCI annotation to the WASM image when building:

...
"annotations": {
 "run.oci.handler": "wasm"
},
...

Crun uses pod_annotations in the containerd configuration and the compat-smart annotation on the K8s object to figure out which workload to run itself and which one to delegate to the Wasm runtime.

Examine the pod’s state and its logs. You should see the same thing in the logs as before:

root@test-master-0:~# kubectl get pods
NAME              READY   STATUS      RESTARTS   AGE
wasm-test-pn4gv   0/1     Completed   0          32s

root@test-master-0:~# kubectl logs wasm-test-pn4gv
Defaulted container "wasm-test" out of: wasm-test, hello (init)
Random number: -158793507
Random bytes: [210, 246, 181, 132, 184, 214, 110, 71, 198, 68, 154, 182, 253, 103, 116, 207, 5, 205, 185, 81, 19, 28, 61, 61, 85, 26, 222, 111, 239, 110, 21, 68, 119, 245, 153, 190, 105, 175, 191, 163, 48, 198, 41, 207, 155, 30, 122, 166, 23, 56, 59, 168, 91, 57, 103, 213, 145, 10, 130, 224, 28, 5, 73, 176, 206, 111, 37, 241, 38, 57, 98, 158, 150, 115, 249, 233, 194, 156, 13, 109, 85, 130, 232, 91, 253, 16, 8, 233, 92, 162, 237, 197, 151, 112, 52, 140, 83, 179, 31, 48, 233, 56, 54, 75, 43, 239, 233, 169, 169, 81, 36, 52, 59, 66, 102, 40, 52, 202, 34, 56, 167, 229, 197, 25, 72, 136, 147, 254]
Printed from wasi: This is from a main function
This is from a main function
The env vars are as follows.
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME: wasm-test-pn4gv
KUBERNETES_PORT: tcp://10.222.0.1:443
KUBERNETES_PORT_443_TCP: tcp://10.222.0.1:443
KUBERNETES_PORT_443_TCP_PROTO: tcp
KUBERNETES_PORT_443_TCP_PORT: 443
KUBERNETES_PORT_443_TCP_ADDR: 10.222.0.1
KUBERNETES_SERVICE_HOST: 10.222.0.1
KUBERNETES_SERVICE_PORT: 443
KUBERNETES_SERVICE_PORT_HTTPS: 443
HOME: /
The args are as follows.
/wasi_example_main.wasm
File content is This is in a file

Check the init container’s logs:

root@test-master-0:~# kubectl logs wasm-test-pn4gv -c hello
Hello, Medium!

Conclusion

Running WebAssembly applications in Kubernetes may not sound like an easy task, but with Deckhouse Kubernetes Platform it becomes a fairly straightforward process. This article delved into setting up the environment, installing the necessary components, and running a test Wasm application. I hope you will find all this information useful.

The DKP provides many features for managing a Kubernetes cluster. We will share new practices and tips in upcoming articles. Stay tuned!

Feel free to ask any questions you have and contribute suggestions in the comments below. You can also submit your question to the Deckhouse Telegram chat or create an Issue in the Deckhouse repository on GitHub. We will be happy to help you. Please star the project if you like it.

Running WebAssembly applications in a Kubernetes cluster managed by Deckhouse was originally published in Deckhouse blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

In werf 2.0 and Nelm, we took it a step further and replaced 3-Way Merge with Server-Side Apply (SSA), a more modern and robust mechanism for updating Kubernetes resources. It solves all 3-Way Merge issues and ensures that resources in the cluster are updated correctly during deployments.

This article discusses the 3WM-related challenges users of Helm 3 have faced. Subsequently, it goes on to show how SSA can help in overcoming them.

Note. Refer to one of our earlier articles to learn more about 3-Way Merge and 2-Way Merge.

Invalid resource updates in Helm 3

If you rely on 3WM (e.g., Helm 3 and/or werf 1.2) to update resources in a cluster, the deployed resources often do not match their description in the Helm chart. Let’s try to simulate such a scenario.

Suppose there is a Helm chart with the Deployment (chart/templates/deployment.yaml):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: main
        image: nginx

… and the Job hook (chart/templates/job.yaml):

apiVersion: batch/v1
kind: Job
metadata:
  name: myjob
  annotations:
    helm.sh/hook: "post-install,post-upgrade"
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: main
        image: alpine
        command: ["echo", "succeeded"]

Let’s release it using the latest Helm 3 version:

$ helm upgrade --install myapp chart
Release "myapp" has been upgraded. Happy Helming!

Now, in the chart’s Deployment, let’s replace the main container with two containers called backend and frontend:

...
      containers:
      - name: backend
        image: nginx
      - name: frontend
        image: nginx

… while also “accidentally” breaking the Job hook:

...
      containers:
      - name: main
        image: alpine
        command: ["fail"]

The second release will fail (just as you’d expect):

$ helm upgrade --install myapp chart
Error: UPGRADE FAILED: post-upgrade hooks failed: 1 error occurred:
       * job myjob failed: BackoffLimitExceeded

Given that we have an error in Job in the chart, let’s fix it:

...
      containers:
      - name: main
        image: alpine
        command: ["echo", "succeeded"]

…and at the same time rename the new containers in the Deployment from backend and frontend to app and proxy (seeing as their original names aren’t really fitting):

...
      containers:
      - name: app
        image: nginx
      - name: proxy
        image: nginx

Now, let’s run a third release (a successful one this time):

$ helm upgrade --install myapp chart
Release "myapp" has been upgraded. Happy Helming!

And check if the Deployment in the chart and in the Helm release match the Deployment in the cluster:

$ cat chart/templates/deployment.yaml
...
      containers:  # correct      
      - name: app
      - name: proxy

$ helm get manifest myapp
...
      containers:  # correct
      - name: app
      - name: proxy

$ kubectl get deploy myapp -oyaml
...
      containers:  # INCORRECT
      - name: app
      - name: proxy
      - name: backend
      - name: frontend

Well, it looks like the Deployment in the chart/release has two containers, but the Deployment in the cluster has four of them for whatever reason: two valid app and proxy containers and two legacy frontend and backend ones.

What is more, repeating the release won’t rid you of the unnecessary frontend and backend containers:

$ helm upgrade --install myapp chart
$ kubectl get deploy myapp -oyaml
...
      containers:
      - name: app
      - name: proxy
      - name: backend
      - name: frontend

Rolling back to the very first revision won’t help either:

$ helm rollback myapp 1
$ kubectl get deploy myapp -oyaml
...
      containers:
      - name: main
      - name: backend
      - name: frontend

At this point, the easiest way to get rid of the unwanted containers is to manually delete them in the cluster via kubectl edit.

Notably, this case is not unique — pretty much the same thing can happen with most of the resources. Meanwhile, a trigger may be not only a failed release but also a canceled release (i.e., when Helm gets an INT, TERM, or KILL signal).

The root of this phenomenon and what to do about it

The thing is that some resource fields are missing in the chart, but they are present in the cluster. It is hard to tell whether Helm should remove those fields or not.

But why not then just delete everything that isn’t in the chart’s resource manifest? The answer is that Kubernetes or Kubernetes operators can make changes to a resource that Helm must never delete. For example, Istio may add an Istio Proxy sidecar container to the Deployment. In this case, Helm should not delete this sidecar container, even though it is not in the chart.

To figure out what to do, Helm must divide up the “extra” fields — those that only exist in the resource in the cluster — into fields it controls and those it does not control. It can delete fields it controls, but it cannot mess with the fields it does not control.

When using helm upgrade, the resource fields from the new release and the previous successful release are considered the fields that Helm controls. Where the issue usually emerges is what if a previous release was unsuccessful or was canceled, but it brought in some important changes, such as new controllable fields?

In the end, the greater the number of releases that fail or are canceled, the more orphaned fields are left in the cluster resources. In some cases, they are perfectly harmless. In others, they can lead to denial of service or even data corruption/loss.

The worst part is that there is no simple solution to this issue within Helm. One way would be to devise a new approach for Helm releases, where for each individual resource, its last applied state would be recorded.

However, there is a better way: replacing 3-Way Merge with Server-Side Apply.

What is Server-Side Apply?

Kubernetes 1.22 introduced a new way to update resources in a cluster called Server-Side Apply. Let’s now compare resource updates via 3WM versus SSA.

Upgrading a resource using 3WM requires you to do the following:

1. Retrieve the resource manifest from the latest successful release.
2. Retrieve the resource manifest from the chart.
3. Retrieve the resource manifest from the cluster.
4. Compose a 3WM patch based on those three manifests.
5. Send an HTTP PATCH request to Kubernetes containing a 3WM patch.

Upgrading a resource using SSA requires you to complete the following two steps:

1. Retrieve the resource manifest from the chart.
2. Send an HTTP PATCH request to Kubernetes containing the resource manifest.

SSA’s advantages include:

• Ease of use.
• No need to keep track of the last applied resource manifest — Kubernetes keeps track of it itself.
• No need to know which resource fields are controlled and which are not. Kubernetes stores this information in the resource’s managedFields field.
• Updating the resource and storing information about the controlled fields in a single atomic operation.

If it were possible to replace 3WM with SSA in Helm, there would be no need to look at manifests from previous releases—except for the cases where you need to figure out which resources need to be deleted entirely if they have already been removed from the chart. Implementing SSA would completely eliminate the issue of orphaned fields in the cluster’s resources.

Server-Side Apply in Helm, werf, and other tools

Flux, Argo CD, and kubectl/kustomize feature SSA support, although so far only Flux has it enabled by default. Unfortunately, SSA was never implemented in Helm 3, although SSA support was in place as early as in Kubernetes 1.16 as Alpha (you could enable it via the feature gate), while in Kubernetes 1.22, it became GA (enabled by default).

In werf 2.0, we developed and implemented a new deployment engine called Nelm that succeeded Helm 3. Not only did we add a lot of new stuff to Nelm, but we completely replaced 3WM with SSA as well.

Introducing SSA helped us resolve a number of other issues, such as this one (which still plagues Helm despite that it originally surfaced in version 2 many years ago). SSA has also allowed us to implement a few features, such as automatically discarding resource changes made manually with kubectl edit.

SSA was introduced in werf 1.2. We have been running it in experimental mode (including in production) for over a year now. All werf 2.0 users use SSA by default. The best part is that all the issues previously associated with 3WM have now been eliminated. At this point, we recommend werf 2.0 and SSA for production use.

As for werf 1.2 users: the migration to werf 2.0 is very easy and, apart from having to validate the Helm charts more rigorously, little needs to be changed.

Server-Side Apply instead of 3-Way Merge: How werf 2.0 solves Helm 3 challenges was originally published in werf blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

A brief reminder of what werf is

Feel free to skip this part of the article if you are an existing werf user or just aware of this project!

werf is an Open Source tool for powering your Kubernetes-based CI/CD pipelines. It works alongside the CI system of your choice and handles the entire CI/CD lifecycle: builds container images, deploys them to Kubernetes clusters, and eventually deletes them.

To deploy your apps with werf, you just need a Git repository with a Helm chart, a simple werf.yaml file and a Dockerfile. With such a repo in place, run the werf converge command to build the images, publish them to the container registry, and deploy them to your Kubernetes cluster.

To make it possible, werf relies on well-known technologies such as Docker, Buildah and Helm (or Nelm since now) under the hood. But it’s more than just a mere wrapper. For example, werf brings several unique features, such as distributed caching out-of-the-box, automatic tagging based on the image content, smart container registry cleanup based on special Git policies, and a number of other niceties.

Since December 2022, werf is a CNCF Sandbox project. In the past month, we’ve seen 10,000 active projects using werf (usually, one such project equals one Git repository where werf is applied). Today, werf boasts almost 4,000 stars on GitHub and 8 years of very active and robust development.

What Nelm is and what the future holds for it

Nelm is the biggest change in werf 2.0, so let’s have a better take on it. We can briefly describe Nelm as our (partial) reimplementation of Helm 4 — the release we all have been waiting for but have never seen.

Helm itself consists of two key components: a chart subsystem and a resource deployment subsystem. We essentially rewrote the deployment subsystem from scratch (while maintaining backward compatibility). We have also improved and continue to refine the chart subsystem.

Note! Currently, you can try Nelm only as part of werf. However, in the future, it will become a standalone tool with a convenient API and the option to integrate it into other CI/CD solutions.

This is what Nelm brings into werf:

• The 3-Way Merge has been replaced by Server-Side Apply — a much more robust mechanism for updating resources in a cluster.
• The werf plan command shows the changes that will be made to the cluster during the next deployment.
• Resource operations (including tracking) during deployment have been efficiently parallelized.
• CRDs deployment has been improved.
• Resource tracking has been significantly improved and revamped.
• Numerous Helm bugs and deployment-related issues (e.g., #6969) have been fixed.

(Here is our GitHub discussion thread where most of these Nelm features were announced as we implemented them.)

We’re working on a few more features, like the ability to set direct resource dependencies instead of using hooks, weights and init containers, as well as the ability for regular resources to use all the advanced hook features. We will announce them later as they become generally available.

You can learn more about Nelm in the next article we will publish soon. (By the way, our Telegram group is a great way to stay tuned for updates!)

How to try werf v2.0

Nelm has slightly different behavior in some cases (compared to Helm), such as stricter validation of charts. That’s why we decided to make it a default engine not in werf v1.2, but in v2.0 only.

werf v2.0 is almost fully compatible with v1.2 — here is the list of backward-incompatible changes (it’s really tiny!). We recommend upgrading to version 2.0, which is much easier than it was when migrating from werf v1.1 to v1.2. What about werf v1.2? It goes into maintenance mode — no new features are planned for it.

Another big change is about version numbering — starting with werf 2.0, we will stick to semantic versioning and plan to release a major version about once a year. This will allow us to streamline and speed up the development without risking compromising backward compatibility in minor or patch versions. On the other hand, this will allow us to be more careful about backward compatibility in minor and patch updates.

Use the following command to try werf v2.0:

source $(trdl use werf 2 stable)

As a reminder, werf comes with several release channels:

• Alpha. Quick to deliver new features, but may be unstable.
• Beta. Best suited for more extensive testing of new features in order to find problems.
• Early-Access. Safe enough for non-critical environments and for local development; allows you to get new features earlier.
• Stable. Generally safe and recommended for widespread use in any environment as a default option.
• Rock-Solid. The most stable channel; recommended for critical environments with strict SLA demands.

The long road from werf v1.2 to v2.0

Now, back to those very “300+ releases” mentioned earlier — werf has indeed accumulated quite a lot of new features and changes over all those years. Here are some of the most significant features that have emerged in the process of making werf 1.2 (not related to Nelm):

1. Building Dockerfiles in werf using Buildah under Linux, Windows, and macOS.
2. Layered caching in registry for Dockerfiles.
3. Out-of-the-box support for building images for arbitrary platforms and for multiple platforms at once.
4. The development mode ( --dev) allowing you no longer worry about determinism and intermediate commits during debugging and developing.
5. A new directive for dependencies images has been added to werf.yaml (as of version 1.2.60).
6. The werf bundle render command to render bundle manifests for further deployment by third-party tools or for debugging.
7. The werf kube-run command, which is similar to werf run, but instead of a local container, it runs a pod in a K8s cluster.
8. Status tracking and event collection for all resource types, not just Deployments/StatefulSets/DaemonSets/Jobs.
9. The option to wait for an external (out-of-release) Kubernetes resource to be ready before deploying a release resource.
10. Migration to the new trdl update manager. By the way, it is another Open Source project cultivated by the werf team.

Official werf resources

• werf website
• werf Telegram chat
• werf repo on GitHub
• Nelm repo on GitHub

Let us know how your migration to werf v2.0 is going and stay tuned for more upcoming news regarding werf v2.0.x & Nelm updates!

werf 2.0 is out with a new deployment engine Nelm replacing Helm was originally published in werf blog on Medium, where people are continuing the conversation by highlighting and responding to this story.