I've worked on Crossplane for almost eight years, since the v0.1 release. In that time I've watched a lot of people use it to put cloud infrastructure behind an API. For the last few months I've been using it to put a particular, demanding kind of infrastructure behind an API: a fleet of GPUs running model inference.

The project is called Modelplane. It lets a platform team turn a pile of accelerators (across clouds, neoclouds, and on-premise) into one fleet. It also lets the ML teams they support deploy a model and get a stable, OpenAI-compatible endpoint without thinking about where it runs.

Modelplane exists because open-weight models have moved inference out of the labs and hyperscalers and into everyone else: neoclouds, regulated enterprises keeping models inside their own walls, and companies trying to get their inference bills under control. The open source stack for serving a model on a single cluster is strong now: vLLM, SGLang, KEDA, Gateway API, DRA. But inference almost never stays in one cluster. Capacity is scattered across hardware types, providers, and regions. The hard problems are now above the cluster: placing models across the capacity you have, provisioning more, routing by cost and locality, moving weights around. The labs and the hyperscalers all built systems to do this, but they built them privately. That's the gap Modelplane fills: an open control plane that sits above your clusters and operates them as one inference fleet.

If the inference part interests you, the Modelplane docs and Bassam's introduction are the place to go. This post is for the Crossplane crowd, because the part I think you'll find interesting is that Modelplane is, top to bottom, a Crossplane configuration. It has no bespoke controllers and no custom operators: it's compositions and composition functions. The same primitives you could use to compose an RDS instance, pushed a lot harder.

I want to cover the problem we set out to solve with Crossplane, the parts of the framework we leaned on hardest, and the edges we hit and fixed upstream.

The problem, in Crossplane terms

Strip away the inference vocabulary and Modelplane's job is one Crossplane users will recognize: take a declarative description of what someone wants, and turn it into composed infrastructure spanning cloud accounts, many Kubernetes clusters, and the workloads on them. Provision an EKS or GKE cluster with the right GPUs. Install an inference stack onto it. Decide which cluster each model runs on, and how many copies. Keep it all converged as clusters come and go and people's inference needs change.

Crossplane was built for that shape of problem. Providers gave us reach: we provision clusters and the infrastructure they need across different clouds, and install software onto them, without needing to write new controllers. Functions allowed us to focus on our business logic, the placement and the scheduling. We didn't have to write the controller plumbing by hand: the watches and requeues and finalizers and drift correction that Crossplane core already handles.

Crossplane v2 helped here too. Modelplane has two clear personas. Platform teams describe the fleet. ML teams describe a model. That split maps onto a scope boundary: an InferenceCluster or InferenceClass is cluster-scoped, a ModelDeployment or ModelService is a plain namespaced composite resource the ML team owns. v2 namespaced composites let us express that directly, with no claim-and-XR duality to explain. That's useful, but it isn't what made the project buildable.

What made it buildable: Developer experience

What really unlocked this project was the new Crossplane CLI and the schemas it generates.

Modelplane's functions are all written in Python. We chose Python because it's the lingua franca of the ML world. We hope it might help folks who aren't yet cloud native experts contribute to the project. Writing functions in Python used to mean giving up a lot of the tooling that makes a codebase feel like a proper project. The new crossplane CLI changed that. It scaffolds a project, generates an XRD from an example resource, and generates typed schema bindings for your APIs.

Those generated models changed how we worked. Our functions read and write typed objects instead of poking at untyped dictionaries and hoping the field is spelled the way we remember. A typo or a wrong type now fails at author time. The models also sped up the coding agents we leaned on while building. A generated type tells the agent the exact shape, so it got field names and types right the first time.

There was friction. We outgrew the CLI's built-in function builders early, and we needed schema generation for one language, not all four. Both of those turned into upstream contributions, which I'll come back to.

Designing the API

The hardest part of Modelplane was designing the API.

People come up to me at conferences worried about how they'll make breaking changes to the APIs they build with Crossplane. My answer is usually that you almost never have to, if you really think the API through before you release it. That discipline pays off: reach for arrays and enums before you think you need them, use required fields sparingly, and leave room to grow without a breaking change.

Take the ModelDeployment, arguably Modelplane's most important API. It's how an ML team describes a model to serve: its engines, what their pods need from a node, and how many replicas to run across the fleet.

apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen3-8b
  namespace: ml-team
spec:
  replicas: 1
  engines:
  - name: qwen3-8b
    members:
    - role: Standalone
      nodeSelector:
        devices:
        - name: gpu
          count: 1
          selectors:
          - cel: device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0
      template:
        spec:
          containers:
          - name: engine
            image: vllm/vllm-openai:v0.23.0
            args:
            - "--model=Qwen/Qwen3-8B"
            - "--served-model-name=qwen"
            # ... engine flags omitted for brevity

I got the engines array wrong at first, and only caught it about two weeks before we released v0.1. Until then a ModelDeployment had a spec.topology block where you'd write tensor: 8 and pipeline: 2. Modelplane would derive engine-specific flags like --tensor-parallel-size and inject them. The problem was that this coupled Modelplane to engine specifics. We could only run the engines whose flags we knew how to inject. It also couldn't express the data and expert parallelism we knew was coming. I caught it only because I sat down to write worked examples for those topologies and found I couldn't. I replaced it with shape: an engine is an array of Standalone, Leader, or Worker members, and the parallelism lives entirely in the flags the user writes. My takeaway: don't rush API design. Work through it with your users and peers, let it sit if you can, and write enough worked examples to confirm you can model everything they need before you commit.

The functions do the work

One thing I really like about composition functions is that they scale with your problem. At the low end a function is a few Go templates or some KCL inlined in the Composition. At the high end it's a program that can do almost anything a traditional Go operator can. Python rides that whole range too, and Modelplane is what the far end looks like once the problem grows.

Modelplane's business logic is all composition functions. Scheduling is an interesting example. I'll walk through it because it shows how far a composition function can be pushed.

Modelplane runs a fleet scheduler in the control plane. Each cluster in the fleet is an InferenceCluster, and each of its node pools references an InferenceClass that declares the pool's hardware. When an ML team creates a ModelDeployment, the fleet scheduler places each replica against that declared node pool capacity, before any nodes necessarily exist. That's a different job from in-cluster DRA, which is a runtime allocator: its drivers publish ResourceSlices about real hardware on real nodes.

Scheduling happens in two layers. The fleet scheduler picks the cluster and node pool; the cluster's own in-cluster scheduler and DRA then place pods on nodes and bind GPUs. That in-cluster scheduler is good at its layer (a single cluster supports up to 5,000 nodes), but a fleet isn't one cluster. It can't be: a cluster doesn't span clouds, and you wouldn't want one blast radius over your whole accelerator footprint even if it could.

Modelplane borrows DRA's vocabulary and lifts it to the fleet layer. DRA's typed, domain-prefixed attribute model and CEL predicate language are a good fit, and they express cleanly in OpenAPI. A platform team's InferenceClass declares what a pool's hardware offers, the same shape DRA uses to describe a real device:

devices:
- name: gpu
  claim: DRA
  driver: gpu.nvidia.com
  deviceClassName: gpu.nvidia.com
  attributes:
    architecture: { string: Ada Lovelace }
  capacity:
    memory: { value: "23034Mi" } # the L4's real usable VRAM

A deployment's nodeSelector then asks for what it needs with a CEL expression:

selectors:
- cel: device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0

Our scheduler evaluates that CEL against an InferenceClass's declared attributes, and DRA evaluates the same expression against a real GPU when the pod binds. The same expression runs at both levels.

A composition function is, by design, a pure function of its inputs. Crossplane hands it the observed composite, and the function returns the desired children. Our scheduler is exactly that, a pure function of observed state, which is what makes it safe to run on every reconcile. A scheduler can't decide placement from the deployment alone, though. It has to see the whole fleet: every InferenceCluster and its published capacity, and every ModelReplica that already exists, including those of other deployments, so it can account for capacity they've already consumed.

That's require_resources (what some of you will know as extra resources), and we couldn't have built the scheduler without it. The function asks Crossplane for the resources it needs to reason over, and Crossplane fetches them and calls the function again with them in hand:

# Every InferenceCluster: candidate clusters with their declared capacity.
response.require_resources(
 rsp, name="clusters",
 api_version="modelplane.ai/v1alpha1", kind="InferenceCluster",
 match_labels=clusters_match_labels,
)
# Every ModelReplica across all deployments: capacity already in use.
response.require_resources(
 rsp, name="all-replicas",
 api_version="modelplane.ai/v1alpha1", kind="ModelReplica",
)

That second call, asking for every resource of a kind with no name or label filter, is one of the things we had to fix upstream.

Where Modelplane pushed Crossplane, and we fixed it upstream

Build something this demanding on Crossplane and you find its rough spots. Modelplane found several. Rather than work around them, we sent fixes upstream, so your configurations get the benefit too.

The first thing we hit was the build itself. The CLI's built-in function builders are great until you outgrow them, and we needed to coordinate function builds with Nix, the rest of our build system. We also only consume one of the four schema languages the CLI generates. We added an explicit functions list that loads pre-built image tarballs instead of building them, and a schemas block that restricts generation to the languages you actually use (crossplane/cli #24). Now our functions build with the same tooling as everything else, and we generate Python and nothing else.

The DRA-style attribute model then walked us into a chain of three related bugs, all rooted in the same thing: a DeviceAttribute has fields named exactly int, bool, string, and version. Those are Python keywords and builtins, and they broke at every layer. First, schema generation emitted Python models that referenced undefined type aliases and wouldn't import at all (crossplane/cli #64); the fix was to bump the code generator to a version that sanitizes such names and preserves the wire name with a Pydantic alias. With the models importable, serialization was next: the SDK emitted bool_: true under the Python attribute name instead of bool: true under the wire alias, so the API server rejected it. Passing by_alias=True, and switching from exclude_defaults to exclude_unset so we serialize the fields the caller actually set rather than the fields that differ from a default, put both right (function-sdk-python #208, with helpers in #205).

The fleet scheduler also needed something the framework couldn't yet express. It has to reason over every InferenceCluster and every ModelReplica in the control plane, not one resource by name or a set by a label match. A require_resources selector with no match field is the natural way to say "all resources of this kind," and the protobuf allows it, but Crossplane rejected it on both the wire and the SDK side. We taught the reconciler to treat an empty selector as match-all (crossplane/crossplane #7241) and relaxed the SDK's validation to match (function-sdk-python #213). That match-all call is now the first thing the scheduler does every reconcile.

Testing functions this complex surfaced one more. crossplane render reimplements the XR reconciler's composition pipeline, and that parallel copy drifts from the real one, so a function can pass render and behave differently in a real control plane. We added a hidden render engine that runs the actual reconciler against an in-memory client, so tools can test against the real composition pipeline rather than a copy of it (crossplane/crossplane #7280).

Finally there's a one-pager up for function-controlled deletion (#7242), which aims to make ordered deletion of composed resources within one XR easier and more expressive than using Usages.

None of these are glamorous. But that's how a framework gets better: you build something hard on it and fix the friction.

What I took away

I went into Modelplane wanting to know whether you could build something this demanding entirely on Crossplane, without dropping to a hand-written controller when the going got hard. You can, though we had to fix the framework in a few places to get there. It's still compositions and functions any adopter can read and extend.

Modelplane is Apache 2.0 and developed in the open. If you run accelerators of any kind, take a look at modelplane.ai, and if you build hard things on Crossplane, find its edges and help make them better.

TL;DR

• The vast majority of v1.x control planes can upgrade to Crossplane v2 right away with no changes. Claims, composite resources (XRs), Compositions, and cluster-scoped managed resources all keep working.
• A small set of v1 features were removed or changed in v2. If your control plane uses one, you'll want to find that out before you upgrade.
• crossplane beta upgrade check is a new, read-only CLI command that scans a live v1.x control plane and reports exactly what (if anything) would break, which resource is responsible, and how to fix it.
• It ships in the v1.20 CLI, so install the latest v1.20 CLI and run it against your cluster before you upgrade.

Ready to upgrade?

When we designed Crossplane v2, backward compatibility was a first-class goal: the vast majority of v1.x control planes can upgrade without changing a thing. Your claims, composite resources (XRs), compositions, and cluster-scoped managed resources all keep working in v2.

But "the vast majority" isn't "everyone." Crossplane v2 does remove or change the behavior of a small set of v1 features, and if your control plane uses any of them, you'll want to catch it before you upgrade. Until now, that meant reading the backward compatibility notes and then spelunking through every composition, package, and resource in your cluster for something that's going away - tedious and easy to get wrong on a control plane of any real size. Understandably, that uncertainty has kept some folks from upgrading to v2.

crossplane beta upgrade check is now available to give you a definitive answer. It's a read-only command that connects to a live v1.x control plane and exhaustively scans it for anything v2 removes or changes in a breaking way. Point it at your cluster before you upgrade and it takes you straight to the resources that need attention, with the exact fix for each.

What it checks

The command covers every breaking change called out in the v2 backward compatibility notes. For each one, it reports the specific resources responsible so you can act on them directly:

• Native patch-and-transform Compositions. Composition Functions are how you compose resources in Crossplane v2. Native patch-and-transform (P&T) Composition, where resources and patches are embedded directly in the Composition, is removed. This is the change most likely to affect you, so any Composition still using it needs to move to a function pipeline.
• ControllerConfig usage. The ControllerConfig type is removed, replaced by DeploymentRuntimeConfig. The check finds both ControllerConfigs themselves and the Providers or Functions that still reference one.
• External secret stores. This alpha feature is removed in v2. The check finds where it's enabled and where your Compositions and resources still publish connection details through it.
• Unqualified package sources. Crossplane v2 drops the implicit default registry, so every package reference (Providers, Configurations, Functions, and their dependencies) must be fully qualified with its registry hostname.
• Composite resource connection details. This one is informational. Legacy XRs and Claims keep aggregating connection details in v2, so the upgrade itself needs no action here. The check highlights resources you'd only need to revisit if you later migrate them to v2-style namespaced XRs, which do not support connection details at the XR level.

The command is also careful about false positives. Crossplane's controllers and the API server inject defaults onto every cluster, and the check knows to look past those and flag only the features you actually opted into. A finding means real usage you'll need to address.

Let's see it in action

Running the command is as simple as pointing it at a cluster. By default it uses your current kubeconfig context and sweeps the entire control plane: all of your Crossplane configuration, cluster-scoped resources, and every namespace:

crossplane beta upgrade check

On a control plane that's already clean, you get the answer you're hoping for and a zero exit code:

[✓] 0 issues, 0 informational, 0 incomplete checks.

There we go - ready to upgrade. On a control plane that still leans on some v1-only features, the report tells you precisely what stands in the way. Here's a representative output from a control plane that is still using native patch-and-transform and external secret stores:

[✗] 4 issues, 0 informational, 0 incomplete checks.

[✗] Native patch-and-transform Compositions - 3 issues
    │
    │  Crossplane v2 removes native patch-and-transform (P&T) Composition. Compositions must use mode: Pipeline with Composition Functions.
    │  Fix:   Migrate to Composition Functions (spec.mode: Pipeline with spec.pipeline steps). Run "crossplane beta convert pipeline-composition" to convert existing Compositions.
    │  Docs:  https://docs.crossplane.io/latest/guides/upgrade-to-crossplane-v2/#native-patch-and-transform-composition
    │         https://docs.crossplane.io/v1.20/cli/command-reference/#beta-convert
    │
    │    NAME                                                            FIELD
    │    composition.apiextensions.crossplane.io/nativepnt-composition   .spec.mode
    │    composition.apiextensions.crossplane.io/nativepnt-composition   .spec.patchSets
    │    composition.apiextensions.crossplane.io/nativepnt-composition   .spec.resources
    └──
[✓] ControllerConfig usage
[✗] External secret stores - 1 issue
    │
    │  Crossplane v2 removes support for external secret stores. Publish connection details as Kubernetes Secrets composed by your Compositions, or adopt External Secrets Operator if
    │  you need an external store.
    │  Fix:   Disable --enable-external-secret-stores on the Crossplane Deployment, replace StoreConfig-based publishing with composed Kubernetes Secrets (or adopt External Secrets
    │         Operator), then delete StoreConfig resources. No automated converter exists.
    │  Docs:  https://docs.crossplane.io/latest/guides/upgrade-to-crossplane-v2/#external-secret-stores
    │         https://docs.crossplane.io/latest/guides/connection-details-composition
    │         https://github.com/external-secrets/external-secrets
    │
    │    NAMESPACE           NAME                         FIELD
    │    crossplane-system   deployment.apps/crossplane   .spec.template.spec.containers[0].args
    └──
[✓] Composite resource connection details
[✓] Unqualified package sources

crossplane: error: blockers found

Each finding is self-contained. The top line summarizes the breaking change in plain language, the Fix: line tells you what to do about it - often naming the exact crossplane beta convert command that will migrate the resource for you - and the Docs: line links straight to the relevant section of the upgrade guide. Below that, a table lists every resource and field responsible, so there's no guesswork about which of your Compositions needs attention.

The command exits non-zero when it finds blockers and zero when it doesn't, and -o json emits the same findings as structured data:

crossplane beta upgrade check -o json

The exit code and JSON output make the check easy to automate anywhere it has access to your cluster: a readiness gate in a CD or promotion pipeline, or a scheduled job watching a fleet of control planes. Most often, though, it's simply something you run by hand once, right before you upgrade.

Performance Tuning

Scanning every managed resource for external secret stores usage is the most expensive part of the run on control planes with many provider CRDs, and these flags let you make a trade off for execution time versus the load the command puts on your API server:

• --skip-managed-resources skips the managed-resource scan entirely. The command still inspects StoreConfig resources, the Crossplane Deployment, and composite resources for external secret stores usage; it just doesn't scan managed resources. This gives the biggest reduction in run time, at the cost of not catching external secret stores usage on managed resources.
• --concurrency (default 10) bounds how many resources the command processes in parallel. Lower it, for example --concurrency 2, to put less load on a busy production control plane you don't want to overload; raise it to finish faster at the cost of more load on the API server.

How to get it

Because this tool is only useful before you upgrade, it ships exclusively in the v1.20 Crossplane CLI - a v2 control plane has nothing left to check, and the v1-only types these checks rely on don't even exist there. We made a deliberate exception to v1.20's critical-fixes-only maintenance policy to ship this in the v1.20.9 patch release, precisely so the whole community can check their control planes before they make the jump.

If you're running a v1.x control plane, you likely already have the CLI on hand. You just need a build that includes upgrade check, which means the latest v1.20 patch release. Install it with:

curl -sL "https://raw.githubusercontent.com/crossplane/crossplane/main/install.sh" | XP_VERSION=v1.20.9 sh

You can also download the binary directly from releases.crossplane.io, or see Download other CLI versions in the docs for more options. Then confirm you're ready to roll:

crossplane beta upgrade check --help

When you're ready to dig into the changes themselves, the Crossplane v2 upgrade guide and the backward compatibility notes walk through each breaking change and how to migrate.

Upgrade with confidence

Most of you will run crossplane beta upgrade check, see a clean report, and upgrade to v2 the same day - which is exactly the outcome all that backward compatibility work was for. For everyone else, it trades guesswork for a precise list of what to change before you go. Either way, you upgrade deliberately, knowing exactly where your control plane stands.

This check is just a first step. It tells you what needs to change, and for some breaking changes it can already point you at a crossplane beta convert command that does the rewrite for you, but it stops short of migrating your control plane end to end. Smoothing that path is where the community should head next.

To lead that effort, we've started a new special interest group (SIG) for v2 migration. Drop into #sig-v2-migration to help shape where we invest in tooling, share best practices, and learn from others making the same move - and to tell us how the upgrade check is working for you, including any case it misses or flags incorrectly.

We love to hear from the community, as they are exactly what makes this project great. Whether you are a developer, user, or just interested in what we're up to, feel free to join us via one of the following methods:

• Website
• Slack
• GitHub
• LinkedIn
• YouTube

The combination of YAML+CEL provides a simple experience to define Kubernetes resources — declare what you want, wire dependencies with CEL expressions, and let the system figure out the order. It's an intuitive model, and we've been hearing from our community that they want this authoring experience inside Crossplane, alongside the many existing languages, functions, and experiences that Crossplane already supports.

We're excited to announce function-kro — a Crossplane composition function that brings the YAML+CEL authoring model of kro into Crossplane's pipeline architecture. Your kro-style resource definitions drop straight into a Composition pipeline step — same syntax, same CEL expressions, unchanged and now running inside Crossplane. function-kro embeds kro's graph builder, CEL evaluator, and runtime engine to offer full feature parity with the latest kro release.

function-kro has been donated to the Crossplane community as a community extension project at crossplane-contrib/function-kro. We've shared it with the kro community and we welcome collaboration and contributions from anyone interested in this experience!

The Function Pipeline

Below is a Crossplane Composition that uses function-kro to define a NetworkingStack platform API — a VPC, Subnet, and SecurityGroup, all wired together with CEL expressions:

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: networkingstack
spec:
  compositeTypeRef:
    apiVersion: example.crossplane.io/v1
    kind: NetworkingStack
  mode: Pipeline
  pipeline:
  - step: kro-run
    functionRef:
      name: function-kro
    input:
      apiVersion: kro.fn.crossplane.io/v1beta1
      kind: ResourceGraph
      status:
        networkingInfo:
          vpcID: ${vpc.status.atProvider.id}
          subnetID: ${subnet.status.atProvider.id}
          securityGroupID: ${securityGroup.status.atProvider.id}
      resources:
      - id: vpc
        template:
          apiVersion: ec2.aws.m.upbound.io/v1beta1
          kind: VPC
          metadata: {}
          spec:
            forProvider:
              region: ${schema.spec.region}
              cidrBlock: 192.168.0.0/16
              enableDnsHostnames: false
              enableDnsSupport: true
      - id: subnet
        template:
          apiVersion: ec2.aws.m.upbound.io/v1beta1
          kind: Subnet
          metadata: {}
          spec:
            forProvider:
              region: ${schema.spec.region}
              cidrBlock: 192.168.0.0/18
              vpcId: ${vpc.status.atProvider.id}
      - id: securityGroup
        template:
          apiVersion: ec2.aws.m.upbound.io/v1beta1
          kind: SecurityGroup
          metadata: {}
          spec:
            forProvider:
              name: my-sg-${schema.metadata.name}
              region: ${schema.spec.region}
              description: Default security group for NetworkingStack
              vpcId: ${vpc.status.atProvider.id}
  - step: auto-ready
    functionRef:
      name: function-auto-ready

This is a two-step pipeline. Step one — function-kro — handles resource composition using kro's YAML+CEL model. Each resource is defined as a template with ${...} CEL expressions that wire dependencies between them. ${schema.spec.region} pulls from the Composite Resource (XR) spec, while ${vpc.status.atProvider.id} creates a dependency on the VPC's output — the subnets and security group won't be created until the VPC is ready and its ID is available. The status block aggregates data from composed resources back to the XR, so consumers of the NetworkingStack API can read the IDs of each resource directly from their XR’s status.

Step two — function-auto-ready — handles automatic readiness detection. But the pipeline could have ten steps. Need to enforce policy on all resources? Add a step. Need to inject cost-allocation tags? Add a step. Need to pull secrets from an external API? Add a step. Each step is independent — you can add the capabilities you need to support your simple resource definitions.

Again, the resources block in that Composition is identical to what you'd write in a standalone kro ResourceGraphDefinition (RGD). If you already have kro resource definitions, they drop into the pipeline input without changes. And when you need more — multi-cloud implementations, safe rollouts, operational controls — you add pipeline steps without rewriting what you already have. Same composition language, with a growth path when you need more.

SimpleSchema for Crossplane

kro bundles schema definition into the RGD using SimpleSchema — a compact shorthand (e.g., region: string | default=us-west-2) that avoids writing full OpenAPIv3. This is a straightforward experience to define a schema, so we’re making it available in Crossplane’s developer tooling as well.

Given a kro-style RGD that includes SimpleSchema:

apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
  name: networkingstack
spec:
  schema:
    apiVersion: v1
    kind: NetworkingStack
    spec:
      region: string
    status:
      vpcID: ${vpc.status.atProvider.id}
      subnetID: ${subnet.status.atProvider.id}
      securityGroupID: ${securityGroup.status.atProvider.id}
  resources:
  - id: vpc
    template:
      # ... same resource templates as above

You’d be able to generate the full Crossplane XRD with OpenAPIv3 schema automatically:

crossplane xrd generate --input=rgd rgd.yaml

In other words, SimpleSchema becomes just another schema authoring option in Crossplane, similar to how function-kro is just another composition logic option. Write your schema and logic the way you prefer, and Crossplane handles the rest.

Running Inside Crossplane's Architecture

Now that we've seen the syntax, let's look at how these resource definitions fit into the broader Crossplane platform. Because function-kro is a function pipeline step, your YAML+CEL definitions automatically participate in the capabilities that Crossplane provides to all compositions. Crossplane does add some structure beyond a standalone RGD — it separates your API schema from its implementation — but that structure is what makes the following possible.

Safe rollouts. CompositionRevisions let you pin existing instances to a known-good revision, roll forward team-by-team, and roll back without a rewrite. When you update your composition logic, it doesn't immediately affect every running instance. You control the blast radius.

Pipeline composability. Add functions from a vast ecosystem of pipeline steps alongside your YAML+CEL definitions. Need to call an external API, query a cloud provider, connect to a secret store, or enforce policy? Add a pipeline step — your resource definitions stay untouched. For cross-cutting concerns that push beyond what CEL can express, general-purpose languages like Go and Python are available as pipeline steps in the same Composition.

Multi-implementation APIs. Define one API backed by multiple Compositions. The same NetworkingStack API can be backed by AWS, GCP, and Azure. One schema, many implementations. Platform teams define the API once and provide region or cloud-specific implementations behind it.

Operational controls. Pausing reconciliation, management policies, and resource lifecycle control. When something goes wrong at 2 AM, you can stop reconciling a specific instance with a single annotation without touching the Composition or affecting other instances.

Developer experience. Shift-left your testing and validation with crossplane render, unit tests, and integration tests. Run a Composition locally against a sample XR before deploying and see the same outputs that would surface at runtime. Diff your logic against a live environment with crossplane-diff. Develop your logic in a modular fashion with Crossplane’s DevEx tooling.

Extensibility Offers Choice

Crossplane's composition function architecture was designed around a simple idea: the platform team should pick the authoring model that fits their needs and skills. function-kro adds YAML+CEL to that already expansive menu.

Today, Crossplane supports KCL for streamlined expressiveness, Go and Python for teams that want full programming languages, YAML templating for those familiar with Helm, HCL for those coming from Terraform, and many others. Now YAML+CEL can be added to that list for teams that like declarative resource definitions with simple expressions to wire them all up.

You don't have to choose one approach for everything - different Compositions can use different functions, and a single pipeline can mix multiple languages across steps.

Get Involved

We're continuing to track upstream kro releases and will keep function-kro current as the kro project evolves. We're also expanding our library of working examples — YAML+CEL has been added to the getting started with composition guide in the Crossplane docs and the function repo includes multiple examples for basic composition, conditionals, readiness checks, collections, and external references to help you get started.

We welcome the kro community to start using and contributing to this new function! We love to hear from all users, as they are exactly what makes this project great. Whether you are a developer, user, or just interested in what we're up to, feel free to reach out via any of the following:

• GitHub — function-kro
• Crossplane website
• Crossplane docs
• Slack
• LinkedIn
• YouTube
• Community meetings

KubeCon + CloudNativeCon Europe is back in Amsterdam, and so are we. This year's edition is shaping up to be one of the most exciting yet for the Crossplane community, featuring end-user stories from major financial institutions and global tech companies, from API-driven infrastructure to self-healing platforms.

Whether you're just getting started with Crossplane, running it in production, or curious about where the project is headed, there's a session (or five) in this schedule with your name on it.

Find Us at the Project Pavilion

Stop by the Crossplane kiosk in the CNCF Project Pavilion (Kiosk #P-16) during Solutions Showcase hours:

• Tuesday, March 24: 10:45–19:00 CET
• Wednesday, March 25: 10:30–17:00 CET
• Thursday, March 26: 10:30–14:00 CET

Come ask questions, see demos of the latest work from the Crossplane community, and connect directly with Crossplane maintainers and community members. If you've been curious about where the project is headed, this is the place to find out.

Crossplane Sessions at KubeCon EU 2026

Start Here: The Crossplane Project Intro and Deep Dive

Crossplane: The Cloud Native Framework for Platform Engineering
Tuesday, March 24 | 15:15 – 15:45 CET | Amtrium 1+2
Jared Watts & Adam Wolfe Gordon, Upbound

New to Crossplane or want to catch up on the latest? This is the session to start with. Jared and Adam, two of Crossplane's core maintainers, will introduce the project, walk through key features, and share what's coming next. Expect demos and walkthroughs of current functionality, real-world use cases, and an interactive opportunity to engage directly with the maintainers. Whether you're at KubeCon to evaluate Crossplane for the first time or you've been building with it for years, this session covers something for everyone.

End-User Stories: Crossplane in Production

API-Driven Infrastructure as Code: Kubernetes APIs as the Contract Bridge Between Teams
Tuesday, March 24 | 11:15 – 11:45 CET | Hall 7, Room C
Florian Hopfensperger, Allianz Technology & Yury Tsarev, Upbound

How does one of the largest technology companies in insurance manage infrastructure across teams at scale? Florian and Yury will share how Allianz Technology is using Kubernetes APIs as the contract layer between platform and application teams, and what it looks like to treat your infrastructure as an API in a large enterprise. If you're navigating the organizational and technical challenges of API-driven infrastructure as code, or just getting started with it, this talk is for you.

From Alert Fatigue to Self-Healing: Building AI-Enabled Control Planes in Banking
Tuesday, March 24 | 16:15 – 16:45 CET | Hall 8, Room D
Nuno Guedes, Millennium bcp & Yury Tsarev, Upbound

Nuno and Yury will walk through how Millennium bcp is moving from reactive alert management toward proactive, AI-enabled, self-healing infrastructure. This is a real-world look at what it takes to build AI-enabled control planes in a highly regulated industry, and what Crossplane makes possible at the foundation.

From GitOps to AIOps: Evolving RBI's Kubernetes Platform with Crossplane and Sharded Kargo
Wednesday, March 25 | 11:00 – 11:30 CET | Forum
Gabor Horvath & Ewald Überall, Raiffeisen Bank International

Another major European financial institution sharing their Crossplane journey at KubeCon. Gabor and Ewald will share how Raiffeisen Bank International is evolving their Kubernetes platform from GitOps patterns toward AIOps, with Crossplane as the control plane backbone, and what Sharded Kargo brings to that picture.

From Projects to Products: The Sociotechnical Journey Behind Sony's Internal Cloud Platform
Wednesday, March 25 | 11:45 – 12:15 CET | Forum
Eugenia Bergman & Hagen Tonnies, Sony Interactive Entertainment

Platform engineering is as much a people and organizational challenge as a technical one. Eugenia and Hagen from Sony Interactive Entertainment will walk through how they transformed their internal cloud platform from a collection of projects into a real product, covering the cultural, structural, and technical shifts that made it possible. If you're on a platform team navigating the "projects to products" transition, this one is worth blocking out time for at KubeCon.

Invisible Guardrails: Enabling Developer Velocity With a Secure Platform
Wednesday, March 25 | 15:00 – 15:30 CET | Hall 8, Room F
James Elías Sigurðarson & Vignir Hafsteinsson, Asana

The best platform guardrails are the ones developers don't notice, until they realize they've been shipping fast without breaking things. The Asana team will show how they've built security and compliance into their platform in a way that accelerates developer velocity rather than blocking it. A must-see for anyone at KubeCon trying to balance governance with a great developer experience.

AI Meets Platform Engineering

Choose Your Own Adventure: AI Meets Internal Developer Platform Tuesday, March 24 | 15:15 – 15:45 CET | Hall 7, Room A
Whitney Lee, Datadog & Viktor Farcic, Upbound

Whitney and Viktor are back at KubeCon with an interactive format exploring what happens when AI becomes an interface layer for your internal developer platform. Expect real demos and a conversation about how AI agents can interact with and through control planes built on Crossplane.

More Crossplane Across the Week

Before KubeCon: Cloud Native Rejekts (Saturday, March 21)

Achieving Platform Engineering Multi-Tenancy with kcp and Crossplane
Cloud Native Rejekts | Saturday, March 21 | 17:00 CET | Room 2
Simon Bein, Kubermatic & Lovro Sviben, Upbound

The unofficial start to KubeCon week happens at Cloud Native Rejekts. Lovro Sviben, a Crossplane contributor, will be presenting on multi-tenancy in platform engineering using kcp and Crossplane. If you're arriving early for the weekend, this is a great way to kick off the conversations that will continue all week.

Project Lightning Talk (Monday, March 23)

Project Lightning Talk: What Can Crossplane Actually Do? A Real World Field Guide
Monday, March 23 | 12:27 - 12:32 CET | Elicium 2
Jonasz Łasut-Balcerzak, Baloise
Five minutes, one practitioner, real-world use cases. Jonasz will give a field guide to what Crossplane can actually do.

Platform Engineering Day (Monday, March 23)

The GitOps Paradox: Why Your Devs Need an API You Don't Want to Build
Monday, March 23 | 15:55 - 16:20 CET | Hall 8 | Room F
Simon Koudijs, ConfigButler

Simon makes the case for why developer-friendly APIs are the missing piece in most GitOps setups, and what it actually takes to build one that your teams will use.

Backend-First IDP: A Production Roadmap With Argo CD, Crossplane, and OPA
Monday, March 23 | 15:20 – 15:45 CET  | Hall 8 | Room G
Abhinav Sharma, KodeKloud & Aditya Soni, Sailpoint

A practical production roadmap for building an internal developer platform using Argo CD, Crossplane, and OPA together, with real lessons from teams who've done it.

Party with the Crossplane Community

After a full day at KubeCon, come unwind with the Crossplane and Kubernetes community on Wednesday, March 25, at 6:00 PM. Upbound is hosting an evening for practitioners, contributors, and cloud-native builders at The Upside Down Amsterdam, an immersive museum just a 3-minute walk from RAI Amsterdam, featuring 25+ interactive rooms, optical illusions, and an LED light ball pit.

Free for KubeCon attendees, but spots are limited; come early. Register here.

See You in Amsterdam

KubeCon + CloudNativeCon Europe runs March 23 – 26. The Crossplane community will be present before day zero, starting at Cloud Native Rejekts on the weekend, through Platform Engineering Day on Monday, all the way to the main conference and our community party on Wednesday evening.

Whether you're stopping by the Project Pavilion kiosk, catching talks throughout the week, or joining us at The Upside Down on Wednesday night, we can’t wait to meet you.

Come find us, ask hard questions, share what you're building, and help us figure out together what platform engineering looks like in 2026 and beyond.

We are excited to announce that Crossplane v2.2.0 has been released and is now available for installation into your control planes. This release is a regular quarterly release focused on maturing key areas of functionality across the project, as Crossplane continues to become more capable, more reliable, and more performant for your production workloads. In this post, we'll dive into the highlights.

Debugging with the Pipeline Inspector (Alpha)

Composition functions give platform builders the power to express arbitrarily complex resource generation logic, but debugging that logic inside a running control plane has historically been difficult. You can write tests and use crossplane render locally, but viewing the requests are going into each function step and the responses coming out required updating the pipeline or the functions themselves.

Crossplane v2.2 introduces a new alpha feature to address this: the pipeline inspector. When enabled, the inspector intercepts every function request and response in a composition pipeline and forwards them over gRPC to a user-configured socket path. This opens the door to a range of debugging and observability use cases — from a local observer that dumps requests to stdout during development, to a production-grade system that stores pipeline traces for auditing or troubleshooting.

The pipeline inspector is disabled by default. To enable it, pass the --enable-pipeline-inspector flag to Crossplane. The default socket address is /var/run/pipeline-inspector/socket, which can be overridden by passing the --pipeline-inspector-socket flag. The full Helm values required to enable the inspector and run a basic pipeline logging sidecar are:

# Enable the pipeline inspector feature flag
args:
  - --enable-pipeline-inspector
  - --pipeline-inspector-socket=/var/run/pipeline-inspector/socket

# Inject the pipeline inspector sidecar
sidecarsCrossplane:
  - name: pipeline-inspector
    image: xpkg.crossplane.io/crossplane/inspector-sidecar:v0.0.3
    args:
      - --socket-path=/var/run/pipeline-inspector/socket
      - --max-recv-msg-size=8388608  # 8MB
    volumeMounts:
      - name: pipeline-inspector-socket
        mountPath: /var/run/pipeline-inspector
    resources:
      requests:
        cpu: 10m
        memory: 64Mi
      limits:
        cpu: 100m
        memory: 128Mi

# Add the shared volume for Unix socket communication
extraVolumesCrossplane:
  - name: pipeline-inspector-socket
    emptyDir: {}

extraVolumeMountsCrossplane:
  - name: pipeline-inspector-socket
    mountPath: /var/run/pipeline-inspector

With this configured, Crossplane will forward a copy of each RunFunctionRequest and RunFunctionResponse to the inspector sidecar as the pipeline runs, giving you full visibility into the data flowing through your function pipeline. This feature remains Alpha in v2.2 and will continue to be matured in future releases.

Runtime Configuration for Package Dependencies

Previously, if you needed to configure a package's runtime you had to set a runtimeConfigRef directly on each package resource. This worked for explicitly installed packages, but had no effect on packages installed as dependencies. This made it difficult to leverage Configuration packages as a top-level construct.

ImageConfig, introduced in Crossplane v1.18, has grown into a powerful centralized mechanism for configuring package installation. New in v2.2, ImageConfig can configure the DeploymentRuntimeConfig used by packages matching a given image prefix, including packages installed as dependencies. The new spec.runtime.configRef field lets you specify a runtime configuration and it applies to all matching packages regardless of how they are installed.

Consider a scenario where you're running on Azure and need your provider pods to use Azure Workload Identity. Previously, you might have disabled dependency resolution to install the provider manually. Now you can express this intent once, declaratively:

---
apiVersion: pkg.crossplane.io/v1beta1
kind: ImageConfig
metadata:
  name: azure-workload-identity
spec:
  matchImages:
    - prefix: xpkg.crossplane.io/crossplane-contrib/provider-azure-
    - prefix: xpkg.crossplane.io/crossplane-contrib/provider-family-azure
  runtime:
    configRef:
      name: azure-workload-identity

---
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: azure-workload-identity
spec:
  serviceAccountTemplate:
    metadata:
      annotations:
        azure.workload.identity/client-id: "12345678-1234-1234-1234-123456789012"
  deploymentTemplate:
    metadata:
      labels:
        azure.workload.identity/use: "true"
    spec:
      selector: {}
      template:
        metadata:
          labels:
            azure.workload.identity/use: "true"
        spec:
          containers:
            - name: package-runtime
              args: []

With this ImageConfig in place, all Azure family providers — whether installed explicitly or as dependencies — will get the azure-workload-identity runtime configuration applied, and therefore have Workload Identity configured on their service accounts. Note that a matching ImageConfig runtime takes precedence over any runtimeConfigRef specified directly on a package's spec.

To confirm the configuration was applied, check the appliedImageConfigRefs field in the package revision's status:

$ kubectl get providerrevision crossplane-contrib-provider-family-azure-e57d1e7b1ce7 -o yaml
# ...
status:
  appliedImageConfigRefs:
  - name: azure-workload-identity
    reason: ConfigureRuntime
# ...

XRD Validation Beyond the Spec

Crossplane XRDs let you define rich OpenAPI schemas for the resources your platform users create. Until now, x-kubernetes-validations (CEL validation rules) could only be used to validate fields within the spec of a composite resource. This left a real gap for platform builders who wanted to enforce naming conventions, require specific labels, or validate other metadata properties at admission time.

Crossplane v2.2 closes this gap by allowing XRDs to configure x-kubernetes-validations outside of spec. This lets you write CEL rules that validate things like the resource's name format or required label values and provide a more complete API contract for your composite resources.

For example, suppose your platform requires all XR names to follow a specific pattern. You can now enforce this directly in the XRD:

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: databases.platform.example.org
spec:
  group: platform.example.org
  names:
    kind: Database
    plural: databases
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object
          x-kubernetes-validations:
            - rule: "self.metadata.name.startsWith('db-')"
              message: "Database names must start with 'db-'"
          properties:
            spec:
              type: object
              properties:
                region:
                  type: string

Any attempt to create an Database with a name that doesn't start with db- will now be rejected at the Kubernetes API server level, giving your users clear, immediate feedback rather than a confusing reconciliation error later.

Using OpenAPI Schemas in Functions

Composition and operation functions may wish to know the OpenAPI schemas for the resources they're working with, for example to make schema-aware decisions, validate resources, or dynamically generate resources. Previously a function could indirectly get the OpenAPI schema for a CRD by requesting the CRD as a RequiredResource (or ExtraResource in Crossplane 1.x). However, schemas were not available for built-in Kubernetes kinds and some manipulation was necessary to extract the OpenAPI schema from the CRD.

Crossplane v2.2 introduces RequiredSchemas in the RunFunctionResponse, allowing functions to request schemas. Analogous to RequiredResources, a function can populate the new field with the API group/version/kind of any resources whose OpenAPI schema it needs and on the next reconciliation Crossplane will include the requested schemas in the function request.

With this expansion of the function gRPC API, Crossplane v2.2 also introduces capability advertisement. The RunFunctionRequest now includes a Capabilities field that tells the function what the running version of Crossplane supports. For example, a function that can optionally use OpenAPI schemas may check whether Capabilities contains CAPABILITY_REQUIRED_SCHEMAS to determine whether schemas are available.

Improvements to crossplane beta trace

Many platform engineers use the crossplane beta trace command is a to understand the state of resources in a Crossplane control plane. v2.2 ships two significant enhancements to make it even more useful.

Trace all resources of a given kind. Previously, crossplane beta trace could take only a single resource instance. Now you can pass a kind (and optionally a namespace) to trace all resources of that type at once. This is helpful when you want to quickly understand the health of every Database or every Provider in your cluster:

# Trace all Database resources in the platform namespace
crossplane beta trace databases.platform.example.org -n platform

# Trace all Providers
crossplane beta trace providers

Watch mode. The trace command now supports a --watch flag (shorthand -w), which keeps the trace output live and refreshes it as resources change, similar to kubectl get --watch. This is especially useful during debugging sessions when you're waiting for a resource to reconcile and want to see the dependency tree update in real time:

crossplane beta trace databases.platform.example.org db-production --watch

Notable Breaking Changes

Before upgrading, be aware of two breaking changes in v2.2:

Function input CRDs are no longer installed. Input CRDs included in Function packages are no longer installed by the package manager, following the xpkg specification. Additionally, unknown or disallowed resource types within a package are now silently ignored instead of causing package installation to fail.

Package cache on-disk structure has changed. The internal structure of the package cache has been updated. This breaks an undocumented behavior that allowed packages to be "side-loaded" directly into Crossplane, which was sometimes used for testing. If you relied on this behavior, please see #6981 and #7147 for details on the necessary changes.

Get Involved

We love contributions from the community in any form: code contributions, issues, questions, feedback on proposals, and many more. Whether you are a developer, user, or just interested in what we're up to, feel free to join us via one of the following methods:

• Crossplane website
• GitHub
• Slack
• LinkedIn
• YouTube
• Reddit
• Email
• Attend a community meeting

AI-assisted development has changed the way engineers create and commit code. But writing code is no longer the bottleneck. The bottleneck is everything that happens after git push.

From infrastructure provisioning, policy enforcement, day-two operations, drift, compliance, to cross-team coordination. That still requires multiple steps, and no new tool will fix it. This is an architecture problem. AI needs APIs, not UIs, and most platforms still aren't built that way.

Current Platforms

Talk to almost any organization, and you'll hear that the desired state lives in Git, while the actual state lives in cloud providers. Policies are buried in pipeline configs. Organizational knowledge exists in wikis no one reads and in engineers who eventually leave.

This has worked up to now because humans worked with humans to navigate the context switching and informal coordination required to get the job done. People fill in the gaps, ask the questions, and translate intent across systems.

But in a world where AI agents are embedded into our organizations, this workflow breaks down. The agent hits a wall, not because it lacks capability, but because the platform wasn't built for programmatic access. It was built for humans who can compensate for inconsistency.

Agents require a unified, structured, machine-readable interface. They need explicit governance rules, readable historical patterns, and discoverable dependencies. Without that structure, autonomy stalls.

Platforms Built on Declarative Control

Kubernetes introduced a simple but powerful control pattern that changes this entirely. Every resource follows a consistent schema:

apiVersion: example.crossplane.io/v1
kind: Database
metadata:
  name: user-db
spec:
  engine: postgres
  storage: 100Gi

Desired state lives in spec, actual state is reflected in status, and controllers observe the difference and reconcile continuously. That reconciliation is consistent and automatic; no human is required to coordinate convergence.

Crossplane extends this model beyond containers to all infrastructure and applications: cloud databases, object storage, networking, SaaS systems, clusters, and custom platform APIs. The result isn't just infrastructure-as-code. It's your entire platform, infrastructure, and applications as a single API. That difference matters.

The three core elements that make this work in practice:

• Desired State: the declarative specification of what we think the world should be. (Example: The frontend service should have 3 replicas with 2 GB of memory each.)
• Actual State: the operational reality of what exists in the infrastructure. (Example: The frontend service has 2 healthy replicas, 1 pending.)
• Policy: the rules and governance that constrain operations. (Example: Production changes require approval between 9 AM and 5 PM PST.)

Controllers continuously reconcile desired state with actual state, and policy is enforced at execution rather than left to manual review. Context becomes part of the system, not something external to it.

Why This Model Works for Agents

An AI agent interacting with a Crossplane-managed platform doesn't need to orchestrate workflows across multiple systems. It interacts with a single API surface.

It can discover resource types via the Kubernetes API, inspect status fields for real-time operational state, watch resources for change events, and submit declarative intent. Since reconciliation handles mechanical execution, agents don't need to coordinate step-by-step logic; they just declare intent and let controllers handle convergence.

This separation of concerns is critical. Controllers handle mechanics, while agents focus on higher-level reasoning. Without a control plane, agents become fragile orchestrators. With one, they become declarative participants.

When the entire platform is accessible through a single, consistent API, the agent has everything it needs. No Slack messages and no tribal knowledge required.

Policy at the Point of Execution

In fragmented platforms, governance follows lots of procedures: reviews, tickets, Slack threads. In a Kubernetes-native control plane, governance is architectural.

RBAC controls who can act. Admission controllers validate changes before they're persisted. Policy engines such as OPA and Kyverno enforce constraints at runtime. Crossplane compositions encode organizational patterns directly into APIs. Every change flows through the same enforcement path, no hidden approval steps, no undocumented exception paths.

This removes ambiguity for agents entirely. The system defines what is allowed. Agents operate within clearly defined boundaries, and the platform enforces them automatically.

Crossplane 2.0: Full-Stack Control

With Crossplane 2.0, compositions can include any Kubernetes resource, not just managed infrastructure. That means a single composite API can provision infrastructure, deploy applications, configure networking, set up observability, and define operational workflows,  all in one place.

apiVersion: platform.acme.io/v1
kind: Microservice
metadata:
  namespace: team-api
  name: user-service
spec:
  image: acme/user-service:v1.2.3
  database:
    engine: postgres
    size: medium
  ingress:
    subdomain: users

Behind that abstraction may live RDS instances, security groups, deployments, services, ingress rules, and monitoring resources. To a human developer or an AI agent, it's a single API. That consistency is what enables automation to scale safely.

Day-two operations follow the same pattern. Crossplane's Operation types bring declarative control to scheduled upgrades, backups, maintenance, and event-driven automation:

apiVersion: ops.crossplane.io/v1alpha1
kind: CronOperation
metadata:
  name: weekly-db-maintenance
spec:
  schedule: "0 2 * * 0"
  operationTemplate:
    spec:
      pipeline:
        - step: upgrade
          functionRef:
            name: function-database-upgrade

Operational workflows are now first-class API objects. Agents can inspect them, trigger them, observe their status, and propose modifications. No need for hidden runbooks.

Where to Start

This doesn't require a start-from-scratch migration. Bring core infrastructure under declarative control first. Your existing resources don't need to be replaced; they just need to be unified behind a consistent API.

For teams using AI-assisted development, engineers express intent and iterate quickly as tools accelerate implementation. As deployment decouples from release, with changes shipping behind feature flags and systems reconciling toward the desired state, the platform must be deterministic and self-correcting, not reliant on someone catching drift or running the right command at the right time.

That is what a declarative control plane provides. Crossplane ensures that intent has somewhere safe, structured, and deterministic to land. Without it, AI will always be bolted onto human-centric workflows. With it, agents become first-class participants in infrastructure operations.

And that starts with a consistent API. Get started by checking out the Crossplane Docs, attending a community meeting, or watching to CNCF's Cloud Native Live on Crossplane 2.0 - AI-Driven Control Loops for Platform Engineering.

We are absolutely thrilled today to announce the graduation of Crossplane within the CNCF! This is a huge milestone for the project as it joins the ranks of other mature, production-ready CNCF projects like Kubernetes, Prometheus, and Helm.

Graduation represents the highest level of maturity in the CNCF and signals that Crossplane is stable, reliable, and broadly deployed in many production environments. Originally created and open-sourced by Upbound, it’s been amazing to see how far Crossplane has come since we donated the project to the CNCF at the Sandbox level in 2020!

Built by the Community

This graduation accomplishment is a testament to an incredible community that has rallied around the vision of making cloud native control planes accessible to everyone. The growth of the Crossplane community has been remarkable. We now have over 3,000 people from more than 480 companies that have contributed to the project - that is a ton of people getting involved!

Compared to other projects, Crossplane ranks #13 out of 231 CNCF projects for number of PR authors, placing us in the top 10% of the CNCF. These aren't just faceless numbers—behind every contribution is a real person who decided to help make Crossplane better. Whether you contributed code, documentation, opened issues, or supported another user, you are a key part of this achievement.

Production-Ready and Battle-Tested

Crossplane is built on community engagement and that combined effort has enabled the project to power real production workloads.

We have over 70 public adopters, including organizations like Nike, Nokia, Grafana, NASA Science Cloud, Elastic, Akamai, SAP, and IBM. These organizations, and many others, trust Crossplane to manage their critical services at scale in production.

The project is also operated according to a robust vendor-neutral governance, completed two comprehensive security audits, achieved an OpenSSF Best Practices badge, and follows a mature release process with a well-defined LTS policy for over 100 total releases to date. Crossplane is definitely built as a project you can bet your cloud native software on.

Building for Long-Term Sustainability

Throughout the rigorous due diligence process for our graduation application, our sponsors on the CNCF Technical Oversight Committee provided thoughtful feedback on how we could strengthen the project's long-term sustainability. We took this guidance to heart and implemented several improvements to the project based on their recommendations.

We established a vendor-neutral community registry to host Crossplane packages at xpkg.crossplane.io and migrated critical release infrastructure to CNCF-owned resources, ensuring the project can continue thriving independently. We also rolled out clear policies for Community Extension Projects to help the community navigate the broad ecosystem of Crossplane extensions.

The result of this effort is that Crossplane is now a stronger and more resilient project. We have the infrastructure and governance to sustain the project for years to come. We are truly grateful for the expertise our TOC sponsors brought to the process—their guidance has made Crossplane better for the cloud native ecosystem.

Beyond Graduation

While graduation is a huge milestone, it isn’t the finish line. It's a recognition of how far we've come on this journey, but also a foundation for the future yet to come.

Thank you to every person who has contributed to Crossplane in any way and any team that has adopted the project—we truly appreciate our amazing community. We are also grateful to the CNCF for all the support they have provided the project over the years.

We're excited to continue building the future of cloud native control planes together. Join us in the Crossplane Slack community, star the project on GitHub, and follow us on Bluesky, X, or LinkedIn.

Here's to the next chapter of Crossplane! 🎉

When we started working on Crossplane back in 2018, we had a simple but ambitious goal: bring the elegance of Kubernetes APIs to infrastructure management. Seven years, 3,000+ contributors, and over 100 releases later, we're thrilled to announce Crossplane 2.0—a release that doesn't just represent new features, but rethinks how platform teams approach control planes.

Crossplane 2.0 is the culmination of everything we've learned from watching real-world platform teams scale from managing a handful of cloud resources to orchestrating entire application ecosystems. It's the result of helping folks debug real production issues, iterating on APIs with hundreds of community members, and constantly asking ourselves: "How can we make this simpler?"

What's New in Crossplane 2.0

• Application support - Crossplane 2.0 extends beyond infrastructure to manage applications alongside your cloud resources
• Broader composition capabilities - Compositions can now include any Kubernetes resource, not just Crossplane-defined resources, enabling full-stack abstractions
• Namespaced by default - Composite Resources (XRs) and Managed Resources (MRs) are now namespaced, providing better isolation and aligning with Kubernetes conventions
• Declarative day two operations - The new Operation type supports one-off, scheduled, and event-driven workflows for upgrades, backups, and maintenance
• Managed resource filtering - Install only the managed resources you need instead of getting everything from a provider

The Problem We Set Out to Solve

When we first built Crossplane, we focused on what seemed like the biggest pain point: giving developers self-service access to cloud infrastructure without the operational overhead. The early versions did this well—platform teams could wrap cloud resources like AWS RDS instances in their own APIs, hide the complexity of security groups and subnet configurations, and let developers get infrastructure with simple YAML declarations.

But as we watched teams adopt Crossplane in production, a pattern emerged. Infrastructure was just the beginning. Crossplane v1 let teams move their infrastructure management into Kubernetes with declarative APIs, but applications and infrastructure remained separate concerns. Platform teams could offer self-service APIs for databases and storage, but those APIs couldn't include applications. Teams still had to manage their apps and infrastructure as distinct, disconnected pieces.

The problem was that Crossplane v1's architecture, while powerful, had become overly opinionated. Claims, cluster-scoped resources, and infrastructure-only compositions had also grown complex and created unnecessary friction. We realized we needed to step back and ask: what would Crossplane look like if we designed it today, with everything we've learned?

What We Built

Crossplane 2.0 represents our answer to that question. Here's what's fundamentally different:

Applications Are First-Class Citizens

The biggest change in Crossplane 2.0 is that compositions can now include any Kubernetes resource, not just Crossplane-managed infrastructure. This means you can define a composite resource that provisions a database, configures networking, deploys an application, and sets up monitoring, all in one cohesive abstraction.

For example, a platform team could now offer their development teams a simple "microservice" API that provisions everything needed for a new service:

apiVersion: platform.acme.io/v1
kind: Microservice
metadata:
  namespace: team-api
  name: user-service
spec:
  image: acme/user-service:v1.2.3
  database:
    engine: postgres
    size: medium
  ingress:
    subdomain: users

Behind the scenes, this creates an RDS instance, configures security groups, deploys the application via a Kubernetes Deployment, creates a Service and Ingress, and even sets up monitoring dashboards. The platform team handles all the complexity; the development team gets exactly what they need.

Namespaces by Default

We've moved away from the cluster-scoped-by-default model that often confused new users. In Crossplane 2.0, both composite resources and managed resources are namespaced by default. This aligns with Kubernetes conventions and makes multi-tenancy much more intuitive.

The old claim/XR duality is gone. Instead, you simply create resources in the namespace where they belong:

apiVersion: example.crossplane.io/v1
kind: Database
metadata:
  namespace: frontend-team
  name: user-db
spec:
  engine: postgres
  storage: 100Gi

For the rare cases where you need cluster-scoped resources like shared networking infrastructure, you can still create them, but now it's an explicit choice rather than the default.

Day Two Operations, Declaratively

One of the most exciting additions is the new Operation type. After years of watching teams struggle with operational tasks like backups, upgrades, and maintenance windows, we realized these shouldn't require custom controllers or external cron jobs.

Operations bring the same declarative approach you know from Crossplane to operational workflows:

apiVersion: ops.crossplane.io/v1alpha1
kind: CronOperation
metadata:
  name: weekly-db-maintenance
spec:
  schedule: "0 2 * * 0"  # Sundays at 2 AM
  operationTemplate:
    spec:
      pipeline:
      - step: upgrade
        functionRef:
          name: function-database-upgrade
        input:
          apiVersion: fn.crossplane.io/v1beta1
          kind: UpgradeInput
          maxDowntime: 5m

We've also introduced WatchOperation for event-driven automation. Want to automatically trigger a backup when someone deploys to production? Or run maintenance tasks when specific resource conditions change? It's all declarative now.

Install Only What You Need

Installing an AWS provider used to mean getting every single AWS resource type, often 100+ CRDs hitting your API server at once. We knew this was a problem, but finding the right solution proved more complex than it initially appeared. Every approach we considered had significant trade-offs.

Crossplane 2.0 introduces ManagedResourceDefinitions (MRDs) and activation policies. We think they're an elegant solution to this longstanding challenge:

apiVersion: apiextensions.crossplane.io/v1alpha1
kind: ManagedResourceActivationPolicy
metadata:
  name: aws-essentials
spec:
  activate:
  - instances.rds.m.aws.crossplane.io
  - buckets.s3.m.aws.crossplane.io
  - "*.ec2.m.aws.crossplane.io"

This not only reduces API server load but also makes it clear to your team exactly which cloud resources are available in your platform.

The Journey Here

Building Crossplane 2.0 required some difficult decisions. We deprecated features that seemed important at the time but added unnecessary complexity in practice. We redesigned core APIs based on thousands of hours of community feedback. Most importantly, we focused on making Crossplane feel more like native Kubernetes—because that's what platform teams consistently told us they wanted.

The engineering effort was substantial. We refactored core controllers, redesigned the composition engine to handle arbitrary Kubernetes resources, and built entirely new systems for operations and provider management. But more than the code, this release represents a shift in philosophy: Crossplane isn't just about infrastructure anymore. It's about giving platform teams the tools to build comprehensive developer experiences.

Backward Compatibility and Migration

We know many of you are running Crossplane in production with business-critical workloads. Crossplane 2.0 maintains backward compatibility with the vast majority of v1.x configurations. Your existing compositions, providers, and composite resources will continue to work.

For new projects, we recommend starting with v2 patterns immediately. For existing workloads, you can migrate gradually—there's no rush. The v1 APIs aren't going anywhere, and we'll continue supporting them as the community needs.

What's Next

Crossplane 2.0 is just the beginning. We're already working on enhanced observability features, improved composition debugging tools, and even tighter integration with the broader cloud-native ecosystem. The community roadmap includes exciting work on composition testing frameworks and advanced policy integration.

But perhaps most importantly, we're continuing to listen. The best features in Crossplane 2.0 came directly from community feedback, production war stories, and late-night Slack conversations about what wasn't working. Keep that feedback coming.

Try It Today

Crossplane 2.0 is available now. You can get started with our updated quick start guide, and the full feature documentation is live at docs.crossplane.io.

We're curious to see how you use these new capabilities. After seven years of working on this project, there's nothing more satisfying than watching the community take our tools and create things we never imagined. Whether you're building internal platforms at a Fortune 500 company or experimenting with control planes in your homelab, Crossplane 2.0 enables the kind of full-stack self-service APIs that platform teams have been asking for.

Join us in the Crossplane Slack community, star us on GitHub, and tell us what works and what doesn't. There's more work ahead.

We are excited to announce the first release of Function Python, a powerful new tool that enables you to write Crossplane compositions using Python. This release opens up new possibilities for platform engineers and developers who prefer Python's simplicity and readability when configuring Crossplane.

Function Python was developed by Upbound and has now been contributed to the Crossplane project. This addition strengthens the growing set of tools available for defining compositions with Crossplane.

What Are Crossplane Composition Functions?

In Crossplane, composition functions are custom programs that template Crossplane resources. They allow you to define how composite resources (XRs) should be composed from other Kubernetes resources. When you create a composite resource, Crossplane invokes these functions to determine the resources to create, update, or delete. This approach provides flexibility and enables advanced logic, such as loops and conditionals, in your infrastructure definitions.

Why Use Python for Composition Functions?

Python is renowned for its ease of use and extensive ecosystem. By writing composition functions in Python, you can leverage:

• Readability: Python's clean syntax makes your composition logic easy to understand and maintain.
• Rich Libraries: Access to a vast array of libraries and frameworks to enhance your compositions.
• Rapid Development: Quickly prototype and iterate on your infrastructure definitions.

With Function Python, you can now harness these benefits within the Crossplane ecosystem.

Getting Started with Function Python

To start using Function Python in your Crossplane compositions:

Install the Function: Apply the following manifest to install Function Python:

apiVersion: pkg.crossplane.io/v1
kind: Function
metadata:
  name: function-python
spec:
  package: xpkg.crossplane.io/crossplane-contrib/function-python:v0.1.0

This will deploy the Function Python package into your Crossplane environment.

Define Your Composition: Create a Composition that references the function-python function. Here's an example snippet:

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: example-composition
spec:
  compositeTypeRef:
    apiVersion: example.org/v1alpha1
    kind: XExample
  mode: Pipeline
  pipeline:
    - step: python
      functionRef:
        name: function-python
      input:
        apiVersion: fn.crossplane.io/v1alpha1
        kind: Script
        source: |
          from crossplane.function import v1alpha1 as fn
          def compose(request: fn.RunFunctionRequest) -> fn.RunFunctionResponse:
            rsp.desired.resources["bucket"].resource.update({
                "apiVersion": "s3.aws.upbound.io/v1beta2",
                "kind": "Bucket",
                "spec": {
                    "forProvider": {
                        "region": req.observed.composite.resource["spec"]["region"]
                    }
                },
            })
            rsp.desired.resources["bucket"].ready = True

Your script has access to function-sdk-python. For example you can import crossplane.function.resource. It also has access to the full Python standard library - use it with care.

The RunFunctionRequest and RunFunctionResponse types provided by the SDK are generated from a Protocol Buffers schema. Their fields behave similarly to built-in Python types like lists and dictionaries, but there are some differences. Read the generated code documentation to familiarize yourself with the the differences.

Create Composite Resources: With the composition in place, you can now create composite resources that utilize your Python-based composition logic.

For a comprehensive guide on using composition functions, refer to the Crossplane documentation.

Join the Community

Function Python is an open-source project under the Crossplane Contrib organization. We welcome contributions, feedback, and suggestions from the community. Visit the GitHub repository to report issues, submit pull requests, or explore the source code.

We look forward to seeing how the community leverages Function Python to build their control planes!

The Crossplane community has always thrived on collaboration, innovation, and openness. Today, we're excited to announce key updates aimed at fostering the growth of community-contributed Crossplane extensions (providers and functions) while reinforcing our commitment to vendor neutrality—just like all CNCF projects.

We recognize the importance of ensuring that all community-driven contributions remain accessible, well-maintained, and free from vendor lock-in. These updates provide clear expectations and a neutral foundation to support the entire Crossplane ecosystem.

What’s Changing?

Clear Expectations for Community-Contributed Extensions

We are introducing operational guidelines for maintaining extensions contributed under the github.com/crossplane-contrib organization. These policies ensure that community extensions are properly maintained, fostering long-term sustainability and reliability.

A Vendor-Neutral, Community-Driven Home for Extensions

To enhance accessibility and neutrality, all community-contributed extensions will now be published, at a minimum, to ghcr.io/crossplane-contrib. Project documentation will reference this location, providing a consistent and independent source that the entire community can trust—regardless of vendor affiliations.

By reinforcing vendor neutrality, we align with the broader CNCF ecosystem’s principles, ensuring that Crossplane remains an open, community-first project. These changes support our shared goal of making Crossplane the best choice for building cloud-native control planes while improving transparency, accessibility, and long-term adoption.

Your contributions and feedback fuel Crossplane’s success. We appreciate your continued support and look forward to building a strong, open, and vendor-neutral ecosystem together!

The Crossplane Ecosystem

Crossplane has a large and thriving ecosystem of providers and functions—collectively referred to as extensions. These extensions can be public or private and maintained by individuals, end-user organizations, or vendors.

A subset of these extensions has been donated to the Crossplane project and resides in the crossplane-contrib GitHub organization. With today's governance updates, these projects will follow the same requirements as the Crossplane core project.

The Goals of crossplane-contrib

From its inception, crossplane-contrib was created to provide a collaborative home for community-driven Crossplane extensions. With today’s changes, we are introducing a clear health criteria for these projects, aligned with the CNCF’s standards. These criteria include:

• Regular builds
• A vendor-neutral home for released OCI images

However, it is not a requirement for all Crossplane extensions to reside in crossplane-contrib. To ensure the ecosystem thrives, there must be opportunities for differentiation through commercial offerings and services around these extensions.

Regular Builds for Community Extensions

A key governance update requires that all extensions donated to github.com/crossplane-contrib provide regular builds, ensuring the community has reliable access to up-to-date extensions. This change will help maintain a curated and well-maintained catalog of community extensions.

Introducing xpkg.crossplane.io

All Crossplane packages—including Crossplane itself and community-contributed extensions—will be hosted on GitHub Container Registry (GHCR) at ghcr.io/crossplane-contrib and proxied via xpkg.crossplane.io.

Previously, much of the community content was served from xpkg.upbound.io. To ensure vendor neutrality, Crossplane and community content will now be published to xpkg.crossplane.io. Additionally, content can be published to other registries such as Upbound and Docker Hub.

API Groups for Extensions

Crossplane providers define managed resources using Kubernetes API conventions. Moving forward:

• All new extension projects in crossplane-contrib must use a crossplane.io API group.
• Existing projects that are later donated to Crossplane will not be required to change their API group, as this would introduce breaking changes for existing compositions.

Updates to Crossplane Documentation and Website

To maintain vendor neutrality, the Crossplane documentation and website will only reference community extensions. Commercial offerings related to Crossplane will be listed separately on a dedicated Enterprise page.

Implementing the New Policies

Over the coming weeks, we will work with community maintainers to bring their Crossplane extensions into compliance with these new policies. Additionally, project documentation will be updated to refer to images from GHCR.

Looking Ahead

We’re excited about the future of Crossplane and the continued growth of community contributions. We look forward to discussing these updates and other new features at this year’s KubeCon in London—be sure to stop by our booth!

The Crossplane community is at the heart of this project’s success. Whether you're a developer, user, or just interested in what we're building, we’d love to hear from you! Connect with us through the following channels:

• Crossplane Website
• GitHub
• Slack

Let’s continue building a strong, open, and vendor-neutral ecosystem together!

Since its launch in 2018, Crossplane has grown into a powerful tool for building Kubernetes-native control planes. Organizations around the world use it to manage cloud infrastructure through declarative APIs, bringing a familiar Kubernetes experience to platform engineering.

With Crossplane v2, we’re introducing a set of thoughtful refinements that make Crossplane more intuitive, flexible, and well-suited for managing applications—all while maintaining compatibility with existing workflows.

What’s Changing in Crossplane v2?

Crossplane v2 simplifies how composite resources (XRs) and managed resources (MRs) are structured:  

• Namespaces by default – Most composite resources (XRs) and all managed resources (MRs) are now namespaced.
• Broader composition capabilities – XRs can now compose any Kubernetes resource, not just MRs.
• Claims are removed – This reduces complexity and aligns Crossplane more closely with Kubernetes conventions.  

These changes make it easier to manage applications and infrastructure in a unified way, without the need for extra cluster-scoped components.

A More Intuitive Model

Many teams use Crossplane to define opinionated self-service APIs for cloud infrastructure, but applications often remain a separate concern. Crossplane v2 makes application composition a first-class use case.  

By moving to a namespaced model, applications and their dependencies can be managed together in a way that feels natural within Kubernetes. The result is a cleaner, more consistent architecture that works equally well for infrastructure and applications.

Backward Compatibility & Migration

While Crossplane v2 introduces new APIs, existing v1 APIs will remain supported. That means:  

• No breaking changes for most users
• Legacy features will continue to work, though they won’t be the default
• Adoption of v2 features is opt-in, allowing teams to migrate at their own pace

The goal is to make Crossplane more intuitive for new users, while ensuring that existing users can transition smoothly when they’re ready.

Get Involved

These changes are designed to make Crossplane more flexible and approachable while staying true to its core strengths. As development progresses, we’ll share more details and provide resources to help teams explore and adopt these improvements.  

We’ve created a draft proposal on GitHub. You can read it here. Please give us your feedback!

We look forward to continuing the conversation with the Crossplane community.

As someone passionate about open-source software, being a mentee in the LFX Mentorship Program, backed by the Linux Foundation, has been one of the most enriching experiences of my career. Through this program, I had the opportunity to work on the Crossplane project, a Kubernetes-based framework that empowers developers to manage and provision infrastructure through Kubernetes' API. This mentorship gave me hands-on experience and invaluable insights into contributing to large-scale, real-world projects.

Application Process

My application process to LFX Mentorships has a very long history to be honest. As far as I know I applied to the program 5 times (first one was to the Crossplane project) and got rejected from 4 of them. Each time I got rejected I learned things about different projects and tried to understand why I was not the selected mentee. These rejections made me learn the open source philosophy and created a mindset that always tries to be “useful” for the team while creating or improving a project.

While I may have been rejected many times, the only number that counts is how I eventually did get accepted. Even before mentorship projects were given to the LFX program by CNCF projects, I started looking for an issue that I thought would be useful, in my skill set, and achievable for a new contributor. This issue was Switch to the Renovate GitHub Action. I started on this issue and got it merged before the mentorship projects were given to the LFX. This greatly improved my chance of being selected to mentorship since I proved that I can be “useful” for the community and the project. I think this shows to the mentors that you are worth their time and they will be very happy to help you learn things and improve the project together.

Contributions to Crossplane

Here are some of the key PRs I worked on during my time in the program:

1. Run Renovate as a Github Action:
   My first PR was about transferring our Renovate tooling from a hosted environment to a self-hosted Github Action so that we would be able to run make generate after Renovate generates PRs that bump dependencies
2. Add unknown field check to beta validate
   This PR focused on adding a frequently requested feature to the validate command. This enables validate to inform developers about the typos and indentation errors at their schemas in addition to already existing validation logics, which have plagued Crossplane users for a very long time.
3. download/cache full graph of dependencies at beta validate
   This PR enabled validate to download all of the deep dependencies of configurations so that we have a closer experience to real world usage and have all resources available while doing validations.
4. Add configuration.meta/crossplane.yaml support for dependencies to beta validate
   This PR enabled validate to have support for meta files so that configuration writers can do validation before sending their configurations to the remote and move the validation experience even further to the left.
5. Use more global cache directory at validate
   This PR is transporting the cache directory of validate from a workspace centered approach to a global cache approach.
6. Download dependencies of providers and functions at beta validate
   This PR is actually very interesting. It started as a PR for adding support for downloading dependencies of providers and functions to a performance improvement PR which improved the speed of validate by 150x. Yes, you read that number correctly. I really enjoyed doing this.

Challenges and Learning

Like any new experience, there were challenges along the way. Familiarizing myself with such a large and complex codebase was hard at first. At times I was feeling like ‘I don't even understand how the things work how am I going to contribute??’ But I was eventually successful with guidance from my precious mentors Ezgi, Jared and others in the Crossplane community (especially Nic, Philippe and Jean), so much on both golang and Crossplane.

One of the biggest lessons I learned was the importance of clear communication. In a very short time I learned how to look at the software not just as a functional product but as a direct source of user experience, shaping how people feel and interact with technology. This realization greatly improved my perspective on software development, as I started to focus more on the user’s journey, ensuring every aspect of the software contributed positively to their overall experience.

The Importance of Mentorship in Open Source

Being a mentee in the LFX Mentorship program has been transformative. It has not only enhanced my technical skills but also introduced me to the collaborative and community-driven nature of open-source projects. I gained first-hand experience in contributing to a project that is widely used by organizations around the world. The mentorship also provided me with a network of passionate contributors from whom I learned a great deal.

Looking Ahead

As I conclude my time in the LFX Mentorship program, I feel prepared to continue contributing to Crossplane and other open-source projects. As a freshly graduated student from college, skills and knowledge I gained during this experience will no doubt influence my future contributions, and I’m excited to remain a part of the open-source community.

I encourage anyone interested in open-source development to apply for the LFX Mentorship program. Whether you are just starting or have prior experience, the mentorship will help you grow as a developer and contribute meaningfully to important projects.

We are excited to announce today that Crossplane v1.16.0 has been released and is now available for installation into your control planes. This latest release of Crossplane focused on maturing a number of key areas of functionality across the project, as Crossplane continues to become more capable, more reliable, and more performant for your production workloads. In this blog post, we will dive into the details for the highlights from all of these major investment areas.

But first, let’s talk about the inspiring progress the community has made in getting involved with the project and keeping Crossplane advancing on an amazing upward trajectory…

Community engagement is 📈🔥

While preparing the release notes for v1.16, one statistic stood out immediately that was quite informative on how folks are getting more directly involved in the project. The v1.16 release has more first time contributors to the project than any release before!

24 different community members decided they wanted to leave their mark on the project for the first time and roll up their sleeves to commit their first Crossplane code change ever in this release. That’s amazing, and is absolutely the way that this project will remain healthy and sustainable far into the future (in addition to our greatly appreciated returning contributors of course!). Thank you everyone that made their first (or second, or 100th) contribution in this release.

Additionally, we saw some community members step up in a big way and start to drive impressive design changes for core functional areas in Crossplane. While these will take some time to get to a design that the entire team feels is the right experience we can support long term, we are truly grateful to see the depth of the investments that @dalton-hill-0 and @TerjeLafton are taking on for the benefit of the entire project. Looking forward to seeing these contributions land in the next release friends!

Finally, we expect to see even more folks continue to get involved as we make deeper investments in contributor enablement in the near future. Contributing to Crossplane should be easy and with the right support in place, we’ll grow an even bigger set of awesome engineers contributing to a healthy project for years to come.

What’s going on with my Managed Resources?

While Crossplane has provided some internal metrics for a long time, these metrics originated from the Kubernetes controller-runtime and were generally applicable to all Kubernetes controllers as opposed to providing insight specific to Crossplane. Starting with v1.16.0, providers in the Crossplane ecosystem are now able to export fine grained data about the operations they are performing on Crossplane managed resources.

These metrics have been incorporated into the upjet framework that most of the community is using to generate new providers for Crossplane, and the providers for the major clouds have also been updated to include these metrics (for example provider-upjet-aws:v1.4.0).

Let’s look at a couple examples of brand new insight that you can now get in your control planes:

• How many resources is this control plane managing? How many of them are ready and synced?
  • crossplane_managed_resource_exists
  • crossplane_managed_resource_ready
  • crossplane_managed_resource_synced
• How long is it taking for each type of resource to be reconciled and to become ready for the first time?
  • crossplane_managed_resource_first_time_to_reconcile_seconds
  • crossplane_managed_resource_first_time_to_readiness_seconds
• How long is it taking for various managed resource types to be deleted?
  • crossplane_managed_resource_deletion_seconds
• How long is it taking to discover that a resource is out of sync and needs to be updated?
  • crossplane_managed_resource_drift_seconds

These new metrics will provide useful new information into the performance and health of your control planes, so make sure to try them out soon.

Clean up of resources in use gets much faster

The Usage API of Crossplane has been very useful for protecting resources from being deleted while they are still in use (depended on) by other resources. However, a frustrating aspect of this experience has been that the eventual clean up of protected resources can take a very long time. People tend to notice this mostly in ephemeral environments that are purposefully torn down and recreated on a regular basis. In some cases, the deletion of the protected resource would take 10 extra minutes, and in extreme cases even more.

The reason for this major slowdown is essentially the exponential back-off Kubernetes uses when retrying operations that fail. While the resource is still in use, the Usage webhook will block attempts to delete it (e.g. by the garbage collector), thus causing the wait time for the next retry to exponentially increase. By the time the resource can finally be cleaned up, it will likely be multiple minutes before Kubernetes will even attempt the deletion again.

This slowdown has been addressed in v1.16 by the addition of a replayDeletion field on the Usage object. When this field is set to true, the controller will replay the deletion attempt of the used object as soon as the Usage object is gone, which bypasses the long back-off waiting period.

The example below captures the dependency on an EKS cluster by a Helm release. Note that replayDeletion: true is set, so the deletion of the EKS cluster will be replayed immediately after the Usage is gone, which drastically speeds up the clean-up process.

apiVersion: apiextensions.crossplane.io/v1alpha1
kind: Usage
metadata:
  name: release-uses-cluster
spec:
  replayDeletion: true
  of:
    apiVersion: eks.upbound.io/v1beta1
    kind: Cluster
    resourceRef:
      name: my-cluster
  by:
    apiVersion: helm.crossplane.io/v1beta1
    kind: Release
    resourceRef:
      name: my-prometheus-chart

One Crossplane user even reported that the clean up of their scenario was now 56 minutes faster while testing this fix! Note that while this is a great improvement to the usability of the Usages API, it remains in Alpha for this v1.16 release and will continue to be matured in future releases.

Providing credentials to Composition Functions

Composition Functions have significantly enhanced the Crossplane platform builder’s experience with capabilities that were not previously possible before their maturity to beta in v1.14. An entire ecosystem of useful functions have bloomed and Crossplane adopters are building complex resource generation pipelines with logic to accomplish basically anything they can dream up.

Functions can even make network calls to external services to discover information relevant to their pipeline logic. However, a missing piece of this end to end flow has been the ability to supply the function with a set of credentials to use when making these remote API calls. The v1.16 release overcomes this limitation and now provides a secure way for a Composition Function to authenticate to external systems.

In the example snippet below, credentials to access an external service are stored in a Kubernetes secret and then passed to the new credentials field of a function pipeline step. The composition machinery will retrieve the credentials from this secret location and then securely pass them over gRPC with mutual TLS to the given composition function so it can then successfully make its external call.

pipeline:
- step: compose
  functionRef:
    name: function-needs-a-secret
  credentials:
  - name: creds-secret
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: ext-svc-creds

Developer Experience gets more 💜

Continuing the trend of the last handful of Crossplane releases, v1.16 also received numerous investments to improve the developer experience and make Crossplane easier to use. The crossplane CLI in particular received the most fixes, mostly in response to feedback from adopters utilizing its functionality to manage their control planes.

This particular functional area had a good amount of first time contributors, so we’re especially grateful for all the passion being shown by the community here. There’s a ton more opportunities to continue contributing in this area in future releases of Crossplane, so we recommend taking a look at the DevEx tracking epic and seeing if anything catches your eye, and then come chat with us more in #SIG-DevEx.

Next stop: v1.17

As we take our first steps towards the next Crossplane release of v1.17, we continue to invite your contributions in many forms: from giving a ⭐ star to the project on GitHub, reporting or commenting on issues, recounting your journey in our adoption stories, to offering feedback on design proposals and pull requests.

We love to hear from the community, as you make this project great. Whether you are a developer, user, or just interested in what we're up to, feel free to join us via one of the following methods:

• Crossplane website
• GitHub
• Slack
• LinkedIn
• Twitter
• YouTube
• Reddit
• Email
• Sign up for the community newsletter

Say hello to Level Up with Crossplane, presented by Upbound—a new event series for the community of engineers interested in learning more and getting hands-on with open source Crossplane control planes. This new series is inspired by Control Plane Day and Crossplane Day events you may have attended in the past!

The first Level Up with Crossplane is on May 7 at 4 PM CEST. Presented by Upbound, this two-hour online event is for early to intermediate Crossplane users to build expertise and connect directly with experts about using the project to build their own platforms.

🎟️Register for the free event.

Agenda

Each event will have its own unique agenda. For the first one on May 7, we’re focusing on Crossplane 1.15, getting in-depth with Crossplane compositions, diving into testing patterns, and an explainer on handling APIs and CRD versioning. Here’s the detailed agenda:

• 16:00 CEST: Viktor Farcic kicks off the event.
• 16:05 CEST: What’s new with Crossplane 1.15 with Philippe Scorsolini. He shares what's new with Crossplane 1.15. Released in February, this latest Crossplane release refined the command line interface (CLI) and introduced functionality that streamlines the development of cloud infrastructure within the control plane paradigm. Updates to composition functions, a new Python SDK, and other improvements will be shared in this brief overview with one of the project leaders. And, catch a little preview of Crossplane 1.16, which is coming soon.
• 16:20 CEST: A Hands-On Crossplane Tutorial – Implementing Crossplane Compositions with Functions. Philippe follows the talk with a hands-on approach to help you take your Crossplane skills to the next level. This tutorial shows implementing a composition with functions. Philippe will guide you through developing a Crossplane composition using new features in the Crossplane CLI and building a Composition Functions pipeline.
• 16:45 CEST: A Tutorial on Crossplane Testing Patterns with Yury Tsarev and Steven Borrelli. Steven will demonstrate how to accelerate Composition development using Crossplane’s render and validate CLI options. Yury will demonstrate uptest, an end-to-end testing framework, and discuss the road to uptest 1.0. Learn techniques and tools for testing Crossplane Compositions from developer desktop to CI integration.
• 17:10 CEST: How to Break Your APIs Without Breaking Your Users and How CRD Versioning Works with David “Murph” Murphy. This is perfect for intermediate and experienced platform engineers. Crossplane enables organizations to create a cloud native platform using Composite Resource Definitions (CRDs) and Compositions. This talk will go through the distinct versioning methods for these resources. Understand how to make non-breaking changes and handle breaking changes as your platform evolves.
• 17:35 CEST: What’s New with Upbound. Murph gives a brief update on what’s New with Upbound, whose product is powered by Crossplane.
• 17:50 CEST: Q&A. Speakers finalize answering any of your questions in a Q&A and Viktor wraps up the event.

Have an idea? Submit your talk for a future Level Up with Crossplane event

Are you a platform engineer using Crossplane to build something cool? Have you successfully implemented Crossplane for your company’s internal developer platform (IDP)? Submit your talk for an opportunity to speak at a future Level Up with Crossplane event!

Submit here by the end of the day on May 7 for consideration in the next event happening at the end of June in North America online. Outside of that deadline, other submissions are reviewed by the Crossplane community team on a rolling basis. We welcome new and seasoned speakers alike!

Register

Stay up to date on future Level Up with Crossplane events and sign up to attend the free Level Up with Crossplane online event happening on May 7 at 4 PM CEST.

🧑‍💻Be sure to attend live—as an appreciation for your time, 50 attendees will have the opportunity to also win limited-edition Crossplane socks. 🧦

What’s the latest with Crossplane, the cloud native control plane framework? Check out the latest talks from the community from the 2024 KubeCon + CloudNativeCon Europe held in Paris last month.

KubeCon Paris and Crossplane

1-Project Lightning Talk: How Crossplane is Accelerating Your Cloud Native Control Plane Journey

Crossplane experts Ezgi Demirel, senior distributed systems engineer at Upbound, and Lovro Sviben, senior distributed systems engineer at Upbound, share a five-minute update on Crossplane.

Abstract: In this talk, we will take a lightning tour through the brand new experience in Crossplane that focuses on helping platform engineers build their own cloud native control plane from start to finish. We've listened to all the feedback from the community about how it can be challenging to get started with Crossplane, so in recent releases we've invested heavily into improving the developer experience and making it quicker and easier to successfully create a cloud native control plane and have it running reliably in production.

We will start the tour by showing off the new init command, through which you can rapidly initialise a new Crossplane Function, Provider, or Configuration project based on common templates. Using post-init hooks to continue the journey, we will customise our Function to start capturing the unique requirements of our control plane. We'll then start the local development cycle using the render command to execute our Function and get early development feedback and rapidly arrive at our desired functionality. We'll further verify the correctness of our control plane's behavior with the new validate command, which will point out any ways that our generated resources may differ from their expected specification.

With our cloud native control plane verified in the local development loop, we deploy to a live production environment. But what if an elusive issue appears there that we can't quite pinpoint? This is where the new trace command shines to give you a tree-like graph of related resources in which you can easily identify and resolve the culprit! Finally, we will quickly show how you can observe the Crossplane health and resource usage using the new top command, further ensuring your production environment remains healthy and reliable.

2-Crossplane Intro and Deep Dive - The Cloud Native Control Plane Framework

Crossplane maintainers Jared Watts, founding engineer at Upbound, and Philippe Scorsolini, senior software engineer at Upbound, give a comprehensive overview of the Crossplane project, fit for all levels, from beginner to intermediate to advanced.  

Abstract: The maintainers of Crossplane, a CNCF Incubating project, will lead this session that will introduce the project to new attendees, as well as dive into the finer details of Crossplane’s functionality and roadmap. We will explain how Crossplane enables you to compose cloud infrastructure and services into your custom platform APIs, and how best to get started building a platform of your own. We will take a tour through the key features included in the latest releases, what problems and use cases they are solving, and how you can adopt them into your control planes. Finally, there will be an interactive opportunity to engage with the maintainers, ask questions, and influence the future of the project direction.

3-Crossplane Observability and Traceability for Effective Multi-Cloud Management

Katharina Sick, senior software engineer at Dynatrace and Viktor Farcic, developer advocate at Upbound, talk about building platforms, managing resources, and what happens after the first day. Think about cooking–gathering ingredients and assembling them to make something meaningful, and how the experience changes at a restaurant–just like we do in our industry as platform engineers. Crossplane is the way you create the menu, which you assemble in a recipe (composition), and provide to your developers to consume. Dive deeper to see an example cluster, taking it to day two operations, integrating observability into the platform, and more through the talk.

Abstract: Companies adopting cloud-native technologies increasingly use declarative definitions of cloud infrastructure. This shift offers excellent scalability, flexibility and agility opportunities, but also comes with challenges: What if resource creation takes longer than expected? What if resources fail to materialize at all? How to identify problems and potential performance optimizations? The answer lies in a familiar strategy: as in software development, we can address these challenges through instrumentation and leveraging observability data to uncover patterns, pinpoint root causes, and drive informed optimizations. Join to learn how to answer those questions using existing metrics, logs and distributed traces emitted by tools like Crossplane, a popular open source CNCF project. The presenters will walk you through their best practices for making declarative cloud infrastructure traceable and leverage this data to improve issue resolution and your cloud infrastructure's reliability.

4-Building a Large Scale Multi-Cloud Multi-Region SaaS Platform with Kubernetes Controllers

Sébastian Guillox, software engineer with Elastic, shares the redesign of the Elastic platform. He uses Crossplane, among other CNCF projects like ArgoCD, Helm, Backstage, cert-manager, and more, to achieve scale in a large-scale, multi-region, multi-platform on top of Kubernetes.

Abstract: Controllers in Kubernetes are implemented through a pattern that reconciles entities towards their desired state. This pattern can actually be extended to cover a large set of use cases, beyond the scope of Kubernetes, while still relying on Kubernetes libraries such as controller-runtime. Come learn how Elastic is building a large-scale multi-cloud multi-region platform on top of Kubernetes, and what design decisions it led to. In this presentation we’ll describe an architecture made of hundreds of Kubernetes clusters, and talk about the challenges we have faced along the way. For example, by going into some details about how some of our controllers don’t actually work with etcd and the apiserver as their backend. They use a different datastore, and reconcile resources in different clusters than the one they are deployed into. We’ll also describe how some CNCF tools help achieve that scale (ArgoCD, Crossplane, Helm, Backstage, cert-manager, etc.).

5-Building AI-Ready Platforms - Symphony for Developer and Platform Engineer

Thomas Vitale, software engineer at Systematic and Lize Raes of LangChain4j share how to extend cloud native platforms to support AI use cases with Backstage, Buildpacks, and Crossplane.

Abstract: You’ve just built a platform, but now AI arrives and brings new requirements for applications wielding the power of large language models. Are you ready to support them? This session bridges the gap between platform engineers and developers, focusing on adapting your platform for AI while providing a smooth developer experience. You’ll learn to:

• Assess the capabilities of polyglot LLM-powered cloud native apps, including prompt engineering, LangChain4j AI Services, and integration of external knowledge.
• Build AI golden paths to streamline integrations and provisioning of models and embedding databases using Backstage, Buildpacks, and Crossplane.
• Implement production-grade capabilities for scaling and LLM observability using Knative and OpenTelemetry.

Join Lize and Thomas in building an AI-ready platform and implementing a composer AI assistant. In a final twist, you’ll choose which movie scene to score, and they’ll compose the music live for it, supported by AI.

6-Building Confidence in Kubernetes Controllers: Lessons Learned from Using E2E-Framework

Matteo Ruina, senior software engineer at Datadog, and Philippe Scorsolini, senior software engineer at Upbound, discuss using the e2e framework and how Datadog and Crossplane adopted it to test their own clusters and controllers.

Abstract: When working with cluster components, or while developing new controllers, it is often necessary to run end-to-end (E2E) tests against real clusters. This allows developers to have confidence in the changes being made, allowing for frequent updates without compromising on reliability. Although Kubernetes has its own E2E framework, it is baked into the Kubernetes source code, making it difficult to consume from other projects. For this reason, the Testing Special Interest Group built the e2e-framework, an out-of-tree Go framework to define end-to-end tests suites that can be used to test Kubernetes components. In this session we will talk about the e2e-framework and will explain how Datadog and Crossplane adopted it to test their own clusters and controllers. We will cover why we chose this framework, how to start writing your own tests, the approach we took, as well as the drawbacks you need to be aware of.

7-Object Storage on Kubernetes? Completed with Provider Ceph

Richárd Kovács, staff software engineer at Akamai, and Conor Nolan, senior software engineer at Akamai discuss their new addition to the Crossplane ecosystem: Provider Ceph. Watch the talk and read more in Conor’s blog post: Object Storage on Kubernetes? Completed with Crossplane Provider Ceph.

Abstract: The challenge of managing multiple Ceph clusters dispersed across various regions from within a single Kubernetes cluster is not a trivial one. Unsurprisingly, it sparked a journey of discovery and innovation. Faced with a unique problem and no existing solution, we embarked on building our own, opting to keep it open source with the hope of benefiting others in similar situations. Enter Crossplane, the framework that proved to be the ideal canvas for our solution - Provider Ceph. In this session, we'll delve into the intricacies of crafting Provider Ceph, a dynamic open source component that sits at the heart of a distributed object storage system. Learn how we leveraged Crossplane's capabilities to give life to a robust S3 bucket manager, seamlessly scaling to asynchronously orchestrate hundreds of thousands of buckets across multiple Ceph clusters.

8-Why Kubernetes is Inappropriate for Platforms, and How to Make It Better

Stefan Schimanski, senior principal engineer, Upbound, Sebastian Scheele, Kubermatic CEO, and MJ, staff engineer, Cast AI discuss the necessary shift in mindset in Kubernetes for platforms, all without reinventing the wheel.

Abstract: The ecosystem is building platforms on Kubernetes now, starting with a hub cluster and then sticking tools for Gitops, for application descriptions and for infrastructure management together, with the goal to create custom APIs for the platform consumers. This works, but hits limits of Kube as a framework quickly. Can we do better? Oh yes, we can! This talk is about extending Kube, adapting its architecture to be a better fit for a world where instead of container orchestration two new personas are at the center: (a) the service & API provider (b) the self-service consumer, often developers or application owners. We focus on 3 dimensions to enable Kube to serve platform engineering better:

• from kcp we take the workspace hierarchy as a vastly better multi-tenancy primitive.
• cross-workspace API exports and bindings tailor-made for the service provider and consumer personas.
• cluster mounting that integrates Kube clusters for a unified user interface and identity management.

But wait, there’s more!

Catch these three talks at the March 19 co-located events.

Platform Engineering Day, App Developer Con and ArgoCon

Crossplane - The Best Kept Secret in Platform Engineering

At Platform Engineering Day, Bassam Tabbara, creator of Crossplane and CEO of Upbound, shares insights about arguably the largest success story in platform engineering, starting in 2002. He connects this story to 22 years later, paralleling the current opportunity to build platforms with Kubernetes and Crossplane, which can give you the same power as the company in this famous success story.

Abstract: So your company has decided to invest in an internal developer platform? This multi-year initiative is likely to bring new levels of developer productivity and decrease overall costs. What recipe should you follow for your platform? In this talk we will share how Crossplane plays a critical role at the heart of every internal developer platform. It’s the best-kept secret within platform engineering teams. It’s no surprise given the world’s largest platforms are built on top of control planes. Come hear how a bank and an athletic company are building their internal platforms to get a glimpse of Crossplane in action. We’ll highlight the benefits of using Crossplane to create self-service, scalable, and efficient platforms that abstract away the complexities of the underlying infrastructure, allowing developers to focus on what they do best: building applications.

Unleashing the Power of Serverless on Kubernetes with Knative, Crossplane, Dapr, KEDA, and Friends

At App Developer Con, Viktor Farcic, developer advocate at Upbound, talks about managing serverless workloads.

Abstract: In the ever-evolving landscape of cloud native computing, serverless computing has emerged as a transformative force, offering a paradigm shift in application development and deployment. This talk delves into the world of serverless computing within the Kubernetes ecosystem, showcasing how open-source projects like Knative, Crossplane, Dapr, KEDA, Shipwright, and others empower developers to harness the power of serverless on top of Kubernetes. This is a hands-on talk, where we will build a serverless application (including infrastructure) from scratch. We will then explore how to extend the application with additional serverless capabilities, such as eventing, messaging, and state management. We will also explore how to leverage the power of Kubernetes operators to extend the Kubernetes API with custom resources and controllers, and how to use these operators to provision and manage serverless applications.

Platform Engineering with the Argo Ecosystem: The Elastic Story

Alejandro Martínez, tech lead - platform tooling, Elastic, Stephen Schmidt, platform tooling - SRE, Elastic, talk about the adventure they had while using a variety of tools in the cloud native ecosystem including Argo and Crossplane.

Cloud Native Rejekts

Get more knowledge with these talks from the Cloud Native Rejekts conference talks from Paris, too.

Crossplane Composition Functions Step by Step

Stefano Borrelli, principal solutions architect at Upbound, share insights and the latest on Composition Functions.

Abstract: Crossplane Composition functions are one of the most consequential new features in the project's history. Crossplane is a CNCF project that allows users to manage anything (like S3 buckets of Gitlab repositories) in a Kubernetes-native manner.

Composition Functions run in a pipeline to create a desired state using any programming language, allowing platform engineers to easily create and manage complex infrastructure deployments. Since graduating to beta in November 2023 functions have been rapidly adopted by the Crossplane community.

This talk will help engineers get up to speed with Composition Functions. We'll go over their design and runtime concerns of Functions in a K8s cluster. We'll then cover how to use functions to build out cloud infrastructure using field patching and Go templates.

Finally we'll discuss the basics of writing distributing your own functions in languages like Python and Go.

Too Many CRDs? I Say Not Enough! Leveraging Crossplane & Cluster API for Effective Platform Delivery

Carlos Mestre del Pino, KCD NL Organizer and platform engineering consultant at ITQ, shares an example, of leveraging Crossplane.

Abstract: Kubernetes is a platform for building platforms, but for some organizations, embracing its distributed and granular nature can easily lead to fragmented platform mayhem.

As platforms engineers tasked to automate the provisioning of infrastructure and/or services, (through IaaC, templating or custom code in pipelines) it sometimes might feel like we are just gluing components together.

Enter Crossplane, which enables you to bake your abstractions within the Kubernetes API without the need for building custom operators. Define the blueprints you need for provisioning infrastructure and the ones your developers need to deploy their services, reducing their cognitive load.

In this talk, I will discuss how you can leverage Crossplane and ClusterAPI for effective platform delivery, while minimizing the coding effort of your platform team and your developer's exposure to the Kubernetes' nitty gritty.

Are you one of those people who think there are too many CRDs? Let me challenge that!

The Magic of Backing Service Provisioning and Consumption With Crossplane and ServiceBindings

Time Salm, senior lead developer experience solutions engineer, EMEA, for VMware Tanzu discusses Crossplane’s capabilities to automate the provisioning of backing services and expose simple, abstracted interfaces for developers to claim those.

Abstract: As the amount of cloud-native applications continues to rise, there is an unprecedented surge in the demand for backing services, amplifying the need for automated provisioning.

We will embark on a journey to demystify the complexity of managing and connecting backing services.

Crossplane, a multi-cloud control plane, will take center stage as we explore its capabilities to automate the provisioning of backing services and expose simple, abstracted interfaces for developers to claim those.

The second component in this magical symphony are ServiceBindings that enable the automated supply of the required information for the connections between applications and their required backing services in a consistent way.

This session is your compass to navigate the enchantment of effortless backing service provisioning and consumption and will illustrate how this dynamic duo enhances developer productivity, reduces operational overhead, and facilitates the seamless deployment of applications.

Have a talk idea?

Don’t forget the KubeCon North America call for speakers is open through June 9, 2024 so get your talks in!

You can also submit your Crossplane-y talks to Level Up with Crossplane, presented by Upbound, an ongoing event series with technical, hands-on talks for platform engineers, DevOps, SREs, and more to get more expertise with Crossplane.

Submit your Level Up With Crossplane talk here.