Skip to content

reaper: updating kubectl image, detection script, rbac - #5444

Merged
jschumacher-wire merged 3 commits into
developfrom
fix/reaper-image-and-rbac
Aug 17, 2026
Merged

reaper: updating kubectl image, detection script, rbac#5444
jschumacher-wire merged 3 commits into
developfrom
fix/reaper-image-and-rbac

Conversation

@jschumacher-wire

Copy link
Copy Markdown
Contributor

https://wearezeta.atlassian.net/browse/WPB-27995

charts/reaper: replace unmaintained image, drop cluster-admin

Why

The chart pinned docker.io/bitnamilegacy/kubectl and bound its
ServiceAccount to cluster-admin via a fixed-name ClusterRoleBinding, giving the
pod read access to every Secret in the cluster.

Changes

Image

  • docker.io/bitnamilegacy/kubectl:1.32.4docker.io/alpine/kubectl:1.36.3
  • scripts/reaper.sh rewritten in POSIX sh so it runs under busybox ash
  • Image is now fully configurable: registry (may be ""), repository, tag,
    digest (wins over tag), pullPolicy, plus imagePullSecrets

RBAC

  • ClusterRoleBinding → cluster-admin replaced by a namespaced Role granting
    get, list, watch, delete on pods.
  • All names are release-scoped (<release>-reaper), so multiple reaper releases
    could coexist in one cluster.

Other

  • podSecurityContext values are now actually applied; the deployment never
    referenced them, so the container ran with cluster defaults
  • Container runs as uid/gid 65534 with a read-only root filesystem
  • Added resource requests/limits (BestEffort → Burstable), nodeSelector,
    tolerations, affinity, and a configmap checksum annotation
  • Poll interval configurable via checkIntervalSeconds, default 15s (was one
    pod LIST per second against the API server)
  • Logs now distinguish "could not reach the API" from "no matching pods" and
    include the underlying error; both previously printed
    Failed to list pods. Skipping this iteration...

Testing

Verified end to end on sde-dev-1 (galaxy, k8s 1.34.4, 3 cannons + redis-ephemeral)
by installing the currently pinned chart 5.14.0-pre.6 and upgrading to this one.

  • Upgrade path — old ClusterRoleBinding and ServiceAccount pruned by Helm, new
    Role/RoleBinding/SA created, image swapped, securityContext and resources applied
  • RBAC boundary — effective grant is exactly pods: [get list watch delete];
    no on secrets in-namespace, secrets cross-namespace, cluster-wide pod list,
    cross-namespace deletes, creating ClusterRoleBindings, and * *
  • Functional reap — deleted the redis-ephemeral pod; all three cannons were
    deleted in sequence and recreated, then the reaper returned to steady state
  • Rollbackhelm rollback restores the old chart and its ClusterRoleBinding;
    rolling forward re-prunes cleanly
  • Multi-namespace — the old chart fails to install into a second namespace
    (ClusterRoleBinding ownership conflict); this chart installs into two namespaces
    simultaneously with zero cluster-scoped objects
  • Fresh install — works standalone, not only as an upgrade
  • read-only rootfs — confirmed against a real API server doing real discovery
  • Script exercised in the alpine/kubectl:1.36.3 image itself as uid 65534 with a
    read-only rootfs, across six scenarios (redis oldest, cannon oldest, no cannons,
    no redis, list failure, delete failure); shellcheck -s sh clean; helm lint and
    helm template clean on kube-version 1.23 and 1.34

Operator notes

  • Anyone overriding image must update the override. The image must contain a
    POSIX shell at /bin/sh, distroless kubectl images will not start.
  • Airgapped installs need the new image mirrored.
  • helm upgrade removes the old reaper-role ClusterRoleBinding and ServiceAccount.

Checklist

  • Add a new entry in an appropriate subdirectory of changelog.d
  • Read and follow the PR guidelines

@jschumacher-wire
jschumacher-wire requested review from a team as code owners August 13, 2026 13:13
Comment thread changelog.d/0-release-notes/reaper-image-and-rbac Outdated
@zebot zebot added the ok-to-test Approved for running tests in CI, overrides not-ok-to-test if both labels exist label Aug 14, 2026
@jschaul
jschaul dismissed smatting’s stale review August 17, 2026 10:16

Feedback has been addressed in latest commits.

@jschaul jschaul left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ok-to-test Approved for running tests in CI, overrides not-ok-to-test if both labels exist

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants