New base cluster

[beyang]

New base k8s cluster, bootstrapped from our existing non-privileged-create cluster.

Desired artifacts

1. a new k8s base cluster with minimal permissions requirements (no root users, no RBAC)
2. an easy, but flexible way to customize the cluster manifest to whatever the customer desires, while preserving a sane and tractable developer experience
3. a migration path from old world to new

We will keep the old legacy cluster, kustomize overlays, and Helm around for existing customers. All new customers should use the new base cluster with the new customization mechanism.

An intermediate milestone is probably to generate the existing (soon legacy) manifest using (2) from (1). That would be a good test of (2), as well. See notes section below for commentary on this.

To-do

• Generate existing cluster from new
• Create equivalent of existing kustomize overlays
• Check with Security to see what modifications we should make to Docker containers
• Odds and ends
  • Set fsGroupChangePolicy to OnRootMismatch (performance)

Notes / outstanding questions

There is the question of how to customize.

• Helm: templated YAML is brittle and yields a very poor dev experience
• Kustomize now supports components, which makes it more composable. There are still 2 issues:
  • You can't easily target multiple containers in a deployment (e.g., to add a securityContext). There is replacement: with create: true in theory, but it's a bit of a hack and I couldn't get it to work
  • You can't dynamically change an env var like SRC_GIT_SERVERS, which we need to do to eliminate the need for service discovery and therefore elevated permissions to create RBAC objects.

Overview of current layout, as of eb28273

• new/resources/sourcegraph - the proposed new base cluster. Still need to remove RBAC and convert some deployments (the ones referenced here) to statefulsets. Once RBAC is removed, we'll have to pass shard names through environment variables like SRC_GIT_SERVERS and our customization mechanism will need a way to do this automatically.
• new/resources/monitoring - moved all the monitoring stuff into a separate directory
• new/k8st - some customization code I copied over from the bl/better-customize branch. This isn't runnable yet, and we can remove this if we decide to move forward with Kustomize

Checklist

• CHANGELOG.md updated
• K8s Upgrade notes updated
• Sister deploy-sourcegraph-docker change:
• All images have a valid tag and SHA256 sum

Test plan