add kubepod driver

[tonistiigi]

Similarly to docker-container driver there could be a support to kubepod driver that bootstraps itself inside a k8s cluster. Kube endpoints in docker context can be used for providing initial configuration. Switching between drivers implemented in #20 . Would be nice if all supported platforms of k8s workers could be also detected automatically.

@AkihiroSuda

[AkihiroSuda]

Bikeshedding: kubepod? kube-pod? k8s-pod? (I prefer kube-pod)

[tonistiigi]

kube-pod seems consistent with docker-container

[AkihiroSuda]

Maybe we should prioritize adding Deployment driver.

• docker buildx create --driver kube --driver-opt replicas=N will createDeployment with N replicas.
  We should not use StatefulSet because creating StatefulSet pod replicas is slow due to the pod creation order constraint.
• docker buildx rm will remove the Deployment
• docker buildx stop should be NOP? Or resize the Deployment to 0 replica?
• docker buildx build will connect to a Pod in Deployment using kubectl exec -it POD buildctl dial-stdio. An alternative way is to set up mTLS (which is mandatory to protect the daemon from the executor containers) and Service, but it is not straightforward to manage the lifecycle of the certificates.
• To choose a Pod in Deployment, we can implement both random and consistenthash. As we are not going to use StatefulSet, consistenthash will break on recreation of ReplicaSet (on rolling update of Deployment). Probably this is not a huge deal.
• docker buildx inspect will show the name of the Deployment.
• The user will be able to do rolling update of the buildkitd image using kubectl set image,
• The user will be able to change the number of the replicas using kubectl scale.

[tonistiigi]

@AkihiroSuda SGTM, except the k8s objects should probably be created on build/bootstrap, not on create.

[AkihiroSuda]

PR #167