OADP does not allow install Velero with feature flag

[phuongatemc]

We want to use velero-plugin-for-vpshere https://github.com/vmware-tanzu/velero-plugin-for-vsphere but it requires additional feature flag (--features=EnableVSphereItemActionPlugin) to function properly. However, OADP won't allow setting any feature flag on the Web-UI when install Velero. This prevents us from using the vSphere plugin mentioned above.

[dymurray]

@phuongatemc FYI we dug intoo this and this seemed to be the case for an older version of the vsphere plugin. You should be able to use OADP 0.2.0 with the latest plugin and not need to set this feature flag.

please let me know if you run into any other issues.

[dymurray]

Also FYI, I don't believe the vsphere plugin supports openshift yet. I could be wrong.

[seema-emc]

@dymurray With the latest vSphere plugin, we need the following feature flag to make sure Vsphere plugin's snapshot manager comes in LocalMode.
- --features=EnableLocalMode

[shubham-pampattiwar]

@seema-emc @phuongatemc I have OADP operator changes ready with me that would enable you to set the value for the --features flag. I tried out the following things for enabling the velero plugin for vsphere via the OADP operator:

• Set the --features=EnableLocalMode I believe this is consumed by the velero plugin for vsphere code here
• Set up an init container of velero plugin for vsphere using the image
• Used upstream Velero image for Velero deployment

After doing the above steps I was able to get the following logs in the Velero pod created by the OADP operator (implying that the velero plugin for vsphere is getting registred):

shubham@localhost ~/oadp-operator -  (master) $ oc logs velero-6cfc5f7f7c-77vsq | grep vsphere
time="2021-03-11T07:50:29Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-vsphere kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/vsphere-pvc-backupper
time="2021-03-11T07:50:29Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-vsphere kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/vsphere
time="2021-03-11T07:50:29Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-vsphere kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/vsphere-pvc-restorer
time="2021-03-11T07:50:29Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-vsphere kind=DeleteItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/vsphere-pvc-deleter

Do the above solution steps and log output fulfill your use case ? Do you just want the ability to set the --features flag values to Velero Deployment Pod or you also want the velero plugin for sphere initContainer creation as well alongside the ability to set features flag ?
Let me know if this solution suffices your use case or If I am misinterpreting or missing anything.

[phuongatemc]

@shubham-pampattiwar Do I need a new/private OADP operator binary from somewhere or simply delete the OADP operator and the latest one with this fix will be used?

[shubham-pampattiwar]

@phuongatemc Once this fix is merged into master there are two ways you can use the latest operator:

1. Delete your existing operator and redeploy using the Non-OLM steps
   OR
2. As We will be doing a minor release of the operator (i.e. once the latest changes are merged in the master we update the operator in community operators), you can use the latest operator via OperatorHub present on OpenShift UI Console after cleaning up the older one.

Also, @phuongatemc Can you please confirm if the proposed solution is the one you wanted and satisfies your usecase ?

[phuongatemc]

The approach looks good to me. Thank you!

[phuongatemc]

I have tested the latest OADP Operator with the feature flag EnableLocalMode set in the velero yaml as below. However, the result Velero pod does not show the EnableLocalMode anywhere in its spec.
Here is my Velero yaml file I used in the step of "Create" Velero in OADP:
apiVersion: konveyor.openshift.io/v1alpha1
kind: Velero
metadata:
name: example-velero
namespace: oadp-operator
spec:
olm_managed: true
custom_velero_plugins:
- image: 'ap-sc.lss.emc.com:8446/dellemc/powerprotect-velero-dd:19.7.0-1-SNAPSHOT'
name: powerprotect-velero-dd
- image: ap-sc.lss.emc.com:8446/vsphereveleroplugin/velero-plugin-for-vsphere:1.1.0-rc4
name: velero-plugin-for-vsphere
default_velero_plugins:
- csi
- openshift
enable_restic: true
velero_feature_flags:
- EnableLocalMode

And here is the result Velero pod:

oc get pod/velero-5968c8f9f9-8jbv4 -n oadp-operator -o yaml

apiVersion: v1
kind: Pod
metadata:
annotations:
k8s.v1.cni.cncf.io/networks-status: |-
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.128.3.120"
],
"dns": {},
"default-route": [
"10.128.2.1"
]
}]
openshift.io/scc: restricted
prometheus.io/path: /metrics
prometheus.io/port: "8085"
prometheus.io/scrape: "true"
creationTimestamp: "2021-03-31T21:04:37Z"
generateName: velero-5968c8f9f9-
labels:
component: velero
pod-template-hash: 5968c8f9f9
name: velero-5968c8f9f9-8jbv4
namespace: oadp-operator
ownerReferences:

• apiVersion: apps/v1
  blockOwnerDeletion: true
  controller: true
  kind: ReplicaSet
  name: velero-5968c8f9f9
  uid: 04ebb456-16d0-4e7c-b685-1b49fe17f940
  resourceVersion: "71886479"
  selfLink: /api/v1/namespaces/oadp-operator/pods/velero-5968c8f9f9-8jbv4
  uid: e9b965f0-64e0-485b-8051-532043073481
  spec:
  containers:
• args:
  • server
  • --restic-timeout
  • 1h
  • --features
  • EnableCSI
    command:
  • /velero
    env:
  • name: LD_LIBRARY_PATH
    value: /plugins
  • name: VELERO_NAMESPACE
    value: oadp-operator
  • name: VELERO_SCRATCH_DIR
    value: /scratch
    image: quay.io/konveyor/velero:konveyor-1.5.2
    imagePullPolicy: Always
    name: velero
    ports:
  • containerPort: 8085
    name: metrics
    protocol: TCP
    resources:
    limits:
    cpu: "1"
    memory: 256Mi
    requests:
    cpu: 500m
    memory: 128Mi
    securityContext:
    capabilities:
    drop:
    • KILL
    • MKNOD
    • SETGID
    • SETUID
      runAsUser: 1000630000
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
  • mountPath: /plugins
    name: plugins
  • mountPath: /scratch
    name: scratch
  • mountPath: /etc/ssl/certs
    name: certs
  • mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    name: velero-token-7hlqf
    readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    imagePullSecrets:
• name: velero-dockercfg-l8snr
  initContainers:
• image: quay.io/konveyor/openshift-velero-plugin:release-1.4.0
  imagePullPolicy: Always
  name: openshift-velero-plugin
  resources: {}
  securityContext:
  capabilities:
  drop:
  - KILL
  - MKNOD
  - SETGID
  - SETUID
  runAsUser: 1000630000
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File
  volumeMounts:
  • mountPath: /target
    name: plugins
  • mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    name: velero-token-7hlqf
    readOnly: true
• image: docker.io/velero/velero-plugin-for-csi:main
  imagePullPolicy: Always
  name: velero-plugin-for-csi
  resources: {}
  securityContext:
  capabilities:
  drop:
  - KILL
  - MKNOD
  - SETGID
  - SETUID
  runAsUser: 1000630000
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File
  volumeMounts:
  • mountPath: /target
    name: plugins
  • mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    name: velero-token-7hlqf
    readOnly: true
• command:
  • sh
  • -ec
  • cp /etc/ssl/certs/* /certs/; ln -sf /credentials/ca_bundle.pem /certs/ca_bundle.pem;
    image: quay.io/konveyor/velero:konveyor-1.5.2
    imagePullPolicy: Always
    name: setup-certificate-secret
    resources: {}
    securityContext:
    capabilities:
    drop:
    • KILL
    • MKNOD
    • SETGID
    • SETUID
      runAsUser: 1000630000
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
  • mountPath: /certs
    name: certs
  • mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    name: velero-token-7hlqf
    readOnly: true
• image: ap-sc.lss.emc.com:8446/dellemc/powerprotect-velero-dd:19.7.0-1-SNAPSHOT
  imagePullPolicy: Always
  name: powerprotect-velero-dd
  resources: {}
  securityContext:
  capabilities:
  drop:
  - KILL
  - MKNOD
  - SETGID
  - SETUID
  runAsUser: 1000630000
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File
  volumeMounts:
  • mountPath: /target
    name: plugins
  • mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    name: velero-token-7hlqf
    readOnly: true
• image: ap-sc.lss.emc.com:8446/vsphereveleroplugin/velero-plugin-for-vsphere:1.1.0-rc4
  imagePullPolicy: Always
  name: velero-plugin-for-vsphere
  resources: {}
  securityContext:
  capabilities:
  drop:
  - KILL
  - MKNOD
  - SETGID
  - SETUID
  runAsUser: 1000630000
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File
  volumeMounts:
  • mountPath: /target
    name: plugins
  • mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    name: velero-token-7hlqf
    readOnly: true
    nodeName: worker2.ocpk8s1.ocpdell.com
    priority: 0
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext:
    fsGroup: 1000630000
    seLinuxOptions:
    level: s0:c25,c15
    serviceAccount: velero
    serviceAccountName: velero
    terminationGracePeriodSeconds: 30
    tolerations:
• effect: NoExecute
  key: node.kubernetes.io/not-ready
  operator: Exists
  tolerationSeconds: 300
• effect: NoExecute
  key: node.kubernetes.io/unreachable
  operator: Exists
  tolerationSeconds: 300
• effect: NoSchedule
  key: node.kubernetes.io/memory-pressure
  operator: Exists
  volumes:
• emptyDir: {}
  name: plugins
• emptyDir: {}
  name: scratch
• emptyDir: {}
  name: certs
• name: velero-token-7hlqf
  secret:
  defaultMode: 420
  secretName: velero-token-7hlqf
  status:
  conditions:
• lastProbeTime: null
  lastTransitionTime: "2021-03-31T21:05:01Z"
  message: 'containers with incomplete status: [velero-plugin-for-vsphere]'
  reason: ContainersNotInitialized
  status: "False"
  type: Initialized
• lastProbeTime: null
  lastTransitionTime: "2021-03-31T21:05:01Z"
  message: 'containers with unready status: [velero]'
  reason: ContainersNotReady
  status: "False"
  type: Ready
• lastProbeTime: null
  lastTransitionTime: "2021-03-31T21:05:01Z"
  message: 'containers with unready status: [velero]'
  reason: ContainersNotReady
  status: "False"
  type: ContainersReady
• lastProbeTime: null
  lastTransitionTime: "2021-03-31T21:04:37Z"
  status: "True"
  type: PodScheduled
  containerStatuses:
• image: quay.io/konveyor/velero:konveyor-1.5.2
  imageID: ""
  lastState: {}
  name: velero
  ready: false
  restartCount: 0
  started: false
  state:
  waiting:
  reason: PodInitializing
  hostIP: 10.235.12.188
  initContainerStatuses:
• containerID: cri-o://3482a0506ef97a72071fa88fb052f00be115cbf821db91f901b5fa16e9a03fde
  image: quay.io/konveyor/openshift-velero-plugin:release-1.4.0
  imageID: quay.io/konveyor/openshift-velero-plugin@sha256:5c24ca12fb7c63e6ca2720755d49b9042254e47d73241e5379f77800d8254ee4
  lastState: {}
  name: openshift-velero-plugin
  ready: true
  restartCount: 0
  state:
  terminated:
  containerID: cri-o://3482a0506ef97a72071fa88fb052f00be115cbf821db91f901b5fa16e9a03fde
  exitCode: 0
  finishedAt: "2021-03-31T21:06:01Z"
  reason: Completed
  startedAt: "2021-03-31T21:06:01Z"
• containerID: cri-o://fbf260ab2bb80a6ef9b4cb3979996f5657cf1c1ae93004d99747d4a47bec5976
  image: docker.io/velero/velero-plugin-for-csi:main
  imageID: docker.io/velero/velero-plugin-for-csi@sha256:138cafba29a77e331c1f8158219508ecdd8ff08613737115bf1551bab03eaa58
  lastState: {}
  name: velero-plugin-for-csi
  ready: true
  restartCount: 0
  state:
  terminated:
  containerID: cri-o://fbf260ab2bb80a6ef9b4cb3979996f5657cf1c1ae93004d99747d4a47bec5976
  exitCode: 0
  finishedAt: "2021-03-31T21:07:01Z"
  reason: Completed
  startedAt: "2021-03-31T21:07:01Z"
• containerID: cri-o://a9fe5f8eb59e3dfdd8b7ebed835201a00a22d2142459be6a7e7c3344bde86352
  image: quay.io/konveyor/velero:konveyor-1.5.2
  imageID: quay.io/konveyor/velero@sha256:c568920b7d6da28c3dc1965e4a4b0b6b90023eea4783b4d9095055c6ff529e5f
  lastState: {}
  name: setup-certificate-secret
  ready: true
  restartCount: 0
  state:
  terminated:
  containerID: cri-o://a9fe5f8eb59e3dfdd8b7ebed835201a00a22d2142459be6a7e7c3344bde86352
  exitCode: 0
  finishedAt: "2021-03-31T21:07:18Z"
  reason: Completed
  startedAt: "2021-03-31T21:07:18Z"
• containerID: cri-o://43b354ee6a6d61596620121d25085881dadb2dc99a96f01ed6d38a6e9052312a
  image: ap-sc.lss.emc.com:8446/dellemc/powerprotect-velero-dd:19.7.0-1-SNAPSHOT
  imageID: ap-sc.lss.emc.com:8446/dellemc/powerprotect-velero-dd@sha256:c39375f019e6e21bf2fec6dbd71b9c8fc17397d06b11a54dc3a33d4797cd962b
  lastState: {}
  name: powerprotect-velero-dd
  ready: true
  restartCount: 0
  state:
  terminated:
  containerID: cri-o://43b354ee6a6d61596620121d25085881dadb2dc99a96f01ed6d38a6e9052312a
  exitCode: 0
  finishedAt: "2021-03-31T21:07:36Z"
  reason: Completed
  startedAt: "2021-03-31T21:07:36Z"
• containerID: cri-o://1b6c6a18f0b602c700d9b05880aee9bbc8ace10a56ad23c10c332817cc427101
  image: ap-sc.lss.emc.com:8446/vsphereveleroplugin/velero-plugin-for-vsphere:1.1.0-rc4
  imageID: ap-sc.lss.emc.com:8446/vsphereveleroplugin/velero-plugin-for-vsphere@sha256:a3fa7eeaf0c04b08c56ebe7023f220a79422361fbfb3aa8c0e907977b80e16ce
  lastState:
  terminated:
  containerID: cri-o://1b6c6a18f0b602c700d9b05880aee9bbc8ace10a56ad23c10c332817cc427101
  exitCode: 2
  finishedAt: "2021-03-31T21:11:56Z"
  reason: Error
  startedAt: "2021-03-31T21:11:55Z"
  name: velero-plugin-for-vsphere
  ready: false
  restartCount: 5
  state:
  waiting:
  message: back-off 2m40s restarting failed container=velero-plugin-for-vsphere
  pod=velero-5968c8f9f9-8jbv4_oadp-operator(e9b965f0-64e0-485b-8051-532043073481)
  reason: CrashLoopBackOff
  phase: Pending
  podIP: 10.128.3.120
  podIPs:
• ip: 10.128.3.120
  qosClass: Burstable
  startTime: "2021-03-31T21:05:01Z"

[shubham-pampattiwar]

@phuongatemc Can you make sure that you are using OADP operator version v0.2.1 ? The operator pod's container image tag should be v0.2.1

[shubham-pampattiwar]

Also, @phuongatemc When using the parameter velero_feature_flags , please keep in mind that it is not a list of flags. So instead of using it as :

velero_feature_flags:
- EnableLocalMode

Use it as follows (correct usage, treat it is as comma separated string for multiple values):

velero_feature_flags: EnableLocalMode

Reference docs: https://github.com/konveyor/oadp-operator#usage-of-velero---features-option

[phuongatemc]

@phuongatemc Can you make sure that you are using OADP operator version v0.2.1 ? The operator pod's container image tag should be v0.2.1

Nope. I have v0.2.0. I just ran "Install" on a new OpenShift Cluster and it pulled in that version. Is there anyway I can specify it to pull v0.2.1?

[shubham-pampattiwar]

@phuongatemc you should be able to get the latest version from the OperatorHub itself. There are 2 things that you can try out:

1. Cleanup the existing instance of the operator using the steps specified here https://github.com/konveyor/oadp-operator#cleanup and try to install it again from the OperatorHub from OpenShift console, the operatorHub must have the latest version as 0.2.1
2. Or You can try out the Alternative install approach by following Non-OLM installation steps specified here: https://github.com/konveyor/oadp-operator#getting-started-with-basic-install-non---olm