Pod restore plugin doesn't take into account `defaultVolumesToRestic` flag

[gmerigo]

I'm trying the latest oadp version availlable in the Openshift Marketplace.
I'm able to backup kubernetes metadata (I tried to backup/restore a full namespace, including imagestreams) but I can't get restic restores to work, the test volume comes out empty, it seems like it's getting reprovisioned and never restored. Note that project and all other objects get correctly restored.

This happens both with OCS storageclasses and with gp2 as the backup/restore target volume.

---

The backup seems fine, I can see [...]/data/ populated in the S3 bucket and I can see the podvolumebackup if I issue the oc get podvolumebackups.velero.io -n oadp-operator command.

The backup has the defaultVolumesToRestic: true option so the volume backups should initiate even without annotating pods.

The Restore object is configured this way:

apiVersion: velero.io/v1
kind: Restore
metadata:
  namespace: oadp-operator
  name: restore-test-restic-all-4
spec:
  backupName: test-restic-all
  includedNamespaces:
    - test-restic
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  restorePVs: true

Note that in the logs there are no errors related to the pv or pvc restores (there's an error in the imagestream but in this case I don't care as the test environment uses a standard openshift image).

Maybe I'm missing some critical parameter in the restore request?

---

These are the versions involved:

Openshift 4.5.18 on AWS
Openshift Container Storage: ocs-operator.v4.5.0
oadp: oadp-operator.v0.2.0
velero: (default image) Version: v1.5.2-konveyor
Storage and Snapshot locations: AWS S3

Note that I can't currently update to a OCS version with snapshot support, so I'm currently unable to use CSI snapshots, so it would be nice to have restic backups and restores working, at least for small volumes.

[dymurray]

@gmerigo I apologize for the delayed response here. If you happen to still be using OADP, could you let me know if a podvolumerestore object was created? I am not able to reproduce this myself so I'd like to learn more about this environment.

[gmerigo]

Thank you Dylan for your answer, my lab has long gone, I can try to setup another test shortly but not immediately if needs be.

Further investigations points out that there are problems when you use defaultVolumesToRestic: true in the backups, it seems that annotating the pods with the explicit list of volumes as per Velero docs works (but it has to be checked further because I didn't do repeatable tests).

Unfortunately this is inconvenient as we need to rely on developers to not replace the deployment/deployment configuration/statefulset in which we set the annotation in the pod templates, and sometimes with some operators and helm charts it's difficult to know if there are options to set custom annotations.

[sseago]

@dymurray I think what's happening is that our pod restore plugin is getting in the way. We discard pods with owner references except when there are restic annotations. But there won't be any restic annotations in the "default to restic" case. When we put the fix in for restore hooks, we probably want to take the "default volumes to restic" situation into account as well (we'll still want to omit restoring DC and DaemonSets though, since DC pods don't work with restic restore right now anyway, and DaemonSet restore of owned pods breaks things post-restore -- but that will probably resolve this for Deployments and StatefulSets.

[dymurray]

+1 I think you hit the nail on the head Scott. That is likely biting us here. I will change the name of the issue to reflect this

[sseago]

@dymurray I noticed the potential problem when investigating the restore hooks issue, but I hadn't gotten around to submitting an issue for it (or looking at existing ones).

[gmerigo]

@dymurray [...] we'll still want to omit restoring DC and DaemonSets though, since DC pods don't work with restic restore right now anyway, and DaemonSet restore of owned pods breaks things post-restore -- but that will probably resolve this for Deployments and StatefulSets.

Can you confirm that currently DC restores do not work? My first use case is to restore a MySQL database and a Wso2 suite, both are DC's (MySQL is one DC, and the latter is four distinct dc with RWX volumes) so it would be pointless to recreate a test environment with those pods.

[sseago]

@gmerigo So DC velero restores themselves work fine. It's restic restores in particular that fail with DeploymentConfigs, even when the restic annotation is set. That's not an issue with our plugin, since when you set the restic annotation, the plugin does restore the pod. However, I confirmed myself when I was testing the OADP 0.2.3 release that restic restores with DCs weren't working I had to modify my test to use a Deployment. The same is reproduced in a pure upstream velero environment. I'll see if I can find the upstream issue where this was reported. But the issue seems to be the DeploymentConfig's post-restore trigger behavior, where the restored pod is shut down when the DC redeploys in response to the config change.

[sseago]

Here's the upstream issue where other (non-OADP, non-Openshift-plugin) users have also reported DC restic restores not working. It's something I'd like to fix upstream, but I haven't had the bandwidth (or the idea for a proper solution) yet.
velero-io/velero#1981

[kdrodger]

Hi @dymurray and @sseago, I just wanted to add another voice here, that we've been experiencing the same issue with defaultVolumesToRestic not working under OADP. We've had to incorporate some (hopefully) temporary work-arounds into our infrastructure and tooling.

I'm thinking now (sent you a Slack message yesterday you may not have seen yet) that this is likely related to another question we have, where the expected OADP default behaviour of selecting a CSI snapshot when available, otherwise defaulting to Restic, is also not behaving as expected.

We're finding we need to work-around it through some relatively expensive logic in our tooling. That's also leading us down a path of a user experience issue, where we need to expose on the front-end that a CSI or restic operation is being requested, and making it the users responsibility to ensure that PVC content actually gets backed up and restored instead of offering a default that reliably just works. Thanks