Post-restore hook for a pod that is managed by a deployment resource is not executed during restore

[auhl-ibm]

Hi,

I have defined a post-restore hook for a pod that is managed by a deployment resource. The OCP plugin is hitting the restore plugin for pods and finding it has an owner reference, then skips the restore of the pod to allow the deployment to recreate the pod automatically to avoid duplicate pods.
 

When looking in the Velero restore log I can see that the restore of the pod was skipped:

time="2021-05-07T15:46:45Z" level=info msg="[pod-restore] skipping restore of pod my-mongodb-58dddc5999-p8zzl, has owner references and no restic backup" cmd=/plugins/velero-plugins logSource="/go/src/github.com/konveyor/openshift-velero-plugin/velero-plugins/pod/restore.go:51" pluginName=velero-plugins restore=velero/restore-k8s-53aadfc4-842e-49be-abf9-86e81e8f0c6f
time="2021-05-07T15:46:45Z" level=info msg="Skipping restore of Pod: my-mongodb-58dddc5999-p8zzl because a registered plugin discarded it" logSource="pkg/restore/restore.go:1074" restore=velero/restore-k8s-53aadfc4-842e-49be-abf9-86e81e8f0c6f

While this is probably expected behavior of the current OCP plugin to avoid duplicate pods, it causes the post-restore hook concept of Velero to be unusable in OCP environments where most of the pods have owner references. So I believe this is a bug.

I can upload more logs and more information if needed.

Thanks

Andreas

[shawn-hurley]

I would like to understand what types of changes you would like to make when a single pod starts running for a given deployment that you can not make when a deployment is restored?

[auhl-ibm]

Lets assume we have 2 pods in the deployment. 1 with a database and the other with an application connecting to the database. Then it would make sense during a restore operation to first restore the database pod, bring it up and recover the database or just bring the database out of suspend state using the post restore hooks before the deployment starts the rest of the application.
During the backup operation of this deployment we for example suspended the database using the pre backup hook and during the restore operation we then need to make sure that the database gets accessible again.
Could such a resume or recover step of the database happen after the deployment was restored and all pods are started? I believe yes, but the Velero post restore hooks are executed on a pod level when the pod gets restored. If a pod is not restored by velero at all, then the post restore hooks are not executed at all.

[kdrodger]

Hi @auhl-ibm . The teams I'm working with have recently uncovered the same problem, also a serious issue for us. @shawn-hurley , I agree with Andreas' comments. While I can partially understand the argument why OADP should behave this way:

• the current behaviour breaks the API promise that Velero hooks will be triggered, where I also work with applications and services dependent on that
• and, what I'm hearing is that there seems to be an asymmetry within OADP recovery today where when a Restic backup is restored the expected post-restore hooks are triggered but during a CSI recovery the hooks are skipped

[shawn-hurley]

Hello,

@kdrodger lets file issues for the difference in behavior for CSI and restic in a different issue.

@auhl-ibm I am confused about the "two pods for a single deployment" statement. AFAIK the deployment spec has only a single template for a single pod. Do you have multiple containers in a single pod?

[auhl-ibm]

Hi @shawn-hurley, you are right. It was confusing. My sample should have been 2 pods in 2 deployments that together make up an application. 

But for the post-restore hook problem, that I'm facing, its enough to just look on one pod in a single deployment. And lets assume we have a single container in the pod.

I hope this makes sense now.

It's really about what Kelly also stated, that the Velero API promises to execute the post restore hooks but it does not when a pod has an owner reference.

[shawn-hurley]

How does the pod/application handle restarts in kubernetes?

[auhl-ibm]

Well...that depends on the pod/application. For applications like MongoDB or other database applications we should assume that they can gracefully handle restarts with a database crash recovery. Other application may just rely on file system crash recovery. 
But for restore operations we should assume that there are applications that cannot handle restores without post-restore hooks.

[shawn-hurley]

Can you please help me understand what types of operations those are? I personally don't see what the post-restore hook is doing here and that is what is confusing me.

If the pod is only a pod, this makes sense as once k8s stops it, it will be stopped. But something that is managed by the deployment will have to handle the exact same situation and I think that is where I am lost.

[kdrodger]

@shawn-hurley as I understand it the design of the OADP/Velero framework is that there can be hooks called post-restore, and application pods are free to use and take advantage of those however they like. That should be true across all storage classes, Restic or CSI, and whether the pods are under the umbrella of an operator or not.

For instance, during an enterprise database backup, a facility such as fsfreeze is insufficient across all storage classes and storage configurations and so DBMS vendors might implement some other kind of suspend-IO or export-to-PVC capability in a backup pre-hook. During a routine pod restart the DBMS could trigger crash recovery but during a restore, operations such as import-from-PVC, or resume-IO and roll-forward may be more appropriate and provide symmetry with the pre-hook.

[sseago]

• the current behaviour breaks the API promise that Velero hooks will be triggered, where I also work with applications and services dependent on that
• and, what I'm hearing is that there seems to be an asymmetry within OADP recovery today where when a Restic backup is restored the expected post-restore hooks are triggered but during a CSI recovery the hooks are skipped

First, regarding the fact that hooks run when a restic backup is restored, in looking into that, I've discovered that the reason for this is that the pod plugin currently makes an exception for the "don't restore pods with owner references" when there's a restic backup annotation on the pod:

	// Check if pod has owner Refs
	if len(ownerRefs) > 0 && pod.Annotations[common.ResticBackupAnnotation] == "" {
		p.Log.Infof("[pod-restore] skipping restore of pod %s, has owner references and no restic backup", pod.Name)
		return velero.NewRestoreItemActionExecuteOutput(input.Item).WithoutRestore(), nil
	}

Second, I have a couple questions about your particular use case (both @auhl-ibm and @kdrodger ):

1. Do you configure your restore hooks using pod annotations or the velero restore spec fields?
2. Do you only do this with deployments, or do you use restore hooks for DeploymentConfigs, DaemonSets, etc?

Regarding the first question, one approach I could take here is to add the presence of restore hook annotatiions to the if condition that governs whether we restore the pod. That would handle cases where you configure hooks by annotation, but it will not handle cases where you use the restore spec.

To handle the restore spec case, we'll further need to look at the Restore resource itself. We could simply restore all owned pods if there are any restore hooks defined on the restore, or we could parse the hooks and determine whether the current pod being restored matches and just make the exception where it does.

Regarding the second question, there's an issue with DaemonSet pods. Without this "skip the pod" logic for DaemonSets, the pods get duplicated post-restore -- we end up with two per node. I'd rather avoid that problem entirely and not enable this for DaemonSets, if there is no current use case that requires them here. Finally, regarding DeploymentConfigs, the issue there is somewhat different. If we restore the DC pod, we won't have a duplicated pod. However, in many cases, the restored DeploymentConfig in the target cluster will trigger a new rollout, resulting in the restored pod being terminated shortly after creation, and a new one rolled out. This puts the restore hooks in an unpredictable state. The hooks may complete before this happens, or the pod may be killed before or during hook execution. We would either need to exclude DCs along with DaemonSets, or include them but document that hook execution isn't guaranteed.

If you have specific needs for restore hooks to work for either DaemonSets or DeploymentCOnfigs, we may have to consider alternatives.

[kdrodger]

Hi @sseago . I'm representing a large set of applications, 40+ at the current time, so some of the specifics might require a significant amount of digging to be completely accurate about. Having said that...

Do you configure your restore hooks using pod annotations or the velero restore spec fields?

I'm not aware of any use of the Velero CR spec fields to accomplish this, and some of our applications definitely do use Velero pod annotations.

Do you only do this with deployments, or do you use restore hooks for DeploymentConfigs, DaemonSets, etc?

This one will require a bit of digging. I've asked a couple of colleagues but off the top of my head I'm not aware of any use of DaemonSets or DeploymentConfigs where we've detected this issue, so that restriction on the final solution may be sufficient. I'll respond back here if I learn anything further.

[shawn-hurley]

The hooks may complete before this happens, or the pod may be killed before or during hook execution

Just to be clear, there is nothing stopping this from happening in pure Kube. I would be worried that if the hook was doing something extra critical with data that that this might be a problem. If it is resolvable then maybe this is a good interim solution.

[sseago]

The hooks may complete before this happens, or the pod may be killed before or during hook execution

Just to be clear, there is nothing stopping this from happening in pure Kube. I would be worried that if the hook was doing something extra critical with data that that this might be a problem. If it is resolvable then maybe this is a good interim solution.

Not only that, but this would also happen with pure upstream velero (without our plugins) with a restore hook on a DeploymentConfig. So if we go the route of restoring pods with hooks, we need to be aware of this possibility with DCs in particular -- it seems much less likely to happen with Deployments.