Problems with region endpoint URL

[hmuchaku]

We use oadp-oeprator with ROKS (managed Openshift on IBMCloud).
We failed to set-up oadp-operator with ICOS(S3 compatible IBM Cloud Object Storage)endpoint due to the following error,

• panic: invalid region provided: jp-tok

This is because jp-tok is an IBM region name that AWS does not have.
I know that we can bypass region check with env "REGISTRY_STORAGE_S3_REGIONENDPOINT",
but the value of s3url has been ignored.
veloro setup YAML:

apiVersion: konveyor.openshift.io/v1alpha1
kind: Velero
metadata:
  name: velero
  namespace: oadp-operator
spec:
  olm_managed: true
  backup_storage_locations:
  - config: 
      region: us-east-1
      s3ForcePathStyle: true
      s3Url: 'https://s3.private.jp-tok.cloud-object-storage.appdomain.cloud'
...

oadp-default-aws-registry env:

...
          env:
            - name: REGISTRY_STORAGE
              value: s3
            - name: REGISTRY_STORAGE_S3_ACCESSKEY
              value: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
            - name: REGISTRY_STORAGE_S3_BUCKET
              value: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
            - name: REGISTRY_STORAGE_S3_REGION
              value: jp-tok
            - name: REGISTRY_STORAGE_S3_SECRETKEY
              value: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
...

We can work around this by manually adding env REGISTRY_STORAGE_S3_REGIONENDPOINT to oadp-default-aws-registry deployment, but I don't think this is recommended since this deployment controlled by an operator.
How should I handle this?

[gmerigo]

I have the same problem with regular AWS region eu-south-1 (Milan), the aws registry exits with invalid region.

oc logs -f pod/oadp-default-aws-registry-668bc8d44c-kds6f -n oadp-operator 
time="2021-03-29T07:32:08.461219176Z" level=info msg="debug server listening :5001"
panic: invalid region provided: eu-south-1

Is there a way to bypass the error or is there a new image we could use to include valid aws regions?

[sseago]

@gmerigo So from the issue as reported, it looks like we need to add an optional REGISTRY_STORAGE_S3_REGIONENDPOINT environment variable to the registry deployment definition. I'm wondering whether an issue with unrecognized AWS-hosted regions is actually an unrelated problem. It may be that for that, the real issue is we need a newer version of the docker registry image that recognizes this region name.

[gmerigo]

@sseago I did just that, I was wondering if the operator would remove the variable when it reconciles the registry.

[sseago]

@hmuchaku So I just started looking at the code this morning. It looks like that env var should be set based on s3URL -- but perhaps there's a bug and something isn't coming across properly. Might just be a typo somewhere else in the templates. I'll have to try it out a bit later to see what happens in my environment

{% if bsl.config.s_3__url is defined %}
          - name: REGISTRY_STORAGE_S3_REGIONENDPOINT
            value: {{ bsl.config.s_3__url }}

https://github.com/konveyor/oadp-operator/blob/master/roles/velero/templates/registry_aws.yml.j2#L37-L39

[sseago]

@hmuchaku So in the OADP Velero CR, the BSL configuration parameters must be written in snake_case rather than camelCase. Taking your above example, the two s3 params should be changes as follows:

apiVersion: konveyor.openshift.io/v1alpha1
kind: Velero
metadata:
  name: velero
  namespace: oadp-operator
spec:
  olm_managed: true
  backup_storage_locations:
  - config: 
      region: us-east-1
      s3_force_path_style: true
      s3_url: 'https://s3.private.jp-tok.cloud-object-storage.appdomain.cloud'
...

To resolve this issue, I will add a note to the doc at https://github.com/konveyor/oadp-operator/blob/master/docs/bsl_and_vsl.md to indicate that this is necessary.

[sseago]

I just realized that was confusing the original poster (@hmuchaku ) and a later commenter (@gmerigo ) in my above comments. I'll edit them accordingly. My apologies for the reversal.
Edit: OK, the comments have been reflected to resolve my earlier confusion.

[sseago]

@gmerigo So your issue does seem to be that with the docker registry image, the list of regions is hard-coded (and missing newer ones). You should be able to get around this by specifying s3_url (rather than s3Url) in the OADP Velero CR as shown above. See distribution/distribution#2870 for confirmation that the underlying docker image we're using has this problem.

[sseago]

Copying a comment from the above docker distribution github issue:

distribution/distribution#2870 (comment)

In case someone needs it, the workaround is to bypass this check by explicitly specifying the endpoint in the environment variables (it can't be done in the config.yml itself but do it in the docker-compose.yml in case you use compose to run this for example):

        # Hack to workaround that eu-north-1 is not in the registry validation table yet, giving this
        # endpoint explicitly bypasses the check... (the eu-north-1 region in config.yml is validated)
        REGISTRY_STORAGE_S3_REGIONENDPOINT: "s3.eu-north-1.amazonaws.com"

That this works fine, leads me to believe that the check could just be dropped, it's not anything the S3 SDK itself cares about as long as it gets its endpoint (though, maybe AWS has a different URL endpoint format for some regions so you simply can't paste it together like this, a bit improbable though).

[sseago]

Doc fix for this issue posted at #87