[File system] Add HuaweiCloud OBS integration

[andybj0228]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Add File System support for HuaweiCloud OBS.

Solution

Integrate with HuaweiCloud OBS Hadoop file system adapter classes.

OBS filesystem configurations should be propagated as is

    remote.data.dir: obs://bucket/fluss
    'fs.obs.endpoint' = 'obs-endpoint-hostname',
    'fs.obs.access.key' = 'obs-access-key',
    'fs.obs.secret.key' = 'obs-secret-key'

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!

[andybj0228]

@luoyuxia When implementing a new filesystem, is the token module below necessary?

[andybj0228]

@luoyuxia

[luoyuxia]

@andybj0228 Thanks for the pr. Sorry for missing this comment.
Yes, it's necessary. Otherwise, client can't access the data in obs. I spent some time to get familiar with obs. Seems this document can help us to obtain the token https://support.huaweicloud.com/api-iam/iam_04_0002.html

[andybj0228]

@luoyuxia Thank you for your detailed response. The code for HuaweiCloud OBS integration is completed. However, while writing test cases, I noticed that the credentials in the OSS test cases are obtained via System.getenv. Are there any requirements for this part?

[andybj0228]

1、I first set the following OBS-related environment variables locally.

2、I run the following Maven command to run the test cases.

@luoyuxia Do the test results above indicate that the entire file system is functioning properly? If I proceed to cover the remaining test cases, can I submit a PR afterward?

Thanks

[luoyuxia]

@luoyuxia Thank you for your detailed response. The code for HuaweiCloud OBS integration is completed. However, while writing test cases, I noticed that the credentials in the OSS test cases are obtained via System.getenv. Are there any requirements for this part?

We manage a OSS bucket for our test purpose. But it's not required for each other filesystems if the filesystem instance to test is not available.

[luoyuxia]

1、I first set the following OBS-related environment variables locally.

2、I run the following Maven command to run the test cases.

@luoyuxia Do the test results above indicate that the entire file system is functioning properly? If I proceed to cover the remaining test cases, can I submit a PR afterward?

Thanks

Sure, you can submit a pr. I'll help review and merge it .
If filesystem instance is not available, of course I think we don't want to have a manage obs instance, the ITCase won't run in our CI. But we believe the contributor to verify in their env.

[andybj0228]

@luoyuxia Please review the pr. Thanks

[luoyuxia]

@andybj0228 Thanks for the pr. Definitely I'll help review & merge it~ Thanks again!

[andybj0228]

@luoyuxia Please review the new pr. Thanks.