e2e/util_test: enable tests to work using external endpoint

[boraberke]

With this PR, e2e tests can now be tested using external endpoint.

To do so, following environment variables need to be set:

S5CMD_ACCESS_KEY_ID
S5CMD_SECRET_ACCESS_KEY
S5CMD_ENDPOINT_URL
S5CMD_IS_VIRTUAL_HOST
S5CMD_REGION
S5CMD_I_KNOW_WHAT_IM_DOING

s5cmd will use S5CMD_ENDPOINT_URL with credentials S5CMD_ACCESS_KEY_ID, S5CMD_SECRET_ACCESS_KEY and S5CMD_REGION to run all e2e tests instead of local gofakess3 server. Additionally, if the endpoint tested uses virtual host style S5CMD_IS_VIRTUAL_HOST needs to be a truthy value and control env variable S5CMD_I_KNOW_WHAT_IM_DOING needs to be set to 1. If any of those environment variables is empty, tests will fallback to use gofakess3 instead.

Additionally, tests are refactored to have the following characteristics:

1. Every test cleanup the bucket(s) after the test is done.
2. Each created test bucket has a unique bucket name.

[igungor]

@boraberke igungor/gofakes3#7 is merged. v0.0.13 release is ready.

[igungor]

This change will be very useful for us to be confident about the tool for certain s3 compatible services. Thank you.

[seruman]

🚀 Thank you!

There's a merge conflict though.

[boraberke]

🚀 Thank you!

There's a merge conflict though.

Just resolved the merge conflict 🏁

[seruman]

For the records, I did try to run tests with Backblaze B2 and there are a couple things to consider -and there might be more-;

• B2 limits maximum buckets per account to 100.
• Versioning is enabled by default. To delete a bucket, all versions of the files had to be deleted.
• Looks like it does not keep spaces between media-type and parameters in Content-Type header;
  • Tests sets and expects text/html; charset=utf-8 but B2 responds with text/html;charset=utf-8.

Do not think these are blockers for this PR. We might update the tests as we start add more compatible services.

[boraberke]

For the records, I did try to run tests with Backblaze B2 and there are a couple things to consider -and there might be more-;

• B2 limits maximum buckets per account to 100.

• Versioning is enabled by default. To delete a bucket, all versions of the files had to be deleted.

• Looks like it does not keep spaces between media-type and parameters in Content-Type header;

  • Tests sets and expects text/html; charset=utf-8 but B2 responds with text/html;charset=utf-8.

Do not think these are blockers for this PR. We might update the tests as we start add more compatible services.

Thanks for testing Backblaze and pointing these out! As you said, after this PR, additional compatible services can be added and tests can be updated accordingly 👍