putFile/putStream API should handle HTTP 307 redirects from AWS S3

[shafdog]

I totally get the "low level" knox API should leave this kind of handling to the client. But I wanted to suggest that the "high level" knox S3 API handle the cases where AWS S3 returns a HTTP 307 status code.

The Amazon's S3 doc says:

If you create a bucket using <CreateBucketConfiguration>, applications that access your
bucket must be able to handle 307 redirects."

This happened to me when I used #53 a few minutes after creating a new bucket in AWS to use. Once DNS is sync'd with S3, the case is likely more rare.

Obviously there are "workarounds" for this, but wanted to bring it up since knox is working pretty well for me otherwise! Thanks!

[domenic]

Can you expand on how we might do this? It sounds like a good idea but I'm not sure exactly what it implies.

[domenic]

Using request might help with this.

[rauchg]

superagent

[shafdog]

Sorry I didn't comment sooner. I wrote this up for completeness, the library works great. I wanted to look at the code before commenting.

But did run into this case. It happened to me right after the bucket was created. My belief was that since AWS S3 relies on the bucket name as part of the DNS hostname, if DNS hasn't replicated, then AWS handles that case by using a 307 while replication is pending. I think it's also happen if there is a problem at AWS where a data center goes, and they need to redirect the S3 traffic.

I've used restler as a HTTP client in other code. Never heard of superagent, but looks like it client side code...

[shafdog]

Found this article that speaks to the underlying case for the 307 HTTP redirects:

http://docs.amazonwebservices.com/AmazonS3/latest/dev/Redirects.html

[magwo]

This is important - I often run into this problem with new buckets. I think using the request library might be a good idea - not only for redirect support, but also for increased sourcecode readability.

[domenic]

@magwo pull request welcome; otherwise I'll try to move this up the priority list.

[magwo]

I think the big question here is whether the knox lib should respect the 307 permanently, or keep going to the original endpoint but following the redirect every time. What do you think?

Edit: Actually since 307 is temporary redirect, it should keep going. But what about 301 or 303 or the like? I'm fairly sure I've gotten that from S3 on several occasions, and the knox version at the time did not handle that gracefully. Does it currently support that?

[magwo]

I might look into implementing this. Switching to internally use the request library might be an idea, but could also be a bit risky since the Node request object is exposed by the API. Maybe if the request library is 100% API compatible with Node requests, this will not be a breaking change. Not sure about the details of request.

[domenic]

I think the big question here is whether the knox lib should respect the 307 permanently, or keep going to the original endpoint but following the redirect every time. What do you think?

Even in the case of 301, I don't think it's the Knox client's responsibility to maintain a table of redirections.

[shafdog]

I think AWS's PermanentRedirect results if you use a bucket name but get the region of the bucket wrong. Didn't try that and haven't seen them however.

Amazon docs are silent on when you'd get a PermanentRedirect (HTTP 301). See http://docs.amazonwebservices.com/AmazonS3/2006-03-01/API/ErrorResponses.html to find out its vague.

But if you create buckets on the fly, it's the 307 that you'll run into.

[magwo]

Agree that it's not knox's responsibility to keep track of which URLs redirect where. However, the API user will probably want to know if there was a permanent redirect that was automatically followed. This should not be done silently. Are there any proper ways to do this in the current API? There is no injected logging etc AFAICS.

[domenic]

If it should not be done silently, then I don't think it should be done at all.