Debugging S3 curl Transfer Issues

"aws" uses "curl" for all it's network IO. In the case of most S3 transactions, aws computes a URL then hands off to curl. If you are having transfer issues, the best way to debug is to remove aws from the loop. You can do it by creating a signed URL that you can use directly with curl. At that point, you can use curl options to get better debugging output.

The following sequence creates a file hello.txt, creates a signed URL, and then uses curl to upload. The --expire-time=1d tells aws to sign the URL, so that it expires after one day. The --request tells aws to output the signed URL rather than execute it.

$ aws rm test681/hello.txt
$ echo hello, world >hello.txt
$ URL=`aws put test681 hello.txt --expire-time=1d --request`
$ echo $URL
https://test681.s3.amazonaws.com/hello.txt?AWSAccessKeyId=AK...AA&Expires=1362586485&Signature=QO...Q%3D
$ curl $URL --upload hello.txt
$ URL=`aws get test681/hello.txt --expire-time=1d --request`
$ echo $URL
https://test681.s3.amazonaws.com/hello.txt?AWSAccessKeyId=AK...AA&Expires=1364912980&Signature=T8...8%3D
$ curl $URL
hello, world