Amazon S3 Transfer Acceleration (opens in new tab)

Make sure you chunk it and use the S3 multi-part upload API. That way, you can break into N chunks, and then upload N in parallel, to get closer to saturating your uplink. The same works for downloads, where you can use byte range GET's to essentially do multi-part download.

try https://github.com/rlmcpherson/s3gof3r you can often get wire speed

I live in southern Japan, with a single thread to a Tokyo bucket I get around 200-300 MBit, which is about what my WiFi network maxes out at

edit: uploading to a Singapore bucket I get around 90 Mbit

Testing in my office (central US) to N. Virginia and Oregon was getting about 90 Mbit/sec. I'm guessing that's the cap on our connection, not an Amazon limit.

I wrote a benchmark a while back: https://github.com/lox/s3-bench

I'm in Singapore, uploading to Oregon, 100mb, took minutes , I changed to upload via signed cloud front requests, takes 15s to upload 100mb now...

900-950 Mbps on a 1000 Mbps line, but only if you use 20 threads and multipart

Even though their Amazon S3 Transfer Acceleration Speed Comparison page [1] do test uploads only, this new feature is documented to accelerate downloads as well.

I guess optimized WAN network routing between S3 regions and their edge locations usually beats regular internet routing.

[1] http://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/e...

We played with this a bit using both cloudfront and other cdns. While we did see some small improvements, it wasn't nearly as good as we were expecting on the upload side. It may be fixed now, but there were also many issues getting the proper headers to pass through cloudfront to s3 for chunked uploads.

You can't upload through Cloudfront. It's a good analogy though, you can think of this feature as a "CDN for uploads", if you'll excuse the abuse of the CDN acronym.

It says on the page: "Note: In general, the farther away you are from an Amazon S3 region, the higher the speed improvement you can expect from using Amazon S3 Transfer Acceleration. If you see similar speed results with and without the acceleration, your upload bandwidth or a system constraint might be limiting your speed."

So I suspect you might be having that problem in your test.

We have 20Gbit of Direct Connect with public peering on. This causes upload for us to be 40% slower in primary region and 2-5% slower in the other regions in the US. We only get any benefit going to APAC, granted we aren't your "typical" or "ideal" user for this

If it makes you feel better, the accelerated ones are slower for me!

http://i.imgur.com/n6DhyHj.png

My ISP hates Europe it seems... http://i.imgur.com/66123xv.png

The accelerated speeds I were getting hovered around 250-300 MBps

"What's the fastest & cheapest way to get data off Amazon's cloud services?"

We[1] may have a good answer for you ...

Let's assume you have terabytes of data, otherwise there's no difficulty, right ?

So, when you combine the HN readers discount pricing for 10TB datasets, which is 4c/GB/mo. with the fact that we support 's3cmd' in our environment:

  ssh user@rsync.net s3cmd get s3://rsync/mscdex.exe

... and the fact that we have no other charges (no transfer/usage/bandwidth charges) ...

... and the fact that we have 10gbps connectivity through he.net ...

It's possible that we would be a good fit.

You fire up a 10TB account for $400/mo and you issue s3cmd commands, over SSH, on your rsync.net account, which "pulls" the data from S3, at pretty much whatever speed Amazon can throw at us.[2]

Email us.

[1] rsync.net

[2] http://arstechnica.com/information-technology/2015/12/rsync-... ... scroll down to the part where they do the speed tests ...

I know of three ways to get data out of amazon's cloud. Whether they are the fastest depends on how much data you're talking about.

1) Download the data and pay egress bandwidth charges. Starts at $90/TB, but gets cheaper with more usage.

2) Use import/export snowball to have disks shipped to you. This is $30/TB + $250 per 80TB device.

3) Use direct connect to connect fiber directly to the region. Costs $1620/mo for each 10G line + $30/TB + an unknown amount to your fiber provider.

It's too bad that export traffic can't be marked as "low priority" with a cheaper cost. I imagine there are times at night when utilization is low and big export jobs could be run during those times. (It's obviously not in AWS's interest to make it cheap to get data out of their data centers.)

It depends entirely on the quantity of data.

For huge datasets, Snowball will be fastest & cheapest (literally shipping data).

Even with that, I don't get it.

"Moving lots of data either requires a huge pipe, or a ton of storage disks."

With that, they offer their Snowball device, which, if I'm understanding correctly, holds up to 50TB (now 80TB), which they physically ship to you, and then you ship back to them. How does this fix either of the constraints (disk space / connection pipe)?

Yes, cloudfront gives you far better control over things like headers, routing for different paths, error pages, etc, than vanilla s3 http access does, and is almost certainly lower latency compared to s3 (assuming your files aren't GB-sized)

You don't actually have a choice:

https://news.ycombinator.com/item?id=11531357

> Anyone know if the quoted cost of $0.04/GB for Transfer Acceleration is instead of the $0.03/GB for standard S3 ingress, or in addition to it? Ie. is the final cost per GB $0.07 or $0.04 when using Transfer Acceleration?

S3 ingress is free. You're thinking of storage pricing.

Data transfer into S3 is free - it's the storage that's $0.03/GB-Month

It will be released by tomorrow.

You can enable in CloudBerry Explorer for Amazon S3 - http://www.cloudberrylab.com/blog/amazon-s3-transfer-acceler...

S3 Website Hosting is distinct from CloudFront; you can create a CloudFront distribution backed by a bucket with any name. You can combine the two, but it isn't required.

http://docs.aws.amazon.com/AmazonCloudFront/latest/Developer...

http://docs.aws.amazon.com/gettingstarted/latest/swh/website...

I think it's simply wildcard SSL getting in the way, if you put a . in the bucket name, the DNS record becomes a.b.c.s3-xxxx.amazonaws.com

The wildcard SSL certificate they use only matches the c.s3-xxxx.amazonaws.com part of the sub domain

You can setup a socks proxy pretty easily with ssh -D 1080 <your server name>, then set localhost:1080 as your socks proxy and run the test.

What stops you from uploading gzip'd data to S3?

Do you have IPv6 only devices you'd like to support?

It's turned on tomorrow.