Introducing Amazon EC2 P3 Instances (opens in new tab)

Dan here (also Paperspace team). Totally agree that transfer costs are a significant pain point which is why we do not charge for it. We can peer with other providers (eg with AWS we can leverage Direct Connect directly from our datacenters) but most of our customers don't implement this unless they're moving major traffic.

>Getting the data into and out of compute services is the most difficult part financially, at least in my experience.

You can never forget that this is entirely because of compute services ripping you off, not because they're providing a valuable service in return for the transfer pricing.

We have definitely been thinking a lot about what that would look like (i.e. is it more of a job architecture, an API, clustering, etc). Would love to hear your thoughts on what GPU Lambda might look like. Feel free to hit me up directly dillon [@] paperspace [dot] com if you want to continue the conversation :)

We're adding sync support to Worker (which has GPU support) at Iron.io soon! This will allow you to run long running background jobs (current behavior) as well as sync serverless/faas Lambda-like functions within a single API.

For ML/deep learning tasks you should definitely use a dedicated GPU. I would recommend our GPU+ (NVIDIA Quadro M4000/P4000) which has 8GB of VRAM and 1664 CUDA cores.

Honestly, I didn't spend enough time delving in to the Python overhead, especially in terms of the framework. Most of it would be an issue of my own causing however rather than the framework's. The original code I wrote was never written with data loading / saving in mind as the source for speed issues so I avoided what would have been premature optimization at the time.

Some of the slowdowns now just seem silly and aren't even listed in the per epoch timings: PyTorch doesn't have an asynchronous torch.save(). This means that if you save your model after each epoch, and the model save takes a few seconds, you're increasing your per epoch timings 5-10% just by saving the damn thing!

Regarding FP16, PyTorch supports, and there's even a pull request that updates the examples repo with FP16 support for language modeling and ImageNet. It's not likely to be merged as it greatly complicates a codebase that's meant primarily for teaching purposes but it's lovely to look at. I also think many of the FP16 issues will get a general wrapper and they'll become far more agnostic to the end user. For the most part they're all outlined in NVIDIA / Baidu's "Mixed Precision Training" paper. Might be useful for DeepLearning4j to go through the most common heavy throughput use cases and get them running (just as an example of how to work around issues really) if customers were using P100s/V100s?

I'm really interested in exploring the FP16 aspect as the QRNN, especially for single GPU, is sitting at basically 100% utilization, with almost all the time spent on matrix multiplications. FP16 is about the only way to speed it up at that stage. This gets a tad more complicated regardless as the CUDA kernel is not written in FP16 (and is not easy to do so) but even converting FP16->FP32->(QRNN element-wise CUDA kernel)->FP16 ("pseudo" FP16) should still be a crazy speedup. I tested that on the P100 and it took per epoch AWD-QRNN from ~28 seconds to ~18.

- PyTorch async save issue: https://github.com/pytorch/pytorch/issues/1567

- PyTorch FP16 examples pull request: https://github.com/pytorch/examples/pull/203

- "Mixed Precision Training": https://arxiv.org/abs/1710.03740

Genuinely curious: Given that Softlayer bare metal server prices start at 700$ per month is there even remote chance of this actually being profitable?

On-demand prices from amazons pricing page https://aws.amazon.com/ec2/pricing/on-demand/ (select Virginia region):

p3.2xlarge: 8 vCPU, 61 GB RAM, $3.06/h

p3.8xlarge: 32 vCPU, 244 GB, $12.24/h

p3.16xlarge: 64v CPU, 488 GB., $24.48/h

Unfortunately, P3 isn't listed yet, but this is my go to site for EC2 pricing: http://www.ec2instances.info/

showing up for me on https://aws.amazon.com/ec2/pricing/ for each of On-Demand Instances, Reserved Instances, Spot Instances, and Dedicated Hosts pricing lists. Are you selecting the regions where this is available - US East (N. Virginia), US West (Oregon), EU West (Ireland) and Asia Pacific (Tokyo)?

You can use the new AWS Deep Learning AMI which has a version of TensorFlow enhanced for CUDA 9 and Volta support https://aws.amazon.com/blogs/ai/announcing-new-aws-deep-lear...

Chris from NV here. You can also get a full compliment of DL framework containers, as well as CUDA 9/CuDNN 7/NCCL 2 base container, optimized for Volta by NVIDIA via this AMI https://aws.amazon.com/marketplace/pp/B076K31M1S?qid=1509089...

TF 1.4 does, but you need to build it yourself. RC1 is out:

https://github.com/tensorflow/tensorflow/releases/tag/v1.4.0...

  All our prebuilt binaries have been built with CUDA 8 and cuDNN 6.
  We anticipate releasing TensorFlow 1.5 with CUDA 9 and cuDNN 7.

Yes, p3.2xlarge in us-east-1b is currently sitting at $0.3204 spot. That's only marginally more than p2.xlarge (at $0.2259 in us-east-1e).

I'm sure this will change with demand, though. :(

There are enough companies out there with deep pockets that want to do some ML. They'll pay pay those prices, no questions asked.

Yep agreed. Didn't want to post the spot pricing since it changes all the time :)

Amazon prices are for the pay as you go model. You can shave a significant amount off the price if you know you're going to be running them for 12 months.

Be careful with the non-CUDA 9 AMIs.

CUDA 8 programs will run, but terribly slowly as they JIT their GPU code without optimization for Volta. You want the CUDA 9 AMI version (https://aws.amazon.com/marketplace/pp/B076TGJHY1?qid=1509090...), but it currently only has MXNet and TF.

If you need other frameworks there's the NVIDIA AMI (https://aws.amazon.com/marketplace/pp/B076K31M1S?qid=1509090...) and Volta optimized containers for NVCaffe, Caffe2, CNTK, Digits, MXNet, PyTorch, TensorFlow, Theano, Torch, CUDA 9/CuDNN7/NCCL.

Didn't know they had an AMI like that. Thank you.

Out of professional curiosity, what are you looking for from ENA?

(I'm an engineer on Google Compute Engine with a deep interest in customer networking use stories, particularly heavy utilization customers, even if they're not my customers :)

what do you mean? 8 or 16 GPUs? That's require changing the code to use distributed tensorflow...

Maybe by being bigger customers... :)

It's not just that Amazon has done the math, it's that sufficiently liquid cryptocurrencies will, by the efficient market hypothesis, quickly gain enough value to make mining on whatever Amazon offers no longer profitable. As soon as you're able to profitably mine without an up-front capital investment, people will take advantage of the arbitrage opportunity until the market adjusts its price, and if the currency is designed at least somewhat competently and has enough of a working market (both of which are definitely true of Bitcoin), that won't take very long.

Cryptocurrencies are the invisible robot hand of the market. (Which is, I think, not a claim about whether they're good, but certainly a claim about whether they are to be feared. If you squint hard enough, the giant Bitcoin mines in China are the work of an unfriendly AI employing people to make paperclips.)