Skip to content

Top Best Ask Show New Jobs

Ask HN: Would you use a shared GPU cloud tier?

6 pointscpeterson421y ago19 comments

I.e. a cloud instance that behaves exactly like a normal GPU instance, where behind the scenes the GPU is shared. The advantage here is you only pay when the process is actively using the GPU (not the whole time the instance is running). The downside is the instance would take ~25% longer to run any GPU task.

19 comments

18 comments · 8 top-level

cpeterson42OP1y ago· 2 in thread

For anyone curious, here is an early prototype of this tech in action:

https://imgur.com/a/2qPN4ru

Would love to hear your thoughts on how we can make this most useful for you!

woah that's really cool! can't wait for the show HN

cpeterson42OP1y ago

Appreciate it! We're reaching out on a smaller scale for now to manage how many signups we get to not break our systems. Happy to share a demo vid and/or more details at carl (at) thundercompute (dot) com.

cheptsov1y ago· 2 in thread

Sounds like an extremely complex technical problem. I also suggest to look at the use cases when this is needed. One of the problems is that loading weights into the GPU will be so slow that it will be really hard to share the GPU between different processes - causing long time to offload and load. Would love to learn more about what you do.

cpeterson42OP1y ago

Hi Cheptsov, know you mentioned wanting to learn more. We have some demo material that we can share if you're interested. Would love to chat at carl (at) thundercompute (dot) com.

cpeterson42OP1y ago

Totally right on all fronts. This has been tough to build and we are still searching for the best use cases. If you are down to chat my email is in my profile, any insights would help us a ton!

lbhdc1y ago· 2 in thread

Yes, I have wanted something like this for a while. I try to avoid using gpus where possible because of the expense, and the ephemeral nature of my use.

cpeterson42OP1y ago

Interesting to hear. What kind of workloads are you running?

lbhdc1y ago

Various forms of content analysis. It is mostly traditional heuristics with ml models sprinkled in. We don't run inference on every request, but it would be great to access a dynamicly provisioned gpu for a single request (kind of like its another serverless container in the system).

einsteinx21y ago· 1 in thread

Depending on the price difference from a standard GPU instance, absolutely.

cpeterson42OP1y ago

Hey Einsteinx2, we're unsure where the pricing will end up at scale, however we are running a beta with a T4 instance + a CPU-only instance for $10/month, so can guarantee a pretty crazy deal for now :)

If you'd be interested would love to chat more at carl (at) thundercompute (dot) com.

JSDevOps1y ago· 1 in thread

Isn’t this just the cloud? You pay for what you use

bmodel1y ago

Essentially yes. The difference would be that we measure "what you use" more granularly since you'll only be charged while something is being run on the GPU, not the entire time the instance is running.

This is where the "shared" aspect would come in, since during unused GPU cycles someone else could use the GPU.

billconan1y ago· 1 in thread

Yes, I would use it if the price is affordable.

cpeterson42OP1y ago

We're running a beta for $10/month for a T4, which is definitely affordable, but we are running at a steep loss for now to test our system at scale. If you're down to be one of the first few users email me at carl (at) thundercompute (dot) com!

DamonHD1y ago· 1 in thread

Maybe prefix with "Ask HN:"?

cpeterson42OP1y ago

Fixed, thank you!

BoredPositron1y ago

Depends on your workload...

j / k navigate · click thread line to collapse