Do We Need a New Orchestration System for GPUs? (opens in new tab)

(nathanleclaire.com)

2 pointszenlikethat3y ago5 comments

5 comments

5 comments · 2 top-level

There is something similar to what you described operating right now: Stable Horde: https://stablehorde.net/

I'm not sure if it has the priority queueing system you mentioned, or how much it has generalized to non-SD loads, but it is functional.

Also, you should consider using a Huggingface diffusers UI/backend instead of the automatic1111 UI, which is based on (and now inextricably linked to) the old Stability AI implementation. Maybe thats fine now, but its already a problem for optimizations people are cooking up (like Facebook's AITemplate, torch.compile, SHARK/MLIR for AMD GPUs, Intel OpenVINO...) which work with diffusers already.

I am bouncing between InvokeAI and VoltaML at the moment (as I cant get any of the optimizations to work in the auto1111 implementation anymore), but I can hardly keep up with everything on Github.

zenlikethatOP3y ago

Oh yeah, Horde is super cool, I’ve been meaning to give it a spin. That’s good info about the web ui! I have noticed that relative to my hardware, the results seem slow. Lots could be causing that though.

brucethemoose23y ago

Another factor is your system libs. An old PyTorch/CUDA combo, for instance, can hold 4000 GPUs back.

I have been quite happy with Arch's build of PyTorch (which targets CUDA 12.1 at the moment).

brianjking3y ago· 1 in thread

I'm only part of the way through the article, and yeah, interesting read.

zenlikethatOP3y ago

Thanks!

j / k navigate · click thread line to collapse

5 comments

5 comments · 2 top-level

brucethemoose23y ago· 2 in thread

There is something similar to what you described operating right now: Stable Horde: https://stablehorde.net/

I'm not sure if it has the priority queueing system you mentioned, or how much it has generalized to non-SD loads, but it is functional.

Also, you should consider using a Huggingface diffusers UI/backend instead of the automatic1111 UI, which is based on (and now inextricably linked to) the old Stability AI implementation. Maybe thats fine now, but its already a problem for optimizations people are cooking up (like Facebook's AITemplate, torch.compile, SHARK/MLIR for AMD GPUs, Intel OpenVINO...) which work with diffusers already.

I am bouncing between InvokeAI and VoltaML at the moment (as I cant get any of the optimizations to work in the auto1111 implementation anymore), but I can hardly keep up with everything on Github.

zenlikethatOP3y ago

Oh yeah, Horde is super cool, I’ve been meaning to give it a spin. That’s good info about the web ui! I have noticed that relative to my hardware, the results seem slow. Lots could be causing that though.

brucethemoose23y ago

Another factor is your system libs. An old PyTorch/CUDA combo, for instance, can hold 4000 GPUs back.

I have been quite happy with Arch's build of PyTorch (which targets CUDA 12.1 at the moment).

brianjking3y ago· 1 in thread

I'm only part of the way through the article, and yeah, interesting read.

zenlikethatOP3y ago

Thanks!

j / k navigate · click thread line to collapse