undefined | Better HN

0 pointssamspenc2y ago0 comments

Ah fascinating, just curious, what's the technical blocker? I thought most of the Llama models were optimized to run on GPUs?

0 comments

mayankchhabra2y ago

It's fairly straightforward to add GPU support when running on the host, but LlamaGPT runs inside a Docker container, and that's where it gets a bit challenging.

stavros2y ago

It shouldn't, nVidia provides a CUDA Docker plugin that lets you expose your GPU to the container, and it works quite well.

dicriseg2y ago

See above if you're interested in that. It does work quite well, even with nested virtualization (WSL2).

1 more reply

j / k navigate · click thread line to collapse