undefined | Better HN

0 pointsnoduerme2y ago0 comments

Agreed. After several rounds of setting up various python environments and tinkering with directory structures and debugging glitches and quantizing models just to end up playing around for a few minutes and getting bored, it would be nice to have the experience just be seamless. I wouldn't try to set up a workflow around seriously using what's out there to run on localhost now.

That said, non-subscription is essential, and that's probably going to be a heavy lift considering how quickly things are evolving.

0 comments

simonw2y ago

I've been trying to push things in that direction with my LLM tool - the idea is to have Python plugins which you can install that do all of the irritating details to get a model setup.

I've not yet been able to solve the challenge of needing CUDA etc for some models though!

Plugins so far: https://llm.datasette.io/en/stable/plugins/directory.html

noduermeOP2y ago

Cool! I've followed your instructions and your blog quite a bit as I've experimented with running local LLMs as well as stable diffusion. It's been especially helpful, as python is not my language or usual environment. Your patience at hacking your way through each new iteration and presenting what's important about them is astonishing; I personally think I'd have gone mad, but you've done great work in charting the territory.

j / k navigate · click thread line to collapse