Ask HN: How to Keep Up with LLMs? [Linux, Self-Hosting, Info]

16 pointsconor_f2y ago4 comments

I've accepted that LLMs are useful when taken with precautions. A few tools I've used recently have convinced me on this fact. Alongside this, I've heard first-hand anecdotal experience of LLMs providing great benefit, from asking about APIs all the way to practising languages or summarizing texts.

I'm now interested enough to try run one myself and see how it suits my personal workflow. So I have a few questions:

1) How can I set up a LLM locally with good effort/reward ratio? I don't want to spend hours setting up something unreliable that needs constant modification - moreso something I can just interact with easily from a web UI/CLI when I need to.

2) Is there an easy way to keep up to date with LLMs so I can update to newer models as they become popular to get the best results?

Note that I'm only looking for self hosted, Linux compatible solutions!

4 comments

version_five2y ago

My first stop would be llama.cpp and compatible models on your own machine. You should be able to run quantized 7B and 13B models, try them out and see if they work.

Though for "personal workflow", unless you want to be able to play with the internals of the models or are worried about privacy, I'd just use ChatGPT (in fact I do, despite having llama.cpp setup to run various models, I always use ChatGPT for personal stuff and programming question)

conor_fOP2y ago

Thanks, I will look into llama.cpp today :)

I don't have much interest in playing with the internals for now, but I generally like keeping my data personal and the services I use self-maintainable as much as reasonably possible! I also feel like I could find the token limits and price limiting with ChatGPT.

tikkun2y ago

1) what do you want to use it for?

2) /r/localllama is good, and then also the “open llm leaderboard” and the “lmsys llm leaderboard”

conor_fOP2y ago

1) Some general use cases:

  - Getting familiar with new APIs

  - Bouncing general knowledge questions off it

  - Having "discussions" to interact with it in a Socratic style

  - Giving some "personality" to automated services by calling it as an API

2) Thanks for the pointers! Will check them out today!

j / k navigate · click thread line to collapse

Ask HN: How to Keep Up with LLMs? [Linux, Self-Hosting, Info]

16 pointsconor_f2y ago4 comments

I'm now interested enough to try run one myself and see how it suits my personal workflow. So I have a few questions:

2) Is there an easy way to keep up to date with LLMs so I can update to newer models as they become popular to get the best results?

Note that I'm only looking for self hosted, Linux compatible solutions!

4 comments

version_five2y ago

My first stop would be llama.cpp and compatible models on your own machine. You should be able to run quantized 7B and 13B models, try them out and see if they work.

conor_fOP2y ago

Thanks, I will look into llama.cpp today :)

tikkun2y ago

1) what do you want to use it for?

2) /r/localllama is good, and then also the “open llm leaderboard” and the “lmsys llm leaderboard”

conor_fOP2y ago

1) Some general use cases:

  - Getting familiar with new APIs

  - Bouncing general knowledge questions off it

  - Having "discussions" to interact with it in a Socratic style

  - Giving some "personality" to automated services by calling it as an API

2) Thanks for the pointers! Will check them out today!

j / k navigate · click thread line to collapse