Ask HN: How are you using llama.cpp?

4 pointsdanielEM1y ago2 comments

For what purposes you use llama.cpp?

What tools do you use with llama.cpp?

Is there anything you recommend to avoid when it comes to llama.cpp?

Want to collect your best practices/experiences and advice around llama.cpp. Eg. if you work with Visual Studio Code - what plugins you recommend, and what not. Etc...

2 comments

2 comments · 2 top-level

roosgit1y ago

I use it to help me write text.

Don't use any tools. I run it from the command line:

./main -f ~/Desktop/prompts/multishot/llama3-few-shot-prompt-10.txt -m ~/Desktop/models/Meta-Llama-3-8B-Instruct-Q8_0.gguf --temp 0 --color -c 1024 -n -1 --repeat_penalty 1.2 -tb 8 --log-disable 2>/dev/null

I prefer `main` to the new `llama-cli` because when searching history for "llama" I want to get commands that contain the "llama" models, not "mistral" ones, for example.

caprock1y ago

I've not used it myself, but I see a lot of people referencing Ollama. It uses llama.cpp (and maybe more).

j / k navigate · click thread line to collapse