undefined | Better HN

0 pointssmcleod9mo ago0 comments

RTX is nice, but it's memory limited and requires to have a full desktop machine to run it in. I'd take slower inference (as long as it's not less than 15tk/s) for more memory any day!

0 comments

diggan9mo ago

I'd love to see more Very-Large-Memory Mac Studio benchmarks for prompt processing and inference. The few benchmarks I've seem either missed to take prompt processing into account, didn't share exact weights+setup that were used or showed really abysmal performance.

chisleu9mo ago

Oh I plan to produce a ton of that. I'll post a blog on it to HN and /r/localllama when I'm done.

j / k navigate · click thread line to collapse

0 pointssmcleod9mo ago0 comments

RTX is nice, but it's memory limited and requires to have a full desktop machine to run it in. I'd take slower inference (as long as it's not less than 15tk/s) for more memory any day!

0 comments

diggan9mo ago

chisleu9mo ago

Oh I plan to produce a ton of that. I'll post a blog on it to HN and /r/localllama when I'm done.

j / k navigate · click thread line to collapse