Show HN: LLM Simulation – Experience TTFT and tokens/SEC before investing (opens in new tab)

(llmsimulation.ht-x.com)

1 pointshertzdog3mo ago2 comments

I built a small tool to simulate the user experience of LLM response speeds, focusing on TTFT (time to first token) and tokens/second.

Instead of reading benchmark numbers, you can feel how fast or slow different configurations are, by adjusting TTFT, token generation rate, and output length. It streams tokens exactly as an LLM would, but without generating real content.

I was wondering which Apple should I buy and then I did it in the weekend, to better feel what does it mean to run locally a model.

The project/toy is public on github too: https://github.com/htxsrl/localllmsimulation

Thanks to the sources (cited) for the real benchmarks that allowed me to set up a small ML model to fit even futuristic hardware (like an imaginary M9 with 2048 Gb RAM and 3000Gb/s bandwidth).

2 comments

ndgold3mo ago

Lovely, yoinked.

Also, I’m seeing check marks next to all quants which confused me a little bit when trying to select.

hertzdogOP3mo ago

Thanks! I added the check marks because when I was testing different quantizations, I often picked a model and only afterwards discovered that it couldn’t even load — it just didn’t fit into RAM or VRAM.

So the check mark simply indicates that the model can actually run under those constraints (fits in memory), not that it’s selected.

j / k navigate · click thread line to collapse