Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon | Better HN

Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon | Better HN

87 comments

simonw3d ago

Suggestion for the maintainers: the comparison table currently lists some pretty old models, Qwen 2.5 14B and Mixtral 8x7B and Llama 3.3 70B.

A lot of people are reporting incredible results with the Qwen 3.5 MoE models on Apple hardware right now (streaming experts - see https://simonwillison.net/2026/Mar/24/streaming-experts/) - it would be great to get some of those models into that table.

Maybe the 1T parameter Kimi K2.5 too if you can get that to work, see https://twitter.com/seikixtc/status/2036246162936910322 and https://twitter.com/danpacary/status/2036480556045836603

The Qwen 3.5 MoE local performance numbers are striking but the cloud pricing picture for the same models is equally interesting. On inference platforms right now Qwen class models are running significantly cheaper than closed equivalents, open source has about an 81% pricing advantage on equivalent platforms. The local vs cloud crossover math gets genuinely interesting for MoE models because the sparse expert loading that makes Hypura useful locally is the same property that makes cloud inference cheaper per token. Worth knowing both sides of that equation when deciding where to run.

tatefOP2d ago

Thanks for sharing this! If you'd be interested in running the benchmark yourself with Hypura I'd happily merge into our stats. Otherwise will add to my todo list :)

Imustaskforhelp3d ago

Simon, A little offtopic but it seems that your website isn't working.

> An error occurred in the application and your page could not be served. If you are the application owner, check your logs for details. You can do this from the Heroku CLI with the command

I get this error when I go to simonwillison.net

Any random blog/link works for example though: https://simonwillison.net/2026/Mar/19/openai-acquiring-astra...

(I checked your website because I wanted to see if you had written something about trivy/litellm as well, I highly recommend checking out what has happened within litellm space if possible as I would love to read your thoughts on it)

Have a nice day simon!

Edit: now the website works but I am not sure what had gone wrong previously, (an issue from heroku maybe?) as its working now

Edit-2: after the website working, I am able to see that you have already made a post about it.

abtinf2d ago

The lack of a token rate metric for the kimi example is disappointing.

vanyaland3d ago

For a lot of local workloads, sub-1 tok/s is useless in foreground and perfectly acceptable in background. If the choice is “this crashes” vs “this finishes overnight,” that’s still a meaningful capability jump.

joelthelion2d ago

How much are you going to spend on electricity though? Is this really going to be more cost-effective than just using openrouter?

vicchenai3d ago

the practical question is whether the read pattern is sequential enough to actually saturate nvme bandwidth or if the attention layer access pattern ends up being random enough to kill throughput. sequential reads on a decent nvme get you 5-7 GB/s, random reads drop to maybe 500 MB/s depending on queue depth.

for a 1T model youd need to stream something like 2TB of weights per forward pass at fp16. even at peak sequential thats 300+ seconds per token which is... not great for interactive use but maybe fine for batch inference where you dont care about latency.

still a cool proof of concept though. the gap between 'can run' and 'runs usefully' is where things get interesting.

p_ing3d ago

4K random read with a queue depth of 1 on an M1 Max is about 65MB/s.

tatefOP3d ago

Yes, definitely agree. It's more of a POC than a functional use case. However, for many smaller MoE models this method can actually be useful and capable of achieving multiple tokens/sec.

zozbot2343d ago

> for a 1T model youd need to stream something like 2TB of weights per forward pass

Isn't this missing the point of MoE models completely? MoE inference is sparse, you only read a small fraction of the weights per layer. You still have a problem of each individual expert-layer being quite small (a few MiBs each give or take) but those reads are large enough for the NVMe.

marksully3d ago

Where does "1T parameter model" come from? I can only see models with 70B params or less mentioned in the repo.

tatefOP3d ago

I'm referencing it as being possible, however I didn't share benchmarks because candidly the performance would be so slow it would only be useful for very specific tasks over long time horizons. The more practical use cases are less flashy but capable of achieving multiple tokens/sec (ie smaller MoE models where not all experts need to be loaded in memory simultaneously)

causal3d ago

Yeah title comes from nowhere in the link. No doubt it's possible but all that matters is speed and we learn nothing of that here...

baq3d ago

Intel Optane rolling in its grave.

Memristors are also missing in this AI hype even when they were around the corner 10 years back.

liuliu3d ago

Still have 4 brand new ones in my storage unit. Just in case these moments.

Joke aside (I do have them tho!), I don't think Optane is that much use (not to mention it is only 256GiB for my unit). It is useful legacy crutch if you have legacy software that is not designed to issue multiple reads / writes in parallel. If you do, it is really not faster than NVMe, especially these modern ones.

speedgoose3d ago

Is it too late for Intel to bring them back to life?

moffkalast3d ago

Wouldn't be Intel if they didn't quit halfway through on a good thing.

Still, couldn't one get a RAID 0 card with four drives to saturate a 16x lane? That's already the max one could push through PCIe anyhow.

0ptan33d ago

pmem

shubhamintech3d ago

The MoE point matters here ie sparse activation means you're not reading all 2TB per forward pass, but the access pattern flips from sequential to random which is exactly the worst case for NVMe. Been thinking about this a lot for agent inference workloads where you want consistent latency more than peak throughput.

dev_tools_lab1d ago

Thanks for this project. Prioritizing MoE models and adding an intelligent NVMe cache could improve efficiency, especially on the M4 Max where bandwidth makes usage more realistic.

This is a pretty cool project! Essentially this is like using Swap memory to extend your RAM, but in a 'smart' way so you don't overload the NVMe unnecessarily.

I do wonder in practice how the 'smarts' pan out, because putting a ton of stress on your NVMe during generation is probably not the best choice for it's longevity.

zozbot2343d ago

This is not putting any stress or wear on the NVMe, it's a pure read workload.

embedding-shape3d ago

> but in a 'smart' way so you don't overload the NVMe unnecessarily

"overloading NVMe"? What is that about? First time I've heard anything about it.

> because putting a ton of stress on your NVMe during generation

Really shouldn't "stress your NVMe", something is severely wrong if that's happening. I've been hammering my SSDs forever, and while write operations "hurt" the longevity of the flash cells themselves, the controller interface really shouldn't be affected by this at all, unless I'm missing something here.

hello

msbhogavi2d ago

"As much memory as possible" is right for model capacity but misses bandwidth. Apple Silicon has distinct tiers: M4 Pro at 273 GB/s, M4 Max at 546 GB/s, M4 Ultra at 819 GB/s. Bandwidth determines tok/s once the model fits in memory. An M4 Max gives you 2x the decode speed of an M4 Pro on the same model.

For what Hypura does, the Max is the sweet spot. 64GB loads a 70B at Q4 with room to spare, and double the bandwidth of the Pro means generation is actually usable instead of just technically possible.

> Consumer hardware (MacBook Pro, Mac Studio) ships with fast unified memory and NVMe storage, but limited capacity. A 32 GB M1 Max cannot naively load a 40 GB model — the OS will swap-thrash until the OOM killer intervenes.

macOS doesn't have an "OOM killer" in that sense. (It has an out of swap space killer but it's pretty weak.)

So what will happen is, either your memory wiring will fail, or else it will get really slow and panic.

zozbot2343d ago

It will be interesting to compare this to https://news.ycombinator.com/item?id=47476422 and https://news.ycombinator.com/item?id=47490070 . Very similar design except that this is apparently using mmap, which according to the earlier experiment incurs significant overhead.

salynchnew3d ago

It was written by an LLM, so... yeah.

jeffybefffy5193d ago

Except this isnt using heavily quantised versions of the model thus reducing quality.

dev_tools_lab2d ago

Nice work on the scheduler. Have you benchmarked parallel inference across multiple models? Running GPT, Claude and Gemini simultaneously on the same input is where latency becomes a real constraint.

zozbot2342d ago

GPT-OSS exists but Claude and Gemini aren't available locally, lol.

dangoodmanUT2d ago

With unified memory and such a strong os-hardware integration, one would hope that swap could handle this task

root_axis3d ago

Are there any 1T parameter open source models?

zozbot2343d ago

Kimi 2.5?

I am curious how the TPS compares vs default OS virtual memory paging

speedgoose3d ago

I wonder how many minutes per token on GLM 5.

This is <1 tok/s for the 40GB model.

Come on, "Run" is not the right word. "Crawl" is.

Headlines like that are misleading.

Could still be useful; maybe for overnight async workloads? Tell your agent research xyz at night and wake up to a report.

smlacy3d ago

Yes, and with virtually zero context, which makes an enormous difference for TTFT on the MoE models.

monksy3d ago

There needs to be something like this from Ollama. At the moment Ollama has a lot of flaws that prevent it from getting great performance. (My understanding is better GPU/CPU splits, etc). But Ollama is the only way to host an LLM and have it switch out on demand. Sigh.

zozbot2343d ago

Ollama has very substandard support for mmap at present, which hurts inference with larger models. There are some recent pull requests in flight that should help address this to at least some extent https://github.com/ollama/ollama/pull/14525 https://github.com/ollama/ollama/pull/14134 https://github.com/ollama/ollama/pull/14864 but progress seems to be stalling. Their support for recent Qwen models seems to also have some bespoke incompatibilities with llama.cpp, which doesn't help matters; it's difficult to test the same model with both.

rubiquity3d ago

llama.cpp and llama-swap do this better than Ollama and with far more control.

EnPissant3d ago

You do not provide any comparison to llama.cpp with mmap.

You do not explain how any kind of predictor can work for MoE experts.

You do not explain how prediction can even be useful. I can predict the layers used in a dense model (all of them are used in order), but that doesn't help me much. It's still bottlenecked on bandwidth (hint: MoE doesn't change this).

anshulbasia273d ago

OS paging would be significantly worse here. The kernel's page fault handler is reactive — it doesn't know you're about to read layer 47's FFN weights, so it can't prefetch. You stall on every fault, wait for the 4KB/16KB page to load, then resume. With 80 layers of dense FFN streaming, that's thousands of cold faults per token.

  What makes this approach faster is that the model's access pattern is completely deterministic during         
  inference. You know exactly which tensors are needed next because transformer layers execute sequentially. So
  you can issue large sequential reads and prefetch the next layer while the current one is computing on Metal. 
  The OS page cache can't do that — it has no concept of "layer N+1 comes after layer N."

  For MoE it's even more stark. The OS would page in all 8 experts on the first token that routes to each one,  
  then evict them under memory pressure with LRU, which has no idea that expert 3 fires 10x more often than
  expert 7. The neuron cache here is basically a domain-specific replacement policy.

zozbot2343d ago

> The kernel's page fault handler is reactive — it doesn't know you're about to read layer 47's FFN weights, so it can't prefetch.

man 2 madvise

EnPissant3d ago

That assumes you have significant work to do between fetches (so you can prefetch while using the current data). With LLM decode you don't.

erikcw3d ago

Simon Willison wrote a good post about Dan Woods’ work on “Autoresearching Apple's "LLM in a Flash" to run Qwen 397B locally”.

[0] https://simonwillison.net/2026/Mar/18/llm-in-a-flash/

j / k navigate · click thread line to collapse