The Cerebras node's actual "RAM" (the 40GB of SRAM) is pretty modest too, but being an enormous chip with the networked storage pools is certainly a better situation than a bunch of A100s reaching out to every other A100.

Honestly, all the AI ASIC makers drastically underestimated the RAM requirements of future models. Graphcore's 4GB and Tenstorrent's 8GB per IC is kinda laughable, and it takes them longer to adjust than Nvidia. And Cerebras' original pitch was "fit the entire model into SRAM!"