There is! See
https://petals.ml/ for inference of models like BLOOM-176B over the internet or
https://arxiv.org/abs/2301.11913 and
https://arxiv.org/abs/2206.01288 that show you how to do pretraining from scratch in the same setting. Disclaimer: I'm a coauthor of these systems (including the one in OP)
