I don't use Erlang, but I have developed an Actor system for C# [1] which is based on its (and Akka's) concepts. Clearly without a static type-checker for the whole distributed system we have to manually get involved and patch the old and new so that we can hot swap processes. Versioning I've found is best done by maintaining the old process that accepts the old message format, maps it to the new one, and then forwards it on to the new process that accepts the new message format. Any other node that is lagging behind will continue to work, and any new one will send to the new address for the process.

This isn't really rocket science, and if you stick to a few basic rules it tends to work out just fine. That doesn't mean that type safety goes out of the window, it just means that in creating a distributed process you must accept that you can't retire the old contract without it causing potential problems.

Apologies if I'm missing your point about OTP, but ultimately it seems that at some point (as the GP says) you are marshalling a message into a text or binary format, and then unmarshalling. At that point if the unmarshalled static type doesn't match the type that the process expects, then it will be off to the dead-letter queue. I don't really see how that's any different to giving the wrong type to a function in a dynamic language, or using an incorrectly typed variable that is picked up by a compiler in a statically typed language. In each case it's type checking at the earliest possible opportunity.

[1] https://github.com/louthy/echo-process

No, that's not how you do it. You marshal things directly into the desired types. Check out either aeson for Haskell or how Go does things via either the json modules or the generic Text/Binary Marshaler/Unmarshaler.

"but also the myriad deployment & upgrade & versioning scenarios."

The answer to all of those things is mostly that even a lot of Erlang shops don't use live upgrading. You really have to have a very particular use case for that to be the best solution vs. a rolling upgrade and server restarts. Even if the language is capable of it, it still requires you to write services that can handle being upgraded, and it's much easier to write services that can handle being restarted, especially since you 100% have to write that anyhow because services get restarted anyhow. Most people don't have that use case. Web services certainly don't have that use case.

Once you drop that, it's a lot simpler.

"Even thinking about something like a gen_server"

gen_server is partially as complicated as it is as a side-effect of other decisions in the language. While the concept of a gen_server is a strength in Erlang, the specific implementation of gen_server as this "behavior" thing is mind-blowingly complicated for what you actually get. (It reminds me of Python's "metaclasses". I spent many hours wrapping my head around what that was, but in the end, all that it amounts to is what is now called a class decorator, which is way more sensible. A metaclass isn't a class decorator in theory, but in practice, class decorators are way easier to understand and cover 99.9% of the use cases, if not 100%.) When I implemented supervisor trees in Go, my solution for gen_server/gen_fsm/gen_* was just to... not. Behaviors are just a very, very weird half-object-ish system with a lot of limitations. They are easily replaced by simply having some sort of "interface" system, be it via conventional classes or interfaces. It's why you don't see "behaviors" as Erlang defines them anywhere else. Erlang has a lot to learn from and copy from, but that part isn't it.

Which is exactly why we want to employ static types: in order to catch the difficulties in implementing it correctly. We describe the complications in the type system, through a model that captures them, to allow compiler errors -- rather than runtime errors -- to guide us in implementing it correctly.

Types only hinder getting an invalid program to compile -- which is exactly what we want.