Also prior to that OSs were proprietary - if you didn't work for a hardware manufacturer you mostly didn't get to play with them much. There was no incentive to make them portable - then along came Unix and it was dropped into the hands of a generation of college kids.
Sure VMS had a better VM story (I both ported Unix and was a VMS systems hack at the time) - but you couldn't pick it up and put on anything other than a vax, it was pretty useless as an OS unless you were DEC, Unix didn't have that yoke around its neck (it had others mostly because AT&T didn't understand what it had)
I can go on and on ...
Unix "won" by becoming the darling of the people not willing to pay money while Windows swept the field of those willing to cough up cash.
However, don't mistake the virality of Unix for it being "better" than VMS.
The thing is you're comparing 1 VMS that didn't evolve much (it did get sort of cloned by MS) with effectively hundreds of Unix descendants.
As I said I've worked intimately with both, and I don't think VMS gets as much love as it's due, but in the long run I think it was a dead end largely because it was held close by a (great!) company that died
The lesson I take from tech of this era is that good enough (and cheap) beats superior over time. Unix over VMS, Ethernet over Token Ring, etc. I posit that Unix's simplicity of that era (you really could hold the entirety of the system in your head back then, and rereading the entire manual set every few months was something considered perfectly feasible and regularly recommended for power users) was the winning formula. It certainly helped that unix was all over academia. 20 years later, when Solaris was the incredibly powerful big dog and linux was the upstart, history repeated itself.
I think this same system oriented view (as distinct from user) is what sunk DEC, HP and Sun.
VMS wasn't all vax assembly either. VMS and a lot of VAX system software was mostly written in BLISS [1], a high level language intended for systems programming that was created about the same time as C. It was all there for customers to read in a big stack of microfiche cards delivered with each machine. I remember being shown the code for the VMS scheduler or maybe the dispatcher which was 'a page or two of (VAX) assembly' - I recall it was pointed out that it was one of the (few) places in VMS that did use assembly.
I don't know how portable the BLISS code for VMS was, but there were BLISS compilers for several other machines besides the VAX.
A similar problem (AFAIK) befell BLISS - BLISS code for the VAX treats all values as 32-bit. There is a separate BLISS-64 compiler for the 64-bit platforms VMS was ported to, but I don't think all the BLISS-32 code was converted over to BLISS-64.[2]
[1] http://www0.mi.infn.it/~calcolo/OpenVMS/ssb71/6467/6467p005.... [2] https://www.cs.tufts.edu/~nr/cs257/archive/ronald-brender/bl...
So the question is why were all these university people running Unix instead of VMS. Were they just dumb?
The "university" people were still running VMS certainly up through 1990 or so. Very few people ran Ultrix on their VAX hardware. And workstations were never widespread in universities as workstations basically cost as much as a car. A lab with 8 workstations was still a big deal in 1990.
Unix didn't sweep away VMS, PCs did. Just like PCs swept away workstations.
Unix didn't "win" until it latched onto PCs as a host. And people also forget that Unix didn't "win"--it lost terribly to both Microsoft and Apple.
