Lots of different OSs. Voyager probably doesn't even have an OS. The booster rockets on the shuttle use 8088's and assembler. The mars rovers use vxworks or it's predecessor. And I've heard rumors of embedded linux and QNX for some projects.

Basically you're going to find an OS reasonably suited to the job. Also, you're going to find OLD stuff as NASA wants something where the bugs are known / already worked out. This is why a few years ago they upgraded Hubble to a 486. And why on the rovers it executes code in debug mode and has hardware to interface with the JTAG ports. Everything that works off planet needs to be as bug free as possible and if something goes wrong you need a way to patch it remotely with latencies in the minutes to hours.