I am not in a position to say why the original NeXT was so different in philosophical underpinnings than the Mac, but there are a few differences one can observe from the outside that may (and I only say may) have contributed to the final outcome:
1. When Jobs left Apple, he was barred from competing with them in the general computing/appliance market. NeXT when it launched was advertised as a product for higher education.
2. Jobs had to bootstrap a company from scratch into what was by then a much more sophisticated industry than existed when the Apple // or mac were launched. Building on top of Unix and Mach was way lower risk than trying to invent an entire operating system from the ground up when your interest was really in the user experience.