Mac OS 8 had no preemptive multitasking or meaningful address space protections. A single bad pointer dereference in user mode took down the entire system, and a single busy loop without a yield locked up the entire system.

Both of these were universally admitted to be bad and outdated by technically minded people.

By 1997 they had looked at replacing it with BeOS or NEXTSTEP, and purchased the latter with the goal of replacing Mac OS. The Rhapsody OS, an OS8 style UI with NeXT underneath, had already been started. Before that, they had also attempted and failed to write a next gen classic Mac OS (Copland).

Windows 9x had a lot of problems, but had preemptive multitasking and much better address space isolation. Windows NT 4 Workstation was also a thing at the time and much better. It did take them two more releases to make it into the consumer product.