- OSDev.org (https://wiki.osdev.org/Expanded_Main_Page) is a great starting point for writing an operating system
- Three Easy Pieces (https://pages.cs.wisc.edu/~remzi/OSTEP/) is a wonderful free online textbook that teaches the concepts of operating systems.
- Modern Operating Systems, 5th Edition by Andrew Tanenbaum (of MINIX fame) and Herbert Bos (https://www.pearson.com/en-us/subject-catalog/p/modern-opera...) is the latest edition of a solid graduate-level textbook on operating system concepts.
It may also be beneficial studying the source code of existing operating systems. I recommend starting with smaller, simpler systems, such as MINIX and xv6 (https://github.com/mit-pdos/xv6-public), before moving on to larger, more complex systems such as the Linux kernel (https://kernel.org/) and its userland (e.g., GNU utilities, systemd, etc.).
Another cool thing is to study the designs of non-Unix operating systems, such as the classic Mac OS, VMS, IBM OS/400, Plan 9 (yes, this is "more Unix than Unix" in many ways, but it's quite a departure from Unix) and its successor Inferno, and Symbolics Genera. Bonus points for reading academic papers on OS concepts such as exokernels.
Good luck! It's a long but very interesting journey!