Why do you like Safari in light mode, or dislike it in dark mode? Are you referring mostly to the app chrome or to the content of the sites you visit?
I would tend towards ease of configuration and consistency, and would only make exceptions for apps/sites with more theme choices than just light/dark.
I think if most of the pages were also dark like the surrounding chrome, because they implemented a dark theme with the prefers-color-scheme media query, that we would find the experience more agreeable. We’re in the early days, but once the dust settles and most sites have implemented dark mode, I’d expect the dark web browsing experience to improve.