If you don’t want Firefox to do even that, you can go to Settings, click on Privacy & Security, then choose the Strict option. It’ll warn you that it might break sites, but you can always turn it off on a site-specific level by clicking on the shield in the address bar and turning off the toggle.
Be aware though that this will make firefox tell sites that you want a light color scheme instead of doing the more sensible thing and indicating no preference.
Though IIRC from some bugginess I was noticing a while ago, only to the JS API, it will still obey the dark color scheme media query (but sometimes inconsistently, I would load the same page in multiple tabs and sometimes get dark or light schemes).
