In other words, assume SDR "Game" mode is set to Backlight 50, SDR "Cinema" mode is set to backlight 25, all other settings are equal, and, therefore, 100% white is considerably brighter in "Game" mode.
Then both values for "white" cannot possibly match a single, fixed level set by any other mode, HDR or otherwise.
It's therefore impossible for Windows or any other input source to "just work" when switching from an arbitrary SDR mode to a preset HDR mode.
Again, assuming your TV works more or less like mine, and mode switches use the most-recently-used preset in the target "mode family" (meaning not only SDR and HDR, but also Dolby Vision, which maintains its own collection of presets), and that the various presets are independent of one another.
And if this is not the case, and presets cannot be set independently, then I'm glad I don't have a newer TV, because some of my presets have color settings that are wildly different from standard calibration (e.g., a preset resetting the display to its native, uncalibrated white point and gamut, used with video players capable of internal HDR tone mapping and color correction given a custom 3D LUT generated from measurements).