I haven't touched that steaming pile of shite in a looong while, so - who knows - they might have managed to minimize the amount of havoc their wreak on their user's systems.
But ... I seem to recall ... Conda tries to install GPU drivers does is not? ... Is that not the case anymore?
Because if it still does, your theory about "Why would your system care" and all that doesn't really hold water.