That's your problem right there.
Virtual environments are the Python ecosystem's solution to the problem of wanting to install different things on the same machine that have different conflicting requirements.
If you refuse to use virtual environments and you install more than one separate Python project you're going to run into conflicting requirements and it's going to suck.
Have you tried pipx? If you're just installing Python tools (and not hacking on them yourself) it's fantastic - it manages separate virtual environments for each of your installations without you having to think about them (or even know what a virtual environment is).