I'm doing something similar.
My development environment for work is defined in a Dockerfile, and I have a small shell script whose only purpose is to call `docker run` with that image, mount a few volumes for caching, mount the CWD in "/workspace", and start a shell in there. Development is done with nvim.
If I need Docker Compose, I run it from the host. For projects that I find unpleasant to work with in this way, I use GitHub Codespaces. I hadn't thought about using a GUI IDE from within a VM, so thanks to your comment (EDIT: and also the submitted article) I now have something new to try!