FWIW there's really nothing wrong with uninstalling this kind of metapackage. I've been uninstalling the "ubuntu-desktop" metapackage as I trim my ubuntu systems, for over a decade. I use a combination of "debfoster -n" and aptitude for the trimming ... it is a bit annoying but it's just once per install.

That said, for the past couple years I've just run Debian and Arch. But I'll probably use ubuntu again for a while when I get a new framework laptop later this month, the hardware is too new, some special/custom kernel package provided for the latest ubuntu will probably be the most convenient solution initially, and I find trimming back ubuntu isn't too bad. Yes, you have to remove the "desktop" meta-package. It's fine.