What if that app is say and open source messenger app that doesn't use GCM/FCM. For that to work you need to maintain a connection with a remote service. You don't need the UI hogging memory the whole time but you definitely want messages to come in immediately. IMO the system should offer a way to completely kill and app but "killing the UI" should not stop critical services utilized by the app.
Okay so maybe you could argue that if you never "swipe kill" the app it should be allowed to have background services executing much like e.g. Slack on macOS. But you can't even do that. Having your ui in the recent tasks list does not exempt you from background restriction.