This is exactly what the ID cards I'm talking about are. You tap them to the phone or a desktop reader and it works. You just invented something that already exists.
eIDAS just takes this one step further and gives you an option to not have to carry your card with you. But if you refuse to have an attested phone, then you pay those 20EUR to get the ID card (which you probably need for other uses as well) and move on with your life.