The problem is (an expectation of) fewer bitcoins chasing more goods. Removing bitcoins while adding labor doesn't counteract that dynamic, it adds to it. (It's also not true that that's the only way bitcoins are removed from circulation - some have already been lost/destroyed - but given my previous point that's simply an aside).

It's a problem because of what money is. It's not an asset to be held as store of value, it's a medium of exchange to make trade work better. If everyone is refusing to spend their bitcoins because they expect them to be more valuable tomorrow, bitcoin is not being a currency and people have to use other currencies to get things done or things aren't getting done, either of which is an issue for bitcoin-as-a-currency (and the latter an issue for everyone). Arguably bitcoin could continue to serve as an asset anyway, but I don't understand that to be the vision.

None of this is to say that I am confident this will be sufficiently an issue that things stop working, but I definitely see it as a risk.