The concern is that there's a large enough chance that it might to be worth planning for the outcome. That chance doesn't need to be high for that to be the case. And there's good reason to believe that the size of number that has been factored up to now is not a reliable indicator that the growth rate will remain very slow.

(The analogy with the Manhattan project is apt: an adversary learning about it would have been wise to adjust their planning around the possibility of it succeeding even if they judged that it was not a given that it would)

> [...] no one has managed to factor a larger number than 21.

Small correction: no one has PUBLICLY managed to factor a larger number than 21.

There could be advances (foreign and domestic) that just don't get published because they represent having an upper-hand with regards to cryptography. So, from Game Theory perspective, not making waves is in the interest of nation states. They'll even try to be dismissive about concerns.