That's arguing that you can just seen an RNG with the current time and use a PRNG as your randomness source - a whitener can't give you randomness out which isn't there to start with.

In the above, fairly extreme case, the risk should be obvious: if someone has a decent guess on what the uptime of your system is, and knows you're doing this, then the search space to crack certificates can be made accessibly small.

Like if you know see a certificate with a Valid From date of say, January 1, 2025 but you know the service definitely wasn't running on January 1, 2024, then by guessing what the PRNG is you've constrained your search space to 1704027600 through 1735650000. So the issue isn't whether the numbers you emit are distinct - it's that an adversary can make it suitably likely that they can produce colliding RSA keys themselves anyway (and remember, they get unlimited attempts at this - they only have to succeed once).

EDIT: And while you can certainly argue that they couldn't predict the exact noise environment of say, your server room, it's also not impossible to model which also might constrain the search space enough to accessible. It's not "haha! we know your every move" it's just making the problem space small enough to brute force.