The thing though is that this doesn't prove that X seconds have passed. It proves that X seconds have passed on whatever baseline hardware has been used to calibrate it. I don't know who actually computes the VDF in the proposed proof-of-stake schemes, though I would assume it's "whoever is proposing a block" (is this the same as the staker? Does this mean every single staker is picking a block and computing their own VDF, meaning everyone is still burning CPU?). And this means the VDF can only establish a minimum CPU requirement. It can say "X seconds have passed on the minimum hardware we're requiring at the moment", but anyone with faster hardware can still compute it faster.
And also because this PoW scheme cannot require more than X seconds for any participant to compute, it means an attacker that starts computing their alternative blockchain at the same time as the block they want to replace faces no difficulty. All this does is interferes with the ability to decide after the fact that you want to attempt to replace history. And even then, if you have hardware faster than the baseline, you can still reach back in time to recalculate a block, you just have to wait longer to do so. And by that I mean if you want to edit a block from 100 minutes ago, and you've got a CPU that's twice as fast as whatever the VDF is tuned for, then it just takes you 100 minutes to compute the replacement blockchain (50 minutes to compute the past 100 minutes, and 50 minutes to compute the new blocks that have been added since you started the attack). So after 100 minutes you now have an alternative chain that everyone thinks took 200 minutes to compute.
Which means now we're just back at the problem of "attacking consensus", where nobody can look at the two blockchains and see within the system which one was calculated first.
---
I suppose the VDF could be calculated by some volunteer with the fastest hardware, though this requires rewarding them for doing so (which means you basically have a monopolist sucking up all of the VDF rewards and no real incentive for nearly all participants to even try and compete). And this is still attackable by someone who can put together hardware that is even just slightly faster than then volunteer. It just takes longer. If the security of the system relies on a volunteer being assumed to have the fastest hardware on the planet, then the system isn't secure. I also question what happens in this scenario when the volunteer goes offline and nobody else has hardware that's as fast. Now the next block isn't ready in X seconds. I assume there's some protocol for "oops nobody has finished computing the VDF in time", but this does provide another avenue of attack for anyone in a position to disrupt the volunteer's connection to the network. Of course, anyone in a position to do that is likely to have access to unusually fast hardware already, but the point is that you cannot rely on the idea that "nobody can possibly calculate this block faster than the VDF is tuned for".
This attack is possible in Bitcoin too, except because Bitcoin is parallelizable, the defense there is that this attack requires spending more money than it is worth as the computing power used to calculate blocks is roughly a function of the value of the network. The danger there is generally in centralizing too much of the computing power among too few participants rather than an outsider breaking the scheme. This attack does work on smaller PoW coins of course, generally by folks who control a chunk of Bitcoin computing power and just redirect it temporarily (if the value in attacking the coin is greater than the expected bitcoin mining rewards for the time it takes to do the attack, then this makes sense).
Honestly, it really seems like we should only have one global PoW network, and everything else should use other systems. Perhaps they should satisfy security by doing things like VDFs for short-term security and storing their blockchain hashes into Bitcoin for long-term security. Bitcoin using up a ton of power is still a problem of course, but maybe there's some sort of approach that can be used to solve the problem of "PoW to establish a global distributed clock" once you remove the "and we want to use this as a currency" part that doesn't invoke a massive arms race. This may involve ditching the idea of "anyone can participate", which also then allows you to change the incentives for running the PoW scheme.
---
Edit: I suppose the VDF's input might not be "the block being computed" but instead "the previous block", and the output then used to elect participants who are then trusted to build the new block. This would allow the new block to indicate whether the VDF actually took longer than expected. But then we're back at a probabilistic function with PoS, where those with the highest stake are now most likely to be trusted and therefore are in a position to abuse that trust.
I suppose reading up more on PoS systems might answer this question. But I really don't want to do that. I've already spent far longer on this than I intended to.