That does seem like the obvious problem with this. For example AWS apparently have GPS time sync to all their datacenters in order to provide accurate NTP time to the instances in them:
https://aws.amazon.com/blogs/aws/keeping-time-with-amazon-ti... It seems like it'd be hard to beat the connection speed and all-around performance of that, especially since there's a cross-internet trip from the third party stratum 1 servers to Cloudflare as well as the one from their servers to you.