I run NTP at work, more specifically, we use GPS time sources to synchronize a wide-area paging network (by wide area I mean over an area nearly a quarter of a million km square). Timing is extremely critical for our application to work correctly over an area that size.

The only viable option for us is about 3 (or more) rackmount NTP appliances from a well known vendor which have the GPS already integrated, support for multiple time source modules and power supplies, and have integrated software to drive the whole thing. This means we need to get GPS antennas installed on the roof of the datacenter and fiber run from there to our racks.

Our remote sites (connected by 4G and VSAT) also have GPS units, but they are connected via RS-232 serial (not USB) and suffer from some of the synchronization issues you describe. They fallback to the time servers in the datacenters in the event that the local GPS time source is rejected.