Especially because an extra 200 miles of fiber only increases round trip time by 3 milliseconds. How far are signals going before they get routed in the right direction?
So now I have to go east 10ms in order to go 70ms west (80ms). If, say, this ISP could also reach transit in Pittsburgh, that might only be 5ms northwest followed by 60ms west (65ms).
Or if HN decided to spin up a second server location in Chicago, then I might only need to go a third of the way across the country instead of all the way (20ms or less).
This is what I believe Dave is referring to when he says that moving data and interconnectivity matters more than deploying fiber (which might save 5-10ms over cable).
So when you are talking about shaving off 6ms from a 70ms connection, it is nowhere near as dramatic as shaving off 8ms from a 10ms connection.
Now, QUIC is doing wonderful things here - 0rtt resume, and in the cloud the overall bandwidth to a given end-user is cached, but there is a really big difference between 2ms and 70! That´s just content. Geoff Huston gave a really good series of talks this week about AS paths getting a lot closer to 1 - 50 miles - across his data set, in the last 4 years: https://www.youtube.com/watch?v=gxO73fH0VqM - hopefully this will become a separate hackernews thread.
Most of my focus, however, is not on content, but communications, where we still see 300+ms glass to glass latency for videoconferencing, or worse - and bufferbloat when you are actually using the link for other things - even fiber networks tend towards 100ms worth of buffering, and fiber has a tendency to bufferbloat the wifi.... I have seen a trend in fiber towards 10ms of buffering which is actually too low and too brittle, unless fq´d and aqm´d also.
To go back to your HN example - HN loads fast because it is ONE IPv6 address (for me) and very lightweight so tcp slow start ramps up pretty darn fast, even going all the way to San Diego.
A packet capture of even a big HN discussion is really lovely, btw. More people should do that and admire it, compared to the sordid mess older websites with 100s of objects have.
The cost of generating crypto is very real when you're talking about single-digit ms latencies :( RSA-2048 TLS certs add about 2-3ms to any connection, just on server-side compute, even on modern CPUs (Epyc Milan). (I believe a coworker benchmarked this after disbelieving how much compute I reported we were spending and found that it's something like 40x slower than ECDSA P256.)
> To go back to your HN example - HN loads fast because it is ONE IPv6 address (for me) and very lightweight so tcp slow start ramps up pretty darn fast, even going all the way to San Diego.
I used HN as an example not because it's bloated, but due to its singly-homed nature to illustrate how much content placement matters. Yeah, we could quibble about 80ms vs 65ms RTT from improving peering but the real win as I mentioned was in server placement. Throwing a CDN or some other reverse proxy in front of that helps as far as cacheability of your content / fanout but also for TCP termination near the users (which cuts down on those startup round trips). This is why I can even talk about Los Angeles for www.google.com serving even though we don't have any core datacenters there that host the web search backends.
(For what it's worth, I picked Chicago as a second location as "nominally good enough for the rest of the US". Could we do better? Absolutely. As you point out, major CDNs in the US have presence in most if not all of the major peering sites in the country, and either peer directly with most ISPs or meet them via IXes.)
Having a second server can definitely have a massive impact on latency, but that doesn't sound like an ISP thing, or a rural thing, or an interconnectivity thing.
Just looking at distances on a map is insufficient for actually characterizing the network path.
Also: bear in mind that you need to double all these numbers when considering round trip time. I recognize I phrased it in such a way that might have been interpreted as one-way latencies, but that wasn't my intent.
Even if you had a direct path from Ashburn to Pittsburgh, speed of light through fiber would be about 3.5 ms to travel 450 miles (there and back). And while you might expect that from just plugging numbers into an equation, I have never seen anything resembling 4ms RTT between DC and New York (which are a comparable distance apart from each other) on Google's production network, even though those are definitely directly connected (6-7ms is more realistic).
