Latency of the connection = sum of latency of every hop along the route.
So in practical terms your ISP directly controls your bandwidth, but latency is mostly out of their hands.
To the closest Measuring Broadband America's measurement servers [1].
(They're currently "hosted by StackPath and were located in ten cities (often with multiple locations within each city) across the United States near a point of interconnection between the ISP’s network and the network on which the measurement server resides.")
[1] https://www.fcc.gov/reports-research/reports/measuring-broad...
I imagine StackPath is thrilled that this is the FCC's metric.
They could invest in peering and other arrangements that improve these paths.
Actually yeah, that sounds like the correct metric to use.
The latency you see is almost always from last-mile software, under the control of the ISP, and can be fixed locally. Before fixing, my local ISP gave me ping-time to the internet interconnect point in downtown Toronto that were typical of a link to Istanbul, Turkey (:-))
They weren't trying, and aren't to this day. Arguably they need a nice unfriendly regulator to require at least a good-faith attempt.
Nearest IXP with at least three Tier 1 ISPs present and where the ISP in question generally peers also seems like a good target. I add the part about peering, because it's not great when I'm near Seattle, and rarher a lot of my traffic to seattle area servers goes through Portland or San Jose, because my ISP apparently doesn't like to peer in Seattle.
A third thing, which would drive more IXP adoption, would be latency between your phone over 5G to your desktop over broadband. Everybody should be able to collect that latter metric, methinks....
And then, go measure the bufferbloat those ways.
Bandwidth tests already deal with the ways ISPs might play games with upstream capacity.
But something else that matters (possibly not so much in the US, but definitely in many other countries) is mini internet 'brown outs', where there's a sudden internet drop out for just a few seconds. I don't know what statistic would be good for measuring that. But when it happens during a meeting it's quite annoying and noticeable. In some parts of the world they seem to occur about once an hour, although YMMV.
So if you're trying to avoid latency spikes, don't use DFS channels on 802.11ac.
Page here has a list of all the Wifi channels on 5GHz: https://www.networkcomputing.com/wireless-infrastructure/cha...
In particular, you can't get a 160Mhz wide channel (8 channels bonded together) on 802.11ac without being on a DFS channel.
So I would vastly prefer that in contended scenarios - an apartment building for example - that the actual channel width provisioned to be not be much more than what the ISP is providing. (and that the wifi be de-bufferbloated: https://lwn.net/Articles/705884/ )
Instead people keep provisioning 160mhz channels with dozens of other APs also living on those channels, and interference and retries go way up...
Above 50Mbits, most of the bufferbloat problem shifts to the wifi.
Availability, i.e., uptime of your link. Frequency of availability interruption etc.
But, luicky I live in big city with multiple ISPs offering. I have 2 ISPs at flat atm, one is ETTH (primary) and second is Cable for backup. ETTH is good, nice peerings, low RTT and very small <1ms jitter. Cable, well.. its cable. They expanded they peerings so RTT dropped, but jitter is noticeable. Its all right, its backup connection after all.
Jitter is a big issue for IP based telecommunications and was a bigger concern than Mbps when I was implementing them in the past.
I have had this on wifi for sure, but never wired with ethernet.
Not an ISP issue.
Sweden. So not USA. But western country where we decided to invest in broadband infra.
I am on DOCSIS/COAX modem/router combo. 200 down 10 up. Ethernet wired to gaming PC and work laptop. Rock solid.
Downspeed is better at work but it being WiFi there and shared with many people it still will suck despite being enterprise Cisco and Unifi hardware on two different networks.
If more of our tests tested up + down + latency - and ISPs and users deployed bufferbloat fixes like fq_codel, cake, and libreqos.io - the internet would be a lot less jittery and feel a lot more fast, all the time.
https://blog.cerowrt.org/post/speedtests/ has some ranting. More ranting about saner uses for fiber here, also: https://blog.cerowrt.org/post/towards_better_broadband/
So the nutrition label scheme is seriously flawed without testing up+down.
They're probably fine here, because I don't think we actually care about consistency (I wouldn't complain if suddenly half my packets had lower latency) we care about the worst case latency and how often it happens.
Standard deviation is what I'd use if I was measuring consistency.
Doing the math, this is 75/(24*3600) = .00086, or availability of 99.914%, which sounds great, but in reality it's garbage.
Instead of p99, which wouldn't tell us anything really, I'd suggest something like average and total packet loss on a continuous 1-second ping by month for the last year, or even by day for non-zero days. Then if a service has dropout problems it will be easy to spot.
But averages obscure what's really important. The biggest indicator of Starlink congestion problems has been how packet loss increases in evenings. Average latency is interesting but much more interesting is the variance of latency or jitter. A steady 50ms is better than a connection varying 20-80ms all the time. As for bandwidth what I've found most useful is a measure of "hours under 20 Mbps download" (1 or 2 a day on my Starlink).
IRTT is a fantastic tool for measuring latency and packet loss. Way better than simple pings. The hassle is you have to run a server too; I have one in the same datacenter as the Starlink terrestrial POP.
It's perfect for a sailboat or a wartime forward operating base, but not so much for a city.
That may not be very relevant now, as barely anyone uses Starlink and as their bandwidth needs are 2024 bandwidth needs rather than 2034 or 2044. But eventually, starting with the densest cells, congestion will occur. If your city is tied to an electrical grid, it's not that much harder to run gobs of fiber.
Starlink could shrink cell size with larger antennas (see Starlink 2.0), but I suspect it's abusing the deep magic of phased array antenna gain to expect a many-phased-array-N-to-many-phased-array-M network connection to scale feasibly at high SNR. Just taking a clever mathematical innovation and asking it to do things in too many dimensions with too many orders of magnitude improvements.
ph'nglui Nyquist mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn
The problems with physical/geometrical high-gain antennas are not so arcane; I wouldn't be surprised if we go back to dishes or equivalently free-space optical networking for fixed antennas. You could run an almost indefinite number of satellite-client connections simultaneously in something as high-gain as a laser/LED launch telescope.
Starlink works well in my rural middle of nowhere and has no competition.
It took a year to get it resolved, and even then I'm not sure if it was deliberate or accidentally fixed. You wind up with all sorts of excuses. "It must be your splitter." "We should recap the end of the cable." "It must be your router." They refuse to do basic diagnostics, like checking your neighbors connections for packet loss (which can be done remotely.)
It would be great to add more to the pool.
The data would be far more interesting if you indicated the distribution of the data, perhaps as ranges of one standard deviation. Then it would succinctly express your point as well as be easy for you to quickly find outliers.
(for reference, Taht's comment is indirectly linked from the submission; https://www.fcc.gov/ecfs/document/12010651418616/1 which has much more detailed technical discussion.)
Dave Taht... that's the guy behind a lot of the bufferbloat work over the past decade or so (and seems to be the submitter?), right?
In my experience, predictable latency is more important than low latency. 50ms RTT can be fine for a video call; it's not fine if it spikes up to 500ms sporadically. Protocol work like QUIC that can reduce per-connection startup overhead definitely helps in this regard.
On the other hand, if you have a highly interactive use case like SSH where every single keystroke must roundtrip between you and a server, that 50ms RTT is very visible.
I think talking about absolute latency might be misguided in that it depends partially on the other end you're talking to. If I'm in the eastern US and communicating to a server in India, there's no way I'm getting < 100ms RTT (I'd be happy if I could get 200ms). On the other hand, if you're using legacy satellite internet like Viasat you might have 600ms latency talking to a server in the same city. (Starlink does better, but it still imposes sizable fixed latency costs on the order of 10s of ms.)
If I had to come up with latency metrics, I might initially suggest "< 20ms within the same city; loaded latency must not impose a penalty greater than 40ms" (I'd love for these numbers to be lower, but they're already very ambitious for Starlink due to physical characteristics of the satellite constellation).
edit: of course, in retrospect, that's my urban bias coming through -- what if you don't live in a city? or consider Hawaii -- is it good enough to just consider latency to Honolulu? Or do you want latency to www.google.com (the nearest frontends are in Los Angeles)?
A lot of city-based fiber dwellers think that just running fiber out to rural areas actually improves performance more than it does. Moving data and interconnectivity out there helps more.
They really should get out more. Or, perhaps, learn how to insert artificial, rural like delays into their networks in the city, to grok it.
(I am very busy today at the metro connect conference in ft lauderdale today, please expect sparse replies if any. If any of y'all are actually here today, I am outside with Bandwidth is a lie t-shirt on!
My example of Hawaii being 70ms away from Google was something that I knew about on paper but didn't truly appreciate until I was there on vacation a few years back (where the hotel wifi injected another 30-50ms on unloaded connections).
Moving interconnectivity to rural areas is tricky because most peering is where you can convince other network operators to meet you. This is a bit of a catch-22, where if you're not big enough then most network operators won't bother, and you can't get big enough unless you get these network operators on board. There's a secondary issue where just having peering isn't useful without servers (or vice versa), and you often don't have the economies of scale to justify spending millions on routing equipment, servers, etc. if you don't expect to serve many users there.
The compromise that some of these big operators (Google, Facebook, Netflix off the top of my head) make is to install cache nodes inside ISP deployments for last-mile delivery. My understanding is that this is generally a win-win for all parties involved: ISPs don't have to build out as much backbone / peering capacity, Google and co can get away with smaller deployments and piggyback off of the ISP's existing network presence, and end users have better quality of experience.
That said, for the most part, Google's cache nodes don't serve www.google.com because we generally view them as less secure, and we would have a very bad day if we had a compromise of any high-value TLS certs. (We mostly use them to serve YouTube videos, since that's where we derive the most value from bandwidth offload.)
Especially because an extra 200 miles of fiber only increases round trip time by 3 milliseconds. How far are signals going before they get routed in the right direction?
Too true. I played so much WoW on 56k that when I finally moved into an area that had high speed internet I basically had to learn to play all over again. IE stop compensating for pings in the ~300ms range.
I love considering latency now.
edit: Cable is still hooked up and it's 17ms-20ms. Not awful.
LTE: GSMA is 25ms-33ms, GSMT is 25ms-51ms.
I don't have satellite, DSL, IDSN or dial up to compare.
For GSMA, I'm ~7k feet to the tower on band 66.
Part of the problem may be that companies who own network infrastructure, and get paid for data usage, are also the ones that are the largest content providers.
This also comes with an electricity cost. We regulate efficiency for refrigerators, it might be time to add some sane limits to the largest content providers, which will also improve connectivity for those stuck with 2 Mbps.
On high latency Internet with otherwise OK speed, pages of "only" 3MB can take several seconds to load.
when did we stop caring about optimization and good technical design?
Of course there be dragons. Oversubscription in various parts of the network can have interesting behaviour. Especially when coordinated user behaviour or complicated packet processing is at play. Think encapsulation or deep packet inspection. My worst case experience includes Cisco Nexus M1 32x10Gbps line cards maxing out at 1-2 Gbps throughput and 1+ sec worst case latency because of OTP. This is a datacenter core switch. And F5 WAF eating random packets because something looks like a VISA card number, causing a retransmit, that again shows up as high latency at higher levels.
too many people in his world think tcp is a function call.
For a while, Google was penalizing slow-loading pages, but that got tied into AMP, which everybody hated.
It only takes 25 Mbps to stream 4K TV, after all.
internet became fast, but instead of faster load times, we got animated UIs.
hyperbole obviously, but not entirely wrong.
I usually explain this to non-techies by inviting them to imagine they own a train full of 1TB drives. This way they can move a million terabytes to a neighbor town in an hour. Naïvely this translates to 277.8 Tbps which seems super fast but there is a catch :-)
However the technology exists to aim for 20ms - even 4ms - of both idle and working latency today, and the 100ms target - as well as the 95% cutoff for measuring latency, are nutty. It is not every day that key members of the ISP industry actually call for stricter metrics - If you cut the ISP working latency off at 99% instead - well, jason livinggood from comcast reported:
...
Looking at the FCC draft report, page 73, Figure 24 – I find it sort of ridiculous that the table describes things as “Low Latency Service” available or not. That is because they seem to really misunderstand the notion of working latency. The table instead seems to classify any network with idle latency <100 ms to be low latency – which as Dave and others close to bufferbloat know is silly. Lots of these networks that are in this report classified as low latency would in fact have working latencies of 100s to 1,000s of milliseconds – far from low latency.
I looked at FCC MBA platform data from the last 6 months and here are the latency under load stats, 99th percentile for a selection of ten ISPs:
ISP A 2470 ms ISP B 2296 ms ISP C 2281 ms ISP D 2203 ms ISP E 2070 ms ISP F 1716 ms ISP G 1468 ms ISP H 965 ms ISP I 909 ms ISP J 896 ms
...
As for the insane 95% FCC working latency cutoff today - in terms of real time performance - what if, you got in your car, for a drive to work, and your steering wheel failed one time in 20. How long would you live? If we want a world with AR, VR, and other highly interactive experiences, 99.9% of no more than 20ms of consistent jitter and latency should the goal for the internet moving forward, and ideally, 4ms.
What is this "25 * 3" and "100 * 20" notation? The next sentence just goes on to say "1 Gigabit" and "500 Mbps" directly.
Having a standard that ignores that is less than useful. It's the standards body showing disrespect for the people it's supposedly creating the standard _for_.
