If you're interested in similar edge cache programs:
https://openconnect.netflix.com/en/
https://www.facebook.com/peering/ (though I don't see FNA specifically mentioned there)
https://peering.google.com/#/options/google-global-cache
https://www.akamai.com/us/en/products/network-operator/akama...
(I suppose the flippant argument is 'the end user' but that's not quite what I mean...) :)
This lets me deduce the ISP is way over-subscribed on their upstream internet connection, but also has a Netflix appliance.
What's frustrating is if they complain enough, the company will send a tech out who will "adjust" their antenna, or suggest it's a line-of-sight problem and they need a bigger tower, should cut down some trees, etc.
It seems like their niche is as an easy to deploy, more traditional looking CDN that can run out in the RAN, which does still have value.
The main requirement is how much traffic you need to be doing with the peer before they give you a rack - Apple states 25Gbps; more mature programs like Netflix are 5Gbps. Sometimes that's negotiable and they'll hand them out for even smaller traffic amounts (I've heard less than 1Gbps for some of those listed, though won't mention which ones). Like most things in this area of networking, there's a lot of variability and personal contacts involved.
Everyone also meets up twice a year at NANOG.
The first kind is the ones that Apple wants - they are the likes of Comcast/Optimum/etc. There are the ISPs that have lots of eyeballs and have captive audience. If Apple has shitty connectivity to them, it is going to be bad for Apple ( because consumer cannot replace such ISP - there's no real competition in those markets ). These are also ISPs that happen to peg their transit PNIs or free peering PNIs as much as possible forcing others to to buy paid peering to have non-shitty connectivity to them. These ISPs are going to charge Apple for colo, power and bandwidth and Apple is going to pay and it is going to pay through the nose ( just like Netflix does and just like Akamai does )
The other set of ISPs are the ones that want Apple a lot more than Apple wants them. There's no way Apple will pay for access to their customers. Those ISPs are going to give Apple space, power and even bandwidth for free. Hell, they may even have to pay Apple.
Source: did that for both ISP side and content provider side (at different times)
Edit: the flip side of this relationship is also interesting - if Apple or whoever doesn't offload enough of their traffic to the rack, then it isn't cost effective and can really annoy the ISP. I've known some ISPs to boot these caches out of their network when the related company wasn't utilizing it effectively.
No peering on immense libraries of HD video content?
Exactly how SSL is handled (or not) varies by provider, but one thing I'll mention is that these will typically never have a cert so important that it can't be easily revoked, and the most important data will likely not be flowing across them - exposure is very limited. Using video streaming as an example, one option is to only do TCP termination for example.com (or to not even terminate that domain on the local cache, but back at your main datacenter), then use subdomains with individual certs for the local cache (eg. isp1.cache.example.com). In that case, service calls like login, retrieving the manifest, etc. are secured by the certs you're keeping in your primary dc, then the manifest has a set of https://isp1.cache.example.com URLs pointing to the local cache only for video segments.
Another tricky aspect is making sure that your main network treats them as untrusted so someone with local access can't use it to get a foothold into the rest of your infra.
I think this is specifically addressed with the introduction of TLS Delegated Credentials[1]. This allows the CDN edge to use a very short lived credential in the place of the certificate's private key.
It's already supported in evergreen browsers and in certificate profiles from commercial CAs like Digicert.
Apple own the intelligent layer, the one that hold the API. Once you query those, it answer with the location and the hash, which allow you to download it from the distributed box and safely verify the content.
- TLS is still important to stop tampering of video content or images, as well as user privacy over what content was specifically viewed.
- Some ISPs have (and still do) intercept plaintext video content -> transcode to a much lower bitrate -> cache that for their users. That hurts the content provider, as they lose visibility (logs/metrics), and the user who may suffer a reduced experience that the content provider can’t easily fix. End-to-end TLS solves that.
Note that I understand the importance of securing private keys in general. I'm just asking about ISPs because you specifically mentioned them.
It's not necessarily that the ISPs will want to break into equipment, just that they are the weakest link bad actors can use to target Apple/Apple Customers.
Also, there's a surprising number of colo facilities that are pretty lax about who is allowed to come and go.
Being able to sign some things as Apple / MITM some Apple URLs strikes me as interesting enough to get state-security level interest, at which point you fall back to ISPs being made of people, and people being coercible and bribeable.
They announced Keyless SSL way back when: https://blog.cloudflare.com/keyless-ssl-the-nitty-gritty-tec...
I'm a fan of Facebook's take on this, though: https://engineering.fb.com/security/delegated-credentials/
Also see, Google's answer to JWTs: Macaroons-- https://hackingdistributed.com/2014/05/16/macaroons-are-bett...
A single domain name and DNS to route is uncommon because it doesn't give you fine-grained control of load - you need to be mindful of the rack's capacity, and you also need to make sure that most of that ISP's customers go to the rack/people who aren't that ISP's customers don't go to it.
Anycasting isn't going to be great for traffic management or long-lived TCP conns, and if you can avoid the complexity of each rack needing a bgp session into the ISP's network you're going to be much better off.
Typically this is going to be directly routed to the rack via a unique DNS name after some form of service call.
Also they could just be signing content at the application layer like APT does, and the transport is plain HTTP.
For example IMGX uses Macs in its data center for image processing.
https://photos.imgix.com/racking-mac-pros
I remember reading somewhere the performance they got with CoreImage was worth the extra hardware cost.
> In the end they are now effectively at 200Gb/s encrypted video streaming from FreeBSD per server.
[1] https://www.phoronix.com/scan.php?page=news_item&px=Netflix-...
RHEL for WebObjects and other servers.
Alas, it is but a dream.
https://en.wikipedia.org/wiki/List_of_Internet_top-level_dom...
I wish there was something up at https://google.google or https://apple.apple
$ whois goog
% IANA WHOIS server
% for more information on IANA, visit http://www.iana.org
% This query returned 1 object
domain: GOOG
organisation: Charleston Road Registry Inc.
address: 1600 Amphitheatre Parkway Mountain View, CA 94043 US
address: United States
contact: administrative
name: Domains Policy and Compliance
organisation: Google Inc
address: 601 N. 34th Street
address: Seattle, WA 98103
address: United States
phone: 1 202 642 2325
fax-no: 1 650 492 5631
e-mail: iana-contact@google.comSadly it always defaults to the first logged in user, which makes it almost entirely useless to me.
Do they have something against using dashes?
That's what they do elsewhere:
https://support.apple.com/en-us/HT202471
https://support.apple.com/guide/terminal/script-management-w...
https://support.apple.com/en-us/HT201954
For less ambiguity, I should have used the term "hyphen," but imagined it would be clear.
"Apple-supplied and -managed hardware"
The product they're presenting isn't significant, and other people had already made fine points on it, so I pointed out something that I care about that was on-topic instead.
I’ve never seen a download from the store faster than about 50 megabits per sec even on a gigabit line.
Popular app and OS updates can be cached on a local Mac, with the recommended machine being on the network using a wired connection and always connected (desktop/Mac mini)
It has the same information as Apple's official documentation, but it highlights the advanced options.
> If your business meets these requirements, request an invitation.
Doesn't that defeat the purpose of being "invitation only" which, to me at least, implies the other party knows who they want to invite? That is, invitation only implies hand picked, or pre-chosen by some prior criteria to me. If it's exclusive to select ISPs that meet the criteria and they have to apply, why not just say that instead of the using wording that requires additional explanation to get past people's likely initial interpretation?
The whole point of saying the party is invitation only is so random people don't ask you to come.
I think the normal way this is normally done is to say "we're being very selective with who we partner with at this point. Please apply if you think you qualify and we'll get back to you."
That they've chosen not to do the obvious seems purposeful, and that it was done so jarringly on purpose is odd.
Imagine that your Internet service is just an Ethernet cable that goes to a router in a datacenter. This is Apple offering to plug their servers into that router. Now you can get to their servers without going out to "the Internet" via a Tier 2 or Tier 1 ISP. That is where the word "internet" comes from -- interconnected networks. More connections is more internetting.
This is all super common. Many big companies are happy to peer with small ISPs if they're already in the same building.
Edit to add: The edge cache thing that this article is about is similar, but not quite the same as what I'm describing. Instead of connecting you to their network, they just put some of their servers in the same datacenter as your network. Even less latency!
This sounds horrifying.
For all other apps (which load static image assets and much smaller dynamic response payloads), meeting 25Gbps minimum peak is going to be a challenge.
Let's do some rough math. Let's say your app needs to load 10MB of assets in every user session. Let's say your user's network speed is not a constraint. Then least number of concurrent users needed to drive 25Gbps of traffic is 25Gbps/10MB = 313 new user sessions per second. If you want to sustain this for 5 minutes or so to register as peak 313 users/second * 5 minutes = 93900 concurrent apple user sessions. Let's say your users realistically have 10Mbps of speed, we will have to multiply 93900 with 8 (because it takes 8 seconds to load 10MB with 10Mbps speed)!
[1]: https://peering.google.com/#/options/google-global-cache
[2]: https://www.akamai.com/us/en/products/network-operator/akama...
Why now? Apple has had hundred of Millions iOS users for years and fast approaching a billion. Why didn't they do this earlier? Or they did but it wasn't public ?
What are the chances this is a Mac Pro rack, even though it is highly unlikely to be running on macOS ?
Do they Cache iCloud Backup, Photos and upload with these Edge Appliance? Same as macOS Server?
Q: are you getting paid for allocating such a cache? Or should you feel honoured that Apple thinks you are eligible to freely distribute their content?