The Principal PM in charge of the "regionalization" effort was asked in a Q&A "why didn't we just switch to IPv6?".
Her answer was something along the lines of "The number of internal networking devices we currently have that cannot support IPv6 is so large that to replace them we would have needed to buy nearly the entire world's yearly output of those devices, and then install them all."[0]
It's easy to presume malicious intent on the IPv4 front from Amazon, but with so many AWS systems being on the scale they are at, I find it easy to believe that replacing all of the old network hardware may just be a project too large to do on a short timescale.
[0] - At least, that's my memory of it. I'm sure that's not an entirely accurate quotation.
I’ve got a slight suspicion you were given some bullshit or at least a creative treatment of facts e.g. everything had IPv6 support but FUD-filled network engineers didn’t want to turn it on.
Most network devices I’ve encountered were dual-stack way before anyone I knew seemed to care about actually using IPv6 — I always assumed it was added for US government/military requirements.
There were also other reasons given, like the amount of internal software that used e.g. IPv4 addresses. Also, AWS likes to have 'lots of small things' instead of one big thing (regions, AZs, cells, two pizza teams, no (official) monorepo) so regionalization was part of that.
Another big reason for regionalization, other than IPv4 exhaustion was that AWS promises customers that AWS regions are completely seperate, but with one big giant network, it turns out there were all sorts of services making calls between regions that nobody had realized. I have a couple of funny examples, but that might make me too identifiable :)
Tables were relatively large internally because AWS was all in on clos networks at that point. And the devices used to build those clos networks were running Broadcom ASICs, not Cisco or other likely vendors.
EDIT: But maybe bugs, IDK.
FUD sounds like a mean way to say unproven in production
You're right about the cost and need to replace network equipment being one of the strong reasons why they didn't. Amazon used its own in-house designed and built network gear for a variety of reasons (IIRC there's a re:invent talk about it), which I'm sure is probably still the case. Every single one of those machines had fixed memory capacity and would need to be replaced to bump up the memory sufficiently large enough to handle IPv6 routing table needs etc. What they had wouldn't even be enough if they'd have chosen to go IPv6 Only (which you couldn't get through except via dual stack IPv4/IPv6 anyway).
I'm not privy to details, but I recall once when a mandate was issued to a Java platform to remove an outdated encryption protocol (mandated by Amazon Infosec). The change was made and rolled out with little fanfare.
A few weeks later, a large outage of Amazon Video (which used said platform) occurred on a Friday evening. Root cause? The network hardware accelerators were only setup to use that outdated protocol, which in turn meant that encryption was happening in software instead. Under load, the video hosting eventually caved.
Might be specific to the hardware used for Amazon retail, but it reinforces the point of their home grown (and now aging) stack.
If that is the case, then Amazon should hold off on charging for IPv4 on a short timescale until they have replaced all the old hardware and can support IPv6 internally everywhere.
surely they started the process...
right? i cannot imagine AWS just sticking head in the ground and ignoring this...
About 18 months ago, the requirement came that federal agencies are required to be IPv6 Only, dropping the dual stack. IIRC they have until 2025 to do that. This has the neat effect of forcing all vendors to make IPv6 a first class citizen. The extra little fun from this is that it applies to the military JWCC contract that all the major clouds have been trying to land. The timescales of JWCC meant that initial offerings are pretty bare, but that won't be allowed to last.
There is no reason any company of any size should run out of IPv4 addresses internally, IF they are doing proper IP management. If I were to wager a guess I'd say there was a lot of waste going on, issuing /24s or larger to teams when all they need are /29s etc. It adds up over time. Once they exhaust private IP space they can always buy more at auction. They are Amazon after all, there's no shortage of money. This is just mismanagement of resources.
If you wanted to assign a single non-routable IP in the 10/8 space to each of those cable modems, they would be 13 million IPs short.
I'd imagine few service teams at Amazon would get very far with a /29, let alone a /24, if they have to put all their stuff on that.
The better reason is the regionalization was probably a way to decrease blast radius in case of a service failure.
Also, AWS definitely did not regionalize all their services in 2016. IAM and certainly not DNS/Rte53 (part of the reason why they had their massive failure in US East 1 2-3 years ago)
Sounds like a perfect opportunity for a market upstart to start out v6-only...
The world is bigger than your apartment.
In the world of their recommendations, even the concept of a "public ip address" is a red flag, and AWS even recommends (for an added cost of course) tooling to flag and "mitigate" them. These provide a strong lock-in effect when customers spend effort to build the complex infrastructure for them in the name of security, even though in reality they hurt security through unnecessary complexity, addressing ambiguity, etc.
I've lost track of all of the "Private Endpoints", "Private Links", "Service Endpoints", "Private Resolvers" and "Virtual WAN" products they've introduced... all to make IPv4 work at scale.
Literally none of those products would be required if they had just made IPv6 work properly.
Instead, they NAT IPv6, so you can't even use it to avoid the NAT forced upon you by IPv4. They also release new products -- in 2023 -- that don't support IPv6 and likely never will.
There is still an entire page[1] of listed limitations in the docs: https://learn.microsoft.com/en-us/azure/virtual-network/ip-s...
Think about how insane it would be if this was the IPX -> IPv4 transition. Imagine if this page said "IPv4 limitations: All VMs must include at least one IPX address, etc..."
Sounds nuts, right? I was migrating customers to IPv4 from IPX in 1999, and IPv6 support materialised in about the 2001-2003 timeframe. It's been decades, but it still feels like 1999 and migrating Novell NetWare where we had to have IPX+IPv4 because "not everything supports IPv4 yet".
[1] This is definitely not all of the limitations. Most of their PaaS products don't support IPv6. Hence, any IaaS+PaaS solutions must use (mostly) IPv4.
If you want port forwarding, they recommend that you do... something. It's not clear what; what you can find on the internet is mostly just people complaining that they insisted to customer support that they needed port forwarding, customer support said they'd do that, and port forwarding still doesn't work.
But, intriguingly, it turns out that a Comcast router will also assign every device on your network at least one public ipv6 address. They also firewall all incoming ipv6 traffic, but unlike the situation with port forwarding, you can disable that firewall on the router admin page. (You can't put up your own firewall.)
Any new competitor would have to cough up a huge upfront cost for IPv4 addresses, if they could even get them at that scale.
I'm not sure what you've been reading, but the concept of a private link has absolutely nothing to do with IPv4 vs IPv6. In fact, practically all your remarks don't involve the issue at all.
The most charitable interpretation of your comment is that you're making the mistake of conflating any application involving a virtual network as something caused or involing IPv4.
I don't understand why, but until some large tech company starts pushing for end to end addressability as best practice, I have no choice but to follow the conventional wisdom to avoid throwing up red flags.
> I don't understand why […]
Excluding Microsoft, all the others find it easier to have as a checkbox to make it easier to confirm that "internal" hosts are actually (theoretically) internal since RFC 1918 isn't allowed outside.
Of course most companies' firewall and NAT rules are probably all sorts of complicated once you get to a certain size (never mind stale open-rules which were never cleaned up), so a bunch stuff is probably accidentally exposed. Also, most attacks are probably from compromised clients nowadays, so even internal hosts need to be locked down as the castle-and-moat security model isn't (as) valid.
But having "internal-only" hosts is low-hanging fruit on the security checklist.
I will resist the urge to be snarky at your expense and politely point out that exposing your LAN to public routing tables is madness, from all perspectives.
It brings no benefits and carries huge risks.
There's nothing preventing you from having a private network using unique address space that's either blocked from accessing the internet via a firewall on a router or just plain not even routed. You could even use ULA networks with stateless prefix translation to avoid using GUA addressing for your private network.
The sad part is that IPv6 support is abysmal on every cloud so just migrating to it imposes serious limitations as addressed by the blog author.
If your (default) gateway from one network segment to another network segment only has one rule, default-deny, then it's not a problem. If you think that's not enough, then use IPv6 ULA (fd00::/8).
But why should the incompetence of some customers limit what all customers can do?
AWS has no incentive to support IPv6 from a sales perspective.
How do you hurt security by preventing external access to your internal services?
In case not: it's about doing it the wrong way (excess complexity and ambiguity -> hard to understand/analyze/monitor). Using globally addressing is orthogonal to controlling access to your internal services - you can do it using firewalling or various other means. Eg on AWS you get a default-deny firewall.
People never interact with these protocols directly and use a layer of indirection such as a HTTP/2 client for HTTP, and the same applies for IPv6: use DNS (or your hosts file).
IPv6 doesn't seem "human usable" sometimes in large part because you aren't actually using it. People adapt. The human skills in pattern matching are robust: there are new tricks to learn, but there were always tricks to learn. (IPv4 addresses aren't "human usable" either if you sit down to truly assess absolutely how many RFCs are involved to build the patterns "everyone" has internalized that seem "easy". They are easy because they are familiar, because you use them often, because you've already adapted to them.)
More like it gets rid of band-aids
Computers do.
The inside of my service is not the internet, even though my service may be exposed on the internet; why would I want internal implementation details externalized?
Thank endless lobbying that make legally mandated monopolies a thing in this arena. They did it with phone companies too.
It's not "just Comcast" by happenstance. It's "just Comcast" by legal regulation.
Funny that you complain about Comcast, when I was with Comcast I actually had native IPv6. With my current provider I only have IPv6 through NAT46.
Maybe Starlink is another option. It has some drawbacks like reduced performance during heavy rain, but I've seen some positive reviews as well.
Most people don't realize there are two IPv6 internets right now, the Cogent side and the Hurricane Electric side. Both are equally sized and refuse to connect to each other, so you need to know that and either buy transit from both or buy transit from a network that buys from both. At least one major provider I know of is still running v6 over tunnels. In many places your v6 traffic is taking suboptimal routes, whereas an enterprise network may have v4 connectivity at each datacenter, v6 all gets sent to that one box under Dave's desk.
But we continue to measure v6 adoption at places like Google and Cloudflare where dedicated teams make sure packets arrive and pat ourselves on the back.
Cogent engages in peering spats on IPv4 too; this dynamic is not new with or unique to IPv6, or limited to Cogent/HE. The lesson here is to not go singlehomed under Cogent, not to reject IPv6.
Just let me strongly associate identities with my workloads and apply policy indicating which workloads should be able to send data with which other workloads.
How data gets from one workload to another should not even be my concern, just make it happen.
I don't think it's that crazy, it's just formally standardizing where we're already going.
IPv6 just works. Amazon, Github, and Azure don't. That's not really a problem in most cases (very few people go IPv6 only because it's just not necessary with CGNAT, and even then network translation tricks can put up IPv6<->IPv4 bridges easily). In Amazon's case, they don't even need to bother setting up a real network, they could abuse an fd00::/8 network to mimic their 10.0.0.0/8 network if they wanted to.
Amazon is terrible at implementing modern standards. Just look at how long it took them to support DNSSEC on their domains, and even that didn't exactly roll out great the first time.
Only via the herculean efforts of a bunch of people having to literally reinvent the world to deal with it. Everything needs IPv6 support specifically. It's such a mess, if IPv6 has just been identical to IPv4 but with larger addresses we would be on it by now. But no they had to make it their religious crusade to eliminate NAT (and now we have NAT66 so clearly a winner) put IPSec in there which is hilarious in the era of Wireguard and eliminate DHCP which is actually insane and makes a stupid number of assumptions about hosts being able to communicate with one another and actually complicates DNS registration.
Can you imagine how trivial it would have been if you could support both v4 and v6 by just supporting v6 and having 0::v4addr be literally equivalent to ipv4? It would be more difficult to not support v6.
There is a reason well resourced companies like google cloud have been slow w IPv6 - and it can be even more hair pulling in smaller settings.
Every other episode seems to be about a different new RFC that's replacing another RFC because the original ended up having a bunch of holes and edge cases. That's somewhat understandable for a new protocol but the protocol have been around for almost 30 years and is just so overly complex that it's rife with these situations.
As an example the most recent such episode was on rfc6724[0] which describes these convoluted algorithms systems are supposed to follow to determine which of their many assigned IPv6 addresses to use for a particular connection and also which of many possible destination addresses to use. Just reading the introduction makes your eyes water with how overly complex and prone to nasty failure cases (what if the source address isn't what you expect and somehow the connection routes around your firewall?) the whole situation they've created is.
I use IPv6 since 2006 and I just can't see how it can give you "a massive headache". I read a blog post about how overly complex HTTP/3 is. Better ignore it forever and never implement it then. ;-) Also, which successful RFC protocol doesn't have a see of follow-up RFCs?
The headache are vendors that still refuse to properly implement IPv6, in 2023.
This distorts the market in eyeball networks and hosting - the former are under little pressure to offer v6, and new entrants to the latter can only offer v6. Competition law in the EU works (I think?) on the principles of consumer benefit and market fairness. On that basis, I'm left wondering why this has never been pursued by the EU's competition authorities.
The European Commission did advocate for IPv6 use, but, the EU being the EU, motivated their recommendation by complaining that law enforcement had issues tracking down people behind CGNAT, and made clear that they wanted every IP address to point to a specific person for law enforcement reasons.
So, yeah, I don't think we should let the EU deal with the specifics of network infrastructure just yet.
I think it's hard to make an economic argument for IPv6. Yes, it's obviously a superior technology, but ISPs can CGNAT for cheap, consumers can still access every server, and the €40 per year a business needs to pay for an IPv4 address isn't exactly breaking the bank either.
Perhaps the EU should force the issue, but I think countries like Lithuania ,where there is practically no IPv6 available (0.58%, according to https://stats.labs.apnic.net/ipv6-zoom, but who knows how accurate that is), will protest any mandate that will force their ISPs to buy new networking equipment.
Not really that cheap. While CPAEX is CAPEX, OPEX is still a thing and operating CGNAT requires efforts. Also some (most?) CGNAT implementations are buggy and is not a good user experience, even for users who don't understand the concept of IP at all.
The problem is customers don't like CGNAT. You can't run Animal Crossing on Nintendo Switch in network mode as a host if you don't place the Switch as a catch-all in the DMZ.
Wish I were joking here - especially due to the security risk involved in running something in all-ports-open on the Internet - but Nintendo doesn't seem to (want to) run STUN/TURN servers.
To me, the question is: what stops me today from spinning up an IPv6-only website and having 99% of the world's browsers use it? If the answer is "nothing", then AWS shouldn't be forced to offer IPv6 (or only IPv6) - IPv4 is just part of what they offer customers. If the answer is "these 7 things" then those 7 things need to be fixed[0] before we pay civil servants to try and force companies to do things that they barely understand.
[0] E.g. in the UK, it's some of the big ISPs that don't do IPv6, so there's no point forcing someone way upstream (and way more optional) in the process to do something https://www.ispreview.co.uk/index.php/2021/11/update-on-ipv6...
Virtually every single application at the company I work at deploys into VPCs without public IPv4 addresses - this seems like a ridiculous claim.
If your target VPC has neither PrivateLink nor public IPv4 connectivity somewhere, I'm not sure how that would work; I'd love to learn how that was built.
This is the definition of cloud bloat. The fact there are tons of systems abusing that kind of architecture probably justifies charging for IPv4.
How nice it would be if you could just create a bunch of load balancers and all that actually meant was that it was just adding config profiles to a single physical load balancer and kept them truly isolated? Right now it's really annoying because load balancer config is global state and everyone has to either be kind neighbors when adding themselves to it or manage them top-down.
You have to set a load-balancer-name annotation https://kubernetes-sigs.github.io/aws-load-balancer-controll... to tie everything together to one load balancer. There is a downside where you have to have a few other annotations be the same value across your ingresses, but once you work around that, you're good to go.
Who is a good provider of ipv6?
For servers, there are plenty; AWS Lightsail, Hetzner and Vultr both provide IPv6 out of the box. If you don’t have an ISP which provides IPv6, you could use a server and set up a wireguard tunnel for IPv6 connectivity.
I don’t know that this is true based on google’s IPv6 adoption data: https://www.google.com/intl/en/ipv6/statistics.html#tab=per-...
It seems like developing nations have the worst IPv6 adoption, at least by a cursory look of how there’s very little green in Africa, for instance.
I think ISP’s in countries with small IPv4 blocks just use CGNAT.
Read the burst/throttling page carefully before you choose this product.
https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amaz...
One can use some public NAT64 services, but that's not very reliable for anything serious. https://nat64.xyz/ . AWS chargers arm and leg for NAT gateways traffic, and I don't think it's possible to configure them so that they only intercept traffic to ipv4-only hosts (please let me know if i'm wrong).
Other than this being IPv6-only in AWS works flawlessly and is cheaper (free egress gateways for private networks). As long as you don't care about IPv4 of course - that's given.
That's because all mobile data connections are on IPv6
Maybe not in AWS, but there are Unique Local IPv6 addresses in fc00::/7 and NAT66 if you really love NAT!
I just recently heard that MS apparently has built everything IPv6 on Azure around NAT. This is so weird.
Yes, for some who understand and have tested it, we do not like or want IPv6, it has no privacy when the device's IP is public on the Internet 24/7. No privacy extensions fix this. Test it, it's not difficult, disable IPv6 in your home router, wait a few hours, the kids will be complaining their search results are messed up, that's just the start of the indication that the advertisers now have a veil where with IPv6 they had clear fully trackable results.
I have a WireGuard server that was dual-stack. I turned off IPV4 to see what would happen, and it kept chugging along very nicely.
Example: we run a bunch of endpoints on ipv4, but get ipv6 IPs in our logs. How? Are there 6-to-4 translators out there at ISP edges?
Unknowns in networking are bad.
Proxies. Your logs are too trusting of X-Forwarded-For headers.
That said, until the cost of IPv4 becomes really huge, few organizations are going to suffer the effort-cost of embracing IPv6.
I would argue that the IPv6 sales story is unmemorable|unclear|weak. Also, it is arguable that most IPv4 addresses are wasted.
I currently use Hugo and my blog is in Markdown in git, but the theme is pretty heavy-weight, and I like this look of the page in OP; Looking at the source, it's so minimal!
This is not so easy to do with a IPv6 address, AWS tends to want to keep it the same
If you are going to mention gateways or other methods make it work please just stop. No end-user is going to do that, or rather no appreciable amount of end users are going to do it. If your fix starts with “why don’t you just…” then please stop living in a fantasy world.
I was excited for IPv6 when it was announced, I was excited years later, I was excited a decade later, now I’m just tired of it. 2024, year of IPv6 and and the Linux desktop, ok sure. My ISP, literally the best available in my area and fairly cutting edge in every other aspect, has zero IPv6 support.
While the idea of every device having its own public IP address was attractive to a younger me, I look at it with a bit of horror now. The privacy/security aspects alone are staggering and you rarely want your device to be publicly available by default. I’m not going to exceed the 16M+ limit of 10.0.0.0/8 so I don’t see why I would ever want to use anything but IPv4 internally for my sanity. Are STUN/TURN servers fun? Is needing some central server ideal? No but the alternative (everyone can talk to everyone directly) makes my head hurt with the implications and footguns.
At the end of the day I’ve started disabling IPv6 as a matter of course. Leaving it on is a landmine I’m laying for my future self. I’ve dealt with too many issues directly myself or for clients/customers which end with “let’s try disabling IPv6, oh it’s working now?” (on my end or theirs) that I’m done. Something drastic would have the happen to get me to change that thinking and seeing how it’s been over 2 decades and major websites I use daily still don’t support IPv6 I’m not holding my breath.
Every time someone brings up this point, I have to assume that they know nothing about IPv6 but the superficial things.
If you work with IPv6 long enough you will remember the addresses, we all remember 192.168.0.* through years of typing it repeatedly and looking at it. Not because it is easily to remember. I can already recall 2606:4700:4700::1111 or 64:ff9b::101:101 from memory.
>My ISP, literally the best available in my area and fairly cutting edge in every other aspect, has zero IPv6 support.
This is almost exclusively an Euro-American phenomenon. I am not sure why you are lashing out on IPv6 when it's the ISPs' fault. In most East or Southeast Asian countries we are looking at double-digit % of IPv6 deployment, the moment you click on the IPv6 checkbox you get IPv6 connectivity here.
>The privacy/security aspects alone are staggering and you rarely want your device to be publicly available by default.
Another one who mistakes "having a globally unique address" with "public accessibility". Boo.
>“let’s try disabling IPv6, oh it’s working now?” (on my end or theirs) that I’m done
Just say that you are lazy in fixing IPv6 problems. I have found that lots of old networking guys would say "it's defo my fault somewhere" when IPv4 fails but when it comes to IPv6 it's always IPv6's fault somehow. Protip: most of time it isn't.
ECS does support dual-stack IPv6, but most other services do not support IPv6 at all.
(Technically S3 does have a separate dual stack endpoint, however it doesn’t really help as I have to change application configuration anyway to deal with this change.)
[0] https://blog.devopstom.com/ipv6-only-ec2/ [1] https://nat64.net and http://v4-frontend.netiter.com
AWS does make it easy to fuck up with its default settings. Subnets that auto-assign EIPs for every instance attached to them should not exist, period. And neither should RDS instances or anything else be reachable from the public Internet by default.
This means that everything else in your VPC has to be dual stack if the lambdas talk to them.
AWS Lambdas don't actually run in your VPC, there is an ENI that is exposed to your VPC that belongs to the VPC-equivalent in the Lambda space.
If you have to run dual-stack just to accomodate lambdas, then you may as well run IPv4 anyway.
I actually keep a cheap Comcast connection as a second WAN just to get IPv6 enabled on my home network. (And also because I live in the woods in New Hampshire and having two ISPs means I have fairly ok uptime)
