Yeah funny you mention this, as the idea of vending infra resources as a common package runs into the issue of how to know if every consumer will be able to successfully ingest updates. From there you have the issue of "blast radius", how to handle rollbacks, etc. The complexity goes up, and now you have a problem. I think the answer is very sophisticated build systems, as well as CI/CD pipelines, which Amazon has. Upfront the team pushing changes can know if the deployment will work, before it gets to production. But this requires a huge amount of tooling, which I don't think most companies have. It also requires alot of cloud expertise.
"I guess the main question I have is what makes this different from any other abstraction?"
For one its all unit testable, with type completion (CDK is generally done in Typescript). The code has to compile, you can diff the new infra with the current infra and see whats changing. You have version control. You can use other design patterns from other teams, much like you would for OOP code, except now its infrastructure. A queue is just a variable in a CDK code base. It demotes infrastructure from a complex thing to manage, into something like a code variable.
