The firms that provided instantaneous-seeming settlement got the orders. And settlement failures were rare and correlated enough that avoiding them wasn't a differentiator. Then we got centralized clearinghouses and deposit insurance and the risk fell so far into the background that it's entirely novel to many market participants today.

> why these transactions aren't being netted and conducted as close to real-time as possible

It is technically difficult. It provides less room for correcting errors. And because real-time payment rails are fundamentally more expensive than batched rails, and if your payments aren't in sync with your settlement you've got a credit component somewhere.

> seems like a significant vulnerability

It does, but I don't think it is. Clearinghouses spread risk across a lot of people. They also help early identify daisy-chains, e.g. a single broker sold ten shares to ten brokers who then sold it to ten more fifty times and that single broker hasn't settled yet, and loops, e.g. I owe you and you owe Larry and Larry owes Bob and Bob owes me, of settlement risk. That's why Dodd-Frank moved many derivatives onto centralized clearinghouses.

That said, we've only had these since the 1970s. Counterfactual: we didn't have the technology to do this until the 1970s.