Mortgages are backed by the asset you are buying. $400k for a house is backed by the house. On top of that, almost every bank can sell off those mortgages and get the cash back immediately.

So, you are correct, I would not lend hundreds of thousands unsecured to a random person, but if I had the disposable capital, though, I WOULD buy a house with that money, and price in their credit rating to their interest rate, and turn around and sell it to Fannie/Freddie and recoup the principal that I lent.

Banks aren't on the hook for any of this (except for mortgages that don't conform, but most banks don't touch those, you have to go to brokers for those).