Have you talked to many landlords that own a dozen units? It's my understanding that owning so many units is hard, requiring you to either work full time or hire a property management company, and you incur a serious risk of massive costs from misbehaving tenants.
You're right that it's not an uncharacteristically complicated business, but all businesses are hard! The idea that business owners sit there sipping whiskey, living an unburdened life like an aristocrat, is almost always false.
Even without debt to service there is are still property taxes to pay and the opportunity cost compared to investing that capital in something else like 30- year treasury bonds that pay ~1%
