You still lose relatively though because the next person with their money in the market made some %

Land and real estate assets are generally a poor investment relative to the market without a ton of active management. There are god knows how many millions of people who own land that would be incredibly valuable if not for the civil engineering lobby getting environmental rules and laws written to their favor.