Saying your physical address is private ignores the reality of the situation: it's on your driver's license that you hand over to people that you don't explicitly trust (getting carded at bars, airports, stores, etc), it's sold between companies that you don't give explicit permission, you give it out to receive goods, and there is no law to force people to remove your address (to your point, try to request removal from sites like 411.info). Given all of that, it's unfortunately a fantasy to think that this is somehow private.

Private information isn't private if you have to give it out.