Correct. Atherton isn't a great example. There's literally no downtown, so any homeless people would have to set up camp right outside some (very rich) person's (giant) single family home on a street with no sidewalk. Even in less affluent parts of the Bay Area unhoused people typically cluster in business or industrial districts rather than residential neighborhoods filled with single family homes.
You see unhoused people in downtown Menlo Park, Palo Alto, and Los Altos which are almost as wealthy as Atherton, and there was temporary outrage last year when it was reported that Menlo Park PD had paid for a one-way cab ride for one of the regular unhoused residents up to Pacifica (they claimed that she asked them to get her there so she could get her hair cut by a friend).