The reputation of ex Googlers was established in the early days when they were tremendously productive, the bar was genuinely very high and turnover was tiny so there were very xooglers to begin with and they were mostly founding startups.
Over time Google grew an enormous amount. Productivity dropped through the floor due to endless headcount expansion, the bar did get lower and "I was at Google for two years" became a much more common thing to hear. But first impressions stick so Google still had done of that early day mystique.
Source: was at Google 8 years, early days, still have friends who work there. Saw the changes with my own eyes.