Trying to force my neuro-divergent brain to work in an office setting where I'm surrounded by noisy people (any people, really), lights that may flicker, and temperatures I can't control is going to limit my career a lot more than failing to have serendipitous conversations in a hallway.
What works for you may not be what works for everybody, and that's the point.
It could be that remote work would have limited your career advancement.
For me, it's the only reason I can have a career at all.