Totally. It's not easy to mix on-site and remote, and most places that do mix have some pain points like you describe.
When I worked remote it was at all-remote jobs. My office jobs have typically had a pretty "standard" policy around how much remote / work from home people can do, and that makes it viable. You definitely do miss out on some things on the days you work from home, though, so it's one of those things you kind of have to decide how much you can afford to take.
