That's not necessarily a bad thing considering that most communication within most office environments, like it or not, is not "productive communication".
When you have many people together in the same environment with a low communication cost it is inevitable that interactions will not always be productive (in the sense, "do we really need to talk about this, right now?").
To some extend the notion of community and belonging is lost when working remotely - as someone else mentioned, is like reading a book vs an e-book. However, if you manage to get past the feeling of not "being real" or not "feeling it" you realise that the function (working) is quite distinct from the form (sharing an office).
Sometimes when I talk with my workmates over skype and listen about time lost on discussions over office politics and needless meetings I feel great, because I'm glad I missing that kind of communication. At the same time, it is difficult to understand the context of a discussion without each party knowing the environment within which the other party is communicating from.
From my experience the downsides of working remotely are far less than the upsides. In addition to that, most downsides can be easily overcome (company or team-wide retreats, better culture management, etc).
The upsides on the other hand, for both parties, are unique in working remotely and cannot be reproduced or copied whilst working on location.
Personally, I enjoy my work the most when I work in two markets: education and music. The country I live in has exactly zero potential for me in those two markets with the skills I have. So working remotely is my only viable option (aside from migrating). Likewise, my employer has a highly skilled individual, who cares deeply about the product he's building, working for him - the type of individual he wasn't able to find locally.
It really is a win-win situation but at the same time, like everything else in life, you win some you lose some.
