There's a huge difference between everyone working remotely, and some. If a big chunks of your direct colleagues are in the office, the remote people will miss out on information, which will draw them to the office.