This argument doesn't work because it's not about being in aspecific physical location, it's about being in the same physical location as all the other workers.
When half the people you need to interact with are remote, there's no point going to the office.