I'm actually working with a distributed team that does this. Everyone is on a shared video conference eight hours a day. If you want to talk to someone you look at your video conference team to see if they are at their desk, on the phone, or talking to someone else. If two people need to talk without disturbing others, they mute their video conference mic and jump on Skype. You can still see them, get their attention, etc. but you don't have everyone talking over each other on the main video conference channel.
It actually works a lot better than I expected.