All three office buildings I've working in during the past 5 years would not let you leave the lobby unless you are escorted by a person from the company you are going to visit. The escort also has to have escort rights.

That is different than an escort coming down to the lobby and saying, "ok you, you and you can come up. You over there have to leave the building". That is something I have never seen happen.

Edit: all three have had metal detectors and airport style scanners, really annoying getting stuck in the security line if you forgot your badge and were running late.