You're 100% correct. Having done multiple red teams I would never attempt to break into a building without 1) the CEO on call, 2) a notarized statement of work identifying my and the client's identity, and 3) notarized authorization from the landlord.
If a client refuses any of these then the physical pillar is quite simply off the table.