Why does it need to be checked on a per-request level?
I'd expect you to be able to give short-lived capability tokens to clients that each machine can verify down the stack without making new rpcs. This would avoid the fan-out of all the internal services.
You can encode capabilities/permissions as scopes in distributed tokens (e.g. OAuth) but this can start to break down if you have very granular, fine-grained permissions (e.g. user:1 has 'editor' access to 1000s of documents/objects). This is similar to the problem that Carta ran into while building out their permissions[1].
In addition, yes - validating permissions on each request makes it so that you can revoke privilege(s) with immediate effect without needing a token to be invalidated.
