Technically you can create one new GitHub account per repo and generate a token for that... But that is highly annoying :)

They need to support IAM / RBAC style policies and tie every authn+z method to those policies, but my guess is they have different auth methods strung all throughout their codebase so implementing it will take a few years. Then of course they have to make it "user friendly" as we all know how painful IAM can be...