The server is not outside of scope.
edit: Do you audit all of the software running on all of the routers between you and keybase's servers? Why or why not? If not, why does this reasoning not extend to the servers? Why would the routers not part of the whole system, top to bottom?
Let me phrase this another way: if there's nothing to hide on the server, why isn't it open source?
We can go back and forth on this forever. My position is simple: strictly speaking, it's more rational to trust open source software. Trusting closed source software ultimately boils down to "trust me". I would love to reduce the degrees to which we have to blindly trust the systems we use.
The answer is that they do have things to hide: anti-spam/anti-scam systems, for example. The question is if they are hiding something that matters for security. You can determine this by auditing only the client.
Sure, open source software is great, and has many uses. In this case, it has no use in ensuring that keybase is secure. Somehow, you don't have to trust a great many components in the secure software you use on a daily basis, and yet you have to trust keybase's servers because.. reasons? And somehow this trust is important even though you'd still be blindly trusting that they're running what you hope.
I won't argue that some people would benefit from the server being open source, but to argue that open sourcing it has anything to do with security is just inane and FUD.