I didn’t dispute the description of Keybase being labeled a walled garden. I opposed it being too-broadly called proprietary, when it’s not — only the backend is. And for anyone only using the official keybase servers, that’s irrelevant from a trust perspective, which is the reason people usually (mistakenly) bring up source code availability.
Now I’ll also partially dispute the accusation of it being a walled garden, since walled gardens don’t have open specifications and documented APIs for third-party client implementations.
The backend source code would be good to have, for the prudent reason you pointed out, as well as for private instances, but that’s not enough: you also need client code modifications to allow configuration for custom servers.
About binaries: anyone who thinks source code is required for determining program behavior probably shouldn’t be auditing software in the first place. (Often having just the source code makes it more difficult, not less.)