Can I easily copy and paste a binary format between text editors and emails and IM? Can I easily write that binary format by hand?
If your answer is "just transform it to/from JSON", I ask this: Why bother, then, since JSON itself works just fine, and avoids unnecessary layers?
With JSON, I get something that can be both read by a human and processed a computer with a minimum of translation and abstraction. I know exactly what is happening every step of the way, and can trivially verify what data is being passed around by analyzing logs, sniffing network packets, and even delving into memory dumps.
I don't think you've dealt with JSON enough. I live and breathe JSON. It is the lingua franca binding together a couple dozen developers working on seven different services written in six different languages (including, incidentally, erlang, which certainly wasn't my idea), and myriad client implementations in at least as many languages and even more runtimes on a dozen different platforms. (And these numbers are actively growing.)
We could not work with a binary format. This is a matter of practicality. There are developers of every skill set and experience level involved, there are third-parties involved (including those with strict control over some platforms that severely limit what kinds of code we can run), there are even customer-service reps involved who see this stuff.
JSON is easy to parse even in languages that make binary data difficult to deal with. It's easy for humans, wizard and muggle alike, to both read and create. As a text format it interacts well with the existing tools the entire world uses on their computers every day.
To say we should just use a binary format because we could conceivably spend time writing new, less-integrated, less-convenient tools to work around the problems it gives us is to miss the forest for the trees. We don't need better performance, and we've got enough problems to deal with without creating more.