And a lot of this stuff is the result of carrying over design decisions, literally, from the 60s. Someone wrote an RFC codifying what they were doing by hand. And that nonsense gets passed forward for zero reason.

Ever wonder why HTTP uses the idiotic "Thu, 28 Jan, 1998, GMT" format for datetime? Because that format makes sense if you're hand-reading email message headers. And HTTP just copied it forward, zero thinking involved. Probably the same reason you can have comments in headers. SIP gives an example:

  Retry-After: 300 (I'm in a meeting)

To an IETF RFC author, that kind of behaviour is "well RFC xxx does it that way", and ultimately comes to "well, like, if you were sending HTTP messages by fax, extra comments might help".

If you stack up enough hacks like that, then it starts to become the clusterfuck it is. I don't think there's any maliciousness involved. Just lack of critical thinking, or lack of experience actually implementing software. Plus the fact that when writing a spec, you've got a blank paper and can just go all-out with crazy ideas you won't have to support.