I could have sworn there was a different and simplified format for segments (datagrams are already about as simple as they can be), but I apparently had hallucinated it all these years. Turns out the only difference is using the larger addresses in the checksum algorithm.
Still, I do have to wonder about a port-free world. Seems a lot of things would get simpler.