undefined | Better HN

0 pointskentonv12y ago0 comments

Hmm, my personal (biased) opinion is that using variable offsets and vtables (as FlatBuffer does) is a lot more complicated than Cap'n Proto's fixed offsets.

"Optional" fields (ones that don't take space on the wire at all) are not clearly much of an advantage. The usual use of optional fields in Protobufs was to emulate unions, which Protobufs itself never supported directly, but Cap'n Proto does. Also, when wire size really matters, Cap'n Proto offers a very fast compression pass which deflates zeros, and unset fields are always zero, largely mitigating this issue.

The lack of VS support in Cap'n Proto is of course a very fair criticism, and a huge sore point in Cap'n Proto adoption. MSVC is finally catching up in its C++11 support so I hope this can be solved soon.

(Disclosure: I'm the author of Cap'n Proto, and also the former maintainer of Protocol Buffers.)

0 comments

4 comments · 2 top-level

mortarion12y ago· 2 in thread

I think you have maybe misunderstood the purpose of why flatbuffers exists; message packing for real time high performance games. It's not a replacement for Protocol Buffers or Cap'n Proto.

You've heard of Valve Software right? Their game engine, being a distant relative of Quake, used the old quake message packing format up until 1 or 2 years ago. Since then they have retrofitted all their popular games to use protocol buffers instead since it allows them to more easily extend their net code without breaking backwards compatibility (up to a point of course). In the context of games, optional fields are very important as are default values.

Also in this context; the author of flatbuffers is Wouter van Oortmerssen, author of the Cube and Cube2 game engines: http://sauerbraten.org

From the white paper:

> In particular, FlatBuffers focus is on mobile hardware (where memory size and memory bandwidth is even more constrained than on desktop hardware), and applications that have the highest performance needs: games.

kentonvOP12y ago

I think there's some confusion here. To be clear, Cap'n Proto allows you to add new fields to a message without breaking backwards compatibility, and Cap'n Proto implements default values.

The question is how a field is represented on the wire if it's defined in the schema but you don't assign it a value before serializing. With FlatBuffers, it won't take any space on the wire (although it still takes space in the vtable, but that can be amortized with many instances of the same structure). With Cap'n Proto, all fields are located at fixed offsets from the start of the struct, therefore they take space even if they aren't set. You can truncate unused fields off the end, but not from the middle.

Also, note that Cap'n Proto uses pointers for variable-width fields. The pointer can be null if the field is not set. So the maximum wasted space for an unset field is 8 bytes, all of which are zero and therefore compress nicely.

Cap'n Proto's approach is exactly the approach used by in-memory data structures in C, C++, and all other high-performance programming languages. If your C struct contains a field you don't use, it still takes space in memory. I would think this would be preferable for games.

judk12y ago

IME, optional fields are used for...optional fields. Like when you have a 50-field data object but only want to transmit 5 of those fields.

1 more reply

ssylvan12y ago

I think looking up values based on a simple offset is far simpler to all the clever rules and "parsing" that happens in capn' proto (some of which no doubt caused by the desire to support streaming "automatically", an anti-feature IMO, streaming should happen at a higher level to take advantage of domain knowledge).

I really don't see how you can even pretend that cap'n proto is less complicated. Yes there's an extra indirection at runtime, but the benefit is great simplicity, and quite possibly better performance when the vtable is hot in the cache.

j / k navigate · click thread line to collapse