Protocol Buffers v3.0.0 released (opens in new tab)

But they didn't keep it internal properly -- the real world has leap seconds for better or for worse, and this library really does stick its head in the sand and pretend they don't exist. Google specifically says that this library is designed to be "the foundation of Google's new API platform". Yet they give a data type (as a headline feature) and a sample usage that is simply incorrect if you don't set your system to work using Google's "leap smear". It also seems quite likely that it'll result in blatantly wrong human-readable strings. I'll even quote a string from timestamp.proto [1]:

9999-12-31T23:59:59Z

That looks like an RFC 3339 string, and it even has the 'Z' suffix, which means it's UTC, which has an agreed-upon international definition. But this is not a valid UTC time. It's a time in a different time zone that Google made up.

Google easily could have done better: publish a spec for a different kind of time like:

9999-12-31T23:59:59s

where the little 's' means 'smeared'. Supply a serializer and deserializer for that. Now there's no ambiguity.

[1] https://github.com/google/protobuf/blob/master/src/google/pr...

Time is perfectly accurate, including all the minutes with 59 or 61 seconds. UTC is perfectly defined as atomic time (TAI) with an offset to keep it within 0.9 seconds of UT1 (time as measured by the rotation of the earth). Every time we increment or decrement this offset, this leads to leap seconds. But since 23:59:60 is a valid time (and distinct from 00:00:00 on days with leap seconds), there is no ambiguity here.

The problem here is how most computers handle this: introducing ambiguity by setting the clock backwards or forwards one second, instead of accounting for the fact that not all minutes have 60 seconds. Google did a pragmatic fix for their use case by squeezing leap seconds into the surrounding seconds, stretching them. It works for them, but now their "seconds" are not actual seconds anymore.

It _seems_ like their "UTC Epoch time" is the same thing as POSIX time, but the Google engineer's terminology is all fubar. The reliance on the Proleptic Gregorian Calendar is further proof as that's a reference to a specific algorithm for calculating calendar dates.

POSIX time says that there are precisely 86400 "seconds" per day, which I think implies the same thing as saying there being precisely 60 "seconds" per minute. The logical consequence is, of course, that in neither case is "second" referring to the SI second.

Once you get over the fact that we're discussing different units of time, then you can see that POSIX time is _perfect_ for recording and manipulating civil calendar time. For the purposes of calendar manipulation, you rarely if ever need to know elapsed time in SI-unit seconds. All you care about is easily calculating past and _future_ calendar information. Your power company and credit card companies don't bill you by SI seconds, they bill you by the hour, day, week, or month.

Conversely, in those situations where you want accurate and precise SI-second measurements, you rarely if ever want to convert or display that data in terms of calendar time. When SpaceX sends a rocket into space, the view screen shows elapsed seconds since launch, not elapsed seconds since lunch. That's a big difference.

Interestingly, in neither case do leap seconds matter! They're irrelevant. Leaps second play no part in either TAI or POSIX time.

There are some cases where you want both pieces of information, but I think it's usually a mistake to conflate them and try to shoehorn them into the same units. That misguided practice is behind all the anxiety about leap seconds in UTC time.

It's also worth noting that as clocks become increasingly precise and accurate that the whole leap second thing will fade away. UTC time is based on the fiction that there's an abstract, universal clock in the world that is measurable in SI seconds. There isn't. At some point the needs of routine industrial measurements will enter the realm where relativity governs, at which point the fiction will be laid bare. Calendar time, of course, doesn't rely on that fiction.

The move to uncouple civil time from solar time is totally misguided, IMO, and only exacerbates the improper way that software engineers conflate the purpose and function of various time measurements.

Using required fields have actually bit Google more than once and were increasingly being considered harmful.

A canonical example is that you add a required field, and then update binaryA in production (which receives messages from binaryB), which immediately crashes or errors out because the new field is missing.

So practically speaking, you can never add required fields to any message where you can't guarantee binary version syncing amongst all instances of the message-dependent services. At scale, this is essentially operationally impossible.

And if you're not running an RPC-based service architecture, then why are you using protos anyway?

You always had to check the individual fields for the zero value. A required field in a proto2 message can be set but also have the default value and pass initialization.

As maintainer of capnproto-rust, I beg to differ. :)

Cap'n Proto is indeed actively maintained, and here at Sandstorm we depend on it every day as a core piece of our infrastructure.

https://github.com/sandstorm-io/capnproto/commits/master https://github.com/sandstorm-io/capnproto/pulse/monthly

Browsers do support binary data using typed arrays but for some reason this isn't commonly used yet. Compatibility, maybe?

(You can find the niche use case in a response to your sibling comment, BTW.)

If protobufs had one canonical encoding, B could unpack the message and re-pack it when done; with the current protobuf implementation, B needs to keep the original blob around. In either case, C needs to check the signature on whatever blob it receives.

(Some details have been changed.)

* No clue about namespacing. If you pick the wrong name for something, you can have name clashes within a protobuf, across uninterpreted option classes, with protobuf source code, with your own source code; and it's different if you're in Python or C. Nowhere are naming restrictions defined.

* The API is maddening and inconsistent, especially in Python. (It's totally different between Python and C.) Some things look like lists but really aren't (e.g. you can't assign a list to a repeated field in Python). Even basic reflection (e.g. to get at uninterpreted options) is a Lovecraftian nightmare, and the docs are wholly unhelpful.

* Good luck serializing a list. There's not really such a thing, despite that the API pretends like there is; there are only repeated fields. So you need a separate flag to distinguish "empty list" from "not present list".

* Abstruse implementation. There are so many layers of indirection in the generated source and the core library that I wouldn't know where to start debugging.

Not sure if they fixed any of these issues with proto3.

But I'm thinking about scripting environments, where the data types used in protobufs don't exist in the host language. Simple things like this. I think in the implementations I've seen, they're just coerced or ignored. That's fine, imo.

But in terms of small codebases: a simple TLV protocol, where only limited data types are implemented, can be 1/10th of the size of any protobufs implementation.

My team has built out a high performance type-length-value system that doesn't require compiled schemas for game development, and we have a very small serialization lib that's smaller than any protobufs implementation for our target language.

I'd like to use protobufs to decrease the amount of modules we have to personally maintain, but I don't see the value in doing so for our particular situation.