Been increasingly using LSP style JSON-RPC 2.0, sure it's got it's quirks and is far from the most wire/marshaling efficient approach but JSON codecs are ubiquitous and JSON-RPC is trivial to implement. In-fact I recently even wrote a stack allocated, server implementation for microcontrollers in Rust https://github.com/OpenPSG/embedded-jsonrpc.
Varlink (https://varlink.org/) is another interesting approach, there's reasons why they didn't implement the full JSON-RPC spec but their IDL is pretty interesting.
Take JSON-RPC and replace JSON with MsgPack for better handling of integer and float types. MsgPack/CBOR are easy to parse in place directly into stack objects too. It's super fast even on embedded. I've been shipping it for years in embedded projects using a Nim implementation for ESP32s (1) and later made a non-allocating version (2). It's also generally easy to convert MsgPack/CBOR to JSON for debugging, etc.
There's also an IoT focused RPC based on CBOR that's an IETF standard and a time series format (3). The RPC is used a fair bit in some projects.
1: https://github.com/elcritch/nesper/blob/devel/src/nesper/ser... 2: https://github.com/EmbeddedNim/fastrpc 3: https://hal.science/hal-03800577v1/file/Towards_a_Standard_T...
Honestly, if protobuf just serialized to a strictly-specified subset of json, I'd be happy with that. I'm not in it for the fast ser/de, and something human-readable could be good. But when multiple services maintained by different teams are passing messages around, a robust schema language is a MASSIVE help. I haven't used Avro, but I assume it's similarly useful.
- dRPC (by Storj): https://drpc.io (also compatible with gRPC)
- Twirp (by Twitch): https://github.com/twitchtv/twirp (no gRPC compatibility)
Me too. My context is that I end up using RPC-ish patterns when doing slightly out-of-the-ordinary web stuff, like websockets, iframe communications, and web workers.
In each of those situations you start with a bidirectional communication channel, but you have to build your own request-response layer if you need that. JSON-RPC is a good place to start, because the spec is basically just "agree to use `id` to match up requests and responses" and very little else of note.
I've been looking around for a "minimum viable IDL" to add to that, and I think my conclusion so far is "just write out a TypeScript file". This works when all my software is web/TypeScript anyway.
State of the art for both gzipped json and protobufs is a few GB/s. Details matter (big strings, arrays, and binary data will push protos to 2x-10x faster in typical cases), but it's not the kind of landslide victory you'd get from a proper binary protocol. There isn't much need to feel like you're missing out.
5-10x is not uncommon, and that's kissing an order of magnitude difference.
That's true of protobufs as much as it is for json, except for skipping over large submessages.
> memory bottleneck
Interestingly, JSON, gzipped JSON, and protobufs are all core-bound parsing operations. The culprit is, mostly, a huge data dependency baked into the spec. You can unlock another multiplicative 10x-30x just with a better binary protocol.
> 5-10x is not uncommon
I think that's in line with what I said. You typically see 2x-10x, sometimes more (arrays of floats, when serialized using the faster of many equivalent protobuf wire encodings, are pathologically better for protos than gzipped JSON), sometimes less. They were aware of and worried about some sort of massive perf impact and choosing to avoid protos anyway for developer ergonomics, so I chimed in with some typical perf numbers. It's better (perf-wise) than writing a backend in Python, but you'll probably still be able to measure the impact in real dollars if you have 100k+ QPS.
Meanwhile if you care about parsing speed, there is MessagePack and CBOR.
If any form of parsing is too expensive for you, you're better off with FlatBuffers and capnproto.
Finally there is the holy grail: Use JIT compilation to generate "serialization" and "deserialization" code at runtime through schema negotiation, whenever you create a long lived connection. Since your protocol is unique for every (origin, destination) architecture+schema tuple, you can in theory write out the data in a way that the target machine can directly interpret as memory after sanity checking the pointers. This could beat JSON, MessagePack, CBOR, FlatBuffers and capnproto in a single "protocol".
And then there is protobuf/grpc, which seems to be in this weird place, where it is not particularly good at anything.
There are just very few spots that actually "need" protobuf at a level of urgency that would justify walking away from self-describing text formats (which is a big, big disadvantage for binary formats!).
[1] Something very well served by JSON
[2] Network routing, stateful packet inspection, on-the-fly transcoding. Stuff that you'd never think to use a "standard format" for.
That means potentially: the majority of devices in the world.
Also JS now has BigInt types and the JSON decoder can be told to use them. So I'd argue it's kind of a moot point at this stage.
What I'd like is to rewind the time machine and undo all the path-dependent brain damage.
Is there a design doc with the rationale for switching back to explicit presence for Edition 2023?
The closest docs I've found are https://buf.build/blog/protobuf-editions-are-here and https://github.com/protocolbuffers/protobuf/tree/main/docs/d....
The implicit presence garbage screwed up the API for many languages, not just C++
What is wild is how obviously silly it was at the time, too - no hindsight was needed.
Organizations often promote fools who don’t second guess their beliefs and think they have it all figured out.
From what I can tell, a major source of the problem was that protobuf field semantics were absolutely critical to the scaling of google in the early days (as an inter-server protocol for rapidly evolving things like the search stack), but it's also being used as a data modelling toolkit (as a way of representing data with a high level of fidelity). And those two groups- along with the multiple language developers who don't want to deal with native code- do not see eye to eye, and want to drive the spec in their preferred direction.
(FWIW nowadays I use pydantic for type descriptions and JSON for transport, but I really prefer having an external IDL unrelated to any specific programming language)
Velocity and stability/maturity are in tension, sure. I think for a foundational protocol like protobuf you want the stability and reliability that come from multiple independent implementations more than you want it to be moving fast and breaking things.
> syntax = "proto3" used implicit presence by default (where cases 2 and 3 cannot be distinguished and are both represented by an empty string), but was later extended to allow opting into explicit presence with the optional keyword
> edition = "2023", the successor to both proto2 and proto3, uses explicit presence by default
The root of the problem seems to be go's zero-values. It's like putting makeup on a pig, your get rid of null-panics, but the null-ish values are still everywhere, you just have bad data creeping into every last corner of your code. There is no amount of validation that can fix the lack of decoding errors. And it's not runtime errors instead of compile-time errors, which can be kept in check with unit tests to some degree. It's just bad data and defaulting to carry on no matter what, like PHP back in the day.
How so? In Go, nil is the zero value for a pointer and is ripe for panic just like null. Zero values do not avoid that problem at all, nor do they intend to.
struct {
data uintptr
len int
cap int
}
Maps are similar, but have internal data structures that require initialization, thus cannot be reliably used when zeroed. This is probably not so much a performance optimization as convention. You see similar instances in the standard library. For example:
var file os.File
file.Read([]byte{}) // panics; file must be initialized first.If what you wanted was to avoid null-panics, you can define the elementary operations on null. Generally null has always been defined as aggressively erroring, but there's nothing stopping a language definition from defining propagation rules like for float NaN.
Is there a way to have your cake and eat it too, and are there real world examples of it?
Pointers are different in that we've decided that the pattern where all bits are 0 is a value that indicates that it's not valid. Note that there's nothing in the definition of the underlying hardware that required this. 0 is an address just like any other, and we could have decided to just have all pointers mean the same thing, but we didn't.
The NULL is just a language construct, and as a language construct it could be defined in any way you want it. You could defined your language such that dereferencing NULL would always return 0. You could decide that doing pointer arithmetic with NULL would yield another NULL. At the point you realize that it's just language semantics and not fundamental computer science, you realize that the definition is arbitrary, and any other definition would do.
As for sum-types. You can't fundamentally encode any more information into an int. It's already completely saturated. What a sumtype does, at a fundamental level, is to bundle your int (which has a default value) with a boolean (which also has a default value) indicating if your int is valid. There's some optimizations you can do with a "sufficiently smart compiler" but like auto vectorization, that's never going to happen.
I guess my point can be boiled down to the dual of the old C++ adage. Resource Allocation is NOT initialization. RAINI.
People generally like to complain about NULL/nil whatever, but they rarely think about what the alternatives mean and what arrangements are completely equivalent. No matter what you do, you have to put some thought into design. Languages can't do the design work for programmers.
In rust, you have:
let s = S{foo: 42, ..Default::default()};
The way you do this is by having types opt in to it, since zero values only make sense in some contexts.
In go, the way to figure out if a type has a meaningful zero value is to read the docs. Every type has a zero value, but a lot of them just nil-pointer-exception or do something completely nonsensical if you try to use them.
In rust, at compiletime you can know if something implements default or not, and so you can know if there's a sensible zero value, and you can construct it.
Go doesn't give you your cake, it gives you doc comments saying "the zero value is safe to use" and "the zero value will cause unspecified behavior, please don't do it", which is clearly not _better_.
Option types specifically allow defaulting (to none) even if the wrapped value is not default-able.
You can very much construct null or zero-ish values in such a langage, but it’s not universal, types have to be opted into this capability.
It was partially reverted with proto3 optional and fully reverted finally. Go's implementation happened to come around the same time as proto3 so allowed struct access, despite behaving quite differently when accessing nil fields. That is also finally reverted. Hopefully more lessons already learned from the Java days will come sooner than later going forward...
For most of my projects, I use a web-framework I built on protobuf over the years but slowly got rid of a lot of the protobufy bits (besides the type + method declarations) and just switched to JSON as the wire format. http2, trailing headers, gigantic multi-MB files of getters, setters and embedded binary representations of the schemas, weird import behaviors, no wire error types, etc were too annoying.
Almost every project I've tracked that tries to solve the declarative schema problem seems to slowly die. Its a tough problem an opinionated one (what to do with enums? sum types? defaults? etc). Anyone know of any good ones that are chugging along? OpenAPI is too resty and JSONSchema doesn't seem to care about RPC.
There are lots of other benefits for non performance-oriented teams and projects: the codegen makes it language independent and it's pretty handy that you can share a single data model across all layers of your system.
If you don't care about the wire format, the standard JSON representation makes it pair well with JSON native databases, so you can get strict schema management without the need need for any clunky ORM.
Why aren't they good for that? They can have very high write throughput, they don't require ORMs, and they can be indexed and queried using standard database methods like the SQL language.
You can even enforce strict schemas on them if you want to, just as you would with an RDBMS.
Coral is a schema definition language, yes. But it’s also a full rpc ecosystem.
Smithy at this point is only really an IDL that (in most cases, at least before I left) is “only” used to generate Coral models and then transitively Coral clients and services. The _vast_ majority of Amazon is still on “native” Coral
I'm curious as to why would you think that?
There's a bit of boilerplate in there if you want to use it for a naive implementation, but I don't find it exceedingly resty.
From my POV, using protobuf as a schema declaration tool (as opposed to being a performance tool) is blind follower behaviour. Getting over all the hurdles doesn't seem worth it for the payoff, and it only becomes less valuable when compared to all the OpenAPI tooling you could be enabling instead.
This being for a web-based problem, where we're solving schema declaration.
Huh? From the Swagger site:
"The OpenAPI Specification defines a standard interface to RESTful APIs..."
From their OpenAI Initiative:
"The OpenAPI Specifications provide a formal standard for describing HTTP APIs"
Not sure I care to understand your POV more given this obvious bit was missed by you.
With Go generics - and equivalent in most other languages - you can write a 5-line helper function that takes a string in that format and either returns a valid proto value (using the generic type param to decide which type to unmarshal) or `t.Fatal()`s. You would never do this in production code, but as a way to represent hand-written proto values it's pretty hard to beat.
For several years I used the GoGo Protobuf SDK. It was vastly superior to the awful Javaesque Go code that the official compiler generated. It allowed structs to be pure data structs, was much more performant, and supported a bunch of options to generate native-feeling, ergonomic Go code.
But the Google team refused to partake in any such improvements, and GoGo was shut down as the burden of following the upstream implementation became too big. [1]
I'm not an expert, but as far as I understand, the extra struct junk is mostly to avoid having a parallel set of types for metadata (including reflection). It's unclear to me why these can't simply be generated as internal types with some nice API on top.
Clearly the new field metadata adds to this extra information, and the Go team is moving in the opposite direction of what I thought the future was — they're doubling down on stuffing metadata into the structs, and making the structs bigger and even less wieldy.
I understand how this might make things more performant, but I was hoping this sort of thing could be solved with the type system, especially now that we have generics. For example, surely lazy field access could be done like this:
type Info struct {
User LazyProto[User]
}
userName := user.Get().Name
Then again I've also seen people do these thousand line in-code literal string protos which really grind my gears.
A given struct is probably faster for protobuf parsing in the new layout, but the complexity of the code probably increases, and I can see this complexity easily negating these gains.
No, the general idea (and practical experience, at least for projects within Google) is that a codebase migrates completely from one API level to another. Only larger code bases will have to deal with different API levels. Even in such cases, your policy can remain “always use the Open API” unless you are interested in picking up the performance gains of the Opaque API.
message M {
string foo = 1;
}
message N {
M bar = 2;
}
There are some options like nilaway if you want static analysis to prevent you from writing this sort of code, but it's difficult to retrofit into an existing codebase that plays a little too fast and loose with nil values. Having code authors and code reviewers do the work is simpler, though probably less accurate.
The generated code's API has never really bothered me. It is flexible enough to be clever. I especially liked using proto3 for data types and then storing them in a kv store with an API like:
type WithID interface { GetId() []byte }
func Put(tx *Tx, x WithID) error { ... }
func Get(tx *Tx, id []byte) (WithId, error) { ... }
It’s good to isolate your dependencies within the code :)
The Go OpenAPI did not do this. For many primative types, it was fine. But for protobuf maps, you had to check if the map had been initialized yet in Go code before accessing it. Meaning, with the Opaque API, you can start just adding items to a proto map in Go code without thinking about initialization. (as the Opaque impl will init the map for you).
This is honestly something I wish Go itself would do. Allowing for nil maps in Go is such a footgun.
This was a mistake. You still want to check whether it was initialized most of the time, and when you do the wrong thing it's even more difficult to see the error.
But I do know the footgun of calling "get" on a Java Proto Builder without setting it, as that actually initializes the field to empty, and could call it to be emitted as such.
Such are the tradeoffs. I'd prefer null-safety to accidental field setting (or thinking a field was set, when it really wasn't).
Like with generics. Now they’re in go. They’re not great, they have some sense but they may as well not exist as far as I’m concerned. I find them pretty useless anyway without lower and upper type bounds.
This is NOT the solution lmao
You can build services internally with gRPC and serve a public graphQL API that aggregates them.
This one small change can result in an order of magnitude improvement.
- Messages (especially opaque ones) are not supposed to be copied. The
recommendation is to use `m := &mypb.Message{}`.
- This would make migrating to use the opaque API more difficult, if the
getters don't return the same type as the old open API fields, much more
code needs to be rewritten, or some wrapper that allocates a new slice on
every get.
- Users expect that `subm := m.GetSubMessages()[2] ;
m.SetSubMessages(append(m.GetSubMessages(), anotherm))) ; subm.SetInt(42) ;
assert(subm.GetInt() == m.GetSubMessages()[2].GetInt())`. This would not be
the case if the API returned a slice of values.
- ...
If we had returned an iterator (new in Go 1.23) instead of an actual slice, then it would've been possible to vary the underlying representation (slice-of-pointers, slice-of-value-chunks, ...). But there are other downsides to that too:
- Allocations when passing iterators to functions that expect a slice.
- Extra API surface for modifying the list (append, getn, len, ...).
Another thing that could be considered is: when decoding, allocate a slice of values ([]mypb.Message), *and* a slice with pointers (or do it lazily): []*mypb.Message. Then initialize:
for i := range valuel {
ptrl[i] = &valuel[i] // TODO: verify that this escape doesn't cause disjoint allocations.
}
So?
> - This would make migrating to use the opaque API more difficult
The opaque API is stupid to begin with. Now the objects are no longer threadsafe. You can't just read a message in one thread and process it in two different threads.
> Users expect
Then don't expect this. If you're breaking the API, then at least break it in a way that makes it better afterwards.
This is not correct. The Opaque API provides the same guarantees as before, meaning you can read a message in one goroutine and then access it (but not modify it) from other goroutines concurrently.
> So?
If you have a []mypb.Message, and range over it in the normal way:
if _, m := range msgs {
// Use.
}
Now you can not use normal Go struct initialization and you'll have to write reams of Set calls.
type LogEntry_builder struct {
BackendServer *string
RequestSize *uint32
IpAddress *string
// contains filtered or unexported fields
}
func (b0 LogEntry_builder) Build() *LogEntryYou still will need to create temporary objects (performance...) and for an unclear gain.
I certainly won’t allow this to be used by the engineering teams under me.
They are called `GetFoo()` instead of the idiomatic `Foo()`, but that is to ensure compatibility with the API where the fields are directly exposed as `Foo`, which also makes sense.
I call this Battlefield versioning, after the Battlefield video game series [1]. I bet the next version will be proto V.
[1]: in order: 1942, 2, 2142, 3, 4, 1, V, 2042
Glad to see this change, for that use case it would've been perfect
Lots of teams creating rest / json APIs, but very few who use code generation to provide compile-time protection.
Of course, code generation is still practical and I'm a lot more likely to trust a third-party writing a code generator like protobufs, OpenAPI specs, etc, but I would not trust an internal team to do so without a very good reason. I've worked on a few projects that lost hundreds of dev hours trying to maintain their code generator to avoid a tiny bit of copy/paste.
Yet another subtlety is that when cross-compiling, you need to build the code generation tool for the local target always even though the main target could be a foreign architecture. And because the code generation tool and the main code could share dependencies, these dependencies need to be built twice for different targets. That again is something many build tools don't support.
To bake endianess and alignment requirements into your protocol.
Currently I'm not quite sold on RPC since the performance benefits seem to show up on a much larger scale than what I am aiming for, so I'm using a proto schema to define my types and using protoc codegen to generate only JSON marshaling/unmarshaling + types for my golang backed and typescript frontend, with JSON transferred between the two using REST endpoints.
Seems to give me good typesafety along with 0 headache in serializing/deserializing after transport.
One thing I also wanted to do was generate SQL schemas from my proto definitions or SQL migrations but haven't found a tool to do so yet, might end up making one.
Would love to know if any HN folk have ideas/critique regarding this approach.
However I can get behind it for the lazy decoding which seems nice, though I doubt its actual usefulness for serious software (tm). As someone else already mentioned, an actual serious api (tm) will have business-scope types to uncouple the api definition from the implementation. And that’s how you keep sane as soon as you have to support multiple versions of the api.
Also, a lot of the benefits mentioned for footgun reductions smell like workarounds for the language shortcomings. Memory address comparisons, accidental pointer sharing and mutability, enums, optional handling, etc are already solved problems and where something like rust shines. (Disclaimer: I run several grpc apis written in rust in prod)