However, I am also in the unusual position that I had to design and implement a package manager for my own little language recently. My original hunch was to just create a Cargo clone, but after reading Russ Cox's writings, I ended up cloning vgo (with a few simplifications). The implementation was wonderfully simple - everything just fit together. The algorithms are trivial. Operationally, it is easy to understand what the package manager is doing (even more so than for vgo, as vgo has do deal with various Go idiosyncracies and backwards compatibility).
This is in stark contrast to my experience with other package managers, which are temperamental beasts at the best of times. When they work, everything's groovy, but their error modes are easily incomprehensible. I think vgo's approach of restricting expressivity and streamlining processes is the right one. But since nobody has used such a package manager before, it remains to be seen whether it works in practice.
For my own experiment, I do have one data point: I showed people the (rather brief) documentation for my vgo-inspired package manager, and they all felt it was very simple to follow and easy understand what it did, and how.
I think it's less that Go has significantly different needs, but it's more that people overestimate what their actual needs are.
I think you are right. And I doubt it hurts that SAT solving is a fun problem!
My main package management experience has been with Haskell, which has used the cabal tool for many years. Cabal was a traditional solver-based tool (with the added pain of a global mutable package database, although that is going away), and it frequently broke down in confusing ways. Cabal hell was a widely used term. A few years ago, another tool arrived on the scene, Stack, which used the same package format and such as cabal, but snapshotted the central package list (Hackage) by gathering subsets of packages and versions that were guaranteed to work together (or at least do not have conflicting bounds). It works well[0], and although it does in principle result in a major loss in flexibility, it's rarely something I miss. Importantly, the improvement in reliability was nothing short of astounding. That certainly helped convince me that flexibility may not be a needed feature for a (language) package manager.
[0]: There are all sorts of socio-political stack/cabal conflicts in the Haskell community now, but I'm not sure they are founded in technical issues.
* Semantic Import Versioning. A package is identified by some name. I think vgo calls it an "import path", but I have also seen "package path" used. Every version of that package must remain compatible with previous versions. You may not break compatibility (i.e. increment the major semver number) without also renaming the package. There is some syntactic sugar that makes it clear that a version 2 is closely related to the original version 1, but from the point of view of vgo, the two versions are completely distinct packages. This also means a program may depend on several major versions of the same package (which Russ claims is necessary when doing gradual migrations of large programs).
* Minimal Version Selection. In contrast to pretty much every other package manager, vgo tries to install the oldest version of a packages that still satisfies any lower bounds specified anywhere in the dependency tree. Upper bounds are not supported. This both makes the solving algorithm trivial (which probably doesn't matter much), but also gives you reproducible builds without using a lockfile (which is nice), and it gives you a very simple operational model for what the package manager is doing (which I think is crucial).
All package managers suck and all package managers break, but I think when vgo breaks, it will be more obvious what is wrong. Time will tell whether it works in practice!
(And I'm not even a Go programmer; I just like the thoughtfulness that goes into the tooling.)
If Go had been designed by community vote from the beginning, it would almost certainly have generics... and operator overloading, and exceptions, and 50 exposed GC knobs, and macros -- and a SAT-based dependency resolver, of course.
I trust Russ, Rob, and the rest to get it right, and they've proven that my trust is justified over and over again.
I can't read that statement without thinking of perl, and the nightmare of developing in a team of perl devs. We don't need ten thousand ways of doing things and not everyone needs their way to be represented. I'm very happy to have Russ enforce one "right" way of doing things and vgo seems to fit that nicely. One less argument for me and my fellow engineers using go to argue about.
It's odd to call out Russ as being disingenuous when he says his concerns were ignored when the dep committee did effectively ignore them. Adding 'effectively' as a qualifier here is somewhat important because it grants that there may have been a deliberation process and that those concerns were not outright dismissed, but the output of a deliberation process must be either in favor or against, and when a concern is deliberated against it is effectively ignored. It has never made sense to me when someone says something along the lines of "we've heard your concerns and have taken them into consideration" and yet did not heed those concerns. Obviously you haven't taken them into consideration and therefore they were effectively ignored.
In this case it seems to me that the burden of proof is on the dep committee to demonstrate that the alleged showstoppers are in fact not stopping the show. You don't get to call a committee meeting, decide to ignore showstoppers by claiming that you don't believe them to be so, and then expect nothing but smooth sailing and cooperation from there on. That makes no sense at all. All this rhetoric about community involvement and "working with us" falls on its face when the implicit terms of engagement are not adhered to.
However, what happened was that he told the dep people "I will build a dep tool myself to understand more on the problem details", which implies "after I understand more, I will come back to you and discuss more", but not "after I build the tool myself, I will just integrate my version into the official tool". For this part, Russ was not executing his own words, or at least delivered the wrong impressions. Maybe this result should be already well expected based on Russ's personality or how the golang team did stuff in the past, but many programmers are not that politically awareing..
It is not the technical result that the "vgo" solution kills "dep" saddens dep people. It is rather the form of communication: "vgo" stabs "dep" in the back, by emerging suddenly out of Russ's pocket, and essentially claims immediate victory using the power of the Go core team. When the vgo posts are out, and the vgo proposal is posted (and it is not really bad technically), there is really no effective way for the rest of the community to reject it.
Russ could have kept his words by going back to the dep people and told them about what he learned from writing "vgo", and then move forward with integrating "vgo" if a consensus cannot be reached. Eventually, Russ might just be able to get the same technical results as today anyways (but maybe with larger communication overhead).
What the dep people essentially says here is that, if Russ really wants, they can also implement exactly "vgo" from "dep", but they never got the chance.
Russ said he was going to go and build a tool to understand the problem better. In no way does that imply that any lessons he learns from the implementation must be communicated back to the dep committee. But for the sake of argument, let's say that did happen. Then what? Does Russ practically force the dep committee to implement vgo out of the remains of dep? Why would he do that when he just spent a month or two implementing vgo from scratch? Why bother going back to a problematic group that doesn't believe their core issues are showstoppers? If that difference of perspective exists and they're not willing to debate that, to say nothing of the fact that they feel that they should be on the receiving end of the burden of proof, what more can be gained from them from Russ's perspective?
In my eyes, Russ is too nice here, and this committee seems like it would've been difficult to work with at an intellectual level, anyhow, judging by this type of griping.
Maybe Russ should hire Linus to deal with situations like this :-)) /s
Was anything major ever decided based on a "community consensus" in Go?
What I've read about, and this case is no exception, is that it's always "a bunch of long-time Go team members". Aside from making libs, and conferences, and such, the community might as well not exist.
Certainly nothing like PEP process, or something like Swift (which despite Apple's thing, goes out of its way to engage the community in feature roadmap) or Rust.
>We did all of this because we wanted to be an exemplar of how the community could step up and solve a problem that was being ignored by the core team. I can’t think of anything else we could have done to be better than we were. But the end result of this effort was no different than if we had done none of it at all: the core team ultimately didn’t engage with us meaningfully on the body of work we’d contributed, and instead insisted on doing the work themselves, as an essentially greenfield project.
Well, have they had any other impression about the prospects of such external effort? The author adds an even more bitter remark later on, and then retracts it saying: "Upon reflection, I think this may be too strongly stated. There are good examples of large contributions that originated outside of the core team, including support for different architectures, like WASM".
But adding a different architecture is not a decision that changes the languages syntax, standard libs, tooling, direction or semantics. It can even be dropped at any time, and none other architecture would even care. It's like proving how an OS is "open" to third parties by pointing out that anybody is welcome to contribute a hardware driver for it.
Good you say that. Not too long ago solutions from swift-server-group is discarded and Apple simply developed swift-nio in-house. I read that server group was blindsided by this development and now more or less disbanded.
"I've seen a surprising amount of people thinking that the server-side Swift community or the Swift Server working group was somehow blind-sided by SwiftNIO. That couldn't be farther from the truth. We had known about SwiftNIO before the first line of Vapor 3 code was even written."
And in any case, even if the server group was blindsided it's not representative of a general tendency, as there are a very active features discussions e.g. in https://forums.swift.org/c/evolution with the community and public members involved and shaping changes.
https://github.com/golang/go/issues/18130
Its also an example of the proposal process which is PEP like.
That said, changes like this are not decided by the community, but by the core team. Go is an opinionated language and far more ideas have been rejected than accepted. For example Ian Lance Taylor's 5 generics proposals: https://github.com/golang/proposal/blob/master/design/15292-....
Russ has asserted this from day one, and has brought several examples out in evidence, but has simply not convinced the committee that it’s true. Calling it a serious problem, let alone a showstopper, is a significant overstatement. The strongest claim that can be made on this point is that it’s a matter of opinion.
OK that's enough for me. "BDFL" clearly correct, random critics totally unwilling to consider the proposition. More language developers are realizing that hierarchical imports with directory name conventions are a good practical thing that just works. Multiple versions come for free with this arrangement. If your tool can't handle that, your tool sucks.
This response is a long one, of which the ending deserves special note:
> I hope this story serves as a warning to others: if you’re interested in making substantial contributions to the Go project, no amount of independent due diligence can compensate for a design that doesn’t originate from the core team.
> Maybe some of you are reading this and thinking to yourself, “well, no duh, Peter, that should be obvious.” Maybe so. My mistake.
It personally made me think of this issue / PR in the Go issue tracker, https://github.com/golang/go/issues/20654, about support for constant time arithmetic with bigints. Currently, operations on bigints in Go may leak secret data via timing channels, because calculations with different values take predictably different time. The maintainers have chosen to specially modify only the P256 curve (but not P224, P384, or P521) to work in constant-time.
The author of the issue has written an extensive (somewhat strawman) PR that includes the constant-time code inside the current implementation of the bigint. That's a choice – for seemingly good reasons laid out in the issue, and of which the author admits that it also could be a separate package – but the rest of the conversation is halted in indecision about how the core team should proceed, without referring back to any of the arguments and reasons the original author has put forth.
The Go crypto libraries have a domain-specialist owner: Filippo Valsorda. If you have questions about constant-time math operations in Go, he's a smart and nice guy; you should maybe ping him.
Go's crypto library is imperfect, but it is the best native crypto library provided by any modern language. (Ring is excellent too, but is not as I understand it part of the Rust standard library).
I think the similarity is as follows:
- there is a desire to fix something that feels like a gap (in this case limited support of constant-time math in standard library)
- there is a change proposal coming from the community
- there are arguments why it should be included and some discussion if it belongs in a separate library or not
- very little or no input from core team
- the whole process is stalled waiting for core team to participate
Now we just need someone on the core team decide they need to research via implementation instead of actively engaging community, end up liking their own solution more and we'll have dep all over again.
It still made me think of it, because in that thread there is a domain expert which lays out careful reasoning for the decision to place constant-time arithmetic inside big/int. Among some other discussion, the main hangup is about the placement of the code, and not any core maintainer is signalling go-ahead with any direction. The issuer author could of course continue to work out the existing strawman proposal, but that risks to be abandoned if the core maintainers turn out to change their mind.
Meanwhile, on March 19 2018 a core maintainer asks if "anybody want to fill out this proposal to see what changes would be required in math/big?", despite that the issue itself has an extensive PR with test suite which reveal most of the necessary changes. (either that or it refers to the Aug 17 2017 comment, but that seems to be exactly bford's proposal with a different public API)
Once a design is set, implementing it is "just work". I'm not sure trying to undervalue that, but the important part is first investigating the design. The package management committee came up with the tried and tested design while Russ was trying to understand the true requirements. Both have their merits, and time will tell if Russ is right here. The thing to do keep in mind is that investigative work is inherently "wasteful".
Calling something “open” a hundred times a day does not remove one bit of corporate control.
Lots of mentions in Peter's post about things the "dep committee" may or may not have agreed with. Isn't this the same appeal to authority he is throwing at Russ? When did the "dep committee" become the gatekeepers of Go dependency discussions and solutions? Looks like a self-elected shadow government, except it didn't have a "commit bit". Someone should have burst their balloon earlier, that is the only fault here. Russ, you are at fault for leading these people on.
Go is better off with Russ's module work and I personally don't care if Sam and Peter are disgruntled.
Note also that he doesn't dispute that Russ tried to bring them into the fold with his concerns and direction but they rejected his arguments. That's a fight they should have known they were going to lose.
Maintainers / BDFLs get the final say, that's just how it is. Just because there is a large community does not mean they get the final say, just because you put in a lot of work does not mean you get the final say. The reason BDFLs are in the position of making that decision is because they have proven through the accumulation of their previous decisions that they get it right more than they get it wrong. And that's the very reason there is a community around them at all.
If you walk into ANY project, public or private, open sourced or closed, and ignore the lead's objections when you go off in a direction they don't agree with, then yes, you are very likely in for some wasted work. That's just how the world works.
On another note I hope this is the last we hear about all this, please lets move on.
I only became aware of this by going to a meetup and seeing a talk. I just want something that's workable and not broken.
The reason BDFLs are in the position of making that decision is because they have proven through the accumulation of their previous decisions that they get it right more than they get it wrong. And that's the very reason there is a community around them at all.
This is the same reason given for totalitarian rule generally. The problem is that the dictator has enough power to make a big mistake, committing the entire communitiy's resources. In this position, if you can avoid making a decision, you should. Toyota doesn't simply make such decisions by fiat. Instead, they have a competition.
I mean in a way didn't the `dep` folks do exactly the same thing to the `glide` and other existing solutions?
Was the competition squashed?
> Russ failed to convince the committee that it was necessary, and we therefore didn’t outright agree to modiying dep to leverage/enforce it.
and
> The community of contributors to dep were ready, willing, able, and eager to evolve it to satisfy whatever was necessary for go command integration.
If you're willing to say: "Sorry BDFL, you didn't convince us this was necessary so we're not doing it." Then you're not willing to do whatever is necessary, BDFL objections are necessities. Peter and Sam just found out what happens when an unstoppable force meets a quite movable object.
I agree with the sentiment that Russ's only mistake here was leading them on for too long. I see this as an argument in favor of Linus' style of BDFLing, in this case he would have cussed them out over their design's flaws and told them in no uncertain terms that it was never getting merged and probably that they were stupid. It would have hurt more at the time, but also probably would have saved them a ton of time and, in the long run, might have even hurt less.
I remember being excited when the “dep committee” was formed, because (if I remember correctly) it was set up with the explicit blessing of the Go Team, with a Go Team Member on it to facilitate two-way communication.
Your characterizations are both inaccurate and ugly.
I think it may be blunt but seems true to me. Sam calling integration of Go module as sad milestone does seem disgruntled.
Dep folks have decided not to take high road which is fine but others are free to call this out.
My impression is that the dep folks understand that too. The problem is that there is no consensus on what was learned from it.
The dep folks seem to have come away convinced that a SAT-solver approach is the better approach. rsc is clearly convinced of the opposite.
Everyone knows it is ultimately rsc's call, so I don't think talking about the power dynamics is very interesting. What I am more interested in is whether or not it's the right call. A good faith interpretation is that the dep folks aren't sad that their solution lost, it's that what they believe is a better solution lost.
Would you really expect those people to be excited about the outcome?
This feels like an overly personal attack.
- Richard Feynman
