How we manage 200 open-source repos (opens in new tab)

> Lesson 4: Age reports keep you honest

> Long-forgotten issues and PRs sap enthusiasm and hinder progress.

There are a truly massive number of open source projects that would benefit from this.

Mozilla is pretty much the poster child for "major issue that usually annoys the hell out of users who come across it, for a decade or more." Firefox and Thunderbird are littered with bugs that are half a decade to sometimes as long as two decades old. There are something like 40,000+ verified bugs in Firefox core.

Instead of those bugs getting fixed, we get shit nobody asked for like massive UI overhauls that everyone hates, integration of SaaS shit like Pocket, secretive data-collecting force-installed plugins for a media conglomerate's TV show (!), and so on.

Maybe they could free up some resources by firing some of the numerous product and project managers, while also moving their offices to places other than "the most expensive zipcodes in the most expensive real estate markets all over the world", while also trimming the CEO's pay, which has gone up even as market share has plunged; she's failing upwards.

I prefer storing the source code of third-party dependencies directly in the monorepo. Treat third-party code as your code, since it is used in your project in the same way as your code. It may contain bugs and security issues, which is easier to spot and fix if the source code is available in your repository. This also prevents you from disaster when third-party code becomes unavailable for some reason (missing internet connection, temporary outage at the third-party code hosting, permanemt deletion or corruption of the third-party code).

Apply custom patches third-party code in the same way as you apply patches to your code. Prefer posting these patches in the upstream project you depend on - this will improve the quality of the upstream prpject and also will reduce the amounts of custom patches on your side.

The hardest part is to update heavily patched third-party code to new releases with significant changes. Fortunately, this happens rarely in real life. Bugfix and security releases for third-party dependencies are usually easy to apply even to heavily patched code in your repository.

Real-life example: we at VictoriaMetrics store all the code for third-party deps inside "vendor" directory. This is very easy because Go provides easy to use tooling for this - just run `go mod vendor` and that's it! It is also very easy to upgrade third-party deps via editing "go.mod" file with subsequent `go mod tidy && go mod vendor` command. It is easy to inspect changes to third-party code for security issues and bugs after this command with `git diff` before commiting them in our repository.

If the projects were my own, I'd consider a monorepo. We use this approach for Steampipe samples - https://github.com/turbot/steampipe-samples

If it's a collection of changes, small improvements, etc to existing projects and repos then personally I'd go for separate forked repos. Then you can track your changes relative to the original project source code and (hopefully) contribute back PRs etc more easily.

As always - there are pros & cons to both - just a matter of choosing the approach that feels best 51% of the time :-). Of course, it's minor in general compared to the value of just keeping on moving on your projects and work!

For a C++ or similar project I would recommend fetching and building the depended-on repos using CMake's ExternalProject feature, with patches that your version in your main repo. I would never recommend submodules.

The examples they give on how this worked for them in practice are also great to follow.

100% - automation assists us to stay on top of it. But, human connection is the key! (In hindsight, our subhead text "automation with a human touch" got it the wrong way around.)

We use our open source project Steampipe [1] to query data using the GitHub plugin [2] and mods [3]. We have also opened up the custom mod we built for the specific charts & reports [4].

I hope they help - let us know if you give them a try!

1 - https://github.com/turbot/steampipe 2 - https://hub.steampipe.io/plugins/turbot/github 3 - https://hub.steampipe.io/mods?q=github 4 - https://github.com/turbot/steampipe-mod-community-tracker

So there is GitHub Scheduled Reminders: https://docs.github.com/en/organizations/managing-organizati... , which has some serious limitations it seems (GitHub will only trigger reminders for up to five repositories per owner and 20 pull requests per repository. Reminders are not sent when changes are merged from upstream into a fork.).

Glad you are enjoying Steampipe and thanks for the kind words :-). If you have any suggestions for where you'd like to see the platform go please let us know!

Monorepos have advantages and we considered that approach - easier mgmt, linear change timeline, consolidation of GitHub stars, etc. But on balance we felt our plugin and mod model benefited more from the separation - specific audience, clearer release history, simpler clone & install (mods), encourages contribution of plugins via repo ownership. Cross-repo tooling helps to offset the negatives of managing many repos, and as you say, the lessons we shared come also from managing our larger repos (e.g. the Steampipe CLI).

High priority tickets are naturally noisy and fixed quickly for impact reasons. Age reports create a good balance by gradually increasing the "noise" of all tickets regardless of priority.

Keeping docs with the code is definitely helpful, we've found that works well for our plugins and mods. All pull requests are then reviewed for consistency and documentation at the same time [1] - e.g. does this new table have good column definitions and clear examples?

1 - https://steampipe.io/docs/develop/plugin-release-checklist

Fair point! Rapid response is definitely valuable for thoughtful issues or PRs. For "questions" we try to just ask for more clarifying details or do a brief pointer to documentation. I find it can also be useful to deliberately slow down responses when getting caught in a low value conversation - kinda like rate limiting undesirable traffic to your website <grin>.

Most of our team just use their text editor to do formatter via plugins. I also find this online SQL formatter to be useful - https://extendsclass.com/sql-formatter.html