Do you have any load tests or synthetic benchmarks that are still capable of producing this?
It would be interesting to see what a more modern Go would do given there have been a bunch of tail latency GC improvements since your older 1.9 Go version... and in an ideal world, it would be nice to file an issue on the tracker if you were still seeing this.
(Maybe that ends up later helping another one of your Go services, or maybe it just helps the community, or maybe it’s a topic for another interesting blog...).
In any event, thanks for taking the time to write up and share this one.