The worst one ever was he wanted to build a kubernetes observability platform from scratch. Specifically, we wanted to filter our application logs for errors and be able to find them. There's a million out of the box ways to do this with free and enterprise tooling, like EFK/ELK setups, datadog, etc. But no. We wanted to do this from scratch.
Well, fine I thought. Good resume builder. His approach was roughly this from what I recall:
- On an EC2 instance we will set up permissions so it can access the control plane of our entire global EKS infrastructure (like 4 dozen large clusters hosting 10k+ containers)
- We will run a series of bash scripts in a background process that hits the control plane every few seconds with Kubectl to get raw pod logs and store them for analysis/error grepping (probably someone is laughing already)
- We would use a similar series of bash scripts to automatically generate a mostly static website that would link to the paths of log files deemed "problematic" in some insane filepath system that was approximately organized by timestamp, the idea being if you thought an incident happened in Cluster XYZ at 11am you'd navigate through this web of raw log files and find the directory that had the precise timestamp you needed (linux epoch of course, not mm/dd/YY)
- Because new files were generated all the time, like every few seconds, we also had to periodically "refresh" the site by rebuilding the static site with all the new links
All told I think it ended up being some absolutely psychotic mess of 50+ scripts and over 15,000 lines of almost all bash and some Go templating.
Surprisingly, it did what it was supposed to, it just looked terrible and had an obviously bad UX. Luckily before we could show it to anyone important the exact thing happened I had warned about and we started crashing some of the clusters' control planes from the sheer number of requests we were sending to it. He panicked and told me to shut it down.
I'm sure I don't even need to go into detail how useless this is. I did become a wizard with kubectl, however, so it was valuable to me. Not so much to the company.
