I agree with the saying as such but I think it's actually a counterpoint to the "do the simplest thing that could possibly work" idea. When building a system initially you want to do the simplest thing that can possible work, given some appropriate definition of "working". Ideally as the systems requirements evolve you should refactor to address the complexity by adding abstractions, making things horizontally scalable, etc. But for any given change the "simplest thing that can possible work" is usually something along the lines of "we'll just add another if-statement" or "we'll just add another parameter to the API call". Before you know it you have an incomprehensible API with 250 parameters which interact in complex ways and a rats nest of spaghetti code serving it.

I prefer the way Einstein said it (or at least I've heard it attributed to him, not sure if he actually said it): "Make things as simple as possible, but no simpler".