Compression is an important part of both experiences. I like to think of it this way:

The soundscape has two axis, one is the overall perception of loudness, the other is the range of frequencies occupied by a given element, or track if you will, in a piece of music. In a stereo, or multi-track recording such as surround sound, there are more axis involving perception of placement, or "imaging" But the basics are all that are needed to consider this compression matter.

When you take one of those tracks, say the drum, or the bass guitar, in isolation a raw playback at volume on a reference system is going to largely reproduce what went in. So far so good.

Now, start adding the various bits and what happens?

That soundscape gets crowded! Little quiet bits you could appreciate are suddenly lost amidst all that is going on, so what to do?

Compress!

And that's a very good thing. First, few people actually have systems that can reproduce high dynamic range material in a quality way, and they don't have a listening environment that would make any sense if they did. So there is that. But more importantly, the details, subtle bits really are quiet! They will get lost, and so we compress the track overall to make sure those are present in the final mix, and pleasing to the ear too. Yes, there is a style and art to this, no doubt. But the compression really is necessary too.

A secondary reason for compression, and light processing by things like auto-tune, is to polish up a performance. Vocalist might be a little soft on part of an expression, or someone playing guitar might not deliver the same solid stroke every time. These things stand right out on a raw recording. If they get mixed in, they will get lost, or feel wrong, weak, etc... Compression can level this out and result in a solid, consistent sound. Again, the minds ear tends to want to hear this.

Ever listen to something a bit distorted and then play it back in your mind? Notice how your mind tosses out a lot of stuff, leaving you with that which you really craved? It's actually pretty difficult to recall something with high aural clarity for most people. Good production involves training the mind and the ear to actually pick this out so it can be managed into something people will really crave over and over.

One might also process things a bit too. This may be done to emphasize some characteristics of the sound with respect to the overall context of the mix. For a vocal, maybe it makes sense to punch the formant frequencies a bit, or dampen them to maximize the color and character in the vocalists vocalizations, for example. Our brain is an awesome audio machine, and it does a lot for us that recordings do not do. So we must bring those out and make them available to listeners of those recordings.

Now, comes more of the basics in the art!

What the minds ear hears is what we want to put on the record. Take that bass, compress it so that it occupies a smaller range in the loudness department (reduce it's overall dynamic range), and set it's LEVEL to one that's appropriate for it's overall contribution to the mix, which itself represents the overall perception of the music. Think two components here. The individual component of the mix, or track, has an overall dynamic range, but it also has a level at which it's present in the mix too. What makes sense here varies a lot and is highly subjective. Good production involves listening to the music and picking out what defines it as good, that which is resonant in terms of style, color, etc... A strong bass in one tune might make great sense, yet on another it might be pushed to the background more, etc... Depends.

A good producer will spread these things out in the soundscape so that the listener can hear them! Bringing up the little details, while bringing down the punch is needed to make room for it all to get in there and have an impact without fatigue or overloading the medium itself. A CD has it's limits, cassette other limits, radio still others, etc... An appropriate balance must be struck here, and that's not always optimal either.

This is what you are hearing when you listen to those older tunes, well produced. And it's damn good stuff too. You aren't wrong about it at all. Just blaming a good tool, when you should be blaming who ever is wielding it poorly.

(this is why remastered recordings exist Here's something fun. Go and get the DVD or Blu Ray of something you really like and compare it to what you might hear on the radio, or off a CD, or download. Often those are pushed out to the edges with the assumption the consumer has much better gear and or will pay for better production. Sometimes, shitty production on a CD can be avoided this way. Interestingly, some video game music gets remastered and I've heard more than a few wall of sound tracks sound great off my PS3...)

When this is done properly, the dynamic range is filled with a lot of things, each occupying their frequency range, sometimes overlapping, sometimes not, and each having an overall level that makes sense and that is aesthetically pleasing to the listener. This is why you can hear a great vocal right on top of that awesome guitar lick and drum set, despite the fact that they may be sharing a significant set of common frequency components.

Someone applied appropriate compression and some processing to make room for everything when it's mixed down into the final track. When they get it right, you don't even notice. It's just fucking good sound you crave. When they get it wrong, it's the tiring wall, or you find yourself straining to hear interesting bits that should just flow.

And I love this done well. To me, it's the most important thing I can say differentiates producers I love. Good production qualities in these areas are what makes a recording "pop" and you feel "there" No joke. This stuff, right here, is what makes a recording "immersive" for you. The sound goes right in, the audio engineers have made sure that's gonna happen, and it tickles your mind, taking you away from the fact that it's a recording.

Now, that brick wall... Take all that nice work, and listen to it on a great system, appropriate volume level, and the music will just stand out there, crisp, clear, every important part audible and enjoyable right?

Right.

A final processing layer can further crush this into a wall of sound, removing more and more of the dynamics to a point where it's all one intense thing, yet people can largely still differentiate the details! This can also be done, and likely and frequently is, in the mixing stage. But I'm mentioning it because commercial radio employs this processing extensively, due the limitations inherent in broadcast. FM, for example, has bandwidth and dynamic range / signal to noise limits that require this kind of processing to combat road and car noise.

newer digital radios are controversial, and a topic for another day, but they do not have those same limits, and can deliver the perception of a much better overall experience that many listeners will say is comparable to a CD. (it's not though, again another day)

It's the abuse that you don't like. Neither do I. Often the clowns even let it clip a little for "grit" or some other BS. That really sucks, because we don't even get the music, crushed as it is! But, let's set that ugly crime aside, and just stay with the brick wall for a moment longer.

It's all still there, LOUD, and that is tiring to you because it does not "breathe", "punch", etc... Graphically, it's a lot like cranking the gamma up on images. More subtle details stand out in more conditions, but the overall depth and feel are lost... and it's tiring to listen to, even at low volume. Our minds expectation of what things sound like can clash with this kind of excessive production, even though the first impression can be good. Producing for that initial, "wow" by maxing out the medium is, IMHO, always an ugly mistake, but the brutal truth is sales metrics select for LOUD over GREAT.

All that said, compression is good. We need it. Vocals, in particular, can very seriously benefit from appropriate compression and processing to really bring out the harmonics inherent in a great set of pipes owned by someone who knows who they are and how to express that well. But it all adds up, and it all needs that bit of compression love to make sure the good stuff isn't lost on the way into your head.

Maybe this helps a little to get at where some of the pain and fatigue really is.