Yes and no. By raw numbers it certainly is in a state of decline. But put in context, it may have been inevitable. It has everything going against it.
DotA is a team game.
It's incredibly complex. Not easy to pick up. Mastery requires 1000's of hours. And it's miserable to play at the start.
The community is toxic. So random teams in a public match when you are new to the game compounds. Even if you aren't new, one way or another random teams can quite often be a frustrating experience that you just don't feel you have control over.
Compunding even further is the fact that once you start a game you are locked in for 35-55+ minutes. That can be 35 painful minutes of feeling crap.
Compare that to rising battle Royale games, and you get solo games you are allowed to be bad at and slowly get better in. And the time you spend in it is directly related to how much fun you have. And the games have a definite time cap. And the mechanics are simple. This holds true for a lot of other up and coming online casual multiplayer games.
Point is, Dota is destined to be niche as a game that is played.
What i would love to see is how the game can change externally to become more accessible to viewers. From client changes to community changes to changes in presenters, valve needs to step up to help people participate as spectators. The potential for the game to generate significant money lies in their pro circuit and it looks like valve is really stepping up their involvement here. Hopefully this continues.
The (somewhat poor) parallel is something like rugby or squash. Fairly arcane rules. A lot of people don't remember when they last played the game. But they can still be heavily invested in watching.
