First, why?
Second, why doesn't the "sound of silence" count? Many composers utilize silence within a composition for a certain effect. Cage just took that to an extreme.
Third, some would argue that the sound of the audience-performer combination constitutes the 'sound' of the composition in this case--a _musical_ breakdown of the fourth wall. (Which is less cliche as an idea 50 years ago than it is today.)
On the whole, I probably agree with you. But there's a certain richness or fullness available to us after we've thought more rigorously about the questions involved than if we hadn't. (Consider the difference between seeing a sweet hack in a system you barely understand v. seeing a sweet hack in a system with which you're very familiar - the later is a much more profound experience.)