This is what kill files were for, which every half-decent news reader of the time had (something that's still lacking in most contemporary web forums).
So trolling wasn't really a problem, because you could easily filter out the trolls. Same with spam, especially once Bayesian spam filtering was invented.
No, the real challenge that Usenet faced was the World-Wide Web. People were enamoured with the new shiny, with hypertext, and with embedded pictures and styling, which Usenet did not have.
Web forums were also a lot easier to use. You didn't need to download or learn to use a news client. All you needed was a web browser, which everyone already had and knew how to use.
Also, web search results on your topic of interest usually pointed you to forums, not to Usenet newsgroups.
If Usenet was better integrated in to the web, it might have stood a chance.