The term may get abused by some people who just want to call anything they dislike toxic, but we shouldn't throw out a perfectly good word (and associated phrases) just because you think "toxic" is toxic.
This lack of clarity is unavoidable in all communication. We can't transplant thoughts directly from one brain to another. But I find that words like "toxic" convey an assessment, rather than description, in contexts where that level of abbreviation is not helpful.
If the more-precise descriptor of behavior was used in the first place, the toxicity of that behavior will likely be self-apparent. But if "toxic" alone is used, we either have to circle the conversation back to pick up the better descriptor, or just assume that the speaker is right.
Toxic is a bad word because it's become a go-to choice for poor articulation. Using it conveys an impression that an assessment of "toxicity" has been made off-screen, but is broad and amorphous enough that I can't trust it implies anything more specific than a snap judgment.