That kind of thinking is misplaced loyalty: you think you're being loyal to your colleagues, but really you're being loyal to the owner. Trying to protect your colleagues from a toxic workplace is just a recipe for more toxicity.
Musk will see or feel a difference if Twitter's technology organization collapses to the point that they have trouble maintaining the site (e.g. return of the fail whale).