Fashion can exist without harming the environment or society even if that society is not yours, there are still people suffering in third world. The argument that without these companies, they won't even have a job is plain bad because they have enough margins to justify paying more and investing in safety but they choose not to and consumers support that by purchasing it.

There is also difference between buying cheap produced out of slave labour vs buying expensive produced out of cheap labor. In former, you might not have enough money to be moralistic but in latter, you have a choice to and you didn't which imo makes you more responsible.