Semiconductor fabrication is nasty. It's way worse than most plastic production. Even the board substrate itself is probably the most toxic kind of plastic out there (other than maybe styrene feed stock) before its cured and being a composite it decomposes the slowest. It's much much worse than most household waste.

There are bigger things that produce way more waste though. Lots of vehicles (especially boats) have huge fiberglass composite components for example.