Its the equivalent of having an 802.11B client that only syncs at 1Mbps and is constantly downloading. There is no airtime left for anyone else!
Things you can try: Block 802.11G and lower clients (increases your minimum WiFi transmission speed), convert multicast packets to unicast (LEDE is apt to support that), or throw the chromecast in the dustbin!