Another factor is that an EV has a noise ceiling, whereas an ICE can get arbitrarily loud — think motorcycles, buses, trucks.
ICEs also use all of their noise range all the time — revving, accelerating. Sudden harsh noise changes like that are much more noticeable and annoying than the sound of a theoretical steady-state ICE engine.