You are either already failing to keep outdoor air out or you are getting too little ventilation. What you want is something like this device with the H13 filter option. Maybe that plus carbon.

Think of it this way, since you aren't dying of asphyxiation when you sleep at night, you're already getting some air exchange with the outside air.

With an ERV you can control how much air you get, and put whatever filter you want in between the outside air and your room.

In fact the fancier systems will let you overpressure your house by running the fan pushing in slightly higher than the air pushing out, ensure air only leaks out of your house not in.