My tests suggest pumping gas wearing a sweater is probably more dangerous than smoking. Igniting vapors with a static spark is actually much easier.
If you car is newer than 2006, there is basically zero chance you are in any danger. The ORVR systems are a really nice piece of engineering that is mandatory.