I don't think it's so much attractive in the physically attractive sense as attractive in the sense of taking care of your appearance. Nerd carries the implication that someone is so obsessed with their passion that they neglect other "unnecessary" aspects of life, the most visually obvious being their appearance, hygiene a close second.
Being "attractive" is usually more than just good looks, it takes a lot of time and commitment. Even the most naturally attractive people can be pretty unattractive if they entirely stop caring about their looks. Hell, just keeping long hair looking somewhat presentable takes a few hours a week (as I found out when I decided to grow my hair out back in highschool, didn't expect that). Add in time to go to the gym, buy groceries, cook and eat properly, find/buy well fitting clothes, etc and it adds up. If you're nerd-level passionate about something, that's all time you could be spending on your passion of choice instead.