Your fears are not uncommon. There are many of us who feel "different" from "normal people" and it's too easy to over-analyze and label our emotions and mental health.
Like you, I have ADHD. Like you, I was misdiagnosed with something ( bipolar disorder, at 16yo ). Like you, I've wondered if my idiosyncrasies are attributed to Aspergers.
In the past, I got caught up with these things. I know a few things: a) I have a classic case of ADHD, b) my ADHD meds appear to stabilize my moods, as I was always a cranky and generally bitter person to be around before I took meds. When I take days off of meds now, that part of me re-emerges. Perhaps the medication makes me calmer and more willing to listen to what other people have to say.
If there's one line of advice I have to give, it's that everything is on a spectrum. I once heard a story about a delusional schizophrenic. A man was standing on the atrium in the psych wards. As each person walked by, he said something along the lines of, "Greetings, ladies and gentlemen! Will you be returning for the show this afternoon? CNN, ABC, CBS, all the broadcasters will be here! Don't forget! At 3 o'clock I give a speech! All the world will be watching!"
I would never advise against seeing an expert if you really have problems. But never forget that everything is on a scale and difficult to measure. There are those who are normal, there are those who are perfect, and there are those with terrible afflictions. Most of us fit somewhere in between ;).