The problem is that you don't know what your situation would be like if you'd
not taken Vitamin D. There are too many variables, generally -- e.g. in your early 20s and 30s you're likely socialising with many more people (so, more exposure chances), and you may also have young children (even worse disease vectors).
I'm similar to you in experience, btw, except that I don't believe correlation is causation. But as it won't do any harm, most of us don't regularly monitor our Vitamin D levels, and Vitamin D supplements are cheap -- it becomes a harmless form of Pascal's wager that I'm happy to keep making. Nothing to lose except a very, very tiny amount of money.