I would say the difference between addiction and supplementation is not in what you take, but rather what happens if you do not take it.

I'm supplementing with Vitamin D but I often forget to take it, sometimes for days.

Now would you be able to forget your caffeine? If yes, I'd say there's no addiction. If you'd start feeling cravings in a few hours, that's a different story ...