I was like you in my 20s, I could drink a lot of caffeine without any effects at all. I was diagnosed with ADHD around 28. At the time my doctor told me that my high caffeine intake was maybe an unconscious way of trying to wrangle with my ADHD, because the treatment for that is stimulants.

Once I started my meds, even a low dose, I basically didn't want coffee anymore. It made a huge difference in my life

Not trying to diagnose you with anything, I just always like to tell heavy coffee drinkers my story in case it might help them if they're feeling like something is off

If so the difference when you are not drinking coffee might be more disorganisation/inattentiveness/impulsivity etc. rather than being less stimulated.

I have learned that I can have four Aeropress brews per day (about 12.5g coffee per cup) and it makes me calmer, more focussed and more able to be organised. I would notice discomfort and jitters at maybe six in a day. But I can fall asleep an hour after drinking one; it has little effect on my sleep and it may even make it a little easier for me to initially fall asleep.

And then I can have nothing the next day and be fine; I only notice withdrawal on day three maybe.

The reason is, it seems, I have otherwise very unmanaged "inattentive" ADHD. (Which is a stupid name for it; I don't have the slightest problem paying attention to things that capture my interest)