Agreed, anyone that already struggles with something like this should quit caffeine. My life is so much better off of it, but I struggle to stay off of it because I'm addicted to the 2hr productivity boost vs the all-day steadiness when you're not on caffeine.
Things that improve for me were: No sense of urgency for every single thing. Significantly improved confidence. Word things better and speak better in general. No hard crash later in the day. All my scattered thoughts become cohesive. No more random heart palpitations.
All of these likely got better due to the overall effect of decreased anxiety and not making ADHD worse. I'm not myself when on caffeine. Nikola Tesla quit all caffeine/other stimulants for a reason.