Doing the math roughly is actually easy.
Every tweet that doesn't come from search endpoints has the user details inline. The user details include the number of followers.
So just consume the filter streaming api with your chosen hashtag. You will get all the tweets, or at least some and a measure of how many you missed (limit notices).
For every new user in said stream, add the number of her followers.
It's so easy you could write the code in the length of this comment. Of course, if you want to avoid counting duplicate followers, you'd have to do an average of 1-2 additional calls per user.