On average, the attacker will have to search half of the hash space to find a given password. So, your 40,000 searches is way too small as long as you require more than 2 character passwords. If you assume lowercase + uppercase + numbers for the passwords, and require at least 6 characters, you get
(26 + 26 + 10) ^ 6 = 56,800,235,584 combinations
searching half of that would be about 28 billion combinations. At 50ms each, that would take 388,888 hours, or 44.36 years.