Captchas don't do anything to stop bots, they just add a small additional cost(~$1.40 per 1000 solved). I am talking about monitoring things that 90% of bots generally do not take precautions against, like tracking mouse movements and other things I won't mention here, that distinguish them from humans.

If you want to learn about the concept and armsrace, here is a paper with plenty of resources (this is in a game context, though not website, although there is the most advanced detection here): http://iseclab.org/papers/botdetection-article.pdf

Here is an open source system demo'd at BlackHat Europe 2011 (that checks it is a proper browser (with DOM/Javascript/etc), also good against DDoS. https://github.com/yuri-gushin/Roboo

Project Honeypot (scans inbound ips) good against spambots: https://www.projecthoneypot.org/

Here are some commercial solutions: CloudFlare's ScrapeShield -https://www.cloudflare.com/apps/scrapeshield

Distil Networks - http://www.distilnetworks.com/

Scrape Sentry - http://www.scrapesentry.com/

Fireblade - http://www.fireblade.com/