Python3 Asyncio for middling developers (opens in new tab)

Yes, in a cooperative async model anything which would block will stop progress of every coroutine, so you would need an async version of the ORM library for this to work. When you await something, you're telling the event loop to stop running this coroutine, and wait for something (usually IO) to happen before resuming execution. Because of this, if you block on IO without using await, the coroutine doesn't know to yield and will simply pause; thus execution of every single coroutine is blocked.

This leads to a sort of infectious need to make everything async as even a single non cooperating coroutine can bring the whole show to a halt. It's essentially the red vs blue function problem of Python. However, there is actually a nice alternative, gevent. gevent will monkey patch all functions in the standard library which would block, e.g. reading from a socket, attaching an implicit await to them. If the author has used gevent, the example Django code would actually work as expected, since the code would execute until the database connection was written to/read from and then immediately await.

Either way, async IO is still a somewhat tricky thing to understand and get right. It took writing a non blocking event loop with epoll in my case to really grok what was going on under the hood of something like asyncio.

Considering everything that is involved in making a request to the internet, multithreading would have to be spectacularly slow to even come close to making serial approach quicker:

  $ python quicktest.py 
  ['http://www.google.com', 'http://news.bbc.co.uk', 'http://news.ycombinator.com', 'http://www.cnn.com', 'http://www.foxnews.com', 'http://www.msnbc.com']
  [<Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>]
  Serial: 1.23853206635
  [<Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>]
  Multiprocess: 0.912357807159
  [<Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>]
  Multithreaded: 0.708998918533

edit: Here's the code:

  import requests
  import time
  from multiprocessing import Pool
  from multiprocessing import Pool as ThreadPool
  
  
  session = requests.Session()
  
  urllist = ['http://www.google.com',
             'http://news.bbc.co.uk',
             'http://news.ycombinator.com',
             'http://www.cnn.com',
             'http://www.foxnews.com',
             'http://www.msnbc.com']
  # Warm up?
  responses = []
  for url in urllist:
      responses.append(session.get(url))
  
  print urllist
  
  start = time.time()
  responses = []
  
  for url in urllist:
      responses.append(session.get(url))
  
  print responses
  print "Serial: {}".format(time.time()-start)
  
  start = time.time()
  
  pool = Pool()
  responses = pool.map(requests.get, urllist)
  
  print responses
  print "Multiprocess: {}".format(time.time()-start)
  
  start = time.time()
  pool = ThreadPool()
  responses = pool.map(requests.get, urllist)
  
  print responses
  print "Multithreaded: {}".format(time.time()-start)

Have _you_ timed it? Not in general, but for this specific case? Thread creation is relatively expensive to some operations, but maybe the speed is entirely irrelevant to the task at hand. In this case, the author of the article is auto-curating some articles from a list of people he finds interesting. If this were done once per day as a cron job, it could almost certainly be done entirely _serially_ with zero concurrency and full blocking and still finish fine. Adding in concurrency is nice, but certainly any method will do with this volume.

This is certainly one of the cases where you should just do whatever is simplest (to _you_ the programmer). The first step is always to optimize for cognitive overhead. I.e. make the code easy to reason about. Next (and relatively rarely) is it necessary to good to optimize for different bottlenecks in your code.