Show HN: Python 3.5 Async Web Crawler Example (opens in new tab)

There are 2 challenges with async def I don't know how to solve elegantly:

1. how to integrate coroutine-based scraping code with on-disk persistent request queues;

2. how to deallocate resources without boilerplate in coroutine-based scraping code.

(1) is easier with callbacks-as-methods because this way state is passed explicitly (it is not in local variables), so Scrapy can choose to save it to disk.

Example of (2) is this code:

    async def parse(self, response):
        resp = await self.fetch(url)
        # ... find another URL to follow
    
        # Here we have the problem: 
        # response object is kept in memory
        # until second response is fully received.
        # This is a problem if 10s and 100s 
        # of requests are processed in parallel
        # and responses are large.
        # Because of refcounting, with callbacks 
        # response would have been kept in 
        # memory only until second request
        # starts - callbacks+refcounting provide
        # an elegant way for resource deallocation.
        resp = await self.fetch(url2)

https://github.com/aosabook/500lines/tree/master/crawler

And a detailed post about its design, co-written with A. Jesse Jiryu Davis, here:

http://aosabook.org/en/500L/a-web-crawler-with-asyncio-corou...