Golang or Python for the large scale distributed crawler?

1 pointsgerenuk9y ago1 comments

To deal better with concurrency which one would you opt for, as for the task queues, I have checked golang has "Machinery" library and for python its celery (we are currently using this). The bottleneck we are seeing is celery is very hard to debug for the issues in case of the gevent.

Sometimes it works fine with 100 threads, and sometimes it completely stalls and become idle without giving any errors etc.

I was thinking about this approach:

- Golang to take care of the requests/HTTP work and store them in the DB. - Python for the NLP tasks etc.

Any insights/suggestion would be helpful.

1 comments

1 comments · 1 top-level

tuxlinuxien9y ago

Are you using 100 threads or coroutines? because 100 threads on a single machine will no give you nice performance at all.

I have started working a crawler in Go and each website to crawl can be configured via YAML files. The quantity of code written is a bit bigger than the python one made before, but a single micro instance on amazon is way enough to "read" few hundred websites every hour. Personally, I think the goroutine system is way easier to deal with compared to the Python one.

So if I was you, I would definitely give the IO processing to golang, and let python deal with the data processing.

j / k navigate · click thread line to collapse