Skip to content
Better HN
Top
Best
Ask
Show
New
Jobs
Search
⌘K
A web crawler that can handle any number of websites
(opens in new tab)
(mixnode.com)
17 points
mixnode
9y ago
2 comments
Save
Share
2 comments
2 comments · 1 top-level
top
newest
oldest
spoondocz
9y ago
· 1 in thread
What's the largest crawl you can handle? meaning your upper limit...
mixnode
OP
9y ago
We've done crawls of over 250TB and 3 billion web pages. If you need to crawl more than this you can simply shoot us an email and we'll provision resources for your mega crawl.
j
/
k
navigate · click thread line to collapse
