pasteurquadrant

Is it because it's not marketed to developers? Is it because of price? Is there a need for a general purpose data analysis platform?

Thinking about starting a developer friendly service that frequently crawls the web at massive scale + a big data analysis cluster (such as Storm) with support for a variety of languages.

Obviously walled gardens like Facebook and Twitter would be off limits at the beginning, but if the service gains traction, then it's possible companies would want to be crawled.

I was trying to analyze the common crawl recently, and the process of getting set up is non trivial. This service would allow more people to more easily analyze the web.

Would welcome any feedback.

I am interested in helping my friend get the word out about his tech blog, and one way would be to send an email to tech companies telling them about it. But is it illegal?