Ask HN: HTTP Proxy pool as a service – Should I bother with this idea?

1 pointsaalhafoudh9y ago4 comments

For one of our project we have built a HTTP proxy pool that inteligently routes the HTTP requests to a proxy server which accessed the target website host the least.

By doing this we can rate limit how much each proxy server in the pool can access specific website in a period of time to avoid abuse and ban.

I am thinking about extracting this part of the project as a paid service. The service would be for companies that do a lot of web scraping, web data analytics and processing.

What do you think about it?

4 comments

mswen9y ago

I know one of my clients - that does a lot of web scraping uses a proxy service - I think maybe 30 or so different proxies and breaks up the jobs manually. So if those ever get blocked they would need to go through the code and update to a bunch of new proxy IP addresses.

I would think a slick service that had hundreds or thousands of proxies and a good way to rotate scrappers through the proxies automatically would be of use to them. I have no idea what they pay for their current service.

Bino9y ago

No, no one uses proxies nowadays, you should aim for a VPN service which does the same. That is, if you want to profit.

aalhafoudhOP9y ago

I may have not explained it enough. The service would not be aimed for end users for browsing websites. The service is for companies that do a lot of web scraping, web data analytics and processing.

Bino9y ago

No you did not :) and yes in that case, as a infrastructure for those companies that may be a good idea.

j / k navigate · click thread line to collapse

Ask HN: HTTP Proxy pool as a service – Should I bother with this idea?

1 pointsaalhafoudh9y ago4 comments

For one of our project we have built a HTTP proxy pool that inteligently routes the HTTP requests to a proxy server which accessed the target website host the least.

By doing this we can rate limit how much each proxy server in the pool can access specific website in a period of time to avoid abuse and ban.

I am thinking about extracting this part of the project as a paid service. The service would be for companies that do a lot of web scraping, web data analytics and processing.

What do you think about it?

4 comments

mswen9y ago

Bino9y ago

No, no one uses proxies nowadays, you should aim for a VPN service which does the same. That is, if you want to profit.

aalhafoudhOP9y ago

I may have not explained it enough. The service would not be aimed for end users for browsing websites. The service is for companies that do a lot of web scraping, web data analytics and processing.

Bino9y ago

No you did not :) and yes in that case, as a infrastructure for those companies that may be a good idea.

j / k navigate · click thread line to collapse