Scaling to 2M API request per day (opens in new tab)

(stackshare.io)

6 pointsmttsn10y ago3 comments

3 comments

Just finished the article. Great read! How are you guys able to retrieve data instantaneously for the personal API? I understand that you can cache the company API but how are you guys caching the Person API as there will be billions of users online across all social networks.

mttsnOP10y ago

Quoting Harlow here:

Great question! When we don't have an email address (or company domain) in cache we use our robust worker platform to doing live lookups. We leverage Sidekiq Pro to process millions of Lookup Jobs per day. We can typically finish a Person Lookup in under 3 seconds by leveraging the massive concurrency that Sidekiq affords us.

timchunght10y ago

Great answer! Thank you for your response. I am a fanboy of Sidekiq and I enjoy its speed and ease of use. What are some places you guys look for personal info. Are they just APIs or do you do some social network scraping? Thanks.

j / k navigate · click thread line to collapse

3 comments

timchunght10y ago

mttsnOP10y ago

Quoting Harlow here:

timchunght10y ago

j / k navigate · click thread line to collapse