bballer

Hey guys wanted to ask a question about what technologies & methodologies you would architect together if you needed to constantly be queuing up 10K+ jobs, distributing the work out and then reporting that it was completed. It would be required that you should never be able to schedule a duplicate job when one is queued/running, and you need to ensure that each job only gets picked up by 1 worker and run once. These jobs could last anywhere (in run time) from 30 seconds to 1 hour.

I've tried googling but my fu is failing me. Would love to hear the thoughts from people who have maybe solved similar problems.

Thanks!

If you were about to start building a very vital brand new application for your company that needs to be maintained long term what tech stack would you choose? Put aside what devs at your company may already be comfortable with and lay out what you think is best.

When I ask tech stack I'm talking the whole dam thing, for example:

DB engine : ie MySQL, Postgres

API layer : ie Node w/joy, Java w/spring, PHP w/laravel, Elixer w/phoenix

Frontend SPA : ie React, Vue (do you use typescript or flow, webpack etc)

Caching : ie Redis, Mongo

PaaS : ie AWS, Google, Azure

Interested to hear your thoughts, and don't want any flame wars stating x is always better than y etc.

Feel free to just specify the stack but better to also back up why you think it is best in a few sentences.