> The other big thing, in comparison to cloud software, is convenience. You can setup a crawl and start it running -and see URLs being crawled - within a minute.

This depends on implementation. If the architecture is modern and well thought, using dynamic scaling or even AWS Lambda, the result should be available much faster on the cloud software due to ability to parallelization. You can only have so much network bandwith / CPU power locally and if you need to crawl hundreds of pages to get your result, it matters a lot.

Disclaimer: I'm building a SaaS tool for SEO which also involves page crawling.

> On cloud software that's simply not possible, due to the way that everything is scheduled.

As someone who works in cloud software this makes me cringe a little.

I have no doubt this is how existing cloud SEO crawlers work but with elastic scaling, web sockets, and serverless there is no reason why this has to be true.

It is not a limitation of cloud software. It is a sign of devs and/or product owners deciding making instant results is not a priority for the product.

Edit: I hear that a lot from industries that are not intimately familiar with web apps. "You can't do that on the cloud"... a typical web software engineer will not be able to do it but there are people out there who can. They are more expensive than your typical developer but if depending on your product they are worth it.