Consuming video instead of text assumes that (1) I can hear and (2) I have the time to sit and listen at your speaking pace and (3) I'm in a place where I can sit with the audio turned up (4) I really want to expend the extra bandwidth for your video.
This trend towards posting video content instead of, rather than along-side, written content is pretty anti-accessbility.
I'm building a SaaS product to try and fix the issues I found with Airflow et al. Feel free to check my profile/reach out if you're interested in trying an alternative.
Disclaimer: I'm the creator.
https://github.com/spotify/luigi
We’ve been using it for complex update workflows for about 5 yrs now, and it just works.
It doesn’t do scheduling or have a fancy ui, but it’s a solid workhorse.
I would be also curious to hear more about alternative and mature open-source solutions to Airflow.
[1] https://medium.com/the-prefect-blog/why-not-airflow-4cfa4232...
Is this a good idea? I don’t think so
Alas, this relies on many things, not the least of which are peering agreements between cloud vendors that do not punish the consumer for using a service on one cloud with services from another.
It is possible to an extent today, but for competitive reasons, cloud companies do not seem to have a built in incentive to collaborate. Perhaps the cost of switching and lock-in will force cloud providers to work together in the long one, else they risk alienating customers, and then lack the ability to capture a larger market.
In other words, competition is good as long as people can realistically take advantage of it. I think cloud vendors have an obligation to see that this competition is encouraged and supported.
I agree.
It would be swell if, as part of the bigger open source movement, Jeff, Bill, and Larry - who at their disposal have AWS, Microsoft, and Google - competed with one another to submit the meritocratically superior implementation of our meritocratically determined open source standards, and then we, as custodians of the open source project, would select the winner, merge it, and use it - to the exclusion of all other implementations - not because of bias, but because of the truth underlying our meritocracy.
The question for me is - when we engage in said meritocracy, with all these BDFLs calling the shots, will they be the ones to negotiate with Jeff, Bill, and Larry? Or will we?
but are largely not, narrowing to sharing core tweaks, and keeping the PaaS layers proprietary
so it's not the maintainers failing to pick & unify, but Google etc employees not giving back
Or people working to make some *nix tools work on different distributions.
Of course their storage backend is going to be S3 and they are going to send logs to Cloudwatch, we have been doing it in a similar way for quite some time and it is what I expect from a solution managed by AWS.
I realize you are buying not just the compute, but the management, but that ends up being something in the cost range of $300-$500 or so per month for the airflow management part of it. Seems a bit steep. $50-$100/mo would be a no brainer for us. For some orgs I can see this being a great solution, but its not really friendly for the little guy (with a min price of $350/mo).
I understand that the premium is paid _every month_ -- and you may not otherwise have an incident every month -- but the AWS premium can also be considered an _insurance premium_ against those outages.
I used to manage an airflow deployment (of which my team was the primary consumer), and it was not enjoyable in the least.
For an enterprise, this pricing would work for some projects, for the reasons you suggest. Although this assumes nothing ever goes wrong with the managed offering, which is unlikely.
But for everyone else, it's a hard sell, especially in these days of infrastructure-as-code - even if you had to rebuild an Airflow server from scratch, it's not going to take very long.
disclosure: co-founder of astronomer
"Environments with CREATING status must complete previous operation before initiating a new operation."
can't email support for help (i only have basic plan)
any AWSMWAA ppl on this thread and can help? the instance name is `airflow-ry-test`.
AWS: “we will build one in 3 months”
See. nobody care about the differentiator in big 2B enterprise business . It’s more about trust and migration cost
I don't think of Glue as a generic workflow system the way the others are - it's definitely much more optimized for ETL use cases.
With time, I'm sure there'll be more detailed guidance on Step Functions vs Apache Airflow, but the simple guidance might be that Step Functions is a fully AWS-native (and serverless) orchestration engine. Whereas, of course Apache Airflow is an open source project with a diverse ecosystem of other plugins.
So far it's been very low maintenance - outside of the few random scares where dag logs filled up my server - but then I researched and found maintenance Dags that prevent a lot of issues.
Wondering when my next outage will be is always fun, but it's been pretty stable so far.
E: I know and appreciate the tech they put into it. It's just too high of a price for me once I get the workers added. I still want migrate mine to fargate workers at some point though.
It has a server/UI component that you can deploy relatively easily on something like Kubernetes. It then makes it easy to configure your flows to run on varying amounts of compute resources.
