undefined | Better HN

0 pointsslap_shot7y ago0 comments

As someone who works very, very closely in this industry, I would just be very careful how much of this you think you want to bite off.

Consider how you trust using dbt more than rolling your own transformation tool. Why wouldn't this apply to the rest of your stack? The 10+ companies that offer data extraction and loading are likely a better choice. Again with Analytics - the dozens of companies that offer BI tools are probably going to be the better choice.

Maybe you can build all these tools better than the hundreds of companies with thousands of employees and millions of dollars. It just seems like the odds that you build the best of each is so unlikely.

I would have been more impressed if your team had designed some API that other tools/platforms could plug in to coordinate a lot of the above jobs with your CI system. There is a SERIOUS need for that and I've had a lot of conversations with companies about what that would look like.

To answer your quest, no, Fivetran does not currently belong in the orchestration area, IMO. I've heard they are soon to release some sort of orchestration tooling to compete with dbt, but it isn't the type of orchestration you get with Airflow.

0 comments

joshlambert7y ago

slap_shot one of our major goals is to provide a solution that startups and small companies can utilize to start putting their data to work.

It shouldn't take weeks of effort, a data engineer, multiple proprietary solutions, and tens of thousands of dollars to answer key questions like CAC or the efficiency of a given marketing campaign.

We're hoping to lower the barrier to entry in both cost and effort, by providing an open source pre-packaged solution.

slap_shotOP7y ago

[Obligatory: Someone is downvoting you, but it isn't me. I upvoted you]

Yeah, I get that. The analytics space is very complex and companies, even ones with good engineering teams, don't have the internal knowledge or resources to typically put all this together.

In addition to working in this space, my copmany helps companies set up their analytics stack.

We typically set them up with one cloud-based data integration tool (the one with the most # of integrations they need at the best price), dbt, and one BI tool (usually Looker or Periscope, in that order). All in, that takes us a few weeks to get them set up and going.

I applaud your effort. I just struggle to understand why you accept punting on transformations (and using dbt (amazing library, by the way - great choice)), but then try to tackle something like integrations or BI tools. The complexity of both of those is massive and there are great open source efforts already out there.

I'm eager to see where this goes.

sytse7y ago

"but then try to tackle something like integrations or BI tools. The complexity of both of those is massive and there are great open source efforts already out there."

I would love to hear your suggestion for a great open source BI tool. We tried Superset and Metabase but both didn't came close to what we could do with Looker. That is why we're giving Meltano Analyze a shot.

BTW Do you want to do a livestreamed video call to discuss further in the 30 next minutes? You have a lot of knowledge. If so please email me and comment here.

Update: He did email and livestream will happen on https://www.youtube.com/watch?v=F8tEDq3K_pE

2 more replies

sciurus7y ago

> designed some API that other tools/platforms could plug in to coordinate a lot of the above jobs with your CI system

That's GitHub's strategy. Don't choose solutions for their customers. Be a platform other tools can plug into.

Gitlab's strategy is to cobble together a bunch of open source software (including their own) to provide a solution out of the box. It's not necessarily the best one for you, but it's certainly less effort for you.

veritas32417y ago

On the analytics side, we're using GitLab CI as our orchestration tool. We're pushing it to its limits and trying to find ways to make it better for us (i.e. data teams) and for GitLab more generally.

I'd love to learn more about what you'd like to see CI be able to do from a dataops perspective.

1 more reply

j / k navigate · click thread line to collapse