How to build a custom data connector in 2 hours (opens in new tab)

(airbyte.io)

99 pointsjeanlaf5y ago32 comments

32 comments

30 comments · 13 top-level

Evidlo5y ago· 6 in thread

Was hoping for something about rapid prototyping of electrical connectors.

afandian5y ago

This is a fun story in that vein: http://www.bigmessowires.com/2016/06/04/db-19-resurrecting-a...

geoduck145y ago

Low supply of DB 19 connectors? I'll bet this is why there are 10k Teslas that can't be delivered right now.

mtricot5y ago

In the end, data is transported as electrical current so it is not completely off :)

tmh88j5y ago

That's what I was expecting as well. Guessing that "connector" an industry term that I'm just not aware of. Seems like some pre-made adapter patterns for ETL's from browsing the docs.

therein5y ago

Same, I was expecting some 3D printing, maybe some experimentation about how much spacing is needed between pins for insulation and how large the contact surfaces should be etc.

I was excited.

buildbot5y ago

I actually would love a service that did this for something mortally/individually affordable.

ducttapecrown5y ago· 2 in thread

What is a connector?

sherifnada5y ago

It's a module which extracts data from a source (an API, database, etc..) or pushes data to a destination (warehouse, blob storage, API, etc..). They're ubiquitous in data engineering & analytics workflows where users often need to consolidate their data in one database or warehouse to build dashboards, run analyses, etc..

tmh88j5y ago

I already responded to a different comment in this thread, but you seem to be knowledgable on the subject. Is this a tool to assist with ETL's between otherwise incompatible sources/destinations?

1 more reply

ncmncm5y ago· 2 in thread

It took a substantial fraction of two hours for the page to load.

jeanlafOP5y ago

Interesting. Where are you located? Thanks!

ncmncm5y ago

In New York. But the slow load (not really an hour, just felt like it) was probably Javascript overhead.

justinzollars5y ago· 2 in thread

One question: who is your customer archetype?

mtricot5y ago

Data engineers, data scientist & analytics engineers. People that be users of Airbyte but also can be contributors.

For data engineers, the objectives is to help them empower other roles to be data fluent and have the freedom to move data without bugging data engineers

hobofan5y ago

How suited is Airbyte for multi-tennant use-cases?

I might have a project coming up to build a semi-analytics solution where you would obviously want to allow the users to easily connect to their data sources, and Airbyte looks like it might be a good fit for that (for now my rough idea was to build some plumbing around Singer taps). Are you aware of any users that use it for such purposes and are there any specific pitfalls too look out for there?

1 more reply

renewiltord5y ago· 1 in thread

Very cool. Honestly, I’ve been using a mostly vanilla Airbyte. There are already enough connectors that I usually don’t need to write a new one.

Pretty thrilled with how fast these guys have been going honestly. Only picked up their stuff a few months ago and there are already more and better connectors today than there were back then.

I’ve been slowly moving over my error prone sync scripts to my Airbyte instance. Life is already better.

jeanlafOP5y ago

Makes our day to read your comment :)

thibaut_barrere5y ago· 1 in thread

Does anyone have some mid-long-term feedback on the use of the community edition of Airbyte in particular?

arjie5y ago

What do you want to know? I've been using since late-Jan / early-Feb.

I fired up an EC2 instance, docker-compose'd up, added the initial few syncs and then let people just add their own.

Mostly without trouble since then, though I'm running an old version as a result of the fire-and-forget.

(edit: also, text me on Keybase or email me since I don't get notifications for responses here)

dvaun5y ago· 1 in thread

This looks great. I’d love to throw up an instance locally and play with it at work, but that likely wouldn’t happen for a long time (local gov -> slow processes).

That being said, I’ve got a fun project in mind which I may find a use case for this. I’ll give it a shot.

mtricot5y ago

Thanks! let us know if we can help.

skadamat5y ago· 1 in thread

Exciting! Hoping to build Reddit & Stack Overflow connectors

mtricot5y ago

Let us know if we can help on slack (https://slack.airbyte.io) #connector-development or we can schedule a coding call call with you

justinzollars5y ago· 1 in thread

cool! Nice work

jeanlafOP5y ago

Thx!

jeanlafOP5y ago

If you want to have a look at the open-source GitHub project, here's the link: https://github.com/airbytehq/airbyte

cgardens5y ago

The next step is to make the tool easy enough to use such that GPT-3 could build a connector in 2 hours.

smarx0075y ago

Been waiting for some practical uses since I read https://news.ycombinator.com/item?id=26650493. I forgot to ask back then if someone had experience with Apache Camel, Spring Integration, or similar and could compare it to the crop of SaaS offerings like Fivetran (that Airbyte itself is taking on by providing an OSS offering).

vincentdm5y ago

Cool!

We built something similar for on-premise data sources, in Clojure. https://basil.net

j / k navigate · click thread line to collapse

32 comments

30 comments · 13 top-level

Evidlo5y ago· 6 in thread

Was hoping for something about rapid prototyping of electrical connectors.

afandian5y ago

This is a fun story in that vein: http://www.bigmessowires.com/2016/06/04/db-19-resurrecting-a...

geoduck145y ago

Low supply of DB 19 connectors? I'll bet this is why there are 10k Teslas that can't be delivered right now.

mtricot5y ago

In the end, data is transported as electrical current so it is not completely off :)

tmh88j5y ago

That's what I was expecting as well. Guessing that "connector" an industry term that I'm just not aware of. Seems like some pre-made adapter patterns for ETL's from browsing the docs.

therein5y ago

Same, I was expecting some 3D printing, maybe some experimentation about how much spacing is needed between pins for insulation and how large the contact surfaces should be etc.

I was excited.

buildbot5y ago

I actually would love a service that did this for something mortally/individually affordable.

ducttapecrown5y ago· 2 in thread

What is a connector?

sherifnada5y ago

tmh88j5y ago

I already responded to a different comment in this thread, but you seem to be knowledgable on the subject. Is this a tool to assist with ETL's between otherwise incompatible sources/destinations?

1 more reply

ncmncm5y ago· 2 in thread

It took a substantial fraction of two hours for the page to load.

jeanlafOP5y ago

Interesting. Where are you located? Thanks!

ncmncm5y ago

In New York. But the slow load (not really an hour, just felt like it) was probably Javascript overhead.

justinzollars5y ago· 2 in thread

One question: who is your customer archetype?

mtricot5y ago

Data engineers, data scientist & analytics engineers. People that be users of Airbyte but also can be contributors.

For data engineers, the objectives is to help them empower other roles to be data fluent and have the freedom to move data without bugging data engineers

hobofan5y ago

How suited is Airbyte for multi-tennant use-cases?

1 more reply

renewiltord5y ago· 1 in thread

Very cool. Honestly, I’ve been using a mostly vanilla Airbyte. There are already enough connectors that I usually don’t need to write a new one.

Pretty thrilled with how fast these guys have been going honestly. Only picked up their stuff a few months ago and there are already more and better connectors today than there were back then.

I’ve been slowly moving over my error prone sync scripts to my Airbyte instance. Life is already better.

jeanlafOP5y ago

Makes our day to read your comment :)

thibaut_barrere5y ago· 1 in thread

Does anyone have some mid-long-term feedback on the use of the community edition of Airbyte in particular?

arjie5y ago

What do you want to know? I've been using since late-Jan / early-Feb.

I fired up an EC2 instance, docker-compose'd up, added the initial few syncs and then let people just add their own.

Mostly without trouble since then, though I'm running an old version as a result of the fire-and-forget.

(edit: also, text me on Keybase or email me since I don't get notifications for responses here)

dvaun5y ago· 1 in thread

This looks great. I’d love to throw up an instance locally and play with it at work, but that likely wouldn’t happen for a long time (local gov -> slow processes).

That being said, I’ve got a fun project in mind which I may find a use case for this. I’ll give it a shot.

mtricot5y ago

Thanks! let us know if we can help.

skadamat5y ago· 1 in thread

Exciting! Hoping to build Reddit & Stack Overflow connectors

mtricot5y ago

Let us know if we can help on slack (https://slack.airbyte.io) #connector-development or we can schedule a coding call call with you

justinzollars5y ago· 1 in thread

cool! Nice work

jeanlafOP5y ago

Thx!

jeanlafOP5y ago

If you want to have a look at the open-source GitHub project, here's the link: https://github.com/airbytehq/airbyte

cgardens5y ago

The next step is to make the tool easy enough to use such that GPT-3 could build a connector in 2 hours.

smarx0075y ago

vincentdm5y ago

Cool!

We built something similar for on-premise data sources, in Clojure. https://basil.net

j / k navigate · click thread line to collapse