Launch HN: Activepieces (YC S22) – Open-Source Zapier Alternative

It has been almost a decade since I've been involved at such an org, so I cannot speak to the state of the art. Folks currently doing the schlep are going to be more knowledgable than myself. With that said, you are likely to be alerted in a few ways of trouble with an integration:

1. Exception handling. Whatever code you have polling API endpoints or processing inbound webhooks is likely to throw an exception if the structure of the API it's consuming or inbound webhook message formats have changed materially. I recommend handling via Sentry or a similar application error reporting mechanism for triage by your SRE or platform team.

2. API responses. There is some peril here, as every API is different. Some APIs will behave as you'd expect with respect to error codes, error messages, and request allowances, while some APIs will reply with code 200 with the error message in the body. Again, this is the value incumbents offer; they know what failure looks like for each API, and they also have a good idea of what success and health looks like at steady state. Build relationships with API partners (do you have a partner team? you eventually should) so that you have open comms with them with regards to breaking changes, and code defensively in general. Tangentially, ensure you have robust logic around deduplication of polling data.

3. User reporting. If your unit tests didn't catch something, nor did your application error mechanisms, your users will absolutely let you know if a piece of JSON element landed where it shouldn't have in a target integration.

I'd encourage you to ask around to others in this space, as their recent knowledge will be more relevant for avoiding sharp edges. Also, once you've built whatever you're building, you'll be able to (or, at least, you should if you've approached this from a systems thinking perspective and wrapped the necessary telemetry and observation tools around the machine) observe at scale what optimal and suboptimal looks like.

Are you thinking of misc. sync errors here (auth issues, temporary downtime, etc.) or API changes?

For API changes: It's not super scalable but I've used a changelog monitoring chat (Slack/Zulip/Discord) channel. Automated notifications for changelog updates for all integrated APIs. Maybe there's an assigned person for each integration and that person scans the changelog updates for their assigned services then emoji reacts to the notification messages when they have been reviewed and cleared (they create issues for required changes or confirm no changes needed).

Maybe https://www.akitasoftware.com/ could help.

Yes, long tail is smaller user base.

> nano id

There are pros and cons to using random IDs as a PK. For RDBMS clustering on the PK (InnoDB), it's a terrible idea. If you're going to sort by the PK, it's usually a terrible idea (UUIDv1 isn't as bad since it includes the timestamp, but that assumes your access pattern is based on insertion time). There is ULID [0] if you'd like something that's sortable. You could also just have a secondary index. An advantage can be that it _can_ be a good way (again, this depends heavily on your access patterns) to do sharding.

My other concern for nano id is twofold, both around their PRNG. They mention using Node's crypto.randomBytes(), but their source code instead references crypto.randomFill() [1]. Node's docs mention that having "surprising and negative performance implications for some applications" [2], related to libuv's thread pool. See my later comment about libuv and containers. Also, Node's crypto.randomBytes() mentions that it "will not complete until there is sufficient entropy available." That sounds suspiciously like they're using `/dev/random` instead of `/dev/urandom`, which at least for this application of it, would be an odd decision. I did note that elsewhere in nano id, they're creating their own entropy pool, so it may not matter either way.

With that out of the way:

If the plan is only for self-hosting, then yeah, you don't really need to consider schema design that carefully. Databases are really good at their job. Also, honestly nearly none of this matters until you have significant scale.

If you plan on starting a SaaS, there's a lot to consider. An incomplete list, in no particular order:

* Foreign keys. They're very handy, but they can introduce performance problems with some access patterns. Consider indexing child table FKs (but not always - benchmark first).

* DDL like ALTER TABLE. I suggest getting intimately familiar with Postgres' locks [3]. The good news is that instant ADD COLUMN with {DEFAULT, NOT NULL} is safer now. The bad news is that it does so by lazy-loading, so if your queries are doing silly things like SELECT *, you're still going to end up a ton of contention.

* Connection pooling. You don't want to eat up RAM dealing with connections. PgBouncer [4] and Pgpool-II [5] are two that come to mind, but there are others as well. The latter also handles replication and load balancing which is nice. If you aren't using that, you'll need to handle replication and load balancing on your own.

* Load balancing. HAProxy [6] is good for load balancing, but has its own huge set of footguns. Read their docs [7]. A few things that come to mind are:

  * Any kind of abstraction away from the CPU, like containers, may cause contention. Same with VMs (i.e. EC2), for that matter, since a noisy neighbor can drop the single-core turbo of Xeons A LOT. Look into CPU pinning if possible.

  * HAProxy really likes fast clocks over anything else, for x86. Xeons will beat Epyc. ARM can beat x86 if tuned correctly.

  * If you're using Kubernetes, look into Intel's CPU Management [8], which is also now native in K8s v1.26 [9].

* Overall for containers, learn about cgroups. Specifically, how they (both v1 and v2) expose the `/proc` filesystem to applications. Then at how your application is detecting that for any kind of multiprocessing. Hint: Node [10] uses libuv, which is calling `/proc/cpuinfo` [11].

* If you have access to the disk (e.g. you're running bare metal or VMs with this capability), think carefully about the filesystem you use and its block size (and record size, if you use ZFS).

Good luck!

[0]: https://github.com/ulid/spec

[1]: https://github.com/ai/nanoid/blob/main/async/index.js#L5

[2]: https://github.com/nodejs/node/blob/main/doc/api/crypto.md#c...

[3]: https://www.postgresql.org/docs/current/explicit-locking.htm...

[4]: https://www.pgbouncer.org/

[5]: https://www.pgpool.net/mediawiki/index.php/Main_Page

[6]: https://www.haproxy.org/

[7]: https://cbonte.github.io/haproxy-dconv/2.4/configuration.htm...

[8]: https://networkbuilders.intel.com/solutionslibrary/cpu-pin-a...

[9]: https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana...

[10]: https://github.com/nodejs/node/blob/main/src/node_os.cc#L100

[11]: https://github.com/libuv/libuv/blob/v1.x/src/unix/linux.c#L8...*

I'm not, to be clear, and I just wrote a giant referenced comment where I clarify that most things don't matter until you're at scale.

But for something small like that, may as well fix it now.

Yes, this! Trigger.dev lives in your code, we offer a visual flow builder, more like Zapier.

When I evaluate software, I see having multiple vendors offering to support/host the software as a good thing. After selecting a piece of software I then begin to evaluate vendor offerings. The vendor being deeply involved in the project- such as being the original creator or otherwise being a major contributor, is a huge plus.

Having 100% of a small pie is often worse than having a good percent of a larger pie. Fostering a large Open Source ecosystem increases the size of the pie- even if you have to share some of it.

Of course, any business is a lot of work, whether your software is Open Source or proprietary. Open Source is a great strategy, but it doesn't guarantee success.

Cheers, I'll take a look!

Thanks