Ask HN: What do you use to move (copy) data between systems?

Take a look at kafka ;-) Just put the file path as the key and the content as the message value.

You get TTL/size garbage collection for free: no need to get/develop a sophisticated client/server.

rsync, rclone

really depends on what formats the data is in, how it will be used, how long it needs to last, how long the code that reads and writes the data will last, how much out of band information vs inband information will be sent along with the data.

You talked about movement, movement is the last of your worries, formats and provenance should rank higher on your list of concerns.

My gotos are sqlite, xml and avro. Try to keep things under 2GB so you don't run into lame data transfer issues.

Many of these sorts of things we will push to an S3 bucket, and have other parts of the system pull from that bucket. We also use "rsync" with SSH keys to copy some data. Some of our vendors provide data to us via FTP, similarly it could be provided via HTTP

Maybe Zapier can help?

Can you configure database replication so your database handles the transfer to the target system, and then make the ecommerce site accessible for orders? The two databases will be kept in sync.

Have you checked out http://n8n.io? Sounds like you’re just pushing JSON blobs around on a bus.

Lots of data integration software is available. In the open source world, Apache Nifi and Apache Camel are two examples.

To add to the rsync answers, I think robocopy in a Windows environment is great.

rsync. Because it's efficient and well understood and emphasises accuracy.