Ask HN: How does your on-premise data analytics/processing stack look like?

Correct. There are regulations in some countries that force organizations to have data in their local infrastructures. There are some organizations that, by their nature, will rather have their data on their private clouds.

The stack differs from organization to organization, depending on the activity.

You'll find everything. Sorry for being generic, but it's kind of wild out there. One thing we do not to disrupt operations in these organizations is to work with them to have our own systems we control, and build connectors to acquire data in some form to work with it. We use Kafka, Spark, but have set Hadoop clusters for some clients, too. We used Python for most projects, Scala for other. We try and adapt to their data sources, for example when the data source is a Windows process in some machine that controls physical, moving, items where the client has hard constraints on the bandwidth of transmitted data.

We also helped some organizations acquire data from physical phenomena by building hardware to do so, and then repurposing software from our other projects to transmit the data from that hardware, and process these signals (protobufs, actors).

We work closely with their trams to decide what level of rawness we want the data. Sometimes we help them anonymize the data, too, working with their legal and security teams.

This also requires deep dives into their domain when the data sources are domain specific. Example: telecommunication billing systems, banking transaction data, reservoir characterization data, rail transportation control systems. We then source that data and route it to something the client wants to use.

We also build ML models, which is why they really hire us and the rest is something we have to do to get there. Then we develop applications to allow their people to either use these models, or train new models with new data.

Sorry for being generic, but it really depends. We have conversations with clients and adapt to their constraints, especially when these are very peculiar.

Now, doing that for many organizations and sometimes at the same time can be really taxing. This can lead people to actually leave the company as there is just too much to handle. Working on different projects is not for everyone. It also is slow and expensive: only extremely large organizations could afford our services. This is the primary reason we set to build our machine learning platform[0]. We want to accelerate this by 10x at least.

- [0]: https://iko.ai