The first two challenges are quite familiar in the financial industry, and kx, with its kdb+ time-series database, is arguably the industry leader. Most people in the open source sector don't pay it much attention, since its proprietary nature and terse APL-inspired syntax [0] lend themselves towards "I'll make my career in this, become a wizard, get Wall Street money, and not be very incentivized to compete with myself by sharing knowledge." To the third question, I'm not sure to what extent it treats topology natively, but kx is certainly positioning themselves as an IoT solution provider [1]. Of course, the cost is likely prohibitive for smaller IoT startups, and more innovation would certainly be good from a supply-demand perspective, but from a technical perspective I'm not sure this is an unsolved problem.

[0] https://a.kx.com/q/d/q.htm

[1] https://kx.com/solutions/utilities/

A friend is working on an IoT problem with two of your three capabilities: storage and analytics. He has voiced similar concerns. (To be fair, he gripes constantly that nothing exists which would work for him, within the budgetary and timing constraints.)

His company provides sensor systems for civil engineering projects. A single large bridge can have sensor packs every 50m or so - per beam. The amount of sensor data coming in for a single municipality or region is already staggering. Vibration and stress analytics are required on a daily basis.

The final requirement, one which you didn't mention, is that this setup should be fairly low maintenance. If you need a team of rocket scientists to operate and just keep it from falling over, the cost structure will be unsustainable.

A service that provided all this in a platform with sane APIs and good BI integration should be making tons of money.

I'd also add, there's a component of streaming analytics that isn't solved either.

One of the points I've tried to make at various companies (we've worked at the same one before) is that streaming solutions and batch solutions need to be fused into a single execution engine.

A streaming system on its own (operating on temporal windows) is not nearly as useful as on that can be joined to a storage engine with data at rest. It also needs to be disk based, so windows can be large, which most people do not want to take on. It also needs to be extremely parallel, and efficient.

Thousands of requests a second per server is not even in the right ball-park (which is lots of current execution engines now). Operating at line rate is generally table stakes IMO. The operations on the stream should be parallelized automatically, up to petabytes a day of input. Humans don't have the necessary context to do the partitioning up front, especially with streams that change.

The issue is (and I've tried to come up with designs to address this, though not in practice), is that co-locating the data at rest, with data that is moving through the system is a tricky problem, especially with complicated joins.

They can be the same engine (and should), but traditional database engines tend to have a problem with streaming queries, since they are just repeatedly executing a query against every new record. They are expressible, just not efficient. There is room to innovate in this space, but most people building these engines either solve the parallelism problem naively, or not at all.

There's also the problem of driving this computation to the edge, which is also something I have a solution for in a way that no one is doing, but have not yet met a company willing to take this level of effort on.

All the points you make about the kernel are apt, as are the points about the distribution algorithms. Also, the protocols used aren't nash safe, so at scale most of these systems become an operational juggling act under pressure.

All streaming systems that I know of do not know enough about the underlying data to gracefully rebalance and co-locate, since they all tend to embody the map/reduce paradigm, which is oblivious to underlying data distribution, at least in current practice.

There is available computer science to solve all these issues, I think some of the spatial algorithms out there can also be applied to the streaming space, especially in join evaluation.

Maybe a general solution just isn't cost effective and may never be cost effective, regardless of how much we bring down actual costs for the resources involved in solving data-intensive problems. The (possible) reason? The performance advantage you gain from tailoring specifics to structural relationships within exceptionally large datasets may just be too large to trade off for some "generalized" solution.

I'm just speculating, of course. I was sort of entertaining your idea and going back and forth between "a sounds-good fantasy" and "No, that would be awesome, why does this not exist?"

Flurry (now part of Yahoo) has experience with IoT storage at that scale (or near it). The "T" just happens to be your phone. Data processing (and staying in business) at that scale is a matter of great teams and solid architecture. This won't be a "service" easily sold to a general audience.

The data processing could become a commodity but the architecture won't be -- its highly tied to your specific industry.

Hi Andrew - What's the status with SpaceCurve?

http://spacecurve.com

What are one or two concrete things you'd actually use this for?