Under the hood we're using NSQ as a queuing layer, S3 for storage and batched uploads, Amazon Aurora (for S3 indexing), DynamoDB for billing and metadata storage, and several distinct Go services that handle batching, transformation, schema updating, deduplication and internal consistency checking.
It's been in beta for several months and we're loading about 10,000 events per second into customers' databases today.
Instead, when we put a new object, we update a table in Aurora which tracks all of the relevant objects. That way, we can query information like "what objects were uploaded in a certain time range" very quickly.
1) We’re lowering the price and opening it up to all our customers to make it more accessible 2) You can bring your own database. This is helpful for customers who already have a data warehouse and want to load Segment data into it. 3) We now support Postgres, in addition to Redshift
The main reason that these companies implement these features to their infrastructure is to provide an alternative way to analyze data within their product in order to prevent losing their existing customers that need more advanced analytics features. (They are usually the biggest paying customers) The funny thing is that when you have an analytical database combined with a stream processing application, you can ask almost all questions you want to ask and get answers you need quickly enough so the value of their core product becomes less valuable when you have this alternative way.
I think that the BI tools such as Periscope and Mode Analytics realized this and started to promote their products as an analytics product rather than an application that creates charts from your data.
[Shameless plug] I'm also working on an open-source analytics platform (https://github.com/buremba/rakam) that collects data from clients (web, mobile or a smartwatch, doesn't matter), transforms (ip-to-geolocation, referrer extraction etc.) and stores data in a database that you specified. (currently there are two alternatives: Postgres and an in-house big data solution that uses PrestoDB as query engine)
Then, you to execute SQL queries, pre-aggregate your data for fast reports with continuous queries and cache query results with materialized views. Once you have these features, you can perform all analytical queries such as funnels, retention, segmentation etc. and create your custom analytics service easily.
http://blog.mparticle.com/mparticle-launches-next-generation...
Wondering if/how that will impact their bigger integration plans that includes a feature to replay data for new integrations you add after the fact.
Your pricing is a bit out of range for most startups IMHO. you go from 0 to 400. I would love a 20$, 99$ and then 400$ tier.
I would go with Alooma if their pricing is right ... And replace most of my other analytics stacks.
Even Amplitude does this in their priced tier..in that you don't need your own redshift cluster (you get query access to your tables in their db).
