Cloud Platform at Google I/O – new Big Data, Mobile and Monitoring products (opens in new tab)

I think App Engine's Datastore is generally an under-appreciated gem. Possibly because you have to use App Engine to use it without sacrificing performance, maybe because it's not easy enough to use if all you have is some JavaScript + JSON and don't want or know how to write Python/Java/Go.

But it's actually the only generally available product I know of that solves all the hard problems (availability, partition tolerance, some - but well defined - consistency with cross entity transactions) with zero hassle for you.

If you read through http://aphyr.com/tags/Jepsen, you get some appreciation for how hard this is to pull off without running into operational nightmares (massive data loss, split brains, etc).

Disclaimer: I work for Google, though not on Datastore.

Most recently: hot hash key. DynamoDB uses the object to be persisted's hash key to route it to the right data cluster.

We're a SaaS company with lots of tiny customers and a few very large customers. We need to keep an index to show a specific customer only their data. That means the index for our largest customers gets hit a lot. The problem with this structure is we have to pay as if all of our customers were as popular as our biggest customers, or we get throttled. And even though the DynamoDB interface shows that you have provisioned 10x above your current usage, you still get throttled, because you're being throttled only in a single cluster.

So, let's say you solve that problem, but now you need to drop the troublesome index on a billion+ row table. With DynamoDB you can't change a table's indexes, so you have to migrate your table to a new table. Doing that without downtime is an incredible challenge.

Which reminds me of when they announced indexes. We were so excited only to find out we couldn't add indexes to our tables, but instead had to recreate them all.

The whole point of SaaS is to make our lives easier, but with DynamoDB our lives were much more difficult than just using Mongo.

Anyway, I need to do a blog post on this -- it's a bit too complicated for a HN comment. :-)

I believe it's Kinesis: http://aws.amazon.com/kinesis/

I presume AWS Data Pipeline. But they have their differences, so perhaps there is no true analog.

http://aws.amazon.com/datapipeline/

Simple Workflow is new to me, so thanks for putting it on my radar!

DataDog is an important part of monitoring at SmarterAgent, where we use their API's and Integrations heavily, especially the CloudWatch integrations for Amazon's Web Services (AWS). Using these, we have been able to put up effective dashboards for new environments in a matter of minutes.

We leverage DD’s API primarily for eventing. For example, deployment notifications are posted to datadog, where they overlay our metrics. This has proven very useful in tracking changes due to deploys and/or configuration changes.

While we do leverage the DataDog agent for standard and custom metrics, DataDog’s ability to put together dashboards (and alerting) for AWS without any modifications to the host is what really closed the deal for us.

I probably will end up building the bare minimum to meet my needs and moving on tbh.

It was basically a monitoring/metrics system to merge how I handle the monitoring of crons, work queue, system metrics, analytics, etc. into a single service. Right now, I'm stuck using 3.

Sure, I could just build something to merge it together ... but at that point, I'm halfway to building my own.

We use datadog at pelotoncycle.com. My past experience was nagios but datadog is so much nicer as well as easier to scale. I looked into other services (cheaper, more expensive) before deciding on datadog.

k.z

Thanks, I'll have a look around at that!