Figured this was bound to come eventually since it's a very very big market and their basic CloudWatch product was lacking in many ways. It's not like Amazon to let an ecosystem eat their lunch.
Few things stand out:
(1) Per-query pricing seems...odd? Likely a good deal for small folks with a low volume of logs (i.e. just need to check actual AWS infrastructure logs vs. application logs), but if you have any actual volume this gets absurdly expensive ($0.005/GB scanned = $5 per query if you need to scan a terabyte. Large enterprises ingest multiple terabytes per day.)
(2) The quote "I pick the first one, click Run query, the logs are scanned and the results are visible within seconds" doesn't sound terribly promising performance-wise. "Seconds" is an eternity in the log management world.
Still, super interesting!
If Logs Insights can match the simplicity of integration that DD and the like services provide, then those services had best watch out. On the other hand, DD's dashboards are pretty slick and I can't imagine AWS's utilitarian UI/X ever competing. I wonder if that is a big enough differentiator. DD can get really expensive as well, but I'd love to see some comparisons on price.
Obviously disk iops are the bottleneck with such a system, so encouraging folks to conserve them by charging for bytes scanned seem like a good measure.
Won’t an enterprise implementing this themselves or using third party tools have to scale out their index nodes based on the amount data queries are scanning?
I wonder if it warns you about potentially expensive button clicks. A little cartesian join, and...ouch.
Say my app generates 1GB of logs per day. If I create a dashboard showing hits/errors for the last 24 hours and have the dashboard refreshing every 5 minutes it might cost me $1.44 per day (0.005 * 12 * 24).
By comparison list price for Sumo Logic is $90/month for 1GB/day of data (stored for 30 days) with queries being "free" (although there are limits to the number you can do).
For $50-$100/month, you get basically unlimited querying, unlimited dashboards, etc., vs. $43.20 from Amazon for a single dashboard that refreshes every 5 minutes in your example.
Looking at the actual numbers now, there's no way this is targeting the same types of users at this pricing level. It's gotta be aimed more at folks looking at meta-logs for AWS services, the volume of which is going to be much less than actual application logs. Otherwise I can't see this being competitive.
Could also be to disuade people from (ab)using it instead of proper metrics. Pretty sure SumoLogic lets you run queries on a cron and use the results for alarming (IMO not a great idea, seen it go wrong). And if there's an API, people will automate it and use it for god knows what. So charging for it seems like a smart enough move, iff the price is sane/competitive.
I just don't get how AWS developers are able to accept the eyesore that is their UX. It's easy to miss things in any AWS service's UI. I keep discovering functionality even after years of use. Google Cloud UX is light years ahead. Also, Google's Stackdriver seems great, haven't used it yet though. Would be great if any Stackdriver users here share how it is better than CloudWatch or other log management tools.
Particularly if you compare to other approaches, talk about technical tradeoffs made, etc.
AWS understands most people using tools like Splunk probably only need a few simple features so AWS just goes and builds that and gives a lot of people excuses to dump expensive licenses for the AWS version of it. It’s a sneaky but highly successful business model.
The feature where you can add the queries to a Cloudwatch dashboard seems to be a bit broken at the moment. First, the version of the query that I copied to a dashboard didn't seem to respect the time limit I'd set so instead of looking at the last hour, it was looking at all time - could get accidentally very expensive! Also, I couldn't see a way to show the visualisation (ie the stats graph) on the dashboard - just the raw table of query results, which is not ideal. Hopefully I've missed something, or those niggles will be sorted out soon.
Overall, very impressive!
