Ask HN: Distributed tracing service with critical path analysis?

10 pointspseudonom-8y ago5 comments

The Dapper paper mentions "An engineer working on long tail latency debugging built a small library which infers hierarchical critical paths from DAPI Trace objects. These critical path structures were then used to diagnose problems and prioritize prospective performance improvements for universal search."

Do any of the distributed tracing services (e.g. Datadog APM, AWS X-Ray) offer this? I don't see any services which tout this functionality.

10 pointspseudonom-8y ago5 comments

Do any of the distributed tracing services (e.g. Datadog APM, AWS X-Ray) offer this? I don't see any services which tout this functionality.

5 comments

5 comments · 3 top-level

programd8y ago· 2 in thread

I think what you're looking for is Zipkin [0]. We use it in our microservers at Nulladmin.com via the opentracing Go library [1] and the zipkin-go-opentracing library [2]

[0] https://github.com/openzipkin/zipkin

[1] https://github.com/opentracing/opentracing-go

[2] https://github.com/openzipkin/zipkin-go-opentracing

pseudonom-OP8y ago

My company actually has and uses Zipkin. The thing that I'm looking for that I think is lacking is a way to identify critical paths without manually clicking through a bunch of traces and eyeballing them.

jjirsa8y ago

+1 for zipkin

As a Cassandra fan, bonus points that it integrates all the way down into the Cassandra internal request tracing features, afaik the only tracing app/library that does so.

itamarst8y ago

See if you can find a tool that creates flamegraph input files from the traces, or if not write one yourself. My causal logging library, Eliot (https://eliot.readthedocs.io), has an experimental branch for that, and it wasn't particularly hard to create the input files.

https://github.com/brendangregg/FlameGraph will then create nice graphs that can help spot bottlenecks.

albertwang8y ago

Disclosure: I work at Datadog --

Tracing requests across services and hosts to identify bottlenecks is one reason why Datadog decided to build a tracing component in APM! A flamegraph is one such method to visually infer critical path analysis that Datadog APM supports. Some obvious things to consider are ease of setup and whether the libraries and languages you use are supported. Some not-so-obvious things to consider are:

  (1) whether you wish to infer this from a single request, or across multiple requests, 
  (2) how to define "interesting" traces -- as you could be collecting millions or billions of traces in production 
  (3) how to create a bigger, "aggregate" picture across individual traces.

We're working on additional ways to easily identify critical paths as well.

j / k navigate · click thread line to collapse