The future of latency profiling in Go

(rakyll.org)

145 points | by rakyll 2438 days ago

4 comments

  • mmclean 2438 days ago
    PM for tracing at Google here.

    As the blog post mentioned, we're collaborating with partners on a common RPC context proposal for tracing systems [1]. Some of the major contributors are currently on vacation, but there'll be be another round of comments and updates once they get back.

    We're also working with the same partners on a project called Census - a set of tracing instrumentation libraries that all vendors can use and contribute to, though we're still in the very early days on this effort. We'll have more to announce later in the year (still need to publish additional libraries, set up a website, etc.), but you can follow the progress on GitHub here [2].

    [1] https://github.com/TraceContext/tracecontext-spec/pull/1/fil... [2] https://github.com/census-instrumentation

    • puzzle 2438 days ago
      OMG, Census! It's one of the frameworks you miss the most once you no longer can use it. There is some preliminary version of it hidden deep inside gRPC, but it's great to hear that it's becoming a standalone project.
    • tonyhb 2438 days ago
      What's the difference between this and opentracing? AFAIK opentracing is built by the people who created Dapper at Google, and the project also has a set of standards for HTTP transport.
      • mmclean 2438 days ago
        They're different in scope and goals. Census is composed of a context propagation format (which the blog post mentioned), a single distribution of language-specific libraries that include instrumentation hooks for popular web / RPC frameworks and metrics + trace exporters for various backends, and an optional agent that you can run locally to view metrics and RPC stats.
    • dmead 2438 days ago
      not having a tool like jvisualvm frustrates me.
  • tonyhb 2438 days ago
    Using Opentracing and uber's Jaeger client is awesome. It's easy to implement, and Jaeger provides multiple different mechanisms for sampling traces (always on, guaranteed throughput etc.).

    It's awesome to be able to see a UI giving you diagrams like this: https://www.dropbox.com/s/b2uhx77urpnuk9g/Screenshot%202017-... for your API.

    Definitely would recommend - it beats old-school log tracing.

    • frik 2438 days ago
      Do I miss anything? OpenTracing mentions "A vendor-neutral open standard for distributed tracing." It reads like it's for implementors - is it useable stand alone + Jaeger as UI?

      How can it be used - can you point me to a tutorial/blog post?

      • jamesmishra 2438 days ago
        Hey frik, former Uber engineer here. I never worked on Jaeger, but I loved using it at Uber.

        If you want to get started, the documentation is at http://jaeger.readthedocs.io/en/latest/

        We also have a blog post that describes the history of Jaegar and some of the design decisions. https://eng.uber.com/distributed-tracing/

        Jaegar is compatible with the OpenTracing standard ( http://opentracing.io/documentation/pages/supported-tracers ) and is also compatible with the Zipkin backend and wire protocol ( http://zipkin.io/ ).

      • tonyhb 2438 days ago
        Mm it's a little sparse right now - had to dig into the opentracing spec, dig in to the implementations and make it piecemeal.

        OpenTracing is for everyone - both tracing implementors (eg. Jaeger engineers) and for you implementing tracing in your system. As an implementor in your system you mainly care about the "vocabulary" and "standards" opentracing defines. So, if you've got two functions (A which calls B) opentracing defines:

        - Syntaxes to say that the span for "B" is a "childOf" "A". This gives you the graph I had above.

        - Syntaxes to attach errors and other information to spans in a standard way (this lets any opentracing-compatible UI display/search errors etc): https://github.com/opentracing/specification/blob/master/sem...

        OpenTracing also allows you to define functions that follow each other: https://github.com/opentracing/specification/blob/master/spe....

        Plus, defines abstract ways of passing tracing contexts through service boundaries (ie. from client => API server => other microservices) via strings and HTTP headers.

        Also, the best part of opentracing (in go) is that you can use it directly to set these fields (https://github.com/tonyhb/personal-starter/blob/master/pkg/a...). You can pull in the "github.com/opentracing/opentracing-go" library and use it to create and propagate your spans and contexts, and set errors using the default vocabulary (https://godoc.org/github.com/opentracing/opentracing-go/ext).

        Been meaning to sort my blog out and write some posts on this as the implementation documentation is sparse. Some time soon.

      • jeffrand 2437 days ago
        Before Jaeger was out I started using this, although I'm considering switching over to using Jaeger: https://github.com/opentracing/opentracing-go

        It should be pretty clear how to use it once you read through the examples.

        Jaeger requires using Cassandra as a datastore (for now), which isn't easily supported at my company, so I went with Zipkin and used MySQL as a datastore.

  • liuliu 2438 days ago
    For 1)., if you use a LLVM backed language, XRay supposedly should be it: https://llvm.org/docs/XRay.html
    • rakyll 2438 days ago
      LLVM is a future consideration for Go. One of the main reasons I don't want to tackle (1) right now if the possibility of benefiting from XRay even though it doesn't sort the case out for the vast majority: the gc users.
  • easytiger 2438 days ago
    what is the latency caused by the call to trace the latency?