Video archive
Lock-free Observations for Prometheus Histograms
Video details
Speaker: Björn Rabenstein
Length: 27:11
Description
Implementing metrics in a client library for Prometheus comes with an interesting challenge: Those metrics are typically updated a lot (like a counter that gets incremented each time a service processes a request) while reading their current value is relatively rare (each time the Prometheus server scrapes the instance, which happens a few times per minute). All those mutations and reads have to be concurrency-safe, but the otherwise quite handy RWMutex is only good for the opposite case: Many reads, few writes. Perhaps lock-free programming can help?
A simple counter only needs updates of a single variable, and that’s perfectly handled by the atomic package . But alas! There are beasts like histograms, where you have to update the buckets, a total count of observations, and the sum of observations – all at the same time! It makes the Prometheus server very unhappy if the scrape delivers some of those values prior to and some after an observation.
Let’s dive a bit deeper into atomic operations, the Go memory model, layout of structs in memory, and more, to finally come up with a sane (or rather insane) way to atomically change multiple values in an efficient way. Warning: This talks includes looking at code that probably runs in binaries near you!
Stay up-to-date with GopherCon UK
Enter your email address to join the GopherCon UK mailing list and be the first to hear our latest news and announcements.
© Vitis Events Ltd. Registered company 09693545 Privacy Policy