This quote from Charity Majors is probably the best summary of the current state of observability in the tech industry - a total, mass confusion. Everyone is confused. What is a trace? What is a span? Is log line a span? Do I need traces if I have logs? Why I need traces if I have great metrics? The list of questions like these goes on. Charity - together with other great folks from observability system called
Thank you so much for writing this. You are so spot on!
- Logs capture information with the intent to diagnose issues. Logs should not capture success cases. They should focus on failure cases.
- Metrics capture aggregates like counters. Metrics should be derived from Events.
- Events are data points in a series + metadata. The metadata enables slicing and dicing data points in innumerable ways.
Events are the unlock. When you start with events, metrics are easy. When you start with metrics, events are impossible. And logs ... blech. What a mess. Logs are the junk drawer of observability.
> Events are the unlock. When you start with events, metrics are easy. When you start with metrics, events are impossible. And logs ... blech. What a mess. Logs are the junk drawer of observability.
Love this.
However reading HN discussion (https://news.ycombinator.com/item?id=39529775) it looks like some folks are offended by calling logs names. So I will avoid this in future. Logs are observability history :)
Yes. I like "wide events" term more because it has this "wide" component which actually highlights 2 things:
- the desire to attach to these events as much information as possible
- the design of the solution that would handle those events. While structured logs one can store in a row-oriented database with a few indexes, it's not wise to store wide events this way - a columnar storage is a must.
When reading some replies on HN discussion about structured logs vs wide events I confirmed this feeling that structured logs are not necessary associated with "a lot" of information. But it's just a personal preference in the end.
Yep, that's the one. There is nothing more in public I think. It's weird how it's so not promoted, given that the paper is published in 2013! It's cool that the paper shows UI screenshots as well and doesn't focus on storage only. Storage-wise I think systems like ClickHouse can do the job good enough.
UI is a different story though. It's simple yet powerful, but in the open source world it's hard to find anything that comes close.
I'm biased because I work for them :) You can take a quick look at a GIF demo here: https://docs.rilldata.com/notes/0.34 which visualises pretty much the experience that you've described in the post (comparing dimensions), I believe.
Looks nice! Would be really awesome if there was a playground, like the one Honeycomb has: https://www.honeycomb.io/sandbox Really useful to get a quick feeling
A colleague of mine recently sent me a link to your post. I couldn't agree with you more which is one reason I have been working on https://github.com/eventrelay/eventrelay.
Horses for courses. I would argue this post covers only one use case for observability. There are more. E.g. when you want to track a specific distributed transaction, you must have a full trace available and be able to look at it as a tree. It's very powerful. But you cannot just sample a random events in a trace though to make it work. Additionally, it poses very different requirements on the collection (the context needs to be passed). Also, all queries that look at traces (rather than individual spans/events) must group the spans into traces
The signals are not just about different semantics associated to them (but eventually meaning the same thing). It's also about relationship of records
Nobody is denying the importance of smart sampling. When we need that certain events always appear together, we should sample based on some common field in the event (traceId).
The Ad Impression example from the post has the same challenge. Once Ad Impression is served, there can be some events associated with this impression: clicks, shares, whatever. And it would be good to be able to see them all together. It's absolutely the same problem of the need for a smart sampling.
And fwiw I don't say that tracing is a bad concept. I'm arguing that it gets presented / defined like a completely different thing, while there is more similarities that differences between wide events / structured logs and traces / spans.
Did Meta ever invest in a tool that could automatically tell you which dimension(s) had abnormality, or how much they lent to a regression? Eg, new OS versions, app versions, countries, carriers etc commonly point to the issue. Seems like doing this automatically, maybe on demand, while computationally expensive, would just give you the answer and prevent manual hunting and pecking.
At Meta while I was there there wasn't such a thing, maybe it has developed it after that. In the tools available outside of Meta I know about BubbleUp feature in Honeycomb which does exactly this: https://docs.honeycomb.io/working-with-your-data/bubbleup/
Indeed AFAIK this feature is uber-popular among Honeycomb customers
You can always have a schema describing the type of each field (may be useful for efficient storage). It just needs to be backward compatible version over version (e.g. only add fields normally with some special process for deprecation).
Some implementations infer schemas from the events itself, see
Depends actually on the language / framework. E.g. in JVM world passing various context data to los it relatively easy (you add data to some context object or whatever, and it gets passed to the logs - something like this).
> Although Kibana does give nice visualizations on logs.
Right. The UX of Kibana is whatever the opposite to "intuitive" IMO. It's not a tool for exploration. One can find the answer on a well-defined question there, but exploring unknonw stuff?.. Would be surprised if many people can do it naturally.
> We do logging for JSON/proto events in BQ but not sure about the UI.
Yep exactly. Logging to somewhere is one thing, but it doesn't give the easy-to-use exploration capabilities. Queries should be easy to make and fast.
I believe more into writing data to ClickHouse for this purpose, because it's faster than BQ. UI is still a question - not sure if there is something available.
Thank you so much for writing this. You are so spot on!
- Logs capture information with the intent to diagnose issues. Logs should not capture success cases. They should focus on failure cases.
- Metrics capture aggregates like counters. Metrics should be derived from Events.
- Events are data points in a series + metadata. The metadata enables slicing and dicing data points in innumerable ways.
Events are the unlock. When you start with events, metrics are easy. When you start with metrics, events are impossible. And logs ... blech. What a mess. Logs are the junk drawer of observability.
> Events are the unlock. When you start with events, metrics are easy. When you start with metrics, events are impossible. And logs ... blech. What a mess. Logs are the junk drawer of observability.
Love this.
However reading HN discussion (https://news.ycombinator.com/item?id=39529775) it looks like some folks are offended by calling logs names. So I will avoid this in future. Logs are observability history :)
Thank you so much for writing this. That`s great!
Love this. It is very difficult to make sense of the open telemetry concepts.
I would say that Wide Events sound a lot like Structured Logging.
Yes. I like "wide events" term more because it has this "wide" component which actually highlights 2 things:
- the desire to attach to these events as much information as possible
- the design of the solution that would handle those events. While structured logs one can store in a row-oriented database with a few indexes, it's not wise to store wide events this way - a columnar storage is a must.
When reading some replies on HN discussion about structured logs vs wide events I confirmed this feeling that structured logs are not necessary associated with "a lot" of information. But it's just a personal preference in the end.
Great article, thanks!
Here is a link to the original Scuba paper I guess: https://research.facebook.com/publications/scuba-diving-into-data-at-facebook/
Maybe you know other public sources with its implementation details?
Yep, that's the one. There is nothing more in public I think. It's weird how it's so not promoted, given that the paper is published in 2013! It's cool that the paper shows UI screenshots as well and doesn't focus on storage only. Storage-wise I think systems like ClickHouse can do the job good enough.
UI is a different story though. It's simple yet powerful, but in the open source world it's hard to find anything that comes close.
Did you check Rill? https://github.com/rilldata/rill
Literally just saw a tweet about their integration with ClickHouse, and it made me wonder :) Do you recommend?
I'm biased because I work for them :) You can take a quick look at a GIF demo here: https://docs.rilldata.com/notes/0.34 which visualises pretty much the experience that you've described in the post (comparing dimensions), I believe.
Looks nice! Would be really awesome if there was a playground, like the one Honeycomb has: https://www.honeycomb.io/sandbox Really useful to get a quick feeling
May be this video can be helpful on the Scuba: https://youtu.be/5XvzuFbKuOc?si=R1WtICvrxMF81wFR
scuba actually reminds me of graylog gelf logs: you can have a message with key/value pairs and then aggregate/filter/group them using graylog UI.
Never used it, but from a quick look sounds like a similar thing yes.
However "native sampling" is a really important Scuba feature - not sure if graylog UI takes into account.
A colleague of mine recently sent me a link to your post. I couldn't agree with you more which is one reason I have been working on https://github.com/eventrelay/eventrelay.
Thank you for sharing!
Horses for courses. I would argue this post covers only one use case for observability. There are more. E.g. when you want to track a specific distributed transaction, you must have a full trace available and be able to look at it as a tree. It's very powerful. But you cannot just sample a random events in a trace though to make it work. Additionally, it poses very different requirements on the collection (the context needs to be passed). Also, all queries that look at traces (rather than individual spans/events) must group the spans into traces
The signals are not just about different semantics associated to them (but eventually meaning the same thing). It's also about relationship of records
Nobody is denying the importance of smart sampling. When we need that certain events always appear together, we should sample based on some common field in the event (traceId).
The Ad Impression example from the post has the same challenge. Once Ad Impression is served, there can be some events associated with this impression: clicks, shares, whatever. And it would be good to be able to see them all together. It's absolutely the same problem of the need for a smart sampling.
And fwiw I don't say that tracing is a bad concept. I'm arguing that it gets presented / defined like a completely different thing, while there is more similarities that differences between wide events / structured logs and traces / spans.
Did Meta ever invest in a tool that could automatically tell you which dimension(s) had abnormality, or how much they lent to a regression? Eg, new OS versions, app versions, countries, carriers etc commonly point to the issue. Seems like doing this automatically, maybe on demand, while computationally expensive, would just give you the answer and prevent manual hunting and pecking.
At Meta while I was there there wasn't such a thing, maybe it has developed it after that. In the tools available outside of Meta I know about BubbleUp feature in Honeycomb which does exactly this: https://docs.honeycomb.io/working-with-your-data/bubbleup/
Indeed AFAIK this feature is uber-popular among Honeycomb customers
How do you deal with schemas and versioning of wide events?
You can always have a schema describing the type of each field (may be useful for efficient storage). It just needs to be backward compatible version over version (e.g. only add fields normally with some special process for deprecation).
Some implementations infer schemas from the events itself, see
- https://vimeo.com/331143124
- https://axiom.co/blog/a-database-hacker-story
Nicely written !!
Adding more fields to log.Errorf() is a pain. Although Kibana does give nice visualizations on logs.
We do logging for JSON/proto events in BQ but not sure about the UI.
> Adding more fields to log.Errorf() is a pain
Depends actually on the language / framework. E.g. in JVM world passing various context data to los it relatively easy (you add data to some context object or whatever, and it gets passed to the logs - something like this).
> Although Kibana does give nice visualizations on logs.
Right. The UX of Kibana is whatever the opposite to "intuitive" IMO. It's not a tool for exploration. One can find the answer on a well-defined question there, but exploring unknonw stuff?.. Would be surprised if many people can do it naturally.
> We do logging for JSON/proto events in BQ but not sure about the UI.
Yep exactly. Logging to somewhere is one thing, but it doesn't give the easy-to-use exploration capabilities. Queries should be easy to make and fast.
I believe more into writing data to ClickHouse for this purpose, because it's faster than BQ. UI is still a question - not sure if there is something available.