How we log everything and simplified answering questions

In this post I will share how we are logging everything and example of how doing this has benefitted us. The inspiration for writing this came from the example, which is something I actually did, as well as the presentation the Elastic team did at the DevOps Brisbane meetup recently.

Why?

The concept of logging everything has been around for a while and really I think the question is why not log everything. I don’t want to (and don’t know that I could) come up with everything that we will need to log and the tools to support logging everything are mature and easy to use. Unless you have a good reason, such as no storage space for log data, it makes sense to log everything.

How

In our .NET applications we use Serilog, a structured event logging library. It is simple, but powerful. Logging structured events is an important part of the setup, as it gives us the ability to go beyond searching for text in a file and run actual queries on our data such as show all events where user equals x.

If you haven’t seen structured logging before, you should try it out in a new or existing project. I’d also recommend downloading Seq to query the logged data with. There’s also a pluralsight course that covers structured logging and these tools.

We also use the SerilogWeb.Classic package in our web applications to enrich our log entries with details from the HTTP request. This creates log events for each request and further enriches other log events with details from the HTTP request (when applicable) such as the request ID, which can help if you need to track the request that caused other log entries.

Our applications log to Elasticsearch. I am fond of Seq, but we ended up choosing Elasticsearch because it can be setup as a cluster with failover capabilities and is easy to scale.

To access the data in Elasticsearch we use Kibana, which is also made by Elastic (the same people who make Elasticsearch) and is a UI to run queries on Elasticsearch.

Example

This is an actual scenario that took place recently. I wanted to find all the user agents that were being used to view our web applications. Thanks to our usage of the SerilogWeb.Classic package the user agent for every HTTP request is being logged, so I had all the required data available.

First I opened Kibana and created and saved a search that retrieved all the logged HTTP requests. The search string for this search was messageTemplate:"HTTP {Method} for {RawUrl}".

The following is an example of the requests that are logged and the data that is stored for each request. Take note of the fields.HttpRequestUserAgent property.

An example of a logged request viewed in Kibana.

I then navigated to the visualise tab in Kibana, and created a new pie chart visualisation using the search that I had just saved. This produced a chart, however there was still a little bit more customisation required to make it display the data I actually wanted to see.

The next steps I took to customise the chart were

  1. Selected split slices as the bucket type.
  2. Selected terms as the aggregation.
  3. Selected files.HttpRequestUserAgent.raw as the field.
  4. Increase the size to display more results.

That produced the following.

Visualising the user agents.

And that was it. I now had a list of the user agents and could also see how many requests came from each user agent. If I want to view the user agents for a different period, I just have to change the time context in Kibana using the button in the top right.

Hopefully you found this interesting. I’m interested in hearing if you’re logging everything and how you are. Also I’d love to see some other cool data visualisations. I enjoyed Joshua Rich’s presentation at the meetup about monitoring and visualising network pings.

Advertisements
How we log everything and simplified answering questions

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s