At first glance, microservices logging may seem simple. You just take the same principles you’ve always followed for monoliths and apply them to each microservice in your application, right?

Well, no. The differences between microservices and monolithic architecture amount to much more than a difference in the number of services involved. To create and manage logs in a way that provides true visibility into a microservices environment, developers and IT engineers must adhere to a different set of practices than they would use when logging for a monolith.

Microservices vs. Monolithic Logging

No matter how exactly you define a microservices architecture (a topic about which there is some debate), there are several distinctions between monolithic and microservices architectures that have critical ramifications for logging.

The first and most obvious is that, in a microservices app, there are more individual components of the application. By extension, there are more logs because each microservice typically generates its own set of log data.

Another, perhaps less obvious, the difference is that in a microservices architecture the various microservices interact constantly in ways that make them dependent on each other. From the perspective of logging, it’s not enough to have logs that only provide visibility into the state of each individual microservice. You also need to use logs to understand how microservices are interacting and to trace the web of dependencies that link microservices together.

A third challenge with microservices logging is that the scope and types of logs generated by each microservice tend not to be the same. Each microservice performs a different function, and some do more work or handle more requests than others. As a result, it usually doesn’t make sense for developers to try to have each microservice structure its logs in a uniform way. Instead, log format and structure are tailored to each microservice, leading to less consistency from the perspective of log management.

The distributed nature of microservices creates an additional challenge. Whereas a monolithic application is typically hosted in just one place—a single server or a single cloud,—microservices applications can be distributed across a variety of locations. The ability to distribute applications in this way is part of the reason why microservices are so advantageous. However, this ability also creates complexity with logging because different microservices store their logs in different places.

In many cases, microservices are deployed using technologies like containers or serverless functions, which lack built-in persistent storage. If a microservice logs data inside of a container or a serverless function execution environment, the logs will disappear when the container or serverless function shuts down unless they are moved somewhere else beforehand.

For all of these reasons, it would be simplistic and ineffective to approach microservices logging in the same way as logging for a monolith.

Best Practices for Microservices Logging

Fortunately, the complexity and challenges can be tamed. The following best practices help ensure that all of the log data generated can be leveraged effectively to provide true visibility into these applications.

Aggregate and analyze all microservices log data

Aggregate all of the log data from an application’s various microservices and analyze it in one place to gain holistic visibility into the application. Knowing information such as the startup time or requests handled by an individual microservice is not very useful for maintaining overall application performance and availability if you don’t know how that one service’s metrics correlate with those of the rest of the application.

Make sure log aggregation is truly centralized

It’s not enough to aggregate some log data into one place (such as a public cloud vendor’s log manager, like CloudWatch) and aggregate other data somewhere else (like a third-party log management tool). Although this approach may seem like the way to go if some microservices run in one location and others run somewhere else, you need all of your log data in a single location if you want to analyze and store it effectively.

Use custom identifiers

Ensure that data logged by microservices includes unique identifiers, such as the name of the microservice or a unique ID for each type of message it generates. This information is immensely valuable for helping to contextualize and trace interactions between microservices. If you don’t have unique identifiers, it is much harder to determine which information within your aggregated logs comes from which microservice. In certain cases, you may need to implement unique identifiers within the microservices’ source code. However, you can also use logging agent features, such as tagging, to associate identifiers to log data without having to modify source code.

Use custom parsing

Because microservices logs are often structured in multiple ways, trying to search through all of your data using generic regexes is typically not very effective. Instead, consider writing custom parsing rules that govern how your log analytics tool identifies relevant trends within log data, even if you are working with logs of varying types or structures.

Log to persistent storage

As noted above, microservices often run inside infrastructure—like a container—that lacks persistent storage. A basic and essential best practice in that case is to ensure that log data is written to somewhere where it will be stored persistently and remain available if the container shuts down.

You could do this by modifying source code or your container configurations to ensure that the logs are written to an external storage volume. An easier approach, however, is to run a logging agent that will collect data from the containerized microservice in real time and aggregate it within a reliable storage location.

This way, you kill two birds with one stone (or logging agent, as it were): You aggregate logs and move log data to persistent storage, all in one step.

Conclusion

Logging for microservices is an entirely different game than logging for monoliths. Not only is there more data and more logs in this type of environment, but the lack of uniformity in log type and structure, the distributed nature of the hosting environments, and the complex interdependence of the services mean that you must use specific logging tools for microservices.

This post is part of a larger series called Logging in the Age of DevOps: From Monolith to Microservices and Beyond. Download the full eBook here.