‍

A DevOps culture’s success includes focusing on your teams’ ability to deliver software continuously and consistently, even if the deployment environment constantly changes and new challenges continuously arise.

Centralized logging plays a critical role in enabling teams to achieve continuous and consistent delivery. Although logging is only one of many processes that reinforce these continuous processes, it's one that DevOps-oriented release teams can't live without, regardless of the type of applications they deliver or how their software environment is structured.

This article explains the role that centralized logging plays in DevOps-oriented delivery workflows. It discusses how centralized logging works, what a well-managed logging process looks like, and how effective logging reinforces the core goals of continuous integration, continuous delivery, and continuous deployment.

What Is Centralized Logging?

Centralized logging is the aggregation of logs from disparate sources into a centralized location where they can be analyzed and managed.

It's possible to manage logs without centralized logging. But it's not efficient, especially in the complex, large-scale environments that DevOps-oriented teams typically manage today. Without centralized logging, engineers have to manage logs individually, wherever they originated.

Not only does this approach require much more effort, but it also deprives teams of the ability to integrate and correlate data from multiple logs when troubleshooting a problem. It is also challenging when dealing with infrastructure like containers, where log data is typically not stored in a persistent location by default and may disappear before the team can analyze it.‍

What Are the Goals of CI/CD?

DevOps as a culture has many goals and philosophies associated with it, including the concepts of continuous integration, continuous delivery, and continuous deployment (which we’ll collectively call CI/CD here for simplicity’s sake). We won't cover them all in detail here; we’ll focus on CI/CD as an overall method. We will, however, explain which CI/CD goals are reinforced by effective centralized logging.

Among the central aims of CI/CD processes is to achieve regular, consistent delivery of new software releases. To do this, release teams need to ensure that the developers who write code and the IT engineers who deploy and manage it operate on the same wavelength and enjoy the same level of visibility into all stages of the software delivery process.

If developers don't know what's happening in production environments, they lack the visibility they need to ensure that the next application release they build addresses the main problems that are occurring in production. Likewise, if IT engineers lack visibility into development workflows, they don't know which releases are coming next, and they can't prepare.‍

Centralized Logging and CI/CD

Centralized logging ensures that both groups—developers and IT engineers—have the end-to-end visibility they need to deliver software continuously and consistently. It does this by aggregating logs from all stages of the software delivery pipeline into a single place where developers and IT engineers alike can view and analyze it.

With centralized logging, in other words, you can collect logs from your dev/test environment (which is where developers test and troubleshoot new releases before handing them off to the IT team) into the same place where you collect logs from production environments. Not only does the centralization of this data make it easier for everyone to view, but it also makes it possible to correlate data in order to gain deeper visibility.

For example, imagine a situation where IT engineers analyze log data from a production environment and detect that the application is unable to authenticate some users. If the IT team can also look at logs from dev/test, they can determine whether the problem occurred there as well, which would suggest that it's an underlying issue in the application code. If the problem is unique to production, it's more likely to be an issue related to the way the production environment is configured, or the variables involved in specific user authentication requests.

Likewise, from a developer’s perspective, the ability to centralize and correlate logs from all stages of the CI/CD process is crucial. For instance, if developers can view logs from production environments, they'll be able to identify the most serious performance problems that occur in production by determining how many users are affected by them. In turn, they'd know which performance improvements to prioritize during the next development cycle. If they could only view logs from dev/test, it would be more difficult to assess the real-world impact of performance issues detected during testing.

To put all of this another way, centralized logging is important to the core tenet of a DevOps culture because it de-silos the visibility data behind disparate parts of the CI/CD cycle. Teams that don't centralize logs have a more difficult time providing all stakeholders—developers, IT engineers, and anyone else who supports software delivery—with across-the-board visibility.

CI/CD, Microservices, and Centralized Logging

It's worth noting as well that centralized logging is particularly important in CI/CD environments where software is deployed into complex, microservices-based architectures. Although release teams don't have to use microservices, CI/CD and microservices tend to go hand-in-hand.

In a microservices environment, the number and complexity of logs increase significantly. Each service may produce its own logs. Each container instance that hosts services, as well as the orchestration tool that manages them, will typically produce logs, too. This means that there are many more logs—and many more types of logs—to manage than there are in a monolithic architecture.

The ability to centralize all of these logs is critical in order to manage them at scale. Otherwise, release teams would be left with a constellation of logs scattered across a complex environment, making it very difficult to collect and analyze them all.

Conclusion

Logging isn't part of the concept of CI/CD, exactly. But it's impossible to do CI/CD effectively without the help of centralized logging. By providing all stakeholders in the software delivery process with end-to-end visibility into all stages of software delivery, centralized logging helps ensure a smooth, continuous delivery process that breaks down the silos, enabling a DevOps way of working.