If you know anything about Kubernetes monitoring, you know that Kubernetes is a mind-bogglingly complex system, and it is challenging to figure out what is happening inside it is very difficult. Simply collecting logs from Kubernetes can feel like an arduous task, let alone making sense of all of the logs that the various components of a Kubernetes cluster produce.

Fortunately, the complexity of Kubernetes monitoring is not impossible. You can master this discipline.

One of the first steps towards having this knowledge is knowing which tools are available to assist with Kubernetes monitoring. These include the native monitoring tools that Kubernetes provides and third-party monitoring and observability solutions that integrate with Kubernetes.

This article provides a concise overview of both categories of Kubernetes monitoring tools.

Native Kubernetes Monitoring Tools

Kubernetes is a container orchestration platform, of course, not a monitoring platform. It alone doesn’t provide all of the toolings that a team would need to monitor Kubernetes resources.

Nonetheless, Kubernetes offers specific tools that can assist in monitoring.

Kubectl

Kubectl, the CLI tool for managing Kubernetes clusters, provides useful commands for basic monitoring tasks.

For example, the following command displays information about the status of a pod:

kubectl describe pod POD NAME

‍

You can run a similar command to display output about node performance:

kubectl describe nodes

‍

Finally, the kubectl top command is handy if you want a glance at current resource utilization data:

kubectl top pod POD NAME

You can sort this output using external tools in your CLI environment (like grep). In addition, the kubectl --sort-by flag also allows you to control the output in ways that may be useful for monitoring. For instance, to display pod statistics sorted according to CPU usage, run:

kubectl top pod POD NAME --sort-by=cpu

Log Files on Nodes

The other easily accessible monitoring resource that Kubernetes provides natively is log files. It stores these files on the master and worker nodes in the cluster, typically in the /var/log directory.

While Kubernetes itself doesn’t provide unique tools for working with these log files, you can access them just as you would any server log file. Use less to display them, tail to display the most recent entries, grep to search for specific values, and so on.

Manually parsing through Kubernetes log files manually is not a practical way to monitor Kubernetes at scale, of course. But if you need to pull up some log data quickly, accessing these log files directly on the nodes comes in handy.

Third-Party Tools for Kubernetes Monitoring

The monitoring functionality that Kubernetes supports natively via tools like kubectl helps gain a quick overview of Kubernetes status, and it may suffice as your sole monitoring solution for tiny, non-production clusters.

To gain complete visibility into Kubernetes, you’ll almost certainly need to use a third-party monitoring tool. Third-party monitoring tools offer critical monitoring features that Kubernetes lacks natively, including:

• Automated log collection and aggregation: You can use external monitoring tools to collect logs instead of accessing or collecting each Kubernetes log manually (which is a lot of work given that all the nodes and pods ) automatically.
• Kubernetes audit logging: Kubernetes provides a robust monitoring framework in the form of auditing, which tracks requests to the Kubernetes API. However, Kubernetes doesn’t provide any native tools for collecting or analyzing audit data; it just provides the facilities for generating the data. To track audit events systematically, you’ll need external tools that can integrate with audit logs or webhooks.
• Storing ephemeral logs: Much of the log data generated in Kubernetes is non-persistent by default. Logs inside running containers disappear when the containers shut down. Even logs stored persistently on nodes will be overwritten after they exceed a mere 10 megabytes (the default maximum log size on most Kubernetes distributions) in most cases. With third-party monitoring tools, you can move logs to an external location and retain them as long as you need.

There are a variety of third-party monitoring tools for Kubernetes available today. Broadly speaking, some treat Kubernetes logs and metrics as they would any other data source. Others, like Mezmo, formerly known as LogDNA, offer unique features like Kubernetes Enrichment for collecting and interpreting Kubernetes logs.

You can also categorize Kubernetes monitoring tools based on which types of environments they support. Some only work with specific Kubernetes deployments; for example, most of the monitoring tools provided by public clouds only support the Kubernetes services that run in those clouds. Other tools are what you might call deployment-agnostic, meaning that they can monitor Kubernetes clusters running in any environment.

Conclusion: Kubernetes Monitoring Tools in a Nutshell

In short, while knowing your way around kubectl is helpful for fundamental Kubernetes monitoring, you’ll typically need an external monitoring tool to keep track of what is happening in complex, fast-changing Kubernetes clusters. The best external tools are those that include monitoring features tailored to the unique nature of Kubernetes. You’ll also want to consider whether the tool or tools you use can support any Kubernetes deployment or if they work only with certain distributions or hosting architectures.