Containers have become the standard framework for delivering microservices-based applications at scale, necessitating the development of more powerful logging and monitoring technologies. Kubernetes has established itself as the de facto container orchestration platform and a vital component of the cloud-native trend. With hundreds of microservices operating on thousands of containers in ephemeral and disposable pods, cloud-native delivers speed, elasticity, and agility to software development; but it also adds complexity. Monitoring such a large, distributed, dynamic system is complex and critical. Fortunately, Mezmo makes gaining observability into your Kubernetes system a breeze.

This tutorial will explain how you can maximize observability in your applications running in Kubernetes and gain control of your data using Mezmo.

Mezmo and Kubernetes Observability

The strength of Kubernetes is also one of its flaws. It abstracts away a lot of complexity to expedite deployment. Still, it blinds you to what's happening, the resources in use, and even the financial consequences of your activities. Furthermore, Kubernetes has far more components — such as servers and services — than traditional infrastructure, making root cause research considerably more complicated when something goes wrong. You'll need an external log collection, analysis, and management platform like Mezmo to get the most out of Kubernetes logging for efficient observability.

Kubernetes Enrichment is a Mezmo feature that provides native, out-of-the-box support for displaying Kubernetes events and metrics alongside log data. That means DevOps teams can obtain visibility into their Kubernetes clusters without writing complex custom queries or separating Kubernetes metrics from other observability data. Upon integration, Mezmo acts as a third-party solution that ensures the centralized logging of pods and clusters and their nodes and resources within the Kubernetes system.

Through Kubernetes Agent and Kubernetes Enrichment, Mezmo's solution gives teams control over their data by centralizing logs from all sources and providing a 360-degree holistic view of their systems and applications. By offering security, compliance, and the automatic archiving of log data in S3, this strategy aids teams in maximizing observability in their Kubernetes settings. Mezmo offers advantages, including the flexibility to prioritize security and assured compliance with SOC 2 Type 2, PCI DSS Level 1, HIPAA, GDPR, Privacy Shield, and CCPA.

How To Achieve Kubernetes Observability With Mezmo

Mezmo is an essential component of the observability stack, providing actionable visibility into Kubernetes clusters so that developers aren't overwhelmed with information they can't utilize. Kubernetes clusters generate a large amount of data, which is beneficial since more data implies greater visibility into the environment. Still, it's also critical that our clients have time-saving shortcuts to actionability.

When determining what to log in to Kubernetes, there are numerous log data types to choose from, including:

Application Logs

Use application logs to gain visibility into what is happening inside the applications running on Kubernetes. This type of log writes log data in the stdout inside the container.

Run the command below to get access to the application logs in Kubernetes. Make sure you specify the pod's name that hosts the app whose logs you wish to view.

Kubernetes Cluster Logs

Several Kubernetes components, including etcd, kube-apiserver, kube-proxy, kubelet, and kube-scheduler, generate their logs at the cluster level. These logs help debug cluster-level problems and give insight into what's going on within Kubernetes clusters.

IBM Cloud's Mezmo integration uses a centralized interface to collect Kubernetes cluster logs and application logs. Then it analyzes them- a far simpler solution than the tedious and time-consuming process of collecting individual logs from each of your nodes via the command line.

Kubernetes Events

Kubernetes keeps track of "events," which are essentially changes in the cluster's object states (such as creating or starting a container) and resource errors. Event logs enable quick access to information about the condition of various objects in your cluster with limited visibility.

To view all event logs, run the following command:

To view pod-specific event logs, you have to specify the pod name, as shown below:

Kubernetes Audit Logs 

You can set the kube-apiserver to log requests in Kubernetes. These are custom requests to Kubernetes resources and running pods. Audit logs provide a lot of visibility if you need to debug a problem with an API call. You can also use them to detect unusual behavior by searching for strange requests, such as a user's repeated failed attempts to access different resources in the cluster. Incidents like this could indicate attempted abuse by someone hunting for poorly secured resources.

Observability with Kubernetes Enrichment

You can use Kubernetes Enrichment to view Kubernetes events, logs, and metrics in Mezmo. You can solve issues faster and skip context switching between logs and other tools because all Kubernetes events, metrics, clusters, and application logs are centralized. The native monitoring feature provided by Kubernetes via tools like kubectl aids in gaining a rapid overview of the Kubernetes state; however, it may not be sufficient for optimal observability. By understanding Kubernetes logs and metrics data, Kubernetes Enrichment delivers new capabilities beyond log management. 

‍

‍

See the methods below for achieving real-time observability into various Kubernetes components.

Viewing Kubernetes Metrics in Logs

You can view Kubernetes metrics directly in the log viewer context menu. With this metadata and these metrics, you'll have a point-in-time snapshot of the Kubernetes cluster state when you sent the logline. Regardless of when an event occurred, the metrics reflect what was happening.

‍

‍

By clicking the arrow next to KUBE STATS in the upper left-hand corner, you can see more node information, including the pod image, namespace, and controller. You can use this information to quickly see resource constraints, unstable pod deployments, or unhealthy nodes that were present when you sent that logline. All metrics stay for the same amount of time, and they are currently collected every 30 seconds.

Viewing Kubernetes Event Logs

Kubernetes Enrichment includes views for categorizing and surfacing Kubernetes events. You can easily find and view historical or current Kubernetes events if your Kubernetes events are in Mezmo. Kubernetes events and the "view in context" option allow you to see the log lines preceding a specific event quickly.

‍

‍

Viewing Kubernetes Cluster-Wide Health

You can get an overview of cluster-level metrics and identify strange behavior in your cluster by using the Screens included in Kubernetes Enrichment. It may take up to 24 hours for metrics to populate on Screens.

‍

Here, you can find a more extensive guide to configuring Kubernetes Enrichment for optimized observability.

Achieving Kubernetes Observability with Mezmo

Kubernetes offers significant benefits for modern cloud-based applications, but organizations need a new approach to monitoring to reap those benefits fully. Kubernetes presents unique observability challenges, and conventional monitoring techniques are not enough to gain insights into cluster health and resource allocation. By understanding the complexities of Kubernetes monitoring, you can better identify a solution that will allow you to derive more value from your Kubernetes deployment.

Given the complexity of Kubernetes systems, Mezmo delivers a simplified log management solution that can help DevOps teams gather insights from systems, applications, and networks. Kubernetes observability through Mezmo also enables you to enhance your security posture, troubleshoot issues faster, and optimize resource utilization. Kubernetes Enrichment provides a single-pane-of-glass observability experience into your underlying Kubernetes infrastructure and deployed services.

‍