Kibana is a free, open-source front-end application part of the Elastic suite of products, specifically at the top of the Elastic (ELK) Stack, made up of Elasticsearch, Logstash, and Kibana. Because of its seamless integration with Elasticsearch and the ELK Stack as a whole, it sees use when turning volumes of data into search-friendly, indexable, and visually helpful information in real-time. Accessing the data indexed by Elasticsearch is made possible through Kibana and its integration to the ELK Stack. Consequently, understanding how to search Kibana means users can support logging and log analytics, container monitoring and infrastructure metrics, analyze, and visualize geospatial data, monitor application performance, and turn data into security and business analytics. 

Searching Logs 

The key to investigating the enormous amount of data being tracked and stored by Elasticsearch stems in part from understanding how to use Kibana to search your logs. This understanding makes it possible to explore and analyze any IT issue you might encounter. Pinpointing metadata from logs that match the type of information you’re searching for helps unlock the answers to any problem you might be experiencing, what caused these issues, and most importantly, how to solve them and prevent them from happening in the future.

You can access the Kibana search via the search bar at the top of the window. Kibana has its query language, KQL (Kibana Query Language), which Kibana converts into Elasticsearch Query DSL. Elasticsearch directly handles Lucene query language, as this is the same qwerty language that Elasticsearch uses to index its data. In Kibana, you can search between KQL and Lucene by clicking the label on the right side of the search box.

The challenge with finding information within Elasticsearch is that it may require parsing vast amounts of data, so specific basic search techniques may lead to results too numerous to solve a problem or find information in a short amount of time. And beyond this, it is also possible that certain types of query language may or may not be able to find the information being searched for, so it is essential to understand the different techniques for searching logs.

Finding Values

The most basic technique for finding results is a values search, where any document with a matching field of the words, integers, or terms entered into the search will appear. In this type of search, the exact term(s) must appear within the document for the query to find the desired results. For example, a search for the term strong will not call up results containing the word “stronger” or “strongest.”

One can further refine their value searches by using AND and OR, and parentheses to outline multiple terms that must appear. Therefore, a Kibana user could use this convention to call up results that contained multiple terms, such as fast and slow, or even stipulate that it included one specific term. They could also use a series of options to modify it, such as car AND (fast OR slow) to find results that contain car and either or both modifying words.

We can narrow the search even more by limiting them to specific categories. To accomplish this, we need to list the category that the search must match. For example, color: red. Incorporating quotation marks can also ensure that an entire phrase or expression turns up in a search result, for example, “fastest car in the world,” where a standard search would pull up results for any of those terms individually but not necessarily the entire phrase altogether.

Kibana Wildcard Searching

A more advanced search technique is called Wildcard Searching, or Wildcarding, which creates the ability to search some  variance in part of a search term that a user is providing. This technique makes use of the asterisk (*)  to use Kibana to search parts of words or phrases to find matching beginnings, middles, or endings of terms. For example, the search query fast* would yield results such as “fast,” “faster,” and “fastest.” Wildcarding is a valuable tool also for finding results from multiple fields, for instance, user.* : fast.  The downside to Wildcards is that you cannot use them for searching phrases; they are limited to results involving variance of one term, although you can use wildcards in conjunction with other search techniques to find more narrow results.

Fuzzy Queries and Proximity

Fuzzy Queries are a search tool that we can use to capture searches involving spelling errors or variance in spelling that can occur in specific entries. These can account for changing or omitting characters or even transposing terms. Fuzziness can be used within a certain distance, meaning it is possible to dictate search results within the stipulated number of characters. Entering fuzziness: 2 in conjunction with a search query would yield results within 2 characters variant, allowing a search for meet to turn up entries including meat, beet, and mean.

Proximity is similar to fuzziness, but it takes two search terms and allows them to fall within a predetermined number of words falling in between them. Proximity happens when using a tilde (~) and the number of words. So a search for fast car ~2 could return queries that could include fast German car and fast Italian sports car.  

Ranges and Comparing Values and Count Query

Searching numerical values in Kibana can be extremely useful as well. And while ranges and comparing values can prove useful for words, they are the most useful for searching numerical values. Range searches follow the same general syntax, falling in between brackets and divided by the word TO with the lowest number on the left and the highest number falling on the right. Square brackets [minimum TO maximum] yield results for inclusive ranges, where the high and low are in the query, where curly brackets {minimum TO maximum} yield exclusive ranges where the highest and lowest values don’t show in the results. Applications for this type of search can include

• Dates: date: [2021-01-01 TO 2021-12-31]
• Dates prior to or dates after: date: [* TO 2021-01-01], date:{2017:12-31 TO *]
• Range of numbers: [count: 1 TO 50]
• Numbers higher than a certain value: count: [5 TO *]
  

We can also find numerical ranges with one side completely open-ended using the greater than (>) or less than (<) sign, or either in conjunction with the equals sign (=) to find results for greater than or equal to, or less than or equal to, respectively. For example price >500 or time <=10.

Kibana count query functions exist as well to help retrieve unique numerical metrics related to search queries. The functions include single value aggregation, single-value, which counts the number of values extracted from a document, and value count, value_count, which does not duplicate values but counts each separate instance of the same value only once. We can use these in conjunction with other aggregators, such as average, avg, or scripts, to aggregate more complex values. These can also be used with histogram fields to create aggregations of all numbers in an array. 

Kibana Regex Searches

Regular expression queries, also known as regex queries, return search results that contain terms matching a regular expression. We use regex queries when matching data patterns that use placeholder characters, known as operators. Many regular expressions are supported within Kibana, which uses Lucene query syntax, and includes all Unicode characters. Although Lucene’s regex engine does not use the Perl Compatible Regular Expressions (PCRE) library, it does incorporate certain standard operators. These operators include matches for any character (.), repeating the preceding character zero or one times (?), repeating the preceding character one or more time (+), repeating the preceding character zero or more time (*) Minimum or maximum number of times the preceding character can repeat ( {xxx} ), OR operator (|), forming of a group ( … ), and matching one of the bracketed characters ( [xxx] ). We can use the flags parameter to enable more of the operators for Lucene’s regex engine. Valid values include all, complement, interval, intersection, and anystring. Lucene’s regex engine does not, however, support anchor operators, like ^ and $ to match a term.