VMware vCenter Log Insight Series Part4 - Creating Analytics

After a bit since last post here's the new article in which we will create some basic analytics from our previously gathered logs.

Interactive Analytics tab in Log Insight is the place you want to go in order to retrieve event information, aggregate logs and present them in a fancy dashboard.
By default Log Insight displays latest logs collected from any connected source ordered in order of occurrence.

In the search box you can insert any string you want in order to search for matches across all logs while in the drop down box near the search button you can also choose the time frame in which to retrieve results, such as latest 5 minutes of data, latest hour, latest 6 hours, etc. The cool aspect of searches is that autocomplete will give you word suggestion and matched words in log entries will be highlighted in order to allow you to fast browse across results.

Filters are probably the most effective way to provide a satisfying search among the (probably) huge amount of collected logs. Filters can enourmously help you in finding the log entry you are searching for. Filters can be added by clicking the green plus icon Add Filter and they can match any generic text inside the log entry or some pre-determined field. The classical logical operators: contains, does not contain, starts with, match regex, etc. can also be used.

Fields used in filters can be easily created, and saved for later uses, by selecting a specific detail presented in a log entry. Select Extract field in order to extract that particular information from the log.

Matched results will be highlighted in green. In the right menu you can also modify the regular expression if needed and choose a proper name to save your extracted field. In this example I used device name as field name.

Extracted fields can now be used in queries.

Let's now produce some dashboard with our retrieved logs. Every log search will automatically produce a bar chart in the upper part of the page. By default Log Insight displays the count of events over time so the displayed bar indicates in a certain time like day, hour, or minute, the number of occurrences of that particular event.

Charts can also be customized, you can retrieve a certain metric grouped by a particular entity. In this example I asked for the average number of scsi latency events grouped by the filter I previously created (device name). With this chart I have a graphical view on the average count of scsi device related events and these are displayed grouped by the disk device name.

Charts aspect can also be modified to fit your style. Depending on the data you are extracting you can choose to represent data as a column like chart, a line chart, area, bar, pie or bubble based.

Once you are satisfied with the obtained result you can save your chart in a dashboard, which basically is a page on which you can have multiple charts grouped together. Before adding a chart to a dashboard you can choose chart title and optionally insert a brief description of what the cart represents.

Your dashboards are accessible by clicking Dashboards button on the top of the pages and by navigating in the left-sided menu.

This could be an example of a populated dashboard, a single pane of glass on which sysadmins can have informations at a rapid glance about how well things are going.

Other articles in this series:
VMware vCenter Log Insight Series Part1 - Introduction
VMware vCenter Log Insight Series Part2 - Installation and Configuration
VMware vCenter Log Insight Series Part3 - Collecting Logs
VMware vCenter Log Insight Series Part4 - Creating Analytics

VMware vCenter Log Insight Series Part3 - Collecting Logs

The most important step in analyzing logs is...collecting logs. In this post I will briefly guide you in how you can collect logs in vCenter Log Insight.
As previously said, Log Insight features a syslog collector, this means that it can ingests logs from literally anything: ESXi hosts, physical infrastructure, virtual machine guest O.S., application logs, etc.

While configuring ESXi hosts and vCenter log collection is an automated task that can be easily performed from within Log Insight itself as seen in previous post, other devices must be manually configured in order to send logs over to Log Insight.

Here's how to collect logs from other sources rather than ESXi hosts and vCenter Server:

1)Windows operating system:

vCenter Log Insight provides a native agent to be used on Windows to collect logs. Windows agent can be retrieved from Log Insight Administration -> Agents page. Agent is a small executable that can be installed on any Windows machine you want to retrieve logs from.

After agent installation has succeeded logs are immediately sent to Log Insight to be collected. Please check Administration -> Agents page to verify that your Windows server has been successfully registered.

Agent is installed by default in C:\ProgramData\VMware\Log Insight Agent. liagent.conf is the configuration file in which you can configure several parameters related on logs collecting such as what kind of logs to collect and what event severity to collect.
Windows agent can also be used to collect specific applications logs from software, residing on the same machine on which agent has been installed, such as Microsoft SQL logs, Apache logs, etc.
This can be done specifying in liagent.conf file the path where to retrieve application specific logs. For further informations regarding app logs collecting have a look at this Log Insight documentation page.

Agent configuration can also be performed in a centralized way via Log Insight Administration -> Agents page. These settings are server-side and will be applied to every client connected to Log Insight, these settings are more authoritative compared to the ones set on client-side, this means that if same setting exists both on client-side and on server-side, the latter is the one that is valid. Otherwise client and server-side settings are merged to create a resulting configuration.

2)Linux operating system:

We will use Rsyslog to send logs from Linux over to Log Insight. Rsyslog can also be used to create a centralized logging server when deploying Log Insight in an hub-spoke architecture.
Rsyslog installation and configuration is quite simple. On RHEL 6 and later is installed by default, on previous versions and on RHEL flavoured distros you can install it using:

 yum install rsyslog  

On Debian and similar distros, such as Ubuntu, installation can be performed with:

 apt-get install rsyslog  

Once installed configure it by editing /etc/rsyslog.conf file. I prefer to use UDP for log transferring even if TCP is also a viable protocol. To enable UDP modules on Rsyslog edit rsyslog.conf by removing comments ("#" at line start) on:

 $ModLoad imudp  
 $UDPServerRun 514  

Add this line at the end of file in order to send logs over to Log Insight:

 *.*  @Log_Insight_IP_Address_or_FQDN:514  

Save file and restart Rsyslog service. If needed add an exception to Linux firewall in order to open port 514 UDP.

 iptables -A INPUT -m state --state NEW -m udp -p udp --dport 514 -j ACCEPT  

Verify that Rsyslog will be automatically started on boot. On RHEL-like distros you can do this with:

 chkconfig rsyslog on  

3)Physical Devices:

Physical devices such as switches, routers, storage nodes and servers typically support sending logs to a remote destination. This of course vary depending on device model and manufacturer and the appropriate documentation must be consulted in order to find out how to offload logs to an external syslog.
This process is usually quite simple and can be done either via a graphical management console either via CLI.

As a general example here's how to configure an HP 29xx physical switch to send logs over to Log Insight.

Connect to switch management interface either via serial cable either via SSH. Enable offsite logging with the following command:

 logging <Log_Insight_IP_Address_or_FQDN>  

That's all. In next post we will get some useful informations by analyzing logs.

Other articles in this series:
VMware vCenter Log Insight Series Part1 - Introduction
VMware vCenter Log Insight Series Part2 - Installation and Configuration
VMware vCenter Log Insight Series Part3 - Collecting Logs
VMware vCenter Log Insight Series Part4 - Creating Analytics

VMware vCenter Log Insight Series Part2 - Installation and Configuration

In this second post regarding VMware vCenter Log Insight I will explain how to deploy it and perform initial setup and configuration.
As first thing bookmark the VMware vCenter Log Insight documentation.

Log Insight installation is a quite straightforward process. Download the OVA file from VMware.com and deploy it in the classic way.
During deployment you will be prompted to choose your deployment size. As discussed in previous article deployment size is related to how much data Log Insight is capable to ingest in a certain amount time. This obviously directly influences the requested CPU number and RAM size to support this certain kind of workload. Please note that the Extra Small configuration option is not supported in production but is only suited for demonstrations only.

Configure IP address, proper subnet mask, gateway, DNS and appliance root password.

Before continuing let's have a brief look at how Log Insight stores data. Once OVF has been deployde if you edit Log Insight virtual machine you will see something similar to this:

Three hard disks are used to store operating system data, application logs and, finally, logs and indexes received by Log Insight. By default, if Small configuration is used, application logs and data are stored in an LVM formatted 120GB disk.

/storage/core is where logs and indexes are stored. This volume can be increased by simply editing the virtual appliance hard disk size. When Log Insight is powered back on LVM will be automatically expanded and will reflect the new size.

Once Log Insight has booted completely point to it using a web browser and you can perform the initial configuration.

Here you can choose either to start a new deployment either to join an existing one. The latter option is used when adding worker nodes to an existing Log Insight cluster and we will see this in detail in another post so at this time I choose the former one.

Insert admin credentials.

Insert license key. You can skip this step at this time and insert the license later but a valid license key must inserted in order to access to collected logs and analytics.

Time configuration is an optional step but I suggest you to configure it to work properly since timestamps in log analysis can be a lifesaver.

SMTP is used by Log Insight to send alerts via email

When setup is complete you will be redirected to the main page of VMware vCenter Log Insight.

In order to get log data from vCenter Server and ESXi hosts connected to it you need to go to Administration -> vSphere and properly fill in the fields pointing to you vCenter Server then check both options in this page. This will automatically retrieve logs from vCenter and configure ESXi hosts to send logs to this instance of Log Insight.

NOTE: If you need to check if an ESXi host sends logs over to Log Insight just connect to it via SSH and run:

 esxcli system syslog config get  

To perform a manual configuration for sending logs from an ESXi host to Log Insight or to a generic syslog run the following command:

 esxcli system syslog config set --loghost='udp://<IP_OF_LOGINSIGHT:514>'  

That's all for now, in the next article we will start using vCenter Log Insight for analyzing logs.


Other articles in this series:
VMware vCenter Log Insight Series Part1 - Introduction
VMware vCenter Log Insight Series Part2 - Installation and Configuration
VMware vCenter Log Insight Series Part3 - Collecting Logs 
VMware vCenter Log Insight Series Part4 - Creating Analytics

VMware vCenter Log Insight Series Part1 - Introduction

It has been a while since my last post, so I decided to write some new content regarding my latest work. Recently I've been intensively working with VMware vCenter Log Insight 2.0 and I've to admit that this is a really valuable tool, so I decided to start a small post series regarding Log Insight.

In this first post I'll give you a brief introduction about this great piece of software.

Log Insight is an analytics tool that acts as a centralized logs server and is deployed in a VMware environment as a virtual appliance. The cool aspect of Log Insight is that it supports the collection of logs either from VMware infrastructure (i.e. ESXi hosts) either from physical infrastructure (i.e. physical servers, physical switches, etc.) either from application (i.e. virtual/physical machines guest operating systems).

Log Insight provides an easy to use interface to analyze, search, and aggregate informations from logs. It supports real-time complex queries against massive log datasets all accessible by using a web browser.

Log Insight use cases are:

-Identify issues during troubleshooting.
-Gain visibility across all infrastructure, either physical, either virtual, from a single log collection point.

Log Insight can be deployed following two reference architectures:

1)Centralized Log Management: where every log is sent directly over to Log Insight. This is best suited to be used in small to medium size environments.

2)Hub-Spoke: this is usually adopted in larger environments, even geographically dispersed, where all logs are first sent over to a syslog collector and only later are offloaded from the syslog collector over to VMware Log Insight.

Log Insight itself is designed to be scaled up and out. Scale-up capability is provided by supporting the size increase of the virtual appliance partition used for storing logs. Scale-out is achieved by native cluster support. Log Insight supports up to six nodes clustered to provide greater throughput and greater availability. A master node exists within the cluster and up to five slaves, referred as worker nodes, can exist.

During the installation of Log Insight you will be prompted to select the appliance size. According to the size of the infrastructure you are up to monitor you can either choose from Small to Large sizing deployment.

Small deployment requires 4 CPUs and 8GB of RAM and it supports the processing of up to 3GB of log files per day (with an average of 200 logs entries per second). Large deployment supports up to 113GB of log files per day with an average of 7500 log entries per second. It also requires 16CPUs and 32GB of RAM.

That's all for this first post, in the coming up articles I will guide you through installation and initial setup, configurations and analytics creation.

Other articles in this series:
VMware vCenter Log Insight Series Part1 - Introduction
VMware vCenter Log Insight Series Part2 - Installation and Configuration 
VMware vCenter Log Insight Series Part3 - Collecting Logs  
VMware vCenter Log Insight Series Part4 - Creating Analytics