VMware vCenter Log Insight Series Part4 - Creating Analytics

After a bit since last post here's the new article in which we will create some basic analytics from our previously gathered logs.

Interactive Analytics tab in Log Insight is the place you want to go in order to retrieve event information, aggregate logs and present them in a fancy dashboard.
By default Log Insight displays latest logs collected from any connected source ordered in order of occurrence.

In the search box you can insert any string you want in order to search for matches across all logs while in the drop down box near the search button you can also choose the time frame in which to retrieve results, such as latest 5 minutes of data, latest hour, latest 6 hours, etc. The cool aspect of searches is that autocomplete will give you word suggestion and matched words in log entries will be highlighted in order to allow you to fast browse across results.

Filters are probably the most effective way to provide a satisfying search among the (probably) huge amount of collected logs. Filters can enourmously help you in finding the log entry you are searching for. Filters can be added by clicking the green plus icon Add Filter and they can match any generic text inside the log entry or some pre-determined field. The classical logical operators: contains, does not contain, starts with, match regex, etc. can also be used.

Fields used in filters can be easily created, and saved for later uses, by selecting a specific detail presented in a log entry. Select Extract field in order to extract that particular information from the log.

Matched results will be highlighted in green. In the right menu you can also modify the regular expression if needed and choose a proper name to save your extracted field. In this example I used device name as field name.

Extracted fields can now be used in queries.

Let's now produce some dashboard with our retrieved logs. Every log search will automatically produce a bar chart in the upper part of the page. By default Log Insight displays the count of events over time so the displayed bar indicates in a certain time like day, hour, or minute, the number of occurrences of that particular event.

Charts can also be customized, you can retrieve a certain metric grouped by a particular entity. In this example I asked for the average number of scsi latency events grouped by the filter I previously created (device name). With this chart I have a graphical view on the average count of scsi device related events and these are displayed grouped by the disk device name.

Charts aspect can also be modified to fit your style. Depending on the data you are extracting you can choose to represent data as a column like chart, a line chart, area, bar, pie or bubble based.

Once you are satisfied with the obtained result you can save your chart in a dashboard, which basically is a page on which you can have multiple charts grouped together. Before adding a chart to a dashboard you can choose chart title and optionally insert a brief description of what the cart represents.

Your dashboards are accessible by clicking Dashboards button on the top of the pages and by navigating in the left-sided menu.

This could be an example of a populated dashboard, a single pane of glass on which sysadmins can have informations at a rapid glance about how well things are going.

Other articles in this series:
VMware vCenter Log Insight Series Part1 - Introduction
VMware vCenter Log Insight Series Part2 - Installation and Configuration
VMware vCenter Log Insight Series Part3 - Collecting Logs
VMware vCenter Log Insight Series Part4 - Creating Analytics

VMware vCenter Log Insight Series Part3 - Collecting Logs

The most important step in analyzing logs is...collecting logs. In this post I will briefly guide you in how you can collect logs in vCenter Log Insight.
As previously said, Log Insight features a syslog collector, this means that it can ingests logs from literally anything: ESXi hosts, physical infrastructure, virtual machine guest O.S., application logs, etc.

While configuring ESXi hosts and vCenter log collection is an automated task that can be easily performed from within Log Insight itself as seen in previous post, other devices must be manually configured in order to send logs over to Log Insight.

Here's how to collect logs from other sources rather than ESXi hosts and vCenter Server:

1)Windows operating system:

vCenter Log Insight provides a native agent to be used on Windows to collect logs. Windows agent can be retrieved from Log Insight Administration -> Agents page. Agent is a small executable that can be installed on any Windows machine you want to retrieve logs from.

After agent installation has succeeded logs are immediately sent to Log Insight to be collected. Please check Administration -> Agents page to verify that your Windows server has been successfully registered.

Agent is installed by default in C:\ProgramData\VMware\Log Insight Agent. liagent.conf is the configuration file in which you can configure several parameters related on logs collecting such as what kind of logs to collect and what event severity to collect.
Windows agent can also be used to collect specific applications logs from software, residing on the same machine on which agent has been installed, such as Microsoft SQL logs, Apache logs, etc.
This can be done specifying in liagent.conf file the path where to retrieve application specific logs. For further informations regarding app logs collecting have a look at this Log Insight documentation page.

Agent configuration can also be performed in a centralized way via Log Insight Administration -> Agents page. These settings are server-side and will be applied to every client connected to Log Insight, these settings are more authoritative compared to the ones set on client-side, this means that if same setting exists both on client-side and on server-side, the latter is the one that is valid. Otherwise client and server-side settings are merged to create a resulting configuration.

2)Linux operating system:

We will use Rsyslog to send logs from Linux over to Log Insight. Rsyslog can also be used to create a centralized logging server when deploying Log Insight in an hub-spoke architecture.
Rsyslog installation and configuration is quite simple. On RHEL 6 and later is installed by default, on previous versions and on RHEL flavoured distros you can install it using:

 yum install rsyslog  

On Debian and similar distros, such as Ubuntu, installation can be performed with:

 apt-get install rsyslog  

Once installed configure it by editing /etc/rsyslog.conf file. I prefer to use UDP for log transferring even if TCP is also a viable protocol. To enable UDP modules on Rsyslog edit rsyslog.conf by removing comments ("#" at line start) on:

 $ModLoad imudp  
 $UDPServerRun 514  

Add this line at the end of file in order to send logs over to Log Insight:

 *.*  @Log_Insight_IP_Address_or_FQDN:514  

Save file and restart Rsyslog service. If needed add an exception to Linux firewall in order to open port 514 UDP.

 iptables -A INPUT -m state --state NEW -m udp -p udp --dport 514 -j ACCEPT  

Verify that Rsyslog will be automatically started on boot. On RHEL-like distros you can do this with:

 chkconfig rsyslog on  

3)Physical Devices:

Physical devices such as switches, routers, storage nodes and servers typically support sending logs to a remote destination. This of course vary depending on device model and manufacturer and the appropriate documentation must be consulted in order to find out how to offload logs to an external syslog.
This process is usually quite simple and can be done either via a graphical management console either via CLI.

As a general example here's how to configure an HP 29xx physical switch to send logs over to Log Insight.

Connect to switch management interface either via serial cable either via SSH. Enable offsite logging with the following command:

 logging <Log_Insight_IP_Address_or_FQDN>  

That's all. In next post we will get some useful informations by analyzing logs.

Other articles in this series:
VMware vCenter Log Insight Series Part1 - Introduction
VMware vCenter Log Insight Series Part2 - Installation and Configuration
VMware vCenter Log Insight Series Part3 - Collecting Logs
VMware vCenter Log Insight Series Part4 - Creating Analytics

VMware vCenter Log Insight Series Part2 - Installation and Configuration

In this second post regarding VMware vCenter Log Insight I will explain how to deploy it and perform initial setup and configuration.
As first thing bookmark the VMware vCenter Log Insight documentation.

Log Insight installation is a quite straightforward process. Download the OVA file from VMware.com and deploy it in the classic way.
During deployment you will be prompted to choose your deployment size. As discussed in previous article deployment size is related to how much data Log Insight is capable to ingest in a certain amount time. This obviously directly influences the requested CPU number and RAM size to support this certain kind of workload. Please note that the Extra Small configuration option is not supported in production but is only suited for demonstrations only.

Configure IP address, proper subnet mask, gateway, DNS and appliance root password.

Before continuing let's have a brief look at how Log Insight stores data. Once OVF has been deployde if you edit Log Insight virtual machine you will see something similar to this:

Three hard disks are used to store operating system data, application logs and, finally, logs and indexes received by Log Insight. By default, if Small configuration is used, application logs and data are stored in an LVM formatted 120GB disk.

/storage/core is where logs and indexes are stored. This volume can be increased by simply editing the virtual appliance hard disk size. When Log Insight is powered back on LVM will be automatically expanded and will reflect the new size.

Once Log Insight has booted completely point to it using a web browser and you can perform the initial configuration.

Here you can choose either to start a new deployment either to join an existing one. The latter option is used when adding worker nodes to an existing Log Insight cluster and we will see this in detail in another post so at this time I choose the former one.

Insert admin credentials.

Insert license key. You can skip this step at this time and insert the license later but a valid license key must inserted in order to access to collected logs and analytics.

Time configuration is an optional step but I suggest you to configure it to work properly since timestamps in log analysis can be a lifesaver.

SMTP is used by Log Insight to send alerts via email

When setup is complete you will be redirected to the main page of VMware vCenter Log Insight.

In order to get log data from vCenter Server and ESXi hosts connected to it you need to go to Administration -> vSphere and properly fill in the fields pointing to you vCenter Server then check both options in this page. This will automatically retrieve logs from vCenter and configure ESXi hosts to send logs to this instance of Log Insight.

NOTE: If you need to check if an ESXi host sends logs over to Log Insight just connect to it via SSH and run:

 esxcli system syslog config get  

To perform a manual configuration for sending logs from an ESXi host to Log Insight or to a generic syslog run the following command:

 esxcli system syslog config set --loghost='udp://<IP_OF_LOGINSIGHT:514>'  

That's all for now, in the next article we will start using vCenter Log Insight for analyzing logs.


Other articles in this series:
VMware vCenter Log Insight Series Part1 - Introduction
VMware vCenter Log Insight Series Part2 - Installation and Configuration
VMware vCenter Log Insight Series Part3 - Collecting Logs 
VMware vCenter Log Insight Series Part4 - Creating Analytics

VMware vCenter Log Insight Series Part1 - Introduction

It has been a while since my last post, so I decided to write some new content regarding my latest work. Recently I've been intensively working with VMware vCenter Log Insight 2.0 and I've to admit that this is a really valuable tool, so I decided to start a small post series regarding Log Insight.

In this first post I'll give you a brief introduction about this great piece of software.

Log Insight is an analytics tool that acts as a centralized logs server and is deployed in a VMware environment as a virtual appliance. The cool aspect of Log Insight is that it supports the collection of logs either from VMware infrastructure (i.e. ESXi hosts) either from physical infrastructure (i.e. physical servers, physical switches, etc.) either from application (i.e. virtual/physical machines guest operating systems).

Log Insight provides an easy to use interface to analyze, search, and aggregate informations from logs. It supports real-time complex queries against massive log datasets all accessible by using a web browser.

Log Insight use cases are:

-Identify issues during troubleshooting.
-Gain visibility across all infrastructure, either physical, either virtual, from a single log collection point.

Log Insight can be deployed following two reference architectures:

1)Centralized Log Management: where every log is sent directly over to Log Insight. This is best suited to be used in small to medium size environments.

2)Hub-Spoke: this is usually adopted in larger environments, even geographically dispersed, where all logs are first sent over to a syslog collector and only later are offloaded from the syslog collector over to VMware Log Insight.

Log Insight itself is designed to be scaled up and out. Scale-up capability is provided by supporting the size increase of the virtual appliance partition used for storing logs. Scale-out is achieved by native cluster support. Log Insight supports up to six nodes clustered to provide greater throughput and greater availability. A master node exists within the cluster and up to five slaves, referred as worker nodes, can exist.

During the installation of Log Insight you will be prompted to select the appliance size. According to the size of the infrastructure you are up to monitor you can either choose from Small to Large sizing deployment.

Small deployment requires 4 CPUs and 8GB of RAM and it supports the processing of up to 3GB of log files per day (with an average of 200 logs entries per second). Large deployment supports up to 113GB of log files per day with an average of 7500 log entries per second. It also requires 16CPUs and 32GB of RAM.

That's all for this first post, in the coming up articles I will guide you through installation and initial setup, configurations and analytics creation.

Other articles in this series:
VMware vCenter Log Insight Series Part1 - Introduction
VMware vCenter Log Insight Series Part2 - Installation and Configuration 
VMware vCenter Log Insight Series Part3 - Collecting Logs  
VMware vCenter Log Insight Series Part4 - Creating Analytics

VMware: alarm notifications using PowerCLI and IFTTT

IFTTT is the short name for If-This-Than-That, a website and an Android/iOS app, that automates task based on the triggering of a certain condition. The idea behind that is really simple yet really powerful: if a <condition> is verified then <do something>.
In this post we are gonna use PowerCLI and IFTTT to provide push notifications in case of an alarm or a warning is triggered in vSphere.

What we are up to do is basically:

-Use a PowerCLI script, intended to run as a scheduled task, to check if alarms or warnings are triggered in our vSphere environment.
-Generate an RSS Atom feed with all alarms/warnings (if any). The feed file must be reachable by the internet via an IP address/DNS name.
-Create a new recipe in IFTTT to check for our RSS feed updates.

If a new alarm or warning is triggered in vSphere the RSS feed will be updated, IFTTT will notify the change and will provide a push notification to the user informing that something that needs to be addressed is happening in vSphere.

Let's start with the PowerCLI script: it checks for any warning/alarm and generates as output an RSS Atom feed which will be monitored by IFTTT in search of updates. The idea is to add this script to a scheduled task which runs automatically and possibly frequently. The $OutputFile variable is where the RSS feed is generated. IFTTT would require an internet accessible feed so in my case I point $OutputFile to the root of one website running on IIS on the same machine that runs the script. Of course the website is publicly accessible.

This is the code, as usual you can also find it in my GitHub Page. IFTTT_Alarms.ps1



The output RSS feed is generated by concatenating three parts: the header which has the xml declaration and the opening tags, the body, which is the main part that is dynamically generated based on any alarm/warning triggered in vSphere, and the footer which closes the feed.

If no alarms or warnings are (hopefully) triggered in your vSphere environment the resulting feed will be like this:

 <?xml version="1.0" encoding="utf-8"?>  
 <feed xmlns="http://www.w3.org/2005/Atom">  
   <title>Triggered Alarms Feed</title>  
   <subtitle>This feed contains all triggered alarms and warnings</subtitle>  
   <link href="http://hostilecoding.blogspot.com" rel="self" />  
   <link href="http://hostilecoding.blogspot.com/" />  
   <updated>2014-06-13T22:38:45.0020183+02:00</updated>  
 </feed>  

If there is any alarm the resulting feed will be similar to this:

 <?xml version="1.0" encoding="utf-8"?>  
 <feed xmlns="http://www.w3.org/2005/Atom">  
      <title>Triggered Alarms Feed</title>  
      <subtitle>This feed contains all triggered alarms and warnings</subtitle>  
      <link href="http://hostilecoding.blogspot.com" rel="self" />  
      <link href="http://hostilecoding.blogspot.com/" />  
      <updated>2014-06-13T22:38:45.0020183+02:00</updated>  
 <entry>  
      <title>Alarm: 192.168.243.144</title>  
                <link href="http://hostilecoding.blogspot.com/" />  
                <link rel="alternate" type="text/html" href="http://hostilecoding.blogspot.com/"/>  
                <link rel="edit" href="http://hostilecoding.blogspot.com/"/>  
      <updated>2014-06-13T22:38:45.0020183+02:00</updated>  
      <summary>Alarm Triggered.</summary>  
                <content type="xhtml">  
                  <div xmlns="http://www.w3.org/1999/xhtml">  
                      <p>Network uplink redundancy lost</p>  
                  </div>  
                </content>  
                <author>  
                      <name>Administrator@vsphere.local</name>  
             </author>  
 </entry>  
 </feed>  

The <entry> part contains the informations about the alarm, which host the alarm refers to, description, and some other infos. Links are present because IFTTT was a bit sensitive to the feed format, if no links are in the feed IFTTT would not recognize it as having a correct syntax and will report an error during recipe creation.

Let's now jump to IFTTT recipe creation, I used Android version but I guess  the iOS counterpart would be identical.

Create a new recipe:

Look for RSS feed -> Neew Feed Item

Insert the URL of the RSS feed. I used feeburner.com to map the public IP address of my IIS server to a URL. I guess you could simply use your IP address.

Android Notifications -> Send Notification

This is the look of the complete recipe:

So, when an alarm/warning will be detected the PowerCLI script will update the feed, IFTTT will notice the new feed entry and will generate a push notification similar to this:

That's all!!

VMware: Automate support bundle collection using PowerCLI

When troubleshooting ESXi issues, especially when dealing with VMware technical support, you (almost) always will be asked to provide support bundles for ESXi host(s) and/or the managing vCenter Server.

Support Bundles can be easily generated by accessing vSphere Client -> File -> Export -> Export System Logs although the process is manual and potentially time consuming.

As almost everything in VMware environment even the support bundle collection process can be easily automated using PowerCLI.
The Get-Log PowerCLI cmdlet is used to retrieve support bundle either from vCenter Server either from ESXi hosts.

When connected to a vCenter Server running:

 Get-Log -Bundle -DestinationPath C:\Users\Paolo\Desktop  

will download in specified location support bundle for vCenter Server.

Support bundle will be named: vcsupport-<Generated_ID>.zip

To download support bundles for specific ESXi host(s) only:

 Get-Log -VMHost (Get-VMHost -Name 192.168.243.144) -Bundle -DestinationPath C:\Users\Paolo\Desktop  

where 192.168.243.144 is the IP address of ESXi host to download support bundle from.

This time file will be named: vmsupport-<Generated_ID>.tgz

To automate bundles retrieval for any ESXi host managed by a specific vCenter Server, including support bundle for vCenter itself, you can use the following script:



As usual you can also find the script above in my GitHub page: DownloadSupportBundles.ps1

Support bundle download is also achieved by simply connecting, using a web browser, to an ESXi host to the following address:

 https://<username>:<password>@<ESXi_host_IP_address>/cgi-bin/vm-support.cgi  

The previous step can be automated using PowerShell (not PowerCLI). This could be useful when in need to get support bundles rapidly using a PC/server where PowerCLI is not installed (only PowerShell is needed).



That's all!!

VMware: A PowerCLI module to massively backup/restore ESXi hosts configurations

Flexibility is one of the greatest advantages of ESXi. Almost every aspect can be customized and tuned using both basic and advanced configurations in order to achieve a custom tailored system.

Configuring settings is a time consuming process, networking configurations for virtual standard switches, iSCSI vmkernel, port binding, NTP configuration, etc.

In case of host reinstall you can save precious time by using a great PowerCLI cmdlet: Get-VMHostFirmware.

This cmdlet creates a tar compressed archive containing all ESXi host's configurations. Conversely, to recover a backupped configuration, Set-VMHostFirmware cmdlet is used.

In this blog post I provide a PowerCLI module that will allow you to backup, and eventually restore, ESXi hosts configurations.

This module uses Get-VMHostFirmware and Set-VMHostFirmware cmdlets introducing the possibility to pass more than a single host as source for backup or target for restore and automatically enters/exits each host into/from maintenance mode before and after a backup restore occurs.

Let's start by briefly explaining how to use the module:

Typically the first step is to connect to a vCenter Server via PowerCLI in order to be able to perform backup or restores of one or more vCenter registered hosts. PowerCLI connection to a single ESXi host is also supported but for obvious reasons you can backup/restore only that specific host.

Once downloaded the script provided below you will have a .psm1 file, which is the common extension for PowerShell modules, that must be imported into PowerCLI in order to use it.

 Import-Module C:\Users\Paolo\WindowsPowerShell\Modules\BackupRestore  

Where C:\Users\Paolo\WindowsPowerShell\Modules\ is the path of BackupRestore.psm1 file on your PC.

BackupRestore module introduces into PowerCLI two new functions: Backup-VMHost and Restore-VMHost.

Backup-VMHost requires as mandatory parameters:

-VMHost: backup source. IP address or FQDN of one or more ESXi hosts you want to backup configurations from.
-FilePath: location where configuration bundles will be saved.

The following example backups the configuration of host 192.168.243.143 and 192.168.243.144 then save their configurations into C:\Users\Paolo\Desktop.

 Backup-VMHost -VMHost 192.168.243.143,192.168.243.144 -FilePath C:\Users\Paolo\Desktop  

Restore-VMHost requires:

-VMHost: Restore destination. IP address or FQDN of one or more ESXi hosts you want to recover configurations to.

-FilePath: location where configuration bundles can be fetched in order to be restored on host(s)

-HostUsername: ESXi host username

-HostPassword: ESXi host password

The following command will first place each host into maintenance mode then restore configuration bundle on ESXi host 192.168.243.143 and 192.168.243.144 taking it from C:\Users\Paolo\Desktop folder.

By default configuration bundles are saved as configBundle-<ESXi_host_IP_address>.tar (for example: configBundle-192.168.243.143.tar) so Restore-VMHost function expects such named files to be present in source folder.
Finally it will wait a few minutes (3 by default), giving to each host time to perform a reboot, then removes host from maintenance mode.

 Restore-VMHost -VMHost 192.168.243.143,192.168.243.144 -FilePath C:\Users\Paolo\Desktop -HostUsername root -HostPassword vmware  

As usual this code is also available on my GitHub repository: BackupRestore.psm1



That's all!!