VMware: Dynamic reports using PowerCLI and AngularJS

In latest articles I provided several ways to produce HTML reports out of data gathered by running a PowerCLI script. Alarms, charts and reports are produced in a simple yet effective way by running a PowerCLI script which retrieve data after querying an entity, usually a vCenter Server, then data is literally outputted as an HTML file by the script itself.
In today's post I will introduce a far more elegant way to produce HTML reports upon data retrieved from an ESXi cluster. These reports are HTML pages based on AngularJS, Google's extremely powerful framework based on MVC pattern. Extremely trivializing, the logic behind AngularJS and an MVC based framework is to separate the logic (Controller) from the content (Model) and from the style (View).

Since I'm a beginner on AngularJS I'm not the most suitable guy to explain these concepts in plain detail so let's proceed to what's the scope of this article: generate a dynamic HTML report.

Let's start by dividing the scenario in two parts:

1)Back-end: where the "magic" happens.
2)Front-end: where you will get in touch with this "magic".

Back-end is composed by GenerateJSON.ps1 PowerCLI script. This script retrieves all virtual machines from a given cluster and exports VM infos in JSON, a format that can be easily processed and consumed by the front-end.

Most of the job is done by this line of PowerCLI code:

 Get-VM -Name $vm | Select * -ExcludeProperty ExtensionData | ConvertTo-Json -Depth 1 > $OutputPath\$($vm.Id).json  

This gets data for each VM in the given cluster and exports it in JSON saving the output in the proper folder. The $OutputPath folder is really important since by default the front-end will search for JSON files in such folder so be careful to set it properly. If you wonder why we exclude ExtensionData from the report that is to prevent an error due to a conflicting naming with an already existing property.

Since this script could take some time depending on how many VMs there are in your cluster be patient.

Another advice is to add this PowerCLI script as a Windows Scheduled Task so you can have VM "fresh-data" once in while and you don't have to run it manually every time.

Front-end comprises a web page that, by using AngularJS, allow us to dynamically list all virtual machines gathered by the back-end with corresponding details like OS they are running, configured amount of memory/CPUs, power state, host on which they are currently running on, etc. AngularJS allow us to do some nice tricks like searching and sorting results without reloading the web page every time.

Who is already skilled on AngularJS will recognize that the front-end is the official AngularJS tutorial application slightly modified and styled. Of course this is just an example to point out how endless are the possibilities offered by PowerCLI for reporting. You should edit/change it according to your needs, but this could be a good starting point.

This is a diagram who gives an idea of what front-end and back-end are:

Here's how the final result will look like:

This is the PowerCLI code composing the back-end and, as usual you can find it in my GitHub repository GenerateJSON.ps1



Front-end is composed by some files, all of them available on my GitHub.

Download AngularJS dynamic reports

This is the folder tree, be careful to download every file in order to make web page work.

Root AngularJS folder\
|__data\
|    |_PLACE HERE GENERATED JSON FILES
|__img\
|    |_logo.png
|__js\
|    |_app.js
|    |_controllers.js
|    |_filters.js
|__partials\
|    |_vm-detail.html
|    |_vm-list.html
|_index.html
|_newstyle.css

In the AngularJS project you will notice that JSON files are fetched "statically", they are placed in data folder and read from there.It would be a nice addition to provide these files to the AngularJS page using a web service, like a PHP page which dynamically calls PowerCLI each time a JSON file is needed.

That's all!!

VMware: PowerCLI to report triggered Alarms

There's one more thing related to management that I've still not covered in previous blog posts about reporting: alarms.
They are extremely important indicators of issues occurring in our virtual infrastructure, they are triggered once there's something wrong, something that need to be resolved in order to make everything work smooth.

Alarms are of two kinds: Warnings and Alerts. A Warning indicates that something is wrong in our environment but it's not yet critical. When it becomes more serious a Warning is replaced by an Alert, dictating something needs to be addressed as soon as possible.

As usual PowerCLI allow us to retrieve and get some informations regarding triggered alarms.

The following PowerCLI script connects to a vCenter Server and retrieves all alarms from any registered host. These alarms are presented in an HTML table and, using JavaScript, we add a little bit a style. By being JavaScript interpreted at client level, browser need to have it enabled in order to work.

The following image depicts an example of what you get after running the script:

For the sake of keeping the script essential I've excluded the CSS from it but, if you have a look at my previous fancy HTML reports blog post, you will get an example about how to use it in your report.

Here's the code for creating reports based on triggered alarms, as usual, you can also find it in my GitHub repository: Alarm Reports.ps1



That's all!!

VMware: VM stats using PowerCLI and Google Charts

Let me start today's post by pointing out that this is more a proof of concept than a real thing, but I hope from this piece of code you get the idea of how many different tasks can be accomplished by using PowerCLI and how versatile it is for reporting capabilities. After producing fancy reports using PowerCLI why not also include some charts in them?

This can be easily accomplished by using Google Charts, JavaScript pieces of code that generates charts/tables just by passing data to them and the resulting charts can be embedded in web pages.

The idea behind this PowerCLI script is to generate an HTML page and retrieve stats from an entity (a virtual machine) by using the Get-Stat cmdlet. These stats will be passed as data upon which charts will be created.

If you have a look at a sample chart code, for example a donut chart, you can slice it up into three different areas: an header in which javascript function is defined, a body, the most important part, in which you insert all the data and some options like title, measures to be used on x-y-axis, etc. Third and final part is a sort of a footer in which javascript and html tags are closed.

The PowerCLI cmdlet Get-Stat retrieve stats from any powered on virtual machine:

 Get-Stat -Entity (Get-VMHost -Name 10.0.1.62 | Get-VM | Where-Object PowerState -match "PoweredOn") -Stat $stat -Start (Get-Date).AddHours(-24) -MaxSamples (10) -IntervalMins 10 | Measure-Object Value -Average  

Where:

-Entity: the object against you want to retrieve stats from. This could be a datacenter, a virtual machine, a host, etc. In the line above it retrieves all powered on VMs running on host 10.0.1.62.
-Stat: the statistic you want to retrieve.

The following values are accepted:

cpu.usage.average
cpu.usagemhz.average
mem.usage.average
disk.usage.average
net.usage.average
sys.uptime.latest

-Start: start time of statistics retrieval
-MaxSamples: how many samples consider for the measurement
-IntervalMins: the amount of time that separates two consecutive measurements
-Measure-Object Value: we need to specify we are going to use this value as a measure.
-Average: how this measure will be considered. Average, Maximum and Minimum are accepted values. Average consider the average resulting value from all collected samples. Please note that for every of the available stats the Average value is a good indicator since the average usage of CPU, Memory, Disk or Network over an amount of time could be considered as a good indicator of how well the measured entity performs. sys.uptime.latest is the only measure for that average is not suited, since we need to retrieve only the last value of uptime in this case Maximum is the correct way to measure it.

The output produced by the PowerCLI script is an HTML file and this means that you can style it applying a proper CSS.

This is an example of a chart report I generated:

The following is the sample PowerCLI code used to generate a donut chart, as usual you can find it also on my GitHub repository: Stats using Google charts.ps1



As you can see this time HTML page is generated by simply concatenating variables $htmlheader, $data and $htmlfooter containing the HTML code. This is another, less elegant way though, to generate the HTML page. Conversely you can use the ConvertTo-Html cmdlet as in fancy reports using PowerCLI.

VMware: A PowerCLI GUI to edit VM hardware version 10

Hardware version 10 was introduced by VMware in late 2013 with ESXi 5.5. It brought several new improvements to VM capabilities like, for example, the maximum size of a single VMDK file up to 62TB au contraire to the 2TB of the hardware version 9 used in ESXi 5.0.

Despite the several indisputable improvements introduced by hardware version 10 some users pointed out some limitations in managing virtual machines adopting this hw version.

As you probably know virtual machine settings can only be modified using vSphere Web Client. Try to modify these VMs using the classic vSphere C# Client and you will incur in this message:

This limitation can be identified in VMware's strategy to soon discontinue the good 'ol vSphere C# Client and adopt the vSphere Flex Web Client as the standard tool to perform any kind of operation on the virtual infrastructure. Despite some features, like Site Recovery Manager, still require the C# Client others are only manageable through the Flex Client like the vSphere Data Protection or the brand new VSAN.

But what about changing a VM setting while a vCenter Server is unavailable? This of course results in Web Client unavailability. Or what about adding a new hard disk to a VM while in extreme hurry?

To answer first question a vCenter Server should NEVER be unavailable since by being the central point of infrastructure management its unavailability could introduce several limitations, not to mention the lack of several capabilities like DRS, vMotion, etc.

Second question instead was brought to my attention by several technicians working at my company. They simply find unbearably slow vSphere Web Client (maybe while editing VMs at customer site in late evening?) so to perform simple VM tasks like increasing CPUs, adding hard disks, etc. they tend to still use the C# Client. But as you know this is not possible for hardware version 10 VMs.

To keep a long story short I first try to encourage them to re-evaluate the Web Client and get used to use it even for the smallest things since this will be the future, but at the same time I embraced theyr cause and developed this little tool to help them to change, hopefully in a way smaller amount of time, settings for hardware version 10 virtual machines.

This tool was scripted in PowerCLI and once run will bring up a .NET like graphical user interface (GUI) from which perform all operations.

To use it you need to insert credentials to connect to a vCenter Server or directly to an host. Then you will have to select the host (in case you are connecting to a vCenter) on which is running/placed the VM you need to edit settings.Select the VM and modify vCPUs, RAM, increase disk size, or add new hardware like a new hard disk or a new network adapter.

Hardware version 10 VMs can be edited using PowerCLI in exactly the same way as for hw version 9 (or older) VMs.

The main cmdlets used in this script to edit VM settings are: 

Set-VM: Used to set many parameters of an existing VM like, for example, the amount of RAM, vCPUs, the High Availability Restart Priority, the hardware version, etc. 
Set-HardDisk: Edit Hard Disk properties/size for a specified Disk of a specified VM
New-HardDisk: Adds a new Hard Disk of a specified size to a specified VM.
Set-NetworkAdapter: Edit Network Adapter properties like wake on LAN, network to which the VM will be connected to, port group, MAC Address(*), etc.
New-NetworkAdapter: Adds a Network Adapter to a specified VM.

(*)In the code below I struggled, and failed, to find a proper way to edit MAC Address of a VM. I always got "Not a valid MAC Address or not in range..." error. Due to time limitations for article release I cannot overcame this obstacle and forced to leave in the code some lines regarding MAC changes: my bad. If you find a proper way to make it work you can also enable the MAC address editing feature.

Here's the PowerCLI code, as usual you can find it in my GitHub repository: Edit hw v10 VMs.ps1



That's all!!

VMware: Fancy HTML reports using PowerCLI

In today's blog post how to create fancy HTML reports using PowerCLI. Sysadmins are well known to be data nerds, they like to constantly monitor all datacenter parameters, either using real time stats monitoring solutions, like VMware vCenter Operations, either producing periodical reports to analyze changes that occurrs in the virtual infrastructure.

Reports must be functional and readable, but if you are not the only one consuming these reports why not to make them look good too?

Fancy reports can be generated using PowerCLI - actually the capability is provided by PowerShell - by using the ConvertTo-Html cmdlet which generates an HTML output.

The trick to generate these good looking reports is to consider them as a webpage, that will be generated by script, to which a Cascading Style Sheet (CSS) will be applied.

Page is composed by an header, a CSS and a body. CSS part is the most important for our purpose because it will describe how output will be styled. Body section of the page will contain correct div IDs assignation and all statistics composing our report and in this example I get infos regarding Datacenter, Cluster and VMs. Of course you can include in the report whatever you want.

This is the expected result:

Here's the code. As usual I've uplaoded it to my GitHub page: Fancy Reports.ps1



$PageBoxOpener, $ReportVmHost, $BoxContentOpener, etc. variables are used to delimit the divs upon which CSS style will be applied.

To better understand how this works the schema of the page generated by the script above can be represented as:

 <div id='title'>  
      PowerCLI Reporting  
 </div>  
 <div id='subtitle'>  
      Report generated: $(Get-Date)  
 </div>  
 <div id='box1'> #Box Delimiting Report  
      <div id='boxheader'> #First Report Header  
           Get-VMHost $LocationName  
      </div>  
      <div id='boxcontent'> #First Report Table  
           Get-VMHost ...  
      </div>  
      <div id='boxheader'>  
           Get-Cluster $ClusterName  
      </div>  
      <div id='boxcontent'>  
           Get-VMHost ...  
      </div>  
      <div id='boxheader'>  
           Get-VM $ClusterName  
      </div>  
      <div id='boxcontent'>  
           Get-VMHost ...  
      </div>  
 </div>  

That's all!!

PowerCLI: Automate Oracle DB install using Invoke-VMScript

I only recently discovered how cool and powerful is Invoke-VMScript, a VMware PowerCLI cmdlet that allows you to run scripts inside the guest VM operating system.
It requires VMware Tools installed and running inside the guest OS in order to successfully inject commands and scripts from the ESXi host to the VM and it is noteworthy to say that not all operating systems are supported by Invoke-VMScript cmdlet.
Supported ones, according to official documentation, are: Windows XP 32bit SP3, Windows Server 2003 32bit SP2, Windows Server 2003 64bit SP2, Windows 7 64bit, Windows Server 2008 R2 64bit and RHEL 5.

Invoke-VMScript syntax is:

Invoke-VMScript [-ScriptText] <String> [-VM] <VirtualMachine[]> [-HostCredential <PSCredential>] [-HostUser <String>] [-HostPassword <SecureString>] [-GuestCredential <PSCredential>] [-GuestUser <String>] [-GuestPassword <SecureString>] [-ToolsWaitSecs <Int32>] [-ScriptType <ScriptType>] [-RunAsync] [-Server <VIServer[]>] [-WhatIf] [-Confirm] [<CommonParameters>]

Where:

HostCredential: username and password for the ESXi host on which resides the virtual machine.
GuestCredential: username and password for the guest OS. The user must have administrative access.
ScriptType: what kind of script you want to invoke: PowerShell, Bat or Bash.

In this article I provide a simple PowerCLI script which allows you to deploy virtual machines from a Win2008 R2 template and perform an automated silent install of Oracle Database 11g R2.

This is just an example of Invoke-VMScript usage and this only provides a software install, neither listener neither instance are configured. This script could also be readapted/improved for RHEL installs.

How this script works:

-Connects to a vCenter or ESXi host.
-Deploys "N" virtual machines from a properly prepared (read a few lines below to find how) Win2008 template.
-Power on the virtual machine.
-Perform a silent, fully automated, install of Oracle Database 11g R2 software by using Invoke-VMScript cmdlet.

Preprequisite for this PowerCLI script are:

-A Windows 2008 R2 virtual machine
-Oracle Database 11g R2 11.2.0.X. Unzip installation files on the Desktop. This script assumes that your installation directory is placed on: C:\Users\Administrator\Desktop\database
-Response file: a response file is a text file which is used by Oracle to perform silent installs. It contains all infos that usually are inserted by the user in a common GUI installation. This script assumes that your response file is: C:\Users\Administrator\Desktop\install.rsp

This is the content of my install.rsp response file.

 oracle.install.responseFileVersion=/oracle/install/rspfmt_dbinstall_response_schema_v11_2_0  
 oracle.install.option=INSTALL_DB_SWONLY  
 UNIX_GROUP_NAME=oinstall  
 INVENTORY_LOCATION=C:\Program Files\Oracle\Inventory\logs  
 ORACLE_HOME=C:\app\Administrator\product\11.2.0\dbhome_1  
 ORACLE_BASE=C:\app\Administrator  
 oracle.install.db.InstallEdition=EE  
 oracle.install.db.isCustomInstall=false  
 oracle.install.db.customComponents=oracle.server:11.2.0.1.0,oracle.sysman.ccr:10.2.7.0.0,oracle.xdk:11.2.0.1.0,oracle.rdbms.oci:11.2.0.1.0,oracle.network:11.2.0.1.0,oracle.network.listener:11.2.0.1.0,oracle.rdbms:11.2.0.1.0,oracle.options:11.2.0.1.0,oracle.rdbms.partitioning:11.2.0.1.0,oracle.oraolap:11.2.0.1.0,oracle.rdbms.dm:11.2.0.1.0,oracle.rdbms.dv:11.2.0.1.0,orcle.rdbms.lbac:11.2.0.1.0,oracle.rdbms.rat:11.2.0.1.0  
 oracle.install.db.DBA_GROUP=dba  
 oracle.install.db.OPER_GROUP=dba  
 SECURITY_UPDATES_VIA_MYORACLESUPPORT=false  
 DECLINE_SECURITY_UPDATES=true  

So, to recap: unzip Oracle DB installation files on the Desktop and place install.rsp in the same location as depicted by the following image.

This is the PowerCLI code, as usual you can also find it on my GitHub repository: Invoking Scripts in Guest-VM.ps1



Run it and deployment of virtual machines from template begins:

After script completion VMs are correctly added to vCenter Server inventory

and Oracle Database is installed inside guest OS.

To perform a full install of Oracle Database the listener and an instance must also be configured. This can be done using a response file and run using PowerCLI Invoke-VMScript.

If you need a standard listener configuration netca.rsp file provided in database\response installation directory does the job.

Listener silent install is performed by running:

 netca -silent -responseFile C:\Users\Administrator\Desktop\database\response\netca.rsp  

Similarly database instance can be added by invoking the Database Configuration Assistant (dbca) in silent install mode using a response file or passing all required configurations as parameters.

For example:

 dbca -silent -createDatabase -templateName General_Purpose.dbc -gdbname orcl -sid orcl -gdbName orcl -memoryPercentage 30 -sysPassword welcome1 -systemPassword welcome1 -dbsnmppassword welcome1 -emConfiguration LOCAL -redoLogFileSize 100 -storageType FS -datafileDestination C:\app\Administrator\oradata -datafileJarLocation C:\app\Administrator\product\11.2.0\dbhome_1\assistants\dbca\templates -characterSet AL32UTF8 -sampleSchema false -totalMemory 512  

That's all!!