Oracle Database: PRVF-0002 Could not retrieve local nodename

If you encounter this error while installing Oracle Database 12c:

PRVF-0002 Could not retrieve local nodename

Having a look at Oracle's error documentation it simply tells us:

Cause: Unable to determine local host name using Java network functions. 
Action: Ensure hostname is defined correctly using the 'hostname' command.

Here's how to solve it:

This is the standard file hosts

[root@vm1 ~]# nano /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

If you ever installed Oracle DB 11g you probably know you need to edit hosts file adding the current machine IP address and hostname. So file hosts resulted in something similar to:

[root@vm1 ~]# nano /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6 
192.168.116.230 vm1 vm1.localdomain

Unfortunately in Oracle DB 12c this is not enough, to solve the error above you need to point loopback address to the machine hostname.

[root@vm1 ~]# nano /etc/hosts

127.0.0.1   vm1 vm1.localdomain localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         vm1 vm1.localdomain localhost localhost.localdomain localhost6 localhost6.localdomain6 
192.168.116.230 vm1 vm1.localdomain

That's all!!

VMware: VLANs and vSwitches

Switches work at Layer2 and basically their job is pretty simple: they just forward Ethernet frames received from one port to another port based on the content of their MAC Address Table.

Let's start giving some basic concepts about Ethernet frames and VLANs.

An IP Packet is encapsulated at Layer2 (Data Link) in an Ethernet Frame.

By default Ethernet packets at Layer3 have a maximum Ethernet payload MTU of 1500bytes. This MTU depends on network protocol and for example is 1478 for PPPoA, 1446 for PPPoE, but it cannot grow over 1500bytes (if it is >= 1501bytes it is a Jumbo Packet).

As depicted in image above an Ethernet frame is composed by:

-Preamble
-SFD (Start of frame delimiter)
-Destination MAC Address
-Source MAC Address
-VLAN ID - IEEE 802.1Q tag (optional)
-Ethertype
-Payload
-FCS (Frame check sequence)

At Layer2, by default, an Ethernet frame is 1518bytes long. Preamble and SFD are not computed in frame size because they are used for synchronization and not for actual data transmission.

When dealing with VLANs an 802.1Q tag of 4bytes is added by the switch between Source MAC Address field and Ethertype field. With VLANs an Ethernet frame grows up to 1522bytes. By default switches drops packets that are greater than 1518bytes so when using VLANs switches must be prior configured.

Actually the condition above doesn't always apply becase for example Cisco switches supports by default Ethernet frames up to 1548bytes to accomodate encapsulation of an Ethernet packet into an ISL packet.

VLAN tagging is performed ONLY by switches and despite a VM itself could tag frames, in this article we consider tagging is ONLY performed by switches.

A switch adds a VLAN ID when it recieves a packet from a port belonging to a VLAN. VLAN tag is added in the correct frame position and, since frame has been modified, Frame check sequence (32bit CRC) has to be recalculated.

When a frame reach the switch to which reciever is directly attached it:

-Read VLAN tag
-Perform a lookup to check if Destination MAC Address from recieved frame is contained in switch MAC Address Table and to which port frame has to be forwarded.
-If step above is successful it recalculates Frame check sequence
-Forward frame to destination

Let's now see how VLANs and vSwitches work in vSphere:

We have two VMs: VM1 and VM2

First scenario: VM1 and VM2 are on the same VLAN, same vSwitch and same PortGroup VM Network 100

Using vswitchInfo.sh from virtuallyGhetto we can retrieve port ID to which VMs are connected to vSwitches in order to allow us to build a MAC Address Table.

Launching from ESXi host via SSH:

~ # ./vswitchInfo.sh -l

We get the following:

This allow us to deduce how the vSwitch MAC Addres Table would look like. We suppose that MAC Addres Table is completely/correctly built so every switch either virtual and physical knows to what interfaces forward packets.

vSwitch1
Vlan    Mac Address          Ports
----    -----------          -----
100    00:50:56:8a:7e:34      50331665
100    00:50:56:8a:25:f9      50331669

Since VM1 and VM2 are both on same PortGroup on same vSwitch when Ethernet frame is send from VM1 to VM2 vSwitch1 recieves frame, it performs a lookup in MAC Addres Table and it find that VM2 is directly attached to it so it forward frame to port ID 50331669 WITHOUT VLAN tag.

VM1 is able to communicate with VM2

Second scenario: VM1 and VM2 are on the same VLAN, same vSwitch but different PortGroup (VM Network 100 and VM Network 100-2)

We have this:

Same MAC Addres Table from first scenario only Port ID has changed for VM2 since it has been moved to another PortGroup.

vSwitch1
Vlan    Mac Address          Ports
----    -----------          -----
100    00:50:56:8a:7e:34      50331665
100    00:50:56:8a:25:f9      50331668

Since VM1 and VM2 are both on same switch vSwitch1 recieves frame, it performs a lookup in MAC Addres Table and it find that VM2 is directly attached to it so it forward frame to port 50331668 WITHOUT VLAN tag.

Third scenario: VM1 and VM2 are on the same VLAN but different vSwitch. Both VM Network 100 and VM Network VLAN 100 PortGroups have VLAN ID=100.

Please note that when VMs are on different vSwitches frames are always forwarded to physical switches.

We have this:

Two MAC Addres Table one for vSwitch1 and one for vSwitch0.

vSwitch1
Vlan    Mac Address          Ports
----    -----------          -----
100    00:50:56:8a:7e:34      50331665

vSwitch0
Vlan    Mac Address          Ports
----    -----------          -----
100    00:50:56:8a:25:f9      33554462

VM1 and VM2 are on different switches. vSwitch1 recieves Ethernet frame from VM1 addressed to VM2. Destination MAC Address is not contained in vSwitch1 MAC Addres Table so vSwitch1:

-Add VLAN tag specifying that frame destined to VM2 belongs to VLAN 100
-Recalculate FCS
-Forward frame to physical switch

Physical switch:

-Recieves frame from vSwitch1
-Forward frame to vSwitch0

Of course even physical switch uses a MAC Addres Table for frames forwarding.
Usually intra switch links are trunks. This means that a single port/cable can carry frames from multiple VLANs.

vSwitch0:

-Recieves frame
-Read VLAN tag
-Perform a lookup to check if VM2 Destination MAC Address (00:50:56:8a:25:f9) is contained in its MAC Address Tableand is connected to port 33554462
-Since previous step is successful it recalculates Frame check sequence
-Forward frame to VM2 via port 33554462


VM1 is able to communicate with VM2

Fourth scenario: VM1 and VM2 are on different VLAN, same vSwitch, different PortGroup (VM Network 100 and VM Network 200)

We have this:

vSwitch1
Vlan    Mac Address          Ports
----    -----------          -----
100    00:50:56:8a:7e:34      50331665
200    00:50:56:8a:25:f9      50331671

VM1 and VM2 are on different VLANs, vSwitch1 recieves frame lookup in MAC Addres Table and find out that VM2 is on a different VLAN than VM1. vSwitch1 drops the frame.

VM1 is NOT able to communicate with VM2

Fifth scenario: VM1 and VM2 are on different VLAN and different vSwitch

We have this:

vSwitch1
Vlan    Mac Address          Ports
----    -----------          -----
100    00:50:56:8a:7e:34      50331665

vSwitch0
Vlan    Mac Address          Ports
----    -----------          -----
200    00:50:56:8a:25:f9      33554462

As in third scenario VM1 and VM2 are on different switches so frames will be forwarded to physical switch. vSwitch1 recieves Ethernet frame from VM1 addressed to VM2.

Destination MAC Address is not contained in vSwitch1 MAC Addres Table so vSwitch1:

-Add VLAN tag specifiyng that frame destined to VM2 belongs to VLAN 100
-Recalculate FCS
-Forward frame to physical switch

Physical switch:

-Recieves frame from vSwitch1
-Forward frame to vSwitch0

vSwitch0:

-Recieves frame
-Read VLAN tag
-Perform a lookup to check if VM2 Destination MAC Address (00:50:56:8a:25:f9) is in its MAC Address Table
-Since VM2 is on a different VLAN vSwitch1 drops the frame

VM1 is NOT able to communicate with VM2

So, it's important to understand that in vSphere when a VM sends packets/frames to another VM on a different vSwitch those packets/frames will always hit physical switch.
Another major concept to remember is that VLAN tagging occurs only when frames leave vSwitch. Intra-vSwitch communications are not VLAN tagged.

That's all!!

VMware: manage networking with esxcfg

In this post I will explain how to manage vNetworking using esxcfg.

For explanation purpouses let's simulate a scenario. Pretend we need a second management network for redundancy's sake (network heartbeating for HA). For this management network we will use a dedicated network adapter.

Here's what we want to achieve:

These are our steps:

-Create an additional vSwitch
-Assign a network adapter to this vSwitch
-Create a new vmkernel
-Enable Jumbo Frame on vmkernel
-Use this vmkernel for Management Traffic

Let's start by connecting to the host via SSH (I use Putty) and listing all network adapters attached to host:

~ # esxcfg-nics -l

List all vSwitches with corresponding network adapters

~ # esxcfg-vswitch -l

As you can see in this scenario I only have one vSwitch named vSwitch0. It would be advised to add the new vmkernel for management to vSwitch0 itselfs but for the sake of this post I will create another vSwitch and then add the management vmkernel to that vSwitch.

Please bear in mind that by creating a new vSwitch you introduce a new element in your environment so when you are in "troubleshooting mode" the less items you have to check the faster you are in identifying trouble causes. A thumb rule is to use as less vSwitches as possible.

Command above told us vmnic3 it's the only available network adapter so we will use that in our new vSwitch.

If there are no vmnics available it is also possible to remove some from vSwitch0.

For example to remove vmnic2 from vSwitch0

~ # esxcfg-vswitch -U vmnic2 vSwitch0

Now let's first create a vSwitch named vSwitch1

~ # esxcfg-vswitch -a vSwitch1

Then we add network adapter vmnic3 to vSwitch1

~ # esxcfg-vswitch -L vmnic3 vSwitch1

Create a Management Network 2 vmkernel in vSwitch1

~ # esxcfg-vswitch -A "Management Network 2" vSwitch1

Assign it an IP/netmask

~ # esxcfg-vmknic -a -i 10.0.1.100 -n 255.255.255.0 "Management Network 2"

Set VLAN ID 300 for Management Network 2 vmkernel

~ # esxcfg-vswitch -v 300 -p "Management Network 2"

Let's pretend we need to use Jumbo Frames. For your interest it's pretty pointless enabling Jumbo Frames in a management only vmkernel since benefits of transmitting Ethernet packets with a bigger payload are primarily pointed out when there is a heavy network activity. Jumbo frames could be useful on vMotion or iSCSI/NFS vmkernels. Anyway to enable Jumbo Frames we need to edit the Maximum Transmission Unit setting it to 9000bytes.

You also need to change MTU on you physical switch in order to use Jumbo Frames. If your vSwitch sends packets with an MTU of 9000bytes and your pSwitch expects them of 1500bytes (default Ethernet packet MTU) packets could be either fragmented or dropped. Here's a good article by Cisco!

~ # esxcfg-vmknic -m 9000 -p "Management Network 2"

We created a "general purpouse" vmkernel and since we need to use this for Management Traffic we need to specify it.

First we retrieve informations on newly created vmkernel:

~ # esxcfg-vmknic -l

As you can see in picture above our vmkernel is vmk3.

William Lam in this VMware Community post explains how to bind specific traffic to a vmkernel.

As for management traffic the following command is used:

~ # vim-cmd hostsvc/advopt/update Net.ManagementIface string vmk3 

To briefly recap we used esxcfg to manage our vSphere networking. We introduced a fictional scenario to point out what commands need to be used in order to edit/create/manage networking in ESXi hosts.

That's all!!

VMware: Locate VMs snapshots using PowerCLI

Here's another PowerCLI post.
Everyone who is dealing with vSphere knows about snapshots pros and cons. To be honest I've had in the past some issues with snapshots, typically when visiting customers datacenters I found out VMs that were running on snapshots so old that they were bigger in size than base vmdk itself or, as once happened, that a backup software created almost 200 snapshots of a VM because, once backup was performed successfully, it was not able to revert back the snapshot.

That's why a common suggestion is to use snapshots just for updating purpouses like OS patching and revert them as soon as possible because as they grow up in size it could be unconfortable to revert them back.

After this introduction here's the main topic of this post: a PowerCLI script that locates snapshots of your VMs.

Save this code as a file with .ps1 extension:

 foreach($vm in ($VMs = Get-VM)){  
 $vm | Get-HardDisk | %{  
   if($_.Filename -like "*-??????.vmdk" -and ($_.Filename -match "\d{2,7}" -eq "True")){  
     $_.Filename |Ft -autosize | out-string -width 4096  
   }  
  }  
 }  

From code above you can see we are retrieving all VMs in our environment, then the hard disk files of that VMs.
Of these hard disks we want to display snapshots only.

As you know VMware snapshots disk are named:

<VM_NAME>-<SNAPSHOT_NUMBER>.vmdk
Example: Win2008-000001.vmdk


So with:

$_.Filename -like "*-??????.vmdk"
we are looking for disks that contains -<SNAPSHOT_NUMBER>.vmdk and with

$_.Filename -match "\d{2,7}" -eq "True"
we are verifying that those esclamation marks *-??????.vmdk are numbers and NOT letters.

Without this control the PowerCLI script would return any vmdk which has a "-" and 6 letters before ".vmdk"

For example it would also return a VM disk named:

Win2008-DVLPMT.vmdk

which clearly is NOT a snapshot disk.

As always to run this script you need to connect to a vCenter server (or ESXi host) by using Connect-VIServer cmdlet.

That's all!!

VMware: Deploy VMs from Template using PowerCLI

After yesterday's introductive post, today I would like to share a simple, yet powerful, PowerCLI script that allows you to deploy VMs from a Template.

This is the script structure:

 1..<N> | Foreach {  
 New-VM -vmhost <ESX_HOST> -Name <VM_NAME> -Template <TEMPLATE> -Datastore <DATASTORE> [-OtherOptions]  
 }  

Where:

<N> is the number of VMs you need to deploy.
<ESX_HOST> is the host where you will deploy your VMs
<TEMPLATE> is the template from which your VMs will be created
<DATASTORE> is the datastore that will contain your VMs
[-OtherOptions] are additional options that can be used in conjunction with New-VM command.

As you can see it's really basic, but New-VM cmdlet is really powerful and it has a lot of options. To go through all on New-VM options:


Get-Help New-VM -Detailed


Here's how I used it today for deploying 6 VMs that will be Oracle RAC nodes.


Open a text editor and paste this code in it saving file with .ps1 extension.

 1..2 | Foreach {  
 New-VM -vmhost esx1.testdomain.com -Name RAC$_ -Template OL6 -Datastore datastore1  
 }  
 3..4 | Foreach {  
 New-VM -vmhost esx2.testdomain.com -Name RAC$_ -Template OL6 -Datastore datastore2 -ResourcePool TestResourcePool  
 }  
 5..6 | Foreach {  
 New-VM -vmhost esx3.testdomain.com -Name RAC$_ -Template OL6 -Datastore datastore3 | Start-VM  
 }  

As you can see we are deploying 6 VMs, named RAC1 to RAC6 from a template called OL6.
RAC1 and RAC2 will be deployed on host esx1.testdomain.com, RAC3 and RAC4 on host esx2.testdomain.com and RAC5 and RAC6 on host esx3.testdomain.com.
RAC3 and RAC4 will be placed in a resource pool called TestResourcePool.
RAC5 and RAC6 will be powered on after deployment due to Start-VM cmdlet.


To run this simple script just open VMware vSphere PowerCLI, connect to your vCenter (or host) using Connect-VIServer cmdlet as stated in VMware: vSphere PowerCLI 101 post.

A progress bar will appear in your PowerCLI console

That's all!!Happy VMware Automation!!

VMware: vSphere PowerCLI 101

These days I'm speding some time learning VMware vSphere PowerCLI. I think vSphere PowerCLI is a great tool for automating tasks among vSphere environments. It's so useful to automate repetitive tasks, it allows IT admins to save huge amounts of time performing common tasks using scripts.

To start using VMware vSphere PowerCLI simply go here:

vSphere PowerCLI: Windows PowerShell interface for managing vSphere

You need to download and install VMware vSphere PowerCLI on your Windows machine in order to start scripting.

Here is the official vSphere PowerCLI Documentation

Another really useful PowerCLI document to read is Getting Started with PowerCLI 4.1 – Automating Your vSphere Environment  which contains a lot of useful commands to perform basic tasks.

Last link I would like to share is the VMware PowerCLI 5.1 Poster which presents all PowerCLI cmdlets in a single panel grouping them by function (Network Operations, Logs/Reporting, etc.)

So, let's start with PowerCLI 101 post...

Connecting/Disconnecting to vCenter Server:

To start using PowerCLI you need to connect to a vCenter Server or to an ESXi host

Connect-VIServer -Server <VCENTER_IP_ADDRESS> -User <USERNAME> -Password <PASSWORD>


Example: Connect-VIServer -Server 192.168.116.155 -User root -Password vmware

To disconnect:

Disconnect-VIServer


Available Commands:

To list all available commands:

Get-Command

Example: Get-Command *VM*  -> To list all commands containing word "VM" in their name.

To get more infos on a command

Get-Help <COMMAND_NAME> -Detailed


Selecting Columns:

<cmdlet> | Select-Object <COLUMNS_TO_SHOW>

Example: Get-VM | Select-Object Name, PowerState -> To display only name and power state of every VM


Prevent truncated columns:

If a column contains many data it could happen that output it's truncated. To prevent this:

<cmdlet> | Select-Object <COLUMNS_TO_SHOW> | Ft -autosize


Redirecting Output:

<cmdlet> > <OUTPUT_FILE>

Example: Get-VM > C:\List_of_VMs.txt -> To create a file named "List_of_VMs.txt" which contains the list of all our VMs


Exporting CSVs:

<cmdlet> | Export-Csv <OUTPUT_FILE>

Example: Get-VM | Export-Csv C:\List_of_VMs.csv -> To export to CSV infos about VMs


Combining Cmdlets:

<cmdlet> | <cmdlet>

Example:  Get-VMHost esx1.testdomain.com | Get-VM -> To show all VMs running on esx1.testdomain.com host


Nested Expressions:

<cmdlet> | Select-Object @{N="<COLUMN_NAME>"; E={<EXPRESSION>}}

Example:  Get-VM | Select-Object Name, @{N="Datastore"; E={$_ | Get-Datastore}} -> Display VM name and datastore on which each VM resides

In the next post I will provide some PowerCLI useful scripts.

That's all!!

ESXi: The operation is not allowed in the current state

Today I stepped into this issue while trying to vMotion a VM to another host. When validating destination host this error is displayed:

The operation is not allowed in the current state

After a brief search I found KB 1003829 which states:

1)Cancel the Enter maintenance mode
My host was not entering maintenance mode

2)Restart the ESX host's management agents
Done but still got error

3)Wait until any pending tasks for the ESX host are complete
No running tasks reported from vSphere Client. Logged into my host using SSH to see if there are pending tasks using this command:

vim-cmd vimsvc/task_list

# vim-cmd vimsvc/task_list
(ManagedObjectReference) [
   'vim.Task:haTask-5-vim...'
]
There was a task still running due to the command above (if there are no ongoing tasks the brackets "[]" are empty), waited until completion but vMotion host validation still failed with message "The operation is not allowed in the current state".

4.a)Disconnect and then reconnect the ESX host to VirtualCenter
Done but still got error

4.b)Restart the VMware VirtualCenter Server service
Finally here is how I solved my issue!

To restart vCenter Server go to:

https://YOUR_VCENTER_SERVER:5480

Click Stop button and wait until services stopping occurs.

Then cick Start and wait for completion.

After this I was able to correctly vMotion guest VMs!!