VMware: vSphere Replication Part 7 - Provision additional Replication Servers

As stated in previous article vSphere Replication architecture comprises two key elements: vSphere Replication Manager and one or more vSphere Replication Servers. Although there is only one vSphere Replication Manager in a vSphere Replication deployment we can add more than just one vSphere Replication Server in our environment to achieve a better load balancing for replicating VMs to target site.

Provisioning more than one vSphere Replication Server concurr in offloading replication tasks to more than just one server leading to a potentially faster replication process.

How can we deploy more than just the standard vSphere Replication Server comprised in vSphere Replication Appliance?

The process is really simple, just login to vSphere Web Client and select vSphere Replication -> Home -> Manage -> vSphere Replication -> Replication Servers and click Deploy Additional RS Server.

The usual OVF deploy form will pop-out, make sure you select the assembly file vSphere_Replication_AddOn_OVF10.ovf that came with vSphere Replication appliance iso downloaded from VMware website.

You will need to provide IP address or leave DHCP assignation if you prefer, and an administrative password to manage the server. Please bear in mind that vSphere Replication Server unlike vSphere Replication Management Server does not have a VAMI so you cannot login to a webpage to configure it but vSphere Replication Management Server will address this for you and will give you the opportunity to configure additional vSphere Replication Servers from it's VAMI.

Once deployed and powered on we can add it as an additional replication server.

Click Register vSphere Replication Server and select the newly deployed standalone server.

Newly deployed vSphere Replication Server will be added and will be listed under Replication Servers page.

Additional vSphere Replication Server can now be selected when performing a replication. Auto-assign mechanisms will always choose least busy vSphere Replication Server (i.e the one with least replications assigned) while manual selection gives you the opportunity to manually choose by which server your VMs will be replicated to target.

Other blog posts in vSphere Replication Series:

vSphere Replication Part 1 - Introduction
vSphere Replication Part 2 - Installation
vSphere Replication Part 3 - Roles & Permissions
vSphere Replication Part 4 - Configuration
vSphere Replication Part 5 - Enable Replication
vSphere Replication Part 6 - Perform Recovery
vSphere Replication Part 7 - Provision additional Replication Servers

VMware: vSphere Replication Part 4 - Configuration

vSphere Replication is ready out-of-the-box, once the appliance has been deployed it is ready to be used but what if you need to change some settings?

vSphere Replication Appliance has a VAMI (VMware Appliance Management Interface) that can be accessed from:

https://<vSphere_Replication_IP_or_FQDN>:5480

All configuration changes are performed through VAMI

The most important tabs are VR -> Configuration

Here you can configure vSphere Replication database. If embedded database is not sufficient for your deployment you can also use and external SQL or Oracle database as well as import existing DB from another vSphere Replication Manager Server.

VRM Host can be edited in case you need to change to another vSphere Replication Manager Server in your environment.
VRM Site Name and vCenter Server Address, vCenter Server Port and vCenter Server Admin Mail can also be customized.

SSL section allows you to accept only signed certificates as well as generate a self-signed certificate and/or upload an existing one.

Service Status section allow you to monitor the state of vSphere Replication Manager Server and restart it.

VR -> Security tab allows you to change vSphere Replication password and VR -> Support to generate the support bundle useful in case of service request.

Network -> Address is used to change vSphere Replication Appliance network settings like hostname, IP address, default gateway, etc.

Other blog posts in vSphere Replication Series:

vSphere Replication Part 1 - Introduction
vSphere Replication Part 2 - Installation
vSphere Replication Part 3 - Roles & Permissions
vSphere Replication Part 4 - Configuration
vSphere Replication Part 5 - Enable Replication 
vSphere Replication Part 6 - Perform Recovery  
vSphere Replication Part 7 - Provision additional Replication Servers 

VMware: vSphere Replication Part 3 - Roles & Permissions

Since vSphere Replication features multisite and cross-vCenter replication capabilities VMware introduced new roles to meet strict security requirements of a deployment scenario with multiple vCenter servers.

New roles are added into vCenter as vSphere Replication Appliance deployment is completed.

These roles will be assigned to specific users and/or groups who will specifically deal with vSphere Replication.
Roles are pre-built to allow users to perform just certain specific tasks they are permitted to. These tasks are such as recovering replicated VMs or viewing replication status with no further permissions.

The following Roles are added:

VRM Replication Viewer:
View replications
Cannot change replication parameters

VRM Virtual Machine Replication User:
View replications
Manage datastores
Configure and unconfigure replications
Manage and monitor replications

VRM Administrator:
Incorporates all vSphere Replication privileges

VRM Diagnostics:
Generate, retrieve, and delete log bundles

VRM Target Datastore User:
Configure and reconfigure replications
Used on target site in combination with the VRM virtual machine replication user role on both sites

VRM Virtual Machine Recovery User:
Recover virtual machines

For any further information regarding these roles refer to vSphere Replication Roles Reference.

To assign to a certain user/group vSphere Replication roles have a look at my previous article: Users, Groups and Roles management in vSphere 5.5.


Other blog posts in vSphere Replication Series:

vSphere Replication Part 1 - Introduction
vSphere Replication Part 2 - Installation
vSphere Replication Part 3 - Roles & Permissions
vSphere Replication Part 4 - Configuration 
vSphere Replication Part 5 - Enable Replication  
vSphere Replication Part 6 - Perform Recovery  
vSphere Replication Part 7 - Provision additional Replication Servers 

VMware: vSphere Replication Part 2 - Installation

Let's continue with vSphere Replication series after last vSphere Replication Part 1 - Introduction post.

vSphere Replication installation is a pretty straigth-forward process since in most cases it is just necessary to deploy a single appliance to have everything setup and properly running.

vSphere Replication can replicate VMs between different datastores managed by the same vCenter or across sites managed by different vCenters.

Test scenario comprises two vCenter Servers managing two different datacenters:

1)SiteA with "localhost" vCenter that manages one host
2)SiteB with "vcenter-test" vCenter that also manages one host

vSphere Replication will be configured from SiteA to SiteB.

Installation will be shown just for SiteA, if your deployment comprises multiple vCenters to install vSphere Replication in SiteB you need to do repeat the same procedure as for SiteA.

Right click Cluster or alternatively one host and select Deploy OVF Template

Select the vSphere Replication OVF file, in this case: vSphere_Replication_OVF10.ovf

Review details

Choose where to deploy

Setup network connections by setting up IP allocation, DNS server, Gateway and Netmask.

Enter administrative password and, if Static - Manual IP Allocation was selected, enter IP address for vSphere Replication Appliance.

Verify that vSphere Replication binds to your correct vCenter identified by IP address or, as best practice, by FQDN.

Press Finish and wait for the appliance to be deployed and powered on. It will automatically integrate with vSpehere Web Client.
A logout/login is required after vSphere Replication startup is finished in order to view the new vSphere Replication plugin enabled in vSpehere Web Client.

vSphere Replication is now deployed and fully functional, ready to replicate VMs.

Other blog posts in vSphere Replication Series:

vSphere Replication Part 1 - Introduction
vSphere Replication Part 2 - Installation
vSphere Replication Part 3 - Roles & Permissions
vSphere Replication Part 4 - Configuration 
vSphere Replication Part 5 - Enable Replication  
vSphere Replication Part 6 - Perform Recovery  
vSphere Replication Part 7 - Provision additional Replication Servers 

VMware: vSphere Replication Part 1 - Introduction

Today I decided to start a vSphere Replication blog post series in which I will cover replication functionalities offered by vSphere.

What is vSphere Replication?

vSphere Replication as name suggest is a replication appliance deployed in vSphere that allows VMs to be replicated between different datastores. Replication can occur within the same vSphere cluster managed by a single vCenter Server, or across clusters managed by different vCenter Servers.

A major benefit of vSphere Replication is the possibility to set RPO of VMs replication between 15mins up to 24hours, this means that in the worst case (assuming RPO of 15mins) you can loose up to 15mins of data.
Since replication can be performed on powered on VMs, and due to quiescing mechanisms involved during replication, more aggressive RPOs under 15mins were avoided since they could introduce performance problems in VMs quiesced too frequently.

vSphere Replication differs from standard array based replication by being hardware agnostic, no hardware prerequisites exist between source and destination storage, plus they don't have to share same communication protocol (i.e. iSCSI,FC,FCoE,etc.) to be connected neither they must be explicitly supported/certified for replication. It also differs from standard array based replication due to the fact that vSphere Replication can copy invidual VMs and not entire LUNs to destination array despite standard array replication offers an hardware offloaded transfer that does not contend for hypervisor's resources.

Another features is the support of replication seeds for VMs copying. Let's assume you have a VM with a 1.5TB vmdk disk. If you are replicating this VM over the internet sending to destination site 1.5TB may involve some time regardless of your internet connection speed. Replication seed allows to transfer this base vmdk using an out-of-band mechanism (i.e. transfer vmdk using an USB HDD) to destination datastore. Once replication is enabled on a VM with a replication seed vSphere Replication will notice vmdk already exists at destination datastore so it will transfer only changes occurred in VM disk using CBT evaluating source vmdk and destination vmdk.

How does it work?

vSphere Replication simply copies VMs from source datastore to destination datastore to meet configured RPOs.
VM at source site is quiesced and then whole vmdk, or delta-vmdk if base disk already exists at destination, is copied to target datastore.

vSphere Replication allows retaining point in time -PIT- copies of the VM that will be seen as VM snapshots once VM is recovered.

As said before vSphere Replication doesen't care if you are replicating VMs within same local site or across the network to a remote site managed by a different vCenter, all you have to do is properly configure Replication Server used by vCenter to connect to in order to set up replication.

vSphere Replication is available as a virtual appliance that comprises a vSphere Replication Management Server that provides replication management, a vSphere Replication Server that is in charge of performing actual data transfer between source and destination, an embedded database to store data and replication schedules, and a plug-in for vSphere Web Client that provides management capabilities for vSphere Replication within Web Client.

vSphere Replication can be managed only from vSphere Web Client.

Up to 10 optional vSphere Replication Servers can be deployed and managed by a single Management Server in order to provide replication load balancing. Up to 500VMs can be replicated from source to destination datastores.

How much does it cost?

vSphere Replication is included in vSphere Essential Plus or higher so it is "free" if you already purchased at least Ess+ license.


Other blog posts in vSphere Replication Series:

vSphere Replication Part 1 - Introduction
vSphere Replication Part 2 - Installation 
vSphere Replication Part 3 - Roles & Permissions
vSphere Replication Part 4 - Configuration
vSphere Replication Part 5 - Enable Replication  
vSphere Replication Part 6 - Perform Recovery  
vSphere Replication Part 7 - Provision additional Replication Servers 

VMware: Increasing VM read performances using vFRC

Virtual Flash Read Cache (vFRC) is a VMware storage solution that leverages host-side SSDs to create a caching layer for VMs.
It allows to offload IOs from SAN to host SSDs. vFRC is defined on a per-VM basis and affects only VM reads while writes are still write-through to SAN.
SSD acts only as caching stage, SAN or DAS is still used as persistent storage. A prerequisite for enabling vFRC on a host is one or more host local SSDs (up to eight) without an existing filesystem on top of them, this is because vFRC will format them using a new filesystem, named Virtual Flash FileSystem (VFFS), a VMFS highly optimized for SSDs.
It is important to say that by enabling vFRC you can still run vMotion, HA and DRS in your cluster.

vFRC reserves a portion of space on VFFS volume per-VM basis and for individual vmdks, this means that if a VM has more drives (i.e. more vmdks) you can select which one, if any, to enable vFRC on. This reservation takes place only when VMs are powered on, when VMs are powered off chaching reserved space is reclaimed and freed.

vFRC has the following requirements:

• vCenter Server 5.5
• At least one ESXi 5.5 host
• vFRC is configured & managed only by using vSphere Web Client
• SSDs without filesystem. If VMFS or other filesystem is already on SSD remove it in order to create VFFS.

And the following maximums:

• One Virtual Flash Resource (VFFS) per vSphere host
• Eight Flash devices per Virtual Flash Resource (VFFS)
• 4TB physical Flash-based device size
• 400GB of Virtual Flash Read Cache per Virtual Machine Disk (VMDK)
• 4TB of Virtual Flash Host Swap Cache per vSphere host
• 32TB Virtual Flash Resource total size


To enable vFRC open vSphere Web Client, select one host then Manage -> Settings -> Virtual Flash -> Virtual Flash Management Resource -> Add Capacity

All available host local SSDs without VMFS on top of them will be available for vFRC.

Select one or more SSDs and choose how much space will be used for vFRC. There is a minimum value of 1GB and a maximum value equal to SSD's capacity, but you can configure any value between these two limits.

Once added Virtual Flash Resource Management will reflect the size of your vFRC pool.

vFRC is enabled on selected host. If needed repeat the same procedure for other hosts in your cluster.

Next step is to define the VMs that will access vFRC caching layer.

Click on desired VM(s), or create a new one like I did, and go to Edit Settings, expand Hard Disk properties and choose how much MB/GB of vFRC space will this disk use.

If you click Advanced option you can also choose blocksize in KB according to your VM workload characterization.

Please bear in mind that vFRC is only used for reads, hence when VM guest OS reads a data block on a vmdk cached by vFRC, vFRC caching software fetches from SAN that data block plus a portion of data that will be likely soon to be used by VM based on principle of locality and stores it on host SSD.

vFRC does not improve write IOs. Data written to vmdks will pass-by vFRC and is written-through directly to SAN.

Another consideration is that not all workloads may benefit from vFRC, it could bring extremely high benefits to certain IOs and not to others, this primarily depends on workload patterns.

VMware: Uplink used by a VM in a LB NIC Teaming

In this post I will explain how to show the uplink used by a VM in a vSwitch with NIC Teaming Load Balancing.

Since we all put our uplinks in a Load Balancing NIC Teaming sometimes it could be useful for troubleshooting purpouses to see what's the specific physical uplink used by that particular VM to access the network.

To explain this I will use a real case scenario that happened to me today.

A customer reported that after vMotioning a VM from one physical host to another this VM was unable to reach a specific IP Address in his network (a router, fyi).

I checked the basic networking settings from vSphere Client like vSwitch configuration and if every vmnic reported network activity (i.e. plugged cable) and everything was fine.

vSwitch configuration was similar to this:

vm4 was the VM reporting issues.

Analyzing vSwitch0 settings this was the result:

As you can see all vmnics (vmnic0, vmnic1, vmnic2, vmnic3) are set as Active and Route Based on originating virtual port ID was selected as Load Balancing method.

Next step was to take a look at physical networking. Customer has a blade enclosure with 3 hosts, each blade (i.e. ESXi physical host) with  4 NIC: vmnic0, vmnic1, vmnic2 and vmnic3.

Each NIC was connected to a different physical switch to achieve path realiability and every switch is connected to the one 'core switch' (second switch from the top in picture below) EXCEPT switch 4...and here all yours bells and whistles will ring.

To make this story short, as most of you have already understood, this was the flaw in this configuration. By per se it's not an issue having a switch not connected to other switches but in this scenario it's a fault because it's used in conjunction with Route Based on originating virtual port ID Load Balancing.

Route Based on originating virtual port ID it's based on a Round Robin uplink assignation. This means that every VM powered on recieve in sequence the next available uplink (vmnic) in the vSwitch.

Take a look at this picture for better understandment:

VM power-on sequence in this case was:

POWER-ON SEQ.  UPLINK ASSIGNED
vm1            vmnic0
vm2            vmnic1
vm3            vmnic2
vm4            vmnic3
vm5            vmnic0

...and so on...let's pretend for example a new vm6 will be powered on this VM will use vmnic1. If vm3  and vm6 are powered off and vm6 is powered on back again it will use vmnic2 as uplink. Pretty simple, right?

In my case vm4 reported network issues reaching the router IP Address (the rightmost grey block on the picture above) because after vMotion from one host to another vm4 ended on a host's vSwitch which assigned it to vmnic3 which is connected to an isolated physical switch. This is easily solvable either by adding the missing intra-switch link or by removing vmnic3 from the Load Balancing active adapters.

After this theorethical prologue let's dive a bit into practice by explaining how it's possible to display which uplink (in a LB NIC Team) is used by a VM.

Let's login into our ESXi host.

To retrieve port informations for a particular VM we need to use the following command:

~ # esxcli network vm port list -w <World_ID>

Where <World_ID> is the World ID of our VM. This World ID uniquely identificates a VM in a host and changes every time a VM is vMotioned and a power status change occurs (i.e. vm is powered off and back on again).

To display World ID for our VMs we need to execute following command. Informations about every VM running in the host are retrieved. To keep things clean I only show you the output of vm4.

~ # esxcli vm process list

vm4
   World ID: 3307679
   Process ID: 0
   VMX Cartel ID: 3307678
   UUID: 42 19 4a 84 a3 52 c7 ca-d9 28 93 bf 20 6c 2d bc
   Display Name: vm4
   Config File: /vmfs/volumes/51dc35ee-c4b96dc0-1c1c-b4b53f5110c0/vm4/vm4.vmx


World ID: 3307679 if the value of interest.

If we run the command above this time specifying World ID we get:

~ # esxcli network vm port list -w 3307679
   Port ID: 33554444
   vSwitch: vSwitch0
   Portgroup: VM Network
   DVPort ID:
   MAC Address: 00:50:56:99:45:3b
   IP Address: 0.0.0.0
   Team Uplink: vmnic3
   Uplink Port ID: 33554438
   Active Filters:

As you can see Team Uplink: vmnic3 indicates that vmnic3 is used by this VM.

That's all!

VMware: Deploy VMs from Template using PowerCLI

After yesterday's introductive post, today I would like to share a simple, yet powerful, PowerCLI script that allows you to deploy VMs from a Template.

This is the script structure:

 1..<N> | Foreach {  
 New-VM -vmhost <ESX_HOST> -Name <VM_NAME> -Template <TEMPLATE> -Datastore <DATASTORE> [-OtherOptions]  
 }  

Where:

<N> is the number of VMs you need to deploy.
<ESX_HOST> is the host where you will deploy your VMs
<TEMPLATE> is the template from which your VMs will be created
<DATASTORE> is the datastore that will contain your VMs
[-OtherOptions] are additional options that can be used in conjunction with New-VM command.

As you can see it's really basic, but New-VM cmdlet is really powerful and it has a lot of options. To go through all on New-VM options:


Get-Help New-VM -Detailed


Here's how I used it today for deploying 6 VMs that will be Oracle RAC nodes.


Open a text editor and paste this code in it saving file with .ps1 extension.

 1..2 | Foreach {  
 New-VM -vmhost esx1.testdomain.com -Name RAC$_ -Template OL6 -Datastore datastore1  
 }  
 3..4 | Foreach {  
 New-VM -vmhost esx2.testdomain.com -Name RAC$_ -Template OL6 -Datastore datastore2 -ResourcePool TestResourcePool  
 }  
 5..6 | Foreach {  
 New-VM -vmhost esx3.testdomain.com -Name RAC$_ -Template OL6 -Datastore datastore3 | Start-VM  
 }  

As you can see we are deploying 6 VMs, named RAC1 to RAC6 from a template called OL6.
RAC1 and RAC2 will be deployed on host esx1.testdomain.com, RAC3 and RAC4 on host esx2.testdomain.com and RAC5 and RAC6 on host esx3.testdomain.com.
RAC3 and RAC4 will be placed in a resource pool called TestResourcePool.
RAC5 and RAC6 will be powered on after deployment due to Start-VM cmdlet.


To run this simple script just open VMware vSphere PowerCLI, connect to your vCenter (or host) using Connect-VIServer cmdlet as stated in VMware: vSphere PowerCLI 101 post.

A progress bar will appear in your PowerCLI console

That's all!!Happy VMware Automation!!

Oracle VM: VM Server status not recognized by VM Manager

Hello, it was a bit since my last post here. Meanwhile I had some weeks of vacation but ...when I came back here's an issue waiting for me: Oracle VM, production environment, guest VMs correctly working but VM Manager completely unable to discover server status (servers reported in stopped status with error) despite servers are running fine.

VM Manager was totally unusable, server discovery processes stucked, I had to manually abort these processes from VM Manager. After aborting these discovery processes VM Manager changed server statuses from "stopped with error" to "starting". If I performed a manually rediscovery of servers I got this error:

https://?uname?:?pwd?@10.0.0.8:8899/api/2 discover_hardware, Status: org.apache.xmlrpc.XmlRpcException: agent.utils.filelock.LockError:Lock file /var/run/ovs-agent/discover_hardware.lock failed: timeout occured.

Logged in to server via SSH and under /var/run/ovs-agent I got:

discover_hardware.lock
monitor.lock

So it became pretty clear this was an issue with ovs-agent which is the agent who performs communication between VM Server and VM Manager.

Please note that if you can afford to stop or move your VMs a server reboot could solve all this without messing with PIDs and services.

If you, like me, don't have such luck you need to deal with PIDs and services.

ovs-agent service is run by a script located at /etc/init.d/ovs-agent and it is registered as a service that can be invoked with:

service ovs-agent [stop | start | restart | status]

So I checked ovs-agent status:

[root@orclvmsrv1 ~]# service ovs-agent status
OVSHA (PID: 6763) is running...
OVSNotificationServer (PID: 6725) is running...
OVSStat (PID: 6767) is running...
OVSAgentServer (PID: 6771) is running...
OVSLogServer (PID: 6709) is running...
OVSMonitor (PID: 6759) is running...
OVSRemaster (PID: 6747) is running...

and all services were reported as running fine even if they actually are not.

My next step was to open a SR with Oracle, which involved sending a lot of logs, from  both VM Server and VM Manager, to support engineers.

One of the requested logs was VM Server "lsof" which is the list of the current open files and a support engineer made me focus on this:

[root@orclvmsrv1 ~]# lsof | grep discover_hardware.lock
python 31309 root 6u REG 104,2 0 31704 /var/run/ovs-agent/discover_hardware.lock

which basically is the PID of the (python) process currently holding "discover_hardware".

This gave me the hint of killing processes involved with ovs-agent services.

So I got all PIDs involved:

[root@orclvmsrv1 ~]# service ovs-agent status
OVSHA (PID: 6763) is running...
OVSNotificationServer (PID: 6725) is running...
OVSStat (PID: 6767) is running...
OVSAgentServer (PID: 6771) is running...
OVSLogServer (PID: 6709) is running...
OVSMonitor (PID: 6759) is running...
OVSRemaster (PID: 6747) is running...

and then killed them:

[root@orclvmsrv1 ~]# kill 6763
[root@orclvmsrv1 ~]# kill 6725
[root@orclvmsrv1 ~]# kill 6767
[root@orclvmsrv1 ~]# kill 6771
[root@orclvmsrv1 ~]# kill 6709
[root@orclvmsrv1 ~]# kill 6759
[root@orclvmsrv1 ~]# kill 6747

and killed the process wich generated file lock seen above.

[root@orclvmsrv1 ~]# kill 31309

Checked status again:

[root@orclvmsrv1 ~]# service ovs-agent status
Procss OVSHA with PID 6763 doesn't exist.

Deleted .pid, .sock, .lock files under /var/run/ovs-agent

[root@orclvmsrv1 ~]# cd /var/run/ovs-agent
[root@orclvmsrv1 ~]# rm *.pid
[root@orclvmsrv1 ~]# rm *.sock
[root@orclvmsrv1 ~]# rm *.lock

Then started ovs-agent again and everything went back to work as usual:

[root@orclvmsrv1 ~]# service ovs-agent start
Starting ovs-agent services:                     [  OK  ]

So, I'm still not able to understand why this happend and what caused this issue with ovs-agent but this was the method I followed to solve this problem.

One last note I would to add is that dealing with ovs-agent DO NOT affect guest VMs running on servers, they will be running fine even if ovs-agent is stopped, restarted, or whatsoever.

Suggestion: if you have more available servers in the pool, live migrate all guest VMs from affected server(s) to working ones and reboot affected servers.

That's all!!

Oracle VM: Some useful tips using XM command

This post states some obvious commands, which are well known by majority of sys admins using XEN hypervisors.
Anyway I whish to share these commands since I think they are pretty useful, infact I had to deal with command line when my VM Manager had troubles.

Let's start saying that XM commands are performed connecting to physical Oracle VM servers and should be used carefully since they are pretty powerful and could potentially destroy your VM environment just with a click.

A good practice is to deal with XM CLI only if VM Manager is unusable, since it's preferrable to execute tasks by VM Manager itself rather than by physical server, especially in an environment with more than one physical server because XM CLI commands could potentially lead to inconsistencies between physical servers.

Another thing to note before using XM CLI is that as stated by server itself: making manual modifications in the management domain might cause inconsistencies between Oracle VM Manager and the server.

XM provide a lot of functionalities and you can check them all by just typing:

[root@orclvmsrv1 ~]# xm

Oracle VM identifies guest VMs by assigning them an unique ID, despite the name you gave to VM during creation, so when you perform an action using XM CLI you need to refer to VMs using that ID instead of the VM name you gave.

I suggest when you create a new VM using VM Manager to take note of "VM Name - assigned ID" because it could be useful for future references.
If you didn't do that don't be worried, there's a solution anyway, you can see what's the VM name on vm.cfg file:

[root@orclvmsrv1 ~]# cd /OVS/Repositories/0004fb0000030000fc2ad4059c973859
[root@orclvmsrv1 0004fb0000030000fc2ad4059c973859]# ls
Assemblies  ISOs  lost+found  Templates  VirtualDisks  VirtualMachines

As you can see above:

0004fb0000030000fc2ad4059c973859 is the REPOSITORY ID, every repository contains the following folders: Assemblies  ISOs  lost+found  Templates  VirtualDisks  VirtualMachines

VM configuration files are placed in VirtualMachines folder.In this case I've 3 VMs in this Repository and these are VM's ID.

[root@orclvmsrv1 0004fb0000030000fc2ad4059c973859]# ls VirtualMachines/
0004fb00000600003ace83d53d5313b0  0004fb0000060000e0f49015b41ead02 0004fb00000600005d4bfe48c8fe0cf9

To check what's the name of a certain VM just have a look at vm.cfg file contained in every of the folders above.

[root@orclvmsrv1 VirtualMachines]# cat 0004fb0000060000e0f49015b41ead02/vm.cfg | grep OVM_simple_name

and search for OVM_simple_name attribute which tells you what's the name you assigned to that VM.

OVM_simple_name = 'VMM'

Now that you know how to identify VMs by theyr ID it's time to do some basic stuff:

1)List all running VMs:

[root@orclvmsrv1 ~]# xm list

Name                                        ID   Mem VCPUs      State   Time(s)
0004fb00000600003ace83d53d5313b0             1  4096     4     -b----   9343.5
0004fb0000060000e0f49015b41ead02             8  4096     4     -b----    832.9
Domain-0                                     0   726    12     r-----  14059.0

As you can see I've two running VMs in my domain.

2)Stop a running VM:

[root@orclvmsrv1 ~]# xm shutdown 0004fb00000600003ace83d53d5313b0

I gracefully stopped a guest VM

3)Kill a running VM:

Same thing as above, but instead of "shutdown" use "destroy"

[root@orclvmsrv1 ~]# xm destroy 0004fb00000600003ace83d53d5313b0

4) Start a (non running) VM:

If you have a VM that is not currently in running state and you wish to start it:

[root@orclvmsrv1 VirtualMachines]# xm create 0004fb00000600005d4bfe48c8fe0cf9/vm.cfg

Please note that in this case I was in VirtualMachines folder.You could need to use full path to vm.cfg file.


5) Live migrate running VM to another Oracle VM server in the same Pool:

This can be done in two ways, specifying destination server IP address or, as I suggest, just typing destination server name.This is useful if you have a lot of servers in your infrastructure, and remembering server names it's usually easier than IP addresses.

You need to add your servers names to host file:

[root@orclvmsrv1 ~]# vi /etc/hosts

Add all your servers:

10.0.0.103              orclvmserver2
10.0.0.104              orclvmserver3
10.0.0.105              orclvmserver4

Save file, then you can migrate your VM to orclvmserver2:

[root@orclvmsrv1 ~]# xm migrate -l 0004fb00000600005d4bfe48c8fe0cf9 orclvmserver2

In a future post I will explain the use of another great CLI: Oracle VM Manager 3 CLI
which was introduced since VM Manager 3.1.1 build 365 and performs almost any task of VM manager but via CLI instead of web interface.

That's all!!

Oracle VM: Upgrade VM Manager to 3.1.1-399

A new patch for VM Manager is released by a few hours on MOS.

This patch as stated on changelog fixes the following issues:

14338549 - Fixes the issue where the log files do not have sufficient information to determine why TCPS fails to be enabled.
14318605 - Ensures that the upgrade process should not continue in case when the Oracle VM Manager cannot be stopped.
14256453 - Fixes the issue in server rediscovery due to the yum repository URL reset.
14222824 - Fixes the issue where iSCSI initiators were no longer associated with the ZFS Storage Appliance Access Groups after a restart of the Oracle VM Manager.

Login to My Oracle Support and download patch ID 14227416

Login to your VM Manager machine and

[root@vmm ~]# unzip p14227416_30_Linux-x86-64.zip
[root@vmm ~]# mount -o loop,ro ovmm-3.1.1-upgrade-b399.iso /mnt/cdrom/
[root@vmm ~]# cd /mnt/cdrom/
[root@vmm cdrom]# ./runUpgrader.sh


Stating OVM Manager upgrade on Thu Aug  2 10:06:59 CEST 2012

Oracle VM Manager 3.1.1.399 upgrade utility
Upgrade logfile : /tmp/upgrade-2012-08-02-06.log


It is highly recommended to do a full database repository backup prior to upgrading Oracle VM Manager ...

Press any key to continue ...


Oracle VM Manager is running ...

Verifying installation status ...
Read Oracle VM Manager config file ...
Skipping database upgrade for the same product version (3.1.1 to 3.1.1)
Found Oracle VM Manager install files ...
Found Oracle VM Manager upgrader ...
Found Oracle WebLogic Server ...
Found Java ...
Using the following information :
  Database Host          : 10.0.0.74
  Database SID           : ORCL
  Database LSNR          : 1521
  Oracle VM Schema       : ovs
  Oracle VM Manager UUID : 0004fb0000010000aef89d7d546a0d5e
  Current Build ID       : 3.1.1.365
  Upgrade from version   : 3.1.1
  Upgrade to version     : 3.1.1
Using /tmp/workdir.OjEwPb3mDR for backup and export location.
Using /tmp/patchdir.bJf4OKPoG for patching.
Undeploying previous version of Oracle VM Manager application ...
Undeploying Oracle VM Manager help ...
Undeploying Oracle VM Manager console ...
Undeploying Oracle VM Manager core ...
Waiting for Oracle VM Manager core to fully undeploy...
Waiting...
Finished undeploying previous version ...
Upgrading Oracle VM Manager ...
Backing up old files to /tmp/ovm-manager-3-backup-2012-08-02-121118...
Removing old files ...
Unpacking Oracle VM Manager 3.1.1.399
Refresh system-jazn-data.xml file ...
Redeploying Oracle VM Manager core container ...
Redeploying Oracle VM Manager console ...
Redeploying Oracle VM Manager help ...
Unpacking Oracle VM Manager OVM CLI Tool
Completed upgrade to 3.1.1.399 ...
Writing updated config in /u01/app/oracle/ovm-manager-3/.config
Restart WebLogic ...
Stopping Oracle VM Manager                                 [  OK  ]
Starting Oracle VM Managernohup: ignoring input and redirecting stderr to stdout
                                                           [  OK  ]

OVM Manager upgrade finished on Thu Aug  2 10:13:55 CEST 2012

Once installation is completed login to your VM Manager and everything should be back as normal and if you go to Help -> About you will see:

Oracle VM Manager
Version: 3.1.1.399
Build: 20120716_399

Oracle VM: Timeout starting OVMM service

I encountered this problem a while ago during a VM Manager restart: OVMM service didn't started anymore going in timeout.
As a general suggestion when this happens first thing to do is to have a look at VM Manager logs, located at:

/u01/app/oracle/ovm-manager-3/machine1/base_adf_domain/servers/AdminServer/logs/AdminServer.log

if file is too long to read you could just give a try grepping only errors

cat /u01/app/oracle/ovm-manager-3/machine1/base_adf_domain/servers/AdminServer/logs/AdminServer.log | grep ERROR

In this case however OVMM Timeout was caused by an issue in /etc/init.d/ovmm file and in detail by this line:

nohup su - oracle -c "$USER_MEM_ARGS DOMAIN_PRODUCTION_MODE=true JAVA_OPTIONS=\"-Djava.awt.headless=true -Xdebug -Xrunjdwp:transport=dt_socket,address=127.0.0.1:8453,server=y,suspend=n -da:org.apache.myfaces.trinidad\" /u01/app/oracle/ovm-manager-3/machine1/base_adf_domain/startWebLogic.sh &" > /dev/null

As you can see by default ovmm start script starts WebLogic server as oracle user, instead of root user, the one used to perform VM Manager installation.
I just fixed it by simply swapping oracle user with root user, this happened despite before VM Manager installation I launched "createOracle.sh" script which performs oracle user creation and permissions assignment.

So, to solve this issue simply replace:

nohup su - oracle -c "$USER_MEM_ARGS DOMAIN_PRODUCTION_MODE=true JAVA_OPTIONS=\"-Djava.awt.headless=true -Xdebug -Xrunjdwp:transport=dt_socket,address=127.0.0.1:8453,server=y,suspend=n -da:org.apache.myfaces.trinidad\" /u01/app/oracle/ovm-manager-3/machine1/base_adf_domain/startWebLogic.sh &" > /dev/null

with:

nohup su - root -c "$USER_MEM_ARGS DOMAIN_PRODUCTION_MODE=true JAVA_OPTIONS=\"-Djava.awt.headless=true -Xdebug -Xrunjdwp:transport=dt_socket,address=127.0.0.1:8453,server=y,suspend=n -da:org.apache.myfaces.trinidad\" /u01/app/oracle/ovm-manager-3/machine1/base_adf_domain/startWebLogic.sh &" > /dev/null


Then start ovmm service again:

service ovmm start

That's all!!

Oracle VM: Using OVM Utils

OVM Utils are a set of utilities created by Wim Coekaerts (for further informations please read Wim's blog) and released on MOS that allow users to perform using Command Line Interface almost every task that can be performed using VM Manager web interface.

Installation is really simple, just download from http://support.oracle.com

Patch 13602094: ORACLE VM 3.0 UTILS RELEASE 0.4

It's a 2.9MB zip that will be installed on the same machine where Oracle VM Manager resides.

Once downloaded extract p13602094_30_Linux-x86-64.zip to

/u01/app/oracle/ovm-manager-3/


[root@vmm ovm-manager-3]# unzip p13602094_30_Linux-x86-64.zip

[root@vmm ovm-manager-3]# unzip ovm_utils_0.5.2.zip

[root@vmm ovm-manager-3]# cd ovm_utils


OVM Utils are 6 different executables:

ovm_managercontrol: Performs actions at VM Manager level such as configuring yum repos and create cpu compatibility group. This features is intresting since using this tool allows you to group to a single server pool server with different CPU family group.
Actions allowed are: yuminfo, yumsetup, addkeystore, keystoreinfo, createcpugroup, removecpugroup, addservertocpugroup, removeserverfromcpugroup, listcpugroups, getsessiontimeout,setsessiontimeout

ovm_poolcontrol: Performs actions at Pool level. Actions allowed are: status, info, events, refresh, list, addserver, removeserver

ovm_repocontrol: Performs actions at Repository level. Actions allowed are: list, status, info, refresh, fixup, create

ovm_servercontrol: Performs actions at Server level. Actions allowed are: start, stop, restart, kill, mainton, maintoff, status, info, lock, upgrade, events, refresh, discover, list, listnfsexports, createnfsexport, deletenfsexport

ovm_vmcontrol: Performs actions at Virtual Machine level. Actions allowed are: start, stop, suspend, resume, status, restart, kill, info, lock, list, migrate, events, vcpuset, vcpuget, gettags, settags, fixcfg, delete

ovm_vmdisks: Query the Virtual Disks associated with a Virtual Machine.

ovm_vmmessage: Allows the user to send a message to a Virtual Machine in the form of a key/value pair or send a query to see if a message with a key was set inside the Virtual Machine.


These utilities have to be called passing some parameters:

./ovm_UTILITYNAME -u ADMIN_USERNAME -p ADMIN_PASSWORD -h VMMANAGER_HOSTNAME -c COMMAND_TO_SEND [-OPTIONAL_PARAMETERS]


For example here are some commands:

1)Perform a server rediscovery. My server IP Address is 10.0.0.102

./ovm_servercontrol -u admin -p password -h localhost -c discover -s 10.0.0.102 -P server_password

2)List all VMs in the Pool

./ovm_vmcontrol -u admin -p password -h localhost -c list

This command produced, in my case, the following output:

VM name : 'Win7'
     uuid   : '0004fb00000600003ace83d53d5313b0'
     server : 'orclvmsrv1'
     pool   : 'CHP'
     status : 'Stopped'

VM name : 'EnterpriseManager'
     uuid   : '0004fb00000600005d4bfe48c8fe0cf9'
     server : 'orclvmsrv1'
     pool   : 'CHP'
     status : 'Running'

VM name : 'VMM'
     uuid   : '0004fb0000060000e0f49015b41ead02'
     server : 'orclvmsrv1'
     pool   : 'CHP'
     status : 'Running'

3)Stop a running VM

./ovm_vmcontrol -u admin -p password -h localhost -c stop -v EnterpriseManager


If you need more informations about OVM Utils you can just read the manual located at

/u01/app/oracle/ovm-manager-3/ovm_utils/man/man8

That's all!!