Why are the Nova Hypervisor statistic not updating after renaming a host while instance are running?

The nova-compute service is periodically updating hardware (VCPU, RAM, DISK) statistics for a host and is using the host name (check with hostname -f in Linux) to update the database with the available resources.

In cases where the host name has been changed while instances are running, all existing instances still reference the old host name inside the node column of the nova.instances table. All those entries need to be updated in order to get the correct amount of available resources for nova  inside the nova MySQL database:

UPDATE nova.instances SET node = '<new host name>' WHERE node = '<old host name>';

Other columns as host, launched_on should be included in a subsequent SQL.

Ever wondering why Windows Guests come up with the wrong time running inside Openstack ?

Openstack is starting the instances in UTC time when using kvm, the simulated guest hardware clock is always to UTC. This is independent from the host clock setting.

Windows OSes only assumes the hardware clock is set to local time so it boots up in UTC time until the time synchronization against the time.microsoft.com finishes and corrects it to the desired time zone.

To change the hardware clock in windows, you can add this registry entry :


That will boot the instance with the correct time, since the hardware inside the guest and host are lining using the same time zone information.

Additionally please note, that Microsoft Windows Server 2008/7 had a High CPU issue when changing to DST.

That has been fixed with the hot fix :

2800213 High CPU usage during DST changeover in Windows Server 2008, Windows 7, or Windows Server 2008 R2


Update per 6/25/2015: This hot-fix is only applicable for older Openstack releases (Havana and lower). Newer releases of Openstack do start the guest inside the localtime zone, local to the host. Additionally Openstack needs to be aware that the image you’re using is a window guest and you have to set the os_type image property to windows. But beware for errors around those glance properties. There are known issues that the image properties are not retained if you create new images from existing nova instances.

Want to schedule AWS snapshots?

For all of you who has a need to schedule AWS snapshots and are not so familiar with Linux shell scripting, here my code how I schedule EC2 snapshots.

I recommend to setup a dedicated script host in your AWS region where you can execute all your scripts. Usually a t1.micro AWS Linux instance will suffice.

Environment variable you need to access AWS EC2 API :

export AWS_CREDENTIAL_FILE=$HOME/.awssecret

The file .awssecret has a simple format :


which you’ll get once you create your user and generate a AWS key at the IAM user management console.

How do you call the script :

source $HOME/.bash_profile ; $HOME/bin/ec2-create-backup.sh

Following a example how I am using the script in a cron :

#Backup of XXX
00 00 * * * ( source $home/.bash_profile ; $home/bin/ec2-create-backup.sh us-west-1 vol-f233464 10 )

Before you run the script, I would test your environment if you have everything correctly setup :

 ec2-describe-snapshots --hide-tags --region us-west-1

Following now the code for my auxiliary script:

# ec2-describe-snapshots --hide-tags --region us-west-1 -F volume-id=vol-xxxxxx
# output:
# SNAPSHOT snap-xxxxxx vol-xxxxxx completed 2013-09-19T23:24:26+0000 100% 519544898336 25 mysql 5.6
export PATH=$PATH:/opt/aws/bin
usage() {
 echo -e "$0\t  ";
 echo -e "$0\tus-west-1 vol-12345 31";
 exit 1;
makeSnapshot() {
 local r=$1, vol=$2
 echo "Creating new snapshot for volume $vol"
 ec2-create-snapshot --region $region $vol -d "Backup $(date +'%Y%m%d%H%M%S') of $vol"
 export RET=$?
deleteSnapshot() {
 local r=$1, snap=$2
 echo "Deleting oldest snapshot $snap"
 ec2-delete-snapshot --region $region $snap
 export RET=$?
test -z $1 && usage
test -z $2 && usage
test -z $3 && backlog=5 || backlog=$3
snaps=( $( ec2-describe-snapshots --hide-tags --region $region -F volume-id=$volume | egrep -o 'snap-[0-9A-Za-z]+' ) )
if [ $nosnaps -lt $backlog ]; then
 makeSnapshot $region $volume
 test $RET -gt 0 && exit 1 || exit 0
 lastsnap=$( let $nosnaps-1 )
 oldestTS=$( ec2-describe-snapshots --hide-tags --region $region -F
 "volume-id=$volume" | egrep -o "Backup [0-9]+ of" | egrep -o '[0-9]+' | sort | head -n1 )
 snap=$( ec2-describe-snapshots --hide-tags --region $region -F "volume-id=$volume" -F "description=*${oldestTS}*" | egrep -o 'snap-[0-9A-Za-z]+' );
 deleteSnapshot $region ${snap}
 test $RET -gt 0 && exit 1
 makeSnapshot $region $volume
 test $RET -gt 0 && exit 1 || exit 0

KVM Live Migration (RedHat)

Live Migration using shared storage

I really love the feature to migrate running VMs from one Linux hypervisor to another without having the burden of pooling or the necessity to have some sort of shared storage attached. Although migrations using shared storage, e.g.NFS, are faster and easier to accomplish. The migration can be initiated using the virt-manager GUI tool or even simpler at the virsh CLI. As a requirement I installed libvirtd and opened the network communication for libvirtd (listen_tcp to 1 @/etc/libvirt/libvirtd.conf). Also check your firewall settings on the hypervisor, if the libvirtd port is open. (as root: netstat -ntlp |grep libvirtd)

Following example shows how to migrate a VM over network using libvirtd. You should always enable TLS for libvirtd, but encryption is not always supported by 3rd party products like Cloudstack :

sudo virsh migrate --live --persistent --p2p --tunnelled <VM> qemu+tcp://<hypervisor>/system

Important is the –persistent option, which ensures the new VM on the target hypervisor stays persistent. If you don’t use the option, the VM configuration will automatically be removed from the target hypervisor and you have to start the VM on the old machine.

I usually use a temporary live migration during hardware maintenance or overload situations, with the intention to run the VM on the old metal afterwards.


Live Migration using local storage

KVM allows you to live migrate a VM from local to local storage. The only requirement is that you have enough RAM and a destination disk image available with the same disk size. This image needs to reside at the same path and file name.

  • Create a new disk on the destination KVM host
sudo qemu-img create -f qcow2 /var/lib/libvirt/images/<VM>.img 2G
  • Start migration on the source KVM host
sudo virsh migrate --live --p2p --tunnelled --persistent --copy-storage-all <VM> qemu+tcp://<hypervisor>/system




Adding Linux VLAN and bridge interfaces using libvirt

Always wanted to now how to add interfaces (VLANs or bridges) to your Linux hypervisor without dealing with the distribution specific network configuration to serve guest networks ?

Just use libvirt or its command line tool virsh to accomplish this tutorial.

First create a XML file containing your physical network layout. In this example I have a bonded Ethernet interface (bond0) and create a new Ethernet interface bond0.10 which tags the Ethernet traffic to VLAN ID 10. It is just a arbitrary number in this example but I always suggest to tag all VM guest traffic using a bridge. Ideally those bridges are running on top a bonding interface which is sometimes called teaming. Using the Linux bonding driver you can aggregate multiple interfaces to a logical interfaces which can enhance bandwidth. Your switch should support IEEE 802.3AD aggregation protocols like LACP otherwise I recommend to use active-passive bonding to enhance reliability against NIC or switch failures.

<interface type='bridge' name='br10'> 
  <start mode='onboot'/> 
    <interface type='vlan' name='bond0.10'> 
      <vlan tag='10'> 
        <interface name='bond0'/> 

Finally create your libvirt/Linux interface

sudo virsh iface-define br10.xml
sudo virsh iface-start br10

Now adding a libvirt network using this XML file. I just create a network called vlan10 and connect it to the previous created bridge.

<network connections='1'>
 <forward mode='bridge'/>
 <bridge name='br10' />

Time to assemble your libvirt network.

sudo virsh net-define vlan10.xml
sudo virsh net-start vlan10
sudo virsh net-autostart vlan10

If everything is done right just check it using virsh again :

virsh # iface-list
Name State MAC Address
bond0 active 00:1d:09:70:a5:a2
br10 active 00:1d:09:70:a5:a2
lo active 00:00:00:00:00:00

virsh # net-list

Name State Autostart Persistent
default active yes yes
vlan10 active yes yes
virsh # net-info vlan10
Name vlan10
UUID a19fa2be-161a-f7cc-a776-e645a990eee2
Active: yes
Persistent: yes
Autostart: yes
Bridge: br10

For the RedHat or CentOS guys who want to know how bonding interfaces can be created, just add the file  ifcfg-bond0 (the number must be incremented with every new interface)

BONDING_OPTS="mode=1 miimon=100"

Finally assign multiple Ethernet interfaces, at least one for mode 1 (active-passive), to this bonding device by adding the following lines in each ifcfg-ethX file:



Install Windows 8 Pro Update without a previous product installed

Some people like me wants to install a fresh installation of Windows 8 without having Windows 7 pre installed. It’s faster and probably cleaner too. The only problem is that the early Windows 8 Pro Keys were Update Keys so they can’t be used for a full installation. After searching at Google I found a way to tell Windows that is was installed through the update process and finally activate my key :

  • Start regedit as administrator (Windows+R key and type regedit)
  • Go to HKEY_LOCAL_MACHINE/Software/Microsoft/Windows/CurrentVersion/Setup/OOBE
  • Edit MediaBootInstall from 1 to 0
  • Start a command prompt as administrator
  • Type slmgr /rearm and reboot with shutdown /r
  • After restart, go into the command prompt as administrator again
  • Type slui.exe 3 and enter your key
  • If the last step doesn’t work, reboot and try the last command again.

    Gun Control heats up again!

    In the shadow of the recent tragedy at Sandy Hook Elementary School in Newtown, Connecticut, the Anti-Gun movement showed now respect and tries to use the News, Media and current situation to push more Anti-Gun laws. For those who believe that’s the answer should carefully listen to this video.
    It can’t better express my thoughts than anything else right now. Don’t get me wrong, everything what can be done to protect our children
    should be discussed but not on costs of taking rights like the Second Amendment away.
    Believe me, I’m originally from Germany and know first hand that laws can not prevent criminal activities like the Winnenden Shooting, Germany. I believe we have to discuss what has changed in our society that leads to those tragedies (Social security, fair changes for everyone, availability of violent media, etc. ) and how we can mitigate those tragedies.

    My truly condolences go to all families who have lost their loved ones.

    Stop California SB 249

    It’s Anti Firearm Season in California, again !

    One Senate Bill is coming into focus of Firearm owners right now, SB249.
    Senate Bill 249 (Yee SD-08 “Firearms: assault weapon conversion kits”) is a California State Senate bill authored by Senator Leland Yee designed to prohibit possession, importation, making, selling, transferring, or loaning, any “conversion kit.” After reading through the SB, which is poorly and sloppy written in my eyes, the intention of this Bill is clear. It should prevent the Mag Magnet or Bullet Button or both in mid 2013. The Bullet Button, or mostly called BB, makes it legal to own a “AR15” style rifle in California which does not fall under the Assault Weapon Law, right now.

    The people in the California Senate still thinks the civilian AR15 is evil and must be prohibited. They probably should consult the FBI Crime Reports to find out that Knifes are more contributing to Homicides than rifles. See:FBI Crime Report 2010

    That said, I’m personally against unclean written laws, having virtually no effect on crime in general. Or do you think if some one who wants to commit to a crime can be stopped by prohibiting a conversion kit which makes the semi-automatic rifle a Assault-Rifle according to California Law. Do you even think, that person is following laws ? That’s why I think this Bill must be stopped.

    For more information, read the background information at stopsb249.org and sign against this law.