Want to schedule AWS snapshots?

For all of you who has a need to schedule AWS snapshots and are not so familiar with Linux shell scripting, here my code how I schedule EC2 snapshots.

I recommend to setup a dedicated script host in your AWS region where you can execute all your scripts. Usually a t1.micro AWS Linux instance will suffice.

Environment variable you need to access AWS EC2 API :

export AWS_CREDENTIAL_FILE=$HOME/.awssecret

The file .awssecret has a simple format :

AWSAccessKeyId=xxxxxxx 
AWSSecretKey=yyyyyy

which you’ll get once you create your user and generate a AWS key at the IAM user management console.

How do you call the script :

source $HOME/.bash_profile ; $HOME/bin/ec2-create-backup.sh

Following a example how I am using the script in a cron :

MAILTO=""
#Backup of XXX
00 00 * * * ( source $home/.bash_profile ; $home/bin/ec2-create-backup.sh us-west-1 vol-f233464 10 )

Before you run the script, I would test your environment if you have everything correctly setup :

 ec2-describe-snapshots --hide-tags --region us-west-1

Following now the code for my auxiliary script:

#!/bin/bash
# ec2-describe-snapshots --hide-tags --region us-west-1 -F volume-id=vol-xxxxxx
# output:
# SNAPSHOT snap-xxxxxx vol-xxxxxx completed 2013-09-19T23:24:26+0000 100% 519544898336 25 mysql 5.6
export PATH=$PATH:/opt/aws/bin
RET=0
 
usage() {
 echo -e "$0\t  ";
 echo -e "$0\tus-west-1 vol-12345 31";
 exit 1;
}
 
makeSnapshot() {
 local r=$1, vol=$2
 echo "Creating new snapshot for volume $vol"
 ec2-create-snapshot --region $region $vol -d "Backup $(date +'%Y%m%d%H%M%S') of $vol"
 export RET=$?
}
 
deleteSnapshot() {
 local r=$1, snap=$2
 echo "Deleting oldest snapshot $snap"
 ec2-delete-snapshot --region $region $snap
 export RET=$?
}
 
region=$1
volume=$2
test -z $1 && usage
test -z $2 && usage
test -z $3 && backlog=5 || backlog=$3
test -z $AWS_CREDENTIAL_FILE && echo "AWS_CREDENTIAL_FILE not set"
snaps=( $( ec2-describe-snapshots --hide-tags --region $region -F volume-id=$volume | egrep -o 'snap-[0-9A-Za-z]+' ) )
nosnaps=${#snaps[@]}
 
if [ $nosnaps -lt $backlog ]; then
 makeSnapshot $region $volume
 test $RET -gt 0 && exit 1 || exit 0
else
 lastsnap=$( let $nosnaps-1 )
 oldestTS=$( ec2-describe-snapshots --hide-tags --region $region -F
 "volume-id=$volume" | egrep -o "Backup [0-9]+ of" | egrep -o '[0-9]+' | sort | head -n1 )
 snap=$( ec2-describe-snapshots --hide-tags --region $region -F "volume-id=$volume" -F "description=*${oldestTS}*" | egrep -o 'snap-[0-9A-Za-z]+' );
 deleteSnapshot $region ${snap}
 test $RET -gt 0 && exit 1
 makeSnapshot $region $volume
 test $RET -gt 0 && exit 1 || exit 0
fi;

KVM Live Migration (RedHat)

Live Migration using shared storage

I really love the feature to migrate running VMs from one Linux hypervisor to another without having the burden of pooling or the necessity to have some sort of shared storage attached. Although migrations using shared storage, e.g.NFS, are faster and easier to accomplish. The migration can be initiated using the virt-manager GUI tool or even simpler at the virsh CLI. As a requirement I installed libvirtd and opened the network communication for libvirtd (listen_tcp to 1 @/etc/libvirt/libvirtd.conf). Also check your firewall settings on the hypervisor, if the libvirtd port is open. (as root: netstat -ntlp |grep libvirtd)

Following example shows how to migrate a VM over network using libvirtd. You should always enable TLS for libvirtd, but encryption is not always supported by 3rd party products like Cloudstack :

sudo virsh migrate --live --persistent --p2p --tunnelled <VM> qemu+tcp://<hypervisor>/system

Important is the –persistent option, which ensures the new VM on the target hypervisor stays persistent. If you don’t use the option, the VM configuration will automatically be removed from the target hypervisor and you have to start the VM on the old machine.

I usually use a temporary live migration during hardware maintenance or overload situations, with the intention to run the VM on the old metal afterwards.

 

Live Migration using local storage

KVM allows you to live migrate a VM from local to local storage. The only requirement is that you have enough RAM and a destination disk image available with the same disk size. This image needs to reside at the same path and file name.

  • Create a new disk on the destination KVM host
sudo qemu-img create -f qcow2 /var/lib/libvirt/images/<VM>.img 2G
  • Start migration on the source KVM host
sudo virsh migrate --live --p2p --tunnelled --persistent --copy-storage-all <VM> qemu+tcp://<hypervisor>/system

 

 

 

Adding Linux VLAN and bridge interfaces using libvirt

Always wanted to now how to add interfaces (VLANs or bridges) to your Linux hypervisor without dealing with the distribution specific network configuration to serve guest networks ?

Just use libvirt or its command line tool virsh to accomplish this tutorial.

First create a XML file containing your physical network layout. In this example I have a bonded Ethernet interface (bond0) and create a new Ethernet interface bond0.10 which tags the Ethernet traffic to VLAN ID 10. It is just a arbitrary number in this example but I always suggest to tag all VM guest traffic using a bridge. Ideally those bridges are running on top a bonding interface which is sometimes called teaming. Using the Linux bonding driver you can aggregate multiple interfaces to a logical interfaces which can enhance bandwidth. Your switch should support IEEE 802.3AD aggregation protocols like LACP otherwise I recommend to use active-passive bonding to enhance reliability against NIC or switch failures.

<interface type='bridge' name='br10'> 
  <start mode='onboot'/> 
  <bridge> 
    <interface type='vlan' name='bond0.10'> 
      <vlan tag='10'> 
        <interface name='bond0'/> 
      </vlan> 
    </interface> 
  </bridge> 
</interface>

Finally create your libvirt/Linux interface

sudo virsh iface-define br10.xml
sudo virsh iface-start br10

Now adding a libvirt network using this XML file. I just create a network called vlan10 and connect it to the previous created bridge.

<network connections='1'>
 <name>vlan10</name>
 <forward mode='bridge'/>
 <bridge name='br10' />
</network>

Time to assemble your libvirt network.

sudo virsh net-define vlan10.xml
sudo virsh net-start vlan10
sudo virsh net-autostart vlan10

If everything is done right just check it using virsh again :

virsh # iface-list
Name State MAC Address
--------------------------------------------
bond0 active 00:1d:09:70:a5:a2
br10 active 00:1d:09:70:a5:a2
lo active 00:00:00:00:00:00

virsh # net-list

Name State Autostart Persistent
-------------------------------------------------
default active yes yes
vlan10 active yes yes
virsh # net-info vlan10
Name vlan10
UUID a19fa2be-161a-f7cc-a776-e645a990eee2
Active: yes
Persistent: yes
Autostart: yes
Bridge: br10

For the RedHat or CentOS guys who want to know how bonding interfaces can be created, just add the file  ifcfg-bond0 (the number must be incremented with every new interface)

DEVICE=bond0
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
BONDING_OPTS="mode=1 miimon=100"

Finally assign multiple Ethernet interfaces, at least one for mode 1 (active-passive), to this bonding device by adding the following lines in each ifcfg-ethX file:

SLAVE="yes"
MASTER="bond0"

 

Install Windows 8 Pro Update without a previous product installed

Some people like me wants to install a fresh installation of Windows 8 without having Windows 7 pre installed. It’s faster and probably cleaner too. The only problem is that the early Windows 8 Pro Keys were Update Keys so they can’t be used for a full installation. After searching at Google I found a way to tell Windows that is was installed through the update process and finally activate my key :

  • Start regedit as administrator (Windows+R key and type regedit)
  • Go to HKEY_LOCAL_MACHINE/Software/Microsoft/Windows/CurrentVersion/Setup/OOBE
  • Edit MediaBootInstall from 1 to 0
  • Start a command prompt as administrator
  • Type slmgr /rearm and reboot with shutdown /r
  • After restart, go into the command prompt as administrator again
  • Type slui.exe 3 and enter your key
  • If the last step doesn’t work, reboot and try the last command again.

    Gun Control heats up again!

    In the shadow of the recent tragedy at Sandy Hook Elementary School in Newtown, Connecticut, the Anti-Gun movement showed now respect and tries to use the News, Media and current situation to push more Anti-Gun laws. For those who believe that’s the answer should carefully listen to this video.
    It can’t better express my thoughts than anything else right now. Don’t get me wrong, everything what can be done to protect our children
    should be discussed but not on costs of taking rights like the Second Amendment away.
    Believe me, I’m originally from Germany and know first hand that laws can not prevent criminal activities like the Winnenden Shooting, Germany. I believe we have to discuss what has changed in our society that leads to those tragedies (Social security, fair changes for everyone, availability of violent media, etc. ) and how we can mitigate those tragedies.

    My truly condolences go to all families who have lost their loved ones.

    Stop California SB 249

    It’s Anti Firearm Season in California, again !

    One Senate Bill is coming into focus of Firearm owners right now, SB249.
    Senate Bill 249 (Yee SD-08 “Firearms: assault weapon conversion kits”) is a California State Senate bill authored by Senator Leland Yee designed to prohibit possession, importation, making, selling, transferring, or loaning, any “conversion kit.” After reading through the SB, which is poorly and sloppy written in my eyes, the intention of this Bill is clear. It should prevent the Mag Magnet or Bullet Button or both in mid 2013. The Bullet Button, or mostly called BB, makes it legal to own a “AR15” style rifle in California which does not fall under the Assault Weapon Law, right now.

    The people in the California Senate still thinks the civilian AR15 is evil and must be prohibited. They probably should consult the FBI Crime Reports to find out that Knifes are more contributing to Homicides than rifles. See:FBI Crime Report 2010

    That said, I’m personally against unclean written laws, having virtually no effect on crime in general. Or do you think if some one who wants to commit to a crime can be stopped by prohibiting a conversion kit which makes the semi-automatic rifle a Assault-Rifle according to California Law. Do you even think, that person is following laws ? That’s why I think this Bill must be stopped.

    For more information, read the background information at stopsb249.org and sign against this law.

    Tactical…

    Tactical,… not always a good choice.

    But wait, that tactical bottle opener looks promising.

    Mission accomplished

    Mission accomplished.

    That’s the first phrase which comes up in my head as writing this entry.  A small reminder, back in 2010 we got a surprising call that my family had won in the Green Card lottery. After a lot of preparation and appointments we finally settled over to US end 2011. First me and after few weeks the rest of the family. Since then we are living in South California and feeling home right now.

    OK, enough of this. I want to publish in the next week more information what private business I’ve joined. I joined the Target Shooting business and want to share with you the exciting activity. So stay tuned and come back for new stuff.