About Xen

Xen is today's leading open-source virtualization software. A Xen server runs as a Hypervisor, a thin layer that sits between the hardware and the operating system. This allows management of virtual machines running multiple operating systems on a single physical machine.

Since Xen is licensed under the GPL, it is available for free and one can download Xen over the web. Xen is also available as a commercial product but for our purposes the free version is more than sufficient.

Xen Alternatives: The only competitor to Xen in the virtualization arena is VMWare. We choose Xen over VMWare mainly for cost and performance reasons. VMWare is only available as a commercial product and many of the popular products require a larger performance penalty for virtualization than Xen.

Terminology

Here is some of the terminology that I'll be using in this document. Hopefully it will eliminate any confusion.

dom0

dom0 is a short name for domain 0. This is the master or first domain that the Xen hypervisor starts. It is special. This is the only domain that has direct access to hardware. New domains can be started from this domain. This domain must run Linux.

domU

domU is a short name for domain U. This is an unprivileged domain and does not have direct access to hardware. All of the resources in the domU are granted to it by the dom0. The domU's kernel comes from the dom0's filesystem not the virtual filesystem. This prevents a domU from upgrading it's own kernel.

Build the dom0

Now that you know a little about how great Xen is, it's time to build a robust Xen server. Let's start with the dom0.

Hardware

Since we are building a virtualization server that will be running many services we want to insure the dom0 is beefy enough to support our uses. Here are the specs of a system that currently runs many CSE services.

• Two Dual-Core 64-bit AMD Opteron CPUs
• 8GB ECC memory
• Two 400GB SATA Disk Drives
• One DVD Drive (or better)
• Redundant Power Supplys

Operating System

We choose to install Ubuntu 6.06 LTS because not only has it proven to be the fastest route to deployment but they also provide a guarantee of five years of security updates.

To install the dom0 you should download and install your Linux OS of choice. This document assumes that you will be using Ubuntu.

Installation

You can install the OS over PXE but it might be easier to use a DVD. We recommend using three partitions for the dom0.

• Root (about 4GB)
• Swap (about 4GB)
• LVM (the rest of the disk)

All three partitions should be configured as a software (or hardware) RAID 1 (mirror). This allows up to a 50% disk failure. The dom0's root partition should be formated as ext3, swap as swap, and LVM as LVM. The LVM partition will be used for all domU virtual partitions including root and swap.

Post Install

In case something goes wrong during the installer you can manually set up LVM using the following commands.

  # pvcreate /dev/md2
  # vgcreate virt /dev/md2

Update the system to the latest patch level.

  # apt-get update
  # apt-get upgrade
  # apt-get install openssh-server
  # reboot

Install Xen

Install packages required for the Xen install.

  # apt-get install iproute python python-twisted bridge-utils debootstrap

Download and unpack the latest stable version on Xen. I should note that v3.0.4 seems to be broken but v3.03 works like a champ.

  # cd /root
  # wget http://bits.xensource.com/oss-xen/release/3.0.3-0/bin.tgz/xen-3.0.3_0-install-x86_64.tgz
  # wget http://bits.xensource.com/oss-xen/release/3.0.3-0/bin.tgz/xen-3.0.3_0-install-x86_64.md5
  # md5sum xen-3.0.3_0-install-x86_64.tgz
  # cat xen-3.0.3_0-install-x86_64.md5
  # tar zxvf xen-3.0.3_0-install-x86_64.tgz
  # cd dist
  # ./install.sh

Create modules.dep and map files for the new kernel. (see /lib/modules for the correct kernel version)

  # /sbin/depmod -a 2.6.16.33-xen

Create an initrd image. Use the same version number as before. See this message for more info.

  # cd /boot
  # mkinitramfs -o initrd.img-2.6.16.33-xen 2.6.16.33-xen

Edit /boot/grub/menu.lst placing the following lines before the Automagic section.

title Xen 3.0 / XenLinux 2.6 
kernel /boot/xen-3.gz 
module /boot/vmlinuz-2.6-xen root=/dev/md0 ro
module /boot/initrd.img-2.6.16-xen

Make Xen start up and autostart selected guests when the system starts up. xend must start before, and must be stopped after xendomains.

  # update-rc.d xend start 30 2 3 4 5 . stop 31 0 1 6 .
  # update-rc.d xendomains start 31 2 3 4 5 . stop 30 0 1 6 .

Rename xen-backend.rules so that it will start up in the correct order.

  # mv /etc/udev/rules.d/xen-backend.rules /etc/udev/rules.d/92-xen-backend.rules

To make sure that /var/run/xenstored and /var/run/xend exist. Edit /etc/init.d/xend and insert the following lines after the check for /proc/xen/capabilities.

if [ ! -d /var/run/xend ] ; then
    mkdir -p /var/run/xend
fi

if [ ! -d /var/run/xenstored ] ; then
    mkdir -p /var/run/xenstored
fi

Edit /etc/init.d/xendomains and change the LOCKFILE line to read

LOCKFILE=/var/lock/xendomains

Now reboot into the Xen kernel. If there are problems during boot the first place to check is your device names in /boot/grub/menu.lst. If the system comes back up OK then run xm list and you should see a listing of all active domains (including the dom0). If all goes well you are done building the dom0.

Build a domU

So, now that you have a newly minted dom0 you are ready to build a new virtual host. In an effort to save time installing a new virtual host we have built a base image as described in the section titled "Configuring the guest domains" in this useful tutorial. We put the base image in an easy-to-find location like /root/dapper_amd64_base.img. It is around 1GB in size. This base image will be used to install all new domU's. First we must build the domU's filesystems.

Create the Filesystem

Our domU's root and swap filesystems will be located on the LVM partition so they can be resized on command. To create these new filesystems execute the following on the dom0:

  # lvcreate -L8192M -n hostname_root virt
  # lvcreate -L2048M -n hostname_swap virt
  # mkfs.ext3 /dev/virt/hostname_root
  # mkswap /dev/virt/hostname_swap

Also, if you ever need to grow the virtual domain's root partition further just run the following. Note: remember to bring the domU down first.

  # lvresize -L 40G /dev/virt/hostname_root
  # fsck.ext3 -f /dev/virt/hostname_root
  # resize2fs /dev/virt/hostname_root

Install the OS

To install the OS on the domU all we have to do is dd the base image to the newly created root partition. After we dump the image we need to remember to resize the partition so it is not a measly 1GB.

  # dd if=/root/dapper_amd64_base.img of=/dev/virt/hostname_root
  # e2fsck -f /dev/virt/hostname_root 
  # resize2fs /dev/virt/hostname_root

Configure the domU

Use the following as a guide to configure the new domU. The config file should be located at /etc/xen/hostname.

kernel = "/boot/vmlinuz-2.6.16-xen"
ramdisk = "/boot/initrd.img-2.6.16-xen"
memory = 512
name = "hostname"
vif = ['bridge=xenbr0,mac=00:16:3E:0B:0F:01']
disk = ['phy:virt/hostname_root,hda1,w','phy:virt/hostname_swap,hda2,w']
vcpus = 2
dhcp='dhcp'
hostname = "hostname"
root = "/dev/hda1 ro"
extra = "4"

Boot the domU

Now the domU is ready to be started. Run xm create hostname -c to start the domU and enter console mode. Leave off the -c if you want to start in standalone mode.

In case you get an error like:

Error: Device 0 (vif) could not be reconnected. Hotplug scripts not working.

or an error in the /var/log/xen-hotplug.log that says:

trap: 53: ERR: bad trap

The fix is to change the first line in all the scripts in /etc/xen/scripts from /bin/sh to /bin/bash.

Upgrade the domU

Since the base image is designed to be configured please read the instructions in /etc/motd on how to configure it.

Offsite Resources

Here are a few offsite resources that I've used to create this document.

• Xen Source
• Xen on Ubuntu How to
• Xen Wiki (Networking)