• Installing Xen on Ubuntu 8.04 and creating Ubuntu 8.04 domU’s (guests)

    by  • January 17, 2010 • linux, virtualisation, xen • 1 Comment

    I’ve recently been looking into virtualisation methods within Linux to find an optimal way of creating a network testbed of virtual machines. The machines are relatively high powered for this task (AMD Athlon 64 3700+ with 2GB RAM) and are more than able to run more than one Linux instance. The only real limitation I can see is the number of PCI slots the motherboard has for network adapters. :)

    The aim is to use the testbed to test and develop routing protocols, implemented through the use of Netfilter kernel modules and a user space module with all the routing logic in it. See my earlier posts on creating simple Netfilter modules.

    My virtualisation wish list was the following:

    1. The ability to dedicate network adapters to virtual instances without the use of virtual network adapters and drivers
    2. Minimize virtualisation overheads (memory, IO, network)
    3. Remotely administer instances in a cluster configuration
    4. Ability to install kernel modules on virtual instances.
    5. Do this without hardware assistance – the Athlon 64 3700+ (socket 939) doesn’t support AMD-V)

    I looked at three possible ways to virtualise in Linux:

    • KVM (kernel-based virtual machine) provides full virtualisation and requires hardware that supports Intel VT or AMD-V extensions. Guests are virtualised without requiring modification, though  paravirtualised disk and network drivers are provided in order to get better performance in VM’s with intensive I/O.
    • Xen supports both para-virtualisation and hardware-assisted virtualisation. In paravirtualisation, computers without Intel VT or AMD-V can run guest OS’, but these OS (a domU in xen terminology) need to be modified to suit.
    • OpenVZ is an operating system-level virtualisation technology where a single kernel running on the host is containerised into multiple, isolated instances, providing near to “bare-metal” performance.

    My wish list pretty much dictated the type of virtualisation I chose:

    • Point 1. ruled out full virtualisation techniques that virtualise the network adapter. I want to ensure that the measurements taken are not skewed from the overheads of running a virtual network adapter. In addition, I want to dedicate a network adapter per virtualised instance. In Xen the documented way is to use pciback on the host to capture the necessary PCI cards, and then assigning the PCI device directly to the guest. In OpenVZ this can be achieved by creating a container with the –netdev_add flag.
    • Point 2. brought Xen and OpenVZ virtualisation to the forefront: Xen provides high performance through paravirtualisation, whilst OpenVZ guests run on a partition of the host kernel with little overheads (1-3% was quoted in various OpenVZ literature).
    • Point 3. Highlighted VM specific distributions such as Proxmox, cluster management tools such as Ganeti and distribution specific VM managers such as virt-manager.
    • Point 4. was a very specific requirement to the testbed and ruled out the use of OpenVZ due to the shared kernel approach.
    • Point 5. ruled out KVM and the hardware-assisted branch of Xen.

    So Xen paravirtualisation was chosen, with Ubuntu 8.04 as the dom0 (host) and domU (guests). This article is based on an aggregation of work from a number of sources, so thanks to these people! :)

    http://www.howtoforge.com/ubuntu-8.04-server-install-xen-from-ubuntu-repositories

    http://bderzhavets.wordpress.com/2008/11/13/backport-intrepid-xen-33-hypervisor-at-ubuntu-hardy-dom0-2624-21-xen/

    https://help.ubuntu.com/community/Xen

    http://www.howtoforge.com/forums/showthread.php?t=5110

    http://markmail.org/message/sk5mvo74kwgssnv2#query:xen%20hangs%20at%20local%20boot%20scripts+page:1+mid:srwmjga4sto7fewo+state:results

    http://deshantm.livejournal.com/29959.html

    http://www.nabble.com/xen-3.3.0-pv-pci-passthrough-co-assigned-problem-td20008460.html

    http://www.ducea.com/2006/08/07/how-to-change-the-hostname-of-a-linux-system/

    http://lists.xensource.com/archives/html/xen-users/2008-09/msg00506.html

    http://wiki.debian.org/Xen#A.27clocksource.2BAC8-0.3ATimewentbackwards.27

    http://www.howtoforge.com/forums/showthread.php?t=23069

    https://bugs.launchpad.net/ubuntu/+source/xen-3.1/+bug/174048

    Installing Ubuntu 8.04.3 with xen-server-3.3 as Dom0, and Ubuntu 8.04.3 as DomU

    Preparing host ubuntu 8.04 for Dom0 kernel

    Install ubuntu 8.04 as normal from server install

    Once installed,

    1. sudo su
    2. /etc/init.d/apparmor stop
    3. update-rc.d -f apparmor remove #removes the firewall running on host
    4. apt-get update && apt-get upgrade #upgrades server to 8.04.3)

    Then, enable backports in /etc/apt/sources.list by removing the comments from the backports lines in this file, to enable access to xen 3.3. Once you’ve done this:

    1. apt-get update
    2. apt-get install ubuntu-xen-server #will install newer 3.3 rather than 3.2

    Fix some error messages:

    mv /lib/tls /lib/tls.disabled

    Line 186 of /etc/init.d/xendomains has a repeated cut statement, replace:

    rest=`echo "$1" | cut cut -d\  -f2-`

    with

    rest=`echo "$1" | cut -d\  -f2-`

    Install some missing dependencies:

    1. apt-get install python-xml python-twisted

    Patch issue with xen add command, still present from xen 3.1.

    1. echo ‘oldxml’ > /usr/lib/python2.5/site-packages/oldxml.pth

    Restart machine, login and check active kernel is now a xen based kernel:

    1. uname –r #should be current kernel version + "-xen"

    Suppress possible error messages when guests start up

    1. vi /etc/sysctl.conf

    Adding the line: xen.independent_wallclock=1

    Preparing xen for install of Ubuntu 8.04 (hardy) guests

    Create a folder for xen images, which can be altered to suit.

    1. mkdir /home/xen #folder for xen images

    Now lets update xen-tools.conf configuration file before creating guests.

    1. vi /etc/xen-tools/xen-tools.conf

    Specifically:
    dir = /home/xen (should be the same as above, and is where guest domains and images will reside)
    dist = hardy
    arch = i386
    gateway = 192.168.x.x(replace with your network config)
    netmask = 255.255.255.0
    broadcast = 192.168.x.255
    mirror = http://gb.archive.ubuntu.com/ubuntu (UK ubuntu mirror; replace with your local mirror)
    size = 20Gb (this one is up to you)

    For each VPC

    1. xen-create-image –hostname=testY –ip=192.168.x.y –ide –force #assigns ip 192.168.x.y to the guest

    Add, or expand the “extra” line in /etc/xen/testy.cfg, to include extra="clocksource=jiffies". Now add and start the VM:

    1. xm create /etc/xen/testY.cfg –c #starts VM and attaches to console

    Now you are in VM:

    1. vi /etc/sysctl.conf

    Add the line: xen.independent_wallclock=1
    Fix some warnings and upgrade the VM:

    1. mv /lib/tls /lib/tls.disabled
    2. passwd
    3. apt-get update && apt-get upgrade #upgrades server VM to 8.04.3

    Fixing bug with tty

    After installing a guest, Xen seems to modify a file on the host it shouldn’t do (a known bug). So, on the host:

    1. vi /etc/event.d/tty1

    And change the line: exec /sbin/getty 38400 xvc0 to exec /sbin/getty 38400 tty1

    Fixing bug in assigning PCI cards to domU

    There seems to be a breaking change made between Xen 3.2 and Xen 3.3 which prevents PCI cards on the same bridge being assigned to different guests. I wanted to assign an Ethernet card to each domU instance, which looked possible, but Xen complained, saying I had to add all PCI cards. Thanks to http://www.nabble.com/xen-3.3.0-pv-pci-passthrough-co-assigned-problem-td20008460.html, the patch submitted restores Xen 3.2 fuctionality in paravirtualised guests. From what I understand this assignment restriction is due to limitations in AMD-V and Intel IOMMV, but it should not affect Xen in paravirtualized mode. The lines I added are in bold type. Make sure you use correct indentation as they are Python files.

    In /usr/lib/python2.5/site-packages/xen/util/pci.py:

    665         def do_FLR(self):
    
    666         """ Perform FLR (Functional Level Reset) for the device.
    
    667         """
    
    668         return
    
    669         if self.dev_type == DEV_TYPE_PCIe_ENDPOINT:

    In /usr/lib/python2.5/site-packages/xen/xend/server/pciif.py:

    375         pci_dev_list = pci_dev_list + [(domain, bus, slot, func)]
    
    376
    
    377         for (domain, bus, slot, func) in pci_dev_list:
    
    378                         continue
    
    379                         try:

    Now reboot the machine and Python should recompile the modified files, without error.

    Seizing PCI devices for use in domU guests

    This is a feature in Xen that allows you to seize a PCI card from the host and allocate it directly to the guest VM, without needing to virtualise the adapter in any way.
    Type lspci to determine the id’s of the PCI cards you want to assign to a domU (VM), and note these down. These id’s are in the form xx:xx.x. In my case these were 01:07.0, 01:08.0 and 01:09.0 corresponding to each of the network adapters.

    Now we’ll edit the boot entry on the Xen host in order to seize the PCI cards from the host.

    1. vi /boot/grub/menu.lst

    Go to the active kernel (i.e. one called Xen 3.3 / Ubuntu 8.04.3 / …) and add the following text to the end of the module part: pciback.hide=(xx:xx.x)(yy:yy.y). In my case, I needed pciback.hide=(01:07.0)(01:08.0)(01:09.0)

    Now reboot, and check pciback has successfully seized the appropriate devices by typing:

    1. dmesg | grep pciback

    Now go to configuration file for the VM you wish to interface to the PCI card, e.g. /etc/xen/test.conf and add in following line – I did so under vnet.

    pci = [’xx:xx.x’,’yy:yy.y’]

    For my configuration, this was:

    pci = [’01:07.0’,’01:08.0’].

    Now start the Xen VM in the normal way. You shouldn’t get any errors about co-assigning a PCI device, as this should have been dealt with by applying the above patch. Type lspci within the VM and you should see your PCI device assigned to the VM.

    Clone VPC

    This assumes the VM to clone is named old_dom and the new VM is new_dom

    1. mkdir /home/xen/domains/new_dom
    2. dd if =”/home/xen/domains/old_dom/disk.img” of=”/home/xen/domains/new_dom/disk.img” bs=4k
    3. dd if=”/home/xen/domains/old_dom/swap.img” of=”/home/xen/domains/new_dom/swap.img” bs=4k
    4. cp /etc/xen/old_dom.cfg /etc/xen/new_dom.cfg

    Edit the new_dom configuration file

    1. vi /etc/xen/new_dom.cfg

    In disk: rename any references from old_dom to new_dom

    In name: rename from old_dom to new_dom

    In vif: you’ll need to use different ip and different mac

    Add cloned VM to Xen:

    1. xm create /etc/xen/new_dom.cfg –c

    The cloned VM will still have the hostname of the old_dom, so to fix:

    1. hostname new_dom

    You might also need to update entries in /etc/hosts.

    About

    .NET developer at thetrainline.com, previously web developer at MRM Meteorite. Awarded a PhD in misbehaviour detection in wireless ad-hoc networks.A keen C# ASP.net developer bridging the gap with APIs and JavaScript frameworks, one web app at a time.

    http://www.paulkiddie.com

    One Response to Installing Xen on Ubuntu 8.04 and creating Ubuntu 8.04 domU’s (guests)

    1. Pingback: xentutorial.com

    Leave a Reply

    Your email address will not be published. Required fields are marked *