Seriously, though, I’ve never had any of these weird goddamned problems with AMD-based systems. Well, whatever.

If you randomly get the ‘Timer ISR/1: Time went backwards’ nonsense, reboot your Dom0, enter BIOS, and make sure you turn off all of Intel’s power saving nonsense. For a Sun Fire x4150, specifically, kill the following crap:

Advanced >
CPU Configuration > Intel(R) SpeedStep(tm) tech
Advanced > CPU Configuration > Intel(R) C-STATE tech

With an x4150, you may end up with a weird hardware clock issue – specifically, /dev/rtc being completely inaccessible. hwclock –debug will return errno=19.

If you’re already infuriated to the point of wanting to put a server through a wall, you can do the following:

Edit /etc/rc.sysinit and /etc/init.d/halt, comment out all lines referring to hwclock
Install and configure ntp
Edit /etc/sysconfig/ntpd, add the following to the first line of options: -x -g

-x will force ntpdate to run first on startup of ntpd, ensuring your eventual completely borked time will be set before ntpd proper starts. -g will ensure that ntpd can continue to mangle your time, even if somehow your drift is several days off the mark.

Would that Xen had a more up-to-date kernel, or that RedHat hadn’t abandoned it in favor of KVM. Or would that KVM was ready for prime time. :p

It provides output similar to the following:

Status report as of: 2010-02-19 09:00:01

dom0 Information:
------------------------------------------------------------------------------

host                   lolhost.irtehrox.com
total_memory           65535
free_memory            9276

Running domU's:
------------------------------------------------------------------------------

NAME:                  MEMORY:   VCPU:
20000-web002              2048       1
20001-web001             12288       4
20001-web002             12288       4
50005-db002               2048       1
50005-web002              4096       1
60015-dev001               256       1
60015-web004              4096       1
80123-web005              8192       1
80123-web006              8192       1

While you certainly could glean this information by looking at the list of systems you should be keeping, or even logging into the server itself, who has time for that? :p

The good stuff:

#!/usr/bin/perl

# Configuration -------------------------------------------------------------
# Domain to send e-mail from; for the actual user, we use our hostname
my $SENDER = "somedomain.foo";

# Address to e-mail report to
my $RECEIVER = 'user@somehost.foo';
# ---------------------------------------------------------------------------

# Global Variables
my $hostname = "unknown";
my $email_body = "";
my $stamp = "unknown";
my $hr = "------------------------------------------------------------------------------";

# Part 1: Intial setup; grab our time, hostname, etc.
open(HOSTNAME, "/bin/hostname |");
while()
{
        # Get rid of the break
        chomp($_);

        # Kill anything after a proper hostname; this may need to be commented out
        # or modified depending on your needs
        $_ =~ s/\..*//g;

        $hostname = $_;
}
close(HOSTNAME);
$stamp = qx(date +"%Y-%m-%d %H:%M:%S");
chomp($stamp);
$email_body = $email_body . "Status report as of: $stamp\n";

# Part 2: Grab key Xen info for the dom0
my (@info_holder);
$email_body = $email_body . "\n\ndom0 Information:\n$hr\n\n";
open(XM, "/usr/sbin/xm info |");
while()
{
        # Get rid of the break
        chomp($_);

        # Selective acquisition of data
        if(($_ =~ /^host/) || ($_ =~ /^total_memory/) || ($_ =~ /^free_memory/))
        {
                # Quick regexp to ensure the line can be split easily
                $_ =~ s/\s+/ /g;

                # Eliminate the colon
                $_ =~ s/: //g;

                # Split our line
                @info_holder = split(/ /, $_);

                # Attach line to e-mail
                $email_body = $email_body . sprintf("%-20s   %s", $info_holder[0],
$info_holder[1]) . "\n";
        }
}
close(XM);

# Part 3: Grab list of running domU's
my (@domu_holder, @domu_out, $i);
$i = 0;
open(XM, "/usr/sbin/xm list |");
while()
{
        # Get rid of the break
        chomp($_);

        # Kill the header and Dom-0 lines
        if(($_ =~ /^Name /) || ($_ =~ /^Domain-0 /))
        {
                next;
        }

        # Quick regexp to ensure the line can be split easily.
        $_ =~ s/\s+/ /g;

        # Insert into our holding array
        $domu_holder[$i] = $_;

        # Increment our counter
        $i++;
}
close XM;
@domu_holder = sort(@domu_holder);
$email_body = $email_body . "\n\nRunning domU's:\n$hr\n\nNAME:                  MEMORY:   VCPU:\n";
foreach(@domu_holder)
{
        # Split the line so we can extract name, vcpu, mem
        @domu_out = split(' ', $_);

        # Format it up!
        $email_body = $email_body . sprintf("%-20s   %7s   %5s", $domu_out[0], $domu_out[2],
$domu_out[3]) . "\n";
}

# Finally, send the e-mail
open(SENDMAIL, "|/usr/sbin/sendmail -t");
print SENDMAIL "To: $RECEIVER\n";
print SENDMAIL "From: $hostname\@$SENDER\n";
print SENDMAIL "Subject: Xen Status Report For $stamp\n";
print SENDMAIL $email_body . "\n";
close(SENDMAIL);

No doubt there’s an easier way to go about doing this, but I have a talent for simplifying the complex and complicating the obvious. Should’ve been a politician.

Basically, my CentOS 5.2 DomU refused to boot, stalling out after initializing SSH. Interactive mode showed it was freezing on starting the local service, or at least shortly afterward – no and continue options had the same result.

The following had to be shoved into the config file for the DomU, and replaced my ‘console=hvc0′ declaration:

extra ="3 xencons=tty"

The solution comes from this message: http://lists.xensource.com/archives/html/xen-users/2009-04/msg00485.html

Since that doesn’t exist, there’s… I won’t even qualify it as the next best thing. More like the lesser of two evils: a completely barebones system. It’s a pain in the ass, but any serious architecture requires building from the ground up after all.

I find myself in this situation and with the need to install Xen 3.4.0 on a CentOS 5.3 x86_64 box. Note that while I’m CPU class specific, if you’re running i386 or some weird hybrid between the two, these instructions may still be useful. Just strip –exclude=*.i?86 from any yum commands.

On with the show:

cd /opt
wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm
rpm -ivh rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm
yum --exclude=*.i?86 -y install zlib-devel openssl-devel ncurses-devel python-devel
   xorg-x11-proto-devel bridge-utils binutils gcc mercurial gettext dev86 patch
   texinfo

In the above block, we’re opening up access to the rpmforge third-party repository. Yes – ouch. Don’t worry. rpmforge is ancient, stable and kick ass. Plus I’m lazy and would rather install mercurial, the package we need, via rpm instead of by hand. We’re also installing a number of other packages that we’ll need. Depending on how bare-bones crazy you are, some of these packages might already be installed – but this is the list of ‘oh crap, my system doesn’t even have gcc!’ requirements.

cd /opt
wget http://bits.xensource.com/oss-xen/release/3.4.0/xen-3.4.0-xen.tar.gz
tar -zxf xen-3.4.0-xen.tar.gz
cd xen-3.4.0
make world
make install

Stupid easy, amirite? The nice thing about Xen is, if you’re missing a package (which you shouldn’t be), it’ll pretty much give you a good idea of what you need. Protip: Open up a CentOS5 mirror and use your browser to search for exact package names. If you can’t find one, chances are the build scripts are shooting you an actual file/executable name. In that case, try:

yum whatprovides *NAMEOFWHATEVER

That should locate what package contains whatever it is you’re missing.

Now execute the following:

ls /boot

Take careful note of the name of the file named ‘vmlinuz-BUNCHOFNUMBERS-xen’ – you need that bunch of numbers.

cd /lib/modules
depmod
mkinitrd --allow-missing /boot/initrd-2.6.18.8-xen.img 2.6.18.8-xen

You know that bunch of numbers? They’re what you put in place of the ’2.6.18.8′ in the mkinitrd command above. –allow-missing *should* be fine for most systems.

cd /boot
ln -s vmlinuz-2.6.18.8-xen vmlinuz-xen
ln -s initrd-2.6.18.8-xen.img initrd-xen

Again, replace the ’2.6.18.8′ with the numbers you retrieved via the ‘ls’ command.

Now open up /boot/grub/grub.conf in your favorite text editor. Add the following as the first block of bootable goodness:

title XEN (Xen Virtualization)
        root (hd0,0)
        kernel /xen.gz
        module /vmlinuz-xen ro root=LABEL=/1
        module /initrd-xen

You can pretty much put anything for the title – standards say to put the kernel version, but I don’t like touching grub.conf, which would have to be done in order to continually update the version each time you update your kernel. Fie on that. Thus the reason for the vmlinuz-xen and initrd-xen symlinks; if/when you upgrade, you need only re-point the symlinks rather than messing with grub.conf.

On that note, if you end up yum updating and it replaces your kernel, that *will* edit grub.conf in my experience. So make it a habit to check your grub.conf to make sure your default remains set at 0 after any yum update that plays with the now unused stock kernels.

chkconfig --level=2345 xend on
chkconfig --level=2345 xendomains on

Finally, we set Xen to start on boot.

Reboot, and assuming your system doesn’t crap out (which is beyond the scope of this post), Bob’s yer Uncle Dick.