Upgrading from CentOS 4.5 to CentOS 5, XEN Style

With CentOS 5 having been out for awhile now, customers are going to start being interested in it. No problems there; I did in fact spend a great deal of last night working on a base CentOS 5 image for future installed feature bloat and distribution. However, setting up clean VEs is a different issue entirely than upgrading old ones.

Sooner or later, someone’s going to want to upgrade from CentOS 4.5 to CentOS 5. Since it’s theoretically possible to do this without setting up a new VE and manually migrating all of a customer’s data (something that can take far, far too much of our time), I’ve decided to use my own system as a test case.

So, go run down the street naked while screaming, “I’m going to upgrade to CentOS 5 – via YUM!” and see what happens. Assuming you don’t get arrested for indecent exposure, and ignoring people who will merely gawk at you… Someone, from a doorway or a random window, might scream, “Yum relaxes the fibres. It’s a scientific fact.”

That (or those) person (or people) will pointedly ignore the fact that, one, you’re on the east coast and your VE is in a datacenter on the west coast. They’ll also ignore the fact that you pointedly want to avoid doing a clean install, because you’d rather risk a small chance of problems than be guaranteed to spend hours migrating your data manually.

You’ll tell them this, and they’ll still shout, “I always recommend a clean install! Yum relaxes the fibres!”

Grass combing buggers; bleeding sodomites. Perhaps you want your fibres relaxed.

At any rate, here’s Howe:

(Yes, I’ve been waiting far, far too long to use that joke.)

Step 1: Preparation

The first thing you need to do is back up anything of any importance to you. Ideally, if you can get a full copy of your VE made, that’s perfect. The way our hardware is set up, each VE is on its own LVM partition. So in my case, all I need to do is bring down the system, mount the partition somewhere and tar it up. Then umount and restart the system.

Of course, I have access to the hardware, being the Harbinger of Upgrades that I am. Not only can I make a full, working snapshot of my VE, but I can restore it at any time I see fit.

You may not be so lucky. If you don’t have access to your hardware, the best thing to do is to try to work with your host. If you explain what you’re about, they should be happy to make time to create an image for you, and later restore it if things go bad. (It’ll be helpful if you mumble about security – hosting companies love it when users are proactive about security and updates.)

Failing that, I’d honestly abort, and start looking for a new hosting provider. If something goes horribly wrong, backups or no, you’re really not going to want to try and repair the damage. Restoring a known-good image from before your attempt at upgrading takes little time and ensures your system is back in a working condition. Attempting to repair serious borkage through yum can take a great deal of time, and isn’t guaranteed to work.

If you’re not that experienced as a sysadmin, you’re really asking for trouble by not having a known-good, complete backup.

If you still want to chance it though, here’s a quick list of stuff that you might want to back up:

• Document Root(s) of Website(s)
• Databases, preferably by creating dumps that can be reimported
• /home directories
• Mail, usually kept in /home directories and /var/spool/mail
• Apache configuration, located in /etc/httpd
• PHP configuration, in /etc/php.ini
• MySQL configuration, in /etc/my.cnf
• Postgres configuration, in /var/lib/pgsql/data

This is by no means a complete list, and the paths given are generic standards – your own files might be somewhere else.

Note: Regardless of having an image available or not, it’s a very good idea to dump your databases anyway; if there’s a major version change, you’ll likely need to re-import the data!

Now that you’ve taken care of a point-of-restore, the next step in preparation is figuring out if you have enough disk space. The ‘du -h’ command is essential here.

You want to have at least 1.5-2GB available.

If you don’t, don’t bother until you free space. The thing is, yum downloads packages before installing them. You can get away with normal yum updating with very little disk space free. The problem is, you’re trying to upgrade from CentOS 4.x to CentOS 5. Think about what that means. That’s right, mate, it’s going to download a massive number of packages – pretty much a package for every package you currently have installed. That’s going to eat a pretty piece of disk (at least as far as your average hosting account is concerned). /var/cache/yum is going to be filled with silks, spice, ambergris – and all the gems of Araby. It’d be a damned awkward situation was you to fill up your disk while updating, and thusly be brought by the lee.

So now you have a point of restoration and have ensured you have enough disk space to continue. Ready? Not yet you aren’t. Make sure you don’t have packages from third party repositories. If you don’t, great.

If you do, there are two ways of dealing with the situation. For example, if you have packages installed via RPM Forge (everyone loves RPM Forge), you’ll be able to simply update the repository information like you’ll be doing with the normal CentOS repository. Well and good, and you should be fine with that.

If you were unwise enough to download .rpm’s from somewhere and install them manually, you do have a problem. Since they’re not in an actual repository, they won’t get updated. What’s more, they may cause things to break, since they will still have dependencies – dependencies which might not be solved with CentOS 5.

The best case here is if you can temporarily remove them (via rpm -e) and download/install CentOS 5-compatible versions again after your upgrade.

And so it begins.

I’m going to start by stopping any services I can – I don’t want any havoc being played during this upgrade.

/etc/init.d/httpd stop
/etc/init.d/postfix stop
/etc/init.d/dovecot stop
/etc/init.d/postgresql stop
/etc/init.d/crond stop

Your list, of course, might be different.

It’s time to upgrade release packages. First I head over to /etc/yum.repos.d and check out my existing repository files. I make sure to take a note of any special configuration I’ve done – that is, stuff like enabling centosplus, includepkg lines, exclude lines. Write this stuff down, and then get ready to update your repositories.

For RPM Forge, I simply need to do this:

rpm -Uvh http://apt.sw.be/packages/rpmforge-release/rpmforge-release-0.3.6-1.el5...

If you’re a user of RPM Forge, you can figure out what you need to do to upgrade the RPM Forge release package by going to http://dag.wieers.com/rpm/FAQ.php.

Next up is CentOS itself. For this, I hit up http://www.centos.org/, browse to a mirror, and to the appropriate 5.0 source.

I’m looking for ‘centos-release’ and ‘centos-release-notes’. I’ll download them, and then carry out the following:

rpm -Uvh centos-release-*.rpm centos-release-notes-*.rpm

After this, hop back to /etc/yum.repos.d and make any adjustments to your repositories that you need to make. In my case, I actually don’t have any adjustments. Go me.

Now it’s time to shed any old kernel packages:

rpm -qa | grep 'kernel'
rpm -e --nodeps kernel kernel-utils

Your rpm -e arguments may look different, depending on the kernel packages you have installed.

Step 2: Execution

It’s already time for the main event, so to speak. Lay the ship alongside, thump in three broadsides and board ‘em in the smoke. Five minutes hearty and she’s ours, mates. Do you want to see a guillotine in Picadilly? Do you want your children to sing La Marseillaise? Do you want to call that raggedy arse Napoleon your King?

Mr. Mowett, Mr. Pullings, yum upgrade.

yum upgrade

This will take some time. Feel free to grab some coffee; by the time you come back, one of two things should happen. Either yum will be ready to rock and waiting for you to hit ‘yes’, or you’ll have dependency errors.

In my case, I have dependency errors:

Error: Missing Dependency: python-abi = 2.3 is needed by package python-elementtree

Ordinarily this should not happen. But as the sea is a harsh mistress, it so happens that CentOS 4.5′s python-elementtree is newer than CentOS 5′s python-elementtree. They’re also of the same version. What this means is that in a fight, CentOS 4.5′s version wins, according to yum. Since CentOS 4.5′s version wins, the CentOS 5.0 version will not be selected for installation. Unfortunately, CentOS 5 is moving to Python 2.4 as opposed to 2.3, which means CentOS 4.5′s version will not work with centOS 5′s python, and vice versa.

The real crux of the issue is this, though: yum needs python-elementtree to operate!

We can’t just remove python-elementtree and then proceed with our upgrade, because yum won’t work. We can’t forcibly replace it with CentOS 5′s package, because yum won’t work. Thankfully, we have one option open to us: Revert to CentOS 4.4′s python-elementtree, which should be older than CentOS 5′s:

wget http://vault.centos.org/4.4/os/i386/CentOS/RPMS/python-elementtree-1.2.6...
rpm -e --nodeps python-elementtree
rpm -ivh --nodeps python-elementtree*.rpm

As of the time of writing this, with CentOS 4.5, this works and continues to allow yum to operate. Your mileage may vary with future/older versions of CentOS.

Now let’s continue on schedule here – execute the upgrade again:

yum upgrade

Okay, now that my problem is fixed, I can tell you, assuming everything goes right, you should see something like:

Total download size: 265 M
Is this ok [y/N]:

Total download size will of course vary depending on what you have installed and what’s being updated. I started my system in a very barebones manner and went to extra pains to remove excess cruft I had no use for. But you can see anywhere from this pitiful amount of downloaded packages to 1+ GB. But anyway, you obviously want to say ‘y’ here.

Now yum will begin replacing packages. This is where it all happens – you are, in fact, upgrading now. This step should take a good long while as well. Go grab some more coffee.

798 actions later… And we may be good to go. Well, there’s only one thing to do – attempt to reboot, and see if the system is going to come back up or not.

shutdown -r now

Step 3: Damage Control

Give it a bit, and try getting back into the system. I have the benefit of being able to see what’s happening during boot through the XEN console from the hardware. Go me. At any rate, once your system is back up, you can either console in or attempt to SSH in.

In my case, I was greeted with the glory of:

CentOS release 5 (Final)
Kernel 2.6.16.29-xen_3.0.3.0 on an i686

trafalgar login:

Hot damn.

Post Upgrade Damage Control

After logging in, I checked to make sure all the appropriate services that should be started, were in fact started. Postgres, of course, failed to start:

/etc/init.d/postgresql start

An old version of the database format was found.
You need to upgrade the data format before using PostgreSQL.
See /usr/share/doc/postgresql-8.1.9/README.rpm-dist for more information.

That’s to be expected, of course. There’s a good reason you should take a database dump regardless of a system image. As it were, I simply moved the existing pgsql directory elsewhere, re-initialized, and re-imported my dump. One thing that bears noting, I had to actually edit my dump – I needed to kill a few improper DELETE commands (pointless since the databases didn’t even exist) and also a function called ‘greatest’ – ‘greatest’ has since become a keyword that cannot be used as a function name. This was provided by Drupal; I’m thinking Postgres finally got around to making a greatest function, and this is thusly unnecessary as well.

With that done, I went to check some stuff out. Surprise – no network. In this case, it was a simple matter of moving /etc/sysconfig/network-scripts/ifcfg-eth0.bak to /etc/sysconfig/network-scripts/ifcfg.eth0 and restarting the network with /etc/init.d/network restart. I should have certainly noticed this earlier if I hadn’t been looking at a XEN console after the reboot… Doh. At any rate, before the reboot, be sure to make sure your network config is set up.

Next up, let’s check yum. Unfortunately, I ended up with:

Loading "installonlyn" plugin

Could not find any working storages.

Interesting if true. But this was fixed by manually downloading the python-sqlite package from a CentOS 5 mirror and installing it:

rpm -Uvh python-sqlite*.rpm --force

After this, yum returned to cromulence.

Next up was my httpd.conf; the updates had made a mockery of it. Simple fix though; I pretty much just copied and pasted my old virtual host configurations directly into the new httpd.conf. Once I did that, I restarted httpd and things were good – News of the Imperium loaded right up.

After this, it was time to get e-mail back up and running. Postfix itself was fine, but dovecot was failing:

Starting Dovecot Imap: Error: Error in configuration file /etc/dovecot.conf
line 21: Unknown setting: imap_listen [FAILED]

Weird. But this is usual with upgrades to Dovecot – configuration has probably changed yet again. Sure enough, we have an .rpmnew for dovecot.conf.

cd /etc
cp dovecot.conf dovecot.old
mv dovecot.conf.rpmnew dovecot.conf

At this point dovecot starts, but I quickly shut it down. I don’t want anything getting screwed up, and since Dovecot only deals with retrieval and sending of mail, I can wait for another time. (The only thing I ever get in my inbox is spam; I’m not waiting eagerly to read it, believe me.)

My trac installation looks good – it loads without error, but I don’t play with it yet. That’s another thing I can wait on looking at.

PHP modules, however, cannot wait. At the least, I need APC installed. Come to think of it, I don’t think I had APC installed before – might’ve been using eAccelerator. But I’ve grown fond of APC… Oops. I guess I did have APC installed. Well, then..

pecl uninstall APC
pecl install APC

After this, I make sure apc.so is still being loaded, and restart Apache again. This is the only non-native PHP module I’m using presently, so I’m done with that.

That’s about it, then. I’m just giving my Drupal sites a once-over; making sure there’s no weird SQL errors popping up in watchdog since there’s been a major upgrade of Postgres. With that done, I’ve just glanced over the logs to make sure nothing funky is happening there… Not a thing. Everything looks good, and despite me needing to figure out what dovecot changed, config wise, postfix is happily shoving spam into my inbox like there’s no tomorrow.