Problem is, I can’t migrate their docroots. Well, I can grab most of the docroots – but curiously, there’s a number of files which are owned by apache:apache and -rx to anything but, wull, Apache.

The virtual server in question has no root access, no sudo, and no way to configure either through their control panel. Last time I had the misfortune of dealing with GoDaddy’s support… Let’s just say, I don’t have three days for another non-resolution.

What to do? Wull, it is a Drupal site…

<?php

?>

Suck it, Trebek. Suck it long, and suck it hard.

(Please note that your mileage may vary depending on how insane the host you’re trying to rescue a client from is. :p)

Yes, clearly.

RHEL/CentOS: Networking is beyond solid, albeit best configured manually in terms of setup. Xen is absolutely archaic, but at least comes with a dom0 kernel – though it seems plenty finicky around scsi devices and VNC.

Fedora: …Hung at the splash screen post-install. On a Sun Fire x4150. Seriously? Points for the upcoming Xenified kernel in Fedora 13, though.

Ubuntu: Networking seems easy enough to configure, and works, as long as you’re not going super-crazy with abstract interconnects. Xen is absolutely archaic, and there’s not even a dom0 kernel. The answer is to pull a Debian kernel (lol) or compile your own kernel, manually patching with OpenSuSE(!) patches. Yeah, no thanks.

OpenSuSE: Xen is up to date, and has a potentially super-up-to-date dom0 kernel. Networking is curiously annoying to deal with via YaST – but the real problem is, right out of the disc, I apparently can’t do ethernet bonding with Xen due to module differences.

It’s 2010.

I’m just trying to run a solid, up-to-date Xen dom0 with simple, bonded ethernet, without having to a) compile Xen myself, or b) compile a dom0 kernel myself.

It’s pointedly not 1995.

Really, if I wanted to compile common software myself, I’d step into a police box and at least get a kick-ass theme song out of it.

Well, OpenSuSE still seems like the best solution for those who are interested in non-broken, easy-to-manage 3.4.x XEN. Time to fiddle – I have no doubt that bonding works and that I’ve just missed the obvious in the installation process.

Initial login was sucessful, the ‘Last logged in…’ line displayed, but after that nothing but a blinking cursor. Commands could be entered, but nothing would execute, and no bash prompt was seen.

The problem was quickly fixed with a strategic rsync, but let this be a lesson:

If you’re messing with root’s bash prompt, either test it on a non-root user, or open a second session to test. I was quite taken aback by this, because it didn’t dawn on me that it was possible to break bash to such a degree via a simple prompt modification.

Linux


36    37   | 38    39   | 40    41   | 42    43   | 44    45   | 46    47
sdab  sdaf | sdt   sdx  | sdar  sdav | sdaj  sdan | sdl   sdp  | sdd   sdh
-----------+------------+------------+------------+------------+-----------
24    25   | 26    27   | 28    29   | 30    31   | 32    33   | 34    35
sdaa  sdae | sds   sdw  | sdaq  sdau | sdai  sdam | sdk   sdo  | sdc   sdg
-----------+------------+------------+------------+------------+-----------
12    13   | 14    15   | 16    17   | 18    19   | 20    21   | 22    23
sdz   sdad | sdr   sdv  | sdap  sdat | sdah  sdal | sdj   sdn  | sdb   sdf
-----------+------------+------------+------------+------------+-----------
00    01   | 02    03   | 04    05   | 06    07   | 08    09   | 10    11
sdy   sdac | sdq   sdu  | sdao  sdas | sdag  sdak | sdi   sdm  | sda   sde
BOOT DEV


Solaris


36    37   | 38    39   | 40    41   | 42    43   | 44    45   | 46    47
c3t3  c3t7 | c2t3  c2t7 | c5t3  c5t7 | c4t3  c4t7 | c1t3  c1t7 | c0t3  c0t7
-----------+------------+------------+------------+------------+-----------
24    25   | 26    27   | 28    29   | 30    31   | 32    33   | 34    35
c3t2  c3t6 | c2t2  c2t6 | c5t2  c5t6 | c4t2  c4t6 | c1t2  c1t6 | c0t2  c0t6
-----------+------------+------------+------------+------------+-----------
12    13   | 14    15   | 16    17   | 18    19   | 20    21   | 22    23
c3t1  c3t5 | c2t1  c2t5 | c5t1  c5t5 | c4t1  c4t5 | c1t1  c1t5 | c0t1  c0t5
-----------+------------+------------+------------+------------+-----------
00    01   | 02    03   | 04    05   | 06    07   | 08    09   | 10    11
c3t0  c3t4 | c2t0  c2t4 | c5t0  c5t4 | c4t0  c4t4 | c1t0  c1t4 | c0t0  c0t4
BOOT DEV


OpenSolaris


36    37    | 38    39    | 40    41    | 42    43    | 44    45   | 46    47
c11t3 c11t7 | c10t3 c10t7 | c13t3 c13t7 | c12t3 c12t7 | c8t3  c8t7 | c7t3  c7t7
------------+-------------+-------------+-------------+------------+-----------
24    25    | 26    27    | 28    29    | 30    31    | 32    33   | 34    35
c11t2 c11t6 | c10t2 c10t6 | c13t2 c13t6 | c12t2 c12t6 | c8t2  c8t6 | c7t2  c7t6
------------+-------------+-------------+-------------+------------+-----------
12    13    | 14    15    | 16    17    | 18    19    | 20    21   | 22    23
c11t1 c11t5 | c10t1 c10t5 | c13t1 c13t5 | c12t1 c12t5 | c8t1  c8t5 | c7t1  c7t5
------------+-------------+-------------+-------------+------------+-----------
00    01    | 02    03    | 04    05    | 06    07    | 08    09   | 10    11
c11t0 c11t4 | c10t0 c10t4 | c13t0 c13t4 | c12t0 c12t4 | c8t0  c8t4 | c7t0  c7t4
BOOT DEV


Nexenta


36    37   | 38    39   | 40    41   | 42    43   | 44    45   | 46    47
c4t3  c4t7 | c3t3  c3t7 | c6t3  c6t7 | c5t3  c5t7 | c1t3  c1t7 | c0t3  c0t7
-----------+------------+------------+------------+------------+-----------
24    25   | 26    27   | 28    29   | 30    31   | 32    33   | 34    35
c4t2  c4t6 | c3t2  c3t6 | c6t2  c6t6 | c5t2  c5t6 | c1t2  c1t6 | c0t2  c0t6
-----------+------------+------------+------------+------------+-----------
12    13   | 14    15   | 16    17   | 18    19   | 20    21   | 22    23
c4t1  c4t5 | c3t1  c3t5 | c6t1  c6t5 | c5t1  c5t5 | c1t1  c1t5 | c0t1  c0t5
-----------+------------+------------+------------+------------+-----------
00    01   | 02    03   | 04    05   | 06    07   | 08    09   | 10    11
c4t0  c4t4 | c3t0  c3t4 | c6t0  c6t4 | c5t0  c5t4 | c1t0  c1t4 | c0t0  c0t4
BOOT DEV


dladm create-aggr -d e1000g0 -d e1000g1 -d e1000g2 -d e1000g3 1
ifconfig aggr1 plumb 10.103.0.13 up
dladm modify-aggr -L active -T short 1

Four Port Link Aggregation on Linux:

modprobe bonding mode=* miimon=100
ifconfig bond0 hw ether randommacaddress
ifconfig bond0 10.103.0.13 up
ifenslave bond0 eth0
ifenslave bond0 eth1
ifenslave bond0 eth2
ifenslave bond0 eth3

I won’t even get into the simplicity of Solaris’ persistent configuration vs. Linux’s persistent configuration – which will change between distributions, even.

The more I play with Solaris and OpenSolaris, the more I’m forced to admit that Linux is a mere toy compared to either.

Three commands and link aggregation just works. No hassle. No debugging. No conflicting instructions on how various bits should be configured. No trying to figure out what mode to use, or why your selected mode isn’t working despite the fact that it should.

It’s unfortunate Solaris requires a logical method of thinking. More people should be using it. The learning curve is supposedly steep – I dare say it is, when you’re used to the nonsensical environments of Linux, Windows or even OS X. Do yourself a favor and try to get past that.

Once you start discovering why things are so obviously set up the way they are, you’ll pause, perhaps scratch your chin, and say, “Damn – why the hell isn’t that other operating system like this?”

To create a USB thumb drive repository, basically, just grab a blank USB thumb drive and – download an existing repository. Hit up a random page on the CentOS mirrors page. For updates, on your USB stick you need to create the following directory structure:

centos/5/updates/

From here, download your version’s (5.3 being the latest as of writing) ‘updates’ directory contents. You need at least one arch (i386 or x86_64) directory as a minimum, including the RPMS and repodata directories under each arch. Good idea to snag both i386 and x86_64, unless you know you’re only using one or the other. (And keep in mind, a default CentOS x86_64 install *will* install i386 rpms – more on killing those later.)

So, your USB stick should now look something like this:

centos/5/updates/i386/
...RPMS/
...repodata/
centos/5/updates/x86_64/
...RPMS/
...repodata/

Obviously, you’ll have individual RPMS in the i386/x86_64 RPMS directories, and repository files in the repodata directories.

Now for the update. First, if you want a *pure* x86_64 system, run this on the system you’re updating:

rpm -qa --queryformat='%{n}-%{v}-%{r}.%{arch}\n' | grep '\.i[3456]86$' | xargs rpm -ev

This command kills all non-x86_64, non-noarch packages.

Insert your USB thumb drive into the box you’re updating.

mkdir -p /mnt/usb
mount /dev/YOURUSBDEVICE1 /mnt/usb

Generally, when you insert a USB thumb drive, some lovely messages will pop up, including the location of the drive. Use that in the above mount command. Remember, you’re mounting a partition, not the entire device – so if the messages spewed when plugging the drive in say it’s located at, say, ‘sdg’, you’d use:

mount /dev/sdg1 /mnt/usb

Now set up yum to use your new repository:

vi /etc/yum.repos.d/CentOS-Base.repo
Comment out the 'mirrorlist' line under the [updates] repo.
Uncomment the 'baseurl' line and set it to:
file:///mnt/usb/centos/$releasever/updates/$basearch/

Save the file, quit. Execute:

yum --disablerepo=* --enablerepo=updates -y update

If you want to undo your repository info (always a good idea, unless you’re going to permanantly be swapping out USB drives for updates), just edit CentOS-Base.repo and uncomment ‘mirrorlist’ and comment out ‘baseurl’ under the updates repo.

Set new UUID on XFS Partition, RHEL/CentOS:

xfs admin -U generate /dev/sdb1

Set new UUID on XFS Partition, SLES/OpenSUSE:

xfs_admin -U generate /dev/sdb1

Set new UUID on ext3 Partition and Display New UUID (RHEL/CentOS, SLES/OpenSUSE):

tune2fs /dev/sdb1 -U random
dumpe2fs -h /dev/sdb1 | grep UUID

Sooner or later, someone’s going to want to upgrade from CentOS 4.5 to CentOS 5. Since it’s theoretically possible to do this without setting up a new VE and manually migrating all of a customer’s data (something that can take far, far too much of our time), I’ve decided to use my own system as a test case.

So, go run down the street naked while screaming, “I’m going to upgrade to CentOS 5 – via YUM!” and see what happens. Assuming you don’t get arrested for indecent exposure, and ignoring people who will merely gawk at you… Someone, from a doorway or a random window, might scream, “Yum relaxes the fibres. It’s a scientific fact.”

That (or those) person (or people) will pointedly ignore the fact that, one, you’re on the east coast and your VE is in a datacenter on the west coast. They’ll also ignore the fact that you pointedly want to avoid doing a clean install, because you’d rather risk a small chance of problems than be guaranteed to spend hours migrating your data manually.

You’ll tell them this, and they’ll still shout, “I always recommend a clean install! Yum relaxes the fibres!”

Grass combing buggers; bleeding sodomites. Perhaps you want your fibres relaxed.

At any rate, here’s Howe:

(Yes, I’ve been waiting far, far too long to use that joke.)

Step 1: Preparation

The first thing you need to do is back up anything of any importance to you. Ideally, if you can get a full copy of your VE made, that’s perfect. The way our hardware is set up, each VE is on its own LVM partition. So in my case, all I need to do is bring down the system, mount the partition somewhere and tar it up. Then umount and restart the system.

Of course, I have access to the hardware, being the Harbinger of Upgrades that I am. Not only can I make a full, working snapshot of my VE, but I can restore it at any time I see fit.

You may not be so lucky. If you don’t have access to your hardware, the best thing to do is to try to work with your host. If you explain what you’re about, they should be happy to make time to create an image for you, and later restore it if things go bad. (It’ll be helpful if you mumble about security – hosting companies love it when users are proactive about security and updates.)

Failing that, I’d honestly abort, and start looking for a new hosting provider. If something goes horribly wrong, backups or no, you’re really not going to want to try and repair the damage. Restoring a known-good image from before your attempt at upgrading takes little time and ensures your system is back in a working condition. Attempting to repair serious borkage through yum can take a great deal of time, and isn’t guaranteed to work.

If you’re not that experienced as a sysadmin, you’re really asking for trouble by not having a known-good, complete backup.

If you still want to chance it though, here’s a quick list of stuff that you might want to back up:

• Document Root(s) of Website(s)
• Databases, preferably by creating dumps that can be reimported
• /home directories
• Mail, usually kept in /home directories and /var/spool/mail
• Apache configuration, located in /etc/httpd
• PHP configuration, in /etc/php.ini
• MySQL configuration, in /etc/my.cnf
• Postgres configuration, in /var/lib/pgsql/data

This is by no means a complete list, and the paths given are generic standards – your own files might be somewhere else.

Note: Regardless of having an image available or not, it’s a very good idea to dump your databases anyway; if there’s a major version change, you’ll likely need to re-import the data!

Now that you’ve taken care of a point-of-restore, the next step in preparation is figuring out if you have enough disk space. The ‘du -h’ command is essential here.

You want to have at least 1.5-2GB available.

If you don’t, don’t bother until you free space. The thing is, yum downloads packages before installing them. You can get away with normal yum updating with very little disk space free. The problem is, you’re trying to upgrade from CentOS 4.x to CentOS 5. Think about what that means. That’s right, mate, it’s going to download a massive number of packages – pretty much a package for every package you currently have installed. That’s going to eat a pretty piece of disk (at least as far as your average hosting account is concerned). /var/cache/yum is going to be filled with silks, spice, ambergris – and all the gems of Araby. It’d be a damned awkward situation was you to fill up your disk while updating, and thusly be brought by the lee.

So now you have a point of restoration and have ensured you have enough disk space to continue. Ready? Not yet you aren’t. Make sure you don’t have packages from third party repositories. If you don’t, great.

If you do, there are two ways of dealing with the situation. For example, if you have packages installed via RPM Forge (everyone loves RPM Forge), you’ll be able to simply update the repository information like you’ll be doing with the normal CentOS repository. Well and good, and you should be fine with that.

If you were unwise enough to download .rpm’s from somewhere and install them manually, you do have a problem. Since they’re not in an actual repository, they won’t get updated. What’s more, they may cause things to break, since they will still have dependencies – dependencies which might not be solved with CentOS 5.

The best case here is if you can temporarily remove them (via rpm -e) and download/install CentOS 5-compatible versions again after your upgrade.

And so it begins.

I’m going to start by stopping any services I can – I don’t want any havoc being played during this upgrade.

/etc/init.d/httpd stop
/etc/init.d/postfix stop
/etc/init.d/dovecot stop
/etc/init.d/postgresql stop
/etc/init.d/crond stop

Your list, of course, might be different.

It’s time to upgrade release packages. First I head over to /etc/yum.repos.d and check out my existing repository files. I make sure to take a note of any special configuration I’ve done – that is, stuff like enabling centosplus, includepkg lines, exclude lines. Write this stuff down, and then get ready to update your repositories.

For RPM Forge, I simply need to do this:

rpm -Uvh http://apt.sw.be/packages/rpmforge-release/rpmforge-release-0.3.6-1.el5...

If you’re a user of RPM Forge, you can figure out what you need to do to upgrade the RPM Forge release package by going to http://dag.wieers.com/rpm/FAQ.php.

Next up is CentOS itself. For this, I hit up http://www.centos.org/, browse to a mirror, and to the appropriate 5.0 source.

I’m looking for ‘centos-release’ and ‘centos-release-notes’. I’ll download them, and then carry out the following:

rpm -Uvh centos-release-*.rpm centos-release-notes-*.rpm

After this, hop back to /etc/yum.repos.d and make any adjustments to your repositories that you need to make. In my case, I actually don’t have any adjustments. Go me.

Now it’s time to shed any old kernel packages:

rpm -qa | grep 'kernel'
rpm -e --nodeps kernel kernel-utils

Your rpm -e arguments may look different, depending on the kernel packages you have installed.

Step 2: Execution

It’s already time for the main event, so to speak. Lay the ship alongside, thump in three broadsides and board ‘em in the smoke. Five minutes hearty and she’s ours, mates. Do you want to see a guillotine in Picadilly? Do you want your children to sing La Marseillaise? Do you want to call that raggedy arse Napoleon your King?

Mr. Mowett, Mr. Pullings, yum upgrade.

yum upgrade

This will take some time. Feel free to grab some coffee; by the time you come back, one of two things should happen. Either yum will be ready to rock and waiting for you to hit ‘yes’, or you’ll have dependency errors.

In my case, I have dependency errors:

Error: Missing Dependency: python-abi = 2.3 is needed by package python-elementtree

Ordinarily this should not happen. But as the sea is a harsh mistress, it so happens that CentOS 4.5′s python-elementtree is newer than CentOS 5′s python-elementtree. They’re also of the same version. What this means is that in a fight, CentOS 4.5′s version wins, according to yum. Since CentOS 4.5′s version wins, the CentOS 5.0 version will not be selected for installation. Unfortunately, CentOS 5 is moving to Python 2.4 as opposed to 2.3, which means CentOS 4.5′s version will not work with centOS 5′s python, and vice versa.

The real crux of the issue is this, though: yum needs python-elementtree to operate!

We can’t just remove python-elementtree and then proceed with our upgrade, because yum won’t work. We can’t forcibly replace it with CentOS 5′s package, because yum won’t work. Thankfully, we have one option open to us: Revert to CentOS 4.4′s python-elementtree, which should be older than CentOS 5′s:

wget http://vault.centos.org/4.4/os/i386/CentOS/RPMS/python-elementtree-1.2.6...
rpm -e --nodeps python-elementtree
rpm -ivh --nodeps python-elementtree*.rpm

As of the time of writing this, with CentOS 4.5, this works and continues to allow yum to operate. Your mileage may vary with future/older versions of CentOS.

Now let’s continue on schedule here – execute the upgrade again:

yum upgrade

Okay, now that my problem is fixed, I can tell you, assuming everything goes right, you should see something like:

Total download size: 265 M
Is this ok [y/N]:

Total download size will of course vary depending on what you have installed and what’s being updated. I started my system in a very barebones manner and went to extra pains to remove excess cruft I had no use for. But you can see anywhere from this pitiful amount of downloaded packages to 1+ GB. But anyway, you obviously want to say ‘y’ here.

Now yum will begin replacing packages. This is where it all happens – you are, in fact, upgrading now. This step should take a good long while as well. Go grab some more coffee.

798 actions later… And we may be good to go. Well, there’s only one thing to do – attempt to reboot, and see if the system is going to come back up or not.

shutdown -r now

Step 3: Damage Control

Give it a bit, and try getting back into the system. I have the benefit of being able to see what’s happening during boot through the XEN console from the hardware. Go me. At any rate, once your system is back up, you can either console in or attempt to SSH in.

In my case, I was greeted with the glory of:

CentOS release 5 (Final)
Kernel 2.6.16.29-xen_3.0.3.0 on an i686

trafalgar login:

Hot damn.

Post Upgrade Damage Control

After logging in, I checked to make sure all the appropriate services that should be started, were in fact started. Postgres, of course, failed to start:

/etc/init.d/postgresql start

An old version of the database format was found.
You need to upgrade the data format before using PostgreSQL.
See /usr/share/doc/postgresql-8.1.9/README.rpm-dist for more information.

That’s to be expected, of course. There’s a good reason you should take a database dump regardless of a system image. As it were, I simply moved the existing pgsql directory elsewhere, re-initialized, and re-imported my dump. One thing that bears noting, I had to actually edit my dump – I needed to kill a few improper DELETE commands (pointless since the databases didn’t even exist) and also a function called ‘greatest’ – ‘greatest’ has since become a keyword that cannot be used as a function name. This was provided by Drupal; I’m thinking Postgres finally got around to making a greatest function, and this is thusly unnecessary as well.

With that done, I went to check some stuff out. Surprise – no network. In this case, it was a simple matter of moving /etc/sysconfig/network-scripts/ifcfg-eth0.bak to /etc/sysconfig/network-scripts/ifcfg.eth0 and restarting the network with /etc/init.d/network restart. I should have certainly noticed this earlier if I hadn’t been looking at a XEN console after the reboot… Doh. At any rate, before the reboot, be sure to make sure your network config is set up.

Next up, let’s check yum. Unfortunately, I ended up with:

Loading "installonlyn" plugin

Could not find any working storages.

Interesting if true. But this was fixed by manually downloading the python-sqlite package from a CentOS 5 mirror and installing it:

rpm -Uvh python-sqlite*.rpm --force

After this, yum returned to cromulence.

Next up was my httpd.conf; the updates had made a mockery of it. Simple fix though; I pretty much just copied and pasted my old virtual host configurations directly into the new httpd.conf. Once I did that, I restarted httpd and things were good – News of the Imperium loaded right up.

After this, it was time to get e-mail back up and running. Postfix itself was fine, but dovecot was failing:

Starting Dovecot Imap: Error: Error in configuration file /etc/dovecot.conf
line 21: Unknown setting: imap_listen [FAILED]

Weird. But this is usual with upgrades to Dovecot – configuration has probably changed yet again. Sure enough, we have an .rpmnew for dovecot.conf.

cd /etc
cp dovecot.conf dovecot.old
mv dovecot.conf.rpmnew dovecot.conf

At this point dovecot starts, but I quickly shut it down. I don’t want anything getting screwed up, and since Dovecot only deals with retrieval and sending of mail, I can wait for another time. (The only thing I ever get in my inbox is spam; I’m not waiting eagerly to read it, believe me.)

My trac installation looks good – it loads without error, but I don’t play with it yet. That’s another thing I can wait on looking at.

PHP modules, however, cannot wait. At the least, I need APC installed. Come to think of it, I don’t think I had APC installed before – might’ve been using eAccelerator. But I’ve grown fond of APC… Oops. I guess I did have APC installed. Well, then..

pecl uninstall APC
pecl install APC

After this, I make sure apc.so is still being loaded, and restart Apache again. This is the only non-native PHP module I’m using presently, so I’m done with that.

That’s about it, then. I’m just giving my Drupal sites a once-over; making sure there’s no weird SQL errors popping up in watchdog since there’s been a major upgrade of Postgres. With that done, I’ve just glanced over the logs to make sure nothing funky is happening there… Not a thing. Everything looks good, and despite me needing to figure out what dovecot changed, config wise, postfix is happily shoving spam into my inbox like there’s no tomorrow.

What is this, and why is it suggesting you install bash for an rpm that doesn’t need it, especially considering you have bash installed already?

It’s very simple: Your RPM DB is completely borked; good luck installing packages now. You’re doomed. Doomed! DoOOoOOoOomed! It’s almost 5 AM, and you need to get stuff working! What will you do?! Oh, crap!

Chances are, no matter how bad this might look, you’ve only got simple corruption. Sure, simple corruption of the RPM DB means ‘completely borked’, but simple corruption also means, ‘easily fixed’. Here’s what you want to do:

First, for the love of Bob, take a backup of the existing DB. If you manage to break things more, you want to be able to restore it to a point of slightly less broken, so you can always blame someone else for the original state you found things in:

# cd /var/lib
# tar czvf /tmp/rpmdb.tar.gz rpm

Once you’ve done that, it’s time for the main event. Don’t go and get some popcorn, because this is fairly quick:

# rpm -vv --rebuilddb

Run this and it’ll rebuild your rpm database with terminal verbosity. To test, just run yum or up2date or manually attempt to rpm -ivh something, and it should act normally again.

If it doesn’t, hit up Google, or as I suggested earlier – restore that backup you made, and just blame somebody else.