Seriously, though, I’ve never had any of these weird goddamned problems with AMD-based systems. Well, whatever.

If you randomly get the ‘Timer ISR/1: Time went backwards’ bullshit, reboot your Dom0, enter BIOS, and make sure you turn off all of Intel’s power saving nonsense. For a Sun Fire x4150, specifically, kill the following crap:

Advanced > CPU Configuration > Intel(R) SpeedStep(tm) tech
Advanced > CPU Configuration > Intel(R) C-STATE tech

With an x4150, you may end up with a weird hardware clock issue – specifically, /dev/rtc being completely inaccessible. hwclock –debug will return errno=19.

If you’re already infuriated to the point of wanting to put a server through a wall, you can do the following:

Edit /etc/rc.sysinit and /etc/init.d/halt, comment out all lines referring to hwclock
Install and configure ntp
Edit /etc/sysconfig/ntpd, add the following to the first line of options: -x -g

-x will force ntpdate to run first on startup of ntpd, ensuring your eventual completely borked time will be set before ntpd proper starts. -g will ensure that ntpd can continue to mangle your time, even if somehow your drift is several days off the mark.

Would that Xen had a more up-to-date kernel, or that RedHat hadn’t abandoned it in favor of KVM. Or would that KVM was ready for prime time. :p

Set up access to the Extra Packages for Enterprise Linux repository. Install ruby, ruby-mysql (you know you want to), rubygems. Generically, you can then gem install whatever else you might need.

Variable tests in s/printf are deliciously C-like:

printf "%s %d\n", vslist[i], enabled[i] == false ? 0 : 1

Escaping double quotes in s/printf is exactly what you’d expect:

printf "\"%s\", %d\n", vslist[i], enabled[i] == false ? 0 : 1

Including Local Files Via require:

This is where Ruby falls down, to be sure. In order to do this, you need to modify the path of require or some such. At any rate, this does it:

$: << File.dirname(__FILE__) + "/wsdl/" unless $:.include? File.dirname(__FILE__) + "/wsdl/"

Basically, this gets placed before the require statement for your local .rb file. In this situation, it's allowing me to access files in a directory titled 'wsdl' that rests under the directory that the Ruby script I'm calling is in. I believe that, if you have the files you're require'ing in the same directory, you should just be able to get rid of the + "/wsdl/" segment in the above statement.

This can be tossed in a .bashrc or /etc/bashrc or somewhere. :p

Server:

First, install the EPEL repository. If this article is dated, be sure to click the install link to grab more up to date instructions from their FAQ. But, for now:

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm

BTW, EPEL is a nice repository filled with all sorts of interesting software that RHEL/CentOS should have, but don’t. It’s basically all the up to date software of Fedora, without the annoyance of Fedora.

Next, you need to enable packages from EPEL for Munin:

echo 'includepkgs=munin munin-common munin-node perl-* rrdtool rrdtool-perl' >> /etc/yum.repos.d/epel.repo

Update: 4/3/2010: I’ve adjusted the includepkgs line to reflect what’s needed in recent updates – mainly, munin-common and moving the perl package includes to perl-* – there’s a lot more perl packages required. That’s not a bad thing – it’s generally better to grab perl packages in RPM form than through cpan, after all – much easier to keep up to date.

Now, install Munin:

yum -y install munin munin-node httpd

We’re installing munin-node as well as munin, so you can immediately see graphing awesomeness by monitoring your Munin server itself. Apache (or some webserver) is necessary to actually view your Munin data. Anyhoo, once installed, edit a few things. Mostly:

/etc/munin/munin.conf

Find the localhost entry and replace it with your monitoring server’s hostname. EG:

[lemuria.taldar.com]
    address 127.0.0.1
    use_node_name yes

Add in an alias to your apache config so you can, you know, see Munin:

echo 'Alias /munin /var/www/html/munin' >> /etc/httpd/conf/httpd.conf
/etc/init.d/httpd graceful

You may want to put that in a specific virtual host, if you have them – or even set up .htaccess protection. That’s outside the scope of this post, however.

Finally, start munin-node and ensure it starts on booth:

/etc/init.d/munin-node start
chkconfig munin-node on

Open your browser, navigate to your Munin installation (http://yourserver/munin if following the listed procedure), and… Chances are, you’ll have to wait a few minutes for Munin to actually collect data. But in a short time, you should have more pretty graphs than you ever dreamed possible.

Client:

Setting up a client is just as easy – you can skip the ‘munin’ part and just install ‘munin-node’. Once done, check:

/etc/munin/munin-node.conf

…for settings. For the most part, all you really need to adjust is:

allow ^127\.0\.0\.1$

Replace 127.0.0.1 with the IP of your munin server. Save, quit, start the node:

/etc/init.d/munin-node start
chkconfig munin-node on

Back on the server, open up:

/etc/munin/munin.conf

…And add an entry for your new client. Example:

[fatov.taldar.com]
    address 192.168.42.69
    use_node_name yes

No need to restart anything on the server, since Munin’s monitoring is cron-based.

If you’ve got iptables running, you probably need to add a rule to allow the client to accept traffic from the server. Open up:

/etc/sysconfig/iptables

..And add:

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --source IPOFYOURMUNINSERVER --dport 4949 -j ACCEPT

Then restart iptables:

/etc/init.d/iptables restart

Note the –source line; this should restrict tcp/4949 to your Munin server. Since nothing else should be talking to your client on 4949, you might as well go the distance in keeping things secure.

Enabling More Stuff

You’ll notice that different servers might have different graphs – you might see MySQL show up on a database server, while it doesn’t show up on a webserver. When you install munin-node, the installer does some checking and enables plugins that seem to make sense. What if you want more stuff? Simply check:

/usr/share/munin/plugins/

…And see what’s available. To enable something – we’ll use ‘netstat’ as example, you just make a symlink in the Munin config plugins directory. Example:

ln -s /usr/share/munin/plugins/netstat /etc/munin/plugins/netstat

Restart the node after this:

/etc/init.d/munin-node start

MySQL

You might have a MySQL plugin enabled by default, but – by default, it won’t pull anything of use. To remedy this, hop on your node, and open up:

/etc/munin/plugin-conf.d/munin-node

Add or modify the following:

[mysql*]
env.mysqlopts -u munin -pSEKRITPASSWORDSHHH

Replace ‘SEKRITPASSWORD…’ with some sort of password. Note the MySQL user (-u munin) you’ll be creating is fairly unprivileged. Speaking of which, let’s create that:

mysql
create user munin@'localhost' identified by 'SEKRITPASSWORDSHHH';
flush privileges;
quit;


…Again, replace the password. After all this – you guessed it, restart the node:

/etc/init.d/munin-node start

Shortly thereafter you should start getting MySQL data on the server.

To create a USB thumb drive repository, basically, just grab a blank USB thumb drive and – download an existing repository. Hit up a random page on the CentOS mirrors page. For updates, on your USB stick you need to create the following directory structure:

centos/5/updates/

From here, download your version’s (5.3 being the latest as of writing) ‘updates’ directory contents. You need at least one arch (i386 or x86_64) directory as a minimum, including the RPMS and repodata directories under each arch. Good idea to snag both i386 and x86_64, unless you know you’re only using one or the other. (And keep in mind, a default CentOS x86_64 install *will* install i386 rpms – more on killing those later.)

So, your USB stick should now look something like this:

centos/5/updates/i386/
...RPMS/
...repodata/
centos/5/updates/x86_64/
...RPMS/
...repodata/

Obviously, you’ll have individual RPMS in the i386/x86_64 RPMS directories, and repository files in the repodata directories.

Now for the update. First, if you want a *pure* x86_64 system, run this on the system you’re updating:

rpm -qa --queryformat='%{n}-%{v}-%{r}.%{arch}\n' | grep '\.i[3456]86$' | xargs rpm -ev

This command kills all non-x86_64, non-noarch packages.

Insert your USB thumb drive into the box you’re updating.

mkdir -p /mnt/usb
mount /dev/YOURUSBDEVICE1 /mnt/usb

Generally, when you insert a USB thumb drive, some lovely messages will pop up, including the location of the drive. Use that in the above mount command. Remember, you’re mounting a partition, not the entire device – so if the messages spewed when plugging the drive in say it’s located at, say, ‘sdg’, you’d use:

mount /dev/sdg1 /mnt/usb

Now set up yum to use your new repository:

vi /etc/yum.repos.d/CentOS-Base.repo
Comment out the 'mirrorlist' line under the [updates] repo.
Uncomment the 'baseurl' line and set it to:
file:///mnt/usb/centos/$releasever/updates/$basearch/

Save the file, quit. Execute:

yum --disablerepo=* --enablerepo=updates -y update

If you want to undo your repository info (always a good idea, unless you’re going to permanantly be swapping out USB drives for updates), just edit CentOS-Base.repo and uncomment ‘mirrorlist’ and comment out ‘baseurl’ under the updates repo.

The scsi-target-utils package distributed with RHEL/CentOS 5.3 is okay for simple stuff, but definitely out of date. And it completely fails when attempting to boot diskless systems via iSCSI over Infiniband. Spent quite a bit of time tonight attempting to figure out why BoIB 2.0.0 with a ConnectX card refused to communicate with my apparently working (export connectable/useable on other machines) iSCSI volume. Replaced tgtadm with the enterprise iSCSI target, and boom – instant satisfaction.

First, download the appropriate drivers for your FC HBA. I’m currently using a QLogic card, and you should be, too. If you’re not, your mileage and sanity may vary. At any rate, a quick trip to the QLogic site provided me with qla2xxx-v8.02.23_4-dist.tgz, which I shoved into /opt.

The thing of primary importance are the Linux rdac drivers. The rdac drivers they provide are out of date and will not compile against any modern kernel. You can snag the updated rdac drivers from: http://lsi.com/rdac/rdac-LINUX-09.02.C5.13-source.tar.gz

Those, too, go into /opt.

Now on with the awesome:

cd /opt
tar -zxf qla2xxx*.tgz
cd qlogic
./drvrsetup
cd qla2xxx-8*
./extras/build.sh install
cd /opt
tar -zxf rdac-LINUX-09.02.C5.13-source.tar.gz
cd linuxrdac-09.02.C5.13
make
make install
y

Open up /boot/grub/grub.conf.

Copy your existing Grub entry and paste it at the bottom of your file. This is your fallback. For the primary entry, modify the initrd line to point to mpp-YOURKERNELVERSION.img. It should look similar to the following:

initrd /mpp-2.6.18-128.1.14.el5.img

Of course, if your kernel is a different version, the numbers will be different.

Open up /etc/multipath.conf and do some pasting:

blacklist {
devnode "*"
}
devices {
device {
vendor SUN
product CSM200_R
product_blacklist "Universal Xport"
features "1 queue_if_no_path"
path_checker rdac
hardware_handler "1 rdac"
path_grouping_policy group_by_prio
prio "rdac"
failback immediate
}
}

This may need to be modified depending on what you want/need; I’m rather new to playing with the 6140 myself so I can’t really make any recommendations. This multipath.conf was actually provided to me by our contacts at Ibrix. Ibrix, as a business, is just pure awesome. They’re on the level of Zeus in terms of willingness to assist potential customers. Absolutely amazing.

Anyhow, make sure you have multipathd starting on boot:

chkconfig multipathd on

Reboot, and you should be able to mount exports from your 6140 on your RHEL/CentOS 5.3 system, and do proper failover, without things freaking the hell out.

Set new UUID on XFS Partition, RHEL/CentOS:

xfs admin -U generate /dev/sdb1

Set new UUID on XFS Partition, SLES/OpenSUSE:

xfs_admin -U generate /dev/sdb1

Set new UUID on ext3 Partition and Display New UUID (RHEL/CentOS, SLES/OpenSUSE):

tune2fs /dev/sdb1 -U random
dumpe2fs -h /dev/sdb1 | grep UUID

Naturally, replace #.#.#.# with the IP addresses of your nameservers.

/etc/sysconfig/network:
NETWORKING=yes
HOSTNAME=BOX
DOMAIN=DOMAIN.FOO

HOSTNAME is, of course, where you put the hostname of your box. This should not be a FQDN; the domain should end up in the DOMAIN variable.

/etc/sysconfig/network-scripts/ifcfg-eth0:
DEVICE=eth0
BOOTPROTO=static
IPADDR=172.16.42.142
NETMASK=255.255.255.0
NETWORK=172.16.42.0
BROADCAST=172.16.42.255
GATEWAY=172.16.42.1
ONBOOT=yes

This assumes the IP of your box is 172.16.42.142 with a netmask of 255.255.255.0. Network and broadcast will be set up in the same fashion, using .0 at the end of network and .255 for broadcast. Gateway may fluctuate depending on your own network setup – usually, gateways are located at .1, but your mileage may vary.

/etc/init.d/networking restart

…And yer off.

Basically, my CentOS 5.2 DomU refused to boot, stalling out after initializing SSH. Interactive mode showed it was freezing on starting the local service, or at least shortly afterward – no and continue options had the same result.

The following had to be shoved into the config file for the DomU, and replaced my ‘console=hvc0′ declaration:

extra ="3 xencons=tty"

The solution comes from this message: http://lists.xensource.com/archives/html/xen-users/2009-04/msg00485.html