So I decided to finally run windows VMs on my server.  When I tried to add it, I found out that I had not enabled VT on the CPU (I’m running dual Intel Xeon E5310 Clovertown 1.6GHz).  So I shut off the machine and turned on the VT in bios.  After I rebooted the machine and when CentOS 5 was loading up with the Xen kernel, it panicked.

What I saw was basically this.

Xen bug at vmcs.c:175

Panic on CPU 4

then it reboots repeatedly in this cycle.

Since this was on a live server with linux VMs, one which was hosting sarNworld, I decided to migrate them to fully play with this issue.  In theory I don’t need to play with windows on it but this was one of those “issues” that bugged me and one that I would *have* to resolve or else it would drive me crazy.

So I started to “google” the errors.  Found alot of interesting things.  What I found out was mostly to do with VMX support in the CPU.  There were also other errors but nothing that really hit home.  I decided to reformat the server and start fresh.

So I reinstalled CentOS 5 and did updates.

While Installing HyperVM on the server I ran into this issue.

Error: kernel-xen conflicts with ecryptfs-utils < 44
error reading information on service xendomains: No such file or directory
error reading information on service libvirtd: No such file or directory

Basically I wasn’t able to install the xen-kernel.  Looked up ecryptfs and it looks like it was not something I need.

wasn’t able to remove the traditional way.

rpm -e ecryptfs-utils-41-1.el5
error: “ecryptfs-utils-41-1.el5″ specifies multiple packages

I found the right command in a thread on centos forum after googling.

rpm -e –allmatches ecryptfs-utils

I switched to the Xen-kernel and rebooted the machine and

still NO LOVE.  Same kernel panic.

I looked up a bit more on VMX, but my cpu was already supporting it.  I started to think that it might be Xen’s fault although I have had zero problem with the latest Xen 3.3.x version.  So whats going on?

Well I found a gem of a thread on the google result page.

http://webui.sourcelabs.com/rhel/issues/427788

Basically the thread had a similar issue as mine and they found out it was a steppings issue on the CPUs.  I realized that I bought the 2nd CPU long after the first as I needed more power.  For clarity I looked up the definition of steppings and found some good threads.  One such one was a tom hardware thread. http://www.tomshardware.com/forum/235792-28-what-stepping

I ran a command to see my steppings.

cat /proc/cpuinfo | grep stepping

What shows up was steppings 7s and 13s.

Well the problem definately could be this and I had a gut feeling this was the issue.  I pulled the server and removed one of the CPU.  After that the machine was booted into the xen-kernel with ZERO issues.

Obviously I won’t be able to fix this issue so my best chance was to upgrade the bios version.

Worse case scenario, I would have to find a cpu with the same stepping which I have no idea how to accomplish it so I was really hoping for the bios to rescue me.

I checked the bios of the motherboard ( supermicro X7DVL-E) and found it was 1.3a.

Went on the mobo site and found out the new bios was 2.1a

Brought in my usb floppy drive and updated the bios.

Crossed my fingers and rebooted it into the xen-kernel.

Lo and behold, it booted in without any problems.

I was overjoyed.  I ran the command again.

cat /proc/cpuinfo | grep stepping

and now all I see is 7s.  It looks like the bios set all the steppings to be the same which was probably the reason why it was working in the xen kernel.

I am loving the supermicro bios today.