Ironically, the next day (today) I got a haircut, beard and all.

My latest was at a client who only calls us for the really strange problems. He is very savvy and setup most of his systems by himself. The Macs live in an AD environment leveraging ADmit Mac and NetBoot for reimaging. He has just started adding in Intel Mac Pros into the mix and needed to create new netboot images for those machines. Strangely, the Intel Mac Pros would hang on the freshly created image. The boot pinwheel would just spin forever.

So, running through my standard process of elimination troubleshooting techniques I began to NetBoot the Mac and then switch it to verbose mode (command + v). I could then watch the boot process for any errors that might occur. It finished with an “Image not found” error pointing to a specific framework and then asking to be rebooted.

I did the only logical thing I could do. I took a working copy of said framework and added it to the NetBoot image. Once rebooted and told to NetBoot the workstation came right up. Simple as that.

The client praised me for my fast fix and I went on my way. I don’t really know that I’m doing anything more than anyone else would but I have definitely gained the title of “Go-To Guy.”
I’m waiting for the day that truly backfires on all of us…

technorati tags:work, mac, NetBoot, troubleshoot

Blogged with Flock

Here’s what I ended up putting in our SR:

[The Client]‘s Xserve is running 10.4.9 and connected to and Xserve RAID with the latest firmware. It is connected to the network via gigabit ethernet and is usually connected to and external LaCie backup drive via firewire 800.

With increasing frequency, the server has been freezing up. Users are unable to connect to file services. The server does not respond to VNC, SSH or ping connections. When a keyboard, mouse and monitor are connected the mouse will move but Aqua and all programs will remain unresponsive, not even the beach ball. By this point, all of the system fans are running at full speed and the only way restore the server is to force it to power down from the front panel.

The server will quickly come back up. There is rarely anything to be found via the console. Crashdump almost never gets a chance to write and entry. Occasionally, Retrospect will warn than a crash or power interruption may have interfered with a backup.

The far majority of these crashes happen at night. Nearly every one can be traced back to the time at which the backup runs. When the backup is manually run during the day it will crash within the first fifteen minutes nearly every time.

During normal operation for [the Client] they is very rarely a crash during business hours.

While trying various backup methods I have noticed what I believe to be the problem.

During sustained read/writes the System Controller Vcore voltages can be seen to increasingly fluctuate as time goes on from Apple Server Monitor.

Normally, it sits at a steady 1.70V, but on this xserve they begin to drop after a while. Then, as they meander up and down, if it drops below 1.00V the server locks up completely.

It seems that the added overhead of local disk writes (including FW and USB) pushes it over the edge during those sustained read/writes.

Network usage (which seems to includes the Fibre Channel for the RAID) does not have quite the overhead so it never (or hasn’t) dropped below 1.00V.

I believe that the System Controller or some associated component is defective and causing an improper drain on the system. When core voltages drop too low, the systems freezes.

Earlier in the day, I was able to run a 50 GB sustained duplication via the network to a backup HD located on another machine. I was also able to run a 200 GB sustained duplication to a second array on the Xserve RAID.

While running both of those I watched the System Controller Vcore voltages fall below 1.70V and fluctuate down to as low as 1.50V but it remained online. Unfortunately, while writing this, I have lost contact with the server as it was trying to run one of the alternate backups that I had hoped would get around the problem.

On another client’s Xserve G5 with a similar setup I ran a sustained duplication while watching the System Controller Vcore and the voltages never once wavered from 1.70V.

(bad server on the left, good one on the right)
Apple would not absolutely confirm my diagnosis without sending someone onsite. They would send us a new Logic Board, for a price. The server was well out of warranty, but the cost of the LB was much better than a new server.

Once the new board was in, I ran several stress tests on the server. With transfers well over 3GB an minute for over an hour, the System Controller Vcore never once waivered from 1.70V.
I wish I would have found this one sooner, but at least it is fixed now.

technorati tags:work, Apple, Xserve

Blogged with Flock

Meanwhile, we have been very busy here at Stamm Business Technologies. Most notably, I just made the new StammWIKI live. It is very bare right now, but I’m hoping that we’ll be able to fill it up quickly.

I’m hoping that as I push the wiki, I can push a resurgence of blog posting as well.

Here’s to new wikies! *clink*

glenn@stammdebian:~$ uptime
16:49:06 up 318 days, 4:27, load average: 0.07, 0.02, 0.00

We made it through. I don’t know how long the power was off, or if it was even off at all. We may have to go through this all over again next week if they didn’t finish.

Just another 47 days of worry…

Linux 2.4.18-bf2.4 #1 Son Apr 14 09:53:28 CEST 2002 i686 unknown

Last login: Wed Feb 17 13:06:58 2006 from [obscured]
glenn@stammdebian:~$ uptime
14:49:53 up 317 days, 2:28, load average: 0.02, 0.04, 0.00

In just a few short month my precious little trooper of a Debian box will have been up, without a reboot, for a year.

Unfortunatly, in the infinte wisdom of those that be, our buildings power will be upgraded on Sunday.

This could easily be a much longer post as I describe my turmoil and the jeers of my fellow co-workers. I had gone so far as to consider renting a gas powered generator and babysitting the damn thing. Time and money have made this an overzealos plan.

Dave is going to power down the Stamm servers, like any other sane person would do and I am going to try to ride out the storm on the then nearly zero load UPS.

We’ll all find out on Monday whether or not it made it over the weekend.

317 days…

Maybe this will catch on, maybe it won’t. Who knows…

As a note, each of these blogs, while being hosted by Stamm Business Technologies and produced by employees of Stamm Business Technologies, the views expressed within are net expressly the views of Stamm Business Technologies.

EDIT: This is an import (along with a handful of other posts) from a failed effort to get a company blog site up and running.