The good, the bad, and the ugly

Tuesday was certainly a mix of a day. On the whole, it was positive, but…

As some of you know, the giant soul-consuming project at work is the relocation of our data centre to a new location. When we first moved into this building 6 years ago, there really was no cost-effective alternative for hosting an operations centre the size of ours besides simply building your own facility. This has changed in recent years, and we’ve entered into an arrangement with a co-location facility which will offer us a lot of things we’ve always wanted, but couldn’t afford. It really is a first class facility, and I’m happy to have use of it.

Unfortunately, the downside is that we actually have to move all our stuff there, while causing as little disruption as possible to our existing customers. We’ve been preparing for this for several weeks, and now have come to the actual shifting of servers, which will occur in phases over the next 4 weeks. Phase one was Tuesday morning.

I woke up a little before midnight, snuggled a bit with kitanzi before getting up and having “breakfast”. I arrived at my office a little before 1am, before anyone else arrived. I started going over my prep stuff and making sure I had my ducks in a row. While there are bound to be some bumps in any operation of this magnitude, I wanted to make sure as few of them as possible were my fault. 🙂

When the rest of the moving party had arrived, we got to work. I had made the moving of one particular piece of gear my personal responsibility. It’s a network-attached filer, that we’ve had for several years, and which hosts the content of all our commercial web accounts. We had decided to move the commercial web first, since it was one of the bigger PITAs of the whole operation, and we wanted to get it out of the way. I was worried about the filer mostly because none of us had ever disassembled and reassembled it before — when it was installed, it was installed by one of the vendor’s engineers, and every major upgrade of it had also been done by the vendor. So this had the potential to be a big learning experience.

We got everything dismantled and loaded into a couple of trucks, then headed over to the new facility, about half an hour away. Checked in through various physical security points. (There are three doors that you have to provide both an id card AND a biometric scan before you get onto the data floor. For some reason, this always makes me hum the Get Smart theme.) For some reason, there was a bit of a delay actually getting our equipment checked into the building, but it finally arrived and I set about reassembling the filer. For all my worries, it actually came back up with relatively little fuss, and no errors. I was majorly relieved.

Ironically, however, the filer — the ONLY thing I had been seriously worried about in this operation — was the ONLY thing that went completely right. Every other server came up complaining of wanting manual fscks. One server had died completely. The servers which needed fscks didn’t seem to want to recognize their root passwords, making me afraid that I had misremembered them. (Ultimately, it must have been me mistyping them, as I finally got in and the machines up and running, but not before I had sent someone back to the old office to get a Solaris disk I could recover the passwords from. Yes, we should have brought that along in the first place.)

There were firewall issues, and we had to recover the dead server’s configs onto a different machine (luckily, we had a webserver that had not yet been put into production, so that was made very easy). There were routing problems and confusion and lots and lots of fun. Finally, it was getting up on 1pm, and we called for relief in the person of my beautiful and talented college, eloren. When she arrived, we briefed her on the outstanding issues, and then headed home to get some rest.

I was looking forward to the evening! Around 8am, I had slipped out for some fresh air and to give a phone call to kitanzi, who had been expecting an early morning phone call, and I wanted to see if she had gotten the news from it we were hoping for. I got her about 10 minutes before she had to leave for work, and she confirmed the excellent news: she was formally offered the job she had interviewed for on Friday! An offer letter was to arrive the next day, and her first day of work at her new job was to be October 13! We made plans to go out for sushi to celebrate that evening!

Unfortunately, as good as I felt about the state of the world, I was entirely too tired. I decided that I would stop and pick up a light lunch on my way home, and headed down Windward Parkway to Planet Smoothie. I almost got there.

I’m still not exactly sure what happened. All I know is I realized a split-second too late that the traffic in the lane I was in had stopped moving. I slammed on brakes and didn’t quite avoid running into the backend of a Lincoln Aviator.

*sigh* After checking that no one was seriously hurt, i called 911 and waited for the police to arrive. They took their time about it, but finally showed up, and we moved off into the parking lot nearby so she could write up the accident report. I received a ticket for “following too closely”, which I intend to pay without argument, but it was a serious bad end of the day. I sincerely hope that the lady I hit (who was very pregnant) was in fact OK — she was going off to get checked over by her doctor as soon as the police let us go.

I went home, called my insurance company, and set everything in motion to get my car fixed. I got a call back from the body shop who indicated that I could bring my car over that afternoon, and went to do that, then came back home.

I’m trying very hard not to let this get me down. We *still* went out to celebrate Kit’s new job, and had a good time doing it. (I waited until she got home to tell her what happened — I didn’t want to bring her mood down any sooner than was absolutely necessary). I especially want to think tigerbright for spending some time keeping me from spiraling completely out of control — by the time Kit got home, I was still miserable, but not as completely depressed as I had been earlier in the afternoon, when I was literally falling apart and playing the lovely self-blame game. Thanks, love, I owe you one. (*smooch*)

So, that’s the news from here. I still count it as an overall positive day. The accident is only a today problem, and Kit’s new job will reward us for years to come, so that’s a net win, don’t you think? I keep telling myself that. I hope that someday very soon I’ll actually believe it.