So, no shit, there I was, nuking e-mail accounts right and left, when….
It recently came to the attention of our DBA crew that there were accounts on our systems that were not properly reflected in the customer database — in essence, potential service users we weren’t billing. Last week, they asked me to give them a list of all the logins on all the customer servers so they could reconcile the two. This is, of course, a perfectly reasonable and sensible thing to want to do, and I approve of the notion wholeheartedly.
Today, I get a list from them of accounts to delete. Over 800 of them. “Are you sure all of these need to be deleted?” I ask cautiously. “Oh yes. We’ve checked them over.” “So everyone one of these is a customer login that isn’t in the database and can be discarded safely?” “Absolutely.”
No problem. Deleting things is fun.
So tonight, I’m merrily wiping out this deadwood, when I get an ICQ message from the support group. They say that for some reason, they suddenly can’t access the database with their command line tools. These tools are perl programs we’ve developed over the years that allow them to, for instance, dump all the db and account information on a customer quickly. I run one of these commands on myself, and get back an Oracle DBI error from Perl. That’s curious. It’s not a “Database is down” error. It’s a missing library error. What on earth could have happened to the…..
You didn’t. You stupid, brainless moron f***s couldn’t have possibly…..
I flip over to the list of accounts to delete. I grep it for a specific login. I know what I’m looking for.
oracle.
They put the freaking ORACLE user on the “unnecessary” list. The user whose homedir contains all the libs necessary for all the provisioning scripts and support tools to work.
Oh, my head.
By now, I have the tech support supervisor on duty in one window and the backup admin in the other. It’s 1am. “We’re going to have to run a restore,” I say. “What’s the directory?” she asks. I tell her. She says what I already know: “Those tapes aren’t in the jukebox. I’ll have to drive in and change them.” It’s 1:15 am.
“Wait a minute!” I say. “I have an idea”
Remember last month when I was moving a bunch of files onto the hardware RAID device I had gotten them to buy because the software disk arrays were old and beginning to get cranky? I never actually deleted any of those files, I just unmounted them and left them sitting there. I quickly mount the old filesystems and check — sure enough, the files are all there. This is pretty static information, so I copy everything back over quickly to it’s original location. A quick test shows all the support programs now work perfectly again.
Backup Admin is now suitably gratified because I saved her a two-hour office visit at 1am on a Friday night. That’s lunch she owes me. Now, on Monday I get to raise holy hell to my boss about how they almost caused us an awful lot of work.
And you thought a sysadmin’s job was dull…..
