Adventures in cc:Mail, Now With Water!

In the mid 90's I was in college, doing all the things that you do in college, like going bowling, roller skating, and servicing minivan engines on the weekends for a little extra cash.¹ Ah, the life of a big city college student!

My roommate helped me get a paid internship at a manufacturing company in the area. They had an extensive Novell Netware based infrastructure, and most of the network in the production facility (they made small vehicle engines in the plant I worked at) was token-ring, but Ethernet (10baseT) was being rolled out in the new engineering facility. I was a newly minted CNA for Novell 3.12, and was having a great time doing network admin tasks. My boss was a great person, ready to help and happy to have a couple of eager "kids" do the scut work that every IT department needs done.

One of the jobs that fell into this description was regular maintenance on the cc:Mail post office. It turns out that in those days there was a hard limit on the upside of the database at the heart of cc:Mail, and that size was 2.5GB of total email volume. Not per user, but for the entire system.² In a facility with over 100 "computer users," this limit was starting to get hit regularly during my time there.

Mind you, this was still the era where a T1 was a blazing fast network connection, and your home computer had a telephone-based modem (probably a card now, not an external box) for Internet access. CD drives were lucky to run at 6x speed, and first generation Pentium computers were just happening (my first college computer was a 486). VHS was still the video state-of-the-art, and email was still mostly plain text.

So one of the glamorous tasks I got to perform on a weekly basis was to come in on Sunday evening and run some tasks against the cc:Mail post office to reclaim space. Sometimes it was enough just to empty the trash and reclaim empty space, other times I had to lop off the oldest emails in the system, since everybody knew email wasn't a file storage system, if you needed something long term you printed it! This usually meant I spent an hour or two waiting for the system to execute my commands. These were commands that had to be run from the console on the cc:Mail server itself, so that meant I was in the data center staring at the KVM CRT for most of that time, waiting to run the next command. Sometimes I'd bring a book, other times I'd puruse the Usenet forums.

It just so happened that our data center was in the basement of the production facility. An old oil sump room had been repurposed. From a climate control perspective it was fantastic, and the room suited the production facility's needs extremely well. We had a raised floor with those 2'x2' tiles that you'd lift up with a suction cup handle to get to the wiring. (Oddly, power was run overhead, it was all data cables under the floor).

So one Sunday night I opened the door to the data center (with the shared 5-digit punch-code on the "state of the art" mechanical lock) and I heard a noise you never want to hear in a data center: running water. OH SHIT! I pulled up a tile and sure enough, there was standing water - not just a puddle, but enough that it was likely across the entire floor by now.

Had our environment been 100% Ethernet by then it probably wouldn't have mattered. Power was run overhead, so that wasn't getting wet yet. But we were still using Type1 connectors on most of our cabling, and that meant that under the floor if you needed a longer cable you just hooked two together - the connectors were "hermaphroditic," meaning there was no "male/female" or "pin/plug" style connection, you could just jam the connectors into each other. That meant we had a bunch of data connections under the floor that weren't watertight, and we were at risk for blowing out the adapter cards in servers and distribution units.

So ~21 year old me runs back to the guard at the front desk and tells him that I need someone from maintenance in the data center ASAP, and gives him a list of names he needs to call immediately to come in.

This ~50 year old guard looks at me like he can't understand what this pimply faced college kid is doing telling him what to do! After I point out to him that he can blame me when people get upset for getting a call Sunday night, but that they'll blame him if the plant can't operate properly Monday morning because he ignored me, he suddenly had interest in making those calls!

After several rounds of "suck the water out with the wet/dry vac," and additional panic on my part, our data center manager and others arrived. Eventually we learned that we had failures on several shelves of the phone system, but overall not much damage. It turned out that someone on the factory floor had left a clean-out sink running, and the drain got clogged up, and had probably been running since Friday night when second shift left for the weekend, letting the water work its way through the floor and into the data center. Oh, and somebody had turned off the water sensors in the data center because they would sometimes throw false positives. I just have to wonder how much damage could have been done with another 12 hours of that water running.

Not sure there's any moral to this story. It's just one I love telling

^{1 Valve stem seal replacement on those Chrysler 2.5l engines were a profitable little side business, especially when all I had to do was show up and help.}

^{2 Daily spam advertising for erectile dysfunction remedies and other delights hadn't really been invented yet, thankfully.}