- When Preparations Pay Off
- Not Quite Prepared Enough?
- Errors and Costs
- Moral of the Story
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Errors and Costs
In hindsight, we can see that we made some subtle errors in judgment. Perhaps we had unrealistic expectations for what we referred to as a RAID "server." It was actually a Fry's Electronics retail store special. (In this case, perhaps fried electronics is a better name. Sorry—couldn't help myself!) The actual nomenclature for the thing is an Iomega StorCenter ix2 network-attached file storage unit. A terabyte of storage and a gigabit Ethernet connection, all in a cool and compact little package. It worked well for us until that fateful weekend, may it rest in peace.
Now let's talk about what it cost us in terms of lost productivity (albeit courtesy of the power company), what it really costs to fix a problem like this, and how it could happen to you. Solving the problem wasn't as simple as just buying another $400 RAID server. After we determined that the spike was fatal (the server wouldn't boot), we began stepping through the process for dealing with the aftermath:
- Plan A. Remove one of the two redundant SATA drives and slap it into one of the office computers. This action quickly exposed the first flaw in our carefully conceived backup plan. Iomega uses a proprietary file system, and none of the office PCs could read it. On to plan B.
- Plan B. Tough it out and call Iomega's data-recovery people. We had six months' worth of data backed up on other servers—kind of a belt-and-suspenders approach, given the fact that the RAID server had two redundant drives. The data was also stored offsite periodically, on an online basis. At Iomega, the help line was answered on the third ring by someone with a good command of English and a pretty good idea of the situation. When asked about the procedure, he explained that we simply had to ship the unit to Iomega, which would perform a free assessment of the problem and call back with a quote to fix the unit and/or resurrect the data. If we didn't want to take any further action, we could simply ask for the unit to be returned to us. Then I asked for a range of costs associated with such a task. The quote? From $700 for a minor issue, up to $3,000 or more for something more complicated.
- Plan C. Being somewhat of a cheapskate, I searched eBay and the factory outlets for the same unit. I reasoned that I could find a good used unit, swap the drives, restore the data, and then drop-kick both units out the back door. I even found one on Iomega's website for $137, but was still unsure whether it would be compatible with the proprietary file structure.
- Plan D. The final answer was "None of the above." I solved the problem myself. Here's how: When we first installed one of the drives in a Windows XP machine (plan A), it didn't show up as a drive letter. It did show up as an unknown drive, however, when we brought up the Administrative Tools > Disk Management function under Windows and looked at the drive layout. Having heard that many manufacturers of these units use the Linux file management system, I pulled out a CD of Slax Linux purchased from an electronics flea market three years ago. Bingo! Not only did it find the Iomega drive, but it read the files. However, it wouldn't create a directory or write to a Microsoft Windows hard drive—perhaps since it was an evaluation copy. Not to be deterred, I pulled out a copy of Xandros Linux. Xandros' claim to fame is as a bridge between Linux and Microsoft environments. That did the trick. Once Xandros was installed, it was a simple matter to copy all of the files from the Linux partition to a Microsoft partition that could be read by the other computers in the office!
The original Fry's Electronics price tag on the box showed $249.99. Spending $3,000 was obviously a different kind of proposition. Lesson learned: The price to fix something isn't based on the dollar value of the unit itself; it's based on the value you place on the data it contains. I've written about this topic for 30 years, but it's another thing altogether to live it firsthand.
In a small office, a $3,000 hit is a big deal. Losing four years of archived data that we thought was safe was even worse to contemplate. Think about a law firm or CPA in a similar predicament. Riddle: Why did the CPA cross the road? Answer: Because he looked in the file and that's what he did last year. Many professions use the same data again and again. My company is one of them.
Bullet dodged, office saved, Leo was the hero. Or was he?
What did this "minor" disaster really cost? First, I had to do the work myself. That meant pulling me off of $250+ per hour assignments for clients for two full days. Total cost: $4,000 (sort of—I caught up on some of my work by putting in overtime for the rest of the week). My best guess is that we broke even on what Iomega would have charged to do the job. But I now have a new appreciation for the costs. When executives in small and medium-sized offices are working on IT problems instead of their jobs, that fully internalizes the scope of the problem. It happens much more often than people imagine. So I learned a valuable lesson, but I also got some great material for this article.