Monday, September 24, 2007

I cant believe we pay these people

We have had a problem with our mail server losing disks, even after new ones have been put it. The disks just went to error as soon as we put them into the server, so we had a Dell engineer out to replace the backplane.

He turned up, replaced it and put a new disk in but the disk didn't come up and start to re-build right away. Instead of thinking why it didn't start to re-build the array onto it, he seamed to take a wild guess and click the online option. The disk then said it was re-initialising, witch didn't seam right but he insisted it was. He then left.

We came to bring all the mail stores back online, and lo and behold we got tons of errors about checksums, invalid data etc, so we rang Dell tech support again. The bloke told my boss that he should not have on-lined the disk, as that just brings it straight into the raid5 array without putting the data on it first. There is a re-build option he should have used.

So, we are now going to have to blow the whole virtual disk away, and restore the data from this afternoon when we took a backup.

If i went round blowing peoples data away I would get sacked, so I hope the bloke who came on site and did this gets a good ass kicking!!

No comments: