Thursday, January 21, 2010

Hyperion - lets focus on infrastructure, client stories....

I've just had 3 clients experience extremely painful outtages due to the ignoring of their server environment.

Client 1: When I last left this client, microsoft mom alerts were running so anytime the server was bounced we (I) would receive an email. This is one way to keep your IT honest. I can't tell you the number of times that IT bounces and touches the Essbase server when they shouldn't be. All produciton outtages should be scheduled, no exceptions. In addition, alerts were running that would notify the business admins of Essbase (business side of the fence) when the hard drives approached 80% capacity. The Essbase hard drive filled up and there was a crash. I got contacted a day later. 3 of 15 apps would not start. I arranged to remotely restore the files from tape if they could place the tape backups in a folder. Guess what? Before this series of outtages from 3 clients at the same time, I would have guestimated a 60% success rate of recovering from tape backup. Now my educated guess has dropped down to 40%. The files they placed in the folder were missing some important files (app, .ind, .pag to name a few). After 8 weeks, I am still battling with this client to ensure backups will be usable. The problem is that the business users had their original data loads scripted and were able to quickly recover. No lesson has been learned yet. I just attempted another restore and their IT still could not replace all the necessary files. This is in an environment where I am giving IT lights out of Essbase for 5 hours a night.

hj

No comments: