Saturday, January 23, 2010

Hyperion - lets focus on infrastructure client story 3

This is one of my favorite clients. It's perhaps one of the typical Hyperion Planning environments. The business folks are running the Hyperion Planning environment. IT at this client site is there to help. The don't put up roadblocks or resent having Hyperion in the building. Well, I last visited this client about 2 years ago. Since that time they have restored using my directions more than a handful of times. It turns out that in the quest to be super responsive to the business environment, a couple of "oops" happened. The planning app ended up getting stuck so they made a copy of the Essbase app and have been running that for a few months. I recieved a call from their IT asking how to get this new Essbase app to show up on the planning web login screen.

Well, I thought that I preached too much when I'm at a site about what Hyperion Planning means to Essbase, but apparently not in this case. Hyperion Planning maintains the outline structure of Essbase. Virtually all outline changes must run through planning first then get pushed to Essbase. In this case they restored the Essbase outline not the entire Hyperion Planning app. The answer to their question is "no. This Essbase app cannot be made available through the planning log in screen". The resolution is that the planning app needs to be rebuilt from the ground up. It turns out this is acceptable to the client as they were ready to rework some dimensions materially. They were also able to function because the extensive business logic in the business rules can be run against native Essbase as well as Hyperion Planning.
Lessons learned:
-development environments are very important. Even if you don't have a development server, at least make a copy of your production app for testing different business requirements.
-always make changes to the outline structure in Hyperion Planning then push them to Essbase via a refresh.


Hyperion - lets focus on infrastructure client story 2

This client is one of the more controlled IT environments in which I've worked. There is proper change control. The business folks have had until recently only read only access to the server. Essbase is running on Unix. There is a full time DBA (contrator 3 years). Sounds pretty good. Well the day came where I needed to restore data from 9 days ago. My typical environment provides the Essbase admins to recover the last 7 days of data and Essbase objects. This set of backups was working properly but I just needed a little bit older data restored. Since we had been in production for a few week, I assumed everything was being backed up. My Essbase admin also believed this. When the call came to restore from tape, it came to light that somehow there was an exclude on the data directory we were using for my Essbase server. So, the good news is that this came to light before a really critical restore was needed. It just caused me 2 days of headaches that resulted in having to assist in validation of data that would not have been required had the restore been functioning.

Thursday, January 21, 2010

Hyperion - lets focus on infrastructure, client stories....

I've just had 3 clients experience extremely painful outtages due to the ignoring of their server environment.

Client 1: When I last left this client, microsoft mom alerts were running so anytime the server was bounced we (I) would receive an email. This is one way to keep your IT honest. I can't tell you the number of times that IT bounces and touches the Essbase server when they shouldn't be. All produciton outtages should be scheduled, no exceptions. In addition, alerts were running that would notify the business admins of Essbase (business side of the fence) when the hard drives approached 80% capacity. The Essbase hard drive filled up and there was a crash. I got contacted a day later. 3 of 15 apps would not start. I arranged to remotely restore the files from tape if they could place the tape backups in a folder. Guess what? Before this series of outtages from 3 clients at the same time, I would have guestimated a 60% success rate of recovering from tape backup. Now my educated guess has dropped down to 40%. The files they placed in the folder were missing some important files (app, .ind, .pag to name a few). After 8 weeks, I am still battling with this client to ensure backups will be usable. The problem is that the business users had their original data loads scripted and were able to quickly recover. No lesson has been learned yet. I just attempted another restore and their IT still could not replace all the necessary files. This is in an environment where I am giving IT lights out of Essbase for 5 hours a night.