HPT Administration: Findings from a typical Hyperion engagement

Below are some findings from a typical customer running Hyperion Essbase. I didn't have the energy to write a more exhaustive list for Hyperion Planning administrators.

Manual steps involved in production processing
Business folks are great at documenting procedures (ie. Writing down the steps to do something). The problem is that since they are business folks, they are used to working harder to get results. I’ve had clients that would on a daily basis, come in, load a file, and calc a cube by hand. The good one’s would then manually tie out the data and then send out an email saying the cube was processed. This was usually completed by around 11:00am. After this they would start their real job. There are 2 different scripting tools available to automate these tasks, esscommand and maxl.

Tieout of hyperion to other systems
There are quite a few customers out there can can’t explain why Essbase doesn’t tie to their source system. They even stop tying it out because it’s always off by x, so why bother. You have to start out on day 1 with a system that does tie. Even if Hyperion does tie out, the validation process is still a manual one. This manual task can be automated into an email being sent out daily after each cube is processed.

Error logs are not empty
A common mistake is that when someone looks at a error log, they see the same error repeated many, many times. “We’ve always had those errors. It’s too much trouble getting rid of them.” Well, by default Essbase only logs 300 errors. So, if you have expected errors that number 200 or > 300, some real errors (like a new customer with activity) never come to light. This is an issue that I’ve seen at numerous clients (>10 through the years).

Maintaining unwieldy hierarchies in the outline manually
While the graphical nature of managing an outline is pleasing to the eye, it’s not the best way to maintain structures in a production environment when multiple cubes are deployed. One option is to create a hierarchy file that is then passed to each cube that should be modified. If you need a little more control you can store the hierarchy inside a relational database and feed it into Hyperion Planning or Essbase via a load rule with sql interface or Integration server. Additional options include writing custom API code, and even purchasing after market tools that perform this task (Star Analytics). There are probably a dozen more options that Oracle offers but I’m not well versed in all the names out there.

Running applications that are not productionized
While it’s a little tricky developing a new application from scratch, it’s not too difficult taking over the administration of a Hyperion Application. Just write down the steps then perform them. Sounds easy, right? Well what typically happens for Essbase, is that once an application is in production, additional copies are made that are just a little bit different. These models then come online. Now the maintenance of your one app has doubled. The change to the hierarchy that took 20 minutes now takes 40 minutes. As cubes multiply this issue becomes a greater burden. If there are changes that must be made to multiple databases consider building these via a load rule so you can make the same change to multiple databases easily. To productionize an app means to standardize the components be they load rules or reports and also to automate as much as possible. I have seen a customer have 90 load rules for 120 different data sources. Now imagine copying this app 4 times. That’s what they did.

Not having backups
In the cases where backups were being performed, the resulting backup would be rendered useless over 50% of the time. Systems I work on will have a 7 day history of data for the business folks to use to recover on their own. Over half of all customers I’ve visited that were running Essbase / Hyperin Planning had improper backups. Recently I came across a customer where they were intentionally not backing up Essbase because they said the data existed in their data mart. While that may be true, the Essbase install was on Unix. They were not qualified to reinstall Essbase on their Unix so the backups would in fact be very important. They were very fortunate in that they never experienced a hardware failure that would have exposed this situation.

No development environment or even a development planning application on production
It’s scary working without a safety net. That’s was working without a development server is like. Even worse is running Hyperion Planning without a copy of the app to test things out on. This is a recipe for disaster. There’s no excuse to not have an extra copy of an Essbase application for tinkering purposes. Another app on production is easy but not best practice. It’s better to have another server whereby development work will not impact production.

No change control process
As Essbase administrators, we usually don’t have to deal with change control (IT speak for taking a long time to get a change into production but making sure nothing bad will happen when implemented), but there is a responsibility to make sure we’ve tested things before placing them into production. After a few screwups, you’ll lose this right, and possibly the admin will start working for IT with a whole bunch of red tape and paperwork.

No remote access to the network
Hyperion software is much more than just an application. There are steps and processes that can take hours to run. Most of these tasks must be performed in the off hours. To maximize user uptime (and satisfaction), it is absolutely essential that Hyperion admins have remote access. This means that they should be able to do everything from home that they can do at work. All too often, IT considers Hyperion administrators as a typical business user. Hyperion software is client server based, and the user community can range in the hundreds.

No one knows how to open a ticket with Oracle support
This can be a 3 day process getting setup for the first time. You need your SSID number and no one can ever find this number. Oracle support is one of many tools you have to help troubleshoot issues. More than half the time, issues that are encountered have nothing to do with a bug in the software but rather a misunderstanding about how a feature works. The support site is metalink3.oracle.com. Your experience with the support group will vary. If your issue is clearly defined in a way that they can understand, you might get lucky, and they’ll quickly inform you that the feature is not designed to work that way, it will be an enhancement request, they can’t reproduce the error, or it might be a bug.

Insufficient access on desktop computer and only 1 pc for the admin
In too many cases IT has gotten overzealous in controlling rights on pc’s. Okay, maybe not overzealous but Hyperion Admins are not a business user. They are themselves a system administrator of sorts and they need certain access and tools to do their job properly. Many of my clients can not even open a dos window because their pc’s are so locked down. I propose that all Hyperion admins have at least 2 machines. They don’t need to be the latest and greatest but given the nature of the administration tasks, rebooting a desktop machine is very, very inconvenient. When I’m running a project if it’s Essbase I request 2 client machines for myself. If it’s planning for my team I request about 5 client machines in addition to using our laptops. It’s best for a customer to provide these machines because consultants in their spare time can perform software compatibility testing while they are working. This task does not fall within the duties of consultants but they can greatly assist in showing how client installs work and help work through issues. Consultants usually get admin rights. The business users and admins very rarely have admin rights. Some of the Hyperion administration software does not work well if it’s not run as admin. These are issues that would take an IT group a long time to resolve on their own.

No access to the server logs / app folder
While arguments can be made that the Hyperion Administrator doesn’t have to have access to the server, who is going to perform the following tasks for them:
-monitor server for .xcp files in the bin directory
-monitor server for .xcp files in the app directories
-schedule batch jobs to run nightly
-analyze the app logs for long retrieval time.
-perform log parsing periodically

I was at one client that had over 300 .xcp app files on their Essbase server. No one ever knew the server was crashing. No one knew to look for these files, IT had the access but not the knowledge and the business folks did not have either the access or the knowledge.

While there are many other items that are important, I find that when I'm at various client sites I end up addressing these issues time and time again.

hj