HPT Administration: Findings from a typical Hyperion engagement

Below are some findings from a typical customer running Hyperion Essbase. I didn't have the energy to write a more exhaustive list for Hyperion Planning administrators.

Manual steps involved in production processing
Business folks are great at documenting procedures (ie. Writing down the steps to do something). The problem is that since they are business folks, they are used to working harder to get results. I’ve had clients that would on a daily basis, come in, load a file, and calc a cube by hand. The good one’s would then manually tie out the data and then send out an email saying the cube was processed. This was usually completed by around 11:00am. After this they would start their real job. There are 2 different scripting tools available to automate these tasks, esscommand and maxl.

Tieout of hyperion to other systems
There are quite a few customers out there can can’t explain why Essbase doesn’t tie to their source system. They even stop tying it out because it’s always off by x, so why bother. You have to start out on day 1 with a system that does tie. Even if Hyperion does tie out, the validation process is still a manual one. This manual task can be automated into an email being sent out daily after each cube is processed.

Error logs are not empty
A common mistake is that when someone looks at a error log, they see the same error repeated many, many times. “We’ve always had those errors. It’s too much trouble getting rid of them.” Well, by default Essbase only logs 300 errors. So, if you have expected errors that number 200 or > 300, some real errors (like a new customer with activity) never come to light. This is an issue that I’ve seen at numerous clients (>10 through the years).

Maintaining unwieldy hierarchies in the outline manually
While the graphical nature of managing an outline is pleasing to the eye, it’s not the best way to maintain structures in a production environment when multiple cubes are deployed. One option is to create a hierarchy file that is then passed to each cube that should be modified. If you need a little more control you can store the hierarchy inside a relational database and feed it into Hyperion Planning or Essbase via a load rule with sql interface or Integration server. Additional options include writing custom API code, and even purchasing after market tools that perform this task (Star Analytics). There are probably a dozen more options that Oracle offers but I’m not well versed in all the names out there.

Running applications that are not productionized
While it’s a little tricky developing a new application from scratch, it’s not too difficult taking over the administration of a Hyperion Application. Just write down the steps then perform them. Sounds easy, right? Well what typically happens for Essbase, is that once an application is in production, additional copies are made that are just a little bit different. These models then come online. Now the maintenance of your one app has doubled. The change to the hierarchy that took 20 minutes now takes 40 minutes. As cubes multiply this issue becomes a greater burden. If there are changes that must be made to multiple databases consider building these via a load rule so you can make the same change to multiple databases easily. To productionize an app means to standardize the components be they load rules or reports and also to automate as much as possible. I have seen a customer have 90 load rules for 120 different data sources. Now imagine copying this app 4 times. That’s what they did.

Not having backups
In the cases where backups were being performed, the resulting backup would be rendered useless over 50% of the time. Systems I work on will have a 7 day history of data for the business folks to use to recover on their own. Over half of all customers I’ve visited that were running Essbase / Hyperin Planning had improper backups. Recently I came across a customer where they were intentionally not backing up Essbase because they said the data existed in their data mart. While that may be true, the Essbase install was on Unix. They were not qualified to reinstall Essbase on their Unix so the backups would in fact be very important. They were very fortunate in that they never experienced a hardware failure that would have exposed this situation.

No development environment or even a development planning application on production
It’s scary working without a safety net. That’s was working without a development server is like. Even worse is running Hyperion Planning without a copy of the app to test things out on. This is a recipe for disaster. There’s no excuse to not have an extra copy of an Essbase application for tinkering purposes. Another app on production is easy but not best practice. It’s better to have another server whereby development work will not impact production.

No change control process
As Essbase administrators, we usually don’t have to deal with change control (IT speak for taking a long time to get a change into production but making sure nothing bad will happen when implemented), but there is a responsibility to make sure we’ve tested things before placing them into production. After a few screwups, you’ll lose this right, and possibly the admin will start working for IT with a whole bunch of red tape and paperwork.

No remote access to the network
Hyperion software is much more than just an application. There are steps and processes that can take hours to run. Most of these tasks must be performed in the off hours. To maximize user uptime (and satisfaction), it is absolutely essential that Hyperion admins have remote access. This means that they should be able to do everything from home that they can do at work. All too often, IT considers Hyperion administrators as a typical business user. Hyperion software is client server based, and the user community can range in the hundreds.

No one knows how to open a ticket with Oracle support
This can be a 3 day process getting setup for the first time. You need your SSID number and no one can ever find this number. Oracle support is one of many tools you have to help troubleshoot issues. More than half the time, issues that are encountered have nothing to do with a bug in the software but rather a misunderstanding about how a feature works. The support site is metalink3.oracle.com. Your experience with the support group will vary. If your issue is clearly defined in a way that they can understand, you might get lucky, and they’ll quickly inform you that the feature is not designed to work that way, it will be an enhancement request, they can’t reproduce the error, or it might be a bug.

Insufficient access on desktop computer and only 1 pc for the admin
In too many cases IT has gotten overzealous in controlling rights on pc’s. Okay, maybe not overzealous but Hyperion Admins are not a business user. They are themselves a system administrator of sorts and they need certain access and tools to do their job properly. Many of my clients can not even open a dos window because their pc’s are so locked down. I propose that all Hyperion admins have at least 2 machines. They don’t need to be the latest and greatest but given the nature of the administration tasks, rebooting a desktop machine is very, very inconvenient. When I’m running a project if it’s Essbase I request 2 client machines for myself. If it’s planning for my team I request about 5 client machines in addition to using our laptops. It’s best for a customer to provide these machines because consultants in their spare time can perform software compatibility testing while they are working. This task does not fall within the duties of consultants but they can greatly assist in showing how client installs work and help work through issues. Consultants usually get admin rights. The business users and admins very rarely have admin rights. Some of the Hyperion administration software does not work well if it’s not run as admin. These are issues that would take an IT group a long time to resolve on their own.

No access to the server logs / app folder
While arguments can be made that the Hyperion Administrator doesn’t have to have access to the server, who is going to perform the following tasks for them:
-monitor server for .xcp files in the bin directory
-monitor server for .xcp files in the app directories
-schedule batch jobs to run nightly
-analyze the app logs for long retrieval time.
-perform log parsing periodically

I was at one client that had over 300 .xcp app files on their Essbase server. No one ever knew the server was crashing. No one knew to look for these files, IT had the access but not the knowledge and the business folks did not have either the access or the knowledge.

While there are many other items that are important, I find that when I'm at various client sites I end up addressing these issues time and time again.

hj

HPT Implementation:web form based security is not sufficient...

I've just come across an instance in a hyperion planning system 9.3.1 implementaion where a few accounts were set to read only by using a row definition setting in a planning web form. Having this feature available to us as developers sounds pretty handy until you think about the consequences. This will lock down the cell for web forms and smartview if used in the web form mode. If however your users will have access to smartview adhoc (if they have smartview they have this) or the excel addin with "essbase write" access provisioned to them, this cell is wide open for users to change the values.

Why might you want a cell to be read only? You could have a business rule that populates the value based on drivers or some other metric, this account could be pre-seeded, or only admin personnel should be able to update this account.

What are some workarounds that are a better best practice?
-in planning under dimensions, give all users read access only to this specific account. (have a group created for all users, preferably 2 groups, all users read and all users write)
-in essbase / planning, make a new account that is a dynamic calc that merely points to this member. By having the property set to dynamic calc, planning will automatically set the cell to read only.

This is similar to a project (large datawarehouse) where everyone thought that securing access to specific web analysis forms and financial reporting reports would limit users access to the entire essbase cube data. While they did not allow access to the excel addin, they did allow users to create their own web analysis reports. I mentioned that while our 85 security filters did limit access to data based on the entitiy dimension it did not eliminate the users from seeing the data that made up these reports. The customer was just overwhelmed with the project and said it was good enough.

hj

HPT Infrastructure: Top 5 wastes of an infrastructure consultant’s time

Top 5 wastes of an infrastructure consultant’s time

There are many reasons a project can go over-budget. In the world of Hyperion Infrastructure the most common ones can be avoided. If you take the time to completely understand all the tasks needed of them before your infrastructure consultant arrived, you would save a great deal of money. Or, perhaps, be able to spend that time learning about your environment and its care and feeding.

1) Download the software before the consultant gets there.

When in doubt, download it. If you download something that you don’t need it is ok, but if you don’t get everything you pay for a consultant to watch a status-bar!

2) Create your databases and give the proper permissions.

Normally you'll receive a list of Databases (or user/schemas in the case of oracle) along with a list of permission. Make sure you follow this, in the case of FDM there are special requirements that need to be met.

3) Order the servers AS SOON AS POSSIBLE.

I know that no-one wants to have capital expenses depreciating on their cost center any longer than they have to. But please remember that your IT department is likely to be very busy with everyone else’s requests. Furthermore next day shipping doesn’t mean that the vender has the servers in stock or that they are even built, it just means when they get around to shipping it, you’ll get it the day after. I've honestly been stuck waiting (and billing) until Wednesday of the *SECOND WEEK* for equipment to be ready. My record before that was noon on a Friday (in that case we were merely waiting for a network cable to be plugged in). In any case I cannot stress enough ORDER YOUR EQUIPMENT EARLY PLEASE!

4) Ensure that you have the appropriate meetings with your infrastructure consultant ahead of time.

Only you can know what’s appropriate. I’ve had simple customers want 10 meetings about something simple (waste of your time) and I’ve had extremely complex customers with complicated (and battling) political factions have only one short meeting. If you have 1 it guy that can do it all, have one good meeting. If you have a complicated it department (even if the environment isn’t) get all the parties involved. It's NOT going to help anything if we finish an installation with automatic deployment, get ready to go live, and find out that your IT department won’t support you because it has to be manually deployed with custom ports!!! This HAS HAPPENED more than once!

5) Ensure that your IT department is on-board with the installation.

I've been at customers who ticked their IT department off so badly that they wouldn’t provide simple DBA support. I can handle this if given the keys to the kingdom, and most likely if your IT department behaves like this your environment will be better off with me doing everything anyways. However, I will one day leave. While I very much appreciate all the after hours billing, you won’t appreciate waiting for me to be done for the day with my present client so that I can put your production environment back together after someone mistakenly trashed it!

6) Get involved with the installation (you didn’t think that I would actually stop at 5 did you?)

The preferred method of installation is for me to build the first environment so that your applications consultants (or internal people) can get working/testing ASAP (remember the applications consultants are billing you too). This is the fastest way to bring an environment up. But on the second installation PLEASE sit with me in a conference room (bring a projector please) and we'll do the installation with you driving. Don’t get me wrong, I’d love to bill you every single day to Webex in and handle your minutia, but it's far more efficient if you understand your environment and can handle these things by yourself. I suggest having at least one IT person in the room and your Hyperion support person as well. This way there is always at least 2 people who understand exactly what Hyperion is doing, and can come to the rescue at quarter end when the CFO is shaking his/her fist at you because they don’t have the numbers to report.

pp