As a Hyperion administrator what skills should I be learning....

In working with Hyperion for the past 15+ years, I've seen a lot of clients and a lot of system administrators. I've even spent significant time trying to backfill my administrator role at various clients as well. You might be surprised at how many folks are out there that say they can run Essbase but they have never written a calc script. There is a great difference between maintaining a Hyperion system and developing a Hyperion system from scratch. In order to make yourself more valuable you need to really understand what's going on under the covers. I've even advised a few business analysts that focusing their efforts on really understanding and learning Essbase would result in a bigger bang for the buck than starting the process of gaining an MBA. In both cases they were able to double their salary in the course of 3 years by learning Hyperion and switching employers without incurring +10,000 in student debt. Usually folks involved in Hyperion at a client don't have any time to goto night school anyway. The right move I believe is for a company to figure out a way to keep this talent. So much business knowledge is learned from creating these systems that it is a real shame for it to go out the door.

Anyway, some things that you should do to make yourself more valuable in the Hyperion arena include:
-fully document your Hyperion Systems and identify / train your backup resource. Go download the outline extractor and export all your outlines to audit all the member formulas in your databases. Many administrators out there have a really tough time taking vacation and an even tougher time taking time off for training as the system seems to stop running when "Bob" is not here.
-devise an issues log to help track all the fires you are putting out. When it comes time for yourself appraisal, you need to have proof of the amount of work your are performing. Saying you are just solving the needs ("keeping them happy") of finance is not tangible. Despite your best efforts it's not possible to keep users happy. By keeping a log you can better explain the amount of work you are performing and this can also give you credibility when you come to the realization that you need to get another head hired to help you out. Logs also help you prioritize and keep track of all the issues. "Prioritize" - the secret that allows IT folks to go home at 5:00pm every night when business folks are always getting stuck working late. It seems on the IT side that resources are always limited while on the business side there is no cap on how hard people work to complete deadlines.....
-learn Essbase VBA to figure out how to automate bulk loading of spreadsheets into Essbase and to streamline other business admin tasks
-see what else VBA can help out with. You can easily create a menu based interface in Excel for report generation.
-understand when web based reports should be used instead of Excel for reporting creation and distribution
-recreating your applications from scratch in development. In so doing, understand the decisions that were made for your application and challenge yourself on how to improve the model.
-reviewing your cube update process and automating every step possible (stop touching the outline by hand when there are more than 1 to update, make a dim build file and run it through batch dimension builds)
-review your data loading process and try to load your data from a relational data store (instead of loading from files, get these files loaded to some kind of sql data store) and perform your etl (scrubbing of your file, adding prefixes, concatenations, etc) in the sql and not the load rule (load rules are disposable and sometimes get corrupted when upgrading. These also are not self documenting).
-learn what esscommand and maxl are (command line based scripting used for automating Essbase administration)
-learn how to perform allocations with calc scripts (spread a value stored at the top of 3 dimensions down to level 0 based on a populated account)
-if you have any calc scripts running, make sure you have validation workbooks to audit every calc that takes place in your models
-review your scripts (might be Essbase only, not sure about HFM) and review the benefits of inserting substitution variables to make your scripts more dynamic
-if you have not created an aggregate storage database do so. If you don't know what an aggregate storage database is then read up on it.
-if using Hyperion Planning review your business rules and web forms to ensure the user experience is as streamlined as possible (not a list of 12 steps of business rules that must be run in order). A user should not need to know how to run more than 1 or 2 business rules. These should all be attached to web forms that run upon save without requiring the end users launching these in the tools->business rules menu.
-also, if you're using Hyperion Planning, go through the effort of configurating and using the application copy utility in System 9 or learn how to use the life cycle management tool of system 11.
-get 2 virtual machines setup, one for windows xp (easy one) and another for windows server 2003. A virtual machine allows you to run different versions of software on your desktop. These are like mini machines that run on your desktop. You can now switch between running Essbase 6.5, 7.x, 9.3.3, and 11.1.3 on your pc. You'll actually need to "fire up" (a vm takes up about 10gig of space on your hard drive and takes a few minutes to start) your vm and probably not run them concurrently. But, VM's can be very valuable in testing compatibility of Hyperion with different versions of IE and Excel.
-install hyperion software on these vm's. There is plenty of help out there on the internet how to install system 9 and more recently system 11. You can install Essbase System 9 on a Windows XP client vm (shared services, essbase, bi+ (workspace).
-learn ODI as this is the replacement for HAL. While this will be a necessity for Planning, ODI can also come in handy for working with Essbase.

ps. If you think you're ready to branch out into the traveling consultant role let me know. I know some pretty good consulting firms to work for. If you think you would rather go independent drop me a line and I'll try to talk you out of it. Some folks have a hard time saving their acorns for the winter months.

hj

HPT Administration: Findings from a typical Hyperion engagement

Below are some findings from a typical customer running Hyperion Essbase. I didn't have the energy to write a more exhaustive list for Hyperion Planning administrators.

Manual steps involved in production processing
Business folks are great at documenting procedures (ie. Writing down the steps to do something). The problem is that since they are business folks, they are used to working harder to get results. I’ve had clients that would on a daily basis, come in, load a file, and calc a cube by hand. The good one’s would then manually tie out the data and then send out an email saying the cube was processed. This was usually completed by around 11:00am. After this they would start their real job. There are 2 different scripting tools available to automate these tasks, esscommand and maxl.

Tieout of hyperion to other systems
There are quite a few customers out there can can’t explain why Essbase doesn’t tie to their source system. They even stop tying it out because it’s always off by x, so why bother. You have to start out on day 1 with a system that does tie. Even if Hyperion does tie out, the validation process is still a manual one. This manual task can be automated into an email being sent out daily after each cube is processed.

Error logs are not empty
A common mistake is that when someone looks at a error log, they see the same error repeated many, many times. “We’ve always had those errors. It’s too much trouble getting rid of them.” Well, by default Essbase only logs 300 errors. So, if you have expected errors that number 200 or > 300, some real errors (like a new customer with activity) never come to light. This is an issue that I’ve seen at numerous clients (>10 through the years).

Maintaining unwieldy hierarchies in the outline manually
While the graphical nature of managing an outline is pleasing to the eye, it’s not the best way to maintain structures in a production environment when multiple cubes are deployed. One option is to create a hierarchy file that is then passed to each cube that should be modified. If you need a little more control you can store the hierarchy inside a relational database and feed it into Hyperion Planning or Essbase via a load rule with sql interface or Integration server. Additional options include writing custom API code, and even purchasing after market tools that perform this task (Star Analytics). There are probably a dozen more options that Oracle offers but I’m not well versed in all the names out there.

Running applications that are not productionized
While it’s a little tricky developing a new application from scratch, it’s not too difficult taking over the administration of a Hyperion Application. Just write down the steps then perform them. Sounds easy, right? Well what typically happens for Essbase, is that once an application is in production, additional copies are made that are just a little bit different. These models then come online. Now the maintenance of your one app has doubled. The change to the hierarchy that took 20 minutes now takes 40 minutes. As cubes multiply this issue becomes a greater burden. If there are changes that must be made to multiple databases consider building these via a load rule so you can make the same change to multiple databases easily. To productionize an app means to standardize the components be they load rules or reports and also to automate as much as possible. I have seen a customer have 90 load rules for 120 different data sources. Now imagine copying this app 4 times. That’s what they did.

Not having backups
In the cases where backups were being performed, the resulting backup would be rendered useless over 50% of the time. Systems I work on will have a 7 day history of data for the business folks to use to recover on their own. Over half of all customers I’ve visited that were running Essbase / Hyperin Planning had improper backups. Recently I came across a customer where they were intentionally not backing up Essbase because they said the data existed in their data mart. While that may be true, the Essbase install was on Unix. They were not qualified to reinstall Essbase on their Unix so the backups would in fact be very important. They were very fortunate in that they never experienced a hardware failure that would have exposed this situation.

No development environment or even a development planning application on production
It’s scary working without a safety net. That’s was working without a development server is like. Even worse is running Hyperion Planning without a copy of the app to test things out on. This is a recipe for disaster. There’s no excuse to not have an extra copy of an Essbase application for tinkering purposes. Another app on production is easy but not best practice. It’s better to have another server whereby development work will not impact production.

No change control process
As Essbase administrators, we usually don’t have to deal with change control (IT speak for taking a long time to get a change into production but making sure nothing bad will happen when implemented), but there is a responsibility to make sure we’ve tested things before placing them into production. After a few screwups, you’ll lose this right, and possibly the admin will start working for IT with a whole bunch of red tape and paperwork.

No remote access to the network
Hyperion software is much more than just an application. There are steps and processes that can take hours to run. Most of these tasks must be performed in the off hours. To maximize user uptime (and satisfaction), it is absolutely essential that Hyperion admins have remote access. This means that they should be able to do everything from home that they can do at work. All too often, IT considers Hyperion administrators as a typical business user. Hyperion software is client server based, and the user community can range in the hundreds.

No one knows how to open a ticket with Oracle support
This can be a 3 day process getting setup for the first time. You need your SSID number and no one can ever find this number. Oracle support is one of many tools you have to help troubleshoot issues. More than half the time, issues that are encountered have nothing to do with a bug in the software but rather a misunderstanding about how a feature works. The support site is metalink3.oracle.com. Your experience with the support group will vary. If your issue is clearly defined in a way that they can understand, you might get lucky, and they’ll quickly inform you that the feature is not designed to work that way, it will be an enhancement request, they can’t reproduce the error, or it might be a bug.

Insufficient access on desktop computer and only 1 pc for the admin
In too many cases IT has gotten overzealous in controlling rights on pc’s. Okay, maybe not overzealous but Hyperion Admins are not a business user. They are themselves a system administrator of sorts and they need certain access and tools to do their job properly. Many of my clients can not even open a dos window because their pc’s are so locked down. I propose that all Hyperion admins have at least 2 machines. They don’t need to be the latest and greatest but given the nature of the administration tasks, rebooting a desktop machine is very, very inconvenient. When I’m running a project if it’s Essbase I request 2 client machines for myself. If it’s planning for my team I request about 5 client machines in addition to using our laptops. It’s best for a customer to provide these machines because consultants in their spare time can perform software compatibility testing while they are working. This task does not fall within the duties of consultants but they can greatly assist in showing how client installs work and help work through issues. Consultants usually get admin rights. The business users and admins very rarely have admin rights. Some of the Hyperion administration software does not work well if it’s not run as admin. These are issues that would take an IT group a long time to resolve on their own.

No access to the server logs / app folder
While arguments can be made that the Hyperion Administrator doesn’t have to have access to the server, who is going to perform the following tasks for them:
-monitor server for .xcp files in the bin directory
-monitor server for .xcp files in the app directories
-schedule batch jobs to run nightly
-analyze the app logs for long retrieval time.
-perform log parsing periodically

I was at one client that had over 300 .xcp app files on their Essbase server. No one ever knew the server was crashing. No one knew to look for these files, IT had the access but not the knowledge and the business folks did not have either the access or the knowledge.

While there are many other items that are important, I find that when I'm at various client sites I end up addressing these issues time and time again.

hj