Sometimes it seems like going over budget on a data warehouse project is an unwritten rule. But very often, there are some simple ways to help avoid making a budget that is doomed to be overrun. By budgeting correctly at the start, and managing cost, the project manager can deliver the benefits that the business was expecting, in the time frame and budget envelope agreed to. Here are a few tips that I’ve found through experience to help;

1) Establish scope clearly before you establish budget. This sounds obvious but its amazing how often I’ve seen this go wrong. There is no way to know what something will cost if you don’t know what it is. If you leave the scope too broad, user expectations will drive feature and scope creep that will wipe out your original budget. There is only one way to have a clear scope- write it down. And the best kind of scope document is one that is signed by all the various players. Make sure everyone understands what is being delivered, when, and at what cost. If there are gray zones, at least flag them as potential areas where additional costs might arise.

2) Take a look at the data and talk to people who work with it. It seems obvious to say that what’s in the data matters. (Or more accurately what’s not in the data). But its amazing how many budgets are made without doing any serious analysis of the data that is actually in the source systems. Don’t look at the data model, see a field called “customer birthday” and design functionality around it. Problems can range from missing data, to mandatory fields that are filled with garbage because “otherwise the system won’t let us put the order through” to differences in interpretation of definitions between groups within the company. For example if all the Asian sales offices classify customers into the same segments, but have slightly different rules then you will need to “reallocate” this segmentation- even though it is the same field, and the same codes. That reallocation is an ETL job you didn’t budget for, unless you found it in a pre-budget data audit. Often the key here is not to launch a massive data audit, but to find the people who have been trying to make the global reports in spreadsheets- they’ve run up against these sorts of issues, and can probably even offer some solutions. (That well respected analyst in head office who has already painstakingly established a cross mapping for for the customer segments working with his colleagues in Asia, for example). These same folks are also going to be key in terms of adoption of the final solution, since they are often the current source of data for the underground data system the new data warehouse is supposed to be improving on. By involving the key people, you gain credibility, save time, and ensure that the final solution addresses business needs.

3) Do proof of concepts for the tricky bits. In many projects there are areas where something is being tried for the first time (certainly in the more interesting ones)- not surprisingly this is often where the issues arise. One way to help quantify how much effort will really be required by these areas is to do some quick and dirty proof of concepts to validate the basic technical and/or functional aspects of the component or system. Often, if it is early in the project and software and hardware selection has not yet been done, your vendors will be willing to assist with a proof of concept (or even do it themselves) as part of the evaluation process. You can learn important things in this stage- For example, if you are doing a reporting project that needs to deliver 300 reports, by doing a proof of concept of 3-4 reports (even if they are not much more than mock-ups) you can at least get a first estimate of how much development effort is involved, how easy the tool is, and how the software performs. I’ve done proof of concepts that doubled my estimates- because when we actually sat down and used the tool, we realized there was additional data cleanup and hardware required to make it work. Better to know that before you set your budget rather than after. The ideal proof of concept is to actually build a “wire frame” version of the key data marts with real data and let the users try it out. Often, its possible to do this quickly, particularly with a tool that lets you do rapid prototypes of ETL transformations.

4) Involve the project team in the budgeting process As a project manager, the responsibility for the budget is ultimately yours, however by getting input from the experts in the various fields you can greatly improve its accuracy. Don’t guess how long it will take to configure the server- go to the infrastructure team and find out how long it took the last three times they did a similar install and configuration. Be aware of the differences, but for many items by talking to the people who have been there and done that you will get good estimates of the true cost/duration of your project line items.

5) Watch out for Infrastructure and User centric costs The following costs are often overlooked, and end up being part of the cost overrun in the end. Don’t get caught by these classic end of project costs;

• Infrastructure costs- We start to take our IT infrastructure for granted, and assume it will accommodate the new system without modifications- but is the network fast enough? Is there enough data storage? Just assuming that the existing server capacity, storage and bandwidth are sufficient may hide significant costs that the project will need to take on just to make the system operational. I’ve also seen cases where three projects that were launched at the same time all checked that the storage was available- but of course each project did not take into account the other two. The last project to go live ended up having to buy more hard drive capacity- and went over budget.
• Training for end users. There are few systems that don’t require some amount of training for end users. Not taking this into account will provide a rude surprise at the end of the project. Costs here include actual training by third parties, but also travel expenses for trainers or trainees if they are not all based in the same location. Web based training can be a cost effective alternative, but results vary- and its difficult to ensure that “attendees” are really attending.
• Transitional support costs. If the project has a wide scope and involves a large number of users, be aware that there may be an initial spike in help desk calls and PC support as the system goes live. Depending on how your help desk is structured, you might end up paying more to your outsourcing company, or need to hire some temporary employees to help handle the extra calls for a few months.

6) Have a contingency amount included in your budget and do everything you can to keep it till the end There are two big mistakes commonly made regarding budget contingencies- first, not having one at all, and second, using it early in the project. Contingencies are often unpopular, first because by increasing the amount that needs to be approved, they might make it harder to get the green light for the project, and secondly because some view them as a “fudge” or a lack of willingness to do a proper cost estimate. However, ideally a contingency is a realistic number that should be based on the risk inherent in the project. Very small, simple projects might only need a 5% contingency. Large, complex projects that involve multiple departments, hundreds or thousands of users and multiple software and hardware vendors need higher contingencies. There are simply more things that can go wrong, and its not realistic to expect that cost analysis can be accurate enough to foresee everything. The key is not to consume your entire contingency with the first scope change- I’ve seen it happen again and again. Contingency is supposed to be for those unforeseen things- for example, a technical problem with the interaction between two vendors packages requires custom development to provide the original functionality envisioned, or requires more hardware than expected.

By spending the time required up front, a realistic, practical budget can be created- and you can get your project started off on the right foot.