Business Intelligence: From Theory to Reality (Part Two)

by Shaku Atre

August 2003

So what does the implementation of a BI decision-support environment look like? Here's an up-close view of what we call the BI Roadmap(see reference). It's composed of six stages; these stages, in turn, take 16 steps. Together, they offer a roadmap for taking BI from Theory to Reality.

These development steps need not be performed in sequence. In fact, most likely a number of them will be performed in parallel. However, since there is a natural order of progression from one engineering stage to another, certain dependencies exist between some of the development steps. In the following illustration, steps stacked atop each other can be performed concurrently. But steps following each other are performed relatively linearly and with less overlap, due to their dependencies.

Here's a closer look at the 16 steps of a successful BI implementation and what they involve:

STAGE I: JUSTIFICATION

Step 1: Business-Case Assessment

The business problem or business opportunity is defined. A BI solution is proposed. Each BI application release is independently cost-justified, and each should either solve a business problem or take advantage of a business opportunity.

STAGE II: PLANNING

Step 2: Enterprise Infrastructure

Since BI applications are cross-organizational initiatives, an enterprise infrastructure must either exist or be developed. This is done while the BI applications are either being considered or being built.

An enterprise infrastructure has two components:

• Technical Infrastructure: This includes hardware, software, middleware, database management systems, operating system, network components, meta data repository management system, and utilities.
• Non-Technical Infrastructure: This includes meta data standards, data-naming standards, enterprise logical data model, methodologies, guidelines, testing procedures, change-control processes, issues-management procedures, and dispute-resolution procedures.

Step 3: Project Planning

BI decision-support projects are dynamic. All sorts of changes – whether scope, staff, budget, technology, business representatives, or sponsors -- can severely affect the project's success. Therefore, detailed project planning is required. Later, the project's progress must be watched closely and reported.

STAGE III: BUSINESS ANALYSIS

Step 4: Project Delivery Requirements

Managing scope is difficult but vital. The desire to have everything instantly is natural. But the requirements for each deliverable must be carefully defined. Expect these requirements to change throughout the development cycle as more is learned about both the possibilities and the limitations of BI technology.

Step 5: Data Analysis

The quality of the source data can make or break a BI project. Bad habits developed over decades are difficult to break, and the damage resulting from these bad habits are expensive, time-consuming, and tedious to correct. Also, in the past, data analysis was confined to the view of a single line of business; it was never consolidated or reconciled with other views in the organization.

Step 6: Application Prototyping

Analysis for the functional deliverables – formerly known as system analysis -- is best done through prototyping. In this way, it can be combined with application design. Today there are new tools and programming languages that enable developers to prove or disprove a concept or idea in relatively little time. Prototyping also lets the business community see the potential -- and limits -- of the technology. This, in turn, lets them adjust both their delivery requirements and expectations.

Step 7: Meta Data Repository Analysis

More tools translate into having more technical meta data and more business meta data. The technical meta data needs to be mapped to the business meta data; then, all meta data must be stored in a meta data repository. Repositories can be either licensed or built. Either way, the meta-data requirements should be documented.

STAGE IV: DESIGN

Step 8: Database Design

Database design schemas must be built to match the information-access needs of business users. One or more BI target databases will be used to store the business data in either detailed or aggregated form, depending on the users' requirements.

Step 9: Extract/Transform/Load (ETL) Design

Four reasons make finishing the ETL process on time a challenge for most organizations. First, the process is complicated. Second, the work is not very glamorous. Third, the processing windows – that is, the time available in which to do the work -- are typically small. Fourth, the poor quality of the source data usually requires a large amount of time to run the transformation and cleansing programs.

Step 10: Meta Data Repository Design

A meta data repository management system from a third-party vendor will most likely need to be enhanced. Most organizations need features that, while documented on the logical meta model, are not provided by the product's vendor. If, on the other hand, the meta data repository management system is being built in-house, then the development team must decide whether to make the meta data repository database design entity-relationship based or object-oriented. Either way, the design will need to meet the requirements of the logical meta model.

STAGE V: CONSTRUCTION

Step 11: Extract/Transform/Load (ETL) Development

Many tools are available for the ETL process -- some sophisticated, others quite simple. But depending on the data-cleansing and data-transformation requirements developed during the Data Analysis and ETL Design steps, an ETL tool may not be the best solution. It might be necessary to implement manual processing. Also, development teams frequently must pre-process the data manually and write extensions to supplement the ETL tool's capabilities.

Step 12: Application Development

Once the prototyping effort is completed, the team can begin to develop the access and analysis application. This may be a simple matter of finalizing an operational prototype. Or it may be more involved, using more robust access and analysis tools. Either way, the front-end application development activities are usually performed in parallel with the activities of back-end ETL development and meta data repository development.

Step 13: Data Mining

This is how organizations can use their BI decision-support environment to the fullest. BI applications are often limited to pre-written reports, some of which are not even new types of reports, but simply replacements of old reports. However, the real payback comes from the BI hidden in the organization’s data. This can be discovered only with data-mining tools.

Step 14: Meta Data Repository Management System Development

If the decision is made to build a meta data repository management system rather than to license (buy) one, a separate team is usually charged with the development process. This becomes a sizable sub-project in the overall BI project.

STAGE VI: DEPLOYMENT

Step 15: Implementation

Once all the components of the BI application are thoroughly tested, the databases and applications are rolled out. Training is scheduled for the business staff and other users of the BI application and meta data repository. Then the support functions are initiated; this includes the functions of a help desk, maintenance of the BI target databases, scheduling and running ETL batch jobs, performance monitoring, and database tuning.

Step 16: Release Evaluation

It's vital to learn lessons from previous projects. Missed deadlines, cost overruns, disputes, and their resolutions should be noted and their causes examined. Adjustments to the processes should then be made before the next release begins. Any tools, techniques, guidelines, and processes that proved unhelpful should be reevaluated and adjusted, possibly even discarded.

It's been said that while a paper airplane may be constructed without a plan, a jet aircraft cannot. By the same token, a standalone computer system can be designed, built, and implemented without formal guidelines, but a cross-functional BI system cannot. The BI Roadmap provides that set of guidelines. It sets a foundation on top of which a BI system can grow and evolve. It also provides a clear path for taking BI from theory to reality.