Extreme scoping: What iterative development really means

by Larissa Moss

May 2008

A data warehouse (DW) is not a DW unless it has two components: data management (cleaning up the data chaos) and data delivery (building BI applications). Unfortunately, most companies put more emphasis on data delivery than on getting their data under control.

There are several reasons for this. Managing data across the enterprise is more difficult and takes more effort, coordination, and resources than delivering silo business intelligence (BI) point solutions. Second, rationalizing redundant and inconsistent data is tedious work and is a lot less impressive on a resume than boasting one’s knowledge of new BI tools and techniques. But the main reason is that business users are pressing for quick deliverables (90 days or less) while constantly “refining” their requirements, and DW teams scramble to oblige. The end result is often a collection of independent (non-integrated) data marts – always with the promise to consolidate them later, which regrettably rarely happens. Does this dilemma have another solution? Can you “have your cake” (address data management) "and eat it too" (produce quick deliverables)? Try “extreme scoping.” It is an iterative development method that uses the “software release” concept. It requires a “self-organizing” team structure, hands-on end user participation, a spiral DW development methodology, and a small cultural adjustment.

Software release concept

The “big bang” development approach for building data warehouses has long been abandoned. It is now generally accepted that data warehouses evolve and grow over time, adding one BI application after another to its environment. What has not yet been generally accepted is that we should not be developing BI applications in a “big bang” manner either. That means that we should not attempt to deliver a fully functioning BI application at the end of a project. Why? Because the data scope of DW projects is usually too large and the project schedule is usually too short to allow time for all the necessary tasks of enterprise-wide data management. It could take 90 days just to analyze and standardize several dozen data elements if their names (synonyms and homonyms), definitions (semantic meaning), domains (set or range of valid values), and business rules (data and process rules) are in disagreement or in dispute among several departments and business users. At that time, project teams often abandon any enterprise-wide data management activities just to get the functionality of the BI application completed on schedule.
A much better approach is to divide the BI application into multiple 30-45 day deliverables; that is, into multiple 30-45 day software releases. Each release is its own project. Each release delivers a fully standardized and inventoried subset of requested data elements with a subset of requested functionality. It may take several releases to deliver a fully functioning BI application. However, this approach allows the DW team to work on enterprise-wide data management tasks, while it gives business users the opportunity to hone their requirements during the development process.

Self-organizing team structure

A software release approach requires a more dynamic team structure than usual. Traditional project teams can be relatively large; teams of ten or more members with specialized skills are not unusual. They depend on the project manager to coordinate and assign tasks to individual project team members and to make project-related decisions. The team members work separately on their assigned tasks, and, once a week, the entire team meets for a status review. Working in this way is much too slow. Communicating and coordinating activities among all the team members is inefficient and error-prone; not to mention the lack of synergy and knowledge transfer among team members.
A much better team structure is a small SWAT team of four to five generalists. This is a self-organizing team, which means that they assign tasks to each other, they coordinate and review each other’s work artifacts, they collaborate on project activities, they make project-related decisions (together), and they take on another team member’s tasks when needed. In order to do all that their combined skill set must include the following expertise: very strong project management skills, senior technical knowledge, data administration proficiency (not to be confused with database administration or database design), and business representation (someone with decision-making authority from the business community).

Hands-on business user participation

A prerequisite for software releases and dynamic team structures is the participation of business people on project activities. This requires much more involvement by the business users than usual. Traditionally, business people participate in requirements-gathering interviews, JAD sessions, project reviews, and user acceptance testing. Other than that, the technicians do all the development work with no involvement from the business people. Besides creating an “US versus THEM” atmosphere, this limited degree of involvement forces IT to make assumptions that lead to unsatisfactory results. This is bad enough on projects with well-defined scopes and deliverables, but on DW projects, where scopes and deliverables are often a moving target, it can be catastrophic. Just think of how many times we hear that DW projects are late, over budget, too costly, and too complicated, and that the deliverables don’t meet business users’ expectations.
The best way to remedy this situation is for business people to be active, full-time members of DW project teams. That means that they actively contribute to project activities, such as participating in project planning, analyzing the source data, modeling the data from a business perspective, naming and defining the data, creating the business rules and domain values, determining data cleansing logic, writing test cases for integration and regression testing, evaluating test results, and so on. Naturally, business people would not be expected to perform these activities in isolation but would collaborate with other project team members.

Spiral DW development methodology

It is a well-known fact that traditional system development lifecycle methodologies don’t work on DW projects. These old “waterfall” methodologies work well for producing stand-alone products, like silo systems, but they don’t support iterative development of a DW, and they don’t address enterprise-wide data management. On top of that, waterfall methodologies are anything but agile.
In contrast, spiral DW methodologies have flexible entry and exit points, which are designed to support iterative development. These new methodologies contain tasks specific to data warehousing and enterprise data management, including tasks to be performed by business people. They also support parallel development tracks, such as ETL, front-end application, metadata solution, and so on. The most significant benefit of agile methodologies is that they provide the necessary framework for software releases, self-organizing team structures, and hands-on user participation.

Small cultural adjustment

“Extreme scoping” works. It has been tested and proven by many DW teams, as well as by operational development teams who subscribe to agile development and extreme programming (XP). It is not a difficult technique to adopt, but it does require a small (in some companies, not so small) cultural adjustment. DW project teams must learn about agile methods and XP techniques, and then experiment with new team structures, shifted roles and responsibilities, data administration principles, and a new development approach based on software releases. Business people must abandon their “hands-off” attitude and take ownership of the BI program. Ownership is expressed in the degree of interest and involvement in projects. Looking back at ERP conversions a few years ago, it would be difficult to find an ERP project without active, full-time business user participation in actual ERP project activities, from data mapping to testing. DW projects that suffer from the syndrome where business people “don’t have time” to participate (and find it incredulous to even be asked), actually suffer from executive lip service. These projects do not have true executive sponsorship for enterprise data management, which is an integral part of data warehousing. They may be condemned to appease their business people with silo BI point solutions, at least for the time being. But for those companies that wish to tackle data management in addition to data delivery, the cultural adjustment should not be that difficult.

Conclusion

DW project teams cannot afford to keep biting off more than they can chew (without choking). They must gently reeducate and redirect business users who insist on “minimum required” scopes (complete functionality and the data to support it) in ever shorter timeframes. “Extreme scoping” is an effective vehicle to do just that. Business people will continue to get “quick hits” while learning about the complexities and the benefits of building an integrated BI environment. They will have a chance to “live” the DW projects as actively participating project team members and to experience the same challenges and triumphs that DW teams have experienced in the past. In addition, this new development approach will go a long way toward obliterating the current rift between business and IT.