Integrating Business Intelligence into the Enterprise (Part I)

by Mike Ferguson

June 2002

It is fair to say that in many organisations today, the business intelligence (BI) systems have taken root and been in production for some time.

These systems may have been built in-house, purchased as one or more packaged analytic applications or comprise a combination of both in-house built and packaged analytic applications. Typically BI systems are built by using Extract, Transform and Load (ETL) and data re-engineering tools to extract, clean, integrate/match, transform and load data from multiple operational source systems into one or more analytical data stores. Analytic application packages or front end reporting, OLAP and data mining tools are then used to analyse data in these data stores to produce customer intelligence or business performance management intelligence in areas such as marketing, sales, finance (e.g. payment and cash flow analysis, debt management), human resources and supply chain operations.

To date, the vast majority of users of BI systems have typically been management and business analyst power users who analyse data, interpret the intelligence produced and make decisions. However, for many organisations, distributing business intelligence to these ‘first tier’ users has not delivered sufficient business benefit, at least not on the scale they perhaps originally expected. One factor contributing to this lack of benefit is the fact that there are clear signs that business users are struggling with too much competition for their attention, particularly from event driven systems. For example, many people are now taking up to two hours every morning just answering email, never mind looking at business intelligence or conducting some kind of analysis. Another reason is too much information to analyse. The introduction of e-business systems has not helped this situation. Increasing numbers of web visitors performing self-service business transactions and accessing content via various internet connected devices has meant that E-business is generating huge data volumes in web logs, cookies and web forms etc. But there is a recognised bigger problem in that power users are often not close enough to front line business operations to understand how intelligence can best be leveraged in core business processes to deliver maximum effect. For example, customer service personnel or sales force representatives rarely have access to BI about the customers they are dealing with on a day-to-day basis. In the case of contact centre representatives for example, they are normally a cost to a business. However providing them with customer intelligence at the right time could turn them into revenue earning customer focussed professionals. Traditional data warehousing systems put obstacles in the way of such progress. BI tools are an example. Contact centre staff do not have time to use BI tools in such a real-time customer-facing role. If they are to be effective in contributing towards the increase in customer value and provide more personalised customer service, then BI must just be there on the screen available for them to use without the need for them to have to learn new tools or applications. What they require is instant intelligence (e.g. live product/service recommendations etc.) on customers while they are interacting with them to help them increase customer value or improve service to keep the customer loyal.

In the back office, employees are often unaware of demand in front office channels. Hence they may miss opportunity to increase inventory to meet surges in demand or may estimate incorrectly and overspend in procurement by purchasing too much inventory. These are just a few examples of cases where intelligence may well be in the hands of power users in organisations but is not well enough integrated into core business operations where it could be even more effective. There are so many examples of this that companies are now demanding the need to go further. They want to tightly integrate business intelligence into business operations to improve efficiency and the effectiveness of front-office, corporate and back-office business operations. In short, companies now have the desire to leverage intelligence everywhere so as to become an intelligent business. This article looks at ways that this can be achieved to maximum effect.

The Incomplete Analytical Process

Before we look at ways in which we can integrate intelligence into enterprise systems it is important to point out the difference between data warehousing and business intelligence and understand what most companies have achieved to date. Data warehousing differs from business intelligence in that data warehouses contain integrated information acquired from various source systems to be made available for analysis. Business intelligence, on the other hand, is produced by conducting such analyses on data in data warehouses and as such is ‘downstream’ so to speak from the data warehouse build process. This implies an analytical process whereby data is first acquired and integrated before it is analysed to produce intelligence – an obvious statement you may think. The current typical analytical process for most organisations today producing business intelligence is therefore a two-step process:

Acquire → Analyse

This process goes some way to being useful in that it can produce BI. However it is not a process that can integrate with enterprise systems. It is ‘stand-alone’ and depends on people needing to be involved in analysing, interpreting intelligence and then taking action. While this works to some extent it is not enough. This is because most companies overlook the fact that the process of being effective with BI means that we need to go way beyond just acquiring data and analysing it. In we are to ever reach the goal of BI integration into enterprise business operations, the process has to be extended to being a complete BI integration process that ties right into enterprise operational systems:

Acquire → Analyse → Interpret → Decide → Act/Recommend

When you look at the above extended analytical process flow, three things are obvious. Firstly, there are three addition steps in the process – interpret, decide and act/recommend. Second, most businesses today use people to interpret, decide and act/recommend when it is technology that is also needed to perform these functions if we are to ever leverage intelligence to the full in business operations. Ironically, this technology is available today but in most cases is ignored or not linked data warehousing systems to really increase effectiveness. The third is that there is a workflow linking the steps in the process to tie the whole process together or parts of the process together. Very few companies have ever thought to use workflow to chain together data warehousing, business intelligence and decisioning technologies to cause BI integration into business operations. What is also interesting is that this process has an input and an output. The input is data and output is actions or recommendations. Furthermore the steps from Analyse through to Act/recommend can be integrated and invoked as whole process for use in real time business operations. Clearly, the more current the data is, the more responsive the actions/recommendations can be. In fact data could be passed into the process as soon as updates occur in source systems and before that data gets to a data warehouse (i.e. en route to a data warehouse). Such an option would make the process the basis for leveraging real time data and real time analytics. Alternatively, integrated data in warehouse systems can be passed into the process at the Analyse step. The power here is significant and as yet most are really just scratching the surface.

In addition actions can of course take many forms. It could be a visual or audible alert, a series of alerts to multiple people, a application event, an internal or external transaction invocation, producing one or more reports, sending emails, delivering other content to one or more users or an entire workflow consisting of all of these things. In other words the above ‘extended’ analytical process can drive business operations if it is all done via technology. One final point is that this process, like any other in business operations, could be invoked by an event occurring such as being triggered by a database update, being called by an operational/e-business application or specifically invoked by a user. It is this point that indicates that BI systems can be almost transparently integrated into business operations to make the entire business an intelligent business.

Therefore this extended analytical process can cause the closed loop with enterprise systems. There are however a number of approaches and architectures to achieve this. Let’s examine them to understand the pros and cons of each as well as the business implications and limitations of these approaches.

Architectures for Integrating Data Warehousing and BI into Enterprise Systems

There are several ways to integrate business intelligence into the enterprise. These are as follows:

• Simple reporting on operational systems
• Attaching pre-computed BI to operational data and accessing it from multiple devices
• Embedding analytics in operational applications during application development
• Introducing BI web services and XML for dynamic integration with enterprise applications
• Integrating BI into portals
• Real time analytics – deploying on-demand analysis and decision engines for automated decisions, recommendations and alerting

Simple Operational Reporting

The first option is not really a closed loop option. However I have added it deliberately because sophisticated technology options are often raved about among IT professionals when in many cases basic core every day need is overlooked. Simple operational reporting on what orders are on the books and what is the current inventory compared with orders are real world every day examples. This is key operational information often simply provided by accessing operational systems using general reporting tools. No clever technology, just simple access to important information. In the current tough climate many people can so easily dismiss such basic core business needs by getting swept up in the hype of technology. This is why the option of simple reporting on operational systems should never be ignored. It is generally cheap, quick and easy to implement and for many it is plenty enough to get by with in every day business operations. Woe betides any company that chooses to sacrifice or ignore simple solutions using this option. As much as 80% of the time straight forward operational reporting is perfectly fine for many needs in most companies albeit that it does not lend itself to more sophisticated analytical studies.

Pre-Computed Intelligence

The second approach is also straight forward and simple to implement. This involves pre-computing intelligence and then attaching it to records by writing it back into the data warehouse or to operational systems or both e.g. attaching customer intelligence to each customer record in an operational single customer view database. This concept is sometimes called ‘write back’. The clear advantages of this approach are the simplicity (again) and the performance when accessing the intelligence. Traditional data warehousing and analysis takes place but the results of the analysis are simply attached to data available to users in operational systems or data warehouses. Leveraging the intelligence in operations is now simply a job of picking up pre-computed intelligence and using it. Of course, the intelligence may need to be updated on a regular basis to keep it current, but nevertheless this straightforward option is very effective. Indeed, several case studies exist today indicating that this approach is widespread and proving simple but effective. The recent winner of the Information Management Project of The Year Award in the UK uses this approach whereby sales representatives are able to access pre-computed customer intelligence in a data warehouse directly from there mobile phones while on the road. This allows them to access the intelligence they need (in this case approved recommended products suited to each customer) before visiting existing customers and prospects. The purpose of the application is to help the sales force make the correct ‘customer intelligent’ competitive offers to immediately close deals and increase customer value. The strength of this approach is that it is a very simple solution that is proving to be extremely effective in a number of corporations.

Embedded Analytic Components

For those requiring tight integration of intelligence in operational business processes, the third option available to companies it is to embed analytics in operational applications during operational application development. Professional developers can do this by embedding analytic components (in analytic application development platforms) inside operational applications to deliver analytics as part of an operational application. Figure 1 shows an example of this with Oracle BI beans. Here standard BI enterprise java beans (EJBs) can be integrated into packaged and internally developed operational applications to analyse or mine data in analytic data stores and provide intelligence within the enterprise operational application being used in a particular operational business process. Examples of such beans include visualising beans, and analytic application beans all of which are part of the Oracle 9i iBeans repository of pre-built BI components. In this case, operational applications could be updating their own database(s) while leveraging analytic components to provide intelligence within a particular context in the operational application. The BI beans could of course be accessing data in a data warehouse or data mart that may or may not be known to the user. In this example the Java APIs to the BI beans can also be called within Oracle PL/SQL stored procedures. Embedding analytic components in operational applications tightly integrates BI into enterprise applications and therefore into operational business processes. Other examples of platforms for embedding BI analytic components into operational applications during development include SAS AppDev Studio, thinkAnalytics K.Wiz Enterprise and Business Objects Application Foundation.

This approach has its strengths and weaknesses. On the plus side it clearly offers application developers the ability to leverage and deeply integrate intelligence in an application context. On the minus side, it may need to be done for multiple applications and so involves more effort to maintain those applications unless the BI components are shared server components or the shared stored procedure option mentioned above is chosen.