Enterprise Business Intelligence and Data Warehousing

Program Management Essentials

Alan Simon

• Publisher: Morgan Kaufmann
• Year: 2014
• ISBN:
  9780128017463
• Topic: Business
• Subtopic: Business Intelligence
• Index: Business

Corporations and governmental agencies of all sizes are embracing a new generation of enterprise-scale business intelligence (BI) and data warehousing (DW), and very often appoint a single senior-level individual to serve as the Enterprise BI/DW Program Manager. This book is the essential guide to the incremental and iterative build-out of a successful enterprise-scale BI/DW program comprised of multiple underlying projects, and what the Enterprise Program Manager must successfully accomplish to orchestrate the many moving parts in the quest for true enterprise-scale business intelligence and data warehousing.

Author Alan Simon has served as an enterprise business intelligence and data warehousing program management advisor to many of his clients, and spent an entire year with a single client as the adjunct consulting director for a $10 million enterprise data warehousing (EDW) initiative. He brings a wealth of knowledge about best practices, risk management, organizational culture alignment, and other Critical Success Factors (CSFs) to the discipline of enterprise-scale business intelligence and data warehousing.

Chapter 1

The Challenge of Managing and Leading the Enterprise BI/DW Program

Abstract

Enterprise business intelligence and data warehousing (EBI/EDW) has historically been a very difficult proposition, for a number of reasons. This chapter explores these many reasons to set the context for how strong program management can overcome these challenges in today’s and tomorrow’s initiatives.

Keywords

Data

Big Data

business intelligence

BI

analytics

predictive analytics

program manager

program management

project manager

project management

challenges

data warehouse

data warehousing

enterprise data warehouse

EDW

Chapter introduction

We began the book’s Preface with a question, and we will do the same in this first chapter. Our question to now ponder is as follows:

Why, since the dawn of the business intelligence and data warehousing era, has enterprise business intelligence and data warehousing been so incredibly difficult to achieve?

Certainly, we can find instances of successful true enterprise-scale BI/DW efforts (i.e., an EDW that actually contains and manages the critical mass of data used for a company’s or governmental agency’s business intelligence needs, rather than a smaller-scale data warehouse or data mart with the word “enterprise” simply tacked onto its description). These enterprise-scale success stories aren’t nearly as common as we would hope, though, given that at the time of this writing (late 2014) a quarter century has passed since BI and data warehousing came upon the scene as the 1980s gave way to the 1990s. Still, if one were to survey a healthy sample of data warehouses, data marts, operational data stores (ODSs), and other “reporting databases” that have been built at some time during the past quarter century, the overwhelming majority of them will be found to serve departmental-level needs, or the reports and BI needed for only a handful of an organization’s business processes.

In this chapter we will explore why enterprise-scale BI and data warehousing has been so difficult to achieve, as well as – the good news, finally – why the tide seems to be turning in recent years with an increasing number of enterprise-level success stories.

Why the history lesson? Because any enterprise business intelligence and data warehousing (EBI/EDW) program manager or program management team needs to understand the inertia that today’s initiatives need to overcome. The “threats” are still out there, and ignorance of the many challenges EBI/EDW efforts have historically faced and still face puts a program manager at a distinct disadvantage. Quoting philosopher and author George Santayana (1863–1952), “Those who cannot remember the past, are condemned to repeat it.” Since so many of today’s enterprise BI/DW program managers haven’t experienced the full 25 years’ worth of the modern era of business intelligence and data warehousing – and indeed may be relative newcomers to the information technology profession – this chapter helps them “remember” that which they may not have actually experienced.

And for those readers who are indeed long-time BI/DW practitioners, the consolidated collection of EBI/EDW challenges presented in this chapter, along with the discussion of what is changing to help counteract those challenges, will provide that “one-stop shopping” of important information that hallmarks what we try to achieve in the BI/DW world for our end users.

The challenges of enterprise-scale bi and data warehousing

Among the many significant challenges over the years to successfully building data warehouses and BI capabilities at the enterprise scale we find:

• Immature technology, at least in the early days of the modern BI/DW era

• The backlash from early EDW failures

• Companies and governmental agencies also focusing on many other competing high-profile enterprise initiatives

• “The need for speed”

• The devaluation of enterprise-level BI versus “subenterprise” reporting and insights

• The cost and difficulty of unwinding data mart proliferation

• The lack of a “voice of authority” champion for enterprise-scale initiatives

• Economic and external factors in the aftermath of two recessions and business downturns

Let’s briefly look at each of the above in the context of not only what has occurred in the past but also which EBI/EDW program managers need to be aware of, even for brand new initiatives.

Immature Technology

At the dawn of the modern BI/DW era, relational database management systems (RDBMSs) were still very much a relatively new technology. RDBMSs began appearing in the research community in the mid-1970s, and started becoming commercially available in the late 1970s and early 1980s with products from Oracle, IBM, Ingres, Digital Equipment Corporation, and other leading systems and software vendors of the day. Given the nature of RDBMS technology, significant engineering continued to go into the commercial products during the early days to enable those RDBMSs to finally be capable of supporting online transaction processing (OLTP) functionality. Specifically, substantial engineering work went into the query plan optimization necessary to provide acceptable performance for the types of database read/write access commonplace to OLTP applications and their typically normalized or lightly denormalized database schemas.

Given that data warehouses typically structure their data dimensionally (to support OLAP capabilities) rather than in a normalized manner, the types of multitable joins and other database operations commonplace in data warehousing weren’t particularly well suited for the first generation or two of RDBMSs that were performance-tuned for OLTP – particularly for large EDWs with tremendous volumes of data and hundreds or thousands of tables. Data loading and user response times were often unpredictable at best, and very often substandard and unable to meet stated requirements.

The multidimensional technology (e.g., data cubes) developed by vendors such as IRI (Express), Arbor Software (Essbase), and Cognos (PowerPlay) as an alternative to relational-based data warehousing typically worked well up to the tens-of-gigabytes size in early product implementations, but not particularly well for the data volumes early EDW planners envisioned for their systems.

Similarly, in the extraction, transformation, and loading (ETL) space, the first generation of tools often provided a user-friendly, decently architected “shell” for the ETL packages that needed to be developed, but at the same time also required significant amounts of callout code written in SQL or another language for more complex transformation operations. Whereas today’s ETL products support many different patterns through graphical interfaces and drag-and-drop usage, the earlier generation of ETL packages didn’t deliver nearly the degree of productivity and overall ETL lifecycle management to which we’re accustomed today.

The bottom line: whereas corporate and governmental agency strategists originally had lofty goals for the “one-stop shopping of enterprise data” envisioned with EBI/EDW, and while the high-level architectural diagrams representing these goals were relatively easy to envision and draw, that first generation of BI and data warehousing technology was actually much better suited toward smaller-scale initiatives than large, enterprise-scale efforts.

Products and tools in all aspects of BI and data warehousing have dramatically improved over the years, of course, but the legacy of that first generation of enterprise-scale efforts and the overall lack of desired success wound up resulting in companies scaling back their efforts at the enterprise level and making their next BI/DW attempts with smaller-scale initiatives (next).

Backlash From Early EDW Failures

By the latter part of the 1990s and into the early 2000s, EDWs were “out of vogue” for most organizations. Part of the reason for this en masse downscaling of aspirations is because of the boom in competing initiatives (discussed next); however, to a large extent BI/DW proponents faced significant backlash from corporate and governmental budget holders and leaders in light of so many early EDW efforts falling far short of their originally stated objectives.

The term data mart came into vogue by the late 1990s: data warehousing and BI capabilities applied on a much smaller scale than their enterprise-wide predecessors. Data marts typically were built with a handful of source system feeds; significantly smaller planned user community sizes; and capabilities for reporting and data analysis limited to a single business process or a single department, or some other significantly subenterprise scope.

Data marts were typically faster and less expensive to design, build, and implement than EDWs (even on-budget, deliver-by-deadline EDW efforts), and came with inherently less overall risk than an enterprise-scale effort. Despite the scope and scale limitations of data marts versus EDWs, more and more organizations turned their data insight efforts in that direction and away from the original vision of an EDW containing the critical mass of an organization’s data for reporting and BI purposes.

Many Competing Enterprise Initiatives (and the Impact of Packaged Software)

At the same time that the first generation of EBI/EDW efforts was coming up short, the mid- and late 1990s saw the dramatic rise of:

• Enterprise resource planning (ERP) packages from SAP, Oracle, PeopleSoft, Baan, and other vendors – many in response to the looming Y2K situation

• Customer relationship management (CRM) systems from Siebel, Vantive, E.piphany, and other vendors that hallmarked a new era of sales force automation (SFA), call center management, and attempts to better understand and manage the entire customer life cycle

• A new generation of supply chain management (SCM) capabilities

• The commercialization of the Internet and the birth of eCommerce

At any given point from about 1996 to 2000, almost every single company and governmental agency found themselves deluged with significant, expensive, and resource-consuming development efforts in all of the above disciplines – at the same time they were trying to cope with so many less than successful EBI/EDW results. In concert with so many organizations shifting their allegiances to embrace the data mart (vs. data warehouse) concept, most packaged software included starter kits (data models, scripting or graphical user interfaces, etc.) for reports based on their underlying transactional data.

The result: most organizations wound up steadily building out a federation of independent, nonintegrated data marts that as an aggregate provided the types of reporting and data-driven insights sought after from earlier EBI/EDW efforts.

The catch: because of the lack of master data and common business rules across these many data marts, the quest for a “single version of the truth” from reports and BI was typically unachievable. Customer and sales data could often be found in dozens of data marts; the same can be said for product data and many other key subject areas. Even relatively simple reports (e.g., “how many stores did we have at the end of this quarter versus the end of the same quarter last year?”) coming out of different data marts often produced very different results when they should have been the same.

Still, organizational leaders just gritted their te...

Table of contents

• Cover
• Title page
• Table of Contents
• Copyright
• About the author
• Preface
• Chapter 1: The Challenge of Managing and Leading the Enterprise BI/DW Program
• Chapter 2: The Role and Charter of the Enterprise Business Intelligence and Data Warehousing Program Manager
• Chapter 3: Building the EBI/EDW Program’s Initial Project Portfolio
• Chapter 4: Putting the Finishing Touches on the EBI/EDW Program’s Project Portfolio
• Chapter 5: Program-Level Risk Management
• Chapter 6: Program Key Performance Indicators (KPIs) and Key Operating Indicators (KOIs)
• Chapter 7: Conducting the Quarterly EBI/EDW Program Review
• Chapter 8: Considerations for the Big Data Era