Do You Really Want to Wait for a Data Warehouse?

WE WERE RECENTLY asked to critique a plan for a system that would support database marketing. The proposal included a corporate data warehouse that would be the system’s information source. The plan called for a three-tiered relational architecture. The time allotted to complete the project was 18 to 24 months.

The system would be required to profile, model, segment and score customers; plan, design and execute promotions; track results; and provide reports to management.

Here’s what’s needed to accomplish these limited objectives: the identification, collection and storage of relevant marketing data from internal operational systems and external data sources (including mail houses, telemarketing suppliers and data providers); and the development of modeling, promotion and reporting systems.

Given all the direct marketing systems available today (relational, non-relational or proprietary) we don’t believe there’s any need to spend years building a DM database and decision-support system from the ground up-unless, of course, that’s what you want to do.

So why did the plan we reviewed suggest that this project would take one to two years to implement?

The Facts First of all, it could take years to build a corporate data warehouse and the associated data marts and decision-support systems. Why it takes so long has to do with the changes that need to be made in the operational systems providing data to the data warehouse.

For example, if several source systems contain names and addresses, then the warehouse will have to first identify which fields are considered attributes of a customer, then determine which customers can be defined as duplicates, and then, based on some agreed-upon logic, arrive at the correct name and address for that customer. Whether this “correct” name is fed back to each input or source system, or whether the logic in the source system is changed to access the data warehouse for names and addresses, is an open question.

The central fact is that operational systems have to be modified to take advantage of the benefits of the consolidation. The extent that common data is found in more than one input file determines the work that has to be done in the data warehouse and in modifications to the operational systems.

Hence, the decision to set up a data warehouse as opposed to building a basic marketing database-defined here as a consolidation of data from multiple sources, without a feedback mechanism to the source operational systems- caused the scope and extent of this project to change dramatically.

What’s more, because the repository was large and could be used for activities other than marketing, a secondary repository containing only the data needed for marketing was envisioned-a data mart or marketing database.

Finally, in order to meet the performance demands for decision support and management reporting, a third data level (containing summary data and the tools required to access summary data) was required.

In other words, since the volumes were large-a database containing information on over 5 million customers is considered large-and because the database management system was relational, using the detailed data available in the marketing database would not be practical. That is, it could take hours to answer even relatively simple queries.

Looking at it objectively, what started as a request to support marketing grew, inadvertently or otherwise, into a much larger project. Couple this with the choice of relational technology for both the warehouse and the data mart, and you wind up with a two- or three-year job.

So how do you get around this problem?

Let’s assume you are forced or choose to use a relational database management system as your base platform, rather than one of the many proprietary systems designed for DM applications. (We market and support a proprietary product.) You can make a data warehouse or data mart unnecessary by limiting the data going into the system to information used only for marketing purposes. Just make sure the interfaces are one-way feeds from the legacy systems to the database. Thus the data warehouse and the data mart become one and the same.

Another approach might be to have a relational warehouse with a proprietary system as the platform for the second tier. This would make summary data and third-tier tools for fast data access unnecessary.

The third choice is to use one of the proprietary platforms as a marketing database to significantly reduce your development time.

Of course, in order to go proprietary you’ll probably have to overcome the objections of your company’s information technology department. IT may insist on using a specific relational product regardless of its performance characteristics, because it has settled on that relational standard for operational systems and doesn’t wish to support another piece of technology.