David Shepard Associates Blog

We were recently asked to critique a plan to have a database system developed that would support database marketing programs. The recommended solution included upgrading a corporate data warehouse that would be the source of data required to support a marketing decision support system. The plan called for a three-tiered relational architecture. The time frame: 18 to 24  months. 

The essential requirements of the system included the ability to: profile, model, segment and score customers, plan, design and execute promotions, track results and provide management reports.
 
In order to accomplish these limited objectives what’s required is: the identification, collection and storage of relevant marketing data from a variety of internal operational systems and external data sources, including mail houses, telemarketing suppliers, and data providers, and the development of the required modeling, promotion and reporting systems.
 
Given all of the direct marketing systems available today (both relational and non-relational or proprietary) there’s no need (our opinion) to spend years to build a direct marketing database and decision support system from the ground up that would meet the requirements outlined above, unless of course you enjoy doing that kind of thing. 

So, why did the plan we reviewed suggest that it would take one to two years to implement? 
First of all, it could take a year or more just to upgrade the corporate data warehouse, let alone the associated data marts and end user software tools.

Why it takes so long has to do with the amount of change required in the operational systems providing data to the data warehouse.  For example, if a number of source systems contain name and address data, then it will be the job of the warehouse to first identify which fields are considered attributes of a customer, then identify which customers can be defined as duplicates, and then based on some agreed upon logic determine the “correct” name and address for that customer. 

Whether this “correct” name is feed back to each input or source system, or whether the logic in the source system is changed to access the data warehouse for name and address data is an open question. The central fact is that operational systems have to be modified to take advantage of the benefits of the consolidation. The extent that common data is found in more than one input file determines the work that has to be done in the data warehouse and in modifications to the operational systems.
 
Hence, the decision to integrate with a data warehouse as opposed to building a simple marketing database (defined here as a one way consolidation of data from multiple sources, without a feedback mechanism to the source operational systems) caused what was initially a relatively simple request to explode into a complicated, time consuming and expensive project.
 
What’s more, because the repository was large and could be used for other activities in addition to marketing, a secondary repository, just containing the data needed for marketing was envisioned — a data mart or marketing database.
 
Finally, in order to meet performance demands for decision support and management reporting, a third data level, containing summary data and the tools required to access summary data was required.  In other words, because the volumes were large, a database with more than 5 million customers is considered large, and because database management system was relational, using the detailed data available in the marketing database would not be practical, i.e. it could take hours to answer even relatively simple queries. 

Looking at it objectively, what had started as a request to support marketing, grew inadvertently or otherwise into a much larger project.  Couple this factor with the choice of relational technology and you will wind up with a two or a three-year project. 

So, how to get around this problem?  Assuming you are forced or choose to use a relational database management system as your base platform, as opposed to one the many proprietary systems designed for direct marketing applications, you can eliminate the need for a data warehouse/data mart configuration by limiting the data going into the warehouse to just data needed for marketing purposes.

Just make sure that the interfaces are one way feeds from the legacy systems to the database. Thus the data warehouse and the data mart become one in the same thing. Another approach might be to have a relational warehouse and use a proprietary system as the platform for the second tier, eliminating the need for summary data and the third tier tools necessary to access data within acceptable performance standards. 

The third choice is to use to one of the proprietary platforms as your marketing database and significantly reduce your development time and benefit from their high performance characteristics. Of course, in order to go proprietary you will probably have to overcome the objections of your IT department which may insist on using a specific relational product regardless of its performance characteristics, because they have settled on that relational standard for the their operational systems and don’t wish to support another piece of technology. 

“I’m sorry, I know you need this customer profile report, but it requires adding a new data source.  The marketing database design specs were frozen in order to make the system delivery date. If we add this new requirement, the system will be delayed for 2 more months.  You want to explain that to the boss?”

How many of us have heard these words?

The majority of companies implementing a new marketing database system will end up doing a major redesign, sometimes called “Phase 2”, within the first 12 months… and they are the lucky ones, many other companies cannot afford the budget or staff time to do a “Phase 2” causing their marketers have to live with the system as-is.  In defense of the system implementers, there comes a time when you have to stop building requirements and start building the system if you are to going to meet your target date.

Sometimes the problem is that the development schedule did not include sufficient time to collect end user requirements and reconcile them into a comprehensive set of  system requirements.  But, most of the time the problem is that the marketing and implementation teams have an insufficient base of experience to draw upon.

In short, they don’t fully understand what type of data is available, how that data would be combined and updated, what new kinds of marketing programs the data would support, how they would go about implementing those new marketing programs and how best to track and evaluate results. In this situation, defining requirements is much more of a process than an event. The more that new capabilities are considered, the more new opportunities are seen… and this never happens all at once.

In implementing marketing databases, there is usually a push by senior management to get the database running A.S.A.P. so that the system can start generating a return on its investment. As a result, many, companies fall victim to the old problem where there is not enough time to do it right the first time, but there is enough time to do it twice.  Shifting to a two step strategy where a prototype database is built before the final system can help alleviate many of these issues.

A prototype database is essentially a “throw away” system that is built quickly, supports most of the required database functions, doesn’t have to worry about being efficient and represents a short term (not a long term) investment.  A prototype database provides immediate operational benefits while helping to build a base of knowledge to use in defining requirements for the final system.  This system can also be used to develop “proof of concept” information to support the decision to fund a marketing database system.

Depending on available data sources, a prototype database can immediately provide information to support strategic planning through demographic profiles, purchase activity profiles, customer value, customer segments,  etc. The system can also be used tactically to implement promotions, track promotion history, track promotion results and do backend responder analysis. A prototype database offers many significant advantages.

Fast startup.  Since the goal is to get a quick “80% solution”, not to implement a complete, efficient long term production system, the prototype database does not need all of the same operational processes as the final system.

Flexible implementation.  Prototype databases use a large number of ad-hoc processes to make them flexible, since efficiency is not a key issue. Therefore, it is much easier to include new sources of data, add new calculated fields, process new list selection instructions, produce new reports, etc.

Hands-on experience with the data.  Building the prototype database gives a preview of which data is available from which internal and external sources and how it can be combined. This involves investigating file matching criteria (customer number, telephone number, name and address, etc.), determining how data fields from different sources should be combined or reconciled against each other and understanding how each field is coded and what the codes mean. During this process, each data field is profiled to review data formats, identify data errors and define options for resolving errors.

Experience using a marketing database system. 
Besides the production-oriented issues identified above, the end users in the marketing department will start gaining hands-on experience of a different nature. Once the database is built, the marketing staff will start understanding what types of data are available.

They will also gain considerable experience with using the data effectively to run their business. Very often, the new range of possibilities will force a re-thinking of how marketing does its job, ie: the business process.  Through a combination of planning and trial & error, new customer segmentations will be created, new list selection criteria will be implemented, new reports will be created and marketing program success will be defined differently.

We have to recall that the speed and flexibility of implementing prototype databases comes at a price. These systems rely on a number of ad-hoc processes, manual intervention and outside services.  Prototype databases often are based on batch processing and having an experienced IT staff member write programs for each update step, list selection and requested report. This results in slower than desired turnaround times. 

Depending on the type of software used to implement the prototype database, the system may or may not support direct access by end users along with the ability to directly use marketing-oriented software tools.  Finally, depending on the ease of gaining access to data sources, the prototype database may not include all of the data that is desired by the marketing staff.