Implementing Analytical Projects

Goals, Resources, Schedule

At the beginning of each analytical project it is important to be in contact with key stakeholders to delineate the overall project orientation, marketing objectives and priorities. This initial step also identifies key internal and external resources, points of contact and project milestones. Common outputs are a written report summarizing project direction and specific objectives, as well as a finalized project schedule.

It is very easy to become mired down in a theoretical discussion of opinions on analytical techniques. However, time and time again it has been demonstrated that the quality of analytical results are much more impacted by data quality and preparation issues than the particular statistical technique used.

The client usually has data that spans multiple business units, multiple product lines, multiple sales channels, multiple time periods and multiple promotions. If this is the case, then selection of the “right” data set(s), or the weights given to individual data sets for analytical projects will be drastically more important than the choice of statistical technique. We believe that our years of experience in dealing with these kinds of “real world” problems will be of significant value to the client.

Identify Data Sources

In the next step, DSA will work with appropriate client staff to identify and evaluate all potentially relevant sources of data (including data append). The end result will be our recommendation for the data elements that should be used for the project. Included in this recommendation will be a summary of data sources, processes for obtaining data, record layouts, schedules and possible third party data enhancements that we think will be useful.

Exploratory Data Analysis

Once the data has been acquired, the next step will be to complete an exploratory data analysis (EDA). This process involves examining profiles and distributions of each available data element to determine whether any data errors exist, the likely range of values, typical distributions and whether the results are deemed reasonable. Data elements displaying any issues will either be corrected or rejected during this process after review with appropriate client staff. DSA pays special attention to data preparation issues such as validating the data, dealing with missing values, outlier adjustment, linear transforms, multicollinearity and cross products (interaction terms).

Data Reduction

After completing the EDA step, the next task is typically to examine the data set for data reduction opportunities. In most analytical projects we are presented with a large number of variables or characteristics (100+ is typical). However, not every variable represents a unique type of information about the member and including inter-correlated variables in the analysis would corrupt the results. That is why we usually use principal components analysis to determine the amount of unique information represented in the data set.

Results from the P.C.A. will be considered along with data profile reports to identify a set of variables to carry forward into the actual analysis step. Depending on the needs of the project, we can also develop recoding and linear transformation schemes for individual variables if necessary to improve predictive power.

Analytical Technique

The next step involves implementing the actual analytical technique. In the case of developing a predictive model, we will use an appropriate statistical modeling technique such as logit, probit or general linear mixed models. This choice depends on the type of dependent variable (initial response, backend conversion, long term value, etc.). The process of calibrating model can be very labor intensive because calibration is more of an art than a science. Basically, we will evaluate well over a hundred different scenarios to find a solution that delivers a high degree of segmentation and consistent results (validation).

Validating Results

DSA uses a variety of techniques for validating analytical results depending on project objectives and constraints. These can include:

Content comparison against other analytical results
Split Halves Validation based on the model development sample
Bootstrap Validation based on the model validation sample
Backend analysis of live promotion results

Implementing Results

When problems arise with implementing analytical results, the problem is usually traced to the implementation process. DSA works with the appropriate client staff to communicate any relevant list pre-selects, limits to application, statistical confidence intervals, likely range of results, etc. DSA will also work with the technical staff to develop appropriate methods for implementing the analytical results and defining any needed quality assurance practices.

The last step in the process involves ongoing support for designing appropriate in-market tests and then measuring, evaluating and interpreting “real world” results.

DSA strongly believes that model performance should be evaluated each time that the model is used. Key indicators that it is time to recalibrate or redevelop a model can include:

Significant decrease in model performance
Continued downward trend in model performance
Major change in underlying offer, price point or creative
Entry of a significant new competitor
Major change in the consumer marketplace or economic conditions