Is
Predictive Analytics for Marketers really that Accurate
Perhaps the biggest myth underlying predictive
analytics is its ability to accurately predict outcomes. Intuitively, one would think that the level
of accuracy would be high. But let’s really see what this means. Suppose we
build a response model where the average response rate prior to the use of any
predictive model is 2%. If this model is
100% accurate and is being deployed against a universe of 100000 names, we
would expect that the top model-scored 2000 names would all be
responders(100000 X .02). In reality, the experienced practitioner never
observes this and if this indeed were the case, grave concerns would be
considered regarding massive overstatement of the model. In fact, a more likely
scenario would be that the model achieves a 6% response rate in the top 10% of
model-scored names where 30% of all observed responders are classified
correctly. Certainly, the number 30% is better than 10% which would be the case
if the solution were completely random. Yet, the mathematical purist would
still be correct in saying “30% of all
responders is good but what about the other 70% of responders”. Indeed what
about this other 70% and why do many practitioners live with these results and these constraints. The
answer in one word is “NOISE.”
As any mathematician and statistician will tell you,
our tools are about explaining trends and patterns and in effect reducing the
noise. Consider the basic concept of multiple regression which attempts to
interpret the performance or power of a model based on its ability to explain away this noise
relative to the total noise in all the data. In the world of business and
indeed marketing , this ability to truly explain away this noise is severely
limited. It is not unusual to have models or tools that explain only 5% of all
this noise with the result being that
the model is performing well
above average when compared to other models.
Why do we accept these so-called
“dismal” statistics such as 70% of responders not being accounted for and 95% of the noise
not being explained.
The real issue when considering “NOISE” is identifying what is true noise versus
noise that can be effectively explained away through a predictive model . Unfortunately, this is not something that a
textbook in mathematics can explain. It is only by applying models in practice
that one observes this huge disparity between what noise can be explained
versus noise that cannot explained.
In the real
world, predictive analytics is often used to predict outcomes where the
observed “YES” cases are indeed much
greater than the “NO” cases. For instance, response and defection models all
operate under scenarios where the “yes” outcomes typically occur in less than 5
cases out of a hundred cases.. The
extreme proportion of “no” outcomes by
its very nature creates more noise. A
good example of this is that the model’s ability to accurately predict outcomes
increases as the rate increases to 50%. In fact, a good exercise is to try building models where the rate is close to
50% and you will observe that diagnostics explaining the power of the model increases when compared to models
where the rate is small(5% or less). We actually attempted this exercise
ourselves where we built a predictive
model using the same data fields and the same target variable which in this
case was response rate. Listed below is a table depicting our results.
|
Response Rate Model
|
Response Rate of Sample
|
R2
|
0.60%
|
0.34%
|
2.76%
|
1.13%
|
5%
|
1.55%
|
50%
|
7.24%
|
As you can see the above results support our
hypothesis that increasing response rate translates to increased model power
or performance as demonstrated by R2 .
Fraud models present extreme challenges when
building predictive models as there are
often far too many no’s relative to yeses .
One way of improving model
performance is to stratify the sample where the practitioner keeps all the
yeses but extracts a random sample of the no’s thereby increasing the overall
rate of the yeses. Stratified sampling
is widely accepted as a sound data mining practice when dealing with extremely
large proportions of no’s.
Having talked about the ability to increase overall
model performance thru increased rates(i.e. response, defection,etc), we
observe that even in our experiment, the
model performance or R2 peaked
at 7.24% with a response rate of 50%. But
we can easily ask the question: “What about the other 92.7% that the model is not able to explain?.
Is this true random noise or can we
further explain some of this noise. The key in attempting to explain away more
noise is to identify variables that exhibit non linear relationships with the target model variable ,
yet make sense regarding the behavior we
are trying to predict and the particular business where these solutions are to
be deployed. The use of pure mathematics
without an understanding of the business could
result in models that yield non linear type variables which are simply
attempting to explain away true random noise.
The notion of having several validation samples, one that is derived
from the same analytical file as the development sample and another validation
sample that is derived from a different time period, can certainly help to
mitigate model overstatement caused by excessive non linear type variables.
Yet, even in undergoing this type of rigor in enhancing our model performance,
it is unlikely that we can reasonably
expect models to explain more than 10% of the target behaviour.
If predictive analytics has limited powers in the marketing arena,
then why should there be any enthusiasm for its resultant solutions. Forgetting
the statistical limitations as seen above, it is the incremental ROI that is
the huge benefit of predictive analytics. It is not unusual to achieve incremental $
benefits in excess of $100M per campaign. So as an old
business mentor of mine used to say:
“Let’s ground the statistics to the business rather
than the business to the statistics”