Is Predictive Analytics for Marketers  really that Accurate

 How can a data mining professional say such blasphemous words.  After all, doesn’t Big Data Analytics and the use of predictive analytics represent the panacea to our ever-expanding data universe. The many articles and whitepapers along with their accompanying vendors purport that this new frontier  must be embraced by organizations to remain competitive in the 21^st century. Yet, like all emerging transformative disciplines, there needs to be a little of “Let’s step back” and  attempt to “get under the hood of what predictive analytics can and cannot do”  if expectations are going to be  consistently managed.

Perhaps the biggest myth underlying predictive analytics is its ability to accurately predict outcomes.  Intuitively, one would think that the level of accuracy would be high. But let’s really see what this means. Suppose we build a response model where the average response rate prior to the use of any predictive model is 2%.  If this model is 100% accurate and is being deployed against a universe of 100000 names, we would expect that the top model-scored 2000 names would all be responders(100000 X .02). In reality, the experienced practitioner never observes this and if this indeed were the case, grave concerns would be considered regarding massive overstatement of the model. In fact, a more likely scenario would be that the model achieves a 6% response rate in the top 10% of model-scored names where 30% of all observed responders are classified correctly. Certainly, the number 30% is better than 10% which would be the case if the solution were completely random. Yet, the mathematical purist would still be correct in saying  “30% of all responders  is good but what about  the other 70% of responders”. Indeed what about this other 70% and why do many practitioners live with  these results and these constraints. The answer in one word is  “NOISE.”

As any mathematician and statistician will tell you, our tools are about explaining trends and patterns and in effect reducing the noise. Consider the basic concept of multiple regression which attempts to interpret the performance or power of a model based on  its ability to explain away this noise relative to the total noise in all the data. In the world of business and indeed marketing , this ability to truly explain away this noise is severely limited. It is not unusual to have models or tools that explain only 5% of all this noise with the result being   that  the model is performing  well above average when compared to other models.  Why do we accept these so-called  “dismal” statistics such as 70% of responders  not being accounted for and 95% of the noise not being explained.

 

The real issue when considering “NOISE”   is identifying what is true noise versus noise that can be effectively explained away through a predictive model .  Unfortunately, this is not something that a textbook in mathematics can explain. It is only by applying models in practice that one observes this huge disparity between what noise can be explained versus noise  that  cannot explained.

 In the real world, predictive analytics is often used to predict outcomes where the observed  “YES” cases are indeed much greater than the “NO” cases. For instance, response and defection models all operate under scenarios where the “yes” outcomes typically occur in less than 5 cases out of a hundred cases..  The extreme proportion of  “no” outcomes by its very nature creates more noise.  A good example of this is that the model’s ability to accurately predict outcomes increases as the rate increases to 50%. In fact, a good exercise is to try  building models where the rate is close to 50% and you will observe that diagnostics explaining the power  of the model increases when compared to models where the rate is small(5% or less). We actually attempted this exercise ourselves  where we built a predictive model using the same data fields and the same target variable which in this case was response rate. Listed below is a table depicting our results.

	Response Rate Model
Response Rate of Sample	R2
0.60%	0.34%
2.76%	1.13%
5%	1.55%
50%	7.24%

 

 

As you can see the above results support our hypothesis that increasing response rate translates to increased  model power  or performance as demonstrated by R² .

Fraud models present extreme challenges when building predictive models  as there are often far too many no’s relative to yeses .  One way of  improving model performance is to stratify the sample where the practitioner keeps all the yeses but extracts a random sample of the no’s thereby increasing the overall rate of the yeses.  Stratified sampling is widely accepted as a sound data mining practice when dealing with extremely large proportions of no’s.

Having talked about the ability to increase overall model performance thru increased rates(i.e. response, defection,etc), we observe that even in our experiment, the  model performance or R²  peaked at  7.24% with a response rate of 50%. But we can easily ask the question: “What about the other  92.7% that the model is not able to explain?. Is this true random noise or can  we further explain some of this noise. The key in attempting to explain away more noise is  to identify  variables that exhibit non linear  relationships with the target model variable , yet  make sense regarding the behavior we are trying to predict and the particular business where these solutions are to be deployed. The use of  pure mathematics without an understanding of the business could  result in models that yield non linear type variables which are simply attempting to explain away true random noise.  The notion of having several validation samples, one that is derived from the same analytical file as the development sample and another validation sample that is derived from a different time period, can certainly help to mitigate model overstatement caused by excessive non linear type variables. Yet, even in undergoing this type of rigor in enhancing our model performance, it  is unlikely that we can reasonably expect models to explain more than 10% of the target behaviour.

If predictive analytics  has limited powers in the marketing arena, then why should there be any enthusiasm for its resultant solutions. Forgetting the statistical limitations as seen above, it is the incremental ROI that is the huge benefit of predictive analytics. It  is not unusual to achieve incremental $ benefits  in excess of $100M per campaign. So as an old business mentor of mine used to say:

 

 

“Let’s ground the statistics to the business rather than the business to the statistics”

 

   

 

How to Significantly Improving Acquisition Modelling Performance: The Ability to Create Individual-Level Variables

Predictive analytics practitioners will universally agree on data limitations as being the most significant challenge in building acquisition models.  Traditional solutions have utilized aggregated postal area type data either from Statistics Canada or a range of other data service providers which in addition will also provide cluster type variables. In using external data sources such as aggregate postal area demographics only, we do indeed obtain solutions that yield the ability to rank order response rate. As a result, the use of external data sources has always been the critical element in building these type of tools. From the success of using this data to better target prospects in becoming customers, many of the data service providers in this area will also offer predictive analytics solutions as a complementary service to the actual source data. Companies that desire to obtain more targeted acquisition results look to these providers as being their partner in delivering these results.

All practitioners will profess to the significant improvement in predictive analytics results when using individual-level information. One just has to look at customer models such as cross-sell,upsell, and attrition models in order to see the significant improvement in model performance. Because of the greater success here, predictive analytics efforts in many cases have been  more  focused on existing customers with neglible effort on acquisition targeting. But what if there were opportunities within the acquisition data itself that allowed results to improve significantly.

The key in creating these opportunities is to ultimately create that all-important individual-level information from name and address and thereby create new individual-level variables. Of course, the question is how does one do this ? Working with one of our business partners, we were able to create algorithms based on name and address that resulted in gender and age type variables. Furthermore, information was created on the source of the list that yielded the prospect name. As well, we also overlaid the traditional external postal area demographic information to our prospect list and then began our model building process. In our final model, nine variables were identified as the key predictors of acquisition response. Interestingly enough, the external postal area type variables  were the two weakest variables in the model. The model in terms of performance yielded a performance lift in acquisition response of 5 to 1 between the top 5% scored names and the bottom 5% scored names. Using just postal area demographic variables yields only a performance lift in acquisition response of 2 to 1 between the top 5% and the bottom 5%. The use of individual-level information in this case clearly provided that additional response lift advantage.

Of course, more work in employing more  practical examples need to occur before overarching conclusions can be made regarding this approach. But nevertheless, this initial work in this isolated case does point to the need that this approach should be clearly explored in other acquisition type programs. Our belief in identifying significant improvements to predictive analytics solutions has always been less about the mathematical or advanced statistical approach and more about the actual data itself. Our work in this area with this client certainly amplifies this belief and it is our hope that more organizations at least explore this approach of converting name and address information to individual-level information  as a viable option in improving their targetted acquisition efforts.

 

The Data Discovery: Investing in Customer Insight

            The Data Discovery: Investing in Customer Insight

In many of our engagements with new clients, the old Donald Rumsfeld phrase of ‘We don’t know what we don’t know” is very applicable as these organizations commence their journey into database analytics.  In these situations, there is no clear definable objective or goal when undertaking these projects. In fact, these companies look for outside consultation in the creation of a roadmap which represents both strategy and tactics on what database analytics projects they should undertake. These type  of exercises don’t yield an immediate return on investment which can be a barrier to many organizations who fail to appreciate the long term benefits that a data discovery can yield. It is often very difficult to convince these organizations of the longer term benefits of database analytics as they are focused on achieving  short-term gains to resolve an immediate business need. In many ways, this is the true challenge of a data discovery as we seek to strike a balance between the longer term analytics needs versus establishing an immediate ROI.

In undertaking these type of projects, the only common feature is the open-ended nature of the assignment  as an end solution to solving a specific business problem is not necessarily our goal. Instead exploration and discovery is the intent of the project with the goal being to build an analytical roadmap. Yet, even open-ended projects require some process in order to provide guidelines and steps which are necessary for its success. Typically, this process involves four steps which are:

·        Preparation

·        Data Audit

·        Preliminary Analysis

·        Recommendations

Preparation

The first stage of this project represents the portion of the project where the analytics practitioners attempts to increase their  knowledge of the client’s current business and results.  In data mining and analytics, all experts  agree that analytics projects require both domain knowledge and data mining expertise in order to really optimize a given solution. Domain knowledge is specific knowledge which pertains to that business. It represents knowledge which is both unique for the industry sector(finance,retail,etc.) of the client business but also knowledge which is unique for the mechanics of how that client business runs. The preparation stage of the project allows the practitioner to increase their domain knowledge of this business.  Of course, the domain knowledge of the practitioner will never be as exhaustive as the client but the objective here is to obtain an adequate level of this knowledge in order to conduct an effective discovery exercise.

The initial tasks here are to conduct extensive interviews with key business stakeholders from marketing, I/T, analytics(if there is an area), finance, and the executive depending on availability. During these meetings, key business issues and challenges are identified  as well as an understanding of what data is available. Business reports/analyses or any other documents that provide results and meaningful information about their business are shared with the practitioner. At the end of this stage, a data extract is then requested which consists of all the files and fields that will be required for the remainder of this project.

Data Audit

Data audits have been discussed in the past and are a core prequisite in any data discovery exercise. At this stage of the process, the practitioner attempts  to become “intimate” with the data which describes a much stronger relationship with data than the standard phrase of “ data knowledge”.

Once the data extract is received by the practitioner , the data is then loaded into their system whereby  standardized reports are produced that essentially provide the following results:

·        Data completeness or coverage  as indicated by the number of missing values in a variable

·        How do  values or outcomes distribute within a  given variable

·        Data inconsistencies and data gaps. Do values change overtime and are there groups of records where certain data anamolies exist  

From these results, the quality of the data can be assessed in terms of determining what information to use in a future analytics exercise. The results also yield files would be linked together since one objective in this phase is to create an analytical file where we have one record per one customer. With these links determined on how the analytical file would be created, the variable creation exercise can commence. This represents the most labor-intensive portion and arguably most important component of the  entire discovery process as it is here that meaningful variables are created which  can be used in any future analytics exercise.

Besides the exhaustive reports from the data audit, a summary level report is produced which indicates the major findings from the data audit. This would indicate what are the current gaps within the data environment and how it might be improved. Some of these gaps could to some extent be filled by data overlays. Good examples of this are Stats Can data for business to consumer type analytics and perhaps Dun and Bradstreet or Info-Canada data for business to business type analytics. A good example of how data overlays might fill a gap is if income was a key component in any analysis but simply unavailable within the current data environment. Its use here at the postal area level as opposed to being used at the individual level would be  a secondary option in attempting to derive insights based on income. Once this data audit exercise is completed, some preliminary analysis is then conducted which essentially commences our journey into analytics.

Preliminary Analysis

Note how we use the word preliminary here as our objective is to acquire a basic level knowledge or understanding of their current business using our data analytics expertise complemented by our domain knowledge. Obviously, as time goes on and the analytics discipline becomes more entrenched, the level and sophistication of knowledge will increase. 

Our first exercise is to understand at a very basic level how customers are different. A value segmentation or RFM segmentation is a tool we use to stratify customers into deciles with  decile 1 representing the highest performing and  decile 10 representing the lowest performing  customers. From this stratification, we can then arrive value-based or RFM-based segments. Under each scheme, though, we can determine how relevant the 80/20 rule is(i.e. do 20% of my customers account for 80% of my business).

This analysis also helps to point out the migration and retention opportunities. For example, a sensitivity analysis using results from this segmentation analysis and a series of different migration(conversion) rates  would determine the $ opportunity in migrating customers from a lower performing segment to a higher performing segment.  Retention analysis would use somewhat the same approach by looking at  a high performing segment and then determine the $ lost when looking at different potential defection rates. Of course, any sensitivity analysis which looks at a range of  migration rates or defection rates should be reasonable. The reasonableness of these numbers, though, would have been confirmed through  results and comments from key stakeholders that were gleamed from the upfront preparation stage.   

Once these segments are determined, profiling of each of these segments helps to better  understand the key customer characteristics that comprise a given segment. Profiles can be used in two ways. Strategically, the information can be used to develop communication and channel strategies which are more engaging to the mindset of that particular group. For instance, a particular profile of a high value customer may comprise the following characteristics:

·        Older

·         wealthier

·        Tend to engage more in  online media

A heavier online strategy speaking to the concerns of wealthier retirees or people nearing retirement would certainly be very appropriate here.

The second way is tactically where the information is used to create lists of customers that most resemble the profiled segment. Here, we target those low to medium value customers that tend to be older, wealthier, and more engaged online since these customers exhibit the characteristics of a high value customer.

Cohort analysis represents another type of unique analysis as its purpose is to explore  new customer behaviour  over a number of years. Typically, this kind of analysis commences with customers that were new five years ago and then tracks their spending behaviour behaviour in subsequent years. The same analysis is then conducted in subsequent years. Obviously, as we move  forward in time, we have less tracking history with new customers. Through this kind of analysis, we can begin to discern retention,migration,and upsell patterns amongst these new customers. At the same time, we may also find that there are unique behavioural patterns that are associated with a certain cohort group.

Depending on the nature of the industry, basket analysis is another form of analysis to better understand the “event” behaviour of the customer’s purchase. The analysis can reveal that the mix of products purchased can vary between customers. For example, this  product mix will in most cases vary depending on whether they are a high medium or low value customer. Migration opportunities can use the insights generated from this analysis by promoting products that are first relevant for the lower value segment and secondly that will migrate them to the higher value segments.

New customers will also exhibit different  product mix patterns  when compared to older customers. As indicated above, the same migration type of migration analysis is conducted here by identifying those relevant products which should be promoted  so that new customers are on a path towards becoming higher value customers.

Recommendations

With the completion of the preliminary analysis, the actual required work for this discovery exercise is now finished. It is now time to consolidate all this work into a comprehensive document . This document contains all the actual details concerning results which are used to support the many findings and insights generated from this exercise. But  despite the comprehensiveness of the report, it is of no use if it does not clearly lay out actionable next steps.  These actionable next steps are focused in two areas. The first area consists of the data strategy and what the organization needs to do to improve its information environment in order to more fully optimize the analytics discipline. The data strategy needs to be proactive in addressing not only the existing  analytical needs but also any potential analytics needs due to the changing nature of analytics.   

The second area focuses on what specific analytics activities should be undertaken in the first year and in what priority so that the company can effectively commence its analytics journey. The analytics strategy should be short-term in nature so as to provide an initial benchmark from which to commence this discipline. Keeping a  shorter term perspective on analytics just simply recognizes the underlying fact that things can change very quickly within organizations. Obviously, this short-term focus will identify activities that will yield the quickest wins in terms of ROI but also activities that will generate much learning.  It is this second component of learning which ensures that the longer term ROI is not being overlooked.   

By having both a data strategy that optimizes the use of analytics complemented with a set of analytics activities that begin the organization’s analytics journey, the organization has a roadmap or path in which to begin this journey.  With this kind of mindset, investing in a data discovery exercise is an easy decision as long as this process is undertaken with the due diligence and discipline as outlined in this article.

Using Analytics to create Rating Territories

One of the many areas of property and casualty risk analysis is the effective creation of rating territories based on some measure of risk.  In Canada and the U.S., geographical areas are grouped into territories based on claim risk. For example, postal codes in Canada and the comparable equivalent of zip codes in the  U.S. are grouped into rating territories.

As the use of analytics continues to adopt the  more scientific or mathematical techniques, insurance organizations would be remiss in not attempting to deploy these disciplines. However, before even adopting these disciplines, the typical rigors in developing an analytics solutions still need to be conducted . This means that the information or analytical file needs to contain the right data before even using any of the more advanced techniques.  In properly assessing risk, it is self-evident that any rating assessment should reflect the most recent information. Under this scenario, the determination of a proper post period is critical in the development of this solution. For instance, do we have enough data to look at the last 12 months or do we need to broaden the period to 24 or 36 months. For smaller organizations, this can be a significant factor as analysts need to have enough claims in order to build an effective solution. More claims for analysis in theory can translate to more robust models.

 Another factor in this determination may be understanding the cyclicality of claims within a given business. AB claims can often take more than 12 months to determine their exact payout. If loss cost is the key determinant rather than claim frequency, one would want to look at a longer timeframe beyond 12 months  in creating rating territories particularly if AB is a significant driver of overall loss cost. The compromise here is that we are sacrificing recency of information versus information that is more reflective of the specific business conditions of the company.

Besides this determination of the proper post period in analyzing claims, one needs to consider information that is considered extraneous to the solution. Often this  so-called ‘extraneous’ information refers to claims that are classified as catastrophic.  The inclusion of  auto claims that are due to a hailstorm as part of a solution will introduce information that is not really indicative of any behavioural tendencies. Instead these tendencies are solely due to ’Acts of God’. Yet, it is behavioural tendencies that we want to better understand and more importantly to action in future business initiatives. Grouping of geographic areas around behavioural  tendencies yields consistency when projecting this learning going forward. ‘Acts of God represent noise or variation in the data that should not be considered in any analysis. Accordingly, these kind of observations should be excluded from any insurance risk analysis whether it be the development of pricing models or the determination of new rating territories. 

Once this analytical  file is created, the ‘analysis’ portion of this exercise can commence. The information in such a file needs to contain only two fields of information with one field containing loss cost or loss frequency  while the other field contains the specific geographic area.

One creative approach is to use a technique that is traditionally used to develop models. CHAID which stands for Chi-Square Automatic Interaction Detection represents a non parametric approach to building models. Essentially, the solution is a decision tree whereby the nodes or branches represent segments comprised of business rules. Comparing the traditional parametric modeling such as equations with variables and weights assigned to each variable(parametric approach), CHAID’s solution of segments and business rules(non parametric) represent the model solution. Nodes or segments  can be ranked sequentially from the highest loss rate node to the lowest  loss rate node and policies can selected in this manner. In most scenarios,though, there will be more than 4 nodes which allows for more granularity regarding the selection of policies. 

However, one real significant advantage of CHAID is its data exploration capability. Through CHAID, certain character type variables can be grouped into meaningful segments based on some objective function. One obvious application is the grouping of character type data such as  geographic areas into meaningful segments or territories. Here the node output represents groups or segments as determined by the CHAID routine. Loss rate ,though, is the key metrics which drives this statistical output.

Certainly, the use of CHAID as a tool for territory assignment is not to imply that what was done in the past was incorrect. Due diligence has always been exercised by combining both the analysis of historical loss rate data as well as the domain knowledge of the actuary. But rather than discard what has been used in the past, we simply have another option which is more mathematically based on how geographic areas are assigned into territories.

Social Media Analytics: Three Perspectives

The analytics discipline within this channel can comprise a number of different perspectives. These three perspectives are:

• network analysis
• marketing attribution
• text mining  

In the area of network analysis, the analytics approach is somewhat similar to what is done within the telecommunication sector. There are often many deliverables that are related to this kind of analysis within this sector. But one such deliverable is the ability to identify influencers vs. followers. With this kind of knowledge, the telco can perform other analytics such as customer profiling  in order to determine what are the key telco customer characteristics that differentiate influencers vs. followers.

 This approach  can be mimicked in the social media world as each individual has a  network of contacts just like each telco customer has a network of calls that that have been made within a given period of  time. The key in determining whether an individual is an influencer vs. a follower is through examination of both the  breadth and depth of that individual’s network. Breadth refers to the number of contacts within the person’s network while depth refers to the level of engagement that each person has with all their contacts. Various metrics and indices can be calculated that allow the analyst to create an ‘influencer score’. A component of this type of analytics exercise would be to determine the threshold or benchmark from these metrics/indices that classifies someone as an influencer vs. a follower. This, of course, will vary from exercise to exercise.

The second area of marketing attribution attempts to determine how much ROI from a given marketing campaign can be attributed to social media. There may be no right answer or panacea to this challenge. Unless there is some direct method whereby a given person logs onto a certain web page and registers for a given product or service within the social media campaign, it is very difficult to determine direct dollars that can be attributed to social media. Yet, even this functionality exists, one could argue that the  non registrants participating in some fanbook contest might be contributing to ROI by purchasing at a store, through TV or direct mail. These  same kind of challenges exist for the mass advertising industry except the key advantage with social media is that  we do have the denominator in determining the number of people that clicked onto some fan page or contest. Given these limitations, all analysts can do is to provide general direction in whether the campaign created additional engagement and whether or not this additional engagement translated into additional dollars. On the engagement side, metrics such as # of people logging onto a fan page or contest plus how long they are on the site  can represent some kind of proxy and comparisons of these metrics  can be made to other social media campaigns to determine the level of success. By the same token, anecdotal analysis of social media campaigns overtime can determine the overall impact on incremental sales. Classification of campaign periods into high, medium, and low social media marketing allow executives to see the general impact of social media marketing on sales. However, one cannot directly attribute the ROI  back to a specific social media campaign. All we can say is that the campaign generated an improvement in engagement relative to other campaigns. The leap of faith for executives is that this improvement in engagement translates to incremental dollars but we don’t know the precise number.

The third area of text mining presents the most exciting opportunities for marketers in being able to actually analyze what is being said within this medium. The ability to identify key themes or topics as well as sentiment allows marketers to craft communication stategies that better address what is being discussed within this space. At an even higher level, brand strategies can be created that better reflect the needs and wants of its core customers. As an analytics practitioner, most of our deliverables end up producing marketing solutions that provide better targeting of customers. The traditional deliverables of analytics have always yielded solutions that are of a more tactical nature. Text mining in the social media space offers the analyst opportunities to build solutions that are of a more strategic nature which ultimately will have higher profile within any organization. The strategic nature of these type of solutions just serves to reinforce the ever-growing importance of analytics as a key business discipline and more importantly a key competitive advantage within  a very dynamic business environment. 

Black Box Analytics

Organizations are now aware of the significant contribution that predictive analytics brings to the bottom line. Solutions, once viewed with skepticism and doubts, are now embraced by many organizations. Yet, from a practitioner’s perspective, certain solutions are easier to understand than others. The challenge for the practitioner is provide a level of communication to the business end user such that the end user has the requisite degree of knowledge when using the solution. What does this mean? From the business end user’s standpoint, this means two things:

1)      The ability to use the solution in order to attain its intended business benefits

2)      Understanding the Business Inputs

The ability to use the solution in order to attain its intended business benefits

The use of decile reports and gains charts have been discussed in many data mining articles and papers and do provide the necessary information in making business decisions when using these solutions. The ability to rank order records with a given solution yields tremendous flexibility in understanding the profit implications under certain business scenarios. As businesses increasingly adopt predictive analytics, decile reports and gains charts are being recognized as the critical solution outputs in providing the necessary information for key business decisions.   

 

Understanding the business inputs

This area represents the most challenging area as it is here that the analyst can really ‘get under the hood of a solution ’ and actually demonstrate what are the key business inputs to the model. Yet, as the world of predictive analytics has evolved , so have the techniques. Highly advanced mathematical techniques using approaches related to artificial intelligence, which imply the ability to detect non linear patterns, present real challenges in trying to explain the key components and triggers of a model. What do we mean by this?.  Suppose I have a distribution where a variable like tenure has a tri-modal distribution(3 mode points within the distribution) with response. As the practitioner, how do I meaningfully explain the trend to the business user other than telling him or her to simply rely on the output as presented in the overall equation.  Yet in many of these advanced techniques, there are a number of variables which exhibit this high degree of non-linear complexity. With some of these advanced techniques and software, equations are not even the solution output. Instead the practitioner is presented with output in the form of business rules. Further complicating this scenario is the fact that many of these non-linear variables may have interaction with each other whereby the interaction between multiple non-linear variables represents an actual input to the solution. These type of complex inputs pose real difficulties to the practitioner if he is trying to educate the end users on what the model or solution is comprised of.  The practitioner’s typical response is that the   solution is a  ‘BLACK BOX’.

Do I believe the business inputs?

Yet, besides the communication challenge of ‘black box’ solutions, the second challenge is believability. Will the business community really believe that the equations and variables as seen in a black box solutions are simply superior in explaining variation than in the more traditional linear and log-linear techniques. Mathematically, we may be able to explain how a given input best explains the variation. But can we explain it in business sense. For example, if there is a linear relationship between tenure and response, it is far easier to explain that higher tenured people are more likely to respond thereby explaining why tenure has a positive coefficient within a given model equation . Conversely, assuming a linear relationship between income and response, we may conclude that higher income people are less likely to respond thereby explaining the negative coefficient of income within the overall model equation. But the ability to explain the impact of tenure on response becomes more difficult if the trend is a curvilinear one with multiple modes within the distribution. Tenure in this case would be captured in some kind of complex polynomial function that is very difficult to explain in business terms. This situation can even be more complicated if the solution is derived from machine learning as the resulting business rules do not look at trying to explain an overall trend of tenure with response. Instead, the tenure relationship is captured by attempting to optimize its relationship with response along  particular points of a distribution.

With these type of black box solutions, user-driven parameters provide the necessary flexibility to the practitioner in delivering a variety of different solutions. The practitioner can alter the parameters in such a way as to deliver an almost  ‘perfect’ solution since these  these high-end mathematical solutions will attempt to explain all the variation even the variation which is truly random. Of course, this is the fundamental problem with ‘black  box’ solutions and hence the dire need for validation.          

Does it matter that black box solutions are difficult to understand?

Many pundits will state that it does not matter if a proper validation environment has been created. There is merit to this argument in that all predictive analytics practitioners are ‘scientists’ at heart. So the use of more advanced techniques to potentially deliver superior solutions just makes scientific sense. Yet, putting on my practical business hat, one could  argue that the use of these highly advanced techniques are being deployed in environments where the random or error component of variation is very large. The consequence to this is that the level of variation that can be truly explained is quite small which is the reason why linear and logistic techniques in most cases will work just as well as the more advanced techniques. Of course, the key in really assessing the superiority of one solution over another is validation and the resulting gains charts where one can observe how well one solution predicts  behaviour over another solution. The notion of a robust validation environment is one area where practitioners and academics are in complete agreement.

Yet, this inability to explain  the key business inputs can present a barrier to broader   acceptance of using these tools throughout the organization.  Businesses need to have this so-called ‘deeper’ understanding because of the measurement process used to evaluate the performance of a given solution. This process cannot occur effectively if there is a gap in understanding what went into the solution. Of course, this is not a problem if the solution works perfectly. But what happens when solutions do not work. Under these scenarios, deeper forensics is required to understand  what worked and what did not work  which certainly implies a deeper understanding of the key business inputs. This so-called  comprehension gap in being able to effectively measure black box solutions is the reason why most organizations opt for the simpler linear or logistic type solutions. In this type of  environment, the notion of ‘less is more’ is very applicable here since these non black box  solutions are more easily  understood and most importantly can be analyzed in a very detailed manner particularly when these solutions fail to produce the expected results.

The ABC's of Analytics

In our education system, we often look at the two key core disciplines of reading and

mathematics as being essential requirements in any successful career. Yet in both these disciplines, there are basic technical fundamentals that need to be mastered before one can be successful in either of these disciplines.

In the area of mathematics, one needs to master the fundamentals of addition, subtraction, multiplication, and division. In reading, the student needs to master the alphabet or ABC’s before he or she can even attempt to read a story.

The reason that I bring up this notion is that this type of process for mastering a particular discipline is no different than in attempting to master the world of analytics. In our world of  analytics our ABC’s  is DATA,DATA, and DATA. Analytics practitioners must not only have a conceptual understanding of data but a ‘ roll up your sleeves’ approach towards data. The ‘roll up your sleeves’ approach affords the analyst a much deeper understanding of the data as he/she is then better able to handle all the detailed nuances which is a fact of life in the data world. We have all heard  the phrase ‘The Devil is in the Details’. Since data is detail, this phase is easily transformed into the phrase that I often use with clients ‘The Devil is in the Data’.

In any environment, be it the digital or offline arena, analyst need to have a process of better understanding data. How is this done? The first step is the so-called data audit which will be the focus of this discussion. The discussion of data and its importance in data mining and analytics does not end here since a more full blown exhaustive discussion could easily result in a book.

Regardless of whether the data represents new information that is unfamiliar to the analyst or even if it represents a familiar existing data source, a process such as a data audit process needs to take place that allows us to better understand the data at a detailed level. Let’s talk about the first situation where the data represents new information.

The first task in the data audit process is to obtain an initial glimpse of the data by observing a random set of  10 records and all their accompanying fields from a given file.

This simple task accomplished two objectives. First, it determines the fact that we have loaded the data correctly into the existing software analytics application which we are using. Secondly, we begin to determine  our initial ‘feel’ of the data environment. In other words, are we dealing with a very simple data environment or one that is quite exhaustive. Looking at the number of files as well as the number of fields on each file along with examples of the values of these fields begins the analyst’s quest towards a very deep detailed understanding of the data environment.

After this initial glimpse, further data diagnostics are conducted which allow the analyst to determine the number of unique values and number of missing values for each field on each file that is received by the analyst. Further diagnostics such as frequency distributions allow the analyst to better understand the distribution of values within a given field.  

With the information from these data audit reports, the analyst can then begin to determine how to create the analytical file which essentially represents key information in the form of analytical variables. The so-called objective in this exercise is to have all meaningful information at the appropriate record level where any proposed solution will be actioned. In most cases, this represents the customer or individual but is not necessarily confined to that level. For example, pricing models for auto insurance are actioned at the  vehicle level while retail analytics might focus on decisions that are actioned at the store level. More about how data and information is used to create this all-important analytical file will be discussed in future blogs. 

The data audit process should not only be conducted on new information or data sources but also on known data sources that serve as updated or refreshed information for a given analytical solution. Assuming that the refresh or updated data is fine because  you have gone through the data audit process when you first received a new data source for the first time reminds me of a high school teacher’s comments about the word ASSUME. Using the word ASSUME means that U make an ASS out of ME. This phrase has extreme relevance in the data world  in that any data(new or updated) that is being used to generate business solutions needs to have some checks and controls. Although the data audit process used in processing a new data source can be quite comprehensive, a less rigorous process may be used when assessing refresh or updated data. This may be simple as checking the number of records and fields that are received as well as some stock audit reports that look at means or averages of key variables which  are unique for a given client. In any event, the purpose of even doing a shortened version of this data audit report is to establish a means of identifying problems or issues with the data that is currently being used by the analyst.

Data Audits, despite producing the least glamorous type of reporting information, are necessary prequisites in any analytics process. If one believes that all analysis starts with data, then one must exercise extreme diligence and respect for the data. This kind of attitude towards data helps to foster an appreciation of the many kind of data nuances that can appear in a project. If analytics is going to be successful, data audits represent the first critical task within any given project. Future discussions will outline the how this analytical file is created once the data audits are completed.