In 15 years of experience, I have seen countless predictive models—but very few useful ones. Most take too long to build, and then sit unused on the shelf. The model becomes a memory, and a bad one at that. Behind the scenes, countless hours were spent figuring out what to model, getting agreement on definitions and parameters, building the model, optimizing it and preparing for the presentation to senior management. Then “boom”: something goes amiss, all the effort goes down the drain, and yet another model meets its arch nemesis—the shelf. Worse, the analytics group’s reputation suffers for “not producing anything”.
Use Code PATIMES15 for 15% off a two day pass or combo pass. (Excludes workshops & All Access)
So why do some models succeed and many fail? What are the traits of the successful model and how can you build one?
Three levers to build successful predictive models
- Successful predictive models need pre-work. I am a big proponent of using simpler methods, such as correlation analysis, to solve a business problem before moving to more complex methods, such as regression and decision tree. Pre-work with simpler methods not only delivers low hanging fruit quickly while the more complex method is being considered, but it also qualifies the opportunity. Pre-work determines the model’s likelihood of delivering greater results over the simpler method and the potential for an ROI.For example, consider the case of a company’s customer churn analysis. A simpler method—a correlation analysis—reveals the top drivers of churn and addressing those brings the churn percentage close to the industry average. Since very few strong hypotheses remain on possible causes of churn, likely most of the churn now is from uncontrollable factors. In this case, predictive modeling would likely be a wasted effort and should not be undertaken. Unfortunately, most analysts don’t do this pre-work and immediately begin working on a model doomed for the shelf.
- Successful models are built with organizational buy-in. You can build the best model with significant uplift and revenue opportunity, but unless the organization is ready to adopt and operationalize the model or the insights, those gains can’t be realized. Also, predictive models are resource and time-intensive and, to top that, they deteriorate fairly quickly, requiring constant improvement. Given such a high investment, model building should not be undertaken unless the right stakeholders are ready to operationalize it and a proper action plan is already laid out to launch the model and its insights.
- Successful models are built by strong modeler following structured approach. Building predictive models with high accuracy and low misclassification is not trivial. A successful modeler uses Aryng’s 5-step BADIR framework (below) or a close variant.
Five steps to successful model building using the BADIR framework. BADIR is an acronym for the below 5 steps:
- Model building starts with clearly understanding the Business question. What is the model trying to solve for and how is it measured? This needs to be arrived at by input from all the stakeholders and all of them need to agree with the goal for the project.
- Then, brainstorming hypotheses with stakeholders generates the most likely predictors for the model. Hypotheses inform the Analysis plan, which also includes the analysis goal, the methodology, sampling strategy, data specification, timelines, resources and milestones. All the stakeholders need to agree to the analysis plan before the project moves forward. This key step ensures that the model will be ready to be acted upon when the results roll out.
- Next, Data is pulled, cleaned and validated. Surprisingly, this step may take more than 50% of the project time, depending on the maturity of the organization.
Piyanka Jain, Founder, Aryng
Originally published at www.forbes.com
