Originally published in Forbes
When Henry Castellanos first presented his machine learning model to his company’s executives, he found himself fighting off a certain self-doubt that is so common among data professionals, it’s almost universal.
On one hand, his model looked great. It did a sturdy job predicting which dental patients would fail to show for an appointment, so that a medium-large chain of dental offices could strategically double-book high-risk time slots – much the same as airlines overbook flights. The project promised healthy returns. If Henry’s model predicted well enough, the business could dramatically reduce the high cost of empty dental chairs, while largely avoiding the repercussions that result when two patients show up for the same appointment.
But on the other hand, Henry’s model sucked – in comparison to a magic crystal ball. It was about two times better than random guessing, but a magic crystal ball would have well outpredict it by flagging no-shows and only no-shows, without error. To be specific, if you used Henry’s model to flag the top 10% most risky patients, those most likely to be no-shows, then about half would turn out to actually fail to show. That is about two times better than random guessing (since about one quarter of all appointments were no-shows).
It didn’t necessarily bother Henry that a hypothetical clairvoyant model would defeat his. Like pretty much all credentialed data scientists, he knew that crummy models are valuable – that magic crystal balls are only a fantasy and the best you can usually hope for from ML is predicting better than guessing. And yet, predicting better than guessing is generally more than sufficient to improve an operational “numbers game,” delivering a strong bottom-line win.
But Henry still had to sell the business on actively using his model.
As the meeting began, Henry could just feel it: The numbers were sound, but they weren’t going to convince the executives.
“Ultimately, I didn’t really feel confident,” he told me during a video interview. “I didn’t have a direct answer to the question of whether or not my model would be actually valuable. I wondered how I could really communicate what using the model would mean operationally and financially.”
Henry had technically validated his model, which is an industry norm that’s generally – but wrongly – considered sufficient. The internal tug-of-war that he then experienced is endemic to the predictive AIprofession. In this field of technology, we’re taught to make sound models, but then to screen them only with regard to their relativepredictive performance – that is, their technical performance – without any data-driven estimation of the absolute business value they would deliver if used.
This standard but flawed practice fails to heed an obvious universal maxim: You can’t sell something without first understanding the customer’s problem and seeing things from their perspective. When it comes to predictive AI projects, what’s being sold is the use of a model. And a data scientist’s business-side counterpart just doesn’t care that a model predicts “two times better than guessing.”
Instead, they care about money – or other KPIs.
To be clear, as a nerd myself, I do care indeed about that kind of technical measure, which confirms relatively good performance. It means that the model works as it was trained to do. ML has found patterns that hold in general – now encoded as a model that can be employed to tip the odds in the numbers game known as “doing business.” Predicting two times better than guessing means the model has a lift of two. Lift is one of a handful of standard metrics the ML industry uses to evaluate models. Other metrics include precision, recall, F-score and AUC.
But any of these highfalutin metrics alone fail to serve the customer and fail to serve the business. They all accomplish a variation on the same theme: They tell you that a model performs relatively well – yet reveal almost nothing about its potential absolute value. They’re helpful, but not sufficient.
And so, by sticking to these standard metrics, data scientists usually fail to answer the most obvious question about the model they’re trying to sell: “How good is it?” Without pinning model performance to value, the answer to this question of model goodness remains subjective. Without an estimation of business value, you could just as easily argue that a model is “bad” as it is “good.”
What an irony: The most formal, technical metrics leave things fuzzy. Without further insight, they leave the buyer’s decision to the mercy of whim and whimsy. Usually, reason prevails and the underinformed decision maker scrubs the launch. The model is never used and the project realizes no value.
This dire mishap persists and persists. After decades of advancements and numerous waves of hype, predictive AI is still stuck, routinely following a process doomed to failure:
After this typical data scientist experience of a nagging feeling that their sales pitch is missing something – and an initially dazzled but ultimately lukewarm reception from the stakeholder (aka customer) – Henry made a decisive, fundamental shift. He moved to showing his executive something that matters: profit. The model was projected to create an additional $500k in annual revenue.
Henry provided visibility into exactly what the model was expected to do. By double-booking the appointments that it flagged, the business would avoid a certain number of empty dental chairs, saving hundreds of dollars on each occasion. This process would also sometimes wrongly double-book, each time causing inconvenience as well as some monetary loss (such as losing a dissatisfied patient) – but the bottom-line payoff looked great.
Walking into this meeting was an entirely different experience. Henry felt the self-assurance of a professional armed with the basis to land a sale. “I felt like this provided the validation that I needed to have the confidence to go into meetings and say, ‘This model would make money.’” Henry’s boss – and his boss’s boss – were psyched.
The lesson is clear: Data scientists, that nagging feeling, a certain lack of confidence, is telling you something. The solution to a business problem isn’t just to predict relativelywell. The solution is to predict well enough to demonstrate it’s absolutely valuable. When data professionals take the old but prevalent route and don’t provide visibility into the potential value, they’re very unlikely to sell the business on using their ML model.
For more detail about Henry’s project, view this video webinar, demo and interview.
About the author
Eric Siegel is a leading consultant and former Columbia University professor who helps companies deploy machine learning. He is the founder of the long-running Machine Learning Week conference series, the instructor of the acclaimed online course “Machine Learning Leadership and Practice – End-to-End Mastery,” executive editor of The Machine Learning Times and a frequent keynote speaker. He wrote the bestselling Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die, which has been used in courses at hundreds of universities, as well as The AI Playbook: Mastering the Rare Art of Machine Learning Deployment. Eric’s interdisciplinary work bridges the stubborn technology/business gap. At Columbia, he won the Distinguished Faculty award when teaching the graduate computer science courses in ML and AI. Later, he served as a business school professor at UVA Darden. Eric also publishes op-eds on analytics and social justice. You can follow him on LinkedIn
