Originally published in Forbes
As I covered recently, Jev and its competitors – a new breed of “judgement models” – have recently taken the AI world by storm. They provide the power to leverage modern foundation models for predictive AI projects. This includes the capability to perform zero-shot outcome prediction, eliminating a major barrier to entry for predictive AI: training a customized predictive model. Jev’s creator announced yesterday that, only a few weeks after launching, it raised $870M at a valuation of $7.5B and that 29.4% of the F500 are already using Jev.
But most Jev users make a grave error: They measure only technical performance such as accuracy rather than business performance. For example, the talent marketplace Jack & Jill replaced a leading foundation model with Jev within its candidate-matching pipeline, citing comparable performance in terms of the common technical metric AUC. Such metrics fail to directly measure how good a Jev-based system is for business.
If you aren’t measuring business value, you aren’t pursuing business value.
Accuracy and AUC tell you almost nothing about AI’s business value. Say you are predicting which customer messages convey an intention to cancel. 20% of the messages are positive (yes, those that convey this negative sentiment are considered “positive”) and 80% are negative. If an AI system predicts “no” for every customer, it achieves an 80% accuracy, even without correctly identifying a single at-risk customer.
Instead, measure the potential money you’d make by using the model.
Using a model is the only way it realizes value. Operationalize it, deploy it, launch it or put it to production. All different ways to say the same thing. You need to determine what the system will do when it receives a customer complaint such as:
“I am writing to report a significant problem with the centralized account management portal, which currently appears to be offline. This outage is blocking access to account settings, leading to substantial inconvenience. I have attempted to log in multiple times using different browsers and devices, but the issue persists.”
If this is for B2B, such as a payroll service, you could respond with a retention offer of, say, $250 off their subscription. That might pay off well, averaging $700 of net future earnings from customers correctly targeted and convinced to stick around.
Of course, the reason you’re using AI in the first place is that detecting (predicting) a customer’s intent to leave based on such a message is fuzzy. It’s a judgement call. It’s probabilistic. It’s a job for predictive AI.
Accordingly, judgement models such as Jev – and machine learning models in general – respond with a probability. That’s potentially useful – if you can determine how to best threshold on its probabilities. Do you invest the $250 discount when the probability of customer cancelation is above 80%? Or above 60%? Or what? Each wrongly targeted discount costs you $250 – the customer wasn’t going to leave and you gave them $250 off anyway.
Simply picking a threshold that “feels right” is a grave error. Why do all this data-driven AI work only to top it all off with a subjective “gut check”? It’s true that some thresholds might feel good – 90% seems pretty darn confident – but the business realities are situational and only by playing them out arithmetically can you drive the decision with true objectivity. Yet even interactive Jev explainers have you focusing on setting the confidence threshold without considering the resulting business value of the operations that would be driven on that basis.
There’s a fundamental, straightforward view that’s missing from most Jev projects – and, in fact, from most predictive AI projects in general: A profit curve.
The potential earnings from retaining at-risk customers depends on how many you invest in – represented by the X-coordinate. Think of the most risky customers on the left, and least risky on the right. If you draw the line at 10%, that corresponds with offering a retention discount to the most risk 10% – according to Jev’s probabilities. The estimated net earnings at that position is about $4.5 million. This is calculated over 10,000 customer messages from a public synthetic testbed, scaled to an assumed 500,000 messages, taking into account that each correctly targeted offer provides a $700 average win, and each incorrectly targeted offer costs $250 in lost revenue.
More pain, more gain. If you and your colleagues are willing to be more aggressive, you can maximize the earnings by extending the offer to 17%, in which case you’ll profit an estimated $5.2 million. By delving further down the list, this means you’re providing the offer to less risky customers, which means more frequent incorrect targeting. If the bottom-line profit is the only consideration, that tradeoff is worth it. Indeed, the profit curve (more generally, value curve) of a predictive AI projects usually looks something like this. There’s a sweet-spot Goldilocks zone: Don’t treat too few and don’t treat too many.
In some circumstances, you may want to be even more aggressive. Say your near-term priority is growing the customer population. Retaining customers is often a much more cost-effective way to do so than acquiring brand new customers “off the street.” So, if you go all the way to 46%, the estimated profit is around zero. This would be the best approach for retaining the most customers at net zero cost.
A business metric – such as the financial win – must be the project’s driving force. Not accuracy. Predictive AI projects have, for decades now, measured predictive performance almost exclusively in terms of technical metrics that are at best poor proxies for business values, such as accuracy, precision, recall and a slew of others familiar to all data scientists. But if you aren’t measuring business value, your project can’t be pursuing business value – and, perhaps most importantly, your project can’t be selling the potential value to business stakeholders that must understand the value in order to authorize deployment. For that reason, most predictive AI projects fail to launch.
Why are business metrics so rare in predictive AI? After all, they’re obviously the originating motivator for most any project. The reason is, a business metric such as profit depends on business factors that are subject to change or uncertainty. In the simple example covered so far, each case’s cost/benefit is set at $250 and $700, respectively. But those are at best only solid estimates. A slightly more sophisticated approach that would better estimate the upside of retaining any given customer would vary each upside based on that customer’s history – that is, based on the expected value for retaining them specifically. Some customers are more valuable to retain than others.
Uncertainty brings up trust issues. Since the value calculation hinges on uncertain factors, it can be hard to trust. Some of your colleagues will inevitably express skepticism, and the potential value of operationalizing your predictive AI scheme may begin to feel obscured by uncertainty.
But we can’t let uncertainty derail these projects. Predictive AI is the very act of playing the odds better. It is the data-driven streamlining of organizational micro-decisions. If we hid from opening a can of “uncertainty” worms, we’d be stuck with the clearly sub-optimal practice of driving decisions by the gut.
The antidote to this uncertainty hurdle is to interactively explore the effects of business-factor “wiggle room.” For example, the figure below shows how things look if we assume the average win for rightly targeting a retention offer is $800 rather than $700.
At the previously-optimal point of targeting the 17% most risky, the profit jumps from $5.2 to $8 million. And yet, that would still miss some opportunity, since now the peak estimated profit would be $9.3 million, by targeting almost 30%. Within this “wiggle room” range, there’s uncertainty – but whichever setting you think is more precise, we’re compelled to deploy. We’re seeing viable business value rather than only presuming or hoping for it.
The profit curve – more generally, “value curve,” since profit isn’t always the main KPI – is only one piece of the business console needed for running a predictive AI project. Zooming out a bit, here’s some of the bigger picture:
There aren’t a million things to consider, but there are more than a couple. Now we’re comparing three different models – in this case, all variations on Jev usage:
In the zoomed-out console shown, some other options and considerations have also come into view. The business factors and their “wiggle room” are shown at the very bottom, where sliders allow you to change (“wiggle”) them. Changing their setting affects changes to the shapes of the three overlapping profit curves.
Finally, the top shows the value of three other business metrics besides profit: The number of customers given the offer, the number of times offers were rightly done and the number done wrongly. Each such value depends on where the decision boundary is set, namely, the horizontal position within the profit curve. As shown, there is a decision boundary, a vertical line on the profit curve – moving it right or left alters the values in the table accordingly.
You can access this console setup immediately here and try it out interactively, moving the sliders and decision boundary. Just find the “Jev” example on that webpage and click.
Taking predictive AI’s business value seriously means making the value explicit, and interacting with potential deployment options and their ramifications. Let’s get concrete with value and drive the use of Jev – and of all predictive models – according to value. With that, we’ll welcome a long-overdue industry make-over that predictive AI demands. Then we’ll have finally achieved the maturity and professionalization that this tech deserves.
About the author
Eric Siegel, Ph.D., is a former Columbia University professor who helps companies deploy machine learning. He is the cofounder and CEO of Gooder AI, the founder of the long-running Machine Learning Week conference series, the instructor of the acclaimed online course “Machine Learning Leadership and Practice – End-to-End Mastery,” executive editor of The Machine Learning Times, and a frequent keynote speaker. He wrote the bestselling Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die, which has been used in courses at hundreds of universities, as well as The AI Playbook: Mastering the Rare Art of Machine Learning Deployment. Eric’s interdisciplinary work bridges the stubborn technology/business gap. At Columbia, he won the Distinguished Faculty award when teaching the graduate computer science courses in ML and AI. Later, he served as a business school professor at UVA Darden. A Forbes contributor, Eric publishes op-eds on analytics and social justice.
Eric has appeared on Bloomberg TV and Radio, BNN (Canada), Cool Science Radio, Israel National Radio, Motley Fool Radio, National Geographic Breakthrough, NPR Marketplace, Radio National (Australia), and TheStreet. A Forbes contributor, Eric and his books have been featured in American Banker, BBC, Big Think, Built In, Businessweek, CBS MoneyWatch, CDO Magazine, Contagious Magazine, The European Business Review, Fast Company, The Financial Times, Fortune, The Globe & Mail, GQ, Harvard Data Science Review, Harvard Business Review, The Huffington Post, The Los Angeles Times, Luckbox Magazine, MIT Sloan Management Review, The New York Review of Books, The New York Times, Newsweek, Quartz, Salon, The San Francisco Chronicle, Scientific American, The Seattle Post-Intelligencer, Trailblazers with Walter Isaacson, The Wall Street Journal, The Washington Post,WSJ MarketWatch. and ZDNET.
