Machine Learning Times
Machine Learning Times
Why You Must Twist Your Data Scientist’s Arm To Estimate AI’s Value
 Originally published in Forbes, June 11, 2024. If you’ve...
3 Ways Predictive AI Delivers More Value Than Generative AI
 Originally published in Forbes, March 4, 2024. Which kind...
AI Success Depends On How You Choose This One Number
 Originally published in Forbes, March 25, 2024. To do...
Elon Musk Predicts Artificial General Intelligence In 2 Years. Here’s Why That’s Hype
 Originally published in Forbes, April 10, 2024 When OpenAI’s...

6 days ago
Why You Must Twist Your Data Scientist’s Arm To Estimate AI’s Value


Originally published in Forbes, June 11, 2024.

If you’ve ever had a data scientist make a machine learning model for you, you probably first experienced excitement, followed by bewilderment.

The potential power is awesome. In its enterprise application, predictive AI is the antidote to information overload. Too many prospective customers? Use a predictive model to prioritize them from most likely to buy down to least likely. Too many web pages that might have the info you need? Google Search uses a model that orders the search results, placing those web pages most likely to pertain at the top. Too many buildings to inspect for fire hazards? The New York City Fire Department’s model tells them which should be prioritized as high risk.

Eventually, you will sit down with your quant to hear how her number crunching went. “The predictive model is great!” she exclaims. “This is a nice dataset, so we ended up with an AUC of 0.83!”

“What’s that?” you’re forced to ask.

“Oh, I mean the area under the receiver operating characteristic curve.”

“Right… so, is 0.83 good?” you ask, hopefully.

“Totally! This model is significantly better than not having a model. It’s, like, much better than random guessing.”

This ML professional is providing you concrete numbers, but she’s not helping you decide what business move to make. You ask, “Okay, so, if we use the model to target our next direct mail send, how much money will it make us?”

“Oh, I’m sure a lot. I mean, that way you’ll be contacting customers more likely to buy, so how could it not?”

It’s strange: Her expertise is credible and you don’t doubt anything she’s saying, but you still feel like you have to corner her. “Bottom line: How much will the marketing profit improve if we only contact the top 20% of our prospect list?”

“Oh… yeah, I could totally estimate that. Let’s meet again tomorrow.”

The next day, you get your answer. “The profit would be an estimated $1.6 million, which is much better than if you did an untargeted mass mailing to the entire list, which would put you in the red by almost $800,000.”

This sounds good, but you feel like, after pressing so hard, you’ve only been given one peek through a keyhole. You need broader visibility. What if you marketed to the top 15% or 35%? Would either of those be better?

Let’s jump to the resolution that would make this awkward yet common conversation unnecessary. The full view that you need is called a profit curve, a rare yet fundamental view of predictive AI’s potential value.

A profit curve estimating the profit of a marketing campaign targeted with a machine learning model.

A profit curve estimating the profit of a marketing campaign targeted with a machine learning model. The horizontal axis represents the number of customers contacted—ordered from most likely to buy if contacted down to least likely—and the vertical axis represents the marketing campaign’s profit. The gray vertical line represents the decision threshold. If set to 30% (as shown), the campaign’s profit is estimated to be $1.7 million, depending on the settings at the bottom: an average of $30 for each positive customer response and a $10 cost per contact. This example is based on the data set used for The Second International Knowledge Discovery and Data Mining Tools Competition. Thanks to Dean Abbott for the predictive model used in this example.

The profit curve shows you the range of options offered by a predictive model. With the example curve shown above, 30% is the best stopping point if your aim is to maximize profit. Or, if immediate-term profit isn’t your only priority and, instead, you’d like to market to many customers, then 84% would be a break-even point—you could contact that many for free, which would arguably be a better choice than contacting all of the list at a loss of almost $800,000, the negative profit shown at the far right side of the curve. Across this range of cut-off options, the position you select is called the decision threshold.

As simple and vital as it is, this visual is rare. No matter the pedigree of your quant, you probably won’t get it unless you press hard. Why? It’s not in the typical data scientist’s wheelhouse. It’s not part of the culture, nowhere to be found within their technical books or university courses. Perhaps more importantly, the ML software they use probably doesn’t have it built in—at least not in a useful, extensible way.

After all, such a business-focused vantage sits outside a data scientist’s proper scope of responsibilities. They know how to navigate technical pitfalls as they guide their machine to learn from data how to predict. And they know how to report on the success of their modeling work—using metrics that report on a model’s pure predictive performance. But what about the potential value of the model in straightforward business terms like profit? Estimating that is someone else’s job, the domain of business leaders who, ideally, will connect the model to their operations, improving the operations with the model’s predictions. The thing is, those leaders consider this a technical step that belongs to the data scientist, and rightly so.

Because visibility into the potential business performance of ML models is rarely granted, enterprise ML projects routinely fail to deploy. This is part and parcel of the stubborn business/tech divide, a disconnect between stakeholders and quants wherein they’re not speaking the same language. But without bridging that gap, it’s almost impossible to make an informed decision about whether and precisely how to deploy an ML model—be it for targeting a marketing campaign or improving any other kind of large-scale operation.

As a business stakeholder, don’t settle. For every ML model that you consider deploying, make sure that your data scientists provide you with a full view of its potential business value—reported in terms of straightforward business metrics such as profit.

Follow me on Twitter or LinkedIn. Check out my website or some of my other work here.

About the author
Eric Siegel is a leading consultant and former Columbia University professor who helps companies deploy machine learning. He is the founder of the long-running Machine Learning Week conference series and its new sister, Generative AI Applications Summit, the instructor of the acclaimed online course “Machine Learning Leadership and Practice – End-to-End Mastery,” executive editor of The Machine Learning Times and a frequent keynote speaker. He wrote the bestselling Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die, which has been used in courses at hundreds of universities, as well as The AI Playbook: Mastering the Rare Art of Machine Learning Deployment. Eric’s interdisciplinary work bridges the stubborn technology/business gap. At Columbia, he won the Distinguished Faculty award when teaching the graduate computer science courses in ML and AI. Later, he served as a business school professor at UVA Darden. Eric also publishes op-eds on analytics and social justice. You can follow him on LinkedIn.

Comments are closed.