How To Overcome Predictive AI’s Everyday Failure

Stakeholder skepticism about machine learning often rings true: If the data scientist hasn’t measured the potential value, then how could the project be pursuing value?

Eric Siegel

Executives know the importance of predictive AI. As Unilever CDO Morgan Vawter wrote, “Its practical deployment represents the forefront of human progress: improving operations with science.”

But there’s bad news for data scientists: Your predictive AI project will probably fail. Your customer probably won’t operationalize the machine learning model you deliver. They won’t use it, act on it or integrate it. Sadly, most models developed for deployment wind up on the shelf.

But you probably already knew that. A plethora of industry research and anecdotal wisdom has let that cat out of the bag.

And yet what most data professionals haven’t come to understand is the true reason why. After all, shouldn’t a great model be a sure bet to deploy?

No. It won’t deploy because you haven’t closed the sale. When you deliver a model, you haven’t necessarily finished selling, no matter how much of a done deal it may seem to be. Until your model actually deploys, you must continue to actively sell it to stakeholders – in compelling, concrete business terms like improved profit – even if they’ve already agreed, signed and paid.

Standard AI Metrics Don’t Launch Models

Standard technical metrics are “fundamentally useless to and disconnected from business stakeholders.”

“The most important metric for your model’s performance is the business metric that it is supposed to influence.”

“Don’t show a confusion matrix to executives!”

“Nothing flies without KPIs.”

Only a concrete projection of value compels a business. Your customer and other decision makers may well have been excited about the project’s intent at its outset. But without visibility into the business value – a view that plainly displays how the model will drive decisions and the expected value to be gained – you’ll ultimately begin to hear excuses not to deploy. Put another way, without the enthusiasm that only a bottom-line promise can produce, the project will be among those first cut when the next inevitable financial crunch arrives. People don’t buy what they don’t understand.

As with any operational improvement, with or without analytics, a business can’t move forward until there’s a credible estimation of how much it’s going to improve those operations – a calculation of the business improvement you stand to gain – in straightforward terms like profit or other KPIs.

Yet most predictive AI projects don’t move beyond standard technical metrics – such as precision, recall, AUC or F-score. These represent the data scientist’s training, tools and comfort zone. And they serve well to establish model soundness, assessing whether the model performs relatively well, substantially better than guessing. If so, the model is potentially valuable.

But these metrics provide little to no insight into how valuable the model would be if used. They are arcane from the standpoint of business professionals and business objectives.

Technical Performance Alone Cannot Sell The Value

You may be the technical heavyweight in the room, but if you’re presenting only standard technical performance metrics, you’re committing a cardinal business sin and deep down your colleagues know it: You’re proposing a systematic operational change with no concrete estimate of its upside.

After all, when you build something, you’ve got to check out how good it is before you use it. You can’t launch a new rocket until you’ve stress-tested it in its intended usage, according to the KPIs that matter. Without a credible estimate of the potential value, the launch would be a shot in the dark. Sensible decision-makers would scrub the launch. Indeed, most predictive AI deployments are scrubbed.

No matter how advanced your analytical method may be, the decision maker’s gut skepticism rings true: If you haven’t measured the potential business value, then how could the project be pursuing business value? They may not feel that they hold the “tech authority” to say so, but they’ll find any of a myriad reasons to not move forward with operationalization.

Instead, make it a no-brainer for your customer: Sell the potential business value. This will give decision makers essentially no choice but to deploy, and will give the fruits of your number crunching a chance to realize a business impact. Everyone already understands, in general terms, the potential value of driving decisions with model predictions – that’s what has brought the project this far. Now it’s time to follow through on your business-oriented swing by establishing how much business value the deployment stands to deliver. This way, you’ll knock it out of the park.