You Must Address These 4 Concerns To Deploy Predictive AI

Predictive AI routinely fails to deploy, so data scientists are spearheading a movement to focus on its business value. But stakeholders need a better understanding.

Eric Siegel (with Meta AI)

Most predictive AI projects fail to launch into production. The number crunching is sound and the data scientist delivers a viable machine learning model – but stakeholder objections sadly preclude deployment.

To better meet stakeholders where they are, ML professionals are spearheading a movement to focus on predictive AI’s business value. Rather than sticking with the traditional technical metrics that report on ML model performance, a proactive minority of data scientists bust out of their nerdy cubicle and deliver estimates of ML’s profit. By reporting on the potential earnings, these quants stand a much better chance of selling model deployment to the business-side counterparts.

But this new move from ML evaluation to ML valuation will face certain objections until the practice is better understood and more widely adopted. Here are four common stakeholder concerns about ML valuation and how to address them.

1) How Can We Trust Profit Forecasts That Rest On Assumptions?

A profit curve provides a more complete view of an ML model’s worth than any single number:

However, a profit curve alone doesn’t solve the business problem of planning and selling deployment. Why? Because it’s usually based on certain business assumptions – such as the false positive and false negative costs – and that can call the entire curve into question.

The solution to this dilemma is to make charts interactive. By moving sliders, the user can vary the settings for such unresolved factors and see how this changes the curve’s shape.

This interaction provides a much-needed intuition, a “feel” for how much these factors matter when making deployment decisions. As the shape of each chart responsively morphs, the user gets to visualize the impact of each factor. In many cases, changes to the curve remain within the range of acceptability, so deployment decisions can be made with confidence. In other cases, a curve may change drastically or detrimentally, signaling that the range of uncertainty is untenable. This means that ranges of uncertainty would need to be narrowed before gaining the confidence in model value needed to greenlight deployment.

This practice empowers you to valuate models despite uncertainties. You may not have direct knowledge of, for example, the monetary loss for each false positive, because it is privy to other business units, or because it would require new investigations or experimental discovery. By interactively altering the value for such variables, you gain instant insights as to how much the uncertainty matters for driving deployment decisions. In this way, you can narrow that range, determining the limits within which the values would have to land for model deployment to be valuable. By viewing how the shape of the curves morph and how other pertinent metrics change, you gain critical intuition as to how big of a difference such factors make, whether a deployment plan may be copasetic nonetheless or whether some factors are “too uncertain” to move forward without additional efforts to narrow the range of uncertainty.

Even if you already hold fairly ideal visibility into the business factors, some of them will inevitably still be subject to potential change or uncertainty – there are always business variables that are subject to such “wiggle room.”

2) Does ML Valuation Perform An Audit On My Predictive AI Project?

Moving from standard ML evaluation to ML valuation does not constitute an audit in the usual sense of the word. In fact, doing so usually strengthens the perception of an ML model, rather than weakening it. The main outcome and purpose is to empower you to maximize deployed value and to demonstrate that potential value to your customers, colleagues and other decision makers. Stakeholders often perceive ML valuation as a validation of business value that they already intuitively believed was there.

This drives deployment. A value-oriented lens on model performance provides vital evidence to help you convince others and ensure that your model gets deployed – and that it gets deployed more optimally.

At the same time, certain “audits” help rather than hurt. Audits can be oriented toward unearthing, proving and communicating potential value – placing a spotlight on an initiative’s purpose and value so that the value will be realized. Moreover, in some cases assessing the potential business value might help you by revealing an addressable weakness in a model.

3) Isn’t Tracking Model Performance After Deployment Sufficient?

Most predictive AI projects plan to only assess the business results after the ML model is already deployed. Accordingly, most fail to deploy. This kind of post-mortem evaluation fails for a couple reasons. The only way to pursue business value during model development is to appraise its business value along the way. And the only way to make prudent business decisions as to whether to deploy, which model to deploy and precisely how to deploy, is to drive those decisions according to business value. Moreover, without an estimation of value, the model will likely never get deployed, so the project won’t ever even get to any post-deployment evaluation.

Explicitly planning for value increases value. It is possible that a model only evaluated technically could turn out to realize value if deployed – but that value would have been left unnecessarily to luck, since the process wouldn’t have explicitly optimized for value. What’s worse, the value would typically be nil, since most models that aren’t valuated aren’t deployed at all. Technical performance fails to compel stakeholders.

ML valuation as a practice also maintains ongoing value after deployment. By monitoring performance in business terms, changes to the model or to its deployment particulars (such as the decision boundary) can be driven to maximize business value. ML projects must be continually revisited and potentially redeployed, so model valuation is a must not only pre-deployment, but also “pre-redeployment.”

4) How Do We Navigate Tradeoffs Between Competing KPIs?

Money is never the only metric. Every predictive AI project must navigate tradeoffs between competing KPIs and strike a balance between them. The best way to do so is to visualize the tradeoff options.

For example, in addition to the bottom-line money saved with fraud detection, there’s another important consideration: the sheer number of times a legitimate transaction is disrupted – aka the number of false positives. A medium-sized bank may stand to win $26 million by placing the decision boundary where the savings curve peaks, but as they say, money isn’t everything. The cost of the disruptions is already factored into that bottom line savings, but they can also incur intangible or longer-term costs that haven’t been accounted for, since they could, for example, contribute to the bank’s reputation for inconveniencing customers in this way.

A small sacrifice to the monetary bottom line can sometimes greatly reduce transactional disruptions. In one case, false positives are reduced by 59% with only a 5% sacrifice in the bottom-line money saved – while also blocking 50% fewer transactions, which means cutting the disruption of commerce in half. You can read here about a similar example for misinformation detection where more misinformation is prevented by way of only a small sacrifice to the bottom line.

Get those models deployed! Addressing these four concerns will go a long way toward establishing ML valuation as a much-needed, widely-adopted best practice – thereby greatly improving predictive AI’s deployment track record.