Getting Machine Learning Projects from Idea to Execution

Guest contributor: Eric Siegel

Original published in Harvard Business Review

Humanity’s latest, greatest invention is stalling right out of the gate. Machine learning projects have the potential to help us navigate our most significant risks — including wildfires, climate change, pandemics, and child abuse. It can boost sales, cut costs, prevent fraud, streamline manufacturing, and strengthen health care.

But ML initiatives routinely fail to deliver returns — or fail to deploy entirely. They stall before deploying, and at great cost. One of the major issues is that companies tend to focus more on the technology than how it should deploy. This is like being more excited about the development of a rocket than its launch.

In this article, I offer an antidote: a six-step practice for ushering machine learning projects from conception to deployment that I call bizML. This framework is an effort to establish an updated, industry-standard playbook for running successful ML projects that is pertinent and compelling to both business professionals and data professionals.

Shifting a Misplaced Focus — from Technology to Deployment

ML’s problem is in its popularity. For all the hoopla about the core technology, the gritty details of how its deployment improves business operations are often glossed over. In this way, ML is now too hot for its own good. After decades of consulting and running ML conferences, the lesson has sunk in.

Today’s hype about ML is overzealous because it feeds a common misconception: the ML fallacy. It goes like this: Since ML algorithms can successfully generate models that hold up for new, unseen situations (which is both amazing and true), their models are intrinsically valuable (which is not necessarily true). The value of ML comes only when it creates organizational change — that is, when an ML-generated model is deployed to actively improve operations. Until a model is used to actively reshape how your organization works, it’s use-less — literally. A model doesn’t solve any business problems on its own and it ain’t gonna deploy itself. ML can be the disruptive technology it’s cracked up to be, but only if you disrupt with it.

Unfortunately, businesses often fail to bridge the business/tech “culture gap,” a disconnect between data scientists and business stakeholders that precludes deployment and leads to models collecting dust. On the one hand, data scientists, who perform the model development step, fixate solely on data science and generally prefer to not be bothered with “mundane” managerial activities. Often, they take the deployment of their model for granted and jump past a rigorous business process that would engage stakeholders to collaboratively plan for deployment.

On the other hand, many business professionals — especially those already inclined to forgo the particulars as “too technical” — have been seduced into seeing this stunning technology as a panacea that solves problems on its own. They defer to data scientists for any project specifics. But when they’re ultimately faced with the operational change that a deployed model would incur, it’s a tough sell. Taken off-guard, the stakeholder hesitates before altering operations that are key to the company’s profitability.

With no one taking proactive ownership, the hose and the faucet fail to connect. Far too often, the data scientist delivers a viable model, but the operational team isn’t ready for the pass and they drop the ball. There are wonderful exceptions and glowing successes, but the generally poor track record we witness today forewarns of broad disillusionment with ML — even a dreaded AI winter.

The Solution: BizML

The remedy is to rigorously plan for deployment from the inception of each ML project. Laying the groundwork for the operational change that deployment would bring to fruition takes more preaching, socializing, cross-disciplinary collaboration, and change-management panache than many, including myself, initially realized.

To accomplish this, a knowledgeable team must collaboratively follow an end-to-end practice that begins by backward planning for deployment. As I mentioned above, I call this practice bizML and it consists of the following six steps.

1. Establish the deployment goal.

Define the business value proposition: how ML will affect operations in order to improve them (i.e., operationalization or implementation).

Example: UPS predicts which destination addresses will receive a package delivery in order to plan a more efficient delivery process.

2. Establish the prediction goal.

Define what the ML model will predict for each individual case. Each detail matters from a business perspective.

Example: For each destination, how many packages across how many stops will be required tomorrow? For example, a group of three office buildings with 24 business suites at 123 Main St. will require two stops with three packages each by 8:30 a.m.

3. Establish the evaluation metrics.

Determine the salient benchmarks to track during both model training and model deployment and determine what performance level must be achieved for the project to be considered a success.

Examples: Miles driven, gallons of fuel consumed, tons of carbon emitted, and stops-per-mile (the more densely a route is packed with deliveries, the more value is generated from each mile of driving).

4. Prepare the data.

Define what the training data must look like and get it into that form.

Example: Assemble a large number of positive and negative examples from which to learn – both destinations that did receive deliveries on certain days and others that did not.

5. Train the model.

Generate a predictive model from the data. The model is the thing that’s “learned.”

Examples: decision trees, logistic regression, neural networks, and ensemble models.

6. Deploy the model.

Use the model to render predictive scores (probabilities) — thereby applying what’s been learned to new cases — and then act on those scores to improve business operations.

Example: By accounting for predicted packages along with known packages, UPS improved its system that assigns packages to delivery trucks at shipping centers. This improvement annually saves an estimated 18.5 million miles, $35 million, 800,000 gallons of fuel, and 18,500 metric tons of emissions.

These six steps define a business practice that charts a shrewd path to ML deployment. Anyone who wishes to participate in ML projects must be familiar with them, no matter whether they’re in a business or technical role.

After culminating with step 6, deployment, you have finished…starting something new. BizML only begins an ongoing journey, a new phase of running improved operations — and of keeping things working. Once launched, a model requires upkeep: monitoring it, maintaining it, and periodically refreshing it.

Following these six steps in this order is almost a logical inevitability. To understand why, let’s start with the end. The final two culminating steps, steps 5 and 6, are the two main steps of ML, model training and deployment. BizML ushers the project through to their completion.

The step just before those two — Step 4: Prepare the data — is a known requirement that always precedes model training. You must provide ML software with data in the right form in order for it to work. That step has always been an integral part of modeling projects, ever since linear regression was first applied by businesses in the 1960s.

Before the technical magic, you must perform business magic. That’s where the first three steps come in. They establish a greatly needed “preproduction” phase of pitching, socializing, and collaborating in order to jointly agree on how ML will be deployed and how its performance will be evaluated. Importantly, these first steps go much further than only agreeing on a project’s business objective. They ask business professionals to dive into the mechanics that define exactly how predictions will alter operations — and they ask data scientists to reach beyond their usual sphere and work closely with business-side personnel. This cross-disciplinary team is uniquely equipped to navigate to a deployment plan that is both technically feasible and operationally viable.

Bringing Business Stakeholders into the Fold

Following all six of the steps of the bizML practice is uncommon, but hardly unheard of. Many ML projects succeed wildly, even if they’re in the minority. While a well-known, established framework has been a long time coming, the ideas at the heart of the bizML framework are not new to many experienced data scientists.

And yet the folks who need it the most — business leaders and other business stakeholders — are least likely to be familiar with it. In fact, the business world in general has yet to become aware of even the need for a specialized business practice in the first place. This is understandable, since the common narrative leads them astray. AI is often oversold as an impenetrable yet exciting cure-all. Meanwhile, many data scientists far prefer to crunch numbers than to take pains to elucidate.

First things first: Business professionals need some edification. Before those in charge can participate in the bizML practice and, ultimately, green-light model deployment with confidence, they must gain a concrete understanding of how an ML project works from end to end: What will the model predict? Precisely how will those predictions affect operations? Which metric meaningfully tracks how well it predicts? and What kind of data is needed? This isn’t the “rocket science” part, but it’s still a modest book’s worth.

Considering the innumerable dollars and resources pumped into ML, how much more potential value could we capture by adopting a universal procedure that facilitates the collaboration and planning needed to reach deployment? Let’s find out.