Why Data Modeling Is Not Enough
MIT Sloan Management Review ·
Otto Dettmer/Ikon Images
Imagine that, as an operations manager at a global retailer, you have invested heavily in data science. Your data science team has built sophisticated models that produce demand predictions for the company’s products with impressive accuracy. Yet, when the CEO, seeking to increase profit, asks you how much inventory should be held and where, the answers are not clear. The models predict demand but do not tell management what to do with those predictions. After investing millions of dollars in data science solutions, the company realizes it has many insights about demand but few decisions to act upon.
This is not a unique scenario. Studies suggest that 60% to 85% of data science projects end in failure. 1 The reason is not usually bad algorithms or incompetent data scientists. More often, there is a disconnect between the models that the data scientists produce and the information that business managers actually need. Often, the data scientist does not speak the language of business, and management does not speak the language of the data scientist. 2 Data scientists are trained to focus on prediction and accuracy metrics, whereas managers’ focus is on making choices that improve business metrics such as cost, market share, or profit. As a result of this communication gap, models often look elegant on paper but sit unused in practice. As one pair of analytics experts put it, “The number one reason data science projects fail is because businesspeople and data scientists, individually and collectively, don’t understand the real business problem at hand.” 3 The focus on the actual business decision is missing.
This is where decision science comes in. Unlike data science, which focuses on modeling the data, decision science models the decisions that business managers must make. This aligns the data model analysis with the business goals. By ensuring that the models are directly related to the business problem, your team can avoid relying only on “convenient data” and tie the model metrics directly to business outcomes.
For business leaders, the message is clear: If your analytics investments are not delivering ROI, stop modeling just your data and start modeling your decisions. Here is what that looks like in practice.
Decision science focuses on improving decision-making using data: As the name suggests, the emphasis is on decisions rather than on data. In a recent study, we examined academic programs and job ads in data science and decision science, with the goal of determining how their skills differ and/or complement each other. 4 We examined required coursework in both data science and decision science master’s degree programs and looked at the skills listed in job ads for each role.
We found that data science places a greater emphasis on skills related to coding, prediction, and machine learning. In contrast, decision science focuses more on directly modeling decisions using data, logic, and mathematics. This skill is known as decision modeling . Decision science, therefore, tends to focus more on problem framing, problem-solving, and business domain knowledge. Decision science relies on data science for predictions and on other data-based relationships as inputs to decision models that ultimately inform and drive decision-making. Your organization needs both skill sets.
The bread and butter of data science is prediction and machine learning. As a previous study of ours confirmed, data scientists are trained with a focus on modeling data, not on decision modeling. As Bipin Chadha, senior vice president of data science at CSAA Insurance Group, said in a presentation at Machine Learning Week 2024, “Many data scientists mistakenly think predictions are decisions.” Indeed, this is the crux of the disconnect between data scientists and management.
This data science disconnect — focusing on the data rather than on the decision to be made — can result in teams solving irrelevant problems and/or using the wrong metrics. This often leads to management rejecting solutions. 5
Models do not create value until managers use them. And managers use them only when the analysis reflects the decisions they face. Shifting the focus from data models to decision models compels analysts to engage directly with leadership, sparking conversations that uncover root causes, align priorities, and build trust in the recommendations.
When a data science team models an organizational decision, not just the data, it forces communication between the analytics team and management, mitigating a big hurdle to project success. Here is how decision models make a difference.
Managers expect the data science team to generate ROI by solving problems and providing the intelligence needed to make better decisions. However, managers often do not have the time to truly understand the root cause of the problem. Managers tend to experience the symptoms of some deeper problem.
A good decision scientist is like a good medical doctor: They must ask a lot of questions about symptoms to navigate to the real underlying problem. Decision modeling forces this discussion and reinforces that the most critical step is identifying the true problem that can then be modeled. The problem must be properly framed before it can be solved.
For example, consider the case of forecasting demand for 50 products over the next four quarters. Your data scientists build a set of 50 highly accurate machine learning forecasting models that include internal and external features. The data science team reports to management that, based on the model validation holdout samples, the mean absolute percentage error (MAPE) on the models ranges from 6% to 9%. The data scientists have done the job that they were trained to do.
As a manager, you may or may not know what MAPE is, but you know that solving the decision problem you face requires accurate demand forecasts. But the decisions you need to make are more complex than that: how much of each product to produce, and how much to hold in inventory to maximize the contribution to profit and fixed costs. Management’s metric is clear: dollars.
The real decision problem, then, is not one of forecast accuracy, though the decision problem is dependent on forecast accuracy. It is something like the following: Maximize contribution to profit and fixed costs while following our business rules.
To model contribution to profit and fixed costs, you will need estimates of the per-unit contribution for each of your products, the opportunity costs of unmet demand, and the cost of holding inventory. Business rules will have to include limited production and storage capacity, material availability, and labor availability. And you will need the demand estimates from the forecasting models because they will inform you about how much you can expect to sell. Importantly, the costs of producing too much and too little often differ significantly, and focusing purely on model accuracy glosses over this reality by treating each error equally.
To model this decision, the decision scientist’s approach would, perhaps, be to use an optimization model in combination with scenarios or simulation to factor in the uncertainty in the model inputs. The result would be a recommendation (how much of each product to produce and hold in inventory each quarter) or, better yet, a set of alternative recommendations that allows you to decide which of several possible plans to implement based on a return-versus-risk trade-off.
A common shortcut in data science modeling is to use the data that is easiest to access rather than first clarifying the decision problem and then asking, “What data would help us solve this problem?” This shortcut leads to elegant models that do not actually model the business decision that will drive business value.
Take, for example, the challenge of customer churn — a common issue for many companies in a variety of industries. Telecommunications companies frequently build models to predict which customers are most likely to leave, drawing on readily available data such as usage history and demographics. These models can generate accurate predictions, but they fall short of answering the managerial question “Which customers are worth retaining, and what actions will convince them to stay?”
Answering that decision problem requires a different lens and different data: Which customers are likely to purchase additional products/services? Which customers are most profitable? Which customers are most valuable to competitors? Which retention offers change customer behavior? These insights are rarely found in the “convenient” data sets already on hand.
By modeling the decision first and then identifying the right data, managers can avoid the convenient data trap . Doing so ensures that analytics not only predict outcomes but also guide actions that deliver measurable business value.
Data science metrics (such as goodness of fit and maximum likelihood) may be used to construct inputs to the decision model, but those metrics are going to be of little interest to management. Decision modeling ensures that the analyst uses metrics that matter to managers.
For example, consider the challenge many financial institutions face when deciding whether to approve or decline a loan request. A typical analytical approach uses data on past loans to build a model that estimates the probability that a new loan holder will default. From there, a threshold is set to create a rule — for example, “If the probability of default is higher than 0.30, then decline the loan request.”
To evaluate how well a model performs, analysts often use a tool called a confusion matrix . In simple terms, the confusion matrix shows where the model gets its predictions right and where it gets them wrong. For example, it identifies cases where the model predicts that loans will be repaid but the loans later default, or predicts that loans will default but they are ultimately repaid. Analysts then weigh these different kinds of errors to fine-tune the model.
The problem is that those measures, while important for statistical accuracy, are not the same metrics that matter to management. Executives do not just care about whether the model is right or wrong in general; they care about the financial consequences of those mistakes.
A decision modeling approach starts with the financial goal, such as maximizing profit within acceptable risk. The predictive model supplies inputs, but the decision model translates those inputs into dollars, showing the financial consequences of different threshold choices. Most importantly, it captures the asymmetry of outcomes: The cost of approving a bad loan is different from the cost of rejecting a good loan.
If your organization’s analytics investments are not delivering the anticipated ROI, here are some practical ways to bring a decision focus to your data science team and projects.
1. Start projects with a decision brief, not a data request. Amazon’s “working backwards” process requires teams to begin a new product or initiative by writing a short mock press release and an FAQ document before building a product. That discipline forces the team to define the intended customer, the problem being solved, the value that will be created, and the tests of success, before the company commits resources. Analytics projects need a similar discipline.
Rather than beginning with a data request or model specification, managers should require a decision brief that works backward from the managerial choice to be improved: who will make the decision, what alternatives are available, what constraints and uncertainties matter, and what business outcomes will define success. In both cases, the value of the upfront document is not bureaucracy; it is clarity. It prevents teams from building something technically impressive that does not solve the problem that matters.
Here is a real-world example from Ben Martin, vice president of data, analytics, and AI at apparel company HanesBrands, who has a history of leading effective analytics and data science teams. Martin has developed a process that he uses for all data science projects: No project goes forward without acceptable responses to the following questions.
Note how the process starts with the business problem, the decisions to be made, and an assessment of the value and impact of solving the problem. Martin’s guiding principle is “Decision analytics drive ROI; everything else is just fun and interesting facts.”
2. Create an enterprise digital twin as a lab for your data science team. Chadha, an engineer by training, started out on CSAA Insurance’s strategy team and later moved to lead its data science efforts. He recognized the disconnect between data scientists and management and the importance of connecting the data scientists to the business. According to Chadha, “You have to learn the business to be focused on decisions. A lot of people say that but don’t do anything about it.”
So Chadha created an enterprise digital twin — a simulation of the company — that effectively became a learning lab for his data scientists to get up to speed on the business. This not only helped his data scientists learn the business and become more decision-focused but also forged a laboratory that allows the data scientists to test the potential impact of results by applying the business rules in the digital twin.
3. Do not accept predictions without prescriptions. Managers should expect analytics teams to translate predictions into recommended actions. For example, UPS’s On-Road Integrated Optimization and Navigation (ORION) system is a route-optimization platform that uses data, advanced analytics, and operations research to recommend more efficient delivery routes for UPS drivers. ORION illustrates the difference between prediction and prescription. The value of the system is not simply that UPS can predict traffic, delivery times, or package flows more accurately. Jack Levis, who led the development of ORION at UPS, estimated that 10% to 25% of ORION’s success can be credited to better predictive models. 6 The real value comes from translating those inputs into optimized route recommendations that drivers can use. In other words, the analytics did not stop at “What is likely to happen?” It answered “What should we do?”
4. Build and lead the analytics team for decision impact. When hiring talent, stop chasing the mythical analytics unicorn. Instead, build teams that combine data science and decision science capabilities. Also, make business translation a leadership responsibility. Data scientists bring coding, data engineering, machine learning, prediction, and statistical modeling skills. Decision scientists bring problem framing, optimization, simulation, decision analysis, and business acumen.
The key is not simply to assemble technical talent but to ensure that the team can connect data, models, business rules, organizational constraints, and managerial choices. Disney’s decision science and integration group provides one example of this model: This centralized internal consulting group uses optimization, simulation, machine learning, data engineering, and AI to address business challenges in pricing, revenue management, and operations.
Leadership matters as well. In many successful analytics organizations, the leader is not necessarily the best technical modeler but rather a talented translator who connects technical work to the decisions that managers actually face. Chadha has degrees in mechanical engineering. Martin studied textile engineering and technology management. Levis was a psychology major. Their examples reinforce an important point: Analytics teams create value when they are built and led for decision impact, not just technical output.
The takeaway for leaders: Data science creates value only when it is connected to the decisions that managers must make. Predictions and insights are valuable inputs, but the end point is a better organizational choice. As a leader, your role is to ensure that analytics teams do not stop at modeling the data. They must model the decision, translate technical performance into business impact, and recommend actions that managers can take. The goal is not simply to become more data-driven but more decision-driven.