Machine Learning Times
Machine Learning Times
EXCLUSIVE HIGHLIGHTS
Effective Machine Learning Needs Leadership — Not AI Hype
 Originally published in BigThink, Feb 12, 2024.  Excerpted from The...
Today’s AI Won’t Radically Transform Society, But It’s Already Reshaping Business
 Originally published in Fast Company, Jan 5, 2024. Eric...
Calculating Customer Potential with Share of Wallet
 No question about it: We, as consumers have our...
A University Curriculum Supplement to Teach a Business Framework for ML Deployment
    In 2023, as a visiting analytics professor...
SHARE THIS:

7 years ago
Analytics 101: Assessing Project Value

 

The key question that organizations ALWAYS ask when undertaking an analytics initiative is “Can they guarantee value from the project?”   They want to know whether the return on their investment will be positive. But they want this guarantee before any data, the technical infrastructure, or other relevant details about the analytics environment have been assessed. Though a prior guarantee is impossible, there are key factors that an organization can assess early to estimate project viability and value – to get at the expected cost and return of the proposed investment.

When organizations hire an analytics consultant or attempt to build an analytics capability, they want a return on that investment. The technical component to achieving an analytics goal is obvious, but there is also a significant non-technical component. Analytics produce a return when it impacts the profit-making processes and this can only happen by deploying a model in the production environment and changing internal processes and systems to make actionable results accessible to decision makers. Understanding potential value for a project requires assessing the factors that influence project cost as well as project return.

Referencing the mining analogy for analytics, when prospecting for value—you don’t know how much value you will find until you start digging and breaking through rock. Early digging takes the form of questions an organization should ask before undertaking an analytics initiative:

  1. Who should be involved with the project?
  2. What level of analytics is required to deliver value?
  3. What software tools should be employed?
  4. What data is available, and is it accessible?

Understanding the answers to these questions will separate success from failure, or separate wild success from a muted success, of a project.

Who Should Be Involved in the Project?
There are several groups of people that are vital to an analytics project:

  • Decision Makers
  • Technical Contacts
  • Data Owners
  • Users of the Analytic Results

It is important that every decision maker and stakeholder understands the value desired for the project and how the team will achieve that goal. Involving decision makers with budget authority is important for success. Having decision makers involved in the process ensures common understanding and the organizational support necessary for project success. If a technical person is the project leader it is important that they have buy-in and ongoing support from their manager and decision makers.  Without proper buy-in it may be challenging to navigate the organizational change necessary to deploy the solution.  Having technical staff such as data owners, IT or data management staff, present throughout the process will help mitigate concerns about data access or security, software deployment, and infrastructure requirements. Finally, it is vital to consider who the ultimate user will be—who will consume the results?  Their early involvement will help guide the most effective technology choices to make the analytic results accessible and actionable.

What Level of Analytics is Required to Deliver Value?
Every project uses analytics, just at different levels, and every level has value. The Ten Levels of Analytics maps the relationship between different analytic approaches. If an organization is only doing level one reporting, building a high-level model is not necessary to have success—an initial analytics project a few levels higher can generate significant value.  Starting with a high level technology increases the risk for success.  It’s better to start small, build trust in the technology, and then build on that success.

 

Understanding the distinction between prescriptive and predictive analytics can help with shaping the project team.  If the project requires an automated decision process (prescriptive analytics) there is no need for a human to understand how to use the analytic results to take action. But a predictive model requires a human to decide what to do with the results. In this case it is important to involve those users early in the process to make sure analytics results will be accessible and understandable.

What Software Tool Should Be Employed?
Has your organization made an investment in an analytics software platform that must be used for the project? Or are you just getting started and open to options? Many commercial (SAS, IBM SPSS, etc.) and open source (R, Python, etc.) options are available and each has strengths and weaknesses. Open-source software is free and has all the latest features, but may not scale as well, or be as maintainable, as the commercial tools over time. Commercial software is often expensive and may be slightly behind the frontier, but has known quality, is more maintainable, and is more readily accepted by IT managers than open-source tools. Using open source tools, when possible, frees up project budget to develop models that will deliver the desired value. All too often we have encountered clients who made a significant investment in analytics software (that wasn’t being used) and had little budget left to build, test, and deploy the models. However, there are costs to choosing an open-source software package that are not obvious up front — for example, the cost of integration.  Some factors to consider when making a software decision:

  • Do you understand the package process of open source tools?
  • How long do you expect the model to live?
  • How big is your data?
  • Do you have IT approval for the software and understand how you will maintain it?

Before making a decision, do your research or discuss options with your analytics team to determine the best solution for your organization.

What Data is Available?
A key consideration is data availability and access. What data is available (storage format, quantity, number of tables, etc.) to support the project and will the analytics team have access to it?  Who owns the data and what is the security around it? You don’t need to wait for perfect data, but it is important to understand if sufficient data exist and how much effort will be required to wrangle the data so that it is usable for analytics.

Understanding Trade-offs are Critical for Success
The most important decision is to choose the right problem. It is easy to aim too high, but sometimes business constraints conspire to undermine lofty goals, such as underestimating the challenge of integrating with technology. If you want to guarantee value and deploy the analytics solution, consider starting with a project with a small rate of return but significantly lower cost. The value derived from a less costly solution can then be reinvested toward a larger project with higher return on investment. By understanding the cost of a technical solution, it is easier to see that a simpler solution, using the tools that you have, may produce more value at less risk moving forward. Think of this initial data analytics project as a sales tool for future full-scale initiatives. It can be used to demonstrate to all levels of management that the benefits of data analytics merit the required investment of time, money, and emotional energy.

 

Author Bio

Dr. Andrew Fast, Chief Scientist, at Elder Research, directs the research and development of new tools and algorithms for mining data, text, and networks. Dr. Fast earned his MS and Ph.D. degrees in Computer Science from the University of Massachusetts Amherst, specializing in algorithms for causal data mining, and for analyzing complex relational data such as social networks. Dr. Fast has published on an array of applications including his “Practical Text Mining” book, written with Dr. John Elder and four others, which won the PROSE Award for top book in the field of Computing and Information Sciences in 2012.

Leave a Reply