June 16th 2017

Wise Practitioner – Predictive Analytics Interview Series: Andrew Burt at Immuta

By: Eric Siegel, Founder, Predictive Analytics World

In anticipation of his upcoming conference presentation, Regulating Opacity: Solving for the Conflict Between Andrew Burt PAW Blog imageLaws and Analytics at Predictive Analytics World for Business New York, Oct 29-Nov 2, 2017, we asked Andrew Burt, Chief Privacy Officer & Legal Engineer at Immuta, a few questions about his work in predictive analytics.

Q: In your work with predictive analytics, what challenges do you most frequently encounter?

A: I consistently come across three types of problems faced by data science teams large and small: access, collaboration, and governance issues. On the access side, data scientists have an incredibly difficult time getting to the data they need—either because of IT architecture issues or institutional issues, where different teams “own” different data sets and have varying incentives to make their data available. The truth is, though, that there are a huge number of reasons why data scientists have a hard time getting access to the data they need. On the collaboration side, data scientists need to work in groups, centralizing their shared knowledge and working towards a common goal. This can be incredibly difficult as well, thanks to distributed teams and high turnover rates.  

Q: And as a lawyer, how do you see the governance challenges come across?

A: Regulatory concerns limit predictive analytics in ways that data science teams and lawyers frequently don’t realize. As organizations move from a business intelligence framework, where analysts were the primary end consumer of data, to a machine-based framework, where machine learning models themselves are replacing analysts in a number of ways, new governance issues are arising that are challenging the way data science gets done.

I’ll cite just one example: the EU’s General Data Protection Regulation, or GDPR, which can impose fines of up to four percent of global revenue, can require that meaningful information be available about the logic of machine learning models, which consumers can have a right to access. Before, you could ask the business intelligence analyst what she or he was doing with the data if you needed to. Increasingly, though, we’re going to need to ask the models themselves, and that requires an entirely different framework for governing and supervising how predictive analytics are applied.

Q: How does predictive analytics deliver value at the organization you work with – what is one specific way in which it actively drives decisions or operations?

A: A host of ways, from consumer identification and retention efforts, to streamlined decision making from the bottom of organizations all the way up to their c-suites. I think what’s most fascinating—and most powerful—about the current state of predictive analytics is the move towards automation. Data science is really eating entire organizations in the sense that data science teams’ products are becoming cross-vocational; you can have one data science team, for example, building models that span multiple areas of expertise, covering logistics and manufacturing to even medical diagnostics. There’s one fascinating example of some researchers at Mount Sinai Hospital in New York, who were able to use unsupervised deep learning to diagnose a range of patients, though no one fully knew how or why the diagnoses were accurate.  

But more to your question: one of our customers was using drone images to manage a large infrastructure project in a remote area, and had serious problems getting that data to data scientists and analysts involved in that project. So they used our platform to provide proper access to, and governance of, their data. And even though the consumers of the data were dispersed all over the world, in multiple regulatory jurisdictions, they were able to perform an infrastructure monitoring and upgrading effort that they would have had to complete in person only a few years ago (and at great cost).

Q: When it comes to specific laws and legal trends, what should data scientists be aware of?

A: I mentioned the EU’s GDPR, but what we’re really seeing is a wave of new efforts to regulate data and the way it’s used. And that last part is crucial—restrictions on how data can be used is the wave of the future from a regulatory standpoint. It used to be that regulations on data focused on security and access. But in a world where our data is increasingly available, and where we generate so much of it, regulations are going to assume that our data is accessible as a baseline, and move to focus on regulating how it’s used. And that’s exactly what the GDPR does, as well as China’s new “cybersecurity law,” among other examples. These new purpose-based restrictions can be hard to enforce with many of today’s data science tools.

Q: Sneak preview: Please tell us a take-away that you will provide during your talk at Predictive Analytics World.

A: The key problem confronting predictive analytics is really transparency. We’re in a world where data science operations are taking on increasingly important tasks, and the only thing holding them back is going to be how well the data scientists who train the models can explain what it is their models are doing. More on that during my talk!

———————

Don't miss Andrew’s conference presentation, Regulating Opacity: Solving for the Conflict Between Laws and Analytics on Tuesday, October 31, 2017 from 4:15 to 5:00 pm, at Predictive Analytics World New York.  Click here to register to attend

By: Eric Siegel, Founder, Predictive Analytics World

No Comments yet »

June 9th 2017

Wise Practitioner – Predictive Analytics Interview Series: Jack Levis at UPS

By: Eric Siegel, Founder, Predictive Analytics World

In anticipation of his upcoming keynote conference presentation, UPS’ Road to Optimization, at Predictive Analytics World for Business New York, Oct 29-Nov 2, 2017, we asked Jack Levis, Senior Director, Process Management at UPS, a few questions Jack Levisabout his work in predictive analytics.

Q: In your work with predictive analytics, what behavior or outcome do your models predict?

A: In order to plan drivers’ day, we predict where deliveries are going to occur as well as how long it will take a driver to complete his / her route.  This opens the door to planning, execution, and analysis tools which we created.  However…  This is NOT the end game.  There is more than prediction. 

Q: How does predictive analytics deliver value at your organization – what is one specific way in which it actively drives decisions or operations?

A: Based on what was described above, UPS reduced 85 million miles driven per year.

Q: Can you describe a quantitative result, such as the predictive lift of your model or the ROI of an analytics initiative?

A: As mentioned, looking forward with predictive data and planning tools, we reduced 85 million miles driven per year while also offering new services to customers.   

Q: What surprising discovery or insight have you unearthed in your data?

A: Prediction is NOT an end game.  Optimization is… By adding optimizations (prescriptive analytics) to our predictive models, UPS reduced an ADDITIONAL 100 million miles driven per year.  This totaled to a 185 million mile reduction annually.  The prescriptive analytics alone is reducing cost of between $300M to $400M annually. 

Q: Sneak preview: Please tell us a take-away that you will provide during your talk at Predictive Analytics World.

A: I will discuss the different types of analytics and how UPS has used each.  I will point out how prescriptive analytics will find solutions that are not readily apparent and often counter intuitive.  I will also go through some best practices.

———————

Don't miss Jack’s keynote conference presentation, UPS’ Road to Optimization on Tuesday, October 31, 2017 at 1:10 to 1:55 pm at Predictive Analytics World New York. Click here to register to attend

By: Eric Siegel, Founder, Predictive Analytics World

No Comments yet »

May 26th 2017

Wise Practitioner – Manufacturing Predictive Analytics Interview Series: Richard Semmes at Siemens PLM

By: Bala Deshpande, Conference Co-Chair, Predictive Analytics World for Manufacturing 2016

In anticipation of his upcoming Predictive Analytics World Manufacturing Chicago, June 19-22, 2017 conferenceRichard Semmes presentation, Closing the Loop with Predictive Product Performance, we interviewed Richard Semmes, Senior Director, R&D at Siemens PLM. View the Q-and-A below for a glimpse of what’s in store at the PAW Manufacturing conference.

Q: What are the challenges in translating the lessons of predictive analytics from other verticals into manufacturing?

A: The objective for predictive analytics in manufacturing is really to enable actionable business decisions that impact the way you design, build, or service your products.  The most successful practitioners of predictive analytics in manufacturing use continuously updated data from many sources throughout their supply chain.  The biggest challenges center on data ETL, aggregation, and continuous updates.

Q: In your work with predictive analytics, what behavior do your models predict?

A: Our models predict the performance of mechatronic products.  We use predictive analytics to connect real world IoT data to the Digital Twin models of the virtual world.  That allows manufacturers of physical goods to proactively manage their businesses by better understanding what is going to happen in their factories as well as their products in the field.

Q: How does predictive analytics deliver value at your organization? What is one specific way in which it actively drives decisions?

A: Predictive analytics serves to find issues with products we did not know existed.  We use predictive models to understand the correlation between product features and product performance.  We use that insight to proactively manage those products in the field as well as optimizing the product through design changes. 

Q: Can you describe a successful result, such as the predictive lift (or accuracy) of your model or the ROI of an analytics initiative?

A: In one instance, we trained models using environmental data as well as IoT data from a very complex machine that produces other products.  The trained model was able to show us the environmental and job characteristics that had the best correlation to job failure.  That information can be used to warn the operator that there is increased risk of failure and it can be used to improve the machines to better handle those adverse situations.

Q: What surprising discovery have you unearthed in your data?

A: The extent to which environmental data should be taken into consideration when creating predictive models.  While it is obvious that weather and other environmental state can influence product performance, the extent to which including environmental conditions helps discover product feature correlations is significant.

Q: Sneak preview: Please tell us a take-away that you will provide during your talk at Predictive Analytics World for Manufacturing.

A: You don’t need an army of data scientists to reap the benefits of predictive analytics in your business.‚Äč

————————

Don't miss Richard’s conference presentation, Closing the Loop with Predictive Product Performance, at PAW Manufacturing, on June 20, 2017 from 1:30 to 2:15 pm. Click here to register for attendance. 

By: Bala Deshpande, Founder, Simafore and Conference Co-Chair of Predictive Analytics World for Manufacturing.

No Comments yet »

May 2nd 2017

Wise Practitioner – Predictive Analytics Interview Series: Edward Shihadeh at Auspice Analytics, LLC

By: Eric Siegel, Founder, Predictive Analytics World

In anticipation of his upcoming conference presentation, How to Revolutionize Your Model Optimization, at Predictive Analytics World for Business Chicago, June 19-22, 2017, we asked Edward Shihadeh, Chief Data Officer at Auspice Analytics, LLC, a few questions about his work in predictive analytics.

Q: In your work with predictive analytics, what behavior or outcome do your models predict?

A: Although we predict a variety of different outcomes, we specialize in predicting retention, calculating the probability that individuals in a university, business, or program will stay or leave. Because the costs of acquiring a customer far exceed the costs of retaining one, this focus allows us to bring great value to our clients.

Q: How does predictive analytics deliver value at your organization – what is one specific way in which it actively drives decisions or operations?

A: We employ our analytics not only in businesses and universities but also in the public policy arena. Working with a major city whose homicide rate was recently 2nd in the country, we used our methodology—which entails not just specialized modeling but also sophisticated appends and measurement of both individual and contextual data—to provide metropolitan police with lists of individuals at risk of committing murder, so that the police could intervene. The result: murder in this city has been driven down by 50% since 2012, translating to 100 lives saved from a tragic end and 100 potential offenders whose lives were turned around.

Q: Can you describe a quantitative result, such as the predictive lift of your model or the ROI of an analytics initiative?

A: We work with a major university for whom the leading predictive analytics solution in the university retention market had failed—leaving the university at risk of losing millions of dollars. By applying our methodology and including contextual (e.g., supra-individual) and behavioral data and measures, we created a model that was more than 99% accurate in predicting the overall percentage of students who would return the next year and 96% accurate in predicting which specific students were at risk of leaving the university. This resulted in millions in savings for this university.

Q: What surprising discovery or insight have you unearthed in your data?

A: The example above, in which the leading predictive analytics solution in the university retention market failed to deliver acceptable results, documents two important findings. First, this industry can serve our clients far better by offering custom solutions than by applying off-the-shelf models. Second, it demonstrates the enormous increase in predictive power that we can gain from including contextual and behavioral data guided by behavioral and social science.

Q: Sneak preview: Please tell us a take-away that you will provide during your talk at Predictive Analytics World.

A: Whether we work in business, health care, financial services, our models predict human behavior; As our successes illustrate, big leaps in the power of predictive analytics come not just from model optimization but also from a sophisticated understanding of the role carefully-selected individual and contextual data, and carefully-crafted measures, will play in predicting outcomes. These gains come not from mindlessly appending data, or blindly applying the latest statistical technique, but from drawing on behavioral and social science to identify data and measures that truly increase predictive power. In other words, it's about the carpenter, not the hammer. 

———————

Don't miss Edward’s conference presentation, How to Revolutionize Your Model Optimization on Monday, June 21, 2017 from 3:30 to 4:15 pm at Predictive Analytics World Chicago. Click here to register to attend

By: Eric Siegel, Founder, Predictive Analytics World

No Comments yet »

April 25th 2017

Wise Practitioner – Predictive Workforce Analytics Interview Series: Emily Pelosi at CenturyLink

By: Greta Roberts, Conference Chair, Predictive Analytics World for Workforce 2017


In anticipation of her upcoming Predictive Analytics World for Workforce conference presentation, How CenturyLink Measures How Well Leaders Manage Their Organizations, we interviewed Emily Pelosi, HR Emily Pelosi IMAGE PAW BlogAnalytics Leader at CenturyLink. View the Q-and-A below to see how Emily Pelosi has incorporated predictive analytics into the workforce of CenturyLink. Also, glimpse what’s in store for the new PAW Workforce conference, May 14-18, 2017.

Q: How is a specific line of business / business unit using your predictive decisions?  How is your product deployed into operations?

A: A product that we created called the Talent Index was shared with our senior leaders earlier this year, and the results contributed to goal planning and people management focus for 2017. The Talent Index is a tool we developed to measure how effectively our leaders are managing their organizations based on our core HR principles. It takes a comprehensive set of HR metrics, groups them into research-based factors, and produces a score through a series of weights and targets that reveals how closely they are aligned with our talent management practices. One aspect of the Index that helped it to be a success was the way it was designed. It was built with the end in mind, which was to give leaders a clear idea of where their people opportunities are. Leaders can look at their scores on the individual factors to identify what is driving their overall index score. Furthermore, they can look at the individual components within these sub scores to see what specific areas are drivers. This allowed our leaders to walk away with a very targeted idea of what they need to improve going forward, whether it be increasing engagement, providing more opportunities for high potential employees, or managing lower performers.

Q: If HR were 100% ready and the data were available, what would your boldest data science creations do?

A: HR has historically struggled with demonstrating the value of investing in people. If more data was available on employee preferences, attitudes, and day-to-day experiences, we could have a better idea of how employees are impacted by the organization. Then, if we have a better idea of how employees are impacted by the organization, we can connect this data to financial and operations targets and make a clear connection between people processes and ROI. This is already being done by some organizations, but not many are doing it well. This is still an area in which HR can make significant progress.  

Q: When do you think businesses will be ready for "black box" workforce predictive methods, such as Random Forests or Neural Networks?

A: “Readiness” for these types of methods can vary between organizations based on their culture, resources, capabilities, and other factors. That being said, assuming the right systems are in place I think businesses are actually ready now. The utility of these methods is driven by the users’ ability to identify meaningful data, connect it to business-critical outcomes, and disseminate results to the movers and shakers in their organizations. In other words, if you use these methods for issues that are actually important to the business and you can articulate what your analysis means and why it matters, you can utilize more advanced workforce predictive methods.   

Q: Do you have suggestions for data scientists trying to explain the complexity of their work, to those solving workforce challenges?

A: Early in my career I had a mentor ask me to explain a complex analysis “like I would explain it to my mom.” Now, my mom is very intelligent, but data science is not her specialty. The point was to consider the perspective of my audience. That has always stuck with me. Stay away from jargon and key words that are specific to the data analysis. You’re telling a story, so don't be afraid to get creative. Make it interesting—use analogies to help explain your work when you can, especially if you know your audience and what would resonate with them. If you can’t avoid including complex terms or details, build up to these concepts by introducing key ideas one at a time. At the end of any presentation, conversation, etc., your goal is for the audience to walk away with the 2-3 key points. Highlight these key points early on in your discussions—don’t keep the audience guessing or lead them down a winding path. 

Q:  What is one specific way in which predictive analytics actively is driving decisions?

A: Predictive analytics is taking the guesswork out of solving workforce challenges. It is reducing the negative impact that results from bias and decision making based on emotions and/or opinions. In HR at CenturyLink, analytics is core to decision making especially for strategic decisions that have a big impact. We’ve leveraged analytics for identifying new engagement initiatives, changing workforce policies, validating our performance process, predicting successful hires, and predicting turnover among other workforce trends. 

Q: How does business culture, including HR, need to evolve to accept the full promise of predictive workforce?

A: One of my favorite books that came out last year is called the “The Rise of HR” by Dave Ulrich, William Schiemann, and Libby Sartain (editors). A chapter written by Seth Kahan offers “12 predictions for a new world,” which proposes the challenges HR will be up against in the future. One of Seth’s predictions is that knowledge execution will become one of the most valuable assets in the world. According to his prediction, the ability to execute on knowledge will be more important than profitability, politics, and other powerful influences. This directly applies to how organizations need to evolve to accept the full promise of predictive analytics. Data has never been more accessible to organizations, and predictive analytics allows us to use this data to obtain knowledge that hasn’t been available before. Businesses that want to be successful in the future need to put predictive analytics at the epicenter of strategy and fully commit to making decisions based on these insights rather than biases and intuition.  In an ideal state, predictive analytics is a central part of strategic decision making by connecting data across multiple business units.    

——————

Don't miss Emily's conference presentation, How CenturyLink Measures How Well Leaders Manage Their Organizations, at PAW Workforce, on Tuesday, May 16, 2017 from 3:55 to 4:40 pm. Click here to register for attendance.  

By: Greta Roberts, CEO, Talent Analytics, Corp. @gretaroberts and Conference Chair of Predictive Analytics World for Workforce

 

No Comments yet »

April 17th 2017

Wise Practitioner – Predictive Analytics Interview Series: Holly Lyke-Ho-Gland and Michael Sims at APQC

By: Eric Siegel, Founder, Predictive Analytics World

In anticipation of their upcoming conference co-presentation, Change Management for Holly Lyke 2Establishing a Data-Driven Culture, at Predictive Analytics World for Business Chicago, June 19-22, 2017, we asked Holly Lyke-Ho-Gland, Principal Research Lead at APQC and Michael Sims, Research Analyst at APQC, a few questions about their work in predictive analytics.

Q: In your work with predictive analytics, what behavior or outcome do your models predict?

A: The organizations we study use predictive analytics to forecast just about anything: Michael Sims 3consumer behavior, employee turnover, exchange rates, etc. For example on of our study participants was able to pinpoint trends in attrition by the employee tenure and potential.

Q: How does predictive analytics deliver value at your organization – what is one specific way in which it actively drives decisions or operations?

A: Predictive analytics allows organizations to make decisions that are a) better informed and b) less prone to cognitive bias. In short, predictive analytics promotes objectivity. Another participant of this study was able to improve its understanding of its customers by integrating quantitative trends as context its traditional qualitative customer feedback. The trends helped decision makers understand what feedback was related to an actual impact on the overall customer experience. 

Q: Can you describe a quantitative result, such as the predictive lift of your model or the ROI of an analytics initiative?

A: Another study participant used language dictionaries and step-wise regression to see if internal social media use could predict employee engagement scores—with the explicit goal of using real-time social media data to create an understanding of real-time employee engagement. The study was a success and the language dictionaries were able to account for approximately 48 percent of the variation in engagement scores.

Q: What surprising discovery or insight have you unearthed?

A: The most surprising thing that we have found is the continued struggle to effectively adopt data-driven decision making in organizations. Though organizations continue to invest in data and analytics capabilities, they still indicate that establishing a data-driven culture continues to be among their greatest challenges. Often, this is a result of poor integration and communications between the business and the analytics sides of the house.

Q: Sneak preview: Please tell us a take-away that you will provide during your talk at Predictive Analytics World.

A: When building a data-driven organization, don’t start with sophisticated tools and technology; begin by creating a data-driven culture. Like any other shift in how an organization operates, a well thought out change management plan is necessary to ensure you can garner the benefits of your investment in data-driven decision making.

———————

Don't miss Holly and Michael’s conference co-presentation, Change Management for Establishing a Data-Driven Culture on Wednesday, June 21, 2017, from 10:00 am to 12:45 am at Predictive Analytics World Chicago. Click here to register to attend. Use

By: Eric Siegel, Founder, Predictive Analytics World

No Comments yet »

April 3rd 2017

Wise Practitioner – Predictive Analytics Interview Series: Natasha Balac at Data Insight Discovery, Inc.

By: Eric Siegel, Founder, Predictive Analytics World

In anticipation of her upcoming conference co-presentation, Identifying Unique Gamer Types Natasha Balac Blog Page IMAGEUsing Predictive Analytics, at Predictive Analytics World San Francisco, May 14-18, 2017, we asked Natasha Balac, CEO and Founder of Data Insight Discovery, Inc., a few questions about her work in predictive analytics.

Q: In your work with predictive analytics, what behavior or outcome do your models predict?

A: It varies, as we work with clients across many verticals from condition based maintenance to forecasting sales.  One great example that we will present at PAW is in the exciting world of Marketing.  This area has been one of the fastest and most electrifying adopters of predictive analytics methods, with countless reports of significant lift and ROI over more traditional approaches.  With the onset of Big Data, we can now utilize even larger and more diverse data to optimize data-driven marketing decisions.

Q: How does predictive analytics deliver value to your clients – what is one specific way in which it actively drives decisions or operations?

A: The ability of companies to sharply focus marketing and PR efforts has improved tremendously with predictive analytics.  The insights provided by predictive analytics reveal the specific pattern of characteristics and behavior profiles of customers most likely to buy a particular product.  Predictive analytics methods take in a wide variety of data, and systemically enable refined customer segmentation and customer persona optimization.  This enables clients to approach customers with a more tailored and personalized message, and focuses marketing resources on the customers more likely to respond.

Q: Can you describe a quantitative result, such as the predictive lift of your model or the ROI of an analytics initiative?

A: Non-disclosure agreements prevent us from sharing specifics regarding projects, but numerous projects showed several orders of magnitude lift in sales through the application of predictive analytics models.  The outcomes from the models allowed the clients to segment and approach the right groups of customers with the right message.

Q: What surprising discovery or insight have you unearthed in your data?

A: Our predictive analytics work for one client produced results that suggested significant, unexpected demographics were being overlooked.  These insights lead to a change in how the client targeted customers.

Q: Sneak preview: Please tell us a take-away that you will provide during your talk at Predictive Analytics World.

A:  Customer behaviors and preferences can be quite surprising, non-intuitive and difficult to predict.  Building personalized offers, and delivering engaging, personalized consumer experiences is the key to successful, optimized marketing campaigns.  Utilizing sophisticated segmentation and data-driven insight allows you to target the right customer with the right message consistently.

———————

Don't miss Natasha’s conference co-presentation, Identifying Unique Gamer Types Using Predictive Analytics on Tuesday, May 16, 2017 at 11:45 am to 12:00 pm at Predictive Analytics World San Francisco. Click here to register to attend

By: Eric Siegel, Founder, Predictive Analytics World

No Comments yet »

March 31st 2017

Wise Practitioner – Predictive Analytics Interview Series: Bryan Bennett at Northwestern University

By: Eric Siegel, Founder, Predictive Analytics World

In anticipation of his upcoming conference presentation, Cross-Enterprise Deployment:  Bryan Bennett IMAGE Blog PageBanking Visualization of Analytics Results – Critical for Communication at Predictive Analytics World for Business Chicago, June 19-22, 2017, we asked Bryan Bennett, Professor at Northwestern University, a few questions about his work in predictive analytics.

Q: In your work with data analytic visualizations, what have you found are the keys to success?

A: The key is to make sure the analytics tells the correct story. Many people get so caught up in the visuals and charts but neglect examining the quality of the information presented. Once people put something in a chart, people tend to accept it without questioning their accuracy. It is critical for managers to continue to ask, “Does this make sense?” That also puts some of the burden on management understanding what analytics can and cannot do, as well as what looks good or makes sense.

Q: How does data analytics visualizations deliver value at your organization – what is one specific way in which it actively drives decisions or operations?

A: If utilized properly, data visualizations can help in the decision-making process. Instead of having to pour over table after table, manager can very easy see which alternative they should choose or which product is selling more or which region is performing better. A good example is using good visualizations for management dashboards. A good dashboard will help management easy see where they need to focus on or ask questions about which leaves them time to focus on other issues that might be pressing in the organization.

Q: What surprising discovery or insight have you unearthed in your visualizations?

A: If visualizations are done properly, they should be able to tell the complete story and enable management to make good decisions. Many people try to use every visualization available to them or put too many visualizations on one page which can confuse the audience. Keep it simple is still the best advice.

Q: Sneak preview: Please tell us a take-away that you will provide during your talk at Predictive Analytics World.

A: Visualizing analytics is a iterative process that requires an understanding of analytics, how people perceive information and how to effectively use colors and shapes to draw the eye to the important stuff.

———————

Don't miss Bryan’s conference presentation, Cross-Enterprise Deployment:  Banking Visualization of Analytics Results – Critical for Communication on Tuesday, June 20, 2017 from 11:20 am to 12:05 pm at Predictive Analytics World Chicago. Click here to register to attend

By: Eric Siegel, Founder, Predictive Analytics World

No Comments yet »

March 27th 2017

Wise Practitioner – Predictive Analytics Interview Series: David Talby at Atigeo

By: Eric Siegel, Founder, Predictive Analytics World

In anticipation of his upcoming conference presentation, Semantic Natural Language Understanding with Spark, David Talby PAW BLOG ImageMachine-Learned Annotators & Deep-Learned Ontologies at Predictive Analytics World San Francisco, May 14-18, 2017, we asked David Talby, Chief Technology Officer at Atigeo, a few questions about his work in predictive analytics.

Q: In your work with predictive analytics, what behavior or outcome do your models predict?

A: Over the past five years I've worked on a wide range of predictive analytics projects in the healthcare space. Clients were mostly healthcare providers – where models were built for patient risk prediction, population health management, forecasting clinical & financial metrics, automated clinical coding and other specialty-specific challenges. For payers, the main application of machine learning was around fraud, waste & abuse – both to augment human experts investigating claims, and to automate the review of free-text clinical notes.

Q: How does predictive analytics deliver value at your organization – what is one specific way in which it actively drives decisions or operations?

A: In healthcare, the two most common goals are to save lives and save money. Quite a few projects do both. Uncovering fraudulent clinicians & pharmacists, for example, is often justified due to its high financial ROI, but also provides major benefits by finding cases where patients are harmed, mistreated or subjected to wasteful procedures.

Q: Can you describe a quantitative result, such as the predictive lift of your model or the ROI of an analytics initiative?

A: Most of the work we've done is confidential, but one published example from 2013 was regarding a readmissions prediction model we've built at the time. We were able to build a completely automated model, that did not apply any curated medical domain expertise and was solely based on our automated feature engineering algorithms, that beat that best performing academically published model at the time by 20% (in terms of AUC improvement). We were then able to beat that model by an additional 45% by building an ensemble between that model and others models that our data science team built. We've seen then further improved both the core algorithms and scalable training pipelines around them, and it is often surprising how much of a lift can be achieved at a fairly short amount of time over commonly accepted benchmarks.  

Q: What surprising discovery or insight have you unearthed in your data?

A: One surprising discovery, for me at least, was the huge variety of clinical language, guidelines and practices across different doctors and hospitals. We have found while human biology is the same across the US, and doctors supposedly follow similar best practices – the effects of healthcare being 'hyper local' are far greater. This has direct implications when applying machine learning – models transfer very poorly across hospitals, provider groups and geographic locations, whether they are on structured or unstructured data. This happens in healthcare to an extent that's far greater than what I've seen in e-commerce, web search and financial systems, which are other verticals in which I've worked before. 

Q: Sneak preview: Please tell us a take-away that you will provide during your talk at Predictive Analytics World.

A: The talk describes the three key tasks that you must perform to build a natural language understanding pipeline: Building an annotations pipeline, training machine learned annotators, and expanding your ontology via deep learning. The talk comes with full source code, available as free Jupyter notebooks that rely only on open source libraries, so that anyone can download and hack away after the talk. The example we walk through is from the healthcare space, but the design and tasks are general and apply to natural language in any domain-specific setting – understanding patents, SEC filings, academic papers, tweets, emails or transcribed phone calls. It's a technical talk and should be fun and useful for people looking to learn how to get this done for their own projects.

———————

Don't miss David’s conference presentation, Semantic Natural Language Understanding with Spark, Machine-Learned Annotators & Deep-Learned Ontologies, on Tuesday, May 16, 2017 from 3:55 to 4:40 pm at Predictive Analytics World San Francisco. Click here to register to attend

By: Eric Siegel, Founder, Predictive Analytics World

No Comments yet »

March 24th 2017

Wise Practitioner – Predictive Workforce Analytics Interview Series: Haig Nalbantian at Mercer

By: Greta Roberts, Conference Chair, Predictive Analytics World for Workforce 2017 


In anticipation of his upcoming Predictive Analytics World for Workforce conference presentation, The Pay Haig_Nalbantian 150Equity Revolution:  How Advanced Analytics are Helping to Close the Gender Pay Gap in Organizations, we interviewed Haig Nalbantian, Senior Partner, Co-leader Mercer Workforce Sciences Institute at Mercer. View the Q-and-A below to see how Haig Nalbantian has incorporated predictive analytics into the workforce of Mercer. Also, glimpse what’s in store for the new PAW Workforce conference, May 14-18, 2017.

Q: How is a specific line of business / business unit using your predictive analytics method to inform decisions? 

A: We’ve been conducting pay equity modeling and assessments either alone or as part of a broader workforce analysis since the early 90s. In the past five years or so, this area of work has grown enormously. More and more of our clients – in the US and, increasingly in Europe as well – are conducting annual pay audits to proactively address pay equity issues for women and minorities. In working with us, they choose to rely on comprehensive predictive models of base pay and total compensation that account for the multiple individual, group and market factors that drive pay in organizations. In this way, they not only isolate the effects of specific demographics themselves, thereby assessing if and to what extent there are unexplained pay disparities associated with gender or race, but also get a deeper insight about explained differences – that is, of the root causes of persistent differences that show up in raw (unadjusted) comparisons of pay levels.

While those concerned with legal challenges regarding pay equity commonly use statistical controls to explain pay differences and reduce estimates of the size of pay disparities, the more strategically-minded leaders in this domain use these same controls to better understand why pay disparities exist and what can be done systematically to reduce or eliminate them in a sustainable way. I am pleased to see more organizations moving away from a predominately legal or compliance view of pay equity to a more expansive strategic view that seeks to address systemic sources of gender and racial disparities in pay. Mercer’s When Women Thrive research has shown that aggressive evaluation and management of pay equity is a leading indicator of greater success in other aspects of employment equity. Specifically, those organizations which have specialized, independent teams using statistical methods to assess and ensure pay equity as part of the annual compensation process are significantly more likely to do better in securing a more diverse workforce and leadership team. Focus on pay equity and you are likely to end up with better diversity outcomes overall.

Many of our clients do, in fact, rely on our predictive modeling approach to pay equity, commissioning us, on an annual basis, to estimate statistical models of pay determination to assess if and to what extent pay disparities exist and make adjustments where bona fide pay gaps are found. They typically do this work as part of the annual compensation review.  

Q: If HR were 100% ready and the data were available, what would your boldest approach to pay equity deliver?

A:  In the best of all worlds, organizations will evaluate and address pay equity in the broader context of what the organization actually rewards. Our team has undertaken analyses of the drivers of pay across literally hundreds of organizations in the US and abroad for almost twenty five years now. We find the drivers of pay vary significantly across and even within organizations. They also vary over time as changing business models and objectives and changing labor market dynamics force organizations to adapt their rewards to help drive corresponding changes in their workforce. Effective pay equity practices must account for such changes and help ensure that pay equity actions align with evolving reward strategies. So, for example, if a new business strategy places a premium on certain new roles, it is important, from a gender pay equity perspective, not only to know that women in those roles are paid on par with comparable men, but that women are getting the opportunity to access these new and valued roles.

If these new roles command higher pay, disproportionate representation of men would end up increasing the raw pay gap and likely diminishing the prospects of women to be successful in the organization. A successful pay equity process will keep tab of underlying changes in what is being valued by the organization to ensure women, minorities and other groups of interest are not systematically disadvantaged by market- or internally-driven shifts in the valuation of skills, knowledge, capabilities, experience, behaviors and roles.

Properly designed, a pay equity assessment is folded into the annual compensation review; it becomes an opportunity to assess the strategic alignment of rewards with business goals. Most our clients pursue this approach. A pay equity review is not a one-time study; it is an ongoing process of rewards review, one that is of significant strategic importance to the organization.    

Q: Do you think "black box" workforce predictive methods will become widely embraced in the pay equity domain?

A: “Black box solutions” are for functional tacticians at best, not practitioners of strategic workforce management. Strategic workforce management requires understanding and effectively communicating the story within the data. By design, black box solutions bypass the story, substituting claims of “predictive validity” instead. Time may prove me wrong, but I have yet to see a compelling human capital storyline emerge from statistical relationships or algorithmically-generated predictions alone. Explanatory analytics – understanding what’s behind relationships detected in the data – is, in my view, central to building and articulating a story that can engage leaders and compel action. Since I view pay equity as fundamental to reward strategy, I am reluctant to embrace the use of automated data analytics as the basis of pay equity assessments. If pay equity is part and parcel of rewards alignment, there is no substitute for careful modeling and interpretation of the drivers of rewards.  

Q: Is there a risk of making the pay equity process too complex?

A: Our domain of workforce analytics always carries the risk of being overwhelmed by complexity of approach or analytical techniques. This has never deterred our team, however, from pursuing a more sophisticated technical solution if we are sure that solution will lead to more accurate conclusions and better results. The proof ultimately is in the results achieved. As I mentioned in my interview last year, sports analytics has definitely added complexity to the statistics tracked and followed by front office professionals, field managers and coaches, players, player representatives and sports journalists, but they have gained speedy adoption in the industry. Few of these stakeholders really grasp the technical dimensions of sports analytics. Nonetheless, they are pervasively used – because they work, because they lead to better decisions and more targeted investments. Staying away from sophisticated analytics on grounds of complexity is a cop out, one that is becoming increasingly untenable in the HR field.

The analytics used for pay equity are not all that complex. Most HR leaders have a basic understanding of multivariate regression analysis. Even if they don’t, they can readily understand that measuring pay disparities and determining their sources requires accounting for other non-demographic factors that also influence pay levels. That’s what good modeling will accomplish. More complex is the way in which the methodology is practically applied and how the results are translated into action.

So, for example; if pay strategies and pay determination are different across business units, functions, geographies, occupations and job families, do you need to model each of these separately? What determines the degree of segmentation used? Technical requirements, such as minimum required population sizes for statistical modeling, may trade off against practical business considerations. There is no pure science to inform such decisions. Similarly, once you identify pay disparities or, for instance, employees who are “under-paid” relative to peers – i.e. “outliers” – how do you close the gaps? Do you address outliers only in groups where demographic disparities have been detected? Should you make adjustments for women and non-whites only? Implementation questions such as these are generally more “complex” and challenging to navigate than are issues around methodology. Seldom do we get drawn into detailed conversations about statistical techniques. On the other hand, we do have extensive discussions about implementation issues and the “philosophy” behind pay actions.

In sum, complexity is not a major barrier for workforce analysts in the pay equity area. A richer explanation of such issues is found in Stefan Gaertner, Greenfield, G and Levine, B. “Pay Equity: New Pressures, New Challenges,” Human Resource Executive Online. April 12, 2016.

Q:  What is one specific way in which predictive analytics is driving workforce decisions?

A: Pay equity is perhaps the area where we see the most tangible results from our predictive modeling work. First of all, clients don’t ask us to do this work if they are not prepared to act on the results. Organizations understand that you don’t sit on pay disparities if you find them. You have to take reasonable action to remedy bona fide pay inequities once found.

Due diligence is always required in implementing pay actions. No statistical model can alone determine if there are pay disparities, certainly not at an individual level. First of all, there is always the potential for error in the raw data on which such models are estimated. Further, there is statistical error in the estimation of the models themselves. Not all relevant factors influencing pay may be captured in the organizations archival workforce (HRIS) data. And some jobs or career levels may be so thinly populated that it is impossible to make accurate statistical comparisons that account for differences in job or role. At a certain point, judgement comes into play.

Once individual outliers are identified, you need to carefully review them to sort out those cases where there are good technical or business explanations for the pay differences observed and those differences related to gender or race that remain unexplained. The modeling helps narrow the field for such hands-on review, but it does not bypass this need entirely. As in most areas of workforce analytics, science and art come together to render the best solution.

Still, there is no question that the analytics delivered here are hugely impactful. When you do this work, you know you are going to have an immediate effect on the client organization and the employees whose pay is at issue. Doing such consequential work is very satisfying. But it carries a huge responsibility. Because you will deliver point estimates of pay differences that may translate into actual payouts to individuals, you cannot rely on large sample sizes to overcome any data error. Precision in working the data you have is critical. Those who do this work have to be on their toes. Always!

Q: How does business culture need to evolve to realize the full promise of predictive workforce analytics such a pay equity modeling?

A: I think I largely answered this question in my response to the first question above where I reference Mercer’s When Women Thrive study. That study showed that pay equity is basically the tip of the spear in organizations’ efforts to secure gender diversity in their leadership and workforce generally. If you don’t get the pay side right, it is unlikely you’ll be doing well on the representation, promotion, retention, hiring or performance sides either. Rewards are consequential. They signal what is valued in an organization. If you don’t signal you value women, minorities or other groups of interest, you are unlikely to secure them as a vital, engaged, representative and effective part of your workforce. So start with pay equity.

But don’t stop there. If I am clear about anything in our field, it is that effective human capital management requires a systems view. The dynamics process that produces your workforce – we call it your “internal labor market”- consists of multiple moving parts that interact with each other continuously to affect the mix of talent embodied in your workforce. What happens on the reward side influences what happens on the retention side, the development side, the performance side; and vice versa. The best analytics will de-mystify this process, help you understand what drives it and, thereby, help you shape your internal labor market to meet the needs of your business. Workforce diversity and pay equity should be seen in this light. In the end, they are all about the business.

Organizations that do in fact recognize their workforce as an asset need to know what’s happening to that asset and the return they’re getting on that asset. Taking a systems view helps deliver and better process this information. Workforce analytics teams can help foster this view in the way they analyze data and communicate results. This approach enhances the power of their work. It also helps engage leadership in a way traditional HR often failed to do. Such engagement makes all the difference in making the resulting strategies successful.

——————

Don't miss Haig’s conference presentation, The Pay Equity Revolution:  How Advanced Analytics are Helping to Close the Gender Pay Gap in Organizations, at PAW Workforce, on Wednesday, May 17, 2017, from 2:15 to 3:00 pm. Click here to register for attendance

By: Greta Roberts, CEO, Talent Analytics, Corp. @gretaroberts and Conference Chair of Predictive Analytics World for Workforce

No Comments yet »

Next »