10 years ago
Stop Searching for That Elusive Data Scientist

Based purely on the data, a really good data scientist will probably tell you the odds are poor that you’ll be able to find and hire really good data scientists. Surveys say there simply aren’t enough people with the unusual blend of software skills and statistical savvy to go around. Arguably even more important, high-impact data scientists bring collaborative temperaments and business acumento data-driven initiatives. Unfortunately, there’s no shortage of individuals with just enough statistical and software knowledge to be data-dangerous. For many organizations, a mediocre data scientist may be worse than none at all.

What to do? My immediate advice: Give up. Stop hunting for that data science unicorn and/or silver bullet. Chances are slim that your organization would even be able take full advantage of their talent. But the opportunities for data-science-enabled efficiencies and innovation are too important to defer or deny. Big organizations can afford — or think they can afford — to throw money at the problem by hiring laid-off Wall Street quants or hiring big-budget analytics boutiques. More frugal and prudent enterprises seem to be taking alternate approaches.

The smartest thing I’ve seen organizations start doing is seed-fund and empower small cross-functional data-oriented teams explicitly charged with delivering tangible and measurable data-driven benefits in relatively short periods of time. The accent is on the word team; the emphasis is on building greater data capability than better digital infrastructures. The goal is to make all of the organization — not just the geeks and quants — more conversant in how to align probability, statistics, technology and business value creation. No black boxes or centers of analytic excellence here; they want data science to be a cultural value, not just a functional expertise.

What I’ve typically observed are small teams not addressing big problems or grand challenges but an imperative to generate insights that could get the organization doing something interestingly valuable really fast. One team, for example, did something as simple as comparing a certain class of tweets from their best customers with their competitor’s customer’s tweets. The overlaps and differences immediately suggested ways to better target and take-away rivals’ customers beyond social media.

An industrial products company started monitoring blogs, boards, and other social media platforms around maintenance and service complaints and then mapping that data to internal client maintenance data. The resulting insights completely changed the internal dialogues between sales, customer support, maintenance engineering.

These were extensive, but not exhaustive, exercises in simple data science and analytics. The tools were crude and, frankly, the individual technical and statistical expertise of the teams was limited. Crudely put, the typical team was collectively less skilled and competent than a typical data scientist. But that collective team learned from each other and sent a message to the rest of the organization that even baby steps in analytics could yield large strides in outcome. Limited ambition did a better job attracting credibility and support than BHAGs.

Without exception, every team I ran across or worked with hired outside expertise. They knew when a technical challenge and/or statistical technique was beyond the capability. But, unsurprisingly, the outside advisors — in one case, an academic, in others, quants from digital consultancies — were better able to collaborate with teams that had really tried to get their minds around a design challenge. The relationship was less of an RFP box-ticking exercise than a shared space for experimental design.

For me, the big takeaway was the way existing software and databases were re-purposed and howcloud and analytics as a service offerings were brought in to address the issues. These teams had minimal experience in putting the disparate pieces together. That said, there was no shortage of vendors, advisors, and service providers who wanted to sell their capabilities into the enterprise.

So are these teams ultimately a short-term fix rather than a more sustainable solution to the data scientist shortage? Yes. But in the same way that the rise of mobile devices has changed how organizations communicate and collaborate internally and externally, the concurrent rise in Big Data and analytic opportunities means that smart organizations would be foolish to outsource this away from the very people who need to be more data-driven. You want to cultivate internal capability, not just hire it.

People don’t need to become data scientists, but they do need to understand and appreciate key principles and practices of data science. In other words, the temporary fix of data science teaming doesn’t solve the problem, but it creates the cultural and organizational context for the necessary hires to follow. Sometimes the data show that “buying time” the right way can be a terrific investment.