The Team Solution to the Data Scientist Shortage

LATEST THINKING

Today’s data scientist shortage won’t go away soon, yet companies need them more than ever to leverage the value of big data. The solution is to build teams of data scientists instead of seeking soloists.

What is a data scientist?

As computer scientists and programmers, they design the intricate models, algorithms and visualizations that can help companies distil insights from huge volumes of chaotic data. Data scientists also do the work of engineers. They acquire external data sets to supplement internal data, and often manage the data they use and maintain the systems which host it. And as their companies’ top experts on generating analytical insights, they guide, train and sometimes manage other quantitative professionals, and help general managers understand what they need to know about big data-era analytics.

Data scientists are in high demand, but research by the Accenture Institute for High Performance has found the world is facing a severe shortage. There is simply not enough PhD talent to fill the jobs. The shortage is especially severe in the U.S. where 80 percent of new data scientist jobs created between 2010 and 2011 have not been filled, according to our analysis. And the shortage is getting worse. One reason is that data scientists require a scarce combination of skills. They must master advanced statistical and quantitative methods and tools, along with the new computing environments, languages and techniques for managing and integrating large data sets. Data scientists must also possess industry knowledge and business acumen to create models and solve real-world problems. And they need excellent communication and data visualization abilities in order to explain their models and findings to others. That combination is hard to find.

DOWNLOAD THE REPORT [PDF, 113 KB]

Training quants to become data scientists and turning to outside sources can help, but both have their limits: training can take years, and many companies cannot accept the risks of opening up their analytical innards to outsiders. But there is another solution which holds real promise for offsetting the shortage: Create a team of people who individually lack the full skill-set of a data scientist, but as a group possess them all. When physicists take on a big project, they do not build super-colliders and analyze the data all by themselves. They bring together a team to design the equipment, run experiments, do the math and analyze the data. Likewise, it makes sense to divide the labor of a data scientist rather than search for that rare combination in a single person.

Creating teams that are a melting-pot mixture of complementary hard and soft skills is the approach Accenture is taking to build up the scale of its own data science capacity.

Some of these roles may already exist within the company under other names or guises. Companies already have people who clean data and operate systems, and “data stewards” who work with business analysts to manage data and help ensure it’s used well. Still, it can be difficult to find even technologists who are familiar with Hadoop and other Big Data technologies. But these teams can be aided by tools that simplify these tasks. These tools are still quite new, and may not necessarily cover all the work that data scientist teams need to do. But as they mature, they will increasingly allow business analysts with less technical know how to be part of a data scientist team.

Most organizations already have a great deal of experience managing teams and projects. But executives should keep the following points in mind for creating effective data engineer-scientist teams:

Widen the recruiting pool

Don’t just look for people who already have these roles and skills in competing companies. Search outside your industry, and even outside the business world. It can sometimes be easier to find people who have used tools to analyze data in the sciences than highly trained and specialized developers in your own industry. Restless academics with strong analytical skills may also be able to find a new home on a data scientist team. The same is true with physics majors, artists and graphic designers. Physics majors are worth seeking for their mathematical imagination and highly speculative minds. Artists, in particular graphic designers, can bring creativity and imagination to data visualization. Consider as well the possibility of including outside specialists on the team—for example, by teaming up with a university or tapping a crowdsourcing service.
Communicate, collaborate, but don’t necessarily co-locate

These teams need to work closely together. In an ideal world, all members work together in the same location and even room. But companies should not give up on teams if they cannot co-locate them. Videoconferencing is just the start. Remote workers can be paired up and set up to share the same screen on their computers, so each can enter text on the same command line and see the same windows.
Boost effectiveness and retention through team learning

On a data scientist team, it’s helpful to encourage members to pick up skills from other members. Over time, this creates flexibility; when one member is unavailable, others can pick up the slack. It can also create a unit that is more resistant to attrition. When everyone is learning new skills from their teammates and thus furthering their careers, team members have more reasons to stay put.

In addition, the time-proven wisdom about managing teams bears repeating: Data scientist teams, like others, flourish best when there is effective leadership, a strong mandate from above and clear goals. They require a path for taking projects from design through implementation. Like many projects in the IT world, they benefit from working in rapid, iterative sprints of preparation, analysis and review. We also recommend starting with short, low-risk projects to learn the ropes before tackling longer, more complex ones. Businesses are long on experience with managing teams. They will remain short on data scientists. Why shouldn’t businesses use what already know to compensate for what they lack?

For further insights read:

The Accenture Institute for High Performance develops and publishes practical insights into critical management issues and global economic trends. Its worldwide team of researchers connects with Accenture’s business leaders to demonstrate how organizations become and remain high performers through original, rigorous research and analysis.

Research Team