By: Alpana Dubey and Alex Kass
Using data-driven analysis to get to know crowd workers
The emergence of online labor markets, such as Upwork, 99Designs or TopCoder, enables on-demand access to workers with specific attributes, supporting what we call “workforce virtualization.” This trend can provide project managers with a new level of agility and cost efficiency, but these advantages come with some tradeoffs.
One tradeoff is the difficulty of estimating when (or even whether) tasks posted to an online labor market will be completed at the required quality level. While a project manager may become very good at estimating how long team members will take to complete a task and how well they will do it, it can be more difficult when sourcing work to a crowd.
When posting a task to a labor market, there are multiple risks that can keep it from being completed. The crowd, for example, might not have enough people who meet the task requirements. Likewise, the price offered might be too high, the task too complex or the constraints too restrictive.
While this unpredictability is not unique to crowdsourcing approaches, the indirect and transient nature of crowdsourcing relationships can amplify these risks. We refer to the uncertain fate of crowdsourced tasks as the “post and hope” model, and believe that it is a significant barrier to the widespread adoption of workforce virtualization.
It’s worth noting this uncertainty does not necessarily imply that crowd workers are less reliable. Crowds may actually be more reliable in some cases, but project owners lack the kind of data-driven analysis that could turn “post and hope” into something more like “post and expect.”
While a project manager may not have detailed experience with every crowd member or even every crowd, there is historical data available from some of the crowdsourcing platforms, from which automated analyses can be generated.
This blog discusses our efforts to demystify this estimation ability with an analysis of data collected from a crowdsourcing platform. As a starting point, we developed a crowd confidence model (see Figure 1). We also collected data about crowd performance and mapped it to this model to determine what factors can be used to determine confidence levels. Our confidence model includes three main elements:
Confidence in job completion within the stipulated time.
Confidence in quality of task done.
Confidence in crowd members, which further breaks down to data protection, IP protection and being free from malicious intention
Figure 1: The elements of our crowd confidence modelBy analyzing historical data and mapping it to a model, we can learn the confidence parameters that can be used to predict the likelihood a given posting to a given crowd will produce the desired results. Such an estimate can help a project manager determine which tasks to crowdsource and to which crowd, as well as the likely implications for a project schedule and other performance factors.
To explore this idea, we conducted studies on various crowdsourcing platforms. Here we present some of the insights drawn from one study, based on publically available data on TopCoder, a crowdsourcing platform for software development.
First, we investigated the impact that task type has on attracting workers, and found that certain job types attract more members of a crowd. We also found that certain job types have better completion rates than others, as depicted in Figure 2. In some ways, these findings reflect that platforms may have workers with specific skill areas in majority; therefore, it makes sense to post certain jobs on certain platforms.
Figure 2: Task uptake and completion vary by job type
For another example factor, we looked at the impact of seasonality on job completion (see Figure 3) and specifically for a demand-supply gap between jobs posted versus jobs completed in different periods of the year. Interestingly, based on the data, the tasks that do not get completed varies significantly from month to month.
Figure 3: Impact of seasonality
Through this analysis, we investigated a number of factors to determine impact on both completion rates and quality of work. The top six factors (in order of importance) that were most important for predicting job completion and quality are shown in Figure 4.
Max rating refers to the maximum rating of a worker who has applied for the job.
Jobs on TopCoder are posted as a competition, with prize money awarded for top submissions. We were a bit surprised to find that although the size of the first prize is a factor, the second prize has more influence on job completion than the first prize. Perhaps winning the competition is a reward in itself because it builds reputation, while a larger second prize increases worker confidence that good work will pay off.
The type of challenge, such as coding, testing or design, was the next most important factor in predicting task completion
The duration of job was also a factor, but perhaps not as big a factor as would be expected.
The last influencer was number of technologies that must be mastered to complete the skill. It is hard to find a crowd member who matches a task that requires a very diverse set of skills.
Quality of completed job had somewhat different predictors. The rating of the worker was the biggest influence, then the prize offered and the ratings points.
In summary, although project managers have previously relied on intuition to “get to know” their workers, our initial studies show that it is possible to get to know the crowd using more quantitative techniques. Project managers who adopt this data-driven approach will be more likely to “post and expect” work to be completed with commensurate quality; they may even realize their understanding of the crowd is more accurate than their intuitive understanding of the individuals with whom they directly work.