How to Design a Successful Crowdsourcing Contest

Image

Crowdsourcing contests work best when competitors win awards for reaching milestones as they go. | Reuters/Ivan Alvarado

In 2009, the Netflix Prize contest awarded $1 million to an outside team that created the most accurate algorithm predicting how much subscribers would enjoy given movies based on their past viewing preferences.

Such “innovation contests” can be used to crowdsource creative, effective solutions to wide-ranging problems, whether carried out in-house or worldwide. “Innovation contests are becoming more popular across domains,” says Kostas Bimpikis, a professor of operations, information, and technology at Stanford Graduate School of Business. “Opening challenging problems up to essentially everyone yields a good breadth of solutions in a very cost-effective way.”

But what kind of innovation contests lead to the best solutions most quickly?

Bimpikis and collaborators Shayan Ehsani and Mohamed Mostagir sought to answer that question by studying when to provide progress-related information to contest participants. Their paper, “Designing Dynamic Contests,” found that offering updated information to participants at regular milestones yielded the best results — a finding with implications beyond the business domain.

Incorporating Uncertainty

Innovation contests are especially useful for addressing open-ended, challenging problems that involve a great deal of uncertainty. “The uncertainty is about both how to solve the problem and whether a solution is even attainable,” Bimpikis says.

The researchers built a model of innovation contests around three main features: uncertainty about the best solution and whether one even existed, intermediate milestones that represent progress along the way to the best solution, and the effort participants put forth.

“We were especially interested in understanding the effects of information provided to participants about their and others’ progress toward the contest goal,” Bimpikis says.

Hearing about others’ progress can have contradictory effects: Knowledge that others are far ahead might discourage participants who feel they have no chance and thus stop trying; but hearing that others are making progress can also encourage competitors because it implies the contest is winnable, stimulating greater effort.

The key question, then, is about how best to share progress-related information to all participants — typically in the form of awards — to reach the best solution faster. “You could have a real-time leaderboard that everyone can see, for example,” Bimpikis says. “Or you can provide no information until someone wins the award — these are extreme ends of the contest-design spectrum.”

Critical Milestones

The best contest design, the research reveals, awards participants for reaching key intermediate milestones.

“If the contest is to achieve an improvement of 10% in something,” Bimpikis says, “you could give an award for reaching the 4% and 8% milestones, for example.”

Wide announcement of such awards indicates the progress certain participants have made. Importantly, it encourages others to exert more effort, because they see that progress is possible and will strive for the next milestone-based award. That leads to the best solution most quickly.

The model also showed that sharing an award among all participants who reached a milestone in a specified timeframe — or randomly granting the award to one — led to the best results, as opposed to presenting the award only to the first milestone-achiever.

From Corporations to Classrooms

The findings here have implications for innovation and learning across domains.

“Sometimes these contests might not be about getting to a certain level of improvement — like 10% — or other specific goals but about getting as far as possible toward some effort, such as how best to build a driverless car,” Bimpikis says.

He points to another real-world example: In 2000, the mining company Goldcorp gave cash awards to participants who helped them find gold on one of their landholdings by using proprietary geological data dating back to 1984. The contest resulted in the discovery of $6 billion in new gold deposits.

In such cases, contest designers might share information about how far participants have gotten and how they’ve gotten there, at regular intervals, with or without prizes for leaders. One downside of such a design, Bimpikis notes, is that participants may be incentivized to follow approaches that tend to work better in the project’s early stages, due to the information shared, which could limit the universe of methods explored.

The researchers’ findings could apply in a school setting as well, in that sharing information on students’ progress toward certain learning goals may optimize results, just as in innovation contests. “Students may be mostly after high grades, while the teacher is interested in maximizing aggregate learning,” Bimpikis says. “So teachers can share information strategically to balance encouraging those who are performing well without deterring those lagging behind.”

In the end, he says, “designing information-disclosure policies carefully for innovation contests has a ton of benefits.” That can be true for everything from improving movie recommendations to finding gold to enhancing student learning — and to other domains yet to be explored.