The balance between crowdsourced and "expert" opinion

Introduction

Some companies are turning towards crowdsourcing as a means of processing large volumes of data at relatively low cost. Crowdsourcing is “the act of taking tasks traditionally performed by an employee or contractor, and outsourcing it to an undefined, generally large group of people or community in the form of an open call.” 1

Although this can prove effective in some cases, there are risks around the quality of the data processing and the extent to which the work being done can be controlled/planned. This page briefly explores the current (2009) state of affairs with regards to crowdsourced data processing and the counter-movement by experts.

Arguments pro and contra crowdsourcing

When deciding between sourcing expertise from an 'anonymous' group and getting an expert opinion, a trade-off needs to made between speed and accuracy of the result.

In his book The Wisdom of Crowds[5] James Surowiecki argues that there are instances when an opinion reached collectively by a crowd is better than an expert opnion. These instances are typically those in which "statistics beat an expert" or differently put: when the average of many peoples' opinions is more likely to be right than an opinion derived through in-depth analysis. Some examples are guessing the number of jellybeans in a jar or estimating the floorspace of a room.

The way Wikipedia deals with content production and moderation (keeping content consistent, correct, free from offensive languiage) gives an important insight into the extent to which an anonymous group of users can 'self regulate' themselves. It took Wikipedia years to achive a situation in which the editors of the site achieved a state of self cleansing among themselves [4] Jimmy Wales, the founder of Wikipedia, had to settle many disputes himself in the early days of Wikipedia, before the self-policing social structure had matured. "You have to create a healthy civil society where there's a balance between having a lot of rules and having total freedom," he says. "And that's about very high-touch customer service."

A 2008 report by Forrester research introduces the concept of "Expert Sourcing" as "Expert sourcing is the practice of tapping specialized, professional-grade know-how and talent." [7] and sees a strong link with innovation: incremental innovation can be achieved through crowdsourcing, but for radical innovation, expert sourcing is required, as the following diagram illustrates:

Conclusion

To conclude the present (2009) consensus appears to be that...

Crowdsourcing makes sense for:

• processing large volumes of non-sensitive data
• processing and interprering data that is hard to process automatically, but simple to process by a human being (e.g. facial recoginition, image interpretation)
• processing data for which the average opinion of a crowd is better than an expert's opinion

Relying on experts makes sense for:

• data processing that relies on industry/niche specific knowledge
• processing large volumes of data iteratively, where patterns/trends in data must be observed through multiple passes over the data using statistical or other methods.

On the IT Strategy blog, Raj Sheelvant draws the following conclusion: "If the idea is evolutionary, then crowd sourcing is just fine. If the idea is revolutionary then expert sourcing is a must." [6]. This is basically the same conclusion as drawn in the Forrester research: radical innovation needs expert sourcing, incremental innovation can benefit from crowdsourcing.

Web Resources

1. Wikipedia on Crowdsourcing
2. Crowdsourcing experiment done by The Guardian
3. Douglas Rushkoff (editot in chief Wired magazine) interviewed on Crowdsourcing
4. Jimmy Wales -founder of Wikipedia- on crowdsourcing
5. James Surowiecki: The Wisdom of Crowds
6. IT Strategy: Crowdsourcing vs. Expert Sourcing
7. Reference to Forrester research on Crowdsorcing and Expert Sourcing