Crowdsourcing provides a convenient and increasingly popular method for gathering large amounts of data and annotations. Amazon's Mechanical Turk and CrowdFlower, games such as the ESP Game, and requests for free annotation help such as LabelMe are just a few examples of crowdsourcing efforts. These attempts have taught us many lessons and brought up yet more questions. How can we most effectively elicit the information we need from a distant and potentially anonymous workforce? What kind of workforce is required for different tasks such as user studies and data set labeling? How can we train and evaluate workers?

The 2012 AAAI Spring Symposium on Wisdom of the Crowd will bring together researchers from robotics, user interfaces, games, computer vision, and other disciplines exploring the core scientific research challenges of crowdsourcing. This symposium will seek to facilitate interaction among researchers and work toward formulating a set of guidelines for future crowdsourcing endeavors.

The symposium will be held at Stanford University, March 26-28, 2012. Full symposium schedule.


Topics of interest include, but are not limited to:

Symposium Format

The symposium will combine a variety of activities to facilitate interaction among participants from different communities and discussion of key challenges, including:

Confirmed Speakers

Ed Chi (Google, Inc.)
Rob Miller (MIT)

Important Dates

Additional information is available on the main AAAI Spring Symposium website.

Submission Information

We invite contributions in the form of full papers (6 pages) and extended abstracts (2 pages). Participants will also be invited to fill in a 1-page position paper including a brief author bio, research interests, and a discussion of research challenges in crowdsourcing. Submissions should be made through the EasyChair Website.


Caroline Pantofaru (Willow Garage, Inc.) (
Sonia Chernova (Worcester Polytechnic Institute) (
Alex Sorokin (CrowdFlower) (