AAAI Spring Symposium 2012:
Wisdom of the Crowd

Crowdsourcing provides a convenient and increasingly popular method for gathering large amounts of data and annotations. Amazon's Mechanical Turk and CrowdFlower, games such as the ESP Game, and requests for free annotation help such as LabelMe are just a few examples of crowdsourcing efforts. These attempts have taught us many lessons and brought up yet more questions. How can we most effectively elicit the information we need from a distant and potentially anonymous workforce? What kind of workforce is required for different tasks such as user studies and data set labeling? How can we train and evaluate workers?

The 2012 AAAI Spring Symposium on Wisdom of the Crowd will bring together researchers from robotics, user interfaces, games, computer vision, and other disciplines exploring the core scientific research challenges of crowdsourcing. This symposium will seek to facilitate interaction among researchers and work toward formulating a set of guidelines for future crowdsourcing endeavors.

The symposium will be held at Stanford University, March 26-28, 2012. Full symposium schedule.

Topics

Topics of interest include, but are not limited to:

• Applications for crowdsourcing:
  • data set annotation,
  • user studies,
  • search relevance,
  • content authoring, and
  • integration of crowdsourcing and AI
• Reward strategies:
  • no direct compensation (the LabelMe data set),
  • low (Mechanical Turk) or high per-task compensation, and
  • making the task fun, such as by using games (the ESP Game).
• Methods for selecting an appropriate workforce:
  • recruiting experts,
  • creating experts and trusted workers,
  • learning worker expertise,
  • changing compensation models, and
  • requiring workers to pass tests.
• Methods for efficiently evaluating results:
  • no evaluation,
  • evaluating each task by hand,
  • allowing workers to evaluate each other, and
  • automated evaluation.

Symposium Format

The symposium will combine a variety of activities to facilitate interaction among participants from different communities and discussion of key challenges, including:

• invited talks,
• individual technical presentations by researchers to serve as case studies,
• open panel discussions and brainstorming around different applications and modalities, and
• working groups to create guidelines and evaluation strategies, providing a common starting point for future research.

Confirmed Speakers

Ed Chi (Google, Inc.)
Rob Miller (MIT)

Important Dates

• October 7, 2011 - Submissions due
• November 4, 2011 - Acceptance notification
• January 20, 2012 - Camera-ready submission
• March 26-28, 2012 - Symposium

Additional information is available on the main AAAI Spring Symposium website.

Submission Information

We invite contributions in the form of full papers (6 pages) and extended abstracts (2 pages). Participants will also be invited to fill in a 1-page position paper including a brief author bio, research interests, and a discussion of research challenges in crowdsourcing. Submissions should be made through the EasyChair Website.

Chairs

Caroline Pantofaru (Willow Garage, Inc.) (pantofaru@willowgarage.com)
Sonia Chernova (Worcester Polytechnic Institute) (soniac@wpi.edu)
Alex Sorokin (CrowdFlower) (syrnick@gmail.com)