Show simple item record

dc.contributor.authorHutt, Hen
dc.contributor.authorEverson, Ren
dc.contributor.authorGrant, Men
dc.contributor.authorLove, Jen
dc.contributor.authorLittlejohn, Gen
dc.date.accessioned2017-05-24T19:15:44Z
dc.date.available2017-05-24T19:15:44Z
dc.date.issued2015-06-18en
dc.identifier.issn1432-7643en
dc.identifier.urihttp://hdl.handle.net/10026.1/9340
dc.description.abstract

© 2014, Springer-Verlag Berlin Heidelberg. The use of citizen science to obtain annotations from multiple annotators has been shown to be an effective method for annotating datasets in which computational methods alone are not feasible. The way in which the annotations are obtained is an important consideration which affects the quality of the resulting consensus annotation. In this paper, we examine three separate approaches to obtaining consensus scores for instances rather than merely binary classifications. To obtain a consensus score, annotators were asked to make annotations in one of three paradigms: classification, scoring and ranking. A web-based citizen science experiment is described which implements the three approaches as crowdsourced annotation tasks. The tasks are evaluated in relation to the accuracy and agreement among the participants using both simulated and real-world data from the experiment. The results show a clear difference in performance between the three tasks, with the ranking task obtaining the highest accuracy and agreement among the participants. We show how a simple evolutionary optimiser may be used to improve the performance by reweighting the importance of annotators.

en
dc.format.extent1541 - 1552en
dc.language.isoenen
dc.titleHow clumpy is my image?: Scoring in crowdsourced annotation tasksen
dc.typeJournal Article
plymouth.issue6en
plymouth.volume19en
plymouth.publication-statusPublisheden
plymouth.journalSoft Computingen
dc.identifier.doi10.1007/s00500-014-1303-zen
plymouth.organisational-group/Plymouth
plymouth.organisational-group/Plymouth/00 All current users
plymouth.organisational-group/Plymouth/00 All current users/Academics
plymouth.organisational-group/Plymouth/00 All current users/Academics/Faculty of Science & Engineering
plymouth.organisational-group/Plymouth/00 All current users/Academics/Faculty of Science & Engineering/School of Biological & Marine Sciences
plymouth.organisational-group/Plymouth/Faculty of Science & Engineering
plymouth.organisational-group/Plymouth/Faculty of Science & Engineering/School of Biological & Marine Sciences
dc.identifier.eissn1433-7479en
rioxxterms.versionofrecord10.1007/s00500-014-1303-zen
rioxxterms.licenseref.urihttp://www.rioxx.net/licenses/all-rights-reserveden
rioxxterms.licenseref.startdate2015-06-18en
rioxxterms.typeJournal Article/Reviewen


Files in this item

Thumbnail
Thumbnail

This item appears in the following Collection(s)

Show simple item record


All items in PEARL are protected by copyright law.
File version is made available in accordance with publisher policies. Please cite only the published version using the details provided on the item record or file.
PEARL is managed by Technology and Information Services.
Theme by 
@mire NV