Bias as a Distinct Factor in Human Ratings of Machine Labeling
Workshop Paper
Eric P.S Baumer, Amin Hosseiny Marani
Abstract
This paper uses an excerpt from a larger analysis to argue that human assessments of machine labeling can reveal bias as a distinct measure separate from other perceptions of label quality. Human subjects were asked to assess the quality of automatically generated labels for a trained topic model. Quality assessments were gathered using 15 distinct self-report questions. Exploratory factor analysis identified a distinct “bias” factor. This point is likely relevant for a wide variety of machine labeling tasks.
PDF
This paper uses an excerpt from a larger analysis to argue that human assessments of machine labeling can reveal bias as a distinct measure separate from other perceptions of label quality. Human subjects were asked to assess the quality of automatically generated labels for a trained topic model. Quality assessments were gathered using 15 distinct self-report questions. Exploratory factor analysis identified a distinct “bias” factor. This point is likely relevant for a wide variety of machine labeling tasks.
Cite
APA
Baumer, E. P. S., & Marani, A. H. Bias as a Distinct Factor in Human Ratings of Machine Labeling.
Chicago/Turabian
Baumer, Eric P.S, and Amin Hosseiny Marani. “Bias as a Distinct Factor in Human Ratings of Machine Labeling” (n.d.).
MLA
Baumer, Eric P. S., and Amin Hosseiny Marani. Bias as a Distinct Factor in Human Ratings of Machine Labeling.