A Field Experiment on Group Bias in Machine Learning Classification
How to cite
Asensio, Omar Isaac; Ha, Sooji; Marchetto, Daniel. 2018. "A Field Experiment on Group Bias in Machine Learning Classification." AEA RCT Registry. AEARCTR-0003425. https://doi.org/10.1257/rct.3425-1.0
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Many algorithms that conceal hidden biases are being used in business and government to make important decisions in society. For example, historical data used to train machine learning (ML) algorithms are known to unintentionally discriminate against minority groups (e.g. by gender, race, etc.) depending on the nature of the observational data. Researchers typically have little to no control over the sample characteristics of the training data and often rely on technical solutions after-the-fact. Field experiments can be an effective alternative to rigorously test and potentially mitigate the effects of algorithmic bias in a number of applications. In this study, we implement a randomized controlled trial (RCT) to detect and mitigate possible bias in the context of electric vehicle (EV) usage. Expert and non-expert participants will complete a classification task using reviews from a popular electric vehicle mobile app. The data collected will be used for training a classification algorithm using neural network-based language models. In addition, policy preferences will be collected during the intervention to determine if the review classification tasks influence participant beliefs as they relate to electric vehicle infrastructure.
Experimental design
We will deploy a survey questionnaire to build a training dataset for machine learning classification tasks. Human raters will be asked to read short text related to electric vehicle experiences and will be asked to classify each review as either positive or negative and identify one or more topic categories that describe the text from a list of predefined choices. All reviews that include potentially sensitive information such as cell phone number or email have been redacted using the terms [phone number] or [email address]. Each participant will classify a total of 20 randomly selected reviews from a set of 127,257 publicly available electric vehicle charging station reviews, as provided by a popular EV charging station locator app. After categorizing the reviews, participants will be asked for short demographic information. Pre- and post- task policy preferences will be collected for analysis. No personally identifiable information will be collected or shared.
Design details
Cited by
No citation data available for this registration yet.