Research shows that evidence-based algorithms perform better than humans in predicting the future. Yet people give less weight to AI advice than they should. By exogenously variating personal experience with AI predictions, we explore how personal experience impacts weight on algorithm advice and how the level of expertise moderates this relationship. Our results will help design algorithms that are better adopted by human decision-makers, and mitigate the biases that experts hold on algorithms.
Experimental design
We will randomly variate each subject's personal experience with AI.
Design details
Randomization method
Randomization will be done by the survey platform.
Randomization unit
individual
Clusters
50 per treatment
According to our pilot test based on N=50 per treatment, the mean in changes in WOA is 0.0227, the standard deviation is 0.2453. We calculate the number of subjects needed to detect the difference with power 0.8 and significance level of 0.1 and conclude that we need 720 subjects in total, or 720/7=103 subjects per treatment.
Observations
50*7=350 subjects
Our pilot study was based on the pre-specified 350 subjects. Then our formal study was based on n=100 per treatment, as calculated above.
Arms
50*7=350 subjects for the pilot
100*5(variation of feedback)*2(experts vs. layman)=1000 subjects for the formal study
Primary outcomes
Weight on advice
Confidence in self
Confidence in algorithm
The belief in own accuracy
The belief in AI accuracy
Cited by
No citation data available for this registration yet.