What Drives Different Assessments of Performance on Coding Tasks?

How to cite

Craig, Ashley; Effenterre, Clémentine Van. 2022. "What Drives Different Assessments of Performance on Coding Tasks?." AEA RCT Registry. AEARCTR-0009816. https://doi.org/10.1257/rct.9816-1.2

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Ashley Craig* , Clémentine Van Effenterre

* Primary investigator

Registry ID
AEARCTR-0009816
DOI
https://doi.org/10.1257/rct.9816-1.2
First registered
Last updated
Status
In development
Trial dates
2022-10-01 – 2025-12-01
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/9816

Abstract

This study focuses on evaluations of in coding interviews, which are used to hire computer programmers. We aim to shed light on the mechanisms underlying these differences in these ratings, including differences in coding quality and style, and coder effort.

Intervention

We aim to assess how software developers evaluate pieces of code written by computer programmers. Our experiment uses a large set of de-identified code blocks from an online coding platform, which span coders of different skill and problems of different levels of difficulty. For each code block, we will have access to the code's objective measures of performance including sub-test results (e.g., whether it runs, whether it produces correct answers to unit tests etc.). Using these data, we will ask evaluators to judge the quality of the code using the same Likert scales as on the platform.

Experimental design

Our design relies on multiple observations per subject. Each participant will evaluate 4 code blocks.

Design details

Randomization method
Randomization will be done by a computer.
Randomization unit
We use a within-subject design: half of the codes seen by an evaluator will be in the treated group, half in the control group. The order in which the treated half of the codes is seen will be randomized. The order of the codes within each condition will be randomized.
Clusters
400 evaluators.
Observations
1600 observations.
Arms
800 treatment code blocks, 800 non-non-treated.
Primary outcomes
Subjective ratings of code quality
Secondary outcomes
Evaluators’ prediction of the candidate’s score from the automated evaluation tool.

Cited by

No citation data available for this registration yet.

Keywords