Comparing long-term effects of direct instruction for performance and instruction for independent learning in weekly R-practicals.

How to cite

Ketel, Joram van; Lent, Max van; Vetten, Arjen de; Koning, Bjorn de. 2026. "Comparing long-term effects of direct instruction for performance and instruction for independent learning in weekly R-practicals.." AEA RCT Registry. AEARCTR-0018183. https://doi.org/10.1257/rct.18183-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Joram van Ketel* , Max van Lent , Arjen de Vetten , Bjorn de Koning

* Primary investigator

Registry ID
AEARCTR-0018183
DOI
https://doi.org/10.1257/rct.18183-1.0
First registered
Last updated
Status
In development
Trial dates
2025-08-21 – 2026-08-20
Countries
Private
Secondary IDs
Preregistration original trial: AEARCTR-0016349
Registry record
http://www.socialscienceregistry.org/trials/18183

Abstract

This preregistration adds long-term measurements to the previously registered RCT AEARCTR-0016349 "Comparing direct instruction for performance and instruction for independent learning in weekly R-practicals: effects on skill-acquisition, experiences and engagement." Initially registered September 16, 2025. In a regular statistics course, students received weekly practicals on R-studio software. Teachers of these practicals used two instructional formats to randomly allocated groups: direct instruction for performance and instruction for independent learning. In a subsequent course with similar practicals 5 months after the initial course in which the intervention took place, we test differences in performance, perceived competence for (independent) learning, time investment and reported use of instructional resources.
The first data is collected after submitting the preregistration of this trial (27-03-2026).

Intervention

Main research question:
To what extent does direct instruction for performance versus instruction for independent learning in weekly R-studio practicals influence students' long-term performance, perceived competence for (independent) learning, and use of instructional resources?
In a regular statistics course, students received weekly practicals on R-studio software. Teachers of these practicals used two instructional formats to randomly allocated groups: direct instruction for performance and instruction for independent learning.
We add long-term measurements at the start and end of a subsequent comparable statistics course five months after the end of the initial course to investigate potential effects beyond the initial trial. Further details are embedded in the original preregistration AEARCTR-0016349.
The first data for this long-term measurement is collected after submitting the preregistration of this trial (27-03-2026).

Experimental design

Detailed descriptions of the course program, instruction conditions, design and measurements in the original RCT are available as supplementary document.

Design details

Randomization method
Below we add the randomization method in the initial course of which this is a long-term measurement (copied from original preregistration AEARCTR-0016349): “All ~10 teachers will be assigned and trained to teach classes in both conditions. At the start of the course, the instructional formats will be randomized over classes within teachers to avoid teacher effects (which are known to be an important factor in previous field-experiments, e.g. van Lent & Souverijn, 2020). Randomisation of the Formats was conducted manually with the following stratification rules: • As equal number as possible within teachers • Language (Dutch or English) of practicals equally divided over formats • Practical groups within one workgroup divided over formats, switching which format for the first/second hour • Diversity in Weekday • Diversity in Time of the day IF possible • Diversity in which format they teach first between more experienced teachers • Diversity in which format they teach first between less experienced teachers”
Randomization unit
Classes of ~19 students each
Clusters
30 classes
Observations
630
Arms
15 each
Primary outcomes
Research Question 7: To what extent does IIL versus DIP in weekly R-studio practicals differ in effect on students’ performance at the start, and performance on the final test in a subsequent statistics course with similar R-studio practicals 5 months later? Hypothesis 7a: Students previously in the DIP condition differ from students in the IIL condition on average performance on a retention task at the start of the subsequent statistics course (two-sided). Like hypothesis 1 in the original preregistration, we see conflicting potential mechanisms for this hypothesis. If any difference at all is to be found at the start of a subsequent course after 5 months, it may be in both directions. Students who received DIP might have a clearer mental representation or clearer external documentation as a result of more efficient instruction during Psychometrics. Students initially in the IIL condition may however better prepare for (expected) independent learning in the upcoming practical, or be more used to addressing resources to complete this preparatory assignment. The latter will be explored in research question 9. Hypothesis 7b: Students in the DIP condition differ on average from students in the IIL condition in their test performance in the subsequent statistics course (two-sided). Just as hypothesis 7a (on the first preparatory assignment) and the preregistered hypothesis 1 (on test performance in the initial course), we expect no clear direction for test performance in this course due to conflicting potential mechanisms. If students previously in the IIL condition showed more signs of engagement in the initial course(Research Question 3 in the original preregistration), any spillover to learning behaviors in this course may benefit final test performance. Contrarily, more efficient instructions in the DIP condition during the initial course may for instance foster prior knowledge and (perceived) competence for learning during the practicals in the current course. Research Question 8: To what extent does IIL versus DIP in weekly R-studio practicals differ in effect on students’ perceived competence for learning and independent learning at the start of a subsequent statistics course with similar R-studio practicals 5 months later? Hypothesis 8a: Students previously in the DIP condition differ from students in the IIL condition on perceived competence for learning in R at the start of the subsequent statistics course (two-sided). Students may feel more competent for learning in a subsequent course after DIP than IIL following the previously preregistered benefit of DIP on metacognitive judgement of perceived learning during the practicals in the initial experiment (hypothesis 2b3). The grade that students received on the final test of the initial course may however have provided an additional que for this metacognitive judgement. Hypothesis 8b: Students previously in the DIP condition differ from students in the IIL condition on perceived competence for independent learning in R at the start of the subsequent statistics course (two-sided). Students may be more experienced with independent learning in a subsequent course after IIL than DIP, following the previously preregistered benefit of IIL on engagement in the initial experiment (hypotheses under Research Question 3). Contrarily, students’ metacognitive perceptions of competence may actually be lower as a result of initially preregistered higher extraneous cognitive load (hypothesis 2b1) and lower perceived learning (hypothesis 2b3). Research Question 9: To what extent does IIL versus DIP in weekly R-studio practicals differ in effect on students’ engagement in terms of preparation time and use of available sources for learning support in the first preparatory assignment of a subsequent statistics course? Hypothesis 9a: Students receiving IIL compared to DIP invest more time in preparing the first practical of the subsequent course. The time to complete the preparatory assignment for week 1 of the subsequent course is not merely a measure of the necessary effort to complete the task (which would presumably be lower after clearer instructions in DIP in line with hypothesis 8a), but in our context more probably a measure of the effort that a student is willing and deems necessary to invest. This is in line with our initial hypothesis 3b1, predicting higher time-investment before the practical in the IIL condition. Hypothesis 9b1: Compared to students who were initially in the DIP condition, students who were initially in the IIL condition use more available instructional support sources in the preparatory assignment for the first practical of the subsequent statistics course. Working in R essentially involves regular use of instructional resources in practice. The execution of a retention task like the preparatory assignment (hypothesis 7a) should therefor naturally allow and include the use of such resources. This rationale was also reason for the initially preregistered hypotheses 3c1-3c6 on the benefit of IIL on use of such resources during practicals. In this way, we specifically test if in this preparatory assignment for the first practical of the subsequent statistics course, more students who were initially in the IIL condition compared to students who were initially in the DIP condition, report the use of Hypothesis 9c1: Information in the workbook, Hypothesis 9c2: the available example code, Hypothesis 9c3: Materials or own notes from a previous course, Hypothesis 9c4: AI, Hypothesis 9c5: help from peers, Hypothesis 9c6: Internet (other than AI or Brightspace).
Secondary outcomes
Intentions for additional analyses are discussed in the analysis plan, available as supplementary document.

Cited by

No citation data available for this registration yet.

Keywords