Comparing direct instruction for performance and instruction for independent learning in weekly R-practicals: effects on skill-acquisition, experiences and engagement.

How to cite

Ketel, Joram van; Lent, Max van; Vetten, Arjen de; Koning, Bjorn de. 2025. "Comparing direct instruction for performance and instruction for independent learning in weekly R-practicals: effects on skill-acquisition, experiences and engagement.." AEA RCT Registry. AEARCTR-0016349. https://doi.org/10.1257/rct.16349-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Joram van Ketel* , Max van Lent , Arjen de Vetten , Bjorn de Koning

* Primary investigator

Registry ID
AEARCTR-0016349
DOI
https://doi.org/10.1257/rct.16349-1.0
First registered
Last updated
Status
On going
Trial dates
2025-08-21 – 2026-08-20
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/16349

Abstract

In a regular statistics course, students receive weekly practicals on R-studio software. Teachers of these practicals will use two instructional formats to randomly allocated groups: direct instruction for performance and instruction for independent learning. We investigate influence of these formats on students' skill-acquisition, (learning) experiences, and engagement in a blended learning course. To this end, we gather data on exam performance, self-report measures during two practicals, data on students' use of resources from the LMS, and a teacher checklist to monitor the teachers’ take-up of the protocol. Next to direct effects, we investigate possible mechanisms and differential learning effects through relationships between outcome variables, planned interaction effects, and data exploration.

Intervention

Main Research Question:
To what extent does direct instruction for performance versus instruction for independent learning in weekly R-studio practicals influence students' skill-acquisition, (learning) experiences, and engagement in a blended learning course?
In a regular course on Psychometrics, second year undergraduate Psychology students receive weekly practicals on R-studio software. Teachers of these practicals will be trained to use two instructional formats: direct instruction for performance and instruction for independent learning. Teachers will teach both versions to randomly allocated groups.
To our knowledge, no research on instruction for skill-acquisition has been replicated in a randomized experiment in the timeframe and setting of an entire academic course. Direct instruction for performance such as modelling of a task generally shows benefits for learning, presumably by providing more efficient instruction (Leppink et al., 2014; Van Gog & Rummel, 2010). This effect might however disappear or even reverse in the natural setting of an eight-week course when compared with instructions for independent learning. We aim to investigate this effect and possible mechanisms through the research questions described under Primary Outcomes and exploration described under Secondary Outcomes.
The first data is collected after submitting the preregistration of this trial (starting September 17, 2025).

Experimental design

Descriptions of the course program, instruction conditions, design and measurements are available as supplementary document.

Design details

Randomization method
All ~10 teachers will be assigned and trained to teach classes in both conditions. At the start of the course, the instructional formats will be randomized over classes within teachers to avoid teacher effects (which are known to be an important factor in previous field-experiments, e.g. van Lent & Souverijn, 2020). Randomisation of the Formats was conducted manually with the following stratification rules: • As equal number as possible within teachers • Language (Dutch or English) of practicals equally divided over formats • Practical groups within one workgroup divided over formats, switching which format for the first/second hour • Diversity in Weekday • Diversity in Time of the day IF possible • Diversity in which format they teach first between more experienced teachers • Diversity in which format they teach first between less experienced teachers
Randomization unit
classes
Clusters
30
Observations
630
Arms
15 classes DIP, 15 classes IIL
Primary outcomes
Primary outcome 1: Test performance Research Question 1: To what extent does direct instruction for performance (DIP) versus instruction for independent learning (IIL) in weekly R-studio practicals differ in effect on students’ test performance? Hypothesis 1: Students in the DIP condition differ on average in their test performance from students in the IIL condition (two-sided). We formulate the alternative hypothesis as any difference between conditions in performance on the test. We see support for contradicting mechanisms in the literature that to our knowledge have not been tested before as randomized controlled trial in the setting and timeframe of a real course. To our knowledge, no research on instruction for skill-acquisition has been replicated in a randomized experiment in the timeframe and setting of an entire course. Direct instruction for performance such as modelling of a task generally shows benefits for learning, presumably by providing more efficient instruction (Leppink et al., 2014; Van Gog & Rummel, 2010). This effect might however disappear or even reverse in the natural setting of an eight-week course when compared with instructions for independent learning. For instance, 1) if students compensate for a lack of modelling through their behavior during and outside practicals because of a lower illusion of understanding (Baars et al., 2020; Dunlosky & Rawson, 2012; Paik & Schraw, 2013), 2) if students develop skills and behavior to learn independently (Glogger-Frey et al., 2017; Roodenrys et al., 2012; Weijers et al., 2023), or 3) if some students may have obtained sufficient expertise in R in previous courses to learn new analyses with less guidance (e.g. Kalyuga et al., 2012). We aim to investigate such possible mechanisms in the other hypotheses, but we have no definite expectation which of the potential effects will have a stronger influence on test performance in our natural setting of an 8-week course. Primary outcome 2: In-class performance and experiences Research Question 2: To what extent does DIP versus IIL in weekly R-studio practicals differ in effect on students’ in-class performance and experiences? Support for direct instruction for performance during such practicals is found in instructional research, but mostly in controlled settings over a shorter period of time (Sweller et al., 2019; Van Gog & Rummel, 2010). Hypothesis 2a: Students receiving DIP perform higher during the practicals than students receiving IIL. We base this expectation on research demonstrating that direct instruction for new skills generally results in more efficient execution of a task, however mostly in controlled settings over a shorter period of time (Dart, 2022; Sweller et al., 2019; Van Gog & Rummel, 2010). We measure performance as progress in the practical, as described in Outcomes (Explanation). We compare conditions on the following learning experiences, measured through a survey at the end of practical. Hypothesis 2b1: Regarding Cognitive Load, we expect lower ratings of extraneous cognitive load in the DIP condition. This would be supported by cognitive and socio-cognitive research stressing modelling for performance as an efficient way for constructing relevant schemas for performing a new task (Sweller et al., 2019; Van Gog & Rummel, 2010), in our study this task is weekly the new type of analysis. Hypothesis 2b2: Regarding Invested Mental Effort, we test an alternative hypothesis of any difference between conditions (two-sided). This outcome is often used for cognitive load in research on instructional effects showing differences between instructional conditions (Ouwehand et al., 2021; Szulewski et al., 2017; Van Der Wel & Van Steenbergen, 2018). In the original publication using this measurement, Paas (1992) however found no difference between conditions in a statistics-education setting. Students’ actually invested effort was concluded to be more closely related to motivation to invest effort than to the requirements of a task. We adopt his original interpretation in our expectation that investment of effort may differ in any direction, or be the same between our conditions as well, even if actual cognitive load demands for learning to perform the analysis in our conditions may differ. Hypothesis 2b3: We expect higher perceived learning in the DIP condition. Higher performance and lower perceived effort are related to higher perceptions of learning and expectations of ease of learning in problem-solving tasks (Baars et al., 2020; Kirk-Johnson et al., 2019). Although we hypothesize no direction for invested mental effort in our setting, we do expect lower perceived required effort for performing the analysis in terms of cognitive load, and hence higher subjective perceptions of learning, with direct instruction for performance. Primary outcome 3: use of resources and out-of-class engagement Research Question 3: To what extent does IIL versus DIP in weekly R-studio practicals differ in effect on students’ engagement in terms of preparation, time-investment, and use of available sources for learning support? Regarding preparation before the practical, we expect the following signs of engagement: Hypothesis 3a1: More students in the IIL condition prepare by watching the lecture before the practical. Hypothesis 3a2: More students in the IIL condition prepare by turning in their preparation assignments. Regarding time investment before and after the practical, we expect the following signs of engagement: Hypothesis 3b1: Students in the IIL condition report higher time investment in the week before a practical. Hypothesis 3b2: Students in the IIL condition report higher planned time investment for the week after a practical. We state the following hypotheses regarding use of support sources: Hypothesis 3c1: Students in the IIL condition use more available sources of support. In our blended course, students have access to many different sources of instruction and learning support (worked examples, lecture slides, opportunities for asking questions and collaboration, answer model etc.). Use of support sources is a subcomponent of different conceptualizations of engagement and self-regulation for learning (Hands & Limniou, 2023; Pintrich, 2004; Pintrich et al., 1993; Wang et al., 2024), that we expect to be enhanced when the teacher provides instructions for independent learning instead of direct instruction for performance. Hypothesis 3c1 tests a general aggregate of use of resources, as patterns of preferred support sources may differ between students. To specifically address the role of worked examples in instruction for independent learning (Chen et al., 2023; Van Gog & Rummel, 2010; Van Harsel et al., 2022), the following hypotheses are separately tested as well. Hypothesis 3c2: More students in the IIL condition access the example code in the week before and after the practical. Hypothesis 3c3: In the IIL condition, more students access the example code during the practical. We also hypothesize that students in the IIL condition seek earlier for the first time access to: Hypothesis 3c4 the example code, Hypothesis 3c5: the slides, Hypothesis 3c6: the answer model after its availability. Following Alhazbi et al. (2024), we investigate not only total use but also timing as measurement of engagement through hypotheses 3c4-3c6. Primary outcome 4: Differential learning effects (expertise reversal) Research Question 4: To what extent can expertise reveal differential learning effects of DIP and IIL in weekly R-studio practicals? Hypothesis 4: We hypothesize that students with lower prior ability in R benefit more from DIP than students with higher prior ability, visible in: Hypothesis 4a: exam performance, Hypothesis 4b: performance during the practical, Hypothesis 4c: perceived extraneous load, Hypothesis 4d: invested mental effort, and Hypothesis 4e: perceived learning. The Expertise Reversal effect states that with higher prior ability, direct instruction may be less beneficial for learning and performance as compared to learners with lower prior ability (Castro-Alonso et al., 2021; Kalyuga, 2007; Tetzlaff et al., 2025; Van Gog & Rummel, 2010). Such a compound effect is generally attributed to learner specific levels of cognitive load for a particular task, and observed in different types of skill acquisition (Kalyuga et al., 2012; Stambaugh, 2011).
Secondary outcomes
To better understand the possible mechanisms that drive effects of both conditions, we plan analyses on relationships between outcomes (Research Question 5), additional analyses on differential effects of instructional formats (Research Question 6) and post hoc explorative analyses. Research Question 5: To what extent are learning experiences, learning outcomes and engagement related in training R through weekly practicals, and can such relationships be influenced by instructional formats? Hypothesis 5a1: We expect a positive relation between perceived learning and test performance. Hypothesis 5a2: We expect a positive relation between measures of engagement and test performance. Hypothesis 5b: We expect a stronger 'illusion of understanding' in the condition with direct instruction for performance as compared to the condition with instructions for independent learning. In essence, direct instruction for performance is expected to create a stronger illusion of understanding. In line with literature on desirable difficulties and monitoring we expect that perceived learning is negatively related to the perceived cognitive effort that was necessary for a task (Baars et al., 2020; Kirk-Johnson et al., 2019), even if the more challenging format (in our case less direct instruction) actually benefits learning (Bjork & Bjork, 2020; Kirk-Johnson et al., 2019; Paik & Schraw, 2013; Seufert et al., 2017). Hypothesis 5c1: We expect a positive relationship between cognitive load and measures of engagement. Hypothesis 5c2: We expect an interaction effect of cognitive load and receiving instruction for independent learning on measures of engagement. Hypothesis 5d1: We expect a negative relation between perceived learning and measures of engagement. Hypothesis 5d2: We expect an interaction effect between perceived learning and receiving instruction for independent learning on measures of engagement. We base these expectations on presumed monitoring in student self-regulation and desirable difficulties literature: students may compensate for such metacognitive judgements of higher perceived load and lower perceived learning through their behavior during and outside practicals (Baars et al., 2020; Dunlosky & Rawson, 2012; Paik & Schraw, 2013). Research Question 6: To what extent can affective personal characteristics reveal differential learning effects of DIP and IIL in weekly R-studio practicals? Hypothesis 6a We hypothesize that students with higher motivation for statistics, will benefit more from IIL, visible in visible in higher exam performance, higher practical performance, higher perceived learning, and higher use of resources than those with lower motivation for statistics. Hypothesis 6b We hypothesize that students with higher fear-of-statistics, will benefit more from DIP, visible in higher exam performance, higher practical performance, and higher perceived learning, than those with lower fear-of-statistics. Motivation and Fear-of-statistics are taken from new scales that are in the process of validation and administered to this sample 6 months before the course in which our experiment takes place. Post hoc analyses are discussed in the analysis plan, available as supplementary document.

Cited by

No citation data available for this registration yet.

Keywords