Through a field experiment in 17 primary schools in China, this study aims to investigate how the access to digital technology in after-school programs affects the performance of K12 students. Specifically, we aim to examine whether interactions with the search engine vs. generative AI in the after-school program lead to better performance in the after-school program and the school curriculums.
Intervention
We will distribute tablets with search and/or GenAI functionalities to students in the after-school program, and evaluate their efficacy on student learning outcomes.
Experimental design
Participants will be randomly assigned to a single-factor, three-level (Interactive device: Generative AI vs. search engines vs. control condition without access to tablets) between-subjects design.
Design details
Randomization method
The teachers are first clustered by city. Within each city, they are sorted by their self-reported years of teaching experience and grouped into triplets. Teachers in each triplet are randomly assigned to the generative AI, search engine, and control group, respectively. The randomization was done in the office by a computer. Our matched triplet design is a generalization of the efficient matched-pair design.
Randomization unit
The randomization is conducted at the teacher level. The teachers are clustered by city.
Clusters
In this experiment, clusters are defined at the group level, with each group consisting of 4–6 students (averaging 5 students per group). The Spring 2025 experiment involves 500 students from 17 schools. Based on this structure, the study consists of approximately 100 clusters. The distribution includes 34 clusters (168 students) in the AI condition, 34 clusters (172 students) in the search engine condition and 32 clusters (160 students) in the control group.
Observations
The Spring 2025 experiment involves students from 17 schools (teachers), with 12 schools assigned to the treatment group (6 to the search engine condition and 6 to the AI condition) and 5 schools assigned to the control condition. 500 students, with 160 in the control group and 340 in the treatment group (168 in the AI condition and 172 in the Baidu condition).
Arms
The Spring 2025 experiment involves students from 17 schools (teachers), with 12 schools assigned to the treatment group (6 to the search engine condition and 6 to the AI condition) and 5 schools assigned to the control condition. 500 students, with 160 in the control group and 340 in the treatment group (168 in the AI condition and 172 in the Baidu condition).
Primary outcomes
Our dependent variables include the level of learning core concepts and the level of innovation. After receiving the intervention, students are required to build science projects according to the course requirements. An example of a project is a model boat made from recycled materials. We will evaluate the extent to which students learn the core science concepts, and the innovation level of the students' works, using raters who are familiar with the domain. Raters will evaluate the following dimensions: (1) Completeness (To what extent do you believe this work is fully completed? 1 = Not at all, 10 =Very much), (2) Functionality (To what extent do you believe this work can achieve its intended functionality? 1 = Not at all, 10 =Very much), 3. Aesthetics (To what extent do you consider this work to be aesthetically pleasing? 1 = Not at all, 10 =Very much), (4) Functionality vs. Aesthetics (Do you think the makers of this product focus more on its functionality or aesthetics? 1 = Being only focusing on functionality, 7 = Being only focusing on aesthetics), (5) innovation (To what extent do you think the work is innovative? (1 = Not at all, 10 =Very much), (6) Overall rating (Please provide an overall rating for this work. 1 = Very poor, 10 =Excellent)
Secondary outcomes
(1) We collect students’ midterm and final exam grades of 2024 Fall and 2025 Spring semesters. The secondary outcome variable is students’ academic performance of core school curriculums such as Chinese, Math, English, Science, and Arts.
(2) We collect students’ prompts when they interact with generative AI and search queries when they use the search engine, from which we create measures of students’ information seeking behavior.
Cited by
No citation data available for this registration yet.