Banking survey experiments

How to cite

Registration, BETA Team. 2025. "Banking survey experiments." AEA RCT Registry. AEARCTR-0015747. https://doi.org/10.1257/rct.15747-2.1

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

BETA Team Registration*

* Primary investigator

Registry ID
AEARCTR-0015747
DOI
https://doi.org/10.1257/rct.15747-2.1
First registered
Last updated
Status
Completed
Trial dates
2025-03-17 – 2025-08-29
Countries
Australia ()
Registry record
http://www.socialscienceregistry.org/trials/15747

Abstract

The Treasury is advising the government on its responses to Australian Competition and Consumer Commission (ACCC)’s Retail deposits inquiry report (ACCC 2023) and Home loan price inquiry report (ACCC 2020). Both reports find a lack of consumer engagement in these markets, despite the significant potential financial gain that could be realised by many customers through regular engagement. Treasury has asked BETA to conduct the consumer testing recommended by ACCC’s reports, to determine the optimal design for prompts aimed at increasing engagement.

BETA is conducting a suite of research activities to support this work: field trial, interviews, surveys. There are two aspects to the suite of work. The first is identifying the best prompt design for enabling customers to engage with the market using an RCT design. The second is understanding frictions and barriers that make it hard for customers to switch to help identify other potential solutions, aside from prompts (separate analyses not PAP relevant). This project focuses on the first part of the work only as prompt designs will be testing using RCTs embedded in the surveys.

As part of this work, BETA is conducting two RCTs embedded in surveys (one on mortgages and one on retail deposits) to test prompts and collect data on consumer experiences with switching. These two RCTs are the focus of this project.

Intervention

This Pre-analyis Plan (PAP) contains analyses for the Randomised Controlled Trial (RCT) components of the survey.

This revised PAP details two main changes:
1. Additional data collection (wave 2) to address a display error discovered in the Savings Treatment 2 condition along with our updated analytical approach. Note that wave 2 data collection commenced on 01 May 2025 prior to the submission of this PAP revision. However, no data analysis has been or will be conducted before finalisation of this revised analysis plan.
2. Changes to the way we code the primary outcome variables. While we originally planned to use a dictionary-matching approach, this was unable to deliver adequate accuracy in testing. Therefore the data will be fully hand-coded.

Each RCT (one for mortgage, one for savings account) will be a three-arm survey experiment. The effects of prompts (attention control + 2 treatments) will be tested via vignettes. Participants are asked to provide advice to their friend in the vignette. A mock-up banking app showing their friend’s financial position is shown and the app screen includes one of the three prompts.
The text in the prompts are as below.
Mortgage stream
Attention control: Check out our great range of products! We have a wide range of products to suit your financial needs. Visit banking.com.au/all-products for more information. Eligibility requirements may apply.
Treatment 1 (simple prompt): A lower interest rate will save you money on your home loan. Visit moneysmart.gov.au/mortgagecalculator to see how much you could save with a lower interest rate. This information is required by the Australian Government.
Treatment 2 (detailed prompt): Your home loan interest rate is higher than average. 7.8% your current rate, 6.5% average rate for similar home loans. Visit moneysmart.gov.au/mortgagecalculator to see how much you could save with a lower interest rate. This information is required by the Australian Government.
Savings stream
Attention control: Check out our great range of products! We have a wide range of products to suit your financial needs. Visit banking.com.au/all-products for more information. Eligibility requirements may apply.
Treatment 1 (simple prompt): Grow your savings with a higher interest rate Visit moneysmart.gov.au to check if you are getting the best interest rate for your savings. This information is required by the Australian Government.
Treatment 2 (detailed prompt): Grow your savings with a higher interest rate A competitive savings account will offer an interest rate of 4% or higher. Visit moneysmart.gov.au to check if you are getting the best interest rate for your savings. This information is required by the Australian Government.

Experimental design

This project includes 2 RCTs (one for mortgage stream, one for savings stream) embedded in surveys. Each RCT will be a 3-arm randomised controlled field trial. All participants will be shown a vignette asking them to recommend actions to help their friend improve their financial position. Part of the vignette includes a banking app screen which contains prompts (interventions).
The trial design for the Savings stream mirrors that of the Mortgage stream except for the differences in the vignette presented.
During the implementation of the original study, we discovered a display error in the Savings T2 condition where a negative sign was missing from the personal loan component display. To address this issue, we are conducting additional data collection (wave 2) with the corrected display. This wave 2 data collection uses an unequal allocation approach to efficiently address this issue:
• Savings T2 condition: 1,000 participants
• Savings Control condition: 40 participants
• Savings T1 condition: 40 participants
The home loan stream conditions are not included in wave 2 data collection as they did not contain any errors in wave 1. All procedures, measures, and randomisation in wave 2 are identical to wave 1, except for the correction of the display error in Savings T2.
Meta-analytic approach for savings stream
To maximise statistical power while addressing the display error in the savings T2 condition, we will use the following analytical approach:
For savings Control and Treatment 1 conditions:
We will calculate the proportion differences (e.g., percentage point difference between treatment and control and cohen’s h) separately for each wave
We will calculate the standard error for each difference and then meta-analytically combine these estimates using a random-effects model
We will report both the separate wave estimates and the combined estimate with 95% CIs
Heterogeneity between waves will be assessed using I^2and Q statistics
For the Savings T2 condition:
Only wave 2 data (with the corrected display) will be used in analyses
Comparisons with other conditions will use the meta-analytically combined estimates for those conditions
For all home loan conditions:
Analyses will use only wave 1 data as no additional data is being collected.
This approach allows us to maintain causal inference while addressing the specific error that occurred, as it preserves the randomised nature of the design in both waves.

Design details

Randomization method
Participants will be randomised into one of the three cells in each RCT within the survey platform, Qualtrics, using the inbuilt randomisation functionality, with roughly equal probability of assignment across the groups (using the ‘evenly present elements’ option in Qualtrics).
Randomization unit
Individual level
Clusters
No clusters
Observations
approximately 4200-4500 participants for the mortgage RCT stream and about 3800-4100 for the savings RCT stream.
Arms
We expect approximately 4000-4500 participants for each RCT (one for mortgage and one for savings) with approximately 1400-1500 participants in each arm of the RCT (except of the savings stream Treatment group 2). As we will not be analysing wave 1 Treatment group 2 data, we will have approximately 1000 participants for this treatment group.
Primary outcomes
Savings stream primary outcome: A binary variable based on recommendations provided by survey respondent about engaging with the market where 1 is engaging with the market (e.g. switching to a higher interest savings account or looking for a better interest rate savings account) and the rest is coded 0. Mortgage stream primary outcome: A binary variable based on recommendations provided by survey respondents about engaging with the market where 1 is engaging with the market (e.g. looking for a better mortgage product, asking for a reprice, switching banks to get better interest rate on home loan) and the rest is coded 0. The question asked to participants for the primary outcome is ‘After looking at your friend's banking app screen below, what are some of the actions you suggest that he takes to improve his financial position?’

Cited by

No citation data available for this registration yet.

Keywords