RCT of Adoption of Digital Accounting and Payment Technology (ADAPT)

How to cite

Kerr, Philip; Delahunty, Luke. 2021. "RCT of Adoption of Digital Accounting and Payment Technology (ADAPT)." AEA RCT Registry. AEARCTR-0007112. https://doi.org/10.1257/rct.7112-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Philip Kerr* , Luke Delahunty

* Primary investigator

Registry ID
AEARCTR-0007112
DOI
https://doi.org/10.1257/rct.7112-1.0
First registered
Last updated
Status
In development
Trial dates
2020-08-01 – 2022-01-31
Countries
Private
Secondary IDs
Innovate UK Project No: 51607
Registry record
http://www.socialscienceregistry.org/trials/7112

Abstract

Randomised Control Trial (RCT) of the Adoption of Digitally Automated Accounting and Payment Technologies (ADAPT) by SMEs in Cheshire, testing whether participation in a “good practice interactive webinar event” (100 SMEs in a treatment group) leads to a greater understanding and progress towards adoption of these technologies compared to a written online information guide only (100 SMEs in control group).

Intervention

Details of the interventions for each group with sufficient detail to allow replication.
200 SMEs in Cheshire will be recruited for the Trial.
Treatment group
Through the Trial, 100 of these SMEs, selected at random, will receive two interventions:
• Access to an online portal, where they can read a static written ‘good practice guide’ on the use of Digitally Automated Accounting and Payment Technologies. Access to the guide will be provided following completion of the EOI form, approval of eligibility for the trial and completion of the baseline survey on the online portal, allowing for immediate access on point of recruitment for all participants (i.e. on a rolling basis over a short window of 4-6 weeks) and prior to the webinar events.
Attendance at an interactive online ‘Good Practice’ webinar event delivered by specialists at Barclays Bank. Once assigned to the treatment group, SMEs will be asked to confirm which event date, from a range of available options, they will attend. This event will provide an opportunity to learn from technology experts from the Bank about good practice in using Barclays’ digital payment technologies. Additionally, the webinar will provide experiential, peer-to-peer learning opportunities to share/compare experiences and learn from each other in a small group environment. The event will include an interactive presentation by Barclays, with Q&A and a facilitated networking discussion. Ten events will be delivered, with 10 SMEs in each event, over a five-day period. The same individual at Barclays will deliver each event to ensure consistency.
Control group
The other 100 SMEs randomly allocated to the Control Group will only get access to the static written ‘good practice guide’ on the use of Digitally Automated Accounting and Payment Technology via the ADAPT portal.

Experimental design

The core intervention of this RCT will be the delivery of a series of virtual ‘Good Practice’ events to be delivered online by Barclays Bank. These events will be hosted by technology experts from the Bank who will share their experience and knowledge of the use of digital payment technologies, and provide opportunities for peer to peer learning.
This proposed intervention is based on an approach tested in a Proof of Concept under Round 1 of the Business Basics Programme. The ADAPT Proof of Concept tested whether it was possible to harness the expertise of larger corporates in the borough, to share their best practice knowledge, to encourage SMEs in Cheshire to adopt productivity boosting digital automation practices and technology. The proof of concept focused on digital automation in manufacturing and marketing businesses.
The evaluation of the Proof of Concept found that the intervention did have a significant effect on SMEs at the early stages of adoption (based on the BEIS Adoption Model). For example, 100% of participants reported that the ADAPT best practice visit had persuaded them of the benefits of adopting digital automation technology.
The concept was however, only tested on a relatively small treatment group and was not compared against any alternative form of treatment. Additionally, due to the time constraints of the Proof of Concept, it was not possible to gather evidence of how the positive changes identified on the ‘immediate outcomes’ of the intervention may lead to changes on ‘intermediate impacts’ such as a decision to invest in/adopt the digital technology and implement the practices within the business.
This RCT will test the intervention with a larger population sample and identify the added average treatment effect to that of a ‘control’ treatment of exchanging good practice, such as the provision of good practice written ‘Information Guides’.
The format of the event has been adapted from an in-person intervention, delivered at the Barclays Global Technology Centre in Knutsford, Cheshire, to virtual, online events, due to restrictions associated with COVID-19.
During the Trial, 100 SMEs from Cheshire will be randomly selected to attend one of the virtual ‘Good Practice’ events delivered online, and receive a ‘Good Practice Guide’ on the use of digital banking and payment technologies to resolve issues with late payments. A further 100 SMEs will be randomly selected to receive only the ‘Good Practice Guide’.
The Trial will last 13 months, starting on 1st January 2021 and ending by 31st January 2022. This will allow for the collection of longitudinal data from both the treatment and control groups on the 'intermediate outcomes/impacts' at appropriate intervals throughout the Trial.
To estimate the effect of the intervention, we will compare the outcomes between the treatment and control groups taking into account the baseline levels of awareness, understanding and adoption of digital payment technologies to assess the net impact of the good practice events. Formal statistical tests (including t-test and regression adjustment, discussed in more detail in section 7) will be used to determine whether the observed differences between the treatment and control groups are statistically significant.

Design details

Randomization method
Description of randomisation methods used to generate the allocation sequence - pure - stratified/blocked (please specify strata) - recommended - paired - cluster (please specify) To reduce the standard errors in our analysis of the core outcomes we will use stratified randomisation. The final decision on the number of strata and the list of stratification variables will be made after the recruitment has been completed (to check how homogeneous the group of recruited businesses is and ensure there are enough businesses in each strata). However, currently we are considering stratification according to the following four characteristics which are likely to explain the outcome variables and the treatment effect: • Exporting vs non-exporting businesses • B2B vs B2C businesses • Baseline results of the ‘knowledge test’ • Baseline levels of technology adoption Depending on the observed differences between participating businesses we may replace some of the characteristics mentioned above with business size and sector.
Randomization unit
Firm level
Clusters
200 firms
Observations
200 firms
Arms
100 in treatment group. 100 in control group.
Primary outcomes
Several outcome measures have been designed to test the intervention’s effect on awareness, understanding and adoption of digital payment technologies. Primary outcomes CO1: SMEs have an increased level of awareness and understanding of digital payment technologies and the associated benefits/risks To measure the effects of the intervention on awareness and understanding of digital payment technologies, in collaboration with Circle Leadership and Barclays, we have developed a short knowledge test which will be included into the baseline and post-intervention surveys. The test will include questions to assess businesses’ awareness of existing technologies and their potential benefits, as well practical aspects of using them. Outcome measure: test result (continuous measure) Analysis metric: change in the test score from baseline to immediate post-intervention survey Method of aggregation: mean per study group (treatment vs control group) Time point of interest: • Before receiving support (baseline survey) • Immediate post-intervention (survey) • 6 month post-intervention (survey) Note: awareness and understanding will be assessed immediately post-intervention and 6-months post-intervention to understand whether this outcome has been sustained. Survey questions: see the baseline survey attached CO2: SMEs make progress towards adopting a new to firm digital Accounting and Payment Technology solution. To measure the effects of the intervention on adoption of digital payment technologies, in collaboration with Circle Leadership and Barclays, we have identified a set of necessary steps/actions needed to progress with adoption of a new digital Accounting and Payment Technology. The information on the number of those actions taken or anticipated by businesses will be collected post-intervention , while the baseline survey will include questions aimed at determining the pre-intervention level of adoption. Digital Accounting and Payment Technology adoption actions taken Action steps • Browsed/considered different digital Accounting and Payment Technology options (undertaken needs analysis and identified potential solutions) • Invested in external technology services/expertise • Initiated contact with technology providers/vendors • Participated in a demonstration of an online Accounting and Payment Technology platform • Met with/held discussions with potential technology providers/vendors • Participated in training and/or invested in staff training • Trialled or piloted the use of new technology • Decision made to continue use of new technology following pilot/trial • Business case developed for investment in technology • Investment made to purchase technology, optimising or expanding adoption across firm • New technology deployed and integrated into company • Other (please specify) Response options • Taken already • Not taken yet but may take in future – follow-on question to estimate the probability action will be taken within the next 12 months • Not taken yet and unlikely to take in future • Not sure • Not applicable Outcome measure: steps/actions taken/anticipated since the intervention Analysis metric: number of steps/actions taken/anticipated to adopt a new digital Accounting and Payment Technology. Method of aggregation: mean per study group (treatment vs control group) Time point of interest: • Pre-intervention level of adoption recorded at the baseline (survey) • Immediate post-intervention (survey) • 6 months post-intervention (survey) Survey questions: see the baseline survey tool attached
Secondary outcomes
We will undertake descriptive and qualitative analysis to answer complementary research questions. This will highlight the elements of the intervention and characteristics of businesses which are associated with positive outcomes. There are two main reasons we are not proposing formal statistical analysis for complementary research questions: a) the nature of the outcomes; b) available sample size. The reasons why the best practice event works for a particular business are inherently heterogeneous and in our opinion are better suited for qualitative analysis. The most successful recruitment methods as well as the types of business benefitting from the intervention the most could be identified through formal statistical analysis of subgroups. However, it is likely the pandemic has narrowed the pool of eligible businesses by increasing usage of digital Accounting and Payment Technologies across the economy and weakened the expected effect of the intervention by forcing the best-practice events to take place online. Therefore, we prefer to use the available sample to analyse the core outcomes and lower the probability of a false negative (rejecting a true null hypothesis that the best practice events do not provide any additional benefit compared to the best practice guides) as much as possible. The sample size calculations are discussed in more detail in section 8. The secondary objectives are: 1. What recruitment methods are most effective in engaging SMEs to participate in a Digitally Automated Accounting and Payment Technology focused business support programme? Analysis metric: scores for CO1and CO2 by type of recruitment method (plus qualitative evidence) Method of aggregation: mean scores for CO1 and CO2 per study group, descriptive analysis Time point of interest: • baseline: recruitment method recorded (survey) • 2-3 months post-intervention (consultations) • 6 months post-intervention (survey) • 7 months post-intervention (case studies) 2. Are the good practice webinar events more effective among certain types of beneficiary SME? If so, what are the characteristics of those businesses (i.e. size/sector/location/exporting etc.)? Analysis metric: scores for CO1and CO2 for different groups of businesses Method of aggregation: mean scores for CO1 and CO2 per study group, descriptive analysis Time point of interest: • baseline: business characteristics recorded (survey) • 6 months post-intervention (survey) 3. For businesses that have progressed through different stages of the adoption process, how/why is a peer-to-peer good practice webinar event effective in influencing behaviours? Analysis metric: testing the assumptions in the Theory of Change (described in section 4) Method of aggregation: qualitative evidence Time point of interest: • 2-3 months post-intervention (consultations) • 6 months post-intervention (survey) • 7 months post-intervention (case studies) Sample size permitting (i.e. depending on the observed levels of attrition and whether the programme is oversubscribed) we will perform formal tests for secondary research objectives i.e. analyse CO1 and CO2 for groups of businesses and recruitment methods. However, we state in advance that these calculations will be secondary, and we should not dismiss the results of the analysis of primary outcomes as happening by chance if they appear statistically significant when the results for the sub-group analysis do not.

Cited by

No citation data available for this registration yet.

Keywords