Does Manager Forecasting Matter?

How to cite

Bloom, Nicholas. 2021. "Does Manager Forecasting Matter?." AEA RCT Registry. AEARCTR-0007058. https://doi.org/10.1257/rct.7058-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Nicholas Bloom*

* Primary investigator

Registry ID
AEARCTR-0007058
DOI
https://doi.org/10.1257/rct.7058-1.0
First registered
Last updated
Status
On going
Trial dates
2019-01-20 – 2021-12-31
Countries
United States of America ()
Registry record
http://www.socialscienceregistry.org/trials/7058

Abstract

In a panel survey of online firms, we find that the ability to correctly forecast sales is highly correlated with firm performance. Despite its apparent importance, firms are remarkably bad at forming accurate predictions – if firms simply accurately reported their sales over the past 3-months as their prediction they would perform twice as well. We posit that simply encouraging them to review the financial data already easily available to them would dramatically improve their prediction performance. We aim to run an RCT testing exactly that and show a causal pathway from monitoring business financials to prediction performance.

Intervention

In conjunction with our partner, Stripe, we have been tracking firms and their forecast abilities over time by comparing 3-month predictions for revenue on Stripe with their actual sales on Stripe. In order to ensure that firms are offering serious predictions, we reward them with a $25 gift card if their predictions are within 10% of their actual sales. We are testing two interventions for significant effects on their predictions: having firms review their prior historical data on the Stripe dashboard before making their prediction and varying the amount of the gift card they can earn with their predictions.

The first intervention tests whether or not having firms review their financial details on the Stripe dashboard as part of the survey prior to forming forecasts can significantly increase their forecast accuracy. In particular, we have them sign-in to their Stripe dashboard and report their revenue over the last 3-months before making their predictions.

On top of this, we are also testing whether or not changing firms earns with a correct prediction can incentivize them to provide more accurate responses. We vary the amounts from $0 to $400.

Experimental design

Our interventions are cross randomized, with the dashboard treatment being run twice. For the first dashboard intervention, we randomly assigned 50% of the firms to control and 50% of the firms to treatment. While taking our quarterly survey, the treatment firms are asked to sign-in to their accounts and report their revenue over the last 3 months in the survey form. Control firms are simply asked to report their revenue and are not asked to consult their financial history. Both treatment and control then continue on to the question on 3-month predictions at that point.

On the next round of the survey, 3-4 months later, we cross randomize again so that 25% are dashboard treatment twice, 25% are treatment then control, 25% are control then treatment, and 25% are control twice.

Once at the prediction question, firms are randomly offered different amounts of money if their prediction for the next 3 months is within 10% of their actual revenue. Approximately 25% of the firms aren’t offered a prize and 25% are offered $25 to match what we have previous offered them. The remaining 50% are evenly split into 6.25% shares for rewards of 50, 100, 150, 200, 250, 300, 350, and 400.

Design details

Randomization method
We used the last few characters of their accounts ID numbers to assign sort them into their treatment arm. These account ID numbers are as good as random.
Randomization unit
Firm
Clusters
8549 Firms
Observations
8549 Firms
Arms
1st Dashboard Treatment: 4271 Treatment, 4278 Control 2nd Dashboard Treatment: 4274 Treatment, 4275 Control Reward Treatment: $0 - 1860 , $25 - $2289, $50 - 504, $100 - 618, $150 - 459 , $200 - 637, $250 - 501, $300 - 510, $350 - 521, $400 - 650
Primary outcomes
Forecast accuracy, winning prediction (i.e. within 10%), prediction bias, time spent forming prediction, checking their financial dashboard
Secondary outcomes
Productivity, growth rates, revenue, survival (having transaction in a 6 month period).

Cited by

No citation data available for this registration yet.

Keywords