Judging the Problem: A Problem-Centric Approach to Evaluating Early-Stage Ventures
How to cite
Zhang, Miaomiao; Lane, Jacqueline. 2025. "Judging the Problem: A Problem-Centric Approach to Evaluating Early-Stage Ventures." AEA RCT Registry. AEARCTR-0015330. https://doi.org/10.1257/rct.15330-1.2
Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.
* Primary investigator
Abstract
Problem formulation has long been acknowledged as a core activity in strategic decision-making, yet how new ventures articulate the problem they aim to solve is often overlooked. This gap in problem formulation is particularly critical for early-stage ventures, which must pinpoint fundamental challenges in their value chain to develop alternative innovative solutions. In this paper, we conduct a field experiment at a prestigious US-based venture competition program that grants over $500,000 to winners. Our randomized control trial includes over 200 professional judges within the accelerator community, who collectively evaluate more than 1,500 ventures at the semi-final stage. Our treatment intervention trains the judges to prioritize problem identification and formulation of an idea, verifying whether the venture has tested their assumptions about the market. In contrast, the control group receives standard guidance on judging with no emphasis on clarifying the problem. We hypothesize that trained judges pay closer attention to avoid “type III” error—fixating on the wrong problem from the outset—thus reducing the odds of endorsing ventures without a clear problem-solution fit. Our findings contribute to the innovation and entrepreneurship literature by concluding that structured training helps expert evaluators better attend to signals of early-stage ideas, with implications for improving the effectiveness of the entrepreneurial support ecosystem involving external stakeholders.
Intervention
We implement a training intervention for judges who participate in the external evaluation of early-stage venture applications during the semi-finalist stage of a large-scale venture competition program.
The training focuses on "problem identification" as a key criterion for assessing the quality of entrepreneurial ventures. Judges are randomly assigned at the individual level to one of two conditions. In the control condition, judges receive standard instructions that reiterate the competition’s evaluation guidelines. In the treatment condition, judges receive guidance emphasizing the importance of clearly defining and validating the problem the venture seeks to address.
This intervention examines whether training expert evaluators to prioritize problem formulation influences their assessments and the feedback they provide to ventures. Specifically, we test whether a problem-focused evaluation approach leads to greater variation in judges’ recommendations and produces feedback that ventures perceive as more constructive and actionable.
Experimental design
This study evaluates whether training expert judges to prioritize problem identification influences their assessments of early-stage ventures. We conduct a randomized controlled trial within a large-scale U.S.-based venture competition program, where judges evaluate semi-finalist applications.
Judges are randomly assigned at the individual level to either a treatment group or a control group:
Treatment Group: Judges receive additional training emphasizing problem identification and why it is important to focus on solving a "right" problem at the first place when evaluating early-stage ventures.
Control Group: Judges receive standard evaluation/onboarding instructions without any additional emphasis on problem formulation.
The intervention aims to assess whether a problem-focused evaluation approach affects:
Judges’ scoring, as well as the weight they assign to problem definition relative to other evaluation criteria leading to their final recommendations.
The quality, length, and focus of qualitative feedback provided to ventures.
Ventures’ perceptions of feedback usefulness, measured through post-competition surveys.
The study contributes to research on the evaluation process of entrepreneurial innovations by testing whether structured training improves expert assessors’ ability to recognize and support early-stage ventures with well-defined problems.
Design details
Cited by
No citation data available for this registration yet.