Judging the Problem: A Problem-Centric Approach to Evaluating Early-Stage Ventures

How to cite

Zhang, Miaomiao; Lane, Jacqueline. 2025. "Judging the Problem: A Problem-Centric Approach to Evaluating Early-Stage Ventures." AEA RCT Registry. AEARCTR-0015330. https://doi.org/10.1257/rct.15330-1.2

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Miaomiao Zhang* , Jacqueline Lane

* Primary investigator

Registry ID
AEARCTR-0015330
DOI
https://doi.org/10.1257/rct.15330-1.2
First registered
Last updated
Status
On going
Trial dates
2024-08-01 – 2025-12-31
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/15330

Abstract

Problem formulation has long been acknowledged as a core activity in strategic decision-making, yet how new ventures articulate the problem they aim to solve is often overlooked. This gap in problem formulation is particularly critical for early-stage ventures, which must pinpoint fundamental challenges in their value chain to develop alternative innovative solutions. In this paper, we conduct a field experiment at a prestigious US-based venture competition program that grants over $500,000 to winners. Our randomized control trial includes over 200 professional judges within the accelerator community, who collectively evaluate more than 1,500 ventures at the semi-final stage. Our treatment intervention trains the judges to prioritize problem identification and formulation of an idea, verifying whether the venture has tested their assumptions about the market. In contrast, the control group receives standard guidance on judging with no emphasis on clarifying the problem. We hypothesize that trained judges pay closer attention to avoid “type III” error—fixating on the wrong problem from the outset—thus reducing the odds of endorsing ventures without a clear problem-solution fit. Our findings contribute to the innovation and entrepreneurship literature by concluding that structured training helps expert evaluators better attend to signals of early-stage ideas, with implications for improving the effectiveness of the entrepreneurial support ecosystem involving external stakeholders.

Intervention

We implement a training intervention for judges who participate in the external evaluation of early-stage venture applications during the semi-finalist stage of a large-scale venture competition program.

The training focuses on "problem identification" as a key criterion for assessing the quality of entrepreneurial ventures. Judges are randomly assigned at the individual level to one of two conditions. In the control condition, judges receive standard instructions that reiterate the competition’s evaluation guidelines. In the treatment condition, judges receive guidance emphasizing the importance of clearly defining and validating the problem the venture seeks to address.

This intervention examines whether training expert evaluators to prioritize problem formulation influences their assessments and the feedback they provide to ventures. Specifically, we test whether a problem-focused evaluation approach leads to greater variation in judges’ recommendations and produces feedback that ventures perceive as more constructive and actionable.

Experimental design

This study evaluates whether training expert judges to prioritize problem identification influences their assessments of early-stage ventures. We conduct a randomized controlled trial within a large-scale U.S.-based venture competition program, where judges evaluate semi-finalist applications.

Judges are randomly assigned at the individual level to either a treatment group or a control group:

Treatment Group: Judges receive additional training emphasizing problem identification and why it is important to focus on solving a "right" problem at the first place when evaluating early-stage ventures.
Control Group: Judges receive standard evaluation/onboarding instructions without any additional emphasis on problem formulation.

The intervention aims to assess whether a problem-focused evaluation approach affects:
Judges’ scoring, as well as the weight they assign to problem definition relative to other evaluation criteria leading to their final recommendations.
The quality, length, and focus of qualitative feedback provided to ventures.
Ventures’ perceptions of feedback usefulness, measured through post-competition surveys.

The study contributes to research on the evaluation process of entrepreneurial innovations by testing whether structured training improves expert assessors’ ability to recognize and support early-stage ventures with well-defined problems.

Design details

Randomization method
The randomization done in office by a computer (R script).
Randomization unit
Individual judges
Clusters
229 Judges
Observations
1,488 judge-venture pairs
Arms
114 judges control, 115 judges treatment
Primary outcomes
The experiment evaluates how a problem-centric training intervention influences judges' assessments of early-stage ventures. The key outcome variables include both quantitative ratings and qualitative feedback provided by judges: 1. Quantitative Ratings (Numerical Scores 1-5 on Evaluation Criteria) Judges score ventures based on the following dimensions: - Problem & Customer Definition: The problem is clearly identified, the solution aligns with the identified problem, and the customer has been well defined and understood. - Solution/Product/MVP: The team has a solution, product, or MVP that they have tested with their customers. - Business Model: There is a business/financial model that appears both achievable and sustainable. - Impact: The idea, once implemented, will have a material impact on the world as measured by directly or indirectly improving the quality of peoples’ lives. - Overall Recommendation: Overall, how strongly do you recommend that this startup moves forward to the PIC Finalist round? 2. Qualitative Feedback (Open-Ended Responses) Judges provide written feedback on venture applications, with specific prompts to capture their evaluation reasoning: - Strengths: What are 1-2 strengths of this team? - Areas for Improvement: What are 1-2 areas where this team can improve? Clarification on Problem Definition: If the judge rates the Problem & Customer Definition below a 3 (on a 5-point scale), they are prompted to explain what could improve it. 3. Optional Connection Question To measure judges’ willingness to engage further with ventures: Yes, I’m open to connecting with this team. Please share my contact info if they are interested.
Secondary outcomes
We will measure completion rates for each judge and time spent by each judge on each venture assigned. We will also categorize venture teams based on whether their applications fall under the opportunity discovery vs market creation means, using the technologies specified, customer/market defined, early tractions, etc. We will also categorize judges based on their expertise (VCs, entrepreneurs, operators) and explore potential heterogeneities. We will also measure venture teams' responses to the feedback judges give via a post-competition survey: Venture teams will report on how they find judges' feedback helpful (or not) and how the additional signal (or noise) affect their decision-making, iteration process, and plan to modify their idea and execution strategy.

Cited by

No citation data available for this registration yet.

Keywords