Judging Judges: How VCs and Entrepreneurs Evaluate Early-Stage Innovations

How to cite

Zhang, Miaomiao; Lane, Jacqueline. 2024. "Judging Judges: How VCs and Entrepreneurs Evaluate Early-Stage Innovations." AEA RCT Registry. AEARCTR-0014748. https://doi.org/10.1257/rct.14748-1.0

Download: BibTeX · RIS — for reference managers without a browser translator. Zotero and most others pick up this page's citation metadata directly.

Miaomiao Zhang* , Jacqueline Lane

* Primary investigator

Registry ID
AEARCTR-0014748
DOI
https://doi.org/10.1257/rct.14748-1.0
First registered
Last updated
Status
On going
Trial dates
2024-10-28 – 2025-03-31
Countries
Private
Registry record
http://www.socialscienceregistry.org/trials/14748

Abstract

Over the last two decades, venture competitions have emerged as a prominent phenomenon of the entrepreneurial ecosystem. Major competitions like the MIT $100K Entrepreneurship Competition, MassChallenge, the Hult Prize, and TechCrunch Disrupt’s Startup Battlefield have attracted thousands of startups, including companies like Grubhub, Cloudflare, and Akamai Technologies. These contests have provided participants with resources and visibility that have contributed to their growth. Increasingly, universities are hosting competitions that go beyond traditional business plans, rewarding projects aimed at creating both financial value and positive social impact. Investors, experienced entrepreneurs, and industry professionals from diverse networks often serve as judges, bringing varied perspectives to the evaluation of these ventures. However, we know little about how evaluators from differing backgrounds and expertise align or misalign in their assessments of early-stage innovations, what they prioritize given the duo objectives of ventures achieving both financial and social returns, and the consequences of convergent or divergent perceptions among experts in shaping venture success.

Intervention

Researchers have designed interventions to judges who are affiliated with an innovation ecosystem support organization during their evaluation of early-stage ventures. Specifically, we collected past years' venture submissions and re-engineer the applications to either focus on the financial or social value of the venture. We randomize four re-framed venture ideas to judges from various industry and occupational backgrounds and expertise. We additionally randomize the order of which "social value" and "financial value" appear in the rubric.

Experimental design

Participants in this study will be recruited from our partner site's judging network of innovation competitions. They will be directed to the study recruitment script and consent form upon RSVP for the [redacted] Judging Discussion & Networking session that our partner organization programs. There will have the options to either complete the study before/after the scheduled live virtual session or during the session. Even if they decide not to participate in the event, they are still invited to join the study. The experiment involves evaluating reframed venture submissions, which have been restructured using large language models to emphasize either financial or social impact dimensions.

Design details

Randomization method
Qualtrics randomization tool
Randomization unit
Individual decisions
Clusters
Each judge will make 4 recommendation decisions responding to the study stimuli (either financially engineered or socially engineered venture submissions), along with other rubric criteria they are asked to give scores for. Each judge will make up their own cluster. We plan to recruit up to 150 judges.
Observations
600 individual decisions, from 150 individual judges.
Arms
This study features within-subject design using mixed-effects model estimation. 300 of the ventures randomly assigned to judges will be financially-engineered and the rest of 300 are socially-focused.
Primary outcomes
Main DVs/sources of mechanism test constructs: - Judges' quantitative ratings of Ingenuity, Social Value, Financial Value, Recommendation as stated in the rubric - Judges' rank ordering of the relative importance of each criteria into their overall recommendation for each of their randomly assigned venture - Judges' qualitative rationale/reasoning that led them to the overall recommendation for the venture Main IDVs: - The random assignment for either financially- or socially-reframed venture idea (four of them for each judge) - Judges' entrepreneurial/investment background as corporate/institutional investors, founders/entrepreneurs, angel/impact investors, operators, and specialists

Cited by

No citation data available for this registration yet.

Keywords