Expert Reviews and Professional Learning: Evidence from a Physician Drug Rating Platform
How to cite
Nakajima, Ryo.
2024. "Expert Reviews and Professional Learning: Evidence from a Physician Drug Rating Platform." AEA RCT Registry. AEARCTR-0014751.
https://doi.org/10.1257/rct.14751-2.0
Download:
BibTeX
·
RIS— for reference managers without a browser
translator. Zotero and most
others pick up this page's citation metadata directly.
This study examines how expert reviews influence product discovery and observational learning in a professional healthcare setting. While previous research has documented how consumers learn about product quality through expert reviews in consumer goods markets, little is known about whether these learning mechanisms persist in professional settings where reviewers have substantial domain expertise. In markets with a vast array of products, expert reviews can serve as a crucial mechanism for product discovery and quality assessment. We conduct a randomized controlled trial on an online medical platform exclusively for licensed physicians, hosting over 600,000 drug reviews. Focusing on a selected set of widely prescribed medications, we causally identify how exposure to reviews from different types of experts—top reviewers, veteran prescribers, and recognized opinion leaders—affects other physicians' rating behaviors and learning processes about drug quality. By focusing on a setting with medical professionals, this study provides novel insights into the role of expert reviews in professional observational learning and contributes to our understanding of how physicians discover and evaluate pharmaceutical products.
Intervention
The intervention is conducted on a secure web-based medication review platform that is accessible exclusively to licensed physicians. On this platform, physicians evaluate medications by submitting a numerical rating (on a five-point overall satisfaction scale) and providing detailed written comments based on their clinical experiences.
To examine how expert reviews influence physicians’ evaluations, participants are randomly assigned at the review level to one of five experimental groups:
Treatment Groups Group A: Shown evaluations from Veteran Reviewers Group B: Shown evaluations from Top Reviewers Group C: Shown evaluations from Authority Reviewers Group D: Shown evaluations with reviewer identity anonymized
Control Group Group O: Shown no expert evaluations
Each expert evaluation includes both a numerical satisfaction rating and a written comment describing the expert's experience with the medication.
Experimental design
The study employs a randomized controlled trial design with four treatment arms and one control arm. The study participants are licensed physicians registered on the web-based review board who actively prescribe medications in the relevant therapeutic areas. Participating physicians are randomly assigned based on their member ID to one of the three expert review types, a review without reviewer type information (treatment), or no expert review (control) when evaluating medications. The study focuses on medications commonly used for treating major chronic diseases and psychiatric disorders.
Design details
Randomization method
Randomization is conducted by computer at the time physicians submit medication reviews on the web-based review board. Using a random number generator, each reviewing physician is randomly assigned to one of the five experimental groups (four treatment groups or control) before they begin their medication evaluation. The randomization is conducted at the physician level to ensure that each physician consistently receives or does not receive expert reviews throughout the study period, preventing potential contamination between treatment and control conditions.
Randomization unit
The unit of randomization is the reviewing physician. When a physician initiates a medication review on the platform, they are randomly assigned to either one of the four treatment groups (shown a review) or the control group (no review shown).
Clusters
Not applicable as randomization occurs at the individual physician-review level rather than cluster level.
Observations
For each medication analyzed, the target minimum sample size is 270 reviews (54 reviews × 5 groups) if control variables are included, or 135 reviews (27 reviews × 5 groups) if control variables are not included.
However, these sample sizes apply to the full experimental scheme, which includes four treatment groups (A, B, C, D) and one control group. If only the partial experimental scheme (with Treatment Groups A, D, and the Control Group) is conducted, the total sample size will be smaller. Nevertheless, we aim to secure at least 54 reviews per group if control variables are included, or 27 reviews per group if control variables are not included.
Arms
The study will collect data from medication reviews across multiple therapeutic areas. For each medication analyzed, we target at least 54 reviews per group (treatment groups A, B, C, D and control group O) to achieve 80% power at a 0.05 significance level for detecting a medium effect size. In this analysis, we perform three separate regression analyses, each comparing one treatment group (A, B, or C) with the control group (O).
For each medication analyzed:
- Treatment Group A (Veteran Reviewer): 54 reviews
- Treatment Group B (Top Reviewer): 54 reviews
- Treatment Group C (Authority Reviewer): 54 reviews
- Treatment Group D (Non-specified Reviewer): 54 reviews
- Control Group O: 54 reviews
This setup allows for sufficient statistical power across three distinct comparisons: (A vs. O), (B vs. O), (C vs. O), and (D v.s O).
For each medication analyzed:
-Treatment Group A (Top Reviewer): 54 reviews
- Treatment Group B (Veteran Reviewer): 54 reviews
- Treatment Group C (Authority Reviewer): 54 reviews
- Control Group O: 54 reviews
This setup allows for sufficient statistical power across three distinct comparisons: (A vs. O), (B vs. O), and (C vs. O).
Primary outcomes
1. Physicians' medication satisfaction ratings (5-point scale)
2. Semantic similarity between general physicians' review comments and expert review comments, measured using natural language processing techniques
Secondary outcomes
1. Time spent completing medication reviews
2. Complexity measures of review comments, including:
- Average sentence length and token count
- Vocabulary diversity (Type-Token Ratio)
- Linguistic sophistication measures
Cited by
No citation data available for this registration yet.