Evaluating performance across diverse clinical settings
For a medical AI system to safely scale globally, it must demonstrate generalizability — maintaining accurate diagnostic performance when introduced to new clinical settings with different patient populations, clinical workflows, and ultrasound hardware variations than those on which the model was trained.
In this study, our original AI model was minimally tuned on only one of our target populations, and then prospectively tested across both target environments:
- Northwestern Medicine: An urban, high-resource academic medical center in Chicago, IL, United States.
- Jacaranda Health: An urban community clinic network in Nairobi, Kenya.
Originally trained on ultrasound data collected in North Carolina and Zambia using a set of Sonosite, GE, and Butterfly Networks commercial ultrasound probes, the research AI system was paired with Clarius C3, a commercial handheld transabdominal probe. All of these sites differed in various care dimensions, including when women first come in for prenatal care, how many routine ultrasounds they receive, and how gestational age is usually dated.
To evaluate efficient system tuning, the study leveraged a small pilot dataset of 120 participants from Chicago — approximately 6% of the original training set—to adapt the model to new probe hardware and extend the gestational age prediction range beyond 36 weeks. To understand generalizability, no data from the Nairobi site was used for fine-tuning, maintaining the Kenyan clinical validation as a strictly independent, external test. To evaluate comprehensive utility, the study also included multiple gestation pregnancies, which could present tracking and measurement challenges for both human operators and automated systems.