CAN AI COMPETE WITH 88 EMBRYOLOGISTS AND EXPERTS ON MORPHOLOGICAL EVALUATION AND RANKING OF BLASTOCYST IMAGES?

OBJECTIVE:

Manual morphological grading is somewhat subjective and weakly correlated with live birth propensity. Can an AI model (Vita Embryo, S Korea), designed for morphological assessment of static images, reduce variability of blastocyst grading, compared to expert or embryologist assessors reviewing the same embryos?

MATERIALS AND METHODS:

Eighty-eight embryologists, from 19 networked clinics, of which 14 were categorised as expert, based on seniority, evaluated 22 day 5 blastocysts for Gardner morphological grade from static images in 3 focal planes, extracted from Embryoscope, Vitrolife. Each assessor determined development stage, (Dev), inner cell mass (ICM) and trophectoderm (TE), with 78 assessors prioritising, 1^st, 2^nd and 3^rd choice based on grade. The AI model generated a morphological score and ranking using the optimal focal image. Consensus grades were defined from the expert scores as the highest frequency grade. Agreement between the consensus grade, experts (n=14), embryologists (n=74) and AI scores, was determined using quadratic weighted Cohen’s kappa (κ). The reference order, 1st choice and top 3 was identified from the expert assessors, using Borda score to compare to all groups, 1^st choice, required an exact match and the top 3, matching to any embryos present in the assessors top 3 regardless of order.

RESULTS:

Two embryos were excluded as ICM and TE grades, were not recorded by >50% of assessors as development stage was determined as early blastocyst. Grade completion for the remaining embryos was Dev (100%), ICM (99.8%) and TE (99%) from 1760 potential assessments per grade (88x20). The AI provided complete scores for all embryos (n=20). Both expert and AI grading agreement was higher for Dev and TE compared to the embryologist; Dev 0.857 ± 0.1 & 0.885 vs 0.824 ± 0.11, TE; 0.808 ± 0.14 & 0.824 vs 0.770 ± 0.15 (k ± SD) respectively. Lower agreement was observed for ICM compared to Dev or TE with similar results across all groups; 0.780 ± 0.17, 0.785 ± 0.13, and 0.706, experts, embryologists and AI. Embryologist ranking was lower, suggesting greater variability for 1st choice and top 3 selection; 0.478 ±0.5 vs 0.545±0.52 and 1.687 ±0.72 vs 1.727 ± 0.9 respectively, with the AI showing perfect ranking to the 1^st choice and matching 2.0 for top 3.

CONCLUSIONS:

The experts demonstrated greater consistency when determining development stage, TE and ranking embryos over the embryologists. ICM grading had greater variability regardless of experience suggesting this important component is inherently more challenging to grade. The AI system showed performance comparable to the expert group and may help remove variability between embryologists with different levels of experience and as grading is instantaneous, could save time.

IMPACT STATEMENT:

AI-based embryo assessment may help improve experience-related consistency and support more standardised embryo evaluation to improve embryo selection practices and correlation to clinical outcomes.