Abstract Study question Could insulin-like peptide 3 (INSL3) predict ovarian reserve and in vitro fertilization (IVF) success in women with unexplained infertility (UI) and diminishing ovarian reserve (DOR)? Summary answer In women with DOR, reduced intrafollicular and serum INSL3 levels correspond with low AMH and high FSH levels, similar to established ovarian reserve markers. What is known already INSL3, belonging to the insulin-like peptide family, is produced by theca interna cells within antral follicles and the corpora lutea. It is hypothesized that INSL3 is integral to the initial development and function of antral follicles, specifically through its regulatory effect on androgen biosynthesis in the thecal cells of these follicles. Moreover, INSL3 is implicated in the modulation of the ovarian microenvironment, which is essential for facilitating the maturation of oocytes. Study design, size, duration This research was carried out at the IVF centers of two tertiary university hospitals, assessing 49 infertile women with either UI (n = 24) or DOR (n = 25) during IVF treatment. A control group (n = 26) comprised women with normal ovarian reserve undergoing IVF due to male or tubal factors. Outcome parameters are ovarian reserve markers (serum FSH, estradiol, AMH, INSL3 concentration and antral follicle count [AFC]), positive pregnancy rate and live birth rate (LBR) (gestational age ≥28 weeks). Participants/materials, setting, methods Women aged between 20-40 with a body mass index (BMI)<30 kg/m² were included. Serum FSH, estradiol, AMH, INSL3 levels and AFC were evaluated on menstrual cycle days 2 or 3, and INSL3 in follicular fluid from dominant follicles (>14 mm) were evaluated during oocyte retrieval. Exclusions included endocrine abnormalities (e.g., polycystic ovary syndrome, hyperprolactinemia, hypo and hyperthyroidism) cycle cancellation due to lack of a viable embryo, uncorrected Mullerian anomalies, recurrent miscarriage history and hydrosalpinx presence. Main results and the role of chance The mean age in the study groups was 28.41±3.5 (UI), 32.40±4.21 (DOR), and 29.65±3.91 (control). In the DOR group, female age (p = 0.001), BMI (p = 0.049), duration of infertility (p = 0.006), previous IVF attempts (p < 0.001), basal FSH (p < 0.001), and basal estradiol (p < 0.001) were higher, while AMH (p < 0.001), AFC (p < 0.001), and follicular fluid INSL3 (p < 0.001) were lower. Serum INSL3 in DOR was lower but not statistically significant. No correlation was found between serum INLS, follicular fluid INSL3, ovarian reserve markers, positive pregnancy rates, miscarriage, and LBR. DOR group had longer stimulation durations (p = 0.017), but lower max estradiol (p < 0.001), fewer oocytes (p < 0.001), mature follicles (p < 0.001), and PN (p < 0.001). Pregnancy rates were 54.1% (UI), 24% (DOR), and 53.84% (control); LBRs were 54.1% (UI), 20% (DOR), and 42.3% (control), with significant differences in pregnancy tests (p = 0.04) and LBR (p = 0.03), both lower in DOR. Univariate analysis identified duration of infertility (p = 0.019) and basal FSH (p = 0.009) as significant LBR predictors. Multivariable analysis confirmed basal FSH (p = 0.033) as significant, with increased duration of infertility and basal FSH level reducing LBR probability. Limitations, reasons for caution The limited sample size of our study potentially undermines the robustness of circulating INSL3 as a predictive biomarker for DOR, evidenced by the non-significant decrease in serum INSL3 levels in the DOR group. Moreover, the lack of INSL3 level assessment during oocyte retrieval further constrains our findings. Wider implications of the findings Our research explored INSL3 as a viable biomarker for predicting ovarian reserve and IVF success in UI and DOR cases. Reduced INSL3 levels in circulation and follicular fluid in DOR patients could be beneficial in clinical practice. Trial registration number not applicable