Background/Objectives: This study aims to create a strong binary classifier and evaluate the performance of pre-trained convolutional neural networks (CNNs) to effectively distinguish between benign and malignant ovarian tumors from still ultrasound images. Methods: The dataset consisted of 3510 ultrasound images from 585 women with ovarian tumors, 390 benign and 195 malignant, that were classified by experts and verified by histopathology. A 20% to80% split for training and validation was applied within a k-fold cross-validation framework, ensuring comprehensive utilization of the dataset. The final classifier was an aggregate of three pre-trained CNNs (VGG16, ResNet50, and InceptionNet), with experimentation focusing on the aggregation weights and decision threshold probability for the classification of each mass. Results: The aggregate model outperformed all individual models, achieving an average sensitivity of 96.5% and specificity of 88.1% compared to the subjective assessment's (SA) 95.9% sensitivity and 93.9% specificity. All the above results were calculated at a decision threshold probability of 0.2. Notably, misclassifications made by the model were similar to those made by SA. Conclusions: CNNs and AI-assisted image analysis can enhance the diagnosis and aid ultrasonographers with less experience by minimizing errors. Further research is needed to fine-tune CNNs and validate their performance in diverse clinical settings, potentially leading to even higher sensitivity and overall accuracy.