Ocean eddies are typical oceanic mesoscale phenomena that are numerous, widely distributed and have high energy. Traditional eddy detection methods are mainly based on physical mechanisms with high accuracy. However, the large number of steps and complex parameter settings limit their applicability for most users. With the rapid development of deep learning techniques, object detection models have been broadly used in the field of ocean remote sensing. This paper proposes a lightweight eddy detection model, ghost eddy detection YOLO (GED-YOLO), based on sea level anomaly data and the “You Only Look Once” (YOLO) series models. The proposed model used ECA+GhostNet as the backbone network and an atrous spatial pyramid pooling network as the feature enhancement network. The ghost eddy detection path aggregation network was proposed for feature fusion, which reduced the number of model parameters and improved the detection performance. The experimental results showed that GED-YOLO achieved better detection precision and smaller parameter size. Its mAP was 95.11% and the parameter size was 22.56 MB. In addition, the test experiment results showed that GED-YOLO had similar eddy detection performance and faster detection speed compared to the traditional physical method.