Understanding fish distribution, behavior, and abundance is crucial for marine ecological research, fishery management, and environmental monitoring. However, the distinctive features of the underwater environment, including low visibility, light attenuation, water turbidity, and strong currents, significantly impact the quality of data gathered by underwater imaging systems, posing considerable challenges in accurately detecting fish objects. To address this challenge, our study proposes an innovative fish detection network based on prior knowledge of image degradation. In our research process, we first delved into the intrinsic relationship between visual image quality restoration and detection outcomes, elucidating the obstacles the underwater environment poses to object detection. Subsequently, we constructed a dataset optimized for object detection using image quality evaluation metrics. Building upon this foundation, we designed a fish object detection network that integrates a prompt-based degradation feature learning module and a two-stage training scheme, effectively incorporating prior knowledge of image degradation. To validate the efficacy of our approach, we develop a multi-scene Underwater Fish image Dataset (UFD2022). The experimental results demonstrate significant improvements of 2.4% and 2.5%, respectively, in the mAP index compared to the baseline methods ResNet50 and ResNetXT101. This outcome robustly confirms the effectiveness and superiority of our process in addressing the challenge of fish object detection in underwater environments.