Genomic prediction (GP) has emerged an effective tool for addressing the many shortcomings of traditional selective breeding, thereby enhancing the selection process. In this study, we optimized GP methods using 5-fold cross-validation to estimate genome-estimated breeding values for the weight traits of olive flounder (Paralichthys olivaceus). To accomplish our goal, we determined the parentage of the target broodstock and the ability of 11 prediction models to predict the weight traits of 1.8-year-old olive flounders, which were genotyped using a 70 K single nucleotide polymorphism (SNP) array. Moreover, our optimization efforts toward the predictive ability of genomic best linear unbiased prediction (GBLUP), Bayesian B (BB), and random forest (RF) methods encompassed changes in various aspects such as fixed effects, SNP quantity, population size, and phenotypic data collected at different fish ages. Additionally, we assessed the predictive ability for the total length and body depth of fish using GBLUP, BB, and RF. Among the 11 prediction methods used in this study, the BB (0.675), Elastic Net (0.679), and RF (0.698) methods exhibited the highest predictive abilities, whereas the GBLUP (0.637) method demonstrated the lowest. Incorporating information regarding fish sex as a fixed effect substantially improved the predictive ability of GBLUP and BB. For mean models, utilizing 3000–5000 random SNP markers resulted in a higher predictive ability, similar to that obtained using 50,000 SNPs. Increasing the population size reduced the standard deviation of the predictive ability. Notably, phenotypic records from 1.8-year-old fish exhibited a significantly higher predictive ability than those from the other age groups. Furthermore, GBLUP, BB, and RF provided higher predictive abilities for length (0.655–0.852) and body depth (0.665–0.861). These findings may significantly shape future olive flounder genomic selection programs and offer valuable insights into GP in aquaculture.