Pepper (Capsicum spp.) fruit-related traits are critical determinants of quality. These traits are controlled by quantitatively inherited genes for which marker-assisted selection (MAS) has proven insufficiently effective. Here, we evaluated the potential of genomic selection, in which genotype and phenotype data for a training population are used to predict phenotypes of a test population with only genotype data, for predicting fruit-related traits in pepper. We measured five fruit traits (fruit length, fruit shape, fruit width, fruit weight, and pericarp thickness) in 351 accessions from the pepper core collection, including 229 Capsicum annuum, 48 Capsicum baccatum, 48 Capsicum chinense, 25 Capsicum frutescens, and 1 Capsicum chacoense in 4 years at two different locations and genotyped these accessions using genotyping-by-sequencing. Among the whole core collection, considering its genetic distance and sexual incompatibility, we only included 302 C. annum complex (229 C. annuum, 48 C. chinense, and 25 C. frutescens) into further analysis. We used phenotypic and genotypic data to investigate genomic prediction models, marker density, and effects of population structure. Among 10 genomic prediction methods tested, Reproducing Kernel Hilbert Space (RKHS) produced the highest prediction accuracies (measured as correlation between predicted values and observed values) across the traits, with accuracies of 0.75, 0.73, 0.84, 0.83, and 0.82 for fruit length, fruit shape, fruit width, fruit weight, and pericarp thickness, respectively. Overall, prediction accuracies were positively correlated with the number of markers for fruit traits. We tested our genomic selection models in a separate population of recombinant inbred lines derived from two parental lines from the core collection. Despite the large difference in genetic diversity between the training population and the test population, we obtained moderate prediction accuracies of 0.32, 0.34, 0.50, and 0.48 for fruit length, fruit shape, fruit width, and fruit weight, respectively. This use of genomic selection for fruit-related traits demonstrates the potential use of core collections and genomic selection as tools for crop improvement.