Actin-binding proteins (ABPs) are essential for the regulation of morphological plasticity required for tumor cells to metastasize. The aim of this study was to perform an unbiased bioinformatic approach to identify the key ABPs significantly associated with the metastatic potential of breast cancer cells. Microarray data from 181 primary breast cancer samples from our hospital were used, and all genes belonging to the Gene Ontology term actin cytoskeleton organization were obtained from QuickGO. Association with metastasis-free survival probability was tested using Cox proportional hazards regression, and pairwise co-expression was tested by Pearson correlations. Differential expression between different subgroups was analyzed using Wilcoxon tests for dichotomous traits and Kruskal-Wallis tests for categorical traits. Validation was performed using four publicly available breast cancer datasets. ARHGAP25 was significantly associated with a low metastatic potential, and CFL1, TMSB15A, and ACTL8 were significantly associated with a high metastatic potential. A significantly higher expression of CFL1, TMSB15A, and ACTL8 mRNA was found in the more aggressive Her2-positive and triple-negative subtypes as well as in ER-negative samples. Also, these genes were co-expressed in the same tumors. However, only mRNA levels of CFL1 were increased in pN1 compared to pN0 patients. External validation revealed that CFL1 and TMSB15A had significant associations with consistent hazard ratios in two breast cancer cohorts, and among these, CFL1 exhibited the highest hazard ratios. CFL1 showed the strongest correlation with the metastatic potential of breast tumors. Thus, targeted inhibition of CFL1 might be a promising approach to treat malignant breast cancer cells.