Esophageal squamous cell carcinoma (ESCC) is one of the most common and deadly cancers in the world. Currently, the five-year survival rate of ESCC remains poor. Studies show that miRNAs plays an important role in various tumor processes. Our study aimed to discover prognostic miRNA and investigate its role in ESCC. Prognosis related miRNA was detected by ABI TaqMan miRNA chip method using serum of 15 ESCC patients with T3-4/N1, M0-1 stage. The cell proliferation was evaluated by Cell Titer 96 AQueous One Solution Cell Proliferation Assay (MTS), and the migration/invasion was evaluated by transwell assay. The bioinformatics methods were applied to predict the targets of miRNA. The mRNA and protein levels of miRNA target were determined by quantitative real time polymerase chain reaction (qRT-PCR) analysis, western blot and enzyme-linked immunosorbent assay (ELISA). Four weeks old NU/NU nude mice were used for the in vivo tumorigenicity study, including subcutaneous injection for tumor volume evaluation and tail vein injection for metastasis observation. Downregulated miR-323a-3p was closely correlated with the shorter survival of ESCC. In most of ESCC cell lines, such as YSE2, KYSE 150, KYSE 510, the expressions of miR-323a-3p were largely lower than esophageal epithelial cells. MiR-323a-3p repressed cell proliferation of KYSE 30 and KYSE 150. Moreover, miR-323a-3p inhibited cell migration and invasion of YES 2 and KYSE 30. These results further confirmed the tumor suppressor status for miR-323a-3p. Through biological predictions, FMR1 was one of the possible targets of miR-323a-3p. The expression of this miRNA and FMR1 showed negative correlation. The compromised cell migration and invasion induced by miR-323a-3p mimic transfection were rescued by transfection of FMR1 expression plasmid in both YSE2 and KYSE 30. In vivo, tumors induced by KYSE 30-miR-323a-3p grew significantly slower and resulted in smaller tumor mass. Metastatic lung colonization could be inhibited by overexpression of miR-323a-3p in KYSE 30. MiR-323a-3p acts as a tumor suppressor by targeting FMR1. The overexpression of miR-323-3p might offer a potential therapeutic target against metastasis in patients with ESCC.