The United States provide Element Sets (ELSET) database in Two-Line Element (TLE) format for public use, which plays an important role in the inversion of atmospheric density in the thermosphere, ballistic coefficient estimation, early-warning and so on. Due to large uncertainties existing in the TLE generation process, space environment changes, and space events, ELSET database contains a large number of abnormal TLE data to be filtered, such as corrected TLE, orbital element outlier, and Bstar outlier. The existing methods to filter out the outliers lack general applicability and are very complicated, which are only applicable to a few space targets in certain orbit regions. To overcome the shortcomings of the existing methods, a filtering method is proposed based on Expectation Maximization (EM) algorithm employing a sliding window and polynomial fitting method, which can detect outliers for different orbital elements and space events. The research shows that the algorithm can effectively single out the outliers in TLE sequences and is suitable for all orbital debris.