Object discrimination plays an important role in an infrared (IR) imaging system. However, at a long observing distance, the presence of detector noise and the absence of robust features make space objects' discrimination difficult to tackle with. In this paper, a multi-scale convolutional neural network (MCNN) is proposed for feature learning and classification. It consists of three parts: transformation, local convolution, and full convolution. Different from previous objects' classification methods, the MCNN can automatically extract features of objects at multi-timescales and multi-frequencies. Low-level features are combined with high-level features to simultaneously capture long-term tendency and short-term fluctuations of the time sequences of IR radiation intensity. Training data are generated from IR radiation models considering micro-motion dynamics and inherent properties of space point objects under different scenarios. The simulation results indicate that our method not only promotes the performance but is also robust to the detector noise. The classification accuracy can reach 96% at a strong noise level (signal-to-noise ratio is 10 dB) in a simulation scenario.