Hard carbon is considered to be one of the most promising anodes for sodium-ion batteries (SIBs) owing to its high capacity and abundant resources. However, the role of particle size on sodium ion storage is unclear, which leads to low capacity and initial coulombic efficiency (ICE) in practical application. In this work, a series of hard carbons with different particle sizes were prepared by an “up to down” strategy using simple grinding and ball-milling method to investigate the effect of particle size on electrochemical response of sodium ion storage. The particle size of hard carbon has negligible effect on initial specific capacity. However, it has a strong effect on the ICE and rate capability. The ICE reduces as the particle size decreases, but the rate performance in reverse. Moreover, impedance analysis and electrochemical kinetics show huge differences for different particle sizes. In-situ Raman technique was also adopted to further illustrate the sodium ion storage mechanism of hard carbon, and an “adsorption-intercalation-pore filling” mechanism is proposed. This work could provide a new perspective for the design of hard carbon materials with suitable structure for efficient sodium ion storage, helping to develop high performance SIBs.