The shrinking and swelling of wood due to moisture changes are intrinsic material properties that control and limit the use of wood in many applications. Herein, hygroscopic deformations of normal and compression wood of Chinese fir (Cunninghamia lanceolata [Lamb.] Hook.) were measured during desorption and absorption processes. The dimensional changes were observed in situ by an environmental scanning electron microscope and analyzed at different hierarchical levels (tissue, cell and cell wall). The relationship between moisture variation and hygroscopic deformation was measured. During initial desorption periods from 95 to 90 or 75% RH, an expansion of the lumen and a shrinkage of the cell wall were observed, revealing a non-uniform and directional deformation of single wood cells. The variation of shrinking or swelling at different hierarchical levels (tissue, cell and cell wall) indicates that the hygroscopic middle lamella plays a role in the deformation at the tissue level. Higher microfibril angles and helical cavities on the cell wall in compression wood correlate with a lower shrinking/swelling ratio. Normal wood showed a more pronounced swelling hysteresis than compression wood, while the sorption hysteresis was almost the same for both wood types. This finding is helpful to elucidate effects of micro- and ultrastructure on sorption. The present findings suggest that the sophisticated system of wood has the abilities to adjust the hygroscopic deformations by fine-tuning its hierarchical structures.