The use of bio-based building materials such as hemp concrete has increased considerably in France in the last few years, due to its hygrothermal and acoustic advantages. Hemp concrete is hygroscopic and very sensitive to relative humidity. This property is responsible for dimensional variations and therefore the degradation of the material. To evaluate these dimensional variations, 2D and 3D imaging techniques can be used. For 3D tools, significant research and financial investments are required. This study is based on the analysis of the 2D microstructure of both hemp concrete and hemp shiv aggregates, in order to understand and estimate these dimensional variations for sorption and desorption cycles. First, the interaction between the different constituents of these bio-based materials was performed using scanning electron microscopy (SEM). Then, a new experimental optical setup was investigated to quantify the hemp shiv and hemp concrete swelling and shrinking by analyzing variations in their microstructure at different relative humidity levels. The quantification is calculated by 2D image analysis function of the fibers direction. Finally, digital image correlation (DIC) was performed to allow better estimation of the hemp concrete swelling. The findings of this study underline the real influence of relative humidity on the hemp concrete, that present a hysteretic swelling due to the porous structure and hygroscopic behavior of this material. The resulting dimensional variations should have to be seriously taken into account when modeling the hygrothermal behavior of this material.