Monocular image-based 3D reconstruction is widely used in virtual reality, augmented reality, and autonomous driving, which benefits from the rapid development of deep learning approaches. Most of the available methods focused on reconstructing the overall shape of the object while ignoring some fine-grained details. Moreover, these methods make it hard to exactly reconstruct complex topological structures. In this paper, we propose a multi-granularity relationship reasoning network (MGRRNet), which aims to recover 3D shapes with high fidelity and rich details via the relationship reasoning between different granularity information. Specifically, our model captures the discriminative and detailed features at different granularities for extracting attentional regions. Then we perform the relationship reasoning between different granularities to reinforce the multi-granularity consistency and inter-granularity correlation. By doing this, our network is able to achieve robust feature representation and fine reconstruction. During the learning process, we jointly optimize procedures of different granularity feature representations via a sequence of inter-granularity cycle loss iterations. Extensive experimental results on two publicly available datasets justify that our approach achieves competitive performance compared to the state-of-the-art methods. Codes and all resources will be publicly available at https://github.com/Ray-tju/MGRRNet.