Silicone adhesive has been widely used in assembly glass curtain walls which leads to its mechanical behaviors under heavy investigation. Due to its complexity, the microstructure of silicone adhesive stays unclear. A detailed description of microstructures of silicone adhesive is the basis to obtain an in-depth understanding of its mechanical behavior and the related mechanisms. In this work, series of multiscale experiments, uniaxial tension, uniaxial compression, simple shear and scanning electron microscopy (SEM) observation, were conducted to characterize mechanical behaviors and microstructure features of silicone adhesive. Stress-strain curves obtained through macroscale experiments were presented. Morphology and dispersion of filler clusters in silicone adhesive were analyzed using Gaussian mixed models. A modification of molecular chain spatial distribution was made to non-affine network model to consider anisotropic damage evolution of polymer matrix. A simplified term was further introduced to account for load-bearing capacity of filler clusters based on fractal theory where fractal dimensions of clusters work as a connection between microscale and macroscale. The modified model was further extended to describe Mullins effect combined with network alteration theory. The results of model validation and comparison demonstrated that due to the consideration of polymer chain anisotropic evolution and filler cluster microscopic features, the proposed model can predict the mechanical behavior of silicone adhesive under different loading conditions as well as the stress softening and permanent set of Mullins effect more accurately.