A multiscale numerical investigation into the origin of the magnetoelectric effect in multiferroic composite materials was performed. Specifically, a well-known but unconfirmed mechanism for generating a macroscopic magnetoelectric effect through transmission of mechanical strain between the ferroelectric and ferromagnetic phases in an inhomogeneous microstructure was quantitatively established through multiscale finite element simulations. An asymptotic homogenization theory was employed for scale bridging between the macrostructure and the microstructure. Focusing first on a polycrystalline BaTiO3/Terfenol-D composite, the relation between the physical properties and the ferroelectric-phase content was analyzed. A uniform magnetic field was then applied to the macrostructure and the mechanical strain in the microstructure was investigated. The specific strain component exhibited an off-centered distribution when divided into ferroelectric and ferromagnetic phases. The simulation revealed that the macroscopic magnetoelectric effect is determined by two factors, the shift of the average strain from zero and the homogenized piezoelectric constant. The macroscopic magnetoelectric effect increases as the product of these two factors increases. This conclusion is applicable to other inhomogeneous structures and it can explain the trend in the magnitude of the magnetoelectric effect for a random polycrystalline structure, a layered polycrystalline structure, and a layered single-crystal structure. The findings are helpful for the functional design of multiferroic composite materials.