We study the electromagnetic scattering problem from multiple arbitrarily shaped cavities filled with inhomogeneous anisotropic media. A local-refined Petrov-Galerkin finite element interface (PGFEI) method is developed for solving the scattering problem. By introducing the transparent boundary condition, the unbounded scattering problem can be reduced into bounded domain problem with coupled boundary conditions on the apertures of multiple cavities. The multiple cavity scattering in transverse magnetic (TM) and transverse electric (TE) polarizations are both investigated based on uniform non-body-fitted meshes. We adopt the level-set functions to capture the irregular cavity wall and the interfaces of inhomogeneous media. The special basis is constructed to satisfy the jump conditions across the interfaces of different materials further. The local-refined algorithm is proposed to enhance the accuracy of the solutions around the discontinuous interfaces. The generation of the grid points facilitates a special numbering scheme which enables a stepwise Schur complement decomposition of the discrete linear system derived by the PGFEI method. It could be an alternative method for solving the scattering problems which have similar structures. The proposed method can handle matrix coefficients caused by the permittivity and permeability tensor of anisotropic media. Numerical experiments illustrate the validity and efficiency of the method for solving the scattering from multiple arbitrarily shaped cavities with inhomogeneous anisotropic media in TM and TE polarization.