We propose a point-to-point convolutional neural network (CNN) model to reconstruct the electromagnetic parameters of multiple cavities filled with inhomogeneous anisotropic media. The input of the model is a grid matrix of electromagnetic field magnitude solved by Petrov–Galerkin finite element interface method, and the output is a distribution matrix of the electromagnetic parameters of the cavities. The uniform non-body-fitted mesh is set in the computational domain, which does not need to be updated repeatedly at the complex interfaces of different media. Compared with the standard finite element, this mesh reduction method effectively saves the cost of mesh generation. Level set functions are applied to capture these arbitrarily shaped interfaces. The matrix coefficients caused by the dielectric constant and permeability tensors of anisotropic media can also be handled with this method. The point-to-point CNN model effectively reconstructs the value and the distribution of electromagnetic parameters in the cavities, also predicting the shape and the position of the interfaces. Three optimized schemes are then proposed to improve reconstruction accuracy. Numerical results show that the size of the convolution kernel and the number of fully connected layers are the crucial parameters in determining interfaces and reducing the number of artefacts.