The paper proposes the use a metamaterial structure for improving the transmission properties of open waveguide probe, developing a frequency selective sensor in microwave frequency X band sensitive to changing of properties of investigated dielectric material. This assembly can be used for properties of dielectric material investigation. The numerical simulations of electromagnetic wave propagation in rectangular waveguide with metamaterial structure were performed using the FEM and FDTD. The numerical results were proved with experiments, testing two types of biological materials.