When solving the problems of deformation solid mechanics, the inhomogeneous composite material is modeled as homogeneous, with averaged mechanical properties − effective characteristics. The purpose of this paper is to develop a technique for determining the effective shear modulus for a viscoelastic fiber composite with a transtropic matrix and fiber. Their isotropy planes coincide and are perpendicular to the fiber axis. The effective shear modulus is defined as a function of the matrix and fiber mechanical properties and the volume content of each of them in a composite. A unidirectional composite material with a hexagonal fiber stacking scheme and a unit cell consisting of a viscoelastic matrix and elastic fiber is considered. The geometric model of a composite is a combination of two coaxial infinite cylinders: a hollow cylinder, modeling the matrix, and a solid one, modeling the fiber inserted into it. The volume of the hexagonal cell is approximated by the volume of the cylinder. The radius of the cylinder is chosen so that the fiber volume content in the hexagonal cell coincides with the value of this characteristic for the cylindrical cell. To describe the viscoelastic properties of a composite, the ratios of the hereditary Boltzmann-Volterra theory are used. The shear modulus is defined as an integral operator with a difference kernel. Two boundary problems are considered: with regard to the longitudinal shear of a transonic viscoelastic solid cylinder modeling the composite, and the joint longitudinal shear of the hollow and solid cylinders that model the matrix and fiber materials, respectively. It is assumed that the displacements and tangential stresses on the contact surface of the matrix and fiber are continuous. A tangential harmonic load is applied on the outer surface of the cylindrical cell. To solve such problems, the Laplace transform is used. As the matching condition, the equality of displacements on the outer surface of the cylinder is used for the two problems. The application of the proposed technique makes it possible to determine the characteristics of the integral operator describing the shear modulus for a viscoelastic composite material. An instantaneous shear modulus and relaxation core parameters are found as the functions of the known mechanical characteristics of the matrix and fiber. As an example, the characteristics of the shear modulus for a composite material consisting of a rubber matrix and polyamide fiber are determined.