To improve methods of laser hyperthermia for the treatment of bulk malignant neoplasms, an urgent task is the development of techniques and devices that automatically control heating at a given tissue depth and ensure its uniformity. The article proposes the concept of a system for performing hyperthermia with real-time spectroscopic temperature control and surface cooling, which allows to record spectra of diffusely scattered radiation and fluorescent signal from various depths of biological tissues by the means of the variation of the angle and distance between the fiber source of laser radiation and the receiving fiber. Theoretical and experimental modeling of the spatial distribution of diffusely scattered radiation and temperature inside the tissue with a fiber optic device providing surface cooling of the irradiated tissue, and recording spectral information from a given depth in real time, is presented. Simulation of radiation propagation in biological tissues, depending on the distance between the source and the receiver and the angle of their tilt, was carried out using the Monte Carlo method. Modeling of the temperature distribution inside the tissues was carried out by means of a numerical solution of the heat conduction equation. Experimental modeling was carried out on phantoms of biological tissues simulating their scattering properties as well as accumulation of the investigated nanoparticles doped with Nd3+ ions. It was shown that inorganic nanoparticles doped with rare-earth Nd3+ ions can be used as temperature labels for feedback to the therapeutic laser. According to the results of the theoretical simulation, optimal configurations of the relative arrangement of the fibers were chosen, as well as the optimum surface cooling temperatures for the given power densities. The heating of the phantom of the neoplasm containing the investigated nanoparticles doped with Nd3+ ions by laser radiation with an 805-nm wavelength and power density of 1W/cm2 up to 42°C at a depth of 1cm while maintaining the surface temperature within the limits of the norm was demonstrated.