In this paper, the properties of the symmetry energy in asymmetric nuclear matter, including the quadratic and quartic symmetry energies as well as their density slopes at the saturation density, are studied by the radioactivity of unstable nuclei. The symmetry energy was initially introduced as a quantum correction energy in the Weizsacker-Bethe liquid-drop model, which was proposed to describe the binding energies of finite nuclei. In order to systematically study the properties of the symmetry energy, the concept of the nuclear matter was introduced, which consists of neutrons and protons with finite neutron and proton densities but infinite neutron and proton numbers. By comparing the properties of the isospin symmetric nuclear matter, in which the neutron density is equal to the proton density, and those of the isospin asymmetric nuclear matter, in which the two densities are unequal, the symmetry energy is found to approximately represent the difference of the energy per nucleon in them. That is to say, the symmetry energy is an evaluation of the energy cost to convert protons in symmetric nuclear matter to neutrons in asymmetric nuclear matter. Since the symmetry energy characterizes the isospin asymmetric effect in nuclear matter, it plays critical roles in not only nuclear physics but also astrophysics. For example, the symmetry energy is closely related to the nuclear masses, the structures and properties of nuclei far away from stability line and near drip lines, the mechanism of heavy-ion reactions, the structures and component of neutron stars, the masses and radii of neutron stars, the cooling process of neutron stars, and so on. Besides of the quadratic symmetry energy, the quartic symmetry energy can have obvious effects on the properties of neutron stars as well. The quartic symmetry energy can significantly affect both the proton fraction in β-stable neutron stars and the critical density for the direct Urca process, which leads to faster cooling of neutron stars. In addition, the quartic symmetry energy is found to be very important for the location of the inner edge of crusts and the core-crust transition density and pressure in neutron stars. So the quartic symmetry energy can also have an influence on the structure of neutron stars. Because of the very important roles of the symmetry energy playing in not only nuclear physics but also astrophysics, it is of great significance to study the properties of both the quadratic and quartic symmetry energies. Based on the fundamental Hugenholtz-Van Hove (HVH) theorem, the properties of the symmetry energy are found to be closely related to the isoscalar and the isovector parts of the nucleon single-particle potentials, where the latter two parts can be extracted from the nuclear radioactivity. So by linking both the symmetry energy and the nuclear radioactivity to the nucleon single-particle potentials, the symmetry energy can be directly studied by nuclear radioactivity. In this paper, both the heavy-cluster radioactivity and the proton radioactivity are employed to constrain the properties of not only the quadratic symmetry energy but also the quartic symmetry energy. Besides, the effect of both the spectroscopic factor of proton radioactivity and the deformation in the daughter nucleus on the extracted results are also discussed, which are subsequently found to be very limited.