The radiative decays of heavy mesons and heavy baryons are studied in a formalism which incorporates both the heavy quark symmetry and the chiral symmetry. The chiral Lagrangians for the electromagnetic interactions of heavy hadrons consist of two pieces: one from gauging electromagnetically the strong-interaction chiral Lagrangian, and the other from the anomalous magnetic moment interactions of the heavy baryons and mesons. Due to the heavy quark spin symmetry, the latter contains only one independent coupling constant in the meson sector and two in the baryon sector. These coupling constants only depend on the light quarks and can be calculated in the nonrelativistic quark model. However, the charm quark is not heavy enough and the contribution from its magnetic moment must be included. Applications to the radiative decays $D^\ast \rightarrow D \gamma~,~B^\ast \rightarrow B \gamma~,~ \Xi^\prime_c \rightarrow \Xi_c \gamma~, \Sigma_c \rightarrow \Lambda_c \gamma$ and $\Sigma_c \rightarrow \Lambda_c \pi \gamma$ are given. Together with our previous results on the strong decay rates of $D^\ast \rightarrow D \pi$ and $\Sigma_c \rightarrow \Lambda_c \pi$, predictions are obtained for the total widths and branching ratios of $D^\ast$ and $\Sigma_c$. The decays $\Sigma^+_c \rightarrow \Lambda^+_c \pi^0 \gamma $ and $\Sigma^0_c \rightarrow \Lambda^+_c \pi^- \gamma $ are discussed to illustrate the important roles played by both the heavy quark symmetry and the chiral symmetry.