The recent measurement of magnetic field strength inside the radiative interior of red giant stars has opened the way toward full 3D characterization of the geometry of stable large-scale magnetic fields. However, current measurements, which are limited to dipolar (ℓ = 1) mixed modes, do not properly constrain the topology of magnetic fields due to degeneracies on the observed magnetic field signature on such ℓ = 1 mode frequencies. Efforts focused toward unambiguous detections of magnetic field configurations are now key to better understand angular momentum transport in stars. We investigated the detectability of complex magnetic field topologies (such as the ones observed at the surface of stars with a radiative envelope with spectropolarimetry) inside the radiative interior of red giants. We focused on a field composed of a combination of a dipole and a quadrupole (quadrudipole) and on an offset field. We explored the potential of probing such magnetic field topologies from a combined measurement of magnetic signatures on ℓ = 1 and quadrupolar (ℓ = 2) mixed mode oscillation frequencies. We first derived the asymptotic theoretical formalism for computing the asymmetric signature in the frequency pattern for ℓ = 2 modes due to a quadrudipole magnetic field. To access asymmetry parameters for more complex magnetic field topologies, we numerically performed a grid search over the parameter space to map the degeneracy of the signatures of given topologies. We demonstrate the crucial role played by ℓ = 2 mixed modes in accessing internal magnetic fields with a quadrupolar component. The degeneracy of the quadrudipole compared to pure dipolar fields is lifted when considering magnetic asymmetries in both ℓ = 1 and ℓ = 2 mode frequencies. In addition to the analytical derivation for the quadrudipole, we present the prospect for complex magnetic field inversions using magnetic sensitivity kernels from standard perturbation analysis for forward modeling. Using this method, we explored the detectability of offset magnetic fields from ℓ = 1 and ℓ = 2 frequencies and demonstrate that offset fields may be mistaken for weak and centered magnetic fields, resulting in underestimating the magnetic field strength in stellar cores. We emphasize the need to characterize ℓ = 2 mixed-mode frequencies, (along with the currently characterized ℓ = 1 mixed modes), to unveil the higher-order components of the geometry of buried magnetic fields and to better constrain angular momentum transport inside stars.