We present a theory to describe conductance fluctuations in ballistic metallic point contacts. The theory is based on interference of electron waves backscattered to the constriction of the device. Since the number of scattering events in a backscattered trajectory is small, the concept of local interference can be applied to make a first-principles calculation of the interference effect in the framework of wave optics. In the theory three different limits can be distinguished, depending on whether there is interference of remote trajectories, of near trajectories, or of a remote and a near trajectory. Interference of near trajectories leads to the estimate of Holweg et al. based on the Landauer formula, which for the amplitude of the fluctuations \ensuremath{\delta}G agrees with the experiments of these authors, but which cannot properly account for the characteristic magnetic-field scale ${\mathit{B}}_{\mathit{c}}$. Interference of remote trajectories, on the other hand, reproduces the result of Maslov et al., also based on the Landauer formula, which can explain ${\mathit{B}}_{\mathit{c}}$, but which predicts a much too small \ensuremath{\delta}G. Finally, a combination of near and remote trajectories leads to the proper description of the experimental data. The remote trajectory, which spreads over a region of the size of the elastic mean free path, controls ${\mathit{B}}_{\mathit{c}}$ of the fluctuations, while the near trajectory causes a substantial enhancement of \ensuremath{\delta}G above the value predicted by Maslov et al. Very good agreement with the experiments is obtained by taking into account enhanced scattering close to the constriction (a likely effect for the type of contacts studied), which gives an additional enhancement of \ensuremath{\delta}G. In this way, our theory successfully predicts both the value of \ensuremath{\delta}G and ${\mathit{B}}_{\mathit{c}}$ in a self-consistent way, lifting previous contradictions.