In this paper, we report large-scale configuration interaction (CI) calculations of linear optical absorption spectra of various isomers of magnesium clusters Mg$_{n}$ (n=2--5), corresponding to valence transitions. Geometry optimization of several low-lying isomers of each cluster was carried out using coupled-cluster singles doubles (CCSD) approach, and these geometries were subsequently employed to perform ground and excited state calculations using either the full-CI (FCI) or the multi-reference singles-doubles configuration interaction (MRSDCI) approach, within the frozen-core approximation. Our calculated photoabsorption spectrum of magnesium dimer (Mg$_{2}$) isomer is in excellent agreement with the experiments both for peak positions, and intensities. Owing to the sufficiently inclusive electron-correlation effects, these results can serve as benchmarks against which future experiments, as well as calculations performed using other theoretical approaches, can be tested.