Breast malignancy is a critical problem that severely affects women’s health globally with a high-frequency rate, necessitating fast, effective, and early diagnostic methods. The present study aims to measure the breast tissue’s optical properties by capturing the spectral signatures from malignant and normal breast tissue for therapeutic and diagnostic applications. The optical imaging system incorporates a hyperspectral (HS) camera to capture the spectral signatures for both the malignant and normal breast tissues within 400 ~ 1000 nm. The system was subdivided into two exploratory (reflection/transmission) to measure the tissue’s diffuse reflectance (Rd) and light transmission (Tr), respectively. The study involved 30 breast tissue (normal/tumor) samples from 30 females in the age range of 46 ~ 72 years, who were optically inspected in the visible and near-infrared (VIS-NIR) spectra. Then, the inverse adding doubling (IAD) method for breast tissue characterization and descriptive analysis (T-test) was exploited to verify the significant difference between the various types of breast tissues and select the optimum wavelength. Finally, comparing the study outcome with the histopathological examination to evaluate the system’s effectiveness by calculation (sensitivity, specificity, and accuracy). The average outcome values demonstrated that the optimal spectral bands distinguishing between the normal and the tumor tissues regarding the reflectance approach were 600 ~ 680 nm and 750 ~ 960 nm at the VIS and NIR spectrum, respectively. Then, for the transmission technique, the optimal spectral bands were 560 ~ 590 nm and 760 ~ 810 nm at the VIS and NIR spectra, respectively. Later, the T-test and the IAD verified that the highest Rd values for discrimination were 600 ~ 640 nm and 800 ~ 840 nm at the VIS and NIR spectra, respectively. On the other side, the highest Tr values were 600 ~ 640 nm and 760 ~ 800 nm at the VIS and NIR spectra, respectively. The investigation’s average reading accuracy, sensitivity, and specificity were 85%, 81.88%, and 88.8%, respectively. The experimental trials revealed that the system could identify the optimal wavelength for therapeutic and diagnostic applications through the light interaction behavior of the breast tissue’s optical properties.