Radiation detection has been a main interest for researchers as all kind of produced particles in atomic and subatomic physics based on the measurement systems so-called detector. Detection efficiency is one of the main parameters in detection system besides many other different parameters of the detector. The detector in experimental physics is an instrument that converts radiation energy into an electrical signal, and this is achieved basically by either ionization or excitation. The choice for any type of a detector (gas-filled, scintillation or semiconductor) for any application depends upon the X-ray of gamma energy range of interest. A working model is therefore developed which is capable of describing the overall NaI(Tl) detection efficiency as a function of several known parameters. The attenuation coefficients for the bismuth borate glasses with different concentrations were measured using gamma spectroscopy technique. The numerical absolute efficiency calibration of a detector can be determined by In-Situ Object Calibration Software (ISOCS) and Monte Carlo Neutral Particle version 5 (MCNP5) techniques which does not require any calibration standards or reference materials. By using the ISOCS and MCNP5 methodologies, the full energy peak efficiency of a scintillator detector (3“X3” NaI (Tl)) exposed to Co-60 and Cs-137 gamma ray sources with average accuracy range 0.126–1.224% for the used samples can be detected. The used materials are ternary and are located between the detector and the source to determine the attenuation coefficients for these samples by using the calculated full energy peak efficiencies of a detector. The average accuracy ranged from − 1.808 to 1.960% for linear attenuation coefficient ( $$\mu $$ ), while it ranged from − 1.999 to 1.888% and from − 1.924 to 1.960% for half value layer (HVL) and mass linear attenuation coefficient ( $$\mu _m$$ ), respectively. The calculated values of the absolute full energy peak efficiency have been used to determine the attenuation coefficients of materials with different concentrations and different densities. The results proved the validation of ISOCS and MCNP to determine the absolute full energy peak efficiency of the detector which can be used to determine the attenuation coefficients for the simulated samples and it is a good tool to be used when experimental methods are not available.