Modeling behavior of broadband (30 to 1000 MHz) frequency modulated near-infrared (NIR) photons through a phantom is the basis for accurate extraction of optical absorption and scattering parameters of biological turbid media. Photon dynamics in a phantom are predicted using both analytical and numerical simulation and are related to the measured insertion loss (IL) and insertion phase (IP) for a given geometry based on phantom optical parameters. Accuracy of the extracted optical parameters using finite element method (FEM) simulation is compared to baseline analytical calculations from the diffusion equation (DE) for homogenous brain phantoms. NIR spectroscopy is performed using custom-designed, broadband, free-space optical transmitter (Tx) and receiver (Rx) modules that are developed for photon migration at wavelengths of 680, 780, and 820 nm. Differential detection between two optical Rx locations separated by 0.3 cm is employed to eliminate systemic artifacts associated with interfaces of the optical Tx and Rx with the phantoms. Optical parameter extraction is achieved for four solid phantom samples using the least-square-error method in MATLAB (for DE) and COMSOL (for FEM) simulation by fitting data to measured results over broadband and narrowband frequency modulation. Confidence in numerical modeling of the photonic behavior using FEM has been established here by comparing the transmission mode's experimental results with the predictions made by DE and FEM for known commercial solid brain phantoms.