As storage density increases, the performance of volume holographic storage channels is degraded, because intersymbol interference and noise also increase. Equalization and detection methods must be employed to mitigate the effects of intersignal interference and noise. However, the output detector array in a holographic storage system detects the intensity of the incident light's wave front, leading to loss of sign information. This sign loss precludes the applicability of conventional equalization and detection schemes. We first address channel modeling under quadratic nonlinearity and develop an efficient model named the discrete magnitude-squared channel model. We next introduce an advanced equalization method called the iterative magnitude-squared decision feedback equalization (IMSDFE), which takes the channel nonlinearity into account. The performance of IMSDFE is quantified for optical-noise-dominated channels as well as for electronic-noise-dominated channels. Results indicate that IMSDFE is a good candidate for a high-density, high-intersignal-interference volume holographic storage channel.