In this paper we formulate the time-dependent many-body theory of photoassociation in an atomic Bose-Einstein condensate with realistic interatomic interactions, using and comparing two approximations: the first-order cumulant approximation, originally developed by Kohler and Burnett [Phys. Rev. A 65, 033601 (2002)], and the reduced pair wave approximation, based on a previous paper [Phys. Rev. A 68 033612 (2003)] generalizing to two channels the Cherny-Shanenko approach [Phys. Rev. E 62, 1046 (2000)]. The two approximations differ only by the way a pair of condensate atoms is influenced by the mean field at short interatomic separations. For these approximations we identify two different regimes of photoassociation: the adiabatic regime and the coherent regime. The threshold for the so-called "rogue dissociation" [Phys. Rev. Lett. 88, 090403 (2002)] (where mean-field theory breaks down) is found to be different in each regime, which sheds new light on the experiment of McKenzie et al [Phys. Rev. Lett. 88, 120403 (2002)] and previous theoretical calculations. We then use the two approximations to investigate numerically the effects of rogue dissociation in a sodium condensate under conditions similar to the McKenzie et al experiment. We find two different effects: reduction of the photoassociation rate at short times, and creation of correlated pairs of atoms, confirming previous works. We also observe that the photoassociation line shapes become asymmetric in the first-order cumulant approximation, while they remain symmetric in the reduced pair wave approximation, giving the possibility to experimentally distinguish between the two approximations.