We theoretically investigate an exciton-coupled electron transfer (XCET) process that is conversion of an exciton into a charge transfer state. This conversion happens in an exciton transfer (XT) process, and the electron moves away in an electron transfer (ET) process in multiple environments (baths). This XCET process plays an essential role in the harvesting of solar energy in biological and photovoltaic materials. We develop a practical theoretical model to study the efficiency of the XCET process that occurs either in consecutive or concerted processes under the influence of non-Markovian baths. The role of quantum coherence in the XT-ET system and the baths is investigated using reduced hierarchal equations of motion (HEOM). This model includes independent baths for each XT and ET state, in addition to a XCET bath for the conversion process. We found that, while quantum system-bath coherence is important in the XT and ET processes, coherence between the XT and ET processes must be suppressed in order to realize that an efficient irreversible XCET process through the weak off-diagonal interaction between the XT and ET bridge sites arises from an XCET bath.