With the ever-increasing penetration of electric vehicles (EVs), extreme fast charging stations (XFCSs) are being widely deployed, wherein battery energy storages (BESs) are also installed for reducing the peak charging power. However, integrating the XFCS with a high-capacity power converter into the power distribution network (PDN) is difficult and uneconomical due to the restrictions regarding urban planning and high investment in PDN expansion. Considering the fluctuation in the EV charging demand and the limited capacity of the power converter, a collaborative policy for real-time EV charging power allocation and BES discharging power control is proposed based on Markov Decision Process (MDP), which is solved by the constraint deep deterministic policy gradient (CDDPG). The proposed model makes it possible to integrate the XFCS with reduced capacity power converter into the PDN with a minimal negative impact on the quality of service (QoS) of EV owners. Finally, the experimental evaluation with real-word data sets demonstrates that the proposed approach is more effective than benchmark methods in dynamically allocating charging power for XFCS.