Abstract Internal combustion Rankine cycle (ICRC) concept implements oxy-fuel combustion, direct water injection (DWI), and waster heat recovery (WHR) into traditional Otto or diesel cycle to realize high thermal efficiency and low emission powertrain. In order to support ICRC realization, this paper is dedicated to investigate the feasibility of implementing oxy-fuel combustion into diffusion combustion which provides fundamental information for future compression ignition (CI)-ICRC engine. The prototype oxy-fuel diffusion combustion engine test bench is established based on a retrofitted diesel engine, and the O2/CO2 mixture intake system, high-pressure common rail fuel injection system, and high-performance electronic controller are designed and installed within engine test bench to investigate the combustion and emission characteristics under different intake oxygen fractions (OF), fuel injection durations, and fuel injection timing. The optimum intake OF and fuel injection strategies are acquired within the selected experimental conditions, a 41.1% brake thermal efficiency (BTE), and 1.2% coefficient of variation (CoV) is achieved utilizing 55% intake OF, 0.7 ms fuel injection duration and 352 °CA (after exchange top dead center (TDC)) fuel injection timing. The oxy-fuel diffusion combustion proved to be a feasible solution for simultaneously reduction in NOX and particulate emissions, and NOX emissions lower than 90 × 10−6 with particulate matters (PM) around 0.1 filter smoke number (FSN) is observed during engine bench testing. The result of this study provides fundamental information for future CI-ICRC prototype engine establishment and optimization, which also could be utilized as reference guidance for potential industrialization of internal combustion engine (ICE) with oxy-fuel combustion mode.