Application of renewable fuels in compression ignition engines provides one solution to reduce greenhouse gas emission in heavy-duty transportation sector. However, it is a big challenge to find a single alternative fuel to replace the traditional diesel under full load conditions. In this study, the combustion characteristics of two kinds of renewable fuels with different reactivity, namely methanol and hydrogenated catalytic biodiesel (HCB), were investigated in an optical engine under partially premixed combustion (PPC) and reactivity-controlled compression ignition (RCCI) modes, respectively. A two-color (2C) method was employed to quantify the in-flame soot formation. Meanwhile, the flame oscillation phenomenon was captured and evaluated. The results indicate that the RCCI mode exhibits reduced peak cylinder pressure, resulting in smoother and more stable combustion compared to that of the PPC mode. Moreover, the more uniform fuel distribution in the RCCI mode further decreases soot formation from the blended fuel. Additionally, with increasing methanol proportion, the rise in oxygen content within the blended fuel significantly reduces soot formation. The flame oscillations are primarily related to the rate of cylinder pressure variation. Under PPC mode, multi-point autoignition causes a rapid increase in cylinder pressure, leading to larger oscillation amplitudes (M15: 0.268, M25: 0.772). Conversely, under the RCCI mode, the heat release process is more stable, resulting in weaken oscillations (M15: 0.161, M25: 0.064) compared to that of PPC.