In this paper, we propose a dual-polarization Mach-Zehnder modulator-based photonic nonlinear analog self-interference cancellation (SIC) technique for in-band full duplex (IBFD) systems. By using the proposed technique, an arbitrary 4th order nonlinear transfer function can be generated, meaning the performance limitation caused by the nonlinearity of the analog SIC circuit can be overcome by imitating the nonlinear transfer function of the analog SIC circuit before cancellation. This paper also presents a performance analysis through simulations and the results of a proof-of-concept demonstration. In the experiment, the proposed nonlinear SIC technique could achieve 29 dB cancellation over 500 MHz bandwidth centered at 1.25 GHz frequency along various degree of distortion caused by nonlinearity. In addition, the performance enhancements achieved by the proposed technique are evaluated in terms of error vector magnitudes (EVMs) and constellations of the signal-of-interest (SOI) in the simulation which is based on the experimental SIC results. More than 3 dB of SOI power gain could be obtained in evaluated EVM performances.