Simultaneous transmit and receive wireless communications have been highlighted for their potential to double the spectral efficiency. However, it is necessary to mitigate self-interference (SI). Considering both the SI channel and remote transmission (RT) channel need to be estimated before equalizing the received signal, we propose two adaptive algorithms for linear and nonlinear self-interference cancellation (SIC), based on a multi-layered joint channel estimator structure. The proposed algorithms estimate the RT channel while performing SIC, and the multi-layered structure ensures improved performance across various interference-to-signal ratios. The M-estimate function enhances the robustness of the algorithm, allowing it to converge even when affected by impulsive noise. For nonlinear SIC, this paper introduces an adaptive algorithm based on generalized Hammerstein polynomial basis functions. The simulation results indicate that this approach achieves a better convergence speed and normalized mean squared difference compared to existing SIC methods, leading to a lower system bit error rate.