Identifying and characterising dynamical regime shifts, critical transitions or potential tipping points in palaeoclimate time series is relevant for improving the understanding of often highly nonlinear Earth system dynamics. Beyond linear changes in time series properties such as mean, variance, or trend, these nonlinear regime shifts can manifest as changes in signal predictability, regularity, complexity, or higher-order stochastic properties such as multi-stability. In recent years, several classes of methods have been put forward to study these critical transitions in time series data that are based on concepts from nonlinear dynamics, complex systems science, information theory, and stochastic analysis. These include approaches such as phase space-based recurrence plots and recurrence networks, visibility graphs, order pattern-based entropies, and stochastic modelling. Here, we review and compare in detail several prominent methods from these fields by applying them to the same set of marine palaeoclimate proxy records of African climate variations during the past 5 million years. Applying these methods, we observe notable nonlinear transitions in palaeoclimate dynamics in these marine proxy records and discuss them in the context of important climate events and regimes such as phases of intensified Walker circulation, marine isotope stage M2, the onset of northern hemisphere glaciation and the mid-Pleistocene transition. We find that the studied approaches complement each other by allowing us to point out distinct aspects of dynamical regime shifts in palaeoclimate time series. We also detect significant correlations of these nonlinear regime shift indicators with variations of Earth's orbit, suggesting the latter as potential triggers of nonlinear transitions in palaeoclimate. Overall, the presented study underlines the potentials of nonlinear time series analysis approaches to provide complementary information on dynamical regime shifts in palaeoclimate and their driving processes that cannot be revealed by linear statistics or eyeball inspection of the data alone.