Aims. Theoretical calculations predict an increased equatorial rotation and more pronounced differential rotation (DR) during the minimum of solar magnetic activity. However, the results of observational studies vary, some showing less and some more pronounced DR during the minimum of solar magnetic activity. Our study aims to gain more insight into these discrepancies. Methods. We determined the DR parameters A and B (corresponding to the equatorial rotation velocity and the gradient of the solar DR, respectively) by tracing sunspot groups in sunspot drawings of the Kanzelhöhe Observatory for Solar and Environmental Research (KSO; 1964–2008, for solar cycles 20–23) and KSO white-light images (2009–2016, for solar cycle 24). We used different statistical methods and approaches to analyse variations in DR parameters related to the cycle and to the phase of the solar cycle, together with long-term related variations. Results. The comparison of the DR parameters for individual cycles obtained from the KSO and from other sources yield statistically insignificant differences for the years after 1980, meaning that the KSO sunspot group data set is well suited for long-term cycle to cycle studies. The DR parameters A and B show statistically significant periodic variability. The periodicity corresponds to the solar cycle and is correlated with the solar activity. The changes in A related to solar cycle phase are in accordance with previously reported theoretical and experimental results (higher A during solar minimum, lower A during the maximum of activity), while changes in B differ from the theoretical predictions as we observe more negative values of B, that is, a more pronounced differential rotation during activity maximum. The main result of this paper for the long-term variations in A is the detection of a phase shift between the activity flip (in the 1970s) and the equatorial rotation velocity flip (in the early 1990s), during which both A and activity show a secular decreasing trend. This indicates that the two quantities are correlated in between 1970 and 1990. Therefore, the theoretical model fails in the phase-shift time period that occurs after the modern Gleissberg maximum, while in the time period thereafter (after the 1990s), theoretical and experimental results are consistent. The long-term variations in B in general yield an anticorrelation of B and activity, as a rise of B is observed during the entire time period (1964–2016) we analysed, during which activity decreased, with the exception of the end of solar cycle 22 and the beginning of solar cycle 23. Conclusions. We study for the first time the variation in solar DR and activity based on 53 years of KSO data. Our results agree well with the results related to the solar cycle phase from corona observations. The disagreement of the observational results for B and theoretical studies may be due to the fact that we analysed the period immediately after the modern Gleissberg maximum, where for the phase-shift period, A versus activity also entails a result that differs from theoretical predictions. Therefore, studies of rotation versus activity with data sets encompassing the Gleissberg extremes should include separate analyses of the parts of the data set in between different flips (e.g., before the activity flip, between the activity and the rotation flip, and after the rotation flip).