AbstractParticipation in educational activities is an important prerequisite for academic success, yet often proves to be particularly challenging in digital settings. Therefore, this study set out to increase participation in an online proctored formative statistics exam by digital nudging. We exploited targeted nudges based on the Fogg Behaviour Model, highlighting the relevance of acknowledging differences in motivation and ability in allocating nudges to elicit target behaviour. First, we assessed whether pre‐existing levels of motivation and perceived ability to participate are effective in identifying different propensities of responsiveness to plain untailored nudges. Next, we evaluated whether tailoring nudges to students' motivation and perceived ability levels increases target behaviour by means of a randomized field experiment in which 579 first‐year university students received 6 consecutive emails over the course of three weeks to nudge behaviour regarding successful participation in the online exam. First, the results point out that motivation explains differences in engagement as indicated by student responsiveness and participation, whereas the perceived ability to participate does not. Second, the results from the randomized field experiment indicate that tailored nudging did not improve observed engagement. Implications for the potential of providing motivational information to improve participation in online educational activities are discussed, as are alternatives for capturing perceived ability more effectively.Practitioner notesWhat is already known about this topic Participation in educational activities is an important prerequisite for academic success, yet often proves to be particularly challenging in digital settings. Students' internal barriers to online participation and persistence in higher education are lack of motivation and perceived ability. Nudging interventions tackle students' behavioural barriers, and are particularly effective when guided by a theory of behaviour change, and when targeting students who suffer most from those barriers. What this paper adds This study examines whether the Fogg Behaviour Model is suited to guide a nudging intervention with the aim to increase student engagement in online higher education. This study examines whether students with different levels of motivation and perceived ability vary in their online behaviour in response to nudges. This study experimentally evaluates whether targeted nudges—targeted at students' motivation and perceived ability—are more effective than plain (not‐targeted) nudges. Implications for practice and/or policy The results indicate the importance of motivation for performing nudged behaviours regarding successful participation in an online educational activity. The results do not provide evidence for the role of perceived digital ability, yet do show prior performance on a similar educational activity can effectively distinguish between students' responsiveness. Targeted nudges were not more effective than plain nudges, but the potential of other motivational nudges and how to increase perceived performance are discussed.