Probabilistic security assessment and real-time dynamic security assessments (DSA) are promising to better handle the risks of system operations. The current methodologies of security assessments may require many time-domain simulations for some stability phenomena that are unpractical in real-time. Supervised machine learning is promising to predict DSA as their predictions are immediately available. Classifiers are offline trained on operating conditions and then used in real-time to identify operating conditions that are insecure. However, the predictions of classifiers can be sometimes wrong and hazardous if an alarm is missed for instance.A probabilistic output of the classifier is explored in more detail and proposed for probabilistic security assessment. An ensemble classifier is trained and calibrated offline by using Platt’ scaling to provide accurate probability estimates of the output. Imbalances in the training database and a cost-skewness addressing strategy are proposed for considering that missed alarms are significantly worse than false alarms. Subsequently, risk-minimised predictions can be made in real-time operation by applying cost-sensitive learning. Through case studies on a real data-set of the French transmission grid and on the IEEE 6 bus system using static security metrics, it is showcased how the proposed approach reduces inaccurate predictions and risks. The sensitivity on the likelihood of contingency is studied as well as on expected outage costs. Finally, the scalability to several contingencies and operating conditions are showcased.