Black Box Research Articles

Design and Analysis of ONLINE Voting System

A democratic country encourages voting system to decide the future leader. Traditional voting system offers lack of user friendliness in review of votes and voters’ identity security. In this present work, for the first time, a web-based voting architecture is designed and developed. The system has a login page, registration page, and dashboard page for voters to access in a user-friendly manner. The login page is the first page that users see when they access the system through registered mobile number, password, and role. The role is used to distinguish between voters and groups or candidates. The dashboard page is the display unit to exhibit voters’ profile information such as name, image, voting status, and mobile number on the left side of the page. On the right side of the page, there will be a list of groups or candidates whom they can cast their votes. In our proposed system, it is ensured that each voter can only vote once and that duplicate voting is not possible. The system uses password protected voting system and Black Box to administer the security issues for inculcating integrity of the voting process. The system is designed to provide a secure and user-friendly interface for voters to cast their votes online and henceforth the architecture will save the time, energy, and costs. T

Hormone replacement therapy perspectives

Hormone replacement therapy (HRT), also known as menopausal hormone therapy (MHT), was looked upon as a fountain of youth that kept women young and reduced cardiovascular disease. This led to a large-scale study called the Women's Health Initiative (WHI) that was conducted to show the cardiovascular benefits of HRT. This study was suspended early because of adverse side effects. The USFDA responded by slapping a “black box” warning on all HRT products. USFDA-approved bioidentical HRT formulations are safe and effective. We propose that these formulations have the “black box” warning removed so that doctors feel more confident in prescribing these products for symptoms of menopause and chronic conditions such as cardiovascular health. We propose eliminating the sale of products containing medroxyprogesterone acetate (MPA) because of the increased risk of heart attacks and breast cancers associated with this medication.

Improved forecasting of the compressive strength of ultra‐high‐performance concrete (UHPC) via the CatBoost model optimized with different algorithms

AbstractThis paper focuses on the applicability of CatBoost models constructed using various optimization techniques for improved forecasting the compressive strength of ultra‐high‐performance concrete (UHPC). Phasor particle swarm optimization (PPSO), dwarf mongoose optimization (DMO), and atom search optimization (ASO), which have been very popular recently, are preferred as optimization algorithms. A comprehensive and reliable data set is used to develop the CatBoost models, which include 785 test results with 15 input features. The performance of the CatBoost models (PPSO‐CatBoost, DMO‐CatBoost, and ASO‐CatBoost) optimized with different algorithms is thoroughly assessed by means of various statistical metrics and error analysis to determine the model with the best forecasting capability, and this model is compared with the models obtained from previous studies. In addition, Shapley additive exPlanations (SHAP) analysis is used to ensure the interpretability of the forecasting models and to overcome the “black box” problem of machine learning (ML) models. The obtained results demonstrate that all CatBoost models outstandingly forecast the compressive strength of UHPC. Among these models, the DMO‐CatBoost model stands out compared to the other models in various performance metrics, such as high coefficient of determination (R2) values, low root mean squared error (RMSE), mean absolute percentage error (MAPE), and mean absolute error (MAE) values, along with a smaller error ratio. In other words, the RMSE, R2, MAPE, and MAE values of the DMO‐CatBoost model for the training set are 3.67, 0.993, 0.019, and 2.35, respectively, whereas those for the test set are 6.15, 0.978, 0.038, and 4.51. Additionally, the performance ranking of the algorithms used to optimize the hyperparameters of the CatBoost model is as follows: DMO > PPSO > ASO. On the other hand, SHAP analysis showed that age, fiber dosage, and cement dosage significantly influence the compressive strength of UHPC. These findings can guide structural engineers in the design and optimization of UHPC, thus assisting them in developing strategies to improve the strength properties of the material. Finally, based on the best forecasting model developed in this work, a graphical user interface has been developed to easily forecast the compressive strength of UHPC in practical applications without additional tools or software.

Understanding the influence of AI autonomy on AI explainability levels in human-AI teams using a mixed methods approach

AbstractAn obstacle to effective teaming between humans and AI is the agent’s "black box" design. AI explanations have proven benefits, but few studies have explored the effects that explanations can have in a teaming environment with AI agents operating at heightened levels of autonomy. We conducted two complementary studies, an experiment and participatory design sessions, investigating the effect that varying levels of AI explainability and AI autonomy have on the participants’ perceived trust and competence of an AI teammate to address this research gap. The results of the experiment were counter-intuitive, where the participants actually perceived the lower explainability agent as both more trustworthy and more competent. The participatory design sessions further revealed how a team’s need to know influences when and what teammates need explained from AI teammates. Based on these findings, several design recommendations were developed for the HCI community to guide how AI teammates should share decision information with their human counterparts considering the careful balance between trust and competence in human-AI teams.

Implementasi Metode Prototype pada Pembuatan Web Portal TEFA House of Health Promotion

Health Promotion is a series of activities that strive for individuals to increase their knowledge related to health so that ultimately they can improve the quality of health. In order for the objectives of health promotion activities to be achieved, media is needed to support these activities. TeFa House of Health Promotion Polije is one of the TeFa (Teaching Factories) managed by the Health Promotion study program, Department of Health. TeFa was created as a forum for communication, information and health education, TeFa is also a means for lecturers to contribute to society by providing consultation services related to the development of public health promotion programs. To support this, a multifunctional platform is needed in the form of a webportal which is used to help TeFa. This research focuses on creating a web portal using one of the SDLC methods, namely prototype. The process of creating a web portal begins with extracting information through interviews with potential users. The results of this excavation process are information related to the features of the web portal, including articles, video, audio, pooling and chat. The process of creating a Web Portal is made using the PHP programming language and Laravel Framework. The testing process is carried out using the black box method to determine the functionality of each feature. The test results showed that the web portal was in accordance with the user's required features and each feature was functioning according to its function.

From Perception to Prediction: Leveraging Explainable AI in Self-Driving Cars for Enhanced Passenger Trust

Self-driving cars hold immense potential for revolutionizing transportation. However, public acceptance hinges on trust in the car's ability to navigate safely and make critical decisions. This trust deficit stems from the "black box" nature of traditional machine learning models used in self-driving cars. Passengers are left in the dark about the car's perception of the environment and the reasoning behind its actions. This research proposes leveraging Explainable Artificial Intelligence (XAI) techniques to enhance passenger trust in self-driving cars. By incorporating explainability into the perception and prediction modules of the car's decision- making system, we aim to provide passengers with real-time insights into how the car perceives its surroundings and translates those perceptions into driving decisions. This paper explores various XAI methods suitable for self-driving car applications. We discuss the integration of these techniques into the perception and prediction pipelines, enabling the car to explain its reasoning behind lane changes, obstacle avoidance maneuvers, and other critical actions. We evaluate the effectiveness of the proposed approach through user studies, assessing how explainability can improve passenger trust and comfort in self-driving vehicles. The ultimate goal of this research is to foster greater transparency and trust in self-driving car technology, paving the way for wider public adoption and a future of safe and reliable autonomous transportation.

Improved Wetland Mapping of a Highly Fragmented Agricultural Landscape Using Land Surface Phenological Features

Wetlands are integral components of agricultural landscapes, providing a wide range of ecological, economic, and social benefits essential for sustainable development and rural livelihoods. Globally, they are vulnerable ecological assets facing several significant threats including water extraction and regulation, land clearing and reclamation, and climate change. Classification and mapping of wetlands in agricultural landscapes is crucial for conserving these ecosystems to maintain their ecological integrity amidst ongoing land-use changes and environmental pressures. This study aims to establish a robust framework for wetland classification and mapping in intensive agricultural landscapes using time series of Sentinel-2 imagery, with a focus on the Gwydir Wetland Complex situated in the northern Murray–Darling Basin—Australia’s largest river system. Using the Google Earth Engine (GEE) platform, we extracted two groups of predictors based on six vegetation indices time series calculated from multi-temporal Sentinel-2 surface reflectance (SR) imagery: the first is statistical features summarizing the time series and the second is phenological features based on harmonic analysis of time series data (HANTS). We developed and evaluated random forest (RF) models for each level of classification with combination of different groups of predictors. Our results show that RF models involving both HANTS and statistical features perform strongly with significantly high overall accuracy and class-weighted F1 scores (p < 0.05) when comparing with models with either statistical or HANTS variables. While the models have excellent performance (F-score greater than 0.9) in distinguishing wetlands from other landcovers (croplands, terrestrial uplands, and open waters), the inter-class discriminating power among wetlands is class-specific: wetlands that are frequently inundated (including river red gum forests and wetlands dominated by common reed, water couch, and marsh club-rush) are generally better identified than the ones that are flooded less frequently, such as sedgelands and woodlands dominated by black box and coolabah. This study demonstrates that HANTS features extracted from time series Sentinel data can significantly improve the accuracy of wetland mapping in highly fragmentated agricultural landscapes. Thus, this framework enables wetland classification and mapping to be updated on a regular basis to better understand the dynamic nature of these complex ecosystems and improve long-term wetland monitoring.

Back to basics to open the black box

Back to basics to open the black box

Predicting cervical cancer risk probabilities using advanced H20 AutoML and local interpretable model-agnostic explanation techniques

Background Cancer is positioned as a major disease, particularly for middle-aged people, which remains a global concern that can develop in the form of abnormal growth of body cells at any place in the human body. Cervical cancer, often known as cervix cancer, is cancer present in the female cervix. In the area where the endocervix (upper two-thirds of the cervix) and ectocervix (lower third of the cervix) meet, the majority of cervical cancers begin. Despite an influx of people entering the healthcare industry, the demand for machine learning (ML) specialists has recently outpaced the supply. To close the gap, user-friendly applications, such as H2O, have made significant progress these days. However, traditional ML techniques handle each stage of the process separately; whereas H2O AutoML can automate a major portion of the ML workflow, such as automatic training and tuning of multiple models within a user-defined timeframe. Methods Thus, novel H2O AutoML with local interpretable model-agnostic explanations (LIME) techniques have been proposed in this research work that enhance the predictability of an ML model in a user-defined timeframe. We herein collected the cervical cancer dataset from the freely available Kaggle repository for our research work. The Stacked Ensembles approach, on the other hand, will automatically train H2O models to create a highly predictive ensemble model that will outperform the AutoML Leaderboard in most instances. The novelty of this research is aimed at training the best model using the AutoML technique that helps in reducing the human effort over traditional ML techniques in less amount of time. Additionally, LIME has been implemented over the H2O AutoML model, to uncover black boxes and to explain every individual prediction in our model. We have evaluated our model performance using the findprediction() function on three different idx values (i.e., 100, 120, and 150) to find the prediction probabilities of two classes for each feature. These experiments have been done in Lenovo core i7 NVidia GeForce 860M GPU laptop in Windows 10 operating system using Python 3.8.3 software on Jupyter 6.4.3 platform. Results The proposed model resulted in the prediction probabilities depending on the features as 87%, 95%, and 87% for class ‘0’ and 13%, 5%, and 13% for class ‘1’ when idx_value=100, 120, and 150 for the first case; 100% for class ‘0’ and 0% for class ‘1’, when idx_value= 10, 12, and 15 respectively. Additionally, a comparative analysis has been drawn where our proposed model outperforms previous results found in cervical cancer research.

PENGEMBANGAN SISTEM FRONT END PADA PMTCINVENTORY BERBASIS MOBILE MENGGUNAKAN METODE WATERFALL

The company inventory is one of the important aspects that receives primary attention in a company. Generally, companies with a vast and diverse inventory cater to clients and local employees. Therefore, company inventory becomes a primary concern for various businesses. However, in some companies, inventory tracking still relies on software such as Microsoft Word and Microsoft Excel. This makes the recording and management of company inventory less efficient. Based on this issue, it is concluded that companies need an electronic inventory system or e-inventory. Hence, the development of a mobile-based inventory system is undertaken. This development utilizes the Waterfall method and the Flutter framework. The stages in the Waterfall method include requirements, design, implementation, testing, and maintenance. The advantage of using this method is that the work is carried out gradually and systematically. The results of the research conducted using the Waterfall method include the creation of five pages or user interfaces: Dashboard, Login, Home, Saya, and Detail Alat. Each page has different functionalities. The user interface is created using the Flutter framework and tested using the black box testing method, resulting in alignment between user preferences and the application created.

The impact of environmental, social and governance performance on brand value: The role of the digitalisation level

Purpose: Studying the role of environmental, social and governance (ESG) performance in enhancing brand value is strategically important for promoting sustainable development.Design/methodology/approach: Using panel data for 126 different brands over 10 years, we empirically analyse the impact of ESG performance on brand value, considering the moderating role of the digitalisation level.Findings/results: Firstly, the impact of ESG performance on brand value exhibits a U-shaped relationship. Secondly, the digitalisation level negatively moderates the U-shaped relationship. Thirdly, the effect of ESG performance on brand value is mainly on the sub-items of environment and social responsibility, while the effect of governance is not significant.Practical implications: Firstly, firms should recognise the complex relationship between ESG performance and brand value, and strive to find a balance between economic benefits and sustainable development. Secondly, firms can utilise digital technology and platforms to communicate and interact with consumers, integrating ESG principles into their daily brand activities. Thirdly, investments in environmental and social responsibility are more meaningful for improving brand value.Originality/value: Firstly, our study adds value to corporate social responsibility (CSR) and ESG literature by investigating the non-linear impact of ESG performance on brand value. Secondly, we replenish the mechanistic study about how ESG performance affects brand value. Thirdly, we open the black box of the effect of single indicators (E/S/G) on brand value, including the specific direction and strength.

Development of interpretable machine learning models to predict in-hospital prognosis of acute heart failure patients.

In recent years, there has been remarkable development in machine learning (ML) models, showing a trend towards high prediction performance. ML models with high prediction performance often become structurally complex and are frequently perceived as black boxes, hindering intuitive interpretation of the prediction results. We aimed to develop ML models with high prediction performance, interpretability, and superior risk stratification to predict in-hospital mortality and worsening heart failure (WHF) in patients with acute heart failure (AHF). Based on the Kyoto Congestive Heart Failure registry, which enrolled 4056 patients with AHF, we developed prediction models for in-hospital mortality and WHF using information obtained on the first day of admission (demographics, physical examination, blood test results, etc.). After excluding 16 patients who died on the first or second day of admission, the original dataset (n=4040) was split 4:1 into training (n=3232) and test datasets (n=808). Based on the training dataset, we developed three types of prediction models: (i) the classification and regression trees (CART) model; (ii) the random forest (RF) model; and (iii) the extreme gradient boosting (XGBoost) model. The performance of each model was evaluated using the test dataset, based on metrics including sensitivity, specificity, area under the receiver operating characteristic curve (AUC), Brier score, and calibration slope. For the complex structure of the XGBoost model, we performed SHapley Additive exPlanations (SHAP) analysis, classifying patients into interpretable clusters. In the original dataset, the proportion of females was 44.8% (1809/4040), and the average age was 77.9±12.0. The in-hospital mortality rate was 6.3% (255/4040) and the WHF rate was 22.3% (900/4040) in the total study population. In the in-hospital mortality prediction, the AUC for the XGBoost model was 0.816 [95% confidence interval (CI): 0.815-0.818], surpassing the AUC values for the CART model (0.683, 95% CI: 0.680-0.685) and the RF model (0.755, 95% CI: 0.753-0.757). Similarly, in the WHF prediction, the AUC for the XGBoost model was 0.766 (95% CI: 0.765-0.768), outperforming the AUC values for the CART model (0.688, 95% CI: 0.686-0.689) and the RF model (0.713, 95% CI: 0.711-0.714). In the XGBoost model, interpretable clusters were formed, and the rates of in-hospital mortality and WHF were similar among each cluster in both the training and test datasets. The XGBoost models with SHAP analysis provide high prediction performance, interpretability, and reproducible risk stratification for in-hospital mortality and WHF for patients with AHF.

Study on urban green development efficiency of Jiangsu, Zhejiang and Fujian in China: a mixed network SBM approach

PurposeBy studying the green development efficiency (GDE) of 33 cities in the provinces of Jiangsu, Zhejiang, and Fujian in China, this study strives to conduct an analysis of the sustainable practices implemented in these developed regions, and derive valuable insights that can foster the promotion of green transformation.Design/methodology/approachFirst, the urban green development system (GDS) was decomposed into the economic benefit subsystem (EBS), social benefit subsystem (SBS), and pollution control subsystem (PCS). Then, a mixed network SBM model was proposed to evaluate the GDE during 20152020, with Moran’s I and Bootstrap truncated regression model subsequently applied to measure the spatial characteristics and driving factors of efficiency.FindingsSubsystem efficiency presents a distribution trend of PCS > EBS > SBS. There is a particular spatial aggregation effect in EBS efficiency, whereas SBS and PCS efficiencies have no significant spatial autocorrelation. Furthermore, urbanization level contributes significantly to the efficiency of all subsystems; industrial structure, energy consumption, and technological innovation play a crucial role in EBS and SBS; external openness is a pivotal factor in SBS; and environmental regulation has a significant effect on PCS.Originality/valueThis study further decomposes the black box of GDS into subsystems including the economy, society, and environment. Additionally, by employing a mixed network SBM model and Bootstrap truncated regression model to investigate efficiency and its driving factors from the subsystem perspective, it endeavors to derive more detailed research conclusions and policy implications.

Development of optimized cascaded LSTM with Seq2seqNet and transformer net for aspect-based sentiment analysis framework

The recent development of communication technologies made it possible for people to share opinions on various social media platforms. The opinion of the people is converted into small-sized textual data. Aspect Based Sentiment Analysis (ABSA) is a process used by businesses and other organizations to assess these textual data in order to comprehend people’s opinions about the services or products offered by them. The majority of earlier Sentiment Analysis (SA) research uses lexicons, word frequencies, or black box techniques to obtain the sentiment in the text. It should be highlighted that these methods disregard the relationships and interdependence between words in terms of semantics. Hence, an efficient ABSA framework to determine the sentiment from the textual reviews of the customers is developed in this work. Initially, the raw text review data is collected from the standard benchmark datasets. The gathered text reviews undergo text pre-processing to neglect the unwanted words and characters from the input text document. The pre-processed data is directly provided to the feature extraction phase in which the seq2seq network and transformer network are employed. Further, the optimal features from the two resultant features are chosen by utilizing the proposed Modified Bird Swarm-Ladybug Beetle Optimization (MBS-LBO). After obtaining optimal features, these features are fused together and given to the final detection model. Consequently, the Optimized Cascaded Long Short Term Memory (OCas-LSTM) is proposed for predicting the sentiments from the given review by the users. Here, the parameters are tuned optimally by the MBS-LBO algorithm, and also it is utilized for enhancing the performance rate. The experimental evaluation is made to reveal the excellent performance of the developed SA model by contrasting it with conventional models.

Inferring the Molecular Basis for Dissolved Organic Matter Photochemical and Optical Properties.

Despite the widespread use of photochemical and optical properties to characterize dissolved organic matter (DOM), a significant gap persists in our understanding of the relationship among these properties. This study infers the molecular basis for the optical and photochemical properties of DOM using a comprehensive framework and known structural moieties within DOM. Utilizing Suwannee River Fulvic Acid (SRFA) as a model DOM, carboxylated aromatics, phenols, and quinones were identified as dominant contributors to the absorbance spectra, and phenols, quinones, aldehydes, and ketones were identified as major contributors to radiative energy pathways. It was estimated that chromophores constitute ∼63% w/w of dissolved organic carbon in SRFA and ∼47% w/w of overall SRFA. Notably, estimations indicate the pool of fluorescent compounds and photosensitizing compounds in SRFA are likely distinct from each other at wavelengths below 400 nm. This perspective offers a practical tool to aid in the identification of probable chemical groups when interpreting optical and photochemical data and challenges the current "black box" thinking. Instead, DOM photochemical and optical properties can be closely estimated by assuming the DOM is composed of a mixture of individual compounds.

Non-Differentiable Loss Function Optimization and Interaction Effect Discovery in Insurance Pricing Using the Genetic Algorithm

Non-Differentiable Loss Function Optimization and Interaction Effect Discovery in Insurance Pricing Using the Genetic Algorithm

Industry-specific stock valuation methods – a literature review

PurposeSurprisingly little is known of the various methods of security analysis used by financial analysts with industry-specific knowledge. Financial analysts’ industry knowledge is a favored and appreciated attribute by fund managers and institutional investors. Understanding analysts’ use of industry-specific valuation models, which are the main value drivers within different industries, will enhance our understanding of important aspects of value creation in these industries. This paper contributes to the broader understanding of how financial analysts in various industries approach valuation, offering insights that can be beneficial to a wide range of stakeholders in the financial market.Design/methodology/approachThis paper systematically reviews existing research to consolidate the current understanding of analysts’ use of valuation models and factors. It aims to demystify what can often be seen as a “black box”, shedding light on the valuation tools employed by financial analysts across diverse industries.FindingsThe use of industry-specific valuation models and factors by analysts is a subject of considerable interest to both academics and investors. The predominant model in several industries is P/E, with some exceptions. Notably, EV/EBITDA is favored in the telecom, energy and materials sectors, while the capital goods industry primarily relies on P/CF. In the REITs sector, P/AFFO is the most commonly employed model. In specific sectors like pharmaceuticals, energy and telecom, DCF is utilized. However, theoretical models like RIM and AEG find limited use among analysts.Originality/valueThis is the first paper systematically reviewing the research on analyst’s use of industry-specific stock valuation methods. It serves as a foundation for future research in this field and is likely to be of interest to academics, analysts, fund managers and investors.

EEGminer: discovering interpretable features of brain activity with learnable filters

Objective. The patterns of brain activity associated with different brain processes can be used to identify different brain states and make behavioural predictions. However, the relevant features are not readily apparent and accessible. Our aim is to design a system for learning informative latent representations from multichannel recordings of ongoing EEG activity. Approach: We propose a novel differentiable decoding pipeline consisting of learnable filters and a pre-determined feature extraction module. Specifically, we introduce filters parameterized by generalized Gaussian functions that offer a smooth derivative for stable end-to-end model training and allow for learning interpretable features. For the feature module, we use signal magnitude and functional connectivity estimates. Main results. We demonstrate the utility of our model on a new EEG dataset of unprecedented size (i.e. 721 subjects), where we identify consistent trends of music perception and related individual differences. Furthermore, we train and apply our model in two additional datasets, specifically for emotion recognition on SEED and workload classification on simultaneous task EEG workload. The discovered features align well with previous neuroscience studies and offer new insights, such as marked differences in the functional connectivity profile between left and right temporal areas during music listening. This agrees with the specialisation of the temporal lobes regarding music perception proposed in the literature. Significance. The proposed method offers strong interpretability of learned features while reaching similar levels of accuracy achieved by black box deep learning models. This improved trustworthiness may promote the use of deep learning models in real world applications. The model code is available at https://github.com/SMLudwig/EEGminer/.

Automatic evolutionary design of quantum rule-based systems and applications to quantum reinforcement learning

Explainable artificial intelligence is a research topic whose relevance has increased in recent years, especially with the advent of large machine learning models. However, very few attempts have been proposed to improve interpretability in the case of quantum artificial intelligence, and many existing quantum machine learning models in the literature can be considered almost as black boxes. In this article, we argue that an appropriate semantic interpretation of a given quantum circuit that solves a problem can be of interest to the user not only to certify the correct behavior of the learned model, but also to obtain a deeper insight into the problem at hand and its solution. We focus on decision-making problems that can be formulated as classification tasks and propose a method for learning quantum rule-based systems to solve them using evolutionary optimization algorithms. The approach is tested to learn rules that solve control and decision-making tasks in reinforcement learning environments, to provide interpretable agent policies that help to understand the internal dynamics of an unknown environment. Our results conclude that the learned policies are not only highly explainable, but can also help detect non-relevant features of problems and produce a minimal set of rules.

Improve robustness of machine learning via efficient optimization and conformal prediction

AbstractThe advance of machine learning (ML) systems in real‐world scenarios usually expects safe deployment in high‐stake applications (e.g., medical diagnosis) for critical decision‐making process. To this end, provable robustness of ML is usually required to measure and understand how reliable the deployed ML system is and how trustworthy their predictions can be. Many studies have been done to enhance the robustness in recent years from different angles, such as variance‐regularized robust objective functions and conformal prediction (CP) for uncertainty quantification on testing data. Although these tools provably improve the robustness of ML model, there is still an inevitable gap to integrate them into an end‐to‐end deployment. For example, robust objectives usually require carefully designed optimization algorithms, while CP treats ML models as black boxes. This paper is a brief introduction to our recent research focusing on filling this gap. Specifically, for learning robust objectives, we designed sample‐efficient stochastic optimization algorithms that achieves the optimal (or faster compared to existing algorithms) convergence rates. Moreover, for CP‐based uncertainty quantification, we established a framework to analyze the expected prediction set size (smaller size means more efficiency) of CP methods in both standard and adversarial settings. This paper elaborates the key challenges and our exploration towards efficient algorithms with details of background methods, notions for robustness measure, concepts of algorithmic efficiency, our proposed algorithms and results. All of them further motivate our future research on risk‐aware ML that can be critical for AI–human collaborative systems. The future work mainly targets designing conformal robust objectives and their efficient optimization algorithms.