Modeling Tasks Research Articles

Interpreting Deep Forest through Feature Contribution and MDI Feature Importance

Deep forest is a non-differentiable deep model that has achieved impressive empirical success across a wide variety of applications, especially on categorical/symbolic or mixed modeling tasks. Many of the application fields prefer explainable models, such as random forests with feature contributions that can provide a local explanation for each prediction, and Mean Decrease Impurity (MDI) that can provide global feature importance. However, deep forest, as a cascade of random forests, possesses interpretability only at the first layer. From the second layer on, many of the tree splits occur on the new features generated by the previous layer, which makes existing explaining tools for random forests inapplicable. To disclose the impact of the original features in the deep layers, we design a calculation method with an estimation step followed by a calibration step for each layer, and propose our feature contribution and MDI feature importance calculation tools for deep forest. Experimental results on both simulated data and real-world data verify the effectiveness of our methods.

Soft-sensor estimation via parameter fitting and dynamic optimization in an experimental batch butadiene homopolymerization reactor

In this work, we address modeling and experimental tasks to develop an industrial process for the anionic polymerization of butadiene, which is a key compound used in the production of high-performance tires. The aim of this study was to determine the processing conditions that lead to the efficient manufacturing of polybutadiene. Accordingly, an experimental polymer reaction facility was constructed to record both butadiene conversion and temperature profiles. A first-principles dynamic mathematical model was developed to track the measurement variables as best as possible. To improve the tracking of recorded variables, some process variables were deployed as manipulated variables. Because good experimental tracking was not acceptable, we used the soft-sensor approach to estimate some additional degrees of freedom. The estimation task was performed by describing the problem as a dynamic optimization problem. The results clearly show that the optimization and control tasks of butadiene polymerization reaction facilities are demanding and require good experimental information to develop reliable mathematical models that can be leveraged to speed up the development of such industrial polymerization systems.

Geometric constructive network with block increments for lightweight data-driven industrial process modeling

Industrial data-driven models may require frequent reconstruction to maintain model performance due to the dynamics, uncertainty, and complexity of industrial processes. The infrastructure of the industrial processes is usually distributed control systems (DCS) with energy-sensitive and resource-constrained. In this context, this article proposes a geometric constructive network with block increments (BI-GCN) to reduce the modeling consumption while achieving comparable accuracy. First, this article proposes a geometric control strategy with block increments, which is capable of adding multiple nodes to the BI-GCN simultaneously. Second, this article demonstrates the universal approximation property of BI-GCN, which in turn guarantees the potential high performance of BI-GCN for modeling tasks. Finally, experiments on benchmark datasets and the grinding process show that BI-GCN can effectively reduce the number of iterations in the modeling process while maintaining comparable accuracy.

Forming the competence of physics and mathematics students using information technology

Relevance. The relevance of the study arises as computers in educational institutions have ceased to be a mere subject of study and have become a tool for learning. The main purpose of using computer-based learning technologies is to increase the efficiency of the learning process. This task is also solved by increasing the motivation of students to study technical graphics. Graduates should be able to use graphics systems to create both blueprint and design documentation and to solve three-dimensional graphic modelling tasks. Purpose. The purpose of the study was to identify the features of the formation of professional competence of non-humanitarian specialities through information technology. Methodology. The methodological basis of the study was general theoretical methods, through which it was possible to determine the fact that the use of information and communication technologies in the classroom allows raising the level of information culture, realising the potential of each student, and forming the professional skills of future specialists. Results. This article covered the main methods of using information technology in mathematics and physics classes, the features of the competency-based approach, the advantages of using innovative computer technology, and the types of technologies used. It has been found that the use of information technology in mathematics and physics classes affects: the activation of students' cognitive activity, development of mathematical logic, variability of reasoning, and concentration of students' intellectual activity on research and investigation. Conclusions. The practical significance of the study lies in the fact that its findings and conclusions improve the training in the development of competence of students of physics and mathematics specialities using information technology in educational activities.

Efficient Masked Autoencoders With Self-Consistency.

Inspired by the masked language modeling (MLM) in natural language processing tasks, the masked image modeling (MIM) has been recognized as a strong self-supervised pre-training method in computer vision. However, the high random mask ratio of MIM results in two serious problems: 1) the inadequate data utilization of images within each iteration brings prolonged pre-training, and 2) the high inconsistency of predictions results in unreliable generations, i.e., the prediction of the identical patch may be inconsistent in different mask rounds, leading to divergent semantics in the ultimately generated outcomes. To tackle these problems, we propose the efficient masked autoencoders with self-consistency (EMAE) to improve the pre-training efficiency and increase the consistency of MIM. In particular, we present a parallel mask strategy that divides the image into K non-overlapping parts, each of which is generated by a random mask with the same mask ratio. Then the MIM task is conducted parallelly on all parts in an iteration and the model minimizes the loss between the predictions and the masked patches. Besides, we design the self-consistency learning to further maintain the consistency of predictions of overlapping masked patches among parts. Overall, our method is able to exploit the data more efficiently and obtains reliable representations. Experiments on ImageNet show that EMAE achieves the best performance on ViT-Large with only 13% of MAE pre-training time using NVIDIA A100 GPUs. After pre-training on diverse datasets, EMAE consistently obtains state-of-the-art transfer ability on a variety of downstream tasks, such as image classification, object detection, and semantic segmentation.

Challenges and opportunities of automated data pipelines for environmental sustainability applications in industrial manufacturing

Energy efficiency analyses and life cycle assessments are two examples of data-driven applications focused on environmental sustainability in the industrial manufacturing context. Both use methodologies based on aggregating operational data, extracted from multiple data sources along a value chain. However, the possibility to source, utilize, and share data is often obstructed by heterogeneous or non-transparent data operations across organizations, as well as poor availability of digital interfaces for data collection. In practice, the underlying application processes lack fidelity and granularity, as a trade-of to simplify the modeling and data collection tasks. Therefore, new opportunities arise for automated workflows on data collection (data pipelines) as a relevant component of every digital transformation strategy. This study reflects on the challenges experienced by two industrial actors collecting operational data for an energy efficiency analysis and a Life-cycle Assessment (LCA), each with a different manufacturing context and architectural approach. Finally, it presents the opportunities for new technologies at affordable costs that potentially ease the development and operation of data pipelines to solve such challenges.

Generalizing Upper Limb Force Modeling With Transfer Learning: A Multimodal Approach Using EMG and IMU for New Users and Conditions.

In the field of EMG-based force modeling, the ability to generalize models across individuals could play a significant role in its adoption across a range of applications, including assistive devices, robotic and rehabilitation devices. However, current studies have predominately focused on intra-subject modeling, largely neglecting the burden of end-user data acquisition. In this work, we propose the use of transfer learning (TL) to generalize force modeling to a new user by first establishing a baseline model trained using other users' data, and then adapting to the end-user using a small amount of new data (only 10% , 20% , and 40% of the new user data). Using a deep multimodal convolutional neural network, consisting of two CNN models, one with high-density (HD) EMG and one with motion data recorded by an Inertial Measurement Unit (IMU), our proposed TL technique significantly improved force modeling compared to leave-one-subject-out (LOSO) and even intra-subject scenarios. The TL approach increased the average R squared values of the force modeling task by 60.81%, 190.53%, and 199.79% compared to the LOSO case, and by 13.4%, 36.88%, and 45.51% compared to the intra-subject case for isotonic, isokinetic and dynamic conditions, respectively. These results show that it is possible to adapt to a new user with minimal data while improving performance significantly compared to the intra-subject scenario. We also show that TL can be used to generalize on a new experimental condition for a new user.

SSR-2D: Semantic 3D Scene Reconstruction from 2D Images.

Most deep learning approaches to comprehensive semantic modeling of 3D indoor spaces require costly dense annotations in the 3D domain. In this work, we explore a central 3D scene modeling task, namely, semantic scene reconstruction without using any 3D annotations. The key idea of our approach is to design a trainable model that employs both incomplete 3D reconstructions and their corresponding source RGB-D images, fusing cross-domain features into volumetric embeddings to predict complete 3D geometry, color, and semantics with only 2D labeling which can be either manual or machine-generated. Our key technical innovation is to leverage differentiable rendering of color and semantics to bridge 2D observations and unknown 3D space, using the observed RGB images and 2D semantics as supervision, respectively. We additionally develop a learning pipeline and corresponding method to enable learning from imperfect predicted 2D labels, which could be additionally acquired by synthesizing in an augmented set of virtual training views complementing the original real captures, enabling more efficient self-supervision loop for semantics. As a result, our end-to-end trainable solution jointly addresses geometry completion, colorization, and semantic mapping from limited RGB-D images, without relying on any 3D ground-truth information. Our method achieves state-of-the-art performance of semantic scene completion on two large-scale benchmark datasets MatterPort3D and ScanNet, surpasses baselines even with costly 3D annotations in predicting both geometry and semantics. To our knowledge, our method is also the first 2D-driven method addressing completion and semantic segmentation of real-world 3D scans simultaneously.

Multimodal High-Order Relationship Inference Network for Fashion Compatibility Modeling in Internet of Multimedia Things

Recent progress in artificial intelligence (AI) have broadened various intelligent application scenarios on the Internet of Multimedia Things (IoMT). Due to the urgent demands for intelligence in the online fashion industry, using AI techniques to explore user’s clothing collocations from fashion data generated by the IoMT system is a challenging task. Under the background, fashion compatibility modeling (FCM), which aims to estimate the matching degree of a given outfit, has attracted great attention in the multimedia analysis field. However, most of the studies often fail to fully leverage multimodal content or ignore the sparse associations between fashion items. In this paper, we propose a novel multimodal high-order relationship inference network (MHRIN) for fashion compatibility modeling task. In MHRIN, we focus on enriching multimodal representations of fashion items by means of incorporating the category correlations and injecting high-order item-item connectivity. Concretely, considering that fashion collocations depend on the semantic relevance patterns between categories, we design a category correlations learning module to adaptively learn category representations. On this basis, multiple modality representations are aggregated by a hierarchical multimodal fusion module to generate visual-semantic embeddings. To address the item-item matching interactions issue, we further refine the final representations by a high-order message propagation module to absorb rich connection information. Experiments on the publicly available dataset demonstrate the superiority of our MHRIN over state-of-the-art methods.

Variations of Training Process in Vanilla Recurrent Neural Network Framework

Recurrent neural networks (RNNs) are capable of learning features and long term dependencies from sequential and time series data and show outstanding performance in sequential modeling tasks. However, training process in RNNs is troubled by issues in learning processes such as slow inference, vanishing gradients and difficulties in capturing long term dependencies. In this paper, we introduce a new learning technique to update the weight set as we change the input sequence which is shifted by certain amount of time in training process, instead of using a traditional way to calculate one set of the weights and bias in training time series with sequences shifted by certain amount of time series. We also consider an algorithm for an evaluation process. In the traditional way, the evaluation process is executed by using final weights and biases calculated in the training process. Instead, during the testing process, the weights and biases are iteratively updated in each sequence as done in the training process. Several numerical experiments demonstrate the efficiency of the proposed techniques.

Enhanced Heart Disease Prediction Using Machine Learning Techniques

This study leverages sophisticated machine learning methodologies, particularly XGBoost, to analyze cardiovascular diseases through cardiac datasets. The methodology encompasses meticulous data pre-processing, training of the XGBoost algorithm, and its performance evaluation using metrics such as accuracy, precision, and ROC curves. This technique represents a notable progression in the realm of medical research, potentially leading to enhanced diagnostic precision and a deeper comprehension of cardiovascular ailments, thereby improving patient care and treatment modalities in cardiology. Furthermore, the research delves into the utilization of deep learning methodologies for the automated delineation of cardiac structures in MRI and mammography images, aiming to boost diagnostic precision and patient management. [24][3][5][6] In assessing machine learning algorithms' efficacy in diagnosing cardiovascular diseases, this analysis underscores the pivotal role of such algorithms and their possible data inputs. Additionally, it investigates promising directions for future exploration, such as the application of reinforcement learning. A significant aspect of our investigation is the development and deployment of sophisticated deep learning models for segmenting right ventricular images from cardiac MRI scans, aiming at heightened accuracy and dependability in diagnostics. Through the utilization of advanced techniques like Fourier Convolutional Neural Network (FCNN) and improved versions of Vanilla Convolutional Neural Networks (Vanilla-CNN) and Residual Networks (ResNet), we achieved a substantial improvement in accuracy and reliability. This enhancement allows for more precise and quicker identification and diagnosis of cardiovascular diseases, which is of utmost importance in clinical practice. Evaluation of Machine Learning Algorithms: We conducted a comprehensive evaluation of machine learning algorithms in the context of cardiovascular disease diagnosis. This assessment emphasized the fundamental role of machine learning algorithms and their potential data sources. We also explored promising avenues, such as reinforcement learning, for future research. Factors Affecting Predictive Models: We highlighted the critical factors affecting the effectiveness of machine learning-based predictive models. These factors include data heterogeneity, depth, and breadth, as well as the nature of the modeling task, and the choice of algorithms and feature selection methods. Recognizing and addressing these factors are essential for building reliable models.

Uni4Eye++: A General Masked Image Modeling Multi-modal Pre-training Framework for Ophthalmic Image Classification and Segmentation.

A large-scale labeled dataset is a key factor for the success of supervised deep learning in most ophthalmic image analysis scenarios. However, limited annotated data is very common in ophthalmic image analysis, since manual annotation is time-consuming and labor-intensive. Self-supervised learning (SSL) methods bring huge opportunities for better utilizing unlabeled data, as they do not require massive annotations. To utilize as many unlabeled ophthalmic images as possible, it is necessary to break the dimension barrier, simultaneously making use of both 2D and 3D images as well as alleviating the issue of catastrophic forgetting. In this paper, we propose a universal self-supervised Transformer framework named Uni4Eye++ to discover the intrinsic image characteristic and capture domain-specific feature embedding in ophthalmic images. Uni4Eye++ can serve as a global feature extractor, which builds its basis on a Masked Image Modeling task with a Vision Transformer architecture. On the basis of our previous work Uni4Eye, we further employ an image entropy guided masking strategy to reconstruct more-informative patches and a dynamic head generator module to alleviate modality confusion. We evaluate the performance of our pre-trained Uni4Eye++ encoder by fine-tuning it on multiple downstream ophthalmic image classification and segmentation tasks. The superiority of Uni4Eye++ is successfully established through comparisons to other state-of-the-art SSL pre-training methods. Our code is available at Github1.

Optimizing postprandial glucose prediction through integration of diet and exercise: Leveraging transfer learning with imbalanced patient data.

In recent years, numerous methods have been introduced to predict glucose levels using machine-learning techniques on patients' daily behavioral and continuous glucose data. Nevertheless, a definitive consensus remains elusive regarding modeling the combined effects of diet and exercise for optimal glucose prediction. A notable challenge is the propensity for observational patient datasets from uncontrolled environments to overfit due to skewed feature distributions of target behaviors; for instance, diabetic patients seldom engage in high-intensity exercise post-meal. In this study, we introduce a unique application of Bayesian transfer learning for postprandial glucose prediction using randomized controlled trial (RCT) data. The data comprises a time series of three key variables: continuous glucose levels, exercise expenditure, and carbohydrate intake. For building the optimal model to predict postprandial glucose levels we initially gathered balanced training data from RCTs on healthy participants by randomizing behavioral conditions. Subsequently, we pretrained the model's parameter distribution using RCT data from the healthy cohort. This pretrained distribution was then adjusted, transferred, and utilized to determine the model parameters for each patient. The efficacy of the proposed method was appraised using data from 68 gestational diabetes mellitus (GDM) patients in uncontrolled settings. The evaluation underscored the enhanced performance attained through our method. Furthermore, when modeling the joint impact of diet and exercise, the synergetic model proved more precise than its additive counterpart. An innovative application of the transfer-learning utilizing randomized controlled trial data can improve the challenging modeling task of postprandial glucose prediction for GDM patients, integrating both dietary and exercise behaviors. For more accurate prediction, future research should focus on incorporating the long-term effects of exercise and other glycemic-related factors such as stress, sleep.

Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment.

While pre-training large-scale video-language models (VLMs) has shown remarkable potential for various downstream video-language tasks, existing VLMs can still suffer from certain commonly seen limitations, e.g., coarse-grained cross-modal aligning, under-modeling of temporal dynamics, detached video-language view. In this work, we target enhancing VLMs with a fine-grained structural spatio-temporal alignment learning method (namely Finsta). First of all, we represent the input texts and videos with fine-grained scene graph (SG) structures, both of which are further unified into a holistic SG (HSG) for bridging two modalities. Then, an SG-based framework is built, where the textual SG (TSG) is encoded with a graph Transformer, while the video dynamic SG (DSG) and the HSG are modeled with a novel recurrent graph Transformer for spatial and temporal feature propagation. A spatial-temporal Gaussian differential graph Transformer is further devised to strengthen the sense of the changes in objects across spatial and temporal dimensions. Next, based on the fine-grained structural features of TSG and DSG, we perform object-centered spatial alignment and predicate-centered temporal alignment respectively, enhancing the video-language grounding in both the spatiality and temporality. We design our method as a plug&play system, which can be integrated into existing well-trained VLMs for further representation augmentation, without training from scratch or relying on SG annotations in downstream applications. On 6 representative VL modeling tasks over 12 datasets in both standard and long-form video scenarios, Finsta consistently improves the existing 13 strong-performing VLMs persistently, and refreshes the current state-of-the-art end task performance significantly in both the fine-tuning and zero-shot settings.

Pattern-Aware Transformer: Hierarchical Pattern Propagation in Sequential Medical Images.

This paper investigates how to effectively mine contextual information among sequential images and jointly model them in medical imaging tasks. Different from state-of-the-art methods that model sequential correlations via point-wise token encoding, this paper develops a novel hierarchical pattern-aware tokenization strategy. It handles distinct visual patterns independently and hierarchically, which not only ensures the full flexibility of attention aggregation under different pattern representations but also preserves both local and global information simultaneously. Based on this strategy, we propose a Pattern-Aware Transformer (PATrans) featuring a global-local dual-path pattern-aware cross-attention mechanism to achieve hierarchical pattern matching and propagation among sequential images. Furthermore, PATrans is plug-and-play and can be seamlessly integrated into various backbone networks for diverse downstream sequence modeling tasks. We demonstrate its general application paradigm across four domains and five benchmarks in video object detection and 3D volumetric semantic segmentation tasks, respectively. Impressively, PATrans sets new state-of-the-art across all these benchmarks, i.e., CVC-Video (92.3% detection F1), ASU-Mayo (99.1% localization F1), Lung Tumor (78.59% DSC), Nasopharynx Tumor (75.50% DSC), and Kidney Tumor (87.53% DSC). Codes and models are available at https://github.com/GGaoxiang/PATrans.

Predictive Analytics in Data Engineering: An AI Approach

Predictive analytics, as a cornerstone of data engineering, has witnessed a paradigm shift with the integration of Artificial Intelligence (AI) methodologies. This abstract provides an overview of the key themes explored in the paper titled “Predictive Analytics in Data Engineering: An AI Approach.” The paper delves into the transformative impact of AI on predictive analytics within the domain of data engineering. Traditional predictive analytics often relied on statistical models and historical data patterns to forecast future trends. The advent of AI technologies, particularly machine learning and deep learning, has revolutionized the predictive analytics landscape by enabling systems to autonomously learn and adapt from data. A central focus of the paper is the exploration of advanced AI algorithms in predictive analytics. Machine learning models, such as regression, decision trees, and ensemble methods, are examined for their efficacy in predictive modeling tasks. Additionally, the integration of deep learning architectures, known for their ability to capture intricate patterns in large datasets, is explored for enhancing predictive accuracy. The convergence of predictive analytics and AI introduces a dynamic dimension to data engineering workflows. The paper outlines how AI-driven predictive analytics not only enhances the accuracy of predictions but also automates feature extraction, identifies complex patterns, and adapts to evolving data structures. The synergy between AI and predictive analytics empowers data engineers to navigate the challenges posed by big data and unstructured datasets. Ethical considerations and interpretability in AI-driven predictive analytics are also scrutinized in the paper. As AI models become increasingly complex, ensuring transparency in decision-making processes and addressing biases are crucial for responsible deployment in real-world scenarios. The findings presented in this paper contribute to the evolving discourse on the integration of AI in predictive analytics within the realm of data engineering. By examining the practical implications, challenges, and ethical dimensions, the paper provides valuable insights for practitioners, researchers, and organizations aiming to harness the full potential of AI in predictive analytics to drive informed decision-making and innovation in data engineering workflows.

Innovations in Ergonomic Risk Assessment and Intervention in Material Handling

This review paper delves into the complex facets of ergonomic risk evaluation and reduction across various industrial sectors. The focus is on identifying the hazards linked to manual labor, especially those contributing to physical risks and musculoskeletal disorders among workers, and proposing inventive solutions to mitigate these threats. The review brings together insights from different studies, emphasizing the application of automation, virtual modeling, Kinect-based evaluations, and other innovative tools in repetitive tasks and material handling operations. It also explores the usage of back-support exoskeletons, observational checklists, and workspace redesigns to decrease risks in construction, manufacturing, vegetable transportation, clothing accessories, and other sectors. The development of risk assessment frameworks for specific roles such as container terminal operators and the examination of musculoskeletal disorders in diverse worker demographics, including older female farmers and factory workers, are also featured. The comprehensive approach of the journal sets the stage for future research, aiming to further reduce the prevalence of musculoskeletal disorders through continued innovation and cross-sector collaboration.

Сочетаемость слова «будущее» как лингвокогнитивный инструмент изучения образной семантики поэтического концепта

The study aims to present various mechanisms of cognitive modelling as a way to detect integrity and stability in the figurative system of poetic language. The paper considers the semantic representation of one element of the figurative-associative level of the FUTURE concept, i.e., the lexeme будущее (the future) in comparison with the element прошлое (the past) of the same conceptual macro-field “Time”. The scientific novelty lies in identifying the general patterns of figurative conceptualization of a poetic reality fragment, i.e., the Future universal. As a result of the study, it was found that the system of figurative paradigms developed on the basis of cognitive metaphors is characterized by stability and reproducibility, serves to store and transmit methods of poetic thinking in linguistic forms. When examining asymmetry of related literary concepts as independent units of literary discourse, collocability becomes an effective tool for identifying the figurative-associative content of the concept. Predicative and non-predicative phrases perform the task of cognitive modelling in different ways. The semantic compatibility of the lexeme representing the concept is stable in poetic language as a secondary semiotic system.

Mathematical Formulation of Learning and Its Computational Complexity for Transformers’ Layers

Transformers are the cornerstone of natural language processing and other much more complicated sequential modelling tasks. The training of these models, however, requires an enormous number of computations, with substantial economic and environmental impacts. An accurate estimation of the computational complexity of training would allow us to be aware in advance about the associated latency and energy consumption. Furthermore, with the advent of forward learning workloads, an estimation of the computational complexity of such neural network topologies is required in order to reliably compare backpropagation with these advanced learning procedures. This work describes a mathematical approach, independent from the deployment on a specific target, for estimating the complexity of training a transformer model. Hence, the equations used during backpropagation and forward learning algorithms are derived for each layer and their complexity is expressed in the form of MACCs and FLOPs. By adding all of these together accordingly to their embodiment into a complete topology and the learning rule taken into account, the total complexity of the desired transformer workload can be estimated.

Capturing natural position relationships: A neural differential equation approach

The Transformer has emerged as the predominant model in Natural Language Processing due to its exceptional performance in various sequence modeling tasks, particularly in handling long-term dependencies. However, the traditional absolute and relative position encoding methods, which do not learn from data, tend to ignore the inherent structure of natural language sequences due to the position embedding layer at the input end of the Transformer model. This paper introduces a novel learnable neural Ordinary Differential Equation Position Encoding (ODEPE) method that can implicitly capture the natural position relationships within a sequence without requiring additional position embeddings. ODEPE can model continuous sequences and leverage differential equations to simulate the evolution of position information along the sequence, enabling position information to flow seamlessly between sequences. Additionally, a highly effective recurrent attention framework is proposed, which hybridizes attention with the ODEPE method to improve model performance. Compared to the Transformer-based sequence modeling network, our framework demonstrates a performance improvement of 24.0 points on the WikiText-103 dataset, while also achieving a performance improvement of 1.06 points on the Enwik8 dataset. This corresponds to an improvement of 4.9% and 0.17%, respectively.