High-quality Datasets Research Articles

Revisiting Bundle Recommendation for Intent-aware Product Bundling

Product bundling represents a prevalent marketing strategy in both offline stores and e-commerce systems. Despite its widespread use, previous studies on bundle recommendation face two significant limitations. Firstly, they rely on noisy datasets, where bundles are defined by heuristics, e.g., products co-purchased in the same session. Secondly, they target specific tasks by holding unrealistic assumptions, e.g., the availability of bundles for recommendation directly. This paper proposes to take a step back and considers the process of bundle recommendation from a holistic user experience perspective. We first construct high-quality bundle datasets with rich metadata, particularly bundle intents, through a carefully designed crowd-sourcing task. We then define a series of tasks that together, support all key steps in a typical bundle recommendation process, from bundle detection, completion and ranking, to explanation and auto-naming, whereby 19 research questions are raised correspondingly to guide the analysis. Finally, we conduct extensive experiments and analyses with representative recommendation models and large language models (LLMs), demonstrating the challenges and opportunities, especially with the emergence of LLMs. To summarize, our study contributes by introducing novel data sources, paving the way for new research avenues, and offering insights to guide product bundling in real e-commerce platforms.

Meta-Fed IDS: Meta-learning and Federated learning based fog-cloud approach to detect known and zero-day cyber attacks in IoMT networks

The Internet of Medical Things (IoMT) is a transformative fusion of medical sensors, equipment, and the Internet of Things, positioned to transform healthcare. However, security and privacy concerns hinder widespread IoMT adoption, intensified by the scarcity of high-quality datasets for developing effective security solutions. Addressing these challenges, we propose a novel framework for cyberattack detection in dynamic IoMT networks. This framework integrates Federated Learning with Meta-learning, employing a multi-phase architecture for identifying known attacks, and incorporates advanced clustering and biased classifiers to address zero-day attacks. The framework's deployment is adaptable to dynamic and diverse environments, utilizing an Infrastructure-as-a-Service (IaaS) model on the cloud and a Software-as-a-Service (SaaS) model on the fog end. To reflect real-world scenarios, we introduce a specialized IoMT dataset. Our experimental results indicate high accuracy and low misclassification rates, demonstrating the framework's capability in detecting cyber threats in complex IoMT environments. This approach shows significant promise in bolstering cybersecurity in advanced healthcare technologies.

Benchmarking compound activity prediction for real-world drug discovery applications

Identifying active compounds for target proteins is fundamental in early drug discovery. Recently, data-driven computational methods have demonstrated promising potential in predicting compound activities. However, there lacks a well-designed benchmark to comprehensively evaluate these methods from a practical perspective. To fill this gap, we propose a Compound Activity benchmark for Real-world Applications (CARA). Through carefully distinguishing assay types, designing train-test splitting schemes and selecting evaluation metrics, CARA can consider the biased distribution of current real-world compound activity data and avoid overestimation of model performances. We observed that although current models can make successful predictions for certain proportions of assays, their performances varied across different assays. In addition, evaluation of several few-shot training strategies demonstrated different performances related to task types. Overall, we provide a high-quality dataset for developing and evaluating compound activity prediction models, and the analyses in this work may inspire better applications of data-driven models in drug discovery.

Prescriptive procedure for manual code smell annotation

– Code smells are structures in code that present potential software maintainability issues. Manually constructing high-quality datasets to train ML models for code smell detection is challenging. Inconsistent annotations, small size, non-realistic smell-to-non-smell ratio, and poor smell coverage hinder the dataset quality. These issues arise mainly due to the time-consuming nature of manual annotation and annotators’ disagreements caused by ambiguous and vague smell definitions.To address challenges related to building high-quality datasets suitable for training ML models for smell detection, we designed a prescriptive procedure for manual code smell annotation. The proposed procedure represents an extension of our previous work, aiming to support the annotation of any smell defined by Fowler. We validated the procedure by employing three annotators to annotate smells following the proposed annotation procedure.The main contribution of this paper is a prescriptive annotation procedure that benefits the following stakeholders: annotators building high-quality smell datasets that can be used to train ML models, ML researchers building ML models for smell detection, and software engineers employing ML models to enhance the software maintainability. Secondary contributions are the code smell dataset containing Data Class, Feature Envy, and Refused Bequest, and DataSet Explorer tool which supports annotators during the annotation procedure.

Feature Selection Techniques in Intrusion Detection: A Comprehensive Review

This investigation aims to explore previous research on the implementation of feature selection in intrusion detection. Feature selection has demonstrated its ability to enhance or sustain comparable classification accuracy levels for intrusion detection systems, while simultaneously improving classification efficiency. The evaluation includes an assessment of filter-based, wrapper-based, and hybrid feature selection techniques. Given that Big Data challenges can affect intrusion detection, feature selection’s classification efficiency can aid in lowering computing requirements. Older KDD intrusion detection datasets have received considerable attention in previous feature selection research. Consequently, researchers need more high-quality datasets that are available to the general public.

Harnessing AI for solar energy: Emergence of transformer models

This review emphasizes the critical need for accurate integration of solar energy into power grids. It meticulously examines the advancements in transformer models for solar forecasting, representing a confluence of renewable energy research and cutting-edge machine learning. It evaluates the effectiveness of various transformer architectures, including single, hybrid, and specialized models, across different forecasting horizons, from short to medium term. This review unveils substantial improvements in forecasting accuracy and computational efficiency, highlighting the models' proficiency in handling complex and diverse solar data. A key contribution is the emphasis on the crucial role of hyperparameters in refining model performance, balancing precision against computational demands. Importantly, the research also identifies critical challenges, such as the significant computational resources required and the need for expansive, high-quality datasets, which limit the broader application of these models. In response, this review advocates for future research directions focused on standardizing model configurations, venturing into longer-term forecasting, and fostering innovations to enhance computational economy. These proposed pathways aim to surmount current challenges, steering the domain towards more accurate, adaptable, and sustainable solar forecasting solutions that can contribute to achieving global renewable energy and climate objectives. This review not only maps the present landscape of transformer models in solar energy forecasting but also charts a trajectory for future advancements. It serves as a pivotal guide for researchers and practitioners, delineating the current advancements and future directions in navigating the complexities of solar data interpretation and forecasting, thereby significantly contributing to the development of reliable and efficient renewable energy systems.

ParisLuco3D: A High-Quality Target Dataset for Domain Generalization of LiDAR Perception

ParisLuco3D: A High-Quality Target Dataset for Domain Generalization of LiDAR Perception

AI-generated synthetic clinical-genomic data for precision oncology research: Validation using a case study on lung adenocarcinoma.

e13627 Background: The analysis of genomic variants is crucial in precision oncology research, offering insights into cancer risks and progression, especially in diverse types such as lung adenocarcinoma (LUAD). However, such research often grapples with balancing patient privacy with the need for comprehensive, high-quality genomic datasets. Our project addresses this by creating synthetic clinical-genomic data, which maintains patient confidentiality and provides a rich resource for genomic cancer research. Methods: Leveraging the GuardantINFORM database, which includes anonymized genomic data and structured payer claims, we focused on generating synthetic data for LUAD patient cohorts. This approach involves processing real patient data into a format compatible with Medisyn’s generative AI models, ensuring the synthetic data retains the original's statistical properties, and processing the output back into the original database structure and format. This method plays a crucial role in maintaining patient privacy and serves as a valuable tool for research by enabling the generation of realistic patients with desired properties on demand. Results: Our synthetic data closely mirrors real-world genomic and claims variable distributions, evidenced by a 0.994 R2 correlation between real and synthetic data along with comparable Oncoprints. Importantly, privacy tests show that patient confidentiality is effectively maintained despite this effective performance. The synthetic data's utility was then demonstrated in a study replicating real-world findings: LUAD patients with KRAS G12C in combination with STK11 mutations showed a significantly higher risk of early mortality. This underscores the potential of synthetic data in advancing cancer research. Conclusions: This research offers a promising avenue for the cancer research community. By providing a method to share privatized, synthetic genomic data, which can be combined and generated on demand, we enable broader, more responsible data sharing. This approach protects patient privacy and offers a rich dataset for groundbreaking research, potentially accelerating advances in cancer diagnosis and treatment. [Table: see text]

The Rise of Data-Driven Weather Forecasting: A First Statistical Assessment of Machine Learning–Based Weather Forecasts in an Operational-Like Context

Abstract Data-driven modeling based on machine learning (ML) is showing enormous potential for weather forecasting. Rapid progress has been made with impressive results for some applications. The uptake of ML methods could be a game changer for the incremental progress in traditional numerical weather prediction (NWP) known as the “quiet revolution” of weather forecasting. The computational cost of running a forecast with standard NWP systems greatly hinders the improvements that can be made by increasing model resolution and ensemble sizes. An emerging new generation of ML models, developed using high-quality reanalysis datasets like ERA5 for training, allows forecasts that require much lower computational costs and that are highly competitive in terms of accuracy. Here, we compare for the first time ML-generated forecasts with standard NWP-based forecasts in an operational-like context, initialized from the same initial conditions. Focusing on deterministic forecasts, we apply common forecast verification tools to assess to what extent a data-driven forecast produced with one of the recently developed ML models (PanguWeather) matches the quality and attributes of a forecast from one of the leading global NWP systems (the ECMWF IFS). The results are very promising, with comparable accuracy for both global metrics and extreme events, when verified against both the operational IFS analysis and synoptic observations. Overly smooth forecasts, increasing bias with forecast lead time, and poor performance in predicting tropical cyclone intensity are identified as current drawbacks of ML-based forecasts. A new NWP paradigm is emerging relying on inference from ML models and state-of-the-art analysis and reanalysis datasets for forecast initialization and model training.

UGEE-Net: Uncertainty-guided and edge-enhanced network for image splicing localization

Image splicing, a prevalent method for image tampering, has significantly undermined image authenticity. Existing methods for Image Splicing Localization (ISL) struggle with challenges like limited accuracy and subpar performance when dealing with imperceptible tampering and multiple tampered regions. We introduce an Uncertainty-Guided and Edge-Enhanced Network (UGEE-Net) for ISL to tackle these issues. UGEE-Net consists of two core tasks: uncertainty guidance and edge enhancement. We employ Bayesian learning to model uncertainty maps of tampered regions, directing the model's focus to challenging pixels. Simultaneously, we employ a frequency domain-auxiliary edge enhancement strategy to imbue localization features with global contour information and fine-grained local details. These mechanisms work in parallel, synergistically boosting performance. Additionally, we introduce a cross-level fusion and propagation mechanism that effectively utilizes contextual information for cross-layer feature integration and leverages channel-level correlations for cross-layer feature propagation, gradually enhancing the localization feature's details. Experiment results affirm UGEE-Net's superiority in terms of detection accuracy, robustness, and generalization capabilities. Furthermore, to meet the growing demand for high-quality datasets in image forensics, we present the HTSI12K dataset, which includes 12,000 spliced images with imperceptible tampering traces and diverse categories, rendering it suitable for real-world auxiliary model training.

A cascaded GRU-based stereoscopic matching network for precise plank measurement

Wooden plank images in industrial measurements often contain numerous textureless areas. Furthermore, due to the thin plate structure, the three-dimensional (3D) disparity of these planks is predominantly confined to a narrow range. Consequently, achieving accurate 3D matching of wooden plank images has consistently presented a challenging task within the industry. In recent years, deep learning has progressively supplanted traditional stereo matching methods due to its inherent advantages, including rapid inference and end-to-end processing. Nonetheless, the acquisition of datasets for stereo matching networks poses an additional challenge, primarily attributable to the difficulty in obtaining accurate disparity data. Thus, this paper presents a novel stereo matching method incorporating three key innovations. Firstly, an enhanced gated recurrent unit network is introduced, accompanied by a redesigned structure to achieve higher matching accuracy. Secondly, an efficient preprocessing module is proposed, aimed at improving the algorithm’s efficiency. Lastly, in response to the challenges posed by datasets acquisition, we innovatively employed image simulation software to obtain a high-quality simulated dataset of wooden planks. To assess the feasibility of our approach, we conducted both simulated and real experiments. The experiments results clearly exhibit the superiority of our method when compared to existing approaches in terms of both stability and accuracy. In the simulation experiment, our method attained a bad1.0 score of 2.1% (compared to the baseline method’s 9.76%); In the real experiment, our method achieved an average error of 0.104 mm (compared to the baseline method’s 0.268 mm). It is worth noting that our study aims to address the challenge of acquiring datasets for deep learning and bridging the gap between simulated and real data, resulting in increased applicability of deep learning in more industrial measurement domains.

Generating high-resolution hyperspectral time series datasets based on unsupervised spatial-temporal-spectral fusion network incorporating a deep prior

Over the past decade, image fusion has emerged as an indispensable tool for surface monitoring due to its capability to reconstruct high-quality surface reflectance. While satisfactory progress has been made in the spatial-spectral and spatial-temporal fusion of remote sensing images, the fusion of spatial-temporal-spectral data remains a challenging task. This challenge arises from the presence of multiple nonlinear relationships caused by inconsistent temporal, spatial, and spectral resolutions. In this paper, we propose a novel approach to address the spatial-temporal-spectral fusion of remote sensing images using a triple discriminator Generative Adversarial Network called TDGAN. Our approach represents the first attempt at spatial-temporal-spectral fusion based on deep learning techniques. Specifically, TDGAN achieves unsupervised learning by incorporating a deep spectral transformation prior, explores the application performance of the Swin Transformer in spectral super-resolution for the first time. Additionally, three discriminators are designed based on the prior consistencies: phenological consistency for data acquired at different times and spatial-spectral consistency for data acquired at the same time. These discriminators effectively enhance the prediction accuracy of missing hyperspectral images. Extensive simulations and real experiments were conducted to evaluate the effectiveness of our proposed TDGAN. The results demonstrate its robustness and comparable performance to other state-of-the-art methods. The proposed approach provides high-quality hyperspectral datasets with high spatial and temporal resolutions, facilitating continuous land cover observations.

Automatic PCB Sample Generation and Defect Detection Based on ControlNet and Swin Transformer.

In order to improve the efficiency and accuracy of multitarget detection of soldering defects on surface-mounted components in Printed Circuit Board (PCB) fabrication, we propose a sample generation method using Stable Diffusion Model and ControlNet, as well as a defect detection method based on the Swin Transformer. The method consists of two stages: First, high-definition original images collected in industrial production and the corresponding prompts are input to Stable Diffusion Model and ControlNet for automatic generation of nonindependent samples. Subsequently, we integrate Swin Transformer as the backbone into the Cascade Mask R-CNN to improve the quality of defect features extracted from the samples for accurate detection box localization and segmentation. Instead of segmenting individual components on the PCB, the method inspects all components in the field of view simultaneously over a larger area. The experimental results demonstrate the effectiveness of our method in scaling up nonindependent sample datasets, thereby enabling the generation of high-quality datasets. The method accurately recognizes targets and detects defect types when performing multitarget inspection on printed circuit boards. The analysis against other models shows that our improved defect detection and segmentation method improves the Average Recall (AR) by 2.8% and the mean Average Precision (mAP) by 1.9%.

High-Accuracy Image Segmentation Based on Hybrid Attention Mechanism for Sandstone Analysis

Mineral image segmentation based on computer vision is vital to realize automatic mineral analysis. However, current image segmentation methods still cannot effectively solve the problem of sandstone grains that are adjoined and concealed by leaching processes, and the segmentation performance of small and irregular grains still needs to be improved. This investigation explores and designs a Mask R-CNN-based sandstone image segmentation model, including a hybrid attention mechanism, loss function construction, and receptive field enlargement. Simultaneously, we propose a high-quality sandstone dataset with abundant labels named SMISD to facilitate comprehensive training of the model. The experimental results show that the proposed segmentation model has excellent segmentation performance, effectively solving adhesion and overlap between adjacent grains without affecting the classification accuracy. The model has comparable performance to other models on the COCO dataset, and performs better on SMISD than others.

Usability of novel beat-by-beat ECG annotation software portal for generating large-scale machine learning training datasets

Abstract Introduction Despite the availability of numerous large publicly available ECG datasets, there is a distinct absence of specialized software to enable the beat-by-beat annotation and labeling, limiting the ability to generate high-quality machine learning training datasets. This study analyzed the usability of such a novel software interface. Through a survey of several cardiologists and clinic staff, several metrics were gauged, including user satisfaction, system and app simplicity, user interface, and effectiveness. The survey results gleaned insight into the users’ experiences, providing an understanding of the software’s user-friendliness and potential in developing a large-scale machine learning annotated ECG training dataset to advance cardiac health research. Methods 3 cardiology physicians, and 9 cardiac telemetry unit staff were surveyed following use of the SafeBeat Annotation Software Platform to annotate a large publicly available dataset ECGs (n=10,646) beat-by-beat. The validated Post-Study System Usability Questionnaire (PSSUQ) and mobile Health App Usability Questionnaire (MAUQ) were used to assess the overall usability of the software interface. Results are reported as mean ± standard deviation. Results Users completed 10,646 ECG beat-by-beat annotations in 62 days, and averaged 164.0 ECGs per day. Physicians rated the software interface highly, with an overall mean PSSUQ of 6.4 ± 0.8 and MAUQ of 6.4 ± 0.7. Staff also rated the software interface highly, with an overall mean PSSUQ of 6.7 ± 0.6 and MAUQ of 6.8 ± 0.5. Overall 92% of the users reported that they strongly agreed they would use the application again and 83% strongly agreed that it was easy to learn how to use the system. Conclusion The usability analysis of this novel beat-by-beat ECG annotation software revealed that it was a highly effective tool for both cardiologists and clinic staff. Both the system and app were reported to be simple to learn, comfortable to use, and easily navigable. With most users responding that they could become productive quickly using this system, there is clear potential for it to be used to develop a large-scale machine learning dataset.Figure 1

HLAIImaster: a deep learning method with adaptive domain knowledge predicts HLA II neoepitope immunogenic responses.

While significant strides have been made in predicting neoepitopes that trigger autologous CD4+ T cell responses, accurately identifying the antigen presentation by human leukocyte antigen (HLA) class II molecules remains a challenge. This identification is critical for developing vaccines and cancer immunotherapies. Current prediction methods are limited, primarily due to a lack of high-quality training epitope datasets and algorithmic constraints. To predict the exogenous HLA class II-restricted peptides across most of the human population, we utilized the mass spectrometry data to profile >223 000 eluted ligands over HLA-DR, -DQ, and -DP alleles. Here, by integrating these data with peptide processing and gene expression, we introduce HLAIImaster, an attention-based deep learning framework with adaptive domain knowledge for predicting neoepitope immunogenicity. Leveraging diverse biological characteristics and our enhanced deep learning framework, HLAIImaster is significantly improved against existing tools in terms of positive predictive value across various neoantigen studies. Robust domain knowledge learning accurately identifies neoepitope immunogenicity, bridging the gap between neoantigen biology and the clinical setting and paving the way for future neoantigen-based therapies to provide greater clinical benefit. In summary, we present a comprehensive exploitation of the immunogenic neoepitope repertoire of cancers, facilitating the effective development of "just-in-time" personalized vaccines.

Situational diversity in video person re-identification: introducing MSA-BUPT dataset

Thanks to the success of deep learning over the past few years, the video person re-identification (ReID) algorithms have achieved high accuracy on multiple public benchmark datasets. However, the available video person ReID datasets cover a limited range of real-world scenarios, and they have several obvious limitations: limited camera viewing angles, tiny variations of the shooting scene, and even errors in manual labels. These disadvantages prevent video person ReID from being widely used in real-life scenarios. In this work, a new high-quality multi-situation video person ReID dataset, named MSA-BUPT, is built to promote the video person ReID task in large-scale urban surveillance. Specifically, MSA-BUPT contains 684 identities, 2,665 trajectories, and nearly 250,000 frames from 200-h videos across various complex scenarios. Person attribute annotations and unannotated video data are also provided for other research perspectives, such as cross-modality ReID, cross-domain ReID, and so on. Furthermore, two plug-and-play components are used to improve retrieval capabilities: a new scenario-based data augmentation method is proposed to alleviate the person misalignment problem; a re-ranking strategy based on person attribute is applied to make secondary adjustments to the content to the results of the model. The extensive experimental results show that the above methods improve the performance of some representative state-of-the-art models on the new dataset.

Revolutionizing Organizational Decision-Making for Banking Sector: A Machine Learning Approach with CNNs in Business Intelligence and Management

This research investigates the transformative impact of deep learning, particularly Convolutional Neural Networks (CNNs) such as VGG16, ResNet50, and InceptionV3, on organizational management and business intelligence within the banking sector. Employing a comprehensive methodology, the study emphasizes the crucial role of high-quality datasets in harnessing deep learning for improved decision-making. Results reveal the superior performance of CNN models over traditional algorithms, with CNN (VGG16) achieving an impressive accuracy rate of 90%. These findings underscore the potential of deep learning in extracting valuable insights from complex data, presenting a paradigm shift in optimizing various banking processes. The article concludes by highlighting the importance of investing in infrastructure and expertise for successful CNN integration, while also addressing ethical and privacy considerations. This research contributes to the evolving discourse on deep learning applications in organizational management, offering valuable insights for banks navigating the challenges of the global market.

On the added value of satellite AOD for the investigation of ground-level PM2.5 variability

To assess and mitigate PM2.5 pollution, it is imperative to accurately characterize its spatiotemporal variability and distribution. As station measurements are spatially selective, additional data sources such as satellite observations or chemical transport models (CTMs) are indispensable for providing essential complementary information. With the increasing accessibility of high-quality reanalysis datasets on air pollutants from CTMs, the use of satellite-based observations with their limited spatial and temporal availability is more and more put into question. Model simulations present a clear advantage concerning spatiotemporal continuity. However, how reliable are these model simulations and how does the information they provide differ from satellite-derived data? By applying a random forest (RF) approach we derive PM2.5 concentrations for Germany based on observations of aerosol optical depth (AOD) and a variety of other atmospheric parameters and systematically compare the satellite-derived PM2.5 dataset to results from the CAMS regional model system. We discern the differences in the information they provide to demonstrate the added value of the satellite AOD-derived data for the investigation of long-term PM2.5 variability. The development in PM2.5 variability is analyzed on a yearly and seasonal base over the time period from 2018 to 2022, including the COVID-19 lockdown in spring 2020. We can demonstrate a better accuracy of the RF-based AOD model compared to CAMS reanalysis data with R2 values of 0.71 and 0.57, respectively. Moreover, we find that the AOD-based data provides more detail in spatial gradients and can better depict local and short-term events of enhanced PM2.5 pollution.

Enhancing citizen science impact in environmental monitoring: Targeted engagement strategies with stakeholder groups

Understanding the motivations and benefits of citizen science (CS) participants is critical to the success of environmental science projects that rely on data collection from engaged citizens. Tailored communication with citizen scientists is essential, leading to the need to target specific societal groups for extensive and high-quality data sets. The purpose of the study was to apply marketing concepts such as stakeholder analysis, value proposition canvas (VPC), and key performance indicators (KPIs) to CS projects. The study examined the extent to which these strategies can be applied to CS projects and add value, resulting in improved recruitment and retention of participants, as well as improved project management. The dynamic landscape of CS projects requires adapted business-oriented approaches supporting ongoing participation with high motivation, acknowledging community needs, and recognizing institutional scientists, akin to targeting potential customers. The case study focuses on the CityCLIM initiative, an EU-funded project collecting urban climate data, especially air temperature, using mobile weather stations. The CityCLIM VPC, analyzing stakeholder groups, facilitated tailored communication strategies by analyzing stakeholder groups and highlighting the effectiveness of the CS cycling initiative within the “Stadtradeln” program. Impact KPIs served as a roadmap for strategic refinement, while data quality KPIs identified deficiencies, guiding adjustments to the campaign. Applied marketing tools improved project planning, engagement, and evaluation, demonstrating the potential of this concept. Adapting marketing tools to recruitment and communication strategies benefits CS projects by targeting specific groups. The holistic integration of stakeholder analysis, VPC, and KPIs streamlines project management and creates a framework for sustainable success. This adaptation forms a robust toolkit for CS project coordinators, contributing to effective communication, engagement, and impact assessment. Applying marketing tools significantly increases CS projects’ reach and impact, ensuring long-term success and meaningful scientific contributions.