Open Dataset Research Articles

Putting People First in Data Quality: Feminist Data Ethics for Open Government Datasets

ABSTRACTOpen government information systems offer great potential for advancing civic life and democracy, but they also reflect and reinforce the biases and systematic inequalities faced by members of socially marginalized groups. We present results from a critical data modeling project that uses a data quality framework to examine open datasets published by police departments in order to understand how data modeling choices shape the social impact of these datasets. Using an arrest record dataset published by the Los Angeles Police Department as a case study, we present we present results detailing the representation of racial data and the presence of children in the dataset. We argue that current data quality frameworks for open government data are insufficient for critical data studies due to an orientation around institutional and computational interests. Incorporating feminist data ethics into data quality analysis provides an approach to data quality that centers people and communities. We propose a definition for data quality of open government datasets based on an ethics of care that centers the needs of vulnerable populations and accountability of institutions toward their communities.

Investigating hunting in a protected area in Southeast Asia using passive acoustic monitoring with mobile smartphones and deep learning

Hunting is one of the most serious threats to wildlife populations. Guns can be used to hunt both terrestrial and arboreal species. While many methods and techniques have been developed to monitor wildlife populations, few techniques have been developed that can meet the demand needed to monitor threats to wildlife populations and support intervention activities; this is particularly true in Southeast Asia. Here, we used smartphones to record gunshots at 64 sites that were systematically spaced in Chu Mom Ray National Park, Vietnam from July to November 2019. Our specific goals were to: 1) quantify temporal and spatial patterns in gunshots in the national park; 2) investigate the correlation between gunshots and seven environmental variables including forest quality, elevation, and distance to the nearest village or ranger post; and 3) compare the performance of three convolutional neural network (CNN) architectures for automatically detecting gunshots. We manually annotated spectrograms of gunshots using Raven Pro 1.6. Using the manual detection approach, we identified 115 gunshots at 30 sites. The number of gunshots recorded at each recording site over two days varied from 0 to 11. On average, there were about 0.93 gunshots recorded per day per recording site. Hunting activity showed a strong temporal trend. The number of gunshots detected increased gradually from early morning, reaching the peak at noon, with the most gunshots recorded from 10:00 to 14:00 local time, equivalent to 0.18 gunshots per hour per site. No gunshots were detected from 22:00 to 04:00 in the morning. There were no strong relationships between the number of gunshots and all environmental variables, with all the correlation coefficients smaller than 0.3. Comparing three CNN architectures — AlexNet, VGG16, and ResNet18— implemented in the ‘torch for R’ ecosystem, we found that both AlexNet and VGG16 architectures led to acceptable performance for automated detection (F1 score > 0.80), but the ResNet18 architecture did not perform well for this task. We found low generalizability, as the models had relatively low performance on an open dataset from Belize (F1 score ∼ 0.52). To implement effective patrols, protected-area managers should make a hunting-risk map. The effort required for this task can be substantially reduced by using a combination of PAM and automated gunshot detection, however these approaches still require creation of a site-specific training and test dataset, along with manual verification of the detections.

Generative model-assisted sample selection for interest-driven visual analytics

We propose interest-driven visual analytics. The core idea is to filter samples with features of interest to analysts from the given dataset for analysis. The approach relies on a generative model (GM) trained using the given dataset as the training set. The GM characteristics make it convenient to find ideal generated samples from its latent space. Then, we filter the original samples similar to the ideal generated ones to explore patterns. Our research involves two methods for achieving and applying the idea. First, we give a method to explore ideal samples from a GM’s latent space. Second, we integrate the method into a system to form an embedding-based analytical workflow. Patterns found on open datasets in case studies, results of quantitative experiments, and positive feedback from experts illustrate the general usability and effectiveness of the approach.

Assessing dynamics of urban solar PV power generation using grid divisional method

The assessment of solar energy potential and urban density has become a crucial prerequisite for urban sustainable development. However, two significant challenges persist: the high computational cost associated with assessments across large urban areas and the lack of detailed rooftop information for each building. This study aims to integrate solar photovoltaic (PV) systems in urban environments of varying built density in an Indian city and assesses the solar energy potential (SEP) using grid divisional method and simulations. The methodology involved extracting, filtering and developing a 2.5 D model with GIS using the open buildings dataset, followed by simulations of grid-connected PV systems using PV-Sol. The results demonstrated the dynamics in correlation of SEP and grid morphology indicators, with built density and roof area showing a strong positive relation. The financial analysis resulted in average payback period of 8–10 years. The study also highlighted significant CO2 emission reductions, with thermal power plants generating 26,982 tons/kWh compared to just 1,513 tons/kWh for solar power. The findings demonstrate the financial viability and environmental benefits of PV adoption in urban areas, offering crucial insights for sustainable energy planning and policy development.

LGTCN: A Spatial–Temporal Traffic Flow Prediction Model Based on Local–Global Feature Fusion Temporal Convolutional Network

High-precision traffic flow prediction facilitates intelligent traffic control and refined management decisions. Previous research has built a variety of exquisite models with good prediction results. However, they ignore the reality that traffic flows can propagate backwards on road networks when modeling spatial relationships, as well as associations between distant nodes. In addition, more effective model components for modeling temporal relationships remain to be developed. To address the above challenges, we propose a local–global features fusion temporal convolutional network (LGTCN) for spatio-temporal traffic flow prediction, which incorporates a bidirectional graph convolutional network, probabilistic sparse self-attention, and a multichannel temporal convolutional network. To extract the bidirectional propagation relationship of traffic flow on the road network, we improve the traditional graph convolutional network so that information can be propagated in multiple directions. In addition, in spatial global dimensions, we propose probabilistic sparse self-attention to effectively perceive global data correlations and reduce the computational complexity caused by the finite perspective graph. Furthermore, we develop a multichannel temporal convolutional network. It not only retains the temporal learning capability of temporal convolutional networks, but also corresponds each channel to a node, and it realizes the interaction of node features through output interoperation. Extensive experiments on four open access benchmark traffic flow datasets demonstrate the effectiveness of our model.

Machine learning-based layer-wise detection of overheating anomaly in LPBF using photodiode data

Overheating anomaly detection is essential for the quality and reliability of parts produced by laser powder bed fusion (LPBF) additive manufacturing (AM). In this research, we focus on the detection of overheating anomalies which can lead to various defects in the part including geometric distortion, and poor surface roughness, among others, using photodiode sensor data. Photodiode sensors can collect high-frequency data from the melt pool, reflecting the process dynamics and thermal history. Hence, the proposed method offers a machine learning (ML) framework to utilize photodiode sensor data for layer-wise detection of overheating anomalies. In doing so, three sets of features are extracted from the raw photodiode data: MSMM (mean, standard deviation, median, maximum), MSQ (mean, standard deviation, quartiles), and MSD (mean, standard deviation, deciles). These three datasets are used to train several ML classifiers. Cost-sensitive learning is used to handle the class imbalance between the “anomalous” layers (affected by overheating) and “nominal” layers in the benchmark dataset. To boost detection accuracy, our proposed ML framework involves utilizing the majority voting ensemble (MVE) approach. First, the top three ML classifiers are identified from an initial pool of classifiers, based on their performance in k-fold cross-validation. Next, final predictions are generated using majority voting from the individual predictions of the top three classifiers. We performed 100 iterations to generate statistically reliable results. This proposed method is demonstrated using a case study including an open benchmark dataset of photodiode measurements from an LPBF specimen with deliberate overheating anomalies at some layers. The results from the case study demonstrate that the MSD features yield the best performance for all classifiers, and the MVE classifier (with a mean F1-score of 0.8654) surpasses the individual ML classifiers. Moreover, our machine learning methodology achieves superior results (9.66% improvement in mean F1-score) in detecting layer-wise overheating anomalies, surpassing the existing methods in the literature that use the same benchmark dataset. Finally, based on our results, we provide useful insights and recommendations for future research on applying machine learning techniques for defect detection in AM.

Standardizing Vehicle Travel Speed Data for Road Safety

ABSTRACTIn 2022, almost 50,000 people died in road crashes in the United States, with speeding implicated in 29% of these fatalities. Despite known links between vehicle speed and crash occurrence and severity, there are no federal guidelines for collecting vehicle travel speed (VTS) data. Cities with open VTS data are using unstandardized datasets, which complicates large‐scale and cross‐jurisdictional analysis. We conducted a qualitative assessment of open data repositories for the 25 largest U.S. cities, using a framework of knowledge representation, evaluated twelve metadata components, and determined the potential usability of these datasets. Our knowledge representation framework includes five data elements: speed metric, timestamp, geospatial representation, posted speed and vehicle type. Findings show that one‐quarter of these cities have open VTS datasets. Of those cities, none has a VTS dataset containing all the elements defined in our framework. This suggests the need to design information policy standards for the collection and sharing of open VTS data.

Community Partnerships in Digital Cultural Heritage: A Systematic Review

ABSTRACTCommunity partnerships with libraries, archives, and museums are increasingly recognized as essential to developing and stewarding complete, accurate, and equitable digital cultural records. While research and practice in this area has burgeoned, there is limited empirical research characterizing the state of the field. This study offers preliminary findings of a systematic, state‐of‐the‐art review of community‐institutional partnerships in digital cultural heritage collections and initiatives. This review systematically identifies wide‐ranging, active community partnerships among relevant institutions and their high‐level objectives, and a correlated array of approaches taken in pursuit of those objectives. The resulting conceptual framework and open access dataset form the base of a toolkit supporting community‐institutional partnerships for a more sustainable, community‐centered cultural record.

Dual edge-embedding graph convolutional network for unified aspect-based sentiment analysis

Aspect-based sentiment analysis (ABSA) mainly involves three elements, i.e., aspect, target, and sentiment. Most existing research focuses on aspect-sentiment or target-sentiment separately, while connection among the three elements is still underexploited. In order to extract three interrelated elements from a sentence simultaneously, we propose a unified end-to-end model, which is based on one aspect via a virtual edge attention mechanism and a graph convolutional network (GCN) to detect related targets and to analyze the corresponding sentiment polarity. The virtual edges are decided by the association intensity between the aspect and the words of the sentence. Instead of naively regarding all connecting edges as binary values, we design a novel GCN with a dual edge-embedding method to help information flow among each word in the sentence and between the aspect category and the sentence. Extensive experiments on two open datasets show that our model outperforms the state-of-the-art models in the aspect-target-sentiment triples detection task and its two subtasks.

Charting Ethical Terrain: The Functon of Artificial Intelligence in Oral and Maxillofacial Imaging

The integration of artificial intelligence (AI) into maxillofacial imaging represents a significant advancement in diagnostics and therapy. This review explores the ethical implications of AI in this specialized area, addressing concerns such as data privacy, informed consent, and algorithmic bias. It highlights the potential benefits of AI for patient outcomes and clinical efficiency while acknowledging risks associated with reliance on automated systems. The review aims to establish a framework for ethical guidelines to ensure that AI enhances patient care. AI's application in various industries has gained momentum, with dentistry, particularly oral and maxillofacial radiology, emerging as a promising field. Recent studies have focused on convolutional neural networks for tasks such as image classification, detection, segmentation, and refinement. These AI systems support radiographic diagnosis, image analysis, forensic dentistry, and image quality enhancement. However, optimal performance requires large, well-labeled datasets, necessitating significant input from oral and maxillofacial radiologists, which can be time-intensive. For AI to be effectively integrated into clinical practice, several challenges must be overcome, including the creation of comprehensive open datasets, understanding AI judgment criteria, and addressing DICOM hacking threats. By developing solutions alongside AI advancements, the technology can significantly evolve, potentially transforming automated diagnosis, treatment planning, and tool development. Oral and maxillofacial radiologists will play a crucial role in shaping AI applications in their field, leveraging their expertise in interpreting radiographic images.

A benchmark for Rey-Osterrieth complex figure test automatic scoring

The Rey–Osterrieth complex figure (ROCF) test is a neuropsychological task that can be useful for early detection of cognitive decline in the elderly population. Several computer vision systems have been proposed to automate this complex analysis task, but the lack of public benchmarks does not allow a fair comparison of these systems. To advance in that direction, we present a benchmarking framework for the automatic scoring of the ROCF test that provides: the ROCFD528 dataset, which is the first open dataset of ROCF line drawings; and experimental results obtained by several modern deep learning models, which can be used as a baseline for comparing new proposals. We evaluate different state-of-the-art convolutional neural networks (CNNs) under traditional and transfer learning paradigms. Experimental quantitative results (MAE = 3.448) indicate that a CNN specifically designed for sketches outperforms other state of the art CNN architectures when the number of examples available is limited. This benchmark can also be a paradigmatic example within the broad field of machine learning for the development of efficient and robust models for analyzing line drawings and sketches not only in classification but also in regression tasks.

A cluster approach to matching the competences of data specialists with skills in demand on the labour market

This paper addresses the challenge of aligning the competences of data specialists with skills in demand on the labour market in the rapidly evolving field of data science. Using an open dataset of 3,744 IT job postings, the study applies K-means clustering to identify key skill groupings for data specialist positions. The optimal number of clusters is determined using the elbow method, resulting in four distinct clusters: Data Analyst & Engineer, Data Platform Engineer, Data Science & Engineering Specialist, and Cloud Data Engineer. The research methodology employs unsupervised learning techniques, specifically K-means clustering, to analyze the distribution of skills across job postings. The clusters are visualized using t-distributed Stochastic Neighbor Embedding (t-SNE), providing insights into the relationships between different skill sets. The study reveals that job titles do not always unambiguously define the required skills, emphasizing the importance of focusing on specific skill sets rather than job titles alone. To bridge the gap between specific subject competences academic programs and industry requirements, the paper proposes a novel approach for comparing the proportion of skills in job clusters with the proportion of professional competencies in academic programs. This method is demonstrated using the Information Systems and Technologies Master's program at Kherson State University as a case study. The chi-square test is applied to confirm the statistical similarity between the skill structure of the Data Science & Engineering Specialist cluster and the competency structure of the academic program. The findings highlight the importance of continuous adaptation of profile of academic program to meet evolving industry needs. The proposed approach provides a data-driven framework for universities to align their programs with labor market demands, potentially improving graduate employability in the data science field. The study also underscores the need for personalized learning paths that can be tailored to individual career goals and skill gaps. Future research directions include the development of an artificial intelligence system to form individualized educational trajectories based on the skills required for specific job clusters. This could further enhance the alignment between education and industry needs, preparing students more effectively for the dynamic data science job market.

Temporal Dynamics of Hepatitis B Infection and Relation of Childhood Vaccination Program in Türkiye: A Longitudinal Study.

Hepatitis B is a significant infectious disease on a global scale. The implementation of vaccination programs and other preventive measures (e.g., serologic screening of blood donors) leads to lower rates of new infections with the hepatitis B virus. This study aimed to investigate and compare the evolution of hepatitis B incidence, prevalence and mortality rates along with vaccination rates in Türkiye and worldwide between 1990 and 2019. The study analyzed open datasets (the Global Burden of Disease and Our World in Data) using descriptive and joinpoint regression analysis to uncover substantial declines in hepatitis B rates from 1990 to 2019, both in Türkiye and globally. The average annual percentage change (AAPC) for incidence rates was -1.81 in Türkiye and -1.481 in the global cohort. For prevalence, the AAPC was -2.5244 in Türkiye and -1.4104 globally. The vaccination rates increased over time and remained at 95% since 2009, suggesting that vaccination efforts effectively reduced the risk of hepatitis B infection among children. This study provided important perspectives for assessing Türkiye's performance in this field and guiding future strategies.

Robust Hashing with Deep Features and Meixner Moments for Image Copy Detection

Copy detection is a key task of image copyright protection. Most robust hashing schemes do not make satisfied performance of image copy detection yet. To address this, a robust hashing scheme with deep features and Meixner moments is proposed for image copy detection. In the proposed hashing, global deep features are extracted by applying tensor Singular Value Decomposition (t-SVD) to the three-order tensor constructed in the DWT domain of the feature maps calculated by the pre-trained VGG16. Since the feature maps in the DWT domain are slightly disturbed by digital operations, the constructed three-order tensor is stable and thus the desirable robustness is guaranteed. Moreover, since t-SVD can decompose a three-order tensor into multiple low-dimensional matrices reflecting intrinsic structure, the global deep feature calculation from the low-dimensional matrices can provide good discrimination. Local features are calculated by the block-based Meixner moments. As the Meixner moments are resistant to geometric transformation and can efficiently discriminate various images, the use of the block-based Meixner moments can make discriminative and robust local features. Hash is ultimately determined by quantifying and combining global deep features and local features. The results of extensive experiments on open image datasets demonstrate that the proposed robust hashing outperforms some state-of-the-art robust hashing schemes in terms of classification and copy detection performances.

RiskPath: Explainable deep learning for multistep biomedical prediction in longitudinal data.

Predicting individual and population risk for disease outcomes and identifying persons at elevated risk is a key prerequisite for targeting interventions to improve health. However, current risk stratification tools for the common, chronic diseases that develop over the lifecourse and represent the majority of disease morbidity, mortality and healthcare costs are aging and achieve only moderate predictive performance. In some common, highly morbid conditions such as mental illness no risk stratification tools are yet available. There is an urgent need to improve predictive performance for chronic diseases and understand how cumulative, multifactorial risks aggregate over time so that intervention programs can be targeted earlier and more effectively in the disease course. Chronic diseases are the end outcomes of multifactorial risks that increment over years and represent cumulative, temporally-sensitive risk pathways. However, tools in current clinical use were constructed in older data and utilize inputs from a single data collection step. Here, we present RiskPath, a multistep deep learning method for temporally-sensitive biomedical risk prediction tailored for the constraints and demands of biomedical practice that achieves very strong performance and full translational explainability. RiskPath delineates and quantifies cumulative multifactorial risk pathways and allows the user to explore performance-complexity tradeoffs and constrain models as required by clinical use cases. Our results highlight the potential for developing a new generation of risk stratification tools and risk pathway mapping in time-dependent diseases and health outcomes by leveraging powerful timeseries deep learning methods in the wealth of biomedical data now appearing in large, longitudinal open science datasets.

An open dataset for individual tree detection in UAV LiDAR point clouds and RGB orthophotos in dense mixed forests

We present an open access dataset for development, evaluation, and comparison of algorithms for individual tree detection in dense mixed forests. The dataset consists of a detailed field inventory and overlapping UAV LiDAR and RGB orthophoto, which make it possible to develop algorithms that fuse multimodal data to improve detection results. Along with the dataset, we describe and implement a basic local maxima filtering baseline and an algorithm for automatically matching detection results to the ground truth trees for detection algorithm evaluation.

Assessment of Dataset Scalability for Classification of Black Sigatoka in Banana Crops Using UAV-Based Multispectral Images and Deep Learning Techniques

This paper presents an evaluation of different convolutional neural network (CNN) architectures using false-colour images obtained by multispectral sensors on drones for the detection of Black Sigatoka in banana crops. The objective is to use drones to improve the accuracy and efficiency of Black Sigatoka detection to reduce its impact on banana production and improve the sustainable management of banana crops, one of the most produced, traded, and important fruits for food security consumed worldwide. This study aims to improve the precision and accuracy in analysing the images and detecting the presence of the disease using deep learning algorithms. Moreover, we are using drones, multispectral images, and different CNNs, supported by transfer learning, to enhance and scale up the current approach using RGB images obtained by conventional cameras and even smartphone cameras, available in open datasets. The innovation of this study, compared to existing technologies for disease detection in crops, lies in the advantages offered by using drones for image acquisition of crops, in this case, constructing and testing our own datasets, which allows us to save time and resources in the identification of crop diseases in a highly scalable manner. The CNNs used are a type of artificial neural network widely utilised for machine training; they contain several specialised layers interconnected with each other in which the initial layers can detect lines and curves, and gradually become specialised until reaching deeper layers that recognise complex shapes. We use multispectral sensors to create false-colour images around the red colour spectra to distinguish infected leaves. Relevant results of this study include the construction of a dataset with 505 original drone images. By subdividing and converting them into false-colour images using the UAV’s multispectral sensors, we obtained 2706 objects of diseased leaves, 3102 objects of healthy leaves, and an additional 1192 objects of non-leaves to train classification algorithms. Additionally, 3640 labels of Black Sigatoka were generated by phytopathology experts, ideal for training algorithms to detect this disease in banana crops. In classification, we achieved a performance of 86.5% using false-colour images with red, red edge, and near-infrared composition through MobileNetV2 for three classes (healthy leaves, diseased leaves, and non-leaf extras). We obtained better results in identifying Black Sigatoka disease in banana crops using the classification approach with MobileNetV2 as well as our own datasets.

Analyzing Large-Scale Political Discussions on Twitter: The Use Case of the Greek Wiretapping Scandal (#ypoklopes)

In this paper, we study the Greek wiretappings scandal, which was revealed in 2022 and attracted significant attention from the press and citizens. Specifically, we propose a methodology for collecting data and analyzing patterns of online public discussions on Twitter. We apply our methodology to the Greek wiretappings use case and present findings related to the evolution of the discussion over time, its polarization, and the role of the Media. The methodology can be of wider use and replicated to other topics. Finally, we publicly provide an open dataset and online resources with the results.

Bee Together: Joining Bee Audio Datasets for Hive Extrapolation in AI-Based Monitoring.

Beehive health monitoring has gained interest in the study of bees in biology, ecology, and agriculture. As audio sensors are less intrusive, a number of audio datasets (mainly labeled with the presence of a queen in the hive) have appeared in the literature, and interest in their classification has been raised. All studies have exhibited good accuracy, and a few have questioned and revealed that classification cannot be generalized to unseen hives. To increase the number of known hives, a review of open datasets is described, and a merger in the form of the "BeeTogether" dataset on the open Kaggle platform is proposed. This common framework standardizes the data format and features while providing data augmentation techniques and a methodology for measuring hives' extrapolation properties. A classical classifier is proposed to benchmark the whole dataset, achieving the same good accuracy and poor hive generalization as those found in the literature. Insight into the role of the frequency of the classification of the presence of a queen is provided, and it is shown that this frequency mostly depends on a colony's belonging. New classifiers inspired by contrastive learning are introduced to circumvent the effect of colony belonging and obtain both good accuracy and hive extrapolation abilities when learning changes in labels. A process for obtaining absolute labels was prototyped on an unsupervised dataset. Solving hive extrapolation with a common open platform and contrastive approach can result in effective applications in agriculture.

Breaking All the Rules: Information Technology Procurement in the Government of Canada

AbstractThe Government of Canada has recently faced intense parliamentary and public scrutiny of the role played by private contractors in its information technology (IT) projects, most notably in the case of the ArriveCAN application. With these ongoing investigations as its backdrop, this article analyzes patterns in federal government IT procurement between 2017 and 2022, drawing on a comprehensive analysis of the federal contracting open dataset. We reveal that the federal government betrays accepted best practice in modern government IT procurement on several key dimensions, including on contract values and lengths; on the diversity of suppliers; on the source of IT expertise; and in the management of intellectual property. We argue that the Canadian approach to IT procurement is an historically overlooked but crucial driver of its failing digital reform efforts. We conclude by turning to IT procurement policy reforms gaining traction outside Canada that may help the Government of Canada improve how it buys and deploys IT going forward—a task we argue is essential if the government wants to avoid future IT contracting scandals and deliver on its long‐standing promise of digital era modernization.