Data Science Community Research Articles

Towards practical artificial intelligence in Earth sciences

AbstractAlthough Artificial Intelligence (AI) projects are common and desired by many institutions and research teams, there are still relatively few success stories of AI in practical use for the Earth science community. Many AI practitioners in Earth science are trapped in the prototyping stage and their results have not yet been adopted by users. Many scientists are still hesitating to use AI in their research routine. This paper aims to capture the landscape of AI-powered geospatial data sciences by discussing the current and upcoming needs of the Earth and environmental community, such as what practical AI should look like, how to realize practical AI based on the current technical and data restrictions, and the expected outcome of AI projects and their long-term benefits and problems. This paper also discusses unavoidable changes in the near future concerning AI, such as the fast evolution of AI foundation models and AI laws, and how the Earth and environmental community should adapt to these changes. This paper provides an important reference to the geospatial data science community to adjust their research road maps, find best practices, boost the FAIRness (Findable, Accessible, Interoperable, and Reusable) aspects of AI research, and reasonably allocate human and computational resources to increase the practicality and efficiency of Earth AI research.

From selfish silo to collaborative culture – embracing data-enabled cancer research

From selfish silo to collaborative culture – embracing data-enabled cancer research Aedin Culhane and Mark Lawler, Co-Leads of the eHealth Hub for Cancer, reflect on their data-enabled cancer research journeys, how their collaborative team science approach has reaped significant dividends in cancer research and policy and how the hub is inducing a paradigm shift in how health data are deployed on the island of Ireland. An article in The Wall Street Journal (not the normal reading material for scientists) in 2011 highlighted a new approach to performing scientific research that was gaining significant credence at the time. Entitled ‘The New Einsteins Will Be Scientists Who Share’ (with the strapline ‘From cancer to cosmology, researchers could race ahead by working together – online and in the open’), the article presaged an unprecedented change in how scientists interact with each other, ushering in a culture of collaboration and cross-disciplinary research. Nowhere was this change more obvious than in the genomics and data science community, where a bottom-up movement led to the creation of the Global Alliance for Genomics and Health (GA4GH), bringing together researchers from around the world to work together to address some of human health’s greatest challenges through the deployment of data and data tools.

In Silico Human Mobility Data Science: Leveraging Massive Simulated Mobility Data (Vision Paper)

Human mobility data science using trajectories or check-ins of individuals has many applications. Recently, we have seen a plethora of research efforts that tackle these applications. However, research progress in this field is limited by a lack of large and representative datasets. The largest and most commonly used dataset of individual human trajectories captures fewer than 200 individuals, while datasets of individual human check-ins capture fewer than 100 check-ins per city per day. Thus, it is not clear if findings from the human mobility data science community would generalize to large populations. Since obtaining massive, representative, and individual-level human mobility data is hard to come by due to privacy considerations, the vision of this work is to embrace the use of data generated by large-scale socially realistic microsimulations. Informed by both real data and leveraging social and behavioral theories, massive spatially explicit microsimulations may allow us to simulate entire megacities at the person level. The simulated worlds, which do not capture any identifiable personal information, allow us to perform “in silico” experiments using the simulated world as a sandbox in which we have perfect information and perfect control without jeopardizing the privacy of any actual individual. In silico experiments have become commonplace in other scientific domains such as chemistry and biology, permitting experiments that foster the understanding of concepts without any harm to individuals. This work describes challenges and opportunities for leveraging massive and realistic simulated alternate worlds for in silico human mobility data science.

Feature selection using metaheuristics made easy: Open source MAFESE library in Python

Artificial intelligence (AI) often relies on feature selection (FS) to recognize and highlight the most relevant and major features in a dataset. The procedure of training and optimizing AI systems with key data points is decisive for its development and efficacy. To address this challenge, the present study introduces MAFESE, an open-source Python library that employs metaheuristic algorithms for selecting the optimal set of attributes, particularly when dealing with complex and high-dimensional data. MAFESE encompasses a wide range of feature selection techniques, including unsupervised-based, filter-based, embedded-based, and wrapper-based methods. Notably, within the wrapper-based category, MAFESE offers users access to over 200 metaheuristic algorithms, empowering them to choose the most suitable algorithm based on their specific datasets and requirements. Additionally, MAFESE incorporates built-in evaluation metrics that enable efficient comparison among various algorithms. Open-source design of MAFESE encourages cooperation within the data science community, allowing for continuous upgrades. This collaborative environment promotes the sharing of ideas, proposals, and changes, resulting in a stronger and more adaptive feature selection framework. MAFESE distinguishes itself with an easy-to-use Python interface that follows object-oriented programming concepts. It supports both experienced researchers and practitioners. MAFESE offers many resources, including documentation, examples, and test cases, for a smooth user onboarding experience. The modular architecture enables users to enhance features and interface with other tools, such as scikit-learn. MAFESE can be a valuable tool for identifying meaningful features in complicated, high-dimensional datasets, making it a significant addition to feature selection. MAFESE utilizes metaheuristic algorithms to help users tackle complex feature selection problems more efficiently and accurately. Through openness and teamwork, MAFESE may become a comprehensive resource that responds to the evolving demands of the data science community. The source code and materials of MAFESE are publicly available in the official GitHub repository at https://github.com/thieu1995/mafese.

Do inclusive incentive systems encourage prosocial or competitive behavior in online communities?

Platform designers create and implement incentive systems to encourage users to contribute content to online communities. This article examines the effect of a multidimensional incentive hierarchy in motivating users to engage in competitive and prosocial activities. Utilizing an external change observed in the data science community, Kaggle, and applying a quasi-experimental design, we compared users’ engagement levels before and after introducing a multidimensional incentive hierarchy. We found that implementing a multidimensional incentive system directed users from submitting answers to Kaggle competitions to participating in Kaggle’s online forum discussions. However, our additional analyses suggest that the most and the least motivated users may be less likely to be impacted by such incentives.

Advancements in Quantum Computing—Viewpoint: Building Adoption and Competency in Industry

The narrative around quantum computing is evolving quickly. First reports of quantum computers able to solve certain scientific problems on-par with the precision of High-Performance Computers are persuading end-users and industry leaders to shift from passive observation to active exploration. Insights are provided here to enable organization and technology leaders in the database and data science community an entry into the field of quantum computing. This article provides an introduction of key concepts, insights into the rapid advancement of the state-of-the-art, an overview of how companies in Germany are approaching the development of competency and adoption of quantum computing, including an overview of European activities and learning materials in the German language.

Opening a conversation on responsible environmental data science in the age of large language models

Abstract The general public and scientific community alike are abuzz over the release of ChatGPT and GPT-4. Among many concerns being raised about the emergence and widespread use of tools based on large language models (LLMs) is the potential for them to propagate biases and inequities. We hope to open a conversation within the environmental data science community to encourage the circumspect and responsible use of LLMs. Here, we pose a series of questions aimed at fostering discussion and initiating a larger dialogue. To improve literacy on these tools, we provide background information on the LLMs that underpin tools like ChatGPT. We identify key areas in research and teaching in environmental data science where these tools may be applied, and discuss limitations to their use and points of concern. We also discuss ethical considerations surrounding the use of LLMs to ensure that as environmental data scientists, researchers, and instructors, we can make well-considered and informed choices about engagement with these tools. Our goal is to spark forward-looking discussion and research on how as a community we can responsibly integrate generative AI technologies into our work.

A Synthetic Time-Series Generation Using a Variational Recurrent Autoencoder with an Attention Mechanism in an Industrial Control System.

Data scarcity is a significant obstacle for modern data science and artificial intelligence research communities. The fact that abundant data are a key element of a powerful prediction model is well known through various past studies. However, industrial control systems (ICS) are operated in a closed environment due to security and privacy issues, so collected data are generally not disclosed. In this environment, synthetic data generation can be a good alternative. However, ICS datasets have time-series characteristics and include features with short- and long-term temporal dependencies. In this paper, we propose the attention-based variational recurrent autoencoder (AVRAE) for generating time-series ICS data. We first extend the evidence lower bound of the variational inference to time-series data. Then, a recurrent neural-network-based autoencoder is designed to take this as the objective. AVRAE employs the attention mechanism to effectively learn the long-term and short-term temporal dependencies ICS data implies. Finally, we present an algorithm for generating synthetic ICS time-series data using learned AVRAE. In a comprehensive evaluation using the ICS dataset HAI and various performance indicators, AVRAE successfully generated visually and statistically plausible synthetic ICS data.

Performance investigation of two-stage detection techniques using traffic light detection dataset

<span>Using a camera to monitor an object or a group of objects over time is the process of object detection. It can be used for a variety of things, including security and surveillance, video communication, traffic light detection (TLD), object detection from compressed video in public places. In recent times, object tracking has become a popular topic in computer science particularly, the data science community, thanks to the usage of deep learning (DL) in artificial intelligence (AI). DL which convolutional neural network (CNN) as one of its techniques usually used two-stage detection methods in TLD. Despite all successes recorded in TLD through the use of two-stage detection methods, there is no study that has analyzed these methods in experimental research, studying the strength and witnesses by the researchers. Based on the needs this study analyses the applications of DL techniques in TLD. We implemented object detection for TLD using 5 two-stage detection methods with the traffic light dataset using a Jupyter notebook and the sklearn libraries. We present the achievements of two-stage detection methods in TLD, going by standard performance metrics used, FASTER-CNN was the best in detection accuracy, F1-score, precision and recall with 0.89, 0.93, 0.83 and 0.90 respectively.</span>

Fraud Detection System Employing Machine Learning Techniques for Credit Card Transactions

This research paper presents a comprehensive demonstration of the Microsoft Fabric data science workflow, illustrating its effectiveness through an end-to-end example centered on the development of an advanced fraud detection system. The primary objective of this study is to construct a robust and efficient fraud detection mechanism by leveraging the capabilities of machine learning algorithms trained on historical data encompassing instances of fraudulent activities. The overarching aim is to discern intricate patterns inherent in fraudulent events, thus empowering the system to swiftly and accurately identify and flag such activities in case of their recurrence. This paper expounds upon a meticulously designed workflow that encompasses a series of pivotal steps, including the installation of custom libraries tailored to the task, meticulous data loading and preprocessing, a comprehensive exploratory data analysis phase aimed at extracting meaningful insights, the intricate process of training a machine learning model using Scikit-Learn and Flow, the critical step of selecting and registering the most performant model, and, finally, the seamless deployment of the trained model for real-time scoring and prediction.The core of the presented workflow centers on the concept of a lake house, wherein the data is sourced from a public blob and subsequently stored for comprehensive analysis. This architectural paradigm underscores the significance of unified data storage, offering a coherent platform for seamless integration and manipulation. The research paper emphasizes that this approach not only elevates the efficiency of data handling but also lays the foundation for consistent and structured analysis, ultimately enhancing the accuracy and applicability of the subsequent machine learning stages.By tackling the complexities of a real-world scenario involving fraud detection, this research paper underscores the versatility and adaptability of the Microsoft Fabric framework. While the primary focus is on the development of a robust fraud detection system, the significance of this approach reverberates across diverse data science endeavors. The intricate process of custom library installation, data ingestion, preprocessing, and model deployment provides an invaluable resource for practitioners navigating the multifaceted landscape of data science workflows within the Microsoft Fabric ecosystem.In a world characterized by an exponential surge in data generation and a corresponding demand for actionable insights, the presented paper serves as more than just a solution to the specific problem of fraud detection. It morphs into a foundational template, empowering data scientists, researchers, and industry professionals with a structured approach to harnessing the potential of complex datasets. The step-by-step elucidation of the workflow equips practitioners with a tangible guide, offering insights into handling intricacies that often accompany large-scale data analysis. Through its detailed exposition, the paper bridges the gap between theory and practice, allowing readers to not only comprehend the theoretical underpinnings of the Microsoft Fabric framework but also practically implement and adapt its methodologies to suit a myriad of data science challenges. In conclusion, this research paper transcends the confines of a traditional study on fraud detection by encompassing a holistic exploration of the Microsoft Fabric data science workflow. While effectively addressing the practical challenge of developing an advanced fraud detection system, it simultaneously contributes a substantive and versatile resource for the data science community at large. As organizations across domains seek to extract maximum value from their data assets, the workflow detailed in this paper offers a beacon of guidance, ensuring that the journey from data to insights is traversed with precision and efficacy.

Distance-Based Analysis of Early Fire Indicators on a New Indoor Laboratory Dataset with Distributed Multi-Sensor Nodes

This work analyzes a new indoor laboratory dataset looking at early fire indicators in controlled and realistic experiments representing different incipient fire scenarios. The experiments were performed within the constraints of an indoor laboratory setting using multiple distributed sensor nodes in different room positions. Each sensor node collected data of particulate matter (PM), volatile organic compounds (VOCs), carbon monoxide (CO), carbon dioxide (CO2), hydrogen (H2), ultraviolet radiation (UV), air temperature, and humidity in terms of a multivariate time series. These data hold immense value for researchers within the machine learning and data science communities who are keen to explore innovative and advanced statistical and machine learning techniques. They serve as a valuable resource for the development of early fire detection systems. The analysis of the collected data was carried out depending on the Manhattan distance between the fire source and the sensor node. We found that especially larger particles (>0.5 μm) and VOCs show a significant dependency with respect to the intensity as a function of the Manhattan distance to the source. Moreover, we observed differences in the propagation behavior of VOCs, PM, and CO, which are particularly relevant in incipient fire scenarios due to the presence of strand propagation effects.

Local-and-Nonlocal Spectral Prior Regularized Tensor Recovery for Cauchy Noise Removal

Cauchy noise removal in multidimension visual data is a fundamental problem in image processing and data science communities. Recently, this problem can be effectively solved by resorting to an appropriate tensor recovery approach and has attracted increasing attention. However, it is not extensively reported whether tensor recovery approaches with multiple priors can achieve even better denosing performance. To address this issue, we propose a novel tensor recovery approach in this paper based on the maximum a posteriori estimation (MAP). Our approach is a nonconvex optimization model that includes a composite regularization term and a data fidelity term. The composite regularization term incorporates nonlocal self-similarity in the spatial and gradient domains of multidimensional visual data, preserving both local image details and non-local structures. The data fidelity term is obtained by computing the negative log-likelihood of Cauchy noise distribution. To solve this proposed tensor recovery model, we develop an effective denoising algorithm with a global convergence guarantee via the well-known nonconvex alternating direction method of multipliers (ADMM). Finally, we conduct extensive numerical experiments to demonstrate that our proposed recovery approach can achieve superior denoising performance, both quantitatively and qualitatively, compared to existing common methods.

Rehabilitating Homeless: Dataset and Key Insights

This paper presents a large anonymized dataset of homelessness alongside insights into the data-driven rehabilitation of homeless people. The dataset was gathered by a large non-profit organization working on rehabilitating the homeless for twenty years. This is the first dataset that we know of that contains rich information on thousands of homeless individuals seeking rehabilitation. We show how data analysis can help to make the rehabilitation of homeless people more effective and successful. Thus, we hope this paper alerts the data science community to the problem of homelessness.

Large language model applications for evaluation: Opportunities and ethical implications

AbstractLarge language models (LLMs) are a type of generative artificial intelligence (AI) designed to produce text‐based content. LLMs use deep learning techniques and massively large data sets to understand, summarize, generate, and predict new text. LLMs caught the public eye in early 2023 when ChatGPT (the first consumer facing LLM) was released. LLM technologies are driven by recent advances in deep‐learning AI techniques, where language models are trained on extremely large text data from the internet and then re‐used for downstream tasks with limited fine‐tuning required. They offer exciting opportunities for evaluators to automate and accelerate time‐consuming tasks involving text analytics and text generation. We estimate that over two‐thirds of evaluation tasks will be affected by LLMs in the next 5 years. Use‐case examples include summarizing text data, extracting key information from text, analyzing and classifying text content, writing text, and translation. Despite the advances, the technologies pose significant challenges and risks. Because LLM technologies are generally trained on text from the internet, they tend to perpetuate biases (racism, sexism, ethnocentrism, and more) and exclusion of non‐majority languages. Current tools like ChatGPT have not been specifically developed for monitoring, evaluation, research, and learning (MERL) purposes, possibly limiting their accuracy and usefulness for evaluation. In addition, technical limitations and challenges with bias can lead to real world harm. To overcome these technical challenges and ethical risks, the evaluation community will need to work collaboratively with the data science community to co‐develop tools and processes and to ensure the application of quality and ethical standards.

Salary Prediction System using Machine Learning

Salary prediction is a popular problem among the Data Science community for complete beginners. Through this regression analysis, we found a perfect linear relationship between the salary and the job experience of the people. It means more job experience results in a higher salary. This research paper presents a software Salary Prediction System that aims to address the challenge of accurately estimating salaries in the software industry. The system provides valuable insights for employers and employees in the face of increasing demand for skilled software professionals and the need for fair compensation.

Quantifying Gender Disparity in Pre-Modern English Literature using Natural Language Processing

Research has continued to shed light on the extent and significance of gender disparity in social, cultural and economic spheres. More recently, computational tools from the data science and Natural Language Processing (NLP) communities have been proposed for measuring such disparity at scale using empirically rigorous methodologies. In this article, we contribute to this line of research by studying gender disparity in 2,443 copyright-expired literary texts published in the pre-modern period, defined in this work as the period ranging from the beginning of the nineteenth through the early twentieth century. Using a replicable data science methodology relying on publicly available and established NLP components, we extract three different gendered character prevalence measures within these texts. We use an extensive set of statistical tests to robustly demonstrate a significant disparity between the prevalence of female characters and male characters in pre-modern literature. We also show that the proportion of female characters in literary texts significantly increases in female-authored texts compared to the same proportion in male-authored texts. However, regression-based analysis shows that, over the 120 year period covered by the corpus, female character prevalence does not change significantly over time, and remains below the parity level of 50%, regardless of the gender of the author. Qualitative analyses further show that descriptions associated with female characters across the corpus are markedly different (and stereotypical) from the descriptions associated with male characters.

Promoting Data Science in Disaster Risk Reduction: Glimpses from the Global South

There are coordinated attempts to use Data Science in disaster risk reduction (DRR) across the world. Different special interest groups are formed within the data science community to formulate different action plans and strategies to use data science in the DRR. Some examples of community-driven initiatives are the CODATA Task Group FAIR Data for Disaster Risk Research (FAIR-DRR), CODATA Task Group Linked Open Data for Global Disaster Risk Research (LODGD), and ISC Integrated Research on Disaster Risk (IRDR). This chapter highlights the data science approaches, models, frameworks, and stakeholders mapping while dealing with the data collection, analysis, and reporting for strengthening data-driven decision-making at different levels. The chapter also explores the FAIR Data Principles that aim at achieving the findable, accessible, interoperable, and reusable data resources at the institutional and national levels. FAIR Data ensures various data users increasingly engage with data related to disasters. We also discuss some case studies and success stories from the Global South, which may inspire other countries in data-driven decision-making while dealing with natural disasters. The much-discussed “Data, Information, Knowledge and Action” (DIKA) model is highlighted in this chapter. For example, the recently launched “Data-Knowledge-Action for Urban Systems” podcast series of CODATA and CEPT Research and Development Foundation (CRDF) in Ahmedabad, talks about how the DIKA model helps in creating intelligent systems for DRR using scientific and quantitative data. On a podcast in related Series, the DRR practitioners and data science researchers discussed the use of big data to predict natural disasters. This chapter highlights notable examples of how big data helped predict natural disasters across the world, particularly in the Global South.

Attending to the Cultures of Data Science Work

This essay reflects on the shifting attention to the “social” and the “cultural” in data science communities. While recently the “social” and the “cultural” have been prioritized in data science discourse, social and cultural concerns that get raised in data science are almost always outwardly focused – applying to the communities that data scientists seek to support more so than more computationally-focused data science communities. I argue that data science communities have a responsibility to attend not only to the cultures that orient the work of domain communities, but also to the cultures that orient their own work. I describe how ethnographic frameworks such as thick description can be enlisted to encourage more reflexive data science work, and I conclude with recommendations for documenting the cultural provenance of data policy and infrastructure.

How Can IJDS Authors, Reviewers, and Editors Use (and Misuse) Generative AI?

How Can <i>IJDS</i> Authors, Reviewers, and Editors Use (and Misuse) Generative AI?

A climate index collection based on model data

Abstract Machine learning (ML) and in particular deep learning (DL) methods push state-of-the-art solutions for many hard problems, for example, image classification, speech recognition, or time series forecasting. In the domain of climate science, ML and DL are known to be effective for identifying causally linked modes of climate variability as key to understand the climate system and to improve the predictive skills of forecast systems. To attribute climate events in a data-driven way, we need sufficient training data, which is often limited for real-world measurements. The data science community provides standard data sets for many applications. As a new data set, we introduce a consistent and comprehensive collection of climate indices typically used to describe Earth System dynamics. Therefore, we use 1000-year control simulations from Earth System Models. The data set is provided as an open-source framework that can be extended and customized to individual needs. It allows users to develop new ML methodologies and to compare results to existing methods and models as benchmark. For example, we use the data set to predict rainfall in the African Sahel region and El Niño Southern Oscillation with various ML models. Our aim is to build a bridge between the data science community and researchers and practitioners from the domain of climate science to jointly improve our understanding of the climate system.