Potential Data Sources Research Articles

A Comprehensive Study of OpenTelemetry Collector: Architecture, Use Cases, and Performance

As microservices and distributed systems gain popularity, observability encompassing metrics, logs, and traces has emerged as a pivotal concern in modern DevOps environments. OpenTelemetry (often abbreviated OTel), an open- source observability framework, seeks to standardize the way telemetry data is collected, processed, and exported. Within OTel’s ecosystem, the OpenTelemetry Collector (OTel Collector) serves as a flexible, pluggable data pipeline, bridging instrumentation signals from multiple sources to various backends. This collector consolidates the ingestion of metrics, logs, and traces in a single, vendor-neutral solution. This paper provides a comprehensive study of the OpenTelemetry Collector, its core architecture, deployment patterns, typical use cases, and performance considerations. We begin by detailing how the Collector interacts with instrumentation libraries, potential data sources, and various backend analysis systems. We then highlight real-world usage scenarios, from sidecar deployments to central aggregator setups, and advanced configurations, including load balancing or pipeline concurrency. Along the way, we examine anti-patterns such as unbounded concurrency, over-collecting data, or ignoring security aspects for sensitive traces. Benchmark results and best practices gleaned from real deployments prior to August 2022 offer guidance on scaling the Collector effectively, ensuring minimal overhead while retaining robust data. Ultimately, this paper aims to equip DevOps engineers, SREs, and software architects with actionable insights to successfully integrate OpenTelemetry Collector as a unified approach to metrics, logs, and traces in cloud-native systems. Keywords OpenTelemetry, Observability, Collector, Tracing, Metrics, Logs, Distributed Systems, DevOps, Performance, Kubernetes

A novel data source for human-caused wildfires in China: extracting information from judgment documents

Information on the spatiotemporal distribution of human-caused wildfires on a national scale is crucial for developing susceptibility map and facilitating decision-making in fire risk management. In China, it is difficult to access a national dataset of wildfires based on ground records at a fine spatiotemporal scale for understanding fire regimes. This study explores the use of judgment documents as a potential alternative long-term key data source, considering human-caused wildfire events as criminal offense in certain cases. Judgment documents are readily accessible, regularly updated, and provide a diverse range of reliable information. To extract information on human-caused wildfires, we develop a rule-based approach for judgment documents. The results indicate the feasibility of the method, allowing for the effective extraction of specific information of event such as the start date and time, location, burn area, and ignition cause. A comparison reveals limited consistency in the spatiotemporal distributions between the novel data and conventional datasets. Notably, judgment documents provide richer attribute information and identify additional human-caused wildfires challenging to capture using conventional datasets. Despite inherent limitations, information sourced from judgment documents proves valuable data for characterizing national patterns of wildfire occurrence at a fine scale.

The Uncharted Waters of International Trade

Abstract The field of international trade has undergone significant theoretical and empirical advancements over the last twenty-five years. A key breakthrough has been the emergence of firm-level approaches to studying exporting, importing, and global value chains. The field has also experienced a quantitative revolution, driven by medium-scale models that rapidly assess the implications of trade cost shocks on real income. Additionally, a branch of the empirical literature has unshackled itself from the discipline of theoretical frameworks and from traditional data sources. Yet, several underexplored areas, or ‘uncharted waters,’ remain in international trade research. I outline new potential areas for theoretical research, including incorporating oligopolistic (strategic) behavior into core models, and fostering greater cross-disciplinary collaboration with other fields in economics and social sciences, such as behavioral economics or political science. I also discuss potential uncharted waters for empirical trade economists, while identifying potential new sources of data and ways in which official trade statistics could be improved. Finally, I explore how big data and artificial intelligence could reshape the design of international trade policy in coming years.

Pharmacoequity measurement framework: A tool to reduce health disparities.

Pharmacoequity is a health system and policy goal of ensuring equitable access to high-quality medications for all individuals, regardless of factors such as race, ethnicity, socioeconomic status, or resource availability to reduce health disparities. Although measurement frameworks have been widely used in health equity contexts, a focused framework for pharmacoequity remains a critical gap. In this article, we introduce a novel pharmacoequity measurement framework anchored in the patient medication-use journey. The framework includes the following domains: (1) access to health care services, (2) prescription generation, (3) primary medication nonadherence, (4) secondary medication nonadherence, and (5) medication monitoring. For each domain, we provide examples of outcome measures and potential data sources that can be used for evaluation. We also outline an implementation workflow of the pharmacoequity measurement framework that population health stakeholders can use across various settings (eg, health systems, health plans). The framework provides a structured approach to identify existing gaps in the path toward achieving pharmacoequity and lay the foundation for targeted interventions. Additionally, it enables ongoing monitoring of progress toward achieving pharmacoequity while identifying interventions that are effective, scalable, and sustainable.

A Systematic Review of Features Forecasting Patient Arrival Numbers.

Adequate nurse staffing is crucial for quality healthcare, necessitating accurate predictions of patient arrival rates. These forecasts can be determined using supervised machine learning methods. Optimization of machine learning methods is largely about minimizing the prediction error. Existing models primarily utilize data such as historical patient visits, seasonal trends, holidays, and calendars. However, it is unclear what other features reduce the prediction error. Our systematic literature review identifies studies that use supervised machine learning to predict patient arrival numbers using nontemporal features, which are features not based on time or dates. We scrutinized 26 284 studies, eventually focusing on 27 relevant ones. These studies highlight three main feature groups: weather data, internet search and usage data, and data on (social) interaction of groups. Internet data and social interaction data appear particularly promising, with some studies reporting reduced errors by up to 33%. Although weather data are frequently used, its utility is less clear. Other potential data sources, including smartphone and social media data, remain largely unexplored. One reason for this might be potential data privacy challenges. In summary, although patient arrival prediction has become more important in recent years, there are still many questions and opportunities for future research on the features used in this area.

Citizen Science for Invertebrate Biodiversity: Mobilizing Spider Occurrence Data through Facebook in India

India accommodates a huge diversity of plant and animal life across a variety of biomes. However, the degree of research, funding, and attention is asymmetric, largely focused on its charismatic vertebrates. Invertebrates, despite their megadiversity, are generally overlooked with some exceptions (for example, lepidoterans). One species-rich group, spiders, exemplifies this knowledge gap. More than 1,800 species from 63 families have been reported in the country (Mondal et al. 2020), even though the true number may be much higher, in part, owing to the need for taxonomic revisions. Several spider systematics and biogeographic studies have pointed out that spatial distribution, seasonality, and natural history data are lacking from India. The lack of foundational biodiversity information has led to poor opportunities to share knowledge between researchers and community scientists. Mining occurrences through scientific publications or databases (in some cases behind paywalls) may require specialized training. This restricted access inhibits more people from using the abundance of information. The prevalence of social media posts showcasing photographs of species in biodiversity contexts has experienced a significant increase, with a multitude of users sharing wildlife photographs on social media sites such as Instagram®, Facebook®, and Flickr®. These observations on social media are a potential source of primary biodiversity data when curated and validated (Barman and Barve 2022, Barman et al. 2022a). Such data can enhance biodiversity monitoring by expanding spatial and temporal coverage and involving a global network of citizen scientists in conservation research (Barve et al. 2023, Roy 2022, Kulkarni 2023). The India Biodiversity Portal (IBP, Vattakaven et al. 2016) has been cataloguing species diversity and spider data through citizen science. Meanwhile, major social media sites report many more species sightings than citizen science platforms. SpiderIndia, a popular Facebook group, has collected over 20,000 observations from 8,500 spider enthusiasts. However, these data remain unorganised and inaccessible to academic researchers and the public. In order to tackle these problems, we implemented a methodical process to enhance the occurrence data on spiders from well-known social media platforms, such as the SpiderIndia project. The procedure involved retrieving the relevant data, and ingesting it through a custom pipeline to parse and extract, scientific names, spatial and temporal data to generate occurrence records on the IBP. The data was then presented on a curation interface, enabling taxonomic experts to verify the records, and finally publishing the verified records on the Global Biodiversity Information Facility (GBIF) (Global Biodiversity Information Facility 2022, Barman et al. 2022b) to improve the records of spider species occurrence in India. This project showcases the capacity of citizen science via social media to involve citizen scientists in generating extensive datasets that make a significant contribution to scientific knowledge and improve our comprehension of invertebrate biodiversity. The final dataset encompasses over 15,000 observations, providing valuable insights into spider diversity and distribution across India (Fig. 1). This data is publicly available on GBIF, facilitating further research on Indian spider populations.

Too rare to dare? Leveraging household surveys to boost research on climate migration

Abstract Nationally representative household surveys are a potential data source that could shed light on the climate–migration nexus. However, they are rarely designed specifically to measure or study migration and often lack the necessary features to identify connections with climate change. This paper offers a critical reflection on current challenges faced by multi-topic household surveys in responding to these needs while also highlighting the many opportunities embedded in their use. Using the Living Standards Measurement Study household survey programme of the World Bank as an example, this paper proposes a methodological agenda and practical guidance to address data gaps and advance research on climate migration.

Applications of Google Trends as a Data Source for Statistical Models

As technology advances, there is a growing number of potential data sources that can provide an alternative to traditional surveys. An example of this is the real time search popularity data made available through Google Trends. This type of data makes it possible to study public opinion, behaviour and attitudes in society or forecast economic phenomena. A definite advantage of using search popularity data is the immediate availability and low cost of obtaining such data. Also of significance is the fact that the Google Trends tool allows for direct research into the behaviour of Internet users, and not just their declarations as in the case of a survey. This can make a difference if respondents consider one of the answers to be more morally correct. Nevertheless, the use of Google Trends requires selecting correct search topics and terms to be included in the study and an awareness of the fact that the research sample is limited to Google search engine users. The paper will present the advantages and disadvantages of Google Trends and review its usefulness as a data source especially in times of higher market volatility.

Comprehensive evaluation of satellite-based precipitation products at hourly scale in Beijing

With the increase of short-duration rainstorms, accurate precipitation data with higher temporal resolution, such as hours or even minutes, has been required for urban hydrological modeling and forecasting. As a potential data source, hourly or half-hour satellite-based precipitation products (SPPs) perform differently in different metropolitan regions, so evaluation is necessary before their application. Based on the observations of 20 meteorological stations from 2015 to 2018 over a typical metropolitan region, Beijing, a comprehensive evaluation of four widely used SPPs at hourly scale was conducted. Results showed that IMERG and GSMaP had the best performance, followed by CMORPH, while PERSIANN performed worst at hourly scale; it was suggested to use data series composed of wet hours for evaluation at hourly scale with more consistent spatial distribution characteristics and reliable results; IMERG, GSMaP and CMORPH performed better during 9:00–12:00 for accuracy and 5:00–9:00 and 20:00–21:00 for detection capability while PERSIANN didn't show obvious diurnal variation. In addition, the short-duration extreme precipitation in Beijing was defined as an event with an hourly precipitation >20 mm. Overall, hourly or half-hour SPPs still need a lot of progress and our study would provide some significant references for developers and potential users.

Understanding the Use of Mobility Data in Disasters: Exploratory Qualitative Study of COVID-19 User Feedback.

Human mobility data have been used as a potential novel data source to guide policies and response planning during the COVID-19 global pandemic. The COVID-19 Mobility Data Network (CMDN) facilitated the use of human mobility data around the world. Both researchers and policy makers assumed that mobility data would provide insights to help policy makers and response planners. However, evidence that human mobility data were operationally useful and provided added value for public health response planners remains largely unknown. This exploratory study focuses on advancing the understanding of the use of human mobility data during the early phase of the COVID-19 pandemic. The study explored how researchers and practitioners around the world used these data in response planning and policy making, focusing on processing data and human factors enabling or hindering use of the data. Our project was based on phenomenology and used an inductive approach to thematic analysis. Transcripts were open-coded to create the codebook that was then applied by 2 team members who blind-coded all transcripts. Consensus coding was used for coding discrepancies. Interviews were conducted with 45 individuals during the early period of the COVID-19 pandemic. Although some teams used mobility data for response planning, few were able to describe their uses in policy making, and there were no standardized ways that teams used mobility data. Mobility data played a larger role in providing situational awareness for government partners, helping to understand where people were moving in relation to the spread of COVID-19 variants and reactions to stay-at-home orders. Interviewees who felt they were more successful using mobility data often cited an individual who was able to answer general questions about mobility data; provide interactive feedback on results; and enable a 2-way communication exchange about data, meaning, value, and potential use. Human mobility data were used as a novel data source in the COVID-19 pandemic by a network of academic researchers and practitioners using privacy-preserving and anonymized mobility data. This study reflects the processes in analyzing and communicating human mobility data, as well as how these data were used in response planning and how the data were intended for use in policy making. The study reveals several valuable use cases. Ultimately, the role of a data translator was crucial in understanding the complexities of this novel data source. With this role, teams were able to adapt workflows, visualizations, and reports to align with end users and decision makers while communicating this information meaningfully to address the goals of responders and policy makers.

An evaluation of the All of Us Research Program database to examine cumulative stress.

To evaluate the NIH All of Us Research Program database as a potential data source for studying allostatic load and stress among adults in the United States (US). We evaluated the All of Us database to determine sample size significance for original-10 allostatic load biomarkers, Allostatic Load Index-5 (ALI-5), Allostatic Load Five, and Cohen's Perceived Stress Scale (PSS). We conducted a priori, post hoc, and sensitivity power analyses to determine sample sizes for conducting null hypothesis significance tests. The maximum number of responses available for each measure is 21 participants for the original-10 allostatic load biomarkers, 150 for the ALI-5, 22476 for Allostatic Load Five, and n = 90583 for the PSS. The NIH All of Us Research Program is well-suited for studying allostatic load using the Allostatic Load Five and psychological stress using PSS. Improving biomarker data collection in All of Us will facilitate more nuanced examinations of allostatic load among US adults.

Automated analysis and assignment of maintenance work orders using natural language processing

Maintenance management forms a core component of building facility management, which is particularly important for specialized and complex healthcare facilities. However, existing maintenance management tends to rely on manual processing, which is time-consuming and human error-prone. Recorded maintenance work orders (MWOs) are recognized as good potential data source to support maintenance activities. In this paper, the authors propose a comprehensive framework utilizing natural language processing (NLP) techniques to automate the interrogation of textual data of MWOs that assist in developing maintenance material management and preventive maintenance strategies and transforming maintenance worker assignment from a labor-intensive process to a more automated one. A set of historical MWOs from a hospital was used for model development, and our model achieved an accuracy of 0.83 on worker assignment. This study addresses the difficulty of utilizing work order information due to the unstructured nature of textual data and contributes to better maintenance management practices in buildings.

Tell your story: Metrics of success for academic data science collaboration and consulting programs

Measuring success plays a central role in justifying and advocating for a statistical or data science consulting or collaboration program (SDSP) within an academic institution. We present several specific metrics to report to targeted audiences to tell the story for success of a robust and sustainable program. While gathering such metrics includes challenges, we discuss potential data sources and possible practices for SDSPs to inform their own approaches. Emphasizing essential metrics for reporting, we also share the metric gathering and reporting practices of two programs in greater detail. New or existing SDSPs should evaluate their local environments and tailor their practice to gathering, analysing and reporting success metrics accordingly. This approach provides a strong foundation to use success metrics to tell compelling stories about the SDSP and enhance program sustainability. The area of success metrics provides ample opportunity for future research projects that leverage qualitative methods and consider mechanisms for adapting to the changing landscape of data science.

The Retrieval of Ground NDVI (Normalized Difference Vegetation Index) Data Consistent with Remote-Sensing Observations

The Normalized Difference Vegetation Index (NDVI) is widely used for monitoring vegetation status, as accurate and reliable NDVI time series are crucial for understanding the relationship between environmental conditions, vegetation health, and productivity. Ground digital cameras have been recognized as important potential data sources for validating remote-sensing NDVI products. However, differences in the spectral characteristics and imaging methods between sensors onboard satellites and ground digital cameras hinder direct consistency analyses, thereby limiting the quantitative application of camera-based observations. To address this limitation and meet the needs of vegetation monitoring research and remote-sensing NDVI validation, this study implements a novel NDVI camera. The proposed camera incorporates narrowband dual-pass filters designed to precisely separate red and near-infrared (NIR) spectral bands, which are aligned with the configuration of sensors onboard satellites. Through software-controlled imaging parameters, the camera captures the real radiance of vegetation reflection, ensuring the acquisition of accurate NDVI values while preserving the evolving trends of the vegetation status. The performance of this NDVI camera was evaluated using a hyperspectral spectrometer in the Hulunbuir Grassland over a period of 93 days. The results demonstrate distinct seasonal characteristics in the camera-derived NDVI time series using the Green Chromatic Coordinate (GCC) index. Moreover, in comparison to the GCC index, the camera’s NDVI values exhibit greater consistency with those obtained from the hyperspectral spectrometer, with a mean deviation of 0.04, and a relative root mean square error of 9.68%. This indicates that the narrowband NDVI, compared to traditional color indices like the GCC index, has a stronger ability to accurately capture vegetation changes. Cross-validation using the NDVI results from the camera and the PlanetScope satellite further confirms the potential of the camera-derived NDVI data for consistency analyses with remote sensing-based NDVI products, thus highlighting the potential of camera observations for quantitative applications The research findings emphasize that the novel NDVI camera, based on a narrowband spectral design, not only enables the acquisition of real vegetation index (VI) values but also facilitates the direct validation of vegetation remote-sensing NDVI products.

Background and Rationale - CDC Guidance for Communities Assessing, Investigating, and Responding to Suicide Clusters, United States, 2024.

To assist community leaders in public health, mental health, education, and other fields with developing a community response plan for suicide clusters or for situations that might develop into suicide clusters, in 1988, CDC published Recommendations for a Community Plan for the Prevention and Containment of Suicide Clusters (MMWR Suppl 1988;37[No. Suppl 6]:1-12). Since that time, the reporting and investigation of suicide cluster events has increased, and more is known about cluster risk factors, assessment, and identification. This supplement updates and expands CDC guidance for assessing, investigating, and responding to suicide clusters based on current science and public health practice. This report is the first of three in the MMWR supplement that describes an overview of suicide clusters, information about the other reports in this supplement, methods used to develop the supplement guidance, and the intended use of the supplement reports. The second report, CDC Guidance for Community Assessment and Investigation of Suspected Suicide Clusters - United States 2024, describes the potential methods, data sources and analysis that communities can use to identify and confirm suspected suicide clusters, and better understand the relevant issues. The final report, CDC Guidance for Community Response to Suicide Clusters - United States, 2024, describes how local public health and community leaders can develop a response plan for suicide clusters. The guidance in this supplement is intended as a conceptual framework that can be used by public health practitioners and state and local health departments to develop response plans for assessing and investigating suspected clusters that are tailored to the needs, resources, and cultural characteristics of their communities.

CDC Guidance for Community Response to Suicide Clusters, United States, 2024.

This is the third of three reports in the MMWR supplement that updates and expands CDC's guidance for assessing, investigating, and responding to suicide clusters based on current science and public health practice. The first report, Background and Rationale - CDC Guidance for Communities Assessing, Investigating, and Responding to Suicide Clusters, United States, 2024, describes an overview of suicide clusters, methods used to develop the supplement guidance, and intended use of the supplement reports. The second report, CDC Guidance for Community Assessment and Investigation of Suspected Suicide Clusters, United States, 2024, describes the potential methods, data sources, and analysis that communities can use to identify and confirm suspected suicide clusters and better understand the relevant issues. This report describes how local public health and community leaders can develop a response plan for suicide clusters. Specifically, the steps for responding to a suicide cluster include preparation, direct response, and action for prevention. These steps are not intended to be explicitly adopted but rather adapted into the local context, culture, capacity, circumstances, and needs for each suicide cluster.

A dataset of global tropical cyclone wind and surface wave measurements from buoy and satellite platforms

There are now a range of potential data sources for wind and surface wave conditions within tropical cyclones. These sources include: in situ buoy data and remote sensing data from satellite altimeters, scatterometers, and radiometers. In addition, data providing estimates of tropical cyclone tracks and wind field parameters are available from best track archives. The present dataset brings together this information in a single archive, providing the available data for each tropical cyclone from each of the data sources in a single file. The data consists of observations in a total of 2927 global tropical cyclones over the period from 1985 to 2017. Global statistics of the observations are provided, along with data on the geographic distribution of tropical cyclones within the database.

Enhanced Heart Disease Prediction Using Machine Learning Techniques

This study leverages sophisticated machine learning methodologies, particularly XGBoost, to analyze cardiovascular diseases through cardiac datasets. The methodology encompasses meticulous data pre-processing, training of the XGBoost algorithm, and its performance evaluation using metrics such as accuracy, precision, and ROC curves. This technique represents a notable progression in the realm of medical research, potentially leading to enhanced diagnostic precision and a deeper comprehension of cardiovascular ailments, thereby improving patient care and treatment modalities in cardiology. Furthermore, the research delves into the utilization of deep learning methodologies for the automated delineation of cardiac structures in MRI and mammography images, aiming to boost diagnostic precision and patient management. [24][3][5][6] In assessing machine learning algorithms' efficacy in diagnosing cardiovascular diseases, this analysis underscores the pivotal role of such algorithms and their possible data inputs. Additionally, it investigates promising directions for future exploration, such as the application of reinforcement learning. A significant aspect of our investigation is the development and deployment of sophisticated deep learning models for segmenting right ventricular images from cardiac MRI scans, aiming at heightened accuracy and dependability in diagnostics. Through the utilization of advanced techniques like Fourier Convolutional Neural Network (FCNN) and improved versions of Vanilla Convolutional Neural Networks (Vanilla-CNN) and Residual Networks (ResNet), we achieved a substantial improvement in accuracy and reliability. This enhancement allows for more precise and quicker identification and diagnosis of cardiovascular diseases, which is of utmost importance in clinical practice. Evaluation of Machine Learning Algorithms: We conducted a comprehensive evaluation of machine learning algorithms in the context of cardiovascular disease diagnosis. This assessment emphasized the fundamental role of machine learning algorithms and their potential data sources. We also explored promising avenues, such as reinforcement learning, for future research. Factors Affecting Predictive Models: We highlighted the critical factors affecting the effectiveness of machine learning-based predictive models. These factors include data heterogeneity, depth, and breadth, as well as the nature of the modeling task, and the choice of algorithms and feature selection methods. Recognizing and addressing these factors are essential for building reliable models.

A new electronic medical record database linked to claims data and discharge abstract data (the RWD database) in Japan: Study design and profile.

This article aims to introduce the Real World Database-a new clinical database in Japan. The Health, Clinic, and Education Information Evaluation Institute and Real World Data Co., Ltd. began developing the Real World Database in 2015. This is an electronic medical record database linked to claims data and discharge abstract data from medical institutions in Japan. The institutions agreed to collect data from 218 medical institutions as of June 2021. In 2019, 82 medical institutions provided data, which showed that 2,184,666 patients received treatment at medical institutions. There were also 334,437 inpatients with at least one hospital stay and 2,011,628 outpatients with at least one visit. More than 200 laboratory test results were available. This database is a potential data source for producing descriptive studies, comparative effectiveness studies, studies of adverse effects, and prediction studies. The Real World Database provides an opportunity and strategy to produce real-world evidence for Japan.

Food and nutrition surveillance actions in Brazil and Portugal: a comparative documentary analysis

Food and nutrition surveillance is fundamental to the formulation, implementation, monitoring, and evaluation of public policies on food and nutrition. The comparative analysis of the evolution of food and nutrition surveillance actions between countries makes it possible to learn about advances, challenges and innovations for the planning of interventions. This study aimed to describe and compare food and nutrition surveillance actions in Brazil and Portugal, countries with different geographical dimensions and socio-economic profiles, but with the common characteristic of having a universal healthcare system. We manually searched for documents describing food and nutrition surveillance actions and for potential data sources on the institutional websites of the Brazilian and Portuguese governments. The recommendations of the READ approach were used (59 documents were identified for Brazil and 29 for Portugal). In Brazil, food and nutrition surveillance actions are included in health policies and in the constraints of conditional cash transfer programs. The Brazilian Households Budget Surveys and health surveys are used simultaneously with the Brazilian National Food and Nutrition Surveillance System (SISVAN). In Portugal, food and nutrition surveillance actions are part of health policies, via the National Program and the Integrated Strategy for the Promotion of Healthy Eating. Among the data sources identified are the Portuguese Food Balance Sheet and household budget, health and food surveys, as well as initiatives to monitor the nutritional status of children and adolescents. In both countries, strategies need to be improved in terms of the regularity of data recording, the harmonization of food consumption indicators and the dissemination of information.