Large-scale Data Analysis Research Articles

Large-Scale Estimation and Analysis of Web Users' Mood from Web Search Query and Mobile Sensor Data.

The ability to estimate the current mood states of web users has considerable potential for realizing user-centric opportune services in pervasive computing. However, it is difficult to determine the data type used for such estimation and collect the ground truth of such mood states. Therefore, we built a model to estimate the mood states from search-query data in an easy-to-collect and non-invasive manner. Then, we built a model to estimate mood states from mobile sensor data as another estimation model and supplemented its output to the ground-truth label of the model estimated from search queries. This novel two-step model building contributed to boosting the performance of estimating the mood states of web users. Our system was also deployed in the commercial stack, and large-scale data analysis with >11 million users was conducted. We proposed a nationwide mood score, which bundles the mood values of users across the country. It shows the daily and weekly rhythm of people's moods and explains the ups and downs of moods during the COVID-19 pandemic, which is inversely synchronized to the number of new COVID-19 cases. It detects big news that simultaneously affects the mood states of many users, even under fine-grained time resolution, such as the order of hours. In addition, we identified a certain class of advertisements that indicated a clear tendency in the mood of the users who clicked such advertisements.

Middle Meningeal Artery Embolization in Adjunction to Surgical Evacuation for Treatment of Subdural Hematomas: A Nationwide Comparison of Outcomes With Isolated Surgical Evacuation.

Surgical evacuation is the standard treatment for chronic subdural hematomas (CSDHs) but is associated with a high risk of recurrence and readmission. Middle meningeal artery embolization (MMAE) is a novel treatment approach which could be performed upfront or in adjunction to surgical evacuation. MMAE studies are limited by small sample sizes. This study aimed to describe and compare outcomes of MMAE in adjunction to surgery with those of surgery alone on a national level. The national Vizient Clinical Database was queried by use of a specific validated set of International Classification of Diseases, Tenth Revision codes (October 2018-June 2022). Patients with the diagnosis of nontraumatic CSDH who received MMAE and surgical drainage in the same hospitalization were identified, and their outcomes were compared with isolated surgical drainage. A total of 606 subjects from 156 institutes and 6340 subjects from 369 institutes were included in the MMAE plus surgery (M&S) and surgery groups, respectively. Average length of stay was significantly longer in the M&S group (9.87 vs 7.53 days; P < .01). There was no significant difference in the in-hospital mortality rate (2.8% vs 2.9%), but the complication rate was significantly higher in the M&S group (8.7% vs 5.5%; P < .01). Complications that were significantly more common in the M&S group included aspiration pneumonia, postoperative sepsis, and anesthesia-related. Mean direct costs were significantly higher in the M&S group (28 834 vs 16 292 US dollars; P < .01). The 30-day readmission rate was significantly lower in the M&S group compared with the surgery group (4.2% vs 8.0%; P < .01). This analysis of large-scale national data indicates that MMAE performed in adjunction to surgery for treatment of CSDH is associated with higher direct costs, higher complication rates, and longer length of stay but lower readmission rates compared with surgical evacuation alone.

Large-scale real-world data analyses of cancer risks among patients with rheumatoid arthritis.

Rheumatoid arthritis (RA) affects 24.5 million people worldwide and has been associated with increased cancer risks. However, the extent to which the observed risks are related to the pathophysiology of rheumatoid arthritis or its treatments is unknown. Leveraging nationwide health insurance claims data with 85.97 million enrollees across 8 years, we identified 92 864 patients without cancers at the time of rheumatoid arthritis diagnoses. We matched 68 415 of these patients with participants without rheumatoid arthritis by sex, race, age and inferred health and economic status and compared their risks of developing all cancer types. By 12 months after the diagnosis of rheumatoid arthritis, rheumatoid arthritis patients were 1.21 (95% confidence interval [CI] [1.14, 1.29]) times more likely to develop any cancer compared with matched enrollees without rheumatoid arthritis. In particular, the risk of developing lymphoma is 2.08 (95% CI [1.67, 2.58]) times higher in the rheumatoid arthritis group, and the risk of developing lung cancer is 1.69 (95% CI [1.32, 2.13]) times higher. We further identified the five most commonly used drugs in treating rheumatoid arthritis, and the log-rank test showed none of them is implicated with a significantly increased cancer risk compared with rheumatoid arthritis patients without that specific drug. Our study suggested that the pathophysiology of rheumatoid arthritis, rather than its treatments, is implicated in the development of subsequent cancers. Our method is extensible to investigating the connections among drugs, diseases and comorbidities at scale.

Performance models of data parallel DAG workflows for large scale data analytics

Directed Acyclic Graph (DAG) workflows are widely used for large-scale data analytics in cluster-based distributed computing systems. The performance model for a DAG on data-parallel frameworks (e.g., MapReduce) is a research challenge because the allocation of preemptable system resources among parallel jobs may dynamically vary during execution. This resource allocation variation during execution makes it difficult to accurately estimate the execution time. In this paper, we tackle this challenge by proposing a new cost model, called Bottleneck Oriented Estimation (BOE), to estimate the allocation of preemptable resources by identifying the bottleneck to accurately predict task execution time. For a DAG workflow, we propose a state-based approach to iteratively use the resource allocation property among stages to estimate the overall execution plan. Furthermore, to handle the skewness of various jobs, we refine the model with the order statistics theory to improve estimation accuracy. Extensive experiments were performed to validate these cost models with HiBench and TPC-H workloads. The BOE model outperforms the state-of-the-art models by a factor of five for task execution time estimation. For the refined skew-aware model, the average prediction error is under 3% when estimating the execution time of 51 hybrid analytics (HiBench) and query (TPC-H) DAG workflows.

A mathematical programming approach for resource allocation of data analysis workflows on heterogeneous clusters

Scientific communities are motivated to schedule their large-scale data analysis workflows in heterogeneous cluster environments because of privacy and financial issues. In such environments containing considerably diverse resources, efficient resource allocation approaches are essential for reaching high performance. Accordingly, this research addresses the scheduling problem of workflows with bag-of-task form to minimize total runtime (makespan). To this aim, we develop a mixed-integer linear programming model (MILP). The proposed model contains binary decision variables determining which tasks should be assigned to which nodes. Also, it contains linear constraints to fulfill the tasks requirements such as memory and scheduling policy. Comparative results show that our approach outperforms related approaches in most cases. As part of the post-optimality analysis, some secondary preferences are imposed on the proposed model to obtain the most preferred optimal solution. We analyze the relaxation of the makespan in the hope of significantly reducing the number of consumed nodes.

Accurate Label-Free Quantification by directLFQ to Compare Unlimited Numbers of Proteomes

Recent advances in mass spectrometry–based proteomics enable the acquisition of increasingly large datasets within relatively short times, which exposes bottlenecks in the bioinformatics pipeline. Although peptide identification is already scalable, most label-free quantification (LFQ) algorithms scale quadratic or cubic with the sample numbers, which may even preclude the analysis of large-scale data. Here we introduce directLFQ, a ratio-based approach for sample normalization and the calculation of protein intensities. It estimates quantities via aligning samples and ion traces by shifting them on top of each other in logarithmic space. Importantly, directLFQ scales linearly with the number of samples, allowing analyses of large studies to finish in minutes instead of days or months. We quantify 10,000 proteomes in 10 min and 100,000 proteomes in less than 2 h, a 1000-fold faster than some implementations of the popular LFQ algorithm MaxLFQ. In-depth characterization of directLFQ reveals excellent normalization properties and benchmark results, comparing favorably to MaxLFQ for both data-dependent acquisition and data-independent acquisition. In addition, directLFQ provides normalized peptide intensity estimates for peptide-level comparisons. It is an important part of an overall quantitative proteomic pipeline that also needs to include high sensitive statistical analysis leading to proteoform resolution. Available as an open-source Python package and a graphical user interface with a one-click installer, it can be used in the AlphaPept ecosystem as well as downstream of most common computational proteomics pipelines.

Accurate drusen segmentation in optical coherence tomography via order-constrained regression of retinal layer heights

Drusen are an important biomarker for age-related macular degeneration (AMD). Their accurate segmentation based on optical coherence tomography (OCT) is therefore relevant to the detection, staging, and treatment of disease. Since manual OCT segmentation is resource-consuming and has low reproducibility, automatic techniques are required. In this work, we introduce a novel deep learning based architecture that directly predicts the position of layers in OCT and guarantees their correct order, achieving state-of-the-art results for retinal layer segmentation. In particular, the average absolute distance between our model’s prediction and the ground truth layer segmentation in an AMD dataset is 0.63, 0.85, and 0.44 pixel for Bruch's membrane (BM), retinal pigment epithelium (RPE) and ellipsoid zone (EZ), respectively. Based on layer positions, we further quantify drusen load with excellent accuracy, achieving 0.994 and 0.988 Pearson correlation between drusen volumes estimated by our method and two human readers, and increasing the Dice score to 0.71 ± 0.16 (from 0.60 ± 0.23) and 0.62 ± 0.23 (from 0.53 ± 0.25), respectively, compared to a previous state-of-the-art method. Given its reproducible, accurate, and scalable results, our method can be used for the large-scale analysis of OCT data.

Comparative transcriptome analysis reveals the core molecular network in pattern-triggered immunity in Sorghum bicolor

Pathogen-associated molecular pattern (PAMP)-triggered immunity (PTI) is the first line of defense in plant disease resistance. However, the molecular mechanisms of plant PTI vary across species, making it challenging to identify a core set of trait-associated genes. This study aimed to investigate key factors that influence PTI and identify the core molecular network in Sorghum bicolor, a C4 plant. We performed comprehensive weighted gene co-expression network analysis and temporal expression analysis of large-scale transcriptome data from various sorghum cultivars under different PAMP treatments. Our results revealed that the type of PAMP had a stronger influence on the PTI network than did the sorghum cultivar. Following PAMP treatment, 30 genes with stable downregulated expression and 158 genes with stable upregulated expression were identified, including genes encoding potential pattern recognition receptors whose expression was upregulated within 1 h of treatment. PAMP treatment altered the expression of resistance-related, signaling, salt-sensitive, heavy metal-related, and transporter genes. These findings provide novel insights into the core genes involved in plant PTI and are expected to facilitate the identification and application of resistance genes in plant breeding studies.

Fathers in Europe: Policies, constructions and practices. Introduction to the Special Collection

Objective: To introduce the readers to the Journal of Family Research’s Special Collection about fatherhood-related political frameworks, social constructions of fatherhood and masculinity, and practices of fathers in Europe. Background: Fatherhood research has proliferated in recent decades and reflects that paternal involvement is closely linked to national policies, to prevailing social normative understandings of fatherhood, and also varies in practice. Method: Except for the review article, the contributions of this Special Collection draw on empirical data, including quantitative analyses of large-scale data, such as the European Social Survey, the QUIDAN-Survey, and the DJI “Growing up in Germany” survey, and qualitative analyses of in-depth interviews. Results: The six contributions vary in focus and illustrate a wide range of approaches to the understanding of fatherhood constructions and practices as well as the political frameworks that shape contemporary fatherhood in Europe. The contributions study fatherhood in the context of the transition to parenthood, parenting practices, the composition of working environments as well as in social work practice. Conclusion: The discussion of fatherhood constructions and practices as well as related political frameworks is crucial to understanding which social conditions facilitate and hinder father involvement in Europe.

Massive Parallel Alignment of RNA-seq Reads in Serverless Computing

In recent years, the use of Cloud infrastructures for data processing has proven useful, with a computing potential that is not affected by the limitations of a local infrastructure. In this context, Serverless computing is the fastest-growing Cloud service model due to its auto-scaling methodologies, reliability, and fault tolerance. We present a solution based on in-house Serverless infrastructure, which is able to perform large-scale RNA-seq data analysis focused on the mapping of sequencing reads to a reference genome. The main contribution was bringing the computation of genomic data into serverless computing, focusing on RNA-seq read-mapping to a reference genome, as this is the most time-consuming task for some pipelines. The proposed solution handles massive parallel instances to maximize the efficiency in terms of running time. We evaluated the performance of our solution by performing two main tests, both based on the mapping of RNA-seq reads to Human GRCh38. Our experiments demonstrated a reduction of 79.838%, 90.079%, and 96.382%, compared to the local environments with 16, 8, and 4 virtual cores, respectively. Furthermore, serverless limitations were investigated.

Curating Training Data for Reliable Large-Scale Visual Data Analysis: Lessons from Identifying Trash in Street View Imagery

Visual data have dramatically increased in quantity in the digital age, presenting new opportunities for social science research. However, the extensive time and labor costs to process and analyze these data with existing approaches limit their use. Computer vision methods hold promise but often require large and nonexistent training data to identify sociologically relevant variables. We present a cost-efficient method for curating training data that utilizes simple tasks and pairwise comparisons to interpret and analyze visual data at scale using computer vision. We apply our approach to the detection of trash levels across space and over time in millions of street-level images in three physically distinct US cities. By comparing to ratings produced in a controlled setting and utilizing computational methods, we demonstrate generally high reliability in the method and identify sources that limit it. Altogether, this approach expands how visual data can be used at a large scale in sociology.

Organizational Agility and Communicative Actions for Responsible Innovation: Evidence from manufacturing firms in South Korea

Responsible Innovation (RI) has increasingly attracted attention in academic research and business practice for UN Sustainable Development Goals (SDGs). Especially, the UN Economic and Social Commission for Asia and the Pacific (UNESCAP) highlights RI as a key concept to address how to institutionalize and organize action programs to solve grand challenges. However, we know little about the determinants of organizing RI in business. By applying agile management and stakeholder theory, this study proposes the critical organizational factors to implement RI, namely: Organizational Agility and Communicative Actions. Based on the large-scale data analysis of 13,234 South Korean manufacturing firms during the period between 1999 and 2019, this paper investigates whether and how organizational factors can affect RI in the Asian context, using the zero-inflated negative binomial regression model. Findings show that well-organized agile management and communicative actions with stakeholders are essential to enhance RI performance in a holistic view. This study contributes to the recently emerging field of RI and offers novel theoretical and practical implications for academics and practitioners regarding the underlying mechanism of organizing RI.

PLDH: Pseudo-Labels Based Deep Hashing

Deep hashing has received a great deal of attraction in large-scale data analysis, due to its high efficiency and effectiveness. The performance of deep hashing models heavily relies on label information, which is very expensive to obtain. In this work, a novel end-to-end deep hashing model based on pseudo-labels for large-scale data without labels is proposed. The proposed hashing model consists of two major stages, where the first stage aims to obtain pseudo-labels based on deep features extracted by a pre-training deep convolution neural network. The second stage generates hash codes with high quality by the same neural network in the previous stage, coupled with an end-to-end hash layer, whose purpose is to encode data into a binary representation. Additionally, a quantization loss is introduced and interwound within these two stages. Evaluation experiments were conducted on two frequently-used image collections, CIFAR-10 and NUS-WIDE, with eight popular shallow and deep hashing models. The experimental results show the superiority of the proposed method in image retrieval.

An empirical analysis of the impact of semiconductor engineer characteristics on outflows and inflows: evidence from six major semiconductor countries

The impact of cross-border migration of semiconductor engineers has become an important concern for many countries’ economic policies. There has been limited large-scale data analysis regarding the movement of semiconductor engineers between countries. This study creates an original engineer database for six major semiconductor countries (U.S., China, Taiwan, Japan, Korea, and Germany) using bibliographic information on patents and papers to analyze their transnational migration. Multiple logistic analysis is conducted on the impact of engineers’ characteristics such as age, skills, and areas of expertise on outflows and inflows. The analysis reveals that (1) the United States, Taiwan, Japan, and Germany have excess outflows of engineers, while China and South Korea have excess inflows, (2) the movement of engineers between the United States and China is the most frequent, and (3) there is a significant outflow of engineers from semiconductor fields in which each country specializes.

PANGEA: a new gene set enrichment tool for Drosophila and common research organisms.

Gene set enrichment analysis (GSEA) plays an important role in large-scale data analysis, helping scientists discover the underlying biological patterns over-represented in a gene list resulting from, for example, an 'omics' study. Gene Ontology (GO) annotation is the most frequently used classification mechanism for gene set definition. Here we present a new GSEA tool, PANGEA (PAthway, Network and Gene-set Enrichment Analysis; https://www.flyrnai.org/tools/pangea/), developed to allow a more flexible and configurable approach to data analysis using a variety of classification sets. PANGEA allows GO analysis to be performed on different sets of GO annotations, for example excluding high-throughput studies. Beyond GO, gene sets for pathway annotation and protein complex data from various resources as well as expression and disease annotation from the Alliance of Genome Resources (Alliance). In addition, visualizations of results are enhanced by providing an option to view network of gene set to gene relationships. The tool also allows comparison of multiple input gene lists and accompanying visualisation tools for quick and easy comparison. This new tool will facilitate GSEA for Drosophila and other major model organisms based on high-quality annotated information available for these species.

Approaches and tools for user-driven provenance and data quality information in spatial data infrastructures

ABSTRACT Geospatial data are fundamental in most global-change and sustainability-related domains. However, readily accessible information on data quality and provenance is often missing or hardly accessible for users due to technical or perceptual barriers, for example, due to unstructured metadata information or missing references. Within an interdisciplinary process encompassing perspectives of data users, data producers, and software developers, we identified major needs to facilitate effective fitness-for-use assessments by data users and developed approaches to address these. We provided a stylized analysis of large-scale land use data to showcase selected approaches. To support data users, interoperable quality and provenance information need to be meaningfully represented. Data producers need efficient workflows and tools supporting them in creating high-quality, structured and detailed quality and provenance information. Our newly developed approaches to increase the availability of structured metadata synthesize new and existing tools to extract metadata or to generate provenance data during processing. Within our approaches to improve interoperability and accessibility we present novel tools to support (i) the creation of curated and linked registers of data quality indicators and thematic terms, and (ii) linked visualization of data quality and provenance information. Following our approaches increases transparency, facilitates fitness-for-use assessments, and ultimately improves research quality.

A national transgender health survey from China assessing gender identity conversion practice, mental health, substance use and suicidality

Gender identity conversion practice (GICP) refers to interventions that intend to alter an individual’s gender identity that is incongruent with societal expectations based on the sex assigned at birth. In this study, the term GICP refers to both professional conversion efforts (also called Gender Identity Conversion Efforts, GICE, performed by psychologists, psychiatrists and so on) and non-professional conversion efforts (performed by family members and so on). Here data were analysed from the Chinese Transgender Health Survey covering transgender, nonbinary and gender diverse (TNG) adolescents and adults, with 7,576 respondents from mainland China entering the analysis following the application of exclusion criteria. Results showed that GICP is a risk factor for multiple mental health problems including depression, anxiety, post-traumatic stress disorder symptoms, suicidal ideation and suicidal plan in the previous 12 months, suicide attempts in both the previous 12 months and in lifetime, non-suicidal self-injury in the previous 12 months, and alcohol use. Participants with experience of professional GICP reported suicide attempts more frequently than those with experience of non-professional GICP. Compared with other age groups, GICP tended to associate with more severe mental health problems in TNG aged ≤17 years old. Evidence suggests that GICP worsens the mental health problems faced by the TNG population (especially adolescents) and reveals the equivalent detrimental effects from both professional GICP and non-professional GICP. It is necessary for the public to become more aware of the devastative impact of GICP on the TNG population. Analysis of large-scale representative data from the Chinese National Transgender Survey finds that exposure to gender identity conversion practice is a risk factor for mental health problems, worsens mental health disorder severity and increases suicidality.

Achieving high yield and nitrogen agronomic efficiency by coupling wheat varieties with soil fertility

Wheat breeding has progressively increased yield potential through decades of selection, markedly increased the capacity for food production. Nitrogen (N) fertilizer is essential for wheat production and N agronomic efficiency (NAE) is commonly index used for evaluate the effects of N fertilizer on crop yield, calculated as the difference of wheat yield between N fertilizer treatment and non-N fertilizer treatment divided by the total N application rate. However, the impact of variety on NAE and its interaction with soil fertility remain unknown. Here, to clarify whether and how wheat variety contributes to NAE, and to determine if soil conditions should be considered in variety selection, we conduct a large-scale analysis of data from 12,925 field trials spanning ten years and including 229 wheat varieties, 5 N fertilizer treatments, and a range of soil fertility across China's major wheat production zones. The national average NAE was 9.57 kg kg−1, but significantly differed across regions. At both the national and regional scales, variety significantly affected NAE, and different varieties showed high variability in their performance among low, moderate, and high fertility soils. Here, superior varieties with both high yield and high NAE were identified at each soil fertility fields. The comprehensive effect of selecting regionally superior varieties, optimizing N management, and improving soil fertility could potentially decrease the yield gap by 67 %. Therefore, variety selection based on soil conditions could facilitate improved food security while reducing fertilizer inputs to alleviate environmental impacts.

Efficient False Positive Control Algorithms in Big Data Mining

The typical hypothesis testing issue in statistical analysis is determining whether a pattern is significantly associated with a specific class label. This usually leads to highly challenging multiple-hypothesis testing problems in big data mining scenarios, as millions or billions of hypothesis tests in large-scale exploratory data analysis can result in a large number of false positive results. The permutation testing-based FWER control method (PFWER) is theoretically effective in dealing with multiple hypothesis testing issues. In reality, however, this theoretical approach confronts a serious computational efficiency problem. It takes an extremely long time to compute an appropriate FWER false positive control threshold using PFWER, which is almost impossible to achieve in a reasonable amount of time using human effort on medium- or large-scale data. Although some methods for improving the efficiency of the FWER false positive control threshold calculation have been proposed, most of them are stand-alone, and there is still a lot of space for efficiency improvement. To address this problem, this paper proposes a distributed PFWER false-positive threshold calculation method for large-scale data. The computational effectiveness increases significantly when compared to the current approaches. The FP-growth algorithm is used first for pattern mining, and the mining process reduces the computation of invalid patterns by using pruning operations and index optimization for merging patterns with index transactions. The distributed computing technique is introduced on this basis, and the constructed FP tree is decomposed into a set of subtrees, each corresponding to a subtask. All subtrees (subtasks) are distributed to different computing nodes. Each node independently calculates the local significance threshold according to the designated subtasks. Finally, all local results are aggregated to compute the FWER false positive control threshold, which is completely consistent with the theoretical result. A series of experimental findings on 11 real-world datasets demonstrate that the distributed algorithm proposed in this paper can significantly improve the computation efficiency of PFWER while ensuring its theoretical accuracy.

Disentangling the rhythms of human activity in the built environment for airborne transmission risk: An analysis of large-scale mobility data.

Since the outset of the COVID-19 pandemic, substantial public attention has focused on the role of seasonality in impacting transmission. Misconceptions have relied on seasonal mediation of respiratory diseases driven solely by environmental variables. However, seasonality is expected to be driven by host social behavior, particularly in highly susceptible populations. A key gap in understanding the role of social behavior in respiratory disease seasonality is our incomplete understanding of the seasonality of indoor human activity. We leverage a novel data stream on human mobility to characterize activity in indoor versus outdoor environments in the United States. We use an observational mobile app-based location dataset encompassing over 5 million locations nationally. We classify locations as primarily indoor (e.g. stores, offices) or outdoor (e.g. playgrounds, farmers markets), disentangling location-specific visits into indoor and outdoor, to arrive at a fine-scale measure of indoor to outdoor human activity across time and space. We find the proportion of indoor to outdoor activity during a baseline year is seasonal, peaking in winter months. The measure displays a latitudinal gradient with stronger seasonality at northern latitudes and an additional summer peak in southern latitudes. We statistically fit this baseline indoor-outdoor activity measure to inform the incorporation of this complex empirical pattern into infectious disease dynamic models. However, we find that the disruption of the COVID-19 pandemic caused these patterns to shift significantly from baseline and the empirical patterns are necessary to predict spatiotemporal heterogeneity in disease dynamics. Our work empirically characterizes, for the first time, the seasonality of human social behavior at a large scale with a high spatiotemporal resolutio and provides a parsimonious parameterization of seasonal behavior that can be included in infectious disease dynamics models. We provide critical evidence and methods necessary to inform the public health of seasonal and pandemic respiratory pathogens and improve our understanding of the relationship between the physical environment and infection risk in the context of global change. Research reported in this publication was supported by the National Institute of General Medical Sciences of the National Institutes of Health under award number R01GM123007.