Intermediate Data Research Articles

Optimal Re-Materialization Strategies for Heterogeneous Chains: How to Train Deep Neural Networks with Limited Memory

Training in Feed Forward Deep Neural Networks is a memory-intensive operation which is usually performed on GPUs with limited memory capacities. This may force data scientists to limit the depth of the models or the resolution of the input data if data does not fit in the GPU memory. The re-materialization technique, whose idea comes from the checkpointing strategies developed in the Automatic Differentiation literature, allows data scientists to limit the memory requirements related to the storage of intermediate data (activations), at the cost of an increase in the computational cost. This paper introduces a new strategy of re-materialization of activations that significantly reduces memory usage. It consists in selecting which activations are saved and which activations are deleted during the forward phase, and then recomputing the deleted activations when they are needed during the backward phase. We propose an original computation model that combines two types of activation savings: either only storing the layer inputs, or recording the complete history of operations that produced the outputs. This paper focuses on the fully heterogeneous case, where the computation time and the memory requirement of each layer is different. We prove that finding the optimal solution is NP-hard and that classical techniques from Automatic Differentiation literature do not apply. Moreover, the classical assumption of memory persistence of materialized activations, used to simplify the search of optimal solutions, does not hold anymore. Thus, we propose a weak memory persistence property and provide a Dynamic Program to compute the optimal sequence of computations. This algorithm is made available through the Rotor software, a PyTorch plug-in dealing with any network consisting of a sequence of layers, each of them having an arbitrarily complex structure. Through extensive experiments, we show that our implementation consistently outperforms existing re-materialization approaches for a large class of networks, image sizes and batch sizes.

To store or not: Online cost optimization for running big data jobs on the cloud

As businesses increasingly rely on cloud-based big data analytics services to drive insights, reducing the cost of storing and analyzing large volumes of data in the cloud has become a major concern. During the execution of big data analysis jobs, some of the generated data can be reused by subsequent jobs. By storing such intermediate data, the cost of running big data jobs can be greatly reduced for businesses using cloud services. An important challenge is how to determine which data should be stored in order to save costs. Existing storing strategies do not differentiate between data with different usage frequencies, resulting in significant storage costs in practical applications. To address the aforementioned challenges, in this paper we propose two online algorithms, one deterministic and the other randomized, which dynamically determine whether to store the data with the aim of saving cost. We show that our proposed deterministic algorithm (resp., randomized) incurs costs within a factor of 2−α′ (resp., 21+α′) times the minimum cost obtained by an optimal offline algorithm which is assumed to know the exact future a priori. Finally, through extensive experiments with real-world workload of big data jobs in Alibaba Cloud environment, we demonstrate that our proposed online algorithms can achieve significant cost savings under common cloud pricing schemes.

Efficient Algorithm Set Forming for the Computing Resources Distribution in Heterogeneous Dynamic Computational Environments Based on the Ontology Usage

Purpose of research. The purpose of this research is to develop an ontology structure as the basis of a database/knowledge base for selecting effective metaheuristic algorithms for solving the problem of load distribution in heterogeneous distributed dynamic computing environments, taking into account the overhead of data transmission over the network.Methods. The main scientific methods used in this study are domain analysis, methods for constructing subject ontologies, numerical optimization methods and computer modeling.Since the literature does not present resource allocation planning models that would take into account geographic distribution, the presence of intermediate data transmission routes, the dynamics of topologies and load, as well as system heterogeneity in terms of criteria for assessing the quality of load distribution, this article proposes a new model that takes into account these features. The complexity of solving a planning problem becomes one of the variable parameters, which has a significant impact on the planning result: with a decrease in the complexity of calculations, the result deteriorates accordingly. Therefore, a greedy strategy is proposed as a solution method: from the optimization methods to be considered, select the least labor-intensive one that would allow obtaining the best result in the allotted time. Test runs of simulated annealing algorithms demonstrate different effectiveness under different initial conditions of the problem; therefore, it is advisable for selected classes of problems to choose algorithms that are effective in terms of solution quality and labor intensity.Results. The result of the study is the structure of the ontology of effective algorithms. Also, the results are instances of simulated annealing algorithms and tasks included in the ontology, related by the “efficiency” relation.Conclusion. This article proposes the structure of an ontology of effective optimization algorithms and an approach to solving the problem of distributing the computational load, taking into account the complexity of the distribution procedure through the “greedy” selection of the most effective optimization algorithms.

Alzheimer's disease detection using data fusion with a deep supervised encoder.

Alzheimer's disease (AD) is affecting a growing number of individuals. As a result, there is a pressing need for accurate and early diagnosis methods. This study aims to achieve this goal by developing an optimal data analysis strategy to enhance computational diagnosis. Although various modalities of AD diagnostic data are collected, past research on computational methods of AD diagnosis has mainly focused on using single-modal inputs. We hypothesize that integrating, or "fusing," various data modalities as inputs to prediction models could enhance diagnostic accuracy by offering a more comprehensive view of an individual's health profile. However, a potential challenge arises as this fusion of multiple modalities may result in significantly higher dimensional data. We hypothesize that employing suitable dimensionality reduction methods across heterogeneous modalities would not only help diagnosis models extract latent information but also enhance accuracy. Therefore, it is imperative to identify optimal strategies for both data fusion and dimensionality reduction. In this paper, we have conducted a comprehensive comparison of over 80 statistical machine learning methods, considering various classifiers, dimensionality reduction techniques, and data fusion strategies to assess our hypotheses. Specifically, we have explored three primary strategies: (1) Simple data fusion, which involves straightforward concatenation (fusion) of datasets before inputting them into a classifier; (2) Early data fusion, in which datasets are concatenated first, and then a dimensionality reduction technique is applied before feeding the resulting data into a classifier; and (3) Intermediate data fusion, in which dimensionality reduction methods are applied individually to each dataset before concatenating them to construct a classifier. For dimensionality reduction, we have explored several commonly-used techniques such as principal component analysis (PCA), autoencoder (AE), and LASSO. Additionally, we have implemented a new dimensionality-reduction method called the supervised encoder (SE), which involves slight modifications to standard deep neural networks. Our results show that SE substantially improves prediction accuracy compared to PCA, AE, and LASSO, especially in combination with intermediate fusion for multiclass diagnosis prediction.

Regional disparities in antenatal care utilization among pregnant women and its determinants in Ethiopia.

Antenatal care primarily focuses on health care checkups, the provision of advice on healthy behaviors, and the delivery of psychological, social, and emotional support for women with pregnancy. The national target set by the Ethiopian government is to achieve 95% of at least four ANC visits. Nevertheless, 43.11% of women had four or more checkups, according to the 2019 Ethiopian Demographic and Health Survey. Despite this achievement, antenatal care visits differ significantly between Ethiopian regions. Consequently, the purpose of this study was to assess regional disparities in pregnant women's utilization of antenatal care and its determinants in Ethiopia. We have used 2019 intermediate Ethiopian Demographic and Health Survey data for analysis. The analysis comprised a total of 3,917 weighted women age 15-49 who had a live birth in the 5 years preceding the survey. Poisson regression analysis was done using SAS software version 9.4. To show the strength and direction of the association, an incidence rate ratio with a 95% confidence interval was used. Variables with a p-value <0.05 were declared as significant factors associated with the number of ANC visits. In Ethiopia, the number of ANC visits differs between regions. With a mean of 4.74 (95% CI: 4.49, 4.99), Addis Ababa reported the highest percentage of ANC visits (82.7%), while the Somali Region reported the lowest percentage (11.3%) with a mean of 0.73 (95% CI: 0.57, 0.88). Maternal age, educational level, religion, household wealth index, place of delivery, and household size show significant associations with the number of antenatal care visits. In Ethiopia, there is the highest regional disparity in the number of ANC visits. The number of ANC visits was influenced by the mother's age, education, religion, household wealth index, place of delivery, and household size. Regarding the ANC visits, there should be initiatives that address the demands of pastoralist and agro-pastoralist communities to increase ANC utilization. As with many other health outcomes, education and low socio-economic status were associated with low ANC visit but these are tied to the overall social development of a country and are not immediately amenable to public health interventions.

40 MHz triggerless readout of the CMS Drift Tube muon detector

The Level-1 trigger scouting system of the CMS experiment aims at intercepting the intermediate data produced by the L1 trigger processors before the final trigger decision. This system can be complemented by adding the raw stream of data collected from the detector front-end, whenever the throughput is manageable. In this work, the triggerless readout of the CMS Drift Tubes (DT) detector is presented. This is realized by reading a sector of the DT which has been equipped with the preproduction of Phase-2 upgrade front-end boards. A Xilinx VCU118 acts as a concentrator of the Phase-2 demonstrator lpGBT links and transmits data to a server via 100G TCP/IP. First results coming from a test-stand mimicking the sector demonstrator are shown.

Efficient asynchronous federated learning with sparsification and quantization

SummaryWhile data is distributed in multiple edge devices, federated learning (FL) is attracting more and more attention to collaboratively train a machine learning model without transferring raw data. FL generally exploits a parameter server and a large number of edge devices during the whole process of the model training, while several devices are selected in each round. However, straggler devices may slow down the training process or even make the system crash during training. Meanwhile, other idle edge devices remain unused. As the bandwidth between the devices and the server is relatively low, the communication of intermediate data becomes a bottleneck. In this article, we propose time‐efficient asynchronous federated learning with sparsification and quantization, that is, TEASQ‐Fed. TEASQ‐Fed can fully exploit edge devices to asynchronously participate in the training process by actively applying for tasks. We utilize control parameters to choose an appropriate number of parallel edge devices, which simultaneously execute the training tasks. In addition, we introduce a caching mechanism and weighted averaging with respect to model staleness to further improve the accuracy. Furthermore, we propose a sparsification and quantitation approach to compress the intermediate data to accelerate the training. The experimental results reveal that TEASQ‐Fed improves the accuracy (up to 16.67% higher) while accelerating the convergence of model training (up to twice faster).

Long-read sequencing-based transcriptomic landscape in longissimus dorsi and transcriptome-wide association studies for growth traits of meat rabbits.

Rabbits are an attractive meat livestock species that can efficiently convert human-indigestible plant biomass, and have been commonly used in biological and medical researches. Yet, transcriptomic landscape in muscle tissue and association between gene expression level and growth traits have not been specially studied in meat rabbits. In this study Oxford Nanopore Technologies (ONT) long-read sequencing technology was used for comprehensively exploring transcriptomic landscape in Longissimus dorsi for 115 rabbits at 84 days of age, and transcriptome-wide association studies (TWAS) were performed for growth traits, including body weight at 84 days of age and average daily gain during three growth periods. The statistical analysis of TWAS was performed using a mixed linear model, in which polygenic effect was fitted as a random effect according to gene expression level-based relationships. A total of 18,842 genes and 42,010 transcripts were detected, among which 35% of genes and 47% of transcripts were novel in comparison with the reference genome annotation. Furthermore, 45% of genes were widely expressed among more than 90% of individuals. The proportions (±SE) of phenotype variance explained by genome-wide gene expression level ranged from 0.501 ± 0.216 to 0.956 ± 0.209, and the similar results were obtained when explained by transcript expression level. In contrast, neither gene nor transcript was detected by TWAS to be statistically significantly associated with these growth traits. In conclusion, these novel genes and transcripts that have been extensively profiled in a single muscle tissue using long-read sequencing technology will greatly improve our understanding on transcriptional diversity in rabbits. Our results with a relatively small sample size further revealed the important contribution of global gene expression to phenotypic variation on growth performance, but it seemed that no single gene has an outstanding effect; this knowledge is helpful to include intermediate omics data for implementing genetic evaluation of growth traits in meat rabbits.

Producing uncertainties and covariance matrix from intermediate data using a Monte-Carlo method

Producing uncertainties and covariance matrix from intermediate data using a Monte-Carlo method

Low-frequency oscillations analysis and big data

The increased observability of processes in electric power systems, provided by the widespread introduction of synchrophasor measurement technology, as well as the use of computational processing of measurement data arrays, has led to changes in the methodology for analyzing the power system operating modes. Along with the development of new approaches to modeling processes and received measurement signals, a class of data-based methods has emerged. In a number of recent studies, the analysis of synchrophasor measurement data at the large power system scale is considered as a big data problem. Despite the controversial nature of this point of view, the application of some big data methods in synchrophasor data analysis yields positive results. In the proposed report, an assessment of the volumes of initial and intermediate data in the analysis of forced low-frequency oscillations is performed. The effect of data explosion at intermediate stages of calculations is shown. The storage and transmission of these data can be replaced by their calculation in the presence of a storable and transmittable configuration of a generalized computational scheme. A method for compact visualization of an array of oscillation dynamic parameters is shown by the example of dissipating energy. In addition, an approach is presented for configuring streams of synthetic test data at the power system scale, which provides a desired result in the analysis of dissipative energy.

Workflow of Genome-Wide Mediation Analysis (GMA) by Incorporating Intermediate Omics Data as Mediators

Workflow of Genome-Wide Mediation Analysis (GMA) by Incorporating Intermediate Omics Data as Mediators

Scientific Discovery Framework Accelerating Advanced Polymeric Materials Design.

Organic polymer materials, as the most abundantly produced materials, possess a flammable nature, making them potential hazards to human casualties and property losses. Target polymer design is still hindered due to the lack of a scientific foundation. Herein, we present a robust, generalizable, yet intelligent polymer discovery framework, which synergizes diverse capabilities, including the insitu burning analyzer, virtual reaction generator, and material genomic model, to achieve results that surpass the sum of individual parts. Notably, the high-throughput analyzer created for the first time, grounded in multiple spectroscopic principles, enables insitu capturing of massive combustion intermediates; then, the created realistic apparatus transforming to the virtual reaction generator acquires exponentially more intermediate information; further, the proposed feature engineering tool, which embedded both polymer hierarchical structures and massive intermediate data, develops the generalizable genomic model with excellent universality (adapting over 20 kinds of polymers) and high accuracy (88.8%), succeeding discovering series of novel polymers. This emerging approach addresses the target polymer design for flame-retardant application and underscores a pivotal role in accelerating polymeric materials discovery.

Notice of Violation of IEEE Publication Principles: Scientific Workflow Partitioning and Data Flow Optimization in Hybrid Clouds

Notice of Violation of IEEE Publication Principles “Scientific Workflow Partitioning and Data Flow Optimization in Hybrid Clouds” by Rubing Duan, Rick Siow Mong Goh, Zheng Qin and Young Liu in Pre-Prints for IEEE Transactions on Cloud Computing After careful and considered review of the content and authorship of this paper by a duly constituted expert committee, this paper has been found to be in violation of IEEE's Publication Principles. Due to the nature of this violation, reasonable effort should be made to remove all past references to this paper and it should not be used for research or citation The scheduling and execution of workflow applications are complex issues because of the dynamic and heterogeneous nature of distributed computing environments like hybrid clouds. While hybrid cloud computing environments provide good potential for achieving high performance and low economic cost, it also introduces a broad set of unpredictable overheads. This paper describes a novel approach to refine workflow structure and optimize intermediate data transfers without changing the scale of scheduled cloud service for large-scale scientific workflows containing thousands (or even millions) of tasks. These methods include pre- and post-partitioning of workflows, data flow optimization, static and dynamic optimization, as well as virtual single execution environment incorporated into a workflow management system.We demonstrate the effectiveness of our methods on the three real applications, Montage, Broadband, and WIEN2k, executed in a real hybrid cloud.

Event and data persistency models for the LHCb Real Time Analysis System

Starting in 2022, the upgraded LHCb detector is collecting data with a pure software trigger. In its first stage, reducing the rate from 30MHz to about 1MHz, GPUs are used to reconstruct and trigger on B and D meson topologies and high-pT objects in the event. In its second stage, a CPU farm is used to reconstruct the full event and perform candidate selections, which are persisted for offline use with an output rate of about 10GB/s. Fast data processing, flexible and custom-designed data structures tailored for SIMD architectures and efficient storage of the intermediate data at various steps of the processing pipeline onto persistent media, e.g. tapes is essential to guarantee the full physics program of LHCb. We present the event model and data persistency developments for the trigger of LHCb in Run 3. Particular emphasis is given to the novel software-design aspects with respect to the Run 1+2 data taking, the performance improvements which can be achieved and the experience of restructuring a major part of the reconstruction software in a large HEP experiment.

Intercalibration: A Cornerstone of the Success of the GEOTRACES Program

The international GEOTRACES program was developed to enhance knowledge about the distribution of trace elements and their isotopes (TEIs) in the ocean and to reduce the uncertainty about their sources, sinks, and internal cycling. Recognizing the importance of intercalibration from the outset, GEOTRACES implemented intercalibration efforts early in the program, and consensus materials were generated that included the full range of TEIs dissolved in seawater, in suspended particles, and from aerosols. The GEOTRACES section cruises include “crossover station(s)” that are occupied by two or more sections and whereby all aspects of sample collection, preservation, and processing can be compared and intercalibrated. Once datasets are generated, an international intercalibration committee reviews intercalibration reports and works with the community to address issues and provide intercalibrated data for intermediate data products. This process has resulted in a highly cooperative community that shares advances in protocols to strengthen capacity building and GEOTRACES outcomes, including an unprecedented oceanic atlas of TEIs, with data quality that is state-of-the-art. This article outlines the development and implementation of the successful GEOTRACES intercalibration process.

Multi-output gaussian processes for the reconstruction of homogenized cross-sections

Deterministic nuclear reactor simulators employing the prevalent two-step scheme often generate a substantial amount of intermediate data at the interface of their two subcodes, which can impede the overall performance of the software. The bulk of this data comprises “few-groups homogenized cross-sections” or HXS, which are stored as tabulated multivariate functions and interpolated inside the core simulator. A number of mathematical tools have been studied for this interpolation purpose over the years, but few meet all the challenging requirements of neutronics computation chains: extreme accuracy, low memory footprint, fast predictions… We here present a new framework to tackle this task, based on multi-outputs gaussian processes (MOGP). This machine learning model enables us to interpolate HXS’s with improved accuracy compared to the current multilinear standard, using only a fraction of its training data – meaning that the amount of required precomputation is reduced by a factor of several dozens. It also necessitates an even smaller fraction of its storage requirements, preserves its reconstruction speed, and unlocks new functionalities such as adaptive sampling and facilitated uncertainty quantification. We demonstrate the efficiency of this approach on a rich test case reproducing the VERA benchmark, proving in particular its scalability to datasets of millions of HXS.

Variant Parallelism: Lightweight Deep Convolutional Models for Distributed Inference on IoT Devices

Two major techniques are commonly used to meet real-time inference limitations when distributing models across resource-constrained IoT devices: (1) model parallelism () and (2) class parallelism (). In, transmitting bulky intermediate data (orders of magnitude larger than input) between devices imposes huge communication overhead. Although solves this problem, it has limitations on the number of sub-models. In addition, both solutions are fault intolerant, an issue when deployed on edge devices. We propose variant parallelism (), an ensemble-based deep learning distribution method where different variants of a main model are generated and can be deployed on separate machines. We design a family of lighter models around the original model, and train them simultaneously to improve accuracy over single models. Our experimental results on six common mid-sized object recognition datasets demonstrate that our models can have 5.87.1× fewer parameters, 4.331× fewer multiply-accumulations (MACs), and 2.513.2× less response time on atomic inputs compared to MobileNetV2 while achieving comparable or higher accuracy. Our technique easily generates several variants of the base architecture. Each variant returns only 2k outputs 1≤k≤#classes2, representing Topk classes, instead of tons of floating point values required in . Since each variant provides a full-class prediction, our approach maintains higher availability compared with and in presence of failure.

The GEOTRACES Intermediate Data Products: Rich Resources for Research, Education, and Outreach

By mid-2023, the international GEOTRACES program had released three intermediate data products (IDP2014, IDP2017, and IDP2021), and in July 2023, an update of the latest product, IDP2021v2, was issued. All IDPs consist of two parts: (1) a compilation of digital data for large numbers of trace elements and isotopes (TEIs), and (2) the eGEOTRACES Electronic Atlas containing almost 1,500 pre-​created section plots and 269 animated three-dimensional scenes that can be browsed via an interactive web interface. GEOTRACES IDPs are used extensively and have proven to be rich resources for research, education, and outreach. Here, we demonstrate how these resources can be used efficiently and effectively via online services. Data browsing, analysis, and visualization occur in the user’s web browser, with the IDP data remaining on a dedicated server. Users simply visit specific resource URLs to access eGEOTRACES visuals and the GEOTRACES digital data directly. We first demonstrate how to navigate the eGEOTRACES Electronic Atlas to view TEI sections and three-dimensional animations. We then focus on two research use cases and provide detailed hands-on instructions for creating publication-ready figures related to the marine Zn cycle.

Community-based intervention for promoting healthy habits through MAHA mobile application in Basque elderly population.

Background: Population over 65 years of age in EU is increasing, rising costs in health and social care systems and making changes in biological, physiological, and psychosocial field. Promoting healthy ageing has become a key to improve the lifelong health and wellbeing of ageing population. Health technologies and eHealth tools can be an effective instrument to address the challenge of ageing. They provide an opportunity to improve the health and well-being of older people, and to prevent, treat and ensure the sustainability of health systems.&#x0D; Digital technologies aimed at promoting active ageing enable access to greater knowledge, better communication and a better relationship with people's environment, if they are designed in a person-centred way. They open up a range of new possibilities and resources that represent a way to reduce the generation gap and bring older people closer to the rest of the community.&#x0D; Methods: The Basque Country region has implemented a community-based intervention to promote active and healthy ageing through the use of a self-managed mobile MAHA App within Gatekeeper European project (H2020, nº857223). MAHA app developed in a co-creation process involving end-users and professionals from community settings is available in Google Play and App Store for iOS. The tool offers activities and materials to reinforce socialization and, promote healthy, meaningful, creative and fulfilling lives among elderly people.&#x0D; This quasi-experimental and longitudinal study is targeted at 10,000 older people and/or their caregivers from the Basque Country (Spain). The community-based intervention in the Basque Country has been designed to involve Primary Health Care services and key agents from different sectors at the community level that can influence the well-being and health of the elderly population. The Support Network created for the intervention has the participation of 39 community organisations and entities belonging to the Basque Health Ecosystem.&#x0D; Results: A mixed methods approach to evaluation will be carried out, using quantitative techniques, through validated questionnaires and usage metrics to assess effectiveness; and qualitative techniques, to collect user experience. An analysis of the implementation process has been executed, to identify the determinants, factors, and aspects that impact the intervention, and the commitment, adherence and dropout rate towards user engagement. The intermediate data shows that the actions implemented are effective, as the intervention has been widely disseminated and many people have been contacted. Although more than half have accepted to participate in the intervention, only 3% finally registered in the application. This intervention also shows low adherence to the intervention and a high level of dropout, as it is understood that once the user accepted to participate, they do not follow the study protocol. &#x0D; Discussion and Conclusions: The community-based intervention implemented will contribute to generating awareness and new data on research into the intention-to-treat, adherence and attrition rate of the use of self-managed eHealth applications by the elderly. Likewise, a tool with these characteristics can be considered an effective health asset for improving elderly people's health, making healthy options more accessible and incorporating new habits in this population.&#x0D; &#x0D;

EFMVFL: An Efficient and Flexible Multi-party Vertical Federated Learning without a Third Party

Federated learning (FL) is a machine learning setting which allows multiple participants collaboratively to train a model under the orchestration of a server without disclosing their local data. Vertical federated learning (VFL) is a special structure in FL. It handles the situation where participants have the same ID space but different feature spaces. In order to guarantee the security and privacy of the local data of each participant, homomorphic encryption (HE) is often used to transmit intermediate parameters or data during the training process. In most VFL frameworks, a trusted third-party server is necessary because the plaintexts of the parameters need to be revealed for the computation. However, it is hard to find such a credible entity in the real world. Existing methods for solving this problem are either communication-intensive or unsuitable for multi-party scenarios. By combining secret sharing (SS) and HE, we propose a novel VFL framework without any trusted third parties called EFMVFL. It allows intermediate parameters to be transmitted among multiple parties without revealing the plaintexts. EFMVFL is applicable to generalized linear models (GLMs) and supports flexible expansion to multiple participants. Extensive experiments under Logistic Regression and Poisson Regression show that our framework is outstanding in communication (reduced by 3.2×– 6.8×) and efficiency (accelerated by 1.6×– 3.1×).