Data Processing Flow Research Articles

Metasurface-enabled 3D imaging via local bright spot gray scale matching using the structured light dot array.

Three-dimensional (3D) imaging is widely utilized in various applications, such as light detection, autonomous vehicles, and machine vision. However, conventional 3D imaging systems often rely on bulky optical components. Metasurfaces, as next-generation optical devices, possess flexible wavefront modulation capabilities and excellent combination with computer vision algorithms. Here, we propose a large field-of-view (FOV) structured light dot array projection device based on a metasurface, covering a 2π-FOV, for projecting coded point clouds in Fourier space. We explore a local bright spot gray scale matching algorithm for depth extraction, enabling 3D imaging. This algorithm simplifies the data processing flow and optimizes depth extraction and feature matching processes through a customized region gray scale comparison. As a result, it effectively reduces computational complexity and enhances tolerance to image quality fluctuations. The proposed approach provides new possibilities for developing compact and high-performance planar 3D optical imaging devices, which will drive the advancement of fields such as computer vision and artificial intelligence.

Full Stack Web Development

Full-stack development encompasses the comprehensive design and implementation of both client-side and server-side components of web applications. Utilizing HTML for structuring content, CSS for styling and layout, and JavaScript for dynamic interactivity, developers create engaging and responsive user interfaces. On the server side, PHP serves as a robust scripting language for handling back-end logic, including data processing and application flow. Coupled with MongoDB, a NoSQL database, developers manage data storage and retrieval efficiently, leveraging its flexibility for complex data structures. This synergy of technologies empowers developers to build scalable and high-performance applications that meet modern web standards. Index Terms—Web Development, Html, CSS, JavaScript, React, Php MongoDB.

Diminished functional segregation and resilience are associated with symptomatic severity and cognitive impairments in schizophrenia: a large-scale study

BackgroundThe research findings on the topological properties of functional connectomes (TP-FCs) in patients with schizophrenia (SZPs) exhibit inconsistencies and contradictions, which can be attributed to limitations such as small sample...

Towards Double-Layer Dynamic Heterogeneous Redundancy Architecture for Reliable Railway Passenger Service System

Researchers have proposed the dynamic heterogeneous redundancy (DHR) architecture, which integrates dynamic, heterogeneous, redundant, and closed-loop feedback elements into the system, to fortify the reliability of the railway passenger service system (RPSS). However, there are at least two weaknesses with the common DHR architectures: (1) they need system nodes with enough computing and storage resources; (2) they have hardly considered the reliability of DHR architecture. To this end, this paper proposes a double-layer DHR (DDHR) architecture to ensure the reliability of RPSS. This architecture introduces a set of algorithms, which are optimized co-computation and ruling weight optimization algorithms for the data processing flow of the DDHR architecture. This set improves the reliability of the DDHR architecture. For the evaluation of the reliability of DDHR architecture, this paper also proposes two metrics: (1) Dynamic available similarity metric. This metric does not rely on the overall similarity of the double-layer redundant executor sets but evaluates the similarity of their performance under the specified interaction paths within a single scheduling cycle. The smaller its similarity, the higher its reliability. (2) Scheduling cycle under dual-layer similarity threshold. This metric evaluates the reliability of the RPSS under actual conditions by setting the schedulable similarity thresholds between the same and different layers of the dual-layer redundant executives in the scheduling process. Finally, analog simulation experiments and prototype system building experiments are carried out, whose numerical experimental results show that the DDHR architecture outperforms the traditional DHR architecture in terms of reliability and performance under different redundancy and dynamically available similarity thresholds, while the algorithmic complexity and multi-tasking concurrency performance are slightly weaker than that of the DHR architecture, but can be applied to the main operations of the RPSS in general.

Key technologies and applications of collaboration between digital power grid and Internet of Things

AbstractThe integration of electricity technology and information technology, such as the Internet of Things (IoT), enables the construction of new power systems, along with the innovation of application scenarios and business scope. The key technologies of power IoT and the data flow process are summarised first. The IoT technology and application scenario requirements of power generation, transmission, loading, and storage of new power systems are studied. Thus, the nature of the collaborative development of the digital power grid and the IoT is demonstrated from the perspective of data processing in power IoT and application requirements in power systems. The key problems and solutions faced by the power IoT under the digital transformation are described, and the cross‐integration of key technologies and promotion of application scenario innovation are prospected. Finally, the key issues of future technological development were discussed, providing reference ideas for fully leveraging the value of energy and electricity data production factors and promoting the construction of a digital electricity ecosystem.

Word frequency statistics based on MapReduce on serverless platforms

This paper investigates the application of serverless computing in conjunction with the MapReduce framework, particularly in machine learning (ML) tasks. The MapReduce programming model has been widely used to process large-scale datasets by simplifying parallel and distributed data processing. This study explores how the combination of these two technologies can provide more efficient and cost-effective ML solutions. Through a detailed analysis of serverless environments and the MapReduce framework, this paper shows how the combination can advance the fields of cloud computing and machine learning. The experimental part includes the implementation of Map-Reduce model on a serverless platform, exploring the impact of different parameter settings on performance and improving efficiency by optimizing the data processing flow. In addition, the paper analyzes the use of memory and CPU resources and derives the relationship between dataset size, memory consumption and processor configuration and execution time. Through these experiments and analyses, this paper provides an empirical basis and theoretical support for the optimization of cloud computing frameworks.

Manajemen Koperasi Gema Nusa Gerakan Pemuda Ansor Kecamatan Cilongok Kabupaten Banyumas Tahun 2023

The aim of this research is to analyze the management of the Gema Nusa Cooperative in an effort to improve the welfare of its members and support local socio-religious organizations. The research method used is a qualitative approach with a case study design. Data collection was conducted through interviews, observations, and documentation studies. Data analysis includes three data processing flows, namely data reduction, data presentation, and conclusion drawing. Data validation uses data triangulation, which is a data collection technique that combines various existing data and sources. The results show that Gema Nusa Cooperative has implemented management functions quite well, but still has some weaknesses in terms of strategic planning and supervision. Therefore, Gema Nusa Cooperative needs to develop a comprehensive organizational strategic planning, develop an effective and efficient organizational structure, and apply the principles of good cooperative governance that includes a performance monitoring and evaluation system, performance feedback, and responsive and collaborative problem solving. Collectively, with the increase in management capacity, it is expected that Gema Nusa Cooperative can be more optimal in improving the welfare of its members.

Retrospective Analysis of Glacial Lake Outburst Flood (GLOF) Using AI Earth InSAR and Optical Images: A Case Study of South Lhonak Lake, Sikkim

On 4 October 2023, a glacier lake outburst flood (GLOF) occurred at South Lhonak Lake in the northwest of Sikkim, India, posing a severe threat to downstream lives and property. Given the serious consequences of GLOFs, understanding their triggering factors is urgent. This paper conducts a comprehensive analysis of optical imagery and InSAR deformation results to study changes in the surrounding surface of the glacial lake before and after the GLOF event. To expedite the processing of massive InSAR data, an InSAR processing system based on the SBAS-InSAR data processing flow and the AI Earth cloud platform was developed. Sentinel-1 SAR images spanning from January 2021 to March 2024 were used to calculate surface deformation velocity. The evolution of the lake area and surface variations in the landslide area were observed using optical images. The results reveal a significant deformation area within the moraine encircling the lake before the GLOF, aligning with the area where the landslide ultimately occurred. Further research suggests a certain correlation between InSAR deformation results and multiple factors, such as rainfall, lake area, and slope. We speculate that heavy rainfall triggering landslides in the moraine may have contributed to breaching the moraine dam and causing the GLOF. Although the landslide region is relatively stable overall, the presence of a crack in the toparea of landslide raises concerns about potential secondary landslides. Our study may improve GLOF risk assessment and management, thereby mitigating or preventing their hazards.

Rapid Classification and Diagnosis of Gas Wells Driven by Production Data

Conventional gas well classification methods cannot provide effective support for gas well routine management, and suffer from poor timeliness. In order to guide the on-site operation in liquid loading gas wells and improve the timeliness of gas well classification, this paper proposes a production data-driven gas well classification method based on the LDA-DA (Linear Discriminant Analysis–Discriminant Analysis) combination model. In this method, considering the requirements of routine management, gas wells are evaluated from two aspects: liquid drainage capacity (LDC) and liquid production intensity (LPI), and are classified into six types. Domain knowledge is used to perform the feature engineering on the on-site production data, and five features are set up to quantitatively evaluate the gas well and to create classification samples. On this basis, in order to specify the optimal data processing flow to establish the gas well classification map, four linear dimensionality reduction techniques, LDA, PCA, LPP, and ICA, are used to reduce the dimensionality of original classification samples, and then, four classical classification algorithms, NB, DA, KNN, and SVM, are trained and evaluated on the low-dimensional samples, respectively. The results show that the LDA space achieves the optimal sample separation and is chosen as the decision space for gas well classification. The DA algorithm obtains the top performance, i.e., the highest Average Macro F1-score of 95.619%, in the chosen decision space, and is employed to determine the classification boundaries in the decision space. At this point, the LDA-DA combination model for sample data processing is developed. Based on this model, gas well classification maps can be established by data mining, and the rapid evaluation and diagnosis of gas wells can be achieved. This method realizes instant and efficient production data-driven gas well classification, and can provide timely decision-making support for gas well routine management. It introduces new ideas for performing gas well classification, expanding the content and scope of the classification work, and presenting valuable insights for further research in this field.

SIMPLE: Efficient Temporal Graph Neural Network Training at Scale with Dynamic Data Placement

Dynamic graphs are essential in real-world scenarios like social media and e-commerce for tasks such as predicting links and classifying nodes. Temporal Graph Neural Networks (T-GNNs) stand out as a prime solution for managing dynamic graphs, employing temporal message passing to compute node embeddings at specific timestamps. Nonetheless, the high CPU-GPU data loading overhead has become the bottleneck for efficient training of T-GNNs over large-scale dynamic graphs. In this work, we present SIMPLE, a versatile system designed to address the major efficiency bottleneck in training existing T-GNNs on a large scale. It incorporates a dynamic data placement mechanism, which maintains a small buffer space in available GPU memory and dynamically manages its content during T-GNN training. SIMPLE is also empowered by systematic optimizations towards data processing flow. We compare SIMPLE to the state-of-the-art generic T-GNN training system TGL on four large-scale dynamic graphs with different underlying T-GNN models. Extensive experimental results show that SIMPLE effectively cuts down 80.5% ~ 96.8% data loading cost, and accelerates T-GNN training by 1.8× ~ 3.8× (2.6× on average) compared to TGL.

MASH-Ocean 1.0: Interactive platform for investigating microbial diversity, function, and biogeography with marine metagenomic data.

A large number of oceanic metagenomic data and environmental metadata have been published. However, most studies focused on limited ecosystems using different analysis tools, making it challenging to integrate these data into robust results and comprehensive global understanding of marine microbiome. Here, we constructed a systematic and quantitative analysis platform, the Microbiome Atlas/Sino-Hydrosphere for Ocean Ecosystem (MASH-Ocean: https://www.biosino.org/mash-ocean/), by integrating global marine metagenomic data and a unified data processing flow. MASH-Ocean 1.0 comprises 2147 metagenomic samples with five analysis modules: sample view, diversity, function, biogeography, and interaction network. This platform provides convenient and stable support for researchers in microbiology, environmental science, and biogeochemistry, to ensure the integration of omics data generated from hydrosphere ecosystems, to bridge the gap between elusive omics data and biological, ecological, and geological discovery, ultimately to foster the formation of a comprehensive atlas for aquatic environments.

Courier Service System

The demand for efficient courier services is growing rapidly, and traditional systems often struggle to meet these demands. This project introduces an Enhanced Courier Service System Application, utilizing cutting-edge technologies such as ReactJS for the frontend, Java Spring Boot for the backend, and MongoDB for the database. The primary objective is to develop a robust and scalable courier service system that offers enhanced features and functionalities to meet the evolving needs of users. The system architecture facilitates seamless communication between frontend, backend, and database components, ensuring smooth data flow and efficient processing. ReactJS for the frontend enhances user experience, while Java Spring Boot provides scalability, security, and performance. MongoDB, a NoSQL database, offers flexibility and scalability for large data storage and retrieval. The modular design allows for easy maintenance and future enhancements, adapting to changing requirements and technologies. By incorporating industry-standard practices like UML diagrams, data flow diagrams, and sample test cases, the project aims to deliver a high-quality, reliable, and user-friendly courier service system application.

Dynamic trust security approach for edge computing-based mobile IoT devices using artificial intelligence

The Industrial Internet of Things (IIoT) encompasses various applications such as smart cities, healthcare, and agriculture, which rely on a multitude of interconnected devices. These applications demand specific performance requirements, including low latency and sufficient bandwidth. Edge computing has emerged as a promising solution to address the challenges associated with data flow processing. Particularly, in scenarios like smart cities, where a massive number of IIoT devices are deployed, edge computing serves as the primary data processing layer. However, the limited resources of edge computing, compared to cloud computing, pose significant security concerns for IIoT applications. In fact, this paper proposes a novel security system to secure the relation between the IIoT devices and edge computing. Dynamic Trust Security Approach (DTSA) leverages Artificial Intelligence to analyze, evaluate and enhance the security and privacy aspects of the IIoT devices’ interaction with edge computing. DTSA improve the security level of the IIoT system and improve the performance with more than 45% compare to the traditional method.

Research on fusion and deformation analysis of decade' HLS-II control network surveying data

To ensure the long-term stability of the beam and further construction needs, it is necessary to re-measure the particle accelerator regularly. Since 2013, we have been measuring the HLS-II control network with laser trackers and the level for 10 years. Taking HLS-II as the object of study, we introduced the data acquisition strategy for the period of 2013–2023. We analyzed the horizontal and vertical deformation trends of each control point. In the vertical direction, we fused the geometric level and elevation data of the laser tracker network by using MC-GMM (a Gauss-Markov model which considers the uncertainty of Monte Carlo simulation). The analysis results show that, after the HLS upgrade, the annual average settlement of the ground control network tends to be stable. The settlement and horizontal variation of the device are all within a reasonable and controllable range. To ensure stability, accuracy, and efficiency in the measurement, we recommend employing the laser tracker for the re-measurement of the HLS-II storage ring. The data processing flow and methods mentioned in the article can provide a reference for deformation analysis of the upcoming construction of the Hefei Advanced Light Facility (HALF) and other large-scale metrology projects.

Design and Architecture of BenguetFresh: An IoT-Enabled Data-Driven System for Sustainable Agriculture

Agriculture plays a vital role in the Philippines, contributing significantly to its economy and providing livelihoods to a substantial portion of the population. Benguet province, known as the "Salad Bowl of the Philippines" for its favorable climate and fertile soil, exemplifies thriving agriculture-based livelihoods. However, challenges persist in the sector, with the low compliance rate of Philippine Good Agricultural Practices (PhilGAP) certification highlighting the need for innovative solutions to enhance sustainable practices, market access, and overall agricultural quality for Benguet farmers.
 This study presents the design and architecture of BenguetFresh a system designed to integrate data-driven monitoring and management practices with sustainable agriculture, while adhering to PhilGAP standards. Through stakeholder engagement, interviews, and systematic review, specific requirements and challenges of farmers in Benguet were identified and addressed.
 BenguetFresh employs cutting-edge technology, Arduino-based smart agriculture monitoring systems, enabling real-time data collection on environmental factors. Data analytics tools process this data to provide valuable insights, facilitating informed decision-making and resource optimization.
 The scalable architecture ensures efficient data flow and real-time processing, while user-friendly interfaces empower farmers with real-time and historical data visualization and data-driven recommendations. Furthermore, the integration of e-commerce functionality fosters direct connections between farmers and consumers, expanding market reach and improving economic outcomes.
 BenguetFresh's design and architecture serve as a blueprint for implementing data-driven, sustainable agriculture. The potential benefits extend to both farmers and consumers, ensuring high-quality, responsibly produced agricultural products. Embracing cutting-edge technology and industry standards, BenguetFresh paves the way for efficient, sustainable, and prosperous agriculture.

Processes of Creating a National Cancer Registry System in the Philippines

AbstractAs a new entrant in the Philippine Cancer Health Information Network, the Philippine Cancer Center (PCC) was pushed to establish the National Cancer Registry as mandated by the National Integrated Cancer Control Act (NICCA). The researchers used a rapid review and rapid realist review approaches which made use of evidence from literature and data from 43 key informant interviews and FGDs to create the National Cancer Registry Development Plan. The results were summarized using a cancer data process flow diagram, context‐view system architecture model, and logic models detailing the roadmap plans for the next 10 years. The study was able to show how the development of a national cancer registry plan that seeks to unite existing population‐based cancer registries (PBCRs) and hospital‐based cancer registries (HBCRs) is only possible through extensive collaboration and effective stakeholder engagement. Using an evidence‐based approach with expert knowledge and validation, the Philippine National Cancer Registry Development Plan was created, an example of an e‐health policy making process from which other developing countries can learn from.

Optimizing event-based neural networks on digital neuromorphic architecture: a comprehensive design space exploration.

Neuromorphic processors promise low-latency and energy-efficient processing by adopting novel brain-inspired design methodologies. Yet, current neuromorphic solutions still struggle to rival conventional deep learning accelerators' performance and area efficiency in practical applications. Event-driven data-flow processing and near/in-memory computing are the two dominant design trends of neuromorphic processors. However, there remain challenges in reducing the overhead of event-driven processing and increasing the mapping efficiency of near/in-memory computing, which directly impacts the performance and area efficiency. In this work, we discuss these challenges and present our exploration of optimizing event-based neural network inference on SENECA, a scalable and flexible neuromorphic architecture. To address the overhead of event-driven processing, we perform comprehensive design space exploration and propose spike-grouping to reduce the total energy and latency. Furthermore, we introduce the event-driven depth-first convolution to increase area efficiency and latency in convolutional neural networks (CNNs) on the neuromorphic processor. We benchmarked our optimized solution on keyword spotting, sensor fusion, digit recognition and high resolution object detection tasks. Compared with other state-of-the-art large-scale neuromorphic processors, our proposed optimizations result in a 6× to 300× improvement in energy efficiency, a 3× to 15× improvement in latency, and a 3× to 100× improvement in area efficiency. Our optimizations for event-based neural networks can be potentially generalized to a wide range of event-based neuromorphic processors.

Imaging tree root systems using ground penetrating radar (GPR) data in Brazil

Trees sequester carbon dioxide from the atmosphere through photosynthesis, storing it in branches, stems, and roots, where the belowground carbon fraction, approximately ¼ of the total amount, exhibits significant interspecies root biomass variability. Estimating the amount of carbon stored in tree roots of different species is key to understanding an important aspect of climate change and exploring how natural forests, urban tree planting policies, and reforestation projects might help to address it. In this context, one of the most prominent Non-Destructive Testing methods capable of estimating the diameter and length of roots at different depths is ground penetrating radar (GPR). It has been widely used for geological, archaeological, and geotechnical studies due to its accuracy in locating buried material in different contexts, although standards for the correct management of datasets related to belowground root systems are still been developed. This paper reports a GPR signal processing flow to estimate the root diameter of three species of tropical forest trees, and to demonstrate the method’s viability, a dataset was collected in a study area with a 900 MHz shielded antenna. A multi-stage data processing flow is then presented, including raw data, file format conversion, zero-time adjustment, background removal, signal gain, Stolt FK migration, and time-to-depth conversion with hyperbolic adjustment velocity. The resulting data were converted from true amplitude data to a trace envelope. High amplitudes on the envelope section, with lateral continuity in parallel sections, were interpreted as roots. However, the interpretation of outcomes encounters notable complexities, primarily attributable to the intricate nature of subsurface root architectures, the soil matrix characterized by significant clay content, and the co-occurrence of buried materials proximate to the arboreal subjects. Consequently, amplitudes discerned within ground penetrating radar (GPR) 2D sections necessitate cautious interpretation, as they are not immediately indicative of subsurface roots. To overcome this difficulty, this study used direct measurements of the roots in the field, to confirm the GPR data. Despite these complexities, the study demonstrates GPR’s efficacy, particularly in the uppermost soil layer-a pivotal carbon reservoir with a 96% correlation (R2) between GPR-derived coarse-root diameter estimates and field measurements.

Assessing the Reproducibility of Research Based on the Food and Drug Administration Manufacturer and User Facility Device Experience Data.

This article aims to assess the reproducibility of Manufacturer and User Facility Device Experience (MAUDE) data-driven studies by analyzing the data queries used in their research processes. Studies using MAUDE data were sourced from PubMed by searching for "MAUDE" or "Manufacturer and User Facility Device Experience" in titles or abstracts. We manually chose articles with executable queries. The reproducibility of each query was assessed by replicating it in the MAUDE Application Programming Interface. The reproducibility of a query is determined by a reproducibility coefficient that ranges from 0.95 to 1.05. This coefficient is calculated by comparing the number of medical device reports (MDRs) returned by the reproduced queries to the number of reported MDRs in the original studies. We also computed the reproducibility ratio, which is the fraction of reproducible queries in subgroups divided by the query complexity, the device category, and the presence of a data processing flow. As of August 8, 2022, we identified 523 articles from which 336 contained queries, and 60 of these were executable. Among these, 14 queries were reproducible. Queries using a single field like product code, product class, or brand name showed higher reproducibility (50%, 33.3%, 31.3%) compared with other fields (8.3%, P = 0.037). Single-category device queries exhibited a higher reproducibility ratio than multicategory ones, but without statistical significance (27.1% versus 8.3%, P = 0.321). Studies including a data processing flow had a higher reproducibility ratio than those without, although this difference was not statistically significant (42.9% versus 17.4%, P = 0.107). Our findings indicate that the reproducibility of queries in MAUDE data-driven studies is limited. Enhancing this requires the development of more effective MAUDE data query strategies and improved application programming interfaces.

Accelerating Medicines Partnership® Schizophrenia (AMP® SCZ): Rationale and Study Design of the Largest Global Prospective Cohort Study of Clinical High Risk for Psychosis.

This article describes the rationale, aims, and methodology of the Accelerating Medicines Partnership® Schizophrenia (AMP® SCZ). This is the largest international collaboration to date that will develop algorithms to predict trajectories and outcomes of individuals at clinical high risk (CHR) for psychosis and to advance the development and use of novel pharmacological interventions for CHR individuals. We present a description of the participating research networks and the data processing analysis and coordination center, their processes for data harmonization across 43 sites from 13 participating countries (recruitment across North America, Australia, Europe, Asia, and South America), data flow and quality assessment processes, data analyses, and the transfer of data to the National Institute of Mental Health (NIMH) Data Archive (NDA) for use by the research community. In an expected sample of approximately 2000 CHR individuals and 640 matched healthy controls, AMP SCZ will collect clinical, environmental, and cognitive data along with multimodal biomarkers, including neuroimaging, electrophysiology, fluid biospecimens, speech and facial expression samples, novel measures derived from digital health technologies including smartphone-based daily surveys, and passive sensing as well as actigraphy. The study will investigate a range of clinical outcomes over a 2-year period, including transition to psychosis, remission or persistence of CHR status, attenuated positive symptoms, persistent negative symptoms, mood and anxiety symptoms, and psychosocial functioning. The global reach of AMP SCZ and its harmonized innovative methods promise to catalyze the development of new treatments to address critical unmet clinical and public health needs in CHR individuals.