Heterogeneous Environmental Data Research Articles

Crop and Weed Segmentation and Fractal Dimension Estimation Using Small Training Data in Heterogeneous Data Environment

The segmentation of crops and weeds from camera-captured images is a demanding research area for advancing agricultural and smart farming systems. Previously, the segmentation of crops and weeds was conducted within a homogeneous data environment where training and testing data were from the same database. However, in the real-world application of advancing agricultural and smart farming systems, it is often the case of a heterogeneous data environment where a system trained with one database should be used for testing with a different database without additional training. This study pioneers the use of heterogeneous data for crop and weed segmentation, addressing the issue of degraded accuracy. Through adjusting the mean and standard deviation, we minimize the variability in pixel value and contrast, enhancing segmentation robustness. Unlike previous methods relying on extensive training data, our approach achieves real-world applicability with just one training sample for deep learning-based semantic segmentation. Moreover, we seamlessly integrated a method for estimating fractal dimensions into our system, incorporating it as an end-to-end task to provide important information on the distributional characteristics of crops and weeds. We evaluated our framework using the BoniRob dataset and the CWFID. When trained with the BoniRob dataset and tested with the CWFID, we obtained a mean intersection of union (mIoU) of 62% and an F1-score of 75.2%. Furthermore, when trained with the CWFID and tested with the BoniRob dataset, we obtained an mIoU of 63.7% and an F1-score of 74.3%. We confirmed that these values are higher than those obtained by state-of-the-art methods.

Edge Federated Optimization for Heterogeneous Data

This study focuses on optimizing federated learning in heterogeneous data environments. We implement the FedProx and a baseline algorithm (i.e., the FedAvg) with advanced optimization strategies to tackle non-IID data issues in distributed learning. Model freezing and pruning techniques are explored to showcase the effective operations of deep learning models on resource-constrained edge devices. Experimental results show that at a pruning rate of 10%, the FedProx with structured pruning in the MIT-BIH and ST databases achieved the best F1 scores, reaching 96.01% and 77.81%, respectively, which achieves a good balance between system efficiency and model accuracy compared to those of the FedProx with the original configuration, reaching F1 scores of 66.12% and 89.90%, respectively. Similarly, with layer freezing technique, unstructured pruning method, and a pruning rate of 20%, the FedAvg algorithm effectively balances classification performance and degradation of pruned model accuracy, achieving F1 scores of 88.75% and 72.75%, respectively, compared to those of the FedAvg with the original configuration, reaching 56.82% and 85.80%, respectively. By adopting model optimization strategies, a practical solution is developed for deploying complex models in edge federated learning, vital for its efficient implementation.

Koncepcja ujednolicenia wzajemnie niekompatybilnych modeli informacyjnych oraz danych przechowywanych w relacyjnych bazach danych zarządców dróg

Road Administrations (RAs) implement Building Information Modelling (BIM) through pilot projects developed for new or reconstructed structures. Each model is processed with respect to BIM standards and practices valid at the time of its creation. Consequently, models are incompatible and cannot be interconnected to create a combined model of the managed network or even its selected parts. Existing structures are often not included in the BIM effort until some major repair is planned. In addition, RAs usually store data on fixed and variable parameters of structures in relational databases. This results in a situation in which a relatively small number of structures are included in mutually incompatible models and data regarding the majority of structures is contained in relational databases. It creates a heterogeneous data environment for RAs. The goals of the paper are as follows: to analyse the described problem, to propose a method of model unification models, a method of creation of simplified compatible information models using data on existing structures stored in relational databases and a method of storing data at the level of the managed network, to support RA asset management systems which can be treated as a dynamic part of BIM.

Ambient-aware continuous aid for mountain rescue activities

Ambient-awareness in conjunction with pervasive computing is a significant challenge for system designers. It follows the necessity of gathering raw, massive and heterogeneous environmental data which we obtained, while middleware processes must merge context modelling and reasoning seamlessly. We proposed a system supporting mountain rescuers which is demanding due to the large number of environmental objects interacting, as well as high data variability. We presented complex context processing embedded in the proposed context life cycle and implemented it in a difficult mountain environment. We introduced five weather scenarios which are a basis for contextual and perceptual processing during the validation of our model. The system merges a message streaming broker for massive data transport, low and high-level processing algorithms, repositories and a logical SAT solver. It constitutes a Context-Aware-as-a-Service (CAaaS) system, offering advanced support for mountain rescue operations. The provided software model defines middleware components which act on a predicted context and transform in situ sensor data into smart decisions, and which could operate as a platform-based cloud computing model. It is an enabler yielding a synergy effect with different software components orchestration when providing pro-activeness and non-intrusiveness concerning smart decisions.

A multimodal dual-fusion entity extraction model for large and complex devices

In the context of large and complex devices with a multi-source heterogeneous data environment, the extraction of network device configuration related entity information from diverse modalities of the Internet of Things data is a crucial and fundamental step towards establishing a domain knowledge graph for global Zero Touch Provisioning of network devices. In this paper, we present a novel multimodal dual-fusion entity extraction model that serves as a foundation for an intelligent and efficient network device configuration process. Firstly, the multimodal data is encoded, followed by the ViLBERT pre-training model to obtain more feature information of entities for multimodal front-end fusion. Next, the attention weights of each modal feature are learned through a multilayer neural network classifier and probabilistic graphical model, facilitating multimodal back-end fusion and reducing information redundancy. Finally, entity recognition is accomplished by employing a cohesive memory module that extracts the essential parameters for device configuration. The simulation results demonstrate that the proposed model performs exceptionally well on the MSCOCO2017 public dataset and the SFZK-Dev data network device dataset, with F1 values of the comprehensive evaluation index of model quality at 94.65% and 96.94%, respectively, indicating high stability. Additionally, the link prediction metrics HITS@1 achieved accuracy levels of 58.21% and 68.04%.

Processing toxic metal source proxies appropriately for better spatial heterogeneity source apportionment

Soil toxic metals have strong spatial heterogeneity, and their sources vary among regions. Thus, this study integrated the Catreg and geographically weighted regression (GWR) models to quantitatively extract the main source proxies (numerical and categorical variables were analyzed simultaneously) for different toxic metals and analyze the spatial heterogeneity of the distributions of these sources. Pb, Cd and Hg were the predominant toxic metals in soil. Of the samples with Pb, Cd and Hg, 84.12 %, 68.03 % and 41.57 % exceeded the background values, and 5.36 %, 6.42 % and 5.43 % were moderately contaminated according to the geoaccmulation index, respectively. Industrial activities, with relative importance values of 17.82 %, 31.54 % and 26.51 % for Cd, Hg and Pb, respectively, were the predominant source of these metals especially, in their high-content cluster areas (central urban areas). Soil available phosphorus was another important factor (relative importance values of 13.03 %, 13.41 % and 25.55 % for Cd, Hg and Pb, respectively), and agricultural activities (especially the overuse of phosphoric fertilizers) were identified as an anthropogenic source of these toxic metals. Soil parent material had the greatest influence on As and Cr, with relative importance values of 19.88 % and 19.09 %, respectively, especially in their high-content accumulation area (the eastern coastal area), indicating that these toxic metals mainly come from natural sources. Slope had important impacts on toxic metal accumulation (relative importance values of 17.48 %, 21.22 %, 12.40 % and 16.13 % for Cd, Hg, Cr and As, respectively) by influencing industrial distribution and pollutant migration. By changing the soil adsorption capacity, soil organic matter (explaining 13.01 % of Pb) and soil pH (explaining 14.58 % of As and 12.40 % of Cr) strongly influenced toxic metal accumulation. This study highlights the benefits of the integrated Catreg-GWR model for analyzing multiple spatially heterogeneous environmental data types (numerical and categorical variables), providing a potential foundation for local pollution prevention.

MG-TAR: Multi-View Graph Convolutional Networks for Traffic Accident Risk Prediction

Due to the continuing colossal socio-economic losses caused by traffic accidents, it is of prime importance to precisely forecast the traffic accident risk to reduce future accidents. In this paper, we use dangerous driving statistics from driving log data and multi-graph learning to enhance predictive performance. We first conduct geographical and temporal correlation analyses to quantify the relationship between dangerous driving and actual accidents. Then, to learn various dependencies between districts besides the traditional adjacency matrix, we simultaneously model both static and dynamic graphs representing the spatio-temporal contextual relationships with heterogeneous environmental data, including the dangerous driving behavior. A graph is generated for each type of the relationships. Ultimately, we propose an end-to-end framework, called MG-TAR, to effectively learn the association of multiple graphs for accident risk prediction by adopting multi-view graph neural networks with a multi-attention module. Thorough experiments on ten real-world datasets show that, compared with state-of-the-art methods, MG-TAR reduces the error of predicting the accident risk by up to 23% and improves the accuracy of predicting the most dangerous areas by up to 27%.

A survey on techniques, methods and security approaches in big data healthcare

A huge percentage of people especially in developed countries spend a good chunk of their wealth in managing their health conditions. In order to adequately administer healthcare, governments and various organizations have embraced advanced technology for automating the health industry. In recent past, electronic health records have largely been managed by Enterprise Resource Planning and legacy systems. Big data framework steadily emerge as the underlying technology in healthcare, which offers solutions that limits capacity of others systems in terms of storage and reporting. Automation through cloud services supported by storage of structured and unstructured health data in heterogeneous environment has improved service delivery, efficiency, medication, diagnosis, reporting and storage in healthcare. The argument support the idea that big data healthcare still face information security concern, for instance patient image sharing, authentication of patient, botnet, correlation attacks, man-in-the-middle, Distributed Denial of Service (DDoS), blockchain payment gateway, time complexities of algorithms, despite numerous studies conducted by scholars in security management for big data in smart healthcare. Some security technique include digital image encryption, steganography, biometrics, rule-based policy, prescriptive analysis, blockchain contact tracing, cloud security, MapReduce, machine-learning algorithms, anonymizations among others. However, most of these security solutions and analysis performed on structured and semi-structured data as opposed to unstructured data. This may affect the output of medical reporting of patients’ condition particularly on wearable devices and other examinations such as computerized tomography (CT) Scans among others. A major concern is how to identify inherent security vulnerabilities in big healthcare, which generate images for transmission and storage. Therefore, this paper conducted a comparative survey of solutions that specifically safeguards structured and unstructured data using systems that run on big data frameworks. The literature highlights several security advancements in cryptography, machine learning, anonymization and protocols. Most of these security frameworks lacks implementation evidence. A number of studies did not provide comprehensive performance metrics (accuracy, error, recall, precision) of the models besides using a single algorithm without validated justification. Therefore, a critique on the contribution, performance and areas of improvements discussed and summarized in the paper.

Cross-platform Tech University Courses Teaching in a Heterogeneous Data Environment

After analyzing the pros and cons of existing online education platforms, we find that the disaggregation between multiple supplementary education platforms has many inconveniences for technical course teaching. We propose and design a multiway collaboration model in a heterogeneous data environment to make the multi-platform teaching approach more convenient. This model helps to address the limitations of the multi-platform teaching and learning approach.

A sustainable deep learning framework for fault detection in 6G Industry 4.0 heterogeneous data environments

The integration of 5G and Beyond 5G (B5G)/6G in Machine-to-Machine (M2M) communications, is making Industry 4.0 smarter. However, the goal of having a sustainable self-monitored industry has not been reached yet. State-of-the-art deep learning-based Fault Detection algorithms cannot handle heterogeneous data, meaning that more than one fault detection computational device has to be used for each data format, in addition to the inability to take advantage of the combination of all the information available in different formats to derive more accurate conclusions. Moreover, these algorithms rely on inefficient hyper-parameters tuning strategies. In this paper, we propose an Advanced Deep Learning framework for Fault Diagnosis in Industry 4.0 (ADL-FDI4), which combines Long Short Term Memory (LSTM), Convolutional Neural Networks (CNN) and graph CNN (GNN), to handle heterogeneous data. Furthermore, our novel framework uses a Branch-and-Bound procedure to guide the learning process. Our experimental results show that ADL-FDI4 outperforms the state-of-the-art solutions in terms of detection rate and running time, and for that, it consumes less energy. In addition to handling heterogeneous data, which implies that one computational device is sufficient to handle all data formats.

An environmental exploration system for visual scenario analysis of regional hydro-meteorological systems

We propose a framework for the 3D exploration of heterogeneous environmental data within a geographical region of interest. Based on Unity, the system can be built for a variety of platforms and works both on regular machines as well as in virtual reality environments, where it can be considered a basis for a virtual geographic environment. Focussing on the catchment of the Müglitz River in south-eastern Germany, a large collection of observation data acquired via a wide range of measurement devices has been integrated in a geographical reference frame for the region. Results of area-wide numerical simulations for both groundwater and soil moisture have been added to the scene and allow for the exploration of the delayed consequences of transient phenomena such as heavy rainfall events and their impact on the catchment scale. This study focusses on the concurrent visualisation and synchronised animation of multiple area wide datasets from different environmental compartments. The resulting application allows to explore the region of interest during specific hydrological events for an assessment of the interrelation of processes. As such, it offers the opportunity for knowledge transfer between researchers of different domains as well as for outreach to an interested public.

NIFL: A Statistical Measures-Based Method for Client Selection in Federated Learning

Federated learning (FL) has been proposed as a machine learning approach to collaboratively learn a shared prediction model. Although, during FL training, only a subset of workers participate in each round, existing approaches introduce model bias when considering the average of local model parameters of heterogeneous workers, which degrades the accuracy of the learned global model. In this paper, we introduce NIFL, a new strategy for worker selection that handles the statistical challenges of FL when local data is Non-Independent and Identically Distributed (N-IID). In NIFL, the server starts sending the signal to the workers that react by sending the number of their samples. The server then selects a percentage of workers with the highest number of samples and requests data statistics such as mean and standard deviation. After that, the server calculates our proposed N-IID index, based on the statistical information collected from the workers without having access to their data, and uses this index as a criterion for worker selection. Finally, the server broadcasts the global model to the selected workers. NIFL takes into account the disparity in the distribution of workers’ data in order to improve the performance of the model in heterogeneous data environment. We have performed several experiments with N-IID data. The obtained results show that both the convergence of our method and the test accuracy increased considerably comparing to the other techniques while keeping a reasonable computation and communication costs.

Empirical Measurement of Client Contribution for Federated Learning With Data Size Diversification

Client contribution evaluation is crucial in federated learning(FL) to effectively select influential clients. Contrary to data valuation in centralized settings, client contribution evaluation in FL faces a lack of data accessibility and consequently challenges stable quantification of the impact of data heterogeneity. To address this instability of client contribution evaluation, we introduce an empirical method, Federated Client Contribution Evaluation through Accuracy Approximation(FedCCEA), which exploits data size as a tool for client contribution evaluation. After several FL simulations, FedCCEA approximates the test accuracy using the sampled data size and extracts the client contribution from the trained accuracy approximator. In addition, FedCCEA grants <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">data size diversification</i> , which reduces the massive variation in accuracy resulting from game-theoretic strategies. Several experiments have shown that FedCCEA strengthens the robustness to diverse heterogeneous data environments and the practicality of partial participation.

A Proposal for Sharing Software Process Provenance Data in Heterogeneous Environments

A Proposal for Sharing Software Process Provenance Data in Heterogeneous Environments

Heterogeneous Clutter Suppression for Airborne Radar STAP Based on Matrix Manifolds

Clutter suppression in heterogeneous environments is a serious challenge for airborne radar. To address this problem, a matrix-manifold-based clutter suppression method is proposed. First, the distributions of training data in heterogeneous environments are analyzed, while the received data are characterized on a Riemannian manifold of Hermitian positive definite matrices. It is indicated that the training data with different distributions with the same power are separated, whereas data with the same distribution are closer together. This implies that the underlying geometry of the data can be better revealed by manifolds than by Euclidean space. Based on these properties, homogeneous training data are selected by establishing a binary hypothesis test such that the negative effects of the use of heterogeneous samples are alleviated. Moreover, as exploiting a geometric metric on manifolds to reveal the underlying information of data, experimental results on both simulated and real data validate that the proposed method has a superior performance with small sample support.

WCL: Client Selection in Federated Learning with a Combination of Model Weight Divergence and Client Training Loss for Internet Traffic Classification

Internet traffic classification (TC) is a critical technique in network management and is widely applied in various applications. In traditional TC problems, the edge devices need to send the raw traffic data to the server for centralized processing, which not only generates a lot of communication overhead but also leads to the privacy leakage and information security issues. Federated learning (FL) is a new distributed machine learning paradigm that allows multiple clients to train a global model collaboratively without raw traffic data sharing. The TC in a FL framework preserves the user privacy and data security by keeping the raw traffic data local. However, because of the different user behaviours and user preferences, traffic data heterogeneity emerges. The existing FL solutions introduce bias in model training by averaging the local model parameters from all heterogeneous clients, which degrades the classification accuracy of the learnt global classification model. To improve the classification accuracy in heterogeneous data environment, this paper proposes a novel client selection algorithm, namely, WCL, in federated paradigm based on a combination of model weight divergence and local model training loss. Extensive experiments on the public traffic dataset QUIC and ISCX have proved that the WCL algorithm obtains, compared to CMFL, superior performance in improving model accuracy and convergence speed on low heterogeneous traffic data and high heterogeneous traffic data, respectively.

Dynamic Fusion Algorithm of Building Surface Data in Heterogeneous Environment

The existing building surface data fusion algorithms do not extract the segmented data features, resulting in inaccurate fusion results. In heterogeneous environment, a Clustering Fusion Algorithm Based on mutual information and fractal dimension is proposed. The regression coefficient is used to express the sequence, and the data feature representation and data dimension reduction are realized. The dynamic data series are processed by similarity measure function method. For the long dynamic data series, the piecewise aggregation approximation method is used to segment the data and then extract the features. Through the incremental clustering processing data based on fractal dimension clustering algorithm, the research of data fusion algorithm is realized. The experimental results show that the accuracy of building surface data fusion is greatly improved by using the dynamic data fusion algorithm, the highest is 0.98, the sum of square error is reduced, and the lowest is only 90.44.

Об опыте комплексного картографирования экологической обстановки урбанизированных территорий с учётом пространственного распространения загрязняющих веществ

The high level of environment pollution is specific to urbanized territories. The efficient solution of city ecological problems requires considering urban environment as a dynamic system and taking its development over time and space into account. This can be achieved through a complex analysis of time and space dynamics of pollutants in accordance with natural conditions and city infrastructure. The study determines an approach to the environmental assessment of the urbanized territory based on geoinformation analysis and 3D cartographic modelling time and space dynamics of pollutants spreading in the urban environment. The approach to assessing it, taking into account the patterns of pollution dissemination is substantiated. The logical sequence of integrating heterogeneous environmental data for pollution distribution modelling is offered. The digital relief and terrain models of Novosibirsk city are formed. The environmental database of Novosibirsk city is developed. The city area zoning by composite index regarding environmental risk is given. A 3D model of pollution dissemination is developed.

Deep Correlation Mining Based on Hierarchical Hybrid Networks for Heterogeneous Big Data Recommendations

The advancement of several significant technologies, such as artificial intelligence, cyber intelligence, and machine learning, has made big data penetrate not only into the industry and academic field but also our daily life along with a variety of cyber-enabled applications. In this article, we focus on a deep correlation mining method in heterogeneous big data environments. A hierarchical hybrid network (HHN) model is constructed to describe multitype relationships among different entities, and a series of measures are defined to quantify the internal correlations within one specific layer or external correlations between different layers. An intelligent router based on deep reinforcement learning framework is designed to generate optimal actions to route across the HHN. An improved random walk with the restart-based algorithm is then developed with the intelligent router, based on the hierarchical influence across network associated with multiple correlations. An intelligent recommendation mechanism is finally designed and applied to support users' collaboration works in scholarly big data environments. Experiments based on DBLP and ResearchGate data show the practicability and usefulness of our model and method.

Enhancing Hadoop Performance in Homogenous and Heterogeneous Big Data Environments by Dynamic Slot Configuration

Hadoop is one of the most famous platform solutions for processing large volume and scale of data in parallel processing in Cloud computing. A Hadoop system can be characterized based on three main factors: cluster, workload and user. Each of these factors can be described as either heterogeneous or homogenous, which reflects the heterogeneity degree of the Hadoop systemThe objective of this proposed research work is to investigate the degree of influence of heterogeneity for each of these factors on the performance of Hadoop based on different schedulers. Three schedulers are considered with different levels of Hadoop heterogeneity and are tested and analyzed: the first algorithm considered is the FIFO (First in First out), the second is the Fair sharing, and the final is the COSHH (Classification and Optimization based Scheduler for Heterogeneous Hadoop). Performance issues are related to Hadoop schedulers and comparative performance analysis between different cases of jobs submission. These jobs are processed in different homogenous or heterogeneous data environments and under fixed or reconfigurable slot between map and reduce tasks for Hadoop MapReduce java programming clustering model. The results showed that when assigning tunable knob between map and reduce tasks under certain schedulers like FIFO algorithm, the performance enhanced significantly especially in cases of heterogeneity environment where the workload decreased significantly and the utilization of computational resources increase was obvious.