Large Data Flow Research Articles

Cyber Resilience Revisited: Law and International Relations

Cyber space is increasingly playing an important role in the global world, affecting the pattern of relations between countries. The issue of non-traditional security threats is shifting towards the threat typology associated with cyber space. The concept of national security began to be complemented by a national cyber security strategy to support the security of its national interests. Where the country needs to ensure the security of cyber ecosystems to maintain national economic stability. The large flow data and information increasingly large and complex, and brings hidden costs in the form of cyber security threats. Cyber security concepts that are considered not responsive and resilient in dealing with and overcoming cyber attacks can occur at any time with patterns and types that continue to evolve. The concept of cyber security should begin to be developed into cyber security that has patterns of recovery, adaptation, and evolution so as to be able to answer the dynamics of challenges in international trade. Interrupted cyber systems in international sphere will potentially cause disruption of international relations because the threat of losses caused not only affects one country. The increasingly complex international law context and involving big data should be one of the top priorities for cyber resilience strategies. This paper starts by explaining the state of play of cyber resilience in international relations and law. Next, analysis why the concept of cyber resilience in the perspective of international relations and international law needs to be re-visited to face challenges in the digital economy.

Availability- and Traffic-Aware Placement of Parallelized SFC in Data Center Networks

Network Function Virtualization (NFV) brings flexible provisioning and great convenience for enterprises outsource their network functions to the Data Center Networks (DCNs). Network service in NFV is deployed as a Service Function Chain (SFC), which includes an ordered set of Virtual Network Functions (VNFs). However, in one SFC, the SFC delay increases linearly as the length of SFC increases. SFC parallelism can achieve high performance of SFC. In this article, we focus on the parallelized SFC placement problem in DCN considering availability guarantee and resource optimization. Firstly, we define the parallelized SFC and propose a multi-flow backup model. The parallelized SFC consists of multiple parallelized sub-SFCs. We split large data flow into multiple small sub-flows, each of them can be transmitted in one sub-SFC. The backup model provides backup sub-SFCs for working sub-SFCs to improve availability. Finally, we design three placement strategies and a Hybrid Placement Algorithm (HPA) aimed at mapping SFCs to DCN. Evaluation results show that our proposed solutions outperform the related work. We can reduce SFC delay (30%) and optimize link consumption (reduce 40%) while guaranteeing availability (99.999%).

20 µs-resolved high-throughput X-ray photon correlation spectroscopy on a 500k pixel detector enabled by data-management workflow.

The performance of the new 52 kHz frame rate Rigaku XSPA-500k detector was characterized on beamline 8-ID-I at the Advanced Photon Source at Argonne for X-ray photon correlation spectroscopy (XPCS) applications. Due to the large data flow produced by this detector (0.2 PB of data per 24 h of continuous operation), a workflow system was deployed that uses the Advanced Photon Source data-management (DM) system and high-performance software to rapidly reduce area-detector data to multi-tau and two-time correlation functions in near real time, providing human-in-the-loop feedback to experimenters. The utility and performance of the workflow system are demonstrated via its application to a variety of small-angle XPCS measurements acquired from different detectors in different XPCS measurement modalities. The XSPA-500k detector, the software and the DM workflow system allow for the efficient acquisition and reduction of up to ∼109 area-detector data frames per day, facilitating the application of XPCS to measuring samples with weak scattering and fast dynamics.

Research of multiclass fuzzy classification of traffic for attacks identification in the networks

Currently, data mining methods such as neural networks, decision trees, genetic algorithms, restricted search algorithms, evolutionary programming, reasoning systems based on similar cases, rule induction, analysis with selective action, logical regression, algorithms for determining associations and sequences, data visualization, combined methods are actively used in various specialized areas. The introduction of analytical methods of data mining is primarily aimed at adapting existing solutions to solve new problems related to the informatization of business processes. One of the actively developing areas that use data mining and artificial intelligence methods is network security. To identify and detect anomalies in networks, it is most effective to create profiles of data flow behavior depending on current conditions. In this work, we developed a method that allows us to identify Exploits, Fuzzers, and Generic attacks based on multiclass fuzzy classification. In the experimental part of the study, it was found that the proposed solution is comparable in accuracy to the Naïve Bayes, SVM, and KNN methods, but it has higher performance and less resource consumption for large data flows. This is quite effective for networks with many devices.

Research on QoS routing method based on NSGAII in SDN

Software Defined Network (SDN) makes routing decisions according to the shortest path. This routing scheme does not consider link information such as link residual bandwidth, forwarding delay, packet loss rate and so on. In the case of large data flow, it is easy to cause network congestion, thus reducing the network quality of service assurance. For this reason, using the ability of SDN network controller to monitor the underlying network information and control the route forwarding, we try to set multiple objective functions according to different QoS parameters, and use the non-dominated sorting multi-objective optimization genetic algorithm (NSGA II) based on the elite strategy to make routing decisions. Compared with other routing algorithms, the proposed multi-objective optimization genetic algorithm reduces forwarding delay and packet loss rate, so can effectively prevent network congestion.

A Single-Shot Generalized Device Placement for Large Dataflow Graphs

With increasingly complex neural network architectures and heterogeneous device characteristics, finding a reasonable graph partitioning and device placement strategy is challenging. There have been prior attempts at learned approaches for solving device placement, these approaches are computationally expensive, unable to handle large graphs consisting over 50000 nodes, and do not generalize well to unseen graphs. To address all these limitations, we propose an efficient single-shot, generalized deep RL method (SGDP) based on a scalable sequential attention mechanism over a graph neural network that is transferable to new graphs. On a diverse set of representative deep learning models, our method on average achieves 20% improvement over human placement and 18% improvement over the prior art with 15× faster convergence. We are the first to demonstrate super human performance on 8-layer recurrent neural network language model and 8-layer GNMT consisting of over 50000 nodes, on 8-GPUs. We provide rationales and sensitivity study on model architecture selections.

Ten-Year Assessment of IASI Radiance and Temperature

The Infrared Atmospheric Sounding Interferometers (IASIs) are three instruments flying on board the Metop satellites, launched in 2006 (IASI-A), 2012 (IASI-B), and 2018 (IASI-C). They measure infrared radiance from the Earth and atmosphere system, from which the atmospheric composition and temperature can be retrieved using dedicated algorithms, forming the Level 2 (L2) product. The operational near real-time processing of IASI data is conducted by the EUropean organisation for the exploitation of METeorological SATellites (EUMETSAT). It has improved over time, but due to IASI’s large data flow, the whole dataset has not yet been reprocessed backwards. A necessary step that must be completed before initiating this reprocessing is to uniformize the IASI radiance record (Level 1C), which has also changed with time due to various instrumental and software modifications. In 2019, EUMETSAT released a reprocessed IASI-A 2007–2017 radiance dataset that is consistent with both the L1C product generated after 2017 and with IASI-B. First, this study aimed to assess the changes in radiance associated with this update by comparing the operational and reprocessed datasets. The differences in the brightness temperature ranged from 0.02 K at 700 cm−1 to 0.1 K at 2200 cm−1. Additionally, two major updates in 2010 and 2013 were seen to have the largest impact. Then, we investigated the effects on the retrieved temperatures due to successive upgrades to the Level 2 processing chain. We compared IASI L2 with ERA5 reanalysis temperatures. We found differences of ~5–10 K at the surface and between 1 and 5 K in the atmosphere. These differences decreased abruptly after the release of the IASI L2 processor version 6 in 2014. These results suggest that it is not recommended to use the IASI inhomogeneous temperature products for trend analysis, both for temperature and trace gas trends.

Pipeline Synthesis and Optimization from Branched Feedback Dataflow Programs

Large dataflow designs are a result of behavioral specification of modern complex digital systems and/or a result of unfolding and transforming looped and branched programs. Since deep-submicron silicon technology provides large amounts of available resources, pipelining optimization without (or with minimal) resource sharing can give significant advantages in performance. High-level synthesis of CAL-programs is particularly popular in computation intensive applications (e.g., image and video processing, cryptography, wireless communication, etc.) where feedback actors with data flows at input and output ports represent loop-like behavior. In this work, we propose techniques for transforming, analysis, speculatively pipelining and optimizing large branched feedback dataflow programs. We develop an accurate algorithm and introduce fast dynamic and mixed static / dynamic heuristics that firstly minimize the number of pipeline stages for a given pipeline-stage time-period, and secondly minimize the overall pipeline registers size by means of appropriate assignment of feedbacks and instructions to pipeline stages. We also propose a genetic algorithm for tuning the heuristics for a particular design. The experimental results show the algorithms we propose give quickly solutions that are very close to accurate solutions and overcomes the earlier developed algorithms regarding computing time and pipeline parameters.

Comprehensive analysis of drugs to treat SARS‑CoV‑2 infection: Mechanistic insights into current COVID‑19 therapies (Review).

The major impact produced by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) focused many researchers attention to find treatments that can suppress transmission or ameliorate the disease. Despite the very fast and large flow of scientific data on possible treatment solutions, none have yet demonstrated unequivocal clinical utility against coronavirus disease 2019 (COVID-19). This work represents an exhaustive and critical review of all available data on potential treatments for COVID-19, highlighting their mechanistic characteristics and the strategy development rationale. Drug repurposing, also known as drug repositioning, and target based methods are the most used strategies to advance therapeutic solutions into clinical practice. Current in silico, in vitro and in vivo evidence regarding proposed treatments are summarized providing strong support for future research efforts.

A Review on Inter-Satellite Links Free Space Optical Communication

Objectives: The need for constricting, lightweight, low-power electronics-aggravated by the thriving demand for large data flow rate and bandwidth is urging the use of optical technologies in defense and space. Free space optical (FSO) communication links have promising eminence over microwave communication links with high speed, low weight, and increased security so that they can be boarded conveniently on the satellite. Methods/findings: The advancement of the optical components and systems has made it very predominant along with its acceptability in space and defense. Yet a couple of significant components to be dealt with are tracking problems with misalignment of the transmitter and receiver apertures and the progressions because of barometrical conditions. Efficiency of the FSO communication system is seriously rotted due to the angle of arrival (AoA) variance and including error pointing. The optical transmitter can be perfectly coordinated with the receiver by utilizing the pointing, acquisition, and tracking (PAT). However, this method is to be carried out under the airborne platform with required restrictions namely size, weight, and power. Application: An overview of promising inter-satellite applications with the challenges have been discussed which sway the performance of the system and make the communication link unattainable. The FSO communication possesses an advantage over two existing technologies optical fiber communication and wireless communication. Hence, the significant use of FSO is intersatellite optical wireless communication (Is OWC) links, which will be deployed in the future in space. Keywords: Angle-of-arrival Variance (AoA), Free-space Optical Communication, Pointing, Acquisition, and Tracking (PAT)

Extrapolation methods for fixed‐point multilinear PageRank computations

SummaryNonnegative tensors arise very naturally in many applications that involve large and complex data flows. Due to the relatively small requirement in terms of memory storage and number of operations per step, the (shifted) higher order power method is one of the most commonly used technique for the computation of positive Z‐eigenvectors of this type of tensors. However, unlike the matrix case, the method may fail to converge even for irreducible tensors. Moreover, when it converges, its convergence rate can be very slow. These two drawbacks often make the computation of the eigenvectors demanding or unfeasible for large problems. In this work, we consider a particular class of nonnegative tensors associated with the multilinear PageRank modification of higher order Markov chains. Based on the simplified topological ε‐algorithm in its restarted form, we introduce an extrapolation‐based acceleration of power method type algorithms, namely, the shifted fixed‐point method and the inner‐outer method. The accelerated methods show remarkably better performance, with faster convergence rates and reduced overall computational time. Extensive numerical experiments on synthetic and real‐world datasets demonstrate the advantages of the introduced extrapolation techniques.

Techniques for the Discovery of Patterns Hidden in Large Data Sets and Process Flow of Data Mining

Data mining procedures are used to find the concealed or brand new patterns to keep the information. We know that data mining can utilize every sector like service, farming, advertising and marketing etc. There are numerous methods for data mining like clustering, distinction, and so on. There are a variety of techniques and also procedures of data mining which could be used on information to build up a brand new setting to enhance the efficiency of existing data and aid to make the brand-new prophecies on the records. Within this newspaper, our team provide procedures for the discovery of patterns concealed in sizable records sets, paying attention to problems relating to their usefulness, effectiveness, effectiveness, and scalability.

Technique of Informative Features Selection in Geoacoustic Emission Signals

Studies of geoacoustic emission in a seismically active region in Kamchatka show that geoacoustic signals produce pronounced pulse anomalies during the earthquake preparation and post-seismic relaxation of the local stresses field at the observation point. The qualitative selection of such anomalies is complicated by a strong distortion and weakening of the signal amplitude. A review of existing acoustic emission analysis methods shows that most often researchers turn to the analysis of more accessible to study statistical properties and energy of signals. The distinctive features of the approach proposed by the authors are the extraction of informative features based on the analysis of time and frequency-time structures of geoacoustic signals and the description of various forms of recognizable pulses by a limited pattern set. This study opens up new ideas to develop methods for detecting anomalous behavior of geoacoustic signals, including anomalies before earthquakes.
 The paper describes a technique of information extraction from geoacoustic emission pulse streams of sound frequency range. A geoacoustic pulse mathematical model, reflecting the signal generation process from a variety of elementary sources, is presented. A solution to the problem of detection of geoacoustic signal informative features is presented by the means of description of signal fragments by the matrixes of local extrema amplitude ratios and of interval ratios between them. The result of applying the developed algorithm to describe automatically the structure of the detected pulses and to form a pattern set is shown. The patterns characterize the features of geoacoustic emission signals observed at IKIR FEB RAS field stations. A technique of reduction of the detected pulse set dimensions is presented. It allows us to find patterns similar in structure. A solution to the problem of processing of a large data flow by unifying pulses description and their systematization is proposed. A method to identify a geoacoustic emission pulse model using sparse approximation schemes is suggested. An algorithmic solution of the problem of reducing the computational complexity of the matching pursuit method is described. It is to include an iterative refinement algorithm for the solution at each step in the method. The results of the research allowed the authors to create a tool to investigate the dynamic properties of geoacoustic emission signal in order to develop earthquake prediction detectors.

Introduce new paper-based outpatient morbidity recording system to outpatient departments of public hospitals in Sri Lanka: feasibility study

Background: Effective evaluation of public health status is reinforced by description of the current health status of the community, ability to detect changes in this status and prediction of trends. Morbidity and mortality are often recorded in patients presenting to hospitals. There are more than 1,100 government owned hospitals in Sri Lanka and these institutions have been providing services to average 50 million out-patients per annum. Morbidity data of patients attending Out Patient Department (OPD) of these health facilities are not routinely recorded. We developed a new paper based OPD morbidity recording system adapted with International Classification of Primary Care (ICPC-2) coding and tested its feasibility in various hospitals. Methods: We used an intervention study design composed of three phases. Pre-intervention was to develop the new paper based OPD morbidity recording system. During the second phase, new recording system was tested in 3 hospitals, which were selected randomly from each hospital level: a primary care hospital, a divisional hospital and a base hospital located in Colombo district. New recording system implemented for a month period in selected three hospitals. We measured the feasibility over the process and acceptability of participants for the new recording system in various aspects during the post-intervention phase. Results: New paper based OPD morbidity recording system consisted with an OPD form, an OPD return and a Tally sheet and data flow was outlined in the pre-intervention. Feasibility assessment showed high perceived satisfaction to layout (97%), timeliness (97.5%), technical competency (75%) and training & orientation provided (98%). All hospitals showed more than 90% of overall completeness for the new OPD at registration table. Although overall completeness of entries made at consultation table had shown more than 68% completeness, it is declining when hospital becomes larger. High accuracy (99%) is shown in direct entering ICPC-2 codes and 98% in overall satisfaction of MO/OPD implied that new OPD recording system is acceptable. The preparation of new OPD return and tally sheet was also well accepted by medical record officers (MRO). Conclusions: New paper based OPD morbidity recording system was accepted as user-friendly and feasible to be implemented in OPD settings and medical record offices in tested hospitals. The readiness of central medical statistic unit for large OPD data flow should be assessed and possibility of incorporating IT solutions in future is recommended.

Integration of Mobile Robot Control System Data Using the Extended Kalman Filter

Purpose of research. The article deals with the adaptation of the algorithm of the extended Kalman filter for the integration of data from sensors of physical values of a mobile robotMethods. Integration of data is the process of information (data) fusion for determination or prediction of the state of an object. Integration provides increased robustness of robot control and accuracy of machine perception of information. This process is similar to repeated experiments in order to determine in direct and/or indirect ways the value of a physical quantity with the required accuracy. In the control system of a mobile robot, the integration of sensor data is carried out by one or more computing devices (for example, processors or microcontrollers) [1-5].Results. Advances in digital signal processing and image processing are based on new algorithms, increasing the speed of data processing by computing devices and increasing the speed of access to data stored in storage (storage devices) and the capacity of the latter. Computing devices also perform averaging and filtering of signals of individual sensors and their further matching. The problem of sustainable integration and processing of information from different measuring devices can be solved with the help of the Kalman filter algorithm. The Kalman linear filter algorithm and, in particular, the extended Kalman filter algorithm perform a large amount of computation in the course of their work. In comparison with the linear Kalman filter, the extended Kalman filter significantly increases the requirements for the computing power of the onboard computer (computing device, computer) of amobile robot.Conclusion. The main effect of integration is to obtain fundamentally new information that cannot be obtained from individual sensors. This approach relieves data channels of large (excessive) data flows coming directly from the sensors, and reduces the requirements for computing power of the computing device of the upper level of the structure of the mobile robot control system.

A prototype framework for parallel visualization of large flow data

Scientific visualization seeks to provide deep insight into the complex pattern underlying big data, while flow visualization plays a crucial role in oceanographic-atmospheric modeling and computational fluid dynamics simulation. As an increasingly important strategy, parallel visualization incorporates data visualization with parallel computing by means of MPI (Message Passing Interface) to achieve efficient visual analysis to facilitate scientific study. This paper presents a prototype framework for parallel visualization of large flow data, involving MPI as the low-level parallel computing paradigm, DIY (Do It Yourself) as a block-oriented data-parallel programming library on top of MPI, OSUFlow as a geometry-based flow visualization engine, and VTK (Visualization Toolkit) for data input, graphics rendering, and scene interaction. It exposes the combined power of DIY and OSUFlow, including parallel yet seamless generation of streamlines as well as pathlines from vector data defined on Cartesian, rectilinear, and curvilinear grids, to a broad community of high-performance flow visualization through VTK. Preliminary results show that this framework is capable of exploiting the horsepower of a vast number of processors to accelerate data processing and visualization for explorative analysis of massive steady/unsteady volume flows.

Dynamic load balancing algorithm for large data flow in distributed complex networks

Abstract Information society brings convenience to people, but also produces a lot of data. Relational databases are not suitable for processing big data due to architecture defects. The most commonly used system to store and process large amounts of data is the NoSQL (Not only Structured Query Language) database. Obviously, it is very important to cooperate with these independent computers to accomplish processing tasks efficiently, which is the function of load balancing. This paper studies the commonly used NoSQL database and load balancing algorithms, and designs and implements a more efficient load balancing algorithm. By introducing the relationship between nodes and the children of their brother nodes, we reduce the height of the whole sorted binary tree. The time cost of the algorithm is reduced versus the commonly used weighted polling algorithm O(N) to O(log N), while the spatial cost remains unchanged. The equalization algorithm synthetically utilizes the characteristics of big data processing systems and has good performance. At the same time, the algorithm can quickly find the sub-optimal nodes when the optimal nodes have been occupied, so it is very suitable for load balancing in highly concurrent systems. Finally, the effectiveness of the proposed load balancing algorithm is verified by simulation.

Rules engine and complex event processor in the context of internet of things for precision agriculture

The Internet of Things (IoT) applications monitor large data flows and events in real time, some raw data is captured from devices located in wireless sensor networks (WSN) and used to make control decisions about actuators. This can be a major problem when the devices grow in number as well as the data that is captured. In this paper, we propose an architecture called “RECEP” for the dynamic processing of events generated in the context of IoT and Precision Agriculture (PA); it is made up of two components: Rules Engine (RE) and Complex Event Processor (CEP). RE allows you to configure dynamic rules conditioning input data from different sources and planning control actions on actuators, alerts, and notifications for end users or applications. The CEP component fuses the input data at the rate at which they arrive, with the rules established in the RE and it performs a prescriptive analysis that consists not only in predicting or detecting patterns of events, but in making automatic decisions. RECEP was implemented in a virtual machine with a 1.9 GHz CPU and 6 GB RAM, then it was integrated into an intelligent irrigation system of an experimental banana plot located in Machala-Ecuador. A WSN simulator was also used to generate sensor data in large quantities, the CEP was evaluated with several test cases, and results show that it consumes computational resources with a growth trend, represented by a logarithmic regression model (r-squared > 0.9); that is, the more events are processed, there is a minimum consumption of resources. It was tested for fifteen days; around 25 thousand events/s were processed. Our RECEP can be implemented in low-cost infrastructure typical of small and large banana producers.

A Multi-Cooperative-Based Approach to Manage Communication in Wireless Instrumentation Systems

Wireless instrumentation systems are a collection of resource-constrained nodes in charge of sensing, processing, and transmitting data. They are often battery powered, and consequently, subject to energy constraints. Since communication is the most energy-consuming task, efforts have to be made during the routing process in order to maximize the network lifespan and avoid network partitioning. In this context, the multi-wireless-agent communication (MWAC) model provides a self-organization process for managing communication in this kind of instrumentation system. Given an organization and a node, MWAC determines a single path to route messages to the destination. This may cause congestions for large data flows. This paper introduces antMWAC, a model designed to improve MWAC. Ant colony optimization is used to balance the traffic load on wireless nodes and insure a multipath routing. Ants interact with nodes to make the communication more efficient. The cooperation between them is increased to find a tradeoff between new path discovering and path reinforcement. The aim is to allow the nodes exploiting the information routed by ants and modifying ant decision parameters. Experiments show that antMWAC diversifies the nodes participating in routing operations, reducing node congestion.

A Robust False Data Detection in Smart Grid Using Trees Decision Algorithm

The smart grid has the objective of improving the control and the supervision of the electric grid with the aim to ensure a reliable and effective functioning; This is possible with a system based on a precise state estimate that accurately reflects the physical aspects of power grids. However, the progress of the grid element, basically the smart meter has considerably increased the number of access point to the electrical grid. As a result, a large flow of false data is generated in the electrical system that compromises the state estimation process. Unlike the conventional detection test, we propose a method capable of handling complex false data. Our analyzes are validated by experiments on a physical bus feeding system performed on PSS / E, then we use Matlab to build false data based on the Jacobian. The collected data is analyzed with a learning technique basing on decision tree method.