Data Processing Flow Research Articles

Intelligent English translation system based on evolutionary multi-objective optimization algorithm

The difficulty of obtaining the characteristics of the corpus database of neural machine translation is a factor hindering its development. In order to improve the effect of English intelligent translation, based on the machine learning algorithm, this paper improves the multi-objective optimization algorithm to construct a model based on the English intelligent translation system. Moreover, this paper uses parallel corpus and monolingual corpus for model training and uses semi-supervised neural machine translation method to analyze the data processing path in detail and focuses on the analysis of node distribution and data processing flow. In addition, this paper introduces data-related regularization items through the probabilistic nature of the neural machine translation model and applies it to the monolingual corpus to help the training of the neural machine translation model. Finally, this paper designs experiments to verify the performance of this model. The research results show that the translation model constructed in this paper is highly intelligent and can meet actual translation needs.

Flexible Production Data Generator for Manufacturing Companies

Advances in technology over last decades have been followed by many challenges of industrial competitiveness at different levels. Production planning and control generally demand too much time to data capture, and it makes it difficult to understand the analysis of the process flow and to obtain satisfactory results. These results can help operations managers make the best decisions and implement them. Companies that do not have tracking systems to automatically capture data from their machines, spend a lot of time trying to achieve a visual correlation which is consistent with the reality of their processes and goals. Companies without this technology can use data generator systems to simulate data variables in different sizes and previously identify suitable graphical correlations. Current’s data generators work with several variables for a lot of situations. On the other hand, it can become confusing and error-prone due to excess of information. However, there is no automated data generator focused on process-flow data which has usability enough to meet the manufacturing company’s needs. This study proposes a flexible production-oriented data generator for different physical arrangements in manufacturing companies, along with graphical correlations of production parameters to aid planning and decision making.

A Standardized ICS Network Data Processing Flow With Generative Model in Anomaly Detection

Industrial control systems (ICS) now usually connect to Wireless Sensor Networks and the Internet, exposing them to security threats resulting from cyber-attacks. However, detecting such attacks is non-trivial task. The high-dimensional network data pose significant challenges on security anomaly detection. In this work, we propose a network flow data processing method, which can make the complex network data more standardized and unified to assist security anomaly detection. Then, data generation method is applied to collect enough training data. We also propose a evaluation method for generated data. Finally, the bidirectional recurrent neural networks with attention mechanism is proposed to extract the latent feature, and give an explainable results in identifying the dominant attributes. Empirical results show our method outperforms the state-of-the-art models.

Techniques for the Discovery of Patterns Hidden in Large Data Sets and Process Flow of Data Mining

Data mining procedures are used to find the concealed or brand new patterns to keep the information. We know that data mining can utilize every sector like service, farming, advertising and marketing etc. There are numerous methods for data mining like clustering, distinction, and so on. There are a variety of techniques and also procedures of data mining which could be used on information to build up a brand new setting to enhance the efficiency of existing data and aid to make the brand-new prophecies on the records. Within this newspaper, our team provide procedures for the discovery of patterns concealed in sizable records sets, paying attention to problems relating to their usefulness, effectiveness, effectiveness, and scalability.

Robust Frame Synchronization Scheme for Continuous-Variable Quantum Key Distribution with Simple Process

In continuous-variable quantum key distribution (CVQKD) systems, high-quality data synchronization between two legitimate parties, Alice and Bob, is the premise of the generation of shared secret keys. Synchronization with specially designed frames is an efficient way, but it requires special modulating devices to generate these special frames. Moreover, the extra requirement of special modulating devices makes it technically impossible for some passive preparation schemes. We propose a novel approach to realize synchronization in this paper, which is different from those special-frame-based methods. In our proposed scheme, Alice publishes parts of the original signals as the synchronization frames and Bob takes these frames to perform the synchronization algorithm. Besides, a synchronization feature is applied to deal with phase shifts. The simulation results based on practical data demonstrate that the proposed synchronization scheme not only maintains a high success rate but simplifies the data processing flow at the same time, which dramatically reduces the computational complexity.

Enhancing stratigraphic, structural and dissolution features in GPR images of carbonate karst through data processing

ABSTRACTObtaining high‐quality ground penetrating radar (GPR) images in karst is difficult because materials resulting from the weathering of carbonate rocks might be electrically conductive. As a consequence, penetration depth and signal resolution might be greatly reduced due to attenuation. In addition, fractures and faults might cause a significant amount of electromagnetic wave scattering. We present a 2D data processing flow which allows improving the quality of GPR images in carbonate karst. The processing flow is composed of the following steps: obtaining a zero‐offset section by removing the direct wave, low‐frequency noise removal, geometrical spreading and exponential gain compensation, spectral balancing, Kirchhoff migration, bandpass filtering, amplitude‐volume enhancement, and topographic correction. For a 200‐MHz dataset, we present in detail each step of the processing flow, exemplifying how to parameterize every step. Spectral balancing is of key importance because it can approximately replenish the high‐frequency content lost due to propagation effects. In this step, we recommend to shift the centroid frequency as much as possible to high‐frequency values, even exceeding the nominal value of the antenna center frequency, but still looking for a band‐limited spectrum as the goal. Despite the difficulty of migrating GPR data, we show that migration (even assuming a constant velocity) might enhance the lateral continuity of the reflection events and allows identification of discontinuities such as faults and fractures. If imaged in a better way, these structures can have special importance as they are often the boundaries of dissolution features. Obtaining images based on amplitude‐volume enhancement techniques allows to better visualize karst voids and deep‐rooted discontinuities because these features are often associated with low‐amplitude zones, which are highlighted in such images. Due to this processing flow, stratigraphic, structural and dissolution features can be enhanced, allowing the interpreter to establish spatial and genetic associations among these elements to obtain a better understanding of the karst formation process.

A Synergetic Orchestration of Objects, Data, and Services to Enable Smart Cities

Smart cities (SCs), as a novel solution built on top of large-scale Internet of Things (IoT) systems, experiences a rapid growth worldwide, in which, a synergetic orchestration among objects, data, and services is emphasized for innovative solutions to elevate the intelligence of cities based on the fusion of multisource and multimodal data. To enable such orchestration, this article proposes a smart service orchestration architecture (SSOA) to coordinate ubiquitous objects, create interlinked data, and implement versatile smart services. As a proof of concept of SSOA, an informed design platform (IDP) is presented to demonstrate how a smart service system can be designed and how a synergetic orchestration can be implemented to support an informed place design. Moreover, two dedicated mechanisms, namely, place utilization analysis mechanism (PUAM) and ensemble-based activity detection mechanism (EADM), are implemented in a multisource data processing flow to illustrate how massive geo-referenced data can be analyzed effectively and efficiently by machine learning algorithms to extract key information for a comprehensive data fusion required in using multimodal IoT systems. As evaluated, PUAM running in a distributed environment can dramatically improve the performance of geospatial clustering about 11 times from the baseline 53.6 to 4.7 s, and EADM with an ensemble activity classifier achieves the highest accuracy about 87.7% and also the highest f-score per activity category. Finally, various insights about the project testbed Jurong East, Singapore, are discussed to reveal its place design context.

Hydrodynamic object identification with artificial neural models

The lateral-line system that has evolved in many aquatic animals enables them to navigate murky fluid environments, locate and discriminate obstacles. Here, we present a data-driven model that uses artificial neural networks to process flow data originating from a stationary sensor array located away from an obstacle placed in a potential flow. The ability of neural networks to estimate complex underlying relationships between parameters, in the absence of any explicit mathematical description, is first assessed with two basic potential flow problems: single source/sink identification and doublet detection. Subsequently, we address the inverse problem of identifying an obstacle shape from distant measures of the pressure or velocity field. Using the analytical solution to the forward problem, very large training data sets are generated, allowing us to obtain the synaptic weights by means of a gradient-descent based optimization. The resulting neural network exhibits remarkable effectiveness in predicting unknown obstacle shapes, especially at relatively large distances for which classical linear regression models are completely ineffectual. These results have far-reaching implications for the design and development of artificial passive hydrodynamic sensing technology.

I can't believe it's not (only) software!

There is a steady increase in the size of data stored and processed as part of data science applications, leading to bottlenecks and inefficiencies at various layers of the stack. One way of reducing such bottlenecks and increasing energy efficiency is by tailoring the underlying distributed storage solution to the application domain, using resources more efficiently. We explore this idea in the context of a popular column-oriented storage format used in big data workloads, namely Apache Parquet. Our prototype uses an FPGA-based storage node that offers high bandwidth data deduplication and a companion software library that exposes an API for Parquet file access. This way the storage node remains general purpose and could be shared by applications from different domains, while, at the same time, benefiting from deduplication well suited to Apache Parquet files and from selective reads of columns in the file. In this demonstration we show, on the one hand, that by relying on the FPGA's dataflow processing model, it is possible to implement in-line deduplication without increasing latencies significantly or reducing throughput. On the other hand, we highlight the benefits of implementing the application-specific aspects in a software library instead of FPGA circuits and how this enables, for instance, regular data science frameworks running in Python to access the data on the storage node and to offload filtering operations.

The Contemporary Ethical and Privacy Issues of Smart Medical Fields

With the development of big data technology, such as data mining and data matching, many industries have started a revolution, including medical field. Big data not only strengthens the accuracy of medical diagnosis, but it also enhances the efficiency of the entire medical system and relevant medical staff. Additionally, with the rethinking of innovation, the application of wearable intelligent device, RFID technology and sensor technology play positive roles in promoting medical interaction between hospital system and wearer. Smart medical provides effective methods for individual health management and promotes the progress of medical information. However, there are also some inevitable ethical problems, e.g., the leakage of privacy information, which cannot be avoided to some extents. The authors recommend some suggestions to reduce the possibilities of ethical problems happened during the data flow process.

An osmotic computing infrastructure for urban pollution monitoring

SummaryUrban pollution control systems suffer from the presence of fixed stations in a greater number than mobile monitoring devices. Data gathered from such stations provide detailed and reliable information, thanks to equipment quality and effective measuring protocols, but these sampled data are gathered from very limited areas and through discontinuous monitoring campaigns. Fortunately, the spread of technologies for mobility has fostered the development of new approaches like mobile crowdsensing (MCS), increasing the chances of using mobile devices, even personal ones, as suitable sensors for the urban monitoring scenario. Nevertheless, one of the open challenges is the management of integrated heterogeneous data flows, differing in terms of typology, technical specifications (eg, transmission protocols), and semantics. The osmotic computing paradigm aims at creating an abstract level between mobile devices/Internet‐of‐Things devices and a cloud platform, which enables opportunistic filtering and the addition of metadata for improving the data processing flow. This work focuses on the design and development of a middleware that integrates data coming from mobile and Internet‐of‐Things devices specifically deployed in urban contexts using the osmotic computing paradigm. Moreover, a component of the osmotic membrane has been developed for security management.

Scientific Data Management in the Age of Big Data: An Approach Supporting a Resilience Index Development Effort.

The increased availability of publicly available data is, in many ways, changing our approach to conducting research. Not only are cloud-based information resources providing supplementary data to bolster traditional scientific activities (e.g., field studies, laboratory experiments), they also serve as the foundation for secondary data research projects such as indicator development. Indicators and indices are a convenient way to synthesize disparate information to address complex scientific questions that are difficult to measure directly (e.g., resilience, sustainability, well-being). In the current literature, there is no shortage of indicator or index examples derived from secondary data with a growing number that are scientifically focused. However, little information is provided describing the management approaches and best practices used to govern the data underpinnings supporting these efforts. From acquisition to storage and maintenance, secondary data research products rely on the availability of relevant, high-quality data, repeatable data handling methods and a multi-faceted data flow process to promote and sustain research transparency and integrity. The U.S. Environmental Protection Agency recently published a report describing the development of a climate resilience screening index which used over one million data points to calculate the final index. The pool of data was derived exclusively from secondary sources such as the U.S. Census Bureau, Bureau of Labor Statistics, Postal Service, Housing and Urban Development, Forestry Services and others. Available data were presented in various forms including portable document format (PDF), delimited ASCII and proprietary format (e.g., Microsoft Excel, ESRI ArcGIS). The strategy employed for managing these data in an indicator research and development effort represented a blend of business practices, information science, and the scientific method. This paper describes the approach, highlighting key points unique for managing the data assets of a smaller scale research project in an era of "big data."

Spatial Interaction Modeling of OD Flow Data: Comparing Geographically Weighted Negative Binomial Regression (GWNBR) and OLS (GWOLSR)

Due to the emergence of new big data technology, mobility data such as flows between origin and destination areas have increasingly become more available, cheaper, and faster. These improvements to data infrastructure have boosted spatial and temporal modeling of OD (origin-destination) flows, which require the consideration of spatial dependence and heterogeneity. Both ordinary least square (OLS) and negative binomial (NB) regression methods have been used extensively to calibrate OD flow models by processing flow data as different types of dependent variables. This paper aims to compare both global and local spatial interaction modeling of OD flows between traditional and geographically weighted OLS (GWOLSR) and NB (GWNBR) modeling methods. From this study with empirical data it is concluded that GWNBR outperforms GWOLSR in reducing spatial autocorrelation and in detecting spatial non-stationarity. Although, it is noted that both local modeling methods show improvement when compared against the equivalent global models.

MP23: Giving medical students what they deserve - a rigorous, equitable and defensible CaRMS selection process

Innovation Concept: The fairness of the Canadian Residency Matching Service (CaRMS) selection process has been called into question by rising rates of unmatched medical students and reports of bias and subjectivity. We outline how the University of Saskatchewan Royal College emergency medicine program evaluates CaRMS applications in a standardized, rigorous, equitable and defensible manner. Methods: Our CaRMS applicant evaluation methods were first utilized in the 2017 CaRMS cycle, based on published Best Practices, and have been refined yearly to ensure validity, standardization, defensibility, rigour, and to improve the speed and flow of data processing. To determine the reliability of the total application scores for each rater, single measures intraclass correlation coefficients (ICCs) were calculated using a random effects model in 2017 and 2018. Curriculum, Tool or Material: A secure, online spreadsheet was created that includes applicant names, reviewer assignments, data entry boxes, and formulas. Each file reviewer entered data in a dedicated sheet within the document. Each application was reviewed by two staff physicians and two to four residents. File reviewers used a standardized, criterion-based scoring rubric for each application component. The file score for each reviewer-applicant pair was converted into a z-score based on each reviewer's distribution of scores. Z-scores of all reviewers for a single applicant were then combined by weighted average, with the group of staff and group of residents each being weighted to represent half of the final file score. The ICC for the total raw scores improved from 0.38 (poor) in 2017 to 0.52 (moderate) in 2018. The data from each reviewer was amalgamated into a master sheet where applicants were sorted by final file score and heat-mapped to offer a visual aid regarding differences in ratings. Conclusion: Our innovation uses heat-mapped and formula-populated spreadsheets, scoring rubrics, and z-scores to normalize variation in scoring trends between reviewers. We believe this approach provides a rigorous, defensible, and reproducible process by which Canadian residency programs can appraise applicants and create a rank order list.

Recent progress in phylogenomic methods

Phylogenomics is the intersection of the fields of phylogenetics and genomics, aiming to use genome-scale molecular data to resolve the phylogenetic relationships among organisms. In this review, we briefly introduced five methods of data collection in phylogenomics by high-throughput sequencing: targeted amplicon sequencing, transcriptome sequencing, reduced-genome-representation sequencing, target sequence capture, and low coverage whole-genome shotgun sequencing. We summarized and compared the characteristics, applications and costs of each method. In addition, we reviewed the current work flow of data processing of a typical phylogenomic analysis, including five steps: data preprocessing and assembly, orthologous sequences identification, sequence alignment, data quality control and phylogenetic tree reconstruction. We also summarized the key points of each step and introduced software that are used in these steps. Phylogenomics is a promising way to reconstruct the Tree of Life. However, it also faces a series of problems at present. For example, the analysis of genome-scale data will lead to a significant increase in computation time, the impact of systematic error will become more serious when the data size increases and different data sets may yield strongly supported but inconsistent results. In the end of this review, we summarized and discussed the latest studies concerning these issues and made some suggestions to deal with these problems in the future.

Joint Task Assignment, Transmission, and Computing Resource Allocation in Multilayer Mobile Edge Computing Systems

In this paper, we propose a multilayer data flow processing system, i.e., EdgeFlow, to integrally utilize the computing capacity throughout the whole network, i.e., the cloud center (CC) on the top layer, the mobile edge computing (MEC) servers on the middle layer, and the edge devices (EDs) on the bottom layer. To realize the efficient data processing in EdgeFlow, we optimally assign the tasks to multiple layers, and allocate the wireless transmission resources between the MEC servers and EDs as well as the wired transmission resources between the CC and MEC servers. We prove that the system is naturally classified into two states, the nonblocking state and the blocking state, according to various data generation speed at the EDs. The system latency is minimized for the nonblocking state even though the problem is nonconvex. As for the blocking state, the recovery time is minimized through solving a min-max problem. Based on the analytical results, the EdgeFlow system is implemented on the universal software radio peripheral and the Intel next units of computing. A typical Internet of Things application, photo recording and face recognition, is used for the simulation and the experiment, and indicates that the EdgeFlow can achieve a low latency and recovery time than the previous distributed frameworks, e.g., the Cloudlet and the Markov decision process.

Visual Web-Oriented Environment of Dynamic Control of Data Flow in Campus of Software Defined Networks

Nowadays new innovative approaches based on the technology of software defined networks (SDN) are gaining popularity in the field of computer networks (CN). SDN provide a flexible approach to the processing and control of data flows in CN by separating the control plane and data plane, as well as centralizing the representation of the entire network. In this paper, we propose a software infrastructure and a visual web-oriented environment (SIVE) for dynamic control of data flows in campus SDN based on OpenFlow protocol. It was proposed to use the SIVE as an integrated segment of the campus network of Ryazan State Radio Engineering University. The aim of the work is the development of the SIVE architecture in the form of UML class diagram description, as well as the creation of software methods for organizing effective network interaction of various software systems in SDN based on OpenFlow protocol. A hardware-software test bench based on HP Aruba 2920-24G equipment was developed to confirm the efficiency and reliability of the proposed SIVE. The offered SIVE is the basis for the development of a large class of software systems and SDN components based on OpenFlow protocol.

EdgeFlow: Open-Source Multi-layer Data Flow Processing in Edge Computing for 5G and Beyond

Edge computing has evolved to be a promising avenue to enhance system computing capability by offloading processing tasks from the cloud to edge devices. In this article, we propose a multi-layer edge computing framework called EdgeFlow. In this framework, different nodes ranging from edge devices to cloud data centers are categorized into corresponding layers and cooperate for data processing. EdgeFlow can deal with the trade-off between the computing and communication capabilities so that the tasks can be assigned to each layer optimally. At the same time, resources are carefully allocated throughout the whole network to mitigate performance fluctuation. The proposed open-source data flow processing framework is implemented on a platform that can emulate various computing nodes in multiple layers and corresponding network connections. Evaluated on the face recognition scenario, EdgeFlow can significantly reduce task finish time and perform more tolerance to run-time variations, compared with pure cloud computing, pure edge computing, and Cloudlet. Potential applications of EdgeFlow, including network function virtualization, Internet of Things, and vehicular networks, are also discussed at the end of this article.

Methods of Preprocessing Tomographic Images Taking into Account the Thermal Instability of the X-ray Tube

For correct numerical interpretation of tomographic images, i.e., estimates of the attenuation coefficients of objects, it is important to obtain reconstruction of high quality, which depends directly on the methods of processing experimental data. Data processing flow begins with its preparation for the application of the reconstruction algorithm. The necessary part of data processing contains the subtraction of the black field, normalization considering empty data, and taking logarithm. This part is not sufficient for obtaining high-quality reconstruction when working with real data since it is not ideal. Real data include noise and distortions due to changes in the setup geometrical parameters during the experiment. We have analyzed two possible types of data distortions during experiment and suggested corrections for them. The first one corrects thermal shifts regarding beam decentralization, and the second eliminates the effect of the polychromatic nature of X-ray radiation on the results of tomographic reconstruction. These methods were tested with both real and synthetic data. Both synthetic and real experiments show that suggested methods improve the reconstruction quality. In real experiments, the level of agreement between the automatic parameter adjustment and experts is about 90%.

From BPMN process models to DMN decision models

The interplay between process and decision models plays a crucial role in business process management, as decisions may be based on running processes and affect process outcomes. Often process models include decisions that are encoded through process control flow structures and data flow elements, thus reducing process model maintainability. The Decision Model and Notation (DMN) was proposed to achieve separation of concerns and to possibly complement the Business Process Model and Notation (BPMN) for designing decisions related to process models. Nevertheless, deriving decision models from process models remains challenging, especially when the same data underlie both process and decision models. In this paper, we explore how and to which extent the data modeled in BPMN processes and used for decision-making may be represented in the corresponding DMN decision models. To this end, we identify a set of patterns that capture possible representations of data in BPMN processes and that can be used to guide the derivation of decision models related to existing process models. Throughout the paper we refer to real-world healthcare processes to show the applicability of the proposed approach.