Data Transformation Process Research Articles

A process-driven platform to manage datasets for research

ABSTRACT
 Objectives
 
 Accumulate, manage and control shared access to research data; 
 Transform and maintain transformation state information about research data; 
 Analyse and investigate data in related sets using open and bespoke tools; 
 Publish extracted data to a secure safe haven environment.
 
 ApproachThe Research Data Management Platform (RDMP) is a set of data structures and processes, sharing a core Catalogue, to manage electronic health records, genomic data and imaging data throughout their lifecycle from identification and acquisition to safe disposal or archival and retention in secured Safe Havens (SH).
 The architecture components of the RDMP consist of the Catalogue and five internal processes: Data Load, Catalogue Management, Data Quality, Data Summary, and Data Extraction. These are designed to enforce rigorous information governance standards relevant to the processing and anonymisation of personal identifiable data. The Catalogue serves as the single ‘source of truth’ about the datasets which all RDMP processes consult. This facilitates repeatable, reliable and auditable operations on the data.
 The novelty of the RDMP is that it dynamically and seamlessly captures and preserves data transformation processes along with the primary research data to promote reuse and curation of continuously accruing research data repositories in a secure SH environment. Thus, the RDMP brings transparency and reproducibility that benefits research programmes in a way that archival of static data objects does not.
 ResultsThe RDMP has been in production use since July 1st 2014. There are 107 datasets configured in the Catalogue, with up to 67 dataset extractions for each of 48 research projects. It has provided data for 32 high-impact journal papers published in the last year.
 Improvements in turnaround time:
 
 Research project data provision reduced from six months to two weeks;
 Data loading reduced from two days to a few hours; 
 Research query response reduced from days to within a day, due to improved and standardised metadata catalogue
 
 ConclusionThe RDMP is a key component in automating the regular release of datasets and rationalising dataset changes over time to ensure reliable delivery of extracts to research projects. The tools and processes comprising the RDMP not only fulfil the RDM requirements of researchers, but also support seamless collaboration of data cleaning, data transformation, data summarisation and data quality assessment activities by different research groups.

Extraction, Transformation, and Loading (ETL) Module for Hotspot Spatial Data Warehouse Using Geokettle

Spatial data warehouse technology is one solution to the problem of big spatial data. Accumulation In the process of making spatial data warehouse, extraction, transformation, and loading (ETL) process has an important role to determine the quality of data. Manual ETL process requires a long time and makes a lot of queries. Therefore, this research uses Geokettle as a spatial ETL tool to integrate spatial data. This research used hotspot dataset of Indonesia from 2006 to 2014 and administrative districts data in Indonesia. This research performed ETL modeling with the simplification, adjustment, and design of ETL scenarios. The result of this research is ETL modeling implemented using Geokettle. SpagoBI Studio was used to create multidimensional data cubes. Moreover ETL testing was conducted using Geokettle, and spatial data warehouse testing was done by comparing the total number of hotspots between SQL query result and spatial analysis hotspot result on Quantum GIS.

Образовательный визуальный потоковый язык для программирования роботов

Visual domain-specific languages usually have low entry barrier. Sometimes even children can program on such languages by working with visual representations. This is widely used in educational robotics domain, where most commonly used programming environments are visual. The paper describes a novel dataflow visual programming environment for embedded robotic platforms. Obviously, complex dataflow languages are not simple for understanding. The purpose of our tool is to "bridge" between lightweight educational robotic programming tools (commonly these tools provide languages which are based on control flow model) and complex industrial tools (which provide languages based on more complex dataflow execution model). We compare programming environments mostly used by robotics community with our tool. After brief review of behavioural robotic architectures, some thoughts on expressing them in terms of our dataflow language are given. Visual language, which is described here, provides opportunity to mix dataflow and control flow models for robotics programming. We believe that it is important for educational purposes. Program on our language consists of different blocks (visual representation of data transformation processes) and "links" which presents data flow between them. Domain-specific modelling approach was used to develop our language. Also, this paper provides the examples of solving two typical robot control tasks in our language.

OPTIMIZATION OF INFORMATION PREPROCESSING IN CLUSTERING SYSTEMS OF HIGH DIMENSION DATA

The methodic of choice of optimal normalization method for object cluster structure of creation, with high dimension of feature space, is shown. The Shannon entropy criterion and entropy relative change were used as main criterions of estimating the data preprocessing quality during the data transformation. Decreasing of feature space dimension of tested objects was realized by component analysis. Model of system clustering with the use of fuzzy C-means algorithm was constructed, which the help of whith the estimate of clustering quality was established by the use of different data preprocessing methods. It’s shown that the best normalization method for tested data is decimal-scaling method, by which the entropy of processed signal gets minimal significance, and relative change of entropy doesn’t exceed permissible norms during the process of data transformation by component analysis.

MAPPing dark networks: A data transformation method to study clandestine organizations

AbstractThere is a growing consensus that dark networks require special methodological considerations, but little attention has been devoted to determining which data transformation processes are best suited to the study of dark networks. Standard approaches to the transformation of multi-modal data “fold” matrices, by binarizing the network and multiplying it against its transpose. Unfortunately, this process produces exaggerated results when applied to weighted networks, and consequently researchers often disregard information on tie-strength when transforming data. This paper evaluates previous attempts to overcome this limitation and assesses projection methods discovered by biologists and physicists, who have studied the problem of transforming weighted multi-modal networks within the specific context of these disciplines. However, the assumptions underlying these transformation processes limit their applicability to dark networks. This paper, therefore, offers the Median Additive Projection Process (MAPP), an approach to data transformation specifically designed for implementation in weighted, multi-modal dark networks. MAPP accounts for the ambiguity inherent to clandestine subject matter by treating relationship strength in a probabilistic fashion. Because agents' one-mode centrality rankings change significantly when different projection processes are applied to the same two-mode network, MAPP allows researchers to more accurately identify central actors in dark networks.

DBA\'s Perspective on Use of Information Visualization in Electronic Health Records

DBA's perspective for Information Visualization (IV) assists in providing a detailed insight about patient history in Electronic Health Records (EHR). These data visualizations are used by different stake holders in public hospitals such as Doctors, Database Administrators (DBA) and visual designers in order to improve health services for patients. This is not possible without the contribution of DBAs, who are directly involved in launching efficient query system and required data retrieval for resulting visualization. EHR database is comprising of different forms of data i.e., numbers, texts, pictures and graphs resulting in a challenge for creating a meaningful visualization for single or multiple patients. DBAs have EHR knowledge gaps, lesser exposure to the transformation process of data into effective visualization and understanding about IV limitations result in poor and less efficient IV tools. This study highlights the challenging factors that affect DBAs in understanding, developing and controlling EHR visualization. These factors are selected after a literature review and further analysis by a questionnaire based study on DBAs in two different government hospitals. This will help in highlighting the limitations of DBAs skills and experience and assist in determining the contribution of their role in the development of future IV applications. This work is a second phase of our ongoing research for designing a model, CARE 1.0, as a future IV solution in multiple EHR.

Data for geoinformation systems

Various purpose and scale geoinformation systems (GIS) are first and foremost interfaces to arrange for data access and user–friendly control of their transformation. As a rule GIS data specifics consists in their geographic reference (bind) i.e. reference to geographic coordinates. The here presented research proposes to consider the complete GIS data transformation cycle to further implement these GIS as interfaces supporting decision making systems. Judging from the so far accumulated experience of GIS implementation and based on the available literature study it could be said that the class of these systems is extremely large. However, a special interest is now drawn to the GIS based decision making systems for managing the municipal aggregations, transportations (maritime, ground); controlling various purposes and scales monitoring systems. In general case the process of data access and transformation is assumed to be considered in the light of three abstracting levels: harmonized, integrated and fused data. The above mentioned levels possess a physical basis and really reflect the information flows propagating from measuring devices up the hierarchy. The paper is only focused on main statements and does not go into details specifying each level.

PRIVACY PRESERVATION FOR ASSOCIATIVE CLASSIFICATION

Privacy preservation is becoming a critical issue to data-mining processes. In practice, a data transformation process is often needed to preserve privacy. However, data transformation would introduce a data quality issue. In this case, the impact on data quality due to the data transformation should be estimated and made clear to the user of the data transformation process. In this article, we consider the problem of k-anonymization transformation in associative classification. The privacy preservation and data quality issues are considered in twofold. First, we propose a frequency-based data quality metric to represent the data quality for associative classification. Second, a novel heuristic algorithm, namely minimum classification correction rate transformation, is proposed. The algorithm is guided by the classification correction rate of the given datasets. We validate our proposed metric and algorithm with University of California–Irvine repository datasets. The experiment results have shown that our proposed metric can effectively demonstrate the data quality for associative classification. The results also show that the proposed algorithm is not only efficient but also highly effective.

Recording Process of Visual Archive for Cultural Heritage

This paper discussed the recording process and context of the visual archive which records the cultural heritage in video. The cultural heritage record has a value for permanent preservation by being a core material for national and cultural identity of a country, and has its own value as information. Visual archive is one of the recording methods for cultural heritage along with the static image like a picture and the reports documenting contents in writing. Visual archive is useful for the efficient delivery of information with visual images and audio system. Context should be considered to distinguish between data and information more clearly in the process of production and utilization of information. The transformation process of data to information was studied by considering context during the process of recording cultural heritage in images. As a result, we expect that cultural heritage is classified by knowledge information and becomes a foundation used actively for creating cultural contents.

Using enterprise systems to realize digital business strategies

PurposeOrganizations invest in enterprise systems (ESs) with an expectation to share digital information from disparate sources to improve organizational effectiveness. This study aims to examine how organizations realize digital business strategies using an ES. It does so by evaluating the ES data support activities for knowledge creation, particularly how ES data are transformed into corporate knowledge in relevance to business strategies sought. Further, how this knowledge leads to realization of the business benefits. The linkage between establishing digital business strategy, utilization of ES data in decision‐making processes, and realized or unrealized benefits provides the reason for this study.Design/methodology/approachThis study develops and utilizes a transformational model of how ES data are transformed into knowledge and results to evaluate the role of digital business strategies in achieving benefits using an ES. Semi‐structured interviews are first conducted with ES vendors, consultants and IT research firms to understand the process of ES data transformation for realizing business strategies from their perspective. This is followed by three in‐depth cases (two large and one medium‐sized organization) who have implemented ESs. The empirical data are analyzed using the condensation approach. This method condenses the data into multiple groups according to pre‐defined categories, which follow the scope of the research questions.FindingsThe key findings emphasize that strategic benefit realization from an ES implementation is a holistic process that not only includes the essential data and technology factors, but also includes factors such as digital business strategy deployment, people and process management, and skills and competency development. Although many companies are mature with their ES implementation, these firms have only recently started aligning their ES capabilities with digital business strategies correlating data, decisions, and actions to maximize business value from their ES investment.Research limitations/implicationsThe findings reflect the views of two large and one medium‐sized organization in the manufacturing sector. Although the evidence of the benefit realization process success and its results is more prominent in larger organizations than medium‐sized, it may not be generalized that smaller firms cannot achieve these results. Exploration of these aspects in smaller firms or a different industry sector such as retail/service would be of value.Practical implicationsThe paper highlights the importance of tools and practices for accessing relevant information through an integrated ES so that competent decisions can be established towards achieving digital business strategies, and optimizing organizational performance. Knowledge is a key factor in this process.Originality/valueThe paper evaluates a holistic framework for utilization of ES data in realizing digital business strategies. Thus, it develops an enhanced transformational cycle model for ES data transformation into knowledge and results, which maintains to build up the transformational process success in the long term.

Utilizing enterprise systems for managing enterprise risks

Enterprise risk management is a critical concept in the current business environment that supports use of tools and processes directed toward monitoring and mitigating organizational risks. Many organizations have embraced enterprise systems (ESs) technology for improving organizational efficiency and effectiveness. ESs provide value by identifying opportunities in operations and assist in managing risks through context sensitive analyses by eliciting relevant information. This research investigates how ES data were transformed into knowledge by a hi-tech manufacturing firm from an ES implementation, and how this knowledge was used to manage risks by utilizing an ES data transformation model from existing literature. Findings indicate that the ES data transformation process resulted from knowledge-leveraging actions at both executive and operational levels. At the executive level, the use of business intelligence module in conjunction with cascades of balanced scorecards helped in assessing progress for achieving goals, and translated decisions into risk-eliminating actions at the operational level. An initial technology-push approach assisted in creating semantically rich representative process models by simulating risk scenarios, leading to a strategy-pull approach for deploying business strategies and decisions. A value assessment strategic model articulates the knowledge-leveraging processes combining human skills with ES tools to optimize enterprise risks.

Enhancing the Quality of Separation in One-Dimensional Peptide Mapping Using Mathematical Transformation

In this study, some practical examples are presented that show the quality of separations using very efficient columns packed with the latest generation of core shell sub-3 μm and fully porous sub-2 μm particles in one-dimensional peptide separations. This paper shows an approach for the analysis of proteins, such as high-resolution separations, and a data transformation process to improve peak recognition and analysis. Applying power functions on raw chromatographic data can be a neat tool in the field of biosimilar analysis, especially in comparability studies regarding the quality (primary structure) of proteins. Based on the results presented here, it can be stated that the use of power functions is beneficial for the comparison of chromatograms when peak areas are considered but has no effect when using peak heights. In this study, the new Acquity CSH columns (C18 and phenyl-hexyl) and the core–shell type wide pore Ascentis Express Peptide ES C18 material were applied with great success in peptide mapping. Finally, using phenyl-hexyl stationary phase in peptide separation seems to be a good alternative to the generally applied C18 or C4 phases.

A Mechanism for Propagation of Semantic Annotations of Multimedia Content

Scientific research is producing and consuming large volumes of multimedia data at an ever growing rate. Data annotations are used, among others, to provide context information and enhance content management, making it easier to interpret and share data. However, raw multimedia data often needs to go through complex processing steps before it can be consumed. During these transformation processes, original annotations from the production phase are often discarded or ignored, since their usefulness is usually limited to the first transformation step. New annotations must be made at each step, and associated with the final product, a time consuming task often carried out manually. The task of systematically associating new annotations to the result of each data transformation step is known as annotation propagation . This paper introduces techniques for structuring and propagating annotations, in parallel to the data transformation processes, thereby alleviating the overhead and decreasing the errors introduced by manual annotation. This helps the construction of new annotated multimedia data sets, preserving contextual information. The solution is based on: (i) the notion of semantic annotations; (ii) a set of transformations rules, based on ontological relations; and, (iii) workflows that deal with interrelated processing steps.

Model Updating for Spectral Calibration Maintenance and Transfer Using 1-Norm Variants of Tikhonov Regularization

In this study, calibration maintenance confronts the problem of updating a model developed in the primary condition to accurately predict the calibrated analyte in samples measured in new secondary conditions. Calibration transfer refers to updating a model based on a primary instrument to predict samples measured on new secondary instruments. A 2-norm variant of Tikhonov regularization (TR) has been used with spectroscopic data to perform calibration maintenance and transfer where just a few samples measured in the secondary condition/instrument are augmented to the primary calibration data to update the primary model. To achieve improved predictions, in this paper we report on 1-norm penalties to create two novel variants of TR for model updating. To solve the multiple penalty minimization numerical problems involved with the new TR variants, data transformation processes are applied, allowing a least absolute shrinkage and selection operator type algorithm to be implemented. Near-infrared spectra measured under two different temperatures represent the calibration maintenance application, and near-infrared spectra measured on two instruments denote the calibration transfer situation. Compared to TR in the recently developed 2-norm penalty mode, validation sample prediction errors are reduced when the 1-norm TR variant that selects wavelengths is used.

A practical data mining method to link hospital microbiology and an infection control database

<h3>Background</h3> Data mining is the process of data exploration, selection, and transformation. Data mining has developed as an important tool in health care for converting immense amounts of data into powerful information to support surveillance, detect and assist with investigation of microbial clusters, and collaborate data for research. Additionally, data mining supports ICPs' commitment to providing quality health care. <h3>Methods</h3> Realizing the existing potential of a current infection control database, current software has been transformed into a system capable of data mining with the addition of a single laboratory information system interface. <h3>Results</h3> Commercial data mining software products are available but at significant cost. With a laboratory interface to a current infection control database, a system capable of data mining was created. Similar to other data mining products, this system achieves electronic surveillance, generates monthly or more frequent reports as required, provides patient diagnosis and culture history, offers unit-specific rates, and provides rapid identification of patients with multidrug-resistant organisms. Additionally, health care system-wide custom report-writing features are highly effective tools used to assist with outbreak investigations and research projects. <h3>Conclusion</h3> Purchasing new software products can be expensive; however, current resources may be used to meet data mining needs. Although weekly queries are optimal, data mining software must provide, at a minimum, monthly reports to track infection-related events in a health care facility. The capability to detect a sudden increase in any specific microorganism and the ability to monitor a known problematic microorganism is a necessity. In addition, data mining software products must include graphing capabilities and ease of use custom report-writing features.

Model-based variance-stabilizing transformation for Illumina microarray data

Variance stabilization is a step in the preprocessing of microarray data that can greatly benefit the performance of subsequent statistical modeling and inference. Due to the often limited number of technical replicates for Affymetrix and cDNA arrays, achieving variance stabilization can be difficult. Although the Illumina microarray platform provides a larger number of technical replicates on each array (usually over 30 randomly distributed beads per probe), these replicates have not been leveraged in the current log2 data transformation process. We devised a variance-stabilizing transformation (VST) method that takes advantage of the technical replicates available on an Illumina microarray. We have compared VST with log2 and Variance-stabilizing normalization (VSN) by using the Kruglyak bead-level data (2006) and Barnes titration data (2005). The results of the Kruglyak data suggest that VST stabilizes variances of bead-replicates within an array. The results of the Barnes data show that VST can improve the detection of differentially expressed genes and reduce false-positive identifications. We conclude that although both VST and VSN are built upon the same model of measurement noise, VST stabilizes the variance better and more efficiently for the Illumina platform by leveraging the availability of a larger number of within-array replicates. The algorithms and Supplementary Data are included in the lumi package of Bioconductor, available at: www.bioconductor.org.

Exploratory mining in cube space

Data Mining has evolved as a new discipline at the intersection of several existing areas, including Database Systems, Machine Learning, Optimization, and Statistics. An important question is whether the field has matured to the point where it has originated substantial new problems and techniques that distinguish it from its parent disciplines. In this paper, we discuss a class of new problems and techniques that show great promise for exploratory mining, while synthesizing and generalizing ideas from the parent disciplines. While the class of problems we discuss is broad, there is a common underlying objective—to look beyond a single data-mining step (e.g., data summarization or model construction) and address the combined process of data selection and transformation, parameter and algorithm selection, and model construction. The fundamental difficulty lies in the large space of alternative choices at each step, and good solutions must provide a natural framework for managing this complexity. We regard this as a grand challenge for Data Mining, and see the ideas discussed here as promising initial steps towards a rigorous exploratory framework that supports the entire process.

Microarray RNA transcriptional profiling: Part II. Analytical considerations and annotation

This review summarizes the various data filtration, transformation and normalization processes for different array platforms (cDNA, oligos, one- and two-color), data analysis methods and their validation, and databases and annotation for RNA transcriptional profiling microarrays. This review is intended to introduce the beginner to the analyses and interpretation of gene expression studies using a nonmathematical approach for easier comprehension. Microarray analysis is not a trivial undertaking as there is no single method that works well for all, and results obtained from these analyses should be considered as a complement to other approaches.

Development of a capillary electrophoretic method for the analysis of low‐molecular‐weight amines from metal working fluid aerosols and ambient air

A method for the determination of low-molecular-weight amines from indoor and ambient air was developed using a concentration device followed by CE coupled with indirect spectrophotometric and mass spectrometric detection that enables a reliable, rapid-response and easy-to-operate method. In indirect detection method, the selected amines were separated from interfering metal ions and amino alcohols present in the samples with an imidazole-based buffer with ethanol and EDTA as modifier. By replacing imidazole with ammonium, the final buffer was applicable for MS detection for the analytes with m/z higher than 50. A novel monolithic polymer material based on poly(methacrylate-acrylate) copolymer was developed for sampling short-chain amines from the gaseous phase. The selected analysis conditions were applied to quantify the selected short-chain amines with detection limits for the whole procedure determined between 1 and 2 microg/filter when 40 L air was sampled with 1 L/min velocity. Improved linearity and precision were obtained when the raw, time-scaled electropherogram data were transformed into mobility-scale applied for the determination of the performance characteristics of the methods. The applicability of the process of data transformation into the mobility scale was demonstrated by studying the matrix effect of water-miscible metal working fluid (stable water-oil emulsion) and of ambient air as real samples. CE-indirect UV and CE-MS, combined with the possibility of rapid air sampling, can be useful for the estimation of short-term exposure of the selected biogenic amines.

NGN에서 오버레이를 이용한 사용자 관점의 End-to-end QoS 지원 구조

This study proposes an Architecture for user level End-to-end Quality of Service(QoS) using overlay In Next Generation Network(NGN). Inexisting NGNs, the IMS of a control plane provides user QoS through direct traffic control and resource-reservation over the IP packet transport network of a user plane. Further, a set of torrent studies are ongoing not only to maximize the QoS for users, but also to minimize the quality deterioration for supporting the user End-to-end QoS. Along with that, an extended QoS in user level must be considered, for Instance, differentiating service quality to support users` expectation, providing optimized contents by users` equipments, and so forth. Accordingly, the Overlay Service Network Architecture proposed by this study provides protocol adaptation for maximum throughput on transport layer by using the most efficient transport layer protocol to various network circumstances. Also, the Overlay Service Network Architecture on application layer distributes processing delay from the data transformation process of the user equipment to the network, and it is capable of intermediate processing depending on user service level. application service feature, and equipment circumstance as well. Thus, this study mainly proposes the Overlay Service Network Architecture for user level end-to-end QoS in NGN with the quality control features both on the transport layer and the application layer, an internal component feature, and a service scenario providing the QoS linking with 3GPP.