Extensible Markup Language Research Articles

Zoonoses and foodborne outbreaks guidance for reporting 2021 data

This technical report of the European Food Safety Authority (EFSA) presents the guidance to reporting European Union (EU) Member States and non-Member States in data transmission using extensible markup language (XML) data transfer covering the reporting of prevalence data on zoonoses and microbiological agents and contaminants in food, foodborne outbreak data, animal population data and disease status data. For data collection purposes, EFSA has created the Data Collection Framework (DCF) application. The present report provides data dictionaries to guide the reporting of information deriving from 2021 under the framework of Directive 2003/99/EC, Regulation (EU) 2017/625 and Commission Implementing Regulation (EU) 2019/627. The objective is to explain in detail the individual data elements that are included in the EFSA data models to be used for XML data transmission through the DCF. In particular, the data elements to be reported are explained, including information about the data type, a reference to the list of allowed terms and any additional business rule or requirement that may apply.

An accessible, efficient, and accurate natural language processing method for extracting diagnostic data from pathology reports

ContextAnalysis of diagnostic information in pathology reports for the purposes of clinical or translational research and quality assessment/control often requires manual data extraction, which can be laborious, time-consuming, and subject to mistakes. ObjectiveWe sought to develop, employ, and evaluate a simple, dictionary- and rule-based natural language processing (NLP) algorithm for generating searchable information on various types of parameters from diverse surgical pathology reports. DesignData were exported from the pathology laboratory information system (LIS) into extensible markup language (XML) documents, which were parsed by NLP-based Python code into desired data points and delivered to Excel spreadsheets. Accuracy and efficiency were compared to a manual data extraction method with concordance measured by Cohen’s κ coefficient and corresponding P values. ResultsThe automated method was highly concordant (90%–100%, P<.001) with excellent inter-observer reliability (Cohen’s κ: 0.86–1.0) compared to the manual method in 3 clinicopathological research scenarios, including squamous dysplasia presence and grade in anal biopsies, epithelial dysplasia grade and location in colonoscopic surveillance biopsies, and adenocarcinoma grade and amount in prostate core biopsies. Significantly, the automated method was 24–39 times faster and inherently contained links for each diagnosis to additional variables such as patient age, location, etc., which would require additional manual processing time. ConclusionsA simple, flexible, and scaleable NLP-based platform can be used to correctly, safely, and quickly extract and deliver linked data from pathology reports into searchable spreadsheets for clinical and research purposes.

A blockchain-based cooperative modeling method for digital twin ontology model of the mechanical product

With the rapid increase of multi-source heterogeneous dynamic data of mechanical products, the digital twin technology is considered to be an important method to realize the deep integration of product data and intelligent manufacturing. As a digital archive of the physical entity in entire life cycle, the mechanical product digital twin model is cross-phased and multi-domain. Therefore, safe and stable cooperative modeling has become a basic technical problem that needs to be solved urgently. In this paper, we proposed a blockchain-based collaborative modeling method for the digital twin ontology model of mechanical products. First, an authorization network was constructed among stakeholders. Then modeling processes of the digital twin were mapped to ontology operations and formatted through extensible markup language. Finally, consensuses were obtained based on practical byzantine fault tolerance. And a material modification process of a helicopter damper bearing was taken as an example to verify. The proposed method enables all participants to accurately obtain the latest state of the digital twin model, and has the advantages of tamper-proof, traceability, and decentralization.

Thermodynamic Modeling with Uncertainty Quantification Using the Modified Quasichemical Model in Quadruplet Approximation: Implementation into PyCalphad and ESPEI

The modified quasichemical model in the quadruplet approximation (MQMQA) considers the first- and the second-nearest-neighbor coordination and interactions, particularly useful in describing short-range ordering in complex liquids such as molten salts, slag in metal processing, and electrolytic solutions. The present work implements the MQMQA into the Python based open-source software PyCalphad for thermodynamic calculations. This endeavor facilitates the development of MQMQA-based thermodynamic database with uncertainty quantification (UQ) using the open-source software ESPEI. A new database structure based on Extensible Markup Language (XML) is proposed for ESPEI evaluation of MQMQA model parameters. Using the KF-NiF2 system as an example, we demonstrate the successful implementation of MQMQA in PyCalphad through thermodynamic calculations of Gibbs energy, equilibrium quadruplet fractions, and phase diagram, as well as database development with UQ using ESPEI. The present implementation offers an open-source capability for performing CALPHAD modeling for complex liquids with short-range ordering using MQMQA.

Medical data integration using HL7 standards for patient's early identification.

Integration between information systems is critical, especially in the healthcare domain, since interoperability requirements are related to patients’ data confidentiality, safety, and satisfaction. The goal of this study is to propose a solution based on the integration between queue management solution (QMS) and the electronic medical records (EMR), using Health Level Seven (HL7) protocols and Extensible Markup Language (XML). The proposed solution facilitates the patient’s self-check-in within a healthcare organization in UAE. The solution aims to help in minimizing the waiting times within the outpatient department through early identification of patients who hold the Emirates national ID cards, i.e., whether an Emirati or expatriates. The integration components, solution design, and the custom-designed XML and HL7 messages were clarified in this paper. In addition, the study includes a simulation experiment through control and intervention weeks with 517 valid appointments. The experiment goal was to evaluate the patient’s total journey and each related clinical stage by comparing the “routine-based identification” with the “patient’s self-check-in” processes in case of booked appointments. As a key finding, the proposed solution is efficient and could reduce the “patient’s journey time” by more than 14 minutes and “time to identify” patients by 10 minutes. There was also a significant drop in the waiting time to triage and the time to finish the triage process. In conclusion, the proposed solution is considered innovative and can provide a positive added value for the patient’s whole journey.

Intelligence Eye for Blinds and Visually Impaired by Using Region-Based Convolutional Neural Network (R-CNN)

Intelligence Eye is an Android based mobile application developed to help blind and visually impaired users to detect light and objects. Intelligence Eye used Region-based Convolutional Neural Networks (R-CNN) to recognize objects in the object recognition module and a vibration feedback is provided according to the light value in the light detection module. A voice guidance is provided in the application to guide the users and announce the result of the object recognition. TensorFlow Lite is used to train the neural network model for object recognition in conjunction with extensible markup language (XML) and Java in Android Studio for the programming language. For future works, improvements can be made to enhance the functionality of the Intelligence Eye application by increasing the object detection capacity in the object recognition module, add menu settings for vibration intensity in light detection module and support multiple languages for the voice guidance.

Adaptive query relaxation and top‐ k result sorting of fuzzy spatiotemporal data based on XML

With the increasing popularity of Extensible Markup Language (XML) for data representation, there is a lot of interest in searching XML data. Due to the structural heterogeneity of XML, it is daunting for users to formulate exact queries and search for accurate answers. Especially when performing queries on fuzzy spatiotemporal data, the imprecision and uncertainty of information make the amount of query results is small or even empty. In this case, query relaxation is necessary. Existing efforts are mainly made on expanding simple queries. Once there are multiple conditions in the query, the existing algorithms often have poor performance. To address this problem, we first propose an adaptive query relaxation algorithm based on XML, which can handle complex queries efficiently. The approach does not depend on any specific domain or user, it can adaptively relax the initial query requirements, and classify the results into two sets after query relaxation. Moreover, the algorithm prestores the potential matching objects in the auxiliary structure, which provides support for efficient query. In addition, after query relaxation, we present the result sorting algorithm to address the problem of information overload. Finally, we report experimental results to show that our algorithms are superior to previous approaches.

Transforming data-centric eXtensible markup language into relational databases using hybrid approach

eXtensible markup language (XML) appeared internationally as the format for data representation over the web. Yet, most organizations are still utilising relational databases as their database solutions. As such, it is crucial to provide seamless integration via effective transformation between these database infrastructures. In this paper, we propose XML-REG to bridge these two technologies based on node-based and path-based approaches. The node-based approach is good to annotate each positional node uniquely, while the path-based approach provides summarised path information to join the nodes. On top of that, a new range labelling is also proposed to annotate nodes uniquely by ensuring the structural relationships are maintained between nodes. If a new node is to be added to the document, re-labelling is not required as the new label will be assigned to the node via the new proposed labelling scheme. Experimental evaluations indicated that the performance of XML-REG exceeded XMap, XRecursive, XAncestor and Mini-XML concerning storing time, query retrieval time and scalability. This research produces a core framework for XML to relational databases (RDB) mapping, which could be adopted in various industries.

Transforming data-centric eXtensible markup language into relational databases using hybrid approach

eXtensible markup language (XML) appeared internationally as the format for data representation over the web. Yet, most organizations are still utilising relational databases as their database solutions. As such, it is crucial to provide seamless integration via effective transformation between these database infrastructures. In this paper, we propose XML-REG to bridge these two technologies based on node-based and path-based approaches. The node-based approach is good to annotate each positional node uniquely, while the path-based approach provides summarised path information to join the nodes. On top of that, a new range labelling is also proposed to annotate nodes uniquely by ensuring the structural relationships are maintained between nodes. If a new node is to be added to the document, re-labelling is not required as the new label will be assigned to the node via the new proposed labelling scheme. Experimental evaluations indicated that the performance of XML-REG exceeded XMap, XRecursive, XAncestor and Mini-XML concerning storing time, query retrieval time and scalability. This research produces a core framework for XML to relational databases (RDB) mapping, which could be adopted in various industries.

A COMPARISON OF CREATING AN ANDROID APPLICATION USER INTERFACE IN XAML AND IN C# CODE

Almost every Android user application has some kind of user interface. Android programmers who create Xamarin.Forms applications and who uses the Microsoft Visual Studio development environment to do so can create user interfaces in the XAML (the Extensible Application Markup Language) or in the C# programming language. This paper deals with a comparison of creating this user interface by the first and the second way. We demonstrate the differences in the creation of Android application user interfaces in XAML and in C# by way of using two of our Android applications which were created by the Microsoft Visual Studio 2019 Enterprise, which work as text editors with the ability to store text to disc, and have user interfaces that are visually identical. However, the user interface of the first Android application was created in XAML with a C# support code, whereas the user interface of the second Android application was created entirely in C#. While comparing the source codes of the user interfaces of both these Android applications, we identify the advantages and disadvantages of both approaches to creating a user interface and we try to find out which of these approaches is better for maintaining and modifying user interfaces. In this study, we also contrast the processes of handling events of controls of a user interface created in the XAML code for the first Android application as well as that of the same user interface created in the C# code for the second Android application. Furthermore, we were interested in determining whether the different ways of creating user interfaces affected the execution time of basic operations that included disc files that were performed on the same data by both the applications. We assume that it does not fundamentally affect the execution time, and so, we performed an experiment to confirm or refute our assumption.

Designing Robust and Automated Treatment Plans Utilizing Dynamic Couch Motion With Automated Treatment Planning: Optimizing Couch Motion for Precise Delivery ― Part A

<h3>Purpose/Objective(s)</h3> This study finds a relationship among couch speed, dose rate, and MLC, allowing gradual start and stop positioning of the couch for treatments utilizing dynamic couch-based dose delivery at static gantry positions. <h3>Materials/Methods</h3> We introduce a gantry static couch motion (GsCM) based technique, which delivers the dose using a dynamic couch and static gantry positions. It is imperative to study couch motion behavior for techniques like GsCM, where the patient is a moving target throughout the dose delivery. Sudden changes in couch speed can result in a shift in position and discomfort for the patient. Therefore, it is necessary to implement a smooth start, a constant speed in the middle, and deceleration at the couch's stopping positioning. We studied the relationship among the following parameters: dose rate, MLC speed, and couch motion. To fulfill this, we simulated a single treatment arc manually in the treatment planning system that includes multiple couch positions at static gantry positions. All the fields were merged into a single deliverable arc while exporting into the Extensible Markup Language (XML) format using the TPS scripting API. Developer mode on a Linac performed the treatment delivery. The second part of this study looked into the various ways to control the couch motion. Methods tried included a fixed couch speed versus controlling the couch speed by limiting the dose rate. Trajectory log files helped analyze the couch positional error (RMSE). <h3>Results</h3> We found couch speed optimization by limiting the dose rate is better than the MLC speed or couch speed. A slow ramping up and down of the speed with a constant speed in the middle of an arc field is achievable. Linac log file analysis showed minimum couch position error (RMSE: median 0.035⁰, range 0.02-0.0.045⁰) for 1° couch speed as compared to system defined couch speed (RMSE: median 0.07⁰, range 0.05-0.12⁰), and (RMSE: median 0.07⁰, range 0.06-0.09⁰) for 2° couch speed respectively. Even though 1° couch speed provided a minimum couch RMSE error but at the expense of a longer treatment time which is not suitable for keeping the patient longer on the treatment table. For the dose rate limiting option, couch positioning error at the start and stop location (RMSE: median 0.008⁰, range 0.006-0.02⁰) showed a gradual and accurate ramping up and down of the couch, and at the middle of the arc remained the same as noticed for the system defined couch speed (RMSE: median 0.09⁰, range 0.065-0.12⁰). <h3>Conclusion</h3> This feasibility study serves as an end-to-end test and is probably the first of its kind that thoroughly validates various ways to control the couch speed and its integration with patient motion during the dose delivery. Treatments utilizing dynamic couch-based dose delivery must include patient immobilization. Our study takes patient motion into account and provides gradual start and stop motions of the couch, which helps minimize the patient motion for treatments utilizing dynamic couch motion.

Designing Robust and Automated Treatment Plans Utilizing Dynamic Couch Motion With Automated Treatment Planning: Delivering Dose to a New Dimension ― Part B

<h3>Purpose/Objective(s)</h3> To develop a novel treatment technique, with automated treatment planning, that will yield superior dose distribution with dynamic couch motion at static gantry positions. <h3>Materials/Methods</h3> Modern treatment planning systems (TPS) cannot design, calculate, or optimize treatment plans utilizing continuous couch-based delivery. Based on the current limitation of modern TPS, the need to plan dynamic couch-based dose delivery arises for our investigation. We implement an automatic method utilizing the TPS scripting application programming interface (ESAPI). Additionally, we introduce a novel gantry static couch motion technique (GsCM) with static gantry positions and dynamic couch motion for brain treatment. The script interface allows selection of PTV, adding MLCs and PTV margin, choosing the number of fields, and initiation of the dose calculation within the TPS environment. We can automatically design custom treatment plans with simple 3-dimentional (3D) planning and conformal arc. This script based automated planning was retrospectively applied to brain patients (n = 5) using 200 cGy for 23 fractions. Initial development required implementing a conversion from the TPS mechanical axes to that of the machine. The readable Extensible markup language (XML) files are generated in the TPS environment using ESAPI by combining the fields at each couch position into a single deliverable field for each gantry angle utilizing continuous couch motion. The entire workflow is feasible within our script interface. The last part of this study focused on verifying the dose delivery. We used an ion chamber array positioned in the vertical direction to avoid side beam entrance. Trajectory log files were also analyzed using root mean square error (RMSE) to verify the MLC, gantry, and couch position accuracy. <h3>Results</h3> All treatment plans satisfy the criteria of V100% ≥ 95% and respect the dose to organs-at-risk (OARs). Automated script-based plans took less than a minute (excluding dose calculation) to design as compared to manual planning (2-8 hours per plan). The delivery was found to be less than 2 minutes for two 90 degrees partial arcs as compared to field-by-field delivery (15-30 minutes based on the plan type). All script-based plans (n = 5) passed patient-specific QA (> 95%) using the criteria of 2 mm distance to agreement and 2% dose difference. The couch rotational RMSE error for step and shoot and continuous delivery was 0.1 degrees and 0.12 degrees, respectively. MLC RMSE was 0.005 mm, and 0.006 mm for step and shoot and continuous delivery. <h3>Conclusion</h3> The results of our study indicate that the automated custom planning with couch motion is feasible and comparable with clinical planning in terms of dosimetric quality. The developed methodology provides a tool that is integratable with the current TPS without any additional resources. It designs script-based plans for dynamic couch motion-based dose delivery.

Facility Planning Optimization Platform, GGOD, for Expandable Cluster-type Micro-grid Installations and Operations

This paper describes the architecture and the utilization for a facility planning optimization platform called GGOD, “Grid of Grids Optimal Designer” and applies it to expandable cluster-type micro-grid installations and operations. The expandable cluster-type micro-grid is defined as a group of micro-grids that are connected by bi-directional power transfer networks. Furthermore, power sources are also networked. Especially, by networking among power sources, powers necessary for social activities in-demand areas are secured. The proposed architecture is based on service-oriented architecture, meaning that optimization functions are executed as services. For flexibility, these services are executed by requests based on extensible mark-up language texts. The available optimizations are written in meta-data, which are accessible to end-users from the meta-data database system called clearinghouse. The meta-data are of two types, one for single optimization and the other for combined optimization. The processes in GGOD are conducted by the management function which interprets descriptions in meta-data. In meta-data, the names of optimization functions and activation orders are written. The basic executions follow sequential, branch, or loop flow processes, which execute combined optimizations, compare more than two kinds of optimization processes, and perform iterative simulations, respectively. As an application of the proposed architecture, the power generation sites and transmission networks are optimized in a geospatial integrated-resource planning scenario. In this application, a structure and a method for the combination of component functions in GGOD are exemplified. Moreover, GGOD suggests promotions of a lot of applications by effective combinations of basic optimization functions.

Decision support system in health care building design based on case-based reasoning and reinforcement learning

The health care building designing is a very difficult process in which there are a great quantity of parameters and variables that it is necessary to consider. The health care building should reach the population needs. Additionally, the design of this type of building and the related facilities involves a great quantity of regulations which they are adapted to the different countries. Thus, the health care facilities should be designed according to numerous and complex regulations. The checking of this regulation is very difficult and usually involves big teams of specialized engineers and architects. The proposed Case-Based Reasoning (CBR) and Reinforcement Learning can analyse the data about building design (provided in an Extended Mark-up Language or XML file, and other compatible formats), checking, and validating the regulations. This approach allows to reduce the specialized and high qualified personnel, providing a report with the checking regulations, and the traceability of warning and faults in the application of regulations.

Data modelling and data processing generated by human eye movements

Data modeling and data processing are important activities in any scientific research. This research focuses on the modeling of data and processing of data generated by a saccadometer. The approach used is based on the relational data model, but the processing and storage of the data is done with client datasets. The experiments were performed with 26 randomly selected files from a total of 264 experimental sessions. The data from each experimental session was stored in three different formats, respectively text, binary and extensible markup language (XML) based. The results showed that the text format and the binary format were the most compact. Several actions related to data processing were analyzed. Based on the results obtained, it was found that the two fastest actions are respectively loading data from a binary file and storing data into a binary file. In contrast, the two slowest actions were storing the data in XML format and loading the data from a text file, respectively. Also, one of the time-consuming operations turned out to be the conversion of data from text format to binary format. Moreover, the time required to perform this action does not depend in proportion on the number of records processed.

Improving the support for XML dynamic updates using a hybridization labeling scheme (ORD-GAP)

Background : As the standard for the exchange of data over the World Wide Web, it is important to ensure that the eXtensible Markup Language (XML) database is capable of supporting not only efficient query processing but also capable of enduring frequent data update operations over the dynamic changes of Web content. Most of the existing XML annotation is based on a labeling scheme to identify each hierarchical position of the XML nodes. This computation is costly as any updates will cause the whole XML tree to be re-labelled. This impact can be observed on large datasets. Therefore, a robust labeling scheme that avoids re-labeling is crucial. Method: Here, we present ORD-GAP (named after Order Gap), a robust and persistent XML labeling scheme that supports dynamic updates. ORD-GAP assigns unique identifiers with gaps in-between XML nodes, which could easily identify the level, Parent-Child (P-C), Ancestor-Descendant (A-D) and sibling relationship. ORD-GAP adopts the OrdPath labeling scheme for any future insertion. Results: We demonstrate that ORD-GAP is robust enough for dynamic updates, and have implemented it in three use cases: (i) left-most, (ii) in-between and (iii) right-most insertion. Experimental evaluations on DBLP dataset demonstrated that ORD-GAP outperformed existing approaches such as ORDPath and ME Labeling concerning database storage size, data loading time and query retrieval. On average, ORD-GAP has the best storing and query retrieval time. Conclusion: The main contributions of this paper are: (i) A robust labeling scheme named ORD-GAP that assigns certain gap between each node to support future insertion, and (ii) An efficient mapping scheme, which built upon ORD-GAP labeling scheme to transform XML into RDB effectively.

Retro-Digitization of Croatian Pre-Standard Grammars

In this article, we will present the rich linguistic heritage of the Croatian language and our attempts to ensure its preservation and presentation to the general public by means of the "Retro-digitization and Interpretation of Croatian Grammar Books before Illyrism ‒ RETROGRAM" project. There is a long tradition of grammatical description in the history of the Croatian language. The first grammar book of the Croatian language was written at the beginning of the 17th century and the first grammar book written in Croatian was compiled in the middle of the 17th century. In later years, when literary and linguistic activity were transferred from the Dalmatian area to the northern and eastern part of Croatia, the Latin model for the description of the Croatian language was still present, even though German was also used. There were a large number of grammars written up to the second half of the 19th century, which are considered pre-standard Croatian grammars. They are the subject of research within the project "Pre-standard Croatian Grammars" at the Institute of Croatian Language and Linguistics. This research proposal "Retro-digitization and Interpretation of Croatian Grammar Books before Illyrism" aims to create a model for the retro-digitization of the chosen eight Pre-standard Croatian Grammars (written from the 17th until the 19th century). The retro-digitization of Croatian grammar books implies the transfer of printed media to computer-readable and searchable text. It also includes a multilevel mark-up of transcribed or translated grammar text. The next step of the project is the creation of a Web Portal of Pre-standard Croatian Grammars, on which both the facsimiles and the digitized text of the grammars will be presented. Our aim is to present to the wider and international public the attainments of the Croatian language and linguistics as an important part of Croatian culture in general. Keywords: pre-standard Croatian grammars, history of the Croatian language, retro-digitization, Extensible mark-up language, Text encoding initiative, web portal of pre-standard Croatian grammars

Improving the support for XML dynamic updates using a hybridization labeling scheme (ORD-GAP).

Background: As the standard for the exchange of data over the World Wide Web, it is important to ensure that the eXtensible Markup Language (XML) database is capable of supporting not only efficient query processing but also capable of enduring frequent data update operations over the dynamic changes of Web content. Most of the existing XML annotation is based on a labeling scheme to identify each hierarchical position of the XML nodes. This computation is costly as any updates will cause the whole XML tree to be re-labelled. This impact can be observed on large datasets. Therefore, a robust labeling scheme that avoids re-labeling is crucial. Method: Here, we present ORD-GAP (named after Order Gap), a robust and persistent XML labeling scheme that supports dynamic updates. ORD-GAP assigns unique identifiers with gaps in-between XML nodes, which could easily identify the level, Parent-Child (P-C), Ancestor-Descendant (A-D) and sibling relationship. ORD-GAP adopts the OrdPath labeling scheme for any future insertion. Results: We demonstrate that ORD-GAP is robust enough for dynamic updates, and have implemented it in three use cases: (i) left-most, (ii) in-between and (iii) right-most insertion. Experimental evaluations on DBLP dataset demonstrated that ORD-GAP outperformed existing approaches such as ORDPath and ME Labeling concerning database storage size, data loading time and query retrieval. On average, ORD-GAP has the best storing and query retrieval time. Conclusion: The main contributions of this paper are: (i) A robust labeling scheme named ORD-GAP that assigns certain gap between each node to support future insertion, and (ii) An efficient mapping scheme, which built upon ORD-GAP labeling scheme to transform XML into RDB effectively.

XML 101 for Journal Production Editors

MODERATOR: Julie L Nash Senior Partner J&J Editorial, LLC Cary, North Carolina SPEAKERS: Michael Casp Director of Business Development, Production Services Coordinator J&J Editorial, LLC Cary, North Carolina Karie Kirkpatrick Associate Publisher, Digital American Physiological Society Rockville, Maryland REPORTER: Heather DiAngelis Associate Publications Director, Transportation Research Board National Academies of Sciences, Engineering, and Medicine Washington, DC This session provided contextual overview, historical background, and best practice suggestions for XML novices in publishing. Michael Casp, Director of Business Development and Production Services Coordinator at J&J Editorial, began the session with a definition of extensible markup language (XML) and gave basic visual examples of how it is both a human-readable and machine-readable language. This mark-up language is everywhere—even in Microsoft Word. His boiled-down definition was “XML is labels.” He went on to explain various versions of XML document type definitions (DTDs) and various ways to organize them, such as NLM, JATS, and custom DTDs for content management systems. Journal article tag suite (JATS) is the most commonly used DTD for scholarly journal content; it has become the de facto XML standard in the scholarly publishing industry. Casp also discussed how peer review systems such as Editorial Manager collect metadata from authors and editorial staff and then convert them to XML files for transmittal. Afterward, a manuscript document is tagged either before or after copyediting, mostly via automation with human review, to create a new XML file of the text, tables, and metadata; as he noted, almost every item relevant to the […]

Efficient processing of complex XSD using Hive and Spark.

The eXtensible Markup Language (XML) files are widely used by the industry due to their flexibility in representing numerous kinds of data. Multiple applications such as financial records, social networks, and mobile networks use complex XML schemas with nested types, contents, and/or extension bases on existing complex elements or large real-world files. A great number of these files are generated each day and this has influenced the development of Big Data tools for their parsing and reporting, such as Apache Hive and Apache Spark. For these reasons, multiple studies have proposed new techniques and evaluated the processing of XML files with Big Data systems. However, a more usual approach in such works involves the simplest XML schemas, even though, real data sets are composed of complex schemas. Therefore, to shed light on complex XML schema processing for real-life applications with Big Data tools, we present an approach that combines three techniques. This comprises three main methods for parsing XML files: cataloging, deserialization, and positional explode. For cataloging, the elements of the XML schema are mapped into root, arrays, structures, values, and attributes. Based on these elements, the deserialization and positional explode are straightforwardly implemented. To demonstrate the validity of our proposal, we develop a case study by implementing a test environment to illustrate the methods using real data sets provided from performance management of two mobile network vendors. Our main results state the validity of the proposed method for different versions of Apache Hive and Apache Spark, obtain the query execution times for Apache Hive internal and external tables and Apache Spark data frames, and compare the query performance in Apache Hive with that of Apache Spark. Another contribution made is a case study in which a novel solution is proposed for data analysis in the performance management systems of mobile networks.