Common Data Structure Research Articles

Analysis of Bi-LSTM CRF Series Models for Semantic Classification of NAVTEX Navigational Safety Messages

NAVTEX is a key component in the Global Maritime Distress and Safety System (GMDSS) that automatically transmits urgent maritime safety information such as navigational and meteorological warnings and forecasts to vessels. For the safe navigation of smart ships, this information from different systems should be shared harmoniously in the Common Maritime Data Structure (CMDS). To share NAVTEX messages as CMDS, words in NAVTEX messages must be semantically classified and placed within the CMDS structure. While traditional parsing methods are typically used to understand message semantics, NAVTEX requires natural language processing methods with deep learning due to its unstructured messages. This paper applies six types of Bi-LSTM CRF-based deep learning models to NAVTEX navigational safety messages and analyzes the results to find the most suitable model for understanding the semantics of each word in NAVTEX messages. This technique can be applied to accurately convey the meaning of NAVTEX navigational safety messages to equipment that requires navigational safety information on smart ships without human intervention.

Common data models to streamline metabolomics processing and annotation, and implementation in a Python pipeline.

To standardize metabolomics data analysis and facilitate future computational developments, it is essential to have a set of well-defined templates for common data structures. Here we describe a collection of data structures involved in metabolomics data processing and illustrate how they are utilized in a full-featured Python-centric pipeline. We demonstrate the performance of the pipeline, and the details in annotation and quality control using large-scale LC-MS metabolomics and lipidomics data and LC-MS/MS data. Multiple previously published datasets are also reanalyzed to showcase its utility in biological data analysis. This pipeline allows users to streamline data processing, quality control, annotation, and standardization in an efficient and transparent manner. This work fills a major gap in the Python ecosystem for computational metabolomics.

Web Mining of Online Resources for German Labor Market Research and Education: Finding the Ground Truth?

The labor market is highly dependent on vocational and academic education, training, retraining, and further education in order to master challenges such as advancing digitalization and sustainability. Further training is a key factor in ensuring a qualified workforce, the employability of all employees, and, thus, national competitiveness and innovation. In the contribution at hand, we explore an innovative way to derive knowledge about learning pathways by connecting the dots from different data sources of the German labor market. In particular, we focus on the web mining of online resources for German labor market research and education, such as online advertisements, information portals, and official government websites. A key question for working with different data sources is how to find the ground truth and common data structures that can be used to make the data interoperable. We discuss how to classify and summarize web data from different platforms and which methods can be used for extracting data, entities and relationships from online resources on the German labor market to build a network of educational pathways. Our proposed solution is based on the classification of occupations (KldB) and related document codes (DKZ), and combines natural language processing and knowledge graph technologies. Our research provides the foundation for further investigation into educational pathways and linked data for labor market research. While our work focuses on German data, it is also useful for other German-speaking countries and could easily be extended to other languages such as English.

Simultaneous variable selection and estimation in semiparametric regression of mixed panel count data.

Mixed panel count data represent a common complex data structure in longitudinal survey studies. A major challenge in analyzing such data is variable selection and estimation while efficiently incorporating both the panel count and panel binary data components. Analyses in the medical literature have often ignored the panel binary component and treated it as missing with the unknown panel counts, while obviously such a simplification does not effectively utilize the original data information. In this research, we put forward a penalized likelihood variable selection and estimation procedure under the proportional mean model. A computationally efficient EM algorithm is developed that ensures sparse estimation for variable selection, and the resulting estimator is shown to have the desirable oracle property. Simulation studies assessed and confirmed the good finite-sample properties of the proposed method, and the method is applied to analyze a motivating dataset from the Health and Retirement Study.

Putting everything in its place: using the INSDC compliant Pathogen Data Object Model to better structure genomic data submitted for public health applications.

Fast, efficient public health actions require well-organized and coordinated systems that can supply timely and accurate knowledge. Public databases of pathogen genomic data, such as the International Nucleotide Sequence Database Collaboration (INSDC), have become essential tools for efficient public health decisions. However, these international resources began primarily for academic purposes, rather than for surveillance or interventions. Now, queries need to access not only the whole genomes of multiple pathogens but also make connections using robust contextual metadata to identify issues of public health relevance. Databases that over time developed a patchwork of submission formats and requirements need to be consistently organized and coordinated internationally to allow effective searches.To help resolve these issues, we propose a common pathogen data structure called the Pathogen Data Object Model (DOM) that will formalize the minimum pieces of sequence data and contextual data necessary for general public health uses, while recognizing that submitters will likely withhold a wide range of non-public contextual data. Further, we propose contributors use the Pathogen DOM for all pathogen submissions (bacterial, viral, fungal, and parasites), which will simplify data submissions and provide a consistent and transparent data structure for downstream data analyses. We also highlight how improved submission tools can support the Pathogen DOM, offering users additional easy-to-use methods to ensure this structure is followed.

Capturing Types

Type systems usually characterize the shape of values but not their free variables. However, many desirable safety properties could be guaranteed if one knew the free variables captured by values. We describe CC < :◻ , a calculus where such captured variables are succinctly represented in types, and show it can be used to safely implement effects and effect polymorphism via scoped capabilities. We discuss how the decision to track captured variables guides key aspects of the calculus, and show that CC < :◻ admits simple and intuitive types for common data structures and their typical usage patterns. We demonstrate how these ideas can be used to guide the implementation of capture checking in a practical programming language.

Formal Specification and Verification of JDK’s Identity Hash Map Implementation

Hash maps are a common and important data structure in efficient algorithm implementations. Despite their wide-spread use, real-world implementations are not regularly verified. In this article, we present the first case study of the IdentityHashMap class in the Java JDK. We specified its behavior using the Java Modeling Language (JML) and proved correctness for the main insertion and lookup methods with KeY, a semi-interactive theorem prover for JML-annotated Java programs. Furthermore, we report how unit testing and bounded model checking can be leveraged to find a suitable specification more quickly. We also investigated where the bottlenecks in the verification of hash maps lie for KeY by comparing required automatic proof effort for different hash map implementations and draw conclusions for the choice of hash map implementations regarding their verifiability.

BioThings Explorer: a query engine for a federated knowledge graph of biomedical APIs.

Knowledge graphs are an increasingly common data structure for representing biomedical information. These knowledge graphs can easily represent heterogeneous types of information, and many algorithms and tools exist for querying and analyzing graphs. Biomedical knowledge graphs have been used in a variety of applications, including drug repurposing, identification of drug targets, prediction of drug side effects, and clinical decision support. Typically, knowledge graphs are constructed by centralization and integration of data from multiple disparate sources. Here, we describe BioThings Explorer, an application that can query a virtual, federated knowledge graph derived from the aggregated information in a network of biomedical web services. BioThings Explorer leverages semantically precise annotations of the inputs and outputs for each resource, and automates the chaining of web service calls to execute multi-step graph queries. Because there is no large, centralized knowledge graph to maintain, BioThings Explorer is distributed as a lightweight application that dynamically retrieves information at query time. More information can be found at https://explorer.biothings.io and code is available at https://github.com/biothings/biothings_explorer.

More Fixpoints! (Functional Pearl)

Haskell’s laziness allows the programmer to solve some problems naturally and declaratively via recursive equations. Unfortunately, if the input is “too recursive”, these very elegant idioms can fall into the dreaded black hole, and the programmer has to resort to more pedestrian approaches. It does not have to be that way: We built variants of common pure data structures (Booleans, sets) where recursive definitions are productive. Internally, the infamous unsafePerformIO is at work, but the user only sees a beautiful and pure API, and their pretty recursive idioms – magically – work again.

Efficient linearizability checking for actor‐based systems

AbstractRecent demand for distributed software had led to a surge in popularity in actor‐based frameworks. However, even with the stylized message passing model of actors, writing correct distributed software is still difficult. We present our work on linearizability checking in DS2, an integrated framework for specifying, synthesizing, and testing distributed actor systems. The key insight of our approach is that often subcomponents of distributed actor systems represent common algorithms or data structures (e.g., a distributed hash table or tree) that can be validated against a simple sequential model of the system. This makes it easy for developers to validate their concurrent actor systems without complex specifications. DS2 automatically explores the concurrent schedules that system could arrive at, and it compares observed output of the system to ensure it is equivalent to what the sequential implementation could have produced. We describe DS2's linearizability checking and test it on several concurrent replication algorithms from the literature. We explore in detail how different algorithms for enumerating the model schedule space fare in finding bugs in actor systems, and we present our own refinements on algorithms for exploring actor system schedules that we show are effective in finding bugs.

An edge re-ordering based acceleration architecture for improving data locality in graph analytics applications

Data structure is the key in Edge Computing where various types of data are continuously generated by ubiquitous devices. Within all common data structures, graphs are used to express relationships and dependencies among human identities, objects, and locations. They are also expected to become one of the most important data infrastructures in the near future. Furthermore, as graph processing often requires random accesses to vast memory spaces, conventional memory hierarchies with caches cannot work efficiently. To alleviate such memory access bottlenecks in graph processing, we present a solution through vertex accesses scheduling and edge array re-ordering, in parallel with the execution of graph processing applications to improve both temporal and spatial locality of memory accesses, especially for edge-centric graph analytics which are popular means in handling dynamic graphs. Our proposed architecture is evaluated and tested through both trace-based cache simulations and cycle-accurate FPGA-based prototyping. Evaluation results show that our proposal has a potential of significantly reducing the Last Level Cache (LLC) misses by 62.60% in general among PageRank and BFS algorithms. Meanwhile, evaluations with the FPGA prototype successfully reduce the quantity of Miss-Per-Kilo-Instructions (MPKI) for LLC by 56.27% on average.

Rethinking databases: The fib project for a connected data repository

AbstractThe need for reliable test data to verify scientific theories together with the necessity to calibrate code‐oriented expressions and their level of safety have pushed researchers to develop test databases. Such databases collect the work of multiple independent scientific initiatives compiling experimental evidences on similar test configurations but with different parameters such as geometry, size of the component tested, boundary conditions, and/or mechanical properties of the materials. Such databases, usually collected for specific purposes, have traditionally faced maintenance, and especially consistency issues. Also, they have given rise to conflicting values due to different interpretations of the same data entries in databases elaborated by different researchers. In addition, such databases are frequently designed and configured to serve for one purpose, although their data could be used in multiple manners. Recent developments regarding machine learning and artificial intelligence have opened the door to exploiting historically collected test results by data mining techniques. In an effort to improve the current situation concerning databases, the fib has launched a wide and open initiative on test data management. It consists of a common data structure, sufficiently flexible to serve for multiple purposes, hosting very different test setups, and meant to be used for cross analyses of data. This allows reusing data originally developed for one purpose to analyze other aspects, enhancing the value of current experimental work. Also, a consistent management structure is defined taking advantage of the fib network, with database editors, collectors and users, allowing for a transparent and documented peer‐review process while incorporating new test data. This paper presents the main basis grounding the test data management as well as details on the first two completed applications, referring to fiber‐reinforced concrete (material data) and to punching of slab‐column connections (structural response), showcasing the framework versatility and universality.

Surface network and drainage network: towards a common data structure

The surface network is an application of the Morse-Smale complex to digital terrain models connecting ridges and thalwegs of the terrain in a planar, undirected graph. Although it provides a topological structure embedding critical elements of the terrain, its application to morphological analysis and hydrology remains limited mainly because the drainage network is the most relevant structure for analysis and it cannot be derived from the surface network. The drainage network is a directed, hierarchical graph formed by streams. Ridges of the surface network are not equivalent to drainage divides, which are not contained in the drainage network, and there is no direct association between thalwegs and streams. Therefore, this paper proposes to extend the surface network into a new structure that also embeds the drainage network. This is done by (1) revising the definition of ridges so that they include drainage divides and (2) assigning a flow direction to each thalweg, taking into account spurious depressions to avoid flow interruption. We show that this extended surface network can be used to compute the flow accumulation and different hydrographic features such as drainage basins and the Strahler order. The drainage network extracted from the extended surface network is compared to drainage networks computed with the traditional D8 approach in three case studies. Differences remain minor and are mainly due to the elevation inaccuracy in flat or slightly convex areas. Hence, the extended surface network provides a richer data structure allowing the use of a common topological data structure in both terrain analysis and hydrology.

The Backbone of Computing: An Exploration of Data Structures

Data structures are the foundation of computing, providing efficient ways to store and manipulate data. They are essential for designing and implementing algorithms that can handle large amounts of information quickly and accurately. In this paper, we explore the world of data structures, examining their basic principles, properties, and uses. We start by introducing the most common data structures, such as arrays, linked lists, stacks, queues, trees, and graphs, and discuss their advantages and limitations.

Hardware/Software Co-Assurance for the Rust Programming Language Applied to Zero Trust Architecture Development

Zero Trust Architecture requirements are of increasing importance in critical systems development. Zero trust tenets hold that no implicit trust be granted to assets based on their physical or network location. Zero Trust development focuses on authentication, authorization, and shrinking implicit trust zones to the most granular level possible, while maintaining availability and minimizing authentication latency. Performant, high-assurance cryptographic primitives are thus central to successfully realizing a Zero Trust Architecture. The Rust programming language has garnered significant interest and use as a modern, type-safe, memory-safe, and potentially formally analyzable programming language. Our interest in Rust particularly stems from its potential as a hardware/software co-assurance language for developing Zero Trust Architectures. We describe a novel environment enabling Rust to be used as a High-Level Synthesis (HLS) language, suitable for secure and performant Zero Trust application development. Many incumbent HLS languages are a subset of C, and inherit many of the well-known security shortcomings of that language. A Rust-based HLS brings a single modern, type-safe, memory-safe, high-assurance development language for both hardware and software. To study the benefits of this approach, we crafted a Rust HLS subset, and developed a frontend to the hardware/software co-assurance toolchain due to Russinoff and colleagues at Arm, used primarily for floating-point hardware formal verification. This allows us to leverage a number of existing hardware/software co-assurance tools with a minimum investment of time and effort. In this paper, we describe our Rust subset, detail our prototype toolchain, and describe the implementation, performance analysis, formal verification and validation of representative Zero Trust algorithms and data structures written in Rust, emphasizing cryptographic primitives and common data structures.

Computational methods for predicting human behaviour in smart environments

This systematic literature review presents the computational methods of human behaviour prediction research from Pentland and Liu’s seminal 1999 paper on human behaviour prediction to the latest research to date. The PRISMA framework for systematic reviews was used as the review methodology to structure this information aggregation. This review provides a high-level summary of the field with key areas identified for new research. The results show that there are frequently used datasets for training predictive models: MavHome, MavLab, LIARA, CASAS, PlaceLab, and REDD. Accuracies in the range of 43.9% to 100% for predictions of varying complexity. Common data structures for modelling behavioural data: Vectors, tables, trees, Markov models, and graphs. Algorithms that fall into three distinct categories: Machine Learning (NN, RL, LSTM), Probabilistic Graphical Models (namely Bayesian and Markov variants), and Statistical and Trend Analysis (ARIMA, Prophet). Additionally, we document other notably useful algorithms that fall outside of these three main categories including Jaro-Winkler and Levenshtein distances. Opportunities identified for further research include the use of audio as the data source for behaviour prediction methods, and applying times-series prediction machine learning algorithms (RNN, LSTM) to the smart home problem space.

A New Framework for Joint Trajectory Planning Based on Time-Parameterized B-Splines

This paper discusses a new, highly effective trajectory planning paradigm for robotics and manufacturing systems. The proposed approach offers a more flexible solution to a problem classically solved in closed-form, defining a generalization of the existing methods by providing a common data structure based on time-parameterized curves to represent exactly or approximately the joint trajectories. In the framework, the most generic approach to constrain or optimize a trajectory is to set up an iterative loop and act on the parameters through computer-aided design concepts, such as shape modeling, symbolic calculation, derivation and integration. The iso-parametric representation allows to define these procedures as macro-instructions to develop scalable and reusable solutions. A benchmark is proposed to confirm the advantages of the new approach over an established solution. Finally, an experimental case study of a single axis motion control is provided to illustrate and clarify the method. • A new Iso-Parametric Trajectory Planning (IPTP) framework is presented. • Time-parameterized univariate B-splines represent trajectories in the joint space. • CAD concepts are applied to design and modify the trajectories. • IPTP allows both to design mechanical cam profiles and to plan the motion of electric axes. • IPTP defines scalable and reusable solutions through macro-instructions definition.

Enabling Process Mining In Global Production Networks

In today's globalized economy, global production networks (GPN) play a central role in ensuring competitiveness. The number of interconnections and size of the networks lead to an ever-increasing complexity. With the help of data-driven methods, such as process mining, transparency over material flows in GPNs can be increased, making the complexity manageable. However, the adoption of process mining is associated with challenges. In GPN, different components are produced across various production stages, each with their own characteristics. In the course of the material flow, the components are then assembled into component groups, which in turn represent intermediate products or the finished end product. Consequently, converging material flows are the result. These circumstances make it difficult to build unique case identifiers and a comprehensive process data model. Furthermore, the identification of the right data sources is a challenge, since in GPN often different information systems are available in individual production lines or locations. We present an approach that supports overcoming these challenges and enables process mining in GPNs. For this purpose, the presented approach contains a process data model that takes into account all GPN specifics and requirements of process mining. This includes the handling of different component types and their properties as well as the merging of process instances in converging material flows. Through this, the individual sub-components of a finished product with their respective sub-processes are traceable. Further, we provide a procedure that supports the user in data identification and extraction by describing common data structures in information systems, which are usable for the generation of process mining event logs in GPN. Based on these data structures, we provide standardized input data tables into which the identified raw data can be extracted. Subsequently, the transfer into the presented process data model is automated by a data transformation algorithm, which takes into account the specifics of material flows in GPN. The approach was applied, implemented and validated in the GPN of an automotive manufacturer.

Implementing FAIR data management practices in shellfish sanitation

In the United States (U.S.), state agencies in charge of mariculture regulation are mandated under the U.S. Food and Drug Administration’s (FDA) National Shellfish Sanitation Program (NSSP) to monitor fecal indicator bacteria (FIB) concentrations, commonly of fecal coliforms, to determine the safety of coastal waters for supporting harvestable shellfish for human consumption. Many states have monitored bacteriological water quality for decades, creating impressive long-term records with the potential to advance foundational understanding of coastal systems and contribute to other complementary monitoring efforts. However, state shellfish sanitation programs differ in how they collect, manage, and share bacteriological monitoring data, resulting in their data typically being available in disparate state-level repositories with non-standardized database structures. Here, we outline three key recommendations as to how shellfish sanitation programs could implement practices to make their data more Findable, Accessible, Interoperable, and Reusable (FAIR), in turn creating new opportunities for the full potential of the data to be realized. We also offer sample materials of a standardized database, ShellBase, to provide an example of how diverse shellfish sanitation data may be integrated with a common data structure.

Data Visualization in Programming Education

"Understanding data structures is fundamental for mastering algorithms. In order to solve problems and tasks, students must be able to choose the most appropriate data structure in which the data is stored and that helps in the process of the solution. Of course, there is no single correct solution, but in many cases, it is an important step to find the most efficient data structure to solve the task. In this paper, I will present the most common data representations and data structures that can help get to know them, to master them, and then to reuse them for solving different tasks."