Intermediate Data Structure Research Articles

The Development of Microhistorical Databases in Norway. A Historiography

Norwegian work on microdata started out with the full count 1801 census and census and vital records from around the capital. Today, most census and ministerial records from 1801 until the mid-20th century have been scanned, transcriptions are being completed, much is encoded and made available via the websites of the Digital National Archives and UiT The Arctic University of Norway. This article complements a previous publication on empirical results from historical microdata. It is primarily organized by technical issues: digitization of source materials, encoding and standardization, building of the Historical Population Register for the period since 1800, record linkage and source criticism as well as GIS. Presently, partner institutions are building the Historical Population Register with prolonged support from the Norwegian Research Council. This will contain longitudinal records of the nine million persons who lived in Norway since 1800. The register increasingly makes it possible to follow the entire population. Unique personal IDs with corresponding URLs to the person page providing links to many sources introduce a new level of historical documentation. Cross-sectional and vital records are being interlinked with automatic and manual record linkage software. Longitudinal data is available for searching as timelines and in Intermediate Data Structure format from UiT The Arctic University and for searching at Histreg.no, which also caters for manual editing. We are well on the way to creating a database that can fill the void in the two centuries before the Central Population Register starts in 1964.

Toward Trustworthy and Privacy-Preserving Federated Deep Learning Service Framework for Industrial Internet of Things

In this article, we propose a trustworthy privacy-preserving federated learning (FL)-based deep learning (DL) service framework for Industrial Internet of Things-enabled systems. FL mitigates the privacy issues of the traditional collaborative learning model by aggregating multiple locally trained models without sharing any datasets among the participants. Nevertheless, the FL-based DL (FDL) model cannot be trusted as it is susceptible to intermediate results and data structure leakage during the model aggregation process. The proposed framework introduces an edge and cloud-powered service-oriented architecture identifying the key components and a service model for residual networks-based FDL with differential privacy for generating trustworthy locally trained models. The service model decomposes the functionality of the overall FDL process as services to ensure trustworthy execution through privacy preservation. Finally, we develop a privacy-preserving local model aggregation mechanism for FDL. We perform several experiments to assess the performance of the proposed framework.

Fast Algorithms for Denial Constraint Discovery

Denial constraints (DCs) are an integrity constraint formalism widely used to detect inconsistencies in data. Several algorithms have been devised to discover DCs from data, as manually specifying them is burdensome and, worse yet, error-prone. The existing algorithms follow two basic steps: building an intermediate data structure from records, then enumerating the DCs from that intermediate. However, current algorithms are often inefficient in computing these intermediates. Also, it is still unclear which enumeration algorithm performs best since some of the available algorithms have not yet been compared to each other. In response, we present a set of new algorithms with improved design choices. We introduce a parallel pipeline for rapidly computing the intermediate using custom data representations, algorithms, and indexes. For DC enumeration, we propose an inverted index, pruning, and parallel search strategies. We present hybrid approaches that integrate our techniques with previous enumeration algorithms, improving their performance in many scenarios. Our experimental study shows that the proposed DC discovery algorithms are consistently much faster (up to an order of magnitude) than the current state-of-the-art.

The Groningen Integral History Cohort Database. Development, Design and Output

The Groningen Integral History project launched in 1987 aimed to sketch the lives of people from various social classes in the Dutch province of Groningen in the 19th and early 20th century. One part was the creation of the Groningen Integral History Cohort Database (GIHCD), reconstructing complete individual life courses of 5,280 persons (RPs) born between 1811 and 1872. The quality of the database has become very high by now, despite the lengthy and difficult process of shaping it over 35 years. More than 98% of the RPs (and for some parts of the database even more than 99%) could be followed until their death or until a migration abroad. Even for the life courses of those moving abroad information is available for most RPs. In this article, we primarily focus on the rural part of the database (n = 4,320), the quality of which is the highest and has had the most significant tangible research impact. Building on information from the Dutch civil registration system (from 1811) and the population registers (from 1850), the database includes multiple individual-level variables. In the technical part of the article, we provide an extensive overview of the available variables and summarize the transformation of the rural part of the database into an Intermediate Data Structure (IDS). Since the early 1990s, historians from the University of Groningen have used GIHCD in quite some publications. At the end of this article, we provide a summary of the main outcomes of these publications.

Reflections on the Intermediate Data Structure (IDS).

The Intermediate Data Structure (IDS) encourages sharing historical life course data by storing data in a common format. To encompass the complexity of life histories, IDS relies on data structures that are unfamiliar to most social scientists. This article examines four features of IDS that make it flexible and expandable: the Entity-Attribute-Value model, the relational database model, embedded metadata, and the Chronicle file. I also consider IDS from the perspective of current discussions about sharing data across scientific domains. We can find parallels to IDS in other fields that may lead to future innovations.

Reflections on the Use of the Intermediate Data Structure (IDS) in Historical Demographic Research

The Intermediate Data Structure (IDS) was developed as a strategy aimed at standardizing the dissemination of micro-level historical demographic data. The structure provides a common and clear data strategy which facilitates studies that consider several databases, and the development and exchange of software. Based on my own experiences from working with the IDS, in this article I provide reflections on the use of IDS to create datasets for analysis and to conduct comparative demographic research.

Historical Databases Now and in the Future

Kees Mandemakers has enriched historical databases in the Netherlands and internationally through the development of the Historical Sample of the Netherlands, the Intermediate Data Structure, a practical implementation of rule-based record linking (LINKS) and personal encouragement of high quality longitudinal data in a number of countries.

High‐performance IO for seismic processing on the cloud

SummaryMost of the applications in the seismology field rely on the processing of up to hundreds of terabytes of data and their performance is strongly affected by IO operations. In this article, we analyze the main file structures currently used to store seismic data and propose a new intermediate data structure to improve IO performance while still complying with established standards. We show that, throughout a common workflow in seismic data analysis, our IO performance gain greatly surpasses the overhead of translating data to the intermediate structure. This approach enables a speedup of up to 208 times in reading time when using classical standards (e.g., SEG‐Y) and our intermediate structure is up to 1.8 times more efficient than modern formats (e.g., ASDF). Considering cache‐friendly applications, our speedups over the direct use of SEG‐Y reach 8000 times. We also performed a cost analysis on the AWS cloud showing that, in our approach, HDDs can be 1.25 times more cost‐effective than SSDs.

The Scanian Economic-Demographic Database (SEDD)

The Scanian Economic-Demographic Database (SEDD) is a high-quality longitudinal data resource spanning the period 1646-1967. It covers all individuals born in or migrated to the city of Landskrona and five rural parishes in western Scania in southern Sweden. The entire population present in the area is fully covered after 1813. At the individual level, SEDD combines various demographic and socioeconomic records, including causes of death, place of birth and geographic data on the place of residence within a parish. At the family level, the data contain a combination of demographic records and information on occupation, landholding and income. The data for 1813-1967 was structured in the model of the Intermediate Data Structure (IDS). In addition to storing source data in the SEDD IDS tables, a wide range of individual- and context-level variables were constructed, which means that most types of analyses using SEDD can be conducted without the need of further elaboration of the data. This article discusses the source material, linkage methods, and structure of the database.

Re-introducing the Cambridge Group Family Reconstitutions.

English Population History from Family Reconstitution 1580-1837 was important both for its scope and its methodology. The volume was based on data from family reconstitutions of 26 parishes carefully selected to represent 250 years of English demographic history. These data remain relevant for new research questions, such as studying the intergenerational inheritance of fertility and mortality. To expand their availability the family reconstitutions have been translated into new formats: a relational database, the Intermediate Data Structure (IDS) and an episode file for fertility analysis. This paper describes that process and examines the impact of methodological decisions on analysis of the data. Wrigley, Davies, Oeppen, and Schofield were sensitive to changes in the quality of the parish registers and cautiously applied the principles of family reconstitution developed by Louis Henry. We examine how these choices affect the measurement of fertility and biases that are introduced when important principles are ignored.

Introduction: Intergenerational Transmissions of Infant Mortality using the Intermediate Data Structure (IDS)

It has previously been shown that infant mortality clusters in a subset of families, a phenomenon which was observed in historical populations as well as contemporary developing countries. A transmission of death clustering across generations has also been shown in Belgium, but it is unknown whether such effects are specific to the studied context or are also found in other areas. The current article introduces a special issue devoted to analysing intergenerational transmissions of infant mortality across the maternal line in Belgium, the Netherlands, northern and southern Sweden, and Norway. Taking advantage of the Intermediate Data Structure (IDS), the five empirical studies created datasets for analysis and ran statistical models using exactly the same programs, which are also published within the special issue. These works are the first set of studies using the IDS on several databases for comparative purposes. Consistent results across the studied contexts were shown: transfers of infant mortality across the maternal line were seen in all five areas. In addition, the works have shown that there are large advantages of adopting the IDS for historical demographic research. The structure has in fact allowed researchers to conduct studies which were fully comparable, transparent and replicable.

Intergenerational Transfers in Infant Mortality in Southern Sweden, 1740-1968


 
 
 Studies conducted in historical populations and developing countries have evidenced the existence of clustering in infant deaths, which could be related to genetic inheritance, early life exposures, and/or to social and cultural factors such as education, socioeconomic status or parental care. A transmission of death clustering has also been found across generations. This paper is one of five studies that analyses intergenerational transmissions in infant mortality by using a common program to create the dataset for analysis and run the statistical models with data stored in the Intermediate Data Structure. The results of this study show that in five rural parishes in Scania, the southernmost province of Sweden, during the years 1740-1968 infant mortality was transmitted across generations. Children whose maternal grandmothers experienced two or more infant deaths had higher risks of dying in infancy. The results remained consistent when restricting the sample only to cases where the grandmother had been observed for her entire reproductive history or when controlling for socioeconomic status. When running sex specific models, significant effects of the number of infant deaths of the grandmother were observed for girls but not for boys.
 
 

Program for Studying Intergenerational Transmissions in Infant Mortality Using the Intermediate Data Structure (IDS)

Studies conducted in historical populations and developing countries have evidenced the existence of clustering in infant deaths, which could be related to genetic inheritance and/or to social and cultural factors such as education, socioeconomic status or parental care. A transmission of death clustering has also been found across generations. One way of expanding the knowledge on intergenerational transfers in infant mortality is by conducting comparable studies across different populations. The Intermediate Data Structure (IDS) was developed as a strategy aimed at simplifying the collecting, storing and sharing of historical demographic data. The current work presents two programs that were developed in STATA to construct a dataset for analysis and run statistical models to study intergenerational transfers in infant mortality using databases that are stored in the IDS. The programs use information stored in the IDS tables and after elaborating such information produce Excel files with results. They can be used with any longitudinal database constructed from church books, civil registers, or population registers.

A heuristic based dependency calculation technique for rough set theory

Feature selection is the process of selecting subset of features that still provide maximum amount of the information that otherwise is provided by the entire set of conditional attributes. Many approaches have been proposed so far in literature for this purpose. Recently the rough set based approaches have become dominant. Majority of these approaches use attribute dependency to find significance of attributes. Problem with this measure is that it uses positive region to calculate dependency which is a computationally expensive job. As a consequence, it degrades the performance of the feature selection algorithms using this measure. In this paper, we have proposed a new heuristic based dependency calculation technique by avoiding the positive region. The proposed method uses a heuristics approach by finding the consistent records regarding each decision class in the dataset. Using this method, allows us to calculate dependency by avoiding the positive region, which ultimately enhances the computational efficiency of the underlying feature selection algorithm thus enabling it to be used for dataset beyond smaller size. In order to calculate dependency by using the proposed method, we have used a two-dimensional grid as intermediate data structure. Number of feature selection algorithms were used with proposed solution on various publically available datasets to justify it. A comparison framework was used to compare the proposed solution with conventional methods. Results have justified the proposed solution both in terms of its efficiency and effectiveness.

The Intergenerational Transfer of Infant Mortality in Northern Norway during the 19th and Early 20th Centuries


 
 
 This paper is one of a series of five studying the intergenerational transfer of infant mortality down the maternal line. All five studies share the same theoretical and methodological design, and use data derived from a standard database format: the Intermediate Data Structure (IDS). The data for the research reported in this paper were derived from a longitudinal dataset covering the 19th and 20th century population of the province of Troms in Northern Norway. Our results suggest that there was an element of intergenerational transmission in women’s risk of experiencing an infant death; the children of a woman whose mother had had a high number of infant deaths also had a greater risk of dying before their first birthday. The risk of an infant death occurring among the children of daughters from such ‘high risk’ families was at least 30 per cent higher than that amongst infants born to the daughters of mothers who had experienced zero infant deaths.
 
 

Mortality Clustering in the Family. Fast Life History Trajectories and the Intergenerational Transfer of Infant Death in Late 19th- and Early 20th-Century Antwerp, Belgium


 
 
 In this article, we investigate to what degree infant mortality risk was transferred from grandmothers to mothers in the Antwerp district, Belgium, during the late nineteenth and early twentieth century. We also investigate some of the determinants of infant mortality and explore the role of the family - paternal factors (presence, age, and social class), mother’s childcare experience, and infant household location - in the survival of infants. The data for this research were retrieved from the Antwerp COR*-database and were transferred into the Intermediate Data Structure (IDS). The results of the survival models show that women whose mother experienced three or more infant deaths had a 77% higher risk of experiencing the loss of an infant themselves, compared to women whose mother experienced zero infant deaths in the past. These results remained robust after controlling for potential mediating and moderating factors. The results on the age of the mother at birth, her marital status, as well as the living environment suggest that at least part of the intergenerational transfer in infant mortality can be explained on the basis of life history theory: women who grew up in a high-risk family tended to reproduce earlier and faster, and often raised their children without a partner. In this way they unconsciously created riskier conditions for the raising of their own infants: the mothers had little life experience, limited resources, and often no assistance from a partner. As a result, their own children were also at an increased risk of dying in infancy.
 
 

Intergenerational Transfers of Infant Mortality in 19th-Century Northern Sweden

This contribution is part of an international comparative initiative with the aim to assess the analytical power of the Intermediate Data Structure (IDS) in a study of possible intergenerational transmissions of death in infancy. An evaluation of the data in applied research will be useful for further development of the IDS structure and for its future use in comparative research. An additional methodological aim for this part of the study is to evaluate and compare different models for statistical analysis of intergenerational transfers. The analysis is based on a cohort of mothers born 1826-1854, whose experiences of infant mortality are compared to the ones of the previous generation, the grandmothers. Data are collected from Swedish parish records, available in the database POPUM at the Demographic Data Base in Umeå. The analysis shows a clear association between infant mortality among mothers and grandmothers. The probability of an infant death for a woman is increased if her mother also had experienced an infant death. Having tested for different approaches of analysis, we found that simple models with few restrictive assumptions gave similar results as more complicated models. Since it is easy to feel confident in the models with the weakest assumptions, we argue that such models are preferred for this type of analysis.

Like Mother, Like Daughter. Intergenerational Transmission of Infant Mortality Clustering in Zeeland, the Netherlands, 1833-1912

The burden of infant mortality is not shared equally by all families, but clusters in high risk families. As yet, it remains unclear why some families experience more infant deaths than other families. Earlier research has shown that the risk of early death among infants may at least partially be transmitted from grandmothers to mothers. In this paper, we focus on the intergenerational transmission of mortality clustering in the Netherlands in the province of Zeeland between 1833 and 1912, using LINKS Zeeland, a dataset containing family reconstitutions based on civil certificates of birth, marriage and death. We assess whether intergenerational transmission of mortality clustering occurred in Zeeland, and if so, whether it can be explained on the basis of the demographic characteristics of the families in which the infants were born. In addition, we explore the opportunities for comparative research using the Intermediate Data Structure (IDS). We find that mortality clustering is indeed transmitted from grandmothers to mothers, and that the socioeconomic status of the family, the survival of mothers and fathers, and the demographic characteristics of the family affected infant survival. However, they explain the heterogeneity in infant mortality at the level of the mother only partially.

Editorial

Historical Life Course Studies, a journal in population studies, aims to stimulate and facilitate the implementation of IDS (Intermediate Data Structure, a standard data format for large historical databases), and to publish the results from (comparative) research with the help of large historical databases. The journal publishes not only empirical articles, but also descriptions (of the construction) of new and existing large historical databases, as well as articles dealing with database documentation, the transformation of existing databases into the IDS format, the development of algorithms and extraction software and all other issues related to the methodology of large historical databases.

Feature selection using rough set-based direct dependency calculation by avoiding the positive region

Feature selection is the process of selecting a subset of features from the entire dataset such that the selected subset can be used on behalf of the entire dataset to reduce further processing. There are many approaches proposed for feature selection, and recently, rough set-based feature selection approaches have become dominant. The majority of such approaches use attribute dependency as criteria to determine the feature subsets. However, this measure uses the positive region to calculate dependency, which is a computationally expensive job, consequently effecting the performance of feature selection algorithms using this measure. In this paper, we have proposed a new heuristic-based dependency calculation method. The proposed method comprises a set of two rules called Direct Dependency Calculation (DDC) to calculate attribute dependency. Direct dependency calculates the number of unique/non-unique classes directly by using attribute values. Unique classes define accurate predictors of class, while non-unique classes are not accurate predictors. Calculating unique/non-unique classes in this manner lets us avoid the time-consuming calculation of the positive region, which helps increase the performance of subsequent algorithms. A two-dimensional grid was used as an intermediate data structure to calculate dependency. We have used the proposed method with a number of feature selection algorithms using various publically available datasets to justify the proposed method. A comparison framework was used for analysis purposes. Experimental results have shown the efficiency and effectiveness of the proposed method. It was determined that execution time was reduced by 63% for calculation of the dependency using DDCs, and a 65% decrease was observed in the case of feature selection algorithms based on DDCs. The required runtime memory was decreased by 95%.