Historical Big Data Research Articles

A Study on Historical Big Data Analysis of Surface Ecological Damage in the Coal Mining Area of Lvliang City Based on Two Mining Modes

Coal mining inevitably causes damage to the surface ecological environment. In response to the characteristics of multiple factors, wide scope, and long time series of surface ecological environment damage in coal mining subsidence areas, how to integrate multiple data sources and monitoring methods to achieve monitoring and trend analysis of ecological damage throughout the entire mining cycle still needs to be studied. In addition, the 110 mining method provides an innovative method for underground coal mining to reduce its surface ecological damage, but the differences in surface damage between the two mining modes and the effectiveness of the 110 method in realizing surface ecological damage-reducing mining still need to be studied in depth. Therefore, this study takes the surface ecological damage in the mining area of Lvliang City, a typical resource city in Shanxi Province, China, as the object. It establishes a four-in-one “Satellite–UAV–Ground–Underground” information monitoring method, proposes a historical big data evolution analysis method, constructs three spatial scales of historical big databases, clarifies the current situation and development trend of damage in coal mining areas in Lvliang City and explores the differences in surface ecological environment damage characteristics in coal mining areas based on the 121 and 110 mining methods. The results show that: (1) The existing coal mining subsidence area in Lvliang City is as high as 92,191.47 hectares, and it is expected to continue to increase to 130,739.55 hectares in the future 2035, with a growth rate of 41.812%, which realizes the goals of mapping the current situation, retracing the history and predicting the future of the ecological damage of the surface in Lvliang City. (2) The surface NDVI of the 110 working face is restored to the average level of the mining area faster than that of the 121 working face. The surface crack width, step displacement, length, distribution density, and surface settlement height of the 110 working face are all smaller than those of the 121 working face. It has been verified that the unique top-cutting and swelling filling effect of the 110 methods can effectively reduce the ecological damage caused by coal mining subsidence. And its widespread application can effectively realize the ecological environmental protection of the coal mine area and contribute to the high-quality development of the coal industry in Lvliang City.

A Dynamic Prediction Framework for Urban Public Space Vitality: From Hypothesis to Algorithm and Verification

Predicting and assessing the vitality of public urban spaces is crucial for effective urban design, aiming to prevent issues such as “ghost streets” and minimize resource wastage. However, existing assessment methods often lack temporal dynamics or heavily rely on historical big data, limiting their ability to accurately predict outcomes for unbuilt projects. To address these challenges, this study integrates previous methodologies with observations of crowd characteristics in public spaces. It introduces the crowd-frequency hypothesis and develops an algorithm to establish a time-dimensional urban vitality dynamic prediction model. Through a case study of the Rundle Mall neighborhood in Adelaide, Australia, the effectiveness of the prediction model was validated using on-site observation sampling and comparative verification. The prediction model framework allows for the determination of urban vitality within specific time ranges by directly inputting basic information, providing valuable support to urban planners and government officials during the design and decision-making processes. It offers a cost-effective approach to achieve sustainable urban vitality construction. Furthermore, machine learning techniques, specifically the decision tree model, were applied to case data to develop a set of preliminary algorithm tools, which enable output of reference urban vitality levels (high-medium-low).

A Lightweight, Secure Big Data-Based Authentication and Key-Agreement Scheme for IoT with Revocability

With the rapid development of Internet of Things (IoT), designing a secure two-factor authentication scheme for IoT is becoming increasingly demanding. Two-factor protocols are deployed to achieve a higher security level than single-factor protocols. Given the resource constraints of IoT devices, other factors such as biometrics are ruled out as additional authentication factors due to their large overhead. Smart cards are also prone to side-channel attacks. Therefore, historical big data have gained interest recently as a novel authentication factor in IoT. In this paper, we show that existing big data-based schemes fail to achieve their claimed security properties such as perfect forward secrecy (PFS), key compromise impersonation (KCI) resilience, and server compromise impersonation (SCI) resilience. Assuming a real strong attacker rather than a weak one, we show that previous schemes not only fail to provide KCI and SCI but also do not provide real two-factor security and revocability and suffer inside attack. Then, we propose our novel scheme which can indeed provide real two-factor security, PFS, KCI, and inside attack resilience and revocability of the client. Furthermore, our performance analysis shows that our scheme has reduced modular exponentiation operation and multiplication for both the client and the server compared to Liu et al.’s scheme which reduces the execution time by one third for security levels of λ = 128 . Moreover, in order to cope with the potential threat of quantum computers, we suggest using lightweight XMSS signature schemes which provide the desired security properties with λ = 128 bit postquantum security. Finally, we prove the security of our proposed scheme formally using both the real-or-random model and the ProVerif analysis tool.

Novel Use of Social Media Big Data and Artificial Intelligence for Community Resilience Assessment (CRA) in University Towns

University towns face many challenges in the 21st century due to urbanization, increased student population, and higher educational institutions’ inability to house all their students on-campus. For university towns to be resilient and sustainable, the challenges facing them must be assessed and addressed. To carry out community resilience assessments, this study adopted a novel methodological framework to harness the power of artificial intelligence and social media big data (user-generated content on Twitter) to carry out remote studies in six university towns on six continents using Text Mining, Machine Learning, and Natural Language Processing. Cultural, social, physical, economic, and institutional and governance community challenges were identified and analyzed from the historical big data and validated using an online expert survey. This study gives a global overview of the challenges university towns experience due to studentification and shows that artificial intelligence can provide an easy, cheap, and more accurate way of conducting community resilience assessments in urban communities. The study also contributes to knowledge of research in the new normal by proving that longitudinal studies can be completed remotely.

Gender Bias in Big Data Analysis

This article combines humanistic "data critique" with informed inspection of big data analysis. It measures gender bias when gender prediction software tools (Gender API, Namsor, and Genderize.io) are used in historical big data research. Gender bias is measured by contrasting personally identified computer science authors in the well-regarded DBLP dataset (1950-1980) with exactly comparable results from the software tools. Implications for public understanding of gender bias in computing and the nature of the computing profession are outlined. Preliminary assessment of the Semantic Scholar dataset is presented. The conclusion combines humanistic approaches with selective use of big data methods.

GECEM Project Database: A digital humanities solution to analyse complex historical realities in early modern China and Europe

Abstract The GECEM Project Database stands out as a new Digital Humanities solution to accurately order and analyse the new historical Big Data gathered in Chinese and European historical archives. Traditional challenges such as capture, storage, analysis, data curation, searching, sharing, transfer, visualization, querying, updating, and information privacy are being tackled and solved within the design of this new multi-relational database. The implementation of this database has as its main innovative elements the capability of coding and cross-referring historical data in Chinese and Western languages. Thus, we present a new database as a digital solution to solve the long-lasting problem in historical research: the maximization and optimization of data collection when analysing historical sources and how to make use of a large amount of registers and information when developing case studies and work hypotheses for historical research. In this paper, we present a solution for data collection and analysis of probate inventories and trade records which is a relevant source when determining changes in patterns of consumption and global trade through the introduction of Chinese goods into Europe and vice versa during the early modern period.

A Survey on Deep Learning for Building Load Forecasting

Energy consumption forecasting is essential for efficient resource management related to both economic and environmental benefits. Forecasting can be implemented through statistical analysis of historical data, application of Artificial Intelligence (AI) algorithms, physical models, and more, and focuses on two directions: the required load for a specific area, e.g., a city, and the required load for a building. Building power forecasting is challenging due to the frequent fluctuation of the required electricity and the complexity and alterability of each building’s energy behavior. This paper focuses on the application of Deep Learning (DL) methods to accurately predict building (residential, commercial, or multiple) power consumption, by utilizing the available historical big data. Research findings are compared to state-of-the-art statistical models and AI methods of the literature to comparatively evaluate their efficiency and justify their future application. The aim of this work is to review up-to-date proposed DL approaches, to highlight the current research status, and to point out emerging challenges and future potential directions. Research revealed a higher interest in residential building load forecasting covering 47.5% of the related literature from 2016 up to date, focusing on short-term forecasting horizon in 55% of the referenced papers. The latter was attributed to the lack of available public datasets for experimentation in different building types, since it was found that in the 48.2% of the related literature, the same historical data regarding residential buildings load consumption was used.

Prediction of Sludge Volume Index in a Wastewater Treatment Plant Using Recurrent Neural Network

Sludge Volume Index (SVI) is one of the most important operational parameters in an activated sludge process. It is difficult to predict SVI because of the nonlinearity of data and variability operation conditions. With complex time-series data from Wastewater Treatment Plants (WWTPs), the Recurrent Neural Network (RNN) with an Explainable Artificial Intelligence was applied to predict SVI and interpret the prediction result. RNN architecture has been proven to efficiently handle time-series and non-uniformity data. Moreover, due to the complexity of the model, the newly Explainable Artificial Intelligence concept was used to interpret the result. Data were collected from the Nine Springs Wastewater Treatment Plant, Madison, Wisconsin, and the data were analyzed and cleaned using Python program and data analytics approaches. An RNN model predicted SVI accurately after training with historical big data collected at the Nine Spring WWTP. The Explainable Artificial Intelligence (AI) analysis was able to determine which input parameters affected higher SVI most. The prediction of SVI will benefit WWTPs to establish corrective measures to maintaining stable SVI. The SVI prediction model and Explainable Artificial Intelligence method will help the wastewater treatment sector to improve operational performance, system management, and process reliability.

Going, but Where? The Resettlement of Japanese Americans from the Heart Mountain Relocation Center

Abstract After the Japanese attack on Pearl Harbor in December 1942, the United States incarcerated (interned) 120,000 people of Japanese descent, two thirds of whom were U.S. citizens. Half of the incarcerated people returned to the West Coast, while the rest were dispersed across the country through the government's resettlement program. In this article, I study the resettlement of Japanese Americans away from the Heart Mountain, Wyoming, incarceration camp through historical big data and network analysis. The overarching goal of this article is to understand post-camp resettlement and its driving forces—or its obstacles—in light of the networks the inmates created at Heart Mountain. In particular, I draw attention to the ways in which resettlement choices could be seen to reflect individual agency: how resettlement decisions and actions mirror a person’s position in the incarceration community. I will uncover a variety and depth of networks that at times are indicative of compliance with authorities, while at times they may enforce separation from the dominant, i.e. White, society. This separation should not automatically be seen as negative or alienating, but rather as resistance to White expectations. Overall, these strategies highlight individual choice and agency.

Trustworthiness Evaluation-Based Routing Protocol for Incompletely Predictable Vehicular Ad Hoc Networks

Incompletely predictable vehicular ad hoc networks is a type of networks where vehicles move in a certain range or just in a particular tendency, which is very similar to some circumstances in reality. However, how to route in such type of networks more efficiently according to the node motion characteristics and related historical big data is still an open issue. In this paper, we propose a novel routing protocol named trustworthiness evaluation-based routing protocol (TERP). In our protocol, trustworthiness of each individual is calculated by the cloud depending on the attribute parameters uploaded by the corresponding vehicle. In addition, according to the trustworthiness provided by the cloud, vehicles in the network choose reliable forward nodes and complete the entire route. The analysis shows that our protocol can effectively improve the fairness of the trustworthiness judgement. In the simulation, our protocol has a good performance in terms of the packet delivery ratio, normalized routing overhead and average end-to-end delay.

Did Hydro-climatic Extremes, Positive Checks, and Economic Fluctuations Modulate the Epidemics Outbreaks in Late Imperial China?

Empirical research has shown that climate-related variables, the decline in economic well-being, and the mutual reinforcement of positive checks are the primary drivers of epidemic outbreaks in recent human history. However, their relative importance in causing the outbreak of epidemics is rarely examined quantitatively in a single study. I sought to address this issue by analyzing the 1402 epidemic incidents in China between 1841 and 1911, which partially overlaps partly with the Third Pandemic period. Fine-grained historical big data, multiple regression, and wavelet coherence analysis were employed. Statistical results show that economic fluctuations drove the country-wide epidemics outbreaks in China in inter-annual and decadal time scales. Economic fluctuations could cause short-term hardship and long-term impoverishment to the underprivileged social groups since a large portion of the Chinese population lived at the subsistence level in the past. The fluctuations might have sustained the repeated waves of epidemic outbreaks during the Third Pandemic period.Supplementary InformationThe online version contains supplementary material available at 10.1007/s10745-021-00272-7.

Object-Oriented and Deep-Learning-Based High-Resolution Mapping from Large Remote Sensing Imagery

Land use and land cover (LULC) mapping is a basic research topic in geography. In deep-learning (DL)-based LULC mapping, there are primarily the following issues: training and testing samples for DL are typically annotated by indoor visual interpretation without field surveys; remotely sensed scene classification based on DL typically lacks fine geometric boundaries for ground objects; historical big data (HBD) (e.g., vector data) are underutilized in DL; studies of large-scale remote sensing mapping (LSRSM) using DL are rare. To solve above issues, this paper proposes an object-oriented (i.e., polygon-based) and DL-based (OODLB) image classification method assisted by HBD for LSRSM to serve for monitoring the soil erosion and water loss (SEWL) in the Yangtze River Basin that includes the following steps: (1) using HBD and OpenStreetMap data, ground objects are vectorized; (2) a remote sensing interpretation key database is established by field surveys and data augmentation; (3) object-oriented (i.e., polygon-based) and Inception-ResNet-V2-based LULC mapping is performed; (4) DL-based classification results are updated by man-machine mutual verification. The experimental results of one county of the Yangtze River Basin show state-of-the-art performance with an overall accuracy of 90.20% in comparison of 75.60% for eCognition. It provides an excellent framework for OODLB large-scene mapping.

A Study of Multiregional Economic Correlation Analysis Based on Big Data—Taking the Regional Economy of Cities in Shaanxi Province, China, as an Example

To enhance the sustainability of the regional economy, this study attempts to integrate historical big data of multiregional and multi-industry economic indicators, aiming to explore and discover the correlations among regions, industries, or cross-regional economic indicators. In this paper, two correlation analysis models (the 2-order correlation model and the elastic-net regularized generalized linear model) are used to conduct a correlation analysis study of multiregional and multi-industry economies, and 20 years of historical data from 9 prefecture-level cities in Shaanxi (778 indicators in total) are analyzed empirically. The results show that the proposed method can mine complex correlations from economic big data.

Technical and Tactical Command Decision Algorithm of Football Matches Based on Big Data and Neural Network

A successful football team not only consists of more than a dozen people on the field but also includes a complete training, analysis, coaching team behind it, and the same basic education and youth training system. With the development of scientific concepts and the advancement of computer technology, more people have begun to study the use of modern technology to replace part of the traditional human work with low creativity and the use of more convenient quantitative analysis, prediction and other technologies to assist football professionals’ decision-making. Based on big data and neural network technology, this paper has designed a novel football technical and tactical command decision algorithm. First, the use of big data technology for analyzing the characteristics of the historical big data of football competitions provides valuable data for the work of this article. Secondly, to formulate scientific and reasonable football technical and tactical command, it requires learning effective offensive or defensive strategies from the big data of football competitions. This article uses deep neural networks to learn massive amounts of football competition data, which can effectively predict the offensive and defensive tactics of each position of the team to a certain extent. In addition, in order to better learn the timing video data of football matches, this paper also has proposed to use long- and short-term memory networks to improve the algorithm of this paper. The proposed method has achieved good results in football technical and tactical command and decision-making and also provides some new ideas for the subject of football combined with computer technology.

Digital Excavation of Mediatized Urban Heritage: Automated Recognition of Buildings in Image Sources

Digital technologies provide novel ways of visualizing cities and buildings. They also facilitate new methods of analyzing the built environment, ranging from artificial intelligence (AI) to crowdsourced citizen participation. Digital representations of cities have become so refined that they challenge our perception of the real. However, computers have not yet become able to detect and analyze the visible features of built structures depicted in photographs or other media. Recent scientific advances mean that it is possible for this new field of computer vision to serve as a critical aid to research. Neural networks now meet the challenge of identifying and analyzing building elements, buildings and urban landscapes. The development and refinement of these technologies requires more attention, simultaneously, investigation is needed in regard to the use and meaning of these methods for historical research. For example, the use of AI raises questions about the ways in which computer-based image recognition reproduces biases of contemporary practice. It also invites reflection on how mixed methods, integrating quantitative and qualitative approaches, can be established and used in research in the humanities. Finally, it opens new perspectives on the role of crowdsourcing in both knowledge dissemination and shared research. Attempts to analyze historical big data with the latest methods of deep learning, to involve many people—laymen and experts—in research via crowdsourcing and to deal with partly unknown visual material have provided a better understanding of what is possible. The article presents findings from the ongoing research project ArchiMediaL, which is at the forefront of the analysis of historical mediatizations of the built environment. It demonstrates how the combination of crowdsourcing, historical big data and deep learning simultaneously raises questions and provides solutions in the field of architectural and urban planning history.

Spatially and temporally disparate data in systems agriculture: Issues and prospective solutions

AbstractBig Data in agriculture is growing rapidly through advancements in metagenomics, precision agriculture, and on‐farm sensor technologies, as well as through increased capacity to collect, process, and store these data. Concurrent with 60% increases in food production demands by 2050 and the need for sustainable intensification, is the increased need for data synthesis across temporal and spatial scales. Therefore, in our data‐rich world, what is lacking is a data management system across spatial and temporal resolutions including workflows, interpretation methodology, and a delivery structure for identifying optimal systems for sustainable intensification or diversification. Consequently, the objective of this paper is to explore the current state of handling spatially and temporally disparate data and offer solutions for developing a platform for bridging component parts (encompassing multiple scales and disciplines) to analyze system functionality for greater resiliency, which may help manage risk. Two datasets were generated using bibliometrics (research articles from systematic literature reviews) evaluated trends and historical Big Data applications in agronomy. Results indicate research and industry progress is advancing towards web‐based real‐time output delivery systems using several well‐established Big Data handling platforms (e.g., Amazon Web Service, Google Cloud, Microsoft Azure), which are not yet widely used by or designed for agronomic researchers. Cloud‐based computing may provide opportunities to extrapolate agricultural research results across larger scales. Authors suggest training and educating agricultural practitioners on Big Data principles, database management, improved data visualization, as well as incentives for data sharing for optimizing Big Data in systems agriculture as these research innovations emerge.

Big data of the past: Analysis of historical freight shipping corridor data in the period 1662–1855

This paper examines the use of big data and data analytics in international transport networks from the perspective of historical big data, focusing on shipping logs from the British, Dutch, Spanish and French fleets in between 1662 and 1855. Based on a large-scale database containing mainly meteorological data collected in the CLIWOC project (2003), we computed travel distances and analyzed historical global maritime networks. This paper focuses on route choice, and consequently the time, distance, speed and reliability of the ships, covering different time periods, seasonal patterns and trade flows. The results reveal a clear picture of the main routes per nationality that is also indicative of the linguistical, cultural and economic colonial heritage that remains in the ‘host’ countries up to this day. The average daily distances covered vary over the countries involved, over the seasons and over different time periods. Also the trip characteristics vary notably over the different countries. Zooming in on the main trade flows, the corridor from the Netherlands to Indonesia stands out, but also considerable differences in average speed and stopover times were found along this route. Related to the complexity of using big data in studying international transport networks, our conclusion is that the degree of permutations and interactions with the dataset is not necessarily less for analyzing historical shipping records. It seems that big data of the past still can inspire future explorations of our historical transport networks on the world's oceans.

Energy and exergy co-optimization of IGCC with lower emissions based on fuzzy supervisory predictive control

This paper presents an energy and exergy co-optimization method of integrated gasification combined cycle (IGCC) based on Fuzzy Supervisory Predictive Control (FSPC). Firstly, a green IGCC process is proposed which contains three principle couplings: air separation unit (ASU), heat recovery steam generator (HRSG) and CO2 capture/storage unit (CCS). From law of thermodynamics, using substance thermophysical parameters, the energy efficiency and exergy efficiency of IGCC are successively defined. The IGCC power station has features such as closed coupling, large time lag and non-linearity, however, faster response speed and lower overshoot are always the unremitting pursuits. Therefore, the Fuzzy Supervisory Predictive Control (FSPC) method is proposed to implement robust control under complex disturbances by pre-considering unmeasurable disturbance and measurable disturbance. The fuzzy rules extracted from historical bigdata are employed in supervisory layer to make the precise control decisions. Finally, the energy and exergy co-optimization model is built and solved for higher efficiency and economic effectiveness. Taking the large-scale (300MW) IGCC for example, after using FSPC, the efficiency of water recovery is increased from 40.7% to 62.1% with the ratio of 52.6% because of waste water recovery (WWR) system. The net efficiency of proposed IGCC system is increased from 37.6% to 41.7% with the ratio of 10.9%. The exergy efficiency of IGCC system is increased from 36.5% to 39.2% with the ratio of 7.4%. The proposed method has great significance for the energy-saving and Near-zero emissions (NZEC) IGCC with high safety and robust control under supercritical (SC) or ultra-super critical (USC) state.

An intelligent decision-making framework for asphalt pavement maintenance using the clustering-PageRank algorithm

With ever-increasing road mileages worldwide, more pavement deterioration and ageing present great challenges to the maintenance and rehabilitation (M&R) of road pavement. In this study, an intelligent decision-making framework is developed for pavement maintenance using the clustering-PageRank algorithm (CPRA) based on historical big data. The proposed model is applied to a 3.5 km pavement (500 road sections) and leads to recommendations for the optimal pavement maintenance plans with appropriate possibilities. The results indicate that seven plans are the same as those obtained by the experience-based maintenance approach, while the other three are similar. The framework is also verified by comparison with the experience-based maintenance activities and is found to have limited reliability when dealing with a small quantity of solutions. The method and results of this study are expected to serve as a reference for decision makers to make well-informed project decisions on the optimum M&R activities.

A Big Data‐Driven Nonlinear Least Squares Four‐Dimensional Variational Data Assimilation Method: Theoretical Formulation and Conceptual Evaluation

AbstractA new nonlinear least squares four‐dimensional variational data assimilation method (NLS‐4DVar) is proposed incorporating the use of “big data.” This distinctive four‐dimensional ensemble‐variational data assimilation method (4DEnVar) is made up of two ensembles, a preprepared historical big data ensemble and a small “online” ensemble. The historical ensemble portrays both the ensemble‐constructed background error covariance and tangent models more accurately, as compared with the standard NLS‐4DVar method, with no heavy increase in computational cost in terms of real‐time operations. The online ensemble maintains the flow dependence of the ensemble‐estimated background error covariance. The ensemble analysis scheme proposed by merging the local ensemble transform Kalman filter scheme with a sophisticated sampling approach is able to adjust the ensemble spreads suitably and maintain them steadily. The updating scheme also largely guarantees the partial flow dependence of the historical ensemble. Experimental results using the shallow‐water equations demonstrate that the new big data method provides substantial performance improvement over the standard NLS‐4DVar method.