Big Data Programming Research Articles

텍스트 마이닝 및 CONCOR 분석을 기반으로 한 충청남도 스포츠 관광 트렌드 분석(2010~2023)

Objectives The purposes of this study were to analyse sports tourism trends in Chungcheongnam-do based on big data analysis, text mining and CONCOR analysis.
 Methods This study used TEXTOM 6.0, a big data program, and the search keywords were 'Chungcheongnam-do+sports+tourism', and the data collection period was classified into three periods (2010~2014, 2015~2019, 2020~2023).
 Results As a result of the text mining analysis, the main keywords were 'Chungcheongnam-do, sports, tourism, presidential tournament, amateur, tournament, Boryeong-si, promotion project, creation', and the CONCOR anal-ysis resulted in a group of keywords related to regional festivals, a group of keywords related to sports com-petitions, and a group of keywords related to Boryeong-si, a sports city.
 Conclusions This study confirmed that promoting sports tourism using e-sports competitions, fostering sports-specific cities such as Boryeong-si, and inducing additional economic effects of sports tourism through attracting sports competitions can be a way to develop regional sports tourism.

A Comparative Study of Parallel and Distributed Big Data programming models: Methodologies, Challenges and Future Directions

According to a survey conducted in 2021, users share about 4 petabytes of data on Facebook daily. The exponential increase in data (called big data) plays a vital role in machine learning, internet of things (IoT), and business intelligence applications. Due to the rapid increase in big data, research in big data programming models gained much interest in the past decade. Today, many programming paradigms exist to handle big data, and selecting an appropriate model for a project is critical for its success. This study provides an in-depth analysis of big data programming models such as MapReduce, Directed Acyclic Graph (DAG), Bulk Synchronous Parallel (BSP), and SQL. We conduct a comparative study of distributed and parallel big data programming models and categorize these models into three classes: traditional data processing, graph-based processing, and query-based processing models. Furthermore, we evaluate these programming models based on different parameters like performance, data processing, storage, fault-tolerant, suitable language, and machine learning support. Finally, we highlight the benchmark datasets used for big data programming models and discuss the challenges of models along with future directions for the research community.

Keyword network analysis of changes in the image of nurses pre- and post-COVID-19 in the media environment.

To analyse and describe the changes in the image of nurses perceived by the public pre- and post-COVID-19 in Korea. Nurses play an important role in disaster situations such as COVID-19. In such disaster situations, we aim to confirm the image of nurses projected in the mass media and promote the professionalism of nurses through the image of nurses as professionals. Qualitative media keyword networks analysis. To understand the change in the image of nurses pre- and post-COVID-19, the big data program TEXTOM was used to collect data. From 19 January to 31 May 2020, data were set as 'post-COVID-19', and 1 year before 18 January 2020, data were set as 'pre-COVID-19'. The keywords from 9533 articles were refined, frequency analysed and social networks analysed using TEXTOM and MS office excel, and the analysis results were visualised using UCINET 6 and the NetDraw program. As for keywords related to nurses pre-COVID-19, those with the most frequent appearance and the highest networking degree in centrality were 'work', 'older adults', 'care', 'time', 'care worker', 'caring labor'. As for keywords related to nurses post-COVID-19, those that appeared most often and had the highest degree of centrality networking were 'COVID-19', 'USA', 'China', 'check', 'patient', 'work'. This study shows that the public image of nurses has changed more positively after COVID-19 due to the media. Individual nurses and nursing organizations should pay attention to the deficiency in the image of nurses and provide a way to reform the public image of professional nurses. The effects of global crises such as COVID-19 on nurses were confirmed, and information delivered through the media was an important way to improve the nursing profession.

Leveraging data-driven decisions: a framework for building intracompany capability for supply chain optimization and resilience

PurposeCompanies are turning to big data (BD) programs to help mitigate supply chain (SC) disruptions and risks that are increasing in frequency and severity. The purpose of this paper is to explore exactly how companies translate data into meaningful information used to manage SC risk and create economic value; an area not well researched. As companies are turning to big-data programs to help mitigate supply chain (SC) disruptions and risks that are increasing in frequency and severity, having the capability to internally integrate SC information is cited as the most critical risk to manage.Design/methodology/approachInformation processing theory and resource-based view are applied to support capability development used to make value-based BD decisions. Semi-structured interviews were conducted with leaders in both the oil and gas industry and logistics SC partners to explore each companies’ BD transformation.FindingsFindings illuminate how companies can build internal capability to more effectively manage SC risk, optimize operating assets and drive employee engagement.Research limitations/implicationsThe oil and gas industry were early adopters of gathering BD; more studies addressing how companies translate data to create value and manage SC risk would be beneficial.Practical implicationsGuidance for senior leaders to proactively introduce BD to their company through a practical framework. Further, this study provides insight into where the maximum benefit may reside, as data intersects with other company resources to build an internal capability.Originality/valueThis study presents a framework highlighting best practices for introducing BD plus creating a culture capable of using that data to reduce risk during design, implementation and ongoing operations. The steps for producing the maximum benefit are laid out in this study.

Spatio Temporal Tourism Tracking System Based on Adaptive Convolutional Neural Network

Technological developments create a lot of impacts in the tourism industry. Emerging big data technologies and programs generate opportunities to enhance the strategy and results for transport security. However, there is a difference between technological advances and their integration into the methods of tourism study. The rising popularity of Freycinet National Park led to a master plan that would not address cultural and environmental issues. This study addresses the gap by using a synthesized application (app) for demographic surveys and Global Navigation Satellite System (GNSS) technology to implement research processes. This article focuses on managing visitors within the famous Freycinet National Park. Extremely comprehensive structured data were analyzed in three phases, (1) identifying groups of visitors who are more likely to use the walking trails, (2) those who are more and less likely to visit during/peak crowding times, and (3) finally creating an integrated Spatio-temporal dependency model via a machine-based learning system for real-time activity. This research examines innovative techniques that can offer energy resources to managers and tourism agencies, especially in detecting, measuring, and potentially relieving crowding and over-tourism.

Research on K-means Clustering Algorithm Based on MapReduce Distributed Programming Framework

As a classical clustering algorithm, K-means algorithm has a profound research background. In the of big data era, K-means algorithms will play a greater advantage, being able to quickly divide similar data into the same cluster. Combining K-means algorithm with MapReduce distributed computing framework and running on Hadoop big data platform can significantly improve the clustering effect. Based on MapReduce framework structure, this paper studies K-means model, including K-means principle, distance calculation, content validity index and external validity index. On this basis, the K-means clustering flow based on MapReduce big data programming framework is proposed, and the execution process of the algorithm flow is described in detail, which provides a guide for the algorithm implementation.

Construction of Higher Education Management Data Analysis Model Based on Association Rules

This paper adopts the analytical method of intelligent programming to conduct in-depth research and analysis on university education management and designs a corresponding university education management model to be applied in actual teaching. Based on the basic theories and methods of big data distributed data mining and machine learning, automated machine learning, and big data programming computational methods, this paper combines the importance and technical challenges of the algorithms themselves and the background of the practical application needs of the industry and firstly selects a series of data mining and machine learning algorithms that are commonly used based on high complexity, outstanding computational efficiency problems, and difficult to design distributed algorithms. The research on efficient large-scale distributed parallelized data mining and machine learning methods and algorithms is carried out. As an innovative teaching model, the experiential teaching model focuses on the cultivation of individual students’ independent learning ability and subjective initiative, which not only can effectively activate the classroom atmosphere and improve the teaching effect but also meet the requirements of classroom teaching for the development of the times. The empirical evidence of the experiential teaching model proposed in this paper is carried out, and the implementation process of the experiential teaching model of high school physics is further improved. In terms of the overall implementation effect, the experiential teaching model proposed in the paper can deepen students’ understanding of concepts and physical laws, thus enabling them to solve physics problems better, and has a positive and active effect on enhancing students’ learning interests and attitudes. In the case of system security, the average comprehensive index of each unit is 61.11, 22 are higher than the average index, accounting for 61.11%, 14 are lower than the average index, accounting for 38.89%, and 23 management departments’ management systems passed the safety monitoring, accounting for 63.89%. At the same time, the proposed model also provides educational practitioners with a more reliable framework for instructional design, which has realistic reference value and significance.

A unified framework to improve the interoperability between HPC and Big Data languages and programming models

One of the most important issues in the path to the convergence of HPC and Big Data is caused by the differences in their software stacks. Despite some research efforts, the interoperability between their programming models and languages is still limited. To deal with this problem we introduce a new computing framework called IgnisHPC, whose main objective is to unify the execution of Big Data and HPC workloads in the same framework. IgnisHPC has native support for multi-language applications using JVM and non-JVM-based languages. Since MPI was used as its backbone technology, IgnisHPC takes advantage of many communication models and network architectures. Moreover, MPI applications can be directly executed in an efficient way in the framework. The main consequence is that users could combine in the same multi-language code HPC tasks (using MPI) with Big Data tasks (using MapReduce operations). The experimental evaluation demonstrates the benefits of our proposal in terms of performance and productivity with respect to other frameworks. IgnisHPC is publicly available for the Big Data and HPC research community.

Centralized data repositories: NOAA’s National Archives for Marine Acoustic Data

The NOAA National Centers for Environmental Information maintains two marine acoustic archives for the long-term stewardship of globally collected water-column sonar and passive acoustic data. These vast datasets are collected across NOAA and academia for a wide range of scientific objectives. The archives document the datasets using standards-driven metadata and preserve them on long-term storage systems. Users can discover, query, and access archived data using the archives’ web-based map viewers. Further, cloud-based access to 200 + TB through the NOAA Big Data Program enables free and immediate download of desired data and allows users to bring processing routines to large volumes of data—from simple statistical analyses to artificial intelligence. Cloud-based tools are being developed in collaboration with our partners in the ocean acoustic community to visualize and analyze data in the archives. This allows researchers of varying backgrounds to easily understand the quality and content of these complex data. Providing free access to data archives and facilitating the utility of these data increases the potential for researchers to address new questions that will advance the field of marine ecosystem acoustics.

Spatio-temporal contrastive learning for acoustic data

Large volumes of water column sonar data have been generated from acoustic surveys of living marine resources. They provide valuable information about marine ecosystems. To leverage them for acoustic target identification, scarcity of annotations is usually an issue. While some annotations have been generated by scientists via manual scrutiny or limited automation, they are limited to certain species. To fill in this gap, we propose a spatio-temporal contrastive learning approach for acoustic data. This unsupervised deep learning technique leverages both acoustic and non-acoustic (spatial and temporal) information for acoustic target classification. We firstly employ the Simple Linear Iterative Clustering (SLIC) algorithm to extract superpixels from acoustic data. Then the spatial and temporal similarity between two superpixels is computed, and the proposed spatio-temporal contrastive learning approach is applied to learn a semantically meaningful representation for superpixels. Finally, the learned representations of superpixels are used in acoustic target classification. We demonstrate that this approach outperforms previous methods with acoustic data collected by the NOAA National Marine Fisheries Service in the Gulf of Maine and Georges Bank region over 10 years. This dataset is archived at NOAA National Centers for Environmental Information and accessed for free through AWS and NOAA Big Data Program.

An adaptive geographic meshing and coding method for remote sensing data

Spatial indexing techniques, inherently data structures, are generally used in portals opened by institutions or organizations to efficiently filter RS images according to their spatial extent, thus providing researchers with fast Remote Sensing (RS) image data discovery ability. Specifically, space-based spatial indexing approaches are widely adopted to index RS images in distributed environments by mapping RS images in two-dimensional space into several one-dimensional spatial codes. However, current spatial indexing approaches still suffer from the boundary objects problem, which leads to multiple spatial codes for a boundary-crossing RS image and thus alleviates the performance of spatial indexes built on top of these spatial codes. To solve this problem, we propose an adaptive geographic meshing and coding method (AGMD) by combining the famous subdivision model GeoSOT and XZ-ordering to generate only one spatial code for RS images with different spatial widths. Then, we implement our proposed method with a unified big data programming model, (i.e., Apache Beam), to enable its execution in various distributed computing engines (e.g., MapReduce, and Apache Spark, etc.) in distributed environments. Finally, we conduct a series of experiments on real datasets, the archived Landsat metadata collection in level 2. The results show that the proposed AGMD method performs well on metrics, including the following aspects: the effectiveness of the storage overhead and the time cost are up to 359.7% and 58.02 %, respectively.

TRANSMUT‐Spark: Transformation mutation for Apache Spark

SummaryThis paper proposesTRANSMUT‐Sparkfor automating mutation testing of big data processing code within Spark programs. Apache Spark is an engine for big data analytics/processing that hides the inherent complexity of parallel big data programming. Nonetheless, programmers must cleverly combine Spark built‐in functions within programs and guide the engine to use the right data management strategies to exploit the computational resources required by big data processing and avoid substantial production losses. Many programming details in Spark data processing code are prone to false statements that must be correctly and automatically tested. This paper explores the application of mutation testing in Spark programs, a fault‐based testing technique that relies on fault simulation to evaluate and design test sets. The paper introducesTRANSMUT‐Sparkfor testing Spark programs by automating the most laborious steps of the process and fully executing the mutation testing process. The paper describes how theTRANSMUT‐Sparkautomates the mutant generation, test execution and adequacy analysis phases of mutation testing. It also discusses the results of experiments to validate the tool and argues its scope and limitations.

Accelerating Spark-Based Applications with MPI and OpenACC

The amount of data produced in scientific and commercial fields is growing dramatically. Correspondingly, big data technologies, such as Hadoop and Spark, have emerged to tackle the challenges of collecting, processing, and storing such large-scale data. Unfortunately, big data applications usually have performance issues and do not fully exploit a hardware infrastructure. One reason is that applications are developed using high-level programming languages that do not provide low-level system control in terms of performance of highly parallel programming models like message passing interface (MPI). Moreover, big data is considered a barrier of parallel programming models or accelerators (e.g., CUDA and OpenCL). Therefore, the aim of this study is to investigate how the performance of big data applications can be enhanced without sacrificing the power consumption of a hardware infrastructure. A Hybrid Spark MPI OpenACC (HSMO) system is proposed for integrating Spark as a big data programming model, with MPI and OpenACC as parallel programming models. Such integration brings together the advantages of each programming model and provides greater effectiveness. To enhance performance without sacrificing power consumption, the integration approach needs to exploit the hardware infrastructure in an intelligent manner. For achieving this performance enhancement, a mapping technique is proposed that is built based on the application’s virtual topology as well as the physical topology of the undelaying resources. To the best of our knowledge, there is no existing method in big data applications related to utilizing graphics processing units (GPUs), which are now an essential part of high-performance computing (HPC) as a powerful resource for fast computation.

Research on the Key Issues of Big Data Quality Management, Evaluation, and Testing for Automotive Application Scenarios

This paper provides an in-depth analysis and discussion of the key issues of quality management, evaluation, and detection contained in big data for automotive application scenarios. A generalized big data quality management model and programming framework are proposed, and a series of data quality detection and repair interfaces are built to express the processing semantics of various data quality issues. Through this data quality management model and detection and repair interfaces, users can quickly build custom data quality detection and repair tasks for different data quality requirements. To improve the operational efficiency of complex data quality management algorithms in large-scale data scenarios, corresponding parallelization algorithms are studied and implemented for detection and repair algorithms with long computation time, including priority-based multiconditional function-dependent detection and repair algorithms, entity detection, and extraction algorithms based on semantic information and chunking techniques, and plain Bayesian-based missing value filling algorithms, and this paper proposes a data validity evaluation algorithm and enhances the validity of the original data in practical applications by adding temporal weights, and finally it passed the experimental validation. Through the comprehensive detection process of data importance, network busyness, duration of transmission process, and failure situation, the efficiency has been increased by 20%, and an adaptive data integrity detection method based on random algorithm and encryption algorithm is designed. After experimental verification, this method can effectively detect the integrity of the data transmission process and improve the application of data value, and the final effect is increased by 30.5%.

DDF Library: Enabling functional programming in a task-based model

In recent years, the areas of High-Performance Computing (HPC) and massive data processing (also know as Big Data) have been in a convergence course, since they tend to be deployed on similar hardware. HPC systems have historically performed well in regular, matrix-based computations; on the other hand, Big Data problems have often excelled in fine-grained, data parallel workloads. While HPC programming is mostly task-based, like COMPSs, popular Big Data environments, like Spark, adopt the functional programming paradigm. A careful analysis shows that there are pros and cons to both approaches, and integrating them may yield interesting results. With that reasoning in mind, we have developed DDF, an API and library for COMPSs that allows developers to use Big Data techniques while using that HPC environment. DDF has a functional-based interface, similar to many Data Science tools, that allows us to use dynamic evaluation to adapt the task execution in run time. It brings some of the qualities of Big Data programming, making it easier for application domain experts to write Data Analysis jobs. In this article we discuss the API and evaluate the impact of the techniques used in its implementation that allow a more efficient COMPSs execution. In addition, we present a performance comparison with Spark in several application patterns. The results show that each technique significantly impacts the performance, allowing COMPSs to outperform Spark in many use cases.

The Legal System and Legal Issues of Marine Environmental Governance in the Age of Big Data

Wang, J., 2020. The legal system and legal issues of marine environmental governance in the age of big data. In: Guido Aldana, P.A. and Kantamaneni, K. (eds.), Advances in Water Resources, Coastal Management, and Marine Science Technology. Journal of Coastal Research, Special Issue No. 104, pp. 313–316. Coconut Creek (Florida), ISSN 0749-0208.China's marine environmental laws and regulations have mainly shown the characteristics of high principle and low rules for a long period of time, thus causing them not to play their full role in dealing with environmental problems. The application of relevant information under big data is of great significance for the active completion of the country's environmental governance objectives, and also helps to establish and improve the system of national and local interactive and cooperative environmental laws and regulations, realize the basic concepts and basic principles of environmental law rule design, at the same time, in the construction and improvement of the basic system of environmental law plays an important role. In the era of big data, the model of Chinese environmental law research sample method is in a co-existence state. In addition, the main reason for the successful application of big data in the study of environmental law in China is that many scholars in China have a clear understanding of the research on the relevance of big data and environmental law in China, and have clear principles and basic requirements for the application of big data programs in the study of environmental law in China.

Key competencies for big data analytics professions: a multimethod study

PurposeThis study aims to identify the precise competencies that employers are seeking for big data analytics professions and whether higher education big data programs enable students to acquire the competencies.Design/methodology/approachThis study utilizes a multimethod approach involving three data sources: online job postings, executive interviews and big data programs at universities and colleges. Text mining analysis guided by a holistic competency theoretical framework was used to derive insights into the required competencies.FindingsWe found that employers are seeking workers with strong functional and cognitive competencies in data analytics, computing and business combined with a range of social competencies and specific personality traits. The exact combination of competencies required varies with job levels and tasks. Executives clearly indicate that workers rarely possess the competencies and they have to provide additional training.Research limitations/implicationsA limitation is our inability to capture workers' perspectives to determine the extent to which they think they have the necessary competencies.Practical implicationsThe findings can be used by higher educational institutions to design programs to better meet market demand. Job seekers can use it to focus on the types of competencies they need to advance their careers. Policymakers can use it to focus policies and investments to alleviate skills shortages. Industry and universities can use it to strengthen their collaborations.Social implicationsMuch closer collaborations among public institutions, educational institutions, industry, and community organizations are needed to ensure training programs evolve with the evolving need for skills driven by dynamic technological changes.Originality/valueThis is the first study on this topic to adopt a multimethod approach incorporating the perspectives of the key stakeholders in the supply and demand of skilled workers. It is the first to employ text mining analysis guided by a holistic competency framework to derive unique insights.

Survey of big data programming frameworks

Advances in ICT today has made data more voluminous and multifarious and its being transferred at high speed. Applications in cloud like Yahoo weather, Facebook photo gallery, and Google search index are changing the IT landscape in a profound way. Reasons for these trend includes scientific organizations solving big problems related to high performance computing workloads, diverse public services being digitized; mobile devices, global positioning systems, sensors, social media, medical imaging, financial transaction logs and lots of them are all sources of massive data generating large sets of complex data. These applications are evolving to be data-intensive, processing very large volumes of data hence, require dynamically scalable, virtualized resources to handle them. This paper compared several of these big data programming frameworks used in data-intensive processes. Area of strengths and weaknesses of these frameworks were highlighted with the intention that future framework will harness all positives from this study for a better system.Keywords: Big Data, Framework, Hadoop, Analytics, Workload, Resource Allocation

Hierarchially Distributed Data Matrix Scheme for Big Data Processing

MapReduce is a programming paradigm and an affiliated Design for processing and making substantial data sets. It operates on a large cluster of specialty machines and is extremely scalable Across the past years, MapReduce and Spark have been offered to facilitate the job of generating big data programs and utilization. However, the tasks in these structures are roughly described and packaged as executable jars externally any functionality being presented or represented. This means that extended roles are not natively composable and reusable for consequent improvement. Moreover, it also impedes the capacity for employing optimizations on the data stream of job orders and pipelines. In this article, we offer the Hierarchically Distributed Data Matrix (HDM), which is a practical, strongly-typed data description for writing composable big data appeals. Along with HDM, a runtime composition is presented to verify the performance of HDM applications on dispersed infrastructures. Based on the practical data dependency graph of HDM, various optimizations are employed to develop the appearance of performing HDM jobs. The empirical outcomes show that our optimizations can deliver increases of between 10% to 60% of the Job-Completion-Time for various types of applications when associated with the current state of the art, Apache Spark.

A Doctor’s Touch: What Big Data in Health Care Can Teach Us About Predictive Policing

We take two professions — police officers and doctors — and place their experiences with big data in dialogue. Policing and medicine, while naturally different in some obvious respects, actually both need to grapple with a lot of the same moral, social, and legal questions that come with adopting big data programs. This because, as we discuss below, both professions generally possess a monopoly over an acute societal vulnerability, be it safety or health, and have accordingly developed a set of settled internal norms to shape individual discretion in service of each respective function. We place the professions side-by-side and try to distill certain insights from the perspective of three key stakeholders — practitioners, policymakers, and the polity.