Development Of Big Data Applications Research Articles

Big Data in multiscale modelling: from medical image processing to personalized models

The healthcare industry is different from other industries–patient data are sensitive, their storage needs to be handled with care and in compliance with regulative, while prediction accuracy needs to be high. This fast expansion in medical image modalities and data collection leads to generation of so called “Big Data” which is time-consuming to be analyzed by medical experts. This paper provides an insight into the Big Data from the aspect of its role in multiscale modelling. Special attention is paid to the workflow, starting from medical image processing all the way to creation of personalized models and their analysis. A review of literature regarding Big Data in healthcare is provided and two proposed solutions are described–carotid artery ultrasound image processing and 3D reconstruction, and drug testing on personalized heart models. Related to the carotid artery ultrasound image processing, the starting point is ultrasound images, which are segmented using convolutional neural network U-net, while segmented masks were further used in 3D reconstruction of geometry. Related to the drug testing on personalized heart model, similar approach was proposed, images were used in creation of personalized 3D geometrical model that is used in computational modelling to determine pressure in the left ventricle before and after drug testing. All the aforementioned methodologies are complex, include Big Data analysis and should be performed using servers or high-performance computing. Future development of Big Data applications in healthcare domains offers a lot of potential due to new data standards, rapid development of research and technology, as well as strong government incentives.

Innovations in Genomics and Big Data Analytics for Personalized Medicine and Health Care: A Review.

Big data in health care is a fast-growing field and a new paradigm that is transforming case-based studies to large-scale, data-driven research. As big data is dependent on the advancement of new data standards, technology, and relevant research, the future development of big data applications holds foreseeable promise in the modern day health care revolution. Enormously large, rapidly growing collections of biomedical omics-data (genomics, proteomics, transcriptomics, metabolomics, glycomics, etc.) and clinical data create major challenges and opportunities for their analysis and interpretation and open new computational gateways to address these issues. The design of new robust algorithms that are most suitable to properly analyze this big data by taking into account individual variability in genes has enabled the creation of precision (personalized) medicine. We reviewed and highlighted the significance of big data analytics for personalized medicine and health care by focusing mostly on machine learning perspectives on personalized medicine, genomic data models with respect to personalized medicine, the application of data mining algorithms for personalized medicine as well as the challenges we are facing right now in big data analytics.

Exploring the Application of Web-based Teaching Platform for Big Data Teaching

Big Data” has existed for some time in various industries such as military, finance and communication, but has attracted attention because of the development of Internet and information industry in recent years. With the extensive development of big data applications has increased the demand for talents, and the number of big data training courses has increased. Due to the unique content of big data teaching, it can be appropriate to teach big data through network teaching platform, so this paper explores the application of big data teaching based on network teaching platform for big data teaching.

Model-Based Big Data Analytics-as-a-Service: Take Big Data to the Next Level

The Big Data revolution promises to build a data-driven ecosystem where better decisions are supported by enhanced analytics and data management. However, major hurdles still need to be overcome on the road that leads to commoditization and wide adoption of Big Data Analytics (BDA). Big Data complexity is the first factor hampering the full potential of BDA. The opacity and variety of Big Data technologies and computations, in fact, make BDA a failure prone and resource-intensive process, which requires a trial-and-error approach. This problem is even exacerbated by the fact that current solutions to Big Data application development take a bottom-up approach, where the last technology release drives application development. Selection of the best Big Data platform, as well as of the best pipeline to execute analytics, represents then a deal breaker. In this paper, we propose a return to roots by defining a Model-Driven Engineering (MDE) methodology that supports automation of BDA based on model specification. Our approach lets customers declare requirements to be achieved by an abstract Big Data platform and smart engines deploy the Big Data pipeline carrying out the analytics on a specific instance of such platform. Driven by customers' requirements, our methodology is based on an OWL-S ontology of Big Data services and on a compiler transforming OWL-S service compositions in workflows that can be directly executed on the selected platform. The proposal is experimentally evaluated in a real-world scenario focusing on the threat detection system of SAP.

Automatic Selection and Parameter Configuration of Big Data Software Core Components Based on Retention Pattern

This paper conducts an in-depth analysis and research on the automatic selection and parameter configuration of the core components of Big Data software by using the retention model and the automatic selection of Big Data components by establishing a standardized requirement index and using the decision tree model to solve the problem of component selection in Big Data application development. By establishing standardized demand indicators and based on the retention model, a data transmission intermediate platform for bidirectional data detection is proposed based on the three demands of user input: storage, computation, and analysis, as well as the problem of undetectable packet loss in data transmission of existing IoT and Web service platforms. The data communication module of the data transmission intermediate platform enables mutual monitoring and detection of data interaction between IoT smart terminals and cloud platforms. The retention mode is built separately to realize the automatic selection of Big Data components. In this paper, we start from several mainstream distributed storage systems and use Cassandra as an example for experiments and tests. We use the multiple regression fitting method to build a corresponding performance model for hardware parameters, take user requirements as input, and use the performance model to configure system hardware parameters; by studying its system principle, architecture, features, and application scenarios, we build a software parameter configuration knowledge base to guide the software. This solves the difficult problem of selecting, deploying, and configuring parameters for Big Data applications.

Забезпечення безпеки залізничного транспорту в умовах цифровізації

У статті розглянуто стан та перспективи безпеки залізничного транспорту в умовах цифровізації. Визначено, що функціонування програми з безпеки залізничного транспорту в умовах цифровізації базується на наступних технологіях та стандартах: ERTMS, ETCS, GSM-R, GoA, ATO, ATP, ATC, ВІМ Rail, CBTC, TMS. Аналіз стану та перспективи безпеки залізничного транспорту в умовах цифровізації встановив необхідність застосування штучного інтелекту. Зазначено, що безпека залізничного транспорту зумовлена спільною взаємодією багатьох технологій, які привели до створення концепцій «Індустрія 4.0» і «Індустріальний Інтернет речей (IIoT)».

Literature review and analysis on big data stream classification techniques

Rapid growth in technology and information lead the human to witness the improved growth in velocity, volume of data, and variety. The data in the business organizations demonstrate the development of big data applications. Because of the improving demand of applications, analysis of sophisticated streaming big data tends to become a significant area in data mining. One of the significant aspects of the research is employing deep learning approaches for effective extraction of complex data representations. Accordingly, this survey provides the detailed review of big data classification methodologies, like deep learning based techniques, Convolutional Neural Network (CNN) based techniques, K-Nearest Neighbor (KNN) based techniques, Neural Network (NN) based techniques, fuzzy based techniques, and Support vector based techniques, and so on. Moreover, a detailed study is made by concerning the parameters, like evaluation metrics, implementation tool, employed framework, datasets utilized, adopted classification methods, and accuracy range obtained by various techniques. Eventually, the research gaps and issues of various big data classification schemes are presented.

Situation and Prospects of Big Data Application on Drilling and Completion of Bohai Oilfield

As a new field in petroleum engineering technology, Big Data and intelligent oilfields have garnered the attention and interest of the petroleum industry and researchers. Drilling and completion specialty in the Bohai oilfield have made significant data theory and technology attempts on the basis of establishing and improving data acquisition hardware, but it still needs to be improved and enriched urgently. Based on the investigation of the drilling and completion technology in the Bohai Sea, the progress of Big Data applications in drilling and completion is summarized. The WellView database, remote assistant decision-making system, cost fine management, and fracturing prediction based on Big Data are introduced. From the aspects of digitalization, intellectualization difficulty, and bottlenecks in technological development in drilling and completion in the Bohai oilfield, the challenges faced by Big Data and intellectualization are analyzed. From the aspects of national energy demand,guarantee of Big Data and intellectualized oilfield advanced technology, frontier interdisciplinary disciplines and foreign advanced experience, etc. The development prospects of Big Data and intelligent oilfields in the Bohai Oilfield are predicted, with a view toward promoting the rapid development of Big Data applications in drilling and completion.

An Innovative Industry 4.0 Cloud Data Transfer Method for an Automated Waste Collection System

Moving to Industry 4.0 involves the collection of massive amounts of data and the development of big data applications that can ensure a quick data flow between different systems, including massive amounts of data and information collection from smart sensors, and sending them to cloud applications that allow real-time data monitoring and processing. Securing and protecting the transmitted data represents a big issue to be discussed and resolved. In the paper, we propose a new method of data encoding and encryption for cloud applications using PNG format images. The proposed method is described in comparison with one of the classical methods of data encoding and transmission used currently. The paper includes a case study in which the proposed method was used to collect and transmit data from an automated waste collection system. The results show that the proposed method represents a secure, fast and efficient way to send and store the data in the cloud compared to the methods currently used. The proposed method is not limited to being used only in waste management but can be used successfully for any type of manufacturing system from smart factories.

Graphical Flow-based Spark Programming

Increased sensing data in the context of the Internet of Things (IoT) necessitates data analytics. It is challenging to write applications for Big Data systems due to complex, highly parallel software frameworks and systems. The inherent complexity in programming Big Data applications is also due to the presence of a wide range of target frameworks, with different data abstractions and APIs. The paper aims to reduce this complexity and its ensued learning curve by enabling domain experts, that are not necessarily skilled Big Data programmers, to develop data analytics applications via domain-specific graphical tools. The approach follows the flow-based programming paradigm used in IoT mashup tools. The paper contributes to these aspects by (i) providing a thorough analysis and classification of the widely used Spark framework and selecting suitable data abstractions and APIs for use in a graphical flow-based programming paradigm and (ii) devising a novel, generic approach for programming Spark from graphical flows that comprises early-stage validation and code generation of Spark applications. Use cases for Spark have been prototyped and evaluated to demonstrate code-abstraction, automatic data abstraction interconversion and automatic generation of target Spark programs, which are the keys to lower the complexity and its ensued learning curve involved in the development of Big Data applications.

A practical solution to clone problem in anonymous information system

Cloning user's identity is always a thorny problem for an information system, especially for an anonymous system. With the development of big data applications, clone behaviors sometimes even become attacks on these systems. But until now, there has been no very satisfactory anti-clone scheme in the anonymous system. After analyzing the problems in existing anti-clone schemes, without any assumptions about physical security, we provide a practical solution to the clone problem in anonymous authentication system. In our scheme, the authentication is not only related to user's private key, but also related to user's current state, which is constantly updated by the system. Therefore, the authentication trajectories of user and clone will inevitably overlap, and it results in information leakage so as to indentify clone behaviors and revoke clone user's credential. Meanwhile, we prove that honest users are truly anonymous and their login behaviors are unlinkable with complete security proofs. According to the analysis of the system function and the system efficiency, our scheme is much more efficient and has the best anti-clone properties comparing with the existing schemes.

Defining medical information privacy in China: a quantitative and qualitative study

BackgroundThe potential value of big data in health care is huge. The Chinese State Council issued the Guides on Promoting and Regulating the Development of Big Data Applications in Health Care to encourage opening and sharing of health data. However, some data controllers in China are unwilling to open and share data because of privacy risks. We aimed to identify medical information that should be removed or processed before sharing of health data, so to ensure that data are shared in a safe and controlled manner. MethodsUsing cluster sampling, we selected ten departments in a hospital in Shanghai between March 5, 2018, and May 1, 2018. 456 patients in the departments were administered a questionnaire that included requests to provide basic personal information and questions on awareness of privacy, control over data, and willingness to share data. In addition, a convenience sample of 50 medical staff, 25 patients, and six management personnel was selected for one-to-one interviews about issues related to medical information privacy. Feedback from patients, doctors, and management personnel was summarised and used in consultation with experts to define categories of medical data that should be considered when opening and sharing health data in China. All analyses were descriptive. FindingsWe defined five categories of identifiable characteristics, including name, identity document number (eg, identification card number and driver's license number), contact information (eg, telephone number and home address), accounts number (eg, social security number and medical record file number), and biometric identification (eg, fingerprint and DNA). We identified eight types of diseases, including reproduction-related diseases, infectious diseases, mental diseases, malignant tumours, hereditary diseases, anal diseases, rare diseases, and other incurable diseases. Finally, we defined three identities, including mother, patient with malignant tumour, and government personnel. Health data were classified by privacy level as summary datasets, limited datasets, and identifiable datasets. InterpretationThe Health Care Commission in China should set regulations to define the content and scope of privacy data in health care. The five categories of identifiable characteristics, eight types of diseases, and three types of identities we have defined could be referred to when setting these regulations so that secondary use of data is possible. FundingNational Natural Science Foundation (grant number 71473164).

AFlux: Graphical flow-based data analytics

Abstract aFlux is a graphical flow-based programming tool designed to support the modelling of data analytics applications. It supports high-level programming of Big Data applications with early-stage flow validation and automatic code generation for frameworks like Spark, Flink, Pig and Hive. The graphical programming concepts used in aFlux constitute the first approach towards supporting high-level Big Data application development by making it independent of the target Big Data frameworks. This programming at a higher level of abstraction helps to lower the complexity and its ensued learning curve involved in the development of Big Data applications.

A review of big data applications of physiological signal data.

The proliferation of smart physiological signal monitoring sensors, combined with the advancement of telemetry and intelligent communication systems, has led to an explosion in healthcare data in the past few years. Additionally, access to cheaper and more effective power and storage mechanisms has significantly increased the availability of healthcare data for the development of big data applications. Big data applications in healthcare are concerned with the analysis of datasets which are too big, too fast, and too complex for healthcare providers to process and interpret with existing tools. The driver for the development of such systems is the continuing effort in making healthcare services more efficient and sustainable. In this paper, we provide a review of current big data applications which utilize physiological waveforms or derived measurements in order to provide medical decision support, often in real time, in the clinical and home environment. We focus mainly on systems developed for continuous patient monitoring in critical care and discuss the challenges that need to be overcome such that these systems can be incorporated into clinical practice. Once these challenges are overcome, big data systems have the potential to transform healthcare management in the hospital of the future.

Fast approaches for semantic service composition in big data environment

The widespread deployment of web services and the rapid development of big data applications bring in new challenges to web service compositions in the context of big data. The large number of web ...

Requirements Engineering in the Context of Big Data Applications

Requirements Engineering (RE) plays an essential role in the software engineering process, being considered as one of the most critical phases of the software development life-cycle. As we might expect, then, the Requirements Engineering would play a similar role in the context of Big Data applications. However, practicing Requirements Engineering is a challenging and complex task. It involves (i) stakeholders with diverse backgrounds and levels of knowledge, (ii) different application domains, (iii) it is expensive and error-prone, (iii) it is important to be aligned with business goals, to name a few. Because it involves such complex activities, a lot has to be understood in order to properly address Requirements Engineering. Especially, when the technology domain (e.g., Big Data) is not yet well explored. In this context, this paper describes a research plan on Requirements Engineering involving the development of Big Data applications. The high-level goal is to investigate: (i) On the technical front, the Requirements Engineering activities with respect to the analysis and specification of Big Data requirements and, (ii) on the management side, the relationship between RE and Business Goals in the development of Big Data Software applications.

A visual programming framework for distributed Internet of Things centric complex event processing

Complex Event Processing (CEP) is a promising approach for real-time processing of big-data streams originating from Internet of Things (IoT) devices. Even though scalability and flexibility are key issues for IoT applications, current studies are mostly based on centralized solutions and restrictive query languages. Moreover, development, deployment and operation of big-data applications require significant amount of technical expertise. Hence, a framework that provides a higher abstraction level programming model and a streamlined execution environment is essential to facilitate widespread user adoption. In this paper, we propose a data-flow based visual programming model that enables development of big-data applications in IoT domain even by users who have limited or no experience in big-data technologies. A scalable CEP engine is also devised to execute the applications developed using this language. A prototype is implemented by using the state-of-the-art big-data and real-time stream processing technologies to evaluate the applicability and usability of the framework.

Big data analytics for proactive industrial decision support

Big data technologies offer new opportunities for analyzing historical data generated by process plants. The development of new types of operator support systems (OSS) which help the plant operators during operations and in dealing with critical situations is one of these possibilities. The project FEE has the objective to develop such support functions based on big data analytics of historical plant data. In this contribution, we share our first insights and lessons learned in the development of big data applications and outline the approaches and tools that we developed in the course of the project.

Big data analytics for proactive industrial decision support

Big data technologies offer new opportunities for analyzing historical data generated by process plants. The development of new types of operator support systems (OSS) which help the plant operators during operations and in dealing with critical situations is one of these possibilities. The project FEE has the objective to develop such support functions based on big data analytics of historical plant data. In this contribution, we share our first insights and lessons learned in the development of big data applications and outline the approaches and tools that we developed in the course of the project.