FORCv2: A High-Throughput Streaming FPGA Accelerator for Optimized Row Columnar File Format Processing in Big Data Engines

Cloud computing and big data: Technologies and applications

Cloud computing and big data: Technologies and applications

The purpose of the work is to study the current problems and prospects of the solution for processing big data received or stored in the Internet (web data), as well as the possibility of practical realization of Data Mining technology for big web data on practical example. Materials and methods. The study included a review of bibliographic sources on big data analysis problems.Data Mining technology was used to analyze large web data, as well as computer modeling of a practical problem using the C # programming language and creating a DDL database structure for accumulating web data.Results. In the course of the work, the specifics of big data were described, the main characteristics of big data were highlighted, and modern approaches to processing big data were analyzed. A brief description of the horizontal-scalable architecture and the BI-solution architecture for big data processing is given. The problems of processing large web data are formulated: limiting the speed of access to data, providing access via network protocols through general-purpose networks.An example showing the approach to processing large web data was also implemented. Based on the idea of big data, the described complexities of web data processing and the methods of Data Mining, techniques were proposed for effectively solving the practical problem of processing and searching patterns in a large data array.The following classes have been developed in the C # programming language:Class of receiving web data via the Internet; Data conversion class;Intelligent data processing class;Created DDL script that creates a structure for the accumulation of web data.A single UML class diagram has been developed.The constructed system of data and classes allows to solve the main part of the problems of processing large web data and perform intelligent processing using Data Mining technology in order to solve the problem posed of identifying certain records in a large array. The combination of object-oriented approach, neural networks and BI-analysis to filter data will speed up the process of data processing and obtaining the result of the studyConclusion. According to the results of the study, it can be argued that the current state of technology for analyzing large web data allows you to efficiently process data objects, identify patterns, get hidden data and get full-fledged statistical data.The obtained results can be used both for the purpose of the initial study of big data processing technologies, and as a basis for developing an already real application for analyzing web data. The use of neural networks and the created universal classes-handlers makes the created architecture flexible and self-learning, and the class declarations and the base DDL structure will greatly simplify the development of program code.

Big Building Data - a Big Data Platform for Smart Buildings

The confluence of science, technology, and medicine in our dynamic digital era has spawned new data applications to develop prescriptive analytics, to improve healthcare personalization and precision medicine, and to automate the reporting of health data for clinical decisions.1 Data science in health care has seen recent and rapid progress along 3 paths: (1) through big data via the aggregation of large and complex data sets including electronic medical records, social media, genomic databases, and digitized physiological data from wireless mobile health devices2; (2) through new open-access initiatives that seek to leverage the availability of clinical trial, research, and citizen science data sources for data sharing3; and (3) in analytic techniques particularly for big data, including machine learning and artificial intelligence that may enhance the analyses of both structured and unstructured data.4 As new data sets are created, analyzed, and become increasingly available, several key questions emerge including the following: What is the quality of unstructured data generation? Will the use of nonstandardized methods in data processing with traditional software and hardware lead to data fragmentation and analyses that are nonreproducible? Will healthcare systems incorporate and use big data especially from new publically and patient-generated sources? How will physicians and researchers learn from new open-sourced data and big-data analytics? And ultimately, How can they acquire the skills to create a knowledge translation in data sciences?5 Practicing in an era of continuous payment reform and decline in research funding, early career investigators are challenged to keep up with the accelerating pace of change in medicine, all while being expected to provide meaningful contributions through productive clinical, educational, and research experiences.6 In this perspective, we aim to highlight how data science can catalyze professional advancement and discuss the implications of big data, open access, …

Advances in experimental techniques have led to an explosion in both the number and size of macromolecular structures in the Protein Data Bank (PDB). For this reason, the transfer and parsing of macromolecular data has become increasingly time-consuming. In this work we present the Macromolecular Transmission Format (MMTF), a new compact, extensible macromolecular file format. MMTF offers over 75% compression over mmCIF, and is over an order of magnitude faster to parse than the standard mmCIF format. We describe the new MMTF format, its Application Programming Interface, and demonstrate its use with Big Data Frameworks to enable nearly interactive data analytics on the PDB archive. Introduction The MacroMolecular Transmission Format (MMTF) ( htt p://mmtf.rcsb.org/ ) [1] was designed with three major aims. First, to minimize data size and network transfer times, the format applies custom encoding and compression techniques [2]. Both a lossless all atom and a lossy representation with reduced level of detail (C-alpha of polypeptides, phosphate backbone of polynucleotides, and all atoms of ligands) are available. Second, the data files should be fast to parse. Third, MMTF was designed to be self-contained, extensible, and interoperable. As a binary, machine-readable format, the preferred access to MMTF is through its APIs, provided in several programming languages. In this work we show that MMTF enables high-performance structural bioinformatics calculations. Methods MMTF is a compact and extensible file format. Compression is enhanced through columnar encoding and custom compression strategies. Dictionary encoding is used to encode repeating residue, bond and atom information. Delta, run-length, and recursive index encoding strategies are used to compress data arrays, such as atomic coordinates and other atomic properties [2]. Decoder libraries are available for Java, JavaScript, Python, C, and C++. The PDB is available as individual MMTF files and as Hadoop Sequence files for parallel processing. The data files are updated weekly ( http://mmtf.rcsb.org/download.html ). Results & Conclusions Using the Apache Spark distributed parallel computing framework, the PDB can be processed in parallel ranging from laptops, workstations, to compute clusters. We show how Big Data operations such as Filter, Map, Reduce, and Broadcasting can be used to implement structural bioinformatics workflows. Further, extracting structure and sequence information into Dataframes enables parallel SQL queries. Acknowledgements This project was supported by the NCI of the NIH under award number U01 CA198942. References Bradley AR, Rose AS, Pavelka A, Valasatava Y, Duarte JM, Prlić A, Rose PW, PLOS Computational Biology 13(6): e1005575. doi: 10.1371/journal.pcbi.1005575 (2017). Valasatava Y, Bradley AR, Rose AS, Duarte JM, Prlić A, Rose PW. PLoS One. 12: e0174846, doi: 10.1371/journal.pone.01748464 (2017).

One of the important features of the “new information age” is “big data”. In the age of big data, with the introduction of student information management systems, colleges and universities are increasingly paying attention to systematically managing student information management and improving their efficiency, effective management of student information and innovative development to build a new model of management system. In studying the evolution of student information management, big data technologies emerging in the information age play an important role. By leveraging big data to improve systematic management of student information, colleges and universities can manage student information faster and more efficiently and make new contributions to the advancement of student information management. In this article, information management is considered as the theoretical basis for research, big data technology is used as the main search algorithm, and its important content is combined to analyze and study the possibilities of the student information management system. In this article, an information management system for university students is used as a research object, which is optimized and improved through big data technologies. Big Data technology can be seen as a kind of algorithm for analyzing and processing data. Therefore, it can be used to create a student information management strategy. The experimental results show that this research has a better impact on the creation and implementation of an information management system for students using Big Data technology and the application of Big Data technology in the information management system for students.

Big data analytics in Industry 4.0 ecosystems

With the emergence of Big Data Technologies (BDT) and the growing application of Big Data Analytics (BDA), Supply Chain Management (SCM) researchers increasingly utilize BDA due to the opportunities from BDT and BDA present. Supply Chain (SC) data is inherently complex and results in an environment with high uncertainty, which presents a real challenge for SC decision-makers. This research study aimed to investigate and illustrate the application of BDA within the existing decision-making process. BDT allowed for the extraction and processing of SC data. BDA aided further understanding of SC inefficiencies and delivered valuable, actionable insights by validating the existence of the SC bullwhip phenomenon and its contributing factors. Furthermore, BDA enabled the pragmatic evaluation of linear and nonlinear regression SC relationships by applying machine learning techniques such as Principal Component Analysis (PCA) and multivariable regression analysis. Moreover, applying more sophisticated BDA time series and forecasting techniques such as Sarimax, Tbats, and neural networks improved forecasting accuracy. Ultimately, the improved demand planning and forecast accuracy will reduce SC uncertainty and the effects of the observed SC bullwhip phenomenon, thus creating a competitive advantage for all the members within the SC value chain.

Various types of data are produced by the shipbuilding and offshore industry. As the number of ships and offshore structures that were built over time increased, an enormous amount of data, called big data, had to be handled. However, it is difficult to handle effectively such big data with the existing methodology for data storage and processing. Therefore, big data technology needs to be applied to the systems of shipyards, such as the product lifecycle management system. On the other hand, the construction of an offshore structure requires a lot of piping, and there are many materials as much as piping. For a shipyard that executes multiple projects at once, it is not easy to correlate so much piping material. A piping designer should check all materials based on his or her understanding of the design characteristics. However, depending on the maturity of the designer, it can be difficult to handle such large data manually; as a result, there can be errors in piping design. In this study, a big data framework applicable to the shipyard is proposed and used for the analysis. That is, the association analysis of piping materials of an offshore structure is performed based on the big data framework that can process a large amount of data to assist piping designers. As an application, after analyzing the material data for one offshore structure, the applicability of this study was evaluated through the results. We believe this study can help piping designers. 1. Introduction 1.1. Research background One of the fastest growing technologies in several industries is big data and its applications. Big data can be defined as a series of computer technologies that can store, process, and manage much more data than were handled before. From this next-generation technology and architecture, data can be easily gathered and analyzed. Big data is often characterized by three different words that start with "V" (Beyer 2011). The first "V" word is volume and it refers to a large volume of data. The next "V" word is velocity, referring to how fast the data processing is. Usually, it should be enough to obtain the result of the analysis in real time without any delay. Also, there are many types of processable data, which can also be unstructured; hence, the last "V" word stands for variety. Recently, two more words are used because we cannot define the characteristics of big data with only three words. One is "veracity"; another is "value." Veracity is for selecting high-quality data from all of it (Villanova University 2018) and value is the main reason why people use big data technology (Mauro et al. 2013). Table 1 describes the characteristics of big data briefly.

The paper examines the special aspects of using Big Data technology in education. The population was made up of 356 third-year university students. To study Big Data technology, a questionnaire was used where respondents rated: cloud technology; apps; Massive Open Online Courses (MOOCs) and digital learning platforms. The study suggested that the education sector is ambitiously applying Big Data technology, both online and offline. All surveyed respondents use apps in Big Data learning and analysis: 73.03% use Moodle, 67.13% use Zoom, 65.17% use Quizlet, 50.84% use Skype, and 35.11% use Slack. MOOCs in education are used by 75% of respondents. Digital learning platforms are used by all respondents. All students use cloud technology. When dealing with Big Data technologies, students preferred apps (8.9 ± 1.33) instead of the cloud (6.9 ± 0.11). Students believe that the important factors for using Big Data in the learning process include: quality of information (85.96%); interest (77.81%); instructor’s support (66.85%). The research findings make it possible to integrate Big Data technology into the learning process, thus improving learning outcomes and providing greater speed in processing reliable and meaningful data.

Internet of things (IoT) and big data technologies are increasingly gaining significance in the implementation of various services and applications. Consequently, much of the research focused on energy efficiency and building management revolves around integrating IoT and big data technologies for data collection and processing. Occupancy detection, comfort, and energy management are the most important services for optimizing building energy consumption in smart buildings, and environmental data play a key role in improving these services. Furthermore, the integration of advanced and recent techniques, such as IoT, big data, and machine learning, is progressively becoming more vital for both researchers and industries. This paper presents and discusses various emerging technologies that will contribute to designing novel IoT-based architectures to improve smart building services. These technologies offer innovative solutions to address the challenges of interoperability, scalability, and real-time data processing within intelligent environments, paving the way for more efficient, adaptive, and user-centric smart building systems. The main aim of this research is to help researchers define an optimal architecture that presents all layers, from sensing to big data stream processing. We established comparative criteria between the most popular data processing techniques to select the appropriate framework for developing intelligent platforms for managing building services, such as occupancy detection systems and occupants’ comfort management, and further, to enhance the deployment of digital twins for critical environment monitoring and anomaly detection. The proposed architecture uses Apache Kafka, Apache Storm, and Apache SAMOA as its core components, creating a comprehensive platform for efficient data collection, monitoring, and processing with high performance in terms of fault tolerance and low latency.

The paper addresses a highly relevant and contemporary topic in the field of data processing. Big data is a crucial aspect of modern computing, and the choice of processing framework can significantly impact performance and efficiency. The technical revolution of big data has changed how organizations handle and value large databases. As data quantities expand quickly, effective and scalable data processing systems are essential. MapReduce and Apache Spark are two of the most popular large data processing techniques. This study compares these two frameworks to determine their merits, shortcomings, and applicability for big data applications. Nearly quintillion bytes of data are created daily. Approximately 90% of data was produced in the previous two years. At this stage, data comes from temperature sensors, social media, movies, photographs, transaction records (like banking records), mobile phone conversations, GPS signals, etc. In this article, all key big data technologies are introduced. This document compares all big data technologies and discusses their merits and downsides. Run trials using multiple data sets of varying sizes to validate and explain the study. Graphical depiction shows how one tool outperforms others for given data. Big Data is data generated by the rapid usage of the internet, sensors, and heavy machinery, with great volume, velocity, variety, and veracity. Numbers, photos, videos, and text are omnipresent in every sector. Due to the pace and amount of data generation, the computing system struggles to manage large data. Data is stored in a distributed architectural file system due to its size and complexity. Big distributed file systems, which must be fault-tolerant, adaptable, and scalable, make complicated data analysis dangerous and time-consuming. Big data collection is called ‘datafication’. Big data is ‘datafied’ for productivity. Organisation alone does not make Big Data valuable; we must choose what we can do with it.

Big data technologies have shown great promise for managing geospatial data in recent years. In order to deal with the growing spatial data, a high performance spatial data processing system layered on big data technologies is needed. In this paper, we present an approach to process big spatial data with Apache Spark, a fast and generic engine for large-scale data processing. We developed a software development kit named SparkSpatialSDK, which takes spatial characteristics of geospatial data into consideration and provides a Spark-enabled spatial data structure and API to allow users easily perform spatial analyses with big spatial data. The spatial data structure couples geometric data structure (point, line, and polygon) with Resilient Distributed Datasets (RDD). An interface, called SpatialRDD, is provided to access big spatial data stored in distributed database systems like HBase and load the data in Spark processing engine. We illustrates the applications of the API using some example processing functions such as the spatial range and spatial k-nearest neighbor queries. The results demonstrate the applicability of using SparkSpatialSDK for big geospatial data processing.

Photo by Josh Riemer on Unsplash
 Introduction
 With the rapid advancements in neurotechnological machinery and improved analytical insights from machine learning in neuroscience, the availability of big brain data has increased tremendously. Neurological health research is done using digitized brain data.[1] There must be adequate data governance to secure the privacy of subjects participating in brain research and treatments. If not properly regulated, the research methods could lead to significant breaches of the subject’s autonomy and privacy. This paper will address the necessity for neuroprotection laws, which effectively govern the use of big brain data to ensure respect for patient privacy and autonomy.
 Background
 Artificial intelligence and machine learning can be integrated with neuroscience big brain data to drive research studies. This integrative technology allows patterns of electrical activity in neurons to be studied in detail.[2]Specifically, it uses a robotic system which can reason, plan, and exhibit biologically intelligent behavior. Machine learning is a method of computer programming where the code can adapt its behavior based on big brain data.[3] The big brain data is the collection of large amounts of information for the purpose of deciphering patterns through computer analysis using machine learning.[4] The information that these technologies provide is extensive enough to allow a researcher to read a patient’s mind. AI and machine learning technologies work by finding the underlying structure of brain data, which is then described by patterns known as latent factors, eventually resulting in an understanding of the brain’s temporal dynamics.[5]
 Through these technologies, researchers are able to decipher how the human brain computes its performances and thoughts. However, due to the extensive and complex nature of the data processed through AI and machine learning, researchers may gain access to personal information a patient may not wish to reveal. From a bioethical lens, tensions arise in the realm of patient autonomy. Patients are not able to control the transmission of data from their brains that is analyzed by researchers. Governing brain data through laws may enhance the extent of patient privacy in the case where brain data is being used through AI technologies.[6] A responsible approach to governing brain data would require a sophisticated legal structure.
 Analysis
 Impact on Patient Autonomy and Privacy 
 In research pertaining to big brain data, the consent forms do not fully cover the vast amounts of information that is collected. According to research, personal data has become the most sought out commodity to provide content to corporations and the web-based service industry. Unfortunately, data leaks that release private information frequently occur.[7] The storage of an individual’s data on technologies accessible on the internet during research studies makes it vulnerable to leaks, jeopardizing an individual’s privacy. These data leaks may cause the patient to be identified easily, as the degree of information provided by AI technologies are personalized and may be decoded through brain fingerprinting methods.[8]
 There has been an extensive growth in the development and use of AI. It is efficient in providing information to radiologists who diagnose various diseases including brain cancer and psychiatric disease, and AI assists in the delivery of telemedicine.[9] However, the ethical pitfall of reduced patient autonomy must be addressed by analyzing current AI technologies and creating more options for patient preference in how the data may be used. For instance, facial recognition technology[10] commonly used in health care produces more information than listed in common consent forms, threatening to undermine informed consent. Facial recognition software collects extensive data and may disclose more information than a person would prefer to provide despite being a useful tool for diagnosing medical and genetic conditions.[11] In addition, people may not be aware that their images are being used to generate more clinical data for other purposes. It is difficult to guarantee the data is anonymized. Consent requirements must include informing people about the complexity of the potential uses of the data; software developers should maximize patient privacy.[12] Furthermore, there is a “human element” in the use of AI technologies as medical providers control the use and the extent to which data is captured or accessed through the AI technologies.[13] People must understand the scope of the technology and have clear communication with the physician or health care provider about how the medical information will be used. 
 Existing Laws for Brain Data Governance 
 A strict system of defined legal responsibilities of medical providers will ensure a higher degree of patient privacy and autonomy when AI technologies and data from machine learning are used. Governing specific algorithmic data is crucial in safeguarding a patient’s privacy and developing a gold standard treatment protocol following the procurement of the information.[14] Certain AI technologies provide more data than others, and legal boundaries should be established to ensure strong performance, quality control, and scope for patient privacy and autonomy. For instance, currently AI technologies are being used in the realm of intensive neurological care. However, there is a significant level of patient uncertainty about how much control patients have over the data’s uses.[15] Calibrated legal and ethical standards will allow important brain data to be securely governed and monitored.
 Once brain signals are recorded and processed from one individual, the data may be merged with other data in Brain Computer Interface Technology (BCI).[16] To ensure a right and ability to retrieve personal data or pull it from the collection, specific regulations for varying types of data are needed.[17] The importance of consent and patient privacy must be considered through giving patients a transparent view of how brain data is governed.[18] The legal system must address discriminatory issues and risks to patients whose data is used in studies. Laws like the General Data Protection Regulation (GDPR) and the California Consumer Privacy Protection Act (CCPA) can serve as effective models to protect aggregated data. These laws govern consumer information and ensure the compliance when personal data is collected.[19] California voters recently approved expansion of the CCPA to health data. The Washington Privacy Act, which would have provided rights to access, change, and withdraw personal data, failed to pass. Other states should improve privacy as well,[20] although a federal bill would be preferable. Scientists at the Heidelberg Academy of Sciences argue for data security to be governed in a manner that balances patient privacy and autonomy with the commercial interests of researchers.[21] The balance could be achieved through privacy protections like those in the Washington Privacy Act. Although the Health Insurance Portability and Accountability Act (HIPAA) provides an overall framework to deter the likelihood of dangers to patient protection and privacy, more thorough laws are warranted to combat pervasive data transfer and analysis that technology has brought to the health care industry.[22] Breaches of patient privacy under current HIPAA regulations include releasing patient information to a reporter without their consent and sending HIV data to a patient’s employer without consent.[23] HIPAA does not cover information being shared with outside contractors who do not have an agreement with technology companies to keep patient data confidential. HIPAA regulations also do not always address blatant breaches on patient data confidentiality.[24] Patients must be provided with methods to monitor the data being analyzed to be able to view the extent of private information being generated via AI technologies. In health research, the medical purposes of better diagnosis, earlier detection of diseases, or prevention are ethical justifications for the use of the data if it was collected with permission, the person understood and approved the uses of the data, and the data was deidentified.
 A standard governance framework is required in providing the fairest system of care to patients who allow their brain data to be examined. Informed consent in the neuroscience field could reaffirm the privacy and autonomy of patients by ensuring that they understand the type of information collected. Laws also could protect data after a patient’s death. Malpractice in the scope of brain data could give people a cause of action critical in safeguarding patient’s rights. Data breach lawsuits will become common but generally do not cover deidentified data that becomes part of big data collection. A more synchronized approach to the collection and consent process will encourage an understanding of how big data is used to diagnose and treat patients. Some altruistic people may even be more likely to consent if they know the largescale data collection is helpful to treat and diagnose people. Others should have the ability to opt out of sharing neurological data, especially when there is not certainty surrounding deidentification.[25]
 Conclusion
 Artificial intelligence and machine learning technologies have the potential to aid in the diagnosis and treatment of people globally by extracting and aggregating brain data specific to individuals. However, the secure use of the data is necessary to build trust between care providers and patients, as well as in balancing the bioethical principles of beneficence and patient autonomy. We must ensure the highest quality of care to patients, while protecting their privacy, informed consent, and clinical trust. More sophis