Large Data Analysis Research Articles

“Empathy for children is often missing”: a mixed methods analysis of a German forum on COVID-19 pandemic measures

BackgroundThe pandemic and its preventive measures disrupted daily routines and posed unforeseen obstacles for families. Users of public online forums chronicled these challenges by freely expressing their sentiments in unrestricted text-length formats. We explored a German COVID-19 forum to understand family perspectives and experiences of pandemic measures, particularly in terms of testing and vaccinating children. Our findings aim to inform future epidemic health policies.MethodsWe retrieved all 11,207 entries from a COVID-19 forum during its lifespan (June 2020 - December 2021), posted during the height of the pandemic. We classified the entries into topic clusters including general pandemic situation, testing, or vaccination using state-of-the-art text embeddings and clustering algorithms. The clusters were selected based on the research’s aims and analysed qualitatively using a health policy triangle framework.ResultsUsers generally appreciated pandemic public health safety measures for everyone’s protection, yet voiced concerns about inconsistent policies and disproportional disadvantages for children compared to other societal groups, like the elderly. Non-compliers were overwhelmingly regarded with skepticism and critiqued. Users found COVID-19 (exit) strategies and information about the benefits of vaccination unclear. This created hurdles for parents and caregivers in navigating their children’s school and social life. Users endorsed vaccinating children mainly for “normalising” children’s lives rather than for their physical health benefits. Some users suggested prioritising teachers and early childhood educators on the vaccination eligibility list to speed up a return to “normality”.ConclusionsDuring pandemics, governments should prioritize addressing the societal and mental health needs of children by implementing participatory and family-oriented public health measures for schools and kindergartens. Clear communication coupled with consistent design and implementation of safety measures and regulations, would be crucial for building trust in the general population and for ensuring compliance regarding testing and vaccination. Communicating the benefits and risks of vaccinating children is of paramount importance for informed decision-making among parents. In future epidemics, computer-aided analysis of large online qualitative data would offer valuable insights into public sentiments and concerns, enabling proactive and adaptive epidemic responses.

Интеллектуальная обработка текстовой информации: обзор автоматизированных методов суммаризации

Interest in innovative technological strategies and modern digital tools has increased significantly due to the need to manage large amounts of unstructured data. This paper reviews current paradigms and services for automated summarization, developed based on interdisciplinary research in linguistics, computer technologies, and artificial intelligence. It focuses on syntactic and lexical techniques employed by neural network models for text compression. The paper presents performance examples of such AI-powered services as QuillBot, Summate.it, WordTune, SciSummary, Scholarcy, and OpenAI ChatGPT. The contemporary automated models proved effective in using extractive and abstractive methods to generate summaries of varying quality and length. The extractive approach relies on identifying the most significant sentences from the original text, while abstractive algorithms create new sentence structures that preserve the main idea of the original content. Automated summarizers effectively utilize text compression techniques that are inherent to human approach to text processing, e.g., they exclude redundant information, simplify complex structures, and generalize data. These technologies provide high accuracy and coherence in the generated summaries, though each summarization model has its limitations. Optimal results depend on the specifics of the task at hand: extractive models provide brevity and precision while abstractive ones allow for deeper semantic processing. Automated summarization is becoming an important tool in various fields that require effective analysis and processing of large text data.

S180 Prediction of New-Onset Diabetes Following Acute and Chronic Pancreatitis Using Machine Learning Algorithms: Analysis of Large Real-World Data

S180 Prediction of New-Onset Diabetes Following Acute and Chronic Pancreatitis Using Machine Learning Algorithms: Analysis of Large Real-World Data

Efficient Nonparametric Estimation of 3D Point Cloud Signals through Distributed Learning

Advancements in technology have elevated the prominence of 3D point cloud data, making its analysis increasingly vital across various applications. This need drives the demand for advanced statistical analytic approaches to handle challenges such as size, sparsity, and irregularity in 3D point clouds while ensuring accurate and efficient information extraction. This paper introduces a novel nonparametric distributed (NPD) learning framework that utilizes trivariate spline smoothing over a triangulation of domain. The proposed NPD algorithm features a straightforward, scalable, and communication-efficient implementation scheme that can achieve near-linear speedup. In addition, we provide rigorous theoretical support for the NPD estimation framework and demonstrate that the NPD spline estimators attain the same convergence rate as the global spline estimators obtained using the entire dataset and achieve the optimal nonparametric convergence rate established by Stone (1982) under some regularity conditions. To evaluate the efficacy of the proposed NPD method, we conduct simulation studies comparing it with several global nonparametric estimation methods used to smooth the 3D data. The results demonstrate the superior performance of the NPD method in accurately and efficiently processing and learning from 3D point clouds, highlighting its potential to advance large and complex data analysis. Supplementary material, which contains related technical details, proofs of the theoretical results, and additional results in simulation studies, is available online.

Intelligent hybrid deep learning models for enhanced shipboard solar irradiance prediction and charging station

Electrifying marine routes with solar energy significantly reduces ocean carbon emissions, playing a vital role in climate regulation. Solar irradiance forecasting ensures reliable power despite unpredictable sea weather, necessitating innovative model development. This research presents a forecasting model designed for the optimal placement of solar charging stations, providing hour-ahead solar irradiance predictions for onboard solar vessels. India (Jawaharlal Nehru Port Trust)– United States (Port of New York and New Jersey) sea route is selected to test our proposed forecasting model, as it aligns with busy shipping operations and long-distance trade requirements for electrification. Two distinct procedures by two case studies are used to select optimum data to predict the suitable locations for solar charging stations along the navigation route. In the first case study (Medium data analysis – MDA), the data are selected based on a statistical analysis of twenty years of hourly solar irradiance data from 201 locations. In the second case study (Large data analysis – LDA), two years of seasonal data from 201 locations are chosen and combined to form a large amount of hourly data. For both case studies, a hybrid solar irradiance forecasting model (ARIMA – Bi – LSTM) is developed by integrating the deep learning model with the time series model and fine-tuning them using optimization algorithms (PSO and GA). In MDA, the PSO optimization achieves mean absolute error (MAE) values of 0.32, 0.85, 1.83, 1.40, and 1.33 W/m2 for locations 1, 2, 3, 4, and 5, respectively, significantly outperforming the GA, which has MAE values of 1.36, 1.65, 3.25, 0.92, and 1.41 W/m2 for the same locations. In LDA, the PSO model achieves MAE values of 0.79, 0.91, 0.28, and 0.39 W/m2 for winter, rain, summer, and spring, respectively, also outperforming the GA, which has MAE values of 1.54, 2.17, 0.99, and 1.46 W/m2 for the same seasons. Comparatively, the LDA reinforces its best alignment to predict optimal charging station locations.

Misclassification of Eyes With Progressive Keratoconus Using the KISA% Index.

To determine the misclassification rate of the keratoconus percentage (KISA%) index efficacy in eyes with progressive keratoconus. This was a retrospective case-control study of consecutive patients with confirmed progressive keratoconus and a contemporaneous normal control group with 1.00 diopters or greater regular astigmatism. Scheimpflug imaging (Pentacam HR) was obtained for all patients. KISA% index and inferior-superior (IS) values were obtained from the Pentacam topometric/keratoconus staging map. Receiver operating characteristic curves were generated to determine the area under the receiver operating characteristic curve (AUROC), sensitivity, and specificity values. There were 160 eyes from 160 patients evaluated, including 80 eyes from 80 patients with progressive keratoconus and 80 eyes from 80 control patients. There were 20 eyes (25%) with progressive keratoconus misclassified by the KISA% index, with 16 eyes (20%) of the progressive keratoconus cohort classified as normal (ie, KISA% < 60). There were 4 eyes (5%) with progressive keratoconus that would classify as having "normal topography" using the published criteria for very asymmetric ectasia with normal topography of KISA% less than 60 and IS value less than 1.45. All controls had a KISA% index value of less than 15. The optimal cut-off value to distinguish cohorts was 15.31 (AUROC = 0.972, 93.75% sensitivity). KISA% index values of 60 and 100 achieved low sensitivity (80% and 73.75%, respectively). The KISA% index misclassified a significant proportion of eyes with progressive keratoconus as normal. Although highly specific for clinical keratoconus, the KISA% index lacks sensitivity, does not effectively discriminate between normal and abnormal topography, and thus should not be used in large data analysis or artificial intelligence-based modeling. [J Refract Surg. 2024;40(9):e614-e624.].

Is it time for a European Convention against Racial Profiling?

Four years after the murder of George Floyd and the global Black Lives Matter protests, racial profiling not only remains prevalent in law enforcement practices in- and outside Europe, but takes on new forms in the wake of technological developments enabling large scale data analysis and the use of algorithmic risk profiles. A few years ago, the European Court of Human Rights (ECtHR) first dealt explicitly with the issue of racial profiling in the twin cases of Basu v. Germany and Muhammad v. Switzerland. More recently, the ECtHR held in Wa Baile c. Suisse that Switzerland had violated the prohibition of discrimination by subjecting the applicant to racial profiling at Zürich train station, as part of a police action to apprehend persons without a valid residence permit. 1 While these judgments, especially in Wa Baile, go some way towards securing international State accountability for racial profiling, they also illustrate the difficulties of fighting such profiling as a form of structural discrimination under the ECHR. Meanwhile other human rights bodies, at the European and international levels, have both acknowledged the structural nature of racial profiling and published numerous policy guidelines and recommendations for positive State action to ensure non-discriminatory law enforcement. Yet, to date all of these instruments remain at the level of soft law. Despite the ECtHR's recent efforts, European human rights law thus does not impose legal obligations on States to actively prevent and combat racial profiling. This column signals this gap, and considers some of the pros and cons of adopting a European Convention against Racial Profiling.

The benefit of adjuvant chemotherapy in pathological T1-3N0M0 rectal mucinous adenocarcinoma: no improvement survival outcomes based on long-term survival analysis of large population data.

Currently, the benefits of the administration of adjuvant chemotherapy (AT) in pathological low-risk rectal mucinous adenocarcinoma (RM) with T1-3N0M0 are unclear. The objective of this study is to retrospectively investigate the clinical significance of AT in terms of survival outcomes for patients with pathological T1-3N0M0 RM using data from a large population. The patient data were collected from the Surveillance, Epidemiology, and End Results (SEER) Program. The Chi-squared test was used to analyze categorical variables. The survival curves were compared using the log-rank test and the Kaplan-Meier method. A multivariate proportional hazards regression (Cox) model was applied to identify the independent prognostic factors of survival outcomes. Propensity score matching (PSM) was utilized to eliminate the differences between groups and estimate AT's effect. The median follow-up duration for the rectal cancer (RC) cohort was 116 months. Multivariate analyses revealed that RM was a significant adverse prognostic factor, correlating with poorer overall survival (OS) and cancer-specific survival (CSS) for RC [hazard ratio (HR): 1.226, 95% confidence interval (CI): 1.094-1.375, P<0.001; HR: 1.446, 95% CI: 1.242-1.683, P<0.001]. Among patients with RM, the rates of 5-year OS and CSS were 68.6% and 79.3% in the AT (-) group, respectively. Additionally, the AT (+) group exhibited similar rates of 65.6% for 5-year OS and 74% for CSS (P=0.80, P=0.26). Subtype analysis according to preoperative therapy status showed that AT also did not significantly affect survival outcomes (P=0.65, P=0.34; P=0.90, P=0.76). Our study found that RM is a poor prognostic factor in pathological T1-3N0M0 RC. However, AT does not appear necessary to improve survival outcomes of pathological T1-3N0M0 RM.

Variations in antihypertensive medication treatment and blood pressure control among Veterans with HIV and existing hypertension

BackgroundHypertension is a leading risk factor for cardiovascular disease among patients living with HIV (PLWH). Understanding the predictors and patterns of antihypertensive medication prescription and blood pressure (BP) control among PLWH with hypertension (HTN) is important to improve the primary prevention efforts for this high-risk population. We sought to assess important patient-level correlates (eg, race) and inter-facility variations in antihypertension medication prescriptions and BP control among Veterans living with HIV (VLWH) and HTN. MethodsWe studied VLWH with a diagnosis of HTN who received care in the Veterans Health Administration (VHA) from January 2018 to December 2019. We evaluated HTN treatment and blood pressure control across demographic variables, including race, and by medical comorbidities. Data were also compared among VHA facilities. Predictors of HTN treatment and control were assessed in 2-level hierarchical multivariate logistic regression models to estimate odds ratios (ORs). The VHA facility random-effects parameters from the hierarchical models were used to calculate the median odds ratios to characterize the variation across the different VHA facilities. ResultsA total of 17,468 VLWH with HTN (mean age 61 years, 97% male, 54% Black, 40% White) who received care within the VHA facilities in 2018-2019 were included. 73% were prescribed antihypertension medications with higher prescription rates among Black vs White patients (75% vs 71%) and higher prescription rates among patients with a history of cardiovascular disease, diabetes, and kidney disease (>80%), and those receiving antiretroviral therapy and with controlled HIV viral load (∼75%). Only 27% of VLWH with HTN had optimal BP control of systolic BP <130 mmHg and diastolic BP <80 mmHg, with a lower rate of control among Black vs White patients (24% v. 31%). In multivariate regression, Black patients had a higher likelihood of HTN medication prescription (OR 1.32, 95% CI: 1.22-1.42) but were less likely to have optimal BP control (OR 0.82; 0.76-0.88). Important positive correlates of antihypertensive prescription and optimal BP control included: number of outpatient visits in prior year, and histories of diabetes, coronary artery disease, and heart failure. There was about 10% variability in both antihypertensive prescription and BP control patterns between VHA facilities for patients with similar characteristics. There was increased inter-facility variation in antihypertensive prescription among those with a history of heart failure and those not receiving antiretroviral therapy. ConclusionIn a retrospective analysis of large VHA data, we found that VLWH with HTN have suboptimal antihypertensive medication prescription and BP control. Black VLWH had higher HTN medication prescription rates but lower optimal BP control.

LEST: Large language models and spatio-temporal data analysis for enhanced Sino-US exchange rate forecasting

Exchange rate forecasting has a significant impact on a country’s balance of international payments, financial security and stability. Under the context of Sino-US trade war, a series of news events have happened thus increasing the abnormal fluctuations in the RMB exchange rate, and these fluctuations have made it more difficult to forecast the exchange rate. The analysis of news events’ impact on exchange rate fluctuations focused on assigning emotional labels to different economic news and political events by building an emotional lexicon. However, the previous method of using an offline emotional lexicon to analyze events could not be updated promptly making it difficult to better exploit implicit information of news events. Also, the offline emotional lexicon requires manual annotation of events which is unavoidable and subjective and therefore influences the sentiment analysis of the event. Therefore, we build a fusion forecasting framework of spatio-temporal and emotional information for accurate, real-time exchange rate forecasting. First, in the preprocessing phase, we leveraged the situational learning capabilities of Large Language Models (LLMs), specifically ChatGPT, combined with prompt engineering to enable ChatGPT to provide real-time, online, and precise automated sentiment analyses of Sino-US trade events. Second, we proposed the SG-TL fusion model, which integrates Spatio-Temporal Graph Convolutional Networks (STGCN) and Gated Recurrent Unit (GRU) models in the feature processing approach to accurately extract exchange rate features in spatial and temporal dimensions. We also employed Transformer and Long Short-Term Memory (LSTM) models in the forecasting phase to forecast the Sino-US exchange rates. Finally, Experiment results demonstrate that the exchange rate forecasting framework LEST can accurately capture the exchange rate fluctuations during the period of Sino-US trade war. Furthermore, its forecasting performance is more precise, efficient, and stable than existing works. The source code of our developed method is publicly available at https://gitee.com/andyham_andy.ham/lest-forecasting-framework.

AEG-1 as a Novel Therapeutic Target in Colon Cancer: A Study from Silencing AEG-1 in BALB/c Mice to Large Data Analysis.

Astrocyte elevated gene-1 (AEG-1) is overexpressed in various malignancies. Exostosin-1 (EXT-1), a tumor suppressor, is an intermediate for malignant tumors. Understanding the mechanism behind the interaction between AEG-1 and EXT-1 may provide insights into colon cancer metastasis. AOM/DSS was used to induce tumor in BALB/c mice. Using an in vivo-jetPEI transfection reagent, transient transfection of AEG-1 and EXT-1 siRNAs were achieved. Histological scoring, immunohistochemical staining, and gene expression studies were performed from excised tissues. Data from the Cancer Genomic Atlas and GEO databases were obtained to identify the expression status of AEG-1 and itsassociation with the survival. In BALB/c mice, the AOM+DSS treated mice developed necrotic, inflammatory and dysplastic changes in the colon with definite clinical symptoms such as loss of goblet cells, colon shortening, and collagen deposition. Administration of AEG-1 siRNA resulted in a substantial decrease in the disease activity index. Mice treated with EXT-1 siRNA showed diffusely reduced goblet cells. In vivo investigations revealed that PTCH-1 activity was influenced by upstream gene AEG-1, which in turn may affect EXT-1 activity. Data from The Cancer Genomic Atlas and GEO databases confirmed the upregulation of AEG-1 and downregulation of EXT-1 in cancer patients. This study revealed that AEG-1 silencing might alter EXT-1 expression indirectly through PTCH-1, influencing cell-ECM interactions, and decreasing dysplastic changes, proliferation and invasion.

Linguistic means of identity construction in media texts (comparative analysis of political interviews)

he study aimed to determine and explain the influence of discourse of political interviews on audience identity construction and their understanding of social and political processes. Methods of cross-cultural analysis, Critical Discourse Analysis (CDA), the method of linguistic analysis of identity markers in media text, and the method of linguistic analysis of identity markers were used in the study. The reliability of the methods was checked with Cronbach's alpha. The increase in the importance of media and social networks in identity construction indicates reorientation of the modern society towards digital platforms as a key factor of public position determination. Increased individualization was fixed in identity formation, where personal choices and clarity are important factors affecting the self-determination process. The study confirms that identity formation in media texts is a complex and multifaceted process, which includes the interaction of different aspects as individual, group, cultural and political identities. Further research perspectives are the study of the role of new technologies, including artificial intelligence and large data analysis in identity construction and interaction with the audience in the digital environment.

A Method for Reducing Training Time of ML-Based Cascade Scheme for Large-Volume Data Analysis.

We live in the era of large data analysis, where processing vast datasets has become essential for uncovering valuable insights across various domains of our lives. Machine learning (ML) algorithms offer powerful tools for processing and analyzing this abundance of information. However, the considerable time and computational resources needed for training ML models pose significant challenges, especially within cascade schemes, due to the iterative nature of training algorithms, the complexity of feature extraction and transformation processes, and the large sizes of the datasets involved. This paper proposes a modification to the existing ML-based cascade scheme for analyzing large biomedical datasets by incorporating principal component analysis (PCA) at each level of the cascade. We selected the number of principal components to replace the initial inputs so that it ensured 95% variance retention. Furthermore, we enhanced the training and application algorithms and demonstrated the effectiveness of the modified cascade scheme through comparative analysis, which showcased a significant reduction in training time while improving the generalization properties of the method and the accuracy of the large data analysis. The improved enhanced generalization properties of the scheme stemmed from the reduction in nonsignificant independent attributes in the dataset, which further enhanced its performance in intelligent large data analysis.

Soft tissue tumor imaging in adults: whole-body staging in sarcoma, non-malignant entities requiring special algorithms, pitfalls and special imaging aspects. Guidelines 2024 from the European Society of Musculoskeletal Radiology (ESSR).

The revised European Society of Musculoskeletal Radiology (ESSR) consensus guidelines on soft tissue tumor imaging represent an update of 2015 after technical advancements, further insights into specific entities, and revised World Health Organization (2020) and AJCC (2017) classifications. This second of three papers covers algorithms once histology is confirmed: (1) standardized whole-body staging, (2) special algorithms for non-malignant entities, and (3) multiplicity, genetic tumor syndromes, and pitfalls. A validated Delphi method based on peer-reviewed literature was used to derive consensus among a panel of 46 specialized musculoskeletal radiologists from 12 European countries. Statements that had undergone interdisciplinary revision were scored online by the level of agreement (0 to 10) during two iterative rounds, that could result in 'group consensus', 'group agreement', or 'lack of agreement'. The three sections contain 24 statements with comments. Group consensus was reached in 95.8% and group agreement in 4.2%. For whole-body staging, pulmonary MDCT should be performed in all high-grade sarcomas. Whole-body MRI is preferred for staging bone metastasis, with [18F]FDG-PET/CT as an alternative modality in PET-avid tumors. Patients with alveolar soft part sarcoma, clear cell sarcoma, and angiosarcoma should be screened for brain metastases. Special algorithms are recommended for entities such as rhabdomyosarcoma, extraskeletal Ewing sarcoma, myxoid liposarcoma, and neurofibromatosis type 1 associated malignant peripheral nerve sheath tumors. Satisfaction of search should be avoided in potential multiplicity. Standardized whole-body staging includes pulmonary MDCT in all high-grade sarcomas; entity-dependent modifications and specific algorithms are recommended for sarcomas and non-malignant soft tissue tumors. These updated ESSR soft tissue tumor imaging guidelines aim to provide support in decision-making, helping to avoid common pitfalls, by providing general and entity-specific algorithms, techniques, and reporting recommendations for whole-body staging in sarcoma and non-malignant soft tissue tumors. An early, accurate, diagnosis is crucial for the prognosis of patients with soft tissue tumors. These updated guidelines provide best practice expert consensus for standardized imaging algorithms, techniques, and reporting. Standardization can improve the comparability examinations and provide databases for large data analysis.

Towards revealing intrinsic vortex-core states in Fe-based superconductors through statistical discovery

In type-II superconductors, electronic states within magnetic vortices hold crucial information about the paring mechanism and can reveal non-trivial topology. While scanning tunneling microscopy/spectroscopy (STM/S) is a powerful tool for imaging superconducting vortices, it is challenging to isolate the intrinsic electronic properties from extrinsic effects like subsurface defects and disorders. Here we combine STM/STS with basic machine learning to develop a method for screening out the vortices pinned by embedded disorder in iron-based superconductors. Through a principal component analysis of large STS data within vortices, we find that the vortex-core states in Ba(Fe0.96Ni0.04)2As2 start to split into two categories at certain magnetic field strengths, reflecting vortices with and without pinning by subsurface defects or disorders. Our machine-learning analysis provides an unbiased approach to reveal intrinsic vortex-core states in novel superconductors and shed light on ongoing puzzles in the possible emergence of a Majorana zero mode.

Strategic Review of the Effectiveness of Open-Source Intelligence at the National Maritime Information Center

This research provides a qualitative analysis of having Open-Source Intelligence (OSINT) at the National Maritime Information Center (NMIC). Indonesia's geographic condition demands strong marine defense and has enormous challenges in the maritime sector, so it is necessary to integrate maritime information data systems through the National Maritime Information Center. Technological advances provide opportunities and threats related to the potential to obtain information quickly. New technologies help localize, process, and disseminate large amounts of data and prepare it for analysis coming from Open-Source Intelligence. NMIC is required to be able to meet the needs of its users in the form of short and fast intelligence reports by utilizing information from OSINT. This is what underlies researchers to conduct research with problem formulations that have the effectiveness of OSINT in the NMIC. This research uses a qualitative method with an exploratory approach to find and analyze more in-depth things that are not yet known about OSINT. Productivity theory is used by researchers to discuss problem formulations assisted by NVIVO 12 tools in processing data and Soft System Methodology (SSM) to analyze data. The results showed that to use OSINT effectively, NMIC must have large data analysis capabilities using artificial intelligence assistance.

Evaluating thermal conductivity of soil-rock mixtures in Qinghai-Tibet plateau based on theory models and machine learning methods

The thermal property of the scum layer (soil-rock mixtures) has dominant influence on the heat exchange efficiency between the lower rock layer and the upper environment in the open-pit mines of the cold regions. This paper presents a series of thermal conductivity tests (560 samples) on the scum particle to investigate the coupling effects of ice (moisture) content, temperature, and particle size distribution on the thermal properties. Previously reported models (47 empirical or theoretical models) were adopted to predicate the thermal conductivity of soil-rock mixtures in order to validate the evaluation ability of these models under the wide testing ranges. The comparison results indicate that the theoretical models, normalized model and linear/non-linear models all can not fully predict experimental results under the wide testing conditions. Three machine learning algorithms were used in the assessment presentation for the thermal properties of soil-rock mixtures. The performance of three machine learning algorithms were contrastively examined by using three important indexes (the coefficient of determination (R2), the root mean square error (RMSE) and the relative error (RE)). Based on the evaluation results, the performance ranking of three machine learning algorithms can be listed (GA-BP > SVR > RFR). This investigation is a beneficial attempt for the large data analysis to introduce the machine learning method into the determination of the thermal conductivity of soil-rock mixture under complex conditions.

Hierarchical dependence modeling for the analysis of large insurance claims data

Hierarchical dependence modeling for the analysis of large insurance claims data

Privacy Challenges and Solutions in Big Data Analytics: A Comprehensive Review

Abstract: Large data collections with a more intricate and diversified structure are referred to as "big data." These traits are typically associated with more challenges when it comes to storing, analyzing, and applying additional procedures or getting results. The technique of analyzing vast quantities of intricate data to uncover patterns or connections that are concealed is known as "big data analytics." Nonetheless, there is a clear conflict between big data's extensive use and its security and privacy. This paper examines privacy needs in big data, distinguishes between privacy and security, and focuses on privacy and security issues in large data. This study addresses the applications of privacy in business by utilizing established techniques like HybrEx, k-anonymity, T-closeness, and L-diversity. Many privacy-preserving techniques have been created for protection of privacy at various phases of the big data life cycle (such as data generation, storage, and processing). This work aims to explain the issues facing current privacy preservation strategies in big data and to give a comprehensive evaluation of them. This study also introduces new approaches to privacy preservation in big data, such as quick anonymization of massive data streams, identity-based anonymization, differential privacy, privacy preserving big data publishing, and hiding a needle in a haystack. This study discusses big data privacy and security issues in healthcare. A comparative analysis of some contemporary large data privacy approaches is also conducted. A comparative analysis of the latest big data privacy approaches is also conducted.

FastBiCmrMLM: a fast and powerful compressed variance component mixed logistic model for big genomic case-control genome-wide association study.

Large sample datasets have been regarded as the primary basis for innovative discoveries and the solution to missing heritability in genome-wide association studies. However, their computational complexity cannot consider all comprehensive effects and all polygenic backgrounds, which reduces the effectiveness of large datasets. To address these challenges, we included all effects and polygenic backgrounds in a mixed logistic model for binary traits and compressed four variance components into two. The compressed model combined three computational algorithms to develop an innovative method, called FastBiCmrMLM, for large data analysis. These algorithms were tailored to sample size, computational speed, and reduced memory requirements. To mine additional genes, linkage disequilibrium markers were replaced by bin-based haplotypes, which are analyzed by FastBiCmrMLM, named FastBiCmrMLM-Hap. Simulation studies highlighted the superiority of FastBiCmrMLM over GMMAT, SAIGE and fastGWA-GLMM in identifying dominant, small α (allele substitution effect), and rare variants. In the UK Biobank-scale dataset, we demonstrated that FastBiCmrMLM could detect variants as small as 0.03% and with α ≈ 0. In re-analyses of seven diseases in the WTCCC datasets, 29 candidate genes, with both functional and TWAS evidence, around 36 variants identified only by the new methods, strongly validated the new methods. These methods offer a new way to decipher the genetic architecture of binary traits and address the challenges outlined above.