File Reading Research Articles

Hadoop-Based Medical Image Storage and Access Method for Examination Series

Under the background of electronic medical data, doctors use electronic images to replace the traditional film for diagnosis, and patients can view examination images at any time through various electronic means. The storage and frequent reading of massive data bring new challenges. Given the characteristics of the size and quantity of image files generated by different examination types, different merging strategies are proposed to improve the storage performance of the files; according to the characteristics of medical data with examination as the basic unit, a two-level model combined with medical imaging information is proposed. The indexing mechanism solves the problem that SEQ files cannot be read randomly without an index; given the time characteristics of data access, an improved 2Q algorithm is proposed to cache the prefetched files and the read files in different cache queues, which improves the efficiency of file reading. In the experimental comparison, the proposed algorithm surpasses the baseline method in storage and access performance.

Massive picture retrieval system based on big data image mining

The traditional picture retrieval system has a slow retrieval speed, poor retrieval accuracy, and a low recall when performing massive picture retrieval. In this paper, we design a massive picture retrieval system using the big data image mining technology. It is constructed with data processing layer, business logic layer and presentation layer and works through three steps of data segmentation, mining and merging. For instance, it runs the distributed file system module in a Master/Slave operation mode and designs file read and write requests according to user interaction. Next, it performs parallel computing of picture data sets based on Map Reduce module to solve the picture matching and similarity metrics and returns to the user sorted picture matching result. Then, it extracts the color and texture features of the target area to generate the final picture retrieval result. We select a large number of pictures on a big data platform as simulation test set. The results show that the system we designed has a good retrieval accuracy and a high retrieval speed, which greatly improves the recall of picture retrieval.

Genomic characterization and phylogenetic analysis of the first SARS-CoV-2 variants introduced in Lebanon.

BackgroundIn December 2019, the COVID-19 pandemic initially erupted from a cluster of pneumonia cases of unknown origin in the city of Wuhan, China. Presently, it has almost reached 94 million cases worldwide. Lebanon on the brink of economic collapse and its healthcare system thrown into turmoil, has previously managed to cope with the initial SARS-CoV-2 wave. In this study, we sequenced 11 viral genomes from positive cases isolated between 2 February 2020 and 15 March 2020.MethodsSequencing data was quality controlled, consensus sequences generated, and a maximum-likelihood tree was generated with IQTREE v2. Genetic lineages were assigned with Pangolin v1.1.14 and single nucleotide variants (SNVs) were called from read files and manually curated from consensus sequence alignment through JalView v2.11 and the genomic mutational interference with molecular diagnostic tools was assessed with the CoV-GLUE pipeline. Phylogenetic analysis of whole genome sequences confirmed a multiple introduction scenario due to international travel.ResultsThree major lineages were identified to be circulating in Lebanon in the studied period. The B.1 (20A clade) was the most prominent, followed by the B.4 lineage (19A clade) and the B.1.1 lineage (20B clade). SNV analysis showed 15 novel mutations from which only one was observed in the spike region.

To Mobilise or to Mediate? Re-assessing the Democratic Role of News Media in Social Protests in China

Protester-sympathising media reports validate activists’ claims, expose official wrongdoing, and mobilise public support, which allows scholars to highlight the importance of the media in promoting democratic participation in authoritarian China. Reaffirming media’s crucial role in sustaining communicative rationality, the article re-evaluates media coverage of four rural protests against land expropriation in China, combining framing analysis of media reports, in-depth interviews, and an extensive reading of court files, etc. It unveils that two storylines—transgressive collective action for maximising economic gains and conflicts inside villages—are tailored off, when information is woven into the dominant media frame “struggle of the weak”. Simplified, but logically coherent, the media narrative is likely to exclude the necessity for public deliberation, reduce the fleeting public activism into anger-venting, and pressures local governments into makeshift concessions at the cost of public good. The one-dimensional civic engagement urges Chinese journalists to consider innovating protest reporting frame.

Avoiding data loss and corruption for file transfers with Fast Integrity Verification

End-to-end integrity verification is used to avoid silent data corruption in file transfers by comparing the checksum of files at source and destination end points. However, it increases transfer times significantly as checksum computation requires reading files back from the storage and running compute-intensive hash computation. In this paper, we propose Fast Integrity VERification (FIVER) algorithm which alleviates the overhead of end-to-end integrity verification by overlapping checksum computation with transfer operation and enabling I/O sharing between the two. The results obtained from various network and dataset settings show that FIVER is able to bring down the cost of end-to-end integrity verification from up to 120% by the state-of-the-art solutions to below 15%. Moreover, existing implementations of end-to-end integrity verification are vulnerable to permanent data loss in the case of an unexpected power outage due to completing the integrity verification process while data is still on memory. FIVER addresses this issue by enforcing dirty data on memory to be flushed to disk before finishing integrity verification such that power loss during any phase of a transfer would cause integrity verification to fail and transfer application to retransfer lost data.

Loos, a Better Tool to Analyze Molecular Dynamics Simulations

The enormous improvements in computational power in recent years means that we can routinely runs simulations of biologically interesting systems out to the microsecond scale or longer. However, in many cases their scientific impact is limited by an inability to extract useful information from this wealth of data. We created LOOS to unlock this previously unrealized potential. Molecular dynamics is unique among biophysical techniques in that the raw data can be interrogated in a multitude of ways; getting the most out of the data usually requires exploration and iterative refinement. LOOS is designed to make this process easy and efficient, to minimize the friction between a user and her data. LOOS can be viewed in two ways: 1) It is a suite of well-validated, easy-to-use tools for analyzing molecular dynamics simulations. These tools include common tasks, such as trajectory manipulation, as well as more advanced techniques such as analysis of simulation convergence, 3D histograms, and Voronoi decomposition, with a particular focus on membranes and membrane-bound proteins. 2) It is a tool for making tools. At its heart, LOOS is a C++ library covered with a Python interface, for the optimal mix of high performance and ease of development. LOOS is designed for rapid application development, so that most common tasks (reading files, selecting atoms, aligning structures, computing distances, etc) are one-line operations. LOOS is licensed under the GPLv3 license and available via GitHub (https://github.com/GrossfieldLab/loos).

A Study of Piano Timbre Teaching in the Context of Artificial Intelligence Interaction.

This paper provides an in-depth analysis and research on piano timbre teaching in the context of artificial intelligence interaction, a bold vision of piano teaching, proposes a feasible solution in terms of teaching modules in intelligent piano teaching for senior teachers, and proposes an implementation path for the integration of intelligent piano and piano teaching from the four main blocks of piano teaching. Based on the multiplicative harmonic model of monophonic signal, combined with the variability of timbre characteristics, an audio synthesis model with editable timbre is proposed, and the experimental results show that editing the timbre parameters in the model can realize timbre modification, and the synthesized timbre conforms to the piano timbre characteristics. Based on the timbre analysis and the timbre synthesis model, a piano timbre library generation system is designed. The detailed design of the software modules such as audio file reading and writing, audio information analysis, timbre parameter acquisition, timbre synthesis, and simulated performance is given. The system can generate piano timbre libraries of different qualities flexibly and meet the requirements of timbre realism. The teaching experiment designed for teaching practice from solo teaching, and the practice target is first-year undergraduate students in the university, and the practice period is six weeks, and finally, the feasibility of intelligent piano teaching application is analysed by combining the experimental results. Through the teaching objectives, teaching content, and teaching methods, teaching environment reflects intelligent piano teaching to make up for the limitations of traditional piano teaching. Analyse the development trend of intelligent piano teaching in the context of artificial intelligence interaction, and explore the value of intelligent piano teaching.

PandAna: A Python Analysis Framework for Scalable High Performance Computing in High Energy Physics

Modern experiments in high energy physics analyze millions of events recorded in particle detectors to select the events of interest and make measurements of physics parameters. These data can often be stored as tabular data in files with detector information and reconstructed quantities. Most current techniques for event selection in these files lack the scalability needed for high performance computing environments. We describe our work to develop a high energy physics analysis framework suitable for high performance computing. This new framework utilizes modern tools for reading files and implicit data parallelism. Framework users analyze tabular data using standard, easy-to-use data analysis techniques in Python while the framework handles the file manipulations and parallelism without the user needing advanced experience in parallel programming. In future versions, we hope to provide a framework that can be utilized on a personal computer or a high performance computing cluster with little change to the user code.

IOData: A python library for reading, writing, and converting computational chemistry file formats and generating input files.

IOData is a free and open-source Python library for parsing, storing, and converting various file formats commonly used by quantum chemistry, molecular dynamics, and plane-wave density-functional-theory software programs. In addition, IOData supports a flexible framework for generating input files for various software packages. While designed and released for stand-alone use, its original purpose was to facilitate the interoperability of various modules in the HORTON and ChemTools software packages with external (third-party) molecular quantum chemistry and solid-state density-functional-theory packages. IOData is designed to be easy to use, maintain, and extend; this is why we wrote IOData in Python and adopted many principles of modern software development, including comprehensive documentation, extensive testing, continuous integration/delivery protocols, and package management. This article is the official release note of the IOData library.

SDMP: A secure detector for epidemic disease file based on DNN

In the era of intelligent office, reading and processing PDF files by mobile devices have become important parts of various businesses. However, due to the universality of mobile device and PDF file, they are also often used by attackers to disguise malicious codes, which makes users in danger. Especially in medical field, once computer virus invades medical experts’ devices, a large number of data with high medical research value will face huge damage and irreparable loss. Therefore, how to ensure the security when users make use of PDF files is a challenging and meaningful task. In this paper, we design an secure detector of malicious PDF file for epidemic disease file based on Deep Neural Network to solve the problem of privacy and security in handling epidemic disease file. Experiment shows that the detection accuracy of our detector can achieve up to 99.3%. Moreover, the time cost on raining and forecasting of the proposed DNN model is extremely low, less than 1s per epoch.

Efficient ensemble data assimilation for coupled models with the Parallel Data Assimilation Framework: example of AWI-CM (AWI-CM-PDAF 1.0)

Abstract. Data assimilation integrates information from observational measurements with numerical models. When used with coupled models of Earth system compartments, e.g., the atmosphere and the ocean, consistent joint states can be estimated. A common approach for data assimilation is ensemble-based methods which utilize an ensemble of state realizations to estimate the state and its uncertainty. These methods are far more costly to compute than a single coupled model because of the required integration of the ensemble. However, with uncoupled models, the ensemble methods also have been shown to exhibit a particularly good scaling behavior. This study discusses an approach to augment a coupled model with data assimilation functionality provided by the Parallel Data Assimilation Framework (PDAF). Using only minimal changes in the codes of the different compartment models, a particularly efficient data assimilation system is generated that utilizes parallelization and in-memory data transfers between the models and the data assimilation functions and hence avoids most of the file reading and writing, as well as model restarts during the data assimilation process. This study explains the required modifications to the programs with the example of the coupled atmosphere–sea-ice–ocean model AWI-CM (AWI Climate Model). Using the case of the assimilation of oceanic observations shows that the data assimilation leads only to small overheads in computing time of about 15 % compared to the model without data assimilation and a very good parallel scalability. The model-agnostic structure of the assimilation software ensures a separation of concerns in which the development of data assimilation methods can be separated from the model application.

A Safe Web in Network Separation Environment

This paper describes a technique for constructing network separation and proposes a method for providing convenience to users by using it. This paper analyzes the background of the Korean government’s adoption of the network separation policy. And there are various techniques for configuring network separation. NAC, media control solution and file transfer system are essential for network separation. Because just running two networks does not guarantee secure. This paper considers ways to provide efficiency and convenience to the extent allowed by the network separation policy. In this paper, Using Headless Browser, transform web pages into electronic documents. The converted electronic document does not contain malicious code. Several Headless browser packages were used to implement and test assuming a virtual network separation. Web scraping technology worked reliably, almost all browsers can read and display PDF file. In this paper, we have defined a safe web prototype in a network separation environment.

RIVA: Robust Integrity Verification Algorithm for High-Speed File Transfers

End-to-end integrity verification is designed to protect file transfers against silent data corruption by comparing checksum of files at source and destination end points using cryptographic hash functions such as MD5 and SHA1. However, existing implementations of end-to-end integrity verification for file transfers fall short to detect undetected disk errors that causes inconsistency between disk and cache memory. In this article, we propose Robust Integrity Verification Algorithm (RIVA) to strengthen the integrity of file transfers by forcing checksum computation tasks to read files directly from disk. RIVA achieves this by invalidating memory mappings of file pages after their transfer such that when the file is read again for checksum calculation, it will be fetched from disk and silent disk errors will be captured. We design and conduct extensive fault resilience experiments to evaluate the robustness of integrity verification algorithms against undetected disk write errors. The results indicate that while the state-of-the-art integrity verification algorithms fail to detect the injected errors for almost all file sizes, RIVA captures all of them with the help of cache invalidation. We further run statistical analysis to assess the probability of missing silent disk errors and find that RIVA reduces the likelihood by 10 to 15 orders of magnitude compared to the existing approaches. Finally, enforcing disk read in integrity verification introduces an inevitable overhead in exchange of increased robustness against silent disk errors, but RIVA keeps its overhead below 15 percent in most cases by running transfer, cache invalidation, and checksum computation processes concurrently for different portions of the same file.

SeisIO: A Fast, Efficient Geophysical Data Architecture for the Julia Language

Abstract SeisIO for the Julia language is a new geophysical data framework that combines the intuitive syntax of a high-level language with performance comparable to FORTRAN or C. Benchmark comparisons against recent versions of popular programs for seismic data download and analysis demonstrate significant improvements in file read speed and orders-of-magnitude improvements in memory overhead. Because the Julia language natively supports parallel computing with an intuitive syntax, we benchmark test parallel download and processing of multiweek segments of contiguous data from two sets of 10 broadband seismic stations, and find that SeisIO outperforms two popular Python-based tools for data downloads. The current capabilities of SeisIO include file read support for several geophysical data formats, online data access using a variety of services, and optimized versions of several common data processing operations. Tutorial notebooks and extensive documentation are available to improve the user experience. As an accessible example of performant scientific computing for the next generation of researchers, SeisIO offers ease of use and rapid learning without sacrificing computational efficiency.

Parallel performance of molecular dynamics trajectory analysis

SummaryThe performance of biomolecular molecular dynamics simulations has steadily increased on modern high‐performance computing resources but acceleration of the analysis of the output trajectories has lagged behind so that analyzing simulations is becoming a bottleneck. To close this gap, we studied the performance of trajectory analysis with message passing interface (MPI) parallelization and the Python MDAnalysis library on three different Extreme Science and Engineering Discovery Environment (XSEDE) supercomputers where trajectories were read from a Lustre parallel file system. Strong scaling performance was impeded by stragglers, MPI processes that were slower than the typical process. Stragglers were less prevalent for compute‐bound workloads, thus pointing to file reading as a bottleneck for scaling. However, a more complicated picture emerged in which both the computation and the data ingestion exhibited close to ideal strong scaling behavior whereas stragglers were primarily caused by either large MPI communication costs or long times to open the single shared trajectory file. We improved overall strong scaling performance by either subfiling (splitting the trajectory into separate files) or MPI‐IO with parallel HDF5 trajectory files. The parallel HDF5 approach resulted in near ideal strong scaling on up to 384 cores (16 nodes), thus reducing trajectory analysis times by two orders of magnitude compared with the serial approach.

Optimized data storage algorithm of IoT based on cloud computing in distributed system

The existing Internet of Things(IoT) uses cloud computing data access storage algorithms, that is, the hash algorithm has defects of low data processing efficiency and low fault tolerance rate. Therefore, HDFS is introduced to optimize cloud computing data access storage algorithms. HDFS is first used to optimize the data access storage architecture according to problems of data access storage architecture in the Internet of Things, in which factors of data access storage distribution in the IoT are fully considered, and hash values are used to optimize the configuration of data access information storage locations, so that data access storage distribution strategy can be optimized. Then, the topology of the IoT is optimized, and data block size is also optimized with effect algorithm. Finally, the design of file storage is optimized. Through simulation experiments, it is proved that the optimized cloud storage method has obvious performance advantages in file read and write speed as well as memory usage. Compared with the traditional hash algorithm, optimization algorithm proposed in the paper greatly improves file upload and download efficiency, data processing efficiency and fault tolerance rate, which fully demonstrates that the proposed cloud computing data access storage optimization algorithm is more superior.

BlobToolKit - Interactive Quality Assessment of Genome Assemblies.

Reconstruction of target genomes from sequence data produced by instruments that are agnostic as to the species-of-origin may be confounded by contaminant DNA. Whether introduced during sample processing or through co-extraction alongside the target DNA, if insufficient care is taken during the assembly process, the final assembled genome may be a mixture of data from several species. Such assemblies can confound sequence-based biological inference and, when deposited in public databases, may be included in downstream analyses by users unaware of underlying problems. We present BlobToolKit, a software suite to aid researchers in identifying and isolating non-target data in draft and publicly available genome assemblies. BlobToolKit can be used to process assembly, read and analysis files for fully reproducible interactive exploration in the browser-based Viewer. BlobToolKit can be used during assembly to filter non-target DNA, helping researchers produce assemblies with high biological credibility. We have been running an automated BlobToolKit pipeline on eukaryotic assemblies publicly available in the International Nucleotide Sequence Data Collaboration and are making the results available through a public instance of the Viewer at https://blobtoolkit.genomehubs.org/view. We aim to complete analysis of all publicly available genomes and then maintain currency with the flow of new genomes. We have worked to embed these views into the presentation of genome assemblies at the European Nucleotide Archive, providing an indication of assembly quality alongside the public record with links out to allow full exploration in the Viewer.

Evaluating Students’ attitudes and usage e of mobile in educational activities at Paramedical Sciences School

Introduction: Mobile learning is a new way of modern teaching method and a subset of e-learning that refers to a change in thinking about the design and planning of learning goals and environments. The purpose of this study was to investigate the attitude of students of the Medical Sciences School toward the mobile phone in educational activities and its usage. Methods: 150 students of Mashhad School of Paramedical Sciences were investigated in this cross-sectional study in 2019-20, using a researcher-made questionnaire including three sections on student demographic information, mobile applications in education, and student attitude toward mobile phone use in educational activities. The questionnaire`s validity was obtained according to 7 experts' opinions in the field of education, and its reliability was calculated using Cronbach's alpha of 0.794. Data were analyzed by descriptive and analytical statistics using SPSS 16 software. Result: From the investigation of 150 students of Mashhad Paramedical Sciences opinions in six educational groups indicated that students of health information technology, laboratory science, physiotherapy, ophthalmology, radiology, speech therapy, medical records, mostly used mobile phones for their educational activities respectively. The most common use of students' mobile phones at university was the use of the Internet and social networks, taking photos and videos, and the most common use in teaching activities was reading files, viewing scores, recording audio and selecting units. From the students' perspective, using mobile phones in educational activities leads to getting higher scores, doing activities faster, make better communication between teacher and student, and speeded up the learning feedback. Conclusion: According to the results, we suggest encouraging teachers to use mobile phones in academic activities and running workshops to increase the awareness of mobile capabilities. Also educational planners, and mobile based learning programs should design educational software to promote the correct culture of mobile use in educational activities.

LOOS: A Suite of Tools to Analyze Molecular Dynamics Simulations

The enormous improvements in computational power in recent years means that we can routinely runs simulations of biologically interesting systems out to the microsecond scale or longer. However, in many cases their scientific impact is limited by an inability to extract useful information from this wealth of data. We created LOOS to unlock this previously unrealized potential. Molecular dynamics is unique among biophysical techniques in that the raw data can be interrogated in a multitude of ways; getting the most out of the data usually requires exploration and iterative refinement. LOOS is designed to make this process easy and efficient, to minimize the friction between a user and her data. LOOS can be viewed in two ways: 1) It is a suite of well-validated, easy-to-use tools for analyzing molecular dynamics simulations. These tools include common tasks, such as trajectory manipulation, as well as more advanced techniques such as analysis of simulation convergence, 3D histograms, and Voronoi decomposition, with a particular focus on membranes and membrane-bound proteins. 2) It is a tool for making tools. At its heart, LOOS is a C++ library covered with a Python interface, for the optimal mix of high performance and ease of development. LOOS is designed for rapid application development, so that most common tasks (reading files, selecting atoms, aligning structures, computing distances, etc) are one-line operations. LOOS is licensed under the GPLv3 license and available via GitHub (https://github.com/GrossfieldLab/loos).

Resistance Sniffer: An online tool for prediction of drug resistance patterns of Mycobacterium tuberculosis isolates using next generation sequencing data

The effective control of multidrug resistant tuberculosis (MDR-TB) relies upon the timely diagnosis and correct treatment of all tuberculosis cases. Whole genome sequencing (WGS) has great potential as a method for the rapid diagnosis of drug resistant Mycobacterium tuberculosis (Mtb) isolates. This method overcomes most of the problems that are associated with current phenotypic drug susceptibility testing. However, the application of WGS in the clinical setting has been deterred by data complexities and skill requirements for implementing the technologies as well as clinical interpretation of the next generation sequencing (NGS) data. The proposed diagnostic application was drawn upon recent discoveries of patterns of Mtb clade-specific genetic polymorphisms associated with antibiotic resistance. A catalogue of genetic determinants of resistance to thirteen anti-TB drugs for each phylogenetic clade was created. A computational algorithm for the identification of states of diagnostic polymorphisms was implemented as an online software tool, Resistance Sniffer (http://resistance-sniffer.bi.up.ac.za/), and as a stand-alone software tool to predict drug resistance in Mtb isolates using complete or partial genome datasets in different file formats including raw Illumina fastq read files. The program was validated on sequenced Mtb isolates with data on antibiotic resistance trials available from GMTV database and from the TB Platform of South African Medical Research Council (SAMRC), Pretoria. The program proved to be suitable for probabilistic prediction of drug resistance profiles of individual strains and large sequence data sets.