Natural Image Processing Research Articles

Advancements and gaps in natural language processing and machine learning applications in healthcare: a comprehensive review of electronic medical records and medical imaging

This article presents a thorough examination of the progress and limitations in the application of Natural Language Processing (NLP) and Machine Learning (ML), particularly Deep Learning (DL), in the healthcare industry. This paper examines the progress and limitations in the utilisation of Natural Language Processing (NLP) and Machine Learning (ML) in the healthcare field, specifically in relation to Electronic Medical Records (EMRs). The review also examines the incorporation of Natural Language Processing (NLP) and Machine Learning (ML) in medical imaging as a supplementary field, emphasising the transformative impact of these technologies on the analysis of healthcare data and patient care. This review attempts to analyse both fields in order to offer insights into the current state of research and suggest potential chances for future advancements. The focus is on the use of these technologies in Electronic Medical Records (EMRs) and medical imaging. The review methodically detects, chooses, and assesses literature published between 2015 and 2023, utilizing keywords pertaining to natural language processing (NLP) and healthcare in databases such as SCOPUS. After applying precise inclusion criteria, 100 papers were thoroughly examined. The paper emphasizes notable progress in utilizing NLP and ML methodologies to improve healthcare decision-making, extract information from unorganized data, and evaluate medical pictures. The key findings highlight the successful combination of natural language processing (NLP) and image processing to enhance the accuracy of diagnoses and improve patient care. The study also demonstrates the effectiveness of deep learning-based NLP pipelines in extracting valuable information from electronic medical records (EMRs). Additionally, the research suggests that NLP has the potential to optimize the allocation of medical imaging resources. The identified gaps encompass the necessity for scalable and practical implementations, improved interdisciplinary collaboration, the consideration of ethical factors, the analysis of longitudinal patient data, and the customization of approaches for specific medical situations. Subsequent investigations should focus on these deficiencies in order to fully exploit the capabilities of natural language processing (NLP) and machine learning (ML) in the healthcare sector, consequently enhancing patient outcomes and the delivery of healthcare services.

A landmark federal interagency collaboration to promote data science in health care: Million Veteran Program-Computational Health Analytics for Medical Precision to Improve Outcomes Now.

In 2016, the Department of Veterans Affairs (VA) and the Department of Energy (DOE) established an Interagency Agreement (IAA), the Million Veteran Program-Computational Health Analytics for Medical Precision to Improve Outcomes Now (MVP-CHAMPION) research collaboration. Oversight fell under the VA Office of Research Development (VA ORD) and DOE headquarters. An Executive Committee and 2 senior scientific liaisons work with VA and DOE leadership to optimize efforts in the service of shared scientific goals. The program supported centralized data management and genomic analysis including creation of a scalable approach to cataloging phenotypes. Cross-cutting methods including natural language processing, image processing, and reusable code were developed. The 79.6 million dollar collaboration has supported centralized data management and genomic analysis including a scalable approach to cataloging phenotypes and launched over 10 collaborative scientific projects in health conditions highly prevalent in veterans. A ground-breaking analysis on the Summit and Andes supercomputers at the Oak Ridge National Laboratory (ORNL) of the genetic underpinnings of over 2000 health conditions across 44 million genetic variants which resulted in the identification of 38270 independent genetic variants associating with one or more health traits. Of these, over 2000 identified associations were unique to non-European ancestry. Cross-cutting methods have advanced state-of-the-art artificial intelligence (AI) including large language natural language processing and a system biology study focused on opioid addiction awarded the 2018 Gordon Bell Prize for outstanding achievement in high-performance computing. The collaboration has completed work in prostate cancer, suicide prevention, and cardiovascular disease, and cross-cutting data science. Predictive models developed in these projects are being tested for application in clinical management. Eight new projects were launched in 2023, taking advantage of the momentum generated by the previous collaboration. A major challenge has been limitations in the scope of appropriated funds at DOE which cannot currently be used for health research. Extensive multidisciplinary interactions take time to establish and are essential to continued progress. New funding models for maintaining high-performance computing infrastructure at the ORNL and for supporting continued collaboration by joint VA-DOE research teams are needed.

Editorial for Vol.32, No.2

The June 2024 (Vol. 32, No. 2) issue of CIT. Journal of Computing and Information Technology brings four papers from the areas of natural language processing, image processing, network security, and biomedical image classification.

Performance Comparison of AI Platforms in Solving Computer Science Problems

In the rapidly evolving landscape of artificial intelligence (AI), its significance and accelerated development are undeniable. AI has emerged as a cornerstone technology with profound implications across various domains, driving innovation and reshaping the way we approach complex problems. Particularly, the utilization of AI in coding tasks has garnered substantial attention, given its potential to streamline development processes and enhance the efficiency of software engineering practices. Against this backdrop, this paper presents a detailed comparative analysis of four different AI platforms, namely ChatGPT, Gemini, Blackbox, and Microsoft Copilot, in addressing key challenges within the realm of computer science, spanning natural language processing, image processing, and cybersecurity. The study focuses on leveraging the C++ programming language to develop solutions for these multifaceted problems across the aforementioned platforms. Each platform's outputs are meticulously evaluated on various parameters including accuracy, execution time, code size, and time complexity to provide a comprehensive understanding of their performance. Furthermore, an iterative optimization methodology is employed, entailing three rounds of refinement for the code produced by each platform, with the resultant outputs subjected to comparative analysis in each iteration. Through this rigorous approach, the paper not only elucidates the efficacy of different AI platforms in addressing diverse computational challenges but also underscores the iterative enhancement process on AI platforms for refining code quality and performance across multiple domains within computer science.

E2SCNet: Efficient Multiobjective Evolutionary Automatic Search for Remote Sensing Image Scene Classification Network Architecture.

Remote sensing image scene classification methods based on deep learning have been widely studied and discussed. However, most of the network architectures are directly reliant on natural image processing methods and are fixed. A few studies have focused on automatic search mechanisms, but they cannot weigh the interpretation accuracy and the parameter quantity for practical application. As a result, automatic global search methods based on multiobjective evolutionary computation have more advantages. However, in the ranking process, the network individuals with large parameter quantities are easy to eliminate, but a higher accuracy may be obtained after full training. In addition, evolutionary neural architecture search methods often take several days. In this article, in order to solve the above concerns, we propose an efficient multiobjective evolutionary automatic search framework for remote sensing image scene classification deep learning network architectures (E2SCNet). In E2SCNet, eight kinds of lightweight operators are used to build a diversified search space, and the coding connection mode is flexible. In the search process, a large model retention mechanism is implemented through two-step multiobjective modeling and evolutionary search, where one step involves the "parameter quantity and accuracy," and the other step involves the "parameter quantity and accuracy growth quantity." Moreover, a super network is constructed to share the weight in the process of individual network evaluation and promote the search speed. The effectiveness of E2SCNet is proven by comparison with several networks designed by human experts and networks obtained by gradient and evolutionary computing-based search methods.

AutoAugHAR: Automated Data Augmentation for Sensor-based Human Activity Recognition

Sensor-based HAR models face challenges in cross-subject generalization due to the complexities of data collection and annotation, impacting the size and representativeness of datasets. While data augmentation has been successfully employed in domains like natural language and image processing, its application in HAR remains underexplored. This study presents AutoAugHAR, an innovative two-stage gradient-based data augmentation optimization framework. AutoAugHAR is designed to take into account the unique attributes of candidate augmentation operations and the unique nature and challenges of HAR tasks. Notably, it optimizes the augmentation pipeline during HAR model training without substantially extending the training duration. In evaluations on eight inertial-measurement-units-based benchmark datasets using five HAR models, AutoAugHAR has demonstrated superior robustness and effectiveness compared to other leading data augmentation frameworks. A salient feature of AutoAugHAR is its model-agnostic design, allowing for its seamless integration with any HAR model without the need for structural modifications. Furthermore, we also demonstrate the generalizability and flexible extensibility of AutoAugHAR on four datasets from other adjacent domains. We strongly recommend its integration as a standard protocol in HAR model training and will release it as an open-source tool1.

Abstract PO2-28-06: An on-line deep learning decision support tool, iBRISK, aimed at improving breast cancer risk estimation and reducing unnecessary biopsies for BI-RADS 4 patients

Abstract Probability of malignancy (POM) for Breast Imaging Reporting and Data System (BI-RADS) category 4 designated breast lesions ranges from 2% – 95% and contributes to a high unnecessary biopsy rate. This is as most clinicians often stick to the biopsy option to rule in or out breast cancer early; withholding biopsy could be risky, and biopsies of BI-RADS 4 lesions serve as a quality metric and performance standard. At 21.1%, biopsy-proven positive predictive value (PPV3) rates for BI-RADS 4 have not improved for decades, translating to high false-positive rates of mammography. Unnecessary biopsies are a big issue in the management of BI-RADS 4 lesions with negative implications including increased medical costs, healthcare wastes, unnecessary psychological burdens to the patients, and potential complications and risks. Objectives 1. Optimize the precision breast cancer risk assessment tool, iBRISK, that utilizes artificial intelligence (AI) technologies, including natural language processing (NLP), image processing, and deep learning, with clinical risk factors and imaging features. 2. Develop a user-friendly web interface for iBRISK to facilitate clinicians or insurers in estimating cancer risk and making informed biopsy decisions for BI-RADS 4 lesions. Methods Our intelligent-augmented breast cancer risk calculator (iBRISK) model was trained on multimodal data collected from 10,778 patients, including demographic factors, historical and clinical characteristics, mammographic features, and pathologic signatures. We validated iBRISK using 4,200 patients from multiple leading hospitals, including Houston Methodist Neal Cancer Center, the University of Texas MD Anderson Cancer Center, and the University of Texas Health San Antonio MD Anderson Mays Cancer Center. The iBRISK was connected to a backend server which is linked to a frontend web user interface using technologies like Hypertext Preprocessor (PHP) for the backend application running on an APACHE HTTP server and React-JavaScript for the front-end web application that communicates with the backend using Representational State Transfer Application Programming Interface (RESTful API) and JavaScript Object Notation (JSON) format. The communication was secured using Secure Sockets Layer (SSL) encryption. Results The iBRISK model demonstrated high sensitivity in malignancy prediction and achieved an accuracy of 89.5%, area under the receiver operating characteristic curve of 0.93 (95% CI: 0.92-0.95), sensitivity of 100%, and specificity of 81%. Only 0.16% of lesions determined to have low POM by the model were malignant. Our multi-center study shows that iBRISK achieves at least 50% reduction in unnecessary biopsies of BI-RADS 4 cases. Data elements for the required 20 features are entered into the interface using a variety of imputation methods including direct text, dropdown menus and radio button selections. The user-friendly web interface provides risk scores (0 – 1), risk levels (low, medium, and high), and associated biopsy recommendations. Conclusion The user-friendly iBRISK web interface is proposed as an adjunct to the BI-RADS system, enhancing the precision of BI-RADS 4 lesion cancer risk stratification. It is expected to reduce unnecessary biopsies, lower health costs, and enhance the quality of health care. This approach aims to tackle a critical issue in breast cancer diagnosis by leveraging advanced AI technologies and big data and providing clinicians with a tool to make more informed and precise biopsy decisions for BI-RADS 4 lesions. Citation Format: Chika Ezeana, Xiaohui Yu, Zhihao Wan, Tiancheng He, Tejal Patel, Virginia Kaklamani, Maryam Elmi, Erika Brigmon, Pamela Otto, Kenneth Kist, Lin Wang, Joe Ensor, Heather Speck, Ya-Chen Shih, Bumyang Kim, I-Wen Pan, David Spak, Wei Yang, Jenny Chang, Stephen Wong. An on-line deep learning decision support tool, iBRISK, aimed at improving breast cancer risk estimation and reducing unnecessary biopsies for BI-RADS 4 patients [abstract]. In: Proceedings of the 2023 San Antonio Breast Cancer Symposium; 2023 Dec 5-9; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2024;84(9 Suppl):Abstract nr PO2-28-06.

Proxyformer: Nyström-Based Linear Transformer with Trainable Proxy Tokens

Transformer-based models have demonstrated remarkable performance in various domains, including natural language processing, image processing and generative modeling. The most significant contributor to the successful performance of Transformer models is the self-attention mechanism, which allows for a comprehensive understanding of the interactions between tokens in the input sequence. However, there is a well-known scalability issue, the quadratic dependency (i.e. O(n^2)) of self-attention operations on the input sequence length n, making the handling of lengthy sequences challenging. To address this limitation, there has been a surge of research on efficient transformers, aiming to alleviate the quadratic dependency on the input sequence length. Among these, the Nyströmformer, which utilizes the Nyström method to decompose the attention matrix, achieves superior performance in both accuracy and throughput. However, its landmark selection exhibits redundancy, and the model incurs computational overhead when calculating the pseudo-inverse matrix. We propose a novel Nyström method-based transformer, called Proxyformer. Unlike the traditional approach of selecting landmarks from input tokens, the Proxyformer utilizes trainable neural memory, called proxy tokens, for landmarks. By integrating contrastive learning, input injection, and a specialized dropout for the decomposed matrix, Proxyformer achieves top-tier performance for long sequence tasks in the Long Range Arena benchmark.

Introduction to Machine Learning

Machine learning has emerged as a transformative technology with wide-ranging applications across diverse domains, from healthcare and finance to autonomous systems and environmental monitoring. This paper provides a comprehensive review of the field of machine learning, encompassing foundational concepts, recent advancements, applications, ethical considerations, challenges, and future directions. The paper begins with an overview of machine learning, highlighting its historical evolution, importance, and applications in various sectors. It then delves into the foundational principles of machine learning, including different types of learning, model evaluation techniques, and statistical foundations. Advances in machine learning algorithms, such as supervised learning, unsupervised learning, reinforcement learning, and deep learning architectures, are discussed in detail. The paper explores the wide-ranging applications of machine learning across different domains, including healthcare, finance, autonomous systems, natural language processing, image and video processing, and environmental monitoring. Ethical and societal implications of machine learning, such as bias and fairness, privacy concerns, automation and job displacement, responsible AI development, and regulatory frameworks, are examined. Challenges and future directions in machine learning, including data quality and quantity issues, interpretability and explainability of models, scalability and resource constraints, addressing ethical and societal concerns, advancements in interdisciplinary research, and opportunities for innovation and collaboration, are discussed. Overall, this paper provides a comprehensive overview of machine learning, highlighting its transformative potential, ethical considerations, challenges, and opportunities for future research and development. It serves as a valuable resource for researchers, practitioners, policymakers, and stakeholders interested in understanding the evolving landscape of machine learning and its impact on society

Deep clustering using 3D attention convolutional autoencoder for hyperspectral image analysis

Deep clustering has been widely applicated in various fields, including natural image and language processing. However, when it is applied to hyperspectral image (HSI) processing, it encounters challenges due to high dimensionality of HSI and complex spatial-spectral characteristics. This study introduces a kind of deep clustering model specifically tailed for HSI analysis. To address the high dimensionality issue, redundant dimension of HSI is firstly eliminated by combining principal component analysis (PCA) with t-distributed stochastic neighbor embedding (t-SNE). The reduced dataset is then input into a three-dimensional attention convolutional autoencoder (3D-ACAE) to extract essential spatial-spectral features. The 3D-ACAE uses spatial-spectral attention mechanism to enhance captured features. Finally, these enhanced features pass through an embedding layer to create a compact data-representation, and the compact data-representation is divided into distinct clusters by clustering layer. Experimental results on three publicly available datasets validate the superiority of the proposed model for HSI analysis.

A Literature Survey on Quadruped AI Assistant: Integrating Image Processing and Natural Language Processing for Emotional Intelligence

At the nexus of artificial intelligence, robotics, image processing, and natural language processing (NLP), quadruped AI assistants provide a revolutionary method of interacting with machines. Intending to give quadruped AI assistants emotional intelligence, this literature review methodically looks over and compiles the body of research, concentrating on how to use natural language processing and image processing together. The review seeks to offer a comprehensive grasp of the developments, challenges, and possible uses in this diverse subject. The initial section of the survey provides an overview of quadruped robots and highlights how they are integrated with image processing technology to provide visual perception. It explores the various locomotion techniques, demonstrating how these robots use picture data to improve their capacity to navigate and adapt to different situations. The conversation also touches on sensor technologies, highlighting their function in obtaining and deciphering visual data for intelligent interaction. In addition, the research delves into how quadruped AI assistants include natural language processing and examines how these robots interpret and react to instructions in human language. One of the main topics of debate is how sentiment analysis and emotional recognition methods might help these assistants become more emotionally intelligent. Finally, this review of the literature offers a comprehensive viewpoint on how natural language processing and image processing are integrated in quadruped AI assistants, providing academics and practitioners with an outline for developing emotional intelligence in these robots. Compiling knowledge from robotics, artificial intelligence, image processing, and natural language processing is essential to creating emotionally competent quadruped AI assistants and ensuring their smooth incorporation into human-centered settings.This research also investigates the integration of Natural Language Processing and Image Processing for Emotional Intelligence in Quadruped AI Assistants and thoroughly assesses the gyroscope functioning on these robotic systems

Progress in protein pre-training models integrating structural knowledge

The AI revolution, sparked by natural language and image processing, has brought new ideas and research paradigms to the field of protein computing. One significant advancement is the development of pre-training protein language models through self-supervised learning from massive protein sequences. These pre-trained models encode various information about protein sequences, evolution, structures, and even functions, which can be easily transferred to various downstream tasks and demonstrate robust generalization capabilities. Recently, researchers have further developed multimodal pre-trained models that integrate more diverse types of data. The recent studies in this direction are summarized and reviewed from the following aspects in this paper. Firstly, the protein pre-training models that integrate protein structures into language models are reviewed: this is particularly important, for protein structure is the primary determinant of its function. Secondly, the pre-trained models that integrate protein dynamic information are introduced. These models may benefit downstream tasks such as protein-protein interactions, soft docking of ligands, and interactions involving allosteric proteins and intrinsic disordered proteins. Thirdly, the pre-trained models that integrate knowledge such as gene ontology are described. Fourthly, we briefly introduce pre-trained models in RNA fields. Finally, we introduce the most recent developments in protein designs and discuss the relationship of these models with the aforementioned pre-trained models that integrate protein structure information.

Self-supervised pre-training with contrastive and masked autoencoder methods for dealing with small datasets in deep learning for medical imaging

Deep learning in medical imaging has the potential to minimize the risk of diagnostic errors, reduce radiologist workload, and accelerate diagnosis. Training such deep learning models requires large and accurate datasets, with annotations for all training samples. However, in the medical imaging domain, annotated datasets for specific tasks are often small due to the high complexity of annotations, limited access, or the rarity of diseases. To address this challenge, deep learning models can be pre-trained on large image datasets without annotations using methods from the field of self-supervised learning. After pre-training, small annotated datasets are sufficient to fine-tune the models for a specific task. The most popular self-supervised pre-training approaches in medical imaging are based on contrastive learning. However, recent studies in natural image processing indicate a strong potential for masked autoencoder approaches. Our work compares state-of-the-art contrastive learning methods with the recently introduced masked autoencoder approach “SparK” for convolutional neural networks (CNNs) on medical images. Therefore, we pre-train on a large unannotated CT image dataset and fine-tune on several CT classification tasks. Due to the challenge of obtaining sufficient annotated training data in medical imaging, it is of particular interest to evaluate how the self-supervised pre-training methods perform when fine-tuning on small datasets. By experimenting with gradually reducing the training dataset size for fine-tuning, we find that the reduction has different effects depending on the type of pre-training chosen. The SparK pre-training method is more robust to the training dataset size than the contrastive methods. Based on our results, we propose the SparK pre-training for medical imaging tasks with only small annotated datasets.

Consistency- and dependence-guided knowledge distillation for object detection in remote sensing images

As one of the challenging tasks in the remote sensing (RS), object detection has been successfully applied in many fields. Convolution neural network (CNN) has recently attracted extensive attention and is widely used in the natural image processing. Nevertheless, RS images have cluttered scenes compared with natural images. As a result, the existing detectors perform poorly in RS images, especially with the complicated backgrounds. Moreover, the detection inference time and model volume of detectors in RS images often go unrecognized. To address the above issues, this study proposes a novel method for object detection in RS images, which is called the consistency- and dependence-guided knowledge distillation (CDKD). To this end, the spatial- and channel-oriented structure discriminative modules (SCSDM) are put forward to extract the discriminative spatial locations and channels to which the teacher model pays attention. SCSDM improves the feature representation of the student model by effectively eliminating the influence of noises and the complicated backgrounds. Then, the consistency and dependence of the features between the teacher model and the student model are constructed under the guidance of SCSDM. Experimental results over public datasets for RS images demonstrate that our CDKD method surpasses the state-of-the-art methods effectively. Most of all, on the RSOD dataset, our CDKD method achieves 92% mean average precision with 3.3M model volume and 588.2 frames per second.

Optimizing University Mobility : An Internal Navigation and Crowd Management System

In the evolving landscape of educational technology, the article explores the critical frontier of indoor navigation systems, focusing on universities. Traditional approaches in higher education often fall short of meeting dynamic user expectations, necessitating revolutionary solutions. This research introduces an innovative internal navigation and crowd management system that seamlessly integrates augmented reality, natural language processing, machine learning, and image processing technologies. The Android platform serves as the foundation, harnessing augmented reality's transformative capabilities to provide real-time visual cues and personalized wayfinding experiences. The voice interaction module, backed by NLP and ML, creates an intelligent, context-aware assistant. The crowd management module, employing advanced image processing, delivers real-time crowd density insights. Personalized recommendations, powered by NLP and ML, offer tailored canteen suggestions based on user preferences. The agmented reality navigation module, using Mapbox, Unity Hub, AR Core, and Vuforia, enriches the user experience with dynamic visual cues. Results reveal the success of each module: the voice interaction module showcases continuous learning, user-centric feedback, contextual guidance excellence, robust security, and multimodal interaction flexibility. The crowd management module excels in video feed processing, image processing with OpenCV, and real-time availability information retrieval. The personalized recommendations module demonstrates high accuracy, equilibrium, and robust performance. The AR navigation module impresses with precision, enriched navigation, and tailored routes through machine learning. This cohesive system sets new benchmarks for user-centric technology in universities. Future work includes multi-university integration, intelligent spatial design, and real-time decision support, paving the way for more efficient, user-centered university experiences and contributing to the advancement of smart university environments. The research serves as a pivotal force in reshaping interactions within university spaces, envisioning a future where technology seamlessly enhances the essence of human interaction in educational environments.

Brain Tumor Classification, Segmentation, and Detection using Deep Learning - A Review

V.Vapnik in 1965 proposed Vector methods. Kimeldorf presented a technique for creating kernel space based on support vectors in 1971. Support Vector Machine (SVM) techniques were initially presented in the 1990s by V. Vapnik in the field of statistical learning. Since then, pattern recognition, natural language processing, image processing and other areas have seen extensive use of SVM. By converting non-linear sample space into linear space via a kernel approach, the algorithm's complexity is reduced. Image classification is a well-known issue in image processing. Predicting the input image categories using the features is the main objective of image classification. There are several different classifiers, including Artificial Neural Networks, Support Vector Machines, and Random Forests, Decision Forests, k-NNs (k Nearest Neighbors), and Adaptive Boost. SVM is one of the best techniques for categorizing any image or pattern. A common non-invasive technique used in the medical sector for the analysis, diagnosis, treatment of brain tissues is magnetic resonance imaging. When a brain tumor is discovered early, the patient's life can be saved by receiving the appropriate care. It becomes difficult to accurately identify tumors in the MRI slices, which requires fussy work..

Mineral identification based on natural feature-oriented image processing and multi-label image classification

Artificial intelligence (AI) technology has significant potential in Earth sciences, particularly in mineral identification for industrial exploration, geological mapping, and archaeological research. However, traditional methods are time-consuming, expensive, and complex. And existing mineral identification methods based on mineral photos face several critical challenges, including lack of consideration for natural image features captured in real environments, limitations of single-label classification which does not align with multi-mineral occurrences in nature, and growing computational complexity as the number of identifiable mineral labels increases. Therefore, this paper proposes an efficient mineral identification model based on multi-label image classification, focusing on natural environmental features. First, realistic feature datasets are created by simulating mineral photos in real environments. Then, the model uses the query-label (Query2Label) framework, with MaxViT-T (Multi-Axis Vision Transformer-Tiny) as the feature extraction network and the asymmetric loss function. Knowledge distillation is employed to improve identification accuracy while reducing computational complexity. The proposed model achieves an impressive average identification accuracy of 84.74% on a dataset of 495,756 mineral photos, surpassing existing models like ResNet-101, ML-GCN (Multi-Label Graph Convolutional Network), and SRN (Spatial Regularization Net). It maintains a lower parameter count and computational complexity. In the end, ablation experiments demonstrate the effectiveness of each optimization scheme.

Back projection deep unrolling network for handwritten text image super resolution

Current super-resolution (SR) methods have demonstrated exceptional advancements in the domain of natural image processing. Nevertheless, these approaches do not fully address the open issues of edge blurring and distortion. In order to address the aforementioned challenges, we propose a novel deep unrolling model, coined back projection deep unrolling network (BPDUN), for super-resolution of handwritten text images leveraging the Algorithm Unrolling paradigm. We design the network model of BPDUN by unfolding and truncating the traditional iterative back projection algorithm (IBPA). We unroll IBPA into a cascade operation of three building blocks (deep denoising module, low-frequency reconstruction module, and residual projection module). BPDUN inherits the interpretability of the iterative optimization algorithm and is also designed to make the reconstructed text image more realistic and natural. Moreover, we propose a new benchmark dataset to address challenging SR problems of handwritten text image (HDT300). Extensive experiments show that BPDUN obtains an enhanced balance between the performance (quantified by PSNR and SSIM) and the cost (as measured by network parameters). Notably, BPDUN sets new benchmarks on the HDT300 dataset, surpassing previous state-of-the-art approaches by achieving up to 0.2 dB gains in PSNR.

The current status and prospects of transformer in multimodality

At present, the attention mechanism represented by transformer has greatly promoted the development of natural language processing (NLP) and image processing (CV). However, in the multimodal field, the application of attention mechanism still mainly focuses on extracting the features of different types of data, and then fusing these features (such as text and image). With the increasing scale of the model and the instability of the Internet data, feature fusion has been difficult to solve the growing variety of multimodal problems for us, and the multimodal field has always lacked a model that can uniformly handle all types of data. In this paper, we first take the CV and NLP fields as examples to review various derived models of transformer. Then, based on the mechanism of word embedding and image embedding, we discuss how embedding with different granularity is handled uniformly under the attention mechanism in multimodal scenes. Further, we reveal that this mechanism will not only be limited to CV and NLP, but the real unified model will be able to handle tasks across data types through pre-training and fine tuning. Finally, on the specific implementation of the unified model, this paper lists several cases, and analyzes the valuable research directions in related fields.

Analysis of Factors Affecting Common Use of Generative Artificial Intelligence-Based Tools by Machine Learning Methods

Artificial Intelligence is a sub-branch of artificial intelligence used to produce new data or content. These methods can create recent examples in different categorical fields such as natural language processing, image processing, music, and video creation by using models from learning clusters with artificial intelligence (AI) tools in this field. AI tools that can solve real-world problems are also created using different methods apart from generative AI methods. With generative-based artificial intelligence tools, it can facilitate people's work in jobs that require creativity. However, they can offer the opportunity to build advanced models that learn from data with other artificial intelligence methods. In the study, the public dataset has been used. This dataset includes trending artificial intelligence tools, AI methods, and user scores. In this study the working area and user trend of the ai tools in the dataset and the effect of generative AI methods on the development of the tool are discussed. Random Forest and Naive Bayes algorithms from classification methods have been used to measure the impact and estimation. Several AI tools help solve real-life problems. Identifying what type of category is needed for AI tools and method selection are interlinked, and the research provides an overview of this connection.