Natural Vision Research Articles

B-spline curve approximation with transformer neural networks

Approximating a curve with a B-spline is a well-known problem with many challenges. Computing parametric values and knot vector that leads to the best approximation of a point sequence is an open problem. Existing methods are usually based on heuristics, genetic algorithms, or meta-heuristics. Nowadays, Deep Neural Networks have demonstrated their usefulness as shown in the use of a Multi-Layer Perceptron in the existing literature. Since its inception, the Transformer architecture has achieved state-of-the-art in multiple domains, like Natural Language Processing and Computer Vision. In this paper, we propose a method for knot placement that focuses on using a Transformer neural network architecture for B-spline approximation. We present and compare the results of our ongoing experimentations with Transformers for B-spline curve approximation. We conclude with possible improvements and modifications to our method for future experiments.

Advancements in Text-to-Video Creation through AI Models: A Comprehensive Review

This review paper explores the dynamic landscape of text-to-video creation facilitated by Artificial Intelligence (AI) models. Examining the intersection of Natural Language Processing (NLP) and Computer Vision (CV), we delve into methodologies, challenges, and advancements shaping this evolving field. From traditional rule-based systems to advanced deep learning architectures like GPT and CLIP, the paper navigates through the diverse spectrum of AI models driving text-to-video synthesis. Challenges, such as context preservation and ethical considerations, are discussed, along with practical applications spanning entertainment, education, and communication. A comparative analysis evaluates the strengths and limitations of different models, offering insights for optimal usage. Looking forward, the paper explores future directions, emphasizing collaborative efforts and ethical considerations. This comprehensive review provides a valuable resource for those engaged in the intersection of NLP, CV, and AI-driven text-to-video technologies. This review delves into AI's role in text-to-video conversion, exploring methods like computer vision and natural language processing. It examines advancements, challenges, and implications in diverse sectors such as marketing, education, and entertainment.

EndoViT: pretraining vision transformers on a large collection of endoscopic images

PurposeAutomated endoscopy video analysis is essential for assisting surgeons during medical procedures, but it faces challenges due to complex surgical scenes and limited annotated data. Large-scale pretraining has shown great success in natural language processing and computer vision communities in recent years. These approaches reduce the need for annotated data, which is of great interest in the medical domain. In this work, we investigate endoscopy domain-specific self-supervised pretraining on large collections of data.MethodsTo this end, we first collect Endo700k, the largest publicly available corpus of endoscopic images, extracted from nine public Minimally Invasive Surgery (MIS) datasets. Endo700k comprises more than 700,000 images. Next, we introduce EndoViT, an endoscopy-pretrained Vision Transformer (ViT), and evaluate it on a diverse set of surgical downstream tasks.ResultsOur findings indicate that domain-specific pretraining with EndoViT yields notable advantages in complex downstream tasks. In the case of action triplet recognition, our approach outperforms ImageNet pretraining. In semantic segmentation, we surpass the state-of-the-art (SOTA) performance. These results demonstrate the effectiveness of our domain-specific pretraining approach in addressing the challenges of automated endoscopy video analysis.ConclusionOur study contributes to the field of medical computer vision by showcasing the benefits of domain-specific large-scale self-supervised pretraining for vision transformers. We release both our code and pretrained models to facilitate further research in this direction: https://github.com/DominikBatic/EndoViT.

Using transformers for multimodal emotion recognition: Taxonomies and state of the art review

Emotion recognition is an aspect of human-computer interaction, affective computing, and social robotics. Conventional unimodal approaches for emotion recognition, depending on single data sources such as facial expressions or speech signals often fall short in capturing the complexity and context-dependent nature of emotions. Multimodal Emotion Recognition (MER), which integrates information from multiple modalities, has emerged as a promising solution to overcome these limitations. In recent years, Transformers-based approaches have gathered significant attention in the fields of natural language processing and computer vision, highlighting their ability to capture long-range dependencies and semantic representations. These models have rapidly achieved the MER state-of-the-art. However, current survey papers that cover MER lack a specific focus on Transformer-based techniques. To bridge this research gap, this review paper provides a comprehensive investigation of Transformers-based approaches for MER. It explores various Transformer architectures and proposes several scenarios for using Transformers at different stages of MER process. In addition, it examines datasets suitable for MER, discusses fusion mechanisms, and introduces novel taxonomies in both MER and Transformer technologies. The review also addresses challenges and future research directions. Through this review, we aim to provide researchers with an inclusive understanding of the current state-of-the-art in Transformers-based approaches for MER, paving the way for further advancements in this rapidly developing field.

Механизм преодоления злоупотребления правом в досудебном уголовном производстве

Abuse of law is a complex legal phenomenon, quite heterogeneous and diverse in its manifestations. There is no consensus in science about its legal nature and essence, but scholars rightly say that such an act must be prevented and suppressed. The absence in the Russian Criminal Procedure Act of a prohibition of abuse of rights by participants in criminal proceedings, and hence of means to prevent and suppress such abuse, makes it very difficult to counter it, which creates the risk of prolonging the length of time in a criminal case, reducing the quality of its investigation and unduly restricting the rights and freedoms of participants in criminal proceedings. Authors offer their own vision of the legal nature of abuse of rights in criminal proceedings, cite typical examples of it, build a mechanism for overcoming abuse of rights in pre-trial criminal proceedings, naming its necessary elements and content, indicating the reasons for its creation. It is concluded that a legal basis for the establishment of such a mechanism in criminal proceedings is necessary. It is summarised that the introduction into the Code of Criminal Procedure of a prohibition against the abuse of rights parties to criminal proceedings will make it possible to counteract its manifestations by means of general and private prevention and the suppression of such an act.

An Exaggeration? Reality?: Can ChatGPT Be Used in Neonatal Nursing?

Artificial intelligence (AI) represents a system endowed with the ability to derive meaningful inferences from a diverse array of datasets. Rooted in the advancements of machine learning models, AI has spawned various transformative technologies such as deep learning, natural language processing, computer vision, and robotics. This technological evolution is poised to witness a broadened spectrum of applications across diverse domains, with a particular focus on revolutionizing healthcare services. Noteworthy among these innovations is OpenAI's creation, ChatGPT, which stands out for its profound capabilities in intricate analysis, primarily facilitated through extensive language modeling. In the realm of healthcare, AI applications, including ChatGPT, have showcased promising outcomes, especially in the domain of neonatal nursing. Areas such as pain assessment, feeding processes, and patient status determination have witnessed substantial enhancements through the integration of AI technologies. However, it is crucial to approach the deployment of such applications with a judicious mindset. The accuracy of the underlying data must undergo rigorous validation, and any results lacking a solid foundation in scientific insights should be approached with skepticism. The paramount consideration remains patient safety, necessitating that AI applications, like ChatGPT, undergo thorough scrutiny through controlled and evidence-based studies. Only through such meticulous evaluation can the transformative potential of AI be harnessed responsibly, ensuring its alignment with the highest standards of healthcare practice.

Generating images using generative adversarial networks based on text descriptions

Modern developments in the fields of natural language processing (NLP) and computer vision (CV) emphasize the increasing importance of generating images from text descriptions. The presented article analyzes and compares two key methods in this area: generative adversarial network with conditional latent semantic analysis (GAN-CLS) and ultra-long transformer network (XLNet). The main components of GAN-CLS, including the generator, discriminator, and text encoder, are discussed in the context of their functional tasks—generating images from text inputs, assessing the realism of generated images, and converting text descriptions into latent spaces, respectively. A detailed comparative analysis of the performance of GAN-CLS and XLNet, the latter of which is widely used in the organic light-emitting diode (OEL) field, is carried out. The purpose of the study is to determine the effectiveness of each method in different scenarios and then provide valuable recommendations for selecting the best method for generating images from text descriptions, taking into account specific tasks and resources. Ultimately, our paper aims to be a valuable research resource by providing scientific guidance for NLP and CV experts.

Fake Social Media Profile Detection and Reporting Using Machine Learning

Our research focuses on utilizing machine learning techniques, encompassing natural language processing and computer vision, to create an automated system for the detection and reporting of fake social media profiles across various platforms. Our approach involves feature extraction from both textual and visual content, followed by the application of machine learning models to classify profiles as fake or genuine. This system operates in real-time, monitoring user activity and promptly flagging suspicious profiles for user- initiated reporting. By combining the power of machine learning with cross-platform compatibility and user feedback, our solution aims to enhance online safety by swiftly identifying and addressing fraudulent social media profiles, thus fostering more secure and trustworthy online communities.

Shelley’s Despairing Lyrics: Unfolding the Layers of Affliction and Resilient Forces

This article investigates how Percy Bysshe Shelley (1792-1822) embodies the diverse aspects of his internal crisis in despairing lyrics based on severe physical illness and tragic family occasions of 1817-8. It argues that the interlinking poems and prose written over these periods reveal the author’s agonizing response to the inevitable facets of the world and his life, which involves the recurrent motifs of mortality, ruination, and despondency. The paper furthers how he confronts this transitory necessity with an awareness of hope and love. The writer intertwines these positive reactions with the speaker’s paradoxical experiences of irony and dilemma when facing the ongoing affliction imposed by uncertainly and instability regarding the anticipation for a liberation from such afflictive moments. Shelley’s poems to depict despondency, however, embed the dynamic progress of the speaker’s mind, which associates with his resilient quest for spiritual freedom unyielding to the physical and internal trapment. This adherence of liberty prefigures the works of his final year 1822, especially A Defence of Poetry where he epitomizes the nature of imagination as a substantial power to bring ultimate freedom and mutual love to the gloom of the present life.

Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding

In recent years, vision language pre-training frameworks have made significant progress in natural language processing and computer vision, achieving remarkable performance improvement on various downstream tasks. However, when extended to point cloud data, existing works mainly focus on building task-specific models, and fail to extract universal 3D vision-language embedding that generalize well. We carefully investigate three common tasks in semantic 3D scene understanding, and derive key insights into the development of a pre-training model. Motivated by these observations, we propose a vision-language pre-training framework 3DVLP (3D vision-language pre-training with object contrastive learning), which transfers flexibly on 3D vision-language downstream tasks. 3DVLP takes visual grounding as the proxy task and introduces Object-level IoU-guided Detection (OID) loss to obtain high-quality proposals in the scene. Moreover, we design Object-level Cross-Contrastive alignment (OCC) task and Object-level Self-Contrastive learning (OSC) task to align the objects with descriptions and distinguish different objects in the scene, respectively. Extensive experiments verify the excellent performance of 3DVLP on three 3D vision-language tasks, reflecting its superiority in semantic 3D scene understanding. Code is available at https://github.com/iridescentttt/3DVLP.

Flood Insights: Integrating Remote and Social Sensing Data for Flood Exposure, Damage, and Urgent Needs Mapping

The absence of comprehensive situational awareness information poses a significant challenge for humanitarian organizations during their response efforts. We present Flood Insights, an end-to-end system that ingests data from multiple non-traditional data sources such as remote sensing, social sensing, and geospatial data. We employ state-of-the-art natural language processing and computer vision models to identify flood exposure, ground-level damage and flood reports, and most importantly, urgent needs of affected people. We deploy and test the system during a recent real-world catastrophe, the 2022 Pakistan floods, to surface critical situational and damage information at the district level. We validated the system's effectiveness through geographic regression analysis using official ground-truth data, showcasing its strong performance and explanatory power. Moreover, the system was commended by the United Nations Development Programme stationed in Pakistan, as well as local authorities, for pinpointing hard-hit districts and enhancing disaster response.

G^2SAM: Graph-Based Global Semantic Awareness Method for Multimodal Sarcasm Detection

Multimodal sarcasm detection, aiming to detect the ironic sentiment within multimodal social data, has gained substantial popularity in both the natural language processing and computer vision communities. Recently, graph-based studies by drawing sentimental relations to detect multimodal sarcasm have made notable advancements. However, they have neglected exploiting graph-based global semantic congruity from existing instances to facilitate the prediction, which ultimately hinders the model's performance. In this paper, we introduce a new inference paradigm that leverages global graph-based semantic awareness to handle this task. Firstly, we construct fine-grained multimodal graphs for each instance and integrate them into semantic space to draw graph-based relations. During inference, we leverage global semantic congruity to retrieve k-nearest neighbor instances in semantic space as references for voting on the final prediction. To enhance the semantic correlation of representation in semantic space, we also introduce label-aware graph contrastive learning to further improve the performance. Experimental results demonstrate that our model achieves state-of-the-art (SOTA) performance in multimodal sarcasm detection. The code will be available at https://github.com/upccpu/G2SAM.

Text-to-Image Generation for Abstract Concepts

Recent years have witnessed the substantial progress of large-scale models across various domains, such as natural language processing and computer vision, facilitating the expression of concrete concepts. Unlike concrete concepts that are usually directly associated with physical objects, expressing abstract concepts through natural language requires considerable effort since they are characterized by intricate semantics and connotations. An alternative approach is to leverage images to convey rich visual information as a supplement. Nevertheless, existing Text-to-Image (T2I) models are primarily trained on concrete physical objects and often struggle to visualize abstract concepts. Inspired by the three-layer artwork theory that identifies critical factors, intent, object and form during artistic creation, we propose a framework of Text-to-Image generation for Abstract Concepts (TIAC). The abstract concept is clarified into a clear intent with a detailed definition to avoid ambiguity. LLMs then transform it into semantic-related physical objects, and the concept-dependent form is retrieved from an LLM-extracted form pattern set. Information from these three aspects will be integrated to generate prompts for T2I models via LLM. Evaluation results from human assessments and our newly designed metric concept score demonstrate the effectiveness of our framework in creating images that can sufficiently express abstract concepts.

Focus-Then-Decide: Segmentation-Assisted Reinforcement Learning

Visual Reinforcement Learning (RL) is a promising approach to achieve human-like intelligence. However, it currently faces challenges in learning efficiently within noisy environments. In contrast, humans can quickly identify task-relevant objects in distraction-filled surroundings by applying previously acquired common knowledge. Recently, foundational models in natural language processing and computer vision have achieved remarkable successes, and the common knowledge within these models can significantly benefit downstream task training. Inspired by these achievements, we aim to incorporate common knowledge from foundational models into visual RL. We propose a novel Focus-Then-Decide (FTD) framework, allowing the agent to make decisions based solely on task-relevant objects. To achieve this, we introduce an attention mechanism to select task-relevant objects from the object set returned by a foundational segmentation model, and only use the task-relevant objects for the subsequent training of the decision module. Additionally, we specifically employed two generic self-supervised objectives to facilitate the rapid learning of this attention mechanism. Experimental results on challenging tasks based on DeepMind Control Suite and Franka Emika Robotics demonstrate that our method can quickly and accurately pinpoint objects of interest in noisy environments. Consequently, it achieves a significant performance improvement over current state-of-the-art algorithms. Project Page: https://www.lamda.nju.edu.cn/chenc/FTD.html Code: https://github.com/LAMDA-RL/FTD

Efficient coding of natural images in the mouse visual cortex

How the activity of neurons gives rise to natural vision remains a matter of intense investigation. The mid-level visual areas along the ventral stream are selective to a common class of natural images—textures—but a circuit-level understanding of this selectivity and its link to perception remains unclear. We addressed these questions in mice, first showing that they can perceptually discriminate between textures and statistically simpler spectrally matched stimuli, and between texture types. Then, at the neural level, we found that the secondary visual area (LM) exhibited a higher degree of selectivity for textures compared to the primary visual area (V1). Furthermore, textures were represented in distinct neural activity subspaces whose relative distances were found to correlate with the statistical similarity of the images and the mice’s ability to discriminate between them. Notably, these dependencies were more pronounced in LM, where the texture-related subspaces were smaller than in V1, resulting in superior stimulus decoding capabilities. Together, our results demonstrate texture vision in mice, finding a linking framework between stimulus statistics, neural representations, and perceptual sensitivity—a distinct hallmark of efficient coding computations.

Using Text Data, Neural Networks are Trained to Develop Emotions that Mimic Human Emotional Understanding

Recent methods of AI have revolutionized the field of computer science. Different sub-sectors of artificial intelligence (AI), like natural language processing (NLP) models, generative AI, computer vision, autonomous and recommendation systems, cybersecurity, quantum computing, etc., have helped automate human tasks, resulting in a tremendous amount of time and energy being saved. Despite the massive development of AI, all AI models lack one major factor, which is emotion. How can emotion be built into AI in order to for it to develop the emotional intelligence of the human brain to interpret and understand emotions so that it could create more human-friendly interactions? In this paper, we hypothesized developing emotions in neural networks as predictive sentiment analysis models using text data in order to replicate the emotional intelligence of the human brain to benefit human relationships. By using the Anaconda Repository, NVIDIA’s CUDA Toolkit, and TensorFlow, we were able to create a sentiment prediction model that achieved an accuracy of 94% and predicted the six basic emotions of joy, sadness, anger, fear, love, and surprise. Concluding this research, we observed that neural networks can develop the habit of recognizing emotions. This can be further fed into complex AI algorithms and systems to fine-tune emotional intelligence, resulting in more natural interactions, benefiting humans in

الخيال التحليلي: استكشاف الخيال في الكتابات القديمة والمعاصرة

The paper examines the concept of imagination from a creative perspective, focusing on a central question concerning the author's level of control over the imaginative nature of his work. Using a comparative textual analysis approach, the discussion looks at specific writings to find similarities and differences in order to clarify the complex relationship between literature and imagination. The paper is organized into an introduction, two analytical sub-sections titled "Imagination" and "Analytical Imagination," and a conclusion. The introduction included a brief literature review as well as the methodological approach, which guided the main research questions about the relationships between literature and imagination and how those connections affect the interpretation process. It concludes with a reflection on the evolution of the Romantic notion of the poetic imagination in modern times (as envisioned by Pound and Beckett) and establishes a link between Aristotle's Mimesis and the concept of analytical imagination.

Artificial intelligence in the diagnosis and treatment of acute appendicitis: a narrative review.

Artificial intelligence is transforming healthcare. Artificial intelligence can improve patient care by analyzing large amounts of data to help make more informed decisions regarding treatments and enhance medical research through analyzing and interpreting data from clinical trials and research projects to identify subtle but meaningful trends beyond ordinary perception. Artificial intelligence refers to the simulation of human intelligence in computers, where systems of artificial intelligence can perform tasks that require human-like intelligence like speech recognition, visual perception, pattern-recognition, decision-making, and language processing. Artificial intelligence has several subdivisions, including machine learning, natural language processing, computer vision, and robotics. By automating specific routine tasks, artificial intelligence can improve healthcare efficiency. By leveraging machine learning algorithms, the systems of artificial intelligence can offer new opportunities for enhancing both the efficiency and effectiveness of surgical procedures, particularly regarding training of minimally invasive surgery. As artificial intelligence continues to advance, it is likely to play an increasingly significant role in the field of surgical learning. Physicians have assisted to a spreading role of artificial intelligence in the last decade. This involved different medical specialties such as ophthalmology, cardiology, urology, but also abdominal surgery. In addition to improvements in diagnosis, ascertainment of efficacy of treatment and autonomous actions, artificial intelligence has the potential to improve surgeons' ability to better decide if acute surgery is indicated or not. The role of artificial intelligence in the emergency departments has also been investigated. We considered one of the most common condition the emergency surgeons have to face, acute appendicitis, to assess the state of the art of artificial intelligence in this frequent acute disease. The role of artificial intelligence in diagnosis and treatment of acute appendicitis will be discussed in this narrative review.

Sparsity in transformers: A systematic literature review

Transformers have become the state-of-the-art architectures for various tasks in Natural Language Processing (NLP) and Computer Vision (CV); however, their space and computational complexity present significant challenges for real-world applications. A promising approach to address these issues is the introduction of sparsity, which involves the deliberate removal of certain parameters or activations from the neural network. In this systematic literature review, we aimed to provide a comprehensive overview of current research on sparsity in transformers. We analyzed the different sparsity techniques applied to transformers, their impact on model performance, and their efficiency in terms of time and space complexity. Moreover, we identified the major gaps and challenges in the existing literature. Our study also highlighted the importance of investigating sparsity in transformers for computational efficiency, reduced resource requirements, scalability, environmental impact, and hardware-algorithm co-design. By synthesizing the current state of research on sparsity in transformer-based models, we also provided valuable insights into their efficiency, impact on model performance, and potential trade-offs, contributing to advancing the field further.

SecureTLM: Private inference for transformer-based large model with MPC

Transformer-based Large Models (TLM), such as generative pre-trained models (GPT), have become increasingly popular for practical applications through Deep Learning as a Service (DLaaS). They have been extensively used in natural language processing and computer vision. However, concerns regarding potential private data leakage arise with this type of inference service. While some private inference techniques can protect privacy, they often introduce high latency and approximate replacements in the design protocols, resulting in changes to the model structure and decreased accuracy. In this research, we present SecureTLM, a private inference method based on secure multi-party computation (MPC) that does not require modifications to the underlying model structure. SecureTLM offers protocols for crucial computations in TLM, such as Multiplication, Softmax, GeLU, and LayerNorm, without altering the model structure. Experimental results demonstrate that SecureTLM ensures data privacy, maintains correctness, and achieves efficiency in private inference tasks.