Transformer-based System Research Articles

Transforming Worlds: The Intersection of Translation Technology and Transformers in the Digital Age

This study provides a comparative analysis of the transformers’ role in translation technology with focus on the advancements of machine translation in digital age using Natural Language Processing (NLP). The study traces the technological advancements from rule-based to statistical methods of machine translation. The advancement is achieved through the integration of transformer models such as BERT, GPT and T5 in improving the efficiency of machine translation such as Neural Machine Translation (NMT). The BERT, GPT and T5 transformer models are characterised by their parallel processing and self-attention mechanisms, have significantly enhanced the accuracy and efficiency of translation, thus contributing to improved global communication. The study applied comparative and interpretative approach and theory of meaning. The study’s population focused on the translation technology and transformers. The study establishes the intersection of the two systems based on the facts and results of the systems from language engineers and translation experts. The discussion revolves around the challenges inherently present in transformer-based systems, including concerns over data efficiency, the handling of rare words, context sensitivity, bias, fairness, and the overarching societal impact of such technologies. The research highlights the development of innovative tools including wearable translation devices, smartphone applications, and emotion recognition systems that aid in surpassing language barriers and fostering international collaboration and understanding. It delves into the societal ramifications of these technologies, advocating for preservation of cultural nuances and the promotion of intercultural dialogue while highlighting ethical considerations such as privacy, security, and misinformation, and their role in shaping the deployment of translation technology. The study concludes that there is a synergy between transformers and translation technology in the digital age. The study traced an evolution from rule-based machine translation to the sophisticated AI powered translation that are revolutionized using transformer models. Through analysis and comparison, it is clear that transformer models beside enhancing accuracy and efficiency of translation, can also process and interpret natural language.

Tomato maturity stage prediction based on vision transformer and deep convolution neural networks

Automated assessment of tomato crop maturity is vital for improving agricultural productivity and reducing food waste. Traditionally, farmers have relied on visual inspection and manual assessment to predict tomato maturity, which is prone to human error and time-consuming. Computer vision and deep learning automate this process by analysing visual characteristics, enabling data-driven harvest decisions, optimising quality, and reducing waste for sustainable and efficient agriculture. This research demonstrates deep learning models accurately classifying tomato maturity stages using computer vision techniques, utilising a novel dataset of 4,353 tomato images. The Vision Transformer (ViT) model exhibited superior performance in classifying tomatoes into three ripeness categories (immature, mature, and partially mature), achieving a remarkable testing accuracy of 98.67% and the Convolution neural network (CNN) models, including EfficientNetB1, EfficientNetB5, EfficientNetB7, InceptionV3, ResNet50, and VGG16, achieved testing accuracies of 88.52%, 89.84%, 91.16%, 90.94%, 93.15%, and 92.27%, respectively, when tested with unseen data. ViT significantly surpassed the performance of CNN models. This research highlights the potential for deploying ViT in agricultural environments to monitor tomato maturity stages and packaging facilities smartly. Transformer-based systems could substantially reduce food waste and improve producer profits and productivity by optimising fruit harvest time and sorting decisions.

FindVehicle and VehicleFinder: a NER dataset for natural language-based vehicle retrieval and a keyword-based cross-modal vehicle retrieval system

Natural language (NL) based vehicle retrieval is a task aiming to retrieve a vehicle that is most consistent with a given NL query from among all candidate vehicles. Because NL query can be easily obtained, such a task has a promising prospect in building an interactive intelligent traffic system (ITS). Current solutions mainly focus on extracting both text and image features and mapping them to the same latent space to compare the similarity. However, existing methods usually use dependency analysis or semantic role-labelling techniques to find keywords related to vehicle attributes. These techniques may require a lot of pre-processing and post-processing work, and also suffer from extracting the wrong keyword when the NL query is complex. To tackle these problems and simplify, we borrow the idea from named entity recognition (NER) and construct FindVehicle, a NER dataset in the traffic domain. It has 42.3k labelled NL descriptions of vehicle tracks, containing information such as the location, orientation, type and colour of the vehicle. FindVehicle also adopts both overlapping entities and fine-grained entities to meet further requirements. To verify its effectiveness, we propose a baseline NL-based vehicle retrieval model called VehicleFinder. Our experiment shows that by using text encoders pre-trained by FindVehicle, VehicleFinder achieves 87.7% precision and 89.4% recall when retrieving a target vehicle by text command on our homemade dataset based on UA-DETRAC [1]. From loading the command into VehicleFinder to identifying whether the target vehicle is consistent with the command, the time cost is 279.35 ms on one ARM v8.2 CPU and 93.72 ms on one RTX A4000 GPU, which is much faster than the Transformer-based system. The dataset is open-source via the link https://github.com/GuanRunwei/FindVehicle, and the implementation can be found via the link https://github.com/GuanRunwei/VehicleFinder-CTIM.

Large-scale application of named entity recognition to biomedicine and epidemiology.

Despite significant advancements in biomedical named entity recognition methods, the clinical application of these systems continues to face many challenges: (1) most of the methods are trained on a limited set of clinical entities; (2) these methods are heavily reliant on a large amount of data for both pre-training and prediction, making their use in production impractical; (3) they do not consider non-clinical entities, which are also related to patient's health, such as social, economic or demographic factors. In this paper, we develop Bio-Epidemiology-NER (https://pypi.org/project/Bio-Epidemiology-NER/) an open-source Python package for detecting biomedical named entities from the text. This approach is based on a Transformer-based system and trained on a dataset that is annotated with many named entities (medical, clinical, biomedical, and epidemiological). This approach improves on previous efforts in three ways: (1) it recognizes many clinical entity types, such as medical risk factors, vital signs, drugs, and biological functions; (2) it is easily configurable, reusable, and can scale up for training and inference; (3) it also considers non-clinical factors (age and gender, race and social history and so) that influence health outcomes. At a high level, it consists of the phases: pre-processing, data parsing, named entity recognition, and named entity enhancement. Experimental results show that our pipeline outperforms other methods on three benchmark datasets with macro-and micro average F1 scores around 90 percent and above. This package is made publicly available for researchers, doctors, clinicians, and anyone to extract biomedical named entities from unstructured biomedical texts.

Human Evaluation of English–Irish Transformer-Based NMT

In this study, a human evaluation is carried out on how hyperparameter settings impact the quality of Transformer-based Neural Machine Translation (NMT) for the low-resourced English–Irish pair. SentencePiece models using both Byte Pair Encoding (BPE) and unigram approaches were appraised. Variations in model architectures included modifying the number of layers, evaluating the optimal number of heads for attention and testing various regularisation techniques. The greatest performance improvement was recorded for a Transformer-optimized model with a 16k BPE subword model. Compared with a baseline Recurrent Neural Network (RNN) model, a Transformer-optimized model demonstrated a BLEU score improvement of 7.8 points. When benchmarked against Google Translate, our translation engines demonstrated significant improvements. Furthermore, a quantitative fine-grained manual evaluation was conducted which compared the performance of machine translation systems. Using the Multidimensional Quality Metrics (MQM) error taxonomy, a human evaluation of the error types generated by an RNN-based system and a Transformer-based system was explored. Our findings show the best-performing Transformer system significantly reduces both accuracy and fluency errors when compared with an RNN-based model.

Solid-state technology: enhancing power system equipment

Some ideas on an evolutionary approach to enhancing functional characteristics of transmission equipment, such as transformers, through the adoption of solid-state switching are explored in this article with the intent of stimulating discussions among specialists in solid-state technology and transformer design and experts in power system planning and operation. This article proposes the use of solid state switching to give transformers the additional capability of var control similar to that in static Var compensators. Incorporating this functionality into transformer-based systems has the potential of adding a very useful system-wide voltage control capability, at a relatively small cost considering mass production and modularity.